Coarse-Fine-Stitched: A Robust Maritime Horizon Line Detection Method for Unmanned Surface Vehicle Applications

The horizon line has numerous applications for an unmanned surface vehicles (USV), such as autonomous navigation, attitude estimation, obstacle detection and target tracking. However, maritime horizon line detection is quite a challenging problem. The pixel points of the horizon line features are far fewer than the pixel points of the entire image, on the one hand. Conversely, the detection results might be impacted negatively by the complex maritime environment, waves, light changing, and partial occlusions due to maritime vessels or islands, for example. To solve these problems, a robust horizon line detection method named coarse-fine-stitched (CFS) is proposed in this paper. First, in the coarse step of CFS, a line segment detection approach using gradient features is applied to build a line candidate pool, which probably contains many false detection results. Then, hybrid feature filtering is designed to pick the horizon line segments from the pool in the fine step. Finally, the fine line segments are stitched to obtain the whole horizon line based on random sample consensus (RANSAC). Using real data in the maritime environment, the experimental results demonstrate the effectiveness of CFS, compared to the existing methods in terms of accuracy and robustness.


Introduction
Currently, with the rapid development of artificial intelligence technology, unmanned surface vehicles (USVs) have become more and more important in maritime systems [1][2][3]. Compared with manned surface vehicles, USVs are more easily manipulated and are more adaptable to the environment with lower risk to staff. USVs have been applied in various military and civilian missions, such as science, mapping, defense and general robotics research [4][5][6]. The vision sensors equipped on the USVs usually are used to perceive surrounding information, which plays an important role in guaranteeing the safety and efficiency of the USVs without human intervention.
The horizon line is an important information source for USVs. When considering real applications, the maritime horizon line can be utilized for autonomous navigation, attitude estimation, obstacle detection, and target tracking, for example [7,8]. However, maritime horizon line detection is quite challenging due to the following two points. First, the pixel points of the horizon line features are far fewer than the pixel points of the entire image; and second, horizon line detection suffers from the complex maritime environment, waves, light changing and partial occlusions by maritime vessels or islands, for example. Sometimes, the horizon line in maritime application is blurry since the image of the sky and the sea is similar. of the fine step, the distinguished features of the horizon line segments are modeled to build hybrid feature filtering, including color and morphology. Finally, a line segment stitching method based on random sample consensus (RANSAC) is proposed to obtain the whole horizon line. Considering the linear relationship of the horizon line segments, the RANSAC method is used to stitch the true results and remove the negative line segments, which can improve the robustness of this approach in a complex maritime environment.
The remainder of this paper is organized as follows. The details of the proposed CFS method are presented in Section 2. Section 3 discusses the experiments used to demonstrate the practical utility of this approach. Finally, the conclusions are shown in Section 4.

Framework
The framework of the proposed coarse-fine-stitched (CFS) method for unmanned surface vehicle (USV) applications is presented in Figure 1. It operates in a three-step way. During the coarse step, the image gradient features are exploited to obtain all the lines of the image, which are all candidates for horizon line segments. In this step, a problem-specific design line segment detection (LSD) algorithm is proposed to extract all the candidates with a short processing time, generating a line pool. Although the coarse detection method can obtain the horizon line candidates of the image, some non-horizon line segments could be selected into the pool. Therefore, in the fine second step of this method, a horizon line filter is designed to eliminate the non-horizon lines from the pool by using hybrid feature filtering, which was designed based on the morphological and color features of the horizon line. Finally, to achieve the accurate identification of the horizon line, the horizon line segments are stitched in this step. Specifically, a random sample consensus-based (RANSAC-based) line segment stitching method is proposed to obtain the whole horizon line, which can improve the robustness of this approach in a complex maritime environment. are modeled to build hybrid feature filtering, including color and morphology. Finally, a line segment stitching method based on random sample consensus (RANSAC) is proposed to obtain the whole horizon line. Considering the linear relationship of the horizon line segments, the RANSAC method is used to stitch the true results and remove the negative line segments, which can improve the robustness of this approach in a complex maritime environment. The remainder of this paper is organized as follows. The details of the proposed CFS method are presented in Section 2. Section 3 discusses the experiments used to demonstrate the practical utility of this approach. Finally, the conclusions are shown in Section 4.

Framework
The framework of the proposed coarse-fine-stitched (CFS) method for unmanned surface vehicle (USV) applications is presented in Figure 1. It operates in a three-step way. During the coarse step, the image gradient features are exploited to obtain all the lines of the image, which are all candidates for horizon line segments. In this step, a problem-specific design line segment detection (LSD) algorithm is proposed to extract all the candidates with a short processing time, generating a line pool. Although the coarse detection method can obtain the horizon line candidates of the image, some non-horizon line segments could be selected into the pool. Therefore, in the fine second step of this method, a horizon line filter is designed to eliminate the non-horizon lines from the pool by using hybrid feature filtering, which was designed based on the morphological and color features of the horizon line. Finally, to achieve the accurate identification of the horizon line, the horizon line segments are stitched in this step. Specifically, a random sample consensus-based (RANSAC-based) line segment stitching method is proposed to obtain the whole horizon line, which can improve the robustness of this approach in a complex maritime environment. The framework of the coarse-fine-stitched (CFS) method for unmanned surface vehicle (USV) applications. The first step of the method is a problem-specific design line segment detector, the second step is a horizon line filter, and the third step is a random sample consensus-based (RANSAC-based) line segment stitching method.

Coarse Detection
Line features are one of the most important features that determine the effectiveness of the coarse-fine-stitched (CFS) method. von Gioi et al. presented a line segment detection (LSD) algorithm which gave accurate results with fast computational speed [21]. The LSD algorithm extracts line segments from an image in three steps: (1) Line-support region: Group connected pixels with gradient values exceeding a threshold into line-support regions by applying a region-growing algorithm; (2) Rectangle approximation: Find the rectangle that best approximates each line-support region; (3) Line applications. The first step of the method is a problem-specific design line segment detector, the second step is a horizon line filter, and the third step is a random sample consensus-based (RANSAC-based) line segment stitching method.

Coarse Detection
Line features are one of the most important features that determine the effectiveness of the coarse-fine-stitched (CFS) method. von Gioi et al. presented a line segment detection (LSD) algorithm which gave accurate results with fast computational speed [21]. The LSD algorithm extracts line segments from an image in three steps: (1) Line-support region: Group connected pixels with gradient values exceeding a threshold into line-support regions by applying a region-growing algorithm; (2) Rectangle approximation: Find the rectangle that best approximates each line-support region; (3) Line segment validation: Validate each rectangle based on a contrary method, which counts the number of aligned points (points with a gradient direction approximately orthogonal to the line segment) and finds the line segments as outliers.
However, the direct application of the LSD algorithm is not appropriate for the horizon line detection problem for the following two reasons.
(1) LSD cannot distinguish the horizon line from other line segments because it just focuses on detecting line segments [22][23][24]. A line that is not the horizon line also will be obtained. Figure 2b gives the result of using LSD for a given image, lines of the island, the clouds and the waves in the image are also extracted, causing a false positive. Therefore, to accurately extract the horizon line from a given image, a filter method should be considered to eliminate non-horizon line segments from all line segments. However, the direct application of the LSD algorithm is not appropriate for the horizon line detection problem for the following two reasons.
(1) LSD cannot distinguish the horizon line from other line segments because it just focuses on detecting line segments [22][23][24]. A line that is not the horizon line also will be obtained. Figure 2b gives the result of using LSD for a given image, lines of the island, the clouds and the waves in the image are also extracted, causing a false positive. Therefore, to accurately extract the horizon line from a given image, a filter method should be considered to eliminate non-horizon line segments from all line segments. (2) LSD ignores the horizon line that is background-like, which might cause a significant false negative. Figure 3b shows the horizon line is totally ignored. During the first step of the LSD algorithm, to improve the exactness of line segment detection, a gradient threshold is selected to eliminate pixels with a small gradient magnitude because the orientation error cannot be ignored for a small gradient magnitude [25], which can create a linear pattern and thereby cause false detections. The threshold on the gradient magnitude in the LSD algorithm is set as [21]: Pixels with a gradient magnitude smaller than the threshold will not be regarded as part of any line segment. However, in this horizon line detection task, the horizon line obtained by the USV might be very background-like, due to vapor or illumination, for example. Their corresponding pixels with gradient magnitude might be under the threshold often. This leads to the horizon line being ignored and causes high false negatives. Figure 3b shows the horizon line is ignored totally in this step, since the gradient magnitude of the pixels do not exceed  (2) LSD ignores the horizon line that is background-like, which might cause a significant false negative. Figure 3b shows the horizon line is totally ignored. During the first step of the LSD algorithm, to improve the exactness of line segment detection, a gradient threshold is selected to eliminate pixels with a small gradient magnitude because the orientation error cannot be ignored for a small gradient magnitude [25], which can create a linear pattern and thereby cause false detections. The threshold on the gradient magnitude in the LSD algorithm is set as [21]: Pixels with a gradient magnitude smaller than the threshold will not be regarded as part of any line segment. However, in this horizon line detection task, the horizon line obtained by the USV might be very background-like, due to vapor or illumination, for example. Their corresponding pixels with gradient magnitude might be under the threshold often. This leads to the horizon line being ignored and causes high false negatives. Figure 3b shows the horizon line is ignored totally in this step, since the gradient magnitude of the pixels do not exceed T g .
Considering that false positive can be processed further in the next step, a smaller threshold is set to avoid ignoring pixels with background-like horizon lines in this step. Demonstrated in the analysis above and inspired by the LSD algorithm, the LSD framework is applied and then the current method focuses on how to redesign the gradient threshold in the first step of the algorithm. A problem-specific LSD design then is proposed to obtain all the lines of a given image.
Regarding the gradient threshold selection problem, an intensity-based method is applied to set the threshold [26]: where µ mean is the mean gradient value of the image, and µ max is the maximum of the mean gradient value of the grid images. The authors divided the image into an 8 × 8 grid. It represents an adaptive threshold that depends on the gradient value of the image. It is evident that T g ≤ 1 ≤ T g , thus, with the smaller threshold, pixels of the background-like horizon line segments will not be ignored in the first step of this method, as shown in Figure 3c. Considering that false positive can be processed further in the next step, a smaller threshold is set to avoid ignoring pixels with background-like horizon lines in this step. Demonstrated in the analysis above and inspired by the LSD algorithm, the LSD framework is applied and then the current method focuses on how to redesign the gradient threshold in the first step of the algorithm. A problem-specific LSD design then is proposed to obtain all the lines of a given image.
Regarding the gradient threshold selection problem, an intensity-based method is applied to set the threshold [26]: where mean  is the mean gradient value of the image, and max  is the maximum of the mean gradient value of the grid images. The authors divided the image into an 8 × 8 grid. It represents an adaptive threshold that depends on the gradient value of the image. It is evident that thus, with the smaller threshold, pixels of the background-like horizon line segments will not be ignored in the first step of this method, as shown in Figure 3c. The problem-specific LSD design is able to extract all the lines of a given image and generate a line pool, Ch, which is composed of all candidates for horizon line segments, denoted as l1, l2, …, lNc, where Nc is the total number of line segments obtained by the coarse detection method, resulting in

Fine Detection
Subsequent to the process of the first step, a line pool Ch is obtained. However, some elements of Ch might be background noise causing a sufficient false positive. To avoid the interference of the non-horizon line, a horizon line filter is applied to suppress the noise and extract horizon line segments from the line pool Ch. The non-horizon line is filtered out from the pool according to the distinguished features of the horizon line segments, including color and morphology.

Morphology Feature
The morphology feature of the horizon line includes the length and direction properties, as well as others [7]. However, in some environments, the horizon line detection might be obscured partially by maritime vessels or islands, as shown in Figure 2a. Thus, the length property would not perform well for horizon line detection. A new direction property is used for fine detection in this paper.
Denote the equation of the line horizon segment li as: where ki and bi are the gradient and the intercept of the line horizon segment li, respectively.
Denoting the direction of the horizon line segment li as di: The problem-specific LSD design is able to extract all the lines of a given image and generate a line pool, C h , which is composed of all candidates for horizon line segments, denoted as l 1 , l 2 , . . . , l Nc , where N c is the total number of line segments obtained by the coarse detection method, resulting in

Fine Detection
Subsequent to the process of the first step, a line pool C h is obtained. However, some elements of C h might be background noise causing a sufficient false positive. To avoid the interference of the non-horizon line, a horizon line filter is applied to suppress the noise and extract horizon line segments from the line pool C h . The non-horizon line is filtered out from the pool according to the distinguished features of the horizon line segments, including color and morphology.

Morphology Feature
The morphology feature of the horizon line includes the length and direction properties, as well as others [7]. However, in some environments, the horizon line detection might be obscured partially by maritime vessels or islands, as shown in Figure 2a. Thus, the length property would not perform well for horizon line detection. A new direction property is used for fine detection in this paper.
Denote the equation of the line horizon segment l i as: where k i and b i are the gradient and the intercept of the line horizon segment l i , respectively. Denoting the direction of the horizon line segment l i as d i : Since the camera system is usually fixed on the unmanned surface vehicle (USV), the direction of the horizon line obtained by the camera system is approximately equal to the roll angle of the USV β. When used in real applications, the angle error between the roll angle of the USV and the direction of the horizon line is the alignment error, which can be measured earlier, denoted as e. Subsequently, the result is: where δ is a very small angle value, which is caused by other factors, such as structural deformation. Denote the maximum value of the roll angle as β max : β max is set as 30 • , and the USV does not risk capsizing.

Color Feature
The color feature is another essential feature to distinguish the horizon line segments from other line segments. Since the horizon line is the apparent edge that separates sea from sky, the greater the color feature difference of the sea and the sky, the easier it is to distinguish the horizon line segments and other line segments. Figure 4 shows the image examples containing the color feature of the horizon line segments. The color feature between the two sides of the horizon line segments is distinguished.
where  is a very small angle value, which is caused by other factors, such as structural deformation.
Denote the maximum value of the roll angle as βmax: βmax is set as 30°, and the USV does not risk capsizing.

Color Feature
The color feature is another essential feature to distinguish the horizon line segments from other line segments. Since the horizon line is the apparent edge that separates sea from sky, the greater the color feature difference of the sea and the sky, the easier it is to distinguish the horizon line segments and other line segments. Figure 4 shows the image examples containing the color feature of the horizon line segments. The color feature between the two sides of the horizon line segments is distinguished.
First, there is a notable color difference between the two sides of the horizon line segments. The greater the color feature variance of the two sides of a horizon line segment, the easier it is to distinguish the horizon line segments and other line segments. Second, the color of each side of the horizon line segment is composed of several categories, for example, the color of the up-side of a horizon line segment is probably blue, white, gray, red, yellow, and more. The color of the down-side of the horizon line segment is probably blue, gray, etc. Third, on the same side of the horizon line segment, the color and texture are roughly the same and, the closer to the horizon line segment, the more important it is to distinguish the result.  Given an image with a horizon line, the line segment pool is obtained through the coarse detection step. The sub-image of line segment li is extracted for fine detection, denoted as Ri. The sub-image with a line segment detected by the coarse detection step is shown in Figure 5. The up-side region and down-side region of line segment li is denoted as Ri,1 and Ri,2, respectively, with the sub-image of li, the color feature is calculated to identify whether it is the horizon line segment. First, there is a notable color difference between the two sides of the horizon line segments. The greater the color feature variance of the two sides of a horizon line segment, the easier it is to distinguish the horizon line segments and other line segments. Second, the color of each side of the horizon line segment is composed of several categories, for example, the color of the up-side of a horizon line segment is probably blue, white, gray, red, yellow, and more. The color of the down-side of the horizon line segment is probably blue, gray, etc. Third, on the same side of the horizon line segment, the color and texture are roughly the same and, the closer to the horizon line segment, the more important it is to distinguish the result.
Given an image with a horizon line, the line segment pool is obtained through the coarse detection step. The sub-image of line segment l i is extracted for fine detection, denoted as R i . The sub-image with a line segment detected by the coarse detection step is shown in Figure 5. The up-side region and down-side region of line segment l i is denoted as R i,1 and R i,2 , respectively, with the sub-image of l i , the color feature is calculated to identify whether it is the horizon line segment.
(c) Given an image with a horizon line, the line segment pool is obtained through the coarse detection step. The sub-image of line segment li is extracted for fine detection, denoted as Ri. The sub-image with a line segment detected by the coarse detection step is shown in Figure 5. The up-side region and down-side region of line segment li is denoted as Ri, 1 and Ri,2, respectively, with the sub-image of li, the color feature is calculated to identify whether it is the horizon line segment.

Figure 5.
Given an image with a horizon line, the line segment pool is obtained through the coarse detection step. The sub-image of line segment then is extracted for fine detection, whose color feature is calculated to identify whether it is the horizon line segment.
Concerning the current method, the first three color moments of the sub-image of li are used for fine detection, which characterize the color distribution of sub-images. It is scaling and rotation invariant. Moreover, since color moments encode both shape and color information, they are a good feature to use under changing lighting conditions [27]. The first three color moments of sub-image Ri are obtained as follows:

Mean
The first color moment of region Ri,1 and region Ri,2 can be calculated as the average color in the image region: Figure 5. Given an image with a horizon line, the line segment pool is obtained through the coarse detection step. The sub-image of line segment then is extracted for fine detection, whose color feature is calculated to identify whether it is the horizon line segment.
Concerning the current method, the first three color moments of the sub-image of l i are used for fine detection, which characterize the color distribution of sub-images. It is scaling and rotation invariant. Moreover, since color moments encode both shape and color information, they are a good feature to use under changing lighting conditions [27]. The first three color moments of sub-image R i are obtained as follows:

Mean
The first color moment of region R i,1 and region R i,2 can be calculated as the average color in the image region: where N i,1 and N i,2 are the number of pixels of region R i,1 and region R i,2 , respectively. p i k,1,j and p i k,2,j are the value of the jth pixel of region R i,1 and region R i,2 at the kth color channel, respectively. The color image is processed and k = 1, 2, 3, corresponding to the three color channels of the image.

Standard Deviation
The second color moment is the standard deviation. The second color moment of region R i,1 and region R i,2 can be obtained by taking the square root of the variance of the color distribution:

Skewness
The third color moment is the skewness. It measures how asymmetric the color distribution is and, thus, it gives information about the shape of the color distribution. Skewness of region R i,1 and region R i,2 can be computed with the following formula: Using Equation (7) to Equation (12), the color feature vector of region R i,1 and region R i,2 can be obtained as the nine-dimension vectors: Using the color feature vectors, the color feature difference between region R i,1 and region R i,2 can be calculated as: where w µ , w σ and w s are the weights for each of the three color moments used. The color feature difference between region R i,1 and region R i,2 will be used to compute a difference score. Considering that the horizon line is the apparent edge that separates sea from sky, the value of d(c i,1 , c i,2 ) should be larger than threshold T d,1 : Since the color of each side of the horizon line segment is composed of several categories, the color feature difference between region R i,1 (or region R i,2 ) and the training data is calculated, which should be smaller than threshold T d, 2 : where m = 1, 2, corresponding to the up-side region R i,1 and down-side region R i,2 , respectively. The training data is composed of 1000 images including the horizon line with different illumination, rain, fog, and occlusion, for example. The authors have T d,1 = 1.5, and T d,2 = 1 used in these experiments.
To improve the effectiveness of the coarse-fine-stitched (CFS) method, the up-side region R i,1 and down-side region R i,2 are divided into a series of sub-regions, as shown in Figure 6.
Similar to the color moments of region R i,1 and region R i,2 , the color feature vector of each sub-region can be calculated as: where c i,1,p and c i,2,p are pth sub-regions of region R i,1 and region R i,2 , respectively. Denote the color feature difference of two sub-regions on the same side as d(c i,m,p , c i,m,q ). As on the same side of the horizon line segment the colors are roughly the same, then the color feature differences of the two sub-regions on the same side should satisfy: where var d(c i,m,p , c i,m,q ) is the variance of d(c i,m,p , c i,m,q ) on the same side of the horizon line segment. The threshold T d,3 is set as 1.
Corresponding to the discussion in this section, hybrid feature filtering is designed and the line segments of the line pool that satisfy Equation (6), Equation (16), Equation (17) and Equation (20) are picked as the horizon line segments for the next step of CFS. The color feature difference between region Ri,1 and region Ri,2 will be used to compute a difference score. Considering that the horizon line is the apparent edge that separates sea from sky, the value of i i d c c should be larger than threshold Td,1: (16) Since the color of each side of the horizon line segment is composed of several categories, the color feature difference between region Ri,1 (or region Ri,2) and the training data is calculated, which should be smaller than threshold Td,2: i m i m t d d T  c c (17) where m = 1, 2, corresponding to the up-side region Ri,1 and down-side region Ri,2, respectively.
The training data is composed of 1000 images including the horizon line with different illumination, rain, fog, and occlusion, for example. The authors have Td,1 = 1.5, and Td,2 = 1 used in these experiments.
To improve the effectiveness of the coarse-fine-stitched (CFS) method, the up-side region Ri,1 and down-side region Ri,2 are divided into a series of sub-regions, as shown in Figure 6. Similar to the color moments of region Ri,1 and region Ri,2, the color feature vector of each sub-region can be calculated as:  (18) (19) where ,1, The threshold Td,3 is set as 1.
Corresponding to the discussion in this section, hybrid feature filtering is designed and the line segments of the line pool that satisfy Equation (6), Equation (16), Equation (17) and Equation (20) are picked as the horizon line segments for the next step of CFS. Figure 6. The up-side region R i,1 and down-side region R i,2 are divided into a series of sub-regions.

Robust Stitching
Through the fine detection step of the coarse-fine-stitched (CFS) method, line segments are picked from the line pool as the horizon line segments, denoted as l 1 , l 2 , . . . , l N f , where N f is the total number of line segments obtained by the fine detection method. Since these candidate horizon line segments have already passed the hybrid feature filtering, most of them have the typical characteristics of the horizon line. However, there are still two problems that need to be resolved to obtain the horizon line result for unmanned surface vehicle (USV) applications.
First, some parts of the horizon line might be missing due to the occlusion of islands and vessels, illumination, or fog, for example. Using this scenario, the horizon line is divided into several pieces of horizon line segments, rather than a complete line.
Second, although most of the line segments are the horizon line segments through fine detection, there inevitably might be some non-horizon line segments, which will have a negative impact on the final result.
To address these challenges, a random sample consensus-based (RANSAC-based) line segment stitching method is proposed to obtain the whole horizon line. Found in real applications, the horizon line is a straight line. Even if it is divided into several pieces, there is still a linear relationship between them. Therefore, a natural idea is that the horizon line can be obtained through straight line fitting, which is used to estimate the best line passing through the candidate line segments after fine line detection. Considering the fine step, it is difficult to completely exclude a false positive, therefore a RANSAC algorithm is applied in this paper for robust line fitting. The advantage of RANSAC is the ability to use redundant consistency to fit the correct results and to reduce the effect of a false positive on the final results [28].
Denote the equation of the horizon line L as: where k and b are the gradient and the intercept of the line horizon L, respectively the parameters k and b are to be determined as follows.
The stitching method is used to solve the two parameters in Equation (21) by minimizing the total mean distance between the line segment l 1 , l 2 , . . . , l N f and the horizon line results: where d i is the distance between the midpoint [x i , y i ] T of line segment l i and the horizon line results: and l i = (x i1 − x i2 ) 2 + (y i1 − y i2 ) 2 is the weight of the distance, which means the result is intended to choose the long-line segment candidates as the final results. This illustrated that the optimal function is a continuous convex function, which can be solved easily by the gradient descent method [29]. Using all line segment candidates, the horizon line detection result can be solved, denoted as L 0 .
Although most of the line segments are horizon line segments through fine detection, there inevitably might be some non-horizon line segments, which will have a negative impact on the final result. To solve this problem, RANSAC is applied for robust stitching. Batches of line segment candidates are randomly selected to solve the equation of the horizon line, denoted as L j , which is the jth iteration of RANSAC.
The principle of the RANSAC-based line segment stitching method is shown in Figure 7. L 0 is the line fitting result using all line segment candidates and L j is the RANSAC-based line fitting. The distance between the midpoint of the line segment candidate l j and L j is larger than the threshold, which is marked as an outlier and excluded for stitching.
Looking at the batch, each distance between the midpoint of the line segment candidate and L j is calculated. When the distance is smaller than a threshold, the line segment is marked as an in-line candidate; otherwise, the line segment is marked as an outlier candidate. The batches of line segment candidates are randomly selected to obtain a batch with the largest number of inline candidates.
The main procedures of the CFS method are summarized as follows: Step  (6), (16), (17) and (20) Step 3. Robust stitching 3.1 Execute RANSAC and solve Equation (23) result. To solve this problem, RANSAC is applied for robust stitching. Batches of line segment candidates are randomly selected to solve the equation of the horizon line, denoted as Lj, which is the jth iteration of RANSAC. The principle of the RANSAC-based line segment stitching method is shown in Figure 7. L0 is the line fitting result using all line segment candidates and Lj is the RANSAC-based line fitting. The distance between the midpoint of the line segment candidate j l  and Lj is larger than the threshold, which is marked as an outlier and excluded for stitching.
  11 11 , T x y   12 12 , Looking at the batch, each distance between the midpoint of the line segment candidate and Lj is calculated. When the distance is smaller than a threshold, the line segment is marked as an in-line candidate; otherwise, the line segment is marked as an outlier candidate. The batches of line segment candidates are randomly selected to obtain a batch with the largest number of inline candidates.

Results and Discussion
Three separate experiments were designed to evaluate the practical utility of the coarse-fine-stitched (CSF) method. The first experiment was to evaluate the performance of the CFS method for images with different backgrounds. The second experiment was the parameter

Results and Discussion
Three separate experiments were designed to evaluate the practical utility of the coarse-fine-stitched (CSF) method. The first experiment was to evaluate the performance of the CFS method for images with different backgrounds. The second experiment was the parameter sensitivity analysis. The third experiment was to test the performance of the method compared with existing methods in terms of accuracy and time-consumption.
The experiments were conducted using a PC with Intel Core i3-6006U CPU 2.00 GHz and 4 GB memory. The testing data was composed of 500 testing images, the public dataset Singapore Maritime Dataset (SMD) [13], and the public dataset Marine Obstacle Detection Dataset (MODD) [18]. The 500 testing images were acquired using a digital camera (Sony-DSC-T70) in Weihai City, China, in July, 2017. The resolution was 640 × 800 pixels, with different backgrounds. Half of the images contained mist or fog and the other half contained relatively clear horizon lines. Additionally, to test the performance of this method, the public dataset SMD and MODD were used for comparison. The training images were composed of more than 1000 images with the horizon line downloaded from http://image.baidu.com/, with different illumination, rain, fog, and occlusion, etc.

Coarse-Fine-Stitched Detection Results with Different Backgrounds
High visual complexity made it difficult to detect the horizon line that was not salient. The authors divided the testing image based on its clutter measure into K sub-images [30]. The measure was based on the mean and standard deviation (µ, σ) of the intensity distribution of the pixels: where I was a given image, (x, y) indicated the pixel index. N was the total number of pixels in sub-image S n .
The clutter measure of an image was given by the average of the intensity variance of the sub-images: The authors divided the image into 4 × 4 sub-images with reference to Candamo's work [31]. The mean clutter of the database was denoted as µ. When the clutter of an image was less than µ, it was considered as low clutter. To test the validation of this approach, the authors used the public datasets Singapore Maritime Dataset (SMD) and Marine Obstacle Detection Dataset (MODD) for benchmarking. The clutter of these images (or frames in video) varied from 1 to 45, and µ = 14. The processing time for an image was a fraction of a second, which permitted the authors to test the method on thousands of images.
Coarse-fine-stitched (CFS) detection results with a low clutter image (clutter < µ) and high clutter image (clutter ≥ µ) are shown in Figure 8. The intermediate process of the CFS method is shown to illustrate their effects. The second, third, and forth column were the output of the coarse detection, fine detection, and robust stitching of CFS, respectively. Figure 8a-d show that the experimental results demonstrated that the method was able to extract the horizon line accurately with different backgrounds. The experiment also showed that there were few false positives for images without the horizon line, even though the background noise was large (Figure 8e). The clutter measure of an image was given by the average of the intensity variance of the sub-images: The authors divided the image into 4 × 4 sub-images with reference to Candamo's work [31]. The mean clutter of the database was denoted as μ . When the clutter of an image was less than  , it was considered as low clutter. To test the validation of this approach, the authors used the public datasets Singapore Maritime Dataset (SMD) and Marine Obstacle Detection Dataset (MODD) for benchmarking. The clutter of these images (or frames in video) varied from 1 to 45, and 14 μ = . The processing time for an image was a fraction of a second, which permitted the authors to test the method on thousands of images.
Coarse-fine-stitched (CFS) detection results with a low clutter image (clutter <  ) and high clutter image (clutter   ) are shown in Figure 8. The intermediate process of the CFS method is shown to illustrate their effects. The second, third, and forth column were the output of the coarse detection, fine detection, and robust stitching of CFS, respectively. Figure 8a-d show that the experimental results demonstrated that the method was able to extract the horizon line accurately with different backgrounds. The experiment also showed that there were few false positives for images without the horizon line, even though the background noise was large (Figure 8e).

Parameter Sensitivity Analysis
There were three main control parameters in the fine detection of the coarse-fine-stitched (CFS) method, such as threshold Td,1 = 1.5, threshold Td,2 = 1 and threshold Td,3 = 1. By fixing two parameters and changing another parameter, the mean height deviation (MHD, pixels) and mean angle deviation (MAD, degrees) of CFS performed on the testing data are shown in Figure 9. Setting a smaller Td,1 resulted in a larger probability of selecting the line segments, even if the difference between the regions was not distinct enough. Setting a larger Td,2 or Td,3 resulted in a larger probability of picking the line segments, even if the similarity between the sub-regions was not enough. Both of these caused false positives and decreased the detection accuracy.

Parameter Sensitivity Analysis
There were three main control parameters in the fine detection of the coarse-fine-stitched (CFS) method, such as threshold T d,1 = 1.5, threshold T d,2 = 1 and threshold T d,3 = 1. By fixing two parameters and changing another parameter, the mean height deviation (MHD, pixels) and mean angle deviation (MAD, degrees) of CFS performed on the testing data are shown in Figure 9. Setting a smaller T d,1 resulted in a larger probability of selecting the line segments, even if the difference between the regions was not distinct enough. Setting a larger T d,2 or T d,3 resulted in a larger probability of picking the line segments, even if the similarity between the sub-regions was not enough. Both of these caused false positives and decreased the detection accuracy.

Methods Comparision
To test the performance of this method compared with the state-of-the-art methods, horizon line detection methods with a processing time for an image that was less than one second was applied. To verify this method, the authors used Wang's algorithm [14], MSCM-LiFe [13], and SSM [18] as the baseline comparisons for the work. The dataset, the Singapore Maritime Dataset (SMD), and Marine Obstacle Detection Dataset (MODD), were applied for comparison in this section.
The performance of the baseline methods and this approach using the authors' dataset in a maritime environment is shown in Figure 10. The figure illustrates that the proposed coarse-fine-stitched (CFS) method can detect the horizon line more accurately and robustly in the presence of various interfering factors, when compared with the other baseline methods. When using the testing image with a relatively simpler background, such as the first column in Figure 10, most of the methods obtained the horizon line accurately. However, when there were several interfering factors, such as illumination changing, clouds, or islands, the CFS method performs best when compared with the other methods.

Methods Comparision
To test the performance of this method compared with the state-of-the-art methods, horizon line detection methods with a processing time for an image that was less than one second was applied. To verify this method, the authors used Wang's algorithm [14], MSCM-LiFe [13], and SSM [18] as the baseline comparisons for the work. The dataset, the Singapore Maritime Dataset (SMD), and Marine Obstacle Detection Dataset (MODD), were applied for comparison in this section.
The performance of the baseline methods and this approach using the authors' dataset in a maritime environment is shown in Figure 10. The figure illustrates that the proposed coarse-fine-stitched (CFS) method can detect the horizon line more accurately and robustly in the presence of various interfering factors, when compared with the other baseline methods. When using the testing image with a relatively simpler background, such as the first column in Figure 10, most of the methods obtained the horizon line accurately. However, when there were several interfering factors, such as illumination changing, clouds, or islands, the CFS method performs best when compared with the other methods.

Methods Comparision
To test the performance of this method compared with the state-of-the-art methods, horizon line detection methods with a processing time for an image that was less than one second was applied. To verify this method, the authors used Wang's algorithm [14], MSCM-LiFe [13], and SSM [18] as the baseline comparisons for the work. The dataset, the Singapore Maritime Dataset (SMD), and Marine Obstacle Detection Dataset (MODD), were applied for comparison in this section.
The performance of the baseline methods and this approach using the authors' dataset in a maritime environment is shown in Figure 10. The figure illustrates that the proposed coarse-fine-stitched (CFS) method can detect the horizon line more accurately and robustly in the presence of various interfering factors, when compared with the other baseline methods. When using the testing image with a relatively simpler background, such as the first column in Figure 10, most of the methods obtained the horizon line accurately. However, when there were several interfering factors, such as illumination changing, clouds, or islands, the CFS method performs best when compared with the other methods. The performance of the baseline methods and this approach using SMD is shown in Figure 11. Wang's algorithm was affected easily by the bottom of the boats. SSM was very sensitive to obstacles when using the segmentation method. MSCM-LiFe was affected easily by waves, as shown in the first column of Figures 10c and 11c. Compared with the other methods, the CFS method performed accurately and robustly for the testing data. The performance of the baseline methods and this approach using MODD is shown in Figure  12. When using the testing image with a relatively simpler background, such as the first to third columns in Figure 12, most of the methods obtained the horizon line accurately. Compared with The performance of the baseline methods and this approach using SMD is shown in Figure 11. Wang's algorithm was affected easily by the bottom of the boats. SSM was very sensitive to obstacles when using the segmentation method. MSCM-LiFe was affected easily by waves, as shown in the first column of Figures 10c and 11c. Compared with the other methods, the CFS method performed accurately and robustly for the testing data.  The performance of the baseline methods and this approach using SMD is shown in Figure 11. Wang's algorithm was affected easily by the bottom of the boats. SSM was very sensitive to obstacles when using the segmentation method. MSCM-LiFe was affected easily by waves, as shown in the first column of Figures 10c and 11c. Compared with the other methods, the CFS method performed accurately and robustly for the testing data.  The performance of the baseline methods and this approach using MODD is shown in Figure  12. When using the testing image with a relatively simpler background, such as the first to third columns in Figure 12, most of the methods obtained the horizon line accurately. Compared with The performance of the baseline methods and this approach using MODD is shown in Figure 12. When using the testing image with a relatively simpler background, such as the first to third columns in Figure 12, most of the methods obtained the horizon line accurately. Compared with other methods, Wang's algorithm and SSM were more sensitive to obstacles or shadow. MSCM-LiFe and this CFS method performed accurately and robustly for the testing data. other methods, Wang's algorithm and SSM were more sensitive to obstacles or shadow. MSCM-LiFe and this CFS method performed accurately and robustly for the testing data. However, when the horizon line was very blurry, such as in the fourth column in Figure 12, this coarse detection method cannot detect the horizon feature and the result was affected entirely by the ground and its shadow. The intermediate process of the CFS method on the very blurry horizon line is shown Figure 13.   Table 1 shows the comparison of the horizon line detection performance on this test data. The detection errors consisted of mean height deviation (MHD) and mean angle deviation (MAD). The However, when the horizon line was very blurry, such as in the fourth column in Figure 12, this coarse detection method cannot detect the horizon feature and the result was affected entirely by the ground and its shadow. The intermediate process of the CFS method on the very blurry horizon line is shown Figure 13. other methods, Wang's algorithm and SSM were more sensitive to obstacles or shadow. MSCM-LiFe and this CFS method performed accurately and robustly for the testing data. However, when the horizon line was very blurry, such as in the fourth column in Figure 12, this coarse detection method cannot detect the horizon feature and the result was affected entirely by the ground and its shadow. The intermediate process of the CFS method on the very blurry horizon line is shown Figure 13.   Table 1 shows the comparison of the horizon line detection performance on this test data. The detection errors consisted of mean height deviation (MHD) and mean angle deviation (MAD). The Figure 13. The performance of this CFS method on the very blurry horizon line. The first column is the original image, and the next three columns are the output of the coarse detection, fine detection, and robust stitching of CFS, respectively. Table 1 shows the comparison of the horizon line detection performance on this test data. The detection errors consisted of mean height deviation (MHD) and mean angle deviation (MAD). The average computing time for each testing was counted for comparison. Table 1 shows that the MHD and MAD of this method was the smallest when compared with the three baseline methods. The MHD of this approach was 49.7%, 82.4% and 39.7% of the results of Wang's algorithm, MSCM-LiFe, and SSM, respectively. The MAD of this approach was less than 0.2 • , which was also the most accurate of all of the methods. Since SSM is the segmentation method, the results were not straight lines. Thus, the MAD of SSM was not calculated in Table 1. The experimental results show that the color feature and RANSAC of this approach played an important role in obtaining low false positives and accurate results. Due to the iteration in RANSAC, the average computing time for this method was 94 ms. The computing time of this method was a little longer than Wang's algorithm and SSM, and was shorter than MSCM-LiFe. Since it included the time for the Hough transform and intensity calculation for the multi-scale median filtering of the image, the computing time of MSCM-LiFe was relatively longer than other methods.

Conclusions
To meet the requirement of horizon line detection for unmanned surface vehicle (USV) applications, a robust horizon line detection method, coarse-fine-stitched (CFS), was proposed in this paper. This CFS method operated in a three-step way. First, in the coarse step, a problem-specific design line segment detection (LSD) algorithm was presented to extract all the candidates, creating a line pool. Second, in the fine step, hybrid feature filtering was designed to select the horizon line segments from the pool. Finally, a random sample consensus-based (RANSAC-based) line segment stitching method was presented to obtain the whole horizon line. Experimental results demonstrate that the CFS outperformed the other state-of-the-art methods in terms of accuracy and robustness. Regarding the authors' future work, a segmentation algorithm will be investigated further for horizon line and obstacle detection in USV applications.