An Effective Lane Detection Algorithm for Structured Road in Urban ...

An Effective Lane Detection Algorithm for Structured Road in Urban Weirong Liu1,2 and Shutao Li1 1

College of Electrical and Information Engineering, Hunan University, 410082 Changsha, P.R. China 2 Lanzhou University of Technology, 730050 Lanzhou, P.R. China [email protected], [email protected]

Abstract. An effective and robust algorithm for structured road in urban is proposed in this paper. Firstly, the adaptive segmentation method is used to determine reasonable Region of Interest (ROI) that covers all candidate lane markings. Secondly, line segments are extracted by Line Segment Detector (LSD) and be used as low level feature to extract the structural information of road scene. Thirdly, non-lane markings are eliminated by clustering on orientation information and vanishing point. Finally, the lanes are extracted from the remaining candidate lane markings. Experimental results on Caltech lane datasets show that the algorithm can extract lanes in complex structured road scenarios. Keywords: Lane detection, Line segment detector, K-means clustering.

1

Introduction

Lane detection is used to extract lane markings from clutter road scenes and filter them to produce a credible estimation of the lanes position, which plays a significant role in driver assistance systems [1]. Many approaches have been proposed for detection of road lane markings over the last decades [2]. Accompanied by the development of computer vision technology, vision-based lane detection and recognition algorithms have been an area of active research. However, it is still a challenge problem due to variations of lane markings appearance, illumination conditions, presence of shadows and other image artifacts. A three-step process for lane detection based on visual cameras was summarized in [3]. The key step is a robust extraction of road features to initialize candidate lane markings and post-processing operation on the initialize candidate lane markings to remove outliers. In most cases, lane markings on structured roads appear as well-defined straight lines or as curves which can be approximated by some short straight lines [4]. Therefore the commonly used methods for lane markings extraction are the Hough Transform (HT) [1, 2] and its variations, such as the randomized HT [5], hierarchical additive HT [4]. However, there are two major shortcomings of HT for lane markings extraction. Firstly, high computational cost is caused by the multiplications and trigonometric operations applied to each pixel in image. Secondly, high false detection rate result from ignoring orientation information and fixed thresholds. J. Yang, F. Fang, and C. Sun (Eds.): IScIDE 2012, LNCS 7751, pp. 759–767, 2013. © Springer-Verlag Berlin Heidelberg 2013

760

W. Liu and S. Li

In order to improve validity of lane markings extraction and reduce the computation complexity, we propose an effective and robust algorithm for lane detection in this paper. Different from previous methods of image intensity or gradient, we use line segments as low-level features to analyze and detect lane markings on structured roads in urban. An efficient Line Segment Detector (LSD) presented in [6] can control false detection rate and require no parameter tuning. Therefore, lane markings on structured roads can be approximated by some line segments from LSD. Due to line segments as low-level features, we do not use any lane model in lane detection. As a result, prior knowledge of the scene constrains are not necessary except for orientation information. The rest of this paper is organized as follows. Section 2 presents the effective lane detection algorithm. Experimental results with real world data sets are provided in Section 3 and Section 4 concludes this paper.

2

Lane Detection Algorithm

2.1

Extraction of ROI

As common scene, all candidate lane markings lie in the middle of the image captured from camera mounted on vehicle. Based on some priors, such as the parameters of camera mounted on vehicle, it is easy to roughly locate the position of Region of Interest (ROI) which covers all candidate lane markings. A rough ROI marked by the red rectangle box is shown in Fig. 1(a). The upper boundary of ROI is determined by fixed ratio and the lower boundary of ROI is determined by position of the front of vehicle. Because scene dynamic varies with vehicle moving, a fixed ratio or position segmentation method can’t precisely capture upper boundary. We analysis the mean of each row in various scenes as shown in Fig. 1(b) and proposed an adaptive ROI location algorithm as follow. Let Fc ∈  m×n be an image captured from camera, where m is the number of rows in Fc and n is the number of columns in Fc . Firstly, the rough ROI F ∈  mr × n is determined by some priors, where mr is the number of rows of F . Secondly, the mean value Fi of each row in F can be represented as (1) n

Fi =  Fij / n , ( i = 1, , mr ) .

(1)

j =1

and then row value d with minimal mean value can be calculated by (2)

d = E(min(Fi )) , ( i = 1, , mr ) .

(2)

where the operator E is used to extract the row value with minimal mean value. Finally, the reasonable ROI Fd ∈ ( mr − d +1)×n extracted by (3)

Fd = Fij , ( i = d , , mr , j = 1,, n ) .

(3)

and the reasonable ROI Fd is shown by rectangle area within red box in Fig. 1(c). As shown in Fig. 1, we can conclude that the ROI in Fig. 1(c) extracted by adaptive ROI segmentation algorithm is more accurate than the rough ROI in Fig. 1(a) extracted by some priors. It is obvious that a more accurate ROI can save computation times and reduce interference from sky, buildings and trees etc.

An Effective Lane Detection Algorithm for Structured Road in Urban

(a)

(b)

761

(c)

Fig. 1. ROI extraction. (a) the rough ROI extracted by some priors, (b) the mean value of each row in rough ROI, (c) the ROI extracted by adaptive ROI segmentation algorithm

2.2

Candidate Lane Markings Extraction Using LSD

Line segments can be used as low level feature to extract the structural information of images, and can help in several problems in image analysis and computer vision [6]. Due to high computational complexity and false detection, those HT methods and its variations are not adequate in many scenarios with real time requirement. Aiming to overcome the disadvantages of traditional line segment detection methods, a good linear-time line segment detector was proposed in [6]. The LSD algorithm can gives accurate results and requires no parameter tuning. The key steps of the LSD are illustrated in Fig. 2. The candidate lane markings extraction using LSD algorithm in ROI Fd are described as follow. Let S define a set of line segments extracted from ROI Fd by (4), (4) S = LSD(Fd ) . where the operator LSD is the lines extraction algorithm as shown in Fig. 2, and S={s1 ,s 2 , , s k } is the line segments extracted from ROI Fd .

Fig. 2. The key steps of the LSD

762

W. Liu and S. Li

Each line segment s i , (i = 1, , k ) is defined as

si = {x1i , y1i , x i2 , yi2 , θi } , (i = 1, , k ) .

(5)

where (x1i , y1i ) and (x i2 , yi2 ) are the starting point coordinate and ending point coordinate of line segment s i respectively. θi ∈ [−90 ,90 ] is the angle of line segment s i and calculate by (6).

θi =

180

π

× arctan(

y i2 -y1i ) , (i = 1, , k ) . x i2 -x1i

(6)

We use the images on different scenario in the first column of Fig. 3 to test the LSD algorithms. The images in first column show the ROI extracted by adaptive ROI segmentation algorithm. There are different types of lanes in ROI such as, straight lanes, curve lanes and dash lanes under the circumstance with shadows, writing, road marks and patches etc. The images in the second and third column show the line segments extracted by HT and LSD, respectively.

Fig. 3. A comparison of HT and LSD method on line segments detection. The images in first column are the ROI, and the images in the second and third column show the results extracted by HT and LSD, respectively.

We can see that firstly the almost full boundaries of different lanes in the ROI are detected by LSD while boundary detected by HT is poor. Secondly, more obvious traces on the road are captured by LSD while missed by HT. Hence the line segments are found by LSD algorithm are greater and more accurate than line segments extracted by HT method, and the detected line segments represent well the geometric structure of ROI Fd , such as candidate lane markings, other road marks and cluttered shadows. 2.3

Non-lane Markings Removed by Orientation Information

Because vehicles usually move on road in relative fixed lanes following the traffic rules, lane markings lie on limited direction in captured image. We find that a range of 45° is enough to cover almost lanes in captured image as shown in Fig. 3. In order to remove irrelevant line segments that represent non-lane markings, we retain the

An Effective Lane Detection Algorithm for Structured Road in Urban

763

～

-70° in the left of ROI and lines that have an orientation angel between -25° between 25° 70° in the right of ROI respectively, i.e.

～

n n SL = {siL | x1i < , x i2 < , θi ∈ [−25 , −70 ]} , (i = 1,  , p ) . (7) 2 2 n n SR = {siR | x1i ≥ , x i2 ≥ , θi ∈ [25 , 70 ]} , (i = 1, , q ) . (8) 2 2 The retained line segments after non-lane markings removed by (7) and (8) are shown in Fig. 4 (a). By comparing the first row third column image in Fig. 3 with image in Fig. 4 (a), we can see that lots of non-lane markings are removed. The candidate lane marks can be more effective cover by the green lines SL and the red lines SR . Furthermore, we find that the retained green lines SL on left can be roughly divided into two categories based on direction. Therefore, we adopt the k-means algorithm to partition the retained lines SL observations into two clusters. The retained red lines SR on right can be separated by the same method. Given a set of line segments SL = {s1L ,s L2 ,,s Lp } (or SR = {s1R ,s R2 ,,s qR } ), k-means clustering aims to partition the p (or q ) line segments into two sets SL = {S1L ,SL2 } (or SR = {S1R ,SR2 } ) so as to minimize the within-cluster sum of squares, i.e. 2

arg min  SL

2

arg min  R

S

 || s

i =1 s Lj ∈SiL

i =1

 || s

L j

− μiL ||2 , ( j = 1, , p ) .

(9)

R j

− μiR ||2 , ( j = 1,  , q ) .

(10)

s Rj ∈SiR

where μiL , (i = 1, 2) is the mean of points in SiL , (i = 1, 2) , and μiR , (i = 1, 2) is the mean of points in SiR , (i = 1, 2) . Since the lanes close to vehicle are more useful than the lanes relative far from vehicle, the results from (9) and (10) which have larger angel are reserved. As shown in Fig. 4(b), the blue line segments reserved by k-means clustering algorithm can more accurately represent the lanes. In some special cases, there are still some line segments from shadow, irregular patches etc. in reserved blue line segments set. These line segments may lie in any direction, while the lane marks always lie in relative fixed orientation and intersect at vanishing point in bird's eye view. For this reason, we can use vanishing point to eliminate the line segments indicating those interference as follows. Let SK define the reserved line segments through k-means algorithm and

V = (x1v , y1v , x 2v , y 2v ) define the vanishing point calculated by V = G(SK ) .

(11)

where the operator G extrapolate SC across the whole ROI Fd , and select the point with maximum intersection number.

764

W. Liu and S. Li

The final line segments SLV (or SRV ) in the left (or right) of Fd are retained by

SLV = {s LVi || s LVi − V ||< ε } , (i = 1, , u ) .

(12)

SRV = {s RVi || s RVi − V ||< ε } , (i = 1,  , w) .

(13)

where ε is the tolerance between line segment and vanishing point, and it is set to 10 in experiments. In Fig. 4 (c), the selected vanishing point V is marked with the red ‘+’ and the reserved lines SLV and SRV are plotted in purple. It can be conclude that the remaining purple line segments are almost the part of lane markings.

(a)

(b)

(c)

Fig. 4. Non-lane markings removed by orientation information, (a) non-lane markings removed by (7) and (8), (b) non-lane markings removed by k-means clustering, (c) non-lane markings removed by vanish point.

2.4

Lane Extraction

After eliminating the non-lane markings by angles and position, the final step is extraction the left L L (or right L R ) from candidate line segments remaining SLV (or

SLV ) following L L = {L L | min(x LV1i ), max(y LV1i ), max(x LVi2 ), min(y LVi2 ), mean(θ VLi )} , (i = 1, , u ) . (14) L R = {L R | max(x RVi1 ), max(y RVi1 ), min(x RVi2 ), min(y RVi2 ), mean(θ VRi )} , (i = 1,  , w) . (15) where min(x •• ) (or max(x •• ) ) is the minimum (or maximum) of horizontal coordinate in all candidate lines S(V• ) , max( y((••)) ) (or max(y(( ••)) ) ) is the minimum (or maximum) of vertical coordinate in all candidate lines S(V• ) and mean(θ ((••)) ) is the mean of angle in all candidate lines S(V• ) . The final detected left lanes L L and right lanes L R are painted in green and red respectively. For the convenience of validation results, the final detected lanes are superimposed on the original image as shown in Fig. 5.

3

Experiments

3.1

Data Sets

In order to evaluate the performance of the proposed method, experiments were performed on sequences with different types of urban streets from Caltech lane datasets in [7]. These data sets include four clips with size of 640×480, totaling 1225 color frames of road, and these clips captured in different scenarios are challenging, such as a lot of curvatures, some writings, different pavement types, lots of shadows and passing cars.

An Effective Lane Detection Algorithm for Structured Road in Urban

3.2

765

Results

We have completed the proposed algorithm in MATLAB R2010b and tested it on Caltech lane datasets. The experiments are implemented on Intel Core 2 Duo CPU T9400 at 2.53GHz with 3GB memory. The detected lanes superimposed on the original image with different scenarios are shown in Fig. 5. Because there is no ground truth of detection on Caltech lane datasets, we qualitatively evaluate performance of the proposed algorithm. From left to right in Fig. 5, four columns of images come from washington1 clip, washington2 clip, cordova1 clip and cordova2 clip in Caltech lane datasets respectively. The results in first row show that the proposed method can accurately and completely detect left (highlighted in green) and right (highlighted in red) straight lane under the circumstance with no or a little interference. It can be seen from the images in the second row that the characters painted on road and irregular patches do not affect the performance of lane detection. Even under scenarios of the messy shadows and moving vehicles as shown in third row, the lanes are exactly extracted by the proposed method. Finally, the results in last row prove that the method is robust to the common road marks such as zebra stripes, stop lines and arrows.

Fig. 5. Examples of the detected lanes

766

W. Liu and S. Li

The results for all clips in Caltech lane datasets are shown in Table 1. The first row shows the numbers of each clips, and the second row to fourth row show the correct detection rate, false detection rate and missing detection rate, respectively. It is should be noted that the average computation time on testing platform mentioned above is under 0.4s/frame. Hence, the results in all cases shown in Fig. 5 and Table 1 clearly demonstrate the potential and practical utility of the proposed method for lane detection under different scenarios. Table 1. The results for all clips in Caltech lane datasets Clips Numbers of frames Correct detection rate False detection rate Missing detection rate

Washington1 Washington2 Cordova1 Cordova2 337 232 250 406 90.80% 93.50% 94.00% 94.30% 5.60% 4.80% 4.48% 3.80% 3.60% 1.70% 1.52% 1.90%

Because the proposed method adopts line segments as low-level and does not use any lane model, the detection results of curving lane are incomplete or false as shown in Fig. 6(a) and Fig. 6(b). For the scenarios with few lanes as shown in Fig.6 (c) or the sun facing the vehicle as shown in Fig.6 (d), the proposed method misses detection of all lanes.

(a)

(b)

(c)

(d)

Fig. 6. False detections and missing detections of lanes. (a) incomplete detection of curving lane, (b) false detection of curving lane, (c) missing detection of very short lanes, (d) missed detection of lanes with very high illumination.

4

Conclusion

In this paper, we have introduced an effective lane detection algorithm for structured road in urban. The algorithm has been tested on Caltech lane datasets and the experimental results show that the algorithm can extract lanes in complex traffic scenarios. The major advantages of the proposed algorithm are that lane markings on structured roads can be precisely represented by some line segments and prior knowledge of the scene constrains are not necessary except for orientation information. Acknowledgement. This paper is supported by the National Natural Science Foundation of China (No. 61172161).

An Effective Lane Detection Algorithm for Structured Road in Urban

767

References 1. Borkar, A., Hayes, M., Smith, M.T.: A Novel Lane Detection System with Efficient Ground Truth Generation. J. IEEE T. Intell. Transp. 13, 365–374 (2012) 2. Gopalan, R., Hong, T., Shneier, M., et al.: A Learning Approach Towards Detection and Tracking of Lane Markings. J. IEEE T. Intell. Transp. (to appear, 2012) 3. McCall, J.C., Trivedi, M.M.: Video-Based Lane Estimation and Tracking for Driver Assistance: Survey, System, and Evaluation. J. IEEE T. Intell. Transp. 7, 20–37 (2006) 4. Satzoda, R.K., Sathyanarayana, S., Srikanthan, T.: Hierarchical Additive Hough Transform for Lane Detection. J. IEEE Embedded Systems Lett. 2, 23–26 (2010) 5. Xu, L., Oja, E., Kultanen, P.: A New Curve Detection Method: Randomized Hough Transform (RHT). J. Pattern Recognit. Lett. 11, 33–338 (1990) 6. Von Gioi, R.G., Jakubowicz, J., Morel, J.M., et al.: LSD: A Fast Line Segment Detector with a False Detection Control. J. IEEE T. Pattern. Anal. 32, 722–732 (2010) 7. Aly, M.: Real Time Detection of Lane Markers in Urban Streets. In: IEEE Intelligent Vehicles Symposium, pp. 7–12. IEEE Press, New York (2008)