Front and Rear Vehicle Detection and Tracking in the ...

Front and Rear Vehicle Detection and Tracking in the Day and Night Times Using Vision and Sonar Sensor Fusion SamYong Kim, Se-Young Oh, JeongKwan Kang and YoungWoo Ryu Department of Electronic and Electrical Engineering Pohang University of Science and Technology Sa31, Hyojadong, Namgu, Pohang, Korea {tripledg, syoh, naroo1, ggr78}@postech.ac.kr Abstract - Recently, active researches on vehicle detection and tracking using a vision sensor are done for the driver assistance systems (DAS) - a collision warning and avoidance, vision enhancement, etc. The vehicle detection and tracking algorithm for DAS require a robust feature extraction and tracking method regardless of the light and road conditions and an exact estimation of vehicle position and velocity regardless of the distance from the ego-vehicle. But most research was carried out in the day time with a good lighting condition and what little research so far done in the night time assumed no interference of headlights from other vehicles. This paper proposes new robust vehicle detection and tracking method regardless of the light and road conditions at any distance using vision and sonar sensors. We use the sonar sensor for detection and distance estimation within 10m and use image sensor over 10m. First, this paper proposes a simple method that can determine the light condition by observing several images and this light condition is used by selecting one of several detection methods. The proposed vehicle detection method in the day time image can extract the shadow region represented by the boundary between a vehicle and the road and further verify by using other vehicle features, such as symmetry rate, vertical edge, and lane information. The vehicle tracking method in the day time uses on-line template matching using the mean image created by several consecutive detection results. The vehicle detection method in the night time extracts bright regions caused by the headlights, taillights, brake lights, etc. and these candidates are verified by observing several consecutive frames. Index Terms – DAS, intelligent vehicle, vehicle detection and tracking, on-line template.

I. INTRODUCTION It is a basic technology for DAS to detect and track vehicles using a vision sensor. However, since the road environment goes through such a huge spectrum of vehicle distribution, vehicle shape, weather conditions, structures by the roadside, tunnel, light change in the day and night times and at dawn, etc., it is vital to develop the image processing and pattern recognition algorithm that can precisely and reliably detect vehicles and extract effective information for DAS. As in the general object detection, the approach to vehicle detection is divided into two branches, that is, the featurebased approach and the appearance-based approach. The feature-based approach is the method that can extract vehicle candidates using vehicle features, like edge and intensity, etc.

Kwangsoo Kim, Sang-Cheol Park and KyongHa Park Telecommunication R&D Center Samsung Electronics Co., Ltd. Maetan-3dong, Yeongtong-gu, Suwon-city, Korea {kwangsoo72.kim, sangcheol.park, kyongha.park}@samsung.com and the appearance-based approach uses pre-designed vehicle templates. Because DAS requires real-time processing, the main approach is to extract vehicle candidates by a featurebased approach and to verify these by the appearance-based approach. There is a great deal of previous research on vehicle detection and tracking using a vision sensor. Betke et al. [1] developed a system that can extract the edge information on the road, track vehicles by a pre-designed objective function, and detect vehicles by rear-light or headlight in the night time. Handmann et al. [2] developed method using a neural network whose input consisted of the results from localoriented coding as well as entropy and shadow analysis. After the initial detection, stable tracking methods using a vehicle template have been proposed [3], [4], [5]. Sukthankar [6] proposed a vehicle detection and distance estimation method in the night time using the taillight pair. The symmetry feature is also widely used in vehicle detection and validation [7], [8]. Ferryman [9] used 3D wire-frame models for correlation with the ROI. Clady [10] and Sotelo [11] used the road and lane information for extracting vehicle candidates. Clady created a binary image for eliminating the non-road region using the road intensity model and then extracted vehicle candidates by analysing the dark area while Sotelo extracted vehicle candidates by edge information extending across the lane boundary and validated these for several frames. Most of the previous vehicle detection and tracking method assumed favourable light conditions where one can easily acquire the vehicle template in the image and cannot apply the dead zone at a close range. Normally, a DAS must extract the vehicle information within a region of interest (ROI) that is defined by the current travelling lane and both side lanes. In the night time, we cannot acquire favourable clear images of the headlight pair, taillight pair and turn lights of other vehicles around the ego-vehicle. Thus, a robust algorithm must be developed to handle these difficult situations. In this paper, we present novel vehicle detection and tracking algorithm that can be applied to the general road condition in the day and night times using vision and sonar sensors. The proposed system improves the detection performance and reduces the processing time using applying a proper algorithm. And it improves the drift problem of template tracking by the online template and improves the

Authorized licensed use limited to: POSTECH. Downloaded on May 03,2010 at 05:26:12 UTC from IEEE Xplore. Restrictions apply.

tracking performance by restores the vehicle information using the degree of trust. In Section II, the in-vehicle sensor system is introduced while in Section III, we present a method that can classify the image view into the day and night conditions at several images and based on this time of a day information, detect vehicles properly. In Section IV, we present a template-based tracking method using an online template. In Section V, the vehicle detection method in the night time is presented. Finally, we present the experimental results using the test DB that consists of various road conditions in Section VI and conclusion is presented in Section VII. II. SYSTEM OVERVIEW The Sensor system used is shown in Fig. 1. The two cameras that can detect vehicles in the medium and far range are installed by the side of a rear-view mirror and at the ceiling above the back seat and the two sonar sensors (its’ beam angle is 15°) that can measure the distance in the near range are installed at the front and rear bumpers. Because the environment of the vehicle changes relatively fast as the speed of a ego-vehicle is high, we acquire 2 images of 1 field with 640×240 for avoiding the motion flow and use 320×240 image by sub-sampling and acquire 2 signals of sonar sensors successively. The overall block diagram of the proposed software system is shown in Fig. 2. Computer Vision sensors Monitor

Sonar sensors Fig. 1. The hardware structure and the test bed

Image

Sonar

Day or night? Day Lane detection

Night

Vehicle detection in the day time

Vehicle detection in the night time

Vehicle tracking in the day time

Vehicle tracking in the night time

No

III. VEHICLE DETECTION A. Determination of the Day and Night Times As stated above, because the environment of a moving vehicle greatly varies, it is so difficult to detect vehicles using a single feature or pre-established vehicle templates. Although we use various features or templates for various environments, we must apply the appropriate algorithm to the current environment and these may be any obstacles to implement the real-time processing system. In this paper, we first divide the environment into two conditions by the intensity of illumination and apply a proper algorithm for each. Detected two lanes by the lane detection algorithm [12] have an intersection point, the vanishing point, and the vanishing line is the horizontal extension line of this point. We first set up a region between the vertical top and the vanishing line. And we calculate the mean intensity M at this region (yellow box in Fig. 3 and observe this value for several frames for eliminate the noise conditions such as the elevated bridge, mountains, wide-spread headlights of the vehicle at the near region, etc. If M exceeds a preset threshold that is determined experimentally, then it is a day time. The degree for the day time is increased or decreased by th1 for each frame continuously. If this score deviates from the threshold th2 for several frames, this system uses other detection method. th1 and th2 is determined experimentally, too. Some sample images are shown in Fig. 3. B. Vehicle Detection in the Day Time The detection system in the day time consists of 4 parts like in Fig. 4, that is, the preprocessing module working on the input raw image, the vehicle candidate extraction module by a shadow region and a template, the validation module by a prior knowledge, and the fusion module for fusing sonar and image data. 1) Preprocessing: We apply histogram equalization below the vanishing line in the raw image. This process can clear the gap between the dark road and other objects on the road as well as can easily extract the shadow region used as the first feature of the vehicle in the day time. After this process, we create a binary image by a low threshold that can eliminate the bright region.

Fail to catch? Yes Vehicle restoration

Fig. 2. System overview

Fig. 3. Determination of the light condition


Image

Sonar

Preprocessing

Lane Detection

Candidates Extraction (a) symmetry rate (b) edge angle map Fig. 6. The symmetry rate and edge angle map

Candidates Validation Fig. 4. Vehicle Detection System in the day time x

I image ( x, y )

y

I world ( x, y )

Extracted candidates

Vertical Scanning Yes

Detected Shadow Filtered candidates y

Lane x

(a) Shadow region detection (b) Splitting and filtering Fig. 5. Vehicle Candidates Extraction

2) Vehicle Candidate Extraction: As the shadow region between the vehicle and the road appears in the day time regardless of weather conditions or low light conditions, and is also a valid region for distance estimation because of plan world assumption, we use the shadow region as the first feature of a vehicle. The shadow region is defined as where the intensity value drops below a certain threshold. As in Fig. 5, the region with a certain width and height is scanned from bottom to top until it satisfies the constraint. First, the histogram profile of x direction is calculated in the input image I ( x, y ) using a horizontal scanning, and then the vertical position of shadow region Vs is extracted using variable shadow height by a vertical scanning like Fig. 5 (a). The final shadow region is determined by the “AND” operation of the Vs that represents indices with a high intensity change and the dark area in the preprocessed binary image. After transforming extracted shadow regions to the world coordinate by the inverse perspective transform (IPT) [13], these regions are split among different lane while some regions in non-ROI are filtered out like Fig. 5 (b). If the lane information does not exist like the failure of the lane detection or due to an intersection, etc., three virtual lanes are generated instead with the proper lane width estimated from the egovehicle position used by the splitting and filtering of shadow regions. Although one vehicle on the lane change may be clustered to two vehicles, this problem is resolved by the symmetry scanning process. 3) Vehicle Candidate Validation: The intensity change by the road repair, road sign, guardrail, oil spill and shadow, by non-vehicle objects in the road side may be mistaken for shadows. Thus the validation of the vehicle candidate is required. And there are many false-positive errors by shadow

pattern at the road boundary and the bottom of the guardrail in real experiments. Because this shadow pattern has a regular histogram in some continuous section, we eliminate this shadow pattern before validation step. First, the symmetry scanning is used for extracting the exact left and right boundaries. For symmetry scanning, we construct an edge angle map using Sobel operators and scan the left and right regions with the center line of the current vehicle position like Fig. 6 (b). The window size for symmetry scanning is the pixel width of the current vehicle. We then find the symmetry axis with the maximum symmetry rate by as follows [10]: Symmetry rate = s 2 / n (1) Where s is the number of the absolute same edge angle and n is total number of edges. By checking the two vertical edges at both sides of the vehicle, this system finally validates the candidates using the vertical edge histogram. If vertical histograms at both sides exceed certain ratio of the pixel width of a vehicle, this candidate is valid. And because DAS is normally interested in the nearby vehicles that drive in the current driving, left and right lanes, and this system remove other vehicles that are not of interest. 4) Vehicle Detection Using Sonar Sensors: Above 10m of distance, this system can extract vehicle candidates by just image, but below 10m of distance, the vehicle detection is supported by a sonar sensor. This region is further divided into two regions - one above 3m and another is below 3m. In the latter region, because the vehicle occupies the bulk of the region of the image, it is difficult to find the vehicle features and thus we rely on the sonar distance information. However, when the distance is above 3m, the sonar distance is fused with the distance calculated from the image coordinates. Then, this region is finally validated by using vehicle features, like symmetry rate, shadow region, and vertical edge histogram. Sample results are shown in Fig. 7.

(a) Below 3m (b) 3m~10m Fig. 7. Vehicle detection at near distance


5) Vehicle Detection at overtaking: This system does not use the existing detection method, like the temporal difference or the optical flow at a predefined ROI. These methods may be malfunction due to the road sign and may miss the long vehicles like trucks and buses that exhibit a similar texture for a long while. Therefore, we overcome these problems by using the same method used to detect forward vehicles explained earlier like Fig. 7 (a) and (b). IV. VEHICLE TRACKING IN THE DAY TIME It is a time consuming task to extract vehicle candidates by the shadow region, which does not always appear especially in the far region of the image. Thus we track the vehicle and in case we intermittently miss the vehicle while tracking, we restore the vehicle by using both the on-line template and the degree of trust (DOT) to be explained in this section. And the detection mode continues to work in lanes with no vehicles. A. Generation of On-Line Templates Each a detected vehicle has its own ID and DOT. DOT is the weight value for making the online template. DOT is set like below: - In case of the initial detection and the detection of an overtaking vehicle: Set DOT to 0 - In case of the continuous detection and tracking of the vehicle with the same ID: Increase DOT by 1 - In case of the tracking failure: Decrease DOT by 1 If the online template is updated at every frame of tracking, the drift problem may occur where the tracking region gradually moves out of the correct region until a total tracking failure. We update the online template (OLT) as follows: OLT(t+1) = aOLT(t) + (1-a) CV a = (DOT-1)/DOT (2) Where OLT(t) is the online template at frame t and CV is the current vehicle candidate region. B. Template-Based Tracking We use the Lucas-Kanade Algorithm (LKA) [3] for tracking. This method minimizes the following error between the pre-established template and the sub-region in the current image like Fig. 8: E (p) = ∑ [ I ( W(x; p + ∆p)) − T (x)] 2 (3) x

W(x;p)

(x1,y1)

(X1,Y1)

(x2, y2) Template

(X2,Y2) Current image

Fig. 8. Template tracking using LKA

Where warping parameter p = ( p1 , p 2 , p3 , p 4 ) T that represents the transform from the template to the sub-region in the image, W(x;p) is the warping function, T(x) is the online template. This research just permit the affine warping in the 2D and p is consisted of a 4D vector.  (1 + p1 ) ⋅ x − p 2 ⋅ y + p3   W(x;p) =  (4)  p 2 ⋅ x + (1 + p1 ) ⋅ y + p 4  (Y − Y + y − y )(x − x ) − ( X 2 − X1 − x2 + x1)( y2 − y1) p2 = 2 1 1 2 2 1 ( y2 − y1) 2 + ( x2 − x1 ) 2 p1 =

X 2 − X1 − x2 + x1 − ( y1 − y2 ) p2 x2 − x1

(5)

p3 = X1 − (1 + p1) x1 + p2 y1 p4 = Y1 − p2 x1 − (1 + p1) y1

The relationship between the bounding box in the image and initial p is defined in (5). Since this p is updated by the LKA in [3], the system can search for the most similar region using the online template. Additionally we adjust the tracking result by horizontal and vertical edge histogram for preventing the drift problem. C. Vehicle Restoration With a sudden unexpected change of the illumination condition, the proposed system may miss the tracked vehicles. In this case, the system can restore them using DOT. If the missed vehicle has a high DOT, the system can restore the vehicle by lowering the threshold parameters in the shadow constraint, vertical histogram constraints than in the previous frames and decreasing the DOT correspondingly. IV. VEHICLE DETECTION IN THE NIGHT TIME A. Vehicle Detection in the Night Time The bright regions generated by headlights, taillights, brake lights and reflected lights around light sources are used as the vehicle feature in the night time. Overall system is shown in Fig. 9. By extracting the pairs of lights, filtering the noise region, and reducing the size according to the size and shape of the bright region by light classification and light extraction step of Fig. 9, we extract vehicle candidates like Fig. 10 (b). For pairing by light shape, we use the light classification step. First, we calculate possible vehicle pixel width (PW: 1.5m is used) at the center of gravity (COG) points of each bright region clusters by IPT. Vehicle light shape that can be appeared in the night time image is divided into 3 shapes. - Small light: Light source by tail lights and brake lights without spreading. - Large light: Reflected light appeared in a vehicle by other light sources - Huge light: Light source by headlight These lights are classified by the ratio of a vehicle size using PW to a light size like Fig. 11. - Small light : light size ≤ (PW/5)×(PW/5) - Large light : small light th ≤ light size ≤ (PW/2)×(PW/2) - Huge light : otherwise case


vehicle candidate, we use the algorithm for the day time. In this light condition, we apply two methods at every 10th frame.

Bright Region Labelling Light Classification

VI. EXPERIMENTAL RESULTS

Vehicle Light Extraction DOT > th Yes No

Light Tracking Vehicle Validation

Fig. 9. Vehicle detection and tracking in the night time

(a)

The test DB consists of the many driving scenarios that were generated on a variety of driving and road environments and there are 1320 vehicles in the test DB, 555 frames. The test DB is shown in Table I. In a performance test, we don’t care vehicles at non-ROI like the opposite lane region, region over the vanishing line. Table II shows the processing time of one image by each part. Some sample results of vehicle detection at various road environments are shown in Fig. 12. The performance of vehicle tracking is evaluated by up to how many frames can be tracked by online templates after the initial detection and is show in Fig. 13. Although there is a small fluctuation of distance estimation the system can stably track by adding the restoring process. TABLE I Test DB DB The number of frames Total Day time 85 Back light (day time) 159 555 Rainy day (day time) 211 Night time 100 TABLE II Breakdown of the Processing Time Processing module Average Processing Time (ms) Preprocessing 16 Vehicle Candidate Extraction 7 Validation 7 Tracking 20

(b)

(c) (d) Fig. 10. Vehicle detection in the night time Vehicle size at the cog point H

PW Cog point of this cluster

S

L

(a) Truck

(b) 2 vehicles (left: sonar distance)

Fig. 11. Light Classification

Reflected part of a huge light is cut off like Fig. 10 (c) and (d). If the vehicle candidate is extracted in the similar region again, DOT of this candidate is increased by 1. And If DOT exceeds certain threshold th, this candidate is tracked. In this paper, th is 3. The system tracks the detected vehicle by applying the same procedure at ROI set in the previous frame. If final vehicle candidates exist in ROI, these candidates are valid. B. Switchover between Day and Night Times Sometimes the division between the day time image and the night time image is vague and shadow features in the day time and bright region features in the night time are mixed in an image at dawn or dusk. In the established boundary of illumination condition, we apply the two detection methods in an image at the same time and select the one method that creates the vehicle candidate. If the both algorithm extract

(c) Rear vehicle (back light)

(d) Rainy day

(e) Night time (rear) (f) Night time (front) Fig. 12. Results of vehicle detection


REFERENCES

Fig. 13. Results of vehicle tracking and restoration TABLE III Performance Rate of Vehicle Detection Detection result Vehicle Non-vehicle Sample Vehicle Non-vehicle

85% 5%

15% 95%

TABLE IV Performance Rate of Vehicle Tracking and Restoration Tracking Vehicle Non-vehicle result Sample Vehicle 96% 4% Non-vehicle 2% 98%

The performance Rate is show in Table III and IV. Table III shows the performance rate of the initial detection stage and Table IV shows the performance rate of the vehicle tracking and restoration stage. And we judge following situations error cases of a performance rate of initial detection: - In case of failing to detect vehicles (maximum 3 vehicles) at ROI in the first frame - In case of detecting vehicles at non-ROI in the first frame Thus a performance rate of initial detection have a large error rate than that of previous researches but that is overcome by observing consecutive frames and a tracking and restoration process like Table IV.

[1] Margrit Betke, Esin Haritaoglu, Larry S. Daivs, “Real-time multiple vehicle detection and tracking from a moving vehicle,” Machine Vision and Applications, vol. 12, no. 2, pp. 69-83, 2000. [2] U. Handmann, T. Kalinke, C. Tzomakas, M. Werner, W.v. Seelen, “An Image processing for drive assistance system,” Image and Vision Computing, vol. 18, issue 5, pp. 367-376, 2000. [3] Iain Matthews, Takahiro Ishikawa, and Simon Baker, “The Template Update Problem”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 810 – 815, June 2004. [4] Tao Xiong and Christian Debrunner, “Stochatic Car Tracking With Lineand Color-Based Features,” IEEE Transactions on Intelligent Transportation Systems, vol. 5, no. 4, pp. 324-328, December 2004. [5] Shai Avidan, “Support Vector Tracking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 8, pp. 1064 – 1072, August 2004. [6] Rahul Sukthankar, “RACCOON : A Real-time Autonomous Car Chaser Operating Optimally at Night,” IEEE Intelligent Vehicles ’93 Symposium, pp. 37-42, July 1993. [7] Thomas Zielke, Michael Brauckmann and Werner von Seelen, “Intensity and Edge-Based Symmetry Detection with an Application to CarFollowing,” Computer Vision and Image Understanding, vol. 58, issue 2, pp. 177-190, September 1993. [8] A. Broggi, P. Cerri and P.C. Antonello, “Multi-Resolution Vehicle Detection using Artificial Vision,” IEEE Intelligent Vehicles 2004 Symposium, pp. 310-314, June 2004. [9] James M. Ferryman, Stephen J. Maybank and Anthony D. Worrall, “Visual Surveillance for Moving Vehicles,” International Journal of Computer Vision, vol. 37, issue 2, pp. 187-197, June 2000. [10]X. Clady, F. Collange, F. Jurie and P. Martinet, “Cars Detection and Tracking with a Vision Sensor,” IEEE Intelligent Vehicles 2003 Symposium, pp. 593-598, June 2003. [11]Aiguel Angel Sotelo and Jose E. Naranjo, “Vision-based Adaptive Cruise Control for Intelligent Road Vehicles,” Proceedings of 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 64-69, September, 2004. [12]JeongKwan Kang, SamYong Kim, Se-Young Oh, “All-around vehicle detection using vision sensors”, Proceedings 2004 ITS World Congress, Nagoya, Oct. [13]Massimo Bertozzi and Alberto Broggi, “GOLD: a Parallel Real-Time Stereo Vision System for Generic Obstacle and Lane Detection,” IEEE Transactions on Image Processing, vol. 7, issue 1, pp. 62-81, January 1998.

VII. CONCLUSION This paper presents the real-time vehicle detection and tracking algorithm that can extract vehicles in various environments with under various illumination conditions, noise on the road, and the coordination of various vehicles. This system first improves the overall detection performance by applying different algorithms between the day and night conditions. It applies other features such as the shadow and bright region for robustness and improves the system stability by online template-based tracking and restoring missed vehicles by using a measure called the DOT. For applying to DAS, this method is required the faithful distance estimation method using a single camera which can access the danger of collision in the future. ACKNOWLEDGMENT This research was supported by Samsung Electronics Co. Ltd.
