IEEE MELECON 2006, May 16-19, Benalmádena (Málaga), Spain
Detection, Tracking and Classification of Road Signs in Adverse Conditions George K. Siogkas
Evangelos S. Dermatas
Dept. of Electrical Engineering & Computer Technology University of Patras Patras, Greece Email:
[email protected]
Dept. of Electrical Engineering & Computer Technology University of Patras Patras, Greece Email:
[email protected] rectangles are used. The shape and colors of a sign define its significance along with the ideogram that it contains.
Abstract—In this paper a complete automatic system for roadsign detection, tracking, and classification is presented and evaluated. The processing of video frames in the L*a*b color space improves significantly the sign detection rate by processing the same frame in different normalized color spaces. The tracking module reduces significantly the processing time by transferring the sign detection information in the next frames and processing different radii signs in parallel. The proposed system is evaluated in normal, raining and night driving conditions. In a total number of 266 roadsign recordings, the complete system track and recognize successfully 216. The main source of system fault appears in city night driving due to the presence of a great number of light sources.
I.
The development of a system that can robustly detect and classify road signs (in other words perform RSR) in real time, has twofold benefits; on one hand it can be used in driver assistance systems (DAS), that help the driver focus more on the navigation of the vehicle by providing the information given by the signs. On the other hand RSR systems can, in the future, be embedded in fully autonomous vehicles. Of course, in order for these systems to be functional, they must have a number of characteristics. First of all, they must be resilient to any change in lighting or weather conditions. They also need to be able to recognize partially occluded signs, as well as signs that are either rotated, or not exactly perpendicular to the camera axis. Finally these systems must be robust to the deterioration of some signs’ color, usually due to their age and bad weather conditions.
INTRODUCTION
The area of road sign recognition (RSR) has attracted the attention of many researchers over the past decade. Its importance lies mainly on the vast amount of car accidents that happen each year all over the world, caused by the drivers’ inability to process all the visual information they receive while driving. Usually, the most important information is provided by the road signs placed in the drivers’ visual field. Road signs are designed to assist the drivers in their effort to navigate their vehicle to their destination efficiently and safely. They can be divided into three main groups: danger proclamation signs, traffic regulation signs and informational signs. Signs that belong to the first group are placed to warn drivers of the dangers that exist ahead on the road, so they can anticipate them. The second group comprises signs that inform the drivers of the special obligations, restrictions or prohibitions they should conform to. The signs of the third group provide information that assists the driver in the navigation task, such as junctions, distances etc. Road signs are designed in a way that helps the drivers to spot them easily in natural scenes. This is achieved by selecting colors and shapes that differentiate the signs from the background. Consequently, the main colors that are used are red, blue, yellow and green, with black or white ideograms. The shapes of the signs are symmetrical. Triangles, circles, octagons, diamonds and
1-4244-0088-0/06/$20.00 ©2006 IEEE
The first important decision when dealing with the problem of road sign detection (RSD) is whether to use color images. In the implementations where grayscale images are processed, the detection is based solely on morphological features, such as symmetry in [1-3], distance transforms from templates generated offline in [4], and border detection using pyramidal structures as in [5-7]. A more sophisticated method of sign detection based on genetic algorithms is proposed in [8]. In color based RSD, the color space plays an important role. In [9-11] the popular RGB color-space, or variations based on it, such as relations between the color coefficients are used. However, the RGB space is not optimized for problems such as RSC, because it is susceptible to lighting changes. Thus, color spaces more immune to such changes are preferred: the HSI is used in [12-18], or the LUV space in [19]. After color segmentation, road signs are detected using circular and triangular shapes [12], neural networks [13,18], genetic algorithms [15,16]. Simulated annealing is used together with genetic algorithms in [17]. Among the most important tracking algorithms for road signs, the authors of [18] use Kalman filters, Kalman-Bucy
537
filtering is used in [4], and a motion-model plus temporal information propagation is proposed in [10]. For RSC the most popular methods are based on template matching, either by cross-correlation [12,17] or by matching pursuit filters [14]. Various type of neural networks for RSC are implemented in [11,13,16,19], and a Bayesian generative modeling is described in [10].
The system processes a given frame, first by converting it to L*a*b format and then by performing color segmentation. Then, the segmented frame is passed on to the detection and tracking modules. The centers and radii of all the detected road signs are used in the tracking stage for the next frame and are used to crop the signs for classification. A. ColorSpace Selection - Color Segmentation The L*a*b color space possesses a number of important features: it is more immune to lighting changes, the a* (red green) and b* (yellow - blue) channels represent perceptual color differences and are linearly spaced.
In this paper a complete system for road-sign detection, tracking, and classification is proposed. The system is founded upon a mixture of widely used methods, like the symmetry detection of [21], and partly modified ones, such as the modified Otsu threshold of [20], used here for color segmentation. Moreover, a number of novel features are introduced, such as the use of L*a*b color space for the color image processing stage, and sign tracking in multiple frames by examining previously selected sub-windows of each frame.
The lighting changes in an image affect mostly the L (Luminosity) channel of the L*a*b space. Thus, an examination of the L channel of an image provides information on the lighting conditions in which it was acquired. This attribute can be used in the segmentation process, as the signs appear lighter than the mean luminosity in dark scenes, and darker than the mean luminosity in welllit scenes. This is especially useful in night driving conditions, as it filters out much unneeded information.
The structure of the paper is as follows: in section II the road-sign detection module is presented, while in section III and IV the tracking and classification modules are discussed. In section V the experimental results are presented, followed by a short discussion. II.
The simple, efficient, and fast Otsu’s thresholding algorithm as proposed in [20] is used to transform the L* channel to a binary image denoted as Lbo. Four binary images are estimated by bisecting the positive and negative part of a* and b* channels, to acquire four chromatic subspaces and the negative subspaces are multiplied by -1 to ensure positive definition for all channels. The four chromatic subspaces are then transformed to binary images by using the Otsu algorithm.
ROAD SIGN DETECTION
The structure of the proposed road-sign detection, tracking and classification system is shown in fig. 1.
The four color-based regions-of-interest (ROIs) are estimated by the intersection of the corresponding binary image and the Lbo, if the mean luminosity of the frame is lower than 40, otherwise the four ROIs are identical to the corresponding binary images. The total segmented area is defined by the union of the four ROIs. B. Symmetry Detection The four chromatic coefficients are scanned for symmetrical shapes, using the fast radial symmetry detection method of [21]. This method has been used in several applications [5-7] due to its computational efficiency and its relevance to the RSD problem. Depended on the types of road signs are detected, some or all chromatic coefficients of the segmented image are used. In the proposed system the symmetry detection method is optimized for circular shapes, but different symmetrical shapes can also be derived by adjusting the radial strictness factor Į. This means that every road sign in the image can be detected by this method, as long as it remains in the image frame after its segmentation. The symmetry detection algorithm scans for shapes of one or more given radii. In order to blindly detect every existing road sign, a large set of radii is used, thus the processing time increases enormously. The computations can be accelerated by taking into account the: (a) capability to perform computations in parallel architecture, as the calculations needed for the symmetry detection can be performed
Figure 1. Flowchart of the proposed system.
538
are defined, circular red, triangular red, circular blue and rectangular blue.
independently for each radius and, (b) information acquired from previous frames. The second approach has been implemented in the proposed system reducing both the number of radii and the size of the searching areas. A more detailed presentation of this module is given in the section discussing the tracking process.
The classification rule is based on the normalized crosscorrelation (NCC) of the cropped road signs with the reference templates for all color channels. The classification process for the circular blue signs uses only the NCC for the b* channel, while for all the other categories the sum of NCC is used for classification.
C. Center Localization – Shape Determination The center of the detected symmetrical shape is performed by a simple circle fitting. This procedure also gives an approximation of the shape’s radius. The error inserted in this process by the assumption that all shapes are circular is not considered to be important for the overall method. Once the center is localized, the sign image is cropped, fill its binary image and perform cross-correlation based template matching with the road sign shapes templates (i.e. circle, triangle, octagon, square). At the end of this stage, the approximate location, radius and exact shape of the sign have been estimated. III.
ROAD SIGN TRACKING
The tracking module has been designed to minimize the computational effort and the tracking errors: The center coordinates, for all symmetrical shapes of a chosen small radius (e.g. 10 pixels), identified as a potential road sign, are passed to the module that processes the next frame. This module now performs symmetry detection and circle fitting as in the RSD module for a sub-window of specified size and centered in the coordinates given by the localization procedure. Thus, once having detected a road sign centered in (x,y), the algorithm scans the next frame in an area around (x,y), for a symmetrical shape with a radius of the nearest integer value above the one calculated for the previously detected sign. This procedure is repeated for every sign detected in the previous frame, regardless if it was its first appearance or if it was tracked from an earlier frame. An obvious flaw of this procedure is the possibility of the temporary loss of visibility of the tracked sign (either total, or partial) in one or more consecutive frames, which results in tracking failure. An efficient, parallel, without backtracking process in previous frames method is used to resolve this problem, by choosing more than one radii in the detection process (e.g. 10, 15 and 20 pixels), thus the sign is relocated at the following frames. This technique slightly increases the computational effort but recovers missing road signs by processing more informative frames. IV.
Figure 2. Detection and tracking (from left to right). (a) Image with detected road-signs, (b) blue chromatic subspace,(c) red chromatic subspace with denoted sub-window for sign tracking, (d) blue symmetry detection for radius of 12 pixels, (e) red symmetry detection for radius of the sign detected in the previous frame, and (f) centers and radii of the detected signs.
V.
EXPERIMENTS
Our experimental data consisted of 50 video clips, acquired from a video camera positioning into a moving vehicle in two different adverse environments and, 75 video clips in a typical condition. Two different video cameras, the SONY HC85 and the SONY DCR-TRV60E were used to record road-signs in the city of Patras and the suburbs using PAL non-interlaced video (frames of 720x576 pixels) at 25 fps. The first set of video clips was shot at noon under shiny weather and was 9 minutes long. The second set of clips was taken in a rainy morning, with light rainfall, and was 2 minutes and 15 seconds long. The third one was shot at night with good weather conditions in the city of Patras and was 3 minutes long. The vehicle was moving with fluctuating speeds and in the first clip there were some parts in which
ROAD SIGN CLASSIFICATION
The proposed system detects and tracks the road signs in the frames sequence taken from a moving vehicle. With a successful detection and tracking procedure, every road-sign in the driver’s visual field is cropped, resized to 64x64 pixels and passed on to the RSC module. The number of potential template matches can be significantly reduced by processing the information of sign color and shape. More specific, four categories of road-signs
539
[3]
the lighting conditions were adverse, due to the sun’s position relative to the camera. A total number of 266 signs were recorded from 21375 frames. The proposed system was evaluated using the complete set of 139 road-signs used in the European Union.
[4]
[5]
[6]
[7]
[8]
[9]
[10]
Figure 3. Sign detection in one normal and three adverse conditions.
In table 1 the recognition rate for the classification module using only one frame, the ROIs located by the sign detection module, is shown for normal, night driving and raining conditions. In night recordings, the detection module locates faulty signs due to the presence of a great number of light sources. TABLE I.
[11]
[12]
[13]
ROAD-SIGN RECOGNITION RATE USING ONE FRAME
[14]
TYPE OF ENVIRONMENT Normal
43.92%
(343/781)
Raining
43.75%
Night driving
(7/16)
6.92%
(11/159)
[15]
The total correct detection rate for all recordings was 95.3% using simultaneous search in three radii 10, 15 and 20 pixels when a single frame is processed. The proposed tracking method increases the sign-detection and recognition rate to 81.2% classifying correctly 216 out of 266 signs.
[16]
[17]
The proposed system usually faults to detect the triangular signs especially in low light and raining conditions and faulty signs are detected in city night driving.
[18]
[19]
REFERENCES [1]
[2]
Sergio Escalera and Petia Radeva, “Fast greyscale road sign model matching and recognition”, Recent Advances in Artificial Intelligence Research and Development, J. Vitria et al. (Eds.) IOS Press, 2004, pp.69-76. G. Loy and N. Barnes, “Fast Shape-based Road Sign Detection for a Driver Assistance System”, Proceedings of International Conference on Intelligent Robots and Systems (IROS), 2004.
[20]
[21]
540
Xavier Baró and Jordi Vitrià, “Fast Traffic Sign Detection on greyscale images”, Congrés Català d’Intelligència Artificial (CCIA’2004). D.M. Gavrila, V. Philomin, “Real-time object detection for smart vehicles”, International Conference on Computer Vision, IEEE September , 1999. ]. H. Austerirmeier, U. Buker, B. Merstching, S. Zimmermann, “Analysis of traffic scenes using the hierarchical structure code”, International Workshop on Structural and Syntactic Pattern Recognition August, 1992. B. Besserer, S. Estable, B. Ulmer, D. Reichqardt, “Shape classification for traffic sign recognition”, First International Workshop on Intelligent Autonomous Vehicles, IFAC April, 1993. U. Buker, B. Mertsching, “Parallel evaluation of hierarchical image databases”, Journal of Parallel and Distributed Computing 31 (2), 1995, pp. 141–152. Y. Aoyagi, T. Asakura, “A study on traffic sign recognition in scene image using genetic algorithms and neural networks”, 22nd International Conference on Industrial Electronics, Control, and Instrumentation, IEEE August, 1996. Rozenn Dahyot, Pierre Charbonnier, Fabrice Heitz, “Unsupervised Statistical Detection of Changing Objects in Camera-In-Motion Video”, ICIP-2001 International Conference on Image ProcessingThessaloniki, Greece - October 7-10, 2001. Claus Bahlmann, Ying Zhu, Visvanathan Ramesh, Martin Pellkofer, and Thorsten Koehler. A System for Traffic Sign Detection, Tracking, and Recognition Using Color, Shape, and Motion Information. In IEEE Intelligent Vehicles Symposium (IV 2005), Las Vegas, NV, June 2005. A. de la Escalera, L. Moreno, M.A. Salichs, J.Ma. Armingol, “Road Traffic Sign Detection and Classification”, IEEE Transactions on Industrial Electronics 44 (6), 1997, pp. 848–859. G. Piccioli, E. de Micheli, P. Parodia, M. Campani, “Robust method for road sign detection and recognition”, Image and Vision Computing 14 (3), 1996, pp. 209–223. C.Y. Fang, C.S. Fuh, P.S. Yen, S. Cherng, S.W. Chen, “An automatic road sign recognition system based on a computational model of human recognition processing”, Elsevier Computer Vision and Image Understanding 96, 2004, pp. 237–268. S.-H. Hsu, C.-L. Huang, ”Road sign detection and recognition using matching pursuit method”, Elsevier Image and Vision Computing 19, 2001, pp. 119–129. A. de la Escalera, J. M. Armingol, M.A. Salichs, “Traffic Sign Detection for Driver Support Systems”, 3rd International Conference on Field and Service Robotics, Espoo, Finland, June 11-13, 2001. A. de la Escalera, J. M. Armingol, M. Mata, “Traffic Sign Recognition and Analysis for Intelligent Vehicles”, Image and Vision Computing, Vol. 11, N. 3, 2003, pp. 247-258. A. de la Escalera, J. M. Armingol, F. Rodriguez, J. M. Pastor, “Visual sign information extraction and identification by deformable models for intelligent vehicles”, IEEE Transactions on Intelligent Transportation Systems, Vol. 5, N. 2, 2004, pp. 57-68. Chiung-Yao Fang, Sei-Wang Chen and Chiou-Shann Fuh, “RoadSign Detection and Tracking”, IEEE Transactions on Vehicular Technology, Vol. 52, No. 5, September 2003, pp. 1329- 1341. D.S. Kang, N.C. Griswold, N. Kehtarnavaz, “An invariant traffic sign recognition system based on sequentialcolor processing and geometrical transformation”, Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation, Dallas, TX, USA, April 1994, pp. 88-93. N. Otsu, “A threshold selection method from gray-level histogram,” IEEE Transactions on System Man Cybernetics, Vol. SMC-9, No. 1, 1979, pp. 62-66. Gareth Loy and Alexander Zelinsky, “Fast Radial Symmetry for Detecting Points of Interest”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 8, August 2003, pp. 959-973.
