Shadow And Highlight Invariant Colour Segmentation ... - IEEE Xplore

Shadow And Highlight Invariant Colour Segmentation Algorithm For Traffic Signs Hasan Fleyeh Department of Computer Engineering Dalarna University Borlänge, Sweden [email protected] Abstract— Shadows and highlights represent a challenge to the computer vision researchers due to a variance in the brightness on the surfaces of the objects under consideration. This paper presents a new colour detection and segmentation algorithm for road signs in which the effect of shadows and highlights are neglected to get better colour segmentation results. Images are taken by a digital camera mounted in a car. The RGB images are converted into HSV colour space and the shadow-highlight invariant method is applied to extract the colours of the road signs under shadow and highlight conditions. The method is tested on hundreds of outdoor images under such light conditions, and it shows high robustness; more than 95% of correct segmentation is achieved. Keywords—colour segmentation, colour detection, road signs, outdoor image.

I.

INTRODUCTION

Colours represent an important part of the information provided to the driver to ensure the objectives of the road sign. Therefore, road signs and their colours are selected to be different from the nature or from the surrounding in order to be distinguishable. Detection of these signs in outdoor images from a moving vehicle will help the driver to take the right decision in good time, which means fewer accidents, less pollution, and better safety. Shadows and highlights represent a big challenge to computer vision researchers. In the first case different parts of the object are exposed to different illumination levels, and in the second case the object reflects some of the light of the illuminant directly to the viewer. The aim of this paper is to study these two cases and develop a colour segmentation algorithm which is invariant to shadows and highlights. Identification of road signs is achieved by two main stages: detection and recognition. In the detection phase, the image is pre-processed, enhanced, and segmented according to the sign properties such as colour or shape. The output is a segmented image containing potential regions which could be recognized as possible road signs. The efficiency and speed of detection are important factors which play a strong role in the whole process, because it reduces the search space and indicates only potential regions. In the recognition stage, each of the candidates is tested against a certain set of features to decide whether it is in the group of road signs or not, and then

1-4244-0023-6/06/$20.00 ©2006 IEEE

according to these features they are classified into different groups. These features are chosen in order to emphasize the differences among the classes. Colours represent the basic cue for road sign detection; however shapes are used by some research groups to detect road signs. Fleyeh [1-3] developed different colour segmentation algorithms to work in different light conditions including poor light conditions such as snow fall and fog. Colour constancy is used in this algorithm to correct the colour of the sign in such poor light conditions. Shadeed et al. [4] proposed a colour segmentation algorithm in which RGB image was converted into HSV and YUV colour spaces, applying proper threshold to H and UV values and the two results were combined by an AND operation. Bénallal and Meunier [5] developed a computer vision system which was embedded in a car and capable of identifying road signs. Segmentation was achieved by the RGB colour space. de la Escalera et al. [6] built a colour classifier based on two look-up tables derived from hue and saturation of an HSI colour space. Fang et al. [7] developed a road sign detection and tracking system in which the colour images from a video camera were converted into the HSI system. Colour features were extracted from the hue using a two-layer neural network. The remaining of the paper is organized as follows. Section 2 describes the properties of the road signs. Section 3 shows the difficulties behind working in the outdoor scenes, and the effect of different factors on the perceived images. Section 4 describes how the colour varies in the outdoor images and the parameters affecting this and in section 5, there is a description of the properties of the hue and how it changes due to light variations. Section 6 describes the theoretical background of the effects of shadows and highlights on colours. Section 7 shows the colour segmentation algorithm, and section 8 shows the result and future research.

II.

TRAFFIC SIGNS: PROPERTIES

Road and traffic signs have been designed using special shapes and colours, very different from the natural environment, which make them easily recognizable by drivers [8]. These may be principally distinguishable from the natural

CIS 2006

and/or man-made backgrounds [9]. They are designed, manufactured and installed according to stringent regulations [10]. They are designed in fixed 2-D shapes like triangles, circles, octagons, or rectangles [11, 12]. The colours are regulated to the sign category (red = stop, yellow = danger) [13]. The information on the sign has one colour and the rest of the sign has another colour. The tint of the paint which covers the sign should correspond to a specific wavelength in the visible spectrum [10, 14]. The signs are located in well-defined locations with respect to the road, so that the driver can, more or less, expect the location of these signs [13]. They may contain a pictogram, a string of characters or both [14]. The road signs are characterized by using fixed text fonts, and character heights. They can appear in different conditions, including partly occulted, distorted, damaged and clustered in a group of more than one sign [11, 14]. III.

TRAFFIC SIGNS: POTENTIAL DIFFICULTIES

Due to the complex environment of roads and the scenes around them, the detection and recognition of road and traffic signs may face some difficulties. The color of the sign fades with time as a result of long exposure to sunlight, and the reaction of the paint with the air [15, 16]. Visibility is affected by weather conditions such as fog, rain, clouds and snow [16]. The color information is very sensitive to the variations of the light conditions such as shadows, clouds, and the sun. [15-17]. It can be affected by the illuminant color (daylight), illumination geometry, and viewing geometry [18]. The presence of objects similar in colors to the road signs in the scene under consideration, like buildings, or vehicles. Signs may be found disoriented, damaged or occulted. If the image is acquired from a moving car, then it is often suffers from motion blur and car vibration. IV.

COLOUR VARIATIONS IN THE OUTDOOR IMAGES

One of the most difficult problems in using colours in outdoor images is the chromatic variation of daylight. As a result of this chromatic variation, the apparent colour of the object varies as daylight changes. The irradiance of any object in a colour image depends on three parameters: The colour of the incident light: Daylight’s colour varies along the CIE curve. It is given by: 2

y = 2.87 x − 3.0 x − 0.275

for 0.25 ≤ x ≤ 0.38

(1)

. The variation of daylight’s colour is a single variable which is independent of the intensity. The reflectance properties of the object: The reflectance of an object s (λ ) is a function of the wavelength λ of the incident light. It is given by:

s ( λ ) = e( λ ) φ ( λ )

(2)

Where e(λ ) is the intensity of the light at wavelength λ , and φ (λ ) is the object’s albedo at each wavelength. The camera properties: The observed intensities depend on the lens diameter d , its focal length f , and the image position of the object measured as angle a off the optical axis. This is given by:

E (λ ) = L(λ ).(π / 4)(d / f ) 2 cos(4a )

(3)

According to this equation, the radiance L(λ ) is multiplied by a constant which will not affect object’s observed colour. By cancelling the camera’s lens chromatic aberration, only the density of the observed light will be affected. As a result, the colour of the light reflected by an object located outdoors is a function of the temperature of the daylight and the object’s albedo, and the observed irradiance is the reflected light surface scaled by the irradiance equation [18, 19]. V.

HUE PROPERTIES AND ADRIFT DUE TO ILLUMINATION CHANGES

Hue plays a central role in the colour detection. This is because it is invariant to the variations in light conditions as it is multiplicative/scale invariant, additive/shift invariant, and it is invariant under saturation changes. But the hue coordinate is unstable, and small changes in the RGB can cause strong variation in hue [13], and it suffers from three problems. Firstly, when the intensity is very low or very high, the hue is meaningless. Secondly, when the saturation is very low, the hue is meaningless. Thirdly, when the saturation is less than a certain threshold, the hue becomes unstable. Vitabile et al. [14] defined three different areas in the HSV colour space: • The achromatic area: characterized by v ≤ 0.2 or v ≥ 0.9 . • The unstable chromatic area: 0.25 ≤ s ≤ 0.5 and 0.2 ≤ v ≤ 0.9 .

s ≤ 0.25 or

characterized

by

• The chromatic area: characterized by s ≥ 0.5 and 0.2 ≤ v ≤ 0.9 . In order to obtain robustness against changes in external light conditions, these areas should be taken into consideration in any colour segmentation system. VI.

REFLECTANCE WITH WHITE ILLUMINATION

A. The reflection model Consider an image of a certain surface patch which is illuminated by an incident light with a certain Spectral Power Density (SPD) denoted e(λ ) . This image is taken by a camera

with RGB sensors characterised by their spectral sensitivities f C (λ ) for C = {R, G, B} . The Cth sensor response of the camera is given by:

∫λ fC (λ ) e(λ ) cb (λ )dλ + ms (n, s, v ) ∫ f C (λ ) e(λ ) cs (λ )dλ λ

C = mb (n, s)

In this equation, Cw is the response of the RGB sensors under the assumption of white light source, and kC is given by: kC =

(4)

for C = {R, G, B} where cb (λ ) and cs (λ ) are the body and surface albedo respectively, λ is the wavelength at which the sensor responds, and n, s, v are unit vectors represent the direction of the normal vector to the surface patch, direction of the source of illumination, and direction of the viewer, respectively, figure 1.

∫λ fC (λ ) cb dλ

(6)

where kC is the compact formulation depending on the sensors and the surface albedo only. If the assumption of white illumination holds, then

∫λ f R (λ ) dλ = ∫λ fG (λ ) dλ = ∫λ f B (λ ) dλ = f

(7)

and the reflection of the surface can be given by: Cw = emb (n, s)kC + ems (n, s, v)cs f

(8)

B. Effect of shadows on colour invariance Based on the discussion in the former section, colour invariant against shadows (surface illuminated by different levels of brightness) is tested for different colour spaces. Consider the body reflection term in equation (8) Cb = emb (n, s)kC for Cb = {Rb , Gb , Bb }

(9)

where Cb is the sensor response for the RGB under the assumption of a white light source. According to equation (9), the colour perceived by the sensor depends on three factors:

Figure 1. Traffic Sign Scene.

Furthermore, the terms mb and ms denote the geometric dependencies on the body and surface reflection component, respectively [20]. Assuming that surface albedo cs (λ ) is constant and independent of the wavelength, and white illumination is used (white illumination means equal energy for all wavelengths within the visible spectrum), then e(λ ) = e and cs (λ ) = cs which are constants. The sensors responses can be modified as: C w = emb (n, s)kC + ems (n, s, v )c s

∫λ f C (λ ) dλ

for Cw = {Rw , Gw , Bw }

Sensor response and surface albedo represented by kC . Illumination intensity e . Object geometry (surface orientation and illumination direction) mb (n, s) . The following colour spaces are examined for variations of brightness. 1) RGB colour space: The RGB colour space is easily affected by all of the factors mentioned above since

(5) C R = emb (n, s)k R

(10)

CG = emb (n, s)kG C B = emb (n, s)k B

(11) (12)

object itself. The first component is called the highlight and it is generated because the surface of the object works like a mirror reflecting the light from the source to the viewer.

Each of the RGB channels is affected by the sensor response, surface albedo, illumination intensity, surface orientation, and illumination direction.

The contribution of the body reflection component Cb and the surface reflection component Cs is added together as shown is equation (8), and hence, the measured colour represents the contribution two components.

2) The intensity I: The intensity I is defined as: I=

R + G + B e mb (n, s)(k R + kG + k B ) = 3 3

(13)

According to this equation, the intensity I depends on the illumination intensity, the object geometry, and the sensor and surface albedo. In other words intensity is variance to illumination intensity and it is affected by shadows. 3) The Nrgb colour space: The Nrgb colour space is insensitive to surface orientation, illumination direction, and illumination intensity as: kR r= k R + kG + k B kG g= k R + kG + k B kB b= k R + kG + k B

(14) (15)

Testing the different colour spaces according the aforementioned argument shows that only the hue feature is invariant to the highlight component of the perceived colour. In the following are some testing results. 1) RGB colour space: The RGB colour space is easily affected by highlight component since Rw = emb (n, s)k R + ems (n, s, v)cs f Gw = emb (n, s)kG + ems (n, s, v)cs f Bw = emb (n, s)k B + ems (n, s, v)cs f

Each of the RGB channels is affected by the sensor response, surface albedo, illumination intensity, surface orientation, and illumination direction. It is also affected by the highlight component. 2) The intensity I: The intensity I is defined by:

(16) Rw + G w + B w 3 3ems (n, s, v )c s f + e mb (n, s)(k R + kG + k B ) = 3

I=

4) Hue and Saturation: The hue and saturation are invariant against variations of brightness which means that they only depend on the sensor response and surface albedo as long as the white illumination condition holds. The hue angle and saturation are given as: ⎧ ⎪

⎫ ⎪ ⎬ 2 ⎪⎩ (k R − k G ) + (k G − k B ) × (k R − k B ) ⎪⎭

θ = cos −1 ⎨

S =1−

0.5 × [(k R − k G ) + (k R − k B )]

3 × min(k R , kG , k B ) (k R + kG + k B )

(19) (20) (21)

(17) (18)

This means that hue and saturation are only affected by sensor response and surface albedo, i.e. they are invariant to shadows. C. Effect of highlights on colour invariance Under certain conditions when the viewer is situated in a position so that the angle between the vectors nv and ns shown in figure 1 are approximately equal, the viewer receives two components of light. The first component is a light from the source reflected on the surface of the object to the viewer, and the second component represents the light reflected by the

(22)

According to equation (22), the intensity I depends on the illumination intensity, the object geometry, and the sensor and surface albedo. Furthermore, it is also affected by the highlight component. 3) The Nrgb colour space: The Nrgb colour space is sensitive to the highlight because it depends on the contribution of the surface reflection component as: e mb (n, s)k R + ems (n, s, v )cs f 3ems (n, s, v )cs f + e mb (n, s)(k R + kG + k B ) e mb (n, s)kG + ems (n, s, v )c s f g= 3ems (n, s, v )c s f + e mb (n, s)(k R + k G + k B ) e mb (n, s)k B + ems (n, s, v )cs f b= 3ems (n, s, v )cs f + e mb (n, s)(k R + kG + k B ) r=

(23) (24) (25)

4) Hue and Saturation: Following the same procedure used in the aforementioned colour spaces, hue is the only invariant feature for surface

reflection component and hence invariant for highlight, and hence it depends only on sensor response and surface albedo. For the case of the hue angle: ⎧ ⎪

⎫ ⎪ ⎬ ⎪⎩ (k R − kG ) 2 + (kG − k B ) × (k R − k B ) ⎪⎭

θ = cos −1 ⎨

0.5 × [(k R − kG ) + (k R − k B )]

(26)

are big enough to initiate at least one seed. Noise and other small object are rejected because of the region growing algorithm. This has an advantage that no more filtering is needed to delete these objected and the remaining objects are only the ones which can be used for recognition. The colour segmentation algorithm is given below: • •

For the case of saturation, it is variance to the surface reflection component, and hence it can not be used for developing any robust algorithm to highlights.

• •

S = 1−

3 × min( R w , G w , B w ) 3em s (n, s, v )c s f + emb (n, s)(k R + k G + k B )

(27)

•

Convert the RGB image into HSV colour space. Normalize the grey level of every pixel in the H image from [0,360] to [0,255]. Normalize the grey level of every pixel in the S image from [0,1] to [0,255]. Normalize the grey level of every pixel in the I image from [0,1] to [0,255]. For all pixels in the H image o

where

Rw = emb (n, s) k R + ems (n, s, v )cs f

(28)

G w = emb (n, s)k G + ems (n, s, v)c s f

(29)

•

Bw = emb (n, s)k B + ems (n, s, v )c s f

• •

VII. COLOUR SEGMENTATION ALGORITHM The Swedish National Road Administration defined the colours used for the signs [21] in CMYK colour space. These values are converted into Normalized Hue and Normalized Saturation as shown in Table 2. The colour segmentation algorithm is carried out by taking RGB images using a digital camera mounted on a moving car. The images are converted to HSV colour space. The hue, saturation, and value are normalized into [0,255]. The HSV colour space is chosen because the Hue feature is invariant to shadows and highlights. While normalized Hue is used as a priori knowledge to the algorithm, normalized Saturation and Value are used to specify and avoid the achromatic subspaces in HSV colour space. When the hue value of colour of the pixel in the input image is with the specified colour range specified in Table 2, and its hue value is not in the achromatic area, then the corresponding value in the output image is set to white. The output image is then divided into a number of 16x16 pixel sub-images and used to calculate the seeds for the region growing algorithm. A seed in initiated if the number of white pixels in the output image is above a certain threshold level. Region growing algorithm is then applied to find all the objects in the output image which

•

If corresponding S_ pixel_value < 40 Then H_pixel_value := 0

For all pixels in the H image o

(30)

Table 1 shows a comparison among different colour models and their abilities to be invariant to different imaging conditions. In this table, the letter ‘Y’ denotes an invariant colour model to the imaging condition and ‘N’ denotes sensitivity of the colour model to that imaging condition.

For all pixels in the H image o

•

If (H_pixel_value >240 AND H_pixel_value = 0 AND H_pixel_value < 10) Then H_pixel_value := 255

If corresponding ( V_ pixel_value < 30) OR ( V_ pixel_value > 230) Then H_pixel_value := 0

Divide the H image into 16x16 pixel subimages. For every subimage o

Calculate number of white pixels

o

If number of white pixels >= 60 Then put a white pixel in the corresponding position in the seed image.

Use seed image and H image, apply region growing algorithm to find proper regions with signs. VIII. RESULTS AND CONCLUSIONS

One of the most difficult jobs in computer vision is to deal with outdoor images. The main reason behind this is the uncontrollable environment including different illumination levels and different illumination geometry. In this paper a new algorithm for colour segmentation for traffic sign recognition is presented. The method is based on invoking the HSV colour space. It is proved that hue and saturation are invariant to the effects of illumination variations and hence they can be used to develop shadow-invariant algorithms for colour segmentation. It is also proved that hue is invariant to the effects of highlights and can be used to develop highlight-invariant colour segmentation algorithms. The algorithm described in this paper invokes merely hue for colour segmentation, but saturation and value features of the HSV colour space are used to define the chromatic subspace in which hue can be used. The method is tested on more hundreds of images under different light conditions (sunny, cloudy, foggy, and snow conditions) and different backgrounds. They established very good robustness.

Segmentation is successfully achieved in more than 95% of the cases under investigation. It fails when the pixel under consideration is not in the chromatic subspace of the HSV colour space. In figure 2, there are some examples. The original images are shown in column a, and the results of segmentation using one shadow-highlight variant algorithm are shown in column b, while column c shows the results of shadowhighlight invariant algorithm. The algorithm is able to delete all the noise and small objects which can not be recognized as traffic signs. This reduces the chances for false alarms and makes the detection better. Combining these results with shape recognition of the road signs, and pictogram recognition, which are parts of the future work, will give a good means to build a complete system which provides the drivers with the information about the signs in real time as part of the intelligent vehicle. TABLE I. Colour Red Yellow Green Light Blue Dark Blue

NORMALIZED HUE AND SATURATION. Normalized Hue [0,255] 250 37 123 157 160

Normalized Saturation [0,255] 207 230 255 255 230

The author wants to express his sincere thanks to Professor Mark Dougherty at Dalarna University for the fruitful discussions and for his time which is given to me freely.

[2]

[3]

[4]

H. Fleyeh, "Color detection and segmentation for road and traffic signs," presented at 2004 IEEE Conf. on Cybernetics and Intelligent Systems, Singapore, 2004. H. Fleyeh, "Road and Traffic Sign Color Detection and Segmentation A Fuzzy Approach," presented at Machine Vision Applications (MVA2005 IAPR), Tsukuba-Japan, 2005. H. Fleyeh, "Traffic Signs Color Detection and Segmentation in Poor Light Conditions," presented at Machine Vision Applications (MVA2005 IAPR), Tsukuba-Japan, 2005. W. Shadeed, D. Abu-Al-Nadi, and M. Mismar, "Road traffic sign detection in color images," presented at 10th IEEE Inter. Conf. on Electronics, Circuits and Systems (ICECS 2003), Sharjah, United Arab Emirates, 2003.

TABLE II.

[6]

[7] [8]

[9] [10]

[11]

[12] [13]

[14]

[15]

[16]

ACKNOWLEDGMENT

[1]

[5]

[17]

[18] [19]

[20] [21]

M. Bénallal and J. Meunier, "Real-time color segmentation of road signs," presented at Canadian Conf. on Electrical and Computer Engineering (IEEE CCECE), Montréal, Canada, 2003. A. de la Escalera, J. Armingol, and M. Mata, "Traffic sign recognition and analysis for intelligent vehicles," Image and Vision Comput., vol. 21, pp. 247-258, 2003. C. Fang, S. Chen, and C. Fuh, "Road-sign detection and tracking," IEEE Trans. on Vehicular Technology, vol. 52, pp. 1329-1341, 2003. G. Jiang and T. Choi, "Robust detection of landmarks in color image based on fuzzy set theory," presented at Fourth Inter. Conf. on Signal Processing, Beijing, China, 1998. N. Hoose, Computer Image Processing in Traffic Engineering. New York: John Wiley & sons Inc., 1991. S. Vitabile and F. Sorbello, "Pictogram road signs detection and understanding in outdoor scenes," presented at Conf. Enhanced and Synthetic Vision, Orlando, Florida, 1998. P. Parodi and G. Piccioli, "A feature-based recognition scheme for traffic scenes," presented at Intelligent Vehicles '95 Symposium, Detroit, USA, 1995. J. Plane, Traffic Engineering Handbook: Prentice-Hall, 1992. M. Lalonde and Y. Li, "Road sign recognition. Technical report, Center de recherche informatique de Montrèal, Survey of the state of Art for sub-Project 2.4, CRIM/IIT," 1995. S. Vitabile, A. Gentile, and F. Sorbello, "A neural network based automatic road sign recognizer," presented at The 2002 Inter. Joint Conf. on Neural Networks, Honolulu, HI, USA, 2002. J. Miura, T. Kanda, and Y. Shirai, "An active vision system for real-time traffic sign recognition," presented at 2000 IEEE Intelligent Transportation Systems, Dearborn, MI, USA, 2000. C. Fang, C. Fuh, S. Chen, and P. Yen, "A road sign recognition system based on dynamic visual model," presented at The 2003 IEEE Computer Society Conf. Computer Vision and Pattern Recognition, Madison, Wisconsin, 2003. S. Vitabile, G. Pollaccia, G. Pilato, and F. Sorbello, "Road sign Recognition using a dynamic pixel aggregation technique in the HSV color space," presented at 11th Inter. Conf. Image Analysis and Processing, Palermo, Italy, 2001. S. Buluswar and B. Draper, "Color recognition in outdoor images," presented at Inter. Conf. Computer vision, Bombay, India, 1998. S. Buluswar and B. Draper, "Non-parametric classification of pixels under varying outdoor illumination," presented at ARPA Image Understanding Workshop, 1994. T. Gevers and A. Smeulders, "Color-based object recognition," Pattern Recog., vol. 32, pp. 453-464, 1999. Swedish-Road-Administration, " http://www.vv.se/vag_traf/vagmarken/farglikare.htm," 2004.

INVARIANCE OF COLOUR MODELS TO IMAGING CONDITIONS. ‘Y’ DENOTES INVARIANCE AND ‘N’ DENOTES SENSITIVITY OF COLOUR MODELS TO IMAGING CONDITIONS [20]. Colour feature I RGB Nrgb H S

Viewing direction N N Y Y Y

Surface orientation N N Y Y Y

Highlight N N N Y N

Illumination direction N N N Y Y

Illumination intensity N N N Y Y

(a)

(b)

(c)

Figure 2. Results. (a) Original Images (b) Images segmented with shadow-highlight variant algorithm [1] (c) Images segmented with shadow-highlight invariant algorithm.