License Plate Detection Method based on HDR

2010 년 제 22 회 영상처리 및 이해에 관한 워크샵

2010.1.27-1.29

License Plate Detection Method based on HDR Image Generation from Multiple Cameras A. Vavilin, K. Deb and K.-H. Jo University of Ulsan {andy, debkaushik99, jkh2009}@islab.ulsan.ac.kr

Abstract In this paper, an approach for automatic detection of license plates (LP) is presented. To improve the detection quality under various lighting conditions high dynamic range (HDR) technique was applied. Two images with different exposures are used as an input. These images are first transformed to a single view point and then combined into one image with recovered details using block-based method. The LP detection method consists of two main stages. Initially, HSI color model is adopted for detecting candidate regions. Segmentation parameters for LP candidate detection are selected using statistical analysis. According to different colored LP, these candidate regions may include LP regions; geometrical properties of LP are then used for classification. The proposed method is able to deal with candidate regions independently from the orientation and scale of the plate. Finally, the decomposition of candidate regions contains predetermined alphanumeric characters by using position in the histogram to verify and detect vehicle LP region. Various LP images were used with variety of conditions to test the proposed method and results proved its effectiveness.

1. Introduction With the rapid development of highway and the wide use of vehicles, people have started to pay more and more attention to the advanced, efficient and accurate intelligent transportation systems (ITSs). The task of recognizing specific object in an image is one of the most difficult topics in the field of computer vision. Vehicle license plate detection (VLPD) task is quite challenging from vehicle images due to the multi-style plate formats, view point changes and the nonuniform outdoor illumination conditions during image acquisition [1, 3]. The VLPD is widely used for detecting speeding cars, security control in restricted areas, in unattended parking zone, for traffic law enforcement and electronic toll collection. Recently, the necessity of vehicle license plate recognition (VLPR) has increased significantly. The LP detection is an important research topic of VLPR system. One of the major problems in LP detection is determining LP systems. This system must guarantee robust detection under various weather and lighting conditions, complex scenes, independent of orientation and scale of the plate. During the past few years, developments dealing with simple images have been achieved with acceptable results. However, recent researches have been addressed to processing complex images with unconstrained conditions [8]. The proposed license plate detection algorithm deals with such vehicle images.

As far as detection of the plate region is concerned, researchers have found many methods of locating license plate. For example, a method for multi-style LP recognition has been presented in [1]. This method has introduced the density-based region growing algorithm for detection LP location. The license plate locations in images are identified by means of integrated horizontal and vertical projections that are scanned using a search window in [2]. Moreover, a character recovery method is exploited to enhance the success rate. Survey paper [3], offers to researchers a link to a public image database to define a common reference point for VLPR algorithmic assessment. In addition, this survey paper discusses about current trends and anticipated research in VLPR system. A regionbased license plate detection method was presented in [4], which firstly applies a mean shift procedure in spatialrange domain to segment a color vehicle image in order to get candidate regions. A method based on image segmentation technique named as sliding windows has also been proposed for detecting candidate region (LP region) in [5]. A technique based on extracts candidate regions by finding vertical and horizontal edges from vehicle region had also been proposed and this segmentation method is named as sliding concentric windows (SCW). Finally, vehicle license plate is verified and detected by using HSI color model and position histogram, respectively in [6]. The proposed paper is focused on improving of LP detection by using HDR technique. Methods for improving


image properties by extending their dynamic range from multiple exposed images were widely studied during last decade. All algorithms could be classified into two main categories: spatially varying (local) and spatially uniform (non-local). Spatially varying approach is based on the fact that humans are capable of viewing high contrasts scenes thanks to the local control sensitivity in the retina. This suggests that a position-dependent scale factor might reduce scene contrasts acceptably for displaying them on a low dynamic range device. This approach converts the original scene of real-world intensities to the displayed image intensities, using a position-dependent multiplying term. Examples of spatial varying approach were described in [9, 11]. Non-local approaches do not imitate local adaptation. Instead, almost all image synthesis, recording, and display processes use an implicit normalizing step to map the original scene intensities to the target display intensities without disturbing any scene contrasts that fall within the range of the display device. This normalizing consists of a single constant multiplier. Image normalizing has two important properties: it preserves all reproducible scene contrasts and it discards the intensities of the original scene or image. Contrast, the ratio of any two intensities, is not changed if the same multiplier scales both intensities. Normalizing implicitly assumes that scaling does not change the appearance, as if all the perceptually important information were carried by the contrasts alone, but scaling display intensities can strongly affect a viewer’s estimates of scene contrasts and intensities. Although this scaling is not harmful for many well-lit images or scenes, discarding the original intensities can make two scenes with different illumination levels appear identical. Normalizing also fails to capture dramatic appearance changes at the extremes of lighting, such as gradual loss of color vision, changes in acuity, and changes in contrast sensitivity. Examples of global-based approach could be found in [12, 13]. Another technique is block-based approach presented in [14, 15]. In this method image is decomposed into rectangular blocks of same size. Then the best exposure is selected for each block. Resulting image is a mosaic combined with selected exposures. Proposed algorithm is based on the similar approach. .

2. Specific features of Korean license plates In this section, the color arrangement of the plate and outline of the Korean VLP that are considered in this study are described. Color arrangement of the plate: Korean license plates are well classified as shown in Table 1. Each style has a different plate and/or character color. However, in all only five distinct colors like white, black, green, yellow, and deep blue used in these license plates. It is worth paying attention to three different plate colors while searching for LP in an input image. Other type of vehicles, such as diplomatic cars and military vehicles, are not addressed since they are rarely seen. Color arrangements for the

2010.1.27-1.29

Korean VLPs are shown in Table 1. Table 1. Styles of Korean license plates Vehicle type Private cars Taxi, bus and truck Government cars

Plate color White Green

Text color Black White

Yellow

Dark blue

Yellow

Black

Outline of the Korean VLP: Standard LP contains Korean alphabets and numbers which are shown in Fig. 1. Few LPs contains Korean alphabets and numbers in two rows; in future this kind of LP has been supposed to be converted into a single row. Where plate color is white and character color is black, they contain seven alphanumeric characters written in a single line. In Fig. 1, where plate color is green and yellow then character color is white and black, respectively, they contain Korean LP in two rows. When plate color is yellow, some LP contains all alphanumeric characters written in a single line.

Fig 1. Outline of the Korean license plate.

3. Algorithm description In this paper we propose an enhanced version of VLPD algorithm described in [7]. While conducting the experiments, nonuniform outdoor illumination condition and varied distances between vehicle and camera often occurred. In that case, improving of LP detection by using HDR technique is proposed. Methods for improving image properties by extending their dynamic range from multiple exposed images were widely studied during last decade. And also improve the traditional LP detection method, as license plates can appear at many different angles to the camera's optical axis, each rectangular candidate region is rotated until they are all aligned in the same way before the candidate decomposition. The proposed algorithm can efficiently determine and adjust the rotation of the license plate as it shown in Fig.3. In the proposed algorithm, detection is based on color properties of LP, shape-based verification and position histogram

3.1 HDR image generation Our algorithm starts with obtaining HDR image from two input images taken with different exposures from different viewpoints. Example of input images I L and I R in case of two viewpoints is shown in Fig.2 (a). These images were taken simultaneously from different


2010.1.27-1.29

viewpoints with exposure difference 3EV. Before getting HDR image we have to transform these images into one viewpoint. To do so, we estimate disparity map using normalized cross correlation. Considering a window around pixel (x,y) normalized cross correlation is defined as

where H k( i ) is an entropy defined as:

E NCC =

1, if a = b g (a, b ) =  0, otherwise (i ) and Dk is level of details defined as:

∑ (I (u, v ) − I )(I (u, v ) − I ) ∑ (I (u, v ) − I ) ∑ (I (u, v ) − I ) L

R

L

R

2

L

(1)

2

L

R

R

where (u,v) are coordinates relative to the window and I is the mean value over the window. Normalized cross correlation (NCC) yields a value of 1 when the pixel intensities over the window around (x,y) are in exact correspondence. It is easy to show what NCC is invariant to exposure changes. Camera response function may be approximated by the gamma function. Thus, the input images may be defined as

I L ( x, y ) = ( I ( x, y ) e L )

1γ

(2)

I R (x, y ) = (I ( x, y )eR )

1γ

255

H k(i ) = ∑ p[ j ]log 2 ( p[ j ])

(9)

j =0

p[ j ] =

1 Nk

∑ g ( j, I ( ) )

( x , y )∈Rk

(10)

i x,y

Dk(i ) = ∑ max(I (ji ) ( x, y ), I (ji ) ( x, y ))

(11)

(12)

j∈C

I x(i ) = I (i ) (x + 1, y ) − I (i ) ( x, y )

(13)

After exposures are selected for each block, they are combined into single image. To make color transactions between regions smooth Gaussian filtering is applied. Example of the resulted image could be found in the Fig.2.(c) More detailed explanation about this method is given in [14,15]

where I is sensor irradiance and eL|R are exposure values for the left and right images. From (2) the following relationship might be derived:

e I R ( x, y ) = I L ( x, y ) R  eL where e RL is the exposure

  

1γ

= I L ( x, y )e1RLγ

(3)

ration between the images.

Using this relationship, the mean over I R is

I R = ∑ I L (u , v )e1RLγ = I L e1RLγ

(4)

Substituting (4) into (1), we get

E NCC =

∑ (I (u, v) − I )(I (u, v)e − I e ) ∑ (I (u, v) − I ) ∑ (I (u, v )e − I e ) L

L

1γ RL

L

2

L

L

L

1γ L RL

1γ RL

(5)

1γ 2 L RL

(a)

1γ

Notice that the factor e RL vanishes; this concludes our proof that normalized cross correlation is invariant to exposure changes if the radiometric response is exactly a gamma curve (ignoring noise and quantization effects). While the real camera response is typically not exactly a gamma curve, it is close enough that normalized cross correlation is practically invariant. Using NCC we determine the disparity map (Fig.2. (b)) between the images and could transform them to the single viewpoint. Obtained image is then decomposed into 16 rectangular blocks (4 in vertical and horizontal directions). For each block we select the best exposure according to utility function (6) which was introduced in [14] (6) U (i, k ) = αH k(i ) + βDk(i ) in which (7) Dk(i ) = Dk(i ) max Dk( j ) (i )

(i )

Hk = Hk

j =1.. N

max H k( j ) j =1.. N

(b)

(c)

(8)

Fig 2. Input images (a), estimated disparities (b) and generated HDR image (c).


Table 2. Filtering properties

3.2 Color segmentation The detection part of the proposed algorithm is based on color segmentation in HSI color space. In this proposed method, LP detection is based on its color properties, namely mean and standard deviation values of hue. For detection of green and yellow LP pixels, hue parameter of HSI color is used in our experiment. To detect white LP pixels hue value is meaningless, hence only saturation and intensity parameters are important for this case. To estimate these properties, we used 30 images of LP taken under different lighting and weather conditions. After training from those sample data, the mean and standard deviation values of hue are computed for detection of green and yellow LP pixels. Detecting white license plate pixels, the mean and standard deviation values of saturation and intensity are computed to detect green, yellow and white LP from vehicle images. For detection of green, yellow, and white LP pixels, the binarization process can be formulated as follows: 1, if  H G ≤ H ( x, y ) ≤ H G   max   min bgreen =  (14) 0, otherwise 1, if  H Y ≤ H ( x, y ) ≤ H Y   max   min byellow =   0, otherwise

[

2010.1.27-1.29

] [

W W 1, if S (x, y ) ≤ S max & I (x, y ) > I min bwhite =  0, otherwise

Filtering parameter

Green CR

Yellow CR

White CR

Bounding box

[0.6,1.0]

[0.7,1.0]

[0.7,1.0]

Aspect ratio

[1.0,3.0]

[1.0,2.0]

[1.0,6.0]

Possible shapes

Rectangle

3.4 Rotation adjustment As license plates can appear at many different angles to the camera's optical axis, each rectangular candidate regions is rotated until they are all aligned in the same way before the candidate decomposition. Following the successful filtering operation in image, measurements such as center of area and the axis of least second moment are employed to solve the rotation adjustment problem.

(15)

] (16)

where H(x, y), S(x, y), I(x, y) is a hue, saturation and intensity components of x-th, y-th pixel, respectively. Threshold values ere obtained using statistical properties of color distribution inside LP. More detailed explanation could be found in your previous paper [7]. Color segmentation parameters are very sensitive in order to detect as much candidates as possible. All false candidates will be filtered out on the next stages.

3.3 Labeling and filtering After the candidate regions are obtained by applying color segmentation, features of each region are to be extracted in order to correctly differentiate the LP regions from others. Next step of proposed algorithm is labeling the connected components. In the proposed method, a recursive algorithm is implemented for connected component labeling operation. In this step we extract candidate regions which may include LP regions from the binary mask obtained in the previous step. During this step, main geometrical properties of LP candidate such as area, bounding box, and aspect ratio are computed. These parameters are used filtering operation to eliminate LP-like objects from candidate list. Filtering operation is done on geometrical properties of LP regions. Candidate regions (CR) are filtered according to the properties shown in Table 2.

Fig. 3 License plate detection process: (a) an input image, (b) color segmentation result (c) detected candidate (d) principal axis (e) rotation adjustment, and (f) extracted candidate. The least second moments provides the principal axis as the orientation with the candidate object. For getting principal axis of detected candidate region, we compute


2010.1.27-1.29

central moments of detected candidate region. We apply this result to obtain a direction of principal axis by centroid of detected candidate region. Angle of principal axis moments is obtained as θ =

1 arctan 2

 2 µ11  µ  20 − µ02

  

(17)

where θ denotes an angle between basis horizontal coordinate and principal axis of region. Fig.3 illustrates a sequence of successful license plate identification.

3.5

Candidate decomposition

Information extracted from image and intensity histograms plays a basic role in image processing, in areas such as enhancement, segmentation and description. In this section, verification and detection of the VLP region as well as character segmentation are considered and discussed in this study. Once the candidate area is binarized the next step is to extract the information. At first, regions without interest such as border or some small noisy regions are eliminated; the checking is made by height comparison with other plate characters height. Fig.4 shows the results for verifying predetermined alphanumeric characters.

Fig. 5 Example images: (a) different illumination, (b) complex scenes, (c) various environment, and (d) damaged license plates

4. Experimental results and conclusions

Fig. 4 Extracting alphanumeric characters (a) extracted candidate region, (b) and (c) are vertical and horizontal position histograms with LP border, (d) horizontal position histogram without LP border, (e) candidate region after removing LP border and noise and (f) character position histogram

All experiments have been done on Pentium-IV 2.4 GHz with 1024 MB RAM. In the experiments, 150 images were used the size is 640 × 480 pixels; some images which are shown in Fig. 5. The images are taken from (a) bad illumination condition (nighttime, strong sunshine and shadow), (b) scenes with a complex environment, where several objects such as trees, motorcycles, and a light post in front of vehicles are presented, (c) various environments in campus parking, access areas and more than one license plate in the same image, and (d) image with corrupted LP (bent or old). They were taken in distance of 2 up to 12 m, and the images were acquired from different viewpoints. Under these conditions, success of LP detection has reached to more than 94%. A common drawback of the proposed VLPD system is the failure to detect the boundaries or border of license plates. This occurs when vehicle bodies and their license plate possess similar colors. In conclusion, we proposed a method to improve of LP detection by using HDR technique. And also a new method


for automatic selection of threshold values for HSI color segmentation based on statistical analysis is adopted in this paper. In the proposed method candidate regions are found by using color segmentation in HSI color space. Detected candidate regions may include LP regions; geometrical properties of LP are then used for classification. The proposed method is able to deal with candidate regions under independent orientation and scale of the plate. Finally, VLP regions containing predetermined LP alphanumeric character are verified and detected by using position histogram. Color arrangement and predetermined LP alphanumeric character of the Korean license plate are important features for verification and detection of license plate regions. While conducting the experiments, different illumination conditions and varied distances between vehicle and camera often occurred. In that case, the result that has been confirmed is very much effective when the proposed approach is used. However, the proposed method is sensitive when vehicle bodies and their license plates possess similar colors. We leave these issues to be considered in future studies. .

Acknowledgements This work was supported (in a part) by a Ministry of Knowledge Economy of Korea under Human Resources Development Program for Convergence Robot Specialists.

References [1] Jiao, J., Ye, Q., Huang, Q.: A Configurable Method for Multi-style License Plate Recognition, Pattern Recognit, 42(3), 358-369 (2009). [2] Huang, Y. -P., Chen, C. -H., Chang, Y. -T, Sandnes, F. E.: An Intelligent Strategy for checking the Annual Inspection Status of Motorcycles based on License Plate Recognition. Expert Syst. with Applications, 36(5), 9260-9267 (2009) [3] Anagnostopoulos, C., Anagnostopoulos, I., Loumos V., Kayafas, E.: License plate-recognition from still images and video sequences: A survey. IEEE Trans. Intell. Transp. Syst, 9(3), 377-391 (2008)

2010.1.27-1.29

[4] Jia, W., Zhang, H., He, X.: Region-based License Plate Detection. J. Network and comput. Applications, 30(4), 1324-1333 (2007) [5] Anagnostopoulos, C., Anagnostopoulos, I., Loumos V., Kayafas, E.: A License Plate-Recognition Algorithm for Intelligent Transportation System Applications. IEEE Trans. Intell. Transp. Syst., 7(3), 377-392 (2006) [6] Deb, K., Chae, H. U., Jo, K. -H.: Vehicle License Plate Detection Method based on Sliding Concentric Windows and Histogram. Journal of Computers, 4(8), 771-777 (2009) [7] Deb, K., Jo, K.-H.: HSI Color Based Vehicle License Plate Detection. In: IEEE ICCAS, pp.687-691, Seoul, Korea (2008) [8] Matas, J., Zimmermann, K.: Unconstrained License Plate and Text Localization and Recognition. In: Proceedings of the IEEE Int. Conf. on Intell. Transp. Syst., IEEE Press, New York (2005) [9] Chiu K., Herf M., Shirley P., Swamy S., Wang C., Zimmerman K.: Spatially Nonuniform Scaling Functions for High Contrast Images. In: Graphics Interface, pp.245-254, (1993) [10] Ferwerda J., Pattanaik S.N., Shirley P. and Greenberg D.P.: A Model of Visual Adaptation for Realistic Image Synthesis. In: SIGGRAPH, pp.249-258, (1996) [11] Guoping Q., Jian G., Jian D. and Min C.: Tone Mapping for HDR Image using Optimization – A New Closed Form Solution. In: Proceedings of the 18th International Conference on Pattern Recognition, (2006) [12] Pattanaik S.N., Ferwerda J, Fairchild M.D., Greenberg D.P.: A Multiscale Model of Adaptation and Spatial Vision for Realistic Image Display. In: SIGGRAPH, ACM, 287-298, (1998) [13] Tumblin, J. and Hodgings, J. K., Guenter, B.: Two Methods for Display of High Contrast Images. ACM Trans. on Graphics, 18(1), pp.56-94, (1999) [14] Vavilin A. and Jo K.-H.: Recursive HDR Image Generation from Differently Exposed Images. In: IEEE ICCAS, Seoul, Korea (2008) [15] Andra's R., Annamaria R., Varkonyi K., Takeshi H., Szilveszter B., Yoshifumi S.: Gradient Based Synthesized Multiple Exposure Time HDR Image. In: Proceedings of IMTC, Warsaw, Poland, May, (2007)