Real Time License Plate Location Based on Edge Detection

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

Real Time License Plate Location Based on Edge Detection Shyang-Lih Chang1,2, Chung-Ju Yeh2, Shen Cherng3, and Sei-Wang Chen2 1

2

Department of Electronic Engineering, St. John’s University, Taiwan Department of Computer Science and Information Engineering, National Taiwan Normal University, Taiwan 3 Department of Electrical Engineering, Cheng Shiu University, Taiwan E-mail: [email protected]

Abstract We present a method that an effective license plate location utilizes edge detection based on FPGA (Field Programmable Gate Arrays) to implement the system so that the system can detect where the vehicle license plate is in real-time and minimize the edge magnitude which is not part of license plate in colorful image. We use the gradient-based technique to compute the three gradients magnitude of the color model of RGB space. For using the property of the relativity of RGB model, the system selects the edge from the inside of the color of the license plate and removes out the complementary color of the license plate color. Finally, the method of edge detection based on the specific vehicle license plate color utilizes the FGPA to implement. Keywords: license plate, edge detection, FPGA

1. Introduction The license plate is the most cardinal and direct authority to identify the vehicles so license plate recognition (LPR) plays an important role in intelligent transport system (ITS). There are two main steps of LPR: 1. license plate location; 2. character recognition. The high-speed and accuracy to find where the license plate is and recognize the characters are to advance the range of application of LPR. This paper is to discuss how to use the information of license plate to detect where the license plate locates and to propose a method of hardware structure to solve the problem. In license plate location, the obvious and reliable feature is the edge of license plate so edge detection is widely using technique for license plate location. The approach of edge detection mostly utilizes the gradient-based detection [2] [3]. However, for capturing the

frame of the vehicle in the real world, the edge detection does not only extract the license plate, but also obtain the edges which are not part of the license plate such the edges of brake light, the mark of vehicle brand, the vent of engine and so on, even the advertise of sidewalk, the pavement or fence may generate the edge in colorful image. Those non license plate edges have a great effectiveness to locate the license plate correctly. If the system eliminates those non license plate edges and only detects the location of license plate, the accuracy of license plate location will be improved. In order to achieve the goal, the information of color is very important so that the system can utilize the specific colors to remove out the edges which are not part of license plate. In the color model of HSI (Hue, Saturation and Intensity), subimage(I) detects the edge first based on [4], and then using the property of white pixel of Saturation (S) that is closed to zero filters non license plate edges so the method can detect the sorts of license plates which are black-white, red-white and green-white. However, there is a problem of the method that can not be detected such the license plate of yellow-background-black-character of the motorcycle (the capacity between 250 to 550 cc) and also be founded the other kind colors of non license plate edge such white-black, white-yellow, white-cyan and so on. In [4], we utilize the relation of the components of RGB in colorful image to detect the specific license plate edge and propose the hardware structure that it is implemented by using FPGA in real-time [5]. However, the method uses the magnitude of gradients of RGB components to look for the specific color edge that we desire but the chief defect is that we detect the complementary color of the edge. For instance, when we look for the color of yellow-back edge, we find the color of

Ⅱ-197

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

blue-white edge at the same time. In order to solving the problem of complementary color, we not only use the magnitude of gradients of RGB to detect the edge, but also add new method that is the property of RGB to extract the edge of license plate. There are the following sections: 2. Edge Detection, 3. Hardware Architecture, 4. Experiment and Result, 5.Conclusion.

gradients of green and blue as ∆g ( x , y ) and

∆b( x , y ) , respectively. ⎧⎪∆r1 = R( x, y ) − R( x −1, y ) ⎨ ⎪⎩∆r2 = R( x , y ) − R( x , y −1) ! ! ⎧∆r1 if ∆r1 ≥ ∆r2 ∆r( x , y ) = ⎨ otherwise ⎩∆r2

Where R(x,y) denotes that it is a R value on the point (x,y). ∆r( x , y ) represents that it is a gradient of

2. Edge Detection of License Plate The main method of this paper is often seen the four specific license plates that are whitebackground-black-characters (capacity between 90 and 250cc motorbike and automobile), whitebackground-red-characters(cab) ,red-backgroundwhite-characters (capacity greater than 550cc motorcycle and sightseeing bus), white-background-green-characters (school bus), green-background-white-characters(capacity less than 90cc motorbike, truck and bus) and yellowbackground-black-characters(capacity between 250 and 550 cc motorcycle). Generally speaking, we have to detect four sorts of license plate colors which are white-black edge (WK), red-white edge (RW), green-white edge (GW) and yellow-black edge (YK). 2.1 Review of Edge Detection on License Plate Based on the previous paper [1] of edge detection on license plate, we can compute the three gradients of RGB to be the edge, where the red gradient (R component) ∆r( x , y ) is shown as equation (1). As the same method as red gradient computation, we can obtain the other two

(1)! ! ! ! ! !

R value on the point (x,y). According to ∆r( x , y ) , ∆g ( x , y ) and ∆b( x , y ) , we can compute the amount of the edge information on frame, but the problem is not all of the edges are required. We only want to detect where the license plate is and also to minimize the existence of the edge which is not part of license plate so that we can use some property of the color model to select what kind of edge we desired. For instance of the black and white edge, the white point is consisted of three RGB values that are closed to each other and those values are high. In opposite, the black point is also made of three RGB values that are closed to each other with lower values. Let’s consider the edge from white to black, that ∆r , ∆g and ∆b are positive and closed to each other. If the edge is from black to white, ∆r , ∆g , ∆b are negative and closed to each other, too. The black-white edge EWK (WK edge) can be represented as equation (2).

⎧Min( ∆r , ∆g , ∆b ) if ∆r ≈ ∆g ≈ ∆b and ∆r , ∆g , ∆b have the same sign EWK = ⎨ otherwise ⎩0 ⎧Min( ∆g , ∆b ) if ∆r < ∆g and ∆r < ∆b and ∆g , ∆b have the same sign ERW = ⎨ otherwise ⎩0 ⎧Min( ∆r , ∆b ) if ∆g < ∆r and ∆g < ∆b and ∆r , ∆b have the same sign EGW = ⎨ otherwise ⎩0 ⎧Min( ∆r , ∆g ) if ∆b < ∆r and ∆b < ∆g and ∆r , ∆g have the same sign EYK = ⎨ otherwise ⎩0 E( x , y ) = Max{CWK × EWK ( x , y ) , C RW × E RW ( x , y ) , CGW × EGW ( x , y ) , CYK × EYK ( x , y ) }

Ⅱ-198

(2)

(3)

(4)

(5)

(6)

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

For the red-white edge (RW edge), R value of the red point is higher than G value and B value. As a point from red to white, it can be represent that ∆g > ∆r and ∆b > ∆r and also the sign bit of ∆g and ∆b are all negative. For the point from white to red, it shows that ∆g > ∆r and ∆b > ∆r and also the sign bit of ∆g and ∆b are all positive. The ERW(RW edge), denotes equation (3). The green-white edge EGW(GW edge) and yellow-black edge EYK(YK edge) are shown as equation (4) and (5), respectively. If we want to detect all of the license plate edge, we select the maxima value of EWK, ERW, EGW and EYK. However, for practical application, instead of finding the entire license plate on image, we aim at detecting one of license plate or specific plate so that the circuit which is control bit Cxy can be added in the system to determinate what sort of license plate we require. Cxy=1 denotes that the system detects the edge of license plate (Exy) as shown in equation (6) 2.2 Edge elimination of complementary color In previous section, we determine to select edge based on the three color model of RGB gradients ∆r , ∆g , ∆b and those sign bits to decide the edge whether we require or not. This method can detect the edge we desired, but we can also find the complementary color simultaneously. For example, the red-white edge is that the three higher RGB values are on the white point and the red point of R value is higher than G and B value so the equation of red-white edge is ∆g > ∆r and ∆b > ∆r with ∆g and ∆b that have the same sign bits.

However, we have to consider the problem that the complementary of red and white are cyan and black. The R value of Cyan is lower than G value and B value ,and the three RGB values of black are all low so the edge from cyan to black can be denoted that ∆g > ∆r and ∆b > ∆r with the sign bits

from

black and cyan, it shows that ∆g > ∆r and ∆b > ∆r with the sign bits

of ∆g and ∆b that are negative. It totally matches equation (3) upon the condition. The conclusion is that if we detect the red-white edge, the cyan-black edge is also detected at the same time. Figure (1) is the original image. The

system detects not only red-white edge but also cyan-black edge based on the equation (3) as shown in figure 1.b. Figure 1.c shows that the system just maintains the white-red edge and eliminates the complementary color with other colors.

(a)

(b)

(c)

Figure 1. Edge detection of red and white, (a) Original image, (b) Red-white edge and its complementary color of edge, (c) Only red-white edge. In order to solve the problem of complementary color, we add one constraint to exam the color of I(x,y). For example, edge detection for red and white, is that we have to determine the point (x,y) whether red or white is or not. If the point (x,y) is red, the constraint will add R(x,y)ɧG(x,y) and R(x,y)ɧB(x,y). In contrast to red point, if the point (x,y) is white, the constraint of R(x,y) ≈ G(x,y) ≈ B(x,y) will be added. Focusing on the edge from white to black, the sign bits of ∆g and ∆b should be positive (0 means positive). According to those above constraints, we modify the equation (7) from equation (3) to obtain the result of edge detection of red and white as shown as Figure 1(c). The equation (8) of edge detection of green-white and equation (9) of edge detection of yellow-black can be modified by equation (4) and (5), respectively.

of ∆g and ∆b that are positive. For the edge

ERW

⎧Min( ∆g , ∆b ) if ∆r < ∆g and ∆r < ∆b and ⎪ ( (R > B and R > G and S g = 1 and Sb = 1 ) or ⎪ ! =⎨ (R ≈ B ≈ G and S g = 0 and Sb = 0 ) ) ⎪ ⎪0 otherwise ⎩

Ⅱ-199

(7)

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

EGW

⎧Min( ∆r , ∆b ) if ∆g < ∆r and ∆g < ∆b and ⎪ ( (G > R and G > B and S r = 1 and Sb = 1 ) or ⎪ ! ! ! =⎨ (R ≈ B ≈ G and S r = 0 and Sb = 0 ) ) ⎪ ⎪0 otherwise ⎩

(8)

⎧Min( ∆r , ∆g ) if ∆b < ∆r and ∆b < ∆g and ⎪ ( ( R > B and G > B and S r = 0 and S g = 0 ) or ⎪ ! ! ! (9) EYK = ⎨ (R ≈ B ≈ G and S r = 1 and S g = 1 ) ) ⎪ ⎪0 otherwise ⎩ In equation (7), (8) and (9), Sr, Sg and Sb denote the sign bit of ∆r , ∆g , ∆b , respectively. EWK

⎧Min( ∆r , ∆g , ∆b ) if ∆r ≈ ∆g ≈ ∆b and ∆r , ∆g , ∆b have the same sign ⎪ =⎨ and R ≈ G ≈ B ⎪0 otherwise ⎩

Although there is no complementary color on black and white, it is easy to detect other sorts of color edge by using equation (2). For instance, the RGB values are 200, 100, 200, respectively that forms pink and the RGB values are 100, 0, 100, respectively that becomes petunia so that the three gradients of RGB( ∆r , ∆g , ∆b ) equal 100, 100 and 100, respectively. It satisfies the condition of equation (2). However, the result is not our expectation of edge detection of black and white. In order to solve the above problem, we have also consider whether the point (x,y) is white or black is or not. Because white and black are classified to gray color, the RGB values of white are close to each other, so are black. Based on the observation above, we have to add one constraint to determine the RGB values of the point (x,y) whether are closed to each or not so that most of points which are not part of white and black can be eliminated. We have modified the equation (10) from equation (2). !

3. Hardware Architecture We propose an effective method of hardware architecture that is edge detection for license plate with FPGA implementation in this section so that the system can detect the license plate in real-time. The hardware architecture is consisted of two components that are Gradient Calculator and Edge Detector. In figure 2, Gradient Calculator mainly computes the results that are the three gradients of RGB ( ∆r , ∆g , ∆b ) with those positive or negative sign bits (Sr, Sg and Sb). The three gradients and those sign bits with the

(10)

RGB values are sent to Edge Detector to determine whether the gradients that belong to license plate should be preserve or not. Of course, the gradient that does not belong to license plate must be eliminated. R(x,y)

R Gradient Calculator

G(x,y)

G Gradient Calculator

B(x,y)

B Gradient Calculator

∆r , S r

∆g , S g

Edge Detector

E(x,y)

∆b , S b

Gradient Calculator

Figure 2. The architecture of license plate edge detection 3.1 Gradient Calculator The main of Gradient Calculator computes the results of the three gradients of RGB values that are consisted of R Gradient Calculator, G Gradient Calculator and B Gradient Calculator. We use R Gradient Calculator to be the following example as shown in Figure 3. R(x,y)

∆r1

Delay R(x-1,y) T

A Subtract S ∆r1 B Bo Sr1

absoluter

Delay WT

A Subtract S ∆r2 Bo Sr2

absoluter ∆r 2

R(x,y-1) B

a comparator b a>b s Multiplexer

∆r( x, y) Sr

Figure 3. The architecture of R Gradient Calculator. We use the equation (2) to implement the hardware of figure 3 to obtain red gradient ∆r and its sign bit of Sr. We utilize the

Ⅱ-200

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

two shift registers to be a buffer. The input signals of the two shift registers delay T time (1 clock cycle) to obtain R(x-1,y) and WT time (W clock cycles, where W is the width of image) to get R(x,y-1), respectively then subtract R(x,y) to gain the differences ∆r1 and ∆r2 with their sign bits Sr1 and Sr2. To enter the differences of ∆r1 and ∆r2 to the circuit of Absoluter, we can get the absolute value of ∆r1 and ∆r2 . After the comparator to compare the magnitude of two absolute value ∆r1 and ∆r2 , the component of Multiplexer selects the higher value and its corresponding sign bit to be two outputs that are the point of gradient ∆r and the sign bit Sr. Similarly, in the same structure, we can also use G and B to get the gradient of G and B( ∆g and ∆b ) with the sign bits (Sg and Sb) in the colorful image as shown in figure 3.

3.2 Edge Detector The structure of Edge Detector which is consisted of four edge detectors by using four specific colors and one Edge Selector is shown in figure 4. There are nine parameters to be computed, that are the input of RGB values, ∆r , ∆g and ∆b from Gradient Calculator and their sign bits of Sr, Sg and Sb. The output is the edge magnitude (E(x,y)) on the image. In figure 4, the four edge detectors are used to perform equation (7), (8), (9) and (10), which can detect the red-white edge (RW Edge), green-white (GW Edge), yellow-black (YK Edge) and white-black (WK Edge), respectively. The four classified colors denote four different sorts of license plate colors. Edge Selector is set by four control bits that are used to select the edge that we are interested in specific color as shown in equation (6). ∆r , ∆g , ∆b S r , S g , Sb

WK Edge

EWK

Detector

RW Edge

ERW

Detector

Edge Selector

E(x,y)

GW Edge Detector

EGW

YK Edge Detector

EYK

CWK CRW CGW CYK

Figure 4. The architecture of Edge Detector. 3.2.a WK Edge Detector WK Edge Detector mainly detects the white-black edge realized by equation (10). From equation (10), there are three properties of white-black edge in the following list: 1. the original of RGB intensities must closed to each other; 2. the three gradients of the RGB components must be approximated; 3. the sign bit of gradient must stay the same. According to the three properties, WK Edge Detector can be designed as shown in figure 5. In figure 5, we utilize two Approximation checkers to exam whether the three gradients of ∆r , ∆g and ∆b are closed to each other or not , so do RGB values. The component of Same sign detector is used to check whether three sign bits of gradients are closed to each other or not. If the three properties are matched, the component of Minimum value detector will choose the minimum value among ∆r , ∆g and ∆b to the edge of white-black; otherwise, Multiplexer will select “0” to be output. ̏∆r| ̏∆g| ̏∆b|

Approximation

checker R(x,y) G(x,y) B(x,y) Sr Sg Sb

0

Minimum value detector

a b Multiplexer

Approximation

checker EWK Same sign detector

Figure 5. The architecture of white-black Edge Detector. ! In figure 5, Minimum value detector can be easily implemented by multiplexers and comparator to accomplish the circuit. The component of Same sign detector directly used to fundamental logic gate. The component of Approximation checker is to exam whether one of RGB values is greater than the other of RGB values multiply by a decimal number which is closed to one, that means that I ≥ αJ , where͉ =b

E

0

Ob

Fig. 8. Architecture for the edge selector.

Ⅱ-202

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

4. Experiment and Results For the system design, we have used the prototype board which contains Cyclone EP1C6Q240C8 Altera FPGA with Qurtus ҈ environment. The entire architecture is fully pipelined. To detect the edge of license plate every frame requires (W+W*H) clocks because it has to wait W time first, where W is width and H is height on image. The clock of FPGA is 48 MHz so for the size image of 640*480, the total processing time is (640+640*480)/48us = 6.413ms. The experimental result is to detect the license plate of white-background-blackcharacters as shown in figure 9. If the system detects all of the edges in colorful image, the system can be not only detected others of vehicles, the walls, the gutter or trees and so on but also can be extracted the edge of the license plate simultaneously as shown in figure 9.(b). Those edges extracted may be interference of license plate location. In the system design, we only detect the white-background-blackcharacters license plate of the white-black edges so that a bundle of edges are eliminated and the license plate can be extracted obviously as shown in figure 9(c). Using the experimental result will help us to find the location of license plate rapidly. In figure 10, it is focus on the three kinds of edge detection that are red-white, green-white and yellow-black license plates as shown in figure 10(a), (b) and (c), respectively. Simultaneously, in the same structure, if we only detect all of the edges, we will obtain the edges of non license plate so that the edges will affect the system judgment of the license plate location

as shown in figure 10 (d), (e) and (f). In the system design, we only detect the specific colors of license plates as shown in figure 10.(h), (i) and (j) respectively. Obviously the experimental results show that the most of edges have been removed out except the edge of license plate so that we can find the position of license plate easily. The system can extract the specific license plate and eliminate the other interferences of license plates. In figure 11.(a), there are two sorts of different license plates on the image. If we only detect the license plate of sightseeing bus, the system will only detect the red-white edges of the image as shown in figure 11.(b). In the same way, if we only detect the license plate of motorcycle, we will detect the white-black edges so that we obtain the location of motorcycle license plate as shown in figure 11.(c).

5. Conclusions and Future Work This paper describes not only the efficiency of the method but also to be implemented by hardware so that the edge of the license plate can be detected faster. Because the architecture is based on SOPC, it is high expansion. In the future, the entire of license plate recognition will integrate in a single chip. Acknowledgments. This work is supported in part by the National Science Council, Taiwan, Republic of China under contract NSC-96-2221-E-129-011.

(a) Original image. (b) Edges detection for all the edges. (c)Only white-black edges Figure 9. Edge detection of white-background-black-characters.

Ⅱ-203

2008亞太華人高速電路設計研討會 (HSCD 2008), 2008年7月22、23日 聖約翰科技大學電子工程系, 台灣淡水

(a)

(b)

(c)

(e)

(f)

(g)

(h)

(i) Figure 10. Different colors of license plate detection.

(j)

(a)original image.

(b) Dtection for red-white edges. (c)Detection for black-white edges. Figure11. Extract the specific color of license plate.

Reference: [1] S.L. Chang, and S.W. Chen, “Edge Detection of License Plates and Its Hardware Architecture”, Conf. on HSPCB’06, Taipei, June 9, 2006. [2] Feng Yang, Zheng Ma, “Vehicle License Plate Location Based on Histogramming and Mathematical Morphology”, Proceeding of the 4th IEEE Workshop on Automatic Identification Advanced Technologies, pp.89 – 94, 17-28 Oct. 2005. [3] Bai Hongliang and Liu Changping,“A hybrid License Plate Extraction Method Based On Edge Statistics and Morphology”, Proceedings of the 17th International

Conference on Pattern Recognition, Volume 2, pp.831-834, 23-26 Aug. 2004. [4] C.C. Lin and W.H. Huang,“Locating License Plate Based on Edge Features of Intensity and Saturation Subimages”, Second International Conference on Innovative Computing, Information and Control, 2007(ICICIC '07), pp. 227 – 227, 5-7 Sept. 2007 [5] S.L. Chang, L.S. Chen, Y.C. Chung, and S.W. Chen, “Automatic License Plate Recognition,” IEEE Transactions on Intelligent Transportation Systems, Vol.5, No.1, pp. 42-53, March 2004.

Ⅱ-204