Farsi License Plate Detection based on Element

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

Farsi License Plate Detection based on Element Analysis and Characters Recognition Mehran Rasooli Sedigheh Ghofrani Electrical Engineering Electrical Engineering Department, Islamic Azad Department, Islamic Azad University, South Tehran University, South Tehran Branch, Tehran, Iran. Branch, Tehran, Iran. [email protected] [email protected]

Emad Fatemizadeh Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran. [email protected]

Abstract In this paper, a safe and powerful method is presented which can detect and identify Farsi license plate irrespective of distance (how far a vehicle is), rotation (angle between camera and vehicle), and contrast (being dirty, reflected, or deformed). In addition, more than one car can be existed in an image. The proposed method extracts edges and then determines the candidate regions by using adaptive image enhancement and applying window movement. Finally by region elements analysis, the license plates are detected. The region elements analysis is working according to the plate geometric structure, continuity and parallelism. After detecting license plates, we estimate rotation angle and try to compensate it. In order to identify a detected plate, every character should be recognized. For this purpose, we present 53 features and use them as input to artificial neural network classifier.

1. Introduction Rapid advancement of technology in artificial intelligence and computer science knowledge and then feel the need to search and security automated systems are because of the appearance of intelligent systems based on image processing and spread this knowledge. One of these intelligent systems is license plate recognition (LPR) system. Usually, LPR systems consist of three main parts: license plate detection, character segmentation, and character recognition. License plate detection is the most important and difficult stage. During recent years, several methods have been proposed for detecting license plate in an image. Wie et al. and Kim et al. [1] - [2] used color processing where the unique color or color combination between the license plate and vehicle body or between the plate background and text color has been considered as the key feature to detect the license plate location. But these methods are not robust to the different environments. Zheng et al., Jia et al. and Ming et al. [3] - [6] used edge analysis. The edge features of a car image are very important and edge density can be used to successfully detect a number plate location due to the characteristics of the number plate. To improve the edge image, eliminate the highest and lowest portions of the edge density to simplify the whole image. But some of the plate region identity will be lost in this method. Park et al. and Wang et al. used neural networks and other classifier for locating the license plate [7] – [8]. These methods can be used as filters for analyzing small windows of an image and deciding whether each window contains a license plate. A post processing combines these filtered images and find the location of the license plates in an image. Anyway, most of existing methods fail in real scenario or strongly depending on input images. In this paper, we have

65

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

considered the content plate characteristics and present a reliable algorithm to detect multiple license plates under various photographs and different conditions. There are numerous algorithms for character recognition such as: statistical classifiers, computational intelligence architectures, and pattern matching techniques. Although there are various types of classifiers for recognizing license plates characters, artificial neural network (ANN) is a well known classifier. ANN needs the value features as input. In this paper, we suggest 53 features and use multilayer perceptron ANN to recognize the license plate characters. The rest of this paper is organized as follows: extracting edges in section 2, extracting candidate regions in section 3, analyzing region elements in section 4, detection experimental results in section 5, compensating rotation in section 6, recognizing characters in section 7, and finally recognition experimental results is presented in section 8. At last, in section 9 we draw some conclusions.

2. Extracting edge In order to extract edges, two different procedures have done. It means image enhancement and image edges detection. They are described in detail in following. 2.1 Image enhancement The advantage of license plate detection system is performing in photographed different conditions, such as low contrast image, dirty license plate, or affected plate by reflection. Therefore, image enhancement is necessary, and for this purpose, the below high frequency emphasis mask is used:

C C C C 9 × ( A − 1) C   C C C We denote that the usual used values for the two parameters C , A of the above mask are C = 1 , and A > 1 . In this work we have obtained them adaptively according to the each input image as follows: A = 1 + (1 − δ ) −A C= 2

(1) (2)

where δ is the standard deviation.

2.2. Image edge detection It seems that, the density of vertical edges is more than horizontal edges for the most characters of any Farsi license plates such as others [4]-[6]. So we have used just the vertical Sobel mask and increased the speed of running algorithm. Figure 1 illustrates the four different pose original images and figure 2 shows extracting edges results.

66

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

3. Extracting candidate regions After detecting edges in section 2, the candidate regions are determined via three steps; it means window movement, region modification, and candidate regions selection as well. Every procedure will be explained in detail in following sub sections. 3.1. Window movement In window movement, at first we select the smallest window size that can be a license plate and move a window on all of the rows and columns in image. Then we increase window size and move on the rows and columns again. The window movement procedure stops when the window size is bigger than the image size. During every window movement, the considered property for license plate candidate is controlled and the contents of a window that could not satisfy the proposed merit would be removed or discard. We apply edge density to determine the presence of license plate in a window, it is obtained as:

∑ ∑ edge − density = x

y

i =1

j =1

| ed ( i, j) |

x×y

(3)

where x and y are window size, and ed (i, j) is edge value which is obtained in section 2. The windows that their edge densities are greater than predefined threshold are selected as license plate probable region and the remaining areas are removed. About 40% of Farsi license plates area is empty and 60% is characters area. So, we have chosen the threshold value equals to 55. Our experimental results confirm the suitability of the selected threshold. Figure 3, shows the window movement output images. 3.2. Candidate regions modification We know license plate includes characters, so we consider this feature to discard wrong candidate regions. First, we perform closing and opening operators on each remaining regions. Then, we subtract the opened and closed images [9]. So the smoothed area is de-emphasized and peaks and valleys are highlighted. Therefore the license plate region can be distinguished from surrounding by using the proper threshold as well. Anyway, Performing opening and closing needs structure elements. Since this Process performs on each of the remaining regions. We determine the structure element for each region adaptively. Let the length and width of a structural element are Y and X respectively, then the structure element for each region, defined by the below equation:  X = round(0.1 × mx)   Y = round(0.2 × my)

(4)

where mx and my are average of active pixels for all of row and column in order in every region. Now, by subtracting the closed and opened images, the characters area in result image are brighter than surrounding. In order to distinguish the character areas those belong to the license plate better, we have chosen the average of difference image as the threshold value for each area separately. Figure 3 shows the result of candidate region modification. As it can be seen, in comparison with Figure 4, some regions those have been selected by mistake now are discarded.

67

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

a)

b)

c) d) Figure 1. Original images are chosen from our database. a) three cars, b) big angle between license plate and camera, c) short distance and rotation, d) low contrast and untidy license plate.

3.3. Candidate regions selection Although the aspect ratio (proportion of length to width) of standard Farsi license plate is 5, it is changed if camera and car make an angle. In this paper, the maximum angle is supposed to be 70 degree. Now, according to this maximum Angle, we suggest using equation (5) to obtain the minimum and maximum value for the aspect ratio:  Minimum aspect ratio = 5 × cot( maximum angle )   Maximum aspect ratio = 5 / cot( maximum angle )

(5)

cos( x ) . So, the maximum and minimum aspect ratios of Farsi license plates sin( x ) regions are 13.7 and 1.8 respectively. In addition, we consider the smallest value for the Farsi license plate area equal to 0.55 and the smallest size equal to 10×50 pixels. So the candidate license plate should have at least 275 (0.55×10×50) pixels. In this paper, we consider these two features (aspect ratio and the smallest plate size) to discard some more regions before arriving in element analysis, Figure 4 shows the result.

where cot( x ) =

68

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

a)

b)

c) d) Figure 2. Extracting edges by using high frequency emphasis mask (enhancement) and vertical Sobel mask.

4. Analyzing region elements As it can be seen in Figure 4, although many wrong candidate regions are removed during execute procedure in section 3, these are some regions in addition to the license plate which should be omitted. So we employ the region elements analysis. In general, it analyzes every candidate regions to decide whether there are any character strings. Region elements analysis includes four stages: binarization, character geometric analysis, character continuity analysis, and at last character parallelism analysis. 4.1.Binarization It is necessary to have a threshold to convert any gray image to binary. Since the general threshold makes losing or merging elements, the threshold value must be determined for each region separately. We use the modified Niblack method [10] to obtain the threshold value for a region as follows: Th = m + (sign( 0.1 − σ ) × k × σ )

(6)

where Th is the desired threshold value, m and σ are average illumination and standard deviation of a region, and k is a constant that considered to be -0.2. In this way, every candidate regions are converted to binary adaptively. After image binarization, the 8 neighbor connect component algorithm are applied for image segmentation and extracting the elements. In this method scroll the image pixels from left

69

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

to right and from top to bottom to extract the connected pixels. Each group of connected pixels is

a)

b)

c) d) Figure 3. In window movement procedure, many regions in addition to license plate are candidate. identified as one element in selected region. In following subsections, we analyze the obtained elements and evaluate the similarity of these elements with license plate characters. Although, there are 32 characters in Farsi language, just 16 characters are used to identify the Farsi license plates. These characters are written in table 1. We have also categorized license plates to three groups. It means private plates those have an owner, public plates those are used for public transportation, and governmental plates those belong to the government. These three different license plates may use different characters and colors as it can be seen in table 1. 4.2. Character geometric analysis This analysis considers two features, character aspect ratio and active pixels per character area (APR). The highest aspect ratio (width to height) in the Iranian standard license plates is related to the characters "‫ " ب‬and "‫ " ت‬and the lowest aspect ratio is related to number "1". These ratios in the worst possible conditions are 3 and 0.1 respectively. So each element can be considered to be a character if it's computed aspect ratio is in interval [0.1 3]. Character area obtains with multiplying major axis by minor axis of character. This rectangular embed on character. On the other hand, active pixels are pixels that formed the character. So for the second feature, we define the active pixels per character area or active pixels ratio (APR) as: APR =

70

active − pixels major − axis × minor − axis

(7)

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

We have considered the Farsi license plate characters in different conditions and determined the maximum and minimum value for APR equal 0.75 and 0.2 in order.

a)

b)

c) d) Figure 4. After region modification, some wrong candidate regions are discarded.

4.3. Character continuity analysis Obviously, those elements belong to license plate set one after another. In other word there is continuity between elements (that belongs to the plate) with before and after element or the distance between license plate elements are not too much. Using this property as a feature, we use Euclidean distance for the centers of mass elements as follows: ( x i+1 − x i ) 2 + ( y i +1 − y i ) 2 < 1.1 × major − axis

(8)

( x i − x i−1 ) 2 + ( y i − y i−1 ) 2 < 1.1 × major − axis

(9)

where x i and y i are width and height of mass centers, and major-axis is the biggest length (horizontal or vertical length) of element. The structure of Farsi characters require that each center of mass element locates between of two neighbor center of mass as smaller than 1.1 its major axis. For the first element just inequality (8) is used and for the last element just inequality (9) is applied, whenever for other elements the both (8) and (9) are considered. Each element which does not satisfy the requirements would be removed from region. The process is done iteratively until all remained elements satisfy the continuity requirements. 4.4. Character parallelism analysis In this section, we evaluate the height and width of region elements. For Farsi characters, the height and width interval ratios via different situations in order are: [0.4 2.5].

71

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

Any not parallelism element with other elements should be removed from region. Parallelism analysis is also an iterative process. At the end of the process, all the remaining characters will be similar in width and height. In addition, we notice that the number of Farsi license plates characters is

a)

b)

c) d) Figure 5. Results of candidate regions selection. There already exist some other regions except license plate. They would be omitted by using the region elements analysis. Table 1. Three types of Farsi license plates and their characters and colors. Plate type private public governmental

Characters ،‫ ص‬،‫ ط‬،‫ س‬،‫ د‬،‫ ج‬،‫ب‬ ‫ ي‬،U ،‫ و‬،‫ ن‬،‫ م‬،‫ ل‬،‫ق‬ ‫ع‬،‫ت‬ \]‫ا‬

Colors white-black yellow-black red-white

8, so any region which includes less or more than 8 elements can not be a license plate and must be removed.

5. Detection experimental results In general, we have defined some elements features to distinguish those belong to Farsi license plate from other. Figure 6 shows the result of region elements analysis. As it can be seen, although these four images are captured in different conditions, our proposed algorithm is able to detect the license plates as well. Any way there is not a standard bank for Farsi license

72

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

plate, so we ourselves prepared the database which includes 300 images. As images are captured in different conditions, we split the database to six categories based on: camera distance (short and normal), camera angle (low and high), low resolution (dirty, reflection, and deformed plates), and multiplicity of plates in image. We have run the explained algorithm and

a)

b)

c) d) Figure 6. The results of region elements analysis for the four selected image. It is seen even for low contrast image (d), the algorithm succeeded to detect the license plate correctly. reported the accuracy for Farsi license plate detection in table 2. We mention that the results of license plate detection are reported in table 2, belong to private cars. As we have mentioned in section 4, Farsi license plates can be categorized to 3 different types. Before continuing to recognize a plate characters, we specify these three license plates in detail more. Basically, the color of their plate background and characters are different. They (background and characters colors) for private, public, and governmental plates are white-black, yellow-black, and redwhite respectively (table 1). Figure 7 shows samples of these Farsi license plates.

a) private

b) public

c) governmental

Figure 7. Three type of Farsi license plate. Although we have processed only the private cars, the same procedure can be done for public or governmental plate detection. We are able to specify the type of detected plate by obtaining the license plate background color and notifying to table 1.

73

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

6. Compensating rotation Extracted plates may have an angle with the horizontal or even vertical line. If we estimate the rotation angle, then compensating is possible. So we label the all elements of a detected plate according to use 8 neighbor connect component which was explained in section 4, then we compute the central mass of each element and compute angle between every side by side elements via using equation (10):

a) the estimated angles are 3,1, and 2 degree in order from top to bottom

c) the estimated angle is 28 degree

b) the estimated angle is 17 degree

d) the estimated angle is 2 degree

Figure 8: The detected plates are shown and estimated angles are written for each one.

a)

b)

c) d) Figure 9: The detected plates after rotation compensating, the plates are ready to be recognized. angle = arctg(

vertical − difference ) horizontal − difference

(10)

where vertical – and horizontal – difference refer to the vertical and horizontal mass difference of the two side by side elements (the reference point with coordinate (0, 0) are considered to be at left-up hand of a detected plate). As we know, any license plate includes just 8 characters, so after computing 7 angles between elements, we determine the average rotation angle and we do rotation counter clockwise. Now the detected plates are shown in figure 8, and the computed angles for every plate are also written. Figure 9 shows the license plates that we compensated

74

International Journal of Signal Processing, Image Processing and Pattern Recognition Vol. 4, No. 1, March 2011

them. We believe this compensation would improve our accuracy for executing the next part; character recognition.

7. Recognizing characters Character recognition means specify exactly letters and numbers which are in a region or in this work detected license plate. For this purpose, at first we should extract characters (the method described in section 4) and then recognize every 8 characters that there are in Farsi license plate.

Table 2. Accuracy of the proposed algorithm for license plate detection. As it can be computed easily, the number of images that the method fails is just 6 while the total numbers of processing images are 300.

Different Situations Short (3 m &