Eye Correction Using Correlation Information - Springer

Eye Correction Using Correlation Information Inho Choi and Daijin Kim Department of Computer Science and Engineering Pohang University of Science and Technology (POSTECH) {ihchoi,dkim}@postech.ac.kr

Abstract. This paper proposes a novel eye detection method using the MCT-based pattern correlation. The proposed method detects the face by the MCT-based AdaBoost face detector over the input image and then detects two eyes by the MCT-based AdaBoost eye detector over the eye regions. Sometimes, we have some incorrectly detected eyes due to the limited detection capability of the eye detector. To reduce the falsely detected eyes, we propose a novel eye verification method that employs the MCT-based pattern correlation map. We verify whether the detected eye patch is eye or non-eye depending on the existence of a noticeable peak. When one eye is correctly detected and the other eye is falsely detected, we can correct the falsely detected eye using the peak position of the correlation map of the correctly detected eye. Experimental results show that the eye detection rate of the proposed method is 98.7% and 98.8% on the Bern images and AR-564 images.

1

Introduction

Face analysis and authentication problem are solved by three different methods [1] as holistic method [2], local method [2,3], and hybrid method [4]. The holistic method identifies a face using the whole face image and needs the alignment in images and the normalization using the facial features. The local method uses the local facial features on face such as eye, nose, and mouth, and needs to localize the fiducial points to analysis the face image. The hybrid method uses the both holistic features and the local features. Because eyes are the stable features on the face, they are used as reliable features for face normalization. So, it is very important to detect and localize the eyes in the applications of face authentication and/or recognition. Brunelli and Poggio [2] and Beymer [5] detected the eyes using the template matching. It used the similarity between the template image and the input image and is largely dependent on the initial position of the template. Pentland et. al. [6] used the eigenspace method to detect the eyes. The eigenspace method showed better eye detection performance than the template matching method. But its detection performance is largely dependent on the choice of training images. Kawaguchi and Rizon [7] detected the iris using the intensity and the edge information. Song and Liu [8] use the binary edge images. They include many technique such Y. Yagi et al. (Eds.): ACCV 2007, Part I, LNCS 4843, pp. 698–707, 2007. c Springer-Verlag Berlin Heidelberg 2007 

Eye Correction Using Correlation Information

699

as the feature template, the template matching, separability filter, and binary valley extraction, and so on and need the different parameters on the different database. So, these methods are not intuitive and not simple. So, we consider a more natural and intuitive method that detects the face region and then search the eyes in the subregion of the detected face. Freund and Schapire introduced the AdaBoost algorithm [9] and showed that it had a good generalization capability. Viola and Jones [10] proposed a robust face detection method using AdaBoost with the simple features and it provided a good performance to locate the face region. Fr¨ oba and Ernst [11] used the AdaBoost algorithm with a modified version of the census transform (MCT). This method is very robust to the illumination change and very fast to find the face. Among these AdaBoost methods, we choose the MCT-based AdaBoost training method to detect the face and the eye due to its simplicity of learning and high speed of face detection. However, sometimes we fail to detect the eyes such that the eyebrows or hairs. To reduce this problem, we propose a novel eye verification process whether the detected eye is true or false eye. The proposed eye verification method employs the MCT-based pattern correlation map. We can verify whether the detected eye is true or false depending on the existence of the noticeable peak in the correlation map. Using this property of the correlation map, we can correct the falsely detected eye using the peak position in the correlation map of the opposite eye. Assume that one eye is correctly detected and the other eye is falsely detected. Then, the correlation map of the correctly detected eye provides the noticeable peak that corresponds to the true location of the falsely detected eye.

2

Eye Detection Using MCT + AdaBoost

The Modified Census Transform (MCT) is a non-parametric local transform which modifies the census transform by Fr¨ oba and Ernst [11]. It is an ordered set of comparisons of pixel intensities in a local neighborhood representing which pixels have lesser intensity than the mean of pixel intensities. We present the eye detection method using the AdaBoost training with MCTbased eye features. In the AdaBoost training, we construct the weak classifier which classifies the eye and non-eye pattern and then construct the strong classifier which is the linear combination of weak classifiers. In the detection, we scan the eye region by moving a 12×8 size of the scanning window and obtain the confidence value corresponding to the current window location using the strong classifier. Then, we determine the window location whose confident value is maximum as the location of the detected eye. Our MCT-based AdaBoost training has been performed using only the left eye and non-eye training images. So, when we are trying to detect the right eye, we need to flip the right subregion of the face image.

700

3

I. Choi and D. Kim

Eye Verification

To remove false detection, we devise an eye verification whether the detected eye is true or false using the MCT-based pattern correlation based on symmetrical property of the human face [12]. 3.1

MCT-Based Pattern Correlation

The MCT represents the local intensity variation of several neighboring pixels around a given pixel in an ordered bit pattern. (See Fig. 1) Because the MCT value is non-linear, the decoded value of the MCT is not appropriate for measuring the difference between two MCT patterns. To solve this problem, we propose the idea of the MCT-based pattern and the MCT-based pattern correlation based on the Hamming distance that measures the difference between two MCT-based patterns. The MCT-based pattern P (x, y) at the pixel position (x, y) is a binary representation of the 3 × 3 pixels in the determined order, from the upper left pixel to the lower right pixel as P (x, y) = [b0 , b1 . . . b8 ],

(1)

where bi is a binary value that is obtained by the comparison function as ¯ y) + α, I(x , y  )), b3(y −y+1)+(x −x+1) = C(I(x,

(2)

¯ y) is the where x = {t|t ∈ {x − 1, x, x + 1}}, y  = {t|t ∈ {y − 1, y, y + 1}}, I(x,   mean of neighborhood pixels and I(x , y ) is the intensity of each pixel. b0 b1 b2 b3 b4 b5 b6 b7 b8 (a)

15 70 15 15 70 15 15 70 15

M BP −−−−−−−−−−−→ if I(x, y) > 33.3

010010010

(b)

Fig. 1. Examples of the MCT-based patterns

We propose the MCT-based pattern correlation to compute the similarity between two different MCT-based patterns. It is based on the Hamming distance that counts the number of positions whose binary values are equal between two MCT-based patterns. The MCT-based pattern correlation between image A and image B is defined as 1  ρ= ρx,y , (3) N x,y where N is the number of pixels in the image and ρx,y is the MCT-based pattern correlation at the pixel position (x, y) as ρx,y =

1 (9 − HammingDistance(PA(x, y), PB (x, y))), 9

(4)

Eye Correction Using Correlation Information

701

Fig. 2. Five face images with different illuminations Table 1. Comparison between the conventional image correlation with histogram equalization and the MCT-based pattern correlation

Face image Face image Face image Face image Face image Face image Face image Face image Face image Face image Mean Variance

1 1 1 1 2 2 2 3 3 4

and and and and and and and and and and

face face face face face face face face face face

image image image image image image image image image image

2 3 4 5 3 4 5 4 5 5

Conventional MCT-based pattern Image correlation correlation with HIST.EQ. 0.873 0.896 0.902 0.862 0.889 0.883 0.856 0.839 0.659 0.827 0.795 0.890 0.788 0.849 0.846 0.870 0.794 0.865 0.627 0.808 0.803 0.859 0.094 0.028

where PA (x, y) and PB (x, y) are the MCT-based patterns of the image A and B, respectively. Fig. 2 shows five different face images with different illuminations. Table 1 compares the conventional image correlation with histogram equalization and the MCT-based pattern correlation between two image pairs. The table shows that (1) the mean of the MCT-based pattern correlation is higher than that of the conventional image correlation and (2) the variance of the MCT-based pattern correlation is much smaller than that of the conventional image correlation. This implies that the MCT-based pattern is more robust to the change of illuminations than the conventional image correlation. Where the MCT-based pattern correlation map is built by sliding a detected eye over the eye region of the opposite side and computing the correlation value in term of the Hamming distance. 3.2

Eye/Non-eye Classification

The detected left and right eye patch can be either eye or non-eye, respectively. In this work, they are classified into eye or non-eye depending on the existence of a noticeable peak in the MCT-based correlation map as follows. If there is a

702

I. Choi and D. Kim

noticeable peak in the MCT-based correlation map, the detected eye patch is an eye. Otherwise, the detected eye patch is a non-eye. Since the eye detector produces two detected eye patches on the left and right eye subregions, respectively, we build two different MCT-based pattern correlation maps as – Case 1(2): Left(right) eye correlation map that is the MCT-based pattern correlation map between the detected left(right) eye patch and the right(left) subregion of the face image. We want to show how the MCT-based pattern correlation maps of the correctly detected eye and the falsely detected eye are different each other. Three images in Fig. 3 are taken to build the left eye correlation map (Case 1), where they are (a) a right eye subregion, (b) a flipped image patch of the correctly detected left eye, and (c) a flipped image patch of the falsely detected left eye (in this case, eyebrow), respectively. Fig. 3-(d),(e) show the correlation maps of the correctly detected left eye patch and the falsely detected left eye patch, respectively. As you see, two correlation maps look very different each other: the true eye patch produces a noticeable peak at the right eye position while the non-eye patch (eyebrow) does not produces any noticeable peak over the entire right eye subregion. From this fact, we need an effective way of finding a noticeable peak in the correlation map in order to decide whether the detected eye patch is eye or non-eye. In this work, we consider a simple way of peak finding based on two predetermined correlation values.

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0 100

0 100 80 60

50

40

80 60

50

40

20 0

(a)

(b)

(c)

0

(d)

20 0

0

(e)

Fig. 3. A typical example of the right eye subregion, the detected eye and non-eye in the left eye subregion

The proposed eye/non-eye classification method is given below. First, we rescale the correlation map whose the highest peak value becomes 1. Second, we overlay a peak finding window Wpeak with a size of w × h at the position with the highest value in the correlation map, where w and h are the width and the height of the detected eye patch. Third, we classify whether the detected eye Edetected patch is eye or non-eye according to the following rule as  eye if R < τ, (5) Edetected = non − eye otherwise,

Eye Correction Using Correlation Information

703

where τ and R are a given threshold and the high correlation ratio, which is defined by a ratio of the number of pixel positions whose correlation value is greater than a given threshold value ρt over the number of total pixel positions within the peak finding window Wpeak as R=

1 N



u+w/2



v+h/2

C(ρ(u , v  ), ρt ),

(6)

u =u−w/2 v  =v−h/2

where N is the number of total pixel positions of Wpeak and C is an comparison function as  1 if ρ(u , v  ) > ρt , C(ρ(u , v  ), ρt ) = (7) 0 otherwise.

4

Falsely Detected Eye Correction

After eye verification, we have four different classification results on the left and right eye regions: 1) eye and eye, 2) eye and non-eye, 3) non-eye and eye, and 4) non-eye and non-eye. In the first and fourth case, we succeed and fail to detect two eyes, respectively. In the fourth case, there is no way to detect the eyes. However, in the case of the second and third cases, we can correct the falsely detected eye as follows. In the second case, we can locate the falsely detected right eye using the peak position of the correlation map of the correctly detected left eye. Similarly, in the third case, we can locate the falsely detected left eye using the peak position of the correlation map of the correctly detected right eye. Fig. 4 shows an example of the falsely detected eye correction, where (a) and (b) show the eye region images before and after falsely detected eye correction, respectively. In Fig. 4-(a), A, B , and C represent the correctly detected left eye, the falsely detected right eye, and the true right eye, respectively. As you see in Fig. 4-(b), the falsely detected right eye is corrected well.

(a)

(b)

Fig. 4. An example of the falsely detected eye correction

5

Experimental Results

For the AdaBoost Training with MCT-based eye features, we used two face databases such as Asian Face Image Database PF01 [13] and XM2VTS Database

704

I. Choi and D. Kim

(XM2VTSDB) [14] and prepared 3,400 eye images and 220,000 non-eye images whose size is 12 × 8. For evaluating the proposed eye detection method, we used the Bern database [15] and the AR face database [16]. Because the accuracy of our face detection method is 100% in the Bern images and the AR image, we consider only the performance of the proposed eye detection method. As a measure of eye detection, we define the eye detection rate as reye =

N 1  di , N i=1

(8)

where N is the total number of the test eye images and di is an indicator function of successful detection as  1 if max(δl , δr ) < Riris , di = (9) 0 otherwise, where and δl and δr are the distance between the center of the detected left eye and the center of the real left eye, and the distance between the center of the detected right eye and the center of the real right eye, respectively, and Tiris is a radius of the eye’s iris. 5.1

Experiments in the Bern Images

Fig. 5-(a) and Fig. 5-(b) show some Bern images whose eyes are correctly and falsely detected by the strong classifier that is obtained by the AdaBoost Training with MCT-based eye features, respectively, where the boxes represent the

(a) Some examples of the (b) Some examples of the (c) An example of falsely correctly detected results falsely detected results detected eye correction Fig. 5. Some examples of results in Bern face database Table 2. Comparisons of various eye detection methods using the Bern face database Algorithms Eye detection rate (%) Proposed method 98.7% Kawaguchi and Rizon [7] 95.3% Template matching [7] 77.9% Eigenface method using 50 training samples [7] 90.7% Eigenface method using 100 training samples [7] 93.3%

Eye Correction Using Correlation Information

705

detected eye patches and the white circles represent the center of the detected eye patches. Fig. 5-(c) shows one example of falsely detected eye correction by the proposed eye correction method, where the left and right figures represent the eye detection results before and after falsely detected eye correction, respectively. Table 2 compares the detection performance of various eye detection methods. 5.2

Experiments in the AR Images

The AR-63 face database contains 63 images (twenty-one people × three different facial expressions) and the AR-564 face database includes 564 images (94 peoples × 6 conditions (3 different facial expressions and 3 different illuminations)). Fig. 6-(a) and Fig. 6-(b) show some AR images whose eyes are correctly and falsely detected by the strong classifier that is obtained by the AdaBoost Training with MCT-based eye features, respectively, where the boxes represent the detected eye patches and the white circles represent the center of the detected eye patches.

(a) Some examples of the correctly de- (b) Some examples of the falsely detected tected results results Fig. 6. Some examples of results in AR-564 face database

(a) Results of the falsely detected

(b) Results of the correction

Fig. 7. Three examples of the falsely detected eye correction

Table 3. Comparisons of various eye detection methods using the AR face database Algorithms Proposed method Song and Liu [8] Kawaguchi and Rizon [7]

AR-63 AR-564 98.4% 98.8% 96.8% 96.8% 96.8% -

706

I. Choi and D. Kim

Fig. 7 shows four examples of falsely detected eye correction by the proposed eye correction method, where (a) and (b) represent the eye detection results before and after falsely detected eye correction, respectively. Table 3 compares the detection performance of various eye detection methods. As you see, the proposed eye detection method shows better eye detection rate than other existing methods and we the improvement of eye detection rate in the case of AR-564 face database is bigger than that in the case of AR-63 face database. This implies that the proposed eye detection method works well under various conditions than other existing eye detection methods.

6

Conclusion

We proposed a eye detection method using the MCT-based pattern correlation. The eye detection method can produce the false detection near the eyebrows or the boundary of hair and forehead in particular. When the existing eye detection method detects the eye in just one subregion, then it does not improve the eye detection rate. To overcome this limitation, we proposed the eye verification and falsely detected eye correction method based on the MCT-based pattern correlation. The MCT-based pattern correlation is based on the Hamming distance that measures the difference between two MCT-based patterns ,where the MCT-based pattern is a binary representation of the MCT. Also, the proposed MCT-based pattern is robust to the illumination changes. To verify detected eye, we proposed the eye/non-eye classification method which classifies into eye or non-eye depending on the existence of a noticeable peak in the MCT-based pattern correlation map. The proposed falsely detected eye correction method uses the peak position in the MCT-based pattern correlation map to correction of the falsely detected eye which is verified by the proposed eye/non-eye classification method. It improves the eye detection rate of the proposed eye detection method. The experimental results show that a eye detection rate of 98.7% and 98.8% can be achieve on the Bern images and AR-564 database. The proposed eye detection method works well under various conditions than other existing eye detection methods.

Acknowledgements This work was partially supported by the Korea Science and Engineering Foundation (KOSEF) through the Biometrics Engineering Research Center (BERC) at Yonsei University. Also, it is financially supported by the Ministry of Education and Human Resources Development (MOE), the Ministry of Commerce, Industry and Energy (MOCIE) and the Ministry of Labor (MOLAB) through the fostering project of the Lab of Excellency.

Eye Correction Using Correlation Information

707

References 1. Tan, X., Chen, S., Zhou, Z., Zhang, F.: Face recognition from a single image per person: A survey. Pattern Recognition 39, 1725–1745 (2006) 2. Brunelli, R., Poggio, T.: Face recognition: features versus templates. IEEE Transaction on Pattern Analysis and Machine Intelligence 15, 1042–1052 (1993) 3. Lawrence, S., Giles, C., Tsoi, A., Back, A.: Face recognition: a convolutional neuralnetwork approach. IEEE Transaction on Neural Networks 8, 98–113 (1997) 4. Martinez, A.: Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Transaction on Pattern Analysis and Machine Intelligence 24, 748–768 (2002) 5. Beymer, D.: Face recognition under varying pose. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 756–761. IEEE Computer Society Press, Los Alamitos (1994) 6. Pentland, A., Moghaddam, B., Starner, T.: View-based and modular eigenspaces for face recognition. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 84–91. IEEE Computer Society Press, Los Alamitos (1994) 7. Kawaguchi, T., Rizon, M.: Iris detection using intensity and edge information. Pattern Recognition 36, 549–562 (2003) 8. Song, J., Chi, Z., Li, J.: A robust eye detection method using combined binary edge and intensity information. Pattern Recognition 39, 1110–1125 (2006) 9. Freund, Y., Schapire, R.: A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 14, 771–780 (1999) 10. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE International Conference on Computer Vision and Pattern Recognition, pp. 511–518. IEEE Computer Society Press, Los Alamitos (2001) 11. Froba, B., Ernst, A.: Face detection with the modified census transform. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 91–96 (2004) 12. Song, Y.J., Kim, Y.G., Chang, U.D., Kwon, H.B.: Face recognition robust to left/right shadows; facial symmetry. Pattern Recognition 39, 1542–1545 (2006) 13. Je, H., Kim, S., Jun, B., Kim, D., Kim, H., Sung, J., Bang, S.: Asian Face Image Database PF01, Technical Report. Intelligent Multimedia Lab, Dept. of CSE, POSTECH (2001) 14. Luettin, J., Maˆıtre, G.: Evaluation Protocol for the Extended M2VTS database (XM2VTSDB), IDIAP Communication 98-05. In: IDIAP, Martigny, Switzerland, pp. 98–95 (1998) 15. Achermann, B.: The face database of University of Bern. Institute of Computer Science and Applied Mathematics, University of Bern (1995) 16. Martinez, A., Benavente, R.: The AR Face Database, CVC Technical Report #24 (1998)