2013 IEEE Student Conference on Research and Development (SCOReD), 16 -17 December 2013, Putrajaya, Malaysia
Automated Eye Blink Detection and Tracking Using Template Matching Muhammad Awais, Nasreen Badruddin, Micheal Drieberg Centre for Intelligent Signal and Imaging Research (CISIR) Department of Electrical and Electronic Engineering, Universiti Teknologi PETRONAS, Malaysia.
[email protected],{nasreen.b, mdrieberg}@petronas.com.my considered to be the more reliable method for the analysis of eye blink features as compared to camera as it gives more information about the eye blink. However, the use of EOG may cause discomfort to the patient because at least three electrodes need to be placed on the head [6] and this may be restrictive. Camera is considered to be more comfortable as compared to EOG especially when it is used by people with disabilities. In this paper, we present a new method for eye blink detection in which eye template is created after face detection. The eye template is then correlated with each frame to find the correlation scores. Based upon correlation scores the eye blinks are detected. The organization of the paper is as follows. Section II briefly describes the existing techniques used to detect eye blinks. Section III discusses how face detection, eye detection and eye blink detection are performed. Section IV presents results and discussion on our findings. Section V describes the conclusions based upon the results.
Abstract— Eye blink detection is considered to be one of the most reliable sources of communication in modern human computer interaction (HCI) systems. This paper proposes a new method for eye blink detection using template matching and similarity measure. In order to minimize the false detection due to changing background in the video frame, face detection is applied before extraction of the eye template. Golden ratio concept is introduced for robust eye detection and is followed by eye template creation for tracking. Eye tracking is performed by template matching between template image and surrounding region. The normalized correlation coefficient is computed for successful eye tracking. Eye blink detection is performed based upon the correlation score as the score changes significantly whenever a blink occurs. The proposed system provides an overall precision of 92.8% and overall accuracy of 99.6% with 0.1% false positive rate in different experimental conditions. Keywords—Eye Blink detection, Template matching, HCI.
I. INTRODUCTION In recent years eye blink detection techniques are widely used in human computer interaction (HCI) systems. Eye blink detection systems are very useful for people who are unable to control computers and electronic devices due to impairment or complete loss of motor functions. Chau et al [1] developed an eye blink detection system for the paralyzed people who are unable to move their body parts except their eyes. This system takes the input from the user’s eye blinks, such as the eye blink duration to produce a mouse click. Grauman et al [2] also used eye blink features as a source of communication between computer and human with disabilities. Human beings can easily distinguish between fake images and live face based upon the liveness clues like eye movement, head movement, facial expression etc. However this process becomes very difficult when a computer is used. Therefore, eye blinks are used for liveness detection against spoofing. Spoofing is the process to defeat the biometric identification system using fake images and finger prints [3]. Eye blink detection algorithm can also be used for driver drowsiness detection [4]. In [5], the researchers showed that eye blink duration increases as a subject moves from alert state to drowsy state. The eye blink duration for alert state varies from 300ms to 350ms whereas it increases significantly beyond 400ms for drowsy state. Mainly, there are two common methods to detect eye blinks, i.e. using a camera or electrooculogram (EOG). EOG is
978-1-4799-2656-5/13/$31.00 ©2013 IEEE
II. RELATED WORK There are many existing techniques which are used to detect eye blinks and each has its own pros and cons. In [7] the authors proposed an eye blink detection system that uses scale invariant feature transform (SIFT) for feature extraction and a Graphics Processing Unit (GPU) to speed up the computations. A Standard CMOS black and white camera (640x480 pixels) was used for Data acquisition. A real time eye tracking system using USB camera has been proposed in [1]. The authors use motion analysis techniques and correlation for eye blink detection and tracking. The experiment yields an overall good accuracy for blink detection but the system is unable to detect the eyes if the distance between the subject and camera is more than 2 feet. Eye blink detection system for drivers is developed by Artem et al [8]. Skin and facial color segmentation based upon neural network is proposed for blink detection and tracking. CCD and CMOS cameras are used for system testing. Eyes and iris positions are monitored continuously in order to compute blink frequency and duration. III. PROPOSED METHOD The proposed method for eye blinks detection is shown in Fig.1. Face detection is performed before eye detection in order to avoid false detections. Furthermore, face detection is also
79
Fig. 3. Eye pair detection using go olden ratio concept Fig. 1. Proposed technique flow diagram
very important in varying background scenaarios when direct eye detection is difficult. A. Face Detection The Viola Jones face detector [9] is used inn the first step to detect the face. It's an object detection approoach resulting in very high detection rate in real time. It consists of three main steps. First, an intermediate image is introduced in the form of integral image which takes the pixels sum to speed up the feature extraction part instead of using rectanggle features which are considered to be slow. Secondly, critiical features are extracted from a large set using AdaBoosst algorithm and results in a very accurate classifier. Finallly, all complex features are combined in a cascaded wayy to discard the unwanted regions of the image that results in vvery accurate face detection. Viola Jones is implemented in the database and the detected face is shown in Fig. 2. The rectangullar box shows the region of the detected face.
Fig. 4. Samples from JZU eye blinks database [3]. (a) Close eyes template image and (b) Open eyes template image.
other eye is also detected as show wn in Fig. 3. The point at golden ratio of height and width giv ves the left eye location and by symmetry, the right eye location n is calculated. In this way the eye pair is successfully detected.. An eye template is created from m the first frame. The eye template from the first frame can be of open eyes or closed eyes therefore, our system consiiders both possibilities to increase system reliability. Close eye e template and open eye template are shown in Fig. 4(a) and (b) respectively. C. Eye Blink Detection and Trackin ng Eye tracking is very essential in order to detect eye blinks accurately. Template matching is very v important in order to track the eyes and achieve high acccuracy. Eye tracking also permits some degree of freedom to the t user to move the face. Grauman et al [2] used normalizeed correlation coefficient to achieve eye tracking. The equation used u to compute correlation for each frame is given as [2]
B. Eye Pair Detection and Template Creationn Eye pair detection is the second step after face detection to detect the eyes. We already know that the eye pair is located in the upper region of the face so the golden ratiio concept [10] is introduced here in order to detect the eye pair. Golden ratio first detects one eye and then, by using facce symmetry, the
R(x, y) =
¦ ¦ ¦ ¦
h
w
y'=0
x'=0
h
w
y'=0
x'=0
T(x', y')I (x + x', y + y') 2
T(x', y')
¦ ¦ h
w
y'=0
x'=0
2
(1)
(I (x + x', y + y')
ഥ , ൫ ̡ ǡ ǡ ̡ ൯ ൌ where ൫ ̡ ǡ ̡ ൯ ൌ ̡ ൫ ̡ ǡ ̡ ൯ െ ̡ ̡ ̡ ̡ ̡ ̡ ൫ ǡ ൯ െ ҧ , ൫ ǡ ൯and ̡ ൫ ̡ ǡ ̡ ൯ are the brightness of the pixels of so ource images and template ഥare th images respectively. ҧ and he average pixel values in
Fig. 2. Samples from JZU eye blinks databasee [3]. (a) Face detection in open eye image (b) Face detection in cloose eye image
80
source and template image respectively. Furthher description of (1) can be found in [2]. In this work, the search region is the areaa around the eye template in the current frame. R ( x, y ) proovides the match between the eye template and the eye searcch region. In this way, values of (x, y) are updated at the pooint of maximum correlation. The eye search regions are updatted 30 times in a second to track the eyes accurately. The eye tracking process and transition from eyes open to closed annd to open again during a blink is shown in Fig. 5. The ssearch region is represented by the black rectangular box aand the template image is represented by blue dashed lined rectaangular box. Successful eye tracking leads towards acccurate eye blink detection and if eye tracked is lost or eyee tracking is not working properly then the false detection rate and missed blink rate increase rapidly. Eye blink detection is baased upon change in normalized correlation score. This leadss to decrease in correlation score in case of open eye templatee and increase in correlation score in case of close eye template.
Fig. 6. Samples from JZU eye blinks database d [3]. (a) Frontal view wearing black rim glasses (b) Frontal vieew without glasses (c) Frontal view wearing thin rim glasses and (d) Up pward view without glasses.
correlation computed from (1) is useed to detect eye blinks. The eye template image acquired from m the first frame can be of open eyes or closed eyes. If the tem mplate image is of open eyes then correlation score will be high in open eyes condition and falls down during a blink. Thiss issue of increasing and decreasing peaks during eye blin nk has been resolved by computing the derivative of correelation scores in order to
IV. RESULTS AND DISCUSSION N We have acquired a publicly available eye blink video database [3], consisting of 80 videos from 20 people including males and females. The data was collected uusing a Logitech camera with frame rate of 30fps and pixel sizee of 320x240. For each individual, four videos, of five secondss, were recorded. These four videos were recorded in differennt conditions and Fig. 6 shows images of various experimental ssetup of database. Fig. 6(a) represents the frontal view of subjeect wearing black frame glasses, Fig. 6(b) represents the frontall view of subject without glasses, Fig. 6(c) represents the frontaal view of subject wearing thin rim glasses and Fig. 6(d) repressents the upward view of subject face without glasses. The num mber of blinks for each clip varies from one to six. The proposed algorithm works very welll for eye blink detection when there is minimal motion annd we tested our algorithm on the JZU eye blinks database [3]. Normalized
1
normalized correlation coefficient score
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
20
40
60 80 number of Fram mes
100
120
140
Fig. 7. Correlation scores with open eyee template 1
rate of change of correlation score
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
Fig 5. Samples from JZU eye blinks database [3]. S Search region is shown by big rectangle and template image is shhown by small dotted rectangle showing eye tracking in (a) Oppen eyes (b) Closing eyes (c) Closed eyes and (d) open eyes afterr blink.
0
20
40
60 80 number of framess
100
120
Fig. 8. Eye blinks detection using open eye e template
81
140
close state, decreases dominantly. After the computation of correlation scores, all the blinks are successfully detected by taking the absolute derivative as shown in Fig. 8. Correlation scores for the closed eyes template are shown in Fig. 9 and eye blinks are detected using derivative of correlation scores as shown in Fig. 10. Correlation scores of closed eye template with image shows completely opposite response to that of open eye. Correlation score rises significantly during a blink as shown in Fig. 9 and eye blinks are successfully detected by taking the absolute derivative as shown in Fig. 10. Eye blink detection results for frontal view wearing black glasses, without glasses, thin rim glasses and upward view without glasses are shown in Table I. The proposed system works very well in various scenarios, i.e., with or without glasses. The performance parameters [10] such as precision, recall, false positive rate (FPR) and accuracy are calculated using TP, FP, TN and FN as shown in (2–5). Performance metrics shows very promising results as shown in Table I, where TP=True Positive, FP=False Positive, TN=True Negative, FN=False Negative.
1
normalized correlation coefficient score
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0
20
40
60 80 number of Frames
100
120
140
Fig. 9. Correlation scores with close eye template 1
rate of change of correlation score
0.9 0.8 0.7 0.6 0.5
Ρrecision =
0.4
TP TP + FP
(2)
0.3 0.2
Re call =
0.1 0
20
40
60 80 number of frames
100
120
(3)
140
FPR =
Fig. 10. Eye blinks detection using close eye template
detect eye blinks. Absolute values of derivative are taken and then peak detection algorithm is applied for eye blinks detection. Fig. 7 shows the correlation graph of an image when the template image is of open eyes and Fig. 8 shows the detected blinks based upon correlation scores in Fig. 7. Horizontal axis shows the number of frames while vertical axis shows the correlation scores in Fig. 7. Three eye blinks, shown in Fig. 7 represents the significant decrease in correlation scores during an eye blink because the template image is of open eyes and its correlation with an eye blink which is eyes TABLE I.
TP TP + FN
FP FP + TN
Accuracy =
(4)
TP + TN TP + TN + FP + FN
Our proposed method gives better results as compared to the experimental results of previous work [4]. Comparative analysis of previous work and our proposed method is shown in Table II. Proposed method increases the overall system accuracy, precision, recall and reduces the FPR significantly.
PERFORMANCE METRICS AND CONFUSSION MATRIX OF DIFFERENT EXPERIMENTAL SETUPS
Confusion Matrix
Experimental Scenarios
(5)
Performance Metrics
TP
FP
TN
FN
Precision
Recall
FPR
Accuracy
Frontal view wearing black frame glasses
67
2
2860
5
97.1%
93.1%
0.1%
99.8%
Frontal view without glasses
56
5
2875
5
91.8%
91.1%
0.2%
99.7%
Frontal view wearing thin frame glasses
61
7
2855
13
89.7%
82.4%
0.2%
99.3%
Upward view without glasses
64
5
2712
4
92.8%
94.1%
0.2%
99.7%
All scenarios
248
19
14157
27
92.8%
90.2%
0.1%
99.6%
82
TABLE II.
[10] C. D. Green, “All that glitters: a review of psychological research on the aesthetics of the golden section,” PERCEPTION, vol. 24, no. 8, pp. 937–968,1995.
COMPARATIVE ANALYSIS OF PREVIOUS WORK WITH OUR PROPOSED METHOD
Performance Parameters
Taner al [4]
et
Our Proposed method
Improvement
Accuracy
94.8%
99.6%
4.8%
Precision
90.7%
92.8%
2.1%
Recall
71.4%
90.2%
18.8%
1%
0.1%
0.9%
False Positive Rate(FPR)
V. CONCLUSION AND FUTURE WORK Eye blink detection is widely used in many applications. We propose a new approach to detect eye blinks. Prior knowledge of face detection and golden ratio computation makes eye detection process very robust. Techniques like motion analysis are not very reliable for eye detection because background or environment changes and varying illumination condition can lead to false eye detection. Using eye tracking, the system is able to detect the eyes, even if the user moves his head slightly. The overall accuracy of the system is very promising and works very well with a frame rate of 30fps. Proposed technique is showing very promising results as compared to the previous work by achieving an overall accuracy of 99.6% with false positive rate of 0.1%. The study will be beneficial and enable people with disabilities, who can only blink their eyes, to perform different day to day activities. Future work of proposed system will focus more on eye blink duration analysis as duration analysis will be used for driver drowsiness detection. REFERENCES [1] [2] [3] [4] [5] [6] [7]
[8] [9]
M. Chau and M. Betke, “Real Time Eye Tracking and Blink Detection with USB Cameras,” Boston University Computer Science, vol. 2215, no. 2005–12, pp. 1–10, 2005. K. Grauman, M. Betke, J. Gips, and G. R. Bradski, “Communication via eye blinks - detection and duration analysis in real time,” in Proc. IEEE Comput. Soc., vol. 1, pp.1010–1017, 2001. G. P. G. Pan, L. S. L. Sun, Z. W. Z. Wu, and S. L. S. Lao, “Eyeblinkbased Anti-Spoofing in Face Recognition from a Generic Webcamera,” in Proc. IEEE int. conf. on Comput. Vision, pp. 1–8, 2007. T. Danisman, I. M. Bilasco, C. Djeraba, and N. Ihaddadene, “Drowsy driver detection system using eye blink patterns,” in Proc. Int. Conf. on Machine and Web Intelligence, pp. 230–233, 2010. Y. S. Kim, H. J. Baek, J. S. Kim, H. B. Lee, J. M. Choi, and K. S. Park, “Helmet-based physiological signal monitoring system.,” European J. of Applied Physiology, vol. 105, no. 3, pp. 365–372, 2009. A. Picot, A. Caplier, and S. Charbonnier, “Comparison between EOG and high frame rate camera for drowsiness detection,” in Proc. IEEE Applicat. of Comput. Vision, pp. 1-6, 2009. M. Lalonde, D. Byrns, L. Gagnon, N. Teasdale, and D. Laurendeau, “Real-time eye blink detection with GPU-based SIFT tracking,” in Proc. 4th Canadian Conf. on Comput. & Robot Vision (CRV), pp. 481–487, 2007. A. A. Lenskiy and J.S. Lee, “Driver’s eye blinking detection using novel color and texture segmentation algorithms,” Int. J. of Control Automation and Syst., vol. 10, no. 2, pp. 317–327, 2012. P. Viola and M. Jones, “Rapid object detection using a boosted cascade of simple features,” in Proc. IEEE Comput. Soc. Conf. on Comput. Vision and Pattern Recognition (CVPR), vol. 1, pp. 511-518, 2001.
83
