Occlusion Detection and Image Restoration in 3D Face Image A.Srinivasan (Corresponding Author),
Balamurugan V
Professor and Head, Department of Information Technology, MNM Jain Engineering College, Anna University, Chennai, Tamil Nadu, India E-mail:
[email protected]
Professor and Head, Department of Computer Science and Engineering, Chandy College of Engineering, Anna University, Thoothukudi, Tamil Nadu, India
[email protected]
Abstract— Face recognition system has emerged as an important field in case of surveillance systems. Since three-dimensional imaging systems have reached a notable growth, we consider the 3D image for face recognition. Occlusion (extraneous objects that hinder face recognition, e.g., scarf, glass, beard etc.,) is one of the greatest challenges in face recognition systems. Other issues are illumination, pose, scale etc., an innovative three dimensional occlusion detection and restoration strategy for the recognition of three dimensional faces partially occluded by unforeseen objects is presented. Normalization provides orientation of the image to frontal view since we require frontal position for face recognition. An efficient method is used for detection of occlusions, which specifies the missing information in the occluded face. A restoration method then eliminates occlusion and renders a restored facial image. It exploits the information provided by the non-occluded part of the face to recover the original face. Restored faces are then applied to a suitable face recognition system. The proposed system will provide better accuracy to eliminate the occlusion and restored facial information method is independent of the face recognition method. Keywords—Face recognition; Restoration; Normalization.
Occlusion;
3D
Face
I. INTRODUCTION Face Recognition is an application to biometric system for identification and verification of faces. A Face Recognition system is supposed to be able to identify or recognize a noncooperative face in uncontrolled environment and an arbitrary situation without the notice of the subject [1]. This generality of environment and situations, however, brought serious challenges to face recognition techniques, e.g., the appearances of a face due to viewing condition changes may vary too much to tolerate or handle. Face recognition is one of the most relevant applications of image analysis. This can be done by either general twodimensional method or the growing 3D methods [2]. The growing availability of three-dimensional imaging systems has paved the way to the use of 3D face models for face recognition. Face recognition involves a number of preprocessing steps prior to recognition which includes localization, normalization, feature extraction, these several issues complicate the task of face recognition.
978-1-4799-4075-2/14/$31.00 ©2014 IEEE
Organization of this paper: Section II, elaborates the related works done in this field and approaches available. Section III reveals the background of the problem. Section IV details the proposed system design. Section V deals with implementation, experiments done using public database and are explained, it also discusses experimental results. Section VI concludes the paper. II. RELATED WORK Colombo A et. al. Proposed a method [3] for face occlusion detection and normalization using GPCA (Gappy Principal Component Analysis) classifier which is robust to detecting occlusions and also efficient in neutralizing facial expressions in 3D faces. It was verified that this method was very efficient for restoration of faces in the presence of occlusions. A method was proposed by Nese Alyuz et al. for 3D face registration and recognition method [4] based on local facial regions called Average Regional Model (ARM) that can provide better accuracy in the presence of expression variations and facial occlusions. Dahua Lin et al. proposed an algorithm [5] to detect and recover the occluded parts in face images, which is guided by a quality assessment model evaluating both global coherence and local details. The drawback of this approach is that it does not address the 3D issue. III. BACKGROUND OF THE PROBLEM The existing work is an innovative three-dimensional detection and restoration strategy for the recognition of three dimensional faces, which may be partially occluded by unforeseen, extraneous objects. A prior knowledge about the occluding objects is not required. The objects may be glasses, hats, scarves and the like, and differ greatly in shape or size, introducing a high level of variability in appearance. The restoration strategy is independent of the method used to detect occlusions and can also be applied to restore faces in the presence of noise and missing pixels due to acquisition inaccuracies. The restoration module recovers the whole face by exploiting the information provided by the non-occluded part of the face, and using a basis that is appropriate for the
face space in which the non-occluded faces lie. The restored faces can be then used by any face recognition method and Eigen faces approach is used for face recognition in the existing system. IV. PROPOSED METHOD OF DESIGN Figure 1 depicts the proposed system, which uses efficient methods to provide better results with respect to the current issues in face recognition and its pre-processing stages. Three-dimensional occluded faces are obtained by applying occluding objects taken from the 3D scanner to the nonoccluded face. An efficient Normalization is applied prior to restoration. Normalization renders a frontal pose of the given 3D image since for the restoration step, it will make the process easier. Restoration module is carried out to retain the features of the face not available due to occlusion. Finally, an effective and efficient recognition is performed. The proposed system performs these functions efficiently, i)3D Acquisition, ii) Face Detection, iii) Face Normalization, iv) Face Restoration and v) Face Recognition.
Since we use the 3D database this range image can reproduce the 3D structure of the face. Viola Jones: From the projected image face detection is done to find out whether the image contains a face region and if so, locate the exact facial region. This has to be done for the test image and the training images. Face detection provides localization of the facial region from the projected image. Here, we have employed the face detection method developed by Viola based on an ada boost and cascade algorithm [9] which uses the rectangular Haar features. The selected Haar like masks will effectively represent particular facial features. After the detection of faces in a given image, face normalization is performed. B. Face Normalization Since the system handles images with 3D structure, and the scope of the system is for face recognition, it is essential that frontal face is to be projected. Normalization method used here is geometric normalization that rotates the 3D shape of the image if necessary (if pose change is present) and delivers a frontal facial image. C. Face Restoration Face restoration is the process by which the occluded regions are located from the given test image and the occlusion is removed to generate original facial image. To achieve this, GPCA face restoration method is used.
Fig. 1. Proposed method and its System Design
A. 3D Acquisition It involves acquiring a 3D face image using a 3D scanner. Similarly occluding 3D object image is captured separately using the same scanner and is applied over the 3D captured facial image. Acquiring three-dimensional images through a scanner is a complicated process. Hence we use UND 3D [6] database, which comprise of acquisitions 3D face images of different types of occlusions. UMB-DB database [7] which contains 3D faces which are occluded by artificial occlusions such as glass, scarf etc., and this database is used for projecting the image in 3D plane and for further processes. Since we use 3D images, projection of the image as a range image is a preliminary process. Range Image: A special class of digital images [8]. They are used to encode the position of surface directly so that shape of a 3D image could be computed easily. Each pixel of a range image expresses the distance between a known reference frame and a visible point in the scene. We encode three dimensional faces as range images, that is, images whose pixels are labelled with the coordinates of a point in 3D space.
1) Gappy Principal Component Analysis(GPCA) The Normalization process provides the necessary information for subsequent face restoration and recognition process. The key idea in restoration is to use the avail- able information provided by the face detection and Normalization. a)
Initial 3D occlusion detection: The preliminary mask is computed by calculating Distance from Feature Space (DFFS), by thresholding vector e. This results in the preliminary mask calculation B. The threshold takes into account the resolution of the imaging device, acquisition noise and the accuracy achieved in normalization.
b) Refined 3D occlusion calculation: Eliminating the regions which are detected in the preliminary mask calculation, we calculate the mask M which provides a more exact facial mask annotation. From this the exact occlusion is determined. c)
Ground Truth: It is the indication of actual occluded parts of faces. After the mask M calculation, the occluded regions are annotated. These points from the generic occlusion. Marking the ground truth makes the restoration process easier.
D. Face Detection Face detection is a process which finds whether the given image contains a facial region and if so present, detects the face. This is done by either locating the features of the face, by using skin color etc., face detection is an essential step in any face- recognition systems. Since we also consider threedimensional images with clutter background, face detection is essential. Also the system involves annotating the facial features in the face restoration module. Hence feature based face detection method called Average Regional Method (ARM) is used, after studying various methods, which are tabulated in Table 2. The scope of the system is for face recognition. The restored faces are then compared with a dataset of images to find which image matches the dataset of images. Average Regional Model (ARM) finds the correct match from the given dataset of images. Average Regional Model(ARM) A part-based 3D face-recognition method which is able to perform robustly under expression variations and in the presence of significant amount of occlusions. This method is adopted from average face model-based (AFM)[10].The average region models (ARMs) is used for face registration. The facial area is divided into several meaningful components such as eye, mouth, cheek and chin regions. Registration of faces is carried out by separate dense alignments to relative ARMs. The dissimilarities between the gallery and test faces obtained for individual regions are then combined to determine the final dissimilarity score. Under extreme variations, such as the ones caused by occlusions, the combination method can automatically determine noisy regions and discard them. Thus the aim of the method is to find regional correspondences between any two face. It consists of the following steps, i) coarse and dense ARMbased registration, and ii) local matching and classifier fusion. The technique uses a two-phase approach: the first phase is a global coarse registration and the second phase is local dense registration. Global coarse registration: It is carried out to roughly align a given 3D face image to the AFM. ARMs are constructed on the AFM by determining the semantic regions manually. The whole facial model is divided into four parts, i) eye/forehead , ii) nose , iii) cheeks and iv) mouth chin regions. Dense registration: This is carried out by aligning local regions with ARMs using the ICP algorithm. Each region over the test face is registered to its corresponding average regional model separately. Registered regions are then regularly re-sampled. Therefore, after local dense registration, facial components are automatically determined over the given facial surface. Local Matching and Classifier Fusion: After the registration phase, each part of the face is labelled, and it is possible to compute regional dissimilarity scores. Since every region is
aligned, we choose to use an approximation of the volumetric difference between local surface pairs with the aid of point set differences. Euclidian distance is calculated for the facial regions separately for the test and the training image which is then compared to find out if the test image corresponds to the training image. Here, we find the Euclidian Distance considering the eye, nose and mouth region of the face to improve the efficiency in face recognition. Thus If occlusions are present in faces, the advantage of using proposed ARMbased 3D face recognizer becomes even more visible. Either by fusing all independent regions or by automatically detecting/removing occluded regions, it is possible to improve the correct classification rate. Hence the resultant face is suitable for face recognition. V. IMPLEMENTATION For implementation, we have used MATLAB, which is an interactive software sys- tem for numerical computations and graphics. In this proposed approach, we tested our system using UMB-DB image database [7]. Face recognition remains one of the most active research topics in pattern recognition. In the past several decades, most work focuses on the source of 2D intensity or color images. Since the accuracy of 2D face recognition is influenced by variations of poses, expressions, illuminations and subordinates, it is difficult to develop a robust automatic 2D face recognition system. The 3D facial data can provide a promising way to understand the characteristics of the human face in 3D domain, and has potential possibility to improve the performance of the recognition system. However, since the 3D cameras are not as common as 2D cameras, it is expensive to build a public 3D face database, which brings the difficulty to validate the proposed methods in a uniform platform. The University of Milano Bicocca 3D face database is a collection of multimodal (3D + 2D colour images) facial acquisitions. The database is available to universities and research centres interested in face detection, face recognition, face synthesis, etc. The classical database called UMB- DB [7] has been acquired with a particular focus on facial occlusions, i.e. scarves, hats, hands, eyeglasses and other types of occlusion, which can occur in real-world scenarios. Figure 2 shows the sample faces of neutral, with expression and occluded.
(a) Neutral Pose
(d) Exp:Smile
(b) Neutral Pose
(e)Exp:Angry
(c) Neutral Pose
(f ) Exp:Bored
(g) Occluded Pose
(h) Occluded Pose
(i) Occluded Pose
(a (a) Projecting the 3D image in 3 co-ordinate plane (j) Occluded Pose
(k) Occluded Pose
(l) Occluded Pose
Fig. 2. Sample Faces (Neutral, Expression and Occluded)
In proposed approach, we have three steps, Step 1 takes an input face image which is an occluded in some way, this image is processed and face portion is detected and given an input to step 2. Step 2 retrieves the face portion from step 1 output. This step also removes the occluded object and produces a non-occluded image as output. Step 3 contemplates the output of step 3 image to 3D image in 3D coordinate plan. Figure 3, 4, 5 shows the output in each step involved in recognizing a 3D image. 1) Experimental Results and Analysis The results are verified using UMB-DB database, which consists of 3D images captured using 3D scanner. The training database consists of around 150 subjects. Each subject is classified with various occlusions like cap, glass, pen, scarf, eyeglass, hair, hand etc,. The database is not only classifies under the occlusion category. It also provides sufficient images classified under different types of Expression. The system is fully implemented with the existing and proposed approaches using UND database [6].
(a) normalized frontal image for face
(b) Detection of face from given test restoration image
(b) Recognized face for the given test image from the database
Fig. 5. Implementation-Step 3
The existing system involves face projection as a range image. The image is normalized and from this, the mask (occluded region) is annotated. The restored face is used to compute the Eigen average face from which face is recognized using the samples in the test database. Test and training databases are classified separately. The input to the system would be an image from test database. The subject follows the subsequent steps, and finally, face recognition is performed with the image from training database. First, every image is tested with the existing system for different subjects handling the various occlusions. The result of the execution (whether recognized or not) is calculated separately and tabulated. It is noted that the system handles occlusions like eyeglass and hat efficiently. Subjects with changes in expression show considerably low recognition. False recognitions are also calculated and highlighted. The proposed system focuses on reducing the false positive rate. Table I shows the false positive rate for an existing and proposed system, and Figure 6 represents the plot using Table I. The below graph represents the false- positive rate for various occlusions handled using UND database for the existing system. From this graph it is seen that the false positive rates are high for mouth occlusion and changes in expression. The proposed system is also discussed with UND database for comparison of efficiency. It is handled with the same type of subjects and occlusions as handled in an existing system for efficiency comparison. TABLE I.
FALSE RECOGNITION RATE - EXISTING VS. PROPOSED SYSTEM
Fig. 3. Implementation-Step 1
(a) normalized frontal image for face Fig. 4. Implementation-Step 2
(b) Restored non occluded face for Restoration given test image
Type of Occlusion Eyeglass Hat Hair Eye Mouth Angry Smile
False Positives Proposed Existing 0.03 0.005 0.04 0.0175 0.05 0.03 0.06 0.0575 0.07 0.04 0.072 0.02 0.06 0.01
The existing and proposed system results are discussed and highlighted. TABLE III.
Type of occlusion Eyeglass Hat Hair Scarf Mouth Angry Smile
Fig. 6. False Recognition Rate - Existing and Proposed System
Table II shows the comparison results for various types of occlusion for the existing and proposed systems for face recognition efficiency. From figure 7, we can note the can see that recognition efficiency of proposed system outperforms the existing system. TABLE II.
. FALSE DETECTION RATE OF EXISTING VS. PROPOSED SYSTEM
Existing System 97.84% 95.56% 90.56% 92.62% 92.85% 88.59% 93.19%
Proposed System 99.03% 98.52% 89.62% 95.39% 95.14% 96.71% 94.04%
FACE RECOGNITION EFFICIENCY EXISTING VS. PROPOSED SYSTEM
Type of occlusion Eyeglass Hat Hair Scarf Mouth Angry Smile
Existing System 97.84% 95.56% 90.56% 92.62% 92.85% 88.59% 93.19%
Proposed System 99.03% 98.52% 89.62% 95.39% 95.14% 96.71% 94.04%
Fig. 7. Comparison of Existing vs. Proposed System - Face Recognition Efficiency
Fig. 8. Comparison of Existing vs. Proposed Systems - False Detection Rate
VI. CONCLUSIONS A face-recognition approach robust to occlusions and other challenges like expressions and pose is developed. Face detection finds the face in the given image and normalization helps to bring the frontal view of the 3D image if the test image does not provide the frontal view. Face restoration is efficient is removing the occlusion and also independent of the further recognition process. The system developed with this approach handles three-dimensional images scanned using a 3D scanner. The proposed system provides efficient results for face recognition, and the results are discussed using 3D UMB-DB database. This system could be further enhanced for face recognition in a way that it can handle 3D occluded images, which are acquired as 2D faces and reconstructed to 3D using an efficient technique and then processed. References
The comparison of false detection rate is provided in the table III. From the figure 8, we can see when there is a decrease in false detection rate, an increase in efficiency of face recognition occurs. The recognition efficiency and false detection rates are considerably improved in proposed system.
[1].
W. Zhao, R. Chellappa, P. J. Phillips, and A. Rosenfeld, (2003), "Face recognition: A literature survey, ACM Computer Survey", vol. 35, no. 4, pp.399–458.
[2].
Abate, Andrea F., et al. "2D and 3D face recognition: A survey" Pattern Recognition Letters 28.14 (2007): 1885-1906. [3]. Colombo, A., Cusano, C., Schettini, R., "ThreeDimensional Occlusion Detection and Restoration of Partially Occluded Faces", J. Math Imaging Vis (2011) [4]. Alyüz, N., Gokberk, B., Akarun, L.: "A 3D face recognition system for expression and occlusion invariance", In: Proceedings of 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, pp. 1–7 (2008) [5]. Lin, D., Tang, X.: "Quality-driven face occlusion detection and recovery", In: Proc. IEEE Conf. on Computer Vision and Pattern Recognition, pp. 1–7 (2007) [6]. http://www.frvt.org/FRGC/ [7]. A. Colombo, C. Cusano, and R. Schettini, "UMB-DB: A Database of Partially Occluded 3D Faces", In Proc. ICCV 2011 Workshops, pp. 2113-2119, 2011. [8]. Stamos, Ioannis, and P. E. Allen. "3-D model construction using range and image data." Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on. Vol. 1. IEEE, 2000. [9]. Viola, Paul, and Michael J. Jones. "Robust real-time face detection." International journal of computer vision 57.2 (2004): 137-154. [10]. M. O. Irfanoglu, B. Gokberk, and L. Akarun, "“3D shape-based face recognition using automatically registered facial surfaces",” in Proc. ICPR, 2004, vol. 4, pp. 183–186.
