Face Sketch Synthesis and Recognition Based on ...

Face Sketch Synthesis and Recognition Based on Linear Regression Transformation and Multi-Classier Technique Alaa Tharwat1,2 , Hani Mahdi2 , Adel El Hennawy2 , Aboul Ella Hassanien3 Faculty of Engineering, Suez Canal University, Ismailia, Egypt 2 Faculty of Engineering, Ain Shams University, Cairo, Egypt Faculty of Computers & Information, Cairo University, Cairo, Egypt 1

3

Abstract.

Biometric technique becomes essential to identify individuals in dierent applications. Face sketch is one of the biometric methods, which are used to identify criminals. In this paper, a face sketch synthesis and recognition model is proposed. In this model, the photo images are transformed to pseudo-sketch images using linear regression technique. Moreover, Gabor lters are used to extract the features from three scales of the images. For each scale, a face sketch image is matched with face pseudo-sketches instead of the original photos to identify an unknown individual. Minimum distance classier is used to match the sketches with pseudo-sketches in each scale. Further, a classication level fusion is used to combine the outputs of the classiers at three scales namely, decision, rank, and score level fusion. CHUK database images is used in our experiments. The experimental results show that the proposed model is superior to other existed models in terms of identication accuracy. Moreover, matching sketch images with pseudo-sketches achieved accuracy better than matching sketch images with the original photo images. The proposed model achieved a recognition rate ranged from 82.95% to 94.32% using single scale, while the accuracy increased to 94.32%, 96.6%, and 97.7% when the decision, rank, and score level fusion, respectively, are used.

Keywords:

Face Sketch, Classier Fusion, Face Sketch Synthesis, Gabor Filters, Linear Discriminant Analysis (LDA), Biometrics.

1

Introduction

Biometrics is the science of measuring and analyzing physiological and behavioral characteristics to identify individuals. The progress in biometric techniques increases the accuracy of security systems. Dierent biological biometrics have dierent weak points. For example, ngerprints and iris need to interact cooperatively with the sensors. While face, gait, and ear recognition can be measured from a distance, which is more convenient and allows remote recognition [1]. Face recognition is one of the most common biometrics and has many applications. Face sketch recognition is one of the recent methods of face recognition

2

Alaa Tharwat1,2 , Hani Mahdi2 , Adel El Hennawy2 , Aboul Ella Hassanien3

methods, which is used to identify criminals to assist law enforcement. In the past, face sketch recognition was based on manual measurements. However, the process of automatic face sketch recognition is used to increase the accuracy and decrease the processing time. Automatic face sketch recognition system starts from draw sketches manually by artists and extract robust features from the sketches and match the extracted features with the face images of the criminals in the database to determine the nearest image to the sketch, hence identify the person. Thus, the artists' drawing skills and experience is one of the most important factors to increase the accuracy of the system [2, 3]. Face sketch system has two types, namely, viewed sketches and forensic sketches. Viewed sketches are sketches that are drawn while viewing a photograph of the person or the person himself. On the other hand, forensic sketches are drawn by asking a witness to know a description of the criminal [4]. Due to dierent modalities, features, and textures between the sketches and faces, many studies proposed to generate a face sketches from the original photos, which is called pseudo-sketches and match the original face sketches with the pseudo-sketches [2]. Zhong et al. synthesized a photo from a sketch by transforming the sketch to a photo, then they used photo-photo recognition to achieve better accuracy. They used 56 color photo-sketch pairs and Embedded Hidden Markov Model (EHMM) to map between photos and sketches and used enginspace to identify sketch images. Their proposed model achieved recognition rate 17.6% when they directly applied photo-sketch recognition while they achieved 88.2% when they applied photos-to-sketch and sketch-to-photo transformation [5]. Sun et al., used Active Shape Modeling (ASM) to extract the 68 features points to transform the photos to sketches automatically [6]. Yong et al. [7], used hand-drawn face sketch recognition based on Principal Component Analysis (PCA). They collected sketches from ve dierent artists and they used 50 subjects each has ve images from dierent ve artists. Mahalanobis distance is used as a classier and score level fusion and achieved 90% recognition rate. Brendan et al. used Scale Invariant Feature Transform (SIFT) and Multiscale Local Binary Patterns (MLBP). They achieved accuracy 99.27%, 98.6%, and 99.47% using SIFT, MLBP, and SIFT+MLBP, respectively [4]. In this paper, linear regression method is used to transform a photo to a pseudosketch. Moreover, Gabor lters are used to extract the features from dierent scales and then the unknown image is identied using each scale individually and the results of the three scales are combined at the classication level. The rest of this paper is organized as follows, the preliminaries are introduced in Section 2. The proposed model is explained in Section 3. Experimental results and discussion are introduced in Section 4. Finally, concluding remarks and future work are introduced in Section 5.

Title Suppressed Due to Excessive Length

2

3

Preliminaries

2.1 Gabor Features Gabor lter method is one of the common methods that are used to extract texture features from grayscale images. Gabor lter is an eective method in texture analysis; hence it is used in many applications such as segmentation, and biometrics [8]. Moreover, Gabor lter method is less sensitive to noise, a small range of translation, rotation, and scaling. A 2D Gabor function g(x, y) is dened as follows [9]:     y2 1 x2 1 + 2 + 2πjW x (1) exp − g(x, y) = 2πσx σy 2 σx2 σy where σx and σy characterize the spatial extent and frequency bandwidth of the Gabor lter, and W represents the frequency of the lter, and g(x, y) is the mother generating function of a Gabor lter family. A set of dierent Gabor functions, gm,n (x, y), can be generated by rotating and scaling g(x, y) to form non-orthogonal basis set, as follows, gm,n (x, y) = a−2m g(x0 , y 0 ), where x ´ = a−m (x cos θn + y sin θn ), y´ = a−m (−x sin θn + y cos θn ), a > 1, θn = nπ/K , m = 0, 1, . . . , S − 1, and n = 0, 1, . . . , K − 1. The parameter S is the total number of scales, and the parameter K is the total number of orientations. Thus, the total number of generated functions is determined by S and K parameters. Gabor lters features for any image (I ) are computed as in Equation (2). XX Gm,n (x, y) = I(x1 , y1 )gm,n (x − x1 , y − y1 )) (2) x1

y1

2.2 Linear Discriminant Analysis (LDA) LDA is one of the common feature extraction and dimensionality reduction methods. The main objective of LDA is to nd the projection space (i.e. LDA space) that have a good discrimination of the original data by increasing the betweenclass variance (SB ) and decreasing the within-class variance (SW ). The redundant and unimportant features are neglected when the original data are projected on the LDA space [10].

2.3 Classier Fusion Combining independent sources of information may help to determine the most suitable decisions. In machine learning, combining data can be performed in many levels such as sensor, feature, and classication level fusion. In classication level, the performance of the systems may be improved if the classiers are independent (i.e. diverse classiers). The outputs of classiers can be combined in decision, rank, or score level. Fusion in decision level considers the simplest fusion method and easiest one to implement because only decisions are combined in this level. One of the most famous combination methods used in decision

4

Alaa Tharwat1,2 , Hani Mahdi2 , Adel El Hennawy2 , Aboul Ella Hassanien3

level fusion is majority voting (MV) [11]. In rank level fusion, the ranked lists, which represent the output of the classiers, are combined. Rank level fusion has data more than decision level, thus it may achieve results better than decision level. The score level fusion has information more than the other two levels and achieved good results [1214].

3

Proposed Model

The proposed model consists of three phases, namely, photo to sketch transformation, training phase, and testing phase. The descriptions of each phase are introduced in the following sections.

3.1 Photo to Sketch Transformation In this phase, the photo image is transformed into pseudo-sketch using linear regression as shown in Fig. (1). As shown from the gure, the photo and the corresponding sketch image are divided into sub-regions (n × n), where n represents odd number (e.g. 3, 5, 7, . . . , etc). Each pixel in the sketch image represents one label and the corresponding feature vector is extracted from the photos. For example, the rst label Y1 represents the pixel value S2,2 , while the corresponding feature vector is denoted by X1 , where X1 represents the rst feature vector (i.e. rst row) and it is calculated from the square region around the center pixel (p2,2 ) as follows, X1 = [p1,1 , p1,2 , p1,3 , p2,1 , p2,2 , p2,3 , p3,1 , p3,2 , p3,3 ], where p2,2 is the value of the pixel in the second row and second column, and X1 is the rst feature vector (i.e. rst row). The detailed steps of the proposed transformation model are summarized in Algorithm (1) and Fig. (1).

3.2 Training Phase In this phase, the photo images or pseudo-sketches are collected in dierent scales (256×256, 128×128, and 64×64), thus each pseudo-sketch is represented by three scales (P S256×256 , P S128×128 , and P S64×64 ), where P Sn×n is the pseudo-sketch in scale n × n. For each scale, Gabor features are extracted. LDA dimensionality reduction method is then used to reduce the dimensions of each scale as shown in Fig. (2).

3.3 Testing Phase In this phase, an unknown image (i.e. sketch image) is collected and resized to be at three dierent scales, thus each sketch represented by three images (S256×256 , S128×128 , and S64×64 ). Gabor features are then extracted from each scale and project each feature vector on its corresponding LDA space (There are three dierent LDA spaces for the three scales), which is calculated in the training phase. After that, match the test feature vector of each scale with its corresponding feature matrix to determine the class label of the sketch image at

Title Suppressed Due to Excessive Length P1,1 P1,2 P1,3 P2,1 P2,2 P2,3 P3,1 P3,2 P3,3

5

Y1=[S2,2]

X1=[P1,1,P2,1,P3,1,P1,2,P2,2,P3,2,P1,3,P2,3,P3,3] X1 X2

Y1 Y2

XN

YN

LinearmRegression Regressionm Coeffecients

PredictmthemSketchmImage

Fig. 1: Block diagram of the proposed transformation method (Photo to Sketch Transformation Using Linear Regression). all three dierent scales as shown in Fig. (2). Finally, combine the outputs of the classiers at all three fusion levels (i.e. decision, rank, and score) to determine the nal class label of the unknown sketch image.

4

Experimental Results and Discussion

In this section, two experiments are performed. The rst experiment is performed to transform the photo images into pseudo-sketches. In the second experiment, face sketch recognition is implemented to match an unknown face sketch with the pseudo-sketch images to determine the nearest person.

4.1 Experimental Setup In this experiment, CHUK dataset is used. CHUK consists of 606 individuals. Each person has a frontal face photo image and a sketch image drawn by an artist. The dataset is divided into two sets, namely, training and testing sets. The training set consists of 306 individuals, while the other images are used as a testing set. Figure (3) shows samples of the used datasets. The training images are used to; (1) build the regression model to transform the photo image to pseudo-sketch image, (2) train or learn the classiers to identify an unknown sketch image.

6

Alaa Tharwat1,2 , Hani Mahdi2 , Adel El Hennawy2 , Aboul Ella Hassanien3

Algorithm 1

: Transformation algorithm from photo to pseudo-sketch using linear regression technique. 1: Divide Photo image (P (N × M )) and sketch image (S(N × M )) into sub-regions, each region (n × n). 2: for (each pixel in sketch image (S) (e.g. rst pixel Y1 = [S2,2 ])) do 3: Extract the corresponding pixel from photo image P and its surrounding pixels (e.g. X1 = [P1,1 , P1,2 , . . . , P3,3 ]. 4: end for 5: X ⇐ X1 , X2 , . . . , XN , X(N × M, n × n). 6: Y ⇐ Y1 , Y2 , . . . , YN , X(N × M, 1). 7: Normalize the feature matrix X . 8: Initialize Learning Rate (α) to zeros. 9: Run Gradient Descent algorithm to compute regression parameters, λi . 10: for (any new photo image Pnew ) do 11: Divide Pnew into sub-regions (n × n). 12: To estimate the value of pixels in the pseudo-sketch image P S , extract the corresponding pixel in photo image Pnew and its surrounding pixels. 13: Predict the pixels in pseudo-sketch image (P S ) as follows: P Snew = Pnew × λi . 14: end for

4.2 First Experiment (Photo to Pseudo-Sketch Transformation) In this experiment, a linear regression technique is used to transform the photo image to pseudo-sketch image. In this experiment, some of the photo images are selected to train the regression model to transform the photo images into pseudo-sketches. As shown in Algorithm (1), the photo and sketch images are divided into sub-regions and the size of each region is n × n. Each sub-region is used to predict one pixel in the pseudo-sketch image as shown in Algorithm (1). In this experiment, the size of the sub-regions will be 3 × 3. Samples of original photos, sketches, and pseudo-sketches are illustrated in Fig. (3).

4.3 Second Experiment (Face Sketch Recognition) The aim of this experiment is to identify individuals by matching the unknown sketch with the pseudo-sketches. In this experiment, Gabor lters are used to extract the features from the sketches, pseudo-sketches, and photo images. This experiment consists of two sub-experiments. The rst sub-experiment aimed to match the sketches with the original photos. While in the second sub-experiment, the sketches are matched with the pseudo-sketches. In this experiment, the images are scaled into three dierent scales (256 × 256, 128 × 128, and 64 × 64) as mentioned in Section 3. Minimum distance classier is used to match the training (i.e. photos or pseudo-sketches) and testing (i.e. sketches) samples. Moreover, dierent levels of fusion are used to combine the outputs of the classication of the three scales. In other words, the extracted features from the three dierent scales are then matched independently (i.e. the photos and sketches of each scale is matched together) and combine the outputs of all classiers in the three dierent levels (decision, rank, and score level) as shown in Fig. (2). In decision level,

Title Suppressed Due to Excessive Length

7

Sketches 256x256

128x128

64x64

Photos))) Extract)Gabor)Features)

128x128 64x64

Extract)Gabor)Features)

Extract)Gabor)Features)

) 256x256

Extract)Gabor)Features)

Extract)Gabor)Features) Linear)Discriminant)Analysis)(LDA))

Extract)Gabor)Features)

Projection)on)LDA)Space

Classifier1)

Classifier2)

Classifier3)

Classifier)Fusion

Class)Label)

Fig. 2: Block diagram of the proposed model. majority voting (MV) is used to combine the nal decisions. While Borda count method is used to combine the ranked lists, which result from matching the unknown sketch with the training samples in dierent scales. Finally, dierent rules such as sum, min, and max rules are used to combine the scores at score level fusion. Accuracy assessment method is used to measure the performance of this experiment. The accuracy represents the ratio between the correctly classied samples and all samples. A summary of this scenario is shown in Table (1).

4.4 Discussion Figure (1) shows the pseudo-sketches, which are generated using linear regression technique. The pseudo-sketches are more similar to the face sketches than photos, which may has a great impact on the accuracy of sketch recognition.

8

Alaa Tharwat1,2 , Hani Mahdi2 , Adel El Hennawy2 , Aboul Ella Hassanien3

Fig. 3: Samples of face photos (top row), original sketches (second row), and pseudo-sketches using the proposed transformation model for dierent four individuals.

From Table (1) many remarks can be seen. First, the accuracy values of the scales are dierent and the scale of 128×128 achieved the highest accuracy, while the 256 × 256 achieved the lowest accuracy. Second, combining dierent scales in classication level (i.e. decision, rank, or score) achieved accuracy better than all single scales. Third, score level fusion achieved the best accuracy, while the decision level achieved the lowest accuracy because the score level has information more than all other classication levels. Fourth, the accuracy of matching sketches with pseudo-sketches is much higher than matching sketch images with the original photos, which reects that pseudo-sketch images are more similar to sketches than original photos. It can also be noticed from Tables (1 and 2) that our proposed model achieved results better than some of the state-of-the-art models, which are listed in Table (2). Finally, we note that the fusion in classication level achieved accuracy better than matching photos with sketches at single scales. Moreover, transforming sketches to pseudo-sketches improves the accuracy of the proposed model.

Title Suppressed Due to Excessive Length

9

Table 1: Accuracy of the proposed model using photo-to-sketch recognition and pseudo-sketch-to-sketch recognition.

Single/Multi scale

Classier

64 × 64 128 × 128 256 × 256

Minimum Distance Minimum Distance Minimum Distance Abstract- MV Rank- Borda Count Measurement-Sum Measurement-Min Measurement- Max

Classier Fusion

Accuracy (%) Using Photos Using pseudo-sketches 52 63 63 77.45 80.25 82.25 80.5 83.12

85.23 94.32 82.95 94.32 96.6 95.5 97.7 93.2

Table 2: Accuracy of some state-of-the-art models.

Author Method Accuracy (in %) Himanshu S. Bhatt et al. [15] EUCLBP+GA 94.12 X. Tang et al. [16] Eigentransform+PCA 75 X. Tang et al. [16] Eigentransform+Bayes 81.3 Qingshan Liu et al. [17] LDA 85 Qingshan Liu et al. [17] PCA 64.33

5

Conclusions and Future Work

In this paper, a face sketch recognition model is proposed. The proposed model is used to transform a face photo image to pseudo-sketch. Two dierent experiments are performed. The goal of the rst experiment is to transform the face photo to pseudo-sketch using linear regression. While in the second experiment, an unknown sketch is identied by matching the unknown sketch with (1) original face photos or (2) pseudo-sketch. In all experiments, Gabor lters method is as a feature extraction method. In the rst sub-experiment, the unknown face sketch is matched with the original photos for each scale individually. While in the second sub-experiment, the face sketches are matched with the pseudosketch images. In the two sub-experiments, the outputs of the three scales are combined at classication level. The experimental results have shown that the classication level fusion in all levels achieved results better than all other single scales. Moreover, score level fusion achieved the best accuracy while the decision level achieved the worst accuracy. Moreover, matching sketches with pseudosketches outperform matching sketches with original photos. Thus, it could be concluded that the proposed model has achieved an excellent accuracy against many state-of-the-art methods. In the future work, we will combine the features of dierent scales at a feature level. Moreover, more robust feature extraction method is used.

10

Alaa Tharwat1,2 , Hani Mahdi2 , Adel El Hennawy2 , Aboul Ella Hassanien3

References 1. Tharwat, A., Ibrahim, A.F., Ali, H.A.: Multimodal biometric authentication algorithm using ear and nger knuckle images. In: Computer Engineering & Systems (ICCES), 2012 Seventh International Conference on, IEEE (2012) 176179 2. Tang, X., Wang, X.: Face sketch recognition. IEEE Transactions on Circuits and Systems for Video Technology 14(1) (2004) 5057 3. Wang, X., Tang, X.: Face photo-sketch synthesis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11) (2009) 19551967 4. Klare, B.F., Li, Z., Jain, A.K.: Matching forensic sketches to mug shot photos. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(3) (2011) 639646 5. Zhong, J., Gao, X., Tian, C.: Face sketch synthesis using e-hmm and selective ensemble. In: IEEE International Conference on Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. Volume 1., IEEE (2007) I485 6. Sun, Y., Miao, Z., Wang, Y., Wang, Z.: Automatic personalized facial sketch based on asm. In: International Conference on Image Analysis and Signal Processing (IASP), 2010, IEEE (2010) 123126 7. Zhang, Y., McCullough, C., Sullins, J.R., Ross, C.R.: Hand-drawn face sketch recognition by humans and a pca-based algorithm for forensic applications. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 40(3) (2010) 475485 8. Tharwat, A., Gaber, T., Hassanien, A.E.: Cattle identication based on muzzle images using gabor features and svm classier. In: Advanced Machine Learning Technologies and Applications. Springer (2014) 236247 9. Han, J., Ma, K.K.: Rotation-invariant and scale-invariant gabor features for texture image retrieval. Image and Vision Computing 25(9) (2007) 14741481 10. Tharwat, A., Gaber, T., Hassanien, A.E., Hassanien, H.A., Tolba, M.F.: Cattle identication using muzzle print images based on texture features approach. In: Proceedings of the 5th International Conference on Innovations in Bio-Inspired Computing and Applications IBICA 2014, Springer (2014) 217227 11. Kuncheva, L.I.: Combining pattern classiers: methods and algorithms. John Wiley & Sons, Second Edition (2014) 12. Tharwat, A., Ibrahim, A., Ali, H.A.: Personal identication using ear images based on fast and accurate principal component analysis. In: 8th International Conference on Informatics and Systems (INFOS). (2012) 5659 13. Tharwat, A., Ibrahim, A., Hassanien, A.E., Schaefer, G.: Ear recognition using block-based principal component analysis and decision fusion. In: Pattern Recognition and Machine Intelligence. Springer (2015) 246254 14. Semary, N.A., Tharwat, A., Elhariri, E., Hassanien, A.E.: Fruit-based tomato grading system using features fusion and support vector machine. In: Intelligent Systems' 2014. Springer (2015) 401410 15. Bhatt, H.S., Bharadwaj, S., Singh, R., Vatsa, M.: On matching sketches with digital face images. In: 4th IEEE International Conference on Biometrics: Theory Applications and Systems (BTAS), 2010, IEEE (2010) 17 16. Tang, X., Wang, X.: Face sketch synthesis and recognition. In: Proceedings of 9th IEEE International Conference on Computer Vision, 2003., IEEE (2003) 687694 17. Liu, Q., Tang, X., Jin, H., Lu, H., Ma, S.: A nonlinear approach for face sketch synthesis and recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005. Volume 1., IEEE (2005) 1005 1010