Spatio temporal Modelling for Face Recognition with

Noname manuscript No. (will be inserted by the editor)

Spatio temporal Modelling for Face Recognition with NeuCube Fahad Bashir Alvi · Russel Pears · Nikola kasabov

Received: date / Accepted: date

Abstract In this paper we research three inter-related problems in face recognition using the Neucube neuromorphic computational platform. We investigate age classification, gender recognition and face verification. The well-known FG-Net image gallery was used and antropometric features were extracted from landmark points on the face. The landmarks were first pre-processed with the procrustes algorithm before feature extraction was performed. The Weka machine learning workbench was used to compare the performance of traditional techniques such as Support VectorMachine (SVM) and MultiLayerPerceptron (MLP) with NeuCube. Our empirical results show that NeuCube performed consistently better across all three problems that we investigated. Keywords Anthropometric Model · Multi Layer Perceptron · More

1 Introduction Aging is a slow process and its effects are visible only after a few years. But in spite of being slow, it remains a spatiotemporal phenomenon. The facial features of a person can be considered as a subspace and the aging over the years of this subspace is in turn a temporal process. It would be very useful to incorporate the temporal as well as spatial, patterns in aging data as an important components in classification. The raw data which has been used in this study is from (FG-NET) image gallery. Face Recognition, Gender Recognition and age group classification have important applications for business managers and law enforcement agencies. In Human Computer Interaction (HCI) gender Fahad Bashir Alvi · Russel Pears · Nikola Kasabov Knowledge Engineering and Discovery Research Institute Auckland University of Technology, Auckland, New Zealand E-mail: [email protected]

2

Fahad Bashir Alvi et al.

Fig. 1 Same Individual at different ages from FG-Net Database [4]

recognition can be used to make it more amenable to both genders. For example, it enables a computer to address a user by their correct title, Mr or Mrs as the case may be. Automatic gender recognition faciltates better interaction with humans as well as saves keystrokes in filling up forms. In Surveillance systems, a gender specific physical locality can be concentrated upon, thereby reducing the area under the observation and making the whole system more efficient. It could also assign higher threat levels to a specified gender location. In content based systems, indexing and searching can be greatly facilitated. For example, today a plethora of digital videos and photographs are produced. Their gender based indexing and searching can be easily carried out. It will also reduce the search effort by limiting it to a particular gender, thereby making it more efficient. The same applies to Biometrics. The automatic collection of demographic data for statistical purposes will be facilitated if the gender can be automatically recognised. Face recognition is also an area of research that is receiving a lot of attention due to biometric extensions[20-25]. There are a number of challenges in making an effective face recognition system which are pose, illumination, expression, occlusions and aging. It is challenging to address all of them simultaneously, because if one error is removed then other crops up. We have adopted the approach of an Anthropometric model which eliminates some of these problems. This model is based on geometrical ratios and distances of fixed number of fiducial landmarks, which in our case corresponds to 68 points on the face. Since Anthropometric model uses geometrical distances it is not affected by texture information, illumination and intensity. Problems caused by occlusions such as moustaches or spectacles are also eliminated as the distance between features will remain unchanged even if such occulsions occur. However, different poses could create problems. We mitigate the effects of this problem by finding an average of all faces and then warping all images to that image with the help of the procrustes algorithm [26]. This allow us to concentrate on effects due to aging. Aging not only changes texture, but also alters geometrical ratios and distance between landmark points. However, such changes over time follow a definite pattern. Thus, our main research hypothesis is that our anthropometric model, after necessary corrections, would provide a realistic basis for studying the effects of aging due to change in geometry of the face. Face Recognition issue has many practical applications.For example, the image of a criminal captured by a camera or video can be searched against a law enforcement database to extract vital information on known associates or areas of abode.

Spatio temporal Modelling for Face Recognition with NeuCube

3

This Research focuses on designing a Novel framework for age group classification, gender recognition and Face recognition. We use an Anthropometric model for feature extraction and extract seven different indexes. These indexes consist of different distances and geometrical ratios of face features. After extracting features we convert the data into discrete spike train. We use the AER encoding method to discretize the continuous signal. This encoding method was applied successfully for the artificial retina sensor [2]. We initialize the Cube by using Small world rule [2]. At unsupervised training stage we used STDP rule [2]. The dynamic evolving Spike Neural Networks (deSNN) is used here as an output classifier.It was observed that the classification achieved with Neucube was significantly better than with traditional techniques such as Support Vector Machine(SVM), Multi Layer Perceptron(MLP), and Naive Bayes(NV) .

2 Literature Review NeuCube arhitecture has also been used for classifying video data into three age groups[28]. [13] introduced a method which developed the concept of coordinate patches and GMMs . These were used to estimate facial ages. In their method, the face image of an individual is encoded as a group of overlapped spatially flexible patches (SFPs). Local features are then extracted by a 2D discrete cosine transform (DCT). The patches are used for integration of local features and coordinate information. These extracted SFPs are then modelled with GMMs. This model is used to estimate the age of a person in the input facial image. He did this by comparing the sum of likelihoods from total SFPs of the hypothetic age.Guo et al[14] designed a local based regression classifier to learn the aging function. He used the concept of manifold learning scheme for estimating the age. He used this manifold to predict age of a given image. Fu et al[15] also proposed a learning technique that involved manifolds. A set of age separate images are taken and low dimensional manifold is developed from it. Linear and quadratic regression functions were applied on the low dimensionalfeature vectors from the respective manifolds. Face estimation of the age was done with these manifolds. Face anthropometry is a science of measuring sizes and proportions on human faces. It is found that it can play an important role in developing facial aging models. These sizes and ratios provide a quantitative description of the craniofacial complex at different ages. This provides numerous options for learning based approaches. These approaches are developed to characterize facial aging. Face anthropometric studies take measurements between key landmarks on human faces at different ages. These ratios have played a critical role in surgical procedures employed on the faces of growing children [16]. Farkas [17] provides a comprehensive overview of face anthropometry and also describes its many significant applications. He has taken 57 selected landmarks on human face that are spread acroos 6 6 regions in the craniofacial complex (head, face, orbits, nose, lips and mouth, ear). The facial measurements are of three kinds: (i) projective measurements

4

Fahad Bashir Alvi et al.

(shortest distance between two land- marks)(ii) tangential measurements (distance between two landmarks mea- sured along the skin surface) (iii) angular measurements. Figure 3 illustrates the kind of data that is collected in face anthropometric studies and further illustrates the different fiducial features across which such data is collected. Since age-based anthropometric data was not available, they prepared it by extracting facial features from passport database. The data was grouped into 5 age groups, namely 21-30 yrs, 31-40 yrs, 41-50 yrs, 51-60 yrs and 61- 70 yrs. Such data was found to be effective was found to be effective in characterizing facial growth based on age, gender, ethnicity etc. It could also characterise instances when individuals gain or lose weight. Thus it describes the process of transforming facial appearances with age progression.[17]. Biswas et al[18] proposed that there is coherency in facial feature drifts across ages, He used it as a measure to perform face verification across ages. They assumed that facial features drifts observed in images that are age separated follows a definite coherent drift pattern. He also observed that the same might not be true for age separated face images of two different individuals. He further suggested that Since fiducially features extracted on the outer contour tend to be very sensitive to head pose variations, therefore a compensation process is required for that also. Ramanathan and Chellappa[19] used eight ratios of distance measures to model age progression in young faces, e.g., 0 to 18 years. These features were used to predict an individual’s appearance across age and to perform face recognition across age progression. Their experiment was performed on a database of 233 images of 109 individuals, partially collected from the FG-NET aging database [16] and partially from their own collection. They reported a face recognition improvement by using the age progression transformation. For example, their results improved from 8 percent (without age prediction) to 15 percent (with age transformation) for the rank 1 face recognition metric. n−gn Facial index ( zy−zy ) Mandibular index ( sto−gn go−go ) en−en Intercantal index ( ex−ex ) Orbital width index ex−en ( en−en ) ps−pi ) – Eye fissure index ( ex−en – Vermilion height index ( ls−sto sto−li ) – Mouth Face width index ch−ch ( zy−zy )

– – – –

3 Methodology We have used Anthropometric model for our study. A set of landmarks (total 68) is taken on face, and their distances and ratios are taken for extracting the

Spatio temporal Modelling for Face Recognition with NeuCube

5

Fig. 2 Schematic representation of the NeuCube-based methodology for mapping, learning, visualisation and classification of FG-NET DATA

features. We have used 7 such features on the face. The details of the model are given below. Since the images in the database are not of the same size, we applied procrustes to align them to a fixed size. This size was determined by averaging all the images of database. The final feature determination was done after application of procrustes on landmarks coordinates. These features wre then fed into NeuCube. To capture the time and space characteristics of spatiotemporal brain data in a spiking neural network (SNN) architecture NeuCube was proposed and experimented with in [9, 1]. The main components of NeuCube are a threedimensional spiking neural network reservoir (SNNr) and an evolving SNN classifier. We have used NeuCube architecture for the data where time length interval spans years rather than seconds or fractions of seconds. A block diagram of the architecture is shown in Fig. 2. There are three parts: an input encoding module, a three-dimension SNNr and an output dynamic evolving spike neural network (deSNN) classifier [9]. In The first stage we adjust the connection weights in SNNc through spatio temporal data. In the second stage, supervised learning is carried out. The modeling process contains the elements of data encoding, reservoir initialization, unsupervised training of the reservoir, supervised training of the classifier, new sample testing.

3.1 Data Encoding In real world applications, the data available is not suitable for spiking neural networks. We convert the data into discrete spike trains. The Address Event Representation (AER) encoding method is used for this purpose. This encoding method was applied successfully for the artificial retina sensor [2]. We obtain both the positive spike train and negative spike train for encoding.

3.2 NeuCube(ST) Initialization We initialize the SNNc by the following rule. Each neuron in the reservoir is connected to its nearby neurons which are within a distance d, where d equals

6

Fahad Bashir Alvi et al.

to the longest distance between any pair of neurons in the reservoir multiplying a parameter r. The initial weights .are set by a random number in [-1, 1]. We randomly select 80

3.3 Training Stage I: This stage deals with Unsupervised Reservoir Training. It encodes hidden spatio-temporal relationships from the input data into neuronal connection weights. According to Hebbian learning rule, if the interaction between two neurons is persisted, then the connection between them will be strengthened. We train the reservoir using STDP learning rule [5]: if neuron j fires before neuron i, then the connection weight from neuron j to neuron i will increase and, on the other hand, the connection from neuron i to neuron j will decrease. It will ensures that the time difference in the input spiking trains will be captured by the neuron firing state and the unsymmetrical connection weights in the reservoir. In the SNNc, when a neuron fires, it emits a spike and then its potential is reduced to 0. Each neuron connecting to this firing neuron will receive a spike.Because of this spike its potential increase with respect to its connection weight to current firing neuron. The potential of each neuron has a small constant rate of leakage along with the time unless it becomes 0. After learning, the connection weights in the reservoir encode temporal relationships from the input spatio-temporal data.

3.4 Training Stage II: In this stage, Supervised Classifier Training is carried out. The deSNN [6, 7]is used here as an output classifier, because deSNN is computationally efficient and emphasizes the importance of the first spike, which has also been observed in biological systems. For each training Once the NeuCube(ST) is trained, all connection weights in the reservoir and in the output classification layer are established. For a given new sample without any class label information, the trained NeuCube(ST) can be used to predict its class label. For the deSNN classifier, there are two algorithms that can be used to determine the class label of the new sample [6, 7]:

4 Experiments In all experiments We have used FG-NET (Face and Gesture Recognition Research Network) aging database [12] comprising of 1002 images of 82 subjects (6 18 images per subject) in the age range 0 - 69 years. In all three experiments, the size of the SNNc is 1000 neurons, a relatively simple 10x10x10 cube. It is trained and tested using a hold out method.

Spatio temporal Modelling for Face Recognition with NeuCube

7

Fig. 3 DataBase of FG-NET

Firstly we converted the data into discrete spike trains using the AER encoding method to discretize the continuous signal, following the example of the silicon retina . The deSNN classied mentioned previously is used here as an output classifier, because deSNN is computationally efficient and emphasizes the importance of the first spike which has been observed to be significant in biological vision systems. We conducted experiments to compare between traditional modeling methods (SVM and MLP) and our pro- posed method for age group classication. We designed two experiments for these baseline algorithms. Note that for these baseline algorithms, the time length of training samples and testing samples have to be the same as these methods cannot tolerate different lengths of feature vectors for training and testing. It was observed that the classication achieved with NeuCube was better than other techniques. See Table 4 for results. Note that the techniques mentioned (other than NeuCube) do not have the capability of representing the spatio-temporal problem space effectively. These traditional techniques are only suitable for static data within a given time segment. Since an eSTDM models the relationships between and within spatio-temporal data, even a small amount of input data will be able to trigger the spiking activities in SNNc, giving rise to a more accurate pattern (class) recognition rate from image data. 4.1 Age Group Classification For this experiment we distributed the FG-NET Database into three subspaces as 0-15, 16-30, 31 to 69. We take a sample size that has time length of 10

8

Fahad Bashir Alvi et al.

divisions. We take 12 samples from each of subspace. Thus we take 120 images from each subspace and give them class 1. Similarly we take next 120 images from subspace 2 and give them class 2 and another 120 images from subspace 3. For each of the image we take 7 features. Each sample provides a 10x7 matrix ie 10 images and each having 7 features. In all there are 36 samples. For the class labels, we take each subspace as one class. So, each of 12 samples has been allotted a class, making a total of 3 classes. The data is then fed to neucube and weka. The results of this experiment are given below

Table 1 Preliminary results of Age Group Classification using a NeuCube eSTDM in comparison with traditional techniques: SVM, MLP, NB. Measure

NeuCube

MLP

SVM

NaiveBayes

Accuracy(%)

72.00

63.30

53.03

55.60

In table 1 NeuCube showed better performance as compared to traditional methods and Multi Layer Perceptron have showed better accuracy among traditional methods. 4.2 Gender Recognition For this experiment we use the FG-NET Database. We model Gender recognition as a two-class classification problem. We examined the whole data manually and gave them a class, namely class 1 for male and class 2 for female. The same seven anthropometric features were used. Given an input image pair, we filled in a set of feature values that were normally distributed within the range of the values of the features from the two images. This data was then fed into the Neucube. We also also used three traditional techniques in Weka for comparative analysis.

Table 2 Results of Gender Classification using a NeuCube eSTDM in comparison with traditional techniques: SVM, MLP, NB. Measure

NeuCube

MLP

SVM

NaiveBayes

Accuracy(%)

81.00

67.30

58.00

66.00

In table 2 NeuCube showed better performance as compared to traditional methods and Multi Layer Perceptron have showed better accuracy among traditional methods.

Spatio temporal Modelling for Face Recognition with NeuCube

9

4.3 Face Recognition As in [28],Whole the FGnet data was divided into three groups based on the age groups, 0-8, 8-18 and 18-69. The age bracket for first two groups was kept small, because face features undergo greater change in younger ages. For each group, the number of persons and their images were determined from the database. For the class labels, each person was allotted one class, either male or female. Thus there are only two classes. Each person had more than one image. Thus we created all possible pairs within images of a person. Thus if a person had 4 images, it would lead to 6 possible pairs C24 . We prepared three sets of data, FG8, FG18, FG69. These were called intra personal pairs. Thereafter, inter personal pairs were also created. In this case, each image of one person was paired with each image of each of other persons. Since there were 69 persons (first age group), so first image of first person would yield 68 inter personal pairs with first image of each of other 68 persons. From all possible pairs, we randomly took a set of pairs. The data used in tests is as follows. – FG8 consists of all the data collected at the ages between 0 and 8. It includes 346 facial images from 69 subjects, in which 580 intra-person pairs and 6000 interperson pairs are randomly generated for verification. – FG18 consists of all the data collected at the ages between 8 and 18. It includes 372 facial images from 78 subjects, in which 580 intra-person pairs and 6000 inter-person pairs are randomly generated for verification. – FG69 consists of all the data collected at the ages 1869. It includes 362 images from 59 subjects, in which 670 interpersonal pairs and about 6000 intra-personal pairs are randomly generated for verification. For each of inter personal or intra personal pair, we took a metric vector of all 7 features. That formed the main content of data. All intra personal pairs were allotted class 1 and all inter personal pairs were allotted class 2. A time length was taken for preparation of data for neucube. We have also implemented three traditional techniques in Weka for comparative analysis.

Table 3 Results of Face Recognition with FG8 using a NeuCube eSTDM in comparison with traditional techniques: SVM, MLP, NB. Measure

NeuCube

MLP

SVM

NaiveBayes

Accuracy(%)

98.00

90.90

90.90

89.40

In table 3 NeuCube showed better performance as compared to traditional methods and almost all traditional methods have showed same accuracy.

10

Fahad Bashir Alvi et al.

Fig. 4 Reservoir connections after training. The red are negative weight connections and the blue are positive weight connections

Fig. 5 the total spikes amount emitted by each neuron during training process. Negative stems mean negative spikes.

Table 4 Results of Face Recognition with FG18 using a NeuCube eSTDM in comparison with traditional techniques: SVM, MLP, NB. Measure

NeuCube

MLP

SVM

NaiveBayes

Accuracy(%)

99.00

87.08

90.09

87.08

In table 4 NeuCube showed better performance as compared to traditional methods and almost all traditional methods have showed same accuracy. In Fig.4,6,8 blue lines represent connections with positive weight and red lines represent connections with negative weights. The line thickness indicates the strength of the connection. The neurons are coloured to indicate their connection strength with other neurons in the reservoir. Brighter neurons have

Spatio temporal Modelling for Face Recognition with NeuCube

11

Fig. 6 Reservoir connections after training. The red are negative weight connections and the blue are positive weight connections

Fig. 7 the total spikes amount emitted by each neuron during training process. Negative stems mean negative spikes.

stronger connections with other neurons while darker neurons have weaker connection with others.

Table 5 Preliminary results of Face Recognition with FG69 using a NeuCube eSTDM in comparison with traditional techniques: SVM, MLP, NB. Measure

NeuCube

MLP

SVM

NaiveBayes

Accuracy(%)

96.00

88.00

88.00

88.00

12

Fahad Bashir Alvi et al.

Fig. 8 Reservoir connections after training. The red are negative weight connections and the blue are positive weight connections

Fig. 9 the total spikes amount emitted by each neuron during training process. Negative stems mean negative spikes.

Discussion From the visualisation of Figures, we can make some conclusions related to a better understanding of the data and the problem. We can observe that connections between nearby input neurons are denser than the connections between far away input neurons. This indicates that there are more interactions between input variables mapped to closer neurons. Neurons in the middle of the reservoir are generally more active than those on the edges. This indicates that the neurons in the reservoir are not uniformly activated that react on the input data. In these experiments we can clearly see that NeuCube has the capability to provide better accuracy as compared to traditional methods. NeuCube can learn spatio temporal aging process. By Visualization we can clearly see that more spikes are emitted at young age because during young age more changes occur in cranio facial growth. But in young and old

Spatio temporal Modelling for Face Recognition with NeuCube

13

age these spikes are constant. NeuCube also validates that more changes will occur in face during childhood and less changes during young and adult age.

5 Conclusion NeuCube has capability to give better performance for face recognition, gender recognition and age group classification because it has capacity to learn spatio temporal relation. We would also implement NeuCUbe arhitecture on Morph Database. In future we would apply real time implementation of NeuCube for age group classification, gender recognition and Face Recognition.

References 1. N. Kasabov, V. Feigin, Z. G. Hou, C. Y., L. Liang, R.Krishnamurthi, M. Othman, and P. Parmar, ”Evolving Spiking Neural Network Method and Systems for Fast Spatio-Temporal Pattern Learning and Classification and for Early Event Prediction with a Case Study on Stroke,” Neurocomputing, 2014. 2. K. Hechenbichler and K. Schliep, ”Weighted k-nearest-neighbor techniques and ordinal classification,” 2004. 3. N. Kasabov, ”Global, local and personalised modeling and pattern discovery in bioinformatics: An integrated approach,” Pattern Recognition Letters, vol. 28, pp. 673-685, 2007. 4. T. Delbruck, B. Christian, and L. Longinotti, ”Real time sensorymotor processing for event-based sensors and systems,” http://sourceforge.net/p/jaer/wiki/Home/, 2007. 5. S. Song, K. D. Miller, and L. F. Abbott, ”Competitive Hebbian learning through spiketiming-dependent synaptic plasticity,” Nature neuroscience, vol. 3, pp. 919-926, 2000. 6. K. Dhoble, N. Nuntalid, G. Indiveri, and N. Kasabov, ”Online spatio-temporal pattern recognition with evolving spiking neural networks utilising address event representation, rank order, and temporal spike learning,” in Neural Networks (IJCNN), The 2012 International Joint Conference on, 2012, pp. 1-7. 7. N. Kasabov, K. Dhoble, N. Nuntalid, and G. Indiveri, ”Dynamic evolving spiking neural networks for on-line spatio-and spectrotemporal pattern recognition,” Neural Networks, 2012. 8. S. Thorpe and J. Gautrais, ”Rank order coding,” in Computational Neuroscience, ed: Springer, 1998, pp. 113-118. 9. N. Kasabov, ”NeuCube evospike architecture for spatio-temporal modelling and pattern recognition of brain signals,” in Artificial Neural Networks in Pattern Recognition, ed: Springer, 2012, pp. 225-243. 10. F. Zhou and F. De la Torre, ”Factorized graph matching,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2012, pp. 127-134. 11. D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schlkopf, ”Learning with local and global consistency,” in NIPS, 2004, pp. 595602. 12. J. Shrager, T. Hogg, and B. A. Huberman, ”Observation of phase transitions in spreading activation networks,” Science, vol. 236, pp. 1092-1094, 1987. 13. S. Yan, X. Zhou, M. Liu, M. Hasegawa-Johnson, and T. Huang. Regression from patchkernel. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 18, 2008. 14. G. Guo, Y. Fu, C. Dyer, and T. .Huang. Image-based human age estimation by manifold learning and locally adjusted robust regression. In IEEE Trans. on Image Processing, pages 11781188, 2008. 15. Y. Fu and T. Huang. Human age estimation with regression on discriminative aging manifold. In IEEE Trans. on Multimedia, volume 10, pages 578584, 2008.

14

Fahad Bashir Alvi et al.

16. L. G. Farkas and I. R. Munro, Anthropometric Facial Proportions in Medicine. Springfield, Illinois, USA: Charles C Thomas, 1987. 17. L. G. Farkas, Anthropometry of the Head and Face. New York: Raven Press, 1994. 18. S. Biswas, G. Aggarwal, and R. Chellappa, A non-generative approach for face recognition across aging, in IEEE Second International Conference on Biometrics : Theory, Application and Systems, 2008 19. Ramanathan, N. and R. Chellappa (2006) Modelling age progression in young faces, in IEEE Conf. on CVPR06, 2006, pp. 387394. 20. Dihong, G., et al. (2013). Hidden Factor Analysis for Age Invariant Face Recognition. Computer Vision (ICCV), 2013 IEEE International Conference on. 21. Jinli, S., et al. (2010). ”A Compositional and Dynamic Model for Face Aging.” Pattern Analysis and Machine Intelligence, IEEE Transactions on 32(3): 385-401. 22. Nayak, J. S., et al. (2012). Modeling self-Principal Component Analysis for age invariant face recognition. Computational Intelligence and Computing Research (ICCIC), 2012 IEEE International Conference on. 23. Syambas, N. R. and U. H. Purwanto (2012). Image processing and face detection analysis on face verification based on the age stages. Telecommunication Systems, Services, and Applications (TSSA), 2012 7th International Conference on. 24. Unsang, P., et al. (2010). ”Age-Invariant Face Recognition.” Pattern Analysis and Machine Intelligence, IEEE Transactions on 32(5): 947-954. Weixin, L., et al. (2010). SuperResolution in Tensor Space and Active Appearance Model Based Simulation of Adult Aging Effects on Face Images. Pattern Recognition (CCPR), 2010 Chinese Conference on. 25. Z. Li, U. Park, and A. Jain. A discriminative model for age invariant face recognition. In IEEE Trans. on Information Forensics and Security, 2011. 26. FG-Net aging database. [Online]. Available:http://sting.cycollege.ac.cy/ alanitis/fgnetaging/index.htm 27. H. Ling, S. Soatto, N. Ramanathan and D. W. Jacobs, Face verication across age progression using discriminative methods, IEEE Trans. Inform. Forensics Security 5(2010) 8291. 28. Evolving Spatio-Temporal Data Machines Based on the NeuCube Neuromorphic Framework: Design Methodology and Selected Applications, Journal of Neural Networks[Preliminary Accepted]