ARTICLE IN PRESS
Neurocomputing 68 (2005) 217–224 www.elsevier.com/locate/neucom
Letters
Ensemble of Parzen window classifiers for on-line signature verification Loris Nanni, Alessandra Lumini DEIS, IEIIT - CNR, Universita` di Bologna, Viale Risorgimento 2, 40136 Bologna, Italy Received 10 March 2005; received in revised form 13 April 2005; accepted 2 May 2005 Available online 18 July 2005 Communicated by R.W. Newcomb
Abstract An on-line signature verification system exploiting global information is presented. Global information is extracted with a feature-based representation and recognized by using an ensemble of Parzen window classifiers. Experimental results obtained on the SUBCORPUS100 MCYT signature database (100 signers, 5000 signatures) show that the machine expert, here proposed, outperforms the state-of-the-art works both for random and skilled forgeries. r 2005 Elsevier B.V. All rights reserved. Keywords: On-line signature; Parzen window classifiers; Ensemble of classifiers
1. Introduction In on-line signature verification the time functions of the dynamic signing process (e.g., position trajectories, or pressure versus time) are available for recognition. The main approaches proposed in the literature in order to extract relevant information from on-line signature data [5,7,10,11] are: (i) feature-based approaches, in which a holistic vector representation consisting of global features is derived from the acquired signature trajectories, and (ii) function-based approaches, in which time sequences describing local properties of the signature are used for recognition (e.g., position trajectory, velocity, acceleration, force, or pressure). In [3] an on-line Corresponding author. Tel.: +349 3511673.
E-mail address:
[email protected] (L. Nanni). 0925-2312/$ - see front matter r 2005 Elsevier B.V. All rights reserved. doi:10.1016/j.neucom.2005.05.004
ARTICLE IN PRESS L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224
218
signature verification system based on fusion of local and global information is presented; in this paper we propose a new approach that extends the previous work with respect to the global feature-based machine expert. The results obtained by the subsystem exploiting global information in [3] were characterized by high variability of the performance changing the system parameter (number of features retained NSe); the new approach is based on an ensemble of Parzen window classifiers (PWC) which makes the recognition performance more stable with respect to the system parameters. Results using all the 5000 signatures from the 100 subjects of the SUBCORPUS-100 MCYT Bimodal Biometric Database [8] are presented, yielding remarkable performance improvement both with random and skilled forgeries. The Machine Expert Based on Global Information is described in Section 2, the system proposed is briefly sketched in Section 3. Experimental procedure and results are given in Section 4. Finally, conclusions are drawn in Section 5.
2. Machine expert based on global information The system here proposed is based on a previous work [3], where the complete set of 100 global features is detailed (a d-dimensional feature vector, d ¼ 100). Note that an on-line signature acquisition process capturing position trajectories and pressure signals both at pen-down and pen-up intervals is supposed. In this paper we use a simple feature ranking which sorts the features that maximize the distance between the centroids of the different signers, as introduced in [3]. For each feature k ðk ¼ 1; . . . ; dÞ we compute the scalar Mahalanobis distance ðdM c;k ¼ dMðmc;k ; Sk ÞÞ between the mean mc;k of the training patterns of each signer c from a total of N c classes (signers), and the set of all the training signatures Sk . The Mahalanobis distance between a pattern x (mc;k in our case) and a set S of patterns with mean mS and covariance matrix CovS is given by dMðx; SÞ ¼ ðx lS ÞT Cov1 S ðx lS Þ.
(1)
Features are then ranked according to the following inter-user class separability measure: Dist F ðkÞ ¼
Nc X Nc X
ðdM i;k þ dM j;k Þ.
(2)
i¼1 j¼1
Given a training set of a signer c composed by a set Dc of feature vectors of its signatures, a non-parametric estimation of its probability density function (p.d.f.) is obtained by using Parzen window [9]. The Parzen window density estimate can be used to approximate the probability density pðxÞ of a vector of continuous random variables X. It involves the superposition of a normalized window function centered on a set of samples. Given a set of n d-dimensional samples D ¼ fx1 ; x2 ; . . . ; xn g the p.d.f. estimate by the Parzen window is given by ^ pðxÞ ¼
n 1X jðx xi ; hÞ, n i¼1
(3)
ARTICLE IN PRESS L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224
219
where jð; Þ is the window function and h is the window width parameter (h ¼ 0:125 in _ our tests). Parzen showed that p ðxÞ converges to the true density if jð; Þ and h are selected properly [2]. The window function is required to be a finite-valued non-negative density function where Z jðy; hÞ dy ¼ 1 (4) and the width parameter is required to be a function of n such that lim hðnÞ ¼ 0
and
n!1
lim nhd ðnÞ ¼ 1.
(5)
n!1
We use Gaussian kernels with covariance matrix S ¼ Identity as window function (detðÞ denotes the determinant of the matrix): T 1 1 y S y jðy; hÞ ¼ exp . (6) d=2 d 1=2 2h2 ð2pÞ h detðSÞ _
We get the estimate of the conditional p.d.f. p ðs=cÞ of each class c using the Parzen window method as 1 X _ p ðs=cÞ ¼ jðs si ; hÞ, (7) #Dc s 2D i
c
where #Dc denotes the cardinality of the set Dc . Given the feature vector s of an input signature and a claimed identity c, the verification procedure of the PWC is performed by evaluating Eq. (7).
3. System proposed In this paper we propose three methods based on an ensemble of PWC for classification: random subspace ensemble of PWC, best subspace ensemble of PWC, clustering subspace ensemble of PWC. Random Subspace Ensemble of PWC (RS): The random subspace method is the combining technique proposed by Ho [4]. This method modifies the training data set (generating K new training sets), builds classifiers on these modified training sets, and then combines them into a final decision rule. The new training sets contain only a subset of all the features. The percentage of the features retained in each training set is denoted by NFe. In this paper a classifier is a non-parametric estimation of the p.d.f. of a signer using the Parzen window method (7). In our experiments, we set K ¼ 100, so we have 100 different non-parametric estimation p^ j ðs=cÞ j ¼ 1; . . . ; K of each signer c, and we combine the classifiers using the ‘‘max rule’’ [6]. Considering the similarity matching score estimated by each of the K classifiers between an input
ARTICLE IN PRESS L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224
220
signature s and its claimed identity c, the max rule selects as final score ðscoreðs; cÞÞ the maximum score of the pool of K classifiers: scoreðs; cÞ ¼ max p^ j ðs=cÞ j¼1;...;K
(8)
Best Subspace Ensemble of PWC (BS): This method modifies the training data set (generating K new training sets), builds classifiers on these modified training sets, and then combines them into a final decision rule. Each individual classifier uses only a subset of all of the features: a percentage of NFe features selected at random as in the Random Subspace method, and the best features selected by the feature ranking method presented in Section 2 (a percentage of NBe features is selected). As in the previous case we combine the classifiers using the ‘‘max rule’’. Clustering Subspace Ensemble of PWC (CS): The patterns (signatures) are independently partitioned into clusters to group together similar signatures. Each signer is assigned to only one cluster (selected as the cluster to which belong most of the signatures of that signer). For each cluster we rank the features (using the feature selection method in Section 2) by their capability to discriminate the different signers belonging to that cluster. We have noted that some people have signatures that belong to only one cluster, while others have signatures that belong to more clusters. For each signer we use 100 PWC trained using Best Subspace, where the best features (in a percentage of NBe) are selected from the cluster to which the signer belongs, and other 100 PWC classifiers trained using Best Subspace where the best features are selected from the whole training set. The similarity between a signature of the test set and a given signer is obtained using the ‘‘max rule’’. This ‘‘mixed selection rule’’ is motivated by the following experimental observation: for signers whose signatures belong to only one cluster the features selected as the best of such a cluster have a high discriminant power, on the contrary the signers whose signatures belong to more than one cluster are better represented by the features selected as the best of the complete training set. In order to take into account both these considerations we use an ensemble of classifiers trained with this mixed selection rule. As concerns the clustering algorithm, in this work Fuzzy C-Means [1], EM [2] and K-Means [2] have been tested obtaining very similar results; the results reported in the experiments have been obtained by Fuzzy C-Means. The number of clusters used for partitioning the signatures is denoted by NCl.
4. Experiments The 100 signers of the SUBCORPUS-100 MCYT database are used for the experiments (100 signers with 25 genuine signatures and 25 skilled forgeries per signer—forgers are provided the signature images of the clients to be forged and, after training with them several times, they are asked to imitate the shape with natural dynamics, i.e., without breaks or slowdowns). Signature corpus is divided into the training and the test sets. In case of considering Skilled Forgeries, the training set comprises the first five genuine signatures and the test set consists of the
ARTICLE IN PRESS L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224
221
remaining samples (i.e., 100 20 client, respectively, and 100 25 impostor similarity test scores). In case of considering Random Forgeries (i.e., impostors are claiming others’ identities using their own signatures), client similarity scores are as above and we use one signature of every other user as impostor data so the number of impostor similarity scores is 100 99. We compare our methods with the approach based on Global Information proposed in [3] (GI), which to our knowledge is the state-of-theart method for signature verification. Please note that GI outperforms [3] the local system based on Hidden Markov Model that was ranked in first and second place, for random and skilled forgeries, respectively, in Signature Verification Competition 2004 [12]. For the performance evaluation we adopt the equal error rate (EER) [5], that is the error rate when the frequency of fraudulent accesses (false acceptance rate, FAR), and the frequency of rejections of people who should be correctly verified (false rejection rate, FRR) assume the same value; it can be adopted as a unique measure for characterizing the security level of a biometric system. In Tables 1–6 we show that our systems permit to obtain improvement in comparison to the state-ofthe-art work [3] of signature matchers. The experimental results show that (1) The performance of the approach based on GI proposed in [3] is strictly dependent on the parameter NSe (number of features retained by the feature selection step). (2) The approach named RS reaches performance very similar to GI, but less dependent on the parameter NFe. (3) Since some features seemed to be very discriminant, we propose to always retain them in each subspace (approach BS): the performance reported in Table 4 is better than that of the previous method (Table 3), also being little dependent on the parameter NBe. (4) The method named CS (Table 6) gains the best performance, further reducing its sensitivity to parameters. (5) In conclusion, relative improvement in the verification performance as high as 21% (for Skilled Forgeries) and 37% (for Random Forgeries) have been obtained as compared to state-of-the-art works (GI [3]). Table 1 Verification performance using the approach based on Global Information in [3], Skilled Forgeries Number of features retained NSe EER
10 15.97
20 14
30 13.5
40 12.35
50 11.8
60 10.6
70 11.5
80 11.8
90 15.8
100 25
Table 2 Verification performance using the approach based on Global Information in [3], Random Forgeries Number of features retained NSe EER
10 9
20 5.7
30 5.2
40 4.9
50 4.7
60 4.7
70 5.3
80 5.5
90 5.4
100 5.5
ARTICLE IN PRESS L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224
222
Table 3 Verification performance using the RS method, Skilled Forgeries NFe (%) EER
80 10
70 9.9
60 9.7
50 9.9
Table 4 Verification performance using the BS method, Random Forgeries ðNFe ¼ 60%Þ NBe (%) EER
5 9.9
10 9.2
15 9.9
20 9.6
25 9.9
30 10
35 9.9
40 10
45 10
50 9.9
Table 5 Verification performance (EER) using the CS method, Skilled Forgeries ðNFe ¼ 60%Þ NBe (%)
5 10 15 20 25 30 35 40 45 50
NCl 2
3
4
5
9.2 9.4 9.3 9.3 9.4 9.5 9.8 9.6 9.4 10.2
9 9.5 9.4 9.2 9.1 9.2 9.5 9.5 9.6 9.9
8.8 9.2 9 8.6 8.7 8.4 8.9 9 9 9.2
9 9.1 9.2 8.8 8.8 8.5 8.8 9 9.1 9
Table 6 Verification performance (EER) using the CS method, Random Forgeries ðNFe ¼ 60%Þ NBe (%)
5 10 15 20 25 30 35 40 45 50
NCl 2
3
4
5
3.2 3.6 3.6 3.7 3.5 3.5 3.7 3.7 3.8 4.4
3.7 3.3 3.3 3.3 3.3 3.3 3.6 3.7 3.8 4.2
2.9 3 3.1 3 2.9 3.1 3.2 3.4 3.5 3.6
3.2 3.1 3.1 3.1 3 3.1 3.1 3.2 3.2 3.3
ARTICLE IN PRESS L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224
223
5. Conclusions An on-line signature recognition system based on global analysis of input signatures has been described. The proposed approaches are based on a nonparametric statistical modelling based on the Parzen window classifier applied to a set of 100 global features extracted from an on-line signature acquisition process. Experimental results are given on the SUBCORPUS-100 MCYT database comprising 5000 different signatures from 100 contributors. Our ensemble of PWC classifiers (CS) permits to obtain a relative improvement in the verification performance as high as 21% (for Skilled Forgeries) and 37% (for Random Forgeries) as compared to state-of-the-art works.
Acknowledgements The authors would like to thank J. Fierrez-Aguilar and J. Ortega-Garcia for sharing SUBCORPUS-100 MCYT data set. References [1] J.C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum, New York, 1981. [2] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, second ed., Wiley, New York, 2000. [3] J. Fierrez-Aguilar, L. Nanni, J. Lopez-Penalba, J. Ortega-Garcia, D. Maltoni, An On-line Signature Verification System Based on Fusion of Local and Global Information, accepted at Audio- and Video-based Biometric Person Authentication, 20–22 July 2005, New York, USA, to appear. [4] T.K. Ho, The random subspace method for constructing decision forests, IEEE Trans. Pattern Anal. Mach. Intell. 20 (8) (1998) 832–844. [5] A.K. Jain, F. Griess, S. Connell, On-line signature verification, Pattern Recognition 35 (12) (2002) 2963–2972. [6] J. Kittler, M. Hatef, R. Duin, J. Matas, On combining classifiers, IEEE Trans. Pattern Anal. Mach. Intell. 20 (3) (1998) 226–239. [7] J. Ortega-Garcia, J. Fierrez-Aguilar, J. Martin-Rello, J. Gonzalez-Rodriguez, Complete signal modelling and score normalization for function-based dynamic signature verification, Audio- and Video-based Biometric Person Authentication 2003, Guilford, UK, 9–11 June 2003, pp. 658–667. [8] J. Ortega-Garcia, J. Fierrez-Aguilar, D. Simon, et al., MCYT baseline corpus: a bimodal biometric database, IEE Proc. Vision Image Sensor Processor 150 (6) (2003) 395–401. [9] E. Parzen, On the estimation of a probability density function and mode, Ann. Math. Statist. 33 (3) (1962) 1064–1076. [10] R. Plamondon, G. Lorette, Automatic signature verification and writer identification—the state of the art, Pattern Recognition 22 (2) (1989) 107–131. [11] D. Sakamoto, et al., On-line signature verification incorporating pen position, pen pressure and pen inclination trajectories, in: IEEE International Conference on Acoustics, Speech, and Signal Processing 2001, Salt Lake City, USA, 9 May 2005, pp. 993–996. [12] D.Y. Yeung, et al., SVC2004: First International Signature Verification Competition, in: International Conference on Biometric Authentication 2004, Fort Lauderdale, USA, 15–17 July 2004, pp. 16–22.
ARTICLE IN PRESS 224
L. Nanni, A. Lumini / Neurocomputing 68 (2005) 217–224 Loris Nanni is a Ph.D Candidate in Computer Engineering at the University of Bologna, Italy. He received his Master Degree cum laude in 2002 from the University of Bologna. In 2002 he started his Ph.D in Computer Engineering at DEIS, University of Bologna. His research interests include pattern recognition, and biometric systems (fingerprint classification and recognition, signature verification, face recognition).
Alessandra Lumini received a degree in Computer Science from the University of Bologna, Italy, on March 26th 1996. In 1998 she started her Ph.D studies at DEIS- University of Bologna and in 2001 she received her Ph.D degree for her work on ’’Image Databases’’. Now she is an Associate Researcher at University of Bologna. She is a member of the BIAS Research Group at the department of Computer Science of the University of Bologna (Cesena). She is interested in biometric systems (particularly fingerprint classification), multidimensional data structures, digital image watermarking and image generation.
