Using Landmarks to Establish a Point-to- Point ... - CiteSeerX

Pattern Analysis & Applications (2000)3:69–75  2000 Springer-Verlag London Limited

Using Landmarks to Establish a Point-toPoint Correspondence between Signatures K. Zhang, I. Pratikakis, J. Cornelis and E. Nyssen ETRO-IRIS, Vrije Universiteit Brussel, Brussels, Belgium Abstract: This paper introduces a new approach for point-to-point correspondence finding, which can be used as pre-processing stage of a handwritten signature verification procedure. This approach provides a solid basis for comparing function features of two handwritten signatures. Corner points of the signatures are first extracted based on velocity information. The characteristics of curvilinear velocity and angular velocity are combined successfully by functions based on membership criteria. The signatures to be compared are then segmented at landmarks obtained by corner matching based on similarity measures. In the last step, the corresponding pairs of segments are mapped by a point-to-point matching algorithm, minimising a curve deformation energy. The techniques described were applied to a set of 188 signatures from 19 volunteers. The resulting point-to-point matching of signature pairs was satisfactory in all cases where there was a visual agreement between the signatures. Keywords: Character recognition; Curve characterisation; Feature extraction; Handwritten text processing; Pattern matching; Signature verification

1. INTRODUCTION In the area of computerised security, signature verification is becoming more and more important as a technique for personal identification. The early work on signature verification can be traced back to the 1960s. Plamondon and Lorette [1] presented a summary of previous work in 1989, and a further summary was made by Leclerc and Plamondon in 1994 [2]. Approaches to signature verification that have been studied can roughly be classified into two groups, according to the different kinds of features dealt with [1]. In the first group, verification is based on comparing ‘function’ features by point-to-point matching. The function features are complete signals (e.g. position x(t), y(t), velocity v(t), etc.) which are represented by functions of time F(t). Values of F(t) directly constitute the feature set. In the second group, the feature set consists of a number of parameters, computed from the signals measured. These two groups of approaches are often referred to as function-based methods and parameter-based methods, respectively. Most of the work that has been done belongs to the second group [3,4]. Hundreds of parameters have been proposed and Received: 1 February 1999 Received in revised form: 5 May 1999 Accepted: 24 May 1999

claimed to be effective, including geometric parameters, timing parameters and other specific ones. But besides the choice of the criterion for parameter selection, there are still some other problems which limit the performance of the parameter-based methods. First, the existing unforeseeable forgeries are not easily characterised in advance. Secondly, the parameters selected with a small set of signers may not work well on a larger set of signers. Function-based methods retain information from the whole signing procedure, and therefore we expect them to be more capable of distinguishing between genuine and forged signatures. The main problem with the function-based methods is the comparison of the function features. Mapping all the corresponding points of two signatures is not an easy task, since in an arbitrary pair of signatures of the same person, portions of these signatures may be significantly different. However, when this mapping is feasible, it is an appropriate basis for the analysis and comparison of any detailed information on the signatures. Little improvement has been made in recent years because of the difficulty of feature comparison. Parizeau and Plamondon [5] compared three main function-based methods: regional correlation; dynamic time warping; and skeletal tree matching. The relatively high error rates (up to 17%), obtained by the global signature validation procedures based on these methods, encouraged us to search for an improvement in the point-to-point signature matching

70

K. Zhang et al.

steps. Recent work on function-based methods can be found elsewhere [6–8]. The techniques described by Martens and Claesen [6] are based on dynamic time warping. Another approach is that of Wu et al [7], where comparison of the signature with the template is based on a split-and-merge mechanism. The results of applying the verification technique to a set of Chinese signatures (from 10 people) show a sensitivity to forgeries of 97.2% and selectivity of 86.5%. The validation of the quality of matching is unfortunately not discussed in this paper. Like our technique, the algorithm Hesketh [8] uses elastic matching to find a point-topoint correspondence between signatures. The approach of Hesketh [8] is based on neural network technology. We derived our algorithm from a recently developed curve matching technique [9], which has shown excellent performance in terms of speed and the quality of matching results, without the prerequisite of a training procedure. In this paper, a novel approach to performing point-topoint mapping is described, which can be used as the preprocessing stage of a function-based signature verification procedure. In Section 2 we introduce a signature segmentation scheme using landmarks. Previous work on signature segmentation can be found elsewhere [10–12]. A method involving membership criteria derived from fuzzy logic is proposed here to locate corner points based on velocity features. In Section 3, we present an algorithm which was recently developed for contour matching [9], and which has been adapted by us for signature matching. Our experimental results are demonstrated and discussed in Section 4; finally, conclusions are presented in Section 5.

2. SEGMENTING SIGNATURES AT LANDMARKS In our approach, a signature is considered as a curve including both the visible and invisible trajectories (i.e. the displacement of the pen, while the pen tip is not touching the paper). To compare the corresponding parts of two signatures, the signature curve should be segmented at corresponding points. Corner points are considered to be more stable and more perceptually important than other types of point [10]. We have therefore chosen to use these corner points as landmarks (markers of the segment endpoints). Instead of geometry parameters, we use the dynamic information of velocity around the corner points. Curvilinear velocity cv(t) and angular velocity av(t) are used, because they are independent of the coordinate system. In our application, the definition of these quantities is based on the local velocity vector v(t), calculated as the first derivative of the pen position (x,y) with respect to time:

have a local maximum at a corner point. However, because of the instability of handwriting, neither cv(t) nor av(t) can alone define all the corner points correctly (see Fig. 1). Two types of problem may complicate the detection of corner points: 1. Occasionally, there is no local minimum of cv(t) at a corner point, or an extra local minimum appears because of an accidental hesitation in normal parts of a signature. 2. Sometimes there is a low and wide wave that has more than one local maximum of av(t) at a round corner of a signature, or a small extra wave appears because of an accidental shake of the pen tip; since successions of multiple maxima occur frequently, only maxima exceeding a given threshold are taken into consideration. One of the processing steps aims at reducing the number of candidate corner points. To achieve this reduction, sets of extrema points are replaced by averaged single extrema points on the basis of the curvilinear distances between the set members. Afterwards, combining the minima of cv(t) with maxima of av(t), we get a reduced number of candidate corner points, either resulting from the combination of a maximum point of av(t) and a minimum point of cv(t), or corresponding to one of the two types of extrema. To identify a corner point reliably, the characteristics of cv(t) and av(t) are combined. We define two membership functions corresponding to these two quantities to evaluate their contribution to the decision as to whether a point is a ‘member of the corner point class’. A function, based on a variant of the fuzzy logic ‘and’ operator and applied to the values of the membership functions, is used to make the final decision. The membership functions are defined in the following way: Definition 1. Assume that C = (C1,C2,%,Cp,%) describes a sequence of minimum points of cv(t), where the value of

cv(t) = 储v(t)储 = 冑v2x(t) + v2y(t) and av(t) = tan−1

vy(t + 1) vy(t) − tan−1 vx(t + 1) vx(t)

Normally, cv(t) will have a local minimum and av(t) will

Fig. 1. Illustration of the distribution of local extrema of av(t) and cv(t), ‘ⴱ’ marks the local maxima of av(t), and ‘o’ marks the local minima of cv(t).

Point-to-Point Correspondence Finding between Signatures

71

Cp is the time index of the p-th minimum point. The membership function of Cp belonging to corner points’ set S is

冦

1 − e−fc(Cp)

−fc(Cp ) ␮c(Cp) = 1 + e

0,

, fc(Cp) ⱖ 0

(1)

fc(Cp) ⬍ 0

with

冘 Ip

fc(Cp) =

(wi · ((cv(Cp + i + 1) − cv(Cp + i))

i=0

+ (cv(Cp − i − 1) − cv(Cp − i)))) where Ip is the size of the neighbouring area of Cp, and wi is a weight, which decreases with i. Obviously, fc is directly proportional to the differences between two adjacent points in the point Cp’s neighbouring area. Equation (1) confines the value of ␮c to the interval [0,1]. Selection of Ip is applied as follows: let us first consider the search for the value of Ipl associated with the left side of Cp. Assume point m1 is located in the middle of Cp−1 and Cp, and m2 is located in the middle of m1 and Cp: 1. Let m = m2; 2. If cv(m − 1) ⬍ cv(m), or m equal m1, then go to step 3; otherwise m = m − 1, repeat step 2; 3. Locate the left border at m, then Ipl = Cp − m. Similarly, we calculate Ipr for the right side, and set Ip = min(Ipl,Ipr). Definition 2. Assume A = (A1,A2,%,Ak,%) is the sequence of av(t) maximum points, where the value of Ak is the time of k-th maximum point. The membership function of Ak belonging to set S is:

␮a(Ak) =

再

fa(Ak ) =

冘

with

1,

兩fa(Ak )兩 ⱖ ⌰

兩fa(Ak )兩/⌰, 兩fa(Ak )兩 ⬍ ⌰

(2)

av(j)

j苸Jk

where Jk is the neighbouring area of Ak.

2. If av(m) has the opposite sign of av(Ak), or av(m − 1) ⬎ av(m), or m equals m1, then go to step 3; otherwise m = m − 1, repeat step 2; 3. Locate the left border of Jk at m. The right border of Jk is located in a similar way. Consider that we obtain the values of ␮c and ␮a for two extrema, combined into one corner point. If the minimum point is missing, we set ␮c to 0. Similarly, we set ␮a to 0 for the missing maximum point. Besides the two membership functions, one could define a final decision function derived from the fuzzy logic ‘and’ of ␮c and ␮a (␮ = ␮c · ␮a). However, due to the fact that ␮a is more important than ␮c, because it is a function of local angular movement, the final decision function is defined as:

␮ = (w · ␮c + ␦) · ␮a

(3)

where w is a weight less than 1, and ␦ is a small positive number. Let us consider that the procedure described so far has been applied on two signatures sb and sm, which have to be compared. In the following, we describe how matching corner points are identified in the signature pair, as a preparation for point-to-point matching. and Cm = (sm1,sm2, Let Cb = (sb1,sb2,%,sbi,%,sblb) %,smk,%,smlm) be two arrays of corners, which correspond to two signatures which have to be matched. The sizes of these arrays are, respectively, lb and lm, and the values of sbi and smk denote time indices. Initially, the coordinates (x,y) of the corners are transformed to make the origin their centre of mass. Matching the corners from the Cb and Cm array is performed using d(i,j): d(i,j) = d1(sbi,smi+j) + d2(sbi,smi+j) + d3(sbi,smi+j) + d4(sbi,smi+j) i = 1,2,%,lb; j = −M,−M + 1,%,−1,0,1,%,M − 1,M

(4)

where dk(·), with k = 1,2,3,4 are measures for the difference of sbi and smi+j: d1 is the Euclidean distance; d2 is a difference measure taking into account curvilinear distances with respect to neighbouring corner points [13]; d3 is the difference between the curvilinear distances with respect to the beginning points; d4 is a normalised difference of the angles formed by connecting the considered corner points with their neighbours [13]. For each corner sbi, we calculate the minimum distance ∀j: jj(i) = min(d(i,j))

(5)

j

Obviously, fa is the local angular movement; parameter ⌰ is a constant value. To define whether Ak is a corner, it is reasonable to compare fa to an angle of about ␲/2. We thus choose ⌰ = 1.6 in our application. Equation (2) confines ␮a values to the interval [0,1]. Selection of Jk is applied as follows: let us first consider the left side of Jk. Assume point m1 is located in the middle of Ak−1 and Ak, m2 is located in the middle of m1 and Ak:

Ideally, sbi and smi+jj should be the corresponding corners of two samples (i.e. two signatures of one person). It may be that a good match does not exist, e.g. when sbi is an extraneous corner point, which is absent in the second signature. Experimental results showed that there was a mismatching error rate of about 5% if no restrictions are imposed upon the choice of jj(i). To match the two signatures correctly, only corners satisfying the following restrictions, involving a chosen threshold dmax, were accepted:

1. Let m = m2;

1. d(i,jj(i)) ⬍ dmax;

72

K. Zhang et al.

2. ∀i⬘ 苷 i and d(i⬘,jj(i⬘)) ⬍ dmax, i + jj(i) 苷 i⬘ + jj(i⬘). After matching, the two signatures are partitioned into segments which have the matched corners as endpoints.

3. POINT-TO-POINT MATCHING

(6)

where Esignature =

冕 再冕 冕冋 s

Esmooth =

s

冎

册

⭸(f(t) − t) 2 dt ⭸t

One can prove that the local minimum of the functional E(f), formulated as the solution of the Euler–Lagrange equation ⵜE(f) = 0 for Eq. (5), satisfies 1 ␭

冕

[cvs(t) − cvs⬘(f(t))]

␦t

The N × N non-linear system obtained this way can be described as follows:

−2w1 + w2 = −h2b1(s1,w1)

 

w1 − 2w2 + w3 = −h2b2(s2,w2)

%% wN−2 − 2wN−1 + wN = −h2bN−1(sN−1,wN−1)

wN−1 − 2wN = −h2bN(sN,wN) − 1

One can write this in matrix notation: Aw = h2b(s,w) The iteration scheme is w(k+1) = h2A−1b(w(k)) 0ⱕ␤ⱕ1

t⬘ = f(t)

f⬙ +

wN+1 = 1

w(k+1) = (1 − ␤)w(k) + ␤w(k+1)

[cvs⬘(t⬘) − cvs(t)]2dt dt

␦t

For i = 1,2,%N, wi and bi are the values of f(t) and b(t), respectively, at discrete positions ti. The boundary conditions become w0 = 0

The corresponding segments obtained by corner matching still have different lengths, and cannot be compared directly. The alignment of the points within the segments is required before comparison. A way of matching the corresponding points of two segments in signatures has been studied, by adjusting and improving a contour matching algorithm studied by Yang et al [9]. The basic idea of this algorithm is to minimise a quadratic fitting criterion E, composed of a bending energy Ecurve and a smoothness energy term Esmooth. In our approach, we adopt the same idea by using the curvilinear velocity and the measured time parameterisation of the signature curves. Assume s and s⬘ are the two corresponding segments of the two signatures to be compared: E = Esignature + ␭Esmooth

wi−1 − 2wi + wi+1 = −bi h2

⭸cvs⬘(f) dt = 0 ⭸f (7)

where the boundary conditions are

According to Yang et al’s [9] analysis for the stability of the iteration algorithm, the iteration scheme is stable if we choose ␤ carefully. Experiments show that matching errors occur only when one tries to match two long segments containing too many corners. This can be illustrated as follows: assume ˆf(t) is the correct matching function. For long segments, a time difference 兩fˆ(t) − t兩 between corresponding points in the front part is often propagated and accumulated into a much larger time difference in the middle. When the time difference is larger than the width of one peak of cv, f(t) may not converge to the correct result ˆf(t) from the initial condition f(t) = t. For example, in Fig. 2, A is the corresponding point of A⬘, but when A⬘ is initialised to match B, matching A

f(t0) = t⬘0 f(te) = t⬘e t0, t⬘0 are the starting times of the segments s and s⬘; te, t⬘e are the ending times. Since we normalise the time of the two segments, the boundary conditions become f(0) = 0 f(1) = 1 Assume b=

1 ␭

冕

Discretising expression:

[cvs(t) − cvs⬘(f(t))]

⭸cvs⬘(f) dt ⭸f

Eq. (7)

the

␦t

yields

following

numerical

Fig. 2. Demonstration of mismatching of two curvilinear velocity curves.

Point-to-Point Correspondence Finding between Signatures

73

fails, and the points in the neighbouring region will also be mismatched. To reduce the error in the initialising step and obtain a good final point-to-point matching result, the segmentation of the signatures according to the corners is necessary, and the length of segments after segmentation should be limited. To reduce the length of the segments, every segment containing more than seven corners was partitioned again by selecting the pairs of corners that have the smallest value with respect to function d (Eq. (4)) among the corners within the segment.

4. EXPERIMENTAL RESULTS We collected 188 genuine signatures from 19 volunteers. A digital pen-tablet device was used for sampling the signatures. Each signature was sampled at a fixed rate of 100 points/second. For each person, a reference signature has been randomly selected for comparison with the other signatures of the same person, resulting in a total of 165 experiments. 1. Corner extraction experiments: the values of the two parameters w and ␦ in Eq. (3) were chosen according to conditions which lead to a value ␮ ⱖ 0.5, whenever a candidate corner point should be accepted as a real corner point: (a) On the one hand, to ensure that ␮ is positive, w and ␦ should be greater than 0, and on the other hand, to obtain values ␮ of the same order of magnitude as ␮a and ␮c, w and ␦ should be less than 1 (0 ⬍ w ⬍ 1, 0 ⬍ ␦ ⬍ 1). (b) If ␮c ⱖ 0.5 and ␮a ⱖ 0.5, we require that ␮ ⱖ 0.5. (c) If ␮c = 0 and ␮a ⱖ 0.9, we require ␮ ⱖ 0.5. This implies that even in the case in which there is no minimum of cv, but the local angular movement is quite large (ⱖ0.9⌰), then the point considered is identified as a corner. According to Condition (c) and Eq. (3), which is the final decision function, we have ␦ ⱖ 5/9. By transforming Condition (b), we get w ⱖ 2 − 2␦. Considering this together with Condition (a), we finally choose ␦ = 0.6, and w = 0.8. (If ␮ is greater than 1 at one point, we truncate the value of ␮ at 1.) The corner extraction technique was applied to the set of signatures, mentioned above. Figure 3 shows the results for two samples, and illustrates that the extraction result is in agreement with the visual perception of the corner points. 2. Corner matching experiments: after segmenting, 87.3% of the segments contained two corners (two end points of the segment), 3.0% contained three corners, 3.4% contained four, 2.7% contained five, 2.0% contained six and 1.7% contained seven, which is the maximum number of corners allowed in one segment. There are 0.1% of pairs of corners mismatched. These percentages were calculated on the basis of the 165 experiments.

Fig. 3. Two samples of the result of corner extraction, ‘ⴱ’ marks the maximum of av(t), and ‘o’ marks the minimum of cv(t); illustrated here: the application of the technique to European (above) and Chinese (below) handwriting.

The reason for mismatching is that the difference between the parts around those corner points is quite large, which led to long segments, and the mismatching occurred in the re-partitioning step when trying to reduce the length of the segments. 3. Point-to-point matching: besides the mismatching of corners, 0.4% of the segments cannot be matched properly because of the large difference between the segments, while all others got a point-to-point matching result in agreement with the visual perception of the matching (see Fig. 4). Figure 4 illustrates that, despite the dissimilarity between some parts of the curves shown, a satisfactory point-to-point alignment between two signatures is established.

74

K. Zhang et al.

The second step consists of a modified existing contour matching algorithm [9], adapted to our application. As shown in Fig. 4, it establishes a nonlinear mapping between corresponding points coping with the existence of differences between the signatures from the same person. Our landmark extraction followed by point-to-point matching inside the segments improves matching robustness, when compared to point-to-point matching applied on the complete signature. As such, it is an effective basis for function-based signature verification methods, which will be the subject of our future work. Acknowledgements

The project has been carried out within the framework of the bilateral agreement between Huazhong University of Science and Technology (P.R.China) and Vrije Universiteit Brussel (Belgium).

References

Fig. 4. Two samples of the point-to-point matching of curvilinear velocity curves; illustrated here: the application of the technique to European (above) and Chinese (below) handwriting.

5. CONCLUSIONS The instability of signatures makes it difficult to compare function features of signatures at all the corresponding points. In our paper, we have presented a novel approach which effectively establishes a point-to-point correspondence between signatures, which consists of two consecutive processing steps. The first step identifies landmarks, splitting the signatures into segments. The landmarks, obtained from dynamic information, coincide with perceptually important points in the static signature pictures, which confirms their suitability as segment borders. The second step deals with the problem of finding a point-to-point correspondence between corresponding segments. In the first step, the extraction of corner points is based on velocity information instead of geometric parameters. A scheme based on membership criteria is proposed in our paper. The fuzzy logic definitions and decision function which have been applied successfully combine the characteristics of the two velocities, and lead to satisfactory results for the location of corners. Corner matching ensures that the signatures can be segmented at corresponding landmarks. The segmentation step significantly reduces the difficulty of signature comparisons. According to our experiments, when a segment contains less than seven corners, it leads to a good performance of the subsequent point-to-point matching step.

1. Plamondon R, Lorette G. Automatic signature verification and writer identification – The state of the art. Patt Recognition 1989; 22(2):107–131 2. Leclerc F, Plamondon R. Automatic signature verification: The state of the art – 1989–1993. Int Journal of Pattern Recognition and Artificial Intelligence 1994; 8(3):643–660 3. Kashi RS, Turin W, Nelson WL. On-line handwritten signature verification using stroke direction coding. Optical-Engineering 1996; 35(9):2526–2533 4. Lee-Luan-Ling. Automatic systems for real-time dynamic signature verification. Controle-&-Automacao 1997; 8(3):95–104 5. Parizeau M, Plamondon R. A comparative analysis of regional correlation, dynamic time warping, and skeletal tree matching for signature verification. IEEE Trans Pattern Analysis and Machine Intelligence 1990; 12(7):710–717 6. Martens R, Claesen L. Dynamic programming optimisation for on-line signature verification. Proc Fourth International Conference on Document Analysis and Recognition (Cat. No.97TB100138), 2, 1997, pp. 653–656 7. Quen-Zong-Wu, Suh-Yin-Lee, I-Chang-Jou. Online signature verification based on split-and-merge matching mechanism. Pattern Recognition Lett 1997; 18(7):665–673 8. Hesketh GB. COUNTERMATCH: a neural network approach to automatic signature verification. IEE Colloquium on Neural Networks for Industrial Applications (Digest No.1997/014), IEE, London, UK, 1997; 2/1–2 9. Xin Y, Truyen B, Pratikakis I, Cornelis J. Hierarchical contour matching in medical images. Image & Vision Computing 1996; 14(6):417–433 10. Brault J-J, Plamondon R. Segmenting handwritten signatures at their perceptually important points. IEEE Trans Pattern Analysis and Machine Intelligentce 1993; 15(9):953–957 11. Chi-Jain-Wen, Hon-Fai-Yau, Bor-Shenn-Jeng. Optimal segmentation of handwritten Chinese signatures using wavelet transforms. Optical Engineering 1996; 35(9):2721–2729 12. Huang K, Yan H. Online signature verification based on dynamic segmentation and global and local matching. Optical Engineering 1995; 34(12):3480–3487 13. Kui Zhang. Study of online handwritten signature verification. Technical Report 51, Vrije Universiteit Brussel, ETRO/IRIS, May 1998

Point-to-Point Correspondence Finding between Signatures

Kui Zhang was born in Beijing, China, in 1969. She received a degree in engineering from the Xi’an Jiaotong University, Xi’an, China in 1990. She finished her masters program of computer science and engineering at Huazhong University of Science and Technology, Wuhan, China, in 1993. Now she is preparing her PhD under the joint supervision of the Vrije Universiteit Brussel, Brussels, Belgium and the School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, P.R. China. Her current research interest is in signature verification.

Ioannis Pratikakis was born in Patras, Greece, in 1967. He received the Diploma degree in electrical engineering from the Democritus University of Thrace, Xanthi, Greece in 1992 and a PhD degree in applied sciences from the Vrije Universiteit Brussel, Brussels, Belgium in January 1999. Since March 1999, he has been with the Vista group at IRISA/INRIA as a postdoctoral fellow. His research interests include mathematical morphology, scale-space theory, deformable models and medical image processing and analysis. He has been a member of the Technical Chamber of Commerce in Greece and the IEEE.

Jan P. H. Cornelis (MD 1973, PhD 1980) was born in Wilrijk, Belgium, in 1950. He is a professor in electronics and digital image processing at the Vrije

75 Universiteit Brussel. He is director of the department of Electronics and Information Processing, ETRO, at the Faculty of Applied Sciences. He also coordinates a research group on image processing and machine vision, IRIS. Current research projects of the IRIS group include: various applications in medical vision; remote sensing; mine- and minefield detection; and design of algorithms and computer architectures for image processing. His current research interest is mainly concentrated in the area of image and video compression.

Edgard Nyssen was born in Brussels, Belgium, in 1953. He received his degree of engineer in mechanics and electrotechnics, with specialisation in electronics, in 1979, from the Faculty of Applied Sciences at the Brussels University (VUB). At the same university he received his Special Masters degree in Informatics in 1981, and his PhD in applied sciences in 1986. Dr Nyssen is lecturer at the Applied Sciences Faculty of the VUB and his topics include digital circuit design, biomedical statistics and pattern recognition. His current research interests include the application of pattern recognition techniques to image analysis and basic research on pattern classifiers.

Correspondence and offprint requests to: K. Zhang, ETRO-IRIS, Vrije Universiteit Brussel, Pleinlaan 2, B-1050 Brussels, Belgium. E-mail: kzhang얀etro.vub.ac.be