in handwriting recognition and signature verification ... added vectorially to generate a word or a signature, ... action plan, made up of virtual targets linked by.
An interactive trajectory synthesizer to study outlier patterns in handwriting recognition and signature verification Moussa Djioua , Christian O’Reilly & Réjean Plamondon École Polytechnique de Montréal, Laboratoire Scribens, Département de Génie Électrique, C.P. 6079, Succursale Centre-Ville, Montréal QC, H3C 3A7, Canada {moussa.djioua, christian.oreilly, rejean.plamondon}@polymtl.ca Abstract A software tool has been developed to simulate complex pen tip trajectories and their corresponding velocity profiles based on the Kinematic Theory of rapid human movements and its lognormal model. The set of equations used to generate the various signals is shortly introduced. The application interface and the functionalities of the main modules are schematized and described. Finally typical simulations results are presented in the context of an interactive comparative analysis of on-line signature data.
1. Introduction Handwriting strokes are used and studied in many fields of research, with many goals in mind. In pattern recognition, for example, many algorithms are designed to recognize handwriting based on the properties of the basic strokes that have been used to generate a character, a letter [7, 8]. Similarly, many signature verification algorithms focus on the idiosyncratic features of the strokes to characterize a signer and check his identity [5]. In forensic sciences, a detailed study of individual stroke patterns, focussing of tiny variations, is often the basis for making a decision about the authenticity of a signature [16]. In education, many teaching methods rely on the production of neat strokes and their concatenation to generate letters and then words [17]. In human motor control, the production of single strokes is studied to get a better understanding of the underlying cognitive and neuromuscular processes involved in their production [2, 19, 20]. In neurosciences, strokes are analysed to characterize neurodegenerative processes like Parkinson and Alzheimer diseases [15, 18]. They are also used as basic patterns to evaluate the recovery in the rehabilitation of brain attack patients [14]. In anthropomorphic robotics, handwriting strokes are used to explore the biomechanical principles used by
human to produce movements and to apply these concepts to control a robot arm [13]. In many of these applications, one needs to focus on specific specimens to test particular aspects of a design and upgrade its performances. For example, an on-line handwriting recognizer or an automatic signature verifier can be greatly improved if one can get a better understanding of the rejection errors. In this paper, we describe an interactive tool that can be used to generate any type of human-like handwriting patterns, both in terms of velocity or position vs time signals. Based on the Kinematic Theory of rapid human movements [6,9,11,12], the system allows a researcher to study interactively any aspects ( biomedical, kinematical or visual ) of a handwriting trajectory and to simulate any variations from that given target pattern. In the next section, we make a brief overview of the trajectory generation model upon which the tool is built. Follow a mathematical description of the theory and its implementation in an interactive and ergonomic environment. The last part presents a typical example of utilization of the handwriting synthesizer, in the context of a signature verification application.
2. The Kinematic Theory The Kinematic Theory of rapid human movements in its vectorial version [4, 10] provides a general framework to study and analyse the variability of handwriting. Using lognormal functions, the theory both describes the basic properties of a single stroke and how strokes can be added vectorially to generate a word or a signature, from a given series of input commands. Fig. 1 schematizes the whole generation process. A motor action plan, made up of virtual targets linked by curved segments, is converted into a series of G vectorial input commands DU (t − t0i ) . These i commands are fed alternatively in two competing neuromuscular systems, one agonist and the other antagonist to the direction of the movement. The overall synergy controls the pen tip velocity to
0-7695-2521-0/06/$20.00 (c) 2006 IEEE
produce a word or a signature. Since the agonist and antagonist systems are not acting in perfect opposition, the resulting velocity emerges from the vectorial summation of the individual velocity vectors of each constituent stroke.
t0i : time occurrence of the input command
µi :
logtime delay; time delay of the neuromuscular system on a logarithmic time scale
σi :
logresponse time; response time of the neuromuscular system on a logarithmic time scale. The direction of the velocity vector is determined by the starting θ si and the ending θie G
Figure 1. The generation of a rapid movement (word, signature, scribble), according to the representation framework of the Kinematic Theory [10]. In other words, a word or a signature is schematized in the human motor memory as an action map made up of a sequence of virtual targets connected by discrete strokes. When a person is about to write, this plan is voluntarily activated to produce a sequence of vectorial commands. Each of these commands stimulates a neuromuscular system that reacts to it with a lognormal impulse response. Depending on the time occurrence of the different commands, the strokes are more or less superimposed and the trajectory is more or less fluent.
directions of the curved input command DU ( t − t 0i ) . i The instantaneous direction
ϕi ( t − t0i ) of the
velocity can thus be expressed by [1]:
ϕi ( t ) = θ is +
θ ie − θis Di
t
∫ v (τ ) dτ i
toi
3.2. Complex trajectory representation and generation In this perspective, a word or a signature is seen as the vectorial summation of M strokes. The pen tip velocity of a complex trajectory is obtained by: M G G v ( t ) = ∑ vi ( t )
3 Trajectory generations These neuromotor concepts have been implemented in an interactive tool that can be used, among other things, to study the variability of genuine handwriting specimens.
This equation can be translated into Cartesian coordinates using: M
3.1. Stroke representation and generation A single stroke i, the basic discrete unit of a continuous trajectory, is described by its velocity
Vy ( t ) = ∑ vi ( t ) sin ϕi ( t )
lognormal function (1): G Di 1 2 exp 2 ln ( t − t0i ) − µi , vi ( t; t0i , µi ,σ i2 ) = σ i 2π ( t − t0i ) 2σ i 0
if t0i < t
(3)
i =1
Vx ( t ) = ∑ vi ( t ) cos ϕi ( t )
G vi ( t − t0 i ) .The module of this vector is a weighted
(2)
(4)
i =1 M
(5)
i =1
Each stroke is thus completely described in 2D by six parameters ( t0i , Di , µi , σ i ,θ si , θei ) . The handwritten image of the trajectory can be recovered by integration: ∞
else
(1)
x ( t ) = ∫ Vx (τ ) dτ
(6)
y ( t ) = ∫ Vy (τ ) dτ
(7)
0 ∞
where
G Di : vector describing the amplitude and the direction of the input command
0
Fig. 2 provides a general overview of the signal synthesizer [3]. A list of parameters describing each discrete stroke is fed into the simulation model to
0-7695-2521-0/06/$20.00 (c) 2006 IEEE
produce three types of output: the ideal velocity profile, as computed from the equations previously introduced, and two simulated signals. The first emulates a digitizer by recovering the pen tip trajectory from the ideal velocity profile, after proper sampling and integration. The resulting x (t) and y (t) signals are then combined, low-pass filtered and timederived to produce what we call the simulated velocity profile. The two simulated signals can be used for comparison with real digitizer data as well as for the testing of various pre-processing algorithms.
Figure 2. Basic modules of the handwriting synthesizer The synthesizer is accessed through an ergonomic interface (see Fig. 3) that allows a user to play with all the parameters of the model. For example, he or she can work at the highest level of movement representation and reshape the action plan by moving the virtual targets, one with respect to the others. Similarly the input commands can be varied by changing their amplitudes, their directions or their time of occurrence. The neuromuscular systems that react to these commands can also be modified by altering their log time delays and their log response times. After each modification, the user can see the effect of his change both on the trajectory and the velocity profile.
Figure 3: The tool interface. The left hand side is dedicated to the modification of the parameters while the right hand side controls the displays.
4. A typical example The flexibility provided by the tool allows a user to analyse specific patterns, for examples a word
rejected by an on-line recognizer, a genuine signature that might look unstable, a forgery specimen accepted by a verification system. In each case a fiducial pattern can be interactively created from the test specimen. Then, this fiducial pattern can be used to generate a set of altered patterns to analyse the limit of a classifier and improve its design. The following example highlights a practical use of the tool for a signature verification application. Fig. 4a illustrates a genuine signature, as collected from a digitizer, as they would appear on the panel interface. The left and right plots depict the original trajectory and the velocity profile respectively. Using this information, a system designer can reconstruct interactively a fiducial pattern that looks quite similar to the original specimen (see Fig. 4b). He can then play with the neuromuscular parameters (Fig. 4c) or the commands (Fig. 4d) to generate a set of signatures that encompass various levels of perturbation. Fig.4.c illustrates the effects of varying t0i and Di while in Fig.4.d,
the µi
and
σ i of
some
lognormal
components have been changed. The variations of t0i and µi tend to shift the components and create overlapping in the velocity profile, and the variations of Di and σ i tend to change their magnitudes, leading to some dilation or compression of the pattern. In a near future, we plan to use similarly modified patterns to study how on-line signature verification systems react to these variations and to exploit the knowledge gained through such an interactive analysis to improve the robustness of some authentication processes
Figure 4. A typical application. Printouts of the display windows:4a), an original signature specimen from a digitizer, 4b)a fiducial reproduction, 4c) and 4d) two typical modifications as generated by changing the neuromuscular or the command parameters respectively.
0-7695-2521-0/06/$20.00 (c) 2006 IEEE
Encyclopedia of Electrical and Electronics Engineering, J.G Webster Ed. John Wiley&Sons, N.Y, 15, 1999,123-146
5. Conclusion We have presented in this paper an interactive tool that can be used to generate any human-like handwriting or signing processes. Based on the Kinematic Theory of rapid human movements [6,9,11,12], the tool allows a system designer to emulate any pen tip trajectory and study its variability as a function of the parameters that have been used to generate such a pattern. This tool could be useful in many practical applications, for examples to study outlier specimens rejected by a cursive handwriting recognizer, to analyse the quality of forgeries in a signature verification application, to generate efficient letter models for interactive handwriting learning, to build strokes database to test different algorithms [21], to generate human-like handwriting to test anthropomorphic robot arm. Acknowledgements: This work was supported by grant RGPIN-915 from NSERC to Réjean Plamondon.
References [1] Djioua, M. . Analyse et synthèse des mouvements humains par l’utilisation du modèle sigma-lognormal, PhD thesis, École Polytechnique de Montréal. 2006, (in press) [2] Meulenbrock, R.G.J, Van Gemmert, A.W.A (Eds), Advanced in the study of drawing and handwriting. Human Movement Science, 2003, 22, 131-220 (special issue) [3] O’Reilly, C. Djioua, M. & Plamondon, R. système de synthèse de l’écriture manuscrite par l’utilisation du modèle sigma-lognormal, Technical Report EPM-/RT-2-5-07, École Polytechnique de Montréal, 2005. [4] Plamondon, R. , Djioua, M. Handwriting Stroke Trajectory Variability in the context of the Kinematic Theory. Proceedings of the International Graphonomics Society, University of Salerno, Italy, 2005,250-254 [5] Plamondon, R. (Ed), Progress in Automatic Signature Verification, Singapore, World Scientific, 1994. [6] Plamondon, R, Feng, C, Woch, A. A Kinematic Theory of rapid human movement: Part IV : A formal mathematical proof and new insights, Biological Cybernetics, 2003, 89, 126-138. [7] Plamondon, R, Srihari, S. On-line and Off-line handwriting recognition : A comprehensive survey, 20th Anniversary Special Issue, IEEE Trans Pattern Analysis and Machine Intelligence , 2000, 22(1), 63-84.
[9] Plamondon, R. A Kinematic Theory of rapid human movements: Part III: Kinetic outcomes. Biological Cybernetics, 1998, 78, 133-145. [10] Plamondon, R, Guerfali, W. The generation of handwriting with delta-lognormal synergies. Biological Cybernetics, 1998,78, 119-132. [11] Plamondon, R. A Kinematic Theory of rapid human movements: Part I: Movement representation and generation. Biological Cybernetics, 1995, 72(4), 295-307. [12] Plamondon, R. A Kinematic Theory of rapid human movements: Part II: Movement time and control. Biological Cybernetics, 1995, 72(4), 309-320. [13] Potkonjac, V. Robotic handwriting. International Journal of Humanoid Robotics, 2005,2(1),105-124. [14] Rohrer, B., Fasoli, S., Krebs, H.I., Hughes, R. Volpe, B., Frontera, W.R. Movement smoothness changes during stroke recovery. The Journal of Neuroscience, 2002, 22, 8297-8304 [15] Schröter, A., Mergl, R., Bürger, K., Hampel, H., Möller, H-J. & Hegerl, U. Kinematic analysis of Handwriting movements in patients with Alzheimer’s disease, Mild cognitive impairment, depression and healthy subjects. Dementia and Geriatric Cognitive Disorders , 2003, 15, 132-142. [16] Simner, M.L, Girouard, P.L. (Eds) Advance in forensic document examination. J. Forensic Document Examination, 2000, 13, 1-14 (special issue). [17] Simner,M.L., Leedham,C.G., Thomassen, A.J.W.M. (Eds) . Handwriting and drawing research : Basic and Applied Issues, Amsterdam : IOS Press, 1996 [18] Teuling, H.L.T, Stelmach, G.E. Control of stroke size, peak acceleration and stroke duration in Parkinsonian handwriting, Human Movement Science, 1991, 10, 315-333. [19] VanGalen G.P., Morasso P.(Eds.) Neuromotor control in Handwriting and Drawing, Acta Psychological ,1998, 100,1-227 (special issue) [20] Van Gemmert, A.W.A., Teuling, H.L.T (Eds) Connecting Sciences using graphonomics research , Motor Control , 2004, 18(4), 367-566 (special issue) [21] Varga, T. , Kilchhofer, D., Bunke, H Template-based synthetic handwriting generation for the training of recognition systems, Proceedings of the International Graphonomics Society, University of Salerno, Italy, 2005, 206-211
[8] Plamondon, R., Lopresti, D., Schomaker, L.R.B. & Srihari R. On-line handwriting Recognition, Wiley
0-7695-2521-0/06/$20.00 (c) 2006 IEEE
