On-line Handwriting Recognition with Support Vector Machines--- A ...

On-line Handwriting Recognition with Support Vector Machines— A Kernel Approach Claus Bahlmann, Bernard Haasdonk and Hans Burkhardt, Computer Science Department, Albert-Ludwigs-University Freiburg, Germany August 6, 2002

Dipl.-Inf. Claus Bahlmann, Computer Science Department, Albert-Ludwigs-University Freiburg, Germany

Overview • Review of classification techniques – (Generative) Bayesian classification – (Discriminative) Support Vector Machine (SVM) classification • Our new SVM-kernel for sequences: the Gaussian dynamic time warping (GDTW) kernel • Examples, simulations and results with UNIPEN data – two-class problems – multi-class problems


1



1



1



1



1



1



1

Bayesian Classification

P (2|X)

2. Choose class with highest posterior probability by Bayes’ rule

PSfrag replacements P (T |l) P (l) P (l|T ) = P (T ) P (X|1) P (X|2)

1.5

P (X|1) 1

P (X|2)

0.5

0

0

0.5

1

1.5

2

2.5

3

3.5

4

3.5

4

1 posterior probability

PSfrag replacements 1. Estimate class conditional density models P (X|l) Pfor (1|X) each class l

class conditional density

The generative approach

0.8

P (2|X)

P (1|X)

0.6 0.4 0.2 0

0

0.5

1

1.5

2

2.5


3

2

Bayesian Classification

P (2|X)

2. Choose class with highest posterior probability by Bayes’ rule

PSfrag replacements P (T |l) P (l) P (l|T ) = P (T ) P (X|1) P (X|2)

1.5

P (X|1) 1

P (X|2)

0.5

0

0

0.5

1

1.5

2

2.5

3

3.5

4

3.5

4

1 posterior probability

PSfrag replacements 1. Estimate class conditional density models P (X|l) Pfor (1|X) each class l

class conditional density

The generative approach

0.8

P (2|X)

P (1|X)

0.6 0.4 0.2 0

0

0.5

1

1.5

2

2.5


3

2

Support Vector Machine (SVM) The Discriminative Approach (Two-class Case)

• Discrimination boundary has widest margin to “closest” training examples (support vectors) • Non-linear extension by implicit problem transformation into higher dimensional space by the “kernel trick”

SVM GUI by (Gunn, 1998) Dipl.-Inf. Claus Bahlmann, Computer Science Department, Albert-Ludwigs-University Freiburg, Germany

3


Kernel: K (T , P ) SVM classification: Sˆ (T ) = sign

X

αiSiK (T , Pi) + b

i

!

SVM training: Determine αi, that maximize the objective function LD =

X i

1X αi − αiαj SiSj K (Pi, Pj ) 2 i,j

with the constraints 0 ≤ αi ≤ C

and

X

α i Si = 0

i


4



X

αiSiK (T , Pi) + b

i

!


X i



and

X

α i Si = 0

i


4



X

αiSiK (T , Pi) + b

i

!


X i



and

X

α i Si = 0

i


4



X

αiSiK (T , Pi) + b

i

!


X i



and

X

α i Si = 0

i


4

Support Vector Machine (SVM) Kernels Vectors Pattern examples

Kernel example K(T , P )

Sequences (On-line handwriting data!)

T

=

(7, 5, 8)

τ

T

=

[7, 5, 8]

P

=

(9, 3, 4)

τ

P

=

[7, 5, 5, 8]

Gaussian kernel

K (T , P ) = exp −γ kT − P k

2


?

5




T

=

(7, 5, 8)

τ

T

=

[7, 5, 8]

P

=

(9, 3, 4)

τ

P

=

[7, 5, 5, 8]

Gaussian kernel

Gaussian DTW (GDTW) kernel

K (T , P ) = exp −γ kT − P k

2

K (T , P) = exp (−γDDTW (T , P))


5




T

=

(7, 5, 8)

τ

T

=

[7, 5, 8]

P

=

(9, 3, 4)

τ

P

=

[7, 5, 5, 8]

Gaussian kernel


K (T , P ) = exp −γ kT − P k

2


(however, GDTW cannot be proven to be positive definite;


5




T

=

(7, 5, 8)

τ

T

=

[7, 5, 8]

P

=

(9, 3, 4)

τ

P

=

[7, 5, 5, 8]

Gaussian kernel


K (T , P ) = exp −γ kT − P k

2


(however, GDTW cannot be proven to be positive definite; but, positive definite in (many) practical evaluations)


5

Dynamic Time Warping Purpose: Aligning temporally distorted patterns

T R

= =

and compute DDTW (T , R)

Warping path: (for samples)

t 1 , . . . , t NT

a

distance

r1 , . . . , r NR

aligning

measure

corresponding

φ : {1, . . . , N } → ({1, . . . , NT } × {1, . . . , NR}) DTW distance:

2 N

1 X

DDTW (T , R) = tφ∗ − rφ ∗

T (n) R(n) N n=1


6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding

PSfrag replacements P φ : {1, . . . , N } → ({1, . . . , NT } × {1, . . . , NR})

T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4


5

6

7

8

9

T

6


T R

= =


t 1 , . . . , t NT

a

distance

φ

r1 , . . . , r NR

measure

P


aligning

corresponding


T

9 8 7 6 5 4 3

DTW distance:

2 1

2 N

1 X


T (n) R(n) N n=1

1

2

3

4

5

6

7

8

9

T


6

Simulations and Results The Database

• UNIPEN Train-R01/V07 corpus • no cleaning from poor quality/mislabeled characters UNIPEN section

number of samples

1a (digits) 1b (upper case characters) 1c (lower case characters)

16000 28000 61000


7

Simulations and Results PSfrag replacements Feature Selection pn+1 2 pn+2 1 pn Feature vector sequence T = t1, . . . , tNT 0 pn−1 Feature vector tn = (˜ xn, yñ, θn)τ pn−2 −1 x 1. normalized x-coordinate x ñ = xnσ−µ θn −1 y 2. normalized y-coordinate yñ =

yn −µy σy

1

pn+1

PSfrag replacements pn

3. tangent slope angle θn = ang ∆nx +

0

pn+2

√

−1 · ∆ny

θn pn−1

pn−2


8

Gaussian DTW (GDTW) Kernel Examples

K (T , Pj ) = exp (−γDDTW (T , Pj )) P1

P2

2 1

P3

2

2

1

1

P4

P5 2

1

1

0

0

0

0

−1

−1 −2

−1

−1

2 1 0

0

1

2

0

2

0 −1

−1

−2

0

2

−2

−1

0

1

2

−2

40

40

40

40

40

30

30

30

30

30

20

20

20

20

20

10

10

10

10

10

0

2

−1

T

−1

0

1

2

DDT W T , Pj K T , Pj

10

20

30

40

10

20

30

10

20

30

10

20

30

40

20

40

0

0.20

0.71

0.99

10.04

1

0.70

0.28

0.17

0.00


60

9

Error Rates of Two-class Problems 1c section (lower case characters), randomly chosen 67 % Train / 33 % Test set

Difficulty easy

difficult

Character pairs

# Tr.-Expls.

# SVs

ESVM−GDTW

ESDTW [BB01]

a↔b

3540

298

0.5 %

0.8 %

d↔m

2595

334

0.1 %

0.4 %

c↔e

5088

351

3.7 %

7.2 %

u↔v

2214

397

9.2 %

6.8 %

y↔g

2088

358

11.2 %

7.7 %

b↔h

2524

275

2.3 %

3.2 %


10


Difficulty easy

difficult

Character pairs

# Tr.-Expls.

# SVs

ESVM−GDTW

ESDTW [BB01]

a↔b

3540

298

0.5 %

0.8 %

d↔m

2595

334

0.1 %

0.4 %

c↔e

5088

351

3.7 %

7.2 %

u↔v

2214

397

9.2 %

6.8 %

y↔g

2088

358

11.2 %

7.7 %

b↔h

2524

275

2.3 %

3.2 %


10


Difficulty easy

difficult

Character pairs

# Tr.-Expls.

# SVs

ESVM−GDTW

ESDTW [BB01]

a↔b

3540

298

0.5 %

0.8 %

d↔m

2595

334

0.1 %

0.4 %

c↔e

5088

351

3.7 %

7.2 %

u↔v

2214

397

9.2 %

6.8 %

y↔g

2088

358

11.2 %

7.7 %

b↔h

2524

275

2.3 %

3.2 %


10

Multi-class SVM DAG (directed acyclic graph)-SVM: combining K · (K − 1) /2 two-class SVMs into one K-class-SVM PSfrag replacements

1 2 3 4

2 3 4

4

not 4

3

not 4

1 2 3 not 4

2 3

3 vs 4 not 3

not 1

2 vs 4 not 2

3 4

1 vs 4

1 vs 3

1 2

2 vs 3 not 2

not 3

not 1

not 3

2

1 vs 2 not 1

not 2

1

figure taken from (Platt, 2000) Dipl.-Inf. Claus Bahlmann, Computer Science Department, Albert-Ludwigs-University Freiburg, Germany

11

Example of a Multi-class SVM-GDTW

Matlab demo


12

Error Rates of Multi-class Problems 1c section (lower case characters) Approach

Error rate E (average of 5 runs)

UNIPEN Database Type

MLP [PLG01]

13.0 % 11.4 % 9.7 % 14.4 %

Train-R01/V07 rand. chosen 10 %/10 % Train/Test rand. chosen 20 %/20 % Train/Test Train-R01/V07 rand. chosen 10 %/10 % Train/Test rand. chosen 20 %/20 % Train/Test rand. chosen 67 %/33 % Train/Test DevTest-R02/V02

HMM-NN hybrid [GADG01]

13,2 %

Train-R01/V07

HMM [HLB00]

14,1 %

Train-R01/V06 4 % ”bad characters” removed

DAG-SVM-GDTW

SDTW [BB01]

11.5 % 12.0 %


13




MLP [PLG01]

13.0 % 11.4 % 9.7 % 14.4 %



13,2 %

Train-R01/V07

HMM [HLB00]

14,1 %


DAG-SVM-GDTW

SDTW [BB01]

11.5 % 12.0 %


13




MLP [PLG01]

13.0 % 11.4 % 9.7 % 14.4 %



13,2 %

Train-R01/V07

HMM [HLB00]

14,1 %


DAG-SVM-GDTW

SDTW [BB01]

11.5 % 12.0 %


13

Complexity multi-class, 1c section (lower case characters), randomly chosen 10 % Train / 10 % Test set Order

Experiments on AMD Athlon 1200 MHz

Time

O M 2 · Tkernel

Training Classification Memory

(K − 1) · Ms · Tkernel K·(K−1) 2

˜ · F · sizeof(float) · MS · N

M: K: Ms : ˜: N F:

81 h 2.5 sec 17.5 MByte

total number of training samples number of classes average number of support vectors average sequence length number of features


14

Conclusion • A discriminative classifier for sequences: SVM with a Gaussian DTW kernel (SVM-GDTW) • Examples, simulations and results – Small training sets: Significant decrease of error rate – Large training sets: Comparable error rates • Remaining potential for improvement • Just a small number of model parameters have to be adjusted • Complexity of SVM-GDTW quite high • Kernel is not positive definite and thus global optimality of the training cannot be guaranteed. • Suitable for all problems with sequences (speech, genome processing, ...) Dipl.-Inf. Claus Bahlmann, Computer Science Department, Albert-Ludwigs-University Freiburg, Germany

15


15


15


15


15


15


15

Future Work • Character recognition −→ word recognition • Improving computational speed • Investigating non-positive definiteness • Investigating additional kernels • Hybrid of generative / discriminative classifier


16



16



16



16



16

References [BB01]

Claus Bahlmann and Hans Burkhardt. Measuring HMM similarity with the Bayes probability of error and its application to online handwriting recognition. In Proc. of the 6th ICDAR, pages 406–411, 2001.

[GADG01] N. Gauthier, T. Artères, B. Dorizzi, and P. Gallinari. Strategies for combining on-line and off-line information in an on-line handwriting recognition system. In Proc. of the 6th ICDAR, pages 412–416, 2001. [HLB00]

Jianying Hu, Sok Gek Lim, and Michael K. Brown. Writer independent on-line handwriting recognition using an HMM approach. Pattern Recognition, 33:133–147, January 2000.

[PCST00] J. Platt, N. Cristianini, and J. Shawe-Taylor. Large margin DAGS for multiclass classification. In S.A. Solla, T.K. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing Systems. MIT Press, 2000. Dipl.-Inf. Claus Bahlmann, Computer Science Department, Albert-Ludwigs-University Freiburg, Germany

17

[PLG01]

Marc Parizeau, Alexandre Lemieux, and Christian Gagné. Character recognition experiments using UNIPEN data. In Proc. of the 6th ICDAR, pages 481–485, 2001.


18

Error Rates of Multi-class Problems UNIPEN section

1a (digits)

1b (upper case)

1c (lower case)

Approach

Error rate E

DAG-SVM-GDTW

3.8 % 3.7 %

MLP [PLG01]

4.5 % 3.2 % 3.0 %

HMM [HLB00]

3.2 %

SDTW [BB01]

DAG-SVM-GDTW

7.4 % 7.3 %

SDTW [BB01]

10.0 % 8.0 %

HMM [HLB00]

6.4 %

DAG-SVM-GDTW

11.5 % 12.0 %

MLP [PLG01] HMM-NN hybrid [GADG01]

13.0 % 11.4 % 9.7 % 14.4 % 13,2 %

HMM [HLB00]

14,1 %

SDTW [BB01]

UNIPEN Database Type Train-R01/V07 rand. chosen 20 %/20 % Train/Test rand. chosen 40 %/40 % Train/Test Train-R01/V07 rand. chosen 20 %/20 % Train/Test rand. chosen 40 %/40 % Train/Test DevTest-R02/V02 Train-R01/V06 4 % ”bad characters” removed Train-R01/V07 rand. chosen 20 %/20 % Train/Test rand. chosen 40 %/40 % Train/Test Train-R01/V07 rand. chosen 20 %/20 % Train/Test rand. chosen 40 %/40 % Train/Test Train-R01/V06 4 % ”bad characters” removed Train-R01/V07 rand. chosen 10 %/10 % Train/Test rand. chosen 20 %/20 % Train/Test Train-R01/V07 rand. chosen 10 %/10 % Train/Test rand. chosen 20 %/20 % Train/Test rand. chosen 67 %/33 % Train/Test DevTest-R02/V02 Train-R01/V07 Train-R01/V06 4 % ”bad characters” removed


19


1a (digits)

1b (upper case)

1c (lower case)

Approach

Error rate E

DAG-SVM-GDTW

3.8 % 3.7 %

MLP [PLG01]

4.5 % 3.2 % 3.0 %

HMM [HLB00]

3.2 %

SDTW [BB01]

DAG-SVM-GDTW

7.4 % 7.3 %

SDTW [BB01]

10.0 % 8.0 %

HMM [HLB00]

6.4 %

DAG-SVM-GDTW

11.5 % 12.0 %


13.0 % 11.4 % 9.7 % 14.4 % 13,2 %

HMM [HLB00]

14,1 %

SDTW [BB01]



19


1a (digits)

1b (upper case)

1c (lower case)

Approach

Error rate E

DAG-SVM-GDTW

3.8 % 3.7 %

MLP [PLG01]

4.5 % 3.2 % 3.0 %

HMM [HLB00]

3.2 %

SDTW [BB01]

DAG-SVM-GDTW

7.4 % 7.3 %

SDTW [BB01]

10.0 % 8.0 %

HMM [HLB00]

6.4 %

DAG-SVM-GDTW

11.5 % 12.0 %


13.0 % 11.4 % 9.7 % 14.4 % 13,2 %

HMM [HLB00]

14,1 %

SDTW [BB01]



19

On-line Handwriting Recognition with Support Vector Machines--- A ...

On-line Handwriting Recognition with Support Vector Machines--- A ...

Suggest Documents

On-line Handwriting Recognition with Support Vector Machines--- A ...

Supernova Recognition using Support Vector Machines - Lawrence ...

support vector machines for speech recognition - CiteSeerX

support vector machines for speech recognition - CiteSeerX

Automatic Target Recognition with Support Vector Machines - CiteSeerX

Support Vector Machines - CiteSeerX

Support Vector Machines

Support Vector Machines (SVM)

Compressed Support Vector Machines

Support Vector Machines

Rating Companies with Support Vector Machines - CiteSeerX

Training of support vector machines with Mahalanobis

SPIKE SORTING WITH SUPPORT VECTOR MACHINES ... - CiteSeerX

Rating Companies with Support Vector Machines

Cascade Support Vector Machines with ...

Sex with Support Vector Machines - NIPS Proceedings

Infinite Ensemble Learning with Support Vector Machines

predicting corporate bankruptcy with support vector machines

optical character recognition system using support vector machines

Face Recognition by Support Vector Machines - Semantic Scholar

Speech Emotion Recognition Using Support Vector Machines (PDF ...

Support Vector Machines for Bass and Snare Drum Recognition

Cost-sensitive Support Vector Machines

Support Vector Machines - R Project