Accumulated-Recognition-Rate Normalization for Combining Multiple ...

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK

Accumulated-Recognition-Rate Normalization for Combining Multiple On/Off-Line Japanese Character Classifiers Tested on a Large Database Ondrej Velek, Stefan Jaeger, Masaki Nakagawa Graduate School of Technology, Tokyo University of Agriculture and Technology 2-24-16 Naka-cho Koganei-shi, Tokyo, 184-8588, Japan E-mail: [email protected], [email protected], [email protected]

Abstract. This paper presents a technique for normalizing likelihood of multiple classifiers, allowing their fair combination. Our technique generates for each recognizer one general or several stroke-number specific characteristic functions. A simple warping process maps output scores into an ideal characteristic. A novelty of our approach is in using a characteristic based on the accumulated recognition rate, which makes our method very robust and stable to random errors in training data and requires no smoothing prior to normalization. In this paper we test our method on a large database named Kuchibue_d, a publicly available benchmark for on-line Japanese handwritten character recognition and very often used for benchmarking new methods.

1.

Introduction

Combining different classifiers for the same classification problem has become very popular during the last years [7,9]. In handwriting recognition, classifier combinations are of particular interest since they allow bridging the gap between on-line and offline handwriting recognition. An integrated on-line/off-line recognition system can exploit valuable on-line information while off-line data guarantees robustness against stroke order and stroke number variations. Since the different nature of on-line and off-line data complicates their combination, most approaches combine both types of information either during pre-processing; i.e. feature computation [1-3] or, like this paper, in post-processing [5-8]. In this paper, we report experiments on the combination of on-line and off-line recognizers for on-line recognition of Japanese characters. Multiple classifiers, especially on-line and off-line classifiers, very often generate likelihood values that are incompatible. The focus of this paper lies on our accumulated-recognition-rate normalization technique, which tries to overcome this problem by aligning the likelihood values with the actual performance of each classifier. At first we define the accumulated recognition rate, which is normalized to a linearly growing function. Using normalized likelihood ensures a fair combination of classifiers. We have introduced our warping technique in [13, 16], and tested its efficiency on a small NTT-AT database. That database is a collection of patterns written by elderly people, usually by incorrect writing style. We have achieved improvement of the recognition rate from 89.07% for the best single classifier to 94.14% by combining seven classifiers in our combination system [13] and in [16] 93.68% by combining two classifiers. However, since the NTT-AT database is not a collection of typical 196

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK

character patterns and the number of samples and categories is low, a good performance cannot be easily generalized for common Japanese character patterns. In this paper we experiment with the benchmark database Kuchibue_d, which contains online Japanese handwritten characters for which many researchers have already published recognition rates. This paper is structured as follows: Section 2 and 3 outline the work described in our previous paper [13] and describe the general theoretical framework of our normalization technique based on accumulated recognition rate. In Section 4 we introduce our improved approach utilizing information about stroke number, firstly published in [16]. Section 5 describes single classifiers and combination schemes used in our experiments. Section 6 presents our new results on the Kuchibue_d benchmark set, which allows comparison of our recognition results on the same benchmark with other research groups or with our previous results on the NTT-AT database (Section 7). Finally, Section 8 concludes this paper with a general discussion of our results.

2.

Comparability of different classifiers

Let A be a classifier that maps an unknown input pattern x to one of m possible classes (ω1, …, ωm), and returns values ai = A(x, ωi) denoting its confidence that x is a member of class ωi,. For an ideal classifier, each returned value ai corresponds to the true probability of ωi given x: Pi(ωi,|x), also called a-posteriori probability, with 0 ≤ ai=Pi(ωi,|x) ≤ 1. In real practice, however, the output values ai can merely be approximations of the correct a-posteriori probabilities. Many classifiers do not even output approximations but only values related to the a-posteriori probabilities. For instance, A(x, ωi) = ai very often denotes the distance between the input pattern x and the class ωi in a high-dimensional feature space under a given metric, sometimes with ai∈[0;∞]. In this case, the best candidate is not the class with the maximum value among all ai, but the class having the smallest ai; i.e., the class with the shortest distance. Also, the scale of output values is generally unknown. For example, from ar = 2ap one cannot predict that class ωr is twice as likely as class ωp. These inadequacies generally pose little problem for a single classifier that needs to find only k-best candidates. If a single classifier rejects patterns with confidence values below a certain threshold, a relation between confidence and a-posteriori probability is useful for setting the threshold. However, for combining several classifiers {A, B, C…} we necessarily need some relation among candidates stemming from different classifiers {ai =A(x, ωi), bi =B(x, ωi), …} in order to compare them and to select the best class. To better describe the output of classifiers, we define two auxiliary functions, n(ak) and ncorrect(ak) counting the overall number of samples and correctly recognized samples for each output value respectively: Function n(ak) returns the number of test patterns classified with output value ak and function ncorrect(ak) returns the number of correctly recognized test patterns. We begin by counting the number of correctly and incorrectly recognized samples for each value: ncorrect(ak) and nincorrect(ak)= nt(ak)ncorrect(ak). Graph 1 and Graph 2 show two exemplary histograms of these numbers for off-line and on-line recognition respectively. An ideal classifier returns only correct answers with likelihood values covering the whole range of possible values. As a matter of fact, most practical classifiers also 197

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK

return incorrect answers and do not cover all output values. Graph 1 and Graph 2 illustrate this for the off-line classifier - most of the correct output is around a =600 -, while the on-line classifier has most of its correct answers around a=900. The peak of ncorrect(a) differs from nincorrect(i) in both classifiers. Also, both classifiers use only a small range of the output interval [0;1000] intensively. Histogram of a confidence value ai

Histogram of a confidence value ai 250

500

200

400

150

300

100

200

50

100 0

0 0

200

400

correct

600

800

incorrect

0

1000 [a i ]

200

400 correct

600

800

incorrect

1000 [ai]

Graph 1: Histogram for an off-line recognizer Graph 2: Histogram for an on-line recognizer.

For a single classifier system, only the recognition rate; i.e., the number of correctly recognized patterns ∑ n correct (i) divided by the number of overall patterns, is of imi

portance. However, for combining multiple classifiers, not only the recognition rate, but also the distribution of ncorrect (a k ) and nincorrect (a k ) is of interest. If some classifiers provide better recognition rates than the best single recognition rate on several sub-intervals, then we can suppose that by combining multiple classifiers the combined recognition rate will outperform the single best rate. Graph 3 and Graph 4 show the recognition rates corresponding to Graph 1 and 2 respectively. Recognition Rate R(ai)

[% ] 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0

200

400

correct

600

inccorect

800

0

1000 [a i ]

200

400

correct

Graph 3: Rcg.Rate for an off-line recognizer.

3.

Recognition Rate R(ai)

[% ] 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

600

inccorect

800

1000 [a i ]

Graph 4: Rcg.Rate for an on-line recognizer.

Normalization and Warping

Our goal is to normalize the output of each classifier so that it better reflects the actual classifier performance for each output value and allows direct comparison and combination with outputs from other classifiers. The main idea is to turn confidence values into a monotone increasing function depending on the recognition rate. After normalization, the highest output should still define the most likely class of an unknown input pattern x; ai > aj should imply that P(ωi,|x) > P(ωj|x). However straightforward normalization of classifier outputs to a monotone increasing function is not possible, because classifier confidence is neither a continuous nor a monotone function. 198

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK In our new approach we solve this problem by defining the accumulated recognition rate (which is always a continuous function) and applying a warping process on it. This process expands and compresses local subintervals of the horizontal likelihood-axis to better match the practical recognition rates achieved, without changing the order and without adding artificial error by additive smoothing. The confidence value is often either smaller than the value suggested by the recognition rate, which means that the confidence value is “too pessimistic,” or it is higher than the actual recognition rate suggests, which is then a “too optimistic” value. We proposed using a second training set to measure the classifier performance for each likelihood value, given an appropriate quantization of likelihood, and then calibrate likelihood in a post-processing step according to the performance measured. Let A={a0,…, ai, …, amax } be a set of likelihood values with a0 being the lowest and amax being the highest likelihood assigned by the classifier. In our experiments, the likelihood values span the integer interval ranging from 0 to 1000, and thus A = [0;1000] , max = 1000 , and ak = k . The calibration replaces the old likelihood values ai by their corresponding recognition rates ri = a inew so that after normalization likelihood and recognition rate are equal: Ai=Ri Using our auxiliary functions, we can state this as follows:

ainew =

ncorrect (a i ) ∗ a max n( a i )

∀i .

Graph 5 and Graph 6 show the accumulated recognition rates before and after normalization computed for the off-line and on-line recognition rates shown in Graph 3 and Graph 4 respectively. The error rates depicted in Graph 5 and Graph 6 show the remaining percentage of misclassified patterns for likelihood values higher than a particular value ai. Accumulated Recognition Rate

[%]

Accumulated Recognition Rate

[%]

0.9

0.9

Accumulated RR(0,1000)=Rec.Rate

0.8

Accumulated RR(0,1000)=Rec.Rate

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0

0 0

200

400

600

800

0

[a i ]

200

400

600

800

[a i ]

correct (before)

incorrect (before)

correct (before)

incorrect (before)

correct (after)

incorrect (after)

correct (after)

incorrect (after)

Graph 5: R() Accumulated Recogni- Graph 6: R() Accumulated Recognition tion Rate for an off-line recognizer before and Rate for an on-line recognizer before and after after normalization. normalization.

Note that the accumulated recognition rate is a monotone growing function over classifier output. Our normalization method equals the accumulated likelihood mass with the corresponding accumulated recognition rate. Using the nomenclature introduced

199

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK above, our normalization ensures the following equation: A0 = R0 , which translates i

i as 1 ∑ n(a ) ∗ a = 1 k k L k =0 N

i

∑n

correct

(a k ) for all i, with

i

N being the overall number of

k =0

max

patterns and L being the overall likelihood mass; i.e., L =

∑ n( a ) * a k

k

.

k =0

For each classifier, we thus normalize the output so that the accumulated probability function R() becomes a function proportional to the classifier output. Accordingly, we adjust each classifier’s output by adding the following adjustment charf(ai):  = a i + a max ∗  

∑

i k =0

ncorrect (a k ) N

∑ ∗

 − a i  = a i + charf (a i )  

∑

i   ncorrect ( a k )  a ' i = a max = a i + a max ∗ k =0 − ai  = a i + charf (ai )   N N   where a max is the maximum possible output of a classifier ( a max = 1000 in our experiments), N is the number of overall patterns, and ri stands for the partially accumulated recognition rate. We call this classifier-specific adjustment “the characteristic function [charfi]” of a classifier. To compute the [charfi] of a classifier, we need another sample set, which should be independent from the training set. Hence, we use data independent from the training set to ensure a proper evaluation, although we observed that a characteristic function depends mostly on the classifier and not the data. To illustrate the effects of normalization we now compare the graphs 1-4 with the corresponding graphs computed after normalization; i.e., after adding the adjustment. Graph 7 and Graph 8 nicely illustrate the uniform distribution of output values after normalization, compared to Graph 1 and Graph 2. In both Graphs 7-8, we see that, after normalization, the whole output spectrum is used for correct answers. Moreover, incorrect answers concentrate in the left, low-value part with a peak near zero. Since both off-line and on-line recognizers show the same output behavior after normalization, this provides us with a standard measure for classifier combination. i

k =0

ncorrect (a k )

Hi st o g r am o f a co nf i d ence val ue a i

Histogram of a confidence value ai 250

600

200

500 400

150

300

100

200

50

100 0

0 0

200

400

correct

600

incorrect

800

0

1000 [a i ]

200

400

cor r ect

600

incor r ect

800

1000 [a i ]

Graph 7: Histogram for an off-line recognizer Graph 8: Histogram for an on-line recognizer after normalization. after normalization

200

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK Finally, Graph 9 and Graph 10 show the likelihood-dependent recognition rates for each likelihood value after normalization, which correspond to the recognition rates shown in Graph 3 and Graph 4 before normalization. Recognition Rate R(ai)

[% ]

Recognition Rate R(ai)

[% ]

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0

200

400 correct

600 inccorect

800

1000

0

[a i ]

200

400 correct

600 inccorect

800

1000 [a i ]

Graph 9: Recognition Rate for an off-line Graph 10: Recognition Rate for an on-line recognizer after normalization. recognizer after normalization

In Graph 9 and Graph 10, the recognition performance quickly increases to a high level, and except for very small likelihood values, the recognition rate is about the same for each likelihood value. The number of correctly recognized patterns is higher than the number of falsely recognized ones, from likelihood values greater than 50 onwards. Again, both off-line and on-line classifiers behave similarly here.

4.

Characteristic of On/Off classifiers according to stroke number

Number of strokes is the basic feature of Chinese and Japanese characters. The right stroke number can be found in a dictionary and varies from 1 to about 30. However, for fluently and quickly written characters, the number of strokes is often lower because some strokes are connected. In some cases the number of strokes can be higher than they should be. An interesting characteristic of stroke number variations is given in [11]. Graph 11-12 show the recognition rate according to the stroke number for an on-line and off-line recognizer respectively. Even from a brief view, we see the big difference between both classifiers. An off-line recognizer is weaker in recognizing characters written with a low number of strokes. The recognition rate grows with increasing complexity of patterns. This is in accordance with the well-known fact that for off-line classifiers it is more difficult to recognize simple patterns like Kana than difficult Kanji. On the contrary, on-line recognizers are very efficient for one-stroke patterns.

201

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK

Recognition rate for off-line recognizer

Recognition rate for on-line recognizer

[%]

[%] 0.9

0.94

0.88

0.92 0.9

0.86

0.88

0.84

0.86 0.84

0.82

0.82 0.8

0.8 1

2

3

4

5

6

7

8

9

1

10 11 12 13 14

2

3

4

number of strokes

5

6

7

8

9

10 11 12 13 14

number of strokes

Graph 11: Recognition rate according to the stroke number for off-line recognizer.

Graph 12: Recognition rate according to the stroke number for on-line recognizer.

From Graph 12 we see that although an average rate of the on-line recognizer is about 5% worse than that of the off-line recognizer, for characters written by one, two, or three strokes the recognition rate is better, or at least similar. Since classifiers efficiency depends on the strokes number, we tried to make not only one general characteristic function, but 14 specific functions, where the last one is for patterns written by fourteen or more strokes. In Graph 13 we show some examples of stroke-dependent characteristic functions [charfi] for an on-line recognizer and in Graph 14 for an offline recognizer. Characteristic functions [chari]

Characteristic functions [chari]

[chari]

[chari]

600

200

#14

150

#14

400

#10

100

#5

0 -50

200

#10

50

0

#2 200

0 400

600

800

0

1000

200

400

-200

#5

600

-100

#1

-200

#1

#2

#5

# 10

[ai]

number of strokes

# 14

#1

Graph 13: Characteristic functions for online recognizer

5.

#1

-600

[ai]

number of strokes

1000

#2

-400

-150

800

#2

#5

# 10

# 14

Graph 14: Characteristic functions for offline recognizer

On-line & off-line classifiers and combination schemes

Our off-line recognizer represents each character as a 256-dimensional feature vector. Every input pattern is scaled to a 64x64 grid by non-linear normalization and smoothed by a connectivity-preserving procedure. Then, a normalized image is decomposed into 4 contour sub-patterns, one for each main orientation. Finally, a 64dimensional feature vector is extracted for each contour pattern from its convolution with a blurring mask (Gaussian filter). A pre-classification step precedes the actual final recognition. Pre-classification selects the 50 candidates with the shortest Euclidian distances between the categories’ mean vectors and the test pattern. The final classification uses a modified quadratic discriminant function (MQDF2) developed by

202

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK Kimura et al. [5] from traditional QDF. The off-line classifier is trained with the offline HP-JEITA database (3,036 categories, 580 patterns per category). The on-line recognizer presented in [12] by Nakagawa et al. employs a two-step coarse classification based on easily extractable on-line features in the first step and four-directional features in the second step. An efficient elastic matcher exploiting hierarchical pattern representations performs the main classification. The recognizer is trained with the on-line Nakayosi_t database[6] (163 patterns for each of 4,438 categories). We investigate two different combination strategies for combining our on-line and off-line recognizers: max-rule and sum-rule. Max-rule takes the class with the maximum output value among each classifier, while the sum-rule adds up the output for each class and selects the one with the highest sum[7].

6.

Benchmarking with the Kuchibue_d database

This section presents experiments evaluating our proposed normalization with respect to the Kuchibue_d database [6], [14], which is a widely acknowledged benchmark for Japanese character recognition. It contains handwritten sentences from a Japanese newspaper. In total, Kuchibue contains more than 1.4 million characters written by 120 writers (11,962 Kanji samples per writer). Kuchibue covers 3,356 Kanji categories including the two phonetic alphabets Hiragana and Katakana, plus alphanumerical characters. Since our recently published results were based on the ETL9B benchmark-database, which does not cover Katakana and alphanumerical characters, we confine our experiments to the 3036 categories of Kuchibue (Kanji and Hiragana) included in ETL9B. On-line patterns processed by the off-line classifier are converted to bitmaps by our calligraphic method [15] painting realistic off-line patterns. Table 1: Combination of an on-line and an off-line recognizer, tested on Kuchibue_d. Benchmark: Kuchibue_d

Single classifier On-line

Off-line

Combination of single classifiers AND

OR

without normalization Normalization by 1 characteristic function Normalization by 14 characteristic function

91.06

94.69

86.88

98.88

Max

Sum

91.44

96.29

95.61

97.13

96.27

97.71

The efficiency of our two single recognizers are 91.06% (on-line) and 94.69% (offline); Table 1, column 1 and 2. The higher recognition rate of the off-line recognizer is partially result of more training patterns per category (580 against 163) and smaller scope of recognizable categories (3036 against 4438). The next two columns show the theoretical worst (AND - a pattern was recognized by both classifiers) and best (OR a pattern was recognized at least by one recognizer) combination schemes, which are the theoretical bounds for any combination rule. Especially the OR combination scheme is important. Although it can never be utilized in a real application, the best theoretical boundary is important for comparing the efficiency of other combination schemes.

203

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK Column 5 and Column 6 of Table 1 show the recognition rates for the max-rule and sum-rule respectively. The sum-rule performs better than max-rule. A theoretical explanation why the sum-rule outperforms the max-rule is given in [7]. The sum-rule is also more resistant to problems of non-normalized likelihood. We have achieved improvements of single recognition rates from 91.06% and 94.69% up to 97.71%. And what is also important, our result is very near to the theoretical maximum 98.88% of the OR combination scheme.

7.

Benchmarking with the NTT-AT database

In this section we compare the results based on Kuchibue_d from Section 6 with results based on the NTT-AT database published in [16]. The NTT-AT database contains data written by elderly people with an average age of 70.5 years, with the oldest writer being 86 years old. These patterns are casually written, very often with an untypical stroke order. Table 2: Combination of on-line and off-line recognizers tested on NTT-AT. Benchmark: NTT-AT

Single classifier On-line

Off-line

Combination of single classifiers AND

OR

without normalization Normalization by 1 characteristic function Normalization by 14 characteristic function

85.67

89.07

79.03

95.71

Max

Sum

92.04

92.08

92.11

93.40

92.19

93.68

Table 2 shows that our normalization significantly improves the recognition rate. From 85.67% (on-line classifier) and 89.07% (off-line classifier) to 93.68%.

8.

Discussion

In this paper we presented an updated version of our accumulated-recognition-ratebased normalization for combining multiple classifiers, which was introduced at IWFHR 2001 [13,16]. A simple warping process aligns confidence values according to the accumulated recognition rate, so that the normalized values allow a fair combination. In our previous experiments on the NTT-AT database, which contains casually written characters often written in untypical writing order, we improved the recognition rate from 89.07% for the best single recognizer to 93.68%. In this paper we used the Kuchibue_d database, which is the well-established benchmark of common casually written Japanese characters. Its size is 23 times bigger than that of the NTT-AT database. The improvement of the recognition rate is from 94.69% for the best single recognizer to 96.29% for the sum-rule combination prior to our normalization, 97.13% when normalized by one common characteristic function, and finally 97.71% if normalized by 14 stroke-number-specific characteristic functions. The main advantages of our normalization based on the accumulated-recognitionrate are as follows: Firstly, the possibility to combine any number of classifiers, without having weaker classifiers degrade the final result. Secondly, it allows easy combi-

204

4th International Workshop on Multiple Classifier Systems (MCS2003) Guildford, UK nation of classifiers with incompatible confidence values. Thirdly, normalizing is a completely automatic process, without empirical parameter settings. And fourthly, normalization is performed only once and does not consume time during recognition. Normalization based on stroke numbers better reflects the nature of each recognizer and thus leads to better results; it requires more training patterns for computing multiple characteristic functions though. Although our accumulated-recognition-rate-based normalization was tested on combined on-line/off-line classifiers for on-line Japanese handwritten characters, it should be useful for combinations in various fields of pattern recognition.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.

M. Hamanaka, K. Yamada, J. Tsukumo, On-Line Japanese Character Recognition Experiments by an Off-Line Method Based on Normalization-Cooperated Feature Extraction, Proc. 2nd ICDAR (1993) 204-207 M. Okamoto, A. Nakamura, K. Yamamoto, Direction-Change Features of Imaginary Strokes for On-Line Handwriting Character Recognition, 14th ICPR (1998) 1747-1751 S. Jaeger, S. Manke, J. Reichert, A. Waibel, On-Line Handwriting Recognition: The Npen++ Recognizer, IJDAR 3(3) (2001) 169-180 F. Kimura, et al., Modified quadratic discriminant function and the application to Chinese characters, IEEE Pattern Analysis and Machine Intelligence, 9(1), pp.149-153, 1987 H. Kang, K. Kim and J. Kim, A Framework for Probabilistic Combination of Multiple Classifiers at an Abstract Level, EAAI 10 (4) (1997) 379-385 Distribution of the on-line databases Kuchibue_d and Nakayosi_t, Nakagawa laboratory, TUAT, Japan: http://www.tuat.ac.jp/~nakagawa/ipdb/ J.Kittler, M.Hatef, R.Duin, J.Matas, On Combining Classifiers, IEEE PAMI 20(3) (1998) 222-239 H. Tanaka, K. Nakajima, K. Ishigaki, K. Akiyama, M. Nakagawa, Hybrid Pen-Input Character Recognition System Based on Integration of On-Line – Off-Line Recognition, Proc. 5th ICDAR (1999) 209-212 T. Ho, J. Hull, S. Srihari, Decision Combination in Multiple Classifier Systems, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 16, no. 1, pp. 66-75, 1994 M. Oberländer, German Patent DE 44 36 408 C1 (in German), 1995, on-line available on the website of the German Patent and Trade Mark Office: www.depatisnet.de K. Matsumoto, T. Fukushima, M. Nakagawa, Collection and analysis of on-line handwritten Japanese character patterns, Proc. 6th ICDAR, Seattle, 2001, pp.496-500. M. Nakagawa, K. Akiyama, L.V. Tu, A. Homma, T. Higashiyama, Robust and Highly Customizable Recognition of On-Line Handwritten Japanese Characters, Proc. 13th ICPR (1996), volume III, 269-273 O. Velek, S. Jaeger, M. Nakagawa, A New Warping Technique for Normalizing Likelihood of Multiple Classifiers and its Effectiveness in Combined On-Line/Off-Line Japanese Character Recognition, Proc. 8th IWFHR(2002), pp. 177-182 S. Jaeger, M. Nakagawa, Two On-Line Japanese Character Databases in Unipen Format, Proc. 6th ICDAR(2001), pp. 566-570 O. Velek, M. Nakagawa, C.-L.Liu, Vector-to-Image Transformation of Character patterns for On-line and Off-line Recognition, International Journal of Computer Processing of Oriental Languages, Vol.15, No2 (2002) 187-209 O. Velek, M.Nakagawa, Using Stroke-Number Characteristics for Improving Efficiency of Combined On-Line and Off-Line Japanese Character Classifiers, 5th DAS (2002), 115-118

205