A hybrid approach of automatic fuzzy rule generation for ... - CiteSeerX

1 downloads 2564 Views 48KB Size Report
German National Research Center for Information Technology (GMD) ... fuzzy rules contain the feature information extracted from a given prototype data set.
Published in the Pre-proceedings of the Fifth Workshop on the Frontiers of Handwriting Recognition (IWFHR5), Colchester, 1996.

A hybrid approach of automatic fuzzy rule generation for handwriting recognition Ashutosh Malaviya, Christoph Leja and Liliane Peters German National Research Center for Information Technology (GMD) Schloß Birlinghoven, 53754 St. Augustin, Germany [email protected], http://set.gmd.de/EIA/fohres.html

Abstract This paper presents a hybrid approach of automatic fuzzy rule generation for on-line handwriting recognition. The fuzzy rules contain the feature information extracted from a given prototype data set. The fuzzy statistical measures and neural networks are used to select the associative features from the input symbols. The final decision is enhanced through additional combination with expert’s knowledge. The rule base is coded in a dedicated fuzzy language for an interactive processing. The proposed method is applied to generate a rule base employing UNIPEN database for online recognition of isolated handwritten characters.

1

Introduction

The human visual system functions successfully even when patterns possess a certain amount of vagueness, slight mismatch and imprecision. It is able to select those specific features which fit the prototypes. To imitate the natural perception mechanism, we have applied fuzzy logic to recognize on-line handwritten characters [4]. Our approach is based on a multi-level fuzzy rule based paradigm [7] which compensates the noise, deviation from the ideal prototype and also incomplete information at each recognition stage. The inherent geometrical shape distortions of handwriting are primarily due to the various handwriting styles and secondly due to the acquisition problems. The first layer or low-level layer of the recognition scheme performs the segmentation of the preprocessed input handwriting profile. The characters are divided into various geometrical domains applying a rule based segmentation method which uses handwriting dynamics and relative curvature information. This information is converted into the linguistic description of sharp corners, isolated dots, ends and pen-up to form the segmentation rules [6]-[7]. Interpretation of the structural vagueness represents the intermediate level. It corresponds to the extraction of geometrical features with the help of fuzzy linguistic values. The shape information for each segmented portion is computed as the degree of membership to basic feature primitives. From these extracted features for all segments, relevant attributive features are combined and complex meaningful features are generated with fuzzy aggregation [7].

FOHRES UNIPEN Data-base

On-line On-line

Preprocessing and Feature Extraction

Stochastic Hybrid and RuleNeural generation Learning

FOHDEL Rule Base

Diagnosis Classification Classification

Figure 1 Overview of the system.

The fuzzy aggregation methods support the integration of scattered and inconsistent information. The fuzzy structural features obtained in the intermediate stage have certain correspondence to each other, but this correspondence does not always have unequivocal meanings. To overcome these syntactical ambiguities, we have developed a dedicated fuzzy language FOHDEL1 [9]. Thus in the learning phase, characteristic features of various symbols are described in the form of a FOHDEL rule-base. Here the relations between the extracted features are defined and combined as FOHDEL rules. In the classification phase the extracted features of unknown symbols are computed as the membership of the parsed symbol rules. The classification output is a response matrix of the five best fits[8]. The advantage of FOHDEL rule-base is that, the handwriting information of various handwriting styles can be represented in a compressed form with a minimal number of features. Fuzzy logic provides a framework for interpreting the vague, linguistic rules. The process of acquisition of such knowledge base is the main task of making reliable rule based systems. The robustness of the recognition system is highly dependent upon the quality of rules. The rules are required to be correct, relevant and complete. Frequently the fuzzy algorithms provided by experts do not satisfy these requirements. On one hand rules are vague and can be misinterpreted, and on the other hand such rule-base may not be complete. With the increasing complexity of the handwriting patterns and number of writers it is increasingly complex to write such rules manually. Our previous experience of automatic rule generation has shown that fully automated rule generation also disrupts the flexibility of the fuzzy recognition system. The rule-base is thus generated by a hybrid approach combining neural network learning, statistical learning and finally expert’s knowledge. (Fig. 1) 2 A hybrid approach of automatic FOHDEL rule generation A number of methods are used to extract fuzzy rules automatically from a given test data. These methods utilize clustering analysis, statistical measures, neural networks and recently genetic algorithms. But in most of these approaches the extracted prototype is highly dependent on the given data set. To overcome this constraint the number of training samples is increased which in turn increases the computational time. Fuzzy logic provides a linguistic rule flexibility such that from a small number of prototypes a widely valid rule base is created. The primary goal of a rule base generation method is to create a minimum number of rules. This is important under two observed aspects in pattern recognition systems. The discrimination power of the classification process is inversely dependent on the number of classification rules. The number of selected features within a rule has a direct impact on the computational time of the recognition process. How can this goal be achieved? Very frequently structure identification methods are used to extract the key information automatically from the raw information data, and subsequently to convert this into a rule set. The aim of these methods is to recognize the cause-effect relations between input and output information. But this is just one aspect of the problem. Moreover there are some points which remain to be addressed such as: validation, redundancy and consistency check of the generated rule base.

Feature Data

Feature Reduction with Statistical Constraints

Neural Network Learning

Rule Aggregation

Rule a1: Human Observation

Statistical Learning

Very High O-Like left

VVH#OL_L

Very High hook at right

VVH#VL_L

. . . .

Figure 2 Hybrid rule generation approach.

1. Fuzzy On-line Handwriting Description Language

O-Like on the medium left

Left Vertical line right Hockey on the right

Right O-Like Figure 3 Linguistic features of two characters ‘a‘ and ‘b‘ . In the proposed method we utilize fuzzy statistical measures and neural networks. The final decision is enhanced through additional combination with expert’s knowledge. Based on the fuzzy statistical measures first we reduce the automatic key feature extraction and statistical rule generation process. The back propagation neural network[2] is used to generate the membership functions for some typical features of symbols [5]. The refinement of the solution is accomplished with an additional correlation method, that detects redundancies within the rule base. Selection of the discriminating information in terms of associative features is the most essential part of the feature extraction process. A rule aggregation module integrates the features generated by various schemes to build the rule base. The FOHRES system gives the possibility to extract over 120 features. These features describe shapes of the unknown symbol [6]. The feature data is the input to our rule generator (Fig. 2). The feature reduction is based on the feature correlation matrix and the fuzzy standard deviation. The output for each given unknown symbol is a list of segments characterized by a given set of features. As the features are complementary like {“left” “right”}, or {“circle” “line”} the belongingness of one symbol segment to all features with the same possibility is impossible. Thus each segment has a specific distribution of shape features. The variability of handwritten characters is shown in Fig.4. The statistical analysis of these features over a training set gives the distribution parameters for each feature. Based on the fuzzy average and the variance a fuzzy measure of feature fitness is computed. This measure gives us the possibility to aggregate the fitness and determine if the feature should be considered for further rule generation [6]. The statistical rules are generated by adapting the distribution values as the membership functions. Similarly the artificial neural networks also produce a large set of rules which are reduced by a statistical evaluation [5]. The next important task is to reduce the number of existing prototype rules. A rule aggregation module combines the generated rules by these methods. The first step is straight forward and consists of comparison of the rules and sorting them based on the above described fitness criterion and weights for each feature. The best suited prototype feature set for a rule is a fuzzy feature set with the highest fitness value. Redundant information, like rule candidates with the same fuzzy feature sets are also discarded. The knowledge based improvement of the rules is the aggregation of rules with the uniform fuzzy feature attribute distribution, e.g. the aggregation of “very high” and “very very high” into “more than very high”. By comparing these fuzzy attributes neighborhood related description can be combined. After this aggregation and redundancy removal process the remaining number of rules are automatically transformed into FOHDEL rules (Fig. 4). The features are combined with their corresponding linguistic attributes. For features with several attributes as they express an alternative an OR operator is used [8]. 3 Results The UNIPEN benchmarks database [3] is utilized to generate the rule base for recognizing isolated handwriting characters. The initial training data-set consisted of ten sets of symbols written by three different writers, i.e. 36 x 10 = 360 symbols. A primary rule base is generated automatically with the statistical analysis and also by the proposed hybrid approach. Later these two rule-bases were used to recognize fifty distinct symbol sets from ten altogether different writers (36 x 50 =1800).

O-Like or C-like at the left and a small hook at the right.

Rule a: (VH#OL_L| VH#CL_L) &

Vertical line at the left and an O-like or D-like at the right.

Rule b: (VH#VL_L | H#VL_M) & (VVH#OL_BR | VH#DL_MR)

A C-like curve in the middle.

(H#HOR_R | H#NS_MR)

Rule c: VVH#C_MM

A O-Like curve at the left, a vertical line in the middle and a hook at the right.

Rule d: VH#OL_L & VH#VL_M & (H#HOR_R | H#NS_MR)

A positive slant upwards or a hook at the left and a C-like curve at the right.

Rule e: (VVH#PS_ML | VH#NS | VH#HL_M) & VVH#CL_M

Figure 4. Sample rule generation from handwriting data (´a´-´e´)

For a fair comparison the same set of data is taken to generate the rules by two methods. The hybrid method has achieved an overall average of 92.3%. On the other hand the automatic method showed relatively poor performance of 87.4%. We have shown the applicability of a hybrid learning method by combining the statistical, neural and human expert’s based knowledge to processing the handwriting information. References 1. 2. 3. 4. 5. 6. 7. 8. 9.

M. Ishikawa, “Structural Learning and its Applications to Rule Extraction,” ICNN, pp.354-359, Orlando, 1994. I. Guyon,”Applications of Neural Networks to Character Recognition,” in Character and Handwriting Recognition: Expanding frontiers,Ed: P.S.P. Wang, World Scientific,pp.353-382,1991. I. Guyon et al., “UNIPEN project of on-line data exchange and recognition benchmarks,’’ 13th IEEE-ICPR, pp. 29-33, Israel, 1994. J. M. Keller et al.,”Evidence Aggregation networks for fuzzy logic inference,” IEEE T. on Neural Networks, vol.3, No.5,pp.761-769,Sept. 1992. C. Leja, Entwicklung und Verifikation von intelligenten Lernverfahren für regelbasierte Systeme sowie einer geeigneten graphischen Benutzeroberfläche, Diplomarbeit, Uni. Siegen, October, 1995. A. Malaviya and L. Peters, “Extracting meaningful handwriting features with fuzzy aggregation method,” 3rd ICDAR, Montreal, 1995. A. Malaviya and L. Peters, “Handwriting recognition with fuzzy linguistic rules,’’ 3rd EUFIT, Aachen, 1995. A. Malaviya et al, “FOHDEL: A New Fuzzy Language for On-line Handwriting Recognition,” FUZZ-IEEE, pp.624-629, Orlando, 1994. A. Malaviya et al, “Automatic generation of fuzzy rule base for online handwriting recognition,” 2nd EUFIT, pp.1060-1065, September, 1994.

Published in the Pre-proceedings of the Fifth Workshop on the Frontiers of Handwriting Recognition (IWFHR5), Colchester, 1996.

A hybrid approach of automatic fuzzy rule generation for handwriting recognition Ashutosh Malaviya, Christoph Leja and Liliane Peters German National Research Center for Information Technology (GMD) Schloß Birlinghoven, 53754 St. Augustin, Germany [email protected], http://set.gmd.de/EIA/fohres.html

Abstract This paper presents a hybrid approach of automatic fuzzy rule generation for on-line handwriting recognition. The fuzzy rules contain the feature information extracted from a given prototype data set. The fuzzy statistical measures and neural networks are used to select the associative features from the input symbols. The final decision is enhanced through additional combination with expert’s knowledge. The rule base is coded in a dedicated fuzzy language for an interactive processing. The proposed method is applied to generate a rule base employing UNIPEN database for online recognition of isolated handwritten characters.

1

Introduction

The human visual system functions successfully even when patterns possess a certain amount of vagueness, slight mismatch and imprecision. It is able to select those specific features which fit the prototypes. To imitate the natural perception mechanism, we have applied fuzzy logic to recognize on-line handwritten characters [4]. Our approach is based on a multi-level fuzzy rule based paradigm [7] which compensates the noise, deviation from the ideal prototype and also incomplete information at each recognition stage. The inherent geometrical shape distortions of handwriting are primarily due to the various handwriting styles and secondly due to the acquisition problems. The first layer or low-level layer of the recognition scheme performs the segmentation of the preprocessed input handwriting profile. The characters are divided into various geometrical domains applying a rule based segmentation method which uses handwriting dynamics and relative curvature information. This information is converted into the linguistic description of sharp corners, isolated dots, ends and pen-up to form the segmentation rules [6]-[7]. Interpretation of the structural vagueness represents the intermediate level. It corresponds to the extraction of geometrical features with the help of fuzzy linguistic values. The shape information for each segmented portion is computed as the degree of membership to basic feature primitives. From these extracted features for all segments, relevant attributive features are combined and complex meaningful features are generated with fuzzy aggregation [7].

FOHRES UNIPEN Data-base

On-line On-line

Preprocessing and Feature Extraction

Stochastic Hybrid and RuleNeural generation Learning

FOHDEL Rule Base

Diagnosis Classification Classification

Figure 1 Overview of the system.

The fuzzy aggregation methods support the integration of scattered and inconsistent information. The fuzzy structural features obtained in the intermediate stage have certain correspondence to each other, but this correspondence does not always have unequivocal meanings. To overcome these syntactical ambiguities, we have developed a dedicated fuzzy language FOHDEL1 [9]. Thus in the learning phase, characteristic features of various symbols are described in the form of a FOHDEL rule-base. Here the relations between the extracted features are defined and combined as FOHDEL rules. In the classification phase the extracted features of unknown symbols are computed as the membership of the parsed symbol rules. The classification output is a response matrix of the five best fits[8]. The advantage of FOHDEL rule-base is that, the handwriting information of various handwriting styles can be represented in a compressed form with a minimal number of features. Fuzzy logic provides a framework for interpreting the vague, linguistic rules. The process of acquisition of such knowledge base is the main task of making reliable rule based systems. The robustness of the recognition system is highly dependent upon the quality of rules. The rules are required to be correct, relevant and complete. Frequently the fuzzy algorithms provided by experts do not satisfy these requirements. On one hand rules are vague and can be misinterpreted, and on the other hand such rule-base may not be complete. With the increasing complexity of the handwriting patterns and number of writers it is increasingly complex to write such rules manually. Our previous experience of automatic rule generation has shown that fully automated rule generation also disrupts the flexibility of the fuzzy recognition system. The rule-base is thus generated by a hybrid approach combining neural network learning, statistical learning and finally expert’s knowledge. (Fig. 1) 2 A hybrid approach of automatic FOHDEL rule generation A number of methods are used to extract fuzzy rules automatically from a given test data. These methods utilize clustering analysis, statistical measures, neural networks and recently genetic algorithms. But in most of these approaches the extracted prototype is highly dependent on the given data set. To overcome this constraint the number of training samples is increased which in turn increases the computational time. Fuzzy logic provides a linguistic rule flexibility such that from a small number of prototypes a widely valid rule base is created. The primary goal of a rule base generation method is to create a minimum number of rules. This is important under two observed aspects in pattern recognition systems. The discrimination power of the classification process is inversely dependent on the number of classification rules. The number of selected features within a rule has a direct impact on the computational time of the recognition process. How can this goal be achieved? Very frequently structure identification methods are used to extract the key information automatically from the raw information data, and subsequently to convert this into a rule set. The aim of these methods is to recognize the cause-effect relations between input and output information. But this is just one aspect of the problem. Moreover there are some points which remain to be addressed such as: validation, redundancy and consistency check of the generated rule base.

Feature Data

Feature Reduction with Statistical Constraints

Neural Network Learning

Rule Aggregation

Rule a1: Human Observation

Statistical Learning

Very High O-Like left

VVH#OL_L

Very High hook at right

VVH#VL_L

. . . .

Figure 2 Hybrid rule generation approach.

1. Fuzzy On-line Handwriting Description Language

O-Like on the medium left

Left Vertical line right Hockey on the right

Right O-Like Figure 3 Linguistic features of two characters ‘a‘ and ‘b‘ . In the proposed method we utilize fuzzy statistical measures and neural networks. The final decision is enhanced through additional combination with expert’s knowledge. Based on the fuzzy statistical measures first we reduce the automatic key feature extraction and statistical rule generation process. The back propagation neural network[2] is used to generate the membership functions for some typical features of symbols [5]. The refinement of the solution is accomplished with an additional correlation method, that detects redundancies within the rule base. Selection of the discriminating information in terms of associative features is the most essential part of the feature extraction process. A rule aggregation module integrates the features generated by various schemes to build the rule base. The FOHRES system gives the possibility to extract over 120 features. These features describe shapes of the unknown symbol [6]. The feature data is the input to our rule generator (Fig. 2). The feature reduction is based on the feature correlation matrix and the fuzzy standard deviation. The output for each given unknown symbol is a list of segments characterized by a given set of features. As the features are complementary like {“left” “right”}, or {“circle” “line”} the belongingness of one symbol segment to all features with the same possibility is impossible. Thus each segment has a specific distribution of shape features. The variability of handwritten characters is shown in Fig.4. The statistical analysis of these features over a training set gives the distribution parameters for each feature. Based on the fuzzy average and the variance a fuzzy measure of feature fitness is computed. This measure gives us the possibility to aggregate the fitness and determine if the feature should be considered for further rule generation [6]. The statistical rules are generated by adapting the distribution values as the membership functions. Similarly the artificial neural networks also produce a large set of rules which are reduced by a statistical evaluation [5]. The next important task is to reduce the number of existing prototype rules. A rule aggregation module combines the generated rules by these methods. The first step is straight forward and consists of comparison of the rules and sorting them based on the above described fitness criterion and weights for each feature. The best suited prototype feature set for a rule is a fuzzy feature set with the highest fitness value. Redundant information, like rule candidates with the same fuzzy feature sets are also discarded. The knowledge based improvement of the rules is the aggregation of rules with the uniform fuzzy feature attribute distribution, e.g. the aggregation of “very high” and “very very high” into “more than very high”. By comparing these fuzzy attributes neighborhood related description can be combined. After this aggregation and redundancy removal process the remaining number of rules are automatically transformed into FOHDEL rules (Fig. 4). The features are combined with their corresponding linguistic attributes. For features with several attributes as they express an alternative an OR operator is used [8]. 3 Results The UNIPEN benchmarks database [3] is utilized to generate the rule base for recognizing isolated handwriting characters. The initial training data-set consisted of ten sets of symbols written by three different writers, i.e. 36 x 10 = 360 symbols. A primary rule base is generated automatically with the statistical analysis and also by the proposed hybrid approach. Later these two rule-bases were used to recognize fifty distinct symbol sets from ten altogether different writers (36 x 50 =1800).

O-Like or C-like at the left and a small hook at the right.

Rule a: (VH#OL_L| VH#CL_L) &

Vertical line at the left and an O-like or D-like at the right.

Rule b: (VH#VL_L | H#VL_M) & (VVH#OL_BR | VH#DL_MR)

A C-like curve in the middle.

(H#HOR_R | H#NS_MR)

Rule c: VVH#C_MM

A O-Like curve at the left, a vertical line in the middle and a hook at the right.

Rule d: VH#OL_L & VH#VL_M & (H#HOR_R | H#NS_MR)

A positive slant upwards or a hook at the left and a C-like curve at the right.

Rule e: (VVH#PS_ML | VH#NS | VH#HL_M) & VVH#CL_M

Figure 4. Sample rule generation from handwriting data (´a´-´e´)

For a fair comparison the same set of data is taken to generate the rules by two methods. The hybrid method has achieved an overall average of 92.3%. On the other hand the automatic method showed relatively poor performance of 87.4%. We have shown the applicability of a hybrid learning method by combining the statistical, neural and human expert’s based knowledge to processing the handwriting information. References 1. 2. 3. 4. 5. 6. 7. 8. 9.

M. Ishikawa, “Structural Learning and its Applications to Rule Extraction,” ICNN, pp.354-359, Orlando, 1994. I. Guyon,”Applications of Neural Networks to Character Recognition,” in Character and Handwriting Recognition: Expanding frontiers,Ed: P.S.P. Wang, World Scientific,pp.353-382,1991. I. Guyon et al., “UNIPEN project of on-line data exchange and recognition benchmarks,’’ 13th IEEE-ICPR, pp. 29-33, Israel, 1994. J. M. Keller et al.,”Evidence Aggregation networks for fuzzy logic inference,” IEEE T. on Neural Networks, vol.3, No.5,pp.761-769,Sept. 1992. C. Leja, Entwicklung und Verifikation von intelligenten Lernverfahren für regelbasierte Systeme sowie einer geeigneten graphischen Benutzeroberfläche, Diplomarbeit, Uni. Siegen, October, 1995. A. Malaviya and L. Peters, “Extracting meaningful handwriting features with fuzzy aggregation method,” 3rd ICDAR, Montreal, 1995. A. Malaviya and L. Peters, “Handwriting recognition with fuzzy linguistic rules,’’ 3rd EUFIT, Aachen, 1995. A. Malaviya et al, “FOHDEL: A New Fuzzy Language for On-line Handwriting Recognition,” FUZZ-IEEE, pp.624-629, Orlando, 1994. A. Malaviya et al, “Automatic generation of fuzzy rule base for online handwriting recognition,” 2nd EUFIT, pp.1060-1065, September, 1994.

Suggest Documents