Intelligent Decision Support System for - Western Michigan University

Intelligent Decision Support System for Eye-Hand Coordination Assessment Janos L Grantner1, Ramakrishna Gottipati1, Norali Pernalete1 1

Electrical and Comp. Eng., Western Michigan University Kalamazoo, Michigan

Abstract - Our goal is to develop an automated assessment and training procedure for children with eye-hand coordination problem. An automated assessment system is expected to reduce the burden and the associated cost of having a trained professional present at any assessment, or training session. The intelligent decision support system will be based upon a fuzzy automaton. By using qualitative (fuzzified) data from the previous test the system will make a decision on the complexity of the next test to be performed. A set of assessment tests, commonly used by occupational therapists, were chosen to implement the various functions using force, inertia and viscosity effects. A test bed will be used for these tasks that consists of a six-degree-of-freedom force-reflecting haptic interface device called PHANToM along with the GHOST SDK Software, and the Intelligent Decision Support System software.

I. INTRODUCTION Learning to write is an important occupation of children [5, 10]. Problems with handwriting or drawing skills (graphomotor skills) are frequently the reason children in public schools are referred to occupational therapy services [8], [9]. There are five identified components that contribute to the quality of handwriting: kinesthesia, motor planning, eyehand coordination, visuomotor integration, and in-hand manipulation [7]. Children with genetic anomalies or neurological disorders can also have problems with eye-hand coordination such as children with Down syndrome, cerebral palsy, and learning disabilities. All these diagnoses require intervention for eye-hand coordination and often for grip strength because these components are necessary for successfully performing activities of daily living such as dressing, feeding, drawing, and writing. Assessing the eye-hand coordination skills of children with disabilities and making decisions on the next, more complex test by using crisp, quantitative terms may not be an optimum approach. Fuzzy logic [13] allows the aggregation of measured data and expert knowledge that is expressed in qualitative terms in a common mathematical model. A fuzzy automaton [14] can help in developing an intelligent decision system by recommending the sequence and complexity of the next test to be given. This recommendation will be based upon the results of the previously performed test and expert knowledge. The decision mechanism can be fine tuned as more test results become available.

George A Fodor2, Sandra Edwards3

2

ABB Automation Technology Products, Vasteras, Sweden 3 Occupational Therapy Western Michigan University Kalamazoo, Michigan

This paper is organized in the following way: Section 2 provides the background of previous applications of robotics for eye-hand coordination assessment and treatment. Section 3 covers a brief description of the standard research design for dexterity assessment tasks and the development of the assistance functions for haptic execution of these tasks. Section 4 describes the Intelligent Decision Support System. Preliminary results and conclusions are presented in Sections 5 and 6, respectively. II. BACKGROUND Extensive research is being done in the field of haptics to improve hand and arm movements. The concepts of motor control can be combined with robotics for a compelling application in rehabilitation and these robotics devices can be used for guiding patients along a desired trajectory [1]. One of the approaches taken was to expose a subject to a perturbing field to develop an internal model of the field as a relation between experienced limb states and forces [2]. In this study, the authors concluded that after-effects persisted after many trials using force field. Another study showed that haptic technologies can give rise to new generation of virtual environment applications for teaching motor skills and manual crafts [3]. The approach taken in that study is based upon a principle of rehabilitation in which simple movements can be improved by constant practice. Previous work by Bardorfer et all [4] shows labyrinths or mazes created in the virtual environment, in which the user has to move the pointer (ball) through it and could feel the reactive forces of the walls. In this paper we discuss the possibility of improving eyehand coordination in children with limited motor skills using a Robotic-Haptic device [6]. Force feedback can be used to guide the subjects hand in a predetermined trajectory when he/she is unable to move in response to a visual feedback. We also incorporate the inertia effect to decrease the tremor in the hand. In the following sections we show various designs developed in virtual environment with visual special effects to increase the concentration on the task and to assist the subjects in improving their eye-hand coordination. III. DESIGN OF EXPERIMENTS

The research approach is based upon the comparison of the performance of an experimental and a control group. The experimental group is exposed to standard occupational therapy tests as well as tests performed on the haptic robot. The control group works only on the standard tests. The sequence of tests given to each group is shown in Table 1. The initial (AInitial) and final (AFinal) data collection sessions will be used in the assessment of the baseline performance and the occupational performance changes of the subjects using the standardized assessments listed below: TABLE I

SEQUENCE OF TESTS Experimental Group Control Group AInitial

B

AFinal

Standardized Occupational Therapy Assessment Tasks (1 week) Intervention using Haptic Device (8 weeks) Standardized Occupational Therapy Assessment Tasks (1 week)

Standardized Occupational Therapy Assessment Tasks (1 week) --

Standardized Occupational Therapy Assessment Tasks (1 week)

A. Description of the Occupational Therapy Assessment Tasks 1) In-Hand Manipulation Test: It is used to assess the quality and speed of in-hand manipulation of children. 2) Sensory Profile (The Short Sensory Profile): The Sensory Profile provides a standard method for professionals to measure the sensory processing abilities of children 5 to 10 years old and to profile the effects of sensory processing on functional performance in the children’s daily lives. The Short Sensory Profile is a 38-item caregiver questionnaire and score sheet designed for use in screening and research protocols. The items are grouped into sensory processing, modulation, and behavioral and emotional responses. 3) Pencil Grasp Identification: The purpose is to determine the type of grasp being used and how functional it is. 4) Tseng Teacher Handwriting Checklist: The purpose of this checklist is to look for treatment carryover in the area of handwriting as a skill necessary in school. 5) Pincher Grip Strength: The purpose is to assess the strength of the fine motor grasps known as the 3-jaw chuck and lateral pinch to see if there is any change as the treatment progresses. 6) Motor Accuracy Test of The Sensory Integration Praxis Test: The Sensory Integration and Praxis Tests (SIPT) measure

the sensory integration processes that underlie learning and behavior. By showing how children organize and respond to the sensory input, SIPT helps pinpoint specific organic problems associated with learning disabilities, emotional disorders, and minimal brain dysfunction. 7) Minnesota Handwriting Assessment Test: This test was selected because of its ability to measure the quality of handwriting (rate, legibility, space, line size and form). The child is required to copy words on the sample (“the quick brown fox jumped over the lazy dogs”) and they represent the entire alphabet. The letters are in mixed order. The subject’s handwriting sample is scored in each category. The score in each category (except for the rate) is based upon the error rate using a maximum score of 34. The rate is graded by counting the number of letters completed in 2.5 minutes. B. Tasks Using the Haptic Device These sessions will be providing carefully selected tasks that make use of the Robotic Haptic Device for eye-hand coordination intervention. The hand grasp and function will be visually identified and noted during every intervention by the therapist who closely monitors for any changes. Each child will have two sessions of evaluation, AInitial and AFinal for 30 minutes each session. The child in the experimental group will receive intervention by means of the haptic robot once a week for 8 weeks, 20 minutes each. The child in the control group will not receive the intervention but will complete the assessments at the same time interval as the intervention group. Various labyrinth trajectories have been developed and tested as shown in Figures 1 and 2. The subject is required to move the ball along the specified trajectory. In addition to testing the free movement of the subject, various effects in form of assistant functions were also implemented such as: velocity assistance, plane constraints, inertia effect, viscosity effect and force feedback effect. C. Assistance Functions Development for the Tasks The underlying idea behind the assistance function concept is the generalization of position and velocity mappings between master and slave manipulators of a telerobotic system. This concept was conceived as a general method for introducing computer assistance in task execution without overriding an operator’s command to the manipulator. [11], [12]. The assistance functions can be classified as regulation of position and contact forces where, as effects, can be classified as inertia and viscosity effects. All of these assistance/effects strategies are accomplished by modification of the system parameters. A simple form of position assistance is scaling, in which the slave workspace is enlarged or reduced as compared to the master workspace whereas the force assistance function consists of imposing some constraints based on attractive or repulsive potential fields.

The inertia effect is implemented by setting the damping coefficient, mass, gravity and spring stiffness where the viscosity effect is implemented by setting a force opposite to the velocity of the PHANToM. The main idea of these effects is to increase the grip and hand strength in children during the task execution.

Fig. 1. Labyrinth Task User Interface

Fig. 2. Letter Pattern Used to test Handwriting The research design without the Intelligent Decision Support System is implemented and tested on a single subject with identified eye hand coordination problems [6]. The AInitial and AFinal of the Minnesota Handwriting Assessment results are shown in Figure 3 and the PHANToM robot used for the research is shown in Figure 4.

Fig. 3. Child’s Handwriting Before/After Therapy IV. DEVELOPMENT OF AN INTELLIGENT DECISION SUPPORT SYSTEM The functional block diagram of the Intelligent Decision Support System (IDSS) is given in Figure 5. In its full

configuration the system will be used for children in the age groups of 5 to 8 years. In the present phase of the research only children of 5-year old are considered.

Fig. 4. PHANToM Haptic Robot The system is made up of two major sections. The Main Controller section includes a computer workstation (PC), the PHANToM robot and the user interface software. The other key section is a simulated fuzzy automaton that is configured to accommodate the testing of the 5-year old children group. The task-level flowchart of the Intelligent Decision Support System is depicted in Figure 6. The notion of fuzzy automaton is based upon the premises as follows: it can stay in more than one crisp (i.e., Boolean) state at once, to a certain degree in each. Those degrees are defined by a state membership function. For each fuzzy state there is just one dominant (crisp) state, though, for which the state membership is a 1 (full membership). Each dominant state is associated with a linguistic model of the plant for inference. For each fuzzy state, a composite linguistic model is derived by composition of all contributing crisp states with a given strength matrix. The transitions between fuzzy states are based upon the transitions defined between their dominant crisp states. There is an underlying Boolean finite state machine to implement the fuzzy automaton. The states of this Boolean automaton are the dominant (crisp) states of the fuzzy automaton. Fuzzy (i.e., continuous signal) inputs are mapped to sets of two-valued logic variables. In addition, other signal input types include two-valued ones and continuous signals with a threshold. Conditions for state transitions are defined by any combination of signals of these three types. Fuzzy outputs are obtained by computing the compositional rule of inference using the composite linguistic models. Defuzzified outputs are derived by calculating a suitable defuzzification algorithm. Two-valued (Boolean) outputs are devised in the usual manner [14].

Fig. 5. Block Diagram of the IDSS

next test to be performed by the child. Based upon its present state and the fuzzified results of the test just performed the fuzzy automaton makes a transition to a new state. In this new state when it becomes the present state only one two-valued output is set, out of a 12-element output vector. This output is then used to choose the next task to be performed by the subject. Depending on his/her performance the subject may get stuck at some particular task level, or may move up to a more sophisticated level. If a subject performs really well the sequence of states will be as follows: 1, 4, 7, 11, and Stop. The level of difficulty in a task increases as the state number increases. However, the approval by the therapist is required to proceed with any new test. The therapist can also abort the test at any point in time. The state transition graph of the fuzzy automaton for the 5-year old children group is shown in Figure 7.

Fig. 6. Task Level Flowchart of the IDSS In this research, a simplified fuzzy automaton is being used. Inference operations (e.g., to classify the eye-hand coordination skills of the children in a qualitative scale) are omitted. The state transition graph of the automaton is used to devise the sequence and the complexity of the tests to be performed. The controller unit consists of the Graphical User Interface (GUI), which allows the therapist to enter the age of the subject and grants the permission to execute the tests suggested by the Intelligent Decision Support System. The main control unit takes the age and assigns the first task to be performed by the subject. It then calculates the accuracy (with respect to the number of times the subject is out of the desired path) and the time (in seconds) taken to carry on the task, then fuzzifies the values and provides them as inputs to the IDSS. The IDSS then makes use of the fuzzy automaton model to recommend the

Fig. 7. State Transition Graph of the Fuzzy Automaton The states are designated as follows: (1): initial state and no effects, or assistance, (2) and (3): design with maximum force assistance and minimum force assistance, respectively, (4): design with curves rather than with sharp edges in the path but no effects, or assistance, (5) and (6): design with minimum inertia effect, and medium inertia effect, respectively. The labyrinth for these states is illustrated in Figure 1. In State (7) the labyrinth in Figure 2 is considered with no effects, or assistance. State (8) is a design with force assistance. States (911) are designs with minimum, medium and maximum inertia

effects, respectively. State Stop is the last possible state in the sequence of tests. The number of states visited depends upon the performance of the subject. This state transition graph has been developed by following the advice of an occupational therapy expert. The state transition conditions are subject to further experiments and refinements. The input membership functions used in evaluating the state transition conditions are given in Figures 8 and 9.

Fig. 10. Raw Input Data for Accuracy

Fig. 8. Membership Functions for Accuracy V. SIMULATION RESULTS For initial verification of the system a test bed has been developed using Matlab. Random inputs have been provided to

Fig. 11. Raw Input Data for Time Taken Fig. 9. Membership Functions for Time Taken the system and the actual state transitions have been verified against the state transition graph developed. The simulation results are shown below in Figures 10-12. The raw input data (before fuzzification) for Accuracy and Time taken to perform the designated task are given in Figures 10 and 11, respectively. Figure 12 illustrates that all states have been reached during the trials. VI. CONCLUSIONS AND FUTURE WORK The initial results of a research to develop an Intelligent Decision Support System are presented. In the next step the IDSS will be loaded as a reconfigurable agent into the Broker Architecture program of the Intelligent Fuzzy Controllers Laboratory [15], [16] at Western Michigan University. The Broker Architecture program and the IDSS system will be integrated with the PHANToM using the Ghost software and will be tested on children with eye-hand coordination problems.

In the future the IDSS system will be extended in phases as follows: first, fuzzy automata will be developed for the other three age groups considered. After it has been successfully done and verified, we will work on the development of a qualitative system to classify the skill-levels of the pool of subjects. It will be done by adding the fuzzy inference capability to the automata. The If Then rules of the knowledge base will be constructed by interviewing experts of occupational therapy and also by extracting knowledge from the test data obtained from a larger pool of children. The final objective for the IDSS system is to develop a distributed system using the Internet. There will be no need for a therapist be present at each location, e.g., in classrooms in a wide geographical area. Instead, only one therapist in a central location will monitor the tests performed at various sites and will make decisions with respect to the test procedures.

Agents for Supervisory Control”, in the Proceedings of the FUZZIEEE’04 Conference, July 25-29, 2004, Budapest, Hungary. [16.] J. L. Grantner, "Intelligent Control Laboratory for Complex Hybrid System", DURIP'2001 Award (DAAD19-01-1-0431)

Fig. 12. States Visited During the Trials REFERENCES [1.] H. Kerbs, N. Hogan, M. Aisen and B. Volpe, “Robot-aided neurorehabilitation,” IEEE Transactions on Rehabilitation Engineering, Vol. 6, pp. 75-87, 1998. [2.] F. Mussa-Ivaldi A., J. Patton L., “Robots can teach people how to move their arm,” International Conference on Robotics and Automation. pp. 300-305, 2000. [3.] G. Prisco M., C. Avizzano A., M. Calcara, S. Ciancio, S. Pinna and M. Bergamasco, “A virtual environment with haptic feedback for the treatment of motor dexterity disabilities,” International Conference on Robotics and Automation. pp. 3721-3726, 1998. [4.] Aleš Bardorfer, Marko Munih, Anton Zupan, Alenka Primožič, “Upper Limb Motion Analysis Using Haptic Interface,” IEEE/ASME Transactions on Mechatronics, VOL.6, NO.3, pp3721-3726 September 2001. [5.] Amundson, S.J. (1995). Evaluation Tool of Children’s Handwriting. Homer, AK: OT KIDS. [6.] N Pernalete, R Gottipati, S Mikkilineni, S Edwards, E McCann, RV Dubey,W Yu, “Eye-Hand Coordination Assessment using a Robotics Haptic Device,” International Conference on Robotics and Automation. 2004 [7.] Cornhill, H., Case-Smith, J. (1996). Factors that relate to good and poor handwriting. American Journal of Occupational Therapy 50, 732-738. [8.] Edwards, S., Buckland, D. & McCoy-Powlen, J. (2002). Developmental and Functional Hand Grasps, New Jersey, Slack. [9.] Diekema, S. M., Deitz, J., & Amundson, S. J. (1998). Evaluation Tool of Children’s Handwriting-Manuscript. American Journal of Occupational Therapy, 52, 248-254 [10.] McHale, K., & Cermack. S.A. (1992). Fine motor activities in elementary school: preliminary findings and provisional implications for children with fine motor problems. American Journal of Occupational Therapy, 46, 898-903. [11.] N. Pernalete. “Development of a Robotic Haptic Interface to Perform Vocational Tasks by People with Disabilities” Ph.D. Dissertation University of South Florida, 2001. [12.] N. Pernalete, Wentao Yu, R.V. Dubey, W.A. Moreno. “Development of a Robotic Haptic Interface to Assist the Performance of Vocational Tasks by People with Disabilities”. The 2002 IEEE International Conference on Robotics and Automation. Washington D.C., May, 2002. [13.] L. A. Zadeh, Fuzzy Sets, Information and Control, Vol. 8, 1965. [14.] J. L. Grantner, G. A. Fodor, “Fuzzy Automaton for Intelligent Hybrid Control Systems”, 2002 World Congress on Computational Intelligence, WCCI'02/FUZZ-IEEE’02, May 12-17, 2002, Hilton Hawaiian Village Hotel, Honolulu, HI, in the CD Proceedings, ISBN: 0-7803-7281-6 [15.] J. L. Grantner, G. A. Fodor, R. Gottipati, “Fuzzy Logic Enabled Software