Modeling Perceptual-Gestural Knowledge for ...

Modeling Perceptual-Gestural Knowledge for Intelligent Tutoring Systems Dedicated to Ill-Defined Domains Ben-Manson Toussaint1, Vanda Luengo1, Francis Jambon1, and Jérôme Tonetti2 Laboratory of Informatics of Grenoble, University Joseph Fourier, Grenoble, France {ben-manson.toussaint, vanda.luengo, francis.jambon}@imag.fr

1

Michallon Hospital, Orthopedic and Trauma Department, Grenoble, France [email protected]

2

Abstract. Perceptual-gestural knowledge is often tacit, empirical and so, difficult to formalize and embed in Intelligent Tutoring Systems. The challenge grows when the underlying domains are ill-defined since most traditional paradigms are not adapted to their specificity. In this paper, we present our methodology for modeling this category of knowledge and tutoring services for an ill-defined domain. Our case study is a simulationbased Intelligent Tutoring System, TELEOS, dedicated to percutaneous orthopedic surgery. We detail our propositions for addressing encountered scientific obstacles, from data acquisition in the real working environment to the modeling of tutoring services. This includes formalizing, representing and diagnosing perceptual-gestural and procedural knowledge as for producing adaptive didactic feedback. Evaluation results showed the pertinence of the implemented tutoring services but highlight some limits like their incompleteness. We discuss these limits and envisaged research solution to address them. Keywords: Ill-Defined Domains, Simulation-Based Tutoring Systems, Perceptual-Gestural Knowledge Modeling

1 Introduction Ill-defined domains [12] have brought great challenges to the TEL community as the traditional paradigms and approaches of the state of the art are not suitable for the implementation of Intelligent Tutoring Systems dedicated to these domains. One example of ill-defined domain is surgery as there is no complete theoretical framework of the domain and there exist different ways to realize a given operation or to react to incoming situations during a surgical intervention. In the work presented here, we are interested to percutaneous orthopedic surgery, specifically vertebroplasty and sacra-iliac screw fixation. Percutaneous orthopedic operations are practiced to treat bones lesion with implants or cement injection from small incisions in the skin. In this type of operation, as opposed to classical surgery, surgeons are guided all along their activity by x-rays. Because of the nerves alongside the targeted anatomic areas, the vertebra and the sacrum, it is crucial that the surgical instruments do not

trespass intra-osseous area. This requires that surgeons master the coordination of anatomic knowledge, radiographic images guidance and various progression insights given by his instruments at different points of insertion: contact with bones, part of the bones being impacted since their structure varies from spongy to stiff. For example, the body (centrum) of a vertebra is spongy and requires low pressures during insertion while cortical are very stiff. In learning activities, teaching surgeons can hardly cover and transmit to their trainees all the possible incoming situations as some solutions and/or ways to arrive at these solutions are not predictable [17]; particularly for perceptual-gestural knowledge which is difficult to address because they are often tacit and impossible to be formulated in a declarative way only. Their construction and validation are empirical: mastering them requires repeated practices. The traditional surgical training approach includes formal learning of declarative knowledge followed by practical trainings on real clinical cases. Novices learn by watching and participating progressively in a more active way as their experience increases but this format induces a gap in the learning process [18]. The work presented in this paper aimed at creating an intermediate phase of learning which provides an operative dimension of knowledge between formal learning and apprenticeship in real situations. The learning environment implemented for that purpose, namely, TELEOS (Technology Enhanced Learning Environment for Orthopedic Surgery), is a simulation-based Intelligent Tutoring System.

It relies on theories from artificial

intelligence, computer science and didactic, and also outcomes from medical imaging and cognitive sciences [19]. To acquire the necessary understanding of the surgical activity, we first proceeded to didactic analyses and psycho-ergonomic studies from observations in real working environment at the University Hospital of Grenoble. We gathered data from simulated environment for observations that were impossible to realize in surgical room like gestures analysis requiring for example dynamometers for measuring applied force on the trocar to penetrate the bones [19]. This study has been completed with interviews with expert teaching surgeons for triggering actions rules, information analysis, controls and reasoning involved in percutaneous surgical operations. The results of these analyses highlighted the worth-considered elements of knowledge in the learning process of the domain. As the application domain relates to a professional practice, theoretical tools brought by didactical research applied to the professional domain of apprenticeship have been used [8] [9], in addition to the theory of didactical situations of Brousseau [3]. Learners can practice freely on training-oriented exercises offered on “problem solving” teaching base. They progress following epistemic feedback generated from their diagnosed knowledge in a constructivist view.

Fig. 1. TELEOS methodology of implementation

2 Related Works One of the most recent works related to ITS for ill-defined domains is CanadarmTutor, a simulation-based Intelligent Tutoring System for training the handling of a robotic arm on the International Spatial Station [5]. This activity is ill-defined since to realize a given task with the robotic arm, many solutions exist with no best predefined beginning point. The teaching model of CanadarmTutor is problem solving. Learners can freely explore the tutoring services and practice training. The reasoning paradigm applied in CanadarmTutor’s tutoring services is an hybrid approach combining expert system, model-tracing and automatic acquisition of partial task [6]. This latter is based on experts’ patterns activity extraction: brute traces from experts and advanced trainees activity are treated to model correct actions on which didactic feedbacks are produced. Another close work is a simulation-based Intelligent Tutoring System for helicopter piloting: IFT (Intelligent Flight Trainer) [15]. This system uses adaptive tutoring services to guide trainees through their progress in mastering the settled procedures of helicopter handling. Declarative and procedural knowledge are involved in this activity. IFT offers training exercises adapted to learners’ knowledge level (novice or advanced) and psychological profile (introvert or extravert). It integrates online feedbacks that target fails in following procedures steps and produce live instructions to recover correct handling regarding these procedures. In our case, automatic acquisition of domain knowledge from experts’ traces would be biased since it is possible for experts to achieve solution using different sets of actions. Consequently, we cannot generate our knowledge model exclusively on experts’ actions or fixed procedures. In our approach, actions leading to failures are as important as correct ones and are considered in knowledge modeling process. In fact, errors as well as successes can be set as controls on which tutoring services focus as applied knowledge underlying actions and decisions and as parameters for their validation.

3 TELEOS Architecture TELEOS has been developed as an opened architecture in order to facilitate evaluation, maintenance and extension of its functionalities independently. The accessible tools to the user are separated from the tutoring services’ models environment. Knowledge model, diagnostic and didactic feedback modules are separated in this architecture as to integrate potential future extensions that we have not addressed yet in our research: for example, tutoring services based on emotional profile. On user’s side, TELEOS environment presents a 3D simulation software [11] providing the trainees with simulation exercises of vertebroplasty and sacra-iliac screw fixation. A database of clinical cases provides learners with patients’ models that he can freely choose for practicing. These latter are complete patients’ information files gathered at the Orthopedic and Traumatology Department of the University Hospital of Grenoble; then modeled and saved in a dedicated database [10]. During a simulation session, the software allows learners to visualize patients’ models and surgical instruments in 3D including the fluoroscope that is used to take x-rays for visual guidance. They can practice surgical gestures and activities involved in real world percutaneous orthopedic surgery such as marking targeted areas on patient’s skin; operate the fluoroscope; generate and visualize xrays; operate the trocar used for cement injection or screw implant.

Fig. 2. TELEOS simulation interface and the real working environment

The background of TELEOS system includes a database of theoretical courses, an authoring tool for the administration of the knowledge model, the knowledge diagnosis module and the didactic feedback module. We will detail further the diagnosis and didactic feedback processes in the next section.

In the current version of the learning environment, the communication flows between the different parts during a simulation session run as follows: the trainee uses the 3D simulation software and the haptic arm to apply various surgical actions that are traced all along the process. At the end of the exercise, the software transfers these traces to the diagnostic module which analyses them to determine the learner’s knowledge profile. The diagnosed traces are then processed to the feedback module which analyses and returns pedagogical suggestions. These latter can be

theoretical courses to revise or practical

simulation to realize focusing on reinforcement, destabilization, hints, scaffolding, etc. depending on knowledge weaknesses or strengths identified by the diagnosis.

Fig. 3. TELEOS Architecture

4 The Learning Environment 4.1 The Authoring Tool for Knowledge Model Administration An authoring tool has been developed in order to allow didactic experts to manage didactic variables of the system with no needed technical or programming skill. The features of this tool include also the administration of knowledge diagnosis rules, and exercises metadata. The knowledge model is based on the ck¢ model [2] that represents concepts of a domain in 4 sets: – – – –

A set of problems P ; A set R of operators (or actions) applied in a problem resolution process; A set Σ of controls structures (or theoretical knowledge) validation or rejection of applied actions rely on ; A set L of representation systems used during a problem resolution process

Table 1. Examples of formalized elements of knowledge with ck¢ model Problems (P) Pb: Operation of a lumbar vertebra

Operators (R) R8: Impact on bone

Representations (L) L2: Body anatomy (3D)

Pc: Operation of a fractured sacrum

R9 : Push the trocar up to 1 cm after midline

L4 : Inlet x-ray (2D)

Controls (Σ) Σ131: If the trocar is below the pedicles, then it damages the vertebral foramina Σ9 : the trocar must pass below the sacral canal

A problem (p) is characterized by didactical variables modeled from situational information like clinical case of the patient (affected vertebra, type of lesion, etc.). For each problem (p), some controls from (Σ) are defined as potentially applicable in the simulation process regarding applied actions from (R). The applied controls are represented in a given representation register from (L). These parameters are manageable by didactic experts of the domain with the implemented authoring tool. This tool has been integrated in respect to the openness of TELEOS’ architecture: it can be maintained, modified or extended without any incidence on the other modules of the learning system.

Fig. 4. Screenshots showing a list of created operators (left) and an operator modification form with its associated controls (right)

4.2 Producing Epistemic Diagnosis The formalized knowledge elements are represented in a Bayesian network to foster analysis of involved uncertainty [4]. In the first version of the diagnosis module (extensively presented in [13]) perceptual-gestural knowledge traces are not treated in the process in spite of their formalization in the model [13] which is one of the limits of the current system.

Problems

Operators

Controls

Situation variables

Registers

Fig. 5. The structure of the Bayesian network

The diagnosis process run in two phases: in the first step, the knowledge’s states of the learner called “situation variables” are computed. The “situation variables” are produced from computing functions regarding related controls. As an example, let’s consider the problem of treating the 12th thoracic vertebra. One of the actions that can be applied during the simulated surgical intervention is resumed as followed: P: Operation of a thoracic vertebra R: Take a front radio L: Radio (2D) Σ : Pedicles must be centered on the radio The user operates the fluoroscope to position it on the right area of the patient’s body. He takes a front x-ray for visual guidance. The action “Take a front x-ray” is then considered as engaged. The radio is displayed on the 2D screen of the interface. The associated control “Pedicles must be centered on the radio” is then activated. The action will be marked as valid only if it is considered in adequacy with the control. This adequacy is tested by a function computing the symmetry of the pedicles regarding the spinous given predefined significant points positions on the 2D display. The result of this test is the situation variable “RF_centeringVertebra_1”.

Controls are stated probabilistically as “Used_Valid (MJV)”, “Used_Invalid (MJI)”, or “Not_Used (NMJ)”. In fact, at the second phase of the diagnosis process, measures are associated to these states with a certain level of uncertainty regarding the current problem (“Operation of a thoracic vertebra” in our example), applied actions (“Take a front x-ray”), and the situations variables computed at the first phase (“RF_centrageVertebre_1”). The diagnosis is produced at this phase as probabilistic evaluation of the control states: MJV= 0.8, MJI = 0.2, PMJ = 0. This action can be stated as correct considering the level of adequacy with the control. The Bayesian network is refreshed at each applied action during exercises taking into account modifications made progressively to achieve solution. In fact, the modifications at time Tn-1 in a good or a wrong way at time T n are considered and independently from the correctness or incorrectness of the same action applied at time T 0. These temporal modifications can then increase inferred measures of the controls states “Used_Valid” or “Used_Invalid”. This temporal dimension impacts associated situation variables and consequently the didactic feedback that will be produced afterward.

4.3 Producing Didactic Feedback The didactic feedback module of the system is based on the decision theory of Skinner [16]. Its construction includes three phases: at the first one, the pedagogical target is identified on the base of provided diagnosed knowledge; at the second phase the feedback format is decided. For this purpose, a node of utility and a node of decision have been added to the Bayesian network in order to generate an influence diagram-like graph. At the third phase, the content of the feedback is decided given the identified pedagogical target and format. This is to select the most pertinent pedagogical elements to return. This decision is computed from the following apprenticeship utility function [14]: Uapp(ei, E) = α.Ustate(ei, E) + β.Utype(ei) + σ.Uorder(ei) + δ.Unature(ei)

(1)

The apprenticeship utility of a given knowledge depends on characteristics like its state, type, order and nature. The variables α, β, σ and δ are “priority variables” that can be modified on didactic hypotheses basis to set a priority order between the characteristics of available pedagogical content to consider when deciding didactic feedback. The returned results are suggestions to review a clinical case, a theoretical course or to realize another simulation exercise. All feedback results are delayed to the end of simulation exercises. One reason for this is the complexity of their computation that would significantly slow down exercises execution if they were to be computed throughout the simulation sessions. We state the hypothesis that perceptual-gestural diagnosis could alleviate the whole process by splitting the computation process into smaller and targeted spot of computation. That would also allow production of live didactic feedbacks closer to the real world learning environment.

5 Tracing Perceptual-Gestural Knowledge Traces To cover perceptual-gestural knowledge involved in the surgical activity we completed the system with an haptic arm and an eye tracker. The haptic arm is configured to render the body and bones resistance at the different levels of insertion as to yield the manipulation realism of the trocar. It traces the gestural handling of the surgical tool, that is, position, force and speed of the gesture. As described in the introduction of this paper, the visual guidance is crucial in percutaneous surgery. The eye-tracker gives feedback on information being taken in account for each applied actions, considering areas and points of interest that should be fixed relatively to these actions. This perceptual information is to give more insights on the level of mastery of learners’ knowledge underlying their actions. To improve the system knowledge diagnosis, it’s important that the distinction between behavioral and epistemic diagnosis [21] is considered. Epistemic models are derived from behavioral models which are constructed on observable events and interactions between learners and learning environment interfaces. Behavioral diagnosis concerns inferences on modeled behaviors in learning situations while epistemic diagnosis reasons on knowledge states related to these behaviors [1]. Thus, accuracy and reliability of epistemic diagnosis strongly depend on the quality of behavioral diagnosis.

5.1 Taking into Account Visual Data in the Diagnosis We aim at modeling and embedding the aforementioned behavioral events in the knowledge analysis process. For visual traces we had to address first the dynamism and unpredictability of areas and points of interest to be considered in a simulation exercise. In fact, these points appear, move or disappear dynamically with the simulation progression and depending on engaged actions. As an example, the coordinates of important parts of a vertebra on the 2D display change following the position angle of the fluoroscope and the types of taken x-rays (face, profile, inlet, outlet). Existing eye-tracking tools can address this dynamic aspect only when evolution in an environment is predefined or predictable [6]. A novel and generic approach has been proposed in [6] to overcome this issue. A tool has been implemented to produce real time visual feedbacks for simulation-based environments using eye-trackers. We actually use this latter for monitoring visual perceptions during simulation sessions in TELEOS. The next step is to treat traced visual perceptions along with other applied actions as to refine the inference of situation variables.

Fig. 7. TELEOS’ real-time visual tracks monitor. Gray circles show current fixation points. Blue lines show groups of near points that have been fixed in a small gap of time. Gray and blue lines show eyes trajectory

6 Conclusion and Perspectives We presented in this paper the methodology for modeling and implementing a simulationbased Intelligent Tutoring System dedicated to an ill-defined domain, percutaneous orthopedic surgery, and our approach to cover involved perceptual-gestural knowledge in the domain by completing the learning software with an haptic arm and an eye-tracker. We detailed the learning environment and its existing features as well as ongoing works to refine knowledge diagnosis inferences with visual data. We stated the hypothesis that perception information will bring more insights on the execution of the surgical activity to be assessed as to determine for example if applied actions have been succeeded randomly or from mastered decision. Visual tracks foster this assessment as they point out if the trainee has followed adequately visual guidance by fixing the right area and points of interest during operation simulation. We also intend to explore in the future real-time visual feedback like hints and warnings throughout simulation exercises which are more in adequacy with real working environment.

Another envisaged evolution concerns knowledge’s model on which the diagnosis inferences are based. The set of formalized elements of knowledge can only be modified or extended by hand. This is a limit to be addressed as, considering the openness of the domain; each executed simulation exercise, either failed or succeeded, can bring potential elements of knowledge that would allow reinforcing existing controls or creating new controls. We state the hypothesis that introducing automatic acquisition of knowledge domain from activity traces would enrich the model and foster more fine-grained diagnosis of learners’ knowledge.

References 1. Balacheff, N. : Contribution de la didactique et de l'épistémologie aux recherches en EIAO. In : Actes des XIIIe Journées francophones sur l'informatique, Grenoble, 9–38 (1991) 2. Balacheff, N. : Conception, propriété du système sujet/milieu. In : Actes de la VIIe école d’été de didactique des mathématiques, 215–229 (1995) 3. Brousseau, G.: Theory of Didactical Situations. Dordrecht: Kluwer Academic Publishers edition (1997) 4. Conati, C., Gertner, A.S., and VanLehn, K.: Using Bayesian Networks to Manage Uncertainty in Student Modeling. In: User Modeling and User-Adapted Interaction 12, 371–417 (2002) 5. Fournier-Viger, P., Nkambou, R., Mephu Nguifo, E.: Building Intelligent Tutoring Systems for Ill-Defined Domains. In: Nkambou, R., Mizoguchi, R., Bourdeau, J. (Eds.). Advances in Intelligent Tutoring Systems, Springer, Heidelberg. 81–101 (2010) 6. Fournier-Viger, P., Nkambou, R., Mayers, A., Mephu Nguifo, E., Faghihi, U. : An Hybrid Expertise Model to Support Tutoring Services in Robotic Arm Manipulations. In: Proc. 10th Mexican International Conference on Artificial Intelligence (MICAI 2011), LNAI 7094, Springer, 478–489 (2011) 7. Jambon, F., Luengo, V. : Analyse oculométrique « on-line » avec zones d’intérêt dynamiques : application aux environnements d’apprentissage sur simulateur. In : Ergo'IHM (2012) 8. Luengo, V., Mufti-Alchawafa, D., Vadcard, L.: The knowledge like the object of interaction in an orthopedic surgery-learning environment, ITS 2004 7th International Conference on Intelligent Tutoring Systems, Maceio, Brésil (2004) 9. Luengo V., Vadcard L., Mufti-Alchawafa, D.: «De l'analyse didactique à la modélisation informatique pour la conception d'un EIAH en chirurgie orthopédique ». In : Extraction et Gestion de ma Connaissance (EGC 2006), Lille, France, (2006) 10. Luengo, V., Vadcard, L., Dubois, M., Mufti-Alchawafa, D. : TELEOS : de l'analyse de l'activité professionnelle à la formalisation des connaissances pour un environnement d'apprentissage. 17e journées francophones d'Ingénierie des connaissances (Nantes), 2006, 101–110 (2006) 11. Luengo, V., Larcher, A., Tonetti, J.: Design and implementation of a visual and haptic simulator in a platform for a TEL system in percutaneous orthopedic surgery. Stud Health Technol Inform 163, 324–328 (2011) 12. Lynch, C., Ashley, K., Aleven, V., Pinkwart, N.: Defining Ill-Defined Domains; A literature survey. In: Proc. Intelligent Tutoring Systems for Ill-Defined Domains Workshop, ITS 2006, 1–10 (2006)

13. Minh Chieu, V., Luengo, V., Vadcard, L., Tonetti, J.: Student Modeling in Orthopedic Surgery Training: Exploiting Symbiosis between Temporal Bayesian Networks and Fine-grained Didactic Analysis. In: International Journal of Artificial Intelligence in Education (IJAIED) 20 (3), (2010) 14. Mufti-Alchawafa D., Luengo V., Vadcard L. Didactic Decision-Making Process In a Surgery Learning Environment (2006) 15. Remolina E., Ramachandran S., Fu D., Stottler R., Howse W. R.: Intelligent SimulationBased Tutor for Flight Training. In: Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) (2004) 16. Skinner D., Introduction to Decision Analysis, Gainesville, Probabilistic Publishing (1999) 17. Tonetti, J., Vadcard, L., Girard, P., Dubois, M., Merloz, P., Troccaz, J.: Assessment of a percutaneus iliosacral screw insertion simulator. J. Orthopaedics & Traumatology: Surgery & Research 95, 572–578 (2009) 18. Vadcard L., Luengo V.: Embedding knowledge in the design of an orthopedic surgery learning environment. In : International Conference on Computer Aided Learning in Engineering education (CALIE 2004), Grenoble (2004) 19. Vadcard L., Luengo V.: Réduire l'écart entre formations théorique et pratique en chirurgie : conception d'un EIAH (2005) 20. Vadcard L., Luengo V.: Interdisciplinary approach for the design of a learning environment. In: E-Learn 2005 - World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education. Vancouver, Canada (2005) 21. Wenger, E. Artificial Intelligence and Tutoring Systems. Morgan Kaufmann Publishers, Los Altos, USA (1987)