Enactive knowledge with gestural annotations Jérôme Olive* Indira Thouvenin *† Stéphane Aubry*† (*)Continental France SNC (*†)Laboratoire Heudiasyc, Université de Technologie Compiègne, France E-mail:
[email protected],
[email protected],
[email protected] Abstract The question of knowledge integration in virtual environments will be a challenge for the next few years for training, design, simulation and architecture… Many knowledge registrations such as symbolic representations (icons, cognitive maps, visualisation metaphors) are present in these spaces for abstract knowledge that we can capitalise. But the gesture is difficult to capitalise other than by its acquisition and complete description. We propose a model of gestural annotation, produced from a 3D annotation model. This model allows the user to explore a virtual environment and gives him the opportunity to constitute his own knowledge by reading and searching the annotations in a knowledge base. The process of searching annotations in a large number of annotations is an enactive process. In addition, the capitalisation of gestures concerned with the virtual mock-up gives a better comprehension of complex technical systems based on time-space navigation.
1. Introduction When navigating in a virtual environment (VE) some metaphors are necessary [8] to guide the user in navigation, manipulation, or exploration. In the case of a virtual mock up manipulation to access specialized knowledge, it is often difficult to understand the context, the gesture for technical applications or the sequence of gestures to perform a task. As in [11], we consider as gestures any motions that can be applied as the cause of a performance, whichever be the producing system. This is a major concern in the case of industrial applications [13] such as maintenance of complex systems. Some VE are linked with knowledge bases for design but there is a large gap between the 2D and the 3D environments which is partly solved in the MATRICS© application [4] MATRICS© proposes a 3D annotation model that is rooted in a virtual mock-up and a system of capitalisation of these annotations with the help of a knowledge base which allows the constitution of an experience by going to and from between the virtual mock-up and the knowledge base. It is an action in the sense of Varela [16] since the user perceives the annotations and must himself act to navigate in the knowledge base to find the annotations that are closely Proceedings of ENACTIVE/07 4th International Conference on Enactive Interfaces Grenoble, France, November 19th-22nd, 2007
related or those that are necessary to understanding it. This enactive process is facilitated by the perceptions of multimodal annotations and by actions that are possible within the VE. On the other hand, the knowledge of a gesture and it capitalisation, indispensable in training situations, is not possible by the model proposed by S. Aubry. That is why a new gestural annotation model allows the acquisition of knowledge of the movement for a precise task by giving rise to the possibility of writing and reading the gesture on a virtual mock-up. We present in this paper the MATRICS project, then the gestural annotation model and the first stage in the implementation of this model. These stages have to be integrated in a localised cognition context.
2. 3D Annotations and virtual environment During collaboration on a virtual object, users exchange informal knowledge about this object. It might be text, images, sounds or more complex information like shapes or gestures. There is a need to capitalise this knowledge in the virtual environment [2]. Our approach is based on collaborative product design situation [15], in which annotations are a central artefact of the collaboration [7]. Therefore, we propose to use 3D annotations, as a support of this knowledge in the virtual space [1].
2.1. Definition of a 3D annotation As the word “annotation” is part of everyday language, several definitions using different criteria can be found in literature [6], [5]. In this paper, we will consider an annotation as a mark or a document with the following properties: It refers to another document (the target) and support, or is the result of an activity about this document. It is not dissociable from its target. However, it is distinct from its target. A 3D annotation needs to have the properties described above, plus these two supplementary properties: The target is a 3D object. It is contextualized in the 3D space. Namely, it is associated to a representation, a location
and interaction methods in the virtual environment. In [3] we propose a theoretical model for 3D annotations; it is based on three components: annotation form, metadata and spatialisation. We will now focus more specifically on 3D annotation with gestures. Gestures are essentially different from 3D for two reasons: first, they include the notion of time, which is not present in 3D shapes. This notion is necessary to express movement, or an order of events.
During the annotation of the virtual mock-up, the user will take a path around it. These movements are decomposed into different sets of alternative movements and captured points of view. These different points of view captured by the user are as many in the context that allows us to understand why the user wanted to annotate the virtual mock-up. The figure 1 presents this process of annotation. The path of the user starts at an initial point of view (point of view of the previous annotation or point of view of entry in the system). This is followed by a series of intermediate movements and points of view which are as many of elementary parts of the cognitive path.
3. Gestural annotations ion tat n no An reatio c
e iat ed ew erm vi Int t of n i po nt me
ENACTIVE/07
ve
3.2. Annotation Process
Mo
Figure 1. 3D annotation process
eo
d Vi d
So
un
tur e Pi c
t
After having navigated (fig. 2) around the virtual mock-up, the user selects the objects to be annotated. Here two choices: either he manipulates the object in question or he annotates with classical media (text, image, sound, video…). The annotation process stops at this moment and can restart with a new annotation.
Te x
What we propose to do is to augment the 3D annotation model with a notion of time which will be as strong as the notion of spacialisation. This addition allows us to capitalize on the position of a 3D entity (object or user) in time. The first knowledge of this type that we want to capitalize is the technical gesture (in particular for industries in cases of maintenance). This capitalization will consist of a systematic saving (whether it be decided by the user or not) of the object manipulations and user movements. This data will be considered as separate annotations and will be linked to an ontology of gestures allowing a semantisation of the gesture. Here, it is not the gesture that we will annotate [9], but the gesture that will be contained within the annotation. In the same manner as with the previous model, we will be able to navigate in the concepts to facilitate the navigation between the annotations. For example, in the scope of maintenance, we will be able to sort all the annotations on the basis of “unscrew”. We would also like to capitalize this process of annotation of the virtual mock-up. Indeed, we think that the path taken by the user in a virtual environment before annotating an object contributes to the definition of the annotation. “What have I seen before annotating?”, “Which objects have I manipulated?” is the knowledge that we would like to capitalize within the annotation. We consider that each of these stages can be assimilated to an elementary annotation, each contributing differently to the global annotation. During the annotation we save each of these elementary annotations (point of view, movement, selection, manipulation).
l w tia Ini f vie o int
3.1. Towards a gestural annotation model
po
In the 3D annotation model previously cited, the only notion of time present is the date of creation or modification of an annotation. A problem arises then: How to capitalize the trace of activities around the virtual mock-up?
Manipulation Selection
Figure 2. 3D annotation creation Our system today allows the capitalization of annotations of movement and manipulation. We save the transformation matrices of all 3D objects (cameras, physical objects…).
3.3. Capitalization of the gestural annotation 3.3.1. Recognition of gestures Thanks to a system of recognition of gestures we can give a meaning to a gesture. Our system is based on a neural network which is capable of differentiating a saved gesture from a base of pre-existing gestures. The system returns the name of the gesture which is then referenced with a base of gestures. 3.3.2. Classification of gestures
The identified gesture is then associated with an annotation by the creation of a link between the gesture base and the annotation. This link allows us to extract a sense of the gestural annotation and to construct a representation of this gesture in the virtual environment.
system replays the manipulation made by expert and access to the annotation. We think that On the figure 3 the user needs to read annotations in red to extract screws before extract the blue piece (white annotation). When he select white annotation to extract the blue piece the annotation on screws become red to tell that they must be read.
3.3.3. Gestural annotation representation To represent annotation on the virtual mock-up, we need an anchor or an “icon”: a 3D object that spatializes the annotation. This icon must show the sense of the annotation. In case of gestural annotation we need to represent the movement. In this way we studied several kinds of movement representation and notation used in choreography. Laban [10] and Benesh [12] have created two notation systems based on a human body cut-out. These two systems describe the body parts positions and movement in space. These systems have a big constraint for our needs, they describe only body movement. Sutton [14] in her early works describes the same data in her Sutton Dance Writing. But she extends her model by introducing objects of the environment and she describes the body position compared to these objects. In her latest model she describes the sign language. In this system she describes hands and face and their relative position and contact. We want to use this latest model of Sign Writing Language to create our representation for interaction between objects and human. We propose to have a representation by annotation of the user’s movement and objects’ manipulation. This representation will use a system of 3D animated anchors. As in the latest Sutton system, our anchors represent interaction, action and gesture.
4. Case study
We will now study an industrial maintenance case of application. We initially consider the work of capitalization of the expert. The expert navigates around a virtual mock-up and fixes his point of view on a specific place. Then he handles an object to reach another one behind. He can now annotate the object or move it etc…. All movements are capitalized as gestural annotations. During the movement capitalization, all movements are analysed by a neural network. If a gesture is recognized only the meaning of the gesture is capitalized. On the contrary, if this one is not recognized then we preserve all the gesture data (data can be added to a gesture base with a posttreatment). In the case of a recognized gesture we adapt the shape of the annotation anchor with the meaning of the gesture. We will now study the case of annotation reading. A user want to access to the annotation created by expert in previous paragraph. He needs to access to the carrier object before. To do that, he can read a gesture annotation (first manipulation in the previous paragraph) on the object occulting. By this action, the
ENACTIVE/07
Figure 3. Read gestural annotations
5. Conclusion In this article, we describe a new method of knowledge integration in virtual environment by an enactive process. This knowledge is registered by means of a 3D annotation model on the virtual mockup; these annotations are capitalized by means of a knowledge base. This allows the user to construct step by step, his cognitive path. We are currently exploring a new annotation model: gestural annotation which brings localised knowledge. Our hypothesis is that this experiment when conducted in a virtual environment enriches the intellectual construction and the hard skills. An initial system of gestural annotation was developed to capitalise the manipulations and movements made. The system will be completed by a gesture interpretation module. In our future work, we will give autonomy to the annotations: - automatic animation of the anchors - presentation of the annotations’ content when needed - adaptative replay of manipulation and navigation - evocation of the closer annotations
Acknowledgment We want to thank Vikram Seshu for his precious and enthusiastic help.
References [1] Aubry, S., Lenne, D., Thouvenin, I., Guénand, A., "VR Annotations for Collaborative Design", in HCII2005, Las Vegas, 22 – 27 July 2005, CD-ROM Proceedings, ISBN 0-8058-5807-5. [2] Aubry, S., Thouvenin, I., Lenne, D., Okawa, S., "Knowledge Enhanced Annotations in Virtual reality Environments", in Virtual Concept 2005, Biarritz, 9 – 10 November 2005. [3] Aubry, S., Thouvenin, I., Lenne, D., Okawa, S., "Knowledge integration for annotating in virtual environments", International Journal of Product Development, Vol. 5, N°6 (2007), pp. 533 - 546. [4] Aubry, S., "Annotations et gestion des connaissances en environnement virtuel collaboratif", PhD thesis, Université de Technologie de Compiègne, 2007. [5] Azouaou, F., Desmoulins, C., Mille, D., "Formalismes pour une mémoire de formation à base d'annotations : articuler sémantique implicite et explicite", EIAH 2003, Strasbourg, France, pp. 43 – 54. [6] Baldonado, M., Cousins, S., Gwizdka, J., Paepcke, A., "Notable: At The Intersection of Annotations and Handheld Technologies", Proceedings of HUC Conference, Bristol, LNCS 1927, Springer Verlag, Berlin, 2000, pp. 100-113. [7] Boujut, J-F., "User-defined annotations: artefacts for coordination and shared understanding in design teams", Journal of Engineering Design, vol 14, n°4, December 2003, pp. 409-419. [8] Bowman D. A., “Interaction techniques for common tasks in immersive virtual environments”, Georgia Institute of Technology, June 1999. [9] Héloir, A., Gibet, S., Courty, N., Raynaud, M., “Système d'annotation et de segmentation de gestes de communication capturés”, TALN 2005, Dourdan (France), 6-10 June 2005. [10] Laban, R., Ullmann, L., “The Mastery of Movement”, Plays, Inc. 1971. [11] Luciani, A., Evrard, M., Couroussé, D., Castagné, N., Cadoz, C., Florens, J-L., “A basic gesture and motion format for virtual reality multisensory”, in the Proceedings of the 1st international GRAPP Conference on Computer Graphics Theoryand Applications, ISBN: 972-8865-39-2, Setubal (Portugal), March 2006. [12] McGuinness-Scott, J., “Movement Study and Benesh Movement Notation: An Introduction to Applications in Dance, Medicine, Anthropology, and Other Studies”, Oxford Univ Pr (November 1983). [13] Olive, J., Thouvenin, I., Lemasson, G., Sbaouni, M., “Tire Manufacturing supported by virtual environment”, Proceedings of Laval Virtual 2006, april 2006, LavalFrance. avril 2006, pp.159-163. [14] Sutton, V., “Sutton Movement Shorthand; Writing Tool For Research”, National Symposium on Sign Language Research & Teaching, Chicago, Illinois, 1977, USA
ENACTIVE/07
[15] Thouvenin, I., Abel, M-H., Ramond, B., Qamhiyah, A., "Environment Improvements for a Better Cooperation in Multi-Culture Collaborative Mechanical Design", In CSCWD 2002, Rio de Janeiro – Brazil, 25 – 27 September 2002. [16] Varela, F., Thomson, E., Rosch, E., « L’inscription corporelle de l’esprit, La couleur des idées », Seuil, 1993, ISBN- 2-02-013492-
