Multimodal Annotation to Support Web Learning ...

19th International Conference on Database and Expert Systems Application

Multimodal annotation to support Web learning activities L. Seta, G. Chiazzese, G. Merlo, S. Ottaviano, G. Ciulla, M. Allegra, V. Samperi, G. Todaro Institute for Educational Technologies, Italian National Research Council Via Ugo La Malfa, 153, Palermo - Italy {luciano.seta; giuseppe.chiazzese; gianluca.merlo; simona.ottaviano; giuseppe.ciulla; mario.allegra; valentina.samperi; giovanni.todaro}@itd.cnr.it by adding new modalities to process educational materials rather than the traditional verbal modality [6], [7]. Within MMHCI approach, Moreno and Mayer [8] proposed the Cognitive-Affective Theory of Learning with Media (CATLM). The most relevant aspect of this model is the activity of processing and reorganizing multimodal representations into a coherent and integrated structure. In this context it is important to use many modalities to process the same information, defined as redundancy, and the synergy described as the merging of different channels in a unique message [9]. The guidelines outlined from the cited frameworks have been concretely applied in many technological environments. These systems offer tools that use many communication channels and support differences in users’ abilities, needs and preferences [10]. Starting from an analysis of these approaches to the use of metacognition in learning activities based on ICT, the Institute for Educational Technologies has defined and developed a new environment called Gym2Learn. Following this vision the environment was designed to permit students to use different modalities to apply various text comprehension strategies during web based activities. This paper describes principally the module of this environment in which the multimodal approach played a very important role: the metacognitive web annotation tool. This tool has some similarity, from the technological and methodological point of view, with other software designed for web annotation, such as Marginalia [11], Gibeo [12], VPEN [13]. But, unlike these, the Gym2Learn tool make self-regulating processes explicit and guides the student in mastering all the steps of executive control of some important comprehension strategies for understanding online resources. Besides, in Gym2Learn the annotation process is based on a multimodal annotation procedure

Abstract This paper describes a multimodal annotation system called Gym2Learn, and its experimentation with deaf students. The system was developed at our Institute in the framework of the project CORFAD “Centro di competenza per la promozione e il testing di metodologie e prototipi di ambienti per l’Orientamento e la Formazione A Distanza”, funded by the Industrial Department through the European Regional Operational Programme for Sicily.

1. Introduction In the last few years, teaching and studying via the Internet are routine and common practices, so much so that the scientific community has carried out much research so as to comprehend how the efficacy of webbased learning activities can be improved. The system described in this paper is part of a research program aimed to support students during web surfing activities. The principle innovative aspect of this program is the design of a new techno-mediated environment in which two well-known research fields, metacognition and MultiModal Human-Computer Interaction (MMHCI), are combined to enhance the students’ learning experience. Metacognitive theories [1], [2], state that the awareness of cognitive processes and the ability to control and monitor them supports students’ performance. These principles have been successfully applied to online learning contexts: if a user selfregulates his cognitive processes, he manages the content that he is studying efficiently and, as a result, the risk of disorientation [3] and cognitive overload decreases [4], [5]. The second field of research, concerning MMHCI, has focused its studies on the student’s cognitive and motivational processes. In particular, according to MMHCI, the comprehension process can be improved

1529-4188/08 $25.00 © 2008 IEEE DOI 10.1109/DEXA.2008.68

166

contents into a predefined typology, Gym2Learn adopts an annotation procedure which allows the student to apply one of the following text comprehension strategies: • previous knowledge; • formulating hypotheses and verifying them during surfing; • asking oneself questions and answering them to verify comprehension; • identifying important parts of the text; Each strategy can be recalled by adding a text, vocal or video note to the web pages. In particular, Figure 1 shows the interface window for formulating a hypothesis. This interface is composed of three tabs: the first tab allows the student to write a textual message, the second to record a voice message and the third to record a video. The video note is an application that opens up access to web annotation systems to people with special needs such us deaf people who can use this type of note to express a concept in sign language. This application will be described in more detail below.

as a way of applying and controlling some text comprehension strategies during a web learning activity. This tool allows students to enrich the visited web pages with textual, audio and video notes. In this paper the Gym2Learn system will be briefly introduced, then the technological architecture for supporting the multimodal annotation will be described. A learning experience with a group of students will be briefly illustrated and in particular a test session with a deaf subject will be discussed to show the benefits available to people with physical disabilities when they interact with multimodal interfaces and technologies.

2. Multimodal Annotation and Gym2Learn System 2.1. The multimodal annotation model used in Gym2Learn system

2.2. The Gym2Learn System: interface and architecture Gym2Learn has been developed as a Firefox addon. The user interface is composed of a sidebar for note visualization and a dropdown menu with three items for accessing the following areas: 1. a self evaluation area consisting of an online questionnaire to evaluate students’ learning strategies, text comprehension strategies and computer skills; 2. a training area in text comprehension strategies; 3. a web learning activity area where a teacher can suggest tasks, procedures and site references. In the first area the student completes a questionnaire and receives a profile describing his way of using the computer, how long he spends studying with the computer, his level of knowledge of text comprehension strategies.

Fig. 1. The windows interface to create a textual, voice and video note Gym2Learn supports students in applying online text comprehension strategies through a multimodal web annotation system. The system has been created to support new learning activities based on the reading and comprehension of web pages. In particular, the system provides features for creating, managing and reviewing notes during surfing. A different approach has been used to annotate the information. While many systems simulate the use of the traditional behaviour of collecting and classifying

167

a pop-up menu which appears by right clicking on the mouse. On the right hand side of the figure an icon indicates the posting of a note, thus enabling a student to return to it later and/or modify it. Figure 4 shows the architecture of Gym2Learn which consists of a Firefox add-on, the Annotea annotation server and Apache FOP, and a streaming server to record and supply the audio/video notes.

Fig. 2. A browser with the sidebar of Gym2Learn. The training area allows the student to practise the strategies previously listed in order to increase his awareness of how to apply them correctly and facilitate his text comprehension during learning activities. In the third area the student starts a new learning activity on the Net and applies the practised strategies through a multimodal annotation method. Figure 2 shows a screenshot of the Gym2learn system with the sidebar on the left for monitoring the student’s annotation activity. Fig. 4. The architecture of the Gym2Learn system. The notes are stored in the Annotea server as a RDF file. Notes in textual form are written in this file, while audio and video notes are stored in the streaming server with only a reference recorded in the RDF file.

Fig. 3. The visualization of a note in the sidebar and the correspondent icon on the web page. Fig. 5. The creation of the content of the sidebar.

The student is able to view the annotations organized by typology using a toolbar located on the top left hand side of the sidebar. Moreover, two buttons are available for deleting the notes and for creating an RTF document containing the annotations selected by the student as being important for the learning objectives. The presence of any audio or video note is indicated by two icons. Figure 3 describes a hypothesis created by the student who can create a web note by selecting a portion of a text or image and clicking on an item from

The Annotea server responds to the client requests by sending the RDF file containing the notes. The RDF file will be visualized following the format described in an XSLT file for generating the HTML file in the sidebar. Using the Apache FOP the student can then obtain a RTF file containing the notes he selected previously.

168

t-value = 2.620, p-value = 0.012) when comparing G4 and G1 pre and post scores. Similar results were obtained also for groups G2 and G3. The next experimentation cycle will aim to study the impact on metacognition of a multimodal web annotation tool. In particular, attention will be paid to the relationship between a specific cognitive strategy used by the students the type of channel activated to annotate the web. The research should show whether some modalities are more suitable than others for supporting specific text comprehension strategies so that the design of interfaces can be adapted to specific metacognitive programs. The rest of this section will be devoted to illustrating a specific use of this annotation tool by a deaf student. Since this experience involved only one individual numerical data will not present. However this single case can offer many interesting clues to the impact that MMHCI and metacognition, used together, can have on the use of web resources by students with special needs.

3. Some experiences using gym2learn the first experimentation cycle the Gym2Learn environment was tested with 69 first year students from four different high schools. The principle aim of this test was to verify the technological efficiency of the environment and the methodological efficacy of the procedure set up. During this trial only text annotations were enabled, and the observation of the use of audio and video annotations was only sporadic. In this way, by limiting the number of factors involved, it was possible to observe the impact of the use of the system on the metacognitive level of the students, as measured by specific metacognitive assessment tests. To study this effect a two-factor quasi-experimental design was set up (Table I), and the students were divided into four groups. One group used the system only during the training phase while another group used it during a real learning activity on the Web (executive phase). A third group used the system in both phases while a control group did all activities without the support of the system. The results were compared to evaluate the effects of using Gym2Learn

3.1. Using Gym2Learn with a deaf subject There is an ever growing awareness of the educational importance of technological tools for students with special needs. The use of new technologies gives all students the opportunity to explore and exploit their abilities regardless of any impairment they may have. These considerations have led to an extension of the Gym2Learn application to contexts where students with special needs may be excluded. The decision to focus on the needs of the deaf originates from the idea that Gym2Learn can make it easier to acquire both knowledge and learning strategies, using video, images, and especially the skill to annotate the text in a visual way. To extend the use of the system to deaf people, some parts of the training phase were changed; in particular, videos were included in the exercises, translating audio and video information into the Italian Sign Language (LIS). Besides, the video annotation mechanism can be used by means of a webcam with which the student can film himself to take note of the most important information. In this way, users of the LIS can annotate by means of their favourite linguistic channel. So it is possible to compare the action of filming themselves while they are using LIS and reviewing the video to a sub-vocalic repetition, that is a silent repetition of words that we use for mnemonic and comprehension purposes. The system functionalities were tested with deaf students. One of these was a 17 year old girl called M.,

TABLE I. Table representing the different levels of the independent variable Use of system during the training phase

Use of the system during the executive phase

Group (or Level)

Number of students

no yes no yes

no no yes yes

G1 G2 G3 G4

12 15 18 24

The experimentation was organized in the following phases: • Phase 1. Metacognitive pre-assessment for all the groups; • Phase 2. Metacognitive training, for groups G2 and G4; • Phase 3. Didactic activity, without Gym2Learn for groups G1 and G2, with Gym2Learn for groups G3 and G4; • Phase 4. Metacognitive post-assessment for all the groups. The analysis of test data highlighted a significant relationship between the use of Gym2learn and an increase in metacognitive scores between pre and post assessments. In particular, the results of the Analysis of Covariance test (ANCOVA) show increments for all experimental groups that used the system with respect to the control group G1, with, for example, a regression coefficient about double (value estimate 2.3,

169

The results of her tasks and of the post-tests, revealed that her awareness level about the strategies she used had improved; similarly, her autonomy in facing difficult texts and tasks had increased. As she stated in the final discussion, the system enabled her to acquire the necessary skills for managing information included in a text using annotation functionalities. In this way she had the opportunity to select only the most interesting information, without getting lost in the information space. M. found the training phase of the system to be very useful, because it allowed her to think about the relevance of the expected comprehension of the content, referring in particular to the titles of the texts and the highlighted parts. In conclusion, we can state that the use of Gym2Learn with deaf students facilitates the development of their awareness and motivational control, stimulating the monitoring of their competences, of the strategies to be applied in the various tasks and in the evaluation of the results. The educational needs for deaf students cannot be separated from the necessity of improving their skills so they have an active role in their own learning processes and in content elaboration; in fact, the need is not only to acquire knowledge but also to strengthen self-esteem and to have a positive perception of their limits.

who uses LIS as her mother tongue, the daughter of deaf parents. M. attends high school supported by a scholastic interpreter, and her school performance has always been satisfactory despite morph-syntactic errors and difficulties with written texts, that are characteristic of deaf pupils [14]. Two tools were used for the metacognitive assessment: the Jr. MAI test [15], and a tool extracted from the Metacomprehension Strategy Index [16] for assessing the level of metacognitive awareness according to individual cognitive strategies stimulated by the system. The results of the tests revealed that this student uses the mechanisms for controlling and verifying the assigned task without awareness, although she is able to assess her work both positively and negatively. Besides, she possesses a set of strategies for the comprehension of the text, but it is difficult for her to apply them because of the intrinsic limits of her deafness. During the seven experimental sessions, when an LIS interpreter was always present, M. showed great interest in the system and acquired awareness of strategies she had already used automatically in her studies. In some of the training phases, M. clearly showed difficulties in comprehending the texts of the assigned tasks and of the assessments, due to the limited vocabulary of the deaf [17]. During the experimentation, we had to distinguish between the evaluation of the task carried out and the quality of the content. In fact, the subject’s writing difficulties resulted in a poor performance from the point of view of the contents of the added notes, but the strategies and the annotations proposed by the system were used in a correct and natural manner. The system was therefore used correctly by the subject who did not require any extra help, apart from some lexical suggestions. To overcome this problem and therefore reduce the use of strategies requiring the interpreter support M. used the Italian dictionary integrated into the system. The use of the video notes is intended to enable students to deal with educational material by means of their habitual learning tools, to manage the contents independently, using only the subject’s resources. At the beginning, M. used the video annotation only a few times, because she was embarrassed about filming herself and using the LIS while being observed. Instead, she used this function better when she was at home, believing it to be effective for comprehending the most difficult parts of the text. During the final discussion, she reported that the explanation in the LIS was more immediate, direct and clear, because she could “see” the words. For these reasons, M. preferred the presence of an LIS interpreter for the most difficult tasks, so using dependent strategies.

4. Conclusion This paper illustrates a multimodal system allowing students engaged in web based didactic activities to enrich the web pages with annotations in different formats; textual, audio and video. The system was designed taking into consideration research results from various disciplines, including cognitive learning theories and particularly the application of metacognition for text comprehension, studies on Human Computer Interaction (HCI) and the use of multi-channel interactions. As a result, a technological architecture was designed and the Gym2Learn system was developed and then tested in various educational settings. Gym2Learn was able to support students during their learning activities and showed considerable flexibility and efficacy. The possibility of choosing between different formats to annotate the web pages proved to be particularly important in supporting the learning of a deaf student; she felt very comfortable using the video mode with sign language to annotate the web, and this enhanced her motivation. Gym2Learn is now in a testing phase and new developments are planned to enable other students with

170

special needs to use the system, thanks to the application of new multimodal features.

[8] R. Moreno, and R. E. Mayer, “Interactive learning environment”, Educational Psychology Review, 19, 2007, pp. 309-326. [9] N. B. Sarter, “Multimodal information presentation: design guidance and research challenges”, International Journal of Industrial Ergonomics, 36 (5), 2006, pp. 439445. [10] A. Jaimes, and N. Sebe, N., “Multimodal humancomputer interaction: a survey”, Computer Vision and Image Understanding, 108 (1-2), 2006, pp. 116-134. [11] C. Xin, and G. Glass, “Enhancing online discussion through web annotation”, in Proceedings of E-Learn 2005, Richards, G. (ed.), Vancouver, Canada, 2005, pp. 3212-3217. [12] W. Hwang, C. Wang and M. Sharples, “A Study of Multimedia Annotation of Web-Based Materials”, in Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications 200,. Chesapeake, VA: AACE. Montreal, Canada, 2005, pp. 4272-4280. [13] J. Miller, “Shared Web Annotation & Content Aware Services”, [Online], available: http://gibeo.net/ [03 Aug 2007], 2004. [14] Fabretti, D. and E. Tomasuolo, Scrittura e sordità, Roma: Carocci, 2006. [15] R. A. Spearling, B. C. Howard, L. A. Miller and C. Murphy, C., “Measures of Children’s Knowledge and Regulation of Cognition”, Contemporary Educational Psychology, 27, 2002, pp. 51-79. [16] M. C. Schmitt, “A questionnaire to measure children’s awareness of strategic reading processes”, The Reading Teacher, 43, 1990, pp. 454-461. [17] Maragna, S., La sordità. Educazione, scuola, lavoro e integrazione sociale, Milano: Hoepli, 2000.

5. References [1] D. Dunning, K. Johnson, J. Ehrlinger, and J. Kruger, “Why people fail to recognize thier own incompetence”, Current Directions in Psychological Science, 12(3), 2003, pp. 83-87. [2] S. Coutinho, K. Wiemer-Hastings, J.J. Skowronski, and M.A. Britt, “Metacognition, need for cognition and use of explanations during ongoing learning and problem solving”, Learning and Individual Differences, 15, 2005, pp. 321-337. [3] S. Puntambekar, and A. Stylianou, “Designing metacognitive support for learning from hypertext: What factors come into play?”, in Hoppe U., F. Verdejo and J. Kay (eds.), Artificial Intelligence in Education: Shaping the future of learning through intelligent technologies. Amsterdam: IOS Press, 2003. [4] K. Clibbon, K., “Conceptually adapted hypertext”, Learning Conference on Human Factors in Computing Systems Denver, New York, United States: ACM, 1995. [5] M. J. Jacobson, C. Maouri, P. Mishra, and C. Kolar, “Learning with hypertext learning environments: Theory, design, and research”, Journal of Educational Multmedia and Hypermedia, 5, 1996, pp. 239–281. [6] Mayer, R. E., Multimedia learning, New York: Cambridge University Press, 2001. [7] J. D. Fletcher, and S. Tobias, “The multimedia principle”, in Mayer, R. E. (ed.), Cambridge handbook of multimedia learning, New York: Cambridge University Press. 2005, pp. 117-133.

171