Digital Storytelling Design with Augmented Reality Technology for Remedial Students in Learning Bahasa Melayu Hafiza Abas Universiti Teknologi Malaysia Malaysia
[email protected] Halimah Badioze Zaman Universiti Kebangsaan Malaysia Malaysia
[email protected]
Abstract: Storytelling is one of the interesting elements that should be append in education process in a formal and structured way. From the observations and interviews made during January until June 2009 in one of the government schools, we found that remedial students have problems in engagement and focus during learning sessions. The main purpose of this study is to design a framework for digital storytelling (DS) as a guideline to follow and adapt during the development process. The first part of this paper is on the research background which gives an overview on research objectives, users, Bahasa Melayu, DS and augmented reality (AR) technology It is also shows the data of difficulty in learning during a reading process. The next section is on the DS framework. DS and AR technology elements are expected to give engagement, focus and stronger user experience in improving listening skills, enriching the vocabulary and ability to narrate. In addition, moral and emotional values and learning theories are also adapted. Keywords: Digital Storytelling, Augmented Reality, Framework, Remedial Students
Introduction Stories are a part of human culture to pass histories, values and entertainment. It consists of background, plots and characters. Stories explore abstract ideas, emotion, life situations, people perspectives (Weinberger, 1996) and enhance literacy development (Wells, 1987).Through storytelling, children can acquire new vocabulary, moral values, learn general knowledge, solve human problems and realize the beauty of the language from the tale (Norhayati et al, 2000). Leslie Rule from Center for Digital Storytelling defines that digital storytelling is the modern expression of the ancient art of storytelling. Its derive their power by weaving images, music, narrative and voice together, thereby giving deep dimension and vivid color to characters, situations, experiences, and insights. Augmented reality (AR) is one of the technologies can be added in digital storytelling. AR shows both real and virtual environments simultaneously (Hafiza & Halimah, 2009). It reduces user’s cognitive load by extending human system and information processing (Neumann & Majoros, 1998). Azuma et al. (2001) defines that AR systems combines real and virtual objects in real environments, runs interactively in real time and register both objects simultaneously. By using AR technology it is hoped that remedial students are more engaged and focused during learning sessions. Remedial students are students with learning difficulties in reading, writing and calculating. They have limited cognitive abilities such as hard to recognize letters and words as well as have problems in decoding and processing information or command from others. In Malaysia, remedial students are addressed by teachers using diagnostic tests. This paper proposes a design framework for digital storytelling with augmented reality technology. Digital storytelling is one of the modules in a prototype for remedial students. It is hoped that it is capable of assisting and bringing a new dimension of learning to remedial students in learning to read Bahasa Melayu (Malay language).
- 3558 -
Research Background Objectives of Research The main objective of developing digital storytelling with AR technology is to motivate remedial students, provide them with an enjoyable experience and keep them constantly engaged in the learning session. The findings can be useful to curriculum developers for teaching them to read in the Bahasa Melayu subject, especially for remedial students. Specifically, the objectives are to: • Develop a digital storytelling with augmented reality technology for teaching and learning how to read Bahasa Melayu. • Design a framework of digital storytelling based on cognitivism and behaviourism approach in teaching and learning how to read Bahasa Melayu. • Develop a digital storytelling module which consists of 2D and 3D animation and augmented reality interaction. • Design and develop a software evaluation checklist for 2D and 3D animation and augmented reality interaction. • Design and develop a questionnaire for remedial students to check their comprehension and ability to tell the tale back. User Remedial students are chosen as target users due to the fact that currently, in Malaysia, there is no courseware in learning how to read Bahasa Melayu for remedial students. Moreover, augmented reality technology is still new in Malaysia and has not yet been implemented in teaching and learning in either government or private schools. From the observations and interviews conducted on January 2009 to June 2009, the findings on the remedial students’ level of reading in the Bahasa Melayu subject is summarized in table 1. Characteristics
Manners during teaching and learning session
Level of reading
• • • • • • • • • • • • • •
Mischievous. Sensitive and emotional. Inactive. Absent-mindedness. Day dreaming. Can focus between 1 to 3 minutes. Easily get bored. Show no interest in reading. Reading is a stress session. Like to chat and gossip. Considered very low. Can read simple sight word such as “buku”(book) and “bola” (ball) . 90% are still in letter naming phase and CV + CV level. 10% can read CV+CVC, CVC+CV.
Speaking
80% has no problems in speaking, 10% problems in switching T to K and K to T. 10% switch G to D. Writing Highest error rate. Switch between words. Hold pencils wrongly, spatial problems between words. Spelling Highest error rate. Wild guessing and try to remember the word’s graphic patterns. Listening Can focus only at one simple instruction at a time. *C= Consonant, V = Vowel Table 1. Characteristics, manners and language components of remedial students during teaching and learning Bahasa Melayu session. From table 1, these students are identified to have learning disabilities have problems not only in reading but also related to other language components such as listening, spelling, writing and speaking. Previous research
- 3559 -
stated that at least 80% of student with learning disabilities have problems in reading (Lyon & Moat, 1997) They have problems with brain ability such as to receive and process information, understand, analyze or even store in short term and long term memory. In terms of behavioural factors, it involves with patterns of errors in reading, seek for extra time to finish work, copying exercises, avoidance of writing and easily getting tired. For this reason, the learning environment to read Bahasa Melayu for remedial students should be motivating and fun for them. This learning environment can be provided by developing a research prototype by implementing AR digital storytelling. Bahasa Melayu as a Teaching and Learning Language to Read Malaysia’s national language is Bahasa Melayu. It is widely used in teaching and learning schools in Malaysia. Students are obliged to pass the Bahasa Melayu subject in order to further studies at university level or pursue a career with the government. Bahasa Melayu is a What You Spells Is What You Pronounce (WYSIWYP) language. It has a seven single word with one syllable pattern, eleven single word with two syllable pattern, nineteen single word with three syllable pattern and twelve single word with four and more syllable pattern (Hafiza & Halimah, 2009). In this storytelling module, remedial students have to focus on understanding the story and not the words. We use sight words with a simple and short stories. No words are displayed on the computer screen. Digital Storytelling Stories are central elements in human thinking and communication (Schank, 1995). Storytelling is an original way of teaching and the oldest form of literature (Pedersen, 1995). It is also a natural way for students to build literacy skills (Kozlvich, 2002) by expanding their vocabulary and understanding grammar. Lynch and Fleming (2007) indicate that: The adaptable and dynamic nature of digital storytelling, which encapsulates aural, visual and sensory elements. This utilises the multitude of cognitive processes that underpin learning-from verbal linguistic to spatial, musical, interpersonal, intrapersonal, naturalist and bodily-kinaesthetic. Digital storytelling enhances the teaching and learning process by using technologies, interaction between users and designers, users and technology tools and also gives more power in terms of audio and visual impact. Thus, it diversifies up the imagination of children as well provide a fun and engaging experience in learning. Integrating digital storytelling in the language curriculum is a creative learning technique that can improve student’s level of reading, writing, speaking and listening (Tsou et al, 2006). However, there are no previous studies that have analyzed the impact of digital storytelling with AR for the remedial student in learning Bahasa Melayu. Augmented Reality AR is not a standalone application. It uses computer, webcam, or head-mounted display or other tools to support itsfunction.AR is a situation where 3D virtual objects complement the real objects or environments. The involvement of physical experience, virtual content, real environment, digital storytelling and imagination gives an ideal learning experience to remedial students (Hafiza & Halimah, 2009). Previous researches showed the benefits of using AR technology in education. The summary of previous researches related to teaching and learning in AR environments is summarized in table 2. Research BlackMagicBook Billinghurst et al. (2001a). eyeMagicBook. Science and Augmented Reality (Kerawalla et al, 2006). Learning Using Augmented Reality Technology: Multiple Means of Interaction for Teaching Children the Theory of Colours (Ucelli et al. 2005).
Description The MagicBook explores how interfaces can be developed that allow for seamless transition between Physical Reality, Augmented Reality (AR), and immersive Virtual Reality (VR) in a collaborative setting. New reading experience in a children picture book. AR for teaching subjects such as earth, sun and moon. Bring the subject to life. It is for primary school students. AR provides a tangible interface metaphor and physical objects are used to communicate meanings.
- 3560 -
Augmented Chemistry (Morten & Benedikt, 2002). Little red:storytelling in mixed reality (Saso et al, 2003). 3D AR Comic book and notes for children using mobile phones (Siddaharth Singh et.al, 2004). AR Volcano (Woods et al., 2004). The Haunted Book (Camille Scherrer et. al 2005) AR-Jam Early literacy (Dunser & Hornecker, 2007). An observational study of children interacting with an augmented story book (Andreas Dünser & Eva Hornecker, 2007). An interactive 3D exploration narrative interface for storytelling (Zhi Ying Zhou et.al, 2004) AR-Based Storybook (Dunser, 2008). Augmented Biochemistry (Medina et al., 2008). AR Book Cover and Bookmark (Inagawa & Fujinami, 2008). The Design of a Mixed-Reality Book: Is It Still a Real Book?(Rapha¨el Grasset et.al, 2008) Edutainment with a Mixed Reality Book: A visually augmented illustrative childrens’ book (Raphaël Grasset et. al, 2008) Vivid Encyclopaedia (Shibata et al, 2004)
Reports on some of the advantages of what tangible interaction can bring to chemistry education. Playground for a virtual storytelling. Enhanced reading experience by using handphone and AR Post-It system. User can interact with the virtual content of volcano. A combination of computer vision and augmented reality. Techniques to animate the illustrations of a poetry book. Use AR to create interactive stories suitable for children aged five to seven. The books pages are augmented using animated virtual 3D characters, sound, and interactive tasks. Children read and interact with AR storybook. Using 3D mixed media story cube as an interface for storytelling. To study how interactive AR affects low ability children. Examine how AR helps student in learning and understand how student learn in AR environments. A virtual illustration system with two types of add-on devices for the flipping detection: book covers and bookmarks. The study of semantics of a mixed reality book, design space and user experience with this interface. This research add the virtual visual and auditoryenhancements to an already published book. It reports on the design and development of a new type of ’mixed reality book’.
Mix reality pictorial book of insects. Vivid Encyclopaedia pages have to be consisted of black background (to detect the camera position) and picture of insects. Virtual Pop-Up Book Based on This book displays 3D virtual objects on thereal book based Augmented Reality (Nobuko et al, 2007). on pose and position estimation of the camera. It does not use any markers. Table 2. Prior research which related to teaching and learning in AR environments. AR motivates the student with it’s intuitive and user friendly interaction, and this could bring a better understanding in teaching and learning process (Chen 2006). In addition, users can view and interact with virtual content in a real environment with AR technology (Dunser 2008).
Designing AR Digital Storytelling Based on user characteristics and previous research with related to AR and Digital storytelling, the research framework in developing storytelling modules for remedial students is designed. Analyze User Analyze Contents Design
Development
• • • • • • • • •
Observation. Interview: Student and teacher. Learning outcomes for each story module. Task Design. Interface Design. Interaction Design. 3D objects. 2D background. AR Story.
- 3561 -
Story Storytelling elements Learning Activities Language Educational theories Holistic Development of Remedial Student Teaching and Learning Medium Assessments
• Cili Merah dan Cili Hijau (Red and Green Chilli) • Aku Sebatang Pen (I am a Pen). Added with moral and educational values. • Fun. • Engagement. Play with the characters in AR environment. Bahasa Melayu. Cognitive, Behaviourist and motivation theory • Cognitive. • Affective. • Psychomotor. Multimedia PC with webcam. Usability • Satisfaction. • Effectiveness. • Eficientcy. Literacy Evaluation • Comprehension. • Ability to tell the story back. • Listening skills.
Table 3. AR Digital Storytelling Framework In developing AR digital storytelling, the main objective is to teach remedial students to read, listen and pick up important words as well as understand the story. Analysis has been performed on the users, contents and creating the tales using Bahasa Melayu language. The next step is the task, interface and interaction design and the development phase. AR storytelling can be used and applied in the Malaysia school systems or as a home based learning because it uses the standard desktops or laptops with web cameras. The web camera identifies the black and white print patterns or ARToolkit markers, translates and displays AR views on the computer screen (Hafiza & Halimah, 2009).
Conclusions
The process of telling stories has changed through the time. Digital storytelling and augmented reality can be combined in order to build a better teaching and learning experience. It has the potential to provide a fun and engaging experience in literacy learning. With the help of learning theories, meaningful AR stories, good design, syllabus matching and a good balance of edutainment, it is believed that in future generation, especially in learning, disable students can learn to read Bahasa Melayu in a fun and enjoyable way through storytelling and available technology.
References 1.
2.
3. 4. 5.
Andreas Dünser, E. H. (2007). An observational study of children interacting with an augmented story book [Electronic Version], from www.ehornecker.de/Papers/edutainment_2007_dunser.pdf Andreas Dünser. (2008). Supporting Low Ability Readers with Interactive Augmented Reality. [Electronic Version]. Retrieved 8 Jan 2009, from http://www.hitlabnz.org/publications/2008.SupportingLowAbilityReadersWithInteractiveAugmentedR eality.pdf. Andreas Dunser, E. H. (2007). An Observational of Children Interacting with an Augmented Story Book. Springer-Verlag, 305-315. Chen C J. (2006). The design, development and evaluation of a virtual reality based learning environment. Australian Journal of Educational Technology, 22(1), 39-63. Eliana Medina, Y.-C. C., Suzanne Weghorst. (2008). Understanding Biochemistry with Augmented Reality [Electronic Version], 5. Retrieved 15 Jan 2009, from http://students.washington.edu/emedina/wordpress/wp-content/uploads/2008/10/edmedia07.pdf.
- 3562 -
6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30.
Hafiza Abas, H. B. Z. (2009). Augmented Reality: A Technology in Helping Reading Disabilities Student Paper presented at the The 1st International Conference on Educational Research and Practice (ICERP) 2009. Issa Saso, T., Iguchi, K., Inakage, M.: Little Red: Storytelling in Mixed Reality, SIGGRAPH 2003 Lucinda Kerawalla, R., Luckin Simon, Seljeflot Adrian, Woolard. (2006). Making it real: Exploring the potential of augmented reality for teaching primary school science. Virtual Real, 10(3), 163-174. Lynch, G., & Fleming, D. (2007) Innovation through design: A constructivist approach to learning. LAB 3000, RMIT University. http://lab.3000.com.au/research/research/index.jsp. Retrieved 5 May 2009. Lyon G. Reid, L. C. M. (1997). Critical Conceptual and Methodological Considerations in Reading Intervention Research. Journal of Learning Disabilities, 30, 578-588. Mark Billinghurst, H., Kato Ivan, Poupyrev. (2001a). MagicBook: transitioning between reality and virtuality. Paper presented at the CHI ' 01 extended abstracts on Human factors in computing systems. Mark Billinghurst, H., Kato Ivan, Poupyrev. (2001b). The MagicBook: Moving seamlessly between reality and virtuality. IEEE Computer Graphics Applications 21(3):6-8. Morten, F., & Benedikt, M. V. (2002). Augmented Chemistry: An Interactive Educational Workbench. Paper presented at the Proceedings of the 1st International Symposium on Mixed and AR. Neumann U. and Majoros, A., 1998: Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance. Proceedings of IEEE Virtual Reality Annual International Symposium ’98.4- 11. Nobuhiro, I., & Kaori, F. (2008). Making Reading Experience Rich with Augmented Book Cover and Bookmark. Paper presented at the Proceedings of the 8th Asia-Pacific,Computer-Human Interaction. Nobuko Taketa, K. H., Hirokazu Kato, Shogo Noshida,. (2007). Virtual Pop-Up Book Based on Augmented Reality. Paper presented at the HCII 2007, Berlin Heidelberg. Norhayati Abd. Mukhti, H. B. Z., Tengku Mohd Tengku Sembok, Siew Pei Hwa, Zurina Muda,. (2000). A Pedagogical Approach to Multimedia Courseware Development to Motivate Reading Habit. IEEE. Raphaael Grasset, A. D., Mark Billinghurst. (2008). Edutainment with a mixed reality book: a visually augmented illustrative childrens'book. Paper presented at the Proceedings of the 2008 International Conference in Advances on Computer Entertainment Technology. Raphael Grasset, A. D., Mark Billinghurst. (2008). The Design of a Mixed-Reality Book: Is It Still a Real Book? IEEE International Symposium on Mixed and Augmented Reality 2008. Reality Technology: Multiple Means of Interaction for Teaching Children the Ronald Azuma, Y. B., Reinhold Behringer,Steven Feiner,Simon Julier, Blair MacIntyre,. (2001). Recent Advances in Augmented Reality. IEEE Comput. Graph. Appl., 21(6), 34-47. Schank, R.C.: Tell Me A Story – Narrative and Intelligence. Northwestern Univ. Press, Evanston, Illinois (1995) Shibata, F., Yoshida, Y., Furuno, K., Sakai, T., Kiguchi, K., Kimura, A., Tamura, H.: Vivid Encyclopedia: MR Pictorial Book of Insects. In: The VR Society of Japan Annual Conference the 9th (in Japanese) pp. 611–612 (2004) Siddaharth Singh, A. D. C., Guo Loong Ng, Farzam Farbiz,. (2004). 3D augmented reality comic book and notes for children using mobile phones. Paper presented at the IDC 2004, Maryland, USA. The Digital Storytelling Association. (2002). The center for digital storytelling. http://www.dsaweb.org. Retrieved 17 September 2009. Tsou, W., Wang, W., & Tzeng, Y. (2006). Applying a multimedia storytelling website in foreign language learning. Computers & Education, 47, 17–28. Ucelli. G, C. G., Amicis R D, Servidio R. (2005). Learning Using Augmented Reality: Multiple Means of Interaction for Teaching Children the Theory of Colours. Proceeding of Intelligent Tecgnologies for Interactive Entertainment pp.193-202. Weinberger, J. (1996). Literacy Goes to School: The Parents Role in Young Children' s Literacy Learning. London: Paul Chapman Publishing Ltd. Wells, G. (1987). The Meaning Makers: Children Learning Language and Using Language to Learn. London: Hodder & Stoughton. Woods. E, M. B., Looser J, Aldridge G, Brown D, Garrie B. (2004). Augmenting the Science Centre and Museum Experience. Paper presented at the Proceedings of 2nd International Conference on Computer Graphics and Interactive Techniques in Australia and SouthEast Asia, Singapore. ZhiYing, Z., Adrian David, C., JiunHorng, P., & Yu, L. (2004). An interactive 3D exploration narrative interface for storytelling. Paper presented at the Proceedings of the 2004 conference on Interaction design and children: building a community.
- 3563 -
