A Mobile Augmented Reality Environment for Collaborative Learning and Training Reiner Wichert ZGDV, Computer Graphics Center Mobile Information Visualization Darmstadt, Germany
[email protected] Abstract: The following work focuses on the requirements for a Collaborative Augmented Reality System with the usage of Internet technologies to create new possibilities for collaborative teaching. In most learning environments, learning is imparted through seeing and hearing. In this paper, Collaborative AR Discussion and personal experience are also integrated in order to achieve a quicker transmission of study material and an increase in memory efficiency. We have analyzed the scenarios occur most frequently in service and maintenance and also the literature of Tele-Teaching. The needed requirements were derived and as a result integrated in a test system. This collaborative game simulates a web-based training and teaching environment in Collaborative AR as a new approach for teachers and trainees and describes scenarios and research perspectives for distance education and training at the same location. The game was used as a testbed to clarify the problems associated with AR and Collaborative Learning.
Introduction Through the integration of Multimedia, study material can be transmitted much faster. In 1965, William Glasser described what influences a student’s ability to retain information and ideas in his book Control Therapy in the Classroom. Respective to this thesis, people remember 10% of what they read, 20% of what they hear, 30% of what they see, 50% of what they see and hear, 70% of what they discuss with others, 80% of what they personally experience and 95% of what they teach others (Glasser 65). With Collaborative AR, learners communicate directly with each other in a face-to-face communication, but also interact with remote learners located in other learning rooms. According to Glasser, and as it relates to the interactive aspect in the learning environment of a Collaborative AR, people can remember information in a very efficient way, because users can discuss and have experiences by interacting with the augmented world in the collaborative learning and training environment. Tele-presence as a multimedia aspect and the access to worldwide distributed information enable a new way of teaching and learning. By combining learning and multimedia, a couple of net-based teaching and training systems have already been realized. By Tele-Teaching, we mean the distribution of knowledge in a synchronous Online Scenario or net of computers, in which the teachers and students work simultaneously. By Tele-Training, we mean the provision of study materials on network servers. These materials are retrieved by students asynchronously. Tele-Teaching is usually associated with university study, while Tele-Training is more suited for further professional training in a cooperate setting (Effelsberg). Augmented Reality (AR) has become an important part of computer graphics. “Unlike Virtual Reality where the physical world is completely replaced with synthetic environments, in Augmented Reality environments, 3D computer graphics objects are mixed with physical objects to become part of the real world” Billinghurst says (Billinghurst 00). Computer Supported Cooperative Work (CSCW) has also been expanded as an identifiable interdisciplinary research field. According to Wilson, CSCW is a generic term, which combines the understanding of the way people work in groups with the enabling technologies of computer networking, and associated hardware, software, services and techniques (Rodden 91). By combining CSCW with Augmented Reality, a new collaboration - Collaborative AR - became possible. Collaborative AR is defined by Reitmayr and Schmalstieg, where co-located users can experience a shared space that is filled with real and virtual objects (Reitmayr 01). The use of CSCW technologies in combination with Tele-Teaching and TeleTraining opened a new application area known as Computer Supported Cooperative Learning (CSCL) (McCONNEL 94). But it is a new idea to take CSCL into Collaborative AR. With Collaborative AR, group discussions are possible, where each user can interact and attain personal experience. In reference to Glasser, it is the best way to achieve a quicker transmission of study materials and an increase in memory efficiency.
Findings Today, the ability to communicate with almost anyone, to exchange knowledge, solve problems, and learn new things via computer networks is widespread. We know that we learn most effectively if we learn together with others in groups. The Internet allows cooperative learning independent of time and place (IPSI). Almost all of the existing e-learning systems do not support AR. So, one of the advantages of the component-based System will be to support individual users in their simultaneous interaction with augmented objects by using Collaborative AR while exchanging knowledge. To specify the requirements for a cooperative AR learning environment, the scenarios that occur most frequently in service and maintenance have been analyzed and two standard situations were identified: 1. cooperative interaction in a master / trainee scenario at the same place with e.g. skilled workers and a teacher having different views to the augmented world, standing and discussing in front of a machine; 2.
direct interaction with a remote expert using AR Technologies like interactive video and having the same view as the skilled worker. The expert can interact with the skilled worker and offer support. From these standard situations we have derived the following needed technical and interactive requirements. A collaborative AR system should give support and assistance to a learning group in their common work. Multiple users can be at the same place, at the same time, interacting together directly in the augmented world. Furthermore, group discussions are supported within the field of Augmented Reality. A common central view is established for each participant in order to get a defined understanding of the problem and to meet the requirements for this specific problem. Another aspect of teaching and collaboration is that an AR system has to provide remote support with the help of Augmented Reality over great distances. However, with a mobile aspect, the system would give multiple users access to information and help at any given time. It should clarify technical aspects, such as how to distribute the functionality in a learning collaboration and which methods can be used to get visual or haptic feedback. To meet the requirement of interaction by a collaborative group, a game could be implemented using the Collaborative AR Prototype System with the advantage of 3D group interaction and individual views for multiple users co-operating at the same place or with a connection over the Internet to remote users. As a result, a 3D Tetris clone was implemented as a testbed for different scenarios. The Tetris game poses a good possibility for realizing the requirements of a collaborative AR learning environment, because it can be played at the same place or from a remote location. It is possible to login dynamically and share the AR Tetris game with other participants. It simulates e.g. the collaboration of skilled production workers and the connection of technicians to a remote expert for support over the Internet. Exploring this research topic within a game presents a significant challenge. The Tetris game is split in the functionality realized by the Tetris Application and the visualization by the rendering component of the AR system. Both are embedded inside the web page, which displays points, the next piece, and the augmented view of the game (Fig. 1). The Tetris application can be started by any player and other users can login dynamically and share the AR Tetris with these participants. To realize this aspect, the status of the game is transmitted into the collaboration component. The first player who starts the game has the role of the master. The other players do not start another game, but get this status at the beginning of their session and initialize their game with these settings. If a player rotates a piece, the orientation of the actual piece will be set into the system. If a translation is done, the new position of the center of the piece is set in the system. A video server captures the picture of the camera and provides it to the rendering component and to the marker-based tracking system. A component was needed to handle the communication between the main components of the base AR Figure 1: 3D AR Tetris clone system and to notify other components or other clients co-operating together when values are changing. In the case of the collaborative learning environment, the values to transmit are the orientation of the scene, the names of the virtual objects used and also the actions like translations or rotations. The values are present on both the client and server side and are compared by way of a Push Mechanism. The Push Mechanism has been conceived to forward a notification to the clients immediately after an event occurs. Both client- and server-sided components can be informed via notification by the Event Mechanism. Another
component is responsible for controlling the collaboration. If two users are changing a value at the same time, this component has to recognize and regulate the collision detection and the common concurrent interaction.
Scenarios This Collaborative AR System can be used in a wide range of educational, e.g. in the field of commerce, to help users in their homes by way of a furniture company consultant, or in the industrial field, to support skilled workers standing in front of a machine by enlisting an expert. By using Collaborative AR to transmit study materials, a Cooperation at the same place is enabled by transmission of the scene, the actions and the orientation of the virtual objects to all participating users in the group; a remote cooperation over great distances is also enabled, requiring additionally the camera image of one of the cooperating users. Thus, training in a collaborative group interacting together at the same place or via distance education can be portrayed in a lot of scenarios. Both forms of cooperation can also be transmitted to the participants synchronously, as well as asynchronously. With an asynchronous connection, a later alteration of the augmented world is possible through assigning and storing scene, actions and orientation to a specific frame in the videostream. In this way, a later correction can be carried out to better portray an error or problem and to heighten the learning experience. In contrast, with synchronous learning in groups, collision recognition in terms of a simultaneous access of the virtual object is necessary. By implementing a Collaborative Augmented Reality learning environment, applications in the field of Tele-Teaching and Tele-Training become both synchronously and asynchronously realizable. For a remote synchronous learning environments, the remote user can have the same model of the real world, which has to be augmented, and can interact the same way as a collaborative user at the same site. In the case of the Tetris game, both users have the model of a pen or a joystick for interaction and the board of the game. Another synchronous possibility is to transmit the video images of one user to the remote user. The interaction can be done with an interactive graphic and will be sent through the event mechanism, as well. In the case of face-to-face collaboration, a pen with markers is used for 3D interaction. This allows a very natural interaction mechanism with the game. In a later version, speech recognition will be integrated to enable a multi-modal interface and, according to Glasser, we can improve the effectiveness of remembering information with the aspect of speech. But it is also possible to realize asynchronous teaching and training collaborations. We need to store the camera images of at least one user and also the actions like translations or rotations of the scene, the orientation of the virtual objects and the name of the scene dependent on the frame number of the images. If we store the images of more than one user, we can also switch between the views of the users. In this asynchronous scenario, the images are retrieved from memory, are identified by the corresponding frame number and are superimposed with the formerly valid objects. However, the virtual objects can be altered later as to position or orientation, or other virtual objects can be loaded. In this way, errors can be better explained and understood. This can occur on-site in the real world by projecting the virtual objects time-dependent to the recording in the real world with the help of a head-mounted display (HMD), or in a remote scenario, displayed on the desktop together with the video content. Discussions concerning a particular action could be subsequently initiated and problems better be solved. Explanations could be offered as to the point in time when something was done wrong and how it could be made better could be visualized. The Tetris game supports multiple users playing one game. This simulates group discussions and the connection to a remote teacher collaborating with a student while assembling a mechanical engine. It also handles the problem of common competing interaction. The actual field of view of other players and their different games can be integrated into the view of a teacher. One special view can be selected and the interaction can be switched to the selected game. This represents the scenario of a master who supervises several trainees learning the AR system or being instructed how to handle a machine. The master can select the view of a trainee and can give him advice and support in his augmented environment. As a result, a master can supervise a few learners. Another aspect is the view of common and private data by the collaboration participants dependent on the expertise of the users. Thus, each user of the learning collaboration can customize the view to his needs or to see different aspects of the same thing adapted to the circumstance. Converted to the Tetris game, the users play the same game and get a private view of the game and their interactions. The common data is the next piece and the score. The users can add and customize visual aspects to their needs like the deleted levels or the time played. How to visualize the result of cooperation by a learning group will be a further step in this work. If many people play the same game remotely, a player wants to know why the piece moves to the left even though
he/she pushed the piece to the right. Therefore, the representation of the results of the interaction is an important field. The best way to do this will be to use multiple indications like color, arrows and numbers (Fig. 2) or the player can get haptic feedback. 37% 21%
42%
Figure 2: Visualization of results
Figure 3: Collision detection and Semaphores
There are a lot of difficulties to solve in sharing virtual learning environments. Mutual exclusions have to be implemented and collision detection is needed. Semaphores have to control the access to interaction mechanisms. This problem can be approached in one game for two players, where two pieces are falling down (Fig. 3). Each person can interact with one piece, gets personalized information and has to coordinate his work with the other player while getting information when collisions are detected.
Conclusions We have analyzed specific scenarios. The needed technical and interactive requirements for these scenarios have been derived. A collaborative AR environment was suggested and realized prototypically as a test system for a Tetris game. The system can be used in a wide range of educational settings and it allows an asynchronous, as well as a synchronous access e.g. to a learning environment that can be used remotely for distance education or in a collaborative group interacting together at the same place in a master – trainee scenario. It is possible to login dynamically to the same group and share the augmented space with other participants at any given time. It was observed that the fast event mechanism can provide the information to other participants in real time by pushing the information. A further step for the future is to evaluate the outcome of this cooperative game and to transfer these initial results to education applications. But until then, a lot of experiences can be garnered and further research work completed.
References Glasser, W. (1965). Control Therapy in the Classroom, Harper & Row: NY, 1986; Reality Therapy:A New Approach to Psychiatry, Harper & Row, NY. Wolfgang Effelsberg, Lehrstuhl für Praktische Informatik IV, Universität Mannheim, Lehren und Lernen im Internet, http://www.igd.fhg.de/igd-a6/itti1.html Billinghurst, M., Poupyrev, I., Kato, H., May, R. (2000). Mixing Realities in Shared Space: An Augmented Reality Interface for Collaborative Computing, ICME2000, New York, USA. Rodden, T. (1991). A survey of CSCW systems, Interacting with computers - the interdisciplinary journal of human-computer interaction, 3(3), pp.319-353. Reitmayr, G., Schmalstieg, D. (2001). Mobile Collaborative Augmented Reality, ISAR2001, New York. McCONNEL, D. (1994). Implementing Computer Supported Cooperative Learning, Kogan Page Publishing. Fraunhofer IPSI, Integrated Publication and Information Systems, Darmstadt, http://ipsi.fhg.de/CSCL/ ARVIKA, Augmented Reality for Development, Production and Servicing, Germany, http://www.arvika.de/
