EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE MARIA ECONOMOU, LAIA PUJOL TOST Department of Cultural Technology and Communication University of the Aegean Harilaou Trikoupi & Faonos Str, Mytilene, 81 100, GREECE Email address:
[email protected] &
[email protected]
Abstract. The paper discusses the potential of Virtual Reality (VR) and the issues related with its application in the cultural sector, particularly its support of learning and social interaction. It examines the lessons learned from the evaluation of VR (and other ICT) applications in formal and informal learning environments and discusses their methodological limitations. It advocates the need for more evaluation research of the effect of Virtual Heritage (VH) on visitors and the development of appropriate methodological strategies.
1. Introduction The use of virtual reality is spreading in museums and cultural heritage settings, offering new opportunities both for archaeological, art-historical and other related scholarship, but also for public education and interpretation activities, which can be impressive and entertaining. Yet, this much advertised potential has in most cases not been fulfilled to its full extend (Stone 2005), due to a variety of practical problems, such as high development and maintenance costs, but also due to conceptual limitations, such as the use of a new medium in traditional and restricting ways, replicating old paradigms, or resistance and suspicion from archaeologists and related experts in the field who still prefer to use the written form for presenting their work (Niccolucci 2002; Staley 2003), as this offers tested ways of checking for validity and is more closely associated with professional credibility and recognition. However, despite these difficulties, the newly developed field of Virtual Heritage (VH), which was formed by the intersection of Virtual Reality (VR) with cultural heritage, continues to develop, borrowing from various disciplines (Roussou 2002). In most cases, the use of VR applications in cultural heritage settings is either a leisure activity or part of the organized visit of a school or other educational
2
M.ECONOMOU, L. PUJOL TOST
organization. Although these experiences are not directly related to a formal educational context, there is nevertheless always the expectation, that these applications will have—at least to some extent—educational content and goals and at the same time be entertaining, belonging to the area of ‘edutainment’, which balances and combines the two. Although several studies have been undertaken about the educational effectiveness of VR in formal educational environments, little is known about informal settings, such as museums and exhibition centers. In the latter case, learning needs to be defined in a broader and more multifaceted way, which moves away from the behaviorist model of set cognitive outcomes and encompasses often difficult to measure effects that are close to the constructivist paradigm, which also stresses interactivity as an important methodology for education and communication (Hein 1998). Interactivity is often considered as an essential part of VH applications and is one of those concepts which has defied a generally accepted definition in the field, despite the many attempts to describe it more specifically by authors from various fields (e.g. Adams and Moussouri 2002, Lévy 1995, Roussou 2004, Sims 1997, Steuer 1995; Pujol Tost 2005). The level of interactivity of the various types of VH applications varies and has so far been often limited. In many cases, even when the program is interactive, navigation and choice of action is controlled by a guide, a member of staff of the organization who acts as a mediator. In this model of interaction, the role he/she plays is very important, affecting the type of experience the visitors will have; yet surprisingly, very little is known about the ways that the complex relationship between visitors, mediator and VR application is constructed. It is ironic in a sense, that despite the very demanding in terms of resources, design of the virtual content, and maintenance of the hardware, one of the most crucial factors in terms of educational benefit, entertainment and construction of meaning for visitors remains the human factor of the guide as presenter and mediator. Whether visitors come to use VH applications as part of organized education groups or at their leisure time, an important parameter affecting their experience, including possible learning outcomes, is the social context of the visit. Although the importance of the social aspect of visiting a cultural heritage settings has been shown by several studies (e.g. Falk and Dierking 1992, 41-54; and the classic early papers by McManus 1987; McManus 1988) and is today generally recognized, new technology applications in this area continue to cater primarily for single users, often isolating them from their group and hindering communication. Unlike other types of ICT applications, such as multimedia kiosks or handheld devices, VR programs are in most cases addressed to groups, yet are rarely designed to actively support or even promote social interaction. These types of interactions (verbal, tactile, non-vocal) have an impact on how the technology is encountered, explored and understood. Augmented (AR) and Mixed Reality (MR), with their closer link to the physical environment and their
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 3
less obtrusive character are supposed to offer the advantages of VR, but without disrupting social interaction. However, systematic evaluation studies are required to investigate this in practice, as some initial evaluation findings seem to contradict these claims (Vom Lehn and Heath 2003), As such studies are extremely limited in number, in our investigations of lessons learned from the evaluation of VR applications for presenting the past and their effects on visitors, we had to broaden our perspective to include also evaluations from different, even though related fields, such as instructional technology and pedagogical studies investigating the effect of VR applications in formal educational settings, or museum visitors studies of new technology (but not exclusively VR) applications. In the analysis of the findings, we took into account the differences and similarities of each context, as well as the methodological limitations. 2. Evaluation studies from formal and informal learning environments 2.1 EVALUATION STUDIES OVERVIEW
The body of work from formal learning environments is considerably larger, as these have a longer tradition of using computers for learning. Their interest is usually directed in investigating how the traditional instructional strategies need to be modified with the inclusion of ICT tools and what is their particular contribution compared to traditional methods. Museum visitor studies (where similar initiatives are considerably fewer in number) do not focus to the same extent on specific cognitive gains or learning outcomes, so if we want to investigate how ICT, and VR in particular, can contribute to the learning process and its key elements (person, task, environment), we have to examine and extrapolate from the findings of mainly formal learning environments. After a first phase in museum evaluation where the influence from behavioral psychology led to a pre-occupation with specific and quantifiable learning outcomes from museum exhibits and applications (particularly in the USA), over the last twenty years there has been a ‘shift of focus from what exhibits do to what visitors make out of exhibits’ (Miles 1993, 28) and how they construct meaning (Hein 1995; Hein 1998). Evaluation in museums and cultural heritage institutions is now focusing more on how ICT applications affect the whole experience of the exhibition: visitors’ behaviour, social dimension of the visit, integration with other exhibits (the effect of high-tech media for the apprehension of exhibition discourses through text, images and hands-on). The studies we investigated in this paper cover a wide range of interfaces and types of applications, from multimedia to immersive virtual reality (IVR), through to desktop simulations. In formal learning evaluations, the most popular
4
M.ECONOMOU, L. PUJOL TOST
applications are multimedia ones (often called, Computer Assisted Learning Environments (CALE) or Virtual Learning Environments (VLE)), as they are the natural development of older systems, transferring to the digital environment the materials traditionally used at school, text and images, adding also hypertextual interactivity. 2.2. METHODOLOGICAL LIMITATIONS
Evaluation studies at formal and informal environments also differ in their methodological approach. The former use a more traditional experimental approach, where the conditions are more controlled and the studies are almost always comparative, while the latter often use observation, focus group discussions and in-depth interviews (to gather qualitative information) and individual questionnaires (for more quantitative data), and often involve the evaluator in the setting as both an observer and a participant. Also, due to the relatively recent introduction of ICT in museums and exhibitions, the lack of resources and suitably trained staff to carry out evaluation research, and the political nature of evaluation with its often sensitive findings, evaluation studies in the cultural sector are limited, but even when they do exist, they tend to be informally managed, produce data of varying quality and depth, do not form part of an overall plan or strategy, and are actually often not even made public (Economou 2004). In many cases, evaluation is planned more as an afterthought, without being integrated in the development process. Furthermore, the lack of generally accepted methodological frameworks affects the comparability of the data and the ability to generalize the results (Economou 2004). In general, the results from both environments are very difficult to universalize because of the methodological characteristics of the studies. These include either statistical limitations, such as insufficient number of users included or non-comparable groups, or the experimental conditions, such as the influence of the test design or lab-based evaluations which ignore the complexity of studying visitors in the natural environment of the exhibition. Despite these shortcomings, it is useful to examine the evaluation findings in the broader field as they still bear great relevance for VH and can guide future developments. 3. Evaluation findings 3.1 USER MOTIVATION AND CONTROL
ICT’s and particularly VR’s power of motivation and popularity over other instruction and communication media is reported by numerous studies of
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 5
different age groups in both formal and informal contexts (e.g. Hilke et al. 1988; Chou and Liu 2005; Johnson et al. 1998; Harper et al. 2000; Wheeler et al. 2002; Jackson and Fagan 2000; Pimentel and Teixeira 1995; Lee et al. 2005). As Chou and Liu (2005) reported in a study of a web-based multimedia CALE for secondary students (with a control group using traditional learning methods), CALE allowed better control of the students’ own learning, which they themselves reported as making them more efficient. This augmented their satisfaction and generated a better learning environment than the traditional techniques. Similar findings were reported at a study of secondary students’ collaborative creation of a web site (Wheeler et al 2002), where the fact that the students and not the teacher or the curriculum were driving the task, provided them with a feeling of control which motivated them to continue working. A few years earlier, Schroeder (1996) also stressed the importance of users’ control in the virtual environments, as this augments significantly their levels of attention, retention, production and motivation and consequently leads them to obtain better scores when learning results are tested. This was also confirmed with young users. A study of children between 6 and 10 using a CAVE in groups of four, where one acted as task coordinator through an avatar, showed that comprehension of the content and the task at hand was a direct function of the level of involvement and engagement: the more engaged in the task (mainly the leaders), the better answers they gave in the post-activity questionnaires (Johnson et al. 1998). Early evaluation studies of museum interactive applications also recorded visitors’ satisfaction with being able to control their navigation and make their own choices through the system (Morrissey 1991; Economou 1998). For virtual heritage environments, these findings indicate that the attraction of direct control of the application (and not through a guide or other mediator) should not be underestimated. Other studies which also reported users’ increased motivation to use the computer-based system, related this more to the competitive way in which they approached the task at hand, as they associated it with games (another lesson which can be incorporated in the design of VH applications). This was the case, for example, with students with learning difficulties (Neuman 1989), senior high-school management students (Yu et al. 2002) or the numerous secondary students at the collaborative VL environments studied by the team at the HOC Laboratory of the Politecnico di Milano (Di Blas et al 2005b). The researchers in the latter case found that the presence of a clear goal stimulated motivation (Di Blas et al. 2005a). On the other hand, when students are over-motivated, this can adversely affect their understanding of the subject. An observational study of secondary education students using a multimedia application with VR as a navigation metaphor showed that too much fascination with the system can lead students to explore the environment too quickly and randomly to allow an acceptable
6
M.ECONOMOU, L. PUJOL TOST
understanding of information (Biosca et al 2002). Similar findings were recorded in the evaluation of a museum multimedia application, which attracted visitors with no prior interest in the subject to the specific exhibition, but did not necessarily lead to increased understanding of the subject, as a portion of these users seem to have initially focused on the process of using the technology, rather than on the content (Economou 1998). This has considerable implications for applications in museum and cultural heritage environments, where the average time spent with the system is usually very limited. 3.2 SOCIAL INTERACTION
Studies of formal learning environments show that educational effectiveness is closely related to the interaction with other users (Di Blas et al 2005a). The team at the HOC lab stress that what is relevant is not the technology but the possibility to establish a link with the other users in the virtual environment (what they call ‘virtual or conceptual presence’). However, they also stress that what cannot fail is this link: their studies of students using the collaborative VR environment in various European secondary schools showed that if there are problems with the connection, the whole process of communication and learning will be negatively affected (Di Blas et al. 2005b). Studies with younger children working in collaborative environments also confirm that the most important aspect is not the presence or not of technology but the equality of conditions when interacting with the material (Scott et al. 2003) Interestingly, collaborative formal learning studies show that the presence of virtual environments did not prevent (Bricken and Byrne 1993), and in some cases, actively encouraged collaboration and social exchange (Di Blas et al. 2005b). Examples with young children showed the importance of social interaction with classmates in the virtual environment which was linked with better results at the post-activity tests (Johnson et al. 1998). Social interaction not only with the peer group but also with an outside helper or expect advisor is also important, assisting for example, in the interpretation of the empirical evidence encountered in the simulation (Reid et al. 2003). However, the picture is different in the cultural setting, where as we have mentioned above, ICT applications are rarely designed to support effectively social interaction. One exception is the evaluation of an experimental system designed from the beginning to support social interaction of local and remote companions in a mixed reality museum and historic house experience which was tested at an exhibition at the Lighthouse Centre for Architecture, Design and the City in Glasgow (Galani and Chalmers 2003). However, this explicit design which supported the co-visiting experience and proved that ‘the museum’s remote presence was treated not strictly as an information space, used in isolation, but also as a social place to visit, enjoy and relate to others’ (Galani
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 7
and Chalmers 2003, 15) is very rare in ICT applications in cultural heritage settings. We already know from early studies that in most cases visitors use museum computer interactives in groups (Doering et al. 1989; Giusti 1994b; McManus 1993b; Menninger 1991; Morrissey 1991). Yet, the study of the use of computer interactives at Explore@Bristol and the Science Museum London showed that it was very difficult for groups to carry out a collaborative exploration (Heath and vom Lehn 2002). The researchers observed that ICT applications have problems to fit the social dimension of the exhibit because their interface is designed for a single user that interacts with the machine through a linear input - output dialog without allowing more sophisticated types of interaction. The study which the same team carried out on the use of mobile computing at a big art gallery in London yielded similar results, showing that these devices monopolize the visitors’ attention, instead of fermenting an egalitarian communication between the different elements involved: user, mobile device, environment, objects and other visitors (vom Lehn and Heath 2003). Similar findings were reported at the evaluation of the use of mobile computing at the Exploratorium in San Francisco, which recorded that users experienced a feeling of isolation either with regard to other visitors or with regard to the rest of the exhibition (Hsi 2003). The researchers also discovered that visitors had problems to establish transferences between the real and the virtual world when there were no reference points to allow the superposition of the two kinds of explanation (Hsi 2003). Along the same lines, Jovet (2003) refers to the difficulty of successfully integrating new media in exhibitions, when she writes that depending on the location of VR in the exhibition, it can break the traditional linearity of the visit by creating a surrounding activity area (parallel to the observation of objects) which, because of its interactivity, generates emotional and physical responses that can still today be perceived as inappropriate inside the exhibition. The VR can interfere with the contemplation of originals and in some cases becomes the protagonist, to the detriment of the message or the objects. On the other hand, research of early multimedia systems in the galleries suggests that, when thoughtfully designed and carefully positioned, interactive systems can actually complement and increase the enjoyment of the exhibits by acting as supplements and enhancements, rather than replacing them (Allison & Gwaltney 1991; Hilke et al. 1988; Mellor 1993; Menninger 1991; Morrissey 1991; Wanning 1991a; Worts 1990; Economou 1998). In any case, the presence of computer applications in the gallery affects the way visitor behave in the exhibition, even those who do not use the application (Hilke et al. 1988; Worts 1990; Morrissey 1991). Museum evaluations suggest that visitors spend considerable time using the computer application, however, this seems to be add-on time, not reducing the
8
M.ECONOMOU, L. PUJOL TOST
normal amount of time they would have spent in the gallery if the interactives were not present (Allison and Gwaltney 1991, Economou 1998). These studies suggest that the use and presence of the computers often encouraged visitors to pay closer attention to the objects and exhibition themes. Similarly, the mixed reality prototype tested at the Lighthouse in Glasgow showed that by allowing different members of the group to share their interests and view the same content, contributed to a shared exploration of the display which led to greater engagement with the exhibition (Galani and Chalmers 2003). Moving back to the formal learning environment (where social interaction and individual learning are perceived differently), it is interesting to examine the contrasting results from a study of collaborative learning with secondary science students which suggests that group interaction is not always the most effective learning strategy and that the linear bidirectional exchange between an individual student and the PC can also obtain very positive results (Chiu 2002). This study suggests that face-to-face collaboration is better for creative exploration of problems and generation of ideas, while cooperation in Cyberspace obtains better results with linking, interpreting and integrating ideas. Another study from a formal learning environment which tested biology students’ perception of virtual versus real fieldtrips showed that they did not consider that virtual learning environments are or have to be a substitution of real field trips, but they believed that they were motivating and useful as learning tools, for example to prepare field trips in advance and therefore take more advantage of them (Spicer 2001). This might have similarities to attitudes towards organized museum and heritage site visits, as the traditional fear of curators and heritage custodians that virtual museums and exhibits will replace the real ones has not been confirmed by evidence so far. The evaluation of mobile computing at the Exploratorium showed that the presence of a virtual guide was a source of motivation and inspiration for users to try new ways of interaction with the exhibition and to pay more attention to the exhibition discourse (His 2003). This brings up a wider issue for virtual heritage applications. How can the loss of direct human contact be compensated in the digital environment? Can avatars, digital guides, links with context experts, artists or other visitors in the virtual space provide the answer? We need more imaginative and thoughtful work in this direction. 3.3 AGE
In terms of the influence of the users’ age, some IVR studies report that young people are more active than adults in the construction of virtual worlds (Bricken and Byrne 1993). More specifically, they also report that 10 to 12-year-olds appear more comfortable with the construction of objects while 13 to 15-year-
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 9
olds work without problems in the context design (Bricken and Byrne 1993), findings which are consistent with evolutionary psychology predictions. VR’s power of attraction seems to appeal to several age groups, from primary education through to university students, but we are missing data from older adults, and again, particularly how they interact in cultural settings. Within this relative young user base, some formal learning studies suggest that there is no relationship between age and use of the system and that in fact, this was less a matter of age than of accumulated experience with computers (Vance Wilson 2000). With few exceptions (Economou 1998), early studies of museum multimedia interactives suggest that users were often young (Allison and Gwaltney 1991; Doering et al. 1989; Giusti 1994a; McManus 1993b; Menninger 1991; Morrissey 1991; Sharpe 1983), but with the continuing use of ICT this is likely to change. 3.4 GENDER
Most studies report no significant differences between the genders in the results or abilities of using ICT applications (Shaw and Marlow 1999; Panagiotakopoulos and Ioannidis 2002); only differences in the goals and approaches. For example, an early study of 13-15 year-old students using an IVR system showed that boys defined first their objectives and then designed the contents following the plan, while girls were more process-oriented (decided a concept, built several objects and chose what would be included) (Bricken and Byrne 1993). In some cases, female users use the system more frequently than male ones, particularly when its main function is communication (Vance Wilson 2000), get more engaged in collaboration than boys who prefer to work in individual environments (Jackson and Fagan 2000), and obtain better learning results when assistants in the system are represented through human faces (Huff 1996). These indicate that gender biases are not due to cognitive but to social reasons. The early study of the program ‘The Caribou connection’ at the National Museum of Natural History in Washington, DC included both direct and indirect users of the application and recorded a phenomenon which is still often observed in cultural exhibitions: while over 60 percent of the direct users of the computer system were male, the indirect users were divided equally between the sexes. Although we need to be careful to take into account the special characteristics of every case, when interpreting these data, it would seem that visitors who voluntarily approach a computer or compete for its use in a busy gallery, are generally those who are already comfortable and familiar with using the technology (McNamara 1986). Until recently, these were more likely to be young and male, but this is changing rapidly. For example, the evaluation of the use of a multimedia interactive at the Museum of Oxford in the mid-1990s
10
M.ECONOMOU, L. PUJOL TOST
showed that female and male visitors were equally likely to use it, while female users were more likely to spend longer with the program (Economou 1998). 3.5 PREVIOUS KNOWLEDGE OF THE SUBJECT, COMPUTER FAMILIARITY AND ATTITUDE TOWARDS TECHNOLOGY
Most formal education studies confirm that there is a positive correlation between previous ICT experience and good perception of such tools (Shaw and Marlow 1999; Ronen and Eliahu 1999), although there are few exceptions which indicate that previous experience with or attitude towards computers did not affect the results (Baxter and Preece 1999). In most cases, as users get more experienced automating the interface manipulation, they are freer to concentrate on the content. Prior knowledge of the subject appears to be a significant influence in procedural as well as factual contents learning, as it is easier for experts to automatically decipher the meaning in the different computer presentation formats (e.g. still graphics, animation, text) and therefore, concentrate on more advanced cognitive tasks; on the contrary, subject novices have to make a bigger effort to decode the meaning of the various representations (Ronen and Eliahu 1999) and, as one study showed, obtained better results with static graphics because these helped them to construct their mental representations, while animation did not improve their learning as it led to quicker saturation of visual memory (which has to process both visual and sequential components) (Chan Lin 2001). In general, the comparisons between experts’ and novices’ results support the idea that the former manipulate and choose more easily different kinds of representation depending on the task because they have a wider and deeper conception of the knowledge domain, while the latter have a more fragmented and factual vision and can only act upon the more superficial aspects (Wood 1999). 3.6 LEARNING STYLES AND TYPE OF LEARNING
One of the advantages of well designed CALEs is that they are respectful of different individual learning styles and rhythms (Wheeler, Waite et al 2002), although there is a suggestion that there might be a negative relationship between certain learning styles and interactivity: one study reported that the more theoretical the students are, the less they like using ICT as a learning tool because they consider it impersonal and prefer more traditional instructional systems (Shaw and Marlow 1999). A small study of 10-11-year-olds with strong visual learning style and difficulties with linguistic lessons showed that use of a visually rich computer environment led to notable improvement of global skills and especially of the capacity to revise text and detect mistakes (Vincent 2001). This indicated that, thanks to their flexibility, virtual environments can compensate for the shortcomings of traditional instructional strategies and
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 11
satisfy the needs of different cognitive styles, especially of the visuospatial one. This has direct implications for the design of virtual heritage applications which can compliment formal education in dynamic and rich ways in this direction. Most researchers agree that one of VR’s greatest advantages is that it can contribute to the understanding of abstract ideas by making them visually concrete (Scanlon et al. 1998). As VR is a visual communication tool, it favors that information is stored, recalled and represented in a visual way (Osberg 1997). In an interesting qualitative study which compared university students’ responses to examining the reproduction of a painting with getting immersed in a virtual reconstruction of it, the group which had worked with VR showed more interest in pictorial techniques, were more likely to spontaneously think about why it was like that instead of what it was (meta-perspective), were able to conceptualize the experience at a more abstract level, and provided more answers based upon their own experience or associative links (suggesting that VR stimulated a more imaginative association of materials). On the other hand, the other group tended to underline the cultural or inferential aspects and adopted a more speculative approach, in which they referred to the author’s work inside its historical context (Antonietti and Cantoia 2000). This has some analogies with the findings of a study of younger users (8-9 year-olds) which compared their recall of a story read traditionally and experienced as an interactive storybook (Trushell et al. 2003). The group who had read the story remembered better its structure, while those who had played the game remembered better the details. A study of even younger users (between 4 and 10 years old) which compared the acquisition of temporal concepts using real objects and working with their virtual reconstructions, indicated that VR was better for showing movement or change in the environment and for improving precision by avoiding irrelevant details that might confuse children, and was more effective when it underlined or emphasized concrete aspects through sounds, graphics, movements (Panagiotakopoulos and Ioannidis 2002). Several studies of IVR systems or VR simulations refer to the effective learning of spatial concepts (Bricken and Byrne 1993), where for example, students obtained better results when using a VR simulation rather than verbal descriptions to understand geometric concepts (Song and Lee 2002). Interactive virtual systems which encourage the creation and use of icons and symbols inside an experimental context lead the students to reason abductively and, through the formulation and verification of hypotheses, to learn from their “errors” without the fear of failure featured in traditional formal learning (Osberg 1997) or cultural heritage contexts which use face-to-face contact with museum staff, guides and educators. Similarly, another study reported on the ability of computer simulation games (through their anonymity and the distance
12
M.ECONOMOU, L. PUJOL TOST
they impose) to reduce the negative emotional feelings which can arise from competitive learning situations where there is direct physical contact (Yu et al. 2002). Although most studies suggest that ICT applications, and VR in particular, can be a useful tool for learning, it appears to encourage certain types of learning (sometimes, to the disadvantage of others), and is not necessarily more effective than traditional strategies. For cultural institutions, this is a reminder that they should not invest all their efforts in VR applications, but examine them as only one of the various possible interpretation media. On the other hand, early studies of multimedia applications in museums suggest that they were influential in stimulating qualitative learning, in some cases, raising new issues and encouraging visitors to challenge their perceptions (McManus 1993b; Peirson Jones 1995) and providing contextual information about the objects (Economou 1998). They also indicate that the interaction with the computer applications had clearly been a memorable experience for the visitors (McManus 1993a; Economou 1998). 3.7 DESIGN AND INTERFACE ISSUES
Despite the general claims about the intuitiveness of VR and mobile computing interfaces, several studies of both formal and informal environments report that users often need help because they do not know what to do inside the virtual environment (Bricken and Byrne 1993; Jackson and Fagan 2000; Scott et al. 2003; Heath and vom Lehn 2002; Hsi 2003). More work is needed in the design of appropriate help in cultural heritage environments, which usually need to be quickly and easily used for a short period of time by users from very different backgrounds and with varying computer experience. 3.7.1 Immersion In a study which summarized the findings from several evaluations of VR in educational settings, Youngblutt (1998) reported that immersion did not seem to produce any specific intellectual effect but in all cases without exception it contributed to considerably augment motivation, which can reinforce indirectly the cognitive gain. Belaën (2003) examined the use of immersion in science museums, but also confirmed the findings from formal learning studies. She reported that immersive devices did not guarantee any immediate positive attitude or acquisition of knowledge: users needed to have the representation codes of the original knowledge domain because otherwise the immersive application would become another source of problems, added to those which arise while trying to understand the content.
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 13
These are interesting findings, especially when we take into account the considerable resources that cultural institutions usually need to spent to create immersive environments. 3.7.2 Realism versus symbolism/abstraction Several studies suggest that simplified or abstract representations can be more effective and better understood than highly realistic ones (Osberg 1997). Realism is not necessarily positive from a strictly cognitive point of view–not considering motivation–as it may distract the user and prevent him/her from focusing on the contents (Lee et al. 2005), which are very important both in formal learning and cultural heritage settings. An early paper by Alessi (1988) confirms the same point: maximal realism can be counter-productive for students with little knowledge, in which case it is more effective to work with simplified representations of objects and phenomena. Interestingly, the same point has been made also about users who become experts in the subject, in which case Hedberg and Alexander (1994) believe that the representation needs to be less realistic because these users can work with increased levels of abstraction. Realism in the virtual environment is crucial only in specific cases, for example when it is used for therapeutic purposes to eliminate phobias by placing the user in ambiguous situations, simulating closely the natural environment, but without any risk (Tarr and Warren 2002). In virtual heritage, however, apart from realistically visualizing sites and architectural details, it is important to also interpret, tell stories, contextualize the objects found, allow artistic means of expression (Roussou and Drettakis 2003), and create an emotional response to users. 4. Conclusions The data from both formal and informal environment clearly show the great motivational power of ICT applications, and particularly of VR, yet the data about its effect on learning are inconclusive, although they suggest that VR encourages specific type of learning. Collaborative applications in classrooms confirm the importance of the social aspect of the experience and the communication with the others in the group (whether onsite or online), but in museum and public exhibition settings the successful integration of the social context of the visit in virtual applications remains problematic. The studies show that age and gender profiles of users are changing with the increased penetration of technology in society and when these biases remain, they are related to social, rather than cognitive reasons. Despite the large and increasing body of evaluation work in formal learning environments, the data we can extrapolate for virtual heritage environments remain limited, as learning in informal settings is not the only preoccupation,
14
M.ECONOMOU, L. PUJOL TOST
and when it is, it takes a wider and more indirect form (encompassing, for example, the change of attitudes, the understanding of other cultures, the emotional response to artistic work, to mention just a few parameters). It is in this area, and particularly in the investigation of the complex and fascinating interrelationships between visitors, virtual applications, mediators, and real objects and sites that we need more systematic and thorough evaluation work. In order for this to succeed in illuminating the multifaceted social phenomenon of heritage use with the application of the technology, it is necessary to develop new and appropriate methodological frameworks, balancing qualitative with quantitative approaches and borrowing, where appropriate, from a range of disciplines. Acknowledgements We would like to thank Maria Roussou for providing access to material and useful input to some early ideas which are discussed in this paper, and Maro Alexaki for discussions on the role of the museum educator in virtual heritage experiences. This work is part of the wider research network CHIRON: Cultural Heritage Informatics Research Oriented Network, 2005-2008, which is supported by the European Community's Sixth Framework Programme under contract number MEST-CT-2004-514539.
REFERENCES Adams, Marianna. and Theano Moussouri. 2002. The interactive experience: Linking research and practice. In Proceedings of an international conference on Interactive Learning in Museums of Art, London, 17-18 May 2002. London: V&A Museum. Internet. Available from http://www.vam.ac.uk/res_cons/research/learning/; accessed 16 October 2005. Alessi, Stephen M. 1988. Fidelity in the design of instructional simulations. Journal of ComputerBased Instruction 15: 40-47. Allison, David K. and Tom Gwaltney. 1991. How People Use Electronic Interactives: “Information Age - People, Information & Technology”. In ICHIM ’91 (Conf. Proceedings, Pittsburgh 14-16 October 1991), ed. David Bearman: 62-73. Pittsburgh, PA: Archives and Museum Informatics. Antonietti, Alessandro and Manuela Cantoia. 2000. To see a painting versus to walk in a painting: an experiment on sense-making through virtual reality. Computers & Education 34(3/4): 213-223. Baxter, J.H. and Preece, P.F.W. 1999. Interactive multimedia and concrete three-dimensional modelling. Journal of Computer Assisted Learning 15: 323-331. Belaën, Florence. 2003. L'immersion au service des musées de sciences. In ICHIM 2003 (International Cultural Heritage Informatics Meeting). Conf. Proceedings published on CD-ROM only, Paris, École du Louvre, 8-12 September 2003. Archives and Museum Informatics. Biosca, Eloi, Elena Cantarell, Marta Sancho and Teresa Vinyoles. 2002. Reconstruyendo el pasado. Enseñar con tecnologías de realidad virtual. Iber: Didactica de las Ciencias
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 15 Sociales, Geografia e Historia 31 (March 2002): 103-111. Bricken, Meredith and Christine M. Byrne. 1993. Summer Students in Virtual Reality. A Pilot Study on Education Applications of Virtual Reality Technology. In Virtual Reality. Applications and Explorations, ed. Alan Wexelblat. Boston: Academic Press Professional: 199-217. Chan Lin, L. 2001. Formats and prior knowledge on learning in a computer-based lesson. Journal of Computer Assisted Learning 17: 409-419. Chiu, Chiung-Hui. 2002. The effects of collaborative teamwork on secondary science. Journal of Computer Assisted Learning 18: 262-271. Chou, Shih-Wei and Chien-Hung Liu. 2005. Learning effectiveness in a Web-based virtual learning environment: a learner control perspective. Journal of Computer Assisted Learning 21(1): 65-76. Di Blas, Nicoletta, Evelyne Gobbo and Paolo Paolini. 2005. 3D Worlds and Cultural Heritage: Realism vs. Virtual Presence. In Museums and the Web 2005, (Conf. Proceedings, Vancouver, Canada 19-22 March 2003) eds David Bearman and Jennifer Trant. Toronto: Archives and Museum Informatics. Internet. Available from http://www.archimuse.com/mw2005/papers/diBlas/diBlas.html; accessed 10 January 2006. Di Blas, Nicoletta, Evelyne Gobbo, Paolo Paolini and Caterina Poggi. 2005. 3D Worlds for education: cooperation and virtual presence. In Virtual reality at Work in the 21st Century. Impact on society, ed. Hal Thwaites: 375-384. Ghent, Belgium: International Society on Virtual Systems and Multimedia. Doering, Zahava D., Pawluklewicz, J. D. and Bohling, K. 1989. The Caribou Connection: Will People Stop, Look, and Question? Washington, DC: Smithsonian Institution. Economou, Maria. 1998. The evaluation of museum multimedia applications: lessons from research. Journal of Museum Management & Curatorship 17(2), 173-187. Economou, Maria. 2004. Evaluation strategies in the cultural sector: the case of the Kelvingrove Museum and Art Gallery in Glasgow. Museum and Society 2 (1) March 2004. Internet. Available from http://www.le.ac.uk/ms/m&s/m&sframeset.html; accessed 10 October 2005. Falk, John. and Lynn Dierking. 1992. The Museum Experience. Washington, DC: Whalesback Books. Galani, Areti and Matthew Chalmers. 2003. Far away is close at hand: Shared mixed reality museum experience for local and remote museum companions. In ICHIM 2003 (International Cultural Heritage Informatics Meeting) Conf. Proceedings published on CD-ROM only, Paris, École du Louvre, 8-12 September 2003. Archives and Museum Informatics. Internet. Available from http://www.dcs.gla.ac.uk/~areti/Galani_ichim03.pdf; accessed 25 October 2005. Giusti, Elen. 1993a. ‘Global Warming: Understanding the Forecast’ - An Assessment. American Museum of Natural History evaluation report, October 1993. Giusti, Elen. 1994b. ‘Hall of Human Biology & Evolution’: Summative Evaluation, American Museum of Natural History evaluation report, 3 March 1994. Harper, Barry, John Hedberg, Robert Corderoy, and Robert Wright. 2000. Employing cognitive tools within interactive multimedia applications. In Computers as cognitive tools: no more walls, ed. Susanne P.Lajoie: 227-246. New Jersey: Lawrence Erlbaum.
16
M.ECONOMOU, L. PUJOL TOST
Hein, George E. 1995. Evaluating teaching and learning in museums. In Museum, Media, Message, ed. Eilean Hooper-Greenhill: 189-203. London: Routledge. Hein, George E. 1998. Learning in the Museum. Routledge, London. Heath, Christian and Dirk vom Lehn. 2002. Misconstruing interactivity. In Proceedings of an international conference on Interactive Learning in Museums of Art, London, 17-18 May 2002. London: V&A Museum. Internet. Available from http://www.vam.ac.uk/res_cons/research/learning/; accessed 16 October 2005. Hilke, D.D., Hennings, E. and Springuel, M. 1988. The Impact of Interactive Computer Software on Visitors’ Experiences: A Case Study. ILVS Review 1 (1): 34-49. Hsi, Sherry. 2003. A study of user experiences mediated by nomadic web content in a museum. Journal of Computer Assisted Learning 19: 308-379. Huff, C. 1996. Gender, computer assisted learning and anxiety: with a little help from a friend. Journal of Educational Computing Research 15(1): 65-69. Johnson, Andrew, Maria Roussos, Jason Leigh, Christina Vasilakis, Craig Barnes, and Thomas Moher. 1998. The NICE Project: learning together in a virtual world. In Proceedings VRAIS'98: 176-183. Jovet, Viviane. 2003. Le multimedia dans l'exposition: la double problématique de l'appropriation et de l'intégration d'un media marginal. In ICHIM 2003 (International Cultural Heritage Informatics Meeting). Conf. Proceedings published on CD-ROM only, Paris, École du Louvre, 8-12 September 2003. Archives and Museum Informatics. Lee, Heebok, Sang-Tae Park, Hee-Soo Kim, and Heeman Lee. 2005. Students' understanding of astronomical concepts enhanced by an immersive Virtual Reality system (IVRS). In 3rd International Conference on Multimedia and Information and Communication Technologies in Education. Caceres, Spain 7-10 June 2005. Recent Research Developments in Learning Technologies. Internet. Available from http://www.formatex.org/micte2005/313.pdf; accessed 14 January 2006. Lévy, Pierre, 1995. Qu'est ce que le virtuel? Paris: La Découverte, coll. ‘Sciences et société’. McNamara, Patricia. 1986. Computers Everywhere - But What Happened to the Research?. The Journal of Museum Education: Roundtable Reports 11 (1), 21-24. McManus, Paulette M. 1987. It's the company you keep ... the social determinants of learningrelated behaviour in a science museum. International Journal of Museum Management and Curatorship 6: 263-70. McManus, Paulette M. 1988. Good companions ... more on the social determination of learningrelated behaviour in a science museum. International Journal of Museum Management and Curatorship 7: 37-44. McManus, Paulette M. 1993a. A Study of Visitors’ Memories of Gallery 33. In Gallery 33: A Visitor Study, ed. Jane Peirson Jones: 56-73. Birmingham: Birmingham Museums and Art Gallery. McManus, Paulette M. 1993b. A Survey of Visitors’ Reactions to the Interactive Video Programme “Collectors in the South Pacific”. In Gallery 33: A Visitor Study, ed. Jane Peirson Jones: 74-114. Birmingham: Birmingham Museums and Art Gallery. Miles, Roger. 1993. Grasping the greased pig: evaluation of educational exhibits. In Museum visitor studies in the 90s, ed. Sandra Bicknell and Graham Farmelo: 24-33. London: Science Museum. Morrissey, Kris. 1991. Visitor Behavior and Interactive Video. Curator 34 (2), 109-118.
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 17 Neuman, Delia. 1989. Computer based education for learning disabled students; teacher's perceptions and behaviors. Journal of Special Education Technology 9(3): 156-166. Niccolucci, Franco. 2002. Virtual Archaeology: An Introduction. In Virtual Archaeology: Proc. of the VAST 2000 Euroconference, Arezzo, Italy, 24-25 November 2000, ed. Franco Niccolucci. BAR International Series 1075, Oxford: Archaeopress. Osberg, Kimberley M. 1997. Constructivism in practice: the case for meaning-making in the virtual world. Ph.D Diss. University of Washington. Internet. Available from http://www.hitl.washington.edu/publications/r-97-47/osberg.rtf; accessed 18 August 2005. Panagiotakopoulos, Chris T. and George S. Ioannidis. 2002. Assessing children's understanding of basic time concepts through multimedia software. Computers & Education 38 (4): 331349. Peirson Jones, Jane. 1995. Communication and learning in Gallery 33: Evidence from a visitor study. In Museum, Media, Message, ed. Eilean Hooper-Greenhill: London: 260-275. Routledge. Pimentel, Ken and Kevin Teixeira. 1995. Virtual Reality: through the new looking glass. New York: McGraw-Hill. Pujol Tost, Laia. 2005. Interactivity in Virtual and Multimedia Environments: a Meeting Point for Education and ICT. In Archaeological Museums. In Virtual Reality at Work in the 21st Century. Impact on society. Proceedings of the 11th International Conference on Virtual Systems and Multimedia, Ghent, Belgium, 3-7 Oct. 2005, ed. Hal Thwaites: 37-52. Ghent: International Society on Virtual Systems and Multimedia. Reid, David J., Jianwei Zhang, and Qi Chen. 2003. Supporting scientific discovery learning in a simulation environment. Journal of Computer Assisted Learning 19 (1): 9-20. Ronen, M. and M. Eliahu. 1999. Simulation as a home learning environment - students' views. Journal of Computer Assisted Learning 15 (4): 258-268. Roussou, Maria. 2002. Virtual Heritage: From the Research Lab to the Broad Public. In Virtual Archaeology. Proceedings of the VAST 2000 Euroconference, Arezzo, Italy, 24-25 November 2000, ed. Franco Niccolucci: 93-100. BAR International Series 1075, Oxford: Archaeopress. Internet. Available from http://makebelieve.gr/mr/research/papers/VAST/VAST_00/mroussou_VAST00_press.pdf; accessed 25 October 2005. Roussou, Maria. 2004. Learning by doing and learning through play: An exploration of interactivity in virtual environments for children. ACM Computers in Entertainment 2 (1). Internet. Available from http://portal.acm.org/citation.cfm?id=973818; accessed 16 October 2005. Roussou, Maria and George Drettakis. 2003. Photorealism and Non-photorealism in Virtual Heritage Representation. In Virtual Reality, Archaeology and Intelligent Cultural Heritage (VAST 2003) and 1st Eurographics Workshop on Graphics and Cultural Heritage, 5-7 November 2003: 46-57. Brighton. Internet. Available from http://www.makebelieve.gr/mr/www/mr_publications.html; accessed 10 September 2005. Scanlon, E., C. Tosunoglu, A. Jones, P. Butcher, S. Ross, J. Greenberg, J. Taylor and P. Murphy. 1998. Learning with computers: experiences of evaluation. Computers & Education 30 (1/2): 9-14. Sharpe, Elen. 1983. Touch-Screen Computers - An Experimental Orientation Device at the National Museum of American History. Office of Public and Academic Programs, National Museum of American History, Smithsonian Institution.
18
M.ECONOMOU, L. PUJOL TOST
Shaw, Graham and Nigel Marlow. 1999. The role of student learning styles, gender, attitudes and perceptions on information and communication technology assisted learning. Computers & Education 33 (4): 223-234. Schroeder, Ralph. 1996. Possible worlds: the social dynamic of virtual reality technology. Boulder: Westview Press. Scott, S. D., R. L. Mandryk and K. M. Inkpen. 2003. Understanding children's collaborative interactions in shared environments. Journal of Computer Assisted Learning 19 (2): 220228. Sims, Rod. 1997. Interactivity: A forgotten art? Instructional Technology Research Online. Internet. Available from http://www2.gsu.edu/~wwwitr/docs/interact/; accessed 16 May 2005. Smith, J. K. 1990. Methods of Measuring Learning. In What research says about learning in science museums, ed. Beverly Serrell: 16-18. Washington, DC: Association of ScienceTechnology Centers. Spicer, John I. and J. Stratford. 2001. Student perceptions of a virtual field trip to replace a real field trip. Journal of Computer Assisted Learning 17 (4): 345-354. Staley, D.J. 2003. Computers, Visualization, and History: How New Technology Will Transform Our Understanding of the Past. In History, Humanities, and New Technologies. Armonk, NYQ: M.E. Sharpe, Inc. Steuer, John. 1995. Defining Virtual Reality: Dimensions Determining Telepresence. In Communication in the Age of Virtual Reality, eds. Frank Biocca and Mark R. Levy: 33-55. Hillsdale: Erlbaum Associates. Stone, Robert. 2005. Serious Gaming – Virtual Reality’s Saviour? In Virtual Reality at Work in the 21st Century. Impact on society. Proceedings of the 11th International Conference on Virtual Systems and Multimedia, Ghent, Belgium, 3-7 Oct. 2005), ed. H. Thwaites: 773784. Ghent: International Society on Virtual Systems and Multimedia. Tarr, Michael J. and William H. Warren. 2002. Virtual Reality in behavioral neuroscience and beyond. Nature 5(neuroscience supplement): 1089-1092. Trushell, John, A. Maitland and C. Burrell. 2003. Pupils' recall of an interactive storybook on CDROM. Journal of Computer Assisted Learning 19 (1): 80-89. Vance Wilson, E. 2000. Student characteristics and computer-mediated communication. Computers & Education 34 (2): 67-76. Vincent, John. 2001. The role of visually rich technology in facilitating children's writing. Journal of Computer Assisted Learning 17 (3): 242-250. vom Lehn, Dirk and Heath, Christian. 2003. Displacing the object: mobile technologies and interpretive resources. In ICHIM 2003 (International Cultural Heritage Informatics Meeting). Conf. Proceedings published on CD-ROM only, Paris, École du Louvre, 8-12 September 2003. Archives and Museum Informatics. Wheeler, Steve, S. J. Waite and C. Bromfield. 2002. Promoting creative thinking through the use of ICT. Journal of Computer Assisted Learning 18 (3): 367-378. Wood, David. 1999. Representing, learning and understanding. Computers & Education 33 (2): 83-90. Worts, Douglas. 1990. The Computer as Catalyst: Experiences at the Art Gallery of Ontario, ILVS Review 1 (2), 91-108.
EDUCATIONAL TOOL OR EXPENSIVE TOY? EVALUATING VR EVALUATION AND ITS RELEVANCE FOR VIRTUAL HERITAGE 19 Youngblut, Cristine. 1998. Educational uses of virtual reality technology. IDA Document D-2128. Institute for Defence Analyses. Alexandria, VA. Internet. Available at http://www.hitl.washington.edu/research/knowledge_base/virtual-worlds; accessed 5 September 2005. Yu, Fu-Yun, L. J. Chang, Y. H. Liu and T. W. Chan. 2002. Learning preferences towards computerised competitive modes. Journal of Computer Assisted Learning 18 (3): 241-250.
