User evaluations on virtual museums (VMs) projects are planned because of the ... evaluations, especially in the case of the âVirtual Museum of the Tiber Valleyâ ...
AN INTEGRATED METHODOLOGICAL APPROACH TO EVALUATE VIRTUAL MUSEUMS IN REAL MUSEUM CONTEXTS Alfonsina Pagano1, Eva Pietroni1, Caterina Poli2 CNR ITABC, Rome, ITALY Università della Tuscia,Viterbo, ITALY 1
2
Abstract User evaluations on virtual museums (VMs) projects are planned because of the necessity to conduct test-beds and user experience surveys to have feedbacks and improve criticisms in the interactive software applications. From a methodological point of view, no specific direction has been taken both in humanities or computer sciences fields, but several and different strategies are employed whenever we deal with VMs, given the diversity and complexity of each of them, the target, the context of use and so on. CNR ITABC has progressively developed a methodological approach for the user experience evaluation, focusing its attention on the pedagogical aspects and cognitive affordances. In the latest CNR evaluations, especially in the case of the “Virtual Museum of the Tiber Valley” and “Livia’s Villa Reloaded”, we come up with solid considerations: what is relevant here is the centrality of the end-user who need to be the main protagonist of any cultural public transmission and thus of any investigation programme. Keywords: On site Virtual Museums, User experience survey, evaluation methodology, quantitative and qualitative analyses, cultural transmission, education.
1
INTRODUCTION
User evaluations on virtual museums (VMs) projects are planned because of the necessity to conduct test-beds and user experience (UE) surveys to have feedbacks and improve criticisms in the interactive digital applications The social communication paradigms are evolving rapidly shaping a society increasingly accustomed to spectacular audio-visual media. The expectations, as well, and the awareness of the public attending museums are evolving too. It is therefore a priority to test the impact, at many levels, that virtual applications in museums produce on the audience: the role and the kind of storytelling, the adopted language, the clarity of contents, the emotional aspects of communication, the interaction, the rhythms of narration and animations, the “embodiment”, the soundscape, the usability and clearness of the graphic user interface, definitively the design of the whole experience in the “performative space”. Likewise, communication strategy, storytelling and pedagogical affordances have been recently included in the studies useful for developers and experts in Digital Cultural Heritage to understand more about audiences, their behaviors and their relationship with such projects. Differently from cinema and game industries, the domain of virtual museums has not a long nor systematic tradition of cognitive studies, user profiling and testbed analyses. Still today we need to improve, step by step, an evaluation methodology starting from concrete case studies to which we apply experimental evaluation procedures, improving along the time. These latter are very helpful to improve also the quality and the impact of future virtual museums. The Italian National Research Council, with the support of national and international partners, has progressively developed a methodological approach for the user experience evaluation, focusing its attention on the pedagogical aspects and cognitive affordances. The massive work carried on under the V - M u s t n e t w o r k [www.v-must.net] a n d o t h e r e u r o p e a n p r o j e c t s a s E t r u s c a n n i n g [www.regolinigalassi.wordpress.com] , o r e x h i b i t i o n s l i k e D i g i t a l H e r i t a g e E x p o [www.digitalheritage2015.org], Archeovirtual [www.archeovirtual.it], or again single project as the Virtual Museum of the Tiber Valley [www.museovirtualevalletevere.it], have allowed to test several evaluation tools and theoretical procedures. Major concerns regard how people interact with multimedia systems, in general which are the social and psychological repercussions, what are the main benefits in terms of memorability and utility of contents, which is the level of learnability and the overall satisfaction [2], [3], [17], [19]. This is what this contribute is about. What is relevant here is the centrality of the end-user who needs to be the main protagonist of any cultural public transmission and thus of any investigation
1
programme [7]. The same concept of VM is not developable if we do not have in mind the final target, the type of interaction we want to provide the public, and the overall experience we want them to live.
2
STATE OF THE ART: AUDIENCE IS THE HEART OF THE VIRTUAL MUSEUM
Audience evaluation is the process of examining information about users and it helps developers, experts and interested researchers to adapt the “message” to be addressed in order to be well understood by the final receivers. A typical communication model is composed by (a) the message (what information is transferred); (b) the source or, better, the sender, who has specific characteristics and who shapes (c) the form of the message (language, style, duration…); (d) the channel (the medium used), and (e) the receiver. In the mid ‘50ies, the researcher Prof. Schramm suggested to examine also the impact that a message has on the end-users, raising this concept by the assumption that communication can be seen as a process of information transmission ruled by three semiotic levels: syntactic (formal properties of signs and symbols - words); pragmatic (relations between the user and the words pronounced); and semantic (relations between signs and symbols and what they represent in a social community). Therefore, communication is conceived as social interaction where at least two interacting agents share a “common” set of signs and semiotic rules. This fruitful situation generates the experience, and it is peculiar of the specific interacting agents or, better, end-users (Fig.1). In case of virtual museums (VMs) projects [16], where cultural heritage, digital domain and technological solutions are strictly blended together, we deal with a wide and complex set of Fig. 1 Schema of the information: multi-layered contents, audio-visual media and communication model. interaction interfaces. It is fundamental that the information we want to transmit is properly “encoded”, balanced according to the selected media and “calibrated” on the final target; beside, the user interface needs to be clear and intuitive, otherwise the communication fails. Audience is thus at the heart of many processes: (a) it influences and practically shapes up the VM’s content creation, (b) it guides operators during the design and assessment of the evaluation programme, and (c) it leads considerations or reasonings upon VM of the next future. But which are the main elements that are affected by targetization in the design of a VM and also in its evaluation? ● For sure the storytelling. Contents needs to be characterized and shaped according to the age and cultural background of the end-users, taking advantage of visual metaphors, easiness of language, linearity of storyline, and dramatization through an evocative soundscape. ● The Graphic User Interface (G.U.I.). The interface layout can be adaptive, according to the user's profile and provenience: one of the future challenges is to make exactly this in VM projects. In any case, GUI needs to refer to principles of usability and clearness, avoiding redundancy and complexity. ● The interaction modes. Usually young users are not frightened by dynamic interaction and alternation among them in the active role; they consider the design of the virtual environment of major importance than the narration itself; generally, technological devices are not considered a barrier by young people, differently from adults, who instead prefer stories upon a prompted interactivity. ● The time of fruition. Time of fruition can be defined either by the interest of individuals to deep the experience or by their ability to use the interactive system. Also the number of users that are allowed to play contemporaneously with the VM installation may affect the timing (i.e. single or collective experience?). ● The same evaluative strategy. According to the final target of the evaluation session, everything must be set up and managed: from the construction of the evaluative tools (i.e. questionnaires, interviews…), the methodology to be used, the multimedia support with which conduct the evaluation, to the operator’s behavior towards users.
2
3
METHODOLOGICAL APPROACHES AND EFFECTIVENESS OF UEs
An experience emerges from the integration of perception, action, motivation, and cognition into an inseparable, meaningful whole [13]. Once generated, the experience becomes an immaterial, personal story which accompanies individuals throughout their life. But the experience is formed by different aspects that must all be taken in consideration when planning and conducting user experience evaluation: for sure, the user’s characteristics, the features of the multimedia system and the contextual factors. These characterize the human-computer interaction highlighting how is the (a) perception of instrumental and non-instrumental qualities of the system, and (b) the emotional user reactions. The former are qualities useful to be investigated like utility of the device and its usability, while the latter refer to the visual aesthetic, the symbolic value that the content of such device can address. The user’s emotional feedback, instead, represent the behavior toward the system, the cognitive approach to its use, and the psychological mechanisms involved. In favorable conditions, user evaluations (UEs) can be planned in three different moments [15]:
● before studying the user experience on a specific VM installation. This is important in order to study the general context of fruition; how the visitors approach the spaces and the objects exposed; which are the preferred points along the visit path and which are neglected; what are the most common “steps” of the experience; how users face themselves with multimedia equipment. The Quality Poll went in this direction when planned in 2013 and 2014 by V-MUST.NET in Rome and in Amsterdam [16]. This evaluation is very helpful to define the concept and the design of the VM before finalizing it, and also to define its arrangement along the paths of visit. ● During the museum visit, once the specific VM installation has been set up. This is helpful to analyze
the practicability of certain tools included into the visit path (usability), their attractiveness and effectiveness compared to traditional panels and other informative elements (educational and pedagogical potential). This has been done for Livia’s Villa Reloaded and Virtual Museum of Tiber Valley projects by CNR ITABC here presented. ● After the cultural visit or VM application experience, to evaluate the user's feedback respect to the acquired knowledge.
A similar evaluative occasion was provided by Digital Heritage Expo, editions 2013 [2] and 2015, and Archeovirtual exhibition, respectively in 2011 [17], [19] and 2012 [2] by VMUST.NET. This is also the case of the Customer Satisfaction evaluation planned in 2014 for “Keys to Rome” traveling exhibition (www.keys2rome.eu) in Sarajevo and Rome locations [18]. The evaluative strategy can involve tools according to the goals of the UE. We can conduct either quantitative or qualitative studies, or both, taking advantage of several methods (likert scales, observations, questionnaires, interviews, focus group…). Quantitative research is based on the numerical data that is possible to collect. Borg and Gall (1996) describe it as the kind of inquiry “that is grounded in the assumption that features of the social environment constitute reality by collecting numerical data on observable behaviors of samples and by subjecting these data to statistical analysis”. An example could be defining which is the medium time spent in front of a museum showcase, that raises up statistical data. Whereas, qualitative research is the "inquiry that is grounded in the assumption that individuals construct social reality in the form of meanings and interpretations, and these constructions tend to be transitory and situational. The dominant methodology is to discover these meanings and interpretations by studying cases by cases and subjecting the resulting data to analytical induction” [11]. Again here an example is analyzing the subjecting feelings of museum visitors when in front of a masterpiece, through open questions and informal speech. Indeed the effectiveness of user evaluations resides mostly on the level of inflexibility of the methods and strategies used, on the final target (their collaboration, predisposition to certain studies and psychological attitude), on the content of the VM (its articulation, complexity of the message) and, finally, on the contextual factors which may influence negatively the whole study (noise, crowd, volume of the system, accessibility). In the following paragraphs two VMs will be taken in consideration: the Virtual Museum of the Tiber Valley and Livia’s Villa Reloaded. They have been developed by CNR ITABC in 2012-2016 following a similar scientific and communicative approach. In fact both of them are virtual reality installations adopting gesture-based interaction (using the Kinect sensor); also the user experience evaluation has been carried on basing on a similar methodological procedures. Both case studies have been the occasion to test and evolve a specific evaluation protocol and consequently to derive precious suggestions useful to understand strength and weakness of communicative choices and improve the efficacy of future VM projects.
3
4
THE EVALUATIONS: THE PROGRAMME AND THE CONDUCTION
4.1 The Tiber Valley Virtual Museum: Science, Art and Technology toward an “expressive unity” The Virtual Museum of the Tiber Valley hosted in the National Etruscan Museum of Villa Giulia, [3], [5], [4] is an integrated system of knowledge, promotion and communication of the cultural landscape of the Tiber river middle valley, in the area north of Rome, (about 60 x 40 km)(https://vimeo.com/129867454). From the collection of existing data, and the digital documentation and topographic survey, the landscape of the middle Tiber valley has been 3D reconstructed, in its geological and anthropogenic evolution through various chronological phases, together with some archaeological sites (Lucus Feroniae, Volusii’s Villa) and contexts of historical and natural interest (Soratte Mountain and the Tevere-Farfa Natural Reserve). The project consists of a virtual reality application using gesture-based interaction and visualized on three 65” aligned screens (Fig.2). The user lives a multi-sensorial experience interacting, through his/her body movements, 4 virtual scenarios, related to the different aspects of the Tiber Valley. Shifting from one avatar to another one, he can: 1) swim like a fish, under the water of the Tiber 2) flying, like a bird, over the territory 3) visiting the Volusii’s Villa rebuilt in Augustan age and 4) walking inside the Roman settlement of Lucus Feroniae rebuilt in the I-II century AD. The investigation was conducted on 117 observed museum visitors of different ages, provenience and technological attitudes. Both active users (46), directly playing with the system, and passive users (40), those who observed the experience led by the active user, were interviewed respecting the social dynamics of fruition - from a single user, to a couple or a group of users.
Fig. 2 Virtual Museum of the Tiber Valley set up at National Etruscan Museum, Rome. Fig.3 Livia's Villa Reloaded set up at the Roman National Museum, Rome. 4.2 Livia’s Villa Reloaded: a multi-perspective story Livia's Villa is an important suburban roman villa, belonged to Livia’s (Augustus' wife) family since the late republican age [12] [6]. It was built on a hill near Prima Porta, at the IX mile of ancient via Flaminia. Differently from the Tiber Valley Virtual Museum, the Livia’s Villa Reloaded application is located in a dark and silent room of the museum, dedicated only to this virtual experience and separated from the main paths of visit along which the public moves. There is an interactive area in front of a large screen where one person at a time can enter, and an area with 25 seats for the attending public, in the back (Fig.3). Starting from digital acquisition on the field, both the archaeological site and the 3D reconstruction of how this context could be in the past, have been modeled in 3D. So people can interact with two main scenarios: 1) flying over the remains and enjoying some educational movies regarding the archaeological site, or 2) walking through the corridors and the rooms of the villa reconstructed during Augustan age, meeting the ancient characters and dialoguing with them while they are attending their daily activities. Again here the gesture-based VR application is intended to be placed to involve visitors in a multisensorial and emotional experience, where perceptive aspects and storytelling play a fundamental roles. For the evaluation 125 museum visitors were interviewed, 63 of them were active users and 62 passive ones; even tough 104 were globally observed. Both applications introduce a new approach, beyond the traditional paradigms of VR environments. In many cases the user is not completely free to move, but he is brought along a predefined path and his
4
freedom in the exploration is limited to stop and look around. This choice derives from the need to keep alive the emotional involvement raised by the storytelling, avoiding the user gets lost or distracted by spatial obstacles. This need has brought to an interesting contamination among different languages and techniques: VR, cinema,virtual set, theatre, augmented reality, and video game. 4.3 Goals For both projects, the goals of the user experience evaluations, made on the general public of the two museums, were to test (a) the attractiveness and (b) the usability of the systems, (c) user satisfaction and (d) educational potential of the contents, while observing also the consistency of individual’s participation once in front of the technology. Evaluations were useful also to understand the (e) efficiency and accessibility of systems included in the museum environment. 4.4 Strategy The methodology used to evaluate the visitors required an innovative combination of techniques: the innovation obviously resides in the choice and match of such specific techniques. The so called multipartitioned analysis saw three different investigative tools working together and overlapping information requested to final users, so to have a better overview and verification of the collected data. These are observations, driven-scenarios and questionnaires, integrated in a complete and comprehensive programme. Their combination was indeed useful to gauge the sincerity of the users’ responses, their awareness and the emotional condition with which they approached the experience, their ability to learn from the system and to verify the different aspects of interaction between the user and the system. The procedure for the user’s evaluation was so organized: when individual entered the application room, the operator observed his behavior to check the visibility and attractiveness of the application in respect to the typical museum setting, by using his observation schema. At this moment, according to the user’s reaction to the system, he was classified as active user (directly interacting) or passive user (spectator). In the case of an active user, once approached the application, the operator firstly observed, his behavior without conditioning it in any way, and after some minutes proposed the driven-scenario, asking the user to perform specific tasks and analyzing the user's memorability, the interaction between him and the system, the time of fruition and the ability to access stories. At the end of the experience, again the operator asked the user (active and passive) to fill in a questionnaire template, to verify the level of involvement and interest, the efficacy of the adopted language and the usefulness of the system in terms of cultural transmission. The data was collected using a paper-based modality which allowed adding comments and opinions aside. Both projects were surveyed during the summer of 2015 during different days and hours. In this way it was possible to analyze and register a well representative sample of visitors. Promotional days were also organized as in Etruscan Museum of Villa Giulia as in the National Roman Museum, in which the public was invited to visit and try the application. Differently from many evaluation experiences managed by other research group, applied to very small samples of users, we strongly believe that a reasonable number of visitors is needed to carry on a reliable evaluation, especially when the goal is to elaborate statistical results on complex dynamics of humancomputer interaction, cognitive impact and educational potential. 4.5 Techniques Observations - Observation, made by the operator, is an investigative technique used to study the general behavior and attitudes of visitors towards the application and the context of fruition. The operator observes the users since when they arrive until the end of the experience; he follows the user’s different actions and reports them in his template form, without being noticed and remaining almost “invisible”. Only in cases where the user asks for help or he is clearly interested in the system, the operator is allowed to intervene. Information are collected on a paper form with a sequence of topics to be filled in. The relevance of such tool is that it could be considered a sort of “control panel”. For the Tiber Valley Museum such form was composed by two sections: one for all users (active and passive) and another for active ones (Fig.4). The same went for Livia’s Villa Reloaded, even if the referring form had a different layout. The form was organized in tasks to be observed containing open questions, multiple-choice answers and blank spaces to write down comments. Few examples are “Where does the visitor go when he/she enters
5
the room of the application?”, “Does the visitor see the poster explaining the project on the wall?”, “Does the visitor understand that there is an interactive area in front of the screens?”. Each form was identified by an ID number corresponding to the user that was playing with the application. In the Tiber Valley case information to be investigated was displayed on a table: in this way every column corresponded to the observation of a single visitor, while the line reported the general result of that specific topic for a group of visitors. For Livia’s Villa case, instead, this work was made after the data collection, on an excel file. At the end of the form there was a section for free comments and observations made by the operator regarding the dynamics of participation and relationship with the other users. In case of an active user, the operator took accurate measure of the length of time of user’s experience, using a stopwatch. The language employed in formulating the questions needed to be simple and Fig. 4 Observation template used by operator to investigate on concise, so to allow operator a quick reporting. Data processing was made end users.Example of the Tiber Valley Virtual Museum. through an excel file; a problem occurred for the free comments, in which the operator reported annotations about the observation made. In fact, in this case, it was necessary to first categorize the free comments and then place them in accordance with the evaluative procedure. Driven-scenario - The driven-scenario is an instrument that implies two kinds of techniques: observation and direct interview. The visitor is asked to solve specific tasks and express his impressions or doubts by the thinking-out-loud method. The operator observes and then asks some questions directly to the user. The behavior of the operator here is crucial for the success of the technique: he need to follow a fixed and predetermined plan. He has to maintain a calm face and also not suggesting with the body, otherwise he could encourage uncertainty or false indications. When the user expresses doubts, the operator must remain passive, except in cases he is asked specific questions. In the case of the Tiber Valley Museum, four scenarios were presented, thus our driven-scenarios were four too. For example, in the Underwater scene the tasks were: 1) Try to understand what you can do and how to move and 2)Try to swim and get to the fragment of collapsed wall (Fig. 5). Similar topics were investigated, both in the direct questions and in the observation phases so to measure the user’s level of sincerity. For Livia’s Villa Reloaded, instead, the main VR scenarios are two, so we presented only two drivenscenarios, following the same template structure and operator’s rules. In both cases, language in driven-scenario was clear and accessible to users of all ages. Everything was structured to obtain simplicity out of the feedback, but at the same time a deep explanation from end users about their experience. The driven-scenario was proposed to the users following these criteria of selection: (a) when the visitor was between 6 and 13 years old; (b) when the user started to interact but was not very interested, to stimulate his motivation and concentration; (c) in case of groups and families, as mean for cooperation and participation. The only problem encountered was related to the technique of “thinking-out-loud” because the majority of users remained silent, concentrating on the task but not commenting. This behavior could have derived from user’s personal character, i.e. shyness. Questionnaire - The questionnaire is a template of questions asking the user to analyze and evaluate the experience just lived. It is provided to users at the end of their experience, and filled out autonomously, without any intervention by the operator. On each questionnaire filled in, the operator has to write the ID number of the user, as done in the observation module, so to cross data in the successive analyses.
6
For the Tiber Valley Museum and Livia’s Villa Reloaded, it was organized in two versions, one for active and one for passive users. The questionnaire was presented as a structured list of questions, giving a lot of attention to the type of language, the typology of the questions and to the logical succession of the themes discussed. It was eligible to gain information about usability of the system, educational potential and appreciation of the system, using both quantitative and qualitative questions. Some example are “Did you understand the function of the colored circles on the ground?”, “Where are the two men in the scene?”, “ Would you be interested in visiting the real places?”. Specifically about the questions made, they were (a) closed questions like Yes/No/I don’t know, True/False and singlechoice; (b) relevance scale, where user needed to assign a number of preference among a series of items; (c) open questions, where the user can answer with his own Fig. 5 Driven -scenario template used by operator to words; (d) exercises, to test the level of investigate on end users. Example of the Tiber Valley memorability and reasoning (i.e. associating Virtual Museum. sentences and images; images recognition, etc.) (Fig.6). Moreover, during the questions’ formulation, it was important to respect the following criteria: - possible answers to questions (a) need to be of the same length; - questions must be formulated not suggesting the correct answer; - questions must be clear and unequivocal (the choice of language and words is fundamental); - questions must not be too complex, otherwise the user will skip them; - questions must not be too many (15 questions is already a high number); - the questionnaire should alternate text and visual questions, so to be more appealing for users; - avoid to group the most difficult questions at the end of the questionnaire. In general, once user approaches the questionnaire, he needs to feel comfortable with it: with few sentences, the importance and necessity of the user’s contribution to the research is underlined. Cases in which the user refuses to fill in the template are rare and usually happen when he spends less than five minutes in the experience. By the way, in our cases, the importance of collecting demographic and personal data of each user was noticed too: information like provenience, age, job, familiarity with technological devices and so on, were useful in order to associate specific results to specific typologies of users. Some difficulties during the data collection and analysis emerged too: (a) the questions referring to the educational potential were not fully compiled, due to low interaction times; (b) the “relevance scale”, failed given that user did not do it. Some example are “Which of the three aspects did you like the most? Indicate 1,2,3 in the box provided...”; (c) open questions were difficult to interpret due to unclear handwriting or the use of a single word insufficient to explain a concept.
Fig. 6 Questionnaire template of the Livia's Villa Reloaded.
7
Fig. 7 Challenge section of Livia's Villa Reloaded questionnaire.
5
DISCUSSION ON THE PROPOSED STRATEGY
About our choice to use both qualitative and quantitative research methods, as noted by Sechrest and Sidani [10], [1], only in the social and behavioral sciences that combination is highlighted. There are more similarities between quantitative and qualitative perspectives than differences, so the two can be efficiently combined to help the work of the user experience researcher - as it was in this contribution. First and foremost, both involve the use of observations to address research questions; secondly, they often attempt to triangulate data [14]; they indeed allow the use of analytical techniques that are designed to obtain the maximal meaning from data collected [9]. Moreover, both quantitative and qualitative inquiries utilize a great group of techniques to verify their data: the former incorporates control procedures on excel Fig. 8 Operator during the evaluation activity. files and graphics to better analyze the results, while the latter uses methods for assessing the credibility of what users are explaining, reformulating the direct questions more than once, and introducing false items to test user’s attention. The decision to divide the public in two categories relies upon the will of authors to verify the effective users’ interest both when they interacted with the system and when they observed the performance. Respecting this categorization, the different techniques, such as observation and questionnaire, allowed us to gather data and clearly analyze them, also to better understand the difference between single or heterogeneous group. The use of observations for both applications has proved to be an important mean to study the public: it allowed the operator to observe in deep several users, quickly and precisely, even in case of groups and couples. Observations also permitted monitoring and managing the flow of users during the research (Fig.8). Strength points of observation form are indeed that it allowed the operator to (a) manage the collection of the data in an efficient and orderly way, thanks to the use of the user number ID, that permits the process of evaluation of every single user to be accessed and followed; then, to (b) analyze and extrapolate a first interpretation of the data: reading the observation table, in the vertical line you know the experience and behavior of the individual visitors, while reading horizontally you analyze the answers and categorize the behavior of the users on each single topic, regarding the attractiveness of the system and the dynamics of participation. About the questionnaire, in the case of Livia’s Villa evaluation a strategy error was done: in the questionnaire we included 37 questions (probably too many) and the most demanding ones, related to the educational potential, a very important topic for us, were grouped in the last part. Thus we obtained a large percentage of evasion, probably incentivized by the tiredness of the user.
8
Regarding the data processing, the operator focused the attention on the transcription of information gathered. In particular, they constantly checked the data and compared them with single users to verify the accuracy of the information acquired. The same practice was used for the Excel files. Data was processed without the use of software given the difficulty to process information of such complexity and to compare information coming from different methods of analysis. In the end, the location of the VM project is also crucial when envisioning the final intense effect on the end-users: the not intimate atmosphere, the overcrowded visitors’ circulation along the paths, and other constraints need to be deeply investigated given that they can badly affect the user experience, especially if the contents hierarchy and narration have been conceived for a deep “immersion”. For Livia’s Villa such requested atmosphere was successful while in the case of the Tiber Valley Museum, this influences the users’ experience and the related evaluation.
6
CONCLUSIONS
There are not “standardized” rules and ready-made solutions in the domain of Vms; nevertheless there a r e best practices. Communication requires a multidisciplinary approach and a continuous creative process that poses always new challenges and a broad spectrum of possibilities. By observing the public reaction, once the VM has been installed in the real museum, we can understand the efficacy of our creation and we can improve our approach. For this reason it is recommended the VMs are evaluated inside the context of fruition for which they have been conceived, especially if the goal is to evaluate their educational potential (memorability, utility, etc), attractiveness and involvement, on a consistent number of visitors. We have seen how a determined but flexible methodology as the multi-partitioned analysis can boost the user evaluation domain. In this contribute we come up with solid considerations: (a) the relevance of the combination of quantitative and qualitative tools to have more material to compare with; (b) the importance of observations as preliminary mean to study the public and categorize them; (c) the practicability of submitting driven-scenarios to analyze user’s memorability and control over the system; (d) the good intuition of dividing the public between active and passive users to better understand the fruition dynamics, also within heterogeneous groups; (e) the importance of collecting demographic and personal data of users (provenience, age, job, familiarity with technological devices) in order to associate specific feedback to specific typologies of users. During the evaluation of the user experience in the museum, the role of (f) the operator is indeed fundamental even if difficult to perform: no emotions need to interfere with the judgments about the user experience; the time of usage makes the difference when analyzing the user’s behavior, so operator needs to be precise; no suggestions or compliance need to be undertaken in driven-scenarios so to not influence the people’s interaction. Language (g) is another problematic issue to care about: it must be customized on the final target (age, background, etc), the support of evaluation’s delivery, the context of submission and the time at disposal; questions and “exercises” in the evaluation schedule have to be carefully planned so to not influence the public feedback. An increasing line of difficulty in the contents needs to be built, useful to understand user’s growing for understanding. All these considerations are part of a bigger apparatus of best practices we constantly update, starting from the past CNR evaluations. Much more can be done in this field, diversifying the typologies of surveys and improving the relative tools and strategies. For sure the multipartitioned analysis, tested up to now - made of observation, questionnaire and driven-scenario - proved to be very functional and efficient as evaluative method, granting an overview on user’s behavior in complex conditions of fruition.
ACKNOWLEDGEMENTS Major thanks go to Arcus Spa, the Regional Direction of the Ministry of Cultural Heritage and the Soprintendenza all'Etruria Meridionale, the colleagues of FBK Fundation, the Museum of the River in Nazzano, and all the team of CNR ITABC. For the evaluation, special mention goes to the partnership between the Tuscia University and CNR for the stagaire who conducted the evaluative programme, Doc. Caterina Poli.
REFERENCES
9
[1] A. J. Onwuegbuzie, N. L. Leech, “On Becoming a Pragmatic Researcher: The Importance of Combining Quantitative and Qualitative Research Methodologies”. In International Journal Of Social Research Methodology, Vol. 8 , Issue 5, 2005. [2] B. Gockel, J. Eriksson, H. Graf, A. Pagano, S. Pescarin, “VMUXE, An Approach to User Experience Evaluation for Virtual Museums”. In Proceedings of The HCI International 2013, 21 - 26 July 2013, Las Vegas, Nevada, USA, Ed. Springer, Heidelberg. [3] E. Pietroni, A. Pagano, C. Poli, “Tiber Valley Virtual Museum: user experience evaluation in the National Etruscan Museum of Villa Giulia”, in Proceedings of 24th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision WSCG 2016, Pilzen, Czech Republic, May 30 – June 3, 2016, ed. By Vaclav Skala, University of West Bohemia, Czech Republic, UNION Agency. [4] E. Pietroni, A. Palombini, H. A. Arnoldus, M. Di Ioia, V. Sanna, “Tiber Valley Virtual Museum: 3D landscape reconstruction in the Orientalising period, North of Rome. A methodological approach proposal”, in Proceedings of Digital Heritage 2013, Vol. II, IEEE, pp. 223-331, 2013. [5] E. Pietroni, D. Ferdani, A. Palombini, M. Forlani, C. Rufa, “Lucus Feroniae and Tiber Valley Virtual Museum: from documentation and 3D reconstruction, up to a novel approach in storytelling, combining virtual reality, theatrical and cinematographic rules, gesture-based interaction and "augmented" perception of the archaeological context”. In Proceedings of 43rd Computer Applications and Quantitative Methods in Archaeology Conference CAA, Siena, 2015. [6] E. Pietroni, M. Forlani, C. Rufa, “Livia's Villa Reloaded: An example of re-use and update of a preexisting Virtual Museum, following a novel approach in storytelling inside virtual reality environments”, In Proceedings of Digital Heritage 2015, IEEE. [7] F. Varela, E. Thompson, E. Rosch, “The Embodied Mind. Cognitive Science and Human Experience”, MIT Press, Cambridge, 1991. [8] H. Graf, J. Keil, A. Pagano, S. Pescarin, “A contextualized educational museum experience connecting objects, places and themes through mobile virtual museums”. In Proceedings of 2015 Digital Heritage, Vol. 1, pp. 337-340), IEEE, 2015. [9] L. C. Dzurec, J. L. Abraham, “The nature of inquiry: Linking quantitative and qualitative research”. In Advances in Nursing Science, 1993, Vol. 16, pp.73–79. [10] L. Sechrest, S. Sidani, “Quantitative and qualitative methods: Is there an alternative?”, In Evaluation and Program Planning, Vol. 18, pp. 77–87, 1995. [11] M. D. Gall, W. R. Borg, J. P. Gall, “Educational Research: An Introduction” (6th ed.). White Plains, NY: Longman, 1996. [12] M. Forte, “La Villa di Livia, un percorso di archeologia virtuale”, ed. Erma di Bretschneider, Roma 2008. [13] M. Hassenzahl, S. Diefenbach, A. Göritz, “Needs, affect, and interactive products – Facets of user experience”. In Interacting with Computers, Volume 22, Issue 5, Pages 353–362, September 2010. [14] N. K. Denzin, “The research act: A theoretical introduction to sociological methods”. New York: Praeger, 1978. [15] S. Pescarin et al., “Del. 7.1 Virtual Museum Quality Labels”, In V-Must.net deliverables’ collection, Ed. 2015. [16] S. Pescarin et al.,“Keys to Rome. Roman Culture, Virtual Museums”, CNR ITABC eds., 2014. [ 1 7 ] S. Pescarin, A. Pagano, M. Wallergard, W. Hupperetz, C. Ray, “Evaluating virtual museums: Archeovirtual case study”. In Archaeology in the Digital Era, vol. 74, p. 12, 2012. [18] S. Pescarin, “Museums and virtual museums in Europe: reaching expectations”. In SCIRES-ITSCIentific RESearch and Information Technology, 4(1), pp. 131-140, 2014. [19] S. Pescarin, A. Pagano, M. Wallergard, W. Hupperetz, C. Ray, “Archeovirtual 2011: An evaluation approach to virtual museums”, in Proceedings of 18th International Conference on Virtual Systems and Multimedia, IEEE, pp.25-32, 2012.
10
