2012 International Conference on Innovations in Information Technology (IIT)
Developing Mobile Mixed Reality Application Based on User Needs and Expectations Amandeep Dhir@, Thomas Olsson#, Said Elnaffar! @
#
Department of Computer Science and Engineering, School of Science, Aalto University, Finland Unit of Human-Centered Technology, Tampere University of Technology, Tampere-33720, Finland ! Faculty of Information Technology, UAE University, Al Ain, United Arab Emirates {
[email protected]}@{
[email protected]}#{
[email protected]}!
Abstract— Mixed reality refers to the unification and interaction of the digital and physical environments giving an immersive view of the environment to its users. The aim of our study is to investigate the needs and expectations of the potential users of a futuristic and a novel technology called Mobile Mixed Reality (MMR). Based on the users’ impressions and expectations, we developed MMR prototypes representing various scenarios and contexts of use. We observed that the most important factors that play a role when it comes to acceptability are usefulness, personalization, reliability, and relevance. We developed different prototypes that were tested in order to evaluate the prototype acceptability. The understanding provided by the study helped in designing early MMR prototypes. The most intrusive elements that should be supported by any MMR application were found to be usefulness (how useful is the information to any individual), personalization (personify the information with such questions as what, how and when to receive any information), reliability (accuracy and authenticity of the presented information), and relevance (utility and value of the information).
Figure 1: Overview of Reality –Virtuality Continuum [3] Therefore, the MR can be viewed as a continuum spectrum whose one end is augmented reality and the other end is augmented virtuality (see Figure 1) [4]. Augmented reality is a technology that enhances the world around us by overlaying important data, usually in real-time. A simple example would be a fighter pilot looking through his visor’s heads-up display at a view of the sky annotated with real-world information, such as the target, altitude and horizon data. The augmented virtuality, at the other end of the Mixed Reality, we find that new controllers from the video game makers such as Nintendo, Microsoft, and Sony enable players to interact with games by tracking real movements in 3D space, providing a more intuitive, natural and overall fun gaming experience.
Keywords-component - Mobile mixed reality, mobile devices, prototyping, user needs, user expectation, user research I.
INTRODUCTION
The ubiquitous nature of mobile devices, particularly smartphones and tablets, has transformed the basic three needs of human lives: food, shelter and clothing. Initially, mobile phones were introduced as a medium for communication, but now they are extended to the full spectrum of social networking, Web surfing, multimedia, global positioning, entertainment, and so on. Gartner Inc has stated that worldwide mobile device sale has reached 1.6 billion units with an increase of 72% in 2010 [1]. This growth has made mobile devices affordable and ubiquitous. This mass scale adoption of mobile devices has created connected spaces involving physical and digital environments that can support the concept of Mobile Mix Reality (MMR). This concept can be more easily explained by understanding the wide spectrum of Mixed Reality (MR) [2][3] where, in different degrees, the physical and digital environments interact and mix together.
978-1-4673-1101-4/12/$31.00 ©2012 IEEE
In this paper, we focus on the augmented reality [5] aspect of the MR but in the context of mobile computing, which is referred to Mobile Mixed Reality (MMR). MMR can provide rich, enjoyable and surprising experiences to its users. Unfortunately, this area did not get its due attention from researchers yet. The last decade, however, has witnessed an increasing interest in MMR services with researchers and service companies interested in investigating MMR. Due to the enormous market potential of MMR, several service
83
currently utilize a few MMR applications. Therefore, we designed the UCD process with the above concern in mind
companies have invested in this physical virtual collaboration through different business models [13, 14]. Some of the notable examples of MMR applications are social gaming, personalized shopping and different augmented events [15]. However, we believe that the business value of MMR can only be fully practically realized if actual user needs and expectations from MMR are understood and met. Currently, there is a lack of understanding the users' perspectives and requirements for MR. It is becoming even more important because the MR domain is at such a level of maturity that end-user applications can start mushrooming. Understanding user expectations is important not only for designing usable products but also for designing products based on novel and futuristic technologies like MMR. The aim of this study is to contribute to the understanding of the potential users’ needs and expectations of MMR. This includes determining what should be presented in an MMR application, what kind of experience any potential MMR user is looking for, and what kind of contexts it could support. II.
RELATED WORK
Currently, research on MR technologies is limited only to introducing enabling technologies that include algorithms, development platforms [6,7], display and output devices [8]. Despite that an extensive research has been carried on MR technologies, we believe that the research community shows shortcoming when it comes to showing potential scenarios and use cases. Current research is also deficient in its overall understanding of what could be presented to a user and what a user can expect from MMR. Furthermore, usability studies are mostly limited to the evaluation of implemented prototypes or working applications [9]. The evaluation includes cognitive studies and other usability aspects such as task performance and clarity of information presentation [10, 11]. There are some studies that have been conducted lately on studying user needs and expectations. For example Olsson et al. [5] conducted user studies in order to gather user needs and expectations using focus discussions that involve different use cases and scenarios on MMR. Vaittinen et al. [12] performed diary studies in order to collect user needs in relation to MMR services. However, both of these studies are limited only to understanding the user needs and expectations related to MMR. So despite these studies, the full potential of MR has not been explored yet. This paper, instead, aims to build early prototypes based on the user research. III.
Figure 2: ISO 13407 UCD process model [16] A. Study Goals In our study, we have three ultimate goals that aim at: 1. 2. 3.
Understanding user needs and expectations for MMR services. Designing and implementing MMR prototypes based on the gleaned user expectations. Prototyping acceptability tests in order to reiterate the design based on user feedback.
B. A Summary of our UCD Process We used the triangulation research principle [17] in the data gathering process, i.e., both qualitative and quantitative research methods were practiced. The UCD process included four focus group discussions (see Figure 4) [18] involving questionnaires [19], scenarios [20] and potential use cases for gaining a comprehensive view of the various users' needs and expectations. The focus group scenarios included tourism, shopping, entertainment and restaurant use cases. Scenarios were carefully designed in order to clearly introduce the MMR concept to the test participants, as they had no prior experience with the MMR before. The gathered data were consolidated using affinity diagramming [21], and the main themes from user research were drawn. Figure 3 presents an overview of the affinity diagrams drawn as a result of our affinity diagram process. Our final MMR prototype designs were inspired by this user research. In the end, we performed prototype acceptability testing [22] by iterating the prototype design twice. The prototype testing was done in order to iterate the developed concepts based on user feedback. Prototype testing fostered our comprehension of users’ attitudes after interacting with the developed MMR prototypes.
USER-CENTERED DESIGN PROCESS
The potential use cases and application of the MMR technology range from safety, leisure, emergency and entertainment. Therefore, it is vital to understand the context of usage. User Centered Design (UCD) methodology, defined by ISO 13407 (see Figure 2) [16], can be used in order to understand the requirements of the users, their tasks and the context of using this technology. We used the UCD methodology for gathering user needs and expectations in the context of the MMR services. Applying the UCD process to the MMR is challenging given that users in their daily routines
84
Figure 4: Focus group arrangement is shown where focus discussions were performed
Figure 3: About one-tenth of the affinity diagrams are shown C. Test Participants Totally 15 participants (6 male and 9 females) participated in the user research phase of the UCD process. The participants’ ages ranged from 18 to 42 years with the mean = 28 years and Standard Deviation (SD) = 4.4. Ten out of 15 test participants were technology-oriented or lead users of any new service or product, while the remaining five were laggards or slow adopters with no previous experience in programming. Prototype acceptability iterations were performed twice in the UCD process. Two young technology-oriented people and three slow adopters in there 30's attended the first iteration while five young technology-oriented people participated in the second iteration. IV.
“If MMR offers a suggestion such as which place is worth watching and which is not, like good or bad, then it is something cool (group session 1, female 24)” “MMR is really useful like while travelling to places like China and Japan. Come on! How many languages can you learn after all? And I am especially clueless when it comes to Chinese (group session 3, female 33)” B. Demand for Timely Infomration Participants appreciated the timely available information provided by MMR technologies and recognized how useful it is to enhancing their existing knowledge about their current environment and surroundings. For example information on real time traffic, weather in any location, newspaper headlines, offers and discounts, rating and recommendations for decision-making, and information on current and future happenings/events in their surroundings. It is not surprising to find such information on any ordinary browser of a smartphone, but the mobile user has to type and search for such information. However, in the case of MMR, all such information is seamlessly presented to the user. The majority of the test participants also expressed the wish that the MMR service be an instant, personalized, trustworthy, effortless and usable. Additionally, some participants emphasized that icons and symbols used for displaying MR contents should be encouraged.
RESULTS FROM USER RESEARCH
After building the affinity diagram, we analyzed the user data from different perspectives such as concept, design, implementation and business. Eventually, we summarized our findings under seven broad themes, as explained next. A. Benefits of using MMR Participants had a lot of discussion on the issue of how MMR can benefit any individual. Discussion on benefits varied from individual and social to business and travelling. Participants agreed that MMR could help in quick decision-making, reduce cognitive load, and that one can also get information without physically visiting places. The majority of the participants agreed that MMR services made them more confident travelers, solved the problem of the language barrier, and also worked as a social application that enabled them to experience social well being. Other benefits agreed on are reduction of boredom, the sense of security and safety provided by 24/7 assistants, and that they were handier than traditional media such as newspapers. Moreover, participants also acknowledged that information dissemination through MMR is fast such that any business should invest in this kind of technology.
Here are some sample quotes from the participants: “MMR is handy as it is always with me in my cell phone, I always keep my mobile with me so I can get suggestions and assistance anytime (group session 4, female 42)” “MMR can show if a restaurant is already full so that I don't waste my time (group session 3, female 44)” C. Demand for Personalization MMR application should be customizable and personable, a feature desired by the majority of our participants. Information
85
However, some participants were willing to share unidentifiable personal information, using nicknames.
overloading can confuse users, and irrelevant information can cause frustration. Test participants reported that customization can include, but not limited to, searches, categories and the presentation of information in general. A few quotes are:
“If my comments are shown to, like, a 25-year female, I don't mind sharing that kind of information…. (Group session 3, female 31)”
“Ratings and recommendations are generally fake, so it is important to know the background of the people who gave those ratings and recommendations. Like, if I want to buy some shoes for my mom, then I would like to see the ratings and suggestions of Finnish females older than 50 (group session 3, female 31)”
G. Unreliability associated with MMR Several participants raised concerns that MMR suffers from an inherent risk of showing out-of-date, incorrect or misleading information. Participants also agreed that the above three concerns are the hurdles for the mass adoption of MMR services. Other less common concerns raised by the test participants were the risk of running out of battery power, behaving socially awkwardly while using MMR on the streets, and the sudden failure of a MMR application in some critical situation.
D. Interaction with MMR Interaction with MMR refers to the different innovative MMR application design concepts and other use cases brought up and discussed by participants during the focus group discussions. Discussions were mainly focused on what kind of interaction is supported by MMR and what they would like to see in MMR. Popular use cases were based on increasing interaction and making users more competent professionally. Less common ideas were related to solving the problem of boredom and making MMR socially competent.
“This kind of application will consume lots of power. If I am in a foreign country having a different language and suddenly MMR stops working then? (Group session 1, male 26)” “When I use this app and it appears like I am clicking pictures; I may freak people around me due to my actions (group session 2, male 26)”
We here provide some samples from the reported experiences by the participants:
V.
“During conferences and social gathering it is very difficult to recall each and everyone with their name, designation, and current work, so it would be nice to see some kind of information tag representing their name and destination. Others can see this kind of information through MMR (group session 2, female 25)”
DEVELOPED PROTOTYPES
We created three prototypes inspired by user research findings. The first two prototypes are semi-functional, while the third is a proof of concept. After the prototype development, we performed two iterations of prototype acceptability test. In the first iteration concepts were drawn on paper and participants were solicited for feedback. However, the second iteration consisted of two prepared test tasks for each of the three prototypes. Both prototype iterations were carried out in a laboratory and participants were asked to think aloud and evaluate the usefulness of each feature supported by the prototypes.
“Introducing some kind of playfulness in MMR such as having a virtual pet (group session 1, male 26)” E. Hindrance to MMR Adoption The majority of the participants agreed that paying for MMR services and the no immediate need to utilize them by local residents in the community (e.g., when the local residents are familiar with their home town and they do not have the language barrier) are among the reasons that discourage the adoption of some MMR applications. Less common opinions were that MMR can affect personal life by disturbing one’s peace of mind, making the user spend more by luring them with more offers and discounts, and the difficulty of carrying around MMR especially for women who may already be carrying heavy shopping bags.
A. Prototype I: The MR Street The first prototype was based on the idea of presenting the MR content in a street view from a particular location as shown in Figure 5. We displayed different digital icons such as a football (representing a stadium), a fork and knife (symbolizing a restaurant) and a wine glass (denoting a bar). We used Google Map API; mark up languages such as HTML and XML, and the Samsung Nexus S browser for creating and hosting this prototype. Through this prototype we aim to give users an immersed view of tagging MR content at their desired locations on the street view. During the prototype acceptability test, participants were asked to attach the available digital icons to a location of their choice on the street map view. Participants could also save the location tagging for future reference.
“I can use this app and pay if I am in a foreign country where I have a language barrier, but in my home country? I doubt (group session 2, female 26)” “I can pay for MMR services as it is an effortless and instant service (group session 3, female 33)” F. Privacy Concerns, again! The majority of the participants were concerned about disclosing and sharing private information, namely, sex, age, name, and email and location while using MMR applications.
86
Figure 5: The MR Street Prototype
Figure 6: An MMR application on a phone
As a result of the acceptability test, we did several changes to our prototype. Moving icons from one location to another was enabled, and the icons and their colors for symbolizing the MR content were changed based on the obtained feedback.
The majority of the participants appreciated the prototype during prototype testing. Participants proposed adding a "Settings" button to personalize the displayed information and suggested associating a particular kind of color-coding with each particular activity.
B. Prototype II: An MMR Application This prototype is meant to show the functionality of MMR on a mobile phone. To that end, the Nokia Flowella [23] prototyping tool was used for creating this prototype with 16 different mock-ups covering different associated functions such as presenting and adding MR content, searching content, tagging information, adding friends and filtering information on certain access levels. During prototype acceptability test, the participants were given an N900 mobile phone that runs the MMR prototype application. Participants were instructed to think aloud while navigating different views and enquiring about the clarity of displayed symbols, icons and displayed text. After trying the prototype for two iterations, we learnt that connecting and chatting with friends was an essential feature for this kind of an application. Tool tips were also added to the icons, similar to what we did to first prototype. Eventually, and as a result of accommodating these suggestions, participants were satisfied with the final prototype, which is depicted in Figure 6.
VI.
DISCUSSION
Understanding user needs and expectations is essential for designing products based on futuristic and novel technology concepts such as MMR. During the user research we found that user needs were pragmatic in nature including relevance, personalization, reliability and usefulness. Users want MMR services to be instant, handy, effortless, useful, meaningful, relevant and usable. Users recognized reducing cognitive load as the main benefit of any MMR service and that an MMR application should have social features of other online social media such as sharing, connecting and chatting with friends. Keeping these needs in mind, the MR Street and the MMR N900 prototypes were created. Users appreciate technology concepts that enhance their existing knowledge of their own surroundings. For example, the MR surrounding fulfills this particular user needs by providing them with up-to-date information about their environment. The results of our study should help us design mobile applications based on futuristic technologies such as MMR by factoring in the genuine user needs and expectations.
C. Prototype III: MR Surroundings The MR surroundings prototype was visualized by creating a short video animation with text annotation (tags) as shown in Figure 7. The MR surroundings prototype presents the MR view of the interior and exterior of buildings. Different types of information are presented in the form of MR information tags such as weather forecast, real time traffic, outside temperature, news headlines and important events happening in the users’ area. As presented earlier, the demand for information and the benefits gained from the MMR applications were the two main motivations for using the MMR technology. Keeping the user research findings in mind, we have re-designed this prototype such that it augments users’ knowledge. It also reduces users’ cognitive overhead as the MR tags enrich the physical space with lots of information (see Figure 7).
VII. CONCLUSION AND FUTURE WORK The MMR provides rich, pleasing, positive and enjoyable experience to its users. This paper has described the study methodology, results and insights on the UCD of MMR applications. The UCD is essential for the development of technical applications that are based on novel and futuristic technologies such as MMR as it help elicit needs and expectations, which can be crucial factors to the success of the adoption of a particular technology. Our study provides useful insights to the application developers and programmers who are interested in exploiting the opportunities provided by MMR. In our study, we found that most MMR augmenting elements render the tested applications more useful, personalized, reliable and relevant.
87
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
Figure 7: The MR Surroundings Prototype [14]
Future work will include more rigorous UCD experiment using multi method research process. The existing concepts will be refined and new ones will be added. Based on these concepts, real mobile phone applications will be developed and the usability testing will be conducted.
[15]
[16]
ACKNOWLEDGMENT The work has been supported by Data communication Software Lab located at the Department of Computer Science and Engineering, Aalto School of Science, Finland. We also thank all the participants who attended the different phases of our experiments.
[17]
[18]
REFERENCES [1]
[2]
[3]
[4]
[5]
[19]
Gartner Says Worldwide Mobile Device Sales to End Users Reached 1.6 Billion Units in 2010; Smartphone Sales Grew 72 Percent in 2010 http://www.gartner.com/it/page.jsp?id=1543014 Bach, C., and Scapin, D. L. 2004 Obstacles and Perspectives for Evaluating Mixed Reality Systems Usability. Acte du Workshop MIXER,IUI-CADUI'04 Milgram, P., and Kishino, A. F. A Taxonomy of Mixed Reality visual Displays. vol. E77-D(12), IEICE Transactions on Information and Systems, pp. 1321-1329. Azuma, R. T. 1997 A Survey of Augmented Reality. Presence: Tele- operators and Virtual Environments. vol. 6, No. 4, pp. 355-385. Thomas Olsson, Pirita Ihamäki, Else Lagerstam, Leena VentäOlkkonen, and Kaisa Väänänen-Vainio-Mattila. 2009. User expectations for mobile mixed reality services: an initial user study. In European Conference on Cognitive Ergonomics, ECCE '09, Article 19 , 9 pages.
[20]
[21]
[22] [23]
88
Azuma, R. T. 1997 A Survey of Augmented Reality. Presence: Tele- operators and Virtual Environments. vol. 6, No. 4, pp. 355-385. Colin Ware and Ravin Balakrishnan. 1994. Reaching for objects in VR displays: lag and frame rate. ACM Trans. Comput.-Hum. Interact. 1, 4, 331-356. Milgram, P., and Kishino, A. F. A taxonomy of mixed reality visual displays. vol. E77-D(12), IEICE Transactions on Information and Systems, pp. 1321-1329 Dünser, A., Grasset, R., and Billinghurst, M. A survey of evaluation techniques used in augmented reality studies. In ACM SIG-GRAPH ASIA 2008 courses (New York, USA, 2008), SIGGRAPH Asia 08, ACM, 27p Drascic D & Milgram P .1996. Perceptual Issues in Augmented Reality. In: Proc. of the Stereoscopic Displays and Virtual Reality Systems III, SPIE, pp 123-134 K. Stanney. 1995. Realizing the full potential of virtual reality: human factors issues that could stand in the way. In Proceedings of the Virtual Reality Annual International Symposium (VRAIS'95) (VRAIS '95). IEEE Computer Society, Washington, DC, USA, 28-34. Vaittinen, T., Karkkainen, T., and Olsson, T. A diary study on annotating locations with mixed reality information. In Proceedings of the 9th International Conference on Mobile and Ubiquitous Multimedia (New York, USA, 2010), MUM '10, ACM, pp. 21:1-21:10 Gurd, J. Can augmented reality be a commercial success for ecommerce?, March 24 2010. http://econsultancy.com/uk/blog/5637-can-augmented-realitybe-a-commercial-success-for-e-commerce Reilly, T. O., and Battelle, J. Web squared: Web 2.0 five years on. Technical Report, O'Reilly Media, Inc., 2009. van Kleef, N., Noltes, J., and van der Spoel, S. Success factors for Augmented Reality Business Models. Study Tour Pixel 2010 - University of Twente, July 2010. http://www.interactief.utwente.nl/studiereis/pixel/files/indepth/KleefSpoelNoltes. pdf ISO/IEC. 13407 Human-Centred Design Processes for Interactive Sys- tems, 1999, Switzerland International Organisation for Standardiza-tion(ISO). Jick, T. D. Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly vol. 24, no. 4, Qualitative Methodology (December, 1979), pp. 602611. Kitzinger, J. The methodology of focus groups: the importance of interaction between research participants. Sociology of Health & Illness vol. 16, no. 1 (1994), pp. 103-121. Flynn, B. B., Sakakibara, S., Schroeder, R. G., Bates, K. A., and Flynn, E. J. Empirical research methods in operations management. Journal of Operations Management vol. 9, no. 2 (1990), pp. 250-284. B ødker, S. Scenarios in User-Centered Design - Setting the Stage for Re ection and Action In Proceedings of the Thirty-Second Annual Hawaii International Conference on System Sciences - Volume 3 (Washington, DC, USA, 1999), IEEE Computer Society, pp. 3053-3064. Beyer, H., and Holtzblatt, K. Contextual Design. Defining Customer Systems. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. pp. 472, 1998. J., N. Heuristic Evaluation, Usability Inspection Methods. Wiley & Sons Inc., New York, USA., 1994, pp. 25-62. Flowella, Nokia rapid prototyping tool, http://www.developer.nokia.com/Resources/Tools_and_downloa ds/Other/Flowella/
