Augmented Graphics for interactive storytelling on a mobile device Marco Marchesi, Bruno Ricc`o∗ University of Bologna CR Categories: H.5.1 [Information Interfaces and Presentation]: Multimedia Information Systems—Artificial, augmented, and virtual realities I.4.9 [Image Processing and Computer Vision]: Applications—; Keywords: Mixed Reality, Computer Vision, Mobile Devices, Interactive Storytelling
1
Introduction
Mobile devices are encountering increasing popularity as systems for interactive storytelling. Novels and comics can be enriched and made interactive by means of touchable interfaces and multimedia contents, so that the usual linear narration can be broken up in many paths and customized, giving rise to alternative developments. Furthermore, with Mixed Reality (MR) the traditional elements of storytelling, i.e. backgrounds, characters and objects, can be partly real and partly virtual, thus allowing new possibilities to renew stories any time they are told, according to the environment and the preference of the viewer. In this context, the present paper introduces the prototype of a new framework, called Augmented Graphics (AG), that exploits Computer Vision (CV) algorithms to recognize real objects that influence a fictional interactive story, as a pioneering example of Augmented Virtuality (AV).
2
Overview
Conventionally, in Augmented Reality (AR) a real environment is enriched by means of virtual elements. On the contrary, in Augmented Virtuality (AV) a virtual background can be “augmented” by introducing real objects and people that modify the virtual elements. The potential superimposition of real actors to a mixed environment has been extensively explored [Charles et al. 2004], and interfaces between virtuality and reality have been investigated [Koleva et al. 2000]. Moreover, Computer Vision algorithms have been applied to MR in gaming [Hammond 2008], where images acquired by a camera were analyzed to recognize objects able to influence the game. In Augmented Graphics a story is enhanced by the interaction of the user with the reality. Because of the leading role played by graphics and the interactive approach given to the storytelling, AG looks oriented to graphic novels and games. As for terminals, mobile devices are the most appropriate choice. Specifically, tablets and smartphones are ideal for enjoying a story augmented by real objects that are taken by the embedded camera, analyzed by image processing algorithms and then eventually placed in the virtual scene. In practice, AG storytelling works as follows: the graphic novel is divided into scenes, in each of which the virtual characters do some action that requires some objects, not immediately present on the scene. AG permits to organize each scene as a collection of graphics, sounds and instructions that set how the Computer Vision algorithms will be applied to analyze the scene and recognize the objects needed to continue the narration. AG works with shape ∗ e-mail:
[email protected],
[email protected]
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. SIGGRAPH Asia 2013, November 19 – 22, 2013, Hong Kong. 2013 Copyright held by the Owner/Author. ACM 978-1-4503-2511-0/13/11
recognition functions and feature detection algorithms, like ORB or SURF, taken from the OpenCV library. Once the story is written and ready to be “played”, the user can start exploring the real world, seeking the objects asked by the story’s characters to accomplish some action necessary to continue, choosing the right shapes, colors or images. Then, the outcome of a search influences the subsequent narration. For instance, in a fantasy story, the amount of time spent to find a door key in a room can influence the chances to survive to the attack of a monster. The working system has been successfully tested on a tablet (iPad 4th generation) that allows to process all the frames captured by the camera stream with a resolution of 1280x720 pixels up to 8-10 fps. For the demonstration of AG, a science-fiction graphic novel is presented, featuring 12 scenes where two characters have to survive to the pitfalls of a desolated land. Besides the particular case of storytelling in a graphic novel, the high interactivity allowed by AG, that encourages the exploration and the discovery of shapes and objects in the surrounding environment, can be used for different applications, such as, for instance, education, gaming and marketing. Future work for AG will be focused on the strength of the implemented algorithms that have to be tested in a variety of light conditions, with the objects recognizable in different scales, orientations and range of colors. We are considering methods to “stylize” any object according to the scene and then place it on stage. How it can be captured, processed with a reasonable delay, consistently displayed and interact with the virtual elements is still an open question.
3
Conclusions
The Augmented Graphics system has been presented. It allows to create interactive stories affected by the objects found in the real world, captured by the camera embedded in the mobile device and recognized thanks to the Computer Vision algorithms taken from OpenCV library. The presented system considerably improves the interaction of the user with a real environment. For this reason, besides entertainment and gaming, Augmented Graphics can also be used for educational purposes as well as for brand marketing.
References C HARLES , F., C AVAZZA , M., M EAD , S. J., M ARTIN , O., NANDI , A., AND M ARICHAL , X. 2004. Compelling experiences in mixed reality interactive storytelling. In Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology, ACM, New York, NY, USA, ACE ’04, 32–40. H AMMOND , B. 2008. A computer vision tangible user interface for mixed reality billiards. In Multimedia and Expo, 2008 IEEE International Conference on, 929–932. ¨ KOLEVA , B., S CHN ADELBACH , H., B ENFORD , S., AND G REEN HALGH , C. 2000. Traversable interfaces between real and virtual worlds. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, ACM, New York, NY, USA, CHI ’00, 233–240.
