Effect of Animated Characters and Adjustable Tags in ...

NICOGRAPH International 2013, pp. 35 - 39

Effect of Animated Characters and Adjustable Tags in AR Storytelling R.P.C. Janaka Rajapakse*

Yoshimasa Tokuyama**

Tainan National University of the Arts*, Tokyo Polytechnic University** [email protected], [email protected], Abstract Mobile computers, smart phones, tablets, and computer software applications have become ubiquitous in academia and daily life. Education institutes and teachers are expanding their teaching materials with computer vision based applications. Augmented reality (AR) applications are very famous and educators use it to produce interactive teaching materials, specially focused on digital storytelling projects. Unfortunately, current AR-driven applications and their contents are not sufficient for the development of storytelling skills. The most common criticism is that the 3D contents are less animated and controlled with predefined and fixed tags which consume considerable amount of time for changing tags as well as not help to translate their correct story ideas. The main objective of this work is to investigate the effect of animated characters and adjustable tags in AR storytelling. This paper presents a simple framework for making manually adjustable AR-tags to represent the different character poses and different states of characters which would help to develop the storytelling skills.

1. Introduction

promising technology for digital storytelling, currently only a few

Augmented Reality (AR) technologies and tools have been widely

applications exploit the potential of developing storytelling skills.

studied in numerous fields, including the game industry, advertising,

The most of AR storytelling applications can be immersed a user, but it

fashion, interior design, education and learning, medical and surgical

is not capable for the development of storytelling skills. The most

operations, product design and manufacturing, archeological

common criticism is that the 3D contents in the virtual worlds are less

restoration and museums, and countless other areas [1, 2]. This

animated and controlled with predefined and fixed tags which

technology allows computer created CG contents to be superimposed

consume considerable amount of time for changing tags as well as not

over real-life live video feeds. While most of AR applications have

help to translate their correct story ideas. Moreover, existing

been oriented to fields such as education or entertainment, mobile

applications change the role of storyteller to interactive presenter. The

computing and high-quality mobile display technologies have gained

pedagogy of storytelling is very different from interactive presentation

enormous progress. Nowadays, mobile computers, smart phones,

[15, 16]. In order to develop an AR-driven interactive pedagogical

tablets, smart books, and pads are often used in academia and daily life.

storytelling application it is important to investigate the weaknesses of

There are large numbers of mobile applications and services designed

the current AR applications. The main contribution of this pilot work

and optimized for edutainment. Within the academia mobile AR

is to examine the effect of animated characters and adjustable tags in

applications and computer vision based applications stand out and

AR storytelling. This paper focuses on simple framework for making

attract considerable attention [3, 4]. Moreover, commercial and

manually adjustable AR-tags to represent the different character poses

open-source AR application development libraries have been

and different states of characters which would help to develop the

introduced that enable a continuous increase of functionality and

storytelling tools.

portability [5, 6, 7, 8, 9, 10].

2. Background

Marker and tag based AR applications have been used to produce interactive teaching materials, specially focused on digital storytelling

2.1 AR Tags and 3D Contents

projects. The traditional workflow of digital storytelling framework focused on pre-production, production and post-production [11]. In

The most common AR applications published so far allow to the

such environment, it is difficult to increase the interactivity. AR can aid

user to interact 3D contents by using several different techniques such

learning and make the overall storytelling process much more

as AR desktop tag (marker)-based, AR desktop tag-less, AR freehand

interesting and interactive [12, 13, 14]. Although AR is still a very

tag-less, AR by mobile, AR by projection etc. Mainly, all technologies 35

NICOGRAPH International 2013, pp. 35 - 39

use tracking methods for live video feeds and control superimposed

deal a lot in expressing “What?, Who?, Where?, and When?” while

3D contents by common geometric transformations such as rotation,

students were trying to express their ideas in “How?” and “Why?”

scaling, and translation, either by tag-based techniques or by tag-less

variety. The problem is that how existing AR contents control the

techniques. The best known tag-based AR applications for storytelling

curiosity issues. Considering curiosity issues and handling rich AR

are based upon the space trend tracking of a graphical tag, usually a

contents, and turning them into an AR application is not an easy

geometric black&white symbol called “confidence” on which we can

process. What does it take to create a successful AR storytelling

hook in real time a two-dimensional or three-dimensional input. Some

application? There are several variables to consider: realistic 3D

AR applications can be controlled without a geometric marker through

contents, animating contents, adjustable tags; rich interactivity,

a process that recognize graphically a generic image or a portion of it,

platform; time, available resources, and target ordinance. Jonassen et al.

after which the tracking system will be able to recognize identity and

defined that the meaningful learning has to be active, intentional,

orientation. The freehand controlled tag-less AR applications that trace

cooperative, authentic, and constructive [20]. A successful AR

position, orientation and direction of the user look, using tracking

storytelling application clearly provides meaningful learning

systems and various movement sensors, associated to natural

environment as defined in [20].

interactive systems such as Microsoft’s Kinect, Wavi Xtion by

2.1 Storytelling and Interactive Media

PrimeSense with Asus3D, Nintendo’s Wii-mote and the Sony’s

How do computer games tell stories, and what does this mean for

PlayStation Move [18]. Recently RFID/NFC and QR codes have been

current digital storytelling? Games are things that are designed

associated in edutainment systems that allowing users to retrieve more

systematically, thoughtfully, artistically for the purpose of being fun.

information using their mobile phones [19]. Therefore, interactive

And the game designers work is not just a matter of pure engineering.

digital edutainment tools and projects are rich with opportunities for

There's a lot of engineering involved, such as a lot of algorithms and

assessment [20]. However, Taiwan elementary school has been proven

interactive techniques. But there's also an artistic experiential side of

to be a traditional and still conservative stage so that new technologies

game design that involves thinking about problems in a certain way.

arrival created somewhat resistance to the regular use of these

Therefore, game design has grown within a rich bed of interacting

interactive edutainment tools. Adopting new technology and finding

trends and traditions in interaction design and games, and that there are

budgets are challenges that inevitably will cause a change in the

already a number of potentially competing, and overlapping concepts.

management of both classrooms and schools. Naturally the experience

Most recently introduced academic term “Gamification” focuses the

says that any big change requires an adjustment phase but it is always

use of game design elements in non-game contexts [21].

very welcome cheap and easy tools for classroom use.

Our focus is storytelling, and expects to investigate on the various degrees to which different media and platforms support stories. Our point is not to settle the ontological status of games, but to focus on their storytelling functions—not to decide what proportion of gaming is storytelling, but to identify how games tell stories. However, looking for storytelling in gaming makes too much of too little. But there are many gaming discussions on narratology vs. ludology. Our background review also experienced that the games primarily as mechanisms for narrative delivery and those emphasizing gaming’s mechanical operations beyond stories. The storytelling application is not only an educational media as fancy content delivery mechanisms, but also a tool that helps kids design and develops their own ideas and

Figure 1: Kids were interacting AR contents in the survey sessions.

improving storytelling skills.

To understand the role of the 3D contents in an AR-driven digital

Creative learning tools such as Scratch [22], NetLogo [23],

storytelling application, it makes sense to examine the kid’s feedback

and LEGO [24], empower kids as self-motivated learners to program,

about the existing applications and their contents. We visited few

build, animate, and design their own creations. LEGO, for example,

elementary schools and did the pilot survey for acquisition of kid’s

would be a perfect tool for constructing an active brain. The purpose of

feedback on AR-contents (see Figure 1). In those surveys, while the

this research is to develop such tools for “storificatoin”, sharing them,

elementary school students were using few AR applications [12], and

and expanding the realm of digital storytelling. Initial step of this

we carefully investigated that how these tools had shaped student’s

research will proceed by developing a series of AR-driven storytelling

own ideas. Students engaged in these sessions not only had an

applications. Each of these developments will guide to invent a story

entertaining experience, but they also asked very important questions.

console and simple character control devices (watch-like) for

After analyzing their questions, we noticed that existing AR contents

interactive pedagogical storytelling. 36

NICOGRAPH International 2013, pp. 35 - 39

3. Effect of Animation

across the surface of a model. For example, in lion character, mane is

The first series of experiments was investigated the effect of

hair-like and longer than other furs. To control exactly where the

animation on AR-contents. The two groups of AR stimuli were

different fur grows, separated models were used (see Figure. 6). The

experimented. The first group of AR stimuli was consisting

several attributes maps were used to control fur parameters such as

non-animated 3D contents and characters, and the second group of

colors, length, baldness, direction, etc.

stimuli had animated contents. A total of fifty six elementary school kids (31 female and 25 male) between 6 to 13 years of age participated in the experiments. In the experiments, the participant’s task was to interact the provided (randomly) AR stimuli from each group (see the Figure 1) and chose the interesting ones. Over 95% of the subjects

Figure 3: Created 3D model of dog character (left), while painting skin

judged that the realistic animated AR-contents were interesting than

weights (middle) and while rigging with FK and IK (right).

the still contents.

In the modeling of human hair, dynamic hair curves in Maya were

3.1 Realistic characters

used to create dynamic motion. To modify and customize created stock

The based on the kid’s evaluation on AR contents, this framework

of characters for breeding many versions with different costumes and

has focused upon the creation of realistic animated AR contents for the

body types, Autodesk Pinocchio tool was used. Figure 4 depicts few

proposed character console. The character modeling and texture

customized characters with different face textures.

composition process were conducted using Autodesk Maya, Mudbox and Adobe Photoshop CS 5 software packages. In order to enhancing realism and visual impact of the virtual characters, UV-mapping techniques and material editing were concerned with real photographs and advanced material models. Figure 4: Customized characters with different facial features. The process of complex motion editing, blending and scaling were carried out by using Autodesk’s MotionBuilder software package. To create more complex body language for biped characters, the captured motion data (from Vicon bonita-10, eight camera MOCAP system) were used and mapped to the different characters as shown in Figure 5.

Figure 2: Creating 3D model & skeletons for skin binding (top), UV-mapping and textured version of character (bottom). Techniques of key-framing and Kinematics (IK/ FK) were used to make animation for peoples and animal characters. To make animated characters look real, anatomical structure guided joint hierarchies known as a skeletons, were created and placed inside the 3D models by using Maya software (Figure 2). Inverse kinematics rigs, technique of smooth binding of skin geometry and Maya muscle were used in order to create realistic deformation for character rigs. Different quadrupeds have varying models of walking and galloping as shown in Figure 3. To adding fur to the animals, polygonal models with properly mapped UV coordinates that lie within a 0 to 1 range in texture space were considered. For some animals, fur is not uniformly

Figure 5: Motion data (left) and mapped characters (middle & right). 37

NICOGRAPH International 2013, pp. 35 - 39

3.2 Multipose and Multistate

All AR stimuli were standalone applications and running on

How do multipose and multistate characters help to storytelling?

notebook PCs equipped with common USB webcams or intergrated

Naturally, multipose has been part of storytelling since early ages of

webcams that provide images at a resolution of 640*480 pixels, with a

handmade puppet shows. Expensive game controllers and gesture

frame rate of 30 fps. Experimental results were indicated that the

based interactive applications can control multi-camera views and

over 90% of participants preferred to use MLT controllers than

character poses [18], but less effort has been spent on multipose and

traditional blac&white makers. MLT control papers provided enough

multistate storification. However, typical AR applications don’t

visual information to easily select the characters and their poses. Over

facilitate such storification functions. The pose of the character, age,

82% of subjects were interested in using MLT-based watch like

and body proportions are very important character attributes for digital

wearable craft controller. However, it was difficult to control with a

storytelling (Figure 6). Our framework has considered providing

low-resolution commodity webcams. Our framework proposed more

adjustable AR tags for controlling multiposes and multistate of

suitable AR controllers for storytelling. As depicted in the Figure 8, multi-MLT tag cylinder is for representing character’s multiposes and

the characters.

multistate, and cube is for different characters. After testing in kid’s workshops, character cube has improved with manually adjustable folded faces for multiposes and multistate as shown in Figure 9.

Figure 8: Multipose & multistate cylinder (left), Character cube (right)

Figure 9: Character cube with folded faces for multipose & multistate.

Figure 6: Different poses obtained from skeleton and IK handling (top and middle), different body propositions in different ages (bottom).

4.1 Implementation

4. Effects of Adjustable AR Tags

This research project is envisaged not only to just investigation, but

Our second series of experiments addressed the effect of adjustable

also motivated to comprise the development of several AR storytelling

AR tags. In the experiments, participant’s task was to judge the most

applications for PC and mobile media. The first phase of the proposed

interesting and suitable AR content controlling method for storytelling.

framework was implemented with a D’Fusion AR authoring tools [25].

Each AR stimuli were controlled by tree methods as shown in Figure 7,

After modeling and confirming all scenes and animations in the Maya

the first by traditional marker in black square(Figure 7-a), the second

software, the D’Fusion exporter has used. It is working as a Maya

was advanced markerless tracking(MLT) by printed images (b), and

plug-in which translate Maya data into Ogre3D data [26]: geometry

the third was also based on MLT, but watch like a paper-made craft (c).

(.mesh), lights, bones (.skeleton), materials (.material), and scenes (.scene). When using the longer MOCAP data in the Maya, timeline has managed according to the character poses and split it pose wise without overlapping (for example clip1 [10-50] and clip2 [51-100]) before exporting for D’Fusion. To enable the MLT functionalities, AR scenarios has configured using D’Fusion studio computer vision tool

(a)

(b)

(c)

elements such as tracking scenario, camera calibration, camera capture.

Figure 7: Experimented AR-content controlling methods.

Basically, this process has composed with three steps: recognition, 38

NICOGRAPH International 2013, pp. 35 - 39

on Ubiquitous Virtual Reality(ISUVR’09) , pp. 7-10, 2009.

initialization, and tracking as shown in the Figure 10. Registering

[6] ARToolKit, A Software Library for Building AR Applications,

imported scene files with corresponding captured images were

http://www.hitl.washington.edu/artoolkit/

implemented with Lua scripting [27] and maintained the scene data

[7] FLARToolKit, ActionScript version of ARToolKit,

base.

http://www.artoolworks.com/products/web/flartoolkit-2/ [8] Layar, Layer Specializes in Mobile Augmented Reality, http://www.layar.com/ [9] Hoppala, Mobile Augmented Reality Platform, http://www.hoppala-agency.com/ [10] Vuforia, AR-based Application Development Environment, https://developer.vuforia.com/resources/sdk/android

Figure 10: Recognition (left), initializing and tracking (right)

[11] Digital Storytelling in the Classroom,

To tracking multiple objects, used few planer tracking at the same

http://edtechteacher.org/index.php/teaching-technology/presentation-

time using the same video capture. The standard 3D transformation

multimedia/digital-storytelling

parameters (position, orientation, scale) were not suitable for all

[12] Zooburst, Digital Storytelling Tool, http://www.zooburst.com/

multiple 3D objects. For restoring initial parameters according to the

[13] ARART, New Art Experience with Augmented Reality,

multiple object proportions, D’Fusion object editor was used. DFusion

http://arart.info/en/

@HomeDesktop Player was used to deploying standalone

[14] Toontastic, http://launchpadtoys.com/toontastic/

applications.

[15] J. C. Kuyvenhoven, In the Presence of Each Other: a Pedagogy of Storytelling,

5. Conclusion

PhD

thesis,

University

of

British

Columbia,

https://circle.ubc.ca/handle/2429/17168 [16] G. M. Deniston-Trochta, The Meaning of Storytelling as

This paper has presented the initial phases of ongoing research project

Pedagogy, Visual Arts Research, Vol. 29, No. 57, 2003.

to develop a framework for interactive pedagogical storytelling. The results were helpful to identify the problems and practical challenges in

[17] K. Matsuo, M. Hagiwara, Entertainment AR Aquarium, Journal

AR storytelling. The development of this pilot prototype as well as

of Art and Science, Vol. 10, No. 4, pp. 226-233, 2011.

initial field surveys has taken an important approach to introduce

[18] B. Lange, S. Rizzo, C-Y. Chang, E. A. Suma, and M. Bola,

advanced media technologies for effective digital storytelling. This

Markerless Full Body Tracking: Depth-Sensing Technology

paper investigated “AR storification” and introduced a simple method

within Virtual Environments, Proc. of Interservice/Industry

for making manually adjustable AR-tags to represent the different

Training, Simulation, and Education conference, paper no.

character poses and different states of the character. This may guides

11363, pp. 1-8, 2011.

and motivates kids to make their own story crafts. The current work is

[19] O. Haberman , A. Damal, R. Pellerin , U. Haberman, E.

implementing the mobile versions of all developed AR story

Gressier-Soudan, Exploring Contemporary Painting through Spatial

applications. Future work would develop a character console and

Annotations Using RFID Tags. Proc. of the 11th Int. Symposium on

simple sensor-driven AR content controller for pedagogical storytelling.

Virtual Reality, Archaeology and Intelligent Cultural Heritage, France, 2010.

References

[20] D. H. Jonassen, D. H., Howland, J., Moor, J., Marra, R. M. (2003).

[1] R. T. Azuma, A survey of Augmented Reality, In Presence:

Learning to Solve Problems with Technology: A Constructivist

Teleoperators and Virtual Environments, Vol. 6, No. 4, pp.

Perspective (2nd Edition), Prentice Hall, USA.

355-385, 1997.

[21] S. Deterding, D. Dixon, R. Khaled, L. Nacke, From Game Design

[2] D. J. Haniff, C. Baber, W. Edmondson, Categorizing

Elements to Gamefulness: defining “Gamification”, Proc. of the 15th

Augmented Reality Systems, International Journal of Three

International Academic MindTrek Conference: Envisioning Future

Dimensional Images, Vol. 14, No. 4, pp. 105-110, 2000.

Media Environments, pp. 9-15, 2011.

[3] T. Höllerer, S. Feiner, Mobile Augmented Reality. In Karimi, H. &

[22] Scratch, http://scratch.mit.edu/#

Hammad, A. (eds.) Telegeoinformatics: Location-based Computing

[23] NetLogo, http://ccl.northwestern.edu/netlogo/

and Services. London: Taylor & Francis Books Ltd., 2004.

[24] Lego, http://www.lego.com/en-us/

[4]

D.

Wagner,

History

of

Mobile

Augmented

[25] D’Fusion Studio, from Total Immersion,

Reality,

http://www.t-immersion.com/products/dfusion-suite/dfusion-studio

Communications, pp. 1-21, 2009

[26]Ogre3D. http://www.ogre3d.or/

[5] W. Wagner, D. Schmalstieg, History and Future of Tracking for

[27] Lua scripting language, http://www.lua.org/

Mobile Phone Augmented Reality, Proc. of International Symposium 39