Developing practical models for teaching Motion Capture Gregory Bennett AUT University
[email protected]
Abstract Motion capture technology is increasingly being used across a range of digital moving image practice, from 3D animation, digital visual effects and gaming, to digitally augmented live performance and dance. Keeping abreast of developing and cutting edge moving image production technologies is often a constant challenge in digital moving image-related teaching programmes, as is the requirement to create curriculum that delivers both a strong foundational technical skill-base, but which is also addresses the meaningful application of the technology to a range of communicative and/or aesthetic principles and outcomes. In addition, there is the importance of sustaining a robust teachingresearch nexus within an academic context. This paper presents a range of preliminary pedagogical and research issues which have arisen at the early stages of developing teaching modules around the use of low cost, entry level motion capture technology at the School of Art and Design, AUT University, New Zealand. The motion capture system being utilised is Optitrack’s economical eight-camera optical motion capture package. This initiative represents a ‘first step’ in the integration of motion capture within an existing digital moving image programme and was conceived to address of a number of core pedagogical and research aims around 3D animation, performance and motion capture. The move to address motion capture in curriculum at AUT University was a result of several factors. The increasing utilisation of motion capture in the New Zealand animation and visual effects industry has meant that there is a need for graduates to have some developed skills in this area. Motion capture is also a developing field in academic research with regard to its use in the entertainment industries, including cinema, gaming and live performance. In addition an opportunity has arisen to develop some key industry partnerships to build a high-end motion capture facility at AUT University. A range of research projects being undertaken by postgraduate students and staff are being used to develop a series of motion capture teaching modules. Experience at a motion capture ‘bootcamp’ at Deakin University in Melbourne, Australia is also being drawn on. Our pedagogical aims around motion capture have two areas of principle focus at this stage. The first is the issue of character animation: the use of motion capture as an aid for teaching animated performance and exploring the relationship between classical animation principles and motion captured movement. The second is concerned with developing director and performer skill sets – how to direct and perform effectively with motion capture for shifting project and performance mode requirements. Within these particular fields of focus a number of issues arise: questions of motion capture versus classical animation technique, stylised or exaggerated versus naturalistic performance modes, motion captured movement applied to Copyright is held by the author / owner(s). SIGGRAPH Asia 2009, Yokohama, Japan, December 16–19, 2009. ISBN 978-1-60558-858-2/09/0012
Andrew Denton AUT University
[email protected]
stylised character models and/or photo-realistic models, the translation of ‘live’ to digital movement, and the digital processing of performance. The paper will present an overview of the range of projects, and their varied applications of motion capture technology, and the related teaching, curriculum development and research considerations arising at this initial stage. Keywords: motion capture, performance, animation, curriculum development
principles
of
Pedagogical issues 1 Character animation Motion capture technology presents some potentially effective ways to deliver curriculum related to principles of character animation and animated character performance. Questions as to the quality or kind of movement required for particular characters in particular projects have naturally arisen as a result of the injection of the possibility of motion capture use in the 3D animation production pipeline. This in turn has created a valuable focus on the issue of character animation and appropriate modes of performance, providing a balance to students’ sometimes overriding concern with details of a character’s rendered appearance at the expense of issues of performance and animation technique. Debates concerning motion capture’s status as ‘animation’ and the quality of captured ‘live’ motion versus ‘crafted’ classical animation style have been common at the conceptual stage of the animation productions of our students, echoing ongoing debates within the animation industry itself (which have ranged from essentialist questions: “Yes, but is it animation?” [Deitch 2008], to the characterisation of motion capture as the “Devil’s rotoscope” [DeGraf and Yilmaz 1999]). We also observed that there was initial student perception of being able to use motion capture purely as a labour-saving substitute for traditional animation. This gave way quickly to practical realisations that motion capture data required both clean-up and further key-framed adjustments (including hand and facial animation from scratch), and also that there were a range of aesthetic factors to consider concerning marrying different forms of motion to different styles of modelled and rendered characters. Thus decisions about character design, shading and texturing in relation to what kind of motion would be used to animate these characters became important in the conceptual and pre-production stages. We have identified that for the students so far seven exemplars have emerged as paradigms for the use of motion capture as manifested in a range of what might be considered as developing emblematic examples of the application of motion capture in 3D moving image up to now:
1.
2.
3.
4.
5.
6.
7.
Motion capture used with photorealistic human characters where the ‘Uncanny Valley’ [Mori 1970] phenomenon is strongly perceptible and is considered to undermine spectator enjoyment. Examples include: Final Fantasy: The Spirits Within (2001), Sakaguchi, H., and The Polar Express (2004), Zemekis, R. Motion capture used with photorealistic human characters where the ‘Uncanny Valley’ phenomenon is not perceivably present, or less noticeable as to not interfere with spectator enjoyment. Examples include: Beowulf (2007), Zemekis, R., and The Curious Case of Benjamin Button (2008), Fincher, D. Motion capture used with human characters which are not photo-realistically rendered. ‘Toon shading’-type rendering would seem to mitigate the ‘Uncanny Valley effect’ in these cases. Examples include: Appleseed (aka Appurushîdo) (2004), Aramake, S., and Vexille (aka Bekushiru: 2077 Nihon sakoku) (2007), Sori, F. Motion capture used with more stylised cartoon-like characters. Examples inlude: Monster House (2006), Kenan, G., and Jane and the Dragon (TV series 2005 6). Motion capture used for particularly complex movement and/or with large numbers of characters, for example dance, crowd scenes, or stunts. Examples include: Happy Feet (2006), Miller, G., The Lord of the Rings trilogy (2001 – 2003), Jackson, P., Spiderman (2002), Raimi, S. Motion capture used with a photorealistic character that is seamlessly integrated with live action performers. Examples include The Lord of the Rings trilogy (2001 – 2003), Jackson, P., and King Kong (2005), Jackson, P. Motion capture used in an experimental and/or interactive performance context, including application to non-figurative moving imagery. For example: Ghostcatching (1999) Kaiser, P. and Eshkar, S.
2.
3.
4.
5.
2 Motion capture and modes of performance So while motion capture has often been cited as a useful time and/or labour-saving technology, questions of its aesthetic appropriateness remain paramount. Central to this question is the type or mode of performance suitable for a particular character and how motion capture might be useful in realising this. A sample here of five 3D animation projects currently undertaken by our post-graduate students illustrates the range of the utilisations of motion capture and some attendant issues relating to character and performance: 1.
A black comedy featuring human characters with a stylised design approach. Performances will be motion captured using actors, facial movement will be keyframe animated. Intention is to combine naturalistic movement with stylised/exaggerated character design. Questions as to whether captured performances will work in a naturalistic mode, or motion capture actors will have to be directed towards a more exaggerated acting style to match the exaggerated character design
and comedic tone of the narrative, or will the need for exaggeration be met by key-frame animation combined with motion capture. Monster House (2006) and The Incredibles (2004) are key influences. A science fiction short featuring photo-realistic human characters. Performances will be motion captured using actors, facial movement will be key-frame animated. Although motion captured movement is seen as a good match for photo-realistically modelled and textured characters, questions as to whether animated characters will suffer from the ‘Uncanny Valley’ syndrome, and to what extent the true complexity of movement and performance is transferred from live performer to digital model. The student is also experimenting with dynamic effects in hair, skin and clothes to further enhance the ‘reality’ effect. A short 3D animation featuring two characters in conversation debating the virtues of motion capture versus traditional animation. One character will be motion captured and one character traditionally animated. An intentionally self-reflective project which intends to highlight the aesthetic differences between motion captured and traditionally animated character movement by their juxtaposition in the same ontological space: a literalisation of the motion capture debate. Questions as to whether these qualitative motion differences will be obvious enough to the audience – will these differences have to be foregrounded through some kind of exaggeration? Also will character design be influenced or affected by the type of movement/animation employed? A short 3D animation with two characters – one with a conventional 3D animation character rig, the other rigged for exaggerated squash and stretch animation. A self-reflexive piece examining character relationships through differential animation performance modes. Motion capture will be utilised to assist with blocking out of performances and identifying key poses. An experimental motion captured dance performance featuring a virtual camera whose movement is driven by the choreographic motion of the dancer.
All of these sample projects illustrate a range of applications of motion capture, both as production pipeline tool, and as aesthetic choice based on the individual creative and communicative requirements of a particular project. For many of these projects questions still remain as to the degree of the role of motion capture at different stages of the pipeline.
3 Directing and performing for motion capture Motion capture effectively introduces a ‘third party’ into the relationship between animator and character. While the animator has traditionally been seen as an “actor with a pencil”, moving image production which employs motion capture has seen the emergence of motion capture actors and the requirement for a still developing skill set around motion capture performance and direction. Animation students who may have previously relied upon themselves to act and embody their characters in video reference
footage, or simply through the use of a mirror, are facing the challenges of communicating with actors and of acquiring directing skills to work with performers and in many cases elicit subtle variations of performance. In addition this is a performance which is further filtered through the digital sampling of the actor’s motion only. This is then applied to a synthetic character which might be of quite different appearance. Dealing with issues of the digital translation of performance from actor to animated character require various levels observation and judgment, and the ability to gauge how a particular live action performance mode will translate appropriately, and according to the requirements of a particular animated character’s personality and actions. Some students have identified that there will be a potential need to gather alternative ‘takes’ of varying performance styles or ‘sizes’ of performance. For an actor experienced in film technique the absence of an identifiable camera to ‘play’ to might impact on the scale of performance. For example a closeup camera would require an actor to underplay gesture as opposed to a wide shot. The motion capture studio might encourage more theatrical-scaled projection from the actor which may be inappropriate for a particular shot. (In response to this particular issue one student has decided to include a motion captured moving ‘camera’ during his capture session, both to give the performer a point of reference, and to generate an ‘authentic’ documentary-style movement to the camera in the final rendered scenes.) Another issue is the fact that motion capture actors are ‘performing’ animated characters, which are sometimes stylised or exaggerated versions of the ‘real’, which also may require adjustments in terms of the ‘naturalism’ of the performance. One particular phenomenon we have observed so far is that raw motion data bears the distinctive imprint of a performer’s signature movement ‘presence’. Idiosyncratic details of attitude, posture, form, and action such as walking and running are present in quite recognisable form in the array of moving motion data points of the raw data. This has challenged assumptions about issues of ‘generic’ movement capture and foregrounded issues of appropriate casting and performance development.
4 Motion capture and 3D animation pipeline So far we have observed that the introduction of motion capture has impacted in two primary ways on the 3D animation teaching environment. Firstly, it has created the possibility to enhance the 3D animation production pipeline: as a useful pre-production aid, and to be in many cases a complement to, rather than replacement of, ‘traditional’ animation process and techniques. Secondly, it has engendered a deeper engagement with issues of character motion, movement and performance, including a renewed examination of classical animation principles in light of the distinct aesthetic qualities of digitised motion data. The so-called ‘twelve principles of animation,’ developed by the Disney studio from the 1930s and detailed in published form in Ollie Johnston and Frank Thomas’ 1981 book The Illusion of Life: Disney Animation, still form the foundation for instruction in creating persuasive animated character performance. The importance of these fundamental principles to 3D computer animation was emphasised by John Lasseter in his 1987 Siggraph paper Principles of Traditional Animation applied to 3D Computer Animation, and additions to these principles have
also been also been proposed (Isaac Kerlow for example outlines an extra five principles for computer animation: [Kerlow 2004]). While these principles provide a set of key guidelines for the production of convincing and engaging animated movement and performance, the observation and analysis of ‘real-life’ motion has always been an important tool for animators. Indeed the development of classical animation technique owed much to the observation and analysis of live action, whether in the flesh, or recorded on film [Lasseter 1987]. The use of video reference of the animator acting out his characters is also a recommended means of developing character performance [Cantor and Valencia 2004, Sullivan et al. 2008, Roberts 2007]. It can be an aid for identifying and blocking key poses in pose-to-pose animation, and also a way for the animator to physically experience and ‘feel’ the performance itself thereby being more effectively able to ‘transmit’ this into more convincing animated motion. It can also be useful for observing and discovering previously unanticipated expressive details to enhance an animated performance. Motion capture provides an opportunity to access performance reference material as digital motion data and to apply and preview it directly on a character rig. Whereas video reference footage is limited to one angle of view (depending on how many cameras are used) in flat 2D space, the ability to examine character movement in ‘navigatable’ 3D form allows for potentially a more comprehensive assessment of motion, both as a reference aid for key-framing pose-to-pose animation, but also as a means of considering performance based on the capture itself. Performance data can also be adjusted through keyframing, adding to the raw data to refine details of movement appropriately. A performance can also be re-captured and adjusted or ‘improved’ based on observation and assessment of the applied data. Motion capture can also be an efficient way of creating an animatic or pre-visualisation - an entire character performance can be captured and applied relatively quickly and then used to assist with developing a visual narrative in 3D animatic form. In one case a student is having trouble pre-visualising his animated project in storyboard form, so has decided to motion capture his entire project and apply the motion data to preliminary character rigs. He will then be able to use this to block out framing, camera movement, and shot timings in a more dynamic way with the advantages of not having to translate conceptually from 2D to 3D. Performance capture itself only takes a few hours, with the balance of time taken up with data cleanup and application of the data to a character rig. Then the camera blocking process can proceed in Maya. Changes can also be made quickly with a swift feedback loop to view updated rendered shots. We already see this impacting on the 3D animation pipeline whether or not a student intends to utilize motion capture as the primary means of animating his or her characters. Motion capture can be employed as a key tool at the development and pre-production stage of the pipeline to create a preliminary 3D animatic. It can be an aid at the shot production stage during the animation process itself by providing ‘live’ reference in digital 3D form. It can also be a key tool in developing character and performance, whether through working with an actor, or in making decisions about how a character design will work with live and/or key-framed motion. We conceive of the use of motion capture as not a discreet ‘stage’ as such in the pipeline, but as a tool to go back and forth to as appropriate needs arise.
5 Virtual Camera and Dance Alongside the use of motion capture in a more traditional animation context, there has been much exploration of its use as a tool in live performance and dance. We choose in this section to focus on an experimental motion capture dance performance work, Virtual Camera and Dance, which is being developed as a research project. The work developed out of initial investigations and brainstorming around the potential for motion capture outside of its traditional realm, that of character animation. We choose to discuss this particular project, because while it sits firmly outside of the so-called traditional application of motion capture technology, the research outcomes and lessons learned in its development directly speak to the aforementioned performance issues. In addition it looks at the technology as a more direct participant in the development of the work. Early into our investigations into motion capture one element that struck us, and research collaborator and choreographer Jennifer Nikolai, was the potential for embedding or infusing the technology, at least psychologically and in a sense physically, on the performers themselves. The term “motion capture” in itself situates alongside the historical terminology in the film industry of “getting the shot”, “capture”, “frame” etc. Where the performer performs to the technology and the performance is fixed in or by the technology. It struck us that an interesting experiment aesthetically, and also sitting alongside a significant pedagogical outcome, was to develop a project, that over a period of time, trained the performer, the director and the choreographer to attain some, and then increasing levels of control over the technology itself in terms of the eventual captured representation. Virtual Camera and Dance acknowledges, that traditionally, motion capture systems acquire movements of the body as data from specialized cameras, and in turn this data is interpreted by processors and human operators to convey this data to a modelled character, and the character is then animated by this movement data. What this project aims to do is take some measure of control back over how this data is applied to the final representation or the literal frame of the final product. We posed the question: What if the performer, through their performance, was able to control a virtual camera or cameras through the data collected from the movement generated from their captured choreography? The final presentation of the project could be developed towards a number of contexts, however the overall aim was to create a movement vocabulary that spoke to the technology in such way as to drive a virtual camera or cameras so that the final moving image was literally composed from the movement data derived from the choreography. Our initial investigations for this project came out of a visit to Deakin University in June 2008. Deakin has purchased a state of the art 24-camera Motion Analysis™ system on their campus and the technology is used throughout their Fine Arts curriculum, with a particular emphasis on dance and movement studies. The system and curriculum is lead by Dr. Kim Vincs, who opened the doors to their Motion Lab to us to ask questions and to demonstrate their own ongoing research into the technology. At that time we were looking into the best pathways for our own university, AUT, to follow towards the purchase of a system or systems for our own research interests. The Deakin University set-up is excellent and their model for
ongoing sustainability through an industry partnership has inspired our own direction at AUT. Beyond the infrastructural lessons we learned from the visit however, were other more critical questions in our brief. What can we do with it? How will it help us build and deliver curriculum? And how do we ‘break’ it make it do things it wasn’t designed to do? The purchase of high end motion capture equipment can be a hefty commitment for an institution, and in terms of research we needed to previsualize, in a fashion, the research potential and the impact on curriculum development that such a capital investment will deliver on. To scaffold on top of this original fact finding visit we arranged in association with AUT, CoLab, and Film Auckland a funded research and training visit back to Deakin University in February 2009. Four staff researchers and six postgraduate students across four disciplines formed a motion capture research cluster that travelled to Melbourne. Each of the participants in the cluster had their own research projects, some in collaboration, and each had their own research questions. After we finished the “bootcamp” component of the visit, the researchers/research teams embarked on their own investigations into the potential for the technology. Each of the research projects opened new avenues to explore but in relation to Virtual Camera and Dance, we will discuss the process we worked towards in its development, and what we have learned to date during its production. The work is a collaboration between Gregory Bennett, Andrew Denton, and Jennifer Nikolai, and was one of the key projects we worked on as a research cluster at Deakin. This project is ongoing so there is to date no final outcome, however we were able to learn a great deal in our first forages into motion capture technology, in terms of its potential for collaboration and creativity, but also in its value as a pedagogical tool, especially in relationship to performance. In a creative context the project conveyed to the group the interaction/relationship between the performer, the choreographer and the director. A primary lesson was developing communication tools between practitioners from different backgrounds. Jennifer Nikolai’s input here was invaluable due to her ongoing research interests into lexicographical transfer between different creative forms, particularly situated between, dance, theatre and moving image, with a focus on technology as a binding force. A process was put into place where we talked about the potential of the technology in terms of its three dimensional and temporal attributes. What data should we utilize to drive the camera in 3D space? And how does time play a role in the delivery of this data. Jennifer worked with an experienced dancer from the Deakin programme to create short movement stanzas that focused on different parts of the body. We then brought these stanzas into the motion capture studio and captured the performance. Out of this experience we looked deeply at the data and movement derived from it. We then went back to studio to sew the stanzas together in order to develop seamless transitions but also to experiment with different temporal signatures for the choreography. Over a period of several sessions a feedback loop was developed between the human participants and the technology. This deep un-packaging of the technology in a very visceral and physical context gave us a greater understanding of how the
system works to capture and deliver data, and it gave us a tangible sense of the limitations of the technology. In addition the technology became a participant in the crafting of the more human side of the work in the choreography. The project is still very much in its initial stages of investigation, as we compile the data and experiment with using different data sets to drive the camera. Once we edit and analyse the movement from this session we will then embark on another choreography and capture session that is informed by our initial findings to present a completed animated short dance sequence. One of the most interesting outcomes, from an anecdotal point of view was how useful the project was in terms of “getting under the hood of the technology.” We introduced this section with questions around translation of performance as an issue to engage with animators and in fact with everyone in the pipeline. This particular project, with its particular emphasis on the impact of human movement on the technology itself, and further to this allowing the technology to feedback into the performance itself, struck a chord in terms of uptake of motion capture to our research cluster. In terms of the current trend of embedding a research teaching nexus into the learning environment the project was and continues to feed us terms of our engagement with the performance issues and conditions of recording performance that are associated with motion capture technology. 6 Syllabus Development The availability of a relatively low-cost system such as Optitrak has made motion capture a viable option for our institution. The system itself has proved so far a robust one, and one which is comparatively quick to set up and calibrate. Students are thus able to master the basic workflow for capture in one introductory class session. The main issue compared with a system with a greater number of cameras would be the limits of the capture volume size (5-10 foot diameter - necessitating the use of a treadmill for walk or run cycles), and only one performer being able to be recorded at a time. Based on our work with the system to date, syllabus has been developed as a six-week introductory module which can be paralleled or integrated as appropriate with existing curriculum. This module covers both the technical set-up and operation of software, as well as basic history and principles: Week 1: Introduction to motion capture Motion capture history and basic principles Introduction to Optitrack motion capture system: setup and calibration, mocap suit marker set-up, skeleton set-up, recording mocap data. Week 2: Motion capture and animation: performance capture Directing and performing for motion capture Trajectorizing data and data cleanup. Week 3: Motion Capture and animation: creating a motion data library - capture using treadmill Importing data to Motion Builder, applying to a model, editing mocap data in Motion Builder Week 4: Motion Builder-to-Maya workflow Editing mocap in Maya using animation layers Week 5: Motion capture as animation tool Using mocap as reference for key-framed animation. Week 6: Motion capture as pre-production tool – creating an animatic/pre-visualisation. Working with ‘virtual’ props
Conclusion As various projects and research endeavours progress teaching modules will be developed to account for the diverse practical and creative/aesthetic issues encountered and identified throughout the production process. At this stage these more advanced modules would meaningfully coalesce around the following areas: • Motion capture as aid in project development. • Motion capture as an aid in teaching animation technique. • Motion capture as a tool for the animator: developing performance. • Directing for motion captured performance. • Acting for motion capture and animation: modes of performance. • Motion Capture and experimental moving image and performance practice.
References CANTOR, J. AND VALENCIA, P. 2004. Inspired 3D Short Film Production. Boston: Thomson Course Technology DEGRAF, B. AND YILMAZ, E. Puppetology: Science or Cult? Animation World Magazine, February 1999. http://mag.awn.com/?article_no=1307 DEITCH, G. Yes, but is It Animation? Animation World Magazine, January 2008. http://mag.awn.com/?article_no=3521 FURNISS, M. 2000. Motion Capture. http://web.mit.edu/commforum/papers/furniss.html KERLOW, I. 2004. The Art of 3D Computer Animation and Effects. New Jersey: John Wiley and sons. LASSETER, J. Principles of traditional animation applied to 3D computer animation. Computer Graphics, pp. 35-44, 21:4, July 1987 (SIGGRAPH 87). MORI, M. 1970. The Uncanny Valley. Energy 7(4): 33-35. ROBERTS, S. 2007. Character Animation: 2D Skills for Better 3D. Oxford and Burlington: Elsevier SULLIVAN, K., SCHUMER, G. AND ALEXANDER, K. (2008). Ideas for the Animated Short: Finding and Building Stories. Burlington & Oxford: Focal Press THOMAS, F. AND JOHNSTON, O. 1981. The Illusion of Life: Disney Animation. New York: Hyperion.
