Motion Capture Data Manipulation and Reuse via B-splines
Sandra Sudarsky1 and Donald House2 1
Computer Science Dept., Texas A&M University
2
Visualization Laboratory, Texas A&M University
[email protected] [email protected]
Abstract. This paper presents an integrated set of tools to facilitate
the manipulation and reuse of motion capture data. A functional representation in terms of nonuniform B-splines provides a compact and ecient description of motion sequences. A collection of B-spline curves attached to a hierarchical structure is used to represent the animation of articulated gures. A set of primitive operators acting on these curves and their application to motion editing is presented. We have successfully used these primitives to generate smooth transitions between motion sequences, motion interpolation and motion cycli cation.
1 Introduction Motion capture is the process of recording motion data in real time from live actors and mapping it into computer characters. Motion capture systems include mechanical, magnetic and optical devices that directly or indirectly track the actor's movements. During the past decade, motion capture has become increasingly popular as a tool to speed up the computer animation process. It has been used to produce high quality animations in relatively short amounts of time[10]. The major drawback of this technique is the lack of tools to eciently edit the captured data. In fact, often the whole acquisition process needs to be repeated in order to modify sections of a motion sequence[5]. Motion capture techniques can also be used to build a database of basic motions ([4], [13], [2]) to be used in interactive 3D character animation applications. In such environments, characters are controlled by the user and should move accordingly. Since only a small subset of all possible motions that the user may select can be stored, the challenging task is to design tools that allow reusing these sequences to generate new animations. Our purpose is to present an integrated set of tools to facilitate the manipulation and reuse of motion capture-based animation data. The design of these tools is based on the following requirements: { Data reduction. The data generated by motion capture systems consists of a value for each degree of freedom at every sampled frame. A compact representation with a reduced number of parameters to be adjusted is essential
2
Sandra Sudarsky and Donald House
to facilitate motion editing and reuse. This representation can also be used to reduce storage and transmission costs. { Data smoothing. The sensitivity of magnetic and optical devices to metals and lights introduces some noise in the captured data. In addition, the postprocessing step to generate three dimensional data and the occlusion problem related to optical systems can add extra artifacts. Some type of ltering mechanism needs to be incorporated to automatically produce smooth motion sequences. Editing operations, such as transitions between two existing motion sequences, must guarantee smoothness of the resulting motion. { Computational eciency. Interactive speeds are required for the system to be of practical use. { Animator Control. One of the major limitations of motion capture is the lack of animator control. High level editing operations together with direct manipulation of the motion parameters need to be provided. { Easy integration. The tools need to be easily integrated with traditional computer animation techniques such as keyframing, forward and inverse kinematics. The basis of our approach is the use of nonuniform B-splines to provide a compact and ecient representation of motion sequences. A set of operators is then applied to these spline curves to support the creation of new animations from previously recorded motion data. The rest of this paper is organized as follows. Section 2 summarizes related previous work. Section 3 describes our approach, presenting a curve tting technique based on B-splines and its application to noise reduction. It also describes a set of primitive operators designed to manipulate spline curves. Section 4 demonstrates the usefulness of our approach for the adaptation and reuse of motion sequences. Section 5 summarizes the work and proposes potential enhancements.
2 Related Work Most of the early editing methods for motion capture-based animation relied on keyframing [5]. However, the large amount of data involved makes this approach undesirable. In what follows, we describe more recently proposed techniques from the computer graphics literature. Using the principle of motion warping introduced simultaneously by Witkin and Popovic [14] and Bruderlin and Williams [4], new motions can be generated from a given motion sequence and a speci ed set of positional (or time) constraints. Gleicher [9] combined motion warping techniques with spacetime constraints to edit motion sequences. As in previous motion warping techniques, the user speci es a set of kinematic constraints that must be satis ed by the edited motion. These constraints together with an objective function describe the spacetime constraint problem. In contrast to other spacetime constraints techniques, interactive speeds are achieved by simplifying the objective function. While these methods allow the interactive creation of new animations they do not provide tools for the automatic generation of new sequences.
Motion Capture Data Manipulation and Reuse via B-splines
3
Bruderlin and Williams [4] applied multiresolution techniques well known in the areas of image and signal processing to edit motion sequences. They use a multiresolution ltering method to transform the motion into frequency bands. Adjusting the amplitude of these bands produces interesting e ects on the motion. Other techniques presented in [4] include waveshaping, which is useful to enforce joint limits of articulated gures, and multitarget motion interpolation that allows the blending of two motion sequences. In the Fourier interpolation approach of Unuma et al. [13], the motion of the joints of an articulated gure are represented by their corresponding Fourier series expansion. Using this representation, they describe a method to interpolate periodic motions of articulated gures and to extract certain characteristics of human behavior. In contrast to this technique, our approach is not based on the frequency domain and therefore it is not restricted to periodic motions. Rose et al. [11] applied spacetime and inverse kinematic constraints to generate smooth and dynamically plausible transitions between sequences of human body motions. Although physically correct, these transitions can not be generated in real time. Recently, Rose et al. [2] proposed a technique for real-time interpolation of motion sequences. Their approach closely relates to our own, it represents the motion of each degree of freedom as a function through time in terms of B-splines. An important distinction in our work is the use of nonuniform splines which provide more exibility during the curve tting process (see section 3.2).
3 Approach The motion of an object can be described as a collection of curves in time. When dealing with articulated gures, these curves represent the trajectories followed by the joints. Motion capture systems transform these curves into discrete-time signals by sampling the position and/or orientation of selected joints at regular time intervals. We use a curve tting technique based on nonuniform B-splines to reconstruct continuous-time curves from the sampled data. These provide a compact representation of motion sequences and allow us to design a set of primitive operators that act on these curves to provide ecient tools to edit the motion.
3.1 Hierarchical Model The captured data is transformed into anatomical rotations corresponding to an articulated gure ([1], [3]). Our model has six degrees of freedom at the root (typically the hips) which allow us to translate and rotate the entire model. The rest of the hierarchy is modeled as a set of revolute joints, each with three rotational degrees of freedom. A typical model with fteen joints and 48 degrees of freedom is shown in Figure 1.
4
Sandra Sudarsky and Donald House
Fig. 1. Hierarchical model with 48 degrees of freedom.
3.2 Curve Fitting and Noise reduction Let f(x ; t ); : : :; (xr ; tr )g be the set of data points generated for a particular 0
0
degree of freedom, where xi is the dataPvalue at time ti . We use a least squares approach to t a spline curve Q(t) = ni=1 ci Bi;k; (t) to the data, where Bi;k; is the ith B-spline basis function of order k associated with the knots i : : : i+k and ci is a weighting coecient. The values of the coecients c = fc1; : : :; cng of Q(t) are found by minimizing the expression r X (1) E = (Q(tj ) ? xj )2 : j =0
The least squares solution of (1) can be found by solving a linear system of the form Ac = r; (2) where A is a positive de nite, symmetric and sparse matrix. The system (2) can be solved eciently using the Cholesky decomposition method [8]. The curve tting problem described above assumes a xed knot vector . In the case that the position of knots are left as variables the problem becomes a constrained optimization problem. A number of algorithms to solve this problem have been proposed (see [7] for a good review). The success of these algorithms depends on the choice of the initial position of the knots and they su er from the existence of many stationary points. For eciency purposes, we select the knot vector prior to solving the least squares problem. The straight forward approach of using a uniform knot vector has several shortcomings. First, in order to re ne a spline de ned in terms of a uniform knot sequence, a knot must be inserted at the midpoint between every pair of existing knots. This restriction forces the addition of knots along the
Motion Capture Data Manipulation and Reuse via B-splines
5
whole curve when only portions of the curve may require re nement. Second, discontinuities can not be represented in terms of uniform knots since the addition of knots with multiplicities greater than one is required. Third, a nonuniform representation usually provides a better approximation when the geometry of the data is \used" to determine the knot placement. Our approach consists of selecting the knot vectors according to the following heuristics:
{ Add knots in regions of high curvature { Avoid knots in regions of noise { Add multiple knots to represent geometric discontinuities of the input data Figure 2 shows 1578 data points describing the position of the hips along the Y axis during a dancing sequence. The corresponding approximating spline de ned in terms of only 218 control points is shown in gure 3. The B-spline representation not only provides a compact representation, but also o ers local control.
Fig. 2. Motion capture data-hips-1578 data points. Although in practice these heuristics produce good results, the curves can always be re ned by introducing extra knots in regions where the data exceeds a speci ed tolerance and solving smaller least squares problems. During the curve tting process, some of the noise inherent to the motion capture data can be ltered out by introducing a weighting factor wj1+1 in equation (1), r X E = ( w 1+ 1 (Q(tj ) ? xj ))2 : (3) j =0 j
The weight wj represents how much deviation exists between the parameter value xj and the average value for that parameter around the neighborhood of
6
Sandra Sudarsky and Donald House
Fig. 3. Approximating spline curve-hips-218 control points. tj . The larger the deviation, the less in uence that point has over the tting process. We use wj = [xj ?2; xj +1] ? [xj ?w ; xj +w?1]; Pj+b jx ?x j where [xj ?a; xj +b] = k=j?1+a b?ka+1 k and 2 w is the width of the window centered at tj . Figure 4 shows the original data representing the rotation of the upper leg during a running sequence. The result of applying the noise reduction approach to approximate this data is shown in Figure 5 (the control points are omitted for readability).
Fig. 4. Motion capture data-Upper leg.
Motion Capture Data Manipulation and Reuse via B-splines
7
Fig. 5. Approximating spline with noise reduction.
3.3 Primitive motion operators We treat motion curves as objects which can be manipulated by a set of primitive operators. Below we describe some of these operators and their application to motion capture-based data manipulation. P The rst set of operators acts on individual curves. Let Q(t) = ni=1 ci Bi;k; (t) be a spline curve de ned by a set of control points c1 ; : : :; cn and a knot vector of size n + k, where k is the order of the spline.
Curve cycli cation. Assume that Q(t) describes a periodic motion (with period ) for a particular parameter. The cycli cation operation generates a new spline S(t) such that S l (t) = S l (t + ), for l = 1; : : :; k ? 2. ()
()
Given the knot vector = 1 ; : : :; n+k of the original spline, we generate the knot vector = 1; : : :; 2n+k as follows: j n+k j = j + n1 + k +1 j 2n+k j ?1 j where j = j ?n ? j ?n?1. Similarly, given the coecients c1 ; : : :; cn of the original spline, the coecients for the resulting spline d1; : : :; d2n are given by: jn dj = ccj 1n j ?n + 1 j 2 n: This process can be repeated to generate spline curves of arbitrary length. Note that in the case where the motion is not exactly periodic, the blending operators described below can be used to smooth out the transitions.
8
Sandra Sudarsky and Donald House
Knot adjustment. Splines provide a parametric curve representation whose length can be controlled by changing the knot vector appropriately. The knot vector can be used to adjust the timing within a sequence or to synchronize two or more motion sequences. For example, we can change the starting time of the animation by shifting the entire knot vector by a constant. Slower and faster animations can be generated by scaling the distance between every pair of knots by a constant factor. Motion synchronization is very important when blending motion sequences together in order to guarantee that important events coincide in time. Suppose ft1; : : :; tm g and fT1; : : :; Tm g represent two sets of time instances specifying corresponding key events on two motion sequences. We can view each pair (ti ; Ti) as a time constraint specifying the time Ti at which the value associated with time ti should occur. This problem is equivalent to the time warping problem described in [14] and [2]. However, instead of transforming the raw data to satisfy the time-constraints, we transform the knot vector to a new knot sequence as follows. For j 2 [ti ; ti+1], the corresponding knot j is given by i+1 ? Ti ) j = Ti + (T (t ? t ) ( j ? ti ): i+1
i
Translation. Translating the curve Q(t) by a constant is equivalent to translating the control points by and computing the spline using the new vertices. The translated curve is given by QT (t) =
n X i=0
(ci + )Bi;k; (t):
Scale. Let S be a scaling factor. Then, the scaled curve can be written as Qs (t) =
n X i=0
(Sci )Bi;k; (t) = SQ(t):
The next set of operators P takes as input two splinePcurves and generate a new curve. Let Q1(t) = ni=0 ciBi;k; 1 (t) and Q2 (t) = mi=0 biBi;k; 2 (t) be two spline curves de ned on the knot vectors 1 and 2 respectively. A new spline curve Q3(t) can be generated in two steps. First, the two original curves are represented with respect toPthe same knot vector 3 = 1 P [ +2 mvia the Oslo +m algorithm [6], i.e. Q1(t) = ni=0 ci Bi;k; 3 (t) and Q2 (t) = ni=0 bi Bi;k; 3 (t), where fbi g and fci g stand for the coecientsPunder the re ned representation. +m Second, we construct the new curve Q3 (t) = ni=0 di Bi;k; 3 (t) in terms of the knot vector 3 . The coecients of the new curve fdi g are computed according to the desired operation. We describe below several operators by specifying how the new coecients are computed.
Motion Capture Data Manipulation and Reuse via B-splines
9
Uniform Blending. This operation consists of a convex combination of the spline coecients,
di = ubi + (1 ? u)ci ; 0 u 1. It generates a curve that lies somewhere in between the two original curves, according to the parameter u. Figure 6 shows a uniform blending of two splines curves. When this operation is applied to each pair of curves representing the motion of two articulated gures, a new animation that corresponds to an interpolant between the two original ones is generated.
Fig.6. Two spline curves (one light gray, the other thin dark gray) and the result of applying the uniform blending operator (shown in black) with u = 0:5.
General Blending. A general blending operation results when the coecients for the new spline are generated by combining the original coecients according to a blending function 0 f(t) 1, di = f(ti )bi + (1 ? f(ti ))ci , where ti corresponds to the value of the ith entry of the knot vector 3 . Figure 7 shows the result of blending two motion curves using f(t) = 2( bt??aa )3 ? 3( bt??aa )2 + 1; where a and b correspond to the value of the rst and last knots of 3 . The above blending function provides smooth changes at the initial and nal stages (slow-in/slow-out) and almost constant rate of change in the middle. Note that the resulting spline (shown in black) slowly transitions from one input spline to the next. We can control the interval during which the blending takes place by supplying two real numbers t0 and t1 . In this case, the new curve Q3 (t) is identical to
10
Sandra Sudarsky and Donald House
Fig.7. Two spline curves (one light gray, the other thin dark gray) and the result (shown in black) of applying the blending operator. Q1(t) for t t0; it corresponds to a smooth transition between Q1 and Q2 for t0 t t1 and it matches Q2 for t > t1. Note that the curves Q1 (t) and Q2(t) must be re ned to share the same knot vector only in the interval [t0; tf ]. The coecients for the new curve are given by di = f(ti ; t0; tf )bi + (1 ? f(ti ; t0; tf ))ci ; where ti corresponds to the value of the ith entry of the knot vector 3 . As an example of a blending function let 81 t < t0 < f(t; t0 ; tf ) = : 2( ttf??tt00 )3 ? 3( ttf??tt00 )2 + 1 t0 t tf 0 tf < t: Figure 8 shows two original spline curves and the resulting curve after the blending operation is applied using a short interval [t0; t1 ]. These blending operators can be used to generate smooth transitions between motion sequences.
Di erence. The di erence (addition) between two curves is found by subtracting (adding) their coecients, di = bi ci : Motion warping. The technique described in [14] and [4] can be easily ex-
tended to work with B-spline curves. Given a curve Q1(t) and a set of displacements di(tk ) = pk for i = 1; : : :; m, we nd a new B-spline curve Q3 (t) by Q3 (t) = Q1 (t) + Q2(t), where Q2(t) is a B-spline curve interpolating the displacements.
Motion Capture Data Manipulation and Reuse via B-splines
11
Fig.8. Two spline curves (one light gray, the other thin dark gray) and the result (shown in black) of applying the blending operator with a speci ed blending interval. When applying these operators to the curves describing the motion of articulated gures, we must guarantee that joint limits are maintained and kinematic constraints are enforced. Due to the convexity and locality property of B-splines, joint limits can be violated only on the local interval controlled by a control point whose value exceeds the limit. In this case, we force the curve to reach the limiting value via motion warping. Kinematic constraints can be enforced via inverse kinematics. Once the joint angles are known, the motion curves for those joints can again be modi ed via motion warping techniques.
4 Results A prototype [12] is being implemented to serve as a testbed for the application of these operators to the manipulation and reuse of motion capture-based animation data. Figures 9 and 10 show original walking and running sequences. Motion is from right to left, with the stick gures representing the character's pose at discrete uniform time steps. Note the greater distance between successive poses in the running sequence, as well as the more extreme swings of arms and legs. Figure 11 shows a smooth transition from one sequence to the other. This new sequence is generated in three steps: First, the starting time of the running sequence is updated by shifting every knot vector de ning the motion curves by a constant factor. Second, the spline curves representing the position of the hip during the running sequence (root of the hierarchy) are translated so that the position of the hip at the beginning of the running sequence closely matches the position of the hip at the end of the walking sequence. The nal step consists of blending (with a carefully chosen interval [t0; tf ]) every corresponding pair of splines representing the motion of the two articulated gures. Note that the interval [t0; tf ] de nes how fast the transition should occur.
12
Sandra Sudarsky and Donald House
Fig. 9. Walk sequence.
Fig. 10. Running sequence. The uniform blending operation (with u = 0:5) applied to the walking and running sequences of gures 9 and 10 generates the accelerated walk shown in Figure 12. This corresponds to an interpolation between the two original sequences. Motion cycli cation is achieved by applying the curve cycli cation operation to every motion curve describing the animation of a periodic motion (except for the curves describing the position of the hips along the X-Z plane which can speci ed by an arbitrary path). Figure 13 shows an original sequence of running steps (in dark gray) and the result of applying the cycli cation operation (in light gray). This operation applied repeatedly can be used to generate animations of arbitrary length.
Motion Capture Data Manipulation and Reuse via B-splines
13
Fig. 11. Smooth transition from walking to running.
Fig. 12. Uniform blending of a walk and a run sequence.
5 Conclusions We mention in closing how the proposed approach allowed us to achieve the desired objectives. { Data reduction. Nonuniform B-splines provide a compact representation since the number of control points is usually far less than the number of input data points. { Data smoothing. The use of cubic B-splines allow us to generate smooth curves having positional, rst and second derivative continuity everywhere except at multiple knots. The incorporation of a ltering mechanism during the curve tting step reduces some of the noise inherent to the data. { Computational eciency. The development of ecient heuristics for nding \good" knot vectors is an important algorithmic improvement, since cur-
14
Sandra Sudarsky and Donald House
Fig. 13. Motion Cycli cation. rently existing spline tting methods are very ecient once a \good" knot vector is provided. The eciency of the editing primitives relies on the fact that the operators are applied directly to the control points. { Animator control. The primitive motion operators provide the animator with tools for motion curve manipulation and reuse. In addition, the local support property inherent to B-splines permits direct and local manipulation of motion curves via control points. { Easy integration. The adoption of a standard hierarchical model to represent articulated gures simpli es the process of incorporating our tools with traditional computer animation algorithms based on forward and inverse kinematics. The techniques described in this paper were applied to motion capture data but they are equally applicable to motion gathered in other ways. Some possibilities include the evaluation of spline curves in a keyframe system or data generated by procedural algorithms. The use of the operators described in this paper facilitates the creation of complex motions from simple animations. We are currently exploring the use of these techniques to design other operators that allow the incorporation of cartoon-like e ects such as exaggeration and squash and stretch into previously recorded motion sequences.
Acknowledgments The authors would like to thank LambSoft for access to motion capture data. They are also grateful to James Abello and Je Trinkle for their suggestions and discussions. Special thanks to Susan Van Baerle and Je Thingvold for their continuous support.
Motion Capture Data Manipulation and Reuse via B-splines
15
References 1. Bodenheimer, B., Rose, C., Rosental, S., Pella, J., The process of Motion Capture: Dealing with the Data, Computer Animation and Simulation'97, Eurographics Animation Workshop, D. Thalmann and M. van de Panne, eds., Springer-Verlag, New York, Sept. 1997, pp. 3-18. 2. Bodenheimer, B., Rose, C., Cohen, M., Verbs and Adverbs: Multidimensional Motion Interpolation, IEEE Computer Graphics and Applications, Sept. 1998, pp. 32-40. 3. Molet, T., Boulic, R., Thalmann, D., A Real-Time Anatomical Converter for Human Motion Capture, In Computer Animation and Simulation'96, Eurographics Animation Workshop, G. Hegron and R. Boulic eds., Springer-Verlag, 1996, pp. 79-94. 4. Bruderlin, A., Williams, L. Motion Signal Processing. In Computer Graphics (SIGGRAPH'95 Proceedings), Aug. 1995, pp. 97-104. 5. Character Motion Systems. In Computer Graphics (SIGGRAPH'93), Course no. 01. 6. Cohen, E., Lyche, T., Riesenfeld, R., Discrete B-splines and Subdivision Techniques in Computer-Aided Geometric Design and Computer Graphics. Computer Graphics and Image Processing, 14(2), Oct. 1980, pp 87-111. 7. Dierckx, P., Curve and Surface Fitting with Splines. Oxford University Press, New York, 1993. 8. George, A., Liu, J., Computer Solution of Large Sparse De nite Systems. PrenticeHall Inc., 1981. 9. Gleicher, M., Motion editing with space-time constraints, In Michael Cohen and David Zeltzer, editors, Proceedings 1997 Symposium on Interactive 3D Graphics, Apr. 1997, pp. 139-148. 10. Motion Capture in Practice. In Computer Graphics (SIGGRAPH'97) Course no. 01. 11. Rose, C., Guenter, B., Bodenheimer, B., and Cohen, M., Ecient Generation of Motion Transitions using Spacetime Constraints. In Computer Graphics (SIGGRAPH'96 Proceedings), Aug. 1996, pp. 147-154. 12. Sandra Sudarsky, Manipulation and Reuse of Motion Data via B-Splines, Ph.D. thesis, Texas A&M University, in preparation. 13. Unuma, M., Anjyo, K., Takeuchi, R. Fourier Principles for Emotion-based Human Figure Animation. In Computer Graphics (SIGGRAPH'95 Proceedings), Aug. 1995, pp. 91-96. 14. Witkin, A., Popovic, Z., Motion Warping. In Computer Graphics (SIGGRAPH'95 Proceedings), Aug. 1995, pp. 105-108.
