An Optimal Control Approach to Reconstruct Human Gait Dynamics from Kinematic Data Martin L. Felis1 , Katja Mombaur1 , and Alain Berthoz2
Abstract— A common approach to record full-body human movement data is by using marker based motion capture systems. To obtain dynamic gait data such as joint torques and ground reaction forces additional measurement devices have to be employed that pose restrictions on where feet have to be placed during the recording. In this paper we use articulated rigid multibody models and optimal control methods to recover dynamic gait data solely from kinematic data. Our approach is independent from the used marker set and creates the rigid multibody model and computes all controls for the model such that when applied to the model, it closely reproduces the originally recorded motion. To achieve this there are two steps involved: i) create a subject-specific rigid multibody model of the recorded person and used marker set and compute the joint kinematics using inverse kinematics. ii) reconstruct the gait dynamics by solving an optimal control problem. For step i) we created a parameterize human model H EI M AN and a graphical user interface P UPPETEER that facilitates creation of the subject specific model and the motion capture mapping. For ii) we use MUSCOD-II , which implements the direct multipleshooting method. We apply our method on 15 emotional human walking motions to compare joint angle and torque patterns of different emotions.
data only has the advantage that the experiments are simpler since less devices are required and that no restrictions on where to place the feet (as it is the case of force plates). In addition, a lot of kinematic data of interesting motions with no corresponding force recordings etc. is already available which could be exploited by such a method.
Fig. 1. Visualization of the postural errors between the kinematic fit (red) and the dynamic reconstructed gait (green).
I. INTRODUCTION Gaining a better understanding of human movement is important for many fields in robotics, such as motion generation and control of humanoid robots, human-robot interaction, and design and control of wearable robots, such as exoskeletons, orthoses and prostheses, or of external physical assistive devices. Some insights can be gained from experimental recordings of human movements by motion capture systems typically based on optical systems or inertial measurement units (IMUs), measurements of external forces e.g. by force plates or pressure soles, EMG recordings etc. However there are many interesting details of human movement that can not be measured directly, such as joint torques, precise trajectories of all limbs, center of mass movement, angular momentum etc. In this context, dynamic whole-body models of the human body are very useful to further look inside human movement by fitting the computer model to available human recordings and evaluating all interesting quantities. In this paper we make a contribution to human movement understanding by dynamical models with a special focus on reconstructing dynamic movement information from purely kinematic measurements. Being able to work with kinematic 1 Optimization in Robotics and Biomechanics (ORB), Interdisciplinary Center for Scientific Computing (IWR), Heidelberg University, Germany, {martin.felis|katja.mombaur}@iwr.uni-heidelberg.de 2 Laboratoire de Physiologie de la Perception et de l‘Action, Coll` ege de France, Paris, France,
[email protected]
Fig. 2.
Schematic of our dynamics reconstruction approach
In particular, we present the following methods and models for human movement understanding: • The efficient rigid body dynamics library RBDL for setting up dynamic model equations of systems of multiple rigid bodies in chain or tree structure as well as models of external contacts and impact. • A customizable human meta-model H EI M AN the parameters of which can be adjusted to subject specific kinematic and dynamic properties using the tool P UP PETEER , based on a semi-automatic adjustment of the unconstrained kinematic model to data. • An optimal control based approach for fitting the constrained dynamic subject specific human model to the given kinematic data. In this paper, we are especially interested in walking motions, but the principal approach is applicable to more general motions. A. Related Work Dynamic gait analysis tools such as OpenSim [22] compute joint actuations by performing inverse dynamics based
c
2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. This is a preprint. The final version is available from: http://dx.doi.org/10.1109/HUMANOIDS.2015.7363490.
on kinematic motion capture data and force plate recordings. However, the computed joint forces are in general not physically consistent with the recorded data, i.e. the measured ground reaction forces and forces excerted by the model do not cancel each other out. These so-called ”residual forces” are then reduced by a separate post-processing step that alters both dynamic properties of the model and the kinematics. Various methods have been proposed to compute dynamic quantities such as ground reaction forces and joint torques from kinematic motion data alone. E.g. [3] computes ground reaction forces from kinematic running data, ground reaction forces and joint torques during the single stance phase of human walking for the lower limbs [7]. By properly treating the indeterminacy of the ground reaction force during the double step phase one can also compute joint torques during a full gait cycle [21], [16]. The proposed methods primarily aim for human gait analysis and are verified with measured data of ground reaction forces, however the joint torques are not used to actually actuate a simulated model. In contrast to these methods our approach relies on a clearly defined multibody system model and leads to reconstructed movement and joint torques that are physically consistent with respect to the dynamic human model used. That means that starting a simulation of the model with the initial conditions for joint angles and velocities and applying the reconstructed torques will yield the same motion of the model as we have reconstructed. This property becomes critical when using optimal control methods to synthesize human walking [1], [18], i.e. to predict how the human model would move with a given optimization criterion and constraint, sometimes also referred to as predictive simulation. We have already used our models in this context as shown in [10], but will not include it here for reasons of space. Another advantage is that our approach works with arbitrary marker sets. Developing RBDL and the metamodel H EI M AN was an important pre-requisite for the optimal control approach used for data fitting since many existing human models do not satisfy the accuracy, efficiency and smoothness conditions that have to be imposed for the use in optimal control. Our methods have been developed in the context of reconstructing the dynamics of human walking under different emotions based on maker data. While the range of applicability of the method is not limited to that domain it shows clearly that our method had to be able to detect subtle differences in the dynamics. A few results for this example will be presented in the paper to demonstrate the validity of the method. We have used datasets expressing Neutral walking as well as the emotions Joy, and Anger during walking. The expressions of the two emotions were each specified in two variants: an expressive variant (JoyExpressive and AngerExpressive that is openly shown and a supressive variant that is held back or directed inwards (JoySelfsufficient and AngerSupressive). For each of the five expressions we used three trials, i.e. we reconstructed 15 motions in total.
II. METHOD OVERVIEW Our approach uses articulated rigid multibody systems to describe the dynamics of human or robotic models. The models we use are underactuated systems that can be described by a tree with one non-actuated designated root body with six degrees of freedom (DOF) and all other bodies are connected with joints that have up to three rotational DOFs. The model further may be subject to external contact constraints. In this work our models are full-body 3-D models with 34 DOFs. To reconstruct the dynamics from marker based motion capture data we first create a subject-specific rigid multibody model that closely represents the kinematic properties in terms segment dimensions and joint center locations of the recorded subject. We then transfer the recorded marker motions onto articulated rigid multibody motion in form of kinematic joint angles of the subject-specific model. From the kinematic properties we also infer a dynamic subjectspecific model by employing tabular data from the biomechanics literature to obtain mass and inertia values for all bodies. The actual dynamics reconstruction is then performed by formulating a multi-phase optimal control problem (OCP) that fits the motion of the dynamic model to the joint angles obtained from the previous step. Each phase of the OCP corresponds to a specific constraint set of the model with the environment. As we focus on bipedal walking here each phase corresponds to certain points of the foot that are in contact with the ground. Our OCP formulation includes single- and double-support and also discontinuities arising from contact gains when heel or forefoot touches the ground. A schematic of our approach is shown in Figure 2. A. Phases of the Human Gait We model the human gait as a sequence of multiple phases. On each phase different contact constraints act on the feet of the human model. In this paper we consider a single step which starts when the right foot is flat on the ground and the left foot just started to swing and the step ends when the right foot lifts off the ground. In total we have five phases: Right Flat, Right Forefoot, Right Forefoot Left Heel and Right Forefoot Left Flat. Additionally we model touch down events of the left heel and left forefoot as discontinuous events. The phases of the step are visualized in Figure 3. Even though we only consider a single step here one should note that it is not a limitation of our model but instead can be extended to a double or a multi-step sequence. Focusing on a single step however leads to fewer variables in the reconstruction optimal control problem. III. M ODELING OF R IGID M ULTIBODY K INEMATICS AND DYNAMICS We use rigid multibody systems to describe human models. A rigid multibody system consists of a set of bodies B0 , . . . , BnB , nB ∈ N that are connected to each other via joints J1 , . . . , JnB . We use reduced coordinates to describe the state of a rigid body system and use q , q˙ , q¨ and τ to
Fig. 3. Phases of human gait that we used for our multi-phase optimal control problem formulation: RF Right Flat, RT Right Forefoot, RT LH Right Forefoot Left Heel, RT LF Right Forefoot Left Flat. Ground collisions occur at TD LH (touchdown Left Heel) and TD LT (touchdown Left Forefoot).
T where x (t) = q (t)T , q˙ (t)T and u(t) are the joint forces of the actuated joints that get mapped onto the generalized forces via a selection matrix T by writing τ = T u. As different set of contact constraints lead to different algebraic constraints we use i to indicate the currently used constraint set. When using numerical integration equation (2b) will not be fulfilled exactly, hence errors will accumulate eventually. This is especially true for large step sizes and/or long simulation durations. In this case one can use Baumgarte stabilization [2]. In this work however no stabilization was used as the integration horizons are relatively short and no large error accumulations have been observed. B. Collision Impacts
denote the generalized positions, generalized velocities, generalized accelerations, and generalized forces of the system. A. Dynamics with External Contacts The equation of motion for a rigid multibody model that is subject to external rigid contact constraints is described by H (q )¨ q + C (q , q˙ ) = τ + G(q )T λ, g (q ) = 0,
(1a) (1b)
∂ g (q ) is the so-called contact Jacobian where G(q ) = ∂q matrix of size m × ndof and λ ∈ Rm are the contact forces. Equation (1) is a differential algebraic equation of (differential) index 3. By differentiating the constraint equation (1b) twice we obtain:
H (q )¨ q + C (q , q˙ ) = τ + G(q )T λ, ˙ )q˙ = 0, G(q )¨ q + G(q
(2a) (2b)
which we can rewrite as a linear system of the unknowns q¨ , λ: # " # #" " q¨ −C (q , q˙ ) + τ H (q ) G(q )T = (3) −λ γ(q , q˙ ) G(q ) 0 This system is always solvable if G(q ) has full rank, which is the case if the constraints in g (q ) are not redundant. The term γ(q , q˙ ) ∈ Rm is the negative right summand of (2b) and is also called contact Hessian. To ensure equivalence of (1) and (3) we have to ensure that the invariants of the constraints are fulfilled: g (q ) = 0, G(q )q˙ = 0.
(4) (5)
It is sufficient to ensure that these conditions are fulfilled only at the beginning of the contact as due to (2b) and therefore (3) the constraint conditions are already fulfilled on the acceleration level. The solution of (3) can be embedded in a ordinary differential equation as x˙ = f i (x (t), u(t))
(6)
The transition from a rigid-body system without contacts to a system that has contacts is called a contact gain. During a contact gain very high forces act on the body for a very short time. In the real world these high forces cause the body to first compress and then expand. After the compression phase, depending on the physical properties of the body, the body remains either in contact (perfect inelastic collision) or bounces off. In rigid-body simulations the compression and expansion is usually neglected and the contact gain is treated as an instantaneous collision. The physical properties of the body are described by the parameter of restitution e ∈ [0, 1]. For e = 0 the collision is a perfect inelastic collision, whereas for e = 1 the collision is perfectly elastic. The contact gain is a discontinuous change in the generalized velocity variables from q˙ − to q˙ + , i.e. the velocity before the collision to the velocity after the collision. The change can be computed using # " #" # " q˙ + H (q )q˙ − H (q ) G(q )T = (7) −Λ G(q ) 0 −eG(q )q˙ − where Λ is the contact impulse. The upper part of this equation implies H (q )q˙ + − H (q )q˙ − = G(q )T Λ,
(8)
which is the change of momentum of the system due to the collision. The lower part G(q )q˙ + = −eG(q )q˙ −
(9)
states the contact velocity after the collision. For e = 0 this is equivalent to a contact velocity of zero, a perfect inelastic collision. C. Numerical Solution of the Contact System The contact systems (3) and (7) have the same matrix on the left-hand side which is regular if G(q ) has full rank. In this section we describe the numerical solution of (3), however the same method can be used to solve (7). There are different ways of solving the linear system. By setting c(q , q˙ , τ ) = −C (q , q˙ ) + τ and omitting the
arguments q , q˙ , and τ we can write the system as # " # #" " q¨ c H GT = , −λ γ G 0
(10)
where H ∈ Rndof ×ndof , G ∈ Rm×ndof , and with 0 < m < ndof being the number of contact constraints. The matrix in the linear system is of the same structure as the so-called Karush-Kuhn-Tucker (KKT) matrix that plays an important role in quadratic optimization problems. The matrix is regular if G has full rank and is furthermore indefinite [20]. The matrix in (10) is symmetric but indefinite. An efficient method to solve the system is the symmetric indefinite LDLT factorization. Nevertheless more efficient methods are available that exploit the structure of the matrix. The so-called range-space method (or sometimes called Schur-complement method) first solves for λ and then q¨ . The former is computed using GH −1 G T λ = γ − GH −1 c
(11)
and the latter is obtained by solving for q¨ using the system H q¨ = c + G T λ.
(12)
(11) can be obtained by multiplying the upper part of (10) with GH −1 and then subtracting the result from the bottom part. The matrices GH −1 G T and H are both symmetric and positive definite which allows us to use a Cholesky decomposition to solve the equations (11) and (12). It suffices to factorize H just once and use the resulting factors to compute the columns of H −1 G T and the vector H −1 c in (11) and additionally when solving (12) to compute the accelerations. As H (q ) has a sparse structure for branched system one can also exploit these so-called branch-induced sparsities to derive even more efficient methods that are described in [8] and [9]. D. Dynamics Implementation For all modeling, kinematics and dynamics computations of rigid multibody systems we created RBDL - the Rigid Body Dynamics Library. It is a highly efficient C++ library that contains state-of-the-art rigid-body dynamics algorithms such as the Recursive Newton-Euler Algorithm (inverse dynamics), the Composite Rigid-Body Algorithm (computation of the joint-space inertia matrix), and the Articulated Body Algorithm (forward dynamics). The library strongly builds on the concept of Spatial Algebra [9], which is a both concise and efficient notation for describing rigid-body kinematics and dynamics. Instead of expressing the motion of the individual bodies using two distinct sets of 3-D equations (one for translations and one for rotations) it combines the two equations in a uniform 6D formulation. This results in less equations but also fewer lines of code.
RBDL uses by default the Eigen3 C++ template library [11] for linear algebra. It automatically makes use of highperformance SSE instructions that greatly speeds up evaluation of the 6-D expressions if supported by the CPU. Computations for the joint-space inertia matrix H (q ) is done using the Composite Rigid-Body Algorithm. The Recursive Newton-Euler Algorithm is used to compute the coriolis and gravitational forces C (q , q˙ ). The library also computes the contact Jacobians G(q ) and derivative terms γ(q , q˙ ). RBDL then builds the linear system (3) and solves it with a range-space method that exploits the branch-induced sparsities of H (q ). The RBDL is freely available under the permissive zlib-License and can be obtained from https://rbdl. bitbucket.org. IV. K INEMATIC M OTION M APPING Mapping marker-based motion capture data onto joint angles is a well-known problem in computer animation, biomechanics, and robotics. Various approaches have been suggested, however either the method or their prerequisites did not suit our use case. E.g. the Plug-in Gait (PiG) model, based on [5]1 , is used by VICONTM motion capture system constructs a body-local coordinate frame to compute orientations for each segment and then computes the joint angles. A problem of this approach is that the individual segments are not rigid, i.e. the distance of two joints that are connected with the same body is not constant. The method described in [13] automatically computes the kinematic structure (i.e. joint centers) and joint angles from recorded markers. The distances between the joints are also constant and arbitrary marker sets can be used. However all joints are modeled as spherical joints, which would require a post-processing step to re-map the data onto 1-DOF joints. Further, the method needs calibration data in which the recorded subject must excite all DOFs of all joints. We created a semi-automatic approach where the subjectspecific model is first manually created from the motion capture data using a graphical user interface and then we map the marker data onto the joint angles by using virtual markers on the model and minimizing their deviations to the measured markers. A. Creation of the Subject-Specific Model To create a subject-specific model from marker data we developed H EI M AN, which is a parameterized meta model for human beings. It has up to 19 segments and joints. In its simplest form it can be used to create rigid multibody model of a human by only specifying height and mass of the human. The rigid multibody model is then created based on tables from the biomechanics literature [6] with the degrees of freedom listed in Table I. In our case it results in a fullbody model with 34 DOFs. The subject-specific model is then created from such a simple human model by adjusting segment lengths (such 1 [5] only describes the computation for the lower-limb kinematics, however for the upper-body we were not able to find corresponding references.
Fig. 4. Schematic of the H EI M AN meta model. Inertial parameters can be initialized from tabular data or specified by custom values. Degrees of freedom can be specified for each joint individually to describe 1-D, 2-D, and 3-D human models.
Fig. 5. Screenshot of P UPPETEER, the motion capture mapping tool we created to create subject-specific models and to perform the inverse kinematics. The posture of the model can be modified using the Model State editor on the left. The degrees of freedom of the currently selected segment are highlighted in green. On the right locations of joint centers can be modified in the ”Object Properties” panel.
B. Mapping the Motion as the length of the Thigh segments, or the width of the Clavicula segments) and segment masses for each segment individually. This allows for a detailed customization of the model to closely represent the recorded subject. The kinematic properties of the subject-specific rigid multibody model are captured by the body-local coordinates of all joint centers. We denote p c ∈ R3nJ , with nJ ∈ N being the number of joints, as the vector that is constructed by the concatenation of all joint centers. We extract segment dimensions either from joint centers (e.g. Thigh, Shank) or markers (e.g. Feet) and scale locations for each segment’s center of mass and inertia matrix as described in [6]. H EI M AN is implemented in Lua [12], which is a lightweight scripting language that can be easily embedded into existing applications. These so-called L UA M ODELs can then be directly loaded into RBDL. Joint Pelvis Hip Knee Ankle
DOFs
Joint
DOFs
Translation: XYZ Rotation: YXZ Rotation: YXZ Rotation: Y Rotation: YXZ
Lumbar Shoulder Elbow Neck
Rotation: Rotation: Rotation: Rotation:
YXZ YXZ Y YXZ
TABLE I D EGREES OF FREEDOM USED BY DEFAULT BY H EI M AN FOR THE JOINTS .
To perform the subject-specific adjustments we created the tool P UPPETEER. It loads motion capture marker data from .c3d files [19] that can be exported from all major motion capture systems. It simultaneously displays the markers from the motion capture data and provides methods to visually adjust the joint centers p c and to assign the virtual markers p m to the model that correspond to the recorded markers. A screenshot of P UPPETEER is shown in Figure 5.
We denote the motion capture marker data by m D ij , i = 1, . . . , nF , j = 1, . . . , nM where nF ∈ N are the number of recorded motion capture frames and nM ∈ N the number of motion capture markers. The subject-specific model is parameterized with joint centers p c and virtual marker locations p m . Let r M (j, q , p c , p m ) ∈ R3 describe the forward kinematics function that computes the cartesian coordinates of the virtual marker j. We can then formulate a least-squares problem to perform the inverse kinematics of motion capture frame i onto the model by: 1 min R(q i ) = r (q i )W r (q i ) (13) qi 2 with M c m mD i,1 − r (1, q i , p , p ) .. ∈ R3nM r (q i ) = . M c m mD − r (n , q , p , p ) M i i,nM (14) being the error between recorded and virtual marker for the current pose q i . A complete motion capture sequence m D ij is then mapped onto the joint angles q i , i = 1, . . . , nF frame by frame. To solve (13) we implemented the method described in [23]. V. O PTIMAL C ONTROL P ROBLEM F ORMULATION FOR THE R ECONSTRUCTION We reconstruct the joint forces and ground reaction forces by solving an optimal control problem (also sometimes referred to a trajectory optimization problem). A. Problem Formulation The optimal control problem uses the dynamics of the subject-specific model and fits it to the kinematic motion obtained by the kinematic motion mapping. This fit is much more complex than the previous fit as it is subject to the following additional constraints:
rigid multibody dynamics of an underactuated system • ground contact constraints • kinematic constraints (such as foot clearance) Let the recorded motion capture data be given as a set of time discrete postures in form of generalized positions ndof qD , j = 0, . . . , m at time points t0 , . . . tm ∈ R and j ∈ R let T = [t0 , tm ] be the time horizon of the motion. Then the dynamics reconstruction problem can be described by •
min
x (·),u(·)
m X 1 j=0
2
2 2 ||q D j − q (tj )||2 + γu ||W u(tj )||2
subject to: x˙ (t)
=
f i (x (t), u(t)),
x (t+ i ) g i (x (t), u(t))
= ≧
c i (x (t− i )), 0,
r (x (tˆ1 ), . . . , x (tˆk ), u(tˆ1 ), . . . , u(tˆk ))
≧
0,
(15a)
t ∈ [ti−1 , ti ], i ∈ I, (15b) i ∈ I, (15c) t ∈ [ti−1 , ti ], i ∈ I, (15d) tˆ1 , . . . tˆk , ∈ T , (15e)
The first term in the objective function (15a) minimizes the sum of squared errors of the joint angles of the model q (tk ) with the joint angles from the motion capture data q D k . The second term of the objective function is used to regularize the problem and minimizes the applied joint forces weighted by the diagonal matrix W = diag {wl } , wl > 0, l = 1, . . . , nu . This weighting is required to account for the different magnitudes of joint forces and the used weights are shown in Table II. We obtained the weights by taking the average joint forces from the solution of (15) with γu = 0. In the actual reconstruction we use a weighting factor γu = 1.0 × 10−2 to ensure that the joint residuals are the major contributor in the objective function. Joint Hip Hip Hip Knee Ankle Ankle Ankle Lumbar Lumbar Lumbar
Axis
w −1 i
Joint
Y X Z Y Y X Z Y X Z
15.5 18.0 3.6 14.4 39.4 5.2 2.5 5.6 13.9 2.5
Shoulder Shoulder Shoulder Elbow Neck Neck Neck
Rnx describe the different dynamics subject to the different constraint sets (see Section III-A) and are influenced by the controls u : T → Rnu . The optimization computes optimal state trajectories x (t) = [q (t), q˙ (t)]T and joint actuations u(t). The duration of the individual phases are prescribed from the motion capture data. Phase transitions, specifically collisions at touch-down events of the foot with the ground, are described using the phase transition functions c i (x (t)). The collisions are evaluated as described in Section III-B with a restitution parameter e = 0. The path constraints (15d) contain upper and lower bounds for the differential states x (t) and controls u(t). The bounds for q (t) are obtained from the motion capture data by using the upper and lower bounds across all recorded emotions and an offset by a margin such that the motion is not constrained by these bounds in the final solution. For the generalized velocities q˙ (t) we used the bounds −15.0 ≤ q˙i (t) ≤ 15.0, i = 1, . . . , ndof . For the controls u(t) we chose bounds such that they were large enough to not affect the solution. B. Numerical Solution of the Optimal Control Problem To solve the optimal control problem (15) we use a direct multiple shooting method [4] which is implemented in the software package MUSCOD-II [15]. It discretizes the continuous formulation (15a)–(15e) for both controls and states by dividing the time horizon in M so-called multiple shooting intervals. The states are parameterized as starting values sj for initial value problems defined for each multiple shooting interval j. The controls are discretized by parameters uj for simple base functions on each multipleshooting interval, such as piecewise constant, piecewise linear or spline functions for each interval. To ensure that the resulting states represent a continuous solution additional continuity conditions
Axis
w −1 i
x(tj+1 ; sj , uj ) − sj+1 = 0
Y X Z Y Y X Z
1.5 1.8 0.2 0.8 0.8 1.0 0.1
are formulated. By doing so the functions x (t) and u(t) were replaced by their finite dimensional counterparts s0 , . . . , sM and u0 , . . . , uM −1 . Further discretization of the constraints and objective function leads to an nonlinear optimization problem of variables y = [s0 , u0 , . . . , uM −1 , sM ]T of the form:
TABLE II W EIGHTING COEFFICIENTS TO ACCOUNT FOR THE DIFFERENT JOINT STRENGTHS IN THE REGULARIZATION TERM OF (15a).
The dynamics of the model during the different phases are described using the ordinary differential equations (15b). The individual right-hand side functions f i : Rnx × Rnu →
min y
subject to: g (y ) h(y )
F (y ) ≥ =
0 0
This problem is then solved by using a specially tailored sequential quadratic programming (SQP) method. We used between 25 and 30 multiple-shooting nodes, depending on the duration of the trial, which results in up to 3692 optimization variables, 2856 nonlinear equality and 7527 inequality
constraints. On a Intel i7-3770 the numerical solution of the discretized reconstruction problem for a single trial took up to 10 hours on a single core. VI. RESULTS A. Kinematic Fitting We applied the method described in Section IV on a motion capture database of emotional walking styles. We used the degrees of freedom listed in Table I for the H EI M AN model and mapped the marker data onto the model by solving the inverse kinematics problem (13). To evaluate how well the subject-specific model can reproduce the recorded motion we compute the error in terms of distances between the virtual and recorded marker data. For all recorded trials we can reproduce the recorded markers using the subject-specific rigid multibody model with an average error of less than 1cm and errors of individual markers are less than 4cm. In Figure 6 we show the trajectories of the joint angles that we obtained using the kinematic fit. For each plot we show the average of each emotion as a solid line and the standard deviation as a lightly shaded area. The expressive motions JoyExpressive and AngerExpressive both feature similar trajectories and a much larger range of values in the plots ElbowRight_RY and ElbowLeft_RY compared to the other expressions. This supports the characterizations of [24], where both Joy and Anger are described as ”expansive” and ”high movement dynamic”. Another interesting observation can be made for the abduction of the right shoulder shown in plot ShoulderRight_RX: here JoyExpressive has the largest range of values and also the overall shape of the trajectory is different compared to the other expressions. B. Dynamic Reconstruction As the kinematic reference motion obtained using the kinematic fit does not perfectly fulfill some of the constraints formulated in the dynamics reconstruction problem (15) one will never perfectly replicate the reference motion. However the joint angles that we obtain from the dynamics reconstruction problem are very close to the reference motion as can be seen from Figure 1. The plots of Figure 8 show the reconstructed joint torques of the emotional expressions. The movements of AngerExpressive generally show the largest amplitudes followed by JoyExpressive. Especially for the torqes of joints that perform flexion and extension, i.e. Lumbar_RY, ShoulderRight_RY, ShoulderLeft_RY, ElbowRight_RY, and ShoulderLeft_RY one can observe high levels of actuations for AngerExpressive compared to the other emotions. In Figure 7 we show the reconstructed computed ground reaction forces. The vertical ground reaction forces show the characteristic M-shape observed for human walking. Overall the force profiles are similar to that of reported measurements in the biomechanics literature such as [14], especially those of the X− and Z− axis of the right leg. The forces of the left leg show the forces starting from heel contact to flat
Fig. 6. Trajectories of the joint angles of the kinematic fit. The first three plots are the translations of the pelvis in m, whereas all others are rotations expressed in rad. Solid lines show the mean value for the corresponding emotion and the shaded area represents the standard deviation.
contact. They start with non-zero values as the initial contact is treated as an instantaneous collision in our walking model. One would expect that the forces of the left leg at the end of the step are similar to those of the right leg at the beginning of the step. In our results however this is not the case. This artefact is caused by our choice of gait modeling: as we only optimize a single step instead of a double-step sequence the optimization omits parts of the preparation for the second step. The ground reaction forces of the right leg show the for human walking characteristic M-shape for the vertical forces (Z-axis). The different emotions exhibit changes in the profile similar to those observed for different walking speeds [17]. Specifically the less pronounced valley of the M at slower speeds (AngerSupressive, Neutral, JoySelfsufficient and the higher first bump for higher walking speeds (AngerExpressive, JoyExpressive). VII. CONCLUSION AND OUTLOOK We have demonstrated how the dynamics can be reconstructed from purely kinematic human movement recordings. A key property of our approach is that we use rigid multibody models throughout our reconstruction. From our observations we can justify the use of 1-DOF knee and elbow joints to model human movement during walking. The ground reaction forces that we reconstructed are
Furthermore we want to thank the Simulation and Optimization research group of the IWR at Heidelberg University for giving us the possibility to work with MUSCOD-II .
R EFERENCES
Fig. 7. Reconstructed ground reaction forces normalized by the subject’s body weight. Solid lines show the mean value for the corresponding emotion and the shaded area represents the standard deviation.
Fig. 8. Trajectories of the normalized joint torques of the reconstructed dynamics. The torques are normalized with respect to the subject’s body weight. Solid lines show the mean value for the corresponding emotion and the shaded area represents the standard deviation.
plausible and show phenomena observed in experiments that used force plates. Nevertheless a proper validation using data that contains force plate data remains to be performed. Our approach so far requires a manual step to create the subject-specific models. An advantage of this is that we are independent of the used marker set during the recording. However automating this step for widely used marker sets such as the VICONTM Plug-In-Gait would facilitate the model creation for motion databases that used this set during recording. ACKNOWLEDGMENT The authors gratefully acknowledge the financial support and the inspiring environment provided by the Heidelberg Graduate School of Mathematical and Computational Methods for the Sciences (HGS), funded by DFG (Deutsche Forschungsgemeinschaft) and the European FP7 project KOROIBOT (grant number 611909).
[1] Frank C. Anderson and Marcus G. Pandy. Dynamic optimization of human walking. Journal of Biomechanical Engineering, 123(01480731 (Linking)):381–390, 2001. [2] Uri M Ascher, Hongsheng Chin, Linda R Petzold, and Sebastian Reich. Stabilization of constrained mechanical systems with daes and invariant manifolds. Journal of Structural Mechanics, 23(2):135–157, 1995. [3] Maarten F. Bobbert, Henk C. Schamhardt, and Benno M. Nigg. Calculation of vertical ground reaction force estimates during running from positional data. Journal of Biomechanics, 24(12):1095 – 1105, 1991. [4] H.G. Bock and K.J. Plitt. A multiple shooting algorithm for direct solution of optimal control problems. In Proceedings 9th IFAC World Congress Budapest, pages 243–247. Pergamon Press, 1984. [5] Roy B Davis, Sylvia Ounpuu, Dennis Tyburski, and James R Gage. A gait analysis data collection and reduction technique. Human Movement Science, 10(5):575–587, 1991. [6] Paolo de Leva. Adjustments to Zatsiorsky-Seluyanov’s segment inertia parameters. Journal of Biomechanics, 29 (9):1223–30, 1996. [7] N. Doriot and L. Cheze. A three-dimensional kinematic and dynamic study of the lower limb during the stance phase of gait using an homogeneous matrix approach. Biomedical Engineering, IEEE Transactions on, 51(1):21–27, Jan 2004. [8] Roy Featherstone. Efficient factorization of the joint-space inertia matrix for branched kinematic trees. The International Journal of Robotics Research, 24(6):487–500, 2005. [9] Roy Featherstone. Rigid Body Dynamics Algorithms. Springer, 2008. [10] Martin L. Felis. Modeling Emotional Aspects in Human Locomotion. PhD thesis, Heidelberg University, July 2015. [11] Ga¨el Guennebaud, Benoˆıt Jacob, et al. Eigen v3. http://eigen.tuxfamily.org, 2010. [12] Roberto Ierusalimschy, Luiz Henrique de Figueiredo, and Waldemar Celes. Lua 5.1 Reference Manual. Lua.Org, 2006. [13] Adam G. Kirk, James F. O’Brien, and David A. Forsyth. Skeletal parameter estimation from optical motion capture data. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) 2005, pages 782–788, June 2005. [14] Song Joo Lee and Joseph Hidler. Biomechanics of overground vs. treadmill walking in healthy individuals. Journal of Applied Physiology, 104(3):747–755, 2008. [15] D.B. Leineweber. Analyse und Restrukturierung eines Verfahrens zur direkten L¨osung von Optimal-Steuerungsproblemen. Master’s thesis, Ruprecht-Karls-Universit¨at Heidelberg, 1995. [16] Urbano Lugr´ıs, Jairo Carl´ın, Rosa P`amies-Vil`a, Josep M. FontLlagunes, and Javier Cuadrado. Solution methods for the doublesupport indeterminacy in human gait. Multibody System Dynamics, 30(3):247–263, 2013. [17] Kei Masani, Motoki Kouzaki, and Tetsuo Fukunaga. Variability of ground reaction forces during treadmill walking. Journal of Applied Physiology, 92(5):1885–1890, 2002. [18] Matthew Millard, John McPhee, and Eric Kubica. Multi-step forward dynamic gait simulation. In Multibody Dynamics, pages 25–43. Springer, 2009. [19] Motion Lab Systems. The c3d file format user guide. Technical report, Motion Lab Systems, 2008. [20] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer Science+Business Media, LLC., 2006. [21] Lei Ren, Richard K Jones, and David Howard. Whole body inverse dynamics over a complete gait cycle based only on measured kinematics. Journal of Biomechanics, 41(12):2750–2759, 2008. [22] Ajay Seth, Michael Sherman, Jeffrey A. Reinbolt, and Scott L. Delp. Opensim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange. Procedia {IUTAM}, 2:212 – 232, 2011. {IUTAM} Symposium on Human Body Dynamics. [23] Tomomichi Sugihara. Solvability-unconcerned inverse kinematics based on Levenberg-Marquardt method with robust damping. In 9th IEEE-RAS International Conference on Humanoid Robots December 7-10, volume 27, pages 984–991, 2009. [24] Harald G. Wallbott. Bodily expression of emotion. European Journal of Social Psychology, 28(6):879–896, 1998.
