Aug 30, 1989 - Dynamic Pattern Recognition of Coordinated. Biological Motion. H. HAKEN,* J. A. S. KELSO,* A. FUCHS, AND A. S. PANDYA*. Universitfit ...
Neural Networks, Vol. 3, pp. 395-401. 1990
1~893-6080/90 $3,(1tl + ,(10 Copyright ,c 1991/ Pergamon Press pie
Printed in the USA. All rights reserved.
ORIGINAL CONTRIBUTION
Dynamic Pattern Recognition of Coordinated Biological Motion H.
HAKEN,* J.
A. S. KELSO,* A. FUCHS, AND A. S. PANDYA*
Universitfit Stuttgart and *Florida Atlantic University
(Received 30 August 1989: revised and accepted 30 November 1989)
Abstract--We develop an algorithm that enables the identification and categorization of visually created patterns of coordinated biological motion, specifically, different multijoint limb trajectories produced by humans. The algorithm uses identified collective variables or order parameters as the basis for encoding these patterns, obtained through experimental studies of phase transitions by Kelso and colleagues. Thus, meaningful information for recognizing dynamic visual patterns resides in attractors of the order parameter dynamics. In a neural net, these order parameters represent different macrostates of the net as a whole (rather than the interactions of single neurons), thereby constituting a synergetically organized neural field, in the fashion of a Gestalt-like process.
evant macroscopic observables that characterize these dynamic, time-varying behavioral patterns. Here we employ a synergetic (Haken, 1977/1983) or dynamic pattern strategy (Kelso & Sch6ner, 1987; Sch6ner & Kelso, 1988) to identify, via a phase transition methodology, the collective variables or order parameters that characterize a particular multi-degree-of-freedom biological motion (see sections 3 and 4). Then we use the identified order parameter as the basis for an algorithm that allows for the recognition and disambiguation of multiple dynamic patterns (section 5). Thereby, in the language of synergetics we demonstrate a parallel between processes of pattern formation and pattern recognition that may go beyond analogy (Haken, 1979; Kelso, 1989).
1. INTRODUCTION
Pattern recognition and associative memory are among the chief targets of interest in artificial neural network research. By and large, the patterns to be recognized are normally static, that is, faces or scenes, although some attention has been devoted to dynamic event recognition, most especially in the case of speech. The reasons for this predisposition to static forms may be several. The area of perception itself has dealt largely with static "snapshot" displays, with motion often considered a source of complication. Yet there may be a remarkable simplicity in dynamic patterns of motion which can be exploited in perception. Presumably, if such simplifying constraints were found and incorporated into neural networks for pattern recognition, speed and efficiency of processing would be greatly enhanced. The dynamic patterns produced by animals and people are a case in point. As perceivers, we can readily tell if a person is walking or running; even a limp is quickly picked up (see also section 2). The information for perception in such situations cannot be arbitrary with respect to the structured or patterned motion produced by animals (e.g., in gaits). A major question, nevertheless, concerns the identification of the tel-
2. PERCEPTION OF STRUCTURE IN BIOLOGICAL MOTION
Classical demonstrations by Johansson (1973) attest to the fact that the visual system can extract rather abstract information from optics. Using only lights m o u n t e d on the joints of a human, Johansson showed that perceivers could easily determine whether the (otherwise invisible) person was walking, climbing, performing press-ups, and so forth. Cutting and colleagues, in an extensive set of studies, using "point light walker displays" showed that even the gender of a walker can be accurately perceived from the relational structure among the lights alone, with all other cues removed (see Cutting & Proffitt, 1981, for a review). Similarly, viewers can recognize themselves and others from these time-varying dis-
The research reported here is supported by NIMH (Neurosciences Research Branch) Grant MH 42900-01 and contract N00014-88-J-1191 from the U.S. Office of Naval Research. Financial support was also provided by the Volkswagenwerk Foundation within the project on synergetics. Requests for reprints should be sent to H. Haken, Institut for Theoretische Physik und Synergetik, Universit~it Stuttgart, Pfaffenwaldring 57, 7000 Stuttgart, FRG.
395
~96
H. Haken et al.
plays which carry, in some sense, an individual's signature. Runeson and Frykholm (1981, 1983) demonstrated the astonishing ability of perceivers to judge the amount of weight lifted by a person based only on the constellation of point lights changing in time. They hypothesize that unique patterns of dot motion result from mechanical constraints on the lifter's motion (see also Bingham, 1987). Hoenkamp (1978) and Todd (1983) in studies of computer-animated stick figures provided evidence that perception of human gait (walk, run, limp) is influenced by certain variables, such as the movements of the lower leg and ankle joint. But with these exceptions (see also Bingham. 1987), the nature of the perceptual information extracted from biological motion appears a mystery. The only certainty is that the visual system can extract whatever the information is and use it to make decisions remarkably quickly (e.g., =
~ ul=l
g..'v{"'L
(7)
0.25 i
0.25
0.50
0.75
[
FIGURE 4. Initial values of ~o and ~j for the different states of the transition, On the left side the antlphase motion is the predominant one; on the right side the algorithm Indicates inphase motion.
400
It. Haken et al. -
[
-
.
.
m
[
.
.
.
. .
0.75
0.50
0.25
0.25
0.00
0.00
-0.25
-0.25 .
.
.
.
.
.
.
.
I
.
.
.
.
:
0.75
Lf
0.50
.
.
.
.
.
.
0.25
.
3
'
O.
0,75
0.25
L ~a
--
~:i
0.50
0,75
t
- - - e(q)
FIGURE S. Simulations of the dyrmmics (9) for the two caus labeled a) and b) in Flgum 4 (compare text).
The corresponding restored and identified pattern is then given by v~"0). 6. APPLICATION AND RESULTS To test our pattern recognition algorithm in detail we use the simulated data corresponding to the pattern generation experiment described in section 4 (Kelso et al. 1989). Figure 2 defines the angles ~ and ~v2. The time series ~ ( t ) , k = i. 2 are determined from the simulation of the multijoint trajectory experiment (section 4) using the Haken et al. (1985) dynamics where a transition from antiphase motion to inphase motion takes place during the course of time. The time series of the angles as well as the difference between them are plotted in Figure 3. Our goal now is the application of the formalism described in the previous section to these time series. that is, the classification of the movement patterns. Because we wished to distinguish between antiphase and inphase motion, we built two prototype vectors v ~a~and v (0 as described above, using the magnitudes and relative phases of a Fourier-transformation from the first and the last cycle of the time series in Figure 3 at t = 0 and t = 1. The components c C"~and c (0. which label the corresponding patterns, are set to - . 5 and .5, respectively, and the adjoint vectors v (a)÷ and v~0+ defined by (7) were calculated. Then each cycle of the time series was Fouriertransformed, and the vectors q, built up with the obtained coefficients, served as test patterns for our recognition algorithm. In these cases the components c(q) were set to 0. Figure 4 shows the time behavior of the order parameters ~, and ¢~, given as the scalar product between q and the adjoint vectors, during the transition. At the crossover point a switch from the antiphase to the inphase motion is identified thereby illustrating that our procedure is capable of classification. Figure 5 shows the temporal evolution of the (s according to the pattern recognition algorithm described previously (Fuchs & Haken, 1988b). In
Figure 5 (left) the antiphase motion is the predominant one. As we see. the corresponding value of ~o increases to a saturation value of 1. whereas the other one. ~i, referring to the inphase motion decays to zero. The label c runs to the value --0.5. indicating antiphase motion. Figure 5 (right) shows the opposite case in which the recognition system indicates inphase motion.
7. CONCLUSIONS In this paper we have shown how behavioral patterns, specifically visually constructed multi joint limb trajectory patterns, can be recognized and categorized. The algorithm can identify both stable patterns and the transition between the patterns. A significant feature of the present approach is to employ the order parameter concept, in this case the relative phases, as a characterization of the behavioral patterns. Order parameters proved to be an adequate macroscopic observable on which to base the recognition of dynamic patterns. It is rather obvious from our results that the whole approach can be appreciably generalized, for instance to more complicated patterns of biological motion, for example, the point light displays created by walking, running, lifting weights, and so forth. For example, procedures that allow for the recovery of connectedness and three-dimensional structure of Johansson-type displays (cf. section 2) along with parameters that encode the length changes of each moving segment can readily be incorporated into the pattern vector of our neural net.
REFERJENCF~ Bingham, G. (1987). Kinematic form and scaling: Further investigations on the visual perception of lifted weight. Journal of
Experimental Psychology: Human Perception and Performance. 13, 153-177. Cutting, J., & Proffitt, D. R. (198t). Gait perception as an ex-
Coordinated Biological Motion ample of how we perceive events. In R. D. Walk and H. L. Pick (Eds.), lntersensory perception and sensory integration, New York: Plenum. Ditzinger, T., & Haken, H, (1989). Oscillations in the perception of ambiguous patterns. Biological Cybernetics, 61,279-287. Fuchs, A., & Haken, H. (1988a). Pattern formation and pattern recognition as dual processes. In J. A. S. Kelso, A. J. Mandell, & M. F. Shlesinger (Eds.), Dynamic patterns in complex systems. Singaporc: World Scientific. Fuchs, A., & Hakcn, lq. (1988b). Pattern recognition and associative memory as dynamical processes in a synergetic system 1 + 1I, Erratum, Biological Qvbernetics, 60, 17-22, 11/7-109, 476. Grossberg, S. (1980). How does the brain build a cognitive code. Psychological Review, 87, 1-51. Haken, H. (1977/1983). Synergetics, an introduction, 3rd ed. Berlin: Springer. Haken, H. (1979). Pattern formation and pattern recognition-an attempt at a synthesis. In Pattern formation by dynamical systems and pattern recognition. Berlin: Springer, 2-i3. Haken, H. (1987). Synergetic computers for pattern recognition and associative memory. In: H. Haken (Ed.) Computational systems--natural and artificial. Berlin: Springer, 2-22. Haken, H. (1988). Information and self-organization. Berlin: Springer. Haken, H., Kelso, J. A. S., & Bunz, H. (1985). A theoretical model of phasc transitions in human hand movements. Biological Cybernetics, 51, 347-356. Haken, H., & Lorenz, W. (to be published). Hoenkamp, E. (1978). Perceptual cues that determine the labeling of human gait. Journal of Human Movement Studies, 4, 5969. Hoffman, D. D., & Flinchbaugh, B. E. (1982). The interpretation of biological motion. Biological Cybernetics, 42, 195-204. Hopfield, J. J. (1982). Neural networks and physical systems with emergent computational abilities. Proceedings of the National Academy of Sciences, USA, 79, 2554-2558. Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception & Psychophysics, 14, 201211, Kclso, J. A. S. 11981). On the oscillatory basis of movement. Bulletin of the Psychonomic Society, 18, 63. Kelso, J. A. S. (1984). Phase transitions and critical behavior in
401 human bimanual coordination. American Journal of Physiol-
ogy, 246, R1000-R1004. Kelso, J. A. S. (1989). Phase transitions: Foundations of behavior. In H. Haken (Ed.), Synergetics o[ cognition. Berlin: Springer, 249-268. Kelso, J. A. S., & Scholz, J. P. (1985). Cooperative phenomena in biological motion. In H. Haken (Ed.), Complex systems-operational approaches. Berlin: Springer, 124-149. Kelso. J, A. S. & Sch6ner, G. S. (1987). Toward a physical (synergetic) theory of biological motion. ,Springer Proceedings in Physics, 19, 224-237. Kelso, J. A, S., Sch6ner, G., Scholz, J. P., & Haken, H. 11987). Phase-locked modes, phase transitions and component oscillators in biological motion. Physica Scripta, 35, 79-87. Kelso, J. A. S., Wallace, S., & Buchanan, J. (1989). Phase transitions and trajectory formation in single, multijoint limb patterns. Psychology c~f Motor Behavior and Sport. Proceedings of the North American Society for Sport and Physical Activity. Kent State University, June 1-4, p. 33. Kelso, J. A. S., & Pan@a, A. S. (1990). Dynamic pattern generation and recognition. In D. Zeltzer, N. Badler, & B. Barsky (Eds.) Making Them Move. San Mateo, CA: Morgan Kaufmann.
Kohonen, T. (1987). Self-organization and associative memory, 2nd ed. Berlin: Springer. Runeson, S., & Frykholm, G. (1981). Visual perception of lifted weights, Journal of Experimental Psychology: Human Perception and Performance, 7, 733-740. Runeson, S., & Frykholm, G. (1983). Kinematic specification of dynamics as an informational basis for person and action perception. Journal of Experimental Psychology: General, 9(4), 581t-610. Sch6ner, G., & Kelso, J. A. S. (1988). Dynamic pattern generation in behavioral and neural svstems. Science, 239, 15131520. Sch6ner, G., Jiang, W., & Kelso, J. A. S. (1990). A synergetic theory of quadrupedal gaits and gait transitions. Journal of Theoretical Biology, 142, 359-392. Todd, J. T. (1983). Perception of gait. Journal of Experimental Psychology: Human Perception and Per/ormance, 9( 1), 31-42. Webb, J. A., & Aggarwal, J. K. (1982). Structure from motion of rigid and jointed objects. Artificial Intelligence, 19, 107130.
