motion primitives and invariants in monkey scribbling

MOTION PRIMITIVES AND INVARIANTS IN MONKEY SCRIBBLING MOVEMENTS: ANALYSIS AND MATHEMATICAL MODELING OF MOVEMENT KINEMATICS AND NEURAL ACTIVITIES Thesis for the degree of Doctor of Philosophy By Felix Polyakov Under the supervision of Professor Tamar Flash Department of Comuter Science and Mathematics The Weizmann Institute of Science

Submitted to the Feinberg Graduate School of The Weizmann Institute of Science Rehovot, Israel September 9, 2006

ii

To my grandmother Riva and grandfather Boris.

iii

Table of Contents Table of Contents

iv

Acknowledgements

ix

Abstract

xi

1 Experimental procedure and data preprocessing 2 Optimality, geometric invariants and parabolic primitives enable to reveal dimensionality reduction through practice in monkey scribbling movements 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Methods and materials . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Some facts from the equiaffine theory of plane curves . . . . . 2.2.2 The models and equiaffine parameters . . . . . . . . . . . . . 2.2.3 Fitting drawings with parabolic pieces . . . . . . . . . . . . . 2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Necessary and sufficient condition on the path implies unique equiaffinely invariant solution in plane: parabolic paths . . . . 2.3.2 Theoretical generalization to the third dimension . . . . . . . 2.3.3 Predicted trajectories for parabolic paths satisfy the two-thirds power-law and vice-versa, drawing parabolic paths according to the two-thirds power-law minimizes jerk . . . . . . . . . . . . 2.3.4 Predicted trajectories define time-warping . . . . . . . . . . . 2.3.5 Examples of empirical paths fitted with parabolic pieces . . . 2.4 Monkey drawings converge to a low-dimensional representation characterized by clusters of parabolic pieces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Quantification of convergence of the fitted parabolic pieces . . iv

8

13 14 15 16 18 20 22 24 24

26 27 28

29 30

2.4.2

2.5

Parabolas possess equiaffine symmetry and equiaffine symmetry of parabolas is isochronous in the minimum-jerk trajectories with one via-point . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Parsimonious representation of piece-wise parabolic movements with constant equiaffine velocity . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 Analyzing representation of different kinematic parameters in the activity of motor cortical units underlying spontaneous planar scribbling movements 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Tuning relationship . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Regression of the kinematic parameters on the neural data for scribbling movements . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Likelihood of surprise in the firing rate (the method was proposed by Moshe Abeles) . . . . . . . . . . . . . . . . . . . . . 3.2.4 Use of partial cross-correlation analysis in order to resolves ambiguity in the encoding of multiple movement features . . . . . 3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Tuning between kinematic and neural activities: actual tuning curves are similar to Poissonian ones for non-directional movement parameters . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 Directional decomposition of the tuning: equiaffine speed is distributed more uniformly w.r.t. the movement direction than Euclidian speed . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 We did not see evidence for single-cell encoding of the endpoints of parabolic pieces, and the question is ill-posed . . . . 3.3.4 Partial cross-correlation reveals stronger representation of the equiaffine speed than Euclidian one in the activity of some of the motor cortical cells . . . . . . . . . . . . . . . . . . . . . . 3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Decision making should be considered in the study of primitives 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Methods and Results . . . . . . . . . . . . . . . . . . . . . 4.2.1 Reward-related dissociation between the behaviors .

v

40 45 46

51 52 54 54 57 60 62 71

71

73 76

78 93

movement . . . . . . . . . . . . . . . . . .

96 97 98 99

4.2.2

4.3

Movement primitive is a movement entity which cannot be intentionally stopped unaccomplished . . . . . . . . . . . . . . . 110 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

5 Hidden Markov modelling of the neural activity and movement primitives 113 5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.1.1 Hidden Markov modelling . . . . . . . . . . . . . . . . . . . . 115 5.1.2 Data preprocessing for HMM analysis of neural data, description of different sets of parameters of the model and initialization of the learning procedure . . . . . . . . . . . . . . . . . . 117 5.1.3 EM algorithm for HMM with mixture of Poissonians observations119 5.1.4 Segmentation of the neural data given the model . . . . . . . 122 5.1.5 MDL-like estimation of the optimal number of hidden states for HMMs with mixture of Poissonian observations . . . . . . . . 122 5.2 Time-lag between the neural activity at hidden states and the corresponding pieces of drawings . . . . . . . . . . . . . . . . . . . . . . . 124 5.2.1 Estimate of the similarity between two pieces based on the timewarping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.2.2 Fast estimate of the similarity . . . . . . . . . . . . . . . . . . 127 5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 5.3.1 Properties of the learned models . . . . . . . . . . . . . . . . . 128 5.3.2 Finding optimal time-lag, based on geometric properties of the paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 5.3.3 Estimation of the optimal time-lag based on separability of the paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6 General discussion and further questions 6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Smoothness and geometry . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 The principle of greater parsimony . . . . . . . . . . . . . . . 6.2.2 More about merging kinematic smoothness with geometry . . 6.3 Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 Rewarded versus non-rewarded trajectories . . . . . . . . . . . 6.4 Does converges to piece-wise parabolic performance reflect an emergence of corresponding attractors in neural networks in the brain? . . 6.5 Further questions and directions . . . . . . . . . . . . . . . . . . . . .

vi

142 142 143 143 145 146 146 147 148

6.5.1 6.5.2 6.5.3

Preliminary perspective for the hierarchical scheme of ment compositionality; ability to co-articulate . . . . . Is there a syntax of hidden states? . . . . . . . . . . . List of questions . . . . . . . . . . . . . . . . . . . . .

move. . . . 148 . . . . 152 . . . . 153

A Topics from equiaffine differential geometry A.1 Some definitions and properties from affine geometry (Shirokov & Shirokov, 1959) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Curvature and element of arc in the geometry of the r-parametric Lie group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2.1 Element of arc and curvature of curve in the geometry of the equiaffine group (Shirokov & Shirokov, 1959) . . . . . . . . . . A.3 Equiaffine theory of plane curves . . . . . . . . . . . . . . . . . . . . A.3.1 Curves of constant equiaffine curvature (Shirokov & Shirokov, 1959) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Main formulae and the main theorem of curves in space (Shirokov & Shirokov, 1959) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

165 165 167 169 170 172 172

B Properties of a parabolic path w.r.t. the minimum-jerk and the 2/3 power-law models 175 C Paths satisfying simultaneously the minimum jerk and the 2/3 powerlaw models 178 C.1 Derivation of the necessary condition for the path . . . . . . . . . . . 178 C.1.1 Planar curves, eliminating non-parabolic solutions . . . . . . . 181 C.1.2 Spatial curves, eliminating non-screw-parabolic solutions . . . 182 C.2 A sufficient condition o the path . . . . . . . . . . . . . . . . . . . . . 184 D Derivation of the parabola passing through three points and having prescribed tangent direction at one of the points 187 E Why the time of moving to the via point τ , in the minimum-jerk model with one via-point, is close to 1/2, which implies isochronous predictions 195 E.1 Relationship between the via-points and τ . . . . . . . . . . . . . . . 195 E.2 Time of passing through the point, at which tangent is parallel to Ox axis is between τ and 0.5, isochrony is stronger for this point . . . . . 199 F Numerical approaches to the constrained minimum-jerk model 205 F.0.1 Optimization procedure . . . . . . . . . . . . . . . . . . . . . 205 vii

G Minimum-jerk model and equiaffine geometry in the analysis of drawing movements, example and results 212 G.0.2 Example: analyzed parameters on a single movement segment 212 G.0.3 Quantitative characterization of changes in the drawn trajectories with practice . . . . . . . . . . . . . . . . . . . . . . . . . 213 H Additional figures H.1 Chapter 2 . . . H.2 Chapter 3 . . . H.3 Chapter 4 . . . H.4 Chapter 5 . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

viii

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

224 224 242 254 265

Acknowledgements I express my deep gratitude to my supervisor Professor Tamar Flash who introduced me to a rich and inspiring scientific problem. Professor Flash contributed to my professional advancement in many meaningful ways, guided my research, and in every step of sometimes thorny path provided me with her help and support, taught me to conduct the research independently, be open-minded, and to recognize important and secondary aspects of the results. I am very grateful to Professor Moshe Abeles who provided the data for this work, raised important research questions in the course of this study, was a member of my PHD committee, and was co-supervising my work during one year. Professor Abeles was always deeply involved in the research, provided essential help, and was always more than just collaborator. I would also like to thank Professor Shimon Ullman, the member of my PHD committee, for his insightful questions, helpful comments and suggestions. I am grateful to Rotem Drori for collecting experimental data and for her help in the framework of our collaboration. I wish also to thank Eran Stark who shared with me his partial correlation code and for his suggestions. I appreciate several discussions with Professor Daniel Bennequin, that made things clearer for me and inspired new questions. I am thankful to to Dr. Jacob Goldberger for his willingness to collaborate and for his suggestions. I am grateful to Victor Kasatkin with whom I had insightful discussions. I thank Dr. Michael Herrmann for our exchange of ideas and for his interest. I am grateful to Dr. Ronen Sosnik who provided me with a part of his data and was always willing to discuss things and to help. I am thankful to Professor Ronen Basri for useful discussion about estimation of similarity among geometric shapes. I thank Ido Cohn for his involvement into part of this work during his summer project with Professor Tamar Flash. I had a great experience of communication with Professor Mark Nagurka, who, in ix

x

part, was teaching a course in which I was a teaching assistant. I am grateful to Dr. Amir Handzel whose PHD work considered the planar hand movements in the framework of the equiaffine geometry for the first time; and who took part in guiding my Master research in its initial stage. My special word of acknowledgement is to the organizers, faculty, and tutors of the OCNC 2004 course. In particular to Professor Kenji Doya for giving me the opportunity to attend this extremely enriching course, to Dr. Masami Tatsuno who was a very good tutor of the course project, and to Professor Barry Richmond whose involvement and encouragement were essential for success during my work on the course project. My many thanks to Dr. Vadim Ostapenko. I am thankful to Ayelet Akselrod, Dr. Armin Biess, Dr. Assaf Dvorkin, Jason Friedman, Ira Kemelmacher, Dan Kushnir, Uri Maoz, Lior Noy, Marina Ousov-Fridin, Denis Simakov, Dr. Michel Vidal-Naquet, and other student peers who maintained creative and supportive environment in the Vision and Robotics laboratory where I spent most of my awake hours for about 4.5 years of PHD studies.

Abstract How do the Motions of the Body follow from the Will, and whence is the Instinct in Animals?

(Isaac Newton, “Opticks”) What rules govern the construction of biological movements and how are these movements represented in the activity of motor cortical neurons? We attempt to bring together two different approaches used to describe hand movement generation in different studies: a) continuous representation of movement parameters in terms of continuous encoding in neural activities, and in terms of the constraints on movement parameters; b) state-dependent representation of the processes in the brain, and the generation of movements by composition of the movement primitives. Previous studies have suggested two main approaches to the description of empirical regularities of complex movements in a compact mathematical way: (1) mathematical models (e.g. the 2/3 power-law, the minimum-jerk model) and (2) the compositionality of movements based on motion primitives. The interplay between these two approaches has remained an open question. The 2/3 power law, whose validity was demonstrated based on hand drawing and motion perception studies in 2D, is consistent with motion at a constant equiaffine velocity (Flash & Handzel, 1996; Pollick & Sapiro, 1997): the piece-wise constant velocity gain factor of this model equals an invariant of equiaffine differential geometry called the equiaffine velocity. Here, we use equiaffine differential geometry to find common features of two different mathematical models of planar hand motion: the constrained minimum-jerk model and the 2/3 power-law, and relate the findings to movement compositionality. We further study compositionality based on the analysis of the scribbling movements recorded from two monkeys in the consecutive recording sessions, and the motor cortical activities recorded during 8 recorded sessions of well-practiced behavior of one of the monkeys. In an earlier study we derived a necessary condition on paths that comply with both models and showed that the prediction of the constrained minimum-jerk model for parabolas satisfies the 2/3 power-law, and that the predicted movements have constant acceleration. Here we show that the

xi

PHD thesis by Felix Polyakov

Abstract

xii

condition is also sufficient for short enough pieces of a solution and prove that parabolic segments constitute the only solution of the necessary condition, which is invariant under arbitrary equiaffine transformations. Other studies showed that parabolas have zero equiaffine curvature (e.g. (Shirokov & Shirokov, 1959)) and are geodesics of the equiaffine plane (Blaschke, 1923; Calabi et al., 1996; Handzel & Flash, 1999a). We generalize our geometric result to 3D, which, in particular, enables us to reveal that parabolic screw line (or cubic parabola) constitutes the only class of equiaffinely invariant 3D curves whose prediction w.r.t the constrained minimum-jerk model has constant spatial equiaffine velocity. Cubic parabola has zero spatial equiaffine curvature and zero equiaffine torsion (Shirokov & Shirokov, 1959). We analyze monkey planar scribbling movements and (for part of the data) the underlying motor cortical activities. The recordings were performed in sequences of consecutive recording sessions. The monkey was rewarded when it hit an invisible randomly chosen target (out of 19 targets). We use parabolic segments in order to approximate monkey drawings and show dimensionality reduction of such representation: the orientations of parabolic pieces fitted to the drawn paths converge to a small number (2-4) of clusters with practice. The convergence of the monkey drawings to piece-wise parabolic lower-dimensional representation may provide evidence for the acquisition of simple geometric primitives and their compositionality rules. We propose a simple scheme for the generation of piece-wise parabolic paths from a single parabolic template, by means of equiaffine transformations. This scheme, based on the geometric feature common to the two mathematical models, provides a parsimonious representation of the stereotypical trajectories. We also propose a general principle: the motor system aims to achieve a more parsimonious representation of the motor plan with practice. We argue that maximal smoothness and greater parsimony may be two independent objectives of the motor system, that culminate simultaneously in piece-wise parabolic performance with close to constant equiaffine velocity. Concerning our 3D theoretical result, we hypothesize that similarly to the empirical observation in 2D, stereotypical pieces of the cubic parabola may emerge through practice of spontaneous movements in 3D and thus provide parsimonious representation. We take a novel axiomatic framework for study and formalization of the notion of movement primitive by considering (1) decision-making during task performance and (2) indivisibility (existence of the point of no return). Assuming tunable movement primitives, the decision-making process stands behind the choice of parameters of a primitive and the way different primitives are composed. We show that the monkey indeed makes decisions based on getting/not getting reward. By locking the activity of single neurons on the event of getting a reward, the existence of motor cortical neurons that alter their activity with respect to the event is shown. All in all we propose that taking decision-making into account is important in studies of the nature of movement primitives.


Abstract

xiii

We discuss the following definition of movement primitives1 : movement primitive is a movement entity, which cannot be intentionally stopped unaccomplished after its initiation. During a number of recording days, after getting a reward at specific locations, the monkey tended to stop along parabolic-like path, which supports our claim that parabolas may be movement primitives that emerge following extensive practice. We have segmented multi-cellular neuronal activities recorded during 8 sessions in an unsupervised way based on hidden Markov modelling (HMM). In one recording session, 28jun00, 1) the segments are relatively similar and meaningful (curved and long) movements resembling parabolas, 2) transitions between some of the states are related to the time of getting a reward. These findings support our proposition that parabolas may be movement primitives and that decision-making (switch of model state in our case) is represented in the activity of motor cortical cells with the decision of which movement primitive to choose. In other days, the segmentation also resulted in relatively long and curved movements, though in the above-mentioned day 28jun00 the segments were most similar and only they complied with the decision-making hypothesis. The duration of the hidden states was shorter than that reported by Gat et al. (1997), and for this reason we did not test the state dependent pairwise cross-correlations. The results also show that motor cortical activities can be segmented into several states. A number of studies of the motor system suggest that a large fraction of primary motor cortical neurons represent simple Euclidian movement-related kinematic quantities in their time-varying activity patterns (e.g. movement direction). However equiaffine velocity, a non-Euclidian empirical invariant, was found valid in both execution and perception of planar movements. Here we examine on a single cell level in what metrics the neural representation is stronger: Euclidian or equiaffine. Using a novel approach to disambiguate the correlation of the interdependent kinematic parameters with the neural activities based on partial cross-correlation (Stark et al., 2006), we bring evidence for the existence of motor cortical units which represent the equiaffine velocity more strongly than the tangential one.

Additionally, we present a number of techniques that we have used to investigate

the neural representation of free planar scribbling movements. Altogether, both behavioral and neurophysiological analyses support the notion that parabolic segments constitute geometrically defined motion primitives subserving the construction of wellpracticed scribbling movements.

Convergence to parabolic clusters may reflect convergence of

the motor cortical activities towards dwelling in stable configurations of activity (“attractors” or “states”). We have also provided the first direct evidence that equiaffine velocity may be neurally 1

Proposed during DIP meeting, 2005, following presentation of the analysis that is a part of this thesis.


Abstract

encoded during the generation of complex planar movements.

xiv

Introduction What can we learn about the principles that underly the planning, generation and neural representation of free planar hand movements from the computational analysis based on the formalized empirical invariants of such movements and experimentally collected kinematic data and motor cortical activities? Firstly we outline the related common knowledge and ask corresponding questions that we have attempted to answer in this work. Then we discuss the specifics of the data we analyze, and our original ideas. Finally, we review some of the properties of the primary motor and premotor cortexes. Each of the following chapters (2, 3, 4, 5) contains its own more specific Introduction.

Empirical constraints led us to parabolic primitives If we turn to the human activity — conscious, but not following the rules of formal logic, i.e. intuitive or seme-intuitive activity, for example to motor reactions, we will find out that high perfection and sharpness of the mechanism of continuous motion is based on the movements of the continuous-geometric type ... One can consider, however, that this is not a radical objection against discrete mechanisms. Most likely the intuition of continuous curves in the brain is realized based on the discrete mechanism.

(Andrey Kolmogorov “Mathematics — science and profession”, 1988) Earlier studies (Lacquaniti et al., 1983; Flash & Hogan, 1985) suggested that planar hand movements satisfy constraints, which enable prediction of planar hand trajectories given partial description of their kinematics. A number of different models were proposed in later works, though the 2/3 power-law (Lacquaniti et al., 1983) and the minimum-jerk model (Flash & Hogan, 1985) are most intuitive and provide an explicit solution. Since these two models are purely kinematic, they imply that the hand movements are preplanned in kinematic, and not in dynamic, variables. The present work is based on these two models. The 2/3 power-law was introduced in the beginning of 80’s. In the beginning of 90’s its relevance to the perception of planar hand movements was shown (Viviani & Stcucchi, 1992). Almost a decade after the 2/3 power-law was mentioned for

1


Introduction

2

the first time, its relationship to non-Euclidian (equiaffine2 ) metric was discovered (Flash & Handzel, 1996; Pollick & Sapiro, 1997). Use of equiaffine differential geometry enabled us to establish a necessary geometric condition for the movement path, which complies with both models, namely the 2/3 power-law and the minimum-jerk model (Polyakov, 2001; Polyakov et al., 2001). We also showed that the prediction of the constrained minimum-jerk model (Todorov & Jordan, 1998) satisfies the 2/3 power-law for parabolic paths. Analysis of the equiaffine parameters of monkey planar scribblings indicated that those parameters were converging with practice towards the values characterizing parabolas (Polyakov, 2001; Polyakov et al., 2001). Parabolas have special properties in equiaffine geometry (see Methods in chapter 2). Our findings related to parabolas motivated us to investigate further the involvement of movement primitives and equiaffine metric in planning, encoding and execution of planar hand movements based on the same type of kinematic data and the underlying motor cortical activities.

More about movement primitives Plausible compositionality of movements represents different type of constraints on the performance. Consider the simplest possible motor invariant — planar point-topoint movements. They are known to be almost straight, with a bell-shaped speed profile, and isochronous (i.e., the duration of such movements are invariant with respect to changes of the movement extent). These properties are captured by a single optimality principle (the minimum-jerk model) (Flash & Hogan, 1985). A sudden change in target location brings about the modification of a point-to-point arm movement. It appears that trajectory modification may involve the vectorial summation of a new plan to the old trajectory plan (Flash & Henis, 1991; Flash et al., 1996). This, therefore, suggests a straight movement with a bell-shaped speed profile as a plausible geometric and kinematic movement primitive with the vectorial summation as a simple rule of compositionality. Geometrically defined movement elements, of which complex movements are composed, might be acquired through extensive practice in monkeys (Polyakov et al., 2001) and humans (Sosnik et al., 2004). 2

Briefly, the term ‘equiaffine’ is related to the objects that are invariant under the affine transformations that preserve area. See the background of the equiaffine geometry in Chapter 2 and in Appendix A.


Introduction

3

Continuous representation of movement parameters in motor cortex Numerous studies have reported representations of different movement variables (e.g. direction, speed, position) in activities of the motor cortical cells, including multiplexing of the variables. A list of different reports can be found in Todorov (2000) (see also introduction to chapter 3). A correlation between movement parameters (e.g. direction of the movement velocity and direction of the movement acceleration) at different time-lags implies ambiguity in observed correlations between the neural activities and those parameters (Todorov, 2000; Stark et al., 2006). The argument that the direct activation of muscle groups by motor cortical activity implies correlation of cortical population output with hand kinematics (Todorov, 2000) caused public discussion (Moran et al., 2000) concerning whether kinematic or dynamic variables are encoded in the motor cortical activities, and the statistical procedures used in the data analysis. The partial cross-correlation method has been applied recently to the data recorded in M1 and PMd areas of monkey tracing complex shapes after moving short snake-like stimulus (Drori, 2005; Stark et al., 2006). The method enabled to disambiguate correlation of the neural activities with a number of interdependent kinematic parameters. Interestingly, the majority of the analyzed neurons represent only one parameter. The represented parameters were: movement speed, direction and magnitude of acceleration; and the most widely represented was movement direction (Drori, 2005). Both the vectorial summation of the point-to-point primitive movements in the double-step paradigm and the representation of movement direction in the neural activity that controls the produced movements provide a clear evidence for representation of the movements in Euclidian metric. However, another empirical constraint, the 2/3 power-law, is directly related to the equiaffine metric. In this work we test the possibility for the representation of the equiaffine velocity in the neural activity.

Discrete (state-wise) representation of movements in motor cortex Gat et al. (1997) applied hidden Markov modeling (HMM) to the simultaneous activity of several cells recorded from the frontal cortex of behaving monkeys. The authors were able to identify the behavioral mode of the animal and directly identify the corresponding collective network activity. They also showed that the segmentation of the data into discrete states provided direct evidence for the state dependence of the short-time correlation functions between the same pair of cells. Earlier work (W.Wu et al., 2002, 2003) applied a Kalman filter to the decoding of the multicellular neural activity underlying slow tracing hand movements. The neural


Introduction

4

activity was modelled as a probabilistic observation of the state, and the state was in turn represented as a vector composed of the kinematic parameters: position, velocity and acceleration. In this thesis, the multicellular motor cortical activity is segmented in an unsupervised way (without any knowledge of the movement kinematics), which is performed by means of HMM in a way similar to (Gat et al., 1997) (see chapter 5 of the thesis). The segmentation of the cortical activity into discrete states implies corresponding segmentation of the movement trajectories and enables to test the behavioral output of plausible movement primitives.

Specifics of the analyzed data rise new questions Other studies of motor cortical activity underlying hand movements involved tightly restricted behavior with explicitly defined goal, e.g. center-out or tracking movements. In our work, monkey movements were constrained to a horizontal plane, and the animal was free in its choice of movement strategy. Additionally, the average speed of free scribbling movements is larger than the average speed in the centerout movements (as we observed comparing performance of the same subject in both tasks). Scribbling movements are much faster than the movements in a tracking task: in our records the average speed among the segments of active motion is above 18 cm/s, and local maxima reach values of 60-80 cm/s, following a period of practice; in the tracking task where underlying neural activity was used for movement reconstruction (W.Wu et al., 2003), local maxima of the speed of an exemplar movement (figure 3 in (W.Wu et al., 2003)) was less than 4 cm/s. Thus we explore the motor cortical representation of hand trajectories for the novel, unstudied type of movements. The speed of motion was used as a modulation factor in the regression model that related the movement direction and speed to the recorded firing rate of a single cell in both reaching and tracing (along a template) tasks (Moran & Schwartz, 1999; Schwartz & Moran, 1999). Therefore large differences in speed that were observed in different tasks may imply need for different approaches to modelling. In the center-out (Georgopoulos et al., 1982), double-step (Flash & Henis, 1991), and many other experimental paradigms in studies with humans and monkeys, the subject gets an explicit cue for every temporal and spatial element of a movement. Therefore, the movements can be easily decomposed into the point-to-point components. Elementary ballistic movements composing trajectories in the tracking task were studied from the point of view of optimal control theory (Hanneton et al., 1997). The analysis of rapid monkey wrist movement recorded during a one-dimensional tracking task suggested the hypothesis concerning the existence of stereotypical movement primitives from which more complex movements are composed (Fishbach et al.,


Introduction

5

2005). Apparent submovements that underlie the generation of continuous arm motions were identified during experimental studies in patients recovering from a single cerebral vascular accident. Kinematic analysis demonstrated the existence of a submovement speed profile that is invariant and could be detected in the drawing movements of patients with different brain lesions (Krebs et al., 1999). A natural question is how does the subject comprise spontaneous movements, in the absence of any explicit cues?

We propose that decision-making has to be accounted for in the studies of the goal-directed movements, especially in the studies of movement primitives In general, the analysis of the higher nervous activity in cybernetics is at the moment focused on two extreme poles. On one hand, the cyberneticists extensively study classical conditioning, that is the simplest type of the higher nervous activity ... The other pole — is the theory of formally-logical decisions. This type of the human higher nervous activity is amenable to the study with mathematical tools, and research in this field has been advanced rapidly with the development of computers and computational mathematics ... The entire huge space between these two poles — the most primitive and the most complicated mental actions (even so simple forms of the synthesis as, say, the mechanisms of the accurately preplanned geometric movement, which was touched upon above, at the moment are very poorly amenable to cybernetic analysis) — is studied so little, not to say is not studied at all.

(A.Kolmogorov, “Mathematics — science and profession”, 1988) The choice of appropriate submovements in free goal-directed drawings may be based on a decision-making process. The scribblings that we analyze are free and goal-directed: if the monkey modifies its movements according to getting/not getting a reward, the chance of getting a reward increases. The monkey’s behavior is not governed by any explicit cue. How does the monkey choose a specific piece of movement from many different possibilities? Several studies of decision-making related to discrimination of the sensory stimuli are mentioned in the Introduction to chapter 4. One of the claims derived from earlier work is that decision-making is tightly related to the working memory. We test the involvement of decision-making in trajectory formation and test neuronal responses for getting a reward in chapter 4. We posit that decision-making is an essential paradigm in the studies of the movement primitives because the tunable primitives have to be adjusted to the task requirements. We show in chapters 4 and 5 how the interpretations of the results of several of our analyses (related to the question of what are movement primitives) converge when we account for decision-making.


Introduction

6

Amazing previsions of Kolmogorov (see above in this Introduction) about plausible discrete nature of “continuous-geometric” movements and about putting classical conditioning together with the decision-making in the two main poles of the huge space of mental processes (e.g. planning of geometric movements) appear to be within the context of our work.

What is there in the motor cortex? What is the difference between the role of cortical and the phylogenetically older brain structures in movement production? Kittens, decorticated during their first few weeks of life (Bjursten et al., 1976), can be kept alive for years. The causal observer finds it difficult to distinguish their movements from those of cats with intact nervous systems. Complex patterns of behavior, such as searching for and finding water or food, or attacking other individuals, require a sequential recruitment of the motor programs coordinating locomotion in combination with other motor patterns. Such patterns of behavior can be so activated without the participation of the cerebral cortex. The neuronal substrate contained in a forebrain devoid of the cerebral cortex is thus able to produce surprisingly complex, goal-directed patterns of behavior. “In the absence of their forebrain mammals like decerebrate cats and rats can be made to walk, trot, and gallop by activation of different locomotor regions in the brainstem. The movements are well coordinated and accompanied by a largely appropriate equilibrium control. In character, however, the movements are robotlike — neither goal directed nor adapted to the environment. The latter qualities are added by neural structures in the forebrain. The neuronal substrate contained in a forebrain devoid of the cerebral cortex is thus able to produce surprisingly complex, goal-directed patterns of behavior” (Grillner et al., 1997). So, it was shown that the production of accurate and smooth movements requires cortical control. Accurate movements require elaborate visuomotor processing, and it appears to depend on intact corticospinal projections. A transection of the pyramidal tract in cats resulted in an inability to locomote on a ladder (which requires precise foot placement), but it had no effect on overground locomotion on a flat floor (Liddell & Phillips, 1944). The pyramidal tract includes projections from different precentral motor areas that are important in the control of a variety of motor patterns including reaching, grasping and fine motor skills. Some types of locomotion resemble reaching and grasping (Georgopoulos & Grillner, 1989). We analyzed the activity of neurons from the motor and dorsal premotor areas recorded in experiments in macaque mulatta. All premotor areas are interconnected with the motor cortex. There is a substantial convergence onto single motorneurons from a wide area of the motor cortex. All precentral motor areas project to the spinal cord; some projections from premotor areas are as dense as those from the motor


Introduction

7

cortex (Dum & Strick, 1991). These findings have invalidated the earlier belief that premotor areas exert their influence on the spinal cord only by way of the motor cortex. It now seems that the precentral motor areas form a parallel system within each premotor area, like the motor cortex, possessesing a direct communication line with the spinal structures (Grillner et al., 1997). In new sensorimotor conditions, monkeys, as humans, show poor generalization of the learned transformation to other directions in space. Considering the related neuronal changes, the tuning curve of motor cortical neurons with preferred direction close to the learned direction was altered during the course of learning. These cells showed a relative increase in their firing rate, and did so only when the monkeys move toward the learned direction and mainly during the preparation for movement (Paz et al., 2003). Thus motor cortex is involved in visuo-motor transformations.

An overview of the following chapters The experimental procedure and data preprocessing are described in chapter 1. Chapter 2 contains analytical and empirical results concerning kinematics of planar drawing movements. In chapter 3 several methods are presented for the analysis of single cell neural activities and the result concerning which metric is represented more strongly in the motor cortical activities. The data analysis related to the plausible decisionmaking in trajectory formation can be found in chapter 4. The treatment of the issue of movement primitives on the level of neural population (HMM approach) constitutes the content of chapter 5, and includes the evidence for involvement of the decisionmaking in the choice of the appropriate movement primitives. Each chapter contains its own specific introduction. Proofs, detailed explanations, and many examples are available in the Appendices.

Chapter 1 Experimental procedure and data preprocessing The experiments were carried out by Rotem Drori, Zoltan Nadasdy and Yoram BenShaul from the group of Professor Moshe Abeles in the Haddasa School of Medicine, the Hebrew University, Jerusalem. Animal care was in accordance with Israeli laws and approved by the ethical committee for animal experimentation of the Hebrew University. Experimental setup Two female monkeys (O and U, macaca fascicularis, 2.6 and 3.5 kg respectively) were used in this study. Each monkey sat in a primate chair with the left hand restrained and the right hand operating a two-joint low-friction manipulandum. The monkeys were trained to create smooth and continuous scribbling movements (an example is depicted in figure 1.2). During the whole session the monkeys saw nothing but a circular cursor (diameter: 1/0.4 cm monkey O/U) that indicated the position of the hand. Monkey O saw a mirror reflection of the cursor projected on a video screen. The mirror was placed obliquely from the monkeys nose down. The video screen was placed above the monkey, the image was aligned to the working zone. This produced a good matching between the cursor and hand position. Monkey U saw the cursor projected on a horizontal screen that was mounted on the neck level just above the shoulder, immediately above the manipulandum. In order to create continuous scribbling movements the working plane was tiled with a grid of 19 possible targets (monkey O circles, monkey U hexagons, both were 2 cm in radius). In the beginning of each session a single target was randomly chosen, each target was equi-probable. As soon as the monkey happened to enter this invisible target, while exploring the area with the manipulandum, a short beep was produced, a juice reward was delivered and the next single target (invisible for the monkey) was randomly selected. The grid 8


Chapter I

9

A

B 60

40

20

0

−20

−40

−60

−80 −80

−60

−40

−20

0

20

40

60

80

Figure 1.1: (A) Experimental setup. (B) Circles — locations of 19 targets for monkey O. for monkey O is depicted in figure 1.1B. The grid for monkey U and the task sequence are depicted in figure 1.2. In a typical session the monkeys worked for 1.5 - 2 hours and got 800 - 1500 rewards. After training, the monkey was prepared for extracellular recording with an array, containing 8 microelectrodes. Electrodes were inserted through a chamber that remained permanently fixed to the skull. The signals from each electrode were fed into an amplifier, filtered and then fed to a spike sorter, which classifies the spike signals into 1-3 groups according to the spikes shape. Firing times of all units were recorded and stored on a computer. Thus a database which contained the firing times of all units and events was constructed. The x-y position of the manipulandum were recorded at a sampling rate of 100Hz.

Estimation of the parameters Coordinate data were smoothed using a Gaussian filter with a low-pass cutoff frequency of 8 Hz. Velocity, acceleration and jerk of the coordinates of the hand trajectories were estimated using finite difference approximation of the first, second and third order derivatives respectively: f (t + ∆t) − f (t) ∆t )≈ . f˙(t + 2 ∆t

(1.0.1)

Numerical integration was performed using trapezoid rule. Numerical calculations of equiaffine parameters were based on the methods developed in (Calabi et al., 1998),


Chapter I

10

B

80

80

40

40

mm

mm

A

0 −40

−40

−80 −120

0

−80 −80

−40

0

40

80

−120

−80

−40

mm

0

40

80

mm

Figure 1.2: Behavioral procedure used in the experiment: grid of possible targets (the hexagoal

Tangential velocity, m/s

grid was used in experiments with monkey U) and an example of scribbling produced by the monkey. The gray hexagon indicates the current single target. Both trajectory and the grid were invisible for the monkey. The only visual stimulus was the cursor (yellow circle), which indicated the hand position in real time. A. The monkey is near the target. B. As soon as the monkey enters the target (which is invisible, so it gets an audio confirmation by a short beep and is rewarded, which does not necessarily destroy the continuity of monkey’s performance), another target is selected randomly.

rest motion threshold

0.5

0.4

0.3

0.2

0.1

0 0

1

2

3

4

5

Time, sec.

Figure 1.3: Three movement segments and four intervals of rest.

6


Chapter I

11

and had been used earlier for the analysis of the human movements (Handzel & Flash, 1999b). Numerical estimates of the equiaffine parameters: equiaffine velocity and equiaffine curvature possess irregularities and have to be regularized. The regularization procedure was developed and described in (Polyakov, 2001).

Movement segments Segmentation of the monkey scribbles The monkey spontaneously altered between periods of rest, with no or very slow motion, and periods of active drawing. Movement segments were detected by the following procedure. 1. We found segments in which the tangential speed was above the threshold 15 cm/s. 2. Portions of the trajectory in which the tangential speed was slower than the threshold for at least 200 msec are considered as rest. We identified the segments of active motion as separated by the segments of rest. 3. We prolonge the identified movement segments for 100 msec (forwards and backwards in time), or to the closest minima in the tangential speed, whichever comes first. Three exemplar movement segments are depicted in Figure 1.3.

Observed differences in the behavior of the monkeys We note that the analysis of recordings performed from the start of practice, which involved equi-affine parameters, showed similar results for both monkeys (see Appendix G). This was so in spite of the following differences in their behavior. 1. Monkey O mostly scribbled counter clock-wise with its right hand. Monkey U mostly scribbled clock-wise with its right hand. The phenomenon was very clear after a first few days of practice. Both monkeys scribbled with the right hand during analyzed days. Both monkeys operated with their right hands during recordings that corresponded to the start of practice. Not that we have also performed that analysis of well-practiced scribblings of monkey U recorded in consecutive experimental sessions together with the underlying neural activities. During that, advanced, period of practice, monkey U operated with its left hand and the drawings were counter clock-wise.

2. The speed of scribbling of O was noticeably larger than the speed of U, see figure 1.4A.


Chapter I

12

Monkey O A. Weighted average values of the tangential speed, [m/s]

Monkey U

0.35 0.3 0.25 0.2

B. Average time intervals between getting reward, [s] 10 8 6 4 2

Consecutive days of practice


Figure 1.4: A. Average values of the tangential velocity. Averaging was performed on the segments of the trajectories where the analysis of the data was performed. The tangential velocity was increasing on average during initial parts of the practice period. B. Average time intervals between getting reward. The averaging was performed for the segments of active motion. That is, if less than 2 reward were obtained within one segments, the information from that segments was not accounted for. We have also depicted the time intervals between getting reward. The averaging was performed for the segments of active motion. That is, if less than 2 rewards were obtained within one movement segment, the information about the inter-rewarding time from that segments was not accounted for.

Chapter 2 Optimality, geometric invariants and parabolic primitives enable to reveal dimensionality reduction through practice in monkey scribbling movements Abstract Previous studies have suggested two main approaches to the description of empirical regularities of complex movements in a compact mathematical way: (1) mathematical models (e.g. the two-thirds power-law, the minimum-jerk model) and (2) compositionality of movements based on the primitives of motion paradigm. The interplay between these two approaches has remained an open question. Here, we use equiaffine differential geometry to find common features of two different mathematical models of planar hand movements: the constrained minimum-jerk model and the two-thirds powerlaw, and to relate the findings to movement compositionality. Using a necessary condition on paths that comply with both models, we prove that segments of parabolas constitute the only solution of the necessary condition, which is invariant under arbitrary equiaffine transformations. We also show that this necessary condition is also sufficient for short enough solutions. The prediction of the constrained minimum-jerk model for parabolas satisfies the two-thirds power-law, and the predicted movements have a constant acceleration. Parabolas have zero equiaffine curvature and are geodesics of the equiaffine plane. Parabolic segments fitted to free planar scribblings recorded in experiments with two monkeys become clustered into a small number, 3 or 4, of clusters with respect to their orientation (as considered for the normal at the vertex of the parabola) after the monkey has undergone a period of practice. Thus, monkey drawings converge to a piece-wise parabolic lower-dimensional representation. This convergence may provide evidence for the acquisition of simple geometric primitives and for their compositionality rules. Based on the observed tendency of the convergence to low-dimensional representation of movements, we formulate the greater parsimony principle: the motor control system tends to achieve more parsimonious control strategies through practice/learning. Theoretical generalization of our geometric result implies that prediction of the constrained minimum-jerk model for cubic parabola and any of its equiaffine images has constant spatial equiaffine velocity. The class

13


Chapter II

14

of cubic parabolas constitutes the only class of spatial curves with such property. We propose a simple scheme for generation of piece-wise parabolic paths from a single parabolic template, based on equiaffine transformations. This scheme, based on the geometric feature common to the two mathematical models, provides a parsimonious representation of the stereotypical trajectories. We point to parabolas as candidates for being motion primitives and raise a question of the involvement of online decision-making in the process of tuning the parameters of individual primitives. A cubic parabola has zero 3-dimensional equiaffine curvature and zero equiaffine torsion, and thus cubic parabola may also be used for parsimonious representation of 3D movements composed of its pieces.

2.1

Introduction

May a common structure be hidden behind different formalized empirical invariants of the planning and generation of hand movements? (1) Pioneering studies by Lacquaniti et al. (1983), and Flash & Hogan (1985) suggested that planar hand movements satisfy constraints, which enable accurate prediction of hand trajectories from its partial description. (2) Possible compositionality of movements represents different type of constraints on the performance. A Sudden change in target location brings about the modification of a point-to-point arm movement. It appears that trajectory modification may involve the vectorial addition of a new plan to the old trajectory plan (Flash & Henis, 1991). This therefore suggests a straight movement with a bell-shaped speed profile as a plausible movement primitive. Geometrically defined movement elements, of which complex movements are composed, might be acquired through extensive practice in monkeys (Polyakov et al., 2001) and humans (Sosnik et al., 2004). In this study, we attempt to combine the two approaches: (1) mathematical models, which establish relationships among different parameters of hand trajectories and (2) movement compositionality into a few elements. Earlier we showed that drawing parabolas with a constant equiaffine velocity is equivalent to drawing with constant acceleration and zero (minimal) jerk. We also derived a necessary mathematical condition on the movement path for which the prediction of the constrained minimum-jerk model (Todorov & Jordan, 1998) satisfies the two-thirds power-law (Lacquaniti et al., 1983). Based on this condition we prove here that parabolas are the unique common geometric paths that obey both the two-thirds power-law and the minimum-jerk model: only parabolas are invariant solutions of the necessary condition under arbitrary equiaffine transformations of the trajectories. Parabolas have zero equiaffine curvature (see references in Methods) and are geodesics of the equiaffine plane (Blaschke, 1923; Calabi et al., 1996; Handzel & Flash, 1999a). We propose a parsimonious scheme that can be used in generating the piece-wise parabolic paths.


Chapter II

15

Parabolas fitted to monkey drawings provide evidence for the development of a low-dimensional representation of the overtrained free scribbling movements. Convergence to a low-dimensional representation of movements also exemplifies that the primate motor control system tends to achieve more parsimonious representations of motor task through repeated practice. The observed tendency of the convergence to low-dimensional representation (we are not familiar with any empirical result contradicting such a tendency) may be placed in the basis of the motor control theory. At the end of the chapter we apply an analytical approach similar to the one we used in the planar case, to three-dimensional trajectories. Easy calculations lead to the conclusion that the two constraints: minimum-jerk and the constancy of equiaffine velocity can be simultaneously satisfied for piece-wise planar drawings only. Obviously, the planar pieces need to be parabolic. In the “Methods” section we introduce two empirical biological constraints: the minimum-jerk model and the two-thirds power-law, and essential definitions and properties from equiaffine differential geometry. The procedure of fitting parabolic paths to arbitrary drawings is presented. In the “Results” section we present our theoretical and data analysis results related to parabolic pieces as plausible path primitives acquired by primates during extensive practice. Implications of the results are discussed in the “Discussion” section of this chapter.

2.2

Methods and materials

The trajectory which satisfies the two-thirds power-law has constant equiaffine velocity, which is invariant under equiaffine transformations. Equiaffine geometry is used throughout our data analysis. Here we outline the definitions and facts from planar equiaffine geometry, which we use further in the text. We formulate the minimumjerk and the two-thirds power-law models and show the relationship between the two-thirds power-law model and equiaffine geometry. Validation of the relationships between the equiaffine geometry and the two models (the two-thirds power-law and the minimum-jerk) needs a procedure applicable to experimental data: recorded trajectory. Following (Richardson & Flash, 2002; Polyakov, 2001) in Appendix F we present several numerical alternatives for reconstruction of the minimum-jerk trajectory given a path. In Appendix G (mostly based on the results from (Polyakov, 2001; Polyakov et al., 2001)) an example and results of applying modelling techniques to drawing movements are shown.


2.2.1

Chapter II

16

Some facts from the equiaffine theory of plane curves

We will use the following notation: for two planar vectors a = {a1 , a2 } and b = {b1 , b2 }, ¯ ¯ ¯ a b ¯ ¯ 1 1 ¯ a × b := ¯ (2.2.1) ¯ = a1 b2 − b1 a2 . ¯ a2 b2 ¯ Planar equiaffine transformations of coordinates are a special case of affine transformations, by demanding to preserve area: ¯ ¯ ¯ α β ¯ x1 = αx + βy + a ¯ ¯ , with ¯ (2.2.2) ¯ = 1. ¯ γ δ ¯ y1 = γx + δy + b The ensemble of the properties of geometric forms, which are conserved under the equiaffine transformations, forms the contents of equiaffine geometry. Consider the arc of the 4 times continuously differentiable curve r(t) = {x(t), y(t)} with an additional condition: the arc does not contain inflection points, i.e. its curvature is never zero c 6= 0. Then, the value (dot means time derivative) Zt p 3 σ(t) = (r˙ × r¨) dt

(2.2.3)

t0

corresponding to the two points p0 = r(t0 ), p = r(t) of the curve does not depend on the choice of the parameter t. It depends only on the points p0 and p on the curve, does not change under equiaffine transformations of the curve and coincides with the equiaffine arc-length of the curve between the points p0 and p (Shirokov & Shirokov, 1959). Consequently, the equiaffine velocity is defined as the time derivative of equiaffine arc-length: p σ˙ = 3 (r˙ × r¨) . (2.2.4) Taking the equiaffine arc-length σ as a parameter (natural parameterization), and dr denoting the derivative w.r.t. the natural parameter with prime (r 0 = dσ ), we get: (r 0 × r 00 ) = 1 .

(2.2.5)

Differentiating both sides with respect to σ and using the chain rule implies: (r 00 × r 00 ) + (r 0 × r 000 ) = (r 0 × r 000 ) = 0, i.e. the vectors r 0 and r 000 are parallel and hence r 000 + κr 0 = 0 .

(2.2.6)


Chapter II

17

The coefficient k is the equiaffine curvature (Shirokov & Shirokov, 1959). Multiplying the above equation by r 00 from the left and noting (A.3.2), we obtain the proportionality coefficient κ for natural parameterization: κ(σ) = (r 00 × r 000 ) .

(2.2.7)

From above, the formula for arbitrary parameterized curve can be obtained (Shirokov & Shirokov, 1959): IV ... ... 3σ˙ 3 (r˙ × r ) + 12σ˙ 3 (¨ r × r ) − 5(r˙ × r )2 κ(t) = . 9σ˙ 8

Noting (A.3.2), let us define the tangent vector t = r 0 and the normal vector n = r 00 (they are tangent and normal in the equiaffine geometric sense, which is in general different from Cartesian case). Then (t × n) = 1 , and we have formulas analogous to the Frenet formulas: ( t0 = n, n 0 = −κt .

(2.2.8)

(2.2.9)

Equation (A.3.3) together with equality (A.3.2) imply the main theorem of the equiaffine theory of plain curves (Shirokov & Shirokov, 1959). Theorem 2.2.1. The (natural) equation κ = f (σ)

(2.2.10)

defines the plane curve up to an arbitrary equiaffine transformation. The theorem means that whenever we can calculate the equiaffine curvature κ(σ) as a function of equiaffine arc-length for two curves and these functions coincide for both curves: κ1 (σ) = κ2 (σ) for every σ, then one curve can be obtained from the other by some equiaffine transformation. The above theorem justifies the use of equiaffine curvature for classification purposes. Analogous is the property of the curvature in Euclidean geometry. Any two planar curves with the same Euclidian curvature can be transformed into each other with some rotational transformation and shift.


Chapter II

18

Curves of constant equiaffine curvature (conics) 1. 0 < κ = c2 , equation (A.3.3) implies: r=

1 (a sin(cσ) + b cos(cσ)) + r0 , c

(2.2.11)

and c2 (a×b) = −1 by (A.3.2). This is an ellipse with the center at r0 (Shirokov & Shirokov, 1959). 2. 0 > κ = −c2 , we get: r=

1 (a sinh(cσ) + b cosh(cσ)) + r0 , c

(2.2.12)

with the condition c2 (a × b) = 1; this is a hyperbola with the center at r0 (Shirokov & Shirokov, 1959). 3. κ = 0 gives r = aσ 2 + bσ + c

(2.2.13)

with the condition 2(a × b) = −1. This is a parabola with its symmetry line parallel to the vector a (Shirokov & Shirokov, 1959). Knowing that curves of zero equiaffine curvature are parabolas, the theorem A.3.1 yields our straightforward Corollary 2.2.2. Any two segments from arbitrary two parabolas can be aligned with some equiaffine transformation whenever the equiaffine arc-lengths of the segments are equal. A more detailed description of relevant topics from equiaffine geometry can be found in Appendix A and elsewhere (Shirokov & Shirokov, 1959; Guggenheimer, 1977; Spivak, 1979).

2.2.2

The models and equiaffine parameters

Two models were used in our analysis: the constrained minimum-jerk model (Todorov & Jordan, 1998) and the two-thirds power-law (Lacquaniti et al., 1983). Let r(s) = (x(s), y(s)) be a planar curve describing the path of the hand during a particular trial, where s is the Euclidian pdistance along the path. The tangential velocity of drawing the curve is s(t) ˙ = x˙ 2 + y˙ 2 (dot means time derivative and boldface signifies vector quantities). The constrained minimum-jerk model assumes


Chapter II

19

that the temporal profile of the movement with duration T is determined by the scalar function s(t) that minimizes the quantity called jerk 1 J= 2

ZT

©...2 ª ... x [s(t)] + y 2 [s(t)] dt .

(2.2.14)

0

Then the temporal profile t(s) can be obtained as inverse function of the optimal s(t). As can be seen, the jerk J is a measure of change of acceleration. The problem of minimizing the jerk given a path was introduced under the name the constrained minimum-jerk model (Todorov & Jordan, 1998). The formulation of the minimization problem in the constrained minimum-jerk model differs from the original formulation of the minimum-jerk model with viapoint(s) in the input of the model. The constrained minimum-jerk model finds the optimal speed profile that minimizes the jerk over a given continuous path r(s). In the original formulation of the minimum-jerk model (with via-point(s)) (Flash & Hogan, 1985) both the optimal path and the optimal speed were computed, when a discrete set of via points and velocity and acceleration at the end-points were specified. That is the discrete input contains a small point-wise geometric (locations of the end-points and locations of the via-points) and kinematic (velocities and accelerations on the boundaries) parameters. The two-thirds power law model expresses the relationship between the angular velocity A of drawing a given path and the Euclidian curvature c = (x¨ ˙y − 2 2 3/2 y¨ ˙ x)/ (x˙ + y˙ ) of that path, namely: A = K c2/3 ,

(2.2.15)

where K is the velocity gain factor which was shown to be piece-wise constant for human drawings (Lacquaniti et al., 1983). The two-thirds power law can be formulated in terms of equiaffine geometry, noting that K from (2.2.15) is exactly the equiaffine velocity σ˙ (Flash & Handzel, 1996; Pollick & Sapiro, 1997; Handzel & Flash, 1999a): p K = Ac−2/3 = (s˙ c) c−2/3 = 3 x¨ ˙ y − xÿ˙ = σ˙ , (2.2.16) where x and y are coordinates of the trajectory parameterized by time. Hence the equiaffine velocity σ˙ is equal to the velocity gain factor K, and is thus piece-wise constant for a motion that satisfies the two-thirds power-law . The property of the fit to the two-thirds power-law is invariant under equiaffine transformations of the trajectory due to invariance of the equiaffine velocity. Equiaffine curvature κ = x00 y 000 − y 00 x000 (here x and y are parameterized by the equiaffine arc-length, prime denotes derivative with respect to σ) defines the curve up to an


Chapter II

20

equiaffine transformations, and thus can be used for classification purposes. Based on the invariance property of the equiaffine parameters, and on the relationship of the equiaffine velocity to the two-thirds power-law model, equiaffine arc-length σ (or correspondingly equiaffine velocity σ) ˙ and equiaffine curvature κ can be used in the analysis of the drawn trajectories.

2.2.3

Fitting drawings with parabolic pieces

Parabolas play a special role in equiaffine geometry, briefly they have zero equiaffine curvature, they are geodesics maximizing equiaffine length among all short arcs1 with prescribed end-points and tangents at the end-points (Handzel & Flash, 2001). We show in Results that parabolas also play a special role in the Motor Control theory as unique equiaffinely invariant paths on which prediction of the minimum-jerk model fits the two-thirds power-law. Therefore we perform fit of the drawings with parabolas by means of a curve fitting procedure that is invariant under equiaffine transformations and preserves the point of maximal curvature of the fitted parabola. The reason to preserve the point of maximal curvature will become clear further in the text, where we show that the fitted parabolic pieces become clustered according to orientation of the parabolas with practice. Orientation is defined as the normal direction to a parabola at its point of maximal curvature. We have constructed an algorithm which fits parabolic pieces to the path in a consistent way. By consistency we mean that each of the fitted parabolic pieces has maximal possible length, is unique in our procedure (greedy search of the curves with maximal length may give different results for similar inputs) and does not depend on the equiaffine transformations of the path, which preserve the point of maximal curvature of the fitted parabolic piece. Thus, our procedure applies in the same manner to all pieces, with a small amount of noise (violations of uniqueness and maximality of length). The fitting procedure was as follows. 1. Segment the data into segments of motion. The segments of motion are separated by the segments of no motion. 2. Fit parabolas around the points of local maxima of the path curvature because parts of the parabola distant from the point of its maximal curvature can be well approximated by short straight segments and we would not need higher order approximation with parabolas. 1 A locally convex arc Γ(p1 ; p2 ), together with its endpoints p1 and p2 , is called a short arc if no two tangent lines to it are parallel. The equivalent statement in Euclidean geometry is that the total turning angle of the tangent to the curve from p1 to p2 is less than π (Handzel & Flash, 2001).


Chapter II

21

Find maxima of the magnitude of Euclidian curvature on the previously calculated segments of motion for the data filtered with the Gaussian filter, cut-off frequency 4 Hz. The reason we apply here a degree of data smoothing higher than in other cases, when the cut-off frequency was 8 Hz, is as follows. We want to decrease the number of points of maximal curvature, which are not relevant for us — several local curvature maxima of this type could belong to a piece of path fitted with a single parabola. 3. Propagate the best fitting parabolas around these maxima of curvature, unless the deviation of a parabola from the path reaches the threshold = 0.4 units. A parabola can be characterized by 4 parameters: 1) the focal parameter p (in canonical coordinate system parabola satisfies the following equation y 2 = −2 p x); 2) the orientation of the normal at the point of maximal curvature; 3) and 4) the location (x, y) of the point of maximal curvature. Thus, only the two first parameters define a unique parabola, up to a translation (see figure 2.1). An estimate of the deviation of the fitted parabola from the path was performed in the canonical coordinate system in which the best fitting parabola is described by the equation y 2 = −2x, the focal parameter p equals 1 (see example of such a parabola in the canonical coordinate system in figure 2.1). Direct verification shows that the focal parameter p equals the radius of curvature at the vertex of a parabola. This procedure has enabled us to unravel important properties of the data, as shown further in the text. Tuning the threshold for deviation from a parabola and incorporating another criteria for deviation, for example varying the threshold along the parabola, may increase the robustness of the algorithm. Exemplar parabolas fitted to a path segment by the above algorithm are depicted in figure 2.2. Red dashed lines depict the best fitting parabolas according to the algorithm, the green dashed lines depict the parabolas fitted as best fitting second order polynomials for the path parameterized with equiaffine length. In all these cases, the two ways of fitting resulted in pairs of parabolas having a similar orientation and focal parameter. The fit according to the equiaffine length does not involve coordinate transformations (note that this remark is solely related to the fitting procedure and is not related to the propagation step), and therefore is computationally faster and easier to program. We propose that finding the best fitting second order polynomial for the path parameterized with the equiaffine arc-length can be used in further applications which involve a fit of parabolas to planar curves. Note that the fitted parabolic pieces, located in the same columns in figure 2.2 have a similar orientation. There are three columns. We show in the “Results” section that with practice the fitted parabolas become clustered according to their orientation. Path pieces fitted with parabolas cover a large part of the paths corresponding to the


Chapter II

22

2.5 y2 = −2 p x, p = 1 2

1.5

1

0.5

Focus (−p/2, 0) 0

−0.5

−1

−1.5

−2

−2.5 −4

−3.5

−3

−2.5

−2

−1.5

−1

−0.5

0

0.5

1

Figure 2.1: Parabola with orientation of the normal at the point of maximal curvature 00 , focal parameter p = 1 and focus at (−1/2; 0).

segments of active motion (see chapter 1) and occasionally overlap. Examples of such overlap can be seen based on the indices of the path samples, corresponding to the start and end of the fitted parabolic pieces correspondingly (see titles of the plots in figure 2.2).

2.3

Results

Common features of the predictions of the two-thirds power-law and the minimumjerk model were initially considered in (Viviani & Flash, 1995). A solution to the problem of identifying the paths for which the prediction of the constrained minimumjerk model exactly satisfies the two-thirds power-law may give additional insight into important features of hand movements, which might be captured simultaneously by these two different models (Todorov & Jordan, 1998), and thus about the features of movement primitives. The statement of the problem of identifying the paths implies that the common features should be of geometric nature.


Chapter II

23

1791 − 1822

1812 − 1840

1830 − 1895

1885 − 1924

1930 − 1946

1963 − 2005

80 60 40 20 0 −20 −40

80 60 40 20 0 −20 −40 −50

0

50

−50

0

50

1985 − 2046

80

y, [mm]

60 40 20 0 −20 −40 −50

0

50 x, [mm] U17JUL0A.065, 17.79−20.32; segm# 202; T=2.53 s ; σ=159 mm2/3; s=802 mm

Figure 2.2: Parabolic pieces fitted to the path. Black dots depict the path, the dots between green square and red circle depict that part of the path, which is fitted with parabolas. Red - fitted parabolas, green - parabolas fitted according to the equiaffine arc-length, based on the end-points of the pieces from the fitting procedure. Square - start of the piece, circle - end of the piece. Indices of the start and end of the fitted part of the path are shown in the title of each plot. Parabolas in the first and second subplots in the first row overlap, which can be verified by the indices: the index of the end-point in the first subplot is larger than the first index in the second subplot. Note the similar direction of the orientation (direction of the normal at the point of maximal curvature) of those parabolas, which are in the same columns.


2.3.1

Chapter II

24

Necessary and sufficient condition on the path implies unique equiaffinely invariant solution in plane: parabolic paths

Earlier we obtained a necessary condition on a path for which the prediction of the constrained minimum-jerk model satisfies the two-thirds power-law (see derivation in Appendix C) (Polyakov, 2001). 2

2

x000 + y 000 − 2x00 x(4) − 2y 00 y (4) + 2x0 x(5) + 2y 0 y (5) = const ,

(2.3.1)

with prime denoting derivative w.r.t. the equiaffine arc-length σ. The equation (2.3.1) should be considered simultaneously with the condition r 0 × r 00 = x0 y 00 − y 0 x00 = 1 .

(2.3.2)

We show here that this necessary condition is also sufficient for short enough pieces of any solution of the necessary condition (Appendix C). Since the x and y components of parabolas are second order polynomials of equiaffine arc σ (2.2.13), parabolic segments satisfy the necessary condition. As is shown further in text, the prediction of the constrained minimum-jerk model for parabolas fits the two-thirds power-law, which means that parabolic paths belong to the set of the above-mentioned common features of the two models. We prove here (Appendix C) that no other paths than parabolas satisfy this necessary condition under the assumption of invariance under arbitrary equiaffine transformations (arbitrary equiaffine transformations of the solutions still satisfy the necessary condition). Therefore, parabolic pieces are unique common feature of the two models in equiaffine geometry. Taking a derivative of the left hand side of (2.3.1), we get another equation with only two terms: x0 x(6) + y 0 y (6) = 0 ,

(2.3.3)

which has a simple geometric interpretation: tangent to the curve is orthogonal to its 6-th derivative. The equality (2.3.3) is satisfied for a harmonic motion along a circle: x(σ) = a + cos σ, y(σ) = a + sin σ. There may exist other solutions. We did not look for them.

2.3.2

Theoretical generalization to the third dimension

The proposed evidence of convergence of free planar drawings of primates to piecewise parabolic behavior points to parabolas as to movement primitives. Still, planar


Chapter II

25

drawings constitute a very special type of hand movements, and a major part of our hand movements are performed with respect to spacial goals, e.g. to bring the food to the mouth. We can easily generalize our theoretical findings to the 3-dimensional case, that is to find the necessary condition for the 3-dimensional paths that simultaneously satisfy the constrained minimum-jerk model and the condition of constancy of the equiaffine speed. It was proposed earlier that spatial equiaffine velocity may also be conserved, and this conservation law was termed the 1/6 power-law (Pollick et al., 1997). Equiaffine arc-length of a 3-dimensional drawing from time t0 till time t equals Zt σ3 (t) =

... |(r˙ r¨ r )|1/6 dt

t0

... (Shirokov & Shirokov, 1959), where (r˙ r¨ r ) denotes mixed product of the three vectors, or equivalently, the volume spanned by the three vectors. Hence 3-dimensional equiaffine velocity equals ... (2.3.4) σ˙ 3 (t) = |(r˙ r¨ r )|1/6 . Introducing the third component z, in addition to the components x and y into the derivations in Appendix C, we get a similar necessary condition on the path: 2

2

2

µ3 = x000 + y 000 + z 000 − 2x00 x(4) − 2y 00 y (4) − 2z 00 z (4) + 2x0 x(5) + 2y 0 y (5) + 2z 0 z (5) = const , (2.3.5) where prime means derivative with respect to the equiaffine length σ3 . Note that the necessary and sufficient condition for 3D case has to be considered together with the following equality: (r 0 r 00 r 000 ) = 1 . (2.3.6) (r 0 r 00 r 000 ) = 1. Our formulations of the necessary and sufficient condition (in 2D and in 3D) enable us to search for the equiaffinely invariant solution by simple substitution of the polynomial expressions of the coordinates to equations (2.3.2, 2.3.6). Correspondingly, we get a unique class of equiaffinely invariant solution, which is described by spatial equiaffine transformations of the following curve parameterized by σ3 : x = σ3 , y = σ3 2 /2, z = σ3 3 /6. This curve is called parabolic screw line or spatial cubic parabola; it has zero spatial equiaffine curvature and zero equiaffine torsion (Shirokov & Shirokov, 1959), as can be easily seen from their formulae (A.4.5) and (A.4.6). Similar to the planar case, the two invariants: spatial equiaffine curvature and equiaffine torsion define a spatial curve up to an equiaffine transformation. We, as in the case of planar parabola, conclude that any two pieces of a cubic parabola can b aligned by some equiaffine transformation whenever their equiaffine lengths are


Chapter II

26

equal. Whether this property may be used for parsimonious representation of piecewise parabolic screw movements in space should be tested in experimental studies. Similarly to (2.3.3), the necessary and sufficient condition (2.3.5) implies the following necessary and sufficient condition: x0 x(6) + y 0 y (6) + z 0 z (6) = 0 .

(2.3.7)

We remind that the condition is sufficient for short enough pieces of any solution of the necessary condition. The geometric interpretation is the same as in the planar case: r 0 is orthogonal to r (6) . Sufficiency of the condition Although we show that the necessary condition (2.3.3) (2D) or (2.3.7) (3D) is sufficient on short enough pieces of any of its solutions (Appendix C), we can formulate a sufficient condition for the entire curve. Based on the formula for the second variation (C.2.2), one can see that the second variation is strictly positive (for a curve without inflection points) if   r 000 2 − 2r 00 r (4) + 2r 0 r (5) = const0 ≥ 0 £ ¤  min 9r 000 2 + 2r 0 r 000 − 24r 00 r 000 = const1 ≥ 0 , 0≤σ≤Σ

where Σ equals the equiaffine length of the curve. In particular, this “global” sufficient condition is satisfied for any circle. For a parabolic screw line, the sufficient condition becomes: 9r 000 2 + 2r 0 r 000 − 24r 00 r 000 = A2 + Bσ + Cσ 2 ≥ 0 for some non-zero constants A, B, C. Thus, motion with constant equiaffine speed along every short enough piece of the parabolic screw line minimizes the jerk cost for that piece.

2.3.3

Predicted trajectories for parabolic paths satisfy the two-thirds power-law and vice-versa, drawing parabolic paths according to the two-thirds power-law minimizes jerk

Let us consider the motion with constant equiaffine speed along a parabola.2 Coordinates of a parabola x = f (σ) and y = g(σ) parameterized by the equiaffine arc-length σ are always polynomials of σ of degree not higher than 2. Substituting the condition 2

Here we consider the constrained minimum-jerk model, in which movement path is provided prior to the cost minimization (Todorov & Jordan, 1998). It should not be mixed with the classical formulation of the minimum-jerk model, which predicts the entire trajectory given end- and viapoints, velocity and acceleration at the boundaries; its solution was found to be composed of the pieces of 5-th order polynomials of time (Flash & Hogan, 1985).


Chapter II

27

of equiaffine speed constancy: σ = α t + β (with arbitrary constants α and β) into functions f and g, we get that time parameterizations of components x = x(t) and y = y(t) are the polynomials of at most second degree too, from which it follows that 1 J= 2

ZT

¡...2 ... ¢ x (t) + y 2 (t) dt = 0 .

0

Therefore, introducing parameterization σ = α t with α = Σ/T for a parabolic piece with equiaffine length Σ and for total duration T of the motion, results in the unique optimal trajectory with zero cost. Any parameterization of motion along a parabola with non-constant equiaffine speed, will result in non-zero third derivatives of the coordinates w.r.t. time and non-zero — not the optimal — cost. This implies that the predicted trajectories along a parabolas have constant equiaffine speed and thus satisfy the two-thirds power-law. In addition, the motion with constant equiaffine speed along parabola, which satisfies the two-thirds power-law, is equivalent to the motion with constant acceleration (x = x(t) and y = y(t) are the polynomials of time of at most second degree). It can be easily shown that any trajectory described by its x and y components as second order polynomials of time, is either a straight line or a piece of parabola. Therefore, zero cost can be achieved only for motion along straight or parabolic paths with constant acceleration. We present derivations for the current subsection in Appendix B.

2.3.4

Predicted trajectories define time-warping

The actual and predicted trajectories differ from each other with respect to the time of passage through a set of sample points along the path (both trajectories follow the same path and have the same total duration). Let s(t) ∈ [0; L] be the distance drawn during time t (L being the total path length of the segment). The velocity of the recorded movement is never zero within a segment; therefore the velocity of the predicted movement is not zero within a segment either. Thus, the functions ta (s) and tp (s) (establishing the correspondence between locations on the path and time it takes to get to this location for actual and predicted trajectories respectively) are well defined within each segment, and enable to define the time-warping between the actual and predicted trajectories w(s) = tp (s) − ta (s) . The difference between corresponding extremal values of the two predicted speed profiles: (1) versus time and (2) versus sample number divided by the recording frequency easily visualizes the time-warping. We generated an exemplar trajectory


Chapter II

28

with constant “drawing” speed and “sampled” it with a constant frequency. The path of the trajectory is depicted in figure 2.3 A. The time-warping between the generated trajectory and the minimum-jerk prediction for the path is shown in figure 2.3 B. This time-warping can be easily visualized in terms of the drawing speeds of the optimal motion: one speed being parameterized by location on the path, and the other being parameterized by predicted time, as shown in figure 2.3 C.

Position − y, [cm]

0 −0.2 −0.4 −0.5

0

0.5

1

Position − x, [cm]

Time−course. [sec.]

B. Actual and predicted time courses versus sample points

A. Path

Actual Predicted

1.5 1 0.5 0

0

50

100

150

Sample point number

C. Actual and predicted tangential velocity profles for the recorded segment 0.016

(a) actual speed versus time (b) predicted speed versus time (c) predicted speed versus sample points

[m/s]

0.014 0.012 0.01 0.008 0.006 0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

Time for (a, b), (Sample point number) / (Sampling frequency) for (a, c)

Figure 2.3: A. Segment path. By square and circle we denote start and end of the segment respectively. B. Had the drawing of the path in A been performed according to the constrained minimum-jerk model, the time evolution along the segment would look like the presented dasheddotted graph. C. Drawing speeds. The two predicted profiles visualize the time-warping, it is exactly the time difference between time instances when the two graphs, (b) and (c), take the same values, extremal values are clear landmarks.

2.3.5

Examples of empirical paths fitted with parabolic pieces

Properties of parabolas in equiaffine geometry and our theoretical results concerning the relationship between the parabolic pieces, the two-thirds power-law, and the minimum-jerk model, motivated us to further analyze the involvement of parabolas in well-practiced drawing movements. Here we show two examples of the empirical trajectories well fitted with parabolic pieces. In the first example, figure 2.4 A, the trajectory corresponds to the experimental paradigm in which a human subject had


Chapter II

29

to perform drawings with several via-points “as fast and as accurate as possible” (Sosnik et al., 2004). We fitted a trajectory drawn during the third day of practice. In the second example, figure 2.4 B, the monkey scribbling was fitted with two parabolas. The fitted data were recorded in the 15-th day of extensive practice. The fit is very good in both examples. A

B Two parabolas fit the segment path

noaE30003445

40

200 180

0

140

Position − y, [mm]

Position − y [mm]

160

120 100

Second parabola → −40

80 60

First parabola → −80

40 150

200

250 Position − x [mm]

300

350

−80

−40

0 Position − x, [mm]

40

80

Figure 2.4: Examples of the paths that emerge during practice and parabolas fitting these paths. A. Exemplar path that emerged during co-articulation of point-to-point human movements with via-points, as described in (Sosnik et al., 2004). This path can be fitted with two parabolic pieces. B. An example of a path pattern (asterisks) and its approximation with two parabolas (dashed lines), after several days of practice of scribbling monkey. Start and end of the part of the path to be fitted are depicted by the square and circle respectively.

2.4

Monkey drawings converge to a low-dimensional representation characterized by clusters of parabolic pieces

We observed that different parameters, such as the magnitude of the equiaffine curvature and the degree to which the predicted (according to the constrained minimumjerk model) trajectories fit the two-thirds power-law, converge to the values characterizing parabolic paths (Polyakov, 2001; Polyakov et al., 2001, 2003). We also saw numerous examples of monkey drawings that can be well fitted with parabolic pieces, similarly to the examples depicted in figures 2.2, 2.4 B. Different indications


Chapter II

30

of convergence of monkey performance to piece-wise parabolic drawings motivated us to implement a direct test of such a convergence. The results of the fit showed additional phenomenon — namely clustering of parabolic pieces into a few clusters with respect to orientation, more details follow. We fitted the paths drawn by two monkeys with parabolic pieces, as described in the “Methods”. A parabola can be characterized by 4 parameters: 1) the focal parameter p (in the canonical coordinate system parabola satisfies the following equation y 2 = −2 p x); 2) the orientation of the normal at the point of maximal curvature; 3) and 4) the location of the point of maximal curvature. Thus, only two first parameters define a unique parabola, up to a translation, see figure 2.1.

2.4.1

Quantification of convergence of the fitted parabolic pieces

We quantized the parameters p and α (orientation of the outer normal at the vertex of a parabola) of the fitted parabolic pieces and observed that, for both monkeys, with practice parabolic pieces become clustered into 2-4 clusters with respect to α. The exemplar color histograms of the counts of parabolic pieces given pair of values (α, p) are depicted in figure 2.5. One can see that for both monkeys, with practice, the fitted parabolic pieces became clustered with respect to the orientation of their main axes. A careful reader will find the histograms for all consecutive recording sessions (18 consecutive days of monkey O, from 17 consecutive days of monkey U from start of practice, 17 consecutive days of the overtrained movements of monkey U, for which we have neural data) in figures H.1, H.2, H.3. Concerning monkey U, initially the parabolas were getting clustered into two clusters, though after the 10-th day of practice, 24jun99, the accumulation into the third cluster can be observed with simultaneous increase of dispersion in the area of the two clusters from earlier days, e.g. u05jul99, u06jul99, figure H.2. For the overtrained drawings of monkey U, figure H.3, the three clusters w.r.t. orientation are sometimes blurred, and for some days, e.g. 28jun00, 04jul00, 06jul00, 11jul0, 17jul00, the clustering is rather sharp. The resolution of the histograms for the start of practice stage is 100 , figures H.1, H.2; for the overtrained period the resolution of orientation is 200 , figure H.3. Clear changes in the performance during the start of practice that were revealed by fitting parabolic pieces motivated us to look deeper at the parameters of the fitted parabolic pieces. Orientation of parabolic pieces Out of the two translational invariants of parabolic pieces (p and α) we saw a clear convergence of orientation α into a few clusters in the two-dimensional histograms.


31

Day 3, u12jun99. Total#=225

Day 1, 07jul97. Total#=523

Angle

Chapter II

5

5

300

4

4

200

3

3

2

2

1

1

100 0

0

0

Day 15, 17jul00. Total#=5095


Day 16, 04aug97. Total#=8280 400

25

300

150

Angle

300

20

200

10

100 0

100

15

200

50

100

0

20

40

p

Monkey O

60

80

0

5 0

20

40

60

p

Monkey U, start

80

0

0

20

40

60

80

p

Monkey U, overtrained

Figure 2.5: Color histograms of the count of the pairs (α, p) for the fitted parabolic pieces in for a recording session. Upper row corresponds to start of practice, lower row corresponds to the recording sessions following a period of practice. First column: monkey O, second column: start of practice of monkey U, third column: overtrained movements of monkey U, for which we have neural data. One can see that through practice parabolic pieces become clustered with respect to their orientation.

Now let us consider the orientation of parabolic pieces irrespective of their focal parameters. The histograms of orientation for the same days as in figure 2.5 are depicted in figure 2.6. One can see that indeed a few (2-4) clusters emerge during practice of both monkeys, with two clusters are more dominant. Again, careful reader is referred to the histograms for all considered days, figures H.4, H.5, H.6. As can be seen in figure 2.6, well separated clusters of the orientation of the fitted parabolas are almost symmetric and resemble Gaussian shape. The histograms were fitted with the optimal (according to the MDL criterion) number of Gaussians based on the Gaussian mixture model (GMM). The histograms were fitted with mixtures consisting of 2 to 10 Gaussians, and the optimal number of Gaussians was chosen based on the minimal description length (MDL) criterion. The idea of MDL is described in the “Methods” of chapter 5. Here the description length was calculated as

0


15

Chapter II

32

10 9

4 Gaussians u12jun99;

07jul97;

3 Gaussians

8 10

5

7 6 5 4 3 2 1

0

0

700

120

300

300

200

60

40

0

200

150

100 20

100

5 Gaussians

400

250 80

17jul00;

5 Gaussians

100

500

u27jun99;

04aug97;

6 Gaussians

600

350

0

50

0

Figure 2.6: Histograms for orientations of the fitted parabolic pieces. Upper row corresponds to start of practice, lower row corresponds to the recording sessions following a period of practice. First column: monkey O, second column: start of practice of monkey U, third column: overtrained movements of monkey U, for which we have neural data. One can see that with practice parabolic pieces become clustered with respect to their orientation into a few (2-4) clusters, with 2 dominant clusters. The histograms are fitted with the optimal (according to the MDL criterion) number of Gaussians. follows: MDL = −Log likelihood +

1 (3 [# of Gaussians] − 1) log[# of data points] . 2

The number 3 stands for 3 parameters (mean, variance and weight), which define each entry of Gaussian in the model, −1 corresponds to the constraint on the weights: they sum up to 1. Exemplar plots with different number of Gaussians for the same histogram of orientation and the plot with the values of the description length are depicted in figure 2.7. Gaussians from the GMM with the optimal number of mixture components are depicted in figures as can be seen in figures H.4, H.5, H.6. Number of the mixture components is indicated in the y-label of each plot. Consider the example with several GMM-s for the same day, figure 2.7. The MDL criterion points to 6 mixture components, and not two or three, though visually the histogram consists of two sharp peaks and an additional shallow one. Nevertheless two Gaussians, that fit the two sharp peaks are preserved along the models with 3 to 10 mixture components and the only slightly change. We may consider the orientations as being drawn from the two clusters around the two peaks and from the third dispersed cluster, the process being corrupted with some noise. The process of drawing a direction from the clusters


Chapter II

33

is ordered because after 2 or 3 days of practice both monkeys were mostly scribbling in one direction — either clockwise (monkey O) or counter-clockwise (monkey U). Exemplar values of the orientation and the focal parameter of parabolic pieces in the sequences of the fitted parabolas are depicted in figure 2.8. The examples correspond to 4 days, and one can see that orientation changes in an ordered way. Sharp clustering of orientation is seen in day 04aug97, and more blurred clusters are observed in other days. No order is seen in the focal parameter p. We also consider the other 2 parameters that define a parabola beside the focal parameter and the orientation, namely the location of the vertex. In figures H.7, H.8, H.9 we depict locations of the vertices and their orientations for every fifth fitted parabola in all analyzed days. The number of parabolas is decreased to make the data observable. As can be seen, for both monkeys clusters with similar orientations of parabolic pieces that emerge during practice also correspond to similar locations of the vertices. Moreover, some type of radial symmetry is observed in the days following a period of practice. By radial symmetry we mean similar orientations for the vertices that lie on a segment of straight line having the same orientation as those parabolic pieces. The most prominent cluster both with respect to the values of orientation (figures H.4 - H.6) and to the locations of the vertex corresponds to the orientations close to 2700 . The most prominent clustering into 3 clusters takes place for the day 15 of the overtrained movements of monkey U (17jul00), this can also be seen in terms of locations of the vertex in figure H.9. In day 12 (10jul00) in the same figure one can see a horizontal line of vertices, near y = 0, which does not appear in the other days. This may be related to the restrictions in the manipulandum during that day. Based on the convergence of the fitted parabolic segments into a small number of directionally identified cluster, We propose a principle of greater parsimony: the motor control system tends to achieve more parsimonious control strategies with practice/learning. Metric properties of the drawings fitted with parabolic pieces Parabolic pieces fitted to the monkey scribblings enable us to capture topological regularities of the overtrained movements in terms of sequences of parabolic pieces taken from a small number of clusters. Still, does it prove that the movements are piece-wise parabolic? What are the properties of the fitted parabolic pieces as independent movement units? Does local isochrony take place? Median values of several metric and temporal characteristics of the fitted parabolic pieces are depicted in figure 2.11. In the first row median values of the equiaffine lengths of the fitted parabolic pieces are depicted together with the median values of the equiaffine lengths of the path segments fitted with those pieces. During the


34

100

50

150

22jul97; 4 Gaussians


150

100

50

100

50

0 150

100

50


0 150


0 150

100

50

100

50

0

0

150

150

100

50

0 0

50

100

150

200

250

300

350


0 150





150

Chapter II

100

50

0 0

50

100

150

200

250

300

350

100

50

0 0

50

100

150

200

250

300

350

4

−1.74

x 10

−1.75

MDL

−1.76

−1.77

−1.78

−1.79

−1.8

2

3

4

5

6

7

8

9

10

Figure 2.7: Histogram of orientations fitted with GMMs with different number of mixture components, from 2 to 10. The description length is depicted in the lower plot. In this day the optimal number of Gaussians in the model was 5.


Chapter II

Monkey U; day # 15 from start of practice, u01jul99

350

350

300

300

Orientation, [deg.]

Orientation, [deg.]

Monkey O; day # 16, 04aug97

250 200 150 100 50

250 200 150 100 50

0

0

100

100

p, [mm]

p, [mm]

35

50

0 550

560

570

580

590

600

610

620

630

640

50

0 50

650

60

70

80

Parabolic piece #

350

350

300

300

Orientation, [deg.]

Orientation, [deg.]

100

110

120

130

140

150

640

650

Monkey U, overtrained; day #15, 17jul00

Monkey U, overtrained; day #3, 28jun00

250 200 150 100

250 200 150 100 50

50

0

0 100

100

p, [mm]

p, [mm]

90

Parabolic piece #

50

0 550

560

570

580

590

600

610

Parabolic piece #

620

630

640

650

50

0 550

560

570

580

590

600

610

620

630

Parabolic piece #

Figure 2.8: Orientation and focal parameter for the sequences of parabolic pieces. Vertical red lines separate between different movement segments. Orientation changes in an ordered way. The focal parameter does not show any ordered behavior. start of practice both types of values increase for both monkeys (see column 1 and column 2, first row). Noticeable differences between the lengths of the paths and the fitted parabolic pieces take place especially during the first days of practice. These differences are due to a large number of the fits to non-appropriate path pieces. Such path pieces can be fitted with parabolic pieces but they are actually closer to straight drawings produced at a small speed. Two examples of such paths, and the parabolas fitting them are depicted in figure 2.12. The phenomenon of such paths is seen frequently during the first days of the start of practice. We show exemplar path segments fitted with parabolic pieces for all days (one plot for one day) in figures H.10, H.11, H.12. The ‘improvement’ of the paths in terms of their parabolicity can be seen in these figures. The histograms with equiaffine lengths of the paths and of the (best) fitted parabolic pieces from the first five days of practice of monkey O are depicted in figure H.13, first and second column. One can see that the histograms for the equiaffine length of the paths become more similar to the histograms of the


Chapter II


36


60

100

40 20

50

y, [mm]

0 −20

0

−40 −60

−50

−80 −100


Day 16, 04aug97. Total#=1656


100

60

100

40

80 50

y, [mm]

20

60

0

40

−20

0

20

−50

−20

−40

0

−60 −80 −100 −100

−40 −150 −50

0

50

100

−100

−50

0

x, [mm]

x, [mm]

50

100 −60 −80 −100 −100

−50

0

50

x, [mm]

Figure 2.9: Data for monkey O, from start of practice. Blue dots - locations of vertices of every fifth parabola, red segments - orientations of those parabolas.

equiaffine length of the best fitted parabolic pieces, which confirms that drawings indeed converge to a piece-wise parabolic representation. Euclidian lengths of both types of paths are depicted in the second row of figure 2.11. Both types of values are very similar, and their histograms are very similar too, as one can see by comparing the 3-d and the 4-th columns in figure H.13. These histograms correspond to the first 5 days of practice of monkey O. The increase in the equiaffine and Euclidian lengths of the fitted parabolic pieces and of the path pieces being fitted also confirms that monkey drawings become better represented in terms of parabolas. As an additional test of convergence to parabolic representation we estimate the parabolic inconsistency. First we compute the values inci = |σpath,i − σpar.,i |/ max{|σpath,i |, |σpar.,i |} for each parabolic piece i, where σpath,i is the equiaffine length of the path being fitted, and σpar.,i is the equiaffine length of the best fitting parabolic piece. The inconsistency ranges from 0 to 2, and should be small for similar values. The median values of inconsistency, for every day analyzed are depicted in the 6-th (last) row of figure 2.11. The inconsistency decreases during the start of practice in both monkeys. Better visualization is given in the histograms of inconsistency for the first 5 days of practice of monkey O depicted in the third column of figure H.14. A decrease of inconsistency is another justification for the piece-wise parabolicity of the monkey drawings that is developed during practice.

100


Chapter II

O07JUL7A.032, 4.52−5.02

U12JUN9A.046, 25.77−26.66

100

37

50

0

−50

−100 O04AUG7A.256, 5.48−7.61

U27JUN9A.083, 33.28−34.77

100

U17JUL0A.134, 4.98−6.8; segm# 513; T=1.82 s ;σ=70 mm2/3; s=397 mm

50

0

−50

−100 −100

−50

0

50

100

Figure 2.10: Monkey O, from start of practice. Exemplar paths fitted with several parabolic pieces. One example for one recording day. Path pieces (blue) and parabolic pieces fitting them. Red - best fitting parabolic piece, green — parabolic piece defined as the best fitting second order polynomial in equi-affine length measured along the path. Considering local isochrony, we estimate the temporal asymmetry relative to the vertex of the fitted parabolas: asymT,i = |T−,i − T+,i |/ max{T−,i , T+,i }, where T−,i is the duration from the start of the fitted parabolic piece to the vertex, and T+,i is the duration from the vertex to the end of the fitted parabolic piece. We consider the vertex here because we obtained clustering of parabolic pieces with respect to their orientation at the vertex. The histograms of the temporal asymmetry for 5 first days of practice of monkey O are depicted in figure H.14, first column. Median values of the asymmetry for all days are depicted in figure 2.11, third row. Parabolic pieces of both monkeys become more temporally symmetric with practice. An increase in the asymmetry following the 10-th day of practice of monkey U corresponds to the emergence of the third cluster in orientation, (see also figure H.5). The initial and final parts of the duration become more similar, after a few days of practice, as can be seen in figure 2.11, 5-th row. The duration of the parabolic pieces decreases for monkey O, which, together with the increase in the equiaffine and Euclidian lengths of parabolic pieces means a considerable increase in the speed of drawing. We also estimated the metric asymmetry of the fitted parabolic pieces as follows: asymσ,i = (y−,i + y+,i )/|y−,i − y+,i |. The values of asymσ,i were considered only for the case when the two endpoints of the parabolic piece were on different sides of the vertex. Here yi is the y-coordinate of the start of the parabolic piece in the canonical coordinate system, and y+ is the y-coordinate of the end of parabolic piece in the


Chapter II

38

canonical coordinate system. Median values of the magnitudes of metric asymmetry are depicted in the fourth row of figure 2.11. As can be seen, the parabolic pieces become more symmetric with practice. Exemplar histograms with metric asymmetry from the first 5 days of practice of monkey O are depicted in the second column of figure H.14. Vertical red lines in the plots show the median values of the magnitudes of the metric asymmetry. Analysis of equiaffine curvature Equiaffine curvature of a curve is zero if and only if the curve is a parabola. The magnitude of our numerical estimates of the equiaffine curvature of monkey drawings becomes closer to zero with practice (Polyakov, 2001). The numerical estimates of equiaffine curvature oscillate near zero, which is partially due to the noise in the recorded data and corresponding filtering procedure. In figure 2.13 we consider a segment of monkey scribbling with small κ0 : |κ0 | ≤ 0.05 mm−2 . Depicted are: the path, speed, Euclidian and equiaffine curvatures (c and κ), κ0 , and equiaffine signature curve κ0 versus κ. As one can see in the third and fourth rows of the figure, equiaffine curvature and its derivative have their minima below zero and maxima above zero. Simultaneous behavior of κ and κ0 can be observed in the signature curve, first row, second column. As a function of time, the signature curve evolutes counter clock-wise around zero, which confirms that minima of both parameters (κ and κ0 ) are negative and maxima are positive. Actually, oscillations of κ around some constant3 imply oscillations of κ0 around zero because every extremum of κ implies zero of κ0 and both functions are considered continuous. Having κ with negative minima and positive maxima, we may propose that noiserelated oscillations corrupt the values of equiaffine curvature of piece-wise parabolic drawings that are very close to zero. In figures H.15, H.16, H.17 we depict the histograms of the values of local maxima and minima of κ and κ0 for all days of recording considered above, on the segments of motion (separated by the segments of no motion). Four plots correspond to each recording day. Positive and negative values of the extremal values are depicted by red and green respectively. The values in the histograms are scaled so that the value at peak equals 1, the scaling factor is shown in the title of each plot. The median value is depicted by the vertical blue line, dashed-dotted vertical lines depict the range of the maximal absolute deviation (MAD = median[median[data] - data]). The values of the median, MAD and their ratio r = median/MAD are depicted in every plot. By ratio r we estimate the tendency of a function to keep the sign of the corresponding extremum constant, that is to have negative minima or to have positive maxima. The range of MAD around 3

By oscillations of f around a constant C we mean that the local maxima of f are always greater than C and local minima of f are always smaller than C.

Median σ, [mm2/3]


20

Median S. Eucl., [mm]

20

10 0

5

10

Median Asymm. t. Median Asymm. y.

5

10

15

5

10

15

0

0

5

10

15

0

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

10

15

0

5

10

15

0

0.4

0.4

0.4

0.3

0.3

0.3

0.2

0.2

0.2

0.1

0.1

0.1

0

0

5

10

15 Total Initial Final

40 20 0

5

10

15

5

10

15

0

60

60

40

40

20

20

0

5

10

15

0

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

5 10 15 First monkey, from start of practice, day #

0

5

10

15

50 Path Parab.

0.6

5

Path Parab.

100

50 Path Parab.

60

Median Durat

0

10

Path Parab.

100

50

0

Median inconsistency

15

39

20

10

Path Parab.

100

0

Chapter II

5 10 15 Second monkey, from start of practice, day #

Path Parab. 5

10

15

5

10

15

5

10

15

5

10

15

0

5 10 15 Second monkey, days with neural data (after a long period of practice), day #

Figure 2.11: Median values of metric and temporal parameters for all days of recordings. First column: days of monkey O, from start of practice. Second column: days of monkey U, from start of practice. Third column: days of monkey U, overtrained performance. First row: median values of the equiaffine length of the path pieces fitted with parabolas (blue, dotted), median values of the equiaffine length of the fitted parabolic pieces (green, dashed.) Second row: same as in the first row, for Euclidian length. Third row: asymmetry, which estimates deviation from local isochrony for the fitted parabolic pieces. Fourth row: metric asymmetry of the fitted parabolic pieces. Fifth row: duration of the parabolic pieces, their initial part, prior to the vertex, and their final part, following the vertex. Sixth row: parabolic inconsistency. It estimates the deviation of the fitted paths from piece-wise parabolicity.


Chapter II

U12JUN9A.031, 1.34−1.93

U12JUN9A.036, 3.1−3.54

50

50

y, [mm]

100

y, [mm]

100

0

0

−50

−50

−100 −150

40

−100

−50

0

x, [mm]

50

100

−100 −150

−100

−50

0

x, [mm]

50

100

Figure 2.12: Two examples of the path pieces (blue) and parabolic pieces fitting them. Red - best fitting parabolic piece, green — parabolic piece defined as the best fitting second order polynomial in equiaffine length measured along the path. These paths are inconsistent with parabolicity.

median does not contain zero whenever |r| ≥ 1. As can be seen, |r| > 1 for minima of κ and for both types of extremum of κ0 . In case of maxima of κ, for some days the magnitude of the ratio |r| is less than 1, e.g. in figure H.15 (b), day #17, 05aug97. There is also no monotonicity in r for the maxima of κ considering the order of days. In figure H.15 r for the maxima of κ increases and gets beyond 1 after two days of practice, and then it decreases to values below 1 starting from day #10, 23jul97. We thus conclude that although with practice the magnitude of κ becomes closer to zero characterizing parabolic paths (Polyakov, 2001), κ has a bias towards negative values characterizing hyperbolic type of conics (see Methods, page 18). Considering the ratios r for κ0 , in all days |r| was above 1, which implies dominance of positive maxima and negative minima. Therefore the phase portraits of κ, or equiaffine signature curves, on the segments of motion corresponds to oscillating behavior of κ of the form similar to the plot in figure 2.13.

2.4.2

Parabolas possess equiaffine symmetry and equiaffine symmetry of parabolas is isochronous in the minimumjerk trajectories with one via-point

Constant equiaffine velocity of the movement implies proportional changes in the equiaffine length and movement duration. Consider movements with one via-point and no motion on the boundaries, as the movement with the path depicted in figure


Chapter II

41

Signature curve 0.05

Path day U17JUL0A.068; time from 17.53 to 18.15. κ, [mm−4/3]

y, [mm]

50 0 −50 −150 −100 −50

0

50

0

100 150

x, [mm]

−0.05 −0.05

0

0.05

∆κ/∆σ, [mm−2] 0.08

0.42

0.06

0.28

0.04

0.14

0.02

0

Eucl. curvature, [1 / mm]

Speed, m/s

Speed and Eucl. curvature 0.56

0 Eq.−aff. curv. (κ), [mm−4/3], red; eq.−aff. increments ∆σ, [mm2/3], blue

1

0.5

0

−0.5 ∆ κ / ∆ σ ~ κ’, [mm−2] 0.3 0.2 0.1 0 −0.1 −0.2 −0.3 17.4

17.6

17.8

18

18.2

18.4

Time

Figure 2.13: A segment with |κ0 | ≤ 0.05. A part of the movement before and after the segment is presented as well. The path of the segment resembles two parabolas. The signature curve (κ versus κ0 ) resembles phase portrait of the one-dimensional oscillating behavior. Speed and Euclidian curvature are depicted in the second row. In the third row increments of the equiaffine arc-length and equiaffine curvature are depicted. Equiaffine curvature oscillates around zero on the segment, its values are close to zero. In the fourth row κ0 is depicted. It is oscillating around zero on the segment. Segment start is depicted by green circle on the path and by leftmost vertical line in rows 2, 3, 4. Segment end is depicted by red circle on the path and by rightmost vertical line in rows 2, 3, 4.


Chapter II

42

2.4A. No motion on the boundaries implies that the movement will not fit the 2/3 power-law per se because equiaffine velocity on the boundaries is zero and is not zero away from the boundaries, thus is not constant. However, note that local isochrony (duration of subunits of a movement defined by the via-point are approximately constant, when overall movement size is changed) was empirically observed and shown analytically for the via-point of such movements (Flash & Hogan, 1985). Here we 1) prove equiaffine symmetry of parabolic pieces, 2) show that the paths of the minimumjerk trajectories with one via-point and zero motion at the boundaries are very close to parabolas, and 3) show that analytical prediction of the local isochrony for the point that divides by half the equiaffine length of the path is better than the prediction for the via-point. The property of equiaffine symmetry for parabolas is a handy tool for fitting parabolic pieces, as we will show below in the text. Parallel lines cut parabolas into parts with equal equiaffine distances. First we provide more explanations about this symmetry and further we show how this property is related to the minimum-jerk trajectories with one via-point. Interestingly, the paths of the minimum-jerk trajectories with one via-point are very close to parabolic pieces. Let us first show that −−→ Observation 2.4.1. If, for three points F , G, H on the parabola, chord F H is parallel to the tangent to the parabola at the intermediate point G, then equiaffine arc along the parabola from F to G is equal to the equiaffine arc along the parabola from G to H, as depicted in figure D.2 A. This symmetry immediately implies that for any two parallel lines that cut a parabola, the equiaffine lengths of the pieces are equal. For the case depicted in figure D.2 B, equiaffine lengths FP and QH along the parabola are equal. Proof. Let parabola be represented in the canonical form, see figure D.2(b), y 2 = −2px . Then, the chord is parallel to the vector [xH − xF yH − yF ] = [−yH 2 /(2p) + yF 2 /(2p) yH − yF ]. The slope of the chord as for the relationship x = x(y) will be (−(yH + yF )) /(2p). Let us find a point on the parabola, at which the tangent is parallel to this direction, that is it has the slope −(yH + yF )/(2p). We have for the parabola x = −y 2 /(2p) , x0y = −y/p . Hence, equating −(yH + yF )/(2p) = −y/p, we get that yG = (yH + yF )/2. Noting that for a parabola in its canonical coordinate system, equiaffine arc-length changes


Chapter II

43

A

B

6

6

↑y

5

4

H

4

3

3

2

2

1

↑y

5

H

Q

1

G →x

0

G →x

0

−1

−1

P −2

−2

F

F

−3 −9

−3 −8

−7

−6

−5

−4

−3

−2

−1

0

1

2

−9

−8

−7

−6

−5

−4

−3

−2

−1

0

1

2

Figure 2.14: A. If tangent is parallel to the chord then, in the canonical coordinate system, y coordinate of the point of touch (G) is average of the y-coordinates of the chord (F H). This implies that equiaffine lengths FG and GH on the parabola are equal. B. The symmetry described in A immediately implies that for any two parallel lines that cut parabola, equiaffine lengths of the pieces are equal. In this case equiaffine lengths FP and QH along the parabola are equal. √ linearly with coordinate y: σ(y) = σ0 + (1/ 3 p)y, and using that yG is average of yF and yH , we prove the observation. Every equiaffine transformation preserves the parallelism. Equiaffine lengths before and after an equiaffine transformation are the same, which follows directly from the formula for the calculation of the equiaffine arc-length. Therefore, parallel lines cut a parabola, ellipse, and hyperbola (that is all curves with a constant equiaffine curvature) into pieces with equal equiaffine lengths. The above follows directly from theorem A.3.1 (the main theorem of the equiaffine geometry of plain curves). An easy proof shows that there exists a unique parabola for given two points and a third point at which the tangent is parallel to the line defined by those two points. We present in Appendix D a detailed exposition of the properties of parabolas with respect to the input that consists of three points and a tangent at a one of the points. The procedure for constructing such a parabola given arbitrary values of the input is presented there. Parabolas defined by such an input are considered in an arbitrary, and not necessarily a canonical coordinate system. In human point-to-point movements with one via-point, the time of passage through the via-point is close to half of the movement duration (Flash & Hogan, 1985). The minimum-jerk model provides a close prediction of this phenomenon (Flash & Hogan, 1985). We show in Appendix E that a geometric construction based on the equiaffine


Chapter II

44

symmetry of parabolas predicts isochrony, and this prediction is superior to the prediction of the minimum-jerk model with one via-point (that is when the time of passing through the via-point is considered). The ‘tangent parallel to chord’ property can be used in order to fit parabolic pieces to the simulated minimum-jerk paths with one via-point. Two examples of such a fit are depicted in figure 2.15. The minimum-jerk path is depicted by blue. The blue dotted line depicts the tangent at the via-point, and the red dotted line depicts the tangent parallel to the chord. The corresponding parabola is depicted by the red dashed line. In both examples (they have different via-points) the fit is very good. xvp00.06; yvp00.995. Fit according to the tangent parallel to the chord.

xvm01.17; yvp00.555. Fit according to the tangent parallel to the chord.

1.4 Minimum−jerk path Tangent at the via−point Parabola Tangent || to FH

1.2

Minimum−jerk path Tangent at the via−point Parabola Tangent || to FH

1.2

1

G 1 0.8

V G

0.6

y, units

y, units

0.8

0.6

0.4

V

0.2 0.4 0

H

F 0.2

−0.2

−0.4

0

H

F −0.6

−0.4

−0.2

0

0.2

x, units

0.4

0.6

0.8

−1

−0.5

0

0.5

x, units

Figure 2.15: Paths of the minimum-jerk trajectories with one via-point and zero velocity and acceleration on the boundaries can be well approximated by parabolas using iso equi-affinity. The minimum-jerk path is depicted by blue line. Blue dotted line depicts the tangent at the via-point, and the red dotted line depicts the tangent parallel to the chord. Corresponding parabola is depicted by the red dashed line. In both examples with different via-points the fit is very good. We performed the fit for a large set of via-points that fill a dense grid with x coordinates ranging from -1.5 to 1, and y coordinates ranging from 0 to 1. Estimates of the deviation of the parabola fitted based on the iso equi-affinity condition from the minimum-jerk path are shown in figure 2.16. The deviations are very small. Please note that simulations of the minimum-jerk with one via-point are not parabolas, though they are close. The reason for the closeness is as follows. The jerk cost is based on the coefficients of the fifth order polynomials that correspond to the monomials of time of degree not lower than 3. Therefore minimization of jerk means minimization of such coefficients by putting as much energy as possible in the

1


Chapter II

45

Figure 2.16: Estimates of the deviation of the parabola fitted based on the iso equi-affinity condition. x and y coordinates in the plot correspond to the location of the via-point.

coefficients near monomials of degree at most 2 that have zero jerk cost. As is shown in Appendix B, x and y coordinates of the drawing being second order polynomials in time imply parabolic shape.

2.4.3

Parsimonious representation of piece-wise parabolic movements with constant equiaffine velocity

A segment of a parabola can be transformed to any other segment of any parabola with some equiaffine transformation whenever the two segments have the same equiaffine arc-length (our corollary 2.2.2). Equiaffine transformations preserve the equiaffine arc-length and equiaffine velocity, therefore the use of parabolic segments can offer a parsimonious representation of the paths and speed in the following way. It is enough to keep a single template of drawing a parabolic piece with constant equiaffine velocity, any other parabolic movement with the same equiaffine velocity can be obtained by a single equiaffine transformation, that is by 5 scalar values. We have shown that different parabolic pieces composing piece-wise parabolic drawings can be generated in a simple way from a single parabolic template. Note that for the template of a parabolic piece, equiaffine arc-length and duration were the only parameters needed to construct the speed profile given a path, assuming that the motion follows the two-thirds power-law.


Chapter II

46

Imagine that the monkey picks a parabolic segment that is part of a piece-wise parabolic movement from some orientation-based cluster. Each cluster has 3 free parameters, namely the focal parameter and the location of the vertex. Considering parabolas as movement primitives, their parameters have to be adjusted (or tuned) to the task demands. If the monkey adjusts its drawings in order to search in an unexplored (after lastly obtained reward) part of the workspace, its chance to get a reward increases. The monkey may need to perform an online decision making (or tuning of the parameters of an upcoming parabolic segment). Therefore, the representation of primitives in neural activity may be essentially related to the representation of the process of decision making, in which parameters of a small set of primitives are to be tuned.

2.5

Discussion

In this chapter we successfully derive candidates for a movement primitive, both theoretically and empirically. The two approaches that are used in studies of movement planning were combined: (1) the modelling approach and (2) the decomposition of the movement trajectories into elementary primitives. We present a necessary and sufficient mathematical condition for a common geometric template of the constrained minimum-jerk and the two-thirds power law; prove that a parabolic piece is a unique common geometric template that is invariant under arbitrary equiaffine transformations; describe a simple scheme of the generation of complex movements based on equiaffine transformations of drawing a single parabolic piece. Analysis of the changes in monkey drawings recorded over a period of practice shows convergence of the geometric properties of the drawn paths towards those characterizing parabolic pieces (see Appendix G). Here we provide an explicit evidence that overtrained monkey scribbling movements become closer to a piece-wise parabolic type. Parabolas, in particular, constitute the simplest type of non-straight polynomial (in σ) curves that can be used to describe the observed dimensionality reduction. Fitting drawings with elementary shapes like parabolas enabled us to find a low-dimensional description of the behavior in terms of a small number of directionally defined parabolic clusters. In this work, the evidence for parabolas being movement primitives acquired during extensive practice are obtained from different perspectives. In the current chapter we show evidence based on geometric properties of two empirical invariants of planar hand drawings and on geometric properties of experimental data.

Equiaffine geometry and the 2/3 power-law This work extends the two-thirds power-law and the constrained minimum-jerk model in proposing an explicit geometric constraint — parabolic shape (which may be a


Chapter II

47

primitive template), in addition to the constraints on the trajectory formulated in terms of the 2/3 power-law and the maximal smoothness. By constraint we mean a reduction in the number of independent variables, that specify the trajectory. The constraint characterized by the two-thirds power-law (2.2.15) is expressed as piecewise constancy of the velocity gain factor K (Lacquaniti et al., 1983). Earlier works proposed that piece-wise constancy of K may reflect segmentation of apparently continuous movements (Viviani & Cenzato, 1985) and raised a discussion about relevance of such a segmentation to the existence of units of motor action (Viviani & Cenzato, 1985; Viviani, 1986), attracting attention to the analysis of the velocity gain factor. The velocity gain factor K from the two-thirds power law equals equiaffine velocity of drawing (Flash & Handzel, 1996; Pollick & Sapiro, 1997; Handzel & Flash, 1999a). Equiaffine velocity is an invariant of the equiaffine transformations. The invariance raises the question, whether another equiaffine invariant, like equiaffine curvature, may constraint the movements. Apparently, parabolic pieces have constant, zero, equiaffine curvature. Parabolic pieces are also geodesics maximizing equiaffine length, among all convex curves connecting two points and having the same tangents at those points (Handzel & Flash, 1999b). In earlier work, equiaffine curvature and equiaffine velocity were, in particular, used to analyze human movements (Handzel & Flash, 1999b). The movements that obey the two-thirds power law are perceived as more uniform (Viviani & Stcucchi, 1992; Levit-Binnun et al., 2005). Do equiaffine invariants contribute to mediation between the visual and motor systems? There is evidence that both primary visual and primary motor cortices possess directional tuning (Hubel & Wiesel, 1962; Georgopoulos et al., 1982) and that directionally tuned cells are organized in columns (DeAngelis et al., 1999; Amirikian & Georgopoulos, 2003). Functional and anatomical similarities between the two cortices (V1 and M1) may underly similarities in the information processing. The information processing, in turn, may be related to the equiaffine geometry.

Parabolic pieces are unique equiaffinely invariant geometric mediators between the two-thirds power-law and the minimumjerk model The first study of the common properties of the two-thirds power-law and the minimumjerk model showed satisfactory convergence between two approaches for three complex geometric shapes (Viviani & Flash, 1995). The smoothness constraint termed the constrained minimum-jerk model explicitly relates arbitrary geometric shape to the speed profile that minimizes the integrated third derivative of position (jerk) (Todorov & Jordan, 1998). The two-thirds power-law is equivalent to setting the projection of the jerk onto the normal to the path to being zero (Todorov & Jordan, 1998). So, which


Chapter II

48

geometric shapes provide exact fit of the predicted speed profile to the two-thirds power-law (Todorov & Jordan, 1998)? Theoretical analysis of the relation between the minimum-jerk model and the two-thirds power-law for certain shapes (e.g. ellipse, cloverleaf) showed that the exponent β in the power-law constraint V = |K| Rβ is close to 1/3 for the optimal trajectories for elliptic shapes, which is equivalent to closeness of the exponent to two-thirds in (2.2.15) (Richardson & Flash, 2002). Our results enable us to use straightforward calculations to test whether some shapes can be candidates for the geometric mediator between the two-thirds powerlaw and the constrained minimum-jerk model. For example, the predicted trajectories for the curves with constant non-zero equiaffine curvature, ellipses or hyperbolas, do not fit the two-thirds power-law exactly. A simple verification of this claim can be performed by substituting their formulae (2.2.11), (2.2.12), into the equation of the necessary and sufficient condition (2.3.1). Such tests can be performed for any geometric shape with known analytic expression, though, the exact fit (β = 1/3) may not be an essential demand due to approximation and measurement errors in any recorded experimental data on which the two-thirds power-law may be tested. We show that parabolic pieces mediate between the two models. Our further investigation shows that parabolas constitute the only solution of the necessary and sufficient condition (2.3.1) under the assumption that the solution is invariant (remains a solution) under arbitrary equiaffine transformations. This assumption means that we eliminate the solutions, which do not possess the properties of efficient equiaffine invariant representation, in a sense described in the Results for parabolic pieces. The remaining solutions, parabolic shapes, are more plausible candidates for providing parsimonious equiaffine invariant representation of movement path in terms of geometric primitives. We show that drawing parabolic pieces with a constant equiaffine velocity is equivalent to drawing at a constant acceleration. An assumption of constant acceleration is used in some motion tracking and prediction systems e.g. (Liu & Lovell, 2001). Though the acceleration of the monkey drawing movements is not piece-wise constant, the change of its direction, for example, is smaller than the change of the movement direction, when a specific parabolic segment is being drawn.

Analysis of experimental data 1. In earlier work we showed (see Appendix G) that for both monkeys, equiaffine invariants of the predicted trajectories for the drawn paths: equiaffine curvature and equiaffine velocity, became closer to those of parabolic paths, during days of practice. 2. With practice, the parabolas fitted to monkey drawings get clustered into 3-4 clusters with respect to orientation of their main axes.


Chapter II

49

3. We introduce a parabolic inconsistency criterion and show that the inconsistency decreases with practice. The development of a low-dimensional representation of free movements based on a small number of parabolic clusters strengthens our proposition that parabolas may be movement primitives in the overtrained planar drawing task. Parabolas fitted to monkey drawings get clustered into 2-4 clusters with respect to orientation of their main axes with practice. We also introduce a parabolic inconsistency criterion and show that the inconsistency decreases with practice. The development of a low-dimensional representation of free movements based on a small number (2 or 3) of parabolic clusters strengthens our proposition that parabolas may be movement primitives in the overtrained planar drawing task.

A geometric point of view on the minimum-jerk model Earlier a theory of movements that suggest that movements tend to minimize the variance of the final position in the presence of signal-dependent noise was proposed (Harris & Wolpert, 1998). The authors of the minimal variance principle claimed that there had been no principled explanation for why the central nervous system should have evolved to optimize such quantities as jerk or torque change, other than that these models predict smooth trajectories. We hypothesize, based on our results, that the objective of the CNS in motor planning might be not (or not only) to minimize the signal-dependent noise, or to maximize the smoothness of the trajectory (which is related to minimization of the signal-dependent noise), but to make the representation of the complex movements more parsimonious. Greater parsimony may be achieved by the simultaneous 1) increase of “parabolicity” of the movements (during the learning phase) to achieve parsimonious piece-wise parabolic representation and by 2) keeping the equiaffine velocity of motion close to constancy. The two together minimize the jerk (see Methods). The principle of greater parsimony may imply piece-wise parabolicity. We propose that the outcome of the objective to move with constant equiaffine velocity along parabolic pieces might ease coordination between the visual and motor systems. It can be argued that with practice the equiaffine velocity of monkey drawings became closer to constant for the predicted and not for the actual trajectories (Appendix G). However, this argument does not imply that the deviations of the actual equiaffine velocity from constancy are so large that they contradict our hypothesis. Our parsimony principle does not mean that the maximization of smoothness is not a goal of the motor system. Both, the maximization of smoothness and maximization of parsimony per se may be important for motor control. It appears that they can also be satisfied simultaneously. Therefore it might be incorrect to say that one is the origin of the other.


Chapter II

50

Parabolas are our candidates to movement primitives In this chapter we have provided evidence for the use of parabolic primitives by the motor system. We present an empirical result which can be explained by our mathematical observations concerning the possibility of parsimonious representation of parabolic primitives on one hand and with parabolas simultaneously capturing geometric features of two different models of planar motion on the other hand (parabolic shapes uniquely provide equiaffinely invariant correspondence between the two models). The use of a small number of elementary tunable primitives, such as parabolas, might be strongly related to the process of decision making. Online decision making, reflected in the tunable parameters of the primitives, is needed in order to adjust the outcome of the composition of the elementary movements and to achieve the purpose of complex movements. We consider decision-making in monkey scribblings in chapter 4. In chapter 5 we implement an unsupervised segmentation of neural data by means of HMM. Movement segments corresponding to the identified states of neuronal activation resulted in clusters of similar geometric shapes, some of them parabolic.

Chapter 3 Analyzing representation of different kinematic parameters in the activity of motor cortical units underlying spontaneous planar scribbling movements Abstract We present in this chapter several methods that we have used to investigate neural representation of free planar scribblings. We present methods for constructing uni- and bi-variate tuning curves, for averaging neural activities for stereotypical segments of drawings, and for fitting regression of the kinematic parameters on the neural data. We describe methods for computing the log-likelihood of change in the firing rate, and for testing which movement parameter is more strongly represented in the neural activity out of several possible candidates. A number of studies of the motor system suggest that the majority of primary motor cortical neurons represent simple Euclidian movementrelated kinematic and dynamic quantities in their time-varying activity patterns. An example of such an encoding relationship is the cosine tuning of firing rate with respect to the direction of hand motion. However a non-Euclidian empirical invariant was found valid in both execution and perception of planar movements. The invariant appears in the empirical constrained called the twothirds power-law: the piece-wise constant velocity gain factor from the model equals the invariant of equiaffine geometry called equiaffine velocity. Equiaffine velocity equals the time derivative of the metric quantity called equiaffine length. We analyze the activity of small neuronal populations in the motor cortex recorded during free planar scribblings. Here we present a single cell level treatment on the encoding of movement primitives in the neural activity and examine in terms of what metrics the neural representation is stronger: Euclidian or equiaffine. Using a novel approach to the studying

51


Chapter III

52

of the strength of representation of kinematic parameters based on partial cross-correlation (Stark et al., 2006), we show preliminary evidence for the existence of motor cortical units which represent the equiaffine velocity more strongly than the tangential one.

3.1

Introduction

What is represented in the motor cortical activities underlying hand movements and how? Different studies propose different, sometimes contradictory, relationships between the discharge frequencies of neurons in the primary motor (M1) cortex and the produced movement. Studies in awake behaving monkeys suggested that the activity of the majority of M1 pyramidal tract neurons was related primarily to the force and its time derivatives and only secondarily to movement direction (Evarts, 1968). Another view argued that single neurons in M1 and M1 population of neurons encode both for the direction (Georgopoulos et al., 1982; Schwartz, 1992) and magnitude (Schwartz, 1994; Moran & Schwartz, 1999) of movement velocity. Velocity and force do not necessarily have the same direction. Following Todorov (2000), “MI firing was also correlated with arm position (Kettner et al., 1988), acceleration (Flament & Hore, 1988), movement preparation (Thach, 1978), target position (Alexander & Crutcher, 1990), distance to target (Fu et al., 1995), overall trajectory (Hocherman & Wise, 1991), ..., serial order (Carpenter et al., 1999) ”, (Ben-Shaul et al., 2004; Lu & Ashe, 2005), “visual target position (Georgopoulos et al., 1989) and joint configuration (Scott & Kalaska, 1995).” The goal of our work was to analyze coding of complex, goal-directed behavioral patterns in primary motor and dorsal premotor cortical areas. Earlier studies of motor cortical activity underlying hand movements involved tightly restricted behavior with explicitly defined goal, e.g. center-out or tracking movements. In our work the monkey movements were constrained to a horizontal plane, and the animal was free in its choice of movement strategy. Additionally, free scribbling movements are faster than the center-out movements (as was observed comparing both tasks for the same subject) and are much faster than the speed in a tracking task: in our records the average speed among the segments of active motion is above 18 cm/s, and local maxima reach values of 60-80 cm/s, following a period of practice; in the tracking task where underlying neural activity was used for movement reconstruction (W.Wu et al., 2003), local maxima of the speed of an exemplar movement (figure 3 in (W.Wu et al., 2003)) was less than 4 cm/s. Thus we explore the motor cortical representation of the hand trajectories for the novel, unstudied type of movements. The material in this chapter is presented in the following order. 1. We analyze the tuning properties of the cells to standard Euclidian movement


Chapter III

53

parameters such as movement direction and tangential speed, based on the scribbling movements — no use of the center-out task. There exist neurons with non-flat tuning curves for both Euclidian parameters. We also consider the empirical invariant of the drawing movements and their perception, the 2/3 power-law (2.2.15). Equi-affine speed (2.2.4) is exactly the piece-wise constant velocity gain factor from the 2/3 power-law (2.2.16) (Flash & Handzel, 1996; Pollick & Sapiro, 1997; Handzel & Flash, 1999a). The tuning properties of the √ 3 motor cortical units w.r.t. the equi-affine velocity σ˙ = x¨ ˙ y − xÿ˙ are analyzed too. 2. Noting that a large part of the analyzed units are directionally tuned, we use the cosine tuning model: f = b0 + b1 cos(θ − θ0 ) (Georgopoulos et al., 1982) to simulate Poissonian spike trains with different values of the preferred direction θ0 . The direction of motion θ is taken from the recorded trajectories. We compare tuning curves for different kinematic parameters that were constructed based on actual and simulated spike trains. Interestingly, actual tuning curves were similar to the simulated tuning curves with certain preferred direction θ0 . 3. Therefore, we also consider two-dimensional actual tuning surfaces, in which one of the parameters is movement direction. 4. In an attempt to find segmentation of the neural data into primitives, peri-event time histograms (PETH) are build for informative landmarks (two endpoints and the point of maximum curvature) of the parabolic pieces from the same cluster, cluster being a pixel in a plot from figures H.1, H.2, H.3. We did not see neural encoding of those three landmarks. This may be explained by lack of the precision in choice of the end-points in the procedure of fitting paths with parabolic pieces, the procedure is described on page 20. 5. We use conditional mutual information between neural and kinematic data, with single time-lag, to see which speeds, Euclidian or equiaffine are more strongly represented in the neural activity. This analysis does not show preferences to any of the two metrics. 6. We apply the partial cross-correlation method (Stark et al., 2006), and indeed find a significant though preliminary evidence that there exist a number of cells that are more strongly tuned to equiaffine velocity than to Euclidian one. This finding is discussed at the end of this chapter. Altogether, this chapter is mainly devoted to the methods we use to analyze the activity of motor cortical cells that underlied free scribbling movements. We consider the preliminary evidence for the existence of motor cortical cells with stronger


Chapter III

54

representation of equiaffine velocity over the tangential velocity as our main finding presented in this chapter.

3.2

Methods

The methods described below were used in order to reveal which movement parameters are represented in the neural activity and how (based on the model in which movement parameters are regressed on the firing rate of units), are movements represented in Euclidian or non-Euclidian metric, and to identify neural encoding of plausible movement primitives. With practice monkey drawings become decomposable into parabolic pieces belonging to a small number (3-4) of clusters, as we show in chapter 2. Parabolas are our candidates for motion primitives and we have intended to analyze further the neural substrate of the properties of the drawn paths based on the properties of the parabolic pieces fitted to the paths. Concerning the neural substrate of movement primitives, we wanted to see on one hand how the parabolic segmentation is represented in the neural activity if at all, and on the other hand to see whether an unsupervised segmentation of the neural activity based on HMM would result in decomposition of the corresponding trajectory pieces into primitive submovements, or maybe even parabolic submovements (see chapter 5 for the results of HMM analysis). First we show how to average the neural data in order to be able to fit the regression model, using the decomposition of the arbitrary scribbling movements into elementary (parabolic) shapes. Next we describe the method of identification of the values of the movement parameters, which correspond to the highest likelihood of change in the firing rate (the method was proposed by Moshe Abeles), knowledge of those values may lead to data segmentation.

3.2.1

Tuning relationship

Data quantization and tuning curves for scribbling movements In order to tune kinematic and neural activities, we quantize movement data. Let p = {p1 , . . . , pn } = {p(t1 ), . . . , p(tn )} be a sequence with values of a parameter p (e.g. speed), evaluated at time instants t1 , . . . , tn . We can divide the range of the values of p into N equal intervals. Let pec 1 , . . . , pec N be the centroids of these intervals. Each of the values of pj will correspond to pej = pec k , k = arg mini |pj − pec i | — the closest centroid. We then replace the vector p with the corresponding vector pe = {e pc i1 , . . . , pec in }, ij can take values from 1 to N . Let τ be a time-lag between neural activity and the value of the movement parameter, and f (t) be a firing rate at time t. For each quantum k, k = 1, . . . , N we now


Chapter III

55

construct a corresponding vector with spike counts: Sk (τ ) =

n X

1ij =k [∆tj f (tj + τ )] ,

(3.2.1)

j=1

negative τ would correspond to the firing before the corresponding movement, ∆tj = 1/(Sampling frequency) = 10 msec. Quantization and assigning spike count to the quanta of a parameter are illustrated in figure 3.1. Vector T = [T1 , . . . , TN ] with Tk =

n X

1ij =k ∆tj

(3.2.2)

j=1

contains durations of the quanta. The average firing rate corresponding to the parameter quantum with the center at pec k , lagged from the neural activity by time τ , equals Sk (τ ) Fk (τ ) = . (3.2.3) Tk Simulation of Poissonian spike trains Let X be a random variables, distributed according to Poissonian distribution. Then αm −α e , m = 0, 1, 2, . . . . (3.2.4) m! Both mean and variance of the random variable distributed according to a Poissonian distribution equal α. Let θ0 be a preferred direction of a single unit. We can simulate a spike train according to the cosine tuning formula (Georgopoulos et al., 1982): P (X = m) =

f (θ) = a + b cos(θ − θ0 )

(3.2.5)

with baseline a and amplitude of oscillation b, preferred direction θ0 and direction of motion θ. For each couple of successive samples ri = (xi , yi ), ri+1 = (xi+1 , yi+1 ) we use the same direction of motion θ between these samples. Rotem Drori proposed to approximate the Poissonian spike trains with binomial distribution. Based on firing rate, in order to generate spike trains, for each time interval [ti ; ti+1 ], q = 100 drawings are simulated according the binomial distribution with probability of success p = f ∆t/q = f /(q × recording frequency). This approximation has high accuracy for large number of drawings q coupled with small value of f ∆t: ¶m µ ¶q−m µ f ∆t f ∆t m . 1− 2 P (m spikes) = Cq q2 q


Chapter III

56

Value of parameter Quantized value of parameter

0 spikes

Value of parameter

−0.5

−0.6

1 spike

−0.7

3 spikes

−0.8

−0.9

Spike −1

0

0.2

τ 0.4

τ

τ 0.6

Time, s

0 spikes

τ 0.8

1

1.2

Figure 3.1: An illustration to the quantization and assigning spike count to the quanta. Values of the movement parameter can be assigned to 4 quanta here. Accounting for the time-lag τ , three spikes were assigned to quantum 2, one spike to quantum 3, 0 spikes to the quanta 1 and 4. In order to assign the spikes, we shift the time of each spike by the time-lag τ , note to which quantum corresponds the movement parameter at the shifted time, and calculate the number of times each quantum is accounted for.

The simulations were performed by means of the matlab function binornd. In case more than 3 spikes occurred within a time interval (∆t = 10 msec), we accounted for 3 spikes only, due to physiological limitations on the neurons. The number of discarded spikes was negligibly small compared with the number of accounted ones. Directional decomposition of the tuning to a single movement parameter Now the question arises whether we can take certain direction of motion and find interesting tuning relationships for this specific direction (or interval of directions, e.g. directions from 300 to 600 ), which will be different from the tuning relationship of the same kinematic parameter for another interval of movement direction. In order to check this, we coupled the movement parameter quantization with the direction of motion. We use 100 directional quanta, that is each directional quantum consists of the angular range of 3.60 . So, for example, in the bivariate tuning we distinguish between tangential speed V in direction θ and in direction θ + 3.60 .


Chapter III

57

The decomposed tuning relationship is of the form: n P

fe(e pc k , θec i , τ ) =

j=1

1pej =pec k , θej =θec i f (tj + τ ) n P j=1

,

(3.2.6)

1pej =pec k , θej =θec i

with f˜ being average firing rate for the quantized data, p - movement parameter (e.g. tangential speed), θ - direction of motion, τ is the time-lag, negative τ corresponds to the neural activity preceding movement. In (3.2.6) we sum firing rate at times of occurrence of the movement parameter p from quantum k and occurrence of the direction of motion θ from quantum i (accounting for the time-lag τ ). The summed firing rates are normalized by the total number of times these two parameters occurre simultaneously during the movement.

3.2.2

Regression of the kinematic parameters on the neural data for scribbling movements

The activity of single units underlying drawing movements, averaged among trials, was analyzed in (Schwartz, 1992). The monkey had to follow predefined template in that study. For us, in order to analyze the activity of units underlying scribbling movements, e.g. to construct regression relationship, which would involve several kinematic parameters, intratrial averaging of the neural activity had to be performed as well. We used similar parabolic shapes to extract spike trains underlying similar shapes drawn and to calculate the firing rate based on these spike trains. Parabolic pieces with similar orientation and focal parameter may differ in location of their vertices. Still, the difference is not very large due to relationship between the orientations of the pieces and locations of the vertices, see for example figure 3.13, figure H.9. Parabolic pieces having similar orientation and focal parameter, still have different end-points in the canonical coordinate system — the unique coordinate system obtained by translation and rotation in the plain, in which parabola is described by the equation y 2 = −2 p x, see figure 2.1. We can use observed clustering of the drawings according to the fitted parabolic pieces in order to extract similar segments. Averaging can be performed for the firing rates and for the kinematic parameters corresponding to the segments of the trajectories fitted with parabolic pieces that have a similar orientation and similar focal parameter p. We want to be able to average at least 10 trials. Therefore, alignment for the movement segments fitted with similar (according to the orientation and to the focal parameter p) parabolic pieces has to be implemented.


Chapter III

58

Pyramids for alignment of parabolic pieces We want to find the longest path contained in at least n parabolas, say n = 10. For this we build the pyramid, in which each level contains the following level, and thus taking level 10 on the levels 1-10 would provide us with the paths for which we can average. An example of the pyramid is depicted in figure 3.3. Abscissa axis in the figure corresponds to the level in the pyramid, ordinate corresponds to the y-coordinates of the end-point of parabola in the canonical coordinate system. The end-points of a parabola are depicted by blue. By green dots we depict the part of the parabola, which is contained in the parabola from the one level above in the pyramid. Actually, the pyramid consists of the segments defined by the green dots. The pyramid is constructed in a greedy way. The first two levels correspond to the longest overlap. Each following level corresponds to the longest overlap with its predecessor. The pyramid is built for parabolas that have similar orientation and focal parameter. Prior to building a pyramid, we define the etalon parabola. Its focal parameter p is the average of the focal parameters of all considered parabolas, the same for its orientation. Next we project the parabolas on this etalon, taking the distances between the samples on the projection proportional to the distances between the samples on the parabola, see two examples in figure 3.2. All further manipulations are considered for the projections on the etalon parabola. For each parabola in the pyramid, we can reparameterize the time-dependent kinematic parameters of the drawing, e.g. time, speed, by the equi-affine arc-length, e.g. t = t(σ). To do so, the inverse of the parameterization σi = σ(ti ) is used (note that no piece contains inflection point and thus the function σ(t) is monotone). Now the average firing rate can be calculated, by averaging the firing rates in all trials, parameterized w.r.t. the equi-affine length. We performed a regression of the form f (t + τ ) = a + b V (t) + c V (t) sin θ(t) + d V (t) cos θ(t)

(3.2.7)

with f , τ , V and θ being the firing rate, time-lag, speed, and direction of motion respectively. The constants a, b, c, and d are the regression coefficients. An example of such a regression is shown in figure 3.4. The fit was quantified by the Euclidian distance between the multidimensional vectors defined by the left- and right-hand sides of (3.2.7), and by the variance not accounted for in the 4-dimensional hyperplane defined by (3.2.7). In figure 3.4 we show an example of the fit that was performed simultaneously to the data from 3 pyramids. The fit was performed for different time-lags, ranging from -0.5 sec. to 0.5 sec. The regression coefficients of the fit in this example are: a = −1.3, b = 15, c = −7, d = 4.5. Left (firing rate, spikes/sec.) and right hand sides of (3.2.7) are depicted in figure 3.4A. The firing rates for 5 different time-lags (-0.5, -0.25, 0, 0.25, 0.5 sec., negative


Chapter III

59

Parabola #27 from the quantum.

Parabola #16 from the quantum.

20 20

10

0

10

Corresponding points on parabola unique for the quantum Path piece Projection on the best fitting parabola

Corresponding points on parabola unique for the quantum Path piece Projection on the best fitting parabola

0

−10

−10

−20

−30 −20 −40

−30

−50

−60

−50

−40

−30

−20

−10

0

−100

−90

−80

−70

−60

−50

−40

−30

−20

−10

Figure 3.2: Two examples of projecting the drawn path on the etalon parabola for the cluster defined by parabolas with similar values of the focal parameter p and orientation α. time-lag corresponds to the neural activity preceding the movement) are depicted in figure 3.4B. The errors of the fit are estimated in two ways: as Euclidian norm and as an unaccounted variance for the PCA on the data normalized to unit variance are plotted in figure 3.4C. In this example, different estimates show different optimal (minimal error) time-lags. The optimal time-lag used in A. corresponds to the PCA estimate and equals -0.12 sec. The time-lag corresponding to the least-squares fit equals 0.16 sec. The three parabolas, which are the 10-th level in the corresponding pyramids, are plotted in figure 3.4D. The corresponding velocities are depicted in figure 3.4E. Here the velocity of the second and the third parabolic pieces change along the straight line. This means that the averaged time-derivative of velocity has constant direction and thus the averaged rule of drawing has acceleration with the direction close to being constant. We saw examples with both good and bad fit to the regression formula (3.2.7). The fit was always very good for short, single pieces. When we considered 3 pieces simultaneously, in many examples the fit was not good. This might be caused by different time-lags in the representation of the drawing of parabolic pieces from different clusters. As we show later, in chapter 5, for the HMM segmented data, this

0


Chapter III

60

50

40

y coordinates of the end−points

30

20

10

0

−10

−20

−30

−40 Par. piece Pyramid −50

0

5

10

15

20

25

30

# of parabola

Figure 3.3: Illustration of the pyramid of parabolas. Blue dots represent the y-coordinates of the end-points of parabolic pieces in the canonical coordinate system, in which y 2 = 2 p x, see figure 2.1. Green dots represent the pyramid. We call this ordered sequence of parabolas pyramid because each of its levels (defined by two green dots) contains the upper level. indeed may be the case. We did not draw any conclusions about the representation of the data, based on our regression procedure. This procedure is presented here as a method for fitting a regression between the neural and kinematic data for non-repeatable drawings, using decomposition into primitive submovements of similar shapes and using a metric pyramid.

3.2.3

Likelihood of surprise in the firing rate (the method was proposed by Moshe Abeles)

Searching for movement primitives, one may analyze the properties of movement segments. Questioning which is the best way to segment the recorded movements, for


Chapter III

61

D. 100

8

y, position, [mm]

F. rate Regr.

6 4 2

50 0 −50 −150 −100

−50

0

50

100

x, position, [mm] B. 10

6

y, velocity, [m/s]

8 f.rate

E.

−0.5 −0.25 0 0.25 0.5

2

tgv; 0.59% in last dim. τL =0.16; τPCA=−0.12; regrs.: −1.3 15 −7 4.5

2

A. (sqrt Σ erri ) / N = 0.0308 27jun00. Unit 21, #4; T1=0.535, T2=0.329, T3=0.3

4 2 0

0

50

100

150

200

250

0.4 0.2 0 −0.2 −0.5

300

0

0.5

x, velocity, [m/s]

Error, values from 0 to 1

C. Pyramid, level #1, 10 parabolas, errors of fit

−0.5

LSQ/18 PCA/29

0

0.5

τ

Figure 3.4: Illustration of the fit of the regression relationship (3.2.7). The fit was quantified by the Euclidian norm of the difference, and by the variance not accounted for in the hyperplane defined by (3.2.7). The fit was performed simultaneously on the data from 3 pyramids, each constructed based on the parabolas from a different parabolic cluster, for different time-lags, ranging from -0.5 sec. to 0.5 sec. The regression coefficients of the fit in this example are: a = −1.3, b = 15, c = −7, d = 4.5. A. The firing rate (spikes/sec.) and the right hand side of (3.2.7). B. Firing rates for 5 different time-lags. C. Errors of the fit versus time-lag, estimated as the squared error, and as an unaccounted variance of the PCA for the data normalized to unit variance. Different estimates show different optimal time-lags. The time-lag used in A. corresponds to PCA and equals -0.12 sec. D. The three parabolas, which are the 10-th level in corresponding pyramids. E. Corresponding velocities. Note that for two parabolic pieces (the second — black and the third — red) the velocity vectors are close to a straight line. This means that the direction of acceleration for them is close to constant.


Chapter III

62

example, based on minimal or maximal values of the speed, we ask the following: for which value of the parameter (speed) the likelihood of change in the firing rate is the highest, accounting for a time-lag, assuming that the neural activity changes abruptly when flipping between different segments. In a different example, we ask whether a segmentation based on primitive geometric shapes (e.g. segmentation based on parabolic pieces) is represented in the neural activity, we can test what is the likelihood for change in the neural activity at the end-points of the segments, accounting for a time-lag. The likelihood of change should be a function of the time-lag too. We follow a procedure suggested by Moshe Abeles (personal communication) for calculating the likelihood of change in the firing rate. The procedure is applied to a spike train of a single unit (spike trains of an ensemble of units within the same time range). Assume that the spike counts are Poissonian. Consider neuron i and time instant t. 1. Calculate the average firing rate λ in the time bin of 200 msec. preceding t, T− = [t − 200msec., t]. 2. Calculate the number of spikes n+ in the time bin of 50 msec following t: T+ = [t, t + 50msec.]. 3. Calculate the Poissonian probability p≤ of n+ or less spikes in the time bin T+ given λ. 4. Calculate the Poissonian probability p≥ of n+ or more spikes in the time bin T+ given λ. 5. Define the log likelihood of surprise in the firing rate for unit i at time t: Li (t) = − log (min{p≤ , p≥ }) . The value min{p≤ , p≥ } is defined as the probability of no change in the firing rate. The illustration for the proposed method is shown in figure 3.5. Log likelihood of surprise for several simultaneously analyzed neurons is defined as a sum of log likelihoods of surprise of individual units.

3.2.4

Use of partial cross-correlation analysis in order to resolves ambiguity in the encoding of multiple movement features

Recently a promising approach has been proposed to resolve ambiguity in encoding of correlated movement features (Stark et al., 2006). Consider the overtrained drawing


Chapter III

63

n+

time λ = n− / T− Figure 3.5: Illustration to calculation of the probability of no change. movements of the monkeys, which are very stereotypical, (see for example the path depicted by black dots in figure 2.2). The direction of the overtrained movements changes in an ordered way, always counter-clockwise or clock-wise, depending on the subject and hand. It was observed that the direction of acceleration is correlated with the direction of velocity. For examples, in figure 3.6 we depict the histogram of direction of acceleration (A) and two-dimensional histogram of the direction of motion and the direction of acceleration (B) for the movements from part of a recording day, without segmenting the data. In (B) the direction of motion corresponds to the angular component, and direction of acceleration corresponds to the radial component via the transformation θacc = −180 + 360 R. Here R is the radial coordinate, which varies from 0 to 1. The direction of motion is a good predictor of the direction of acceleration (Stark et al., 2006) as can be seen in the example depicted in figure 3.7A, where the cosine of the direction of velocity and the cosine of the direction of acceleration are depicted. The direction of velocity is similar to a delayed version of the direction of acceleration. The observation is true for the segments of active motion (defined in chapter 1). In the left part of figure 3.7B we depict an illustration from (Stark et al., 2006). In that study a monkey was required to perform tracing movements. Simultaneously with the movement, the activity of neurons in the motor cortex was recorded. The path of one drawing performed by monkey is depicted in the left part. In the right part crosscorrelation profiles averaged over many trials are depicted. The illustration is related


Chapter III

64

A

B (# of quanta) x 416

17jul00. Unit: 12, # 2, Poiss?=0; τ=0

90 120

Histogram of accel. direction

(# of quanta) x 5033

1 150 0.8

1 60 0.8 0.6 0.4 0.2

0.9 0.8 30

0.7 0.6 0.5

0.6

0.4 0.4 0.2

330

210

0.3 0.2

−100

0

100

240

300 270

Direction of acceleration, [deg.]

Figure 3.6: Here we visualize the relationship between the direction of motion and direction of acceleration. The data correspond to the period of time when the unit 12 from day 17jul00 was classified as stably active. The data were not segmented (e.g. into the segments of active motion). A. Histogram of the direction of acceleration of the scribblings. B. Two dimensional radial histogram of joint distribution of the direction of velocity and direction of acceleration. The angular component corresponds to the direction of motion, and the radial component corresponds to the direction of acceleration, θacc = −180 + 360 R degrees. Here R is the radial coordinate, which varies from 0 to 1. One can see that direction of motion is a good predictor of direction of acceleration. to representation of movement parameters in the activity of a specific neuron. The direction of velocity and the direction of acceleration “are highly correlated at nonzero delays: acceleration precedes velocity by 150 ms (peak of green curve). Ambiguity ensues: does the neuron encode acceleration at 0 delay (black) or velocity at a time delay of 150 ms (red)?” We correlated the cosine of the direction of velocity and the cosine of the direction of acceleration for different time-delays between the direction of velocity and direction of acceleration. Only segments of active motion were considered, and the result for one recording session is depicted in figure 3.7C. Indeed, for large enough time delays, the direction of velocity is correlated with the direction of acceleration for the movements we analyze and therefore whenever we find that certain neuron is tuned to the direction of velocity, the same neuron will be tuned to the direction of acceleration. Thus the problem of correct selection of the movement parameter which is represented in the neural activity (e.g. velocity or acceleration (Stark et al., 2006)) takes place for the data we analyze. Note that the time-shift between the direction of velocity and the direction of acceleration that corresponds to maximal correlation is

0.1


Chapter III

65

about 300 msec. Approximately the same time shift between the direction of velocity and the direction of acceleration corresponds to the maximal correlation in (Stark et al., 2006), as one can see in figure 3.7B. Careful reader will find the plots with the correlations for all recording sessions in figure H.18. Joint probability distributions of other movement parameters than direction of velocity and direction of acceleration, are not uniform as well, as can be seen in an exemplar figure 3.8. The depicted are scaled joint probability distributions for pairs of quantized movement parameters. Number of movement samples from the most probable two-dimensional quantum is displayed in the title of each plot. The names of the corresponding pairs of parameters are given left-side from the plot and in the title of the plot. The distributions correspond to the movement data recorded during stable activity of one of the recorded units. One can see that equi-affine and tangential speeds are correlated. Indeed, in figure 3.9 we depicted the file-wise cross-correlation coefficients between the tangential and equiaffine speeds. The values of the strongest negative and the strongest positive correlation coefficients are shown in the title of every plot. The abscissa axis corresponds to the time-shift between the tangential and equiaffine velocities. Ordinate stands for the index of the file with the data. The two velocities are stronger correlated for smaller time-shifts. The correlation is a bit weaker than that for the direction of acceleration and direction of velocity, figure H.18 but still is relatively high. Correlations for all days are depicted in figure H.19. An intriguing question, which can be studied for spontaneous scribbing movements, is are the movements represented in Euclidian on another, non-Euclidian metric. Note a widespread directional tuning of motor cortical units and an invariance of the point-to-point movements (Euclidian features); and a special role of the equiaffine geometry in the two-thirds power law, our results for parabolic pieces from chapter 2, and a special role of parabolas in equiaffine geometry (Equiaffine features). Therefore we compare the strength of representation of the Euclidian and equiaffine velocities in the neural activity. We chose to analyze these two kinematic parameters because tuning to Euclidian speed is a known property of motor cortical neurons (Moran & Schwartz, 1999), on the other hand the equiaffine speed directly enters the two-thirds power-law (chapter 2, Methods). The curvature, especially equiaffine is much noisier parameter than the speed. Thereafter we did not compare the strength of representation of the Euclidian and equiaffine curvatures. Following (Stark et al., 2006), in the Results we apply the partial cross-correlation method in order to resolve the ambiguity between encoding of Euclidian and equiaffine speeds. Below we present explanation of what partial correlation is. We provide the definition of partial correlation and two examples from (Myers & Well, 2003) to make the explanation more intuitive.


Chapter III

66

A U17JUL0A.035

), cos(θ

0.5

cos(θ

Acc

)

1

−0.5

cos(θVel) cos(θAcc)

Vel

0

Speed, [m/s]

−1

4

5

6

7

8

9

4

5

6

7 Time, [sec]

8

9

0.6 0.4 0.2 0

B

C Day 17, 20jul00; min = −0.77; max = 0.81 1 60

0.8 0.6

50

0.4 40

0.2 0

30 −0.2 −0.4

20

−0.6 10

−0.3

−0.8 −0.2

−0.1

0

t

vel

0.1

0.2

0.3

−1

− tacc

Figure 3.7: A. Cosine of the direction of velocity and of the direction of acceleration. The direction of velocity is similar to a delayed version of the direction of acceleration. The observation is true for the segments of active motion (defined in chapter 1 ). B. Illustration from (Stark et al., 2006). Path of one drawing performed by monkey is depicted in the left part. In the right part cross-correlation profiles averaged over many trials are depicted. Direction of velocity and direction of acceleration “are highly correlated at nonzero delays: acceleration precedes velocity by 150 ms (peak of green curve). Ambiguity ensues: does the neuron encode acceleration at 0 delay (black) or velocity at a time delay of 150 ms (red)?” C. Correlation coefficients between the direction of velocity and the direction of acceleration for for the files with scribbling movements within a day. The values of the strongest negative and the strongest positive correlation coefficients are shown in the title of every plot. The abscissa axis corresponds to the time-shift between the cosine of the direction of acceleration signal and the cosine of the direction of velocity signal. The ordinate stands for the index of the file with the data.


Chapter III

67

Day 15, 17jul00. Unit: 1102, # 1, Poiss?=0 9.2e+02; EuclCurv

4.3e+02; AffVel

1.3e+02; AffCurv

1.7e+02; DAffCurv

1.3e+03; AffVel

7.8e+02; AffCurv

1.1e+03; DAffCurv

2e+02; AffCurv

3e+02; DAffCurv

0.4 0.2 0

0.1

0.2 0.1 0.05 0 −0.05

−0.5

0

0.5

1

1.5 1.5 1 0.5 0

−0.1

0

0.1 5.5e+02; DAffCurv AffCurv

−0.1

AffVel

0

EuclCurv

Tgv

0.6

0.1 0 −0.1 −0.2

0

0.2

Figure 3.8: Joint probability distributions for pairs of quantized movement parameters. The probability of the most probable two-dimensional quantum is displayed in the title of each plot. The names of the corresponding pairs of parameters are given left-side from the plot and in the title of the plot. Activity of different neural units was registered along different time intervals. One can be seen, that equi-affine and tangential speeds are correlated


Chapter III

68

Day 17, 20jul00; min = −0.197; max = 0.66 1 60

0.8 0.6

50

0.4 40

0.2 0

30 −0.2 −0.4

20

−0.6 10

−0.3

−0.8 −0.2

−0.1

0

tang. vel.

− taff. vel.

t

0.1

0.2

0.3

−1

Figure 3.9: Data for monkey U, following extensive practice. Correlation coefficients for the files with scribbling movements within a day. The values of the strongest negative and the strongest positive correlation coefficients are shown in the title of every plot. Abscissa axis corresponds to the time-shift between the tangential and equiaffine velocities. Ordinate stands for the index of the file with the data. The two velocities are stronger correlated for smaller time-shifts. The correlation is a bit weaker than that for the direction of acceleration and direction of velocity, figure 3.7C but still is relatively high. The partial correlation method Two variables may be correlated because they are both influenced directly or indirectly by other variables. For example, hand size is correlated with verbal abilities in children, because both hand size and verbal abilities increase with age. However, one may wish to ask whether there would still be a correlation between hand size and verbal abilities even if the effects of age could somehow be controlled or “partialed out”. For example, one may believe that even at the same ages, students who are more physically mature may tend to be more mentally mature, perhaps because greater physical growth may be an indicator of better health or nutrition, and might therefore be related to mental ability. How can we find a measure of the relation between size and verbal ability that is not contaminated by the effects of chronological age? Let the notation rXY = corr(X, Y ) stand for the correlation between two variables X and Y , then rXY |W , the partial correlation between X and Y with the effects of W partialed out, is given by rXY |W = corr(X|W, Y |W ) , where X|W is the value of X predicted from the regression of X on W ; therefore, X|W is the part of X that is not predictable from W . Similarly, Y |W is the residual that results when Y is regressed on W . In terms of the simple correlations between X, Y , and W (Myers & Well, 2003): rXY − rXW rY W

rXY |W = p

2 (1 − rXW ) (1 − rY2 W )


Chapter III

69

with correlation coefficient (CC) rXY =

h(X − hXi) · (Y − hY i)i , σX · σY

where hXi is the mean value of X and σX is its standard deviation (SD). For example, suppose X represents size, Y represents verbal ability, and W represents age. If the correlations of both size and verbal ability with age were .7 (rXW = rXY = 0.7), and the correlation between size and verbal ability was 0.5 (rXY = .5), rsize,verbal | age would have a value of (0.5 − 0.49)/(l − 0.49) = 0.02. In other words, if we take into account the relation between size and age, and verbal ability and age, the apparent relation between size and verbal ability essentially disappears. Partial correlations are often calculated in an attempt to statistically “control” for the effects of variables that are not of interest. The meaning of a partial correlation can usually be properly understood only in terms of a specific theory or causal model of the situation under investigation. When rXY |W is obtained, what is removed from X and Y are the components that are predictable from a linear regression on W . Suppose, for example, that X measures parents’ education, Y measures their children’s performance in school, and W is the number of books in the home. If rXY |W is considerably smaller than rXY – that is, if the correlation between school performance and parent’s education is much smaller when the number of books in the home is partialed out – this does not necessarily mean that providing the family with lots of books will have much of an effect on performance, or that parental education is unimportant. Partialing books out of the correlation between parental education and school performance removes more than the direct effect of the books; partialing out books removes any components of parental education and children’s school performance predictable from the number of books in the home. The number of books in the home is correlated with parental education and intelligence, as well as with other potentially important variables, such as economic level and parental encouragement of achievement. Therefore, when the number of books is partialed out of the relation between X and Y , some of the effects of these other variables are removed as well. The ideas of partial correlation can be extended to partialing out the effects of more than one variable. Suppose that one wishes to partial out the effects of variables W and Q from the correlation between X and Y , then (Myers & Well, 2003) rXY |W Q = r³

rXY |Q − rXW |q rY W |Q ´³ ´. 2 2 1 − rXW |Q 1 − rY W |Q

Straightforward derivation shows commutativity of rXY |W Q with respect to the partialing parameters (Q and W ): rXY |W Q = rXY |QW .


Chapter III

70

By induction, commutativity takes place for any number of partialing parameters. Disambiguating the correlation between the neural activity and kinematic parameters Using partial cross-correlation we can partial out the the correlation between the neural activity and one movement parameter from the correlation between the neural activity and the other movement parameter on a set of different time-lags. In order to account for different time-lags the partial cross-correlation matrix (PCCMs) M is constructed as follows. Each entry of the matrix equals the partial correlation for two kinematic parameters lagged from the neural data by the corresponding two time-lag τ1 and τ2 : M (τ1 , τ2 ) = rn(t) p1 (t+τ1 )|p2 (t+τ2 ) with two movement parameters denoted by p1 and p2 . Considering the case of the direction of velocity and the direction of acceleration, p1 = θvel and p2 = θacc . PCCMs are used to determine to which movement features the firing rate is related. To this end, a t-statistic is used (Stark et al., 2006), which is a measure of correlation that does not depend on the number of trials, and can be compared between different variables. The t-statistic is computed based on the average value across trials and the SD across trials ¯ ¯ ¯ ¯ hˆ ρ(τ1 , τ2 )i ¯ √ ¯¯ , t(τ1 , τ2 ) = ¯ (3.2.8) SD [ˆ ρ(τ1 , τ2 )] / n where n is the number of trials, and ρˆ(τ1 , τ2 ) = tanh−1 [ρ(τ1 , τ2 )] is a Fisher ztransformed CC at a pair of time delays. Single-trial partial CCs are Fisher ztransformed so that they are distributed normally under the null hypothesis of no correlation between variables. For CCs that are normally distributed this statistic is distributed according to the t-distribution with n − 1 degrees of freedom, hence P values can readily be computed. The statistic is a useful visualization tool. Same as in (Stark et al., 2006), we estimate PCCMs for movement features within the range of -300 to 300 msec, in 10-msec increments. More details on implementation and interpretation of the partial correlation can be found in (Stark et al., 2006). Our calculations were based on the code developed by Eran Stark.


3.3 3.3.1

Chapter III

71

Results Tuning between kinematic and neural activities: actual tuning curves are similar to Poissonian ones for nondirectional movement parameters

Some of the units we analyzed were tuned to kinematic parameters, and some were not Some units change their activity w.r.t. the change of kinematic variables, whereas some do not. In figure 3.10 we display tuning curves from two units. In both (A) and (B) the upper row corresponds to the spike count Sk (τ ) from (3.2.1), and the lower row corresponds to the firing rate Fk (τ ) from (3.2.3). The tuning curves were computed for several kinematic variables: movement direction, tangential velocity, Euclidian curvature, equiaffine velocity, equiaffine curvature and its derivative w.r.t. the equiaffine arc length; they are depicted for different time lags, from -250 msec. to 250 msec. with a time step of 50 msec. Activity of one of the units with the tuning curves depicted in figure 3.10 (a) is tuned to different kinematic parameters, the tuning relationship depends on the time-lag. For another unit (with tuning curves in 3.10 (b)), bimodal tuning is observed for the direction of motion, for other parameters the tuning relationship is shallow and can be due to noise. Tuning to kinematic parameters can be reproduced by simulated neural activity, based on Poissonian spike trains for cosine tuning model In figure 3.10 (a) clear directional tuning can be seen even for different time-lags. Directional tuning is a known property of motor neurons neurons and is a substrate of the population vector approach (Moran & Schwartz, 1999). Can the observed tuning of neuronal activity to kinematic parameters other than the direction of motion be the outcome of non-uniformity of the drawn paths coupled with directional tuning of the neurons? The results of our analysis support this hypothesis, as we show below. However, such a plausibility does not imply that the activity of the recorded units can be explained by pure directional tuning. We drew Poissonian spike trains (see Methods, page 55) based on the cosine tuning model (3.2.5) for the trajectories recorded simultaneously with the neural activity for 18 equi-distant preferred directions: 00 , 200 , . . ., 3400 with parameters a = 15, b = 12. Then we chose the preferred direction with the tuning curves most similar to the actual ones. It appears that the tuning between the kinematic parameters and the simulated neural activity is similar to the one observed for the actual neural activity for all kinematic parameters under consideration, for each of several units we have checked. In figure 3.11 tuning curves for the actual data (A) are contrasted to the tuning curves of the simulated neural data (B); legends are the same as in


Chapter III

72

A 17jul00. Unit: 5104, # 12, PD~0ο,a=0,b=0;DACs?0; Poiss?=0;Pred?=0;gr=−0.25,bl=0,rd=0.25,∆color=0.05 . x 1.1e+03; Theta

x 6.2e+02; Tgv

x 2.9e+03; EuclCurv

x 7.2e+02; AffVel

x 5.9e+02; AffCurv

x 1e+03; DAffCurv

1

Spike count, scaled

0.8 0.6 0.4 0.2 0

−100

0

100

0

0.2

Scale: x 21.5

0.4

0.6

−0.1

x 25.5

0

0.1

0.2−0.5

0

x 21.6

0.5

1

1.5 −0.1

x 27.8

0

0.1 −0.2

x 15.8

0

0.2

x 17.1

F. rate, [sp./sec.], scaled

1 0.8 0.6 0.4 0.2 0

B

−100

0

100

0

0.2

0.4

0.6

−0.1

0

0.1

0.2−0.5

0

0.5

1

1.5 −0.1

0

0.1 −0.2

0

0.2

27jun00. Unit: 31, # 6, PD~0ο,a=0,b=0;DACs?0; Poiss?=0;Pred?=0;gr=−0.25,bl=0,rd=0.25,∆color=0.05 . x 1.9e+02; Theta

x 1.7e+02; Tgv

x 8e+02; EuclCurv

x 2.1e+02; AffVel

x 1.1e+02; AffCurv

x 1.9e+02; DAffCurv

1

Spike count, scaled

0.8 0.6 0.4 0.2 0

−100

0

100

0

0.2

Scale: x 11.1

0.4

0.6

−0.1

x 11.9

0

0.1

0.2−0.5

0

x 12.6

0.5

1

1.5 −0.1

x 9.49

0

0.1 −0.2

x 10.3

0

0.2

x 10.2


1 0.8 0.6 0.4 0.2 0

−100

0

100

0

0.2

0.4

0.6

−0.1

0

0.1

0.2−0.5

0

0.5

1

1.5 −0.1

0

0.1 −0.2

0

0.2

Figure 3.10: Tuning curves for two single units. Green color corresponds to the time-lag τ = −0.25 sec. — movement follows the neural activity, red — to the time-lag τ = 0.25 sec. — neural activity follows the movement. Intermediate values are depicted with ‘intermediate’ colors. Tuning curve for zero time-lag is depicted with blue. The tuning was performed for the following 6 parameters: direction of motion [degrees], tangential speed [m/s], Euclidian curvature [1/mm], equi-affine increments (equivalent to equi-affine velocity) - [mm2/3 ], equi-affine curvature [mm−4/3 ], derivative of the equi-affine curvature w.r.t. the equi-affine arc-length [mm−2 ]. In the first row in A and in B spike counts and in the second row average firing rates are depicted. In A one can see that the unit has preferred direction and that for all kinematic parameters there exist ranges for which the firing rate is higher/lower. Tuning depends on the time-lag. For example, for equi-affine curvature κ, firing rate is higher near κ = 0 for negative time-lag (green), and is lower for positive time-lag (red). In B no preferred direction is observed, and no (or much weaker than in A) tuning is observed for other kinematic parameters than direction of motion.


Chapter III

73

figure 3.10. The tuning curves for both actual and simulated spike trains are similar. Note that their preferred directions are different by about 450 . The tuning curves for the simulated firing, with preferred direction1 3000 are depicted in figure 3.11. The direction 3000 is close to the preferred direction of the actual unit (figure 3.11A), still the tuning curves for the simulated data are different from the ones for the actual data. We suppose that this difference (450 between the preferred directions of the actual and simulated units in figure 3.11) could be due to the non-constant time-lag and due to the possibility that the activity of this actual unit cannot be explained by the directional tuning, but the directional tuning captures much of the observed tuning properties of the unit. Note also the sharp peak in the actual directional tuning curve (e.g. consider the blue curve, which corresponds to the time-lag 0), in the simulated tuning curve the peak is not sharp.

3.3.2

Directional decomposition of the tuning: equiaffine speed is distributed more uniformly w.r.t. the movement direction than Euclidian speed

We found that the observed tuning curves between the recorded neural activity and kinematic variables can be reconstructed as a byproduct of non-uniformity of the drawn paths and cosine tuning of the neurons (which does not necessarily imply that the kinematic variables are not represented in the neural activity). In order to see how the tuning relationship depends on movement directions, the tuning relationship was endowed with an additional variable — movement direction, see Methods, page 56. Exemplar tuning correspondences for directionally decomposed tangential (A) and equiaffine (B) speeds for the same neuron are depicted in the lower rows of A and B in figure 3.11. In the figure, the first column corresponds to spike count, second column — to duration of the corresponding quantum, third column — to average firing rate for corresponding quantum of the decomposed parameter (lower row) and of the non-decomposed parameter (upper row). In the decomposed plots, the radial component stands for the values of the movement parameter transformed via the the relationship on the right. The angular component corresponds to the movement direction. The values in the colored plots should be multiplied by the values, which appear in the titles of the corresponding plot. In all examples we saw that 1) equi-affine speed was distributed more uniformly w.r.t. to the direction of motion than the tangential speed; 2) directionally decomposed tuning is sharper for tangential speed than for equi-affine velocity, as in figure 1

By preferred direction we mean here the direction of motion with highest corresponding firing rate in the tuning curve for scribbling task. This direction is not necessarily the same as the preferred direction of the same unit in the center-out task, which was discovered by Rotem Drori.


Chapter III

74

A 17jul00. Unit: 5104, # 12, PD~340ο,a=15,b=12;DACs?0; Poiss?=1;Pred?=0;gr=−0.25,bl=0,rd=0.25,∆color=0.05 . x 1.5e+03; Theta

x 1.1e+03; Tgv

x 4.2e+03; EuclCurv

x 1.2e+03; AffVel

x 8.4e+02; AffCurv

x 1.4e+03; DAffCurv

1

Spike count, scaled

0.8 0.6 0.4 0.2 0

−100

0

100

0

0.2

Scale: x 27.8

0.4

0.6

−0.1

x 26.6

0

0.1

0.2−0.5

0

x 28.1

0.5

1

1.5 −0.1

x 26.4

0

0.1 −0.2

x 20.2

0

0.2

x 20.9


1 0.8 0.6 0.4 0.2 0

B

−100

0

100

0

0.2

0.4

0.6

−0.1

0

0.1

0.2−0.5

0

0.5

1

1.5 −0.1

0

0.1 −0.2

0

0.2

17jul00. Unit: 5104, # 12, PD~300ο,a=15,b=12;DACs?0; Poiss?=1;Pred?=0;gr=−0.25,bl=0,rd=0.25,∆color=0.05 . x 1.6e+03; Theta

x 1.3e+03; Tgv

x 4.4e+03; EuclCurv

x 1.4e+03; AffVel

x 8.5e+02; AffCurv

x 1.5e+03; DAffCurv

1

Spike count, scaled

0.8 0.6 0.4 0.2 0

−100

0

100

0

0.2

Scale: x 28.5

0.4

0.6

−0.1

x 25.9

0

0.1

0.2−0.5

0

x 26.9

0.5

1

1.5 −0.1

x 28.3

0

0.1 −0.2

x 21

0

0.2

x 21.8


1 0.8 0.6 0.4 0.2 0

−100

0

100

0

0.2

0.4

0.6

−0.1

0

0.1

0.2−0.5

0

0.5

1

1.5 −0.1

0

0.1 −0.2

0

0.2

Figure 3.11: Tuning curves for simulated spike trains, for two preferred directions: 3400 in A., and 3000 in B. The parameters of the cosine tuning formula (3.2.5) are: a = 15, b = 12. Meaning of the graphs is the same as in figure 3.10. The direction of motion is represented w.r.t. the preferred direction. In figure 3.10A, the unit is directionally tuned and has single preferred direction. The unit is also sensitive to other, non-directional movement parameters. The tuning curves in A of the current figure are similar to the tuning curves in figure 3.10A, including non-directional tuning curves, though the spike trains were obtained from cosine tuning to the movement direction. The preferred direction, 3400 , is different from the preferred direction of the actual unit: 2950 . Although the preferred direction in B is close to the preferred direction of the actual unit, the tuning curves for the non-directional parameters are different from the actual tuning curves.


Chapter III

75

17jul00. Unit: 5104, # 7; τ=−0.18; ’dir_decompose_f_rate_main.m −>’ x 1067; Tgv

x 13.15

x 1.241e+04

1

Scaled

0.8 0.6 0.4 0.2 0

Non−decomp. Decomp., no outl. 0

0.2

0.4

0.6

0

0.2

0.4

0.6

Spike count, x 56.5316

# of quanta, x 547.6115

90

90

1

120

0.8

0.8

0.6 150

0.4

30150

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 25.4642

0.6

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

17jul00. Unit: 5104, # 7; τ=−0.18; ’dir_decompose_f_rate_main.m −>’ x 1263; AffVel

x 14.66

x 1.294e+04

1

Non−decomp. Decomp., no outl.

Scaled

0.8 0.6 0.4 0.2 0

−0.5

0

0.5

1

1.5 −0.5

0

0.5


# of quanta, x 460.1082

90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1.5

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330

210 330

330210

210

240 300

240

B

1

0.9

0.6 30150

0.5

F. rate, x 30.5717

1

120

60

0.6

x = −0.75 + 2.25 * R

1

270

300

240

300

0.2 0.1

270

270

Figure 3.12: Directionally decomposed kinematic parameters - tangential speed (A), and equiaffine speed (B), and corresponding spike counts and firing rates of the same unit as in figure 3.11. One can see that 1) equi-affine speed (in B) is distributed more uniformly w.r.t. to the direction of motion than the tangential speed (in A); 2) directionally decomposed tuning is sharper for tangential speed than for equi-affine velocity.


Chapter III

76

3.12. Observation 1) may be interpreted as a uniform span of the parameter range for the performed task, in which tangential speed was not necessarily uniform. Our kinematic interpretation of the observed uniformity is related to the two-thirds power-law model: the tangential speed change is compensated for by the path curvature change to conserve equi-affine velocity.

3.3.3

We did not see evidence for single-cell encoding of the end-points of parabolic pieces, and the question is illposed

We tested whether the end-points of our parabolic segments correspond to segmentation of the neural activity on a single-cell level. By segmentation in the neural activity we mean change in the activity that is not predictable based on the knowledge of its past and future. Peri event time histograms (PETH) were constructed for the end-points of parabolic pieces and their points of maximal curvature. We looked through many examples, non of them justified segmentation of the neural data with respect to the end-points of our parabolic segments. One example of PETH and corresponding parabolic pieces is depicted in figure 3.13. In the first row, three plots with the spike trains are locked on the start of parabolic pieces, on their points of the maximal curvature, and on their end-points, correspondingly. In the second row, the corresponding average firing rates are depicted. They are followed by the likelihood of change in the firing rate, see Methods, page 60. The likelihood attains maximum near the point of maximal curvature, but this maximum in likelihood corresponds to monotonous (predictable) change in the firing rate. The change in the firing rate for this directionally tuned neuron is higher near the point of maximal curvature because the direction of motion is changing faster near that point. The average firing rate reaches its maximum near the end points of the pieces. The end points correspond to the direction around 900 , and the peak of the directional tuning curve for this unit is about 1000 , which is consistent with the claim that the observed increase in the firing rate is due to directional or other kinematic preferences of the unit. The directional preferences can be accounted for because the parabolic pieces considered have similar orientation and thus the direction of motion at the 3 mentioned points (start, vertex and end) is similar among the selected parabolic segments. The time-lag is zero in all plots and the time range of ± 0.5sec. covers any time-lag ever mentioned in the literature. The abscissa axis stands for time relative to the featured point (stat, vertex or end of parabolic piece). In the last, fourth column, average tangential and equiaffine speeds for the segments are depicted. The parabolic pieces are depicted in the plot below. Does our lack of evidence for the segmentation of the neural data underlying parabolic segments mean that no segmentation of the neural data exists in relation

# of the parabolic piece


Chapter III

77

28jun00; # 7, 70; P0; cos0, τ=0; st; MS0 Ang.∈[260;280], #14; p∈[10.1;15.1], #3

function: q_p_p_rasters_similar_parabs.m en hat

−0.5

0

0

0.5 −0.5

0.5 −0.5

0

blue − aver(tgv), [mm/s] 2/3 green − 5*aver(veaff), [mm /s] 0.5

0

200

400

5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85

V

−log (L)/ .:2.7; :12 2 F. rate

50 40 0

20

−50

0 1 0.5 0

−50

0

50

Max. # of spikes: 181−> 30 −0.5

0 [sec]

0.5 −0.5

0 [sec]

0.5 −0.5

0 [sec]

0.5

Figure 3.13: Example of the neural activity locked on the end-points of parabolic pieces and on their points of maximal curvature. First row: spike trains locked on the start point, point of maximal curvature and the end point of the parabolic pieces. Abscissa axis represents time. The first column corresponds to the spike trains locked on the start of the segment (green line), the second — on the vertices of parabolic pieces (blue line), the third — on the end-points of the segments (red line). Fourth column of the first row: average Euclidian speed (blue) and average equi-affine speed multiplied by 5 (green). Second row: average firing rate associated with the corresponding rasters. Third row: the average log likelihood of change in the firing rate is depicted by blue. One can see that the maximal log likelihood corresponds to that portion of the trials, where the time derivative of the firing rate had highest magnitude. Second row, fourth column: parabolic segments to which the spike trains correspond.

600


Chapter III

78

to parabolic segments on the path? The question of the coding of the end-points of parabolic segments from the same quantum (all the segments have similar focal parameter p and similar orientation α) may be ill-posed. Firstly, the segmentation procedure does not provide an exact answer about the end-point because it is based on the fitting properties, and the time instants of the end-points may vary in the range of tens of milliseconds. Such temporal range of variability with respect to the behavioral event (e.g. start or end of the parabolic piece) may not enable to lock the changes in the neural activity in time. Secondly, the segmentation may not be a result of sequential encoding of the movement only, as we will show further in the part related to HMM. Therefore there may be not only a hard flipping of coding of the segments, but also a soft segmentation — parallel encoding with increase of the activity related to parabolic segments from a specific parabolic quantum (parabolas with similar orientation and focal parameter), similar to the encoding of the segments of the figures drawn by monkeys as described in (B.Averbeck et al., 2002). Figure 3.14 contains illustration of parallel encoding of movement segments from (B.Averbeck et al., 2002). The monkey in that study was required to draw geometric shapes composed of a number of straight segments (e.g. triangle, trapezoid). The posterior probability of a segment of the shape, based on the discriminant analysis, was called the strength of representation. Each plot shows the strength of the representation of each segment for each time bin of the task. Plots show parallel representation of segments before initiation of copying. Further, rank order of strength of representation before coping corresponds to the serial position of the segment in the series. The rank order evolves during the drawing. We claim that the PETH analysis may not be relevant for such type of coding.

3.3.4

Partial cross-correlation reveals stronger representation of the equiaffine speed than Euclidian one in the activity of some of the motor cortical cells

It was reported in earlier studies (e.g. (Schwartz, 1992)) that single units in motor cortex are tuned to the direction and speed of motion. Both direction of motion and its speed characterize motion in the Cartesian frame of reference. Earlier kinematic studies (e.g. (Lacquaniti et al., 1983)) reported that the velocity gain factor K in (2.2.15) is piece-wise constant for drawing movements. The value K exactly equals the equiaffine speed of motion (2.2.16) and thus is invariant under equiaffine transformations. Superiority of representation of the equiaffine speed over Euclidian speed may indicate use of equiaffine metric in the neural representation of movements. On average, the tangential and equiaffine speeds increase and decrease simultaneously, in other words they are correlated. For illustration see exemplar plot with different movement parameters for one movement segment in figure 2.13, exemplar


Chapter III

79

Figure 3.14: Illustration from (B.Averbeck et al., 2002). “Plots for all four shapes of strength of representation vs. time ... Plots show parallel representation of segments before initiation of copying.” The posterior probability of the segment of the shape, based on the discriminant analysis, was called the strength of representation. (for one day) plot with pairwise joint distributions of different kinematic parameters recorded simultaneously with the recordings of a unit of neural activity in figure 3.8, and file-wise values of cross-correlation for all days in figure H.19. Comparison of the representation strength of the tangential and equiaffine speeds in neural activity by means of conditional mutual information (“naive approach”) When comparing the information capacity of neural representation of the tangential and equi-affine speeds, one has to account for correlation between these two parameters. Here we attempt to resolve the ambiguity by means of conditional mutual information, when the information content of one of the movement parameters is partialed out of the mutual information between the neural activity and the other movement parameter. For the quantized firing rate f , tangential speed V and equiaffine speed |σ|: ˙ IV = I(f ; V | |σ|) ˙ I|σ| = I(f ; |σ| ˙ |V) ˙


Chapter III

80

where for random variables X, Y, Z I(X; Y |Z) =

X x,y,z

p(x, y, z) log

p(x, y|z) p(x|z)p(y|z)

(3.3.1)

(Cover & Thomas, 1991). A Venn diagram of the conditional mutual information I(X; Y |Z) for information content of the random variables X, Y , Z is shown in figure 3.15. The case IV < I|σ| ˙ is interpreted as stronger representation of the equi-affine speed in the neural activity than the tangential speed because then the information content of |σ| ˙ in the inference relationship f → |σ| ˙ → V would be higher than the information content of V in the inference f → V → |σ|. ˙ We compared the values I(X;Y|Z) H(Y)

H(X) H(Z)

Figure 3.15: Venn diagram for conditional mutual information I(X; Y |Z). of the mutual information IV and I|σ| ˙ and did ˙ for the same time-lags for V and σ not observe superiority for the information capacity of any of the two parameters. However, as the resent work (Stark et al., 2006) shows, independent time-lags should be considered in such type of analysis. Equiaffine velocity is represented stronger than tangential velocity in the activity of some of the motor cortical cells, as revealed with the method of partial cross-correlation The above calculations involving conditional mutual information are based on the same time-lags for both kinematic parameters: tangential and equiaffine velocities. The approach with the same time-lags is misleading. Several (in our case 2) independent time-lags should be used, one for each parameters. In addition, as use of the t-statistic for partial correlation shows (Stark et al., 2006), the estimates of the significance of correlation are more pronounced than the averaged (among trials) correlation coefficients. Correlation coefficients are relatively small, though the values


Chapter III

81

of t-statistic have high peaks corresponding to the time-lag between the neural and kinematic activities. Additionally, use of independent time-lags may provide a good visualization of the result (Stark et al., 2006) via partial cross-correlation matrix. For the case when the strengths of representation of the tangential and equiaffine speeds in the neural activity are compared, and thus the time-lags for the tangential velocity are independent from the time-lags for the equiaffine velocity — the pairs (τ1 , τ2 ) cover the rectangular grid. For the case of conditional mutual information that has been considered above, both time-lags are equal thus belonging to a diagonal of a square. Partial correlations account for linear effects only. Ideally, we would measure the mutual information on the (τ1 , τ2 ) grid by directly applying equation (3.3.1). In practice, however, the joint probability distributions are not available, and one has to use instead the frequency table computed on the basis of parameter-response triplets, PN (f, V, σ). ˙ If PN (f, V, σ) ˙ is simply inserted in equation (3.3.1) in place of p(f, V, σ), ˙ it is known that information is usually grossly overestimated because of the undersampling due to the limited number of trials usually available (Panzeri et al., 1999). Here we show that there exist motor cortical units for which the linear regression models based on the equiaffine velocity show significant correlation even if partialed out by the tangential velocity. Partial correlation between the activity of the same units and tangential velocity, partialed out by the equiaffine velocity is considerably less significant. In figures 3.16, 3.17, values of the t-statistic for the PCCMs for a unit from PMd area are depicted. In both figures, the kinematic data correspond to the segments of active motion, positive time-lags correspond to neural activity preceding movement. For each unit PCCMs correspond to several sets of movement parameters. The interpretation of the figures depends on existence of continuous stripe with high values of the t-statistic of correlation. In plot 1.A, the stripes in favor of stronger representation of the direction of motion/speed need to be horizontal, however in the plot 1.B, the stripes in favor of stronger representation of the the direction of acceleration/magnitude of acceleration should be vertical. In other plots, horizontal stripe in the left column, and vertical stripe in the right column indicate stronger representation of the corresponding movement parameter. The depicted parameters are as follows. 1. Comparison of representation of direction and magnitude of velocity and acceleration of motion, the same as in (Stark et al., 2006). A: (left) direction of motion is conditioned on speed, direction and magnitude of acceleration; (right) speed is conditioned on direction of motion, direction and magnitude of acceleration. B: (left) direction of acceleration is conditioned on direction and magnitude of velocity and magnitude of acceleration; (right) magnitude of acceleration is conditioned on direction and magnitude of velocity


Chapter III

82

and direction of acceleration. 2. (left) direction of velocity is conditioned on direction of acceleration; (right) direction of acceleration is conditioned on direction of velocity. 3. (left) equiaffine velocity is conditioned on movement direction; (right) movement direction is conditioned on equiaffine velocity. 4. (left) speed is conditioned on curvature; (right) curvature is conditioned on speed. 5. (left) speed is conditioned on equiaffine velocity; (right) equiaffine velocity is conditioned on speed. 6. (left) log(speed) is conditioned on log(curvature); (right) log(curvature) is conditioned on log(speed). 7. (left) speed conditioned on the magnitude of the equiaffine velocity; (right) equiaffine velocity is conditioned on speed. 8. (left) log(magnitude of the equiaffine velocity) is conditioned on log(curvature); (right) log(curvature) is conditioned on the log(magnitude of the equiaffine velocity). 9. (left) log(magnitude of the equiaffine velocity) is conditioned on log(speed); (right) log(speed) is conditioned on the log(magnitude of the equiaffine velocity). 10. A: (left) direction of motion is conditioned on speed (with the same time-lag as the direction of motion) and equiaffine velocity; (right) speed is conditioned on the direction of motion (with the same time-lag as the speed) and on equiaffine velocity. B: (left) equiaffine velocity is conditioned on the direction and magnitude of the velocity. 11. A: (left) direction of acceleration is conditioned on the magnitude of acceleration (with the same time-lag as the direction of acceleration) and on the equiaffine velocity; (right) magnitude of acceleration is conditioned on the direction of acceleration (with the same time-lag as the magnitude of acceleration) and equiaffine velocity. B (left) equiaffine velocity is conditioned on the direction and magnitude of acceleration. 12. (left) speed is conditioned on the direction of motion; (right) direction of motion is conditioned on speed.


Chapter III

83

13. A: (left) direction of motion is conditioned on equiaffine velocity (the same timelag as the direction of motion) and on the speed; (right) equiaffine velocity is conditioned on the direction of motion (with the same time-lag as the equiaffine velocity) and speed. B: (left) speed is conditioned on the direction of motion and equiaffine velocity (the last two have the same time-lag).


Chapter III

27jun;10_13;#1;D:20;m t(ρ(θvel | . ));t(ρ(speed | . ))

t(ρ(θvel(sin, cos) | . ));t(ρ(θacc(sin, cos) | . )) 27jun;10_13;#1;D:20;m

0.3

0.3

0.2

15

0.2

15

0.1

0.1

10

0

10

0

−0.1

−0.1

5

−0.2 −0.3 −0.3

−0.2

acc

τacc

t(ρ(θacc | . ))

t(ρ(|acc| | . ))

−0.1

1.A

τ

0

0.1

0.2

0.3 −0.3

−0.2

−0.1

0

0.1

0.2

0

0.3

5

−0.2 −0.3 −0.3

−0.2

−0.1

2.

τ

0

0.1

0.2

0.3 −0.3

−0.2

−0.1

acc

τacc 0

0.1

0.2

0.3

27jun;10_13;#1;D:20;m t(ρ(|∆σ|[r] | . ));t(ρ(θV[r] | . )) 0.3

0.3 0.2

0.1

10

0

10

0.2

15

0.1

0

5

−0.1

−0.1

5

−0.2 −0.3 −0.3

−0.2

−0.1

1.B

τacc 0

0.1

0.2

0.3 −0.3

27jun;10_13;#1;D:20;m

−0.2

−0.1

τacc 0

0.1

0.2

0 −0.1 −0.2

−0.1

4.

τ|c| 0

0.1

0.2

0.3 −0.3

−0.2

−0.1

τ|c| 0

−0.2

0.1

0.2

0.3

τθ [reg] −0.1

0.1

0.2

0.3 −0.3

−0.2

τθ [reg] −0.1

0

0.1

0.2

0

0.3

V

27jun;10_13;#1;D:20;m

t(ρ(speed | . ));t(ρ(∆σ | . ))

10

0.3

8 6 4 2 0

0.2 0.1

5

0 −0.1 −0.2 −0.3 −0.3

−0.2

−0.1

5.

τ∆σ 0

0.1

0.2

0.3 −0.3

−0.2

−0.1

τ∆σ 0

0.1

0.2

0

0.3

27jun;10_13;#1;D:20;m t(ρ(speed | . ));t(ρ(|∆σ| | . ))

t(ρ(log(speed) | . ));t(ρ(log(|c|) | . )) 27jun;10_13;#1;D:20;m

8

0.3

0

V

t(ρ(speed | . ));t(ρ(|c| | . ))

0.1

−0.2

−0.3 −0.3

3.

0.2

−0.3 −0.3

−0.2

0

0.3

0.3

0.3 0.2

0.2

6

0.1

0.1 0

4

0

−0.1

−0.1

2

−0.2 −0.3 −0.3

−0.2

6.

τlog(|c|)

−0.1

0

0.1

0.2

0.3 −0.3

−0.2

τlog(|c|)

−0.1

0

0.1

0.2

−0.2 −0.3 −0.3

0.3

−0.1

τ

0

0.1

0.2

0.3 −0.3

−0.2

−0.1

|∆σ|

27jun;10_13;#1;D:20;mt(ρ(log(|∆σ|) | . ));t(ρ(log(|c|) | . ))

τ|∆σ| 0

0.1

0.2

0.3

27jun;10_13;#1;D:20;mt(ρ(log(|∆σ|) | . ));t(ρ(log(V) | . ))

6

6

0.2

0.1

4

0.1

2

−0.1

0

4

0

−0.1 −0.2

2

−0.2

−0.2

−0.1

τ|c| 0

0.1

0.2

0.3 −0.3

−0.2

−0.1

τ|c| 0

0.1

0.2

0.3

8 6 4 2 0

0.3

0.2

−0.3 −0.3

−0.2

7.

0.3

8.

84

0

−0.3 −0.3

9.

−0.2

−0.1

τV 0

0.1

0.2

0.3 −0.3

−0.2

−0.1

τV 0

0.1

0.2

0.3

Figure 3.16: PCCMs for different movement parameters, see details in text on page 81.

0


Chapter III

27jun;10_13;#1;D:20;m t(ρ(θV[r] | . ));t(ρ(speed[r] | . ))

27jun;10_13;#1;D:20;m t(ρ(θacc[r] | . ));t(ρ(|acc| | . ))

0.3

0.3

15

0.2

15

0.2

0.1

0.1

10

0 −0.1 −0.2 −0.3 −0.3

−0.2

10.A

τ∆σ[reg] −0.1

0

0.1

0.2

0.3 −0.3

−0.2

τ [reg] −0.1

0

0.1

0.2

0.3

5

−0.1

0

−0.3 −0.3

∆σ

t(ρ(∆σ[r] | . ))

10

0

5

−0.2

τ∆σ[reg]

−0.2

−0.1

11.A

t(ρ( | . ))

0

0.1

0.2

0.3 −0.3

τ [reg]

−0.2

−0.1

0

0.1

0.2

0.3

0

∆σ

t(ρ(∆σ[r] | . ))

0.3

t(ρ( | . ))

0.3

15

0.2

15

0.2

0.1

0.1

10

0 −0.1 −0.2 −0.3 −0.3

−0.2

10.B

τ∆σ[reg] −0.1

0

0.1

0.2

0.3 −0.3

−0.2

τ∆σ[reg] −0.1

0

0.1

0.2

0.3

5

−0.1

0

−0.3 −0.3

5

−0.2

τ∆σ[reg]

−0.2

−0.1

11.B 0.3

0.2

0.2

10

0.1

0

0.1

0.2

0.3 −0.3

27jun;10_13;#1;D:20;m

0.3

τ∆σ[reg]

−0.2

−0.1

0

0.1

0.2

0.3

t(ρ(θV[r] | . ));t(ρ(∆σ[r] | . ))

20 10

0

5

−0.1

0

0.1

0

−0.1

−0.2 −0.3 −0.3

10

0

27jun;10_13;#1;D:20;m t(ρ(speed[r] | . ));t(ρ(θV[r] | . ))

12

85

−0.2

−0.2

τθ [reg] −0.1

0

V

0.1

0.2

0.3 −0.3

−0.2

τ [reg] −0.1

0

θ

V

0.1

0.2

0.3

0

−0.3 −0.3

13.A

τ

−0.2

−0.1

0

0.1

0.2

[reg]

0.3 −0.3

speed

τ

−0.2

−0.1

0

0.1

0.2

[reg]

0.3

0

speed

t(ρ(speed[r] | . ))

t(ρ( | . ))

20

0.3 0.2 0.1

10

0 −0.1 −0.2 −0.3 −0.3

13.B

τ

−0.2

−0.1

0

0.1

0.2

[reg]

speed

0.3 −0.3

τ

−0.2

−0.1

0

0.1

0.2

[reg]

0.3

speed

Figure 3.17: PCCMs for different movement parameters, see details in text on page 81.

0


Chapter III

86

One can see from 5. that equiaffine velocity is represented more strongly than speed in the activity of this unit. Moreover, when conditioned on the direction of motion or on the direction of acceleration, a clear stripe of significance corresponds to equiaffine velocity2 . The 2/3 power-law in the logarithmic form: log V = log |σ| ˙ + β log c motivated us to depict PCCMs for log(equiaffine velocity), log(speed) and log(curvature). Stripes in the plots with such PCCMs also contain significant stripes corresponding to equiaffine velocity. More examples are depicted in Appendix H. The data that we analyze are different from the data analyzed in (Stark et al., 2006): it is non-uniform in terms of the movement directions. Some directions occur more often than others, see for example histograms in figure 3.12. The PCCM method accounts for linear relationship between the variables and therefore additional tests should be performed for non-uniformly distributed data. Following (Stark et al., 2006), we simulated the neural activities, which are noisy version of the linear dependence based on the equiaffine velocity. The PCCMs showed preferred representation of the equiaffine velocity at the time-lag used in simulation. The simulations based on the firing rate being cosine tuned to the direction of acceleration also resulted in the stripes for the equiaffine velocity in their PCCMs. However, in the actual data we do not observe preferential representation of the direction of acceleration, as we do in the PCCMs for the simulated data. Anyway, more tests based on simulated firing rates should be performed. What is encoded in the activity of the exemplar unit, for which PCCMs provide evidence for stronger representation of equiaffine speed in their activity when partialed on tangential velocity? The directionally decomposed tuning relationship with tangential and equiaffine speeds for the unit from figures 3.16, 3.17 is depicted in figure H.23. This unit has directional preferences. Still, as we saw from PCCMs, it does represent equiaffine velocity even if equiaffine velocity is conditioned on direction of motion, or direction of acceleration (figures 3.16(3); 3.17(10, 11, 13)). In figure H.23, the depicted are: directionally decomposed kinematic parameters - tangential speed (A), and equi-affine speed (B): corresponding spike counts, duration of twodimensional quanta, firing rates. Circle in the center of the decomposed equiaffine speed in (B) corresponds zero equiaffine speed (straight path or no motion). Onedimensional tuning curves are depicted in the upper row. Black tuning curves for the firing rate correspond to estimation based on the directionally decomposed data: P P Sk i i Sk i P fk = i Tk i 6= . Tk i i

2

Note that we have shown above that for some units their tuning curves for some movement parameters are similar to the tuning curves obtained from Poissonian spike trains based on cosine directional tuning only, see figure 3.11. Therefore we disambiguate the directional component in the comparison of the strength of representation of the Euclidian and equiaffine speeds by means of PCCM. In case of this unit, disambiguation indicates that the representation of equiaffine velocity in the activity of this unit is not a result of cosine tuning


Chapter III

87

27jun00. Unit: 10−13, # 1; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 181.2; Tgv

x 7.83

x 2960

1


Scaled

0.8 0.6 0.4 0.2 0

0

0.2

0.4

0.6

0

0.2

0.4

0.6


# of quanta, x 75.5137

90

90

1

120

0.8

0.8

0.6 150

0.4

30150

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 12.7667

0.6

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

27jun00. Unit: 10−13, # 1; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 192.4; AffVel

x 8.559

x 3479

1


Scaled

0.8 0.6 0.4 0.2 0

−0.5

0

0.5

1

1.5 −0.5

0

0.5



90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 21.8449

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270

Figure 3.18: Directionally decomposed kinematic parameters - tangential speed (A), and equiaffine speed (B): corresponding spike counts, duration of two-dimensional quanta, firing rates. Circle in the center of the decomposed equiaffine speed in (B) corresponds zero equiaffine speed (straight path or no motion). One-dimensional tuning curves are depicted in the upper row. Black tuning curves for the firing rate correspond to estimation based on the directionally decomposed data: P P Sk i i Sk i P fk = i Tk i 6= Ti i . The plots are for the same unit as in figures 3.16, 3.17. i


Chapter III

88

Finally, we performed PCCM analysis, the same as in figures 3.16, 3.17 for all 138 units in our disposal. The results (time-lags) were put in tables 3.1, 3.2, 3.3, 3.4. Empty entries correspond to no stripe in the corresponding PCCM. Each row corresponds to one unit. The meaning of the columns is as follows. 1. Ordered number of the unit. 2. Recording session. 3. Unit. 4. Direction of motion conditioned on the direction of acceleration (same as 1.A and 2, left). 5. Direction of acceleration conditioned on the direction of motion (same as 1.A and 2, right). 6. Speed conditioned on the equiaffine velocity, combined from 5, 7, 9. 7. Equiaffine velocity conditioned on speed, combined from 4, 5, 7, 9. 8. Euclidian speed conditioned on the equiaffine speed. 9. Equiaffine speed conditioned on the Euclidian speed. 10. Curvature conditioned on speed or on equiaffine velocity, combined from 4, 6, 8. 11. Speed conditioned on movement direction. Combined from 1, 10, 12, 13. 12. Equiaffine velocity conditioned on movement direction. Combined from 3, 10, 13. 13. Magnitude of acceleration (1B, left is used). Bold text corresponds to the parameters which are represented more strongly than the other parameters in the group. The groups are: (θV el /θAcc , θAcc /θV el ); (V /σ, ˙ σ/V ˙ , c/V, σ); ˙ (V /θ, σ/θ). ˙ Two values for curvature correspond to the time-lag from 4 (left value), and to the time-lags from 6, 8 (right value). One can see that there are quite a few units whose PCCMs show stronger representation of the equiaffine velocity over the Euclidian speed. However, the number of such units with simultaneous representation of equiaffine velocity partialled by movement direction is noticeably smaller.


#

Day

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

26jun00 26jun00 26jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 27jun00 28jun00 28jun00 28jun00 28jun00 28jun00 28jun00 28jun00 28jun00 29jun00 29jun00 29jun00 29jun00 30jun00 30jun00 30jun00 30jun00 30jun00 30jun00 30jun00 30jun00 30jun00

Unit 3 10 40 50 10 13 12 20 21 30 31 40 42 50 61 71 81 82 40 41 50 51 60 62 70 71 10 11 50 51 10 11 22 31 30 32 50 53 60 80

Chapter III

89

θV el /θAcc

θAcc /θV el

V /σ˙

σ/V ˙

V /|σ| ˙

|σ|/V ˙

c/V, σ˙

V /θ

σ/θ ˙

4

5

6

7

8

9

10

11

12

13

0.1

0.12

0.1

0.12

0.1

0.12

-0.26

0.2

0 -0.05

0.15

0.12

0.1 0 0 0.2 0

0.05 0.15 0.1

0 -0.05 0.18 -0.3;0.05 0

-0.3;0 0.08 0.18

0.3

0.2

0.05 -0.05

0.1

-0.26

0.1

0; 0.2; -0.05

0 -0.05

0.05

;-0.07

0.2 0

|Acc|

0.05 0.05 -0.08

0.08; -0.05 0.3

0.15 0

-0.25 0.22 0.15

0.25 -0.05

-0.05

-0.3 0

0.08 -0.2 -0.12 -0.07 -0.25 0

-0.07

-0.3

0.2

-0.1

0

-0.5; -0.05;-0.15

0.2

0.08 -0.2

-0.12

0.3 -0.3 0

-0.2 -0.15

-0.2

-0.08 -0.15; 0.15

0.1

0.1 ;-0.1

-0.1

-0.16

Table 3.1: Time-lags for the stripes from the PCCMs, for different parameters. See more explanations in text on page 88.


#

Day

1

2

Unit 3

38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69

02jul00 02jul00 02jul00 02jul00 02jul00 02jul00 02jul00 02jul00 03jul00 03jul00 03jul00 03jul00 03jul00 03jul00 03jul00 03jul00 03jul00 03jul00 04jul00 04jul00 04jul00 04jul00 04jul00 05jul00 05jul00 05jul00 05jul00 05jul00 05jul00 05jul00 05jul00 05jul00

40 50 52 60 62 70 71 80 10 11 30 43 50 61 62 71 80 81 21 61 62 71 82 11 30 31 41 60 70 71 80 82

Chapter III

90

θV el /θAcc

θAcc /θV el

V /σ˙

σ/V ˙

V /|σ| ˙

|σ|/V ˙

c/V, σ˙

V /θ

σ/θ ˙

|Acc|

4

5

6

7

8

9

10

11

12

13

0.3

-0.3

61

33

;0.18 0;

-0.05 0.15

63

-0.05 0.15 -0.12 0.25;0.1

0.09

-0.08 0.15 -0.12 0.1

0.06 0

0.1

0.15

;-0.08

;-0.05 0.1

0.09

0.15

-0.3

0.05

0.1

23

-0.03

0.12;

-0.06 -0.2

;0.12 ;-0.1

-0.12

;-0.03



#

Day

1

2

70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102

06jul00 06jul00 06jul00 06jul00 06jul00 06jul00 06jul00 09jul00 09jul00 09jul00 09jul00 09jul00 09jul00 10jul00 10jul00 10jul00 10jul00 10jul00 10jul00 10jul00 10jul00 10jul00 10jul00 10jul00 11jul00 11jul00 11jul00 11jul00 11jul00 11jul00 11jul00 11jul00 11jul00

Unit 3 11 20 21 33 41 51 61 11 31 41 60 61 62 71 11 20 30 50 51 60 61 70 71 80 81 11 21 22 41 43 60 61 70 80 81

Chapter III

91

θV el /θAcc

θAcc /θV el

V /σ˙

σ/V ˙

V /|σ| ˙

|σ|/V ˙

c/V, σ˙

V /θ

σ/θ ˙

|Acc|

4

5

6

7

8

9

10

11

12

13

-0.25

-0.25

-0.3

0

0.1

0.25 0.1

;0.03 0

0

0.25 0.17

;-0.8

;0.08 0.1

0.1

0.12

0.2

0.1

0.12

0.2

-0.25 0.25 0

0

0

;0.05 -0.03; 0.03



#

Day

1

2

103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138

16jul00 16jul00 16jul00 16jul00 16jul00 16jul00 16jul00 16jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 17jul00 18jul00 18jul00 18jul00 18jul00 18jul00 18jul00 18jul00 18jul00 20jul00 20jul00 20jul00 20jul00 20jul00 20jul00 20jul00

Unit 3 10 12 21 51 70 71 80 81 1102 1105 3101 3103 4101 5101 5104 6101 6105 7101 7103 8101 8102 10 50 51 52 61 70 71 80 10 11 20 30 40 41 80

Chapter III

92

θV el /θAcc

θAcc /θV el

V /σ˙

σ/V ˙

V /|σ| ˙

|σ|/V ˙

c/V, σ˙

V /θ

σ/θ ˙

|Acc|

4

5

6

7

8

9

10

11

12

13

-0.3 & 0.3

0.3 0.2 0.1

-0.2 0.12

0.12

0.12

-0.06

0.15 0.1;

-0.03

0.07

-0.03

0.12

0.12 0.1 0.25

0.12 0.08

0.12

-0.1

0

0.1 3102 3104 5102 6102 6106

0.12 0.08

0.12

-0.1

0

-0.05 -0.05 0.05

0.12 0.1

-0.05 -0.2

7104 8105 8104

0; -0.06 -0.2

-0.25 -0.25

0

0.1 0.15 -0.1 0.2 0.2 -0.1

0.12 -0.05 0.08 0.08 -0.2

0.15 0.08 0.25

-0.2

-0.2

-0.3 -0.3 0.3

0.1

0.07 0

-0.1 0.22 -0.25

0.15

0.1

0.1

-0.05

-0.1

-0.05

0.2 0

0.25 -0.05

0.25 0

0.1;

0.2

0.1 0.05 -0.1 0.2

0.15 -0.05

0.2 ; 0.1


0.1


Chapter III

93

We summarize the observations in tables 3.1 - 3.4 in the histogram presented in figure

Figure 3.19: Here the results presented in the tables 3.1 - 3.4 are summarized. For each type of data analysis (4-13), the number of units with dominant and non-dominant representation are shown. Numbers of such units are given in the table below the histogram.

3.4

Discussion

Our study indicates that some single neurons in the premotor and primary motor cortices may encode movements rather in terms of equiaffine than in terms of Euclidian geometry. We showed that there exist dorsal premotor and primary motor cortical units whose activity is correlated stronger with equiaffine speed rather than with Euclidian speed. There also exist neurons for which both equiaffine and tangential speeds have similar strength of representation, and neurons which represent Euclidian speed stronger than equiaffine. All in all, in this chapter we present methods that can be used in the analysis of neural representation of spontaneous drawings: analysis of tuning properties of units related to a single movement parameter, tuning properties with respect to a pair of movement parameters, simulation of Poissonian spike trains based on a tuning model,


Chapter III

94

regression equations for stereotypical drawing patterns, log-likelihood of change in the neural activity (the method was proposed by Moshe Abeles in personal communication), use of conditional mutual information and partial correlation to disambiguate tuning preferences of the single units. The majority of the analyzed neurons possessed directional sensitivity, as seen in their directional tuning curves (e.g. figure 3.10, first column). The directionally tuned neurons were also tuned to other movement parameters, e.g. curvature or speed. Such a “multi-parametric” tuning can be explained by stereotypy in the drawings after a period of practice (see figure 2.2, parabolas in figure 3.13) together with the 2/3 power-law relating the speed of drawing with its curvature. Emergence of stereotypical movements resulted in the emergence of regularities between movement direction and non-directional movement parameters (e.g. speed, see figure 3.12A, where directionally decomposed tangential speed is not uniform w.r.t. movement direction). Therefore apparent tuning to the equiaffine parameters, such as equiaffine velocity or equiaffine curvature, could be a byproduct of nothing more than a directional tuning (we show further by means of the partial cross-correlation method that this is not the case for a number of units). In addition, the existence of a certain type of tuning relationship does not imply representation of a specific movement parameter in the neural activity and can be due to correlation between different movement parameters. Our test consisted of simulating Poissonian spike trains according to a cosine tuning model based on the recorded drawing movements. We saw that the tuning curves for the non-directional movement parameters were similar to the Poissonian tuning curves. The partial cross-correlation analysis enables us to disambiguate the linear relationship between the considered movement variable and the correlation between another movement variable with the neural activity. We showed by means of PCCMs that equiaffine velocity and other parameters are represented in the activity of some of the units, when partialled on movement direction A pivotal question of our study is whether the metric in which movements are represented in the neural activity is Euclidian or equiaffine? Tangential and equiaffine speeds are correlated at small time-lags (figure 3.9) and disambiguating should be performed. As we found, the conditional mutual information did not give a clearcut answer when the tangential and equiaffine speeds were considered at the same time-lags from the neural activity. In a recent study partial correlation was applied in order to resolve the velocity-acceleration ambiguity (Stark et al., 2006). The clear-cut answer about which parameter is represented more strongly followed from (1) considering two independent time-lags for different movement parameters and (2) considering the significance of the partial correlation and not only its inter-trial average. Calculation of the partial cross-correlation has several advantages over calculation of conditional mutual information. There is no need to estimate the three(!)-dimensional joint probability distributions. Estimation of significance for correlation coefficients


Chapter III

95

can be readily performed based on standard statistical methods. Our analysis based on partial cross-correlation matrices revealed neurons in the areas PMd and M1 with the following property: equiaffine speed was represented in their activity more strongly than the Euclidian one; when conditioned on movement direction, the significant trace of representation was still there (therefore directional tuning is not the source of representation!). Interestingly, velocity or acceleration directional (when conditioned on the other parameter) preferences were found for a very few units. The data we analyzed are very non-uniform, as can be seen from the spike counts and durations of quanta in figures related to tuning relationships. The PCCM method accounts for linear relationship between the variables and therefore additional tests should be performed for non-uniformly distributed data. We performed several simulations, which also showed stronger representation of the parameter encoded in the simulated neural activity, and the time-lag was reconstructed correctly in the PCCMs. Interestingly, directionally decomposed equiaffine speed was always distributed more uniform w.r.t. movement direction than tangential speed, see figure 3.12 as an example. If equiaffine metric is used in the representation of movements, more uniform distribution of the equiaffine invariant w.r.t. the movement direction may mean that the movements are not as constrained in equiaffine metric (for example, due to inertial properties of the limbs) as they are in the Euclidian metric. We show in chapter 2 that with practice monkey movements become piece-wise parabolic, and that parabolas are clustered into three clusters with respect to their orientation. Considering parabolic pieces as path templates to which monkey movements converge with practice, or even as geometric primitives, the method of PCCMs can be applied to separate clusters of parabolic pieces. Thus neural encoding of different clusters of geometric primitives would be considered separately. Similarly, PCCMs should be applied to the segments from different states of HMM (chapter 5). Additional analysis should be implemented in order to understand what is there behind stronger representation of equiaffine velocity over tangential one revealed by partial correlation method: whether an additive form with certain weights relates speeds in two metrics with the neural activity, or multiplicative form, or combined, or something else.

Chapter 4 Decision making should be considered in the study of movement primitives Abstract The movement primitives paradigm gets more and more attention in the fields related to biological and robotic movements. The underlying assumption is that the brain evolves in its development as a parsimonious system for perception of the environment and for control of body actions. An attempt to give a rigorous definition of movement primitive is an attempt to solve a “chicken and egg” problem: in order to show which are the movement primitives one should find all their possible different types, but how does one know that what he has found constitutes all possible movement primitives if he does not have a formal definition for it? In case one provides a constructive definition of movement primitive, this would imply that movement primitives exist in the biological movement control systems and are of specific form. However, so far no study has succeeded to formalize answers to the questions ’what’ and ’how’ in the entire bunch of different biological movements. Claiming that movement primitives exist as an elementary finite alphabet of movements because their existence would make the control parsimonious establishes framework for the answers but does not introduce explicit descriptions and definitions. In this chapter we take a novel axiomatic framework for the study and formalization of the notion of movement primitives by considering (1) decision-making during task performance and (2) indivisibility. Assuming tunable movement primitives, the decision-making process influences the choice of parameters of a primitive and the way different primitives are composed. We show that the monkey indeed makes decisions based on getting/not getting a reward. By locking the activity of single neurons on the event of getting a reward, existence of the motor cortical neurons that alter their activity with respect to the event is shown. All in all we propose that taking decisionmaking into account is important in studies of the nature of the movement primitives. We discuss the following definition of movement primitives: movement primitive is a movement entity, which cannot be intentionally stopped unaccomplished after its initiation. During a number of recording

96


Chapter IV

97

days, after getting a reward at specific locations, the monkey tended to stop along parabolic-like path, which supports our claim that parabolas may be movement primitives that emerge during extensive practice.

4.1

Introduction

In our attempt to answer the question: what is represented and how in the motor cortical activities, we treat the representation of plausible elementary building blocks of movements — the so called movement primitives, together with the representation of the instantaneous movement signal (for latter see chapter 3). Consider the simplest possible motor invariant — planar point-to-point movements. They are known to be almost straight, with bell-shaped speed profiles, and isochronous, all these properties are captured by a single optimality principle called the minimum-jerk model (Flash & Hogan, 1985). The studies of double-step movements (Flash & Henis, 1991; Flash et al., 1996) report vectorial superposition of two point-to-point movements as the result of the switch in the first target prior to task completion. Thus, straight trajectories with bell-shaped speed profiles is a fair candidate for movement primitives, with a simple vectorial compositionality rule. In the center-out (Georgopoulos et al., 1982), double-step (Flash & Henis, 1991), and many other experimental paradigms in studies with humans and monkeys, the subject gets an explicit cue for every temporal and spacial element of her/his/its movement. Therefore, the movements can be easily decomposed in the point-to-point components. Apparent submovements that underlie the generation of continuous arm motions were identified during experimental studies in patients recovering from a single cerebral vascular accident. Kinematic analysis demonstrated the existence of a submovement speed profile that is invariant and could be detected in the drawing movements of patients with different brain lesions (Krebs et al., 1999). However, how does the subject comprise spontaneous movements, in the absence of any explicit cues? The choice of appropriate submovements in free goal-directed drawings, might be based on the decision-making process. The scribblings that we analyze are indeed free and goal-directed: the monkey modifies its movements according to the feedback related to the goal (what the goal is, will be specified in the Results). Monkey behavior is not ruled by any explicit cue. We have identified the candidates to movement primitives in chapter 2 and now wonder how the monkey chooses a specific piece of movement from many different possibilities. For example, consider a cluster (defined by the orientation of parabolic pieces) from some day in figure H.3. The free dimension of a cluster corresponds to variations in the focal parameter of parabolic pieces. Locations of their vertices variate along the direction of the normal, see for example figure H.9.


Chapter IV

98

Decision-making often requires the accumulation and maintenance of evidence over time. Previous work (Shadlen & Newsome, 2001) has suggested that neural activity in the lateral intraparietal area (LIP) of the monkey brain reflects the formation of perceptual decisions in a random dot direction-discrimination task in which monkeys communicate their decisions with eye-movement responses. It was demonstrated that LIP is involved in neural time integration underlying the accumulation of evidence in this task (Huk & Shadlen, 2005). Temporal integration may be a fundamental computation underlying higher cognitive functions that are dissociated from immediate sensory inputs or motor outputs (Huk & Shadlen, 2005). The clearest neural correlate of working memory during frequency discrimination is found in the prefrontal cortex, which contains neurons that increase their activity in a frequency-dependent manner during the delay period between the two flutter stimuli. This activity does not seem to be related to the impending motor response. The prefrontal cortex might not be the only structure in which such a mnemonic correlate exists, as neurons with similar activity have been found in the secondary somatosensory cortex and in the medial premotor cortex (Salinas, 2003). In this chapter we test the hypothesis that the monkey had to modify its drawings according to getting/not getting a reward. In other words, consider that the monkey had performed scribblings in a certain part of the workspace and was not rewarded. If it continues to search in that same area, its chances to get a reward are low comparing with the chances to get a reward in a novel part of the workspace. We also test whether getting a reward is reflected in the activity of single neurons.

4.2

Methods and Results

How do we dissociate motor outcomes of different decisions? The behavior of the monkey became more regular with practice, as is evidenced by the emergence of clusters of parabolic pieces. Emergence of the regularities in behavior imply existence of movement strategies in the well-practiced movements. Monkey movements are caused by its desire to get a reward. We ask the following question: is the developed movement strategy influenced by the behavioral event — the monkey got a reward? This is a plausible consequence because if the monkey explores only part of the workspace, does not get a reward there, and does not enter the unexplored parts, the probability of getting a reward is smaller comparing with the strategy in which the monkey visit an unexplored part of the workspace. We propose that the monkey may need to tune the parameters of the primitives it uses due to getting/not getting the reward. In order to show the differences in the motor behavior based on the above-mentioned event, we propose below a method of dissociating between the trajectories following two types of the plausible behavior-influencing event: getting or not getting a reward.


4.2.1

Chapter IV

99

Reward-related dissociation between the behaviors

We propose that the reward-related change of the movement strategy influences the area where the monkey scribbles. Note also that scribbling in the distal1 (from the monkey) part of the workspace needs more physical effort than scribbling in the proximal (from the monkey) part. Therefore we dissociate the “rewarded” trajectories (pieces of drawing that follow rewarding the monkey) from non-rewarded ones based on the metric proximity of the rewarded locations ({(x(ti ), y(ti ))}, the reward was obtained at times ti ). The targets for monkey O and for monkey U were different: 19 overlapping circles for monkey O, as in figure 1.1B, and 19 non-overlapping hexagons as in figures 4.1, 1.2. Correspondingly, the procedures of comparison of the rewarded and nonrewarded trajectories are slightly different for the two monkeys. The monkey ought to get rewards near the boundaries of the targets. The locations where both monkeys actually got the reward do not exactly lie near the grid, see for example figure 4.1B. We cluster the rewarded locations according to proximity. Monkey U The rewarded locations are labelled with the indices of the corresponding targets (1-19), figure 4.1C. Given a label of the target (1-19), we define the “likely” area for the monkey to get a reward related to the given label as follows. Consider all locations where the reward associated with the label was obtained. The likely area is defined as an ellipse with the center at the mean of the considered locations, and with lengths of the principle axes beingqthe unbiased standard deviation along the corresponding direction: Pn 2 vaxis, i = j=1 (rj · vj ) /(n − 1) vi , exemplar ellipses are depicted in figure 4.1D. Monkey O The rewarded locations, due to overlap of the targets, are clustered without any a-priori knowledge (of the labels) with the normalized cut algorithm. Locations of the points and the desired number of clusters is the only input to the algorithm. An exemplary result of the clustered rewarded locations is depicted in figure 4.2. Kinematic study of the primitives in the framework of decision-making We indeed observed many cases when the rewarded trajectories differ from the nonrewarded ones. Two examples with paths and speed profiles of rewarded and nonrewarded trajectories are depicted in figure 4.3. The paths and speed profiles of 1

The lowest target (figure 4.1A) is supposed to be the closest to the monkey. Therefore, by distal locations we mean the locations further from the lowest row of targets, and by proximal locations those, which are closer to the lowest row of the targets.


Chapter IV

100

A

B 28jun00

Hexagonal grid

120

100

80

y position, [mm]

60

40

20

0

−20

−40 −80

−60

−40

−20

0

20

40

60

80

x position, [mm]

C

D 28jun00

120

120

100

100

80

80

60

60

y position, [mm]

y position, [mm]

28jun00

40

20

40

20

0

0

−20

−20

−40

−40

−100

−80

−60

−40

−20

0

20

x position, [mm]

40

60

80

100

−100

−80

−60

−40

−20

0

20

40

60

80

x position, [mm]

Figure 4.1: A. The hexagonal grid. B. The locations at which the rewards were obtained. They do not exactly form the grid. C. The rewarded locations are colored according the the label of the target they correspond to. D. The ellipses of the typical locations.

100


Chapter IV

101

06aug97 50

0

−50

−100 −80

−60

−40

−20

0

20

40

60

80

20

40

60

80

A 06aug97 50

0

−50

−100 −80

−60

−40

−20

0

B Figure 4.2: (A) An example of the rewarded locations from a day of well-practiced (#17) recordings of monkey O. (B) The rewarded locations clustered and the corresponding ellipses.


Chapter IV

102

the rewarded trajectories, with rewards being inside the same ellipse, are depicted in the left column. Paths and speed profiles of 8 randomly selected non-rewarded trajectories crossing the same ellipse are depicted in the right column. Zero in the plots with speed profiles corresponds to the time instant of getting rewards (rewarded trajectories) or to the time instant equal to the average of the time instants of crossing the ellipse boundary (non-rewarded trajectories). One can see that the rewarded and the non-rewarded trajectories are different. Hypothesis #1: the monkey might expect that it is going to get a reward Below we present an evidence for the observation that the monkey might have expected that it is going to get a reward at distant locations of the workspace. This observation is interesting for us because this is an additional argument for the online decision making and involvement of working memory based on getting/not getting a reward, it also supports the claim about sequential behavior of the monkey. Our reward-related observations open a new way of thinking about trajectory formation in this task. There are totally 19 ellipses, that are related to the rewarded locations, as described earlier. For each non-rewarded crossing of the ellipse we calculate the average of the time of crossing the boundary of the ellipse (time of entering and time of leaving) tnr . For well practiced movements of monkey U, we plotted the histograms — one histogram for an ellipse — of the time it takes the monkey to get from the location where it got the reward to the time instants {tnr } for the corresponding ellipse (there are no other rewards on the way, of course) and of the time. The time it takes the monkey to get from the rewarded locations to the following rewarded locations are depicted on the same plots by green. The histograms for one day of recordings are shown in figure 4.4. Locations of the histograms correspond to the locations of the targets in figure 4.1A. Considering the three most distal targets, there is a time interval ≈ [0.5 2] sec. in which the monkey did not get any rewards. This phenomenon appears in 10 days for which we have neural data out of the 17 analyzed. Our hypothesis is that following 2 seconds or more of scribbling, the monkey expected that it is going to get a reward and therefore it produced the movement towards those distal targets. Indeed, the end-points of many segments from state 6 in figure 5.6A fall near these three most distal ellipses and are related to getting reward, see figure 5.6E, that is getting reward in these locations corresponds to the end of the hidden state of HMM (careful description of HMM is provided in chapter 5). After getting a reward, the monkey may first explore the proximal part of the workspace, and then decide to explore the distal part of the workspace. Intersections with the three distal ellipses that occur shortly (sooner than 0.5 sec.) after getting reward take place when the monkey was heading to the distal part of the workspace


Chapter IV

103

A 26jun00, target #16 Reward was obtained

analyze_trajs_around_rewards.m . Reward was not obtained, passed through the ellipse. 100

y position − [cm]

y position − [cm]

100

50

0

−50

50

0

−50

−100

−50

0

50

100

−100

60

Speed, [cm/sec]

70

60

Stop

50 40 30 20 10

Reward 0

−50

0

50

100

x position − [cm]

70

50 40 30 20 10

0.2

0.4

0.6

0.8

1

0

Time from the event, [sec.]

0.2

0.4

0.6

0.8

1


B analyze_trajs_around_rewards.m . Reward was not obtained, passed through the ellipse.

120

120

100

100

80

80

y position − [cm]

y position − [cm]

26jun00, target #1 Reward was obtained

60 40 20 0 −20 −40

60 40 20 0 −20 −40

−60

−60

−80 −100

−50

0

50

−80 −100

100

x position − [cm]

−50

0

50

100

0.6

0.8

x position − [cm]

60

60

50

50

Speed, [cm/sec]

Speed, [cm/sec]

Speed, [cm/sec]

x position − [cm]

40

30

20

10

40

30

20

10

−0.2

0

0.2

0.4

0.6


0.8

1

−0.2

0

0.2

0.4

1


Figure 4.3: Two examples (A and B) of the rewarded (left column) and non-rewarded (right column) trajectories: their paths and speed profiles. The ellipse with rewards is depicted by blue. Zero time at the graphs with velocity profiles corresponds to the time of getting reward. By event we mean either the time of getting a reward (rewarded trajectories), or the mean time of crossing the ellipse (non-rewarded trajectories).


Chapter IV

104

and after getting a reward in the nearby target did not change its strategy within such a short time interval. As the histograms in figure 4.4 show, the monkey never gets to the distal targets before at least 2 seconds of explorations in the more proximal part of the workspace. Note that a cycle (closed drawing) is performed usually within about 1.5 seconds at a normal speed. In figure 4.5 two examples (from two different recording days) of the rewarded and non-rewarded trajectories are depicted, for ellipse corresponding to the target located in the proximal part of the workspace. The meaning of the plots is the same as in figure 4.3. All rewarded trajectories stay in the proximal part of the workspace. Part of the same number (as rewarded trajectories) randomly chosen non-rewarded trajectories pass through the distal part of the workspace. An alternative hypothesis (#2): monkey’s strategy might be based on stopping in a specific location in the workspace after getting a reward, during certain days of practice An alternative explanation to the observations presented above may be the following: after getting a reward, the monkey tended to stop in the specific part of the workspace. Such stops also provide an explanation to the time interval during which the most distal locations were not visited after a reward was obtained, see figure 4.4. PSTH for times of getting a reward Our following step in the analysis of how the behavior is altered by getting the reward, which we suppose is related to the process of decision-making, was to observe the neural activity locked on the time when the reward was obtained. Out of totally 138 units analyzed, we found 5 units with clear sensitivity to reward. The figures 4.6, H.29, H.31, H.33, H.35 contain the PSTH for these 5 sensitive units (1 unit for a couple of figure; the first figure in the couple corresponds to scribbling movements, the second figure corresponds to center-out movements). Part of the figures was put in Appendix H. Each figure contains 4 raster plots. The upper left corner corresponds to the actual activity of the rewarded trial. The plots in the lower left corner correspond to the actual activity of the non-rewarded trials. The plots on the right correspond to Poissonian spike trains, which are based on the directional tuning curves obtained for scribbling task. Poissonian spike trains were simulated as described on page 55. The spike trains are sorted according to the label of the reward. Average firing rate and the log-likelihood of surprise in the firing rate are depicted below the raster plot correspondingly. The average and median speed profiles, for every label of the targets, are on the right from the raster plot. Zero time corresponds to the time of getting a reward. In PSTH for the non-rewarded trials, zero corresponds to the average of the time of crossing the ellipse boundaries. One can see that the neural activity for the


Chapter IV

28jun00. Non−rew/rew. misses:29/1 Non−rew/rew. misses:41/1 Non−rew/rew. mean=3/4.4 Non−rew/rew. mean=3.3/3.8 30 30 20

20

10

10

105

Non−rew/rew. misses:25/3 Non−rew/rew. mean=3.8/3.4 10

5

0 0 5 Non−rew/rew. misses:44/2 Non−rew/rew. mean=2.7/4 40

0 10 15 0 5 Non−rew/rew. misses:53/1 Non−rew/rew. mean=3.1/3.3


10 15 Non−rew/rew. misses:46/3 Non−rew/rew. mean=3.5/3.7

30

30

30

20

20

20

10

10

10

20

0


0 5 Non−rew/rew. misses:29/0 Non−rew/rew. mean=3.5/4.1 10

40

5

20

0

0

20

0 10 15 0 5 Non−rew/rew. misses:23/1 Non−rew/rew. mean=3/3.4

30

30

20

20

10

10

10 15 Non−rew/rew. misses:40/1 Non−rew/rew. mean=3.5/4.3 20

10


5


0 10 15 0 5 Non−rew/rew. misses:54/1 Non−rew/rew. mean=2.7/1.9 60



20

40

10

20

10

40 10 20 0

0


5


15

60

10

40

5

20


10

15

40

20

0

0 0 5 10 15 0 Time, [s]. function: analyze_trajs_around_rewards.m

5

10

15

0

0

5

10

15

Figure 4.4: As described in text above, an ellipse corresponds to the rewarded locations associated with a specific target. There are totally 19 targets, implying 19 ellipses. In this figure the depicted are the histograms of how long it takes for the monkey to get into an ellipse after the last reward obtained. The time of getting into the ellipse is calculated as the average of the time of crossing the boundaries of the ellipse. The time for trials, for which the monkey got a reward, are depicted by green. One can see that in the three most distal targets there is a time interval ≈ [0.5 2] sec. in which the monkey did not get any rewards. The monkey got rewards shortly before entering an ellipse (shorter than half a second), and when the time between getting reward and entering ellipse was larger than 2 seconds. This phenomenon appears in 10 days for which we have neural data out of 17 analyzed. Our hypothesis is that the monkey predicted that it was going to get a reward after it had explored more proximal areas during, for instance, 2 seconds.

15


Chapter IV

106

A Reward was obtained

Reward was not obtained, passed through the ellipse.

120

120

100

100

Silent area

80

y position − [cm]

y position − [cm]

80 60 40 20 0 −20 −40

60 40 20 0 −20 −40 −60

−60 −80 −100

−50

0

50

100

−80 −100

−50

80

70

70

60

60

Speed, [cm/sec]

Speed, [cm/sec]

80

50 40 30

50

100

50 40 30

20

20

10

10

0

0

x position − [cm]

x position − [cm]

0.5

1

0


0.5

1


B 28jun00, target #18 Reward was obtained

analyze_trajs_around_rewards.m . Reward was not obtained, passed through the ellipse.

120

120

100

100

Silent area

80

y position − [cm]

y position − [cm]

80 60 40 20 0 −20 −40

60 40 20 0 −20 −40

−60

−60

−80 −100

−50

0

50

−80 −100

100

−50

0

50

100

x position − [cm]

80

80

70

70

60

60

Speed, [cm/sec]

Speed, [cm/sec]

x position − [cm]

50 40 30 20

50 40 30 20

10

10

−0.2

0

0.2

0.4

0.6


0.8

1

−0.2

0

0.2

0.4

0.6

0.8

1


Figure 4.5: Two examples (A and B) of the rewarded (left column) and non-rewarded (right column) trajectories: their paths and speed profiles. The ellipse with rewards is depicted by blue. Zero time at the graphs with velocity profiles corresponds to the time of getting reward. All rewarded trajectories stay in the proximal part of the workspace. Part of the same number (as rewarded trajectories) randomly chosen non-rewarded trajectories pass through the distal part of the workspace.


Chapter IV

107

rewarded trials changes abruptly shortly (within 200 msec.) after getting a reward. The change (if it takes place) is different in the neural activity for the non-rewarded trials, and the difference is not explained by Poissonian spike trains, which depend on the geometric properties (direction of motion) of the trajectories. One can argue that the number of neurons sensitive to getting reward is small comparing with the total number of units analyzed (138); and that a change in the neural activity for the rewarded trajectories may simply reflect the projection from auditory cortex. However similar changes occur in case of the center-out movements (figures 4.7, H.30, H.32, H.34, H.36) for one unit only out of 5: #5, 62, 09jul00, see figures 4.6, 4.7. The neural activity in the figures corresponding to center-out movements is depicted for 2 seconds preceding getting the reward (0.2 sec. for scribblings) in order to cover the preparation period and the center-out movements entirely. Consider the parameters of the HMM depicted in figure 5.6. State 11 labels low maximal a-posterior probabilities that cannot be classified to the states 1-8, for which the model was learned. The peak near 0 in the histograms with numbers of rewards corresponding to state 11, figure 5.6, D, E, indicates that soon after getting reward the neural activity was altered in many cases. Such a peak was not observed in the histograms for other days, in which the reward-related activity was not revealed. We present PSTH for one more unit in figures H.37, H.38 to exemplify the changing firing rate, though the neural activity can be related rather to movement speed, as seen for both scribbling and center-out movements. The meaning of plots is the same as in other figures with reward-related PSTH.

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

4.7%

3

5.4%

4

5.4%

35 5

6.2%

23 29

41

7.8%

7

10.1%

59 3.9% 3.1% 3.1% 2.3% 5.4%

95 14

7.8%

5.4%

101

15 107 16 113 17 119

3.9% 2.3% 3.1%

18

5.4%

125 19

7%

PETH firing rate, [spikes/sec] 6

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

#20 is the total aver., median

2

5 1

4.7%

3

5.4%

4

5.4%

35 5

6.2%

29 4

41

0

7.8%

2

17 23

6

65 8 9 71 10 77 11 12 83 89 13

0

2

2

47 53

1

11

averaged (blue), max=0.44 [m/s]; median(black), max=0.47 [m/s]

2

17

0

7.8%

0

Normalized speeds for the targets

Rasters. 09jul00; # 5, 62; r1. P1


2

5 1 11

108


Rasters. 09jul00; # 5, 62; r1. P0 0

Chapter IV

6

7.8%

7

10.1%

47 6

53 59

8

10

12

65 8 9 71 10 77 11 12 83 89 13

3.9% 3.1% 3.1% 2.3% 5.4%

95 14

7.8%

5.4%

101

15 107 16 113 17

14

119

3.9% 2.3% 3.1%

18

5.4%

125 19

7%

16


5

18

0

1

2

2



4

6

8

10

12

14

16

18

4

4 3 20

Locations of the events Arrows ~ aver. veloc.

1

0

Likelihoods of change in firing rate, normed by: 0.65; 32 100

1 y, [mm]

0.8 0.6

0.6 0

0.2

0.4

−50

0.6

0.8 1 1.2 Time, [sec.] function: rewards_peth.m

1.4

1.6

1.8

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2.6%

0

3

6.7%

4

5.6%

355 5

6.3%

425 6

4.9%

495 7 565 8

5.8% 1.4%

9

6.8%

10 11

3.5% 3.9%

12

6.6%

705 775

13

4%

14

7%

915

1055 15

5.7%

1125 16

7.3%

1195 17

3.2%

1265 18

6.2%

1335

0

1

2

0.2

0.4

19

5.8%

PETH firing rate, [spikes/sec]

−50

0.6


1.4

1.6

1.8

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

4

6

2.6%

0

6.7%

4

5.6%

355 5

6.3%

425 6

4.9%

495 7 565 8

5.8% 1.4%

9

6.8%

10 11

3.5% 3.9%

12

6.6%

13

4%

14

7%

705

8

775

915 985

12

14

1055 15

5.7%

1125 16

7.3%

1195 17

3.2%

1265 18 1335

1

2

2

845 10

0

6.6%

3

635

50


2

215 285

0 x, [mm]


Rasters. 09jul00; # 5, 62; r0. P1 5 1

145 2

845

985

0

75 2


635

0

6.6%

215 285

Max. # of spikes: 37−> 30

50


2

75 2 145

0 x, [mm]


Rasters. 09jul00; # 5, 62; r0. P0 5 1

0 −50

0.2


50

Aver/0.56 All spikes/8

0.4

0.2

0

100

0.8

−50

0

Likelihoods of change in firing rate, normed by: 0.56; 8

1

50


0.4


1

y, [mm]

0

20

2

2

6.2%

19

5.8%

16



3

4

6

8

10

12

14

16

4 18

18

3

3 2

2

20

0

Likelihoods of change in firing rate, normed by: 0.46; 2.3 100

1 y, [mm]

0.8 0.6

0.2

0.4

100

0.8

50

0.6

0.6


1.6

1.8

2

−50

0.2 −50

1.4

0 x, [mm]

50

Max. # of spikes: 296−> 30 0

50 0

Aver/0.45 All spikes/2.3

0.4

Max. # of spikes: 298−> 30 0

Likelihoods of change in firing rate, normed by: 0.45; 2.3

1

−50

0.2 0


0


0.4

1

y, [mm]

0

20


1

0

0.2

0.4

0.6


−50 1.4

1.6

1.8

2

0 x, [mm]

50

Figure 4.6: PSTH for the time when the monkey got a reward, scribbling movements. See explanations in text on page 104.



−0.5

0

0.5

1

1.5

−1 0 5

1

2

20 3


9.7%

−1

30 4

17.7%

35 40 5

21%

50

6

19.4%

60


0.5

1

1.5


2

1

9.7%

4 20 3

30 4

17.7%

5

21%

35 40

10

45 12 50 55

14

2

4

14.5%

25

8

1

2

15 2

6

0

17.7%

10

6

19.4%

60 16


18

6

0

−1 0

14.5%

25

−0.5

2

17.7%

15 2

55

1

5

10

45

0


Rasters. 09jul00; # 5, 62; r1. P1


2

109


Rasters. 09jul00; # 5, 62; r1. P0 −1

Chapter IV

6

8

10

12

14

16

18

5 4

4

3

20

20

2

2

1 0



1 y, [mm]

0.8 0.6

100 50 0


0.4

0.6

−200

0 x, [mm]

200


0.4

−200

0

0.5 1 Time, [sec.] function: rewards_peth.m

1.5

0 −1

2

−0.5

0

0.5

1

−0.5

Normalized speeds for the targets 1.5

−1 0

1

2

5.5%

3

6.8%

20 25 4

35 40 45 5

26%

50 55 60 65 6

21.9%

70


6

−1

−0.5

0

0.5

1

0 x, [mm]

200

2



2

−1 0 5 1 10

2

27.4%

1.5

2

12.3%


5 1

0


Rasters. 09jul00; # 5, 62; r0. P1


2

0

15 4

2

5.5%

3

6.8%

20 25

6

30

4

35 40 45 5

26%

50 55

12

60 65 6

14

21.9%

70 16

PETH firing rate, [spikes/sec] 6 5

18

1

2

2

27.4%

8

10

0

12.3%


−0.5

Rasters. 09jul00; # 5, 62; r0. P0 −1

30

100 50 0

0.2

0 −1

15


0.8

0.2

10


1 y, [mm]

0

4

6

8

10

12

14

16

18

4 4

3 20

20 2

2

1 0



1 y, [mm]

0.8 0.6 0.4


−200

0.2 0 −1

100 50 0



1 0.8

y, [mm]

0

0.6 0 x, [mm]

200

0.4


100 50 0 −200

0 x, [mm]

200

0.2

−0.5

0


1.5

2

0 −1

−0.5

0


1.5

2

Figure 4.7: PSTH for the time when the monkey got a reward, center-out movements. See explanations in text on page 104.


4.2.2

Chapter IV

110

Movement primitive is a movement entity which cannot be intentionally stopped unaccomplished

120

120

100

100

80

80

y position − [cm]

y position − [cm]

Many examples of comparison between the rewarded and non-rewarded trajectories corresponding to certain targets (e.g. 1, 16, see for examples figures 4.3, 4.5) provide evidence for a decrease in movement speed to nearly zero in cases of the rewarded trajectories. One more example, with the same meaning of the plots as in figure 4.3 is depicted in figure 4.8. In all cases, the rewarded trajectories in this example tend to stop in the leftmost part of the trajectory, in contrast to the non-rewarded trajectories.

60 40 20 0 −20 −40

60 40 20 0 −20 −40

−60

−60

−80 −100

−50

0

50

100

−80 −100

−50

80

80

70

70

60

60

50 40 30

30

10

10

0.2

0.4

0.6

100

40

20

0

50

50

20

−0.2

0

x position − [cm]

Speed, [cm/sec]

Speed, [cm/sec]

x position − [cm]

0.8


1

−0.2

0

0.2

0.4

0.6

0.8

1


Figure 4.8: An example of the rewarded (left column) and non-rewarded (right column) trajectories: their paths and speed profiles. The ellipse with rewards is depicted by blue. Zero time at the graphs with velocity profiles corresponds to the time of getting reward.


Chapter IV

111

We propose the following possible definition of a movement primitive: movement primitive is a movement entity which cannot be stopped unaccomplished. Adopting the definition, one can note that the rewarded trajectories between the rewarded and stopping locations look similar to parabolas, providing an additional evidence for parabolas being movement primitives acquired during extensive practice. The non-rewarded trajectories may thus be a composition of the corresponding parabolic movements with another movement component.

4.3

Discussion

We have shown in this chapter that 1. Monkey trajectories are influenced by getting/not getting a reward. 2. During some days of practice, monkey U pursues the following strategy: after getting a reward it first explores the proximal (with respect to its body) part of the workspace, and then explores the distal (with respect to its body) part of the workspace. Such a strategy may be an outcome of the more general strategy: stop in the specific location of the workspace after being rewarded. Similar analysis should be performed for monkey O. 3. There are units in M1 and PMd areas, which change their activity as the monkey gets a reward. The main proposition of this chapter is that neural representation of the tunable movement primitives is inevitably related to neural processes underlying decisionmaking. Monkey U changes its strategy based on getting/not getting reward, which implies existence of decision-making in planning of monkey drawings. We were able to identify geometric templates which are repeatable, clusterable into a small number of clusters and thus may be geometric primitives. Further analysis of neural representation of decision making may reveal neural representation of tuning the parabolic pieces according to the online decision-making (modification of trajectory according to rewarding or other events). Correlates of decision-making processes appear within motor regions such as premotor cortex during tasks in which tactile perceptual decisions are reported by arm movements (Hernandez et al., 2002; Romo et al., 2004). Recent review (Cisek, 2005) proposes that “the neural processes of motor planning appear to be inextricably entwined in the processes of decision-making”; based on results of different studies (see references in (Cisek, 2005)) which contradict the assumption that decisions are made prior to, and by different mechanisms than motor planning. Cisek & Kalaska (2005)


Chapter IV

112

showed that when a monkey is presented with two mutually-exclusive potential directions for a reaching movement, directional signals corresponding to both of these actions are simultaneously represented in PMd. When a cue is presented that identifies the correct action, PMd activity associated with the selected direction increases while activity associated with the other sharply decreases, only 110 ms after the decision cue is given. “ ... movement-related regions of the cerebral cortex can become active, and can represent multiple possible actions, even before a cognitive decision is made about which action to take” (Cisek, 2005). We observed an abrupt change of the neural activity locked on getting reward, which fits the framework of multiple choices followed by a decision after the reward is obtained. Interestingly, out of 5 ’reward-related’ units (figures H.29 - H.36), the activity of one of them (70, 18jul00, figures H.31, H.32) was recorded in the primary motor cortex. We are not aware of any study, which reports about reward-related, or decision-making related activity in the primary motor cortex. Going back to the question of what is represented in the motor cortex, which we discuss in chapter 3, this reward-related finding brings an additional novel variable to be considered.

Chapter 5 Hidden Markov modelling of the neural activity and movement primitives Abstract Formation of movements by means of combinations of a number of elementary building blocks would imply the existence of neural representation of the separate elementary movement primitives. According to the cell assembly hypothesis suggested by Hebb in 1949, the basic unit of information processing in the brain is an assembly of cells, which can act briefly as a closed system, in response to a specific stimulus. This hypothesis proposed that a repeated stimulation of specific receptors will lead slowly to the formation of an assembly of association-area cells, which can act briefly as a closed system after stimulation has ceased. Analogously, repeated performance of stereotypical movements, e.g. similar parabolic pieces, may lead to the formation of an ‘assembly’ of cells, which can act briefly as a closed system. Our work follows a successful attempt (Gat & Tishby, 1993; Abeles et al., 1995; Gat et al., 1997) to characterize this supposed neural population activity using a hidden Markov model (HMM) that enabled to identify the behavioural mode of the monkey and directly identify the corresponding collective network activity. In our study, the process at hand was the simultaneous activity of several cells recorded in the primary motor or dorsal premotor areas of the monkey during scribbling movements. The segmentation of the neural data into the discrete states resulted in (time-lagged) segmentation of the corresponding drawings. In one recording session (out of 8), 28jun00, 1) the segmentation resulted in relatively similar and meaningful (curved and long) movements resembling parabolas, 2) the endpoints of the segments were related to the time of getting a reward. For 3 out of the 8 hidden states, the corresponding movement paths were similar to parabolas. These findings justify our proposition that parabolas may be movement primitives and that decision-making (switch of model state in our case) is represented in the activity of motor cortical cells with the decision of which movement primitive to choose. In other days, the segmentation also resulted in relatively long and curved movements, though in the above-mentioned day 28jun00 the segments were most similar and only they complied with the decision-making hypothesis. The duration of the hidden states was shorter than that reported in (Gat et al., 1997), and for this reason we did not test the state dependent pairwise cross-correlations.

113


Chapter V

114

Introduction Single cell based analysis of the neural activity corresponding to the end-points of parabolic pieces (chapter 3) was our initial attempt to relate the parabolic segmentation of the kinematic data with the neural recordings. However our procedure of fitting parabolic pieces to the drawings allows variability in the end-points of the fitted pieces in terms of tens of milliseconds. In behaving monkeys, the time intervals between spikes, measured in correspondence to a specific behavior, may be controlled to within the milliseconds range (Shmiel et al., 2005). Therefore the question of segmenting the neural activity based on the events defined as the end-points of parabolic pieces is ill-posed. In the study of interactions among groups of neurons in primary motor cortex (MI) that may convey information about motor behavior, during armreaching task, information-theoretic analysis demonstrated that interactions caused by correlated activity carry additional information about movement direction beyond that based on the firing rates of independently acting neurons (Maynard et al., 1999). These results showed that cortical representations incorporating higher order features of population activity would be richer than codes based solely on firing rate, if such information can be exploited by the nervous system. We analyze the activity of groups of simultaneously recorded neurons. Neural population carries reacher information about the movement than single units. Therefore we used hidden Markov modelling (HMM), in order to 1) account simultaneously for the activity of several neurons and their interactions and go beyond single cell data 2) segment the neural activity without any prior knowledge of the kinematic data and to see further how the neural and kinematic segmentations correspond to each other. The idea of applying HMM to cortical data was first introduced by Gat & Tishby (1993), where basic idea of using multivariate Poisson HMM for multielectrode recordings was shown. Further works (Abeles et al., 1995; Gat et al., 1997) presented a detailed exposition of the technique, and showed the relationship between the segmentation of the data into states of collective activity and the pairwise correlations between the cells. State-dependent information processing in the cortex is related to the cell-assembly hypothesis first introduced by Hebb in the late forties and has since gained support in later works. Following (Gat et al., 1997), “according to this hypothesis, it is proposed first that a repeated stimulation of specific receptors will lead slowly to the formation of an assembly of association-area cells, which can act briefly as a closed system after stimulation has ceased; this prolongs the time during which the structural changes of learning can occur and constitutes the simplest instance of a representative process (image or idea) ... The cell assembly is further characterized by stronger than average connections between the cells which comprise it ... The cell assembly is a group of interconnected cells, whose participation in the cell assembly is not determined by


Chapter V

115

proximity. One cell may participate in several cell assemblies, and it is possible for cells which are next to each other not to be part of the same cell assembly.” Analogously, repeated performance of stereotypical movements, e.g. similar parabolic pieces, may lead to the formation of an ‘assembly’ of cells, which can act briefly as a closed system. In (Gat et al., 1997) HMM was applied to to the simultaneous activity of several cells recorded from the frontal cortex of behaving monkeys. The authors were able to identify the behavioral mode of the animal and directly identify the corresponding collective network activity. They also showed that the segmentation of the data into discrete states provides direct evidence for the state dependence of the short-time correlation functions between the same pair of cells. We apply the HMM based (unsupervised) segmentation to multicellular motor cortical data. We show that the derived segments of the neural activity justify our ideas about movement primitives, proposed in previous chapters. We also propose a computationally fast estimation of similarity among a set of geometric shapes, which we apply in order to find the optimal time-lag between the neural and movement data. In Methods, a brief background for HMM and MDL is presented, the learning algorithm for HMM with vector of independent mixtures of Poissonian observations is derived, and the procedure for fast estimation of similarity among geometric shapes is proposed. In the Results section we establish the relationship between the HMMbased temporal segments and the time-lagged movements and estimate the correlation between the single-cell activities and the model.

5.1 5.1.1

Methods Hidden Markov modelling

In the brief description of HMM presented below, we use (Gat et al., 1997) and illustration from that source. The classical introduction text about HMM is (Rabiner, 1989). Other relevant references can be found in (Gat et al., 1997). HMM is a stochastic modelling technique for the study of complex time series. It is essentially a stochastic function of a Markov chain, i.e. a conditioned probability distribution on the possible output observations is attached to each state of the Markov process. The general HMM is constructed of the following elements: • The states of the Markov process change according to specific transition probabilities. • The states cannot be observed directly. Only the stochastic output of the state at each point in time is observable. The resulting model is a double embedded stochastic process with an underlying stochastic process that is not observable.


Chapter V

116

Figure 5.1: Illustration from (Gat et al., 1997). A hidden Markov model with three states. The Si are the different states of the model, the a(i, j) are the transition probabilities from state Si to state Sj , and the bj are the observation probabilities in each state Si .

• The available data are the stochastic output of the Markov process. For convenient reasons, this output is usually transformed into a feature space, in which the essential information found in the data is preserved, while its size is reduced. The transformed data are referred to as the the observations of the model. One type of such a process is the first-order HMM, in which the transition between states is governed by the current state and the next state only. An example of such a process can be seen in figure 5.1. The double-embedded feature of that modelling is shown in that figure in the following manner: the states are shown to be hidden by the dash-dotted horizontal line presented in the upper part of the figure, where the observation of each state is shown as a dotted arrow emerging from the state and crossing that horizontal line. An HMM is characterized, therefore, with the following elements: • N , the number of states in the model. • M , the number of distinct observation symbols, i.e. the discrete alphabet size. In the case of continuous observation, M denotes the number of parameters of the distribution function. • A, the state transition probability distribution matrix.


Chapter V

117

• B, the observation symbol probability distribution matrix. In the case of continuous observations, this matrix is replaced by the distribution functions. • π, the initial state distribution vector. Given appropriate values of N, M, A, B and π, the HMM can be used as a generator to give an observation sequence O = o1 , o2 , o3, . . . , oT , and an unobservable sequence of states Q = q1 , q2 , q3 , . . . , qT , where each observation ot is one of the symbols, and T is the number of observations in the sequence. Three fundamental problems The use of the HMM requires the solution to the following computational problems. I. Given the observation sequence O and a model Λ = (A, B, π), how do we efficiently compute P (O|Λ), i.e. the probability of the observation sequence, given the model? This problem is referred to as the evaluation problem. II. Given the observation sequence O and a model Λ = (A, B, π), how do we choose a corresponding state sequence Q which is optimal in some meaningful sense (i.e. best explains the observations)? This problem is referred to as the decoding problem. III. How do we adjust the model parameters Λ = (A, B, π) to maximize P (O|Λ) (likelihood function)? This problem is referred to as the learning problem, or the estimation problem. Efficient solution of these problems is critical for the practical implementation of the model. HMMs can be used for classification. When several alternative models exist and an observation sequence is presented, the evaluation problem is used to determine the most probable model for the data. After finding the best model for the data, the solution to the decoding problem helps to reveal the underlying structure of observations, i.e. the hidden states as a function of time. The learning problem can be solved by training the models on given observation sequences.

5.1.2

Data preprocessing for HMM analysis of neural data, description of different sets of parameters of the model and initialization of the learning procedure

Here we show the derivation of the learning algorithm and data preprocessing implemented prior to learning of the model. The number of hidden states in HMM is a parameter provided as input to the learning algorithm. The results of segmentation


Chapter V

118

depend on the predefined number of hidden states. Therefore, HMMs with different numbers of hidden states (2-13) were learned and further the most appropriate model was chosen out of them based on the minimal description length (MDL) like criterion (Bishop, 1995). The MDL was calculated for the learning data set and log likelihood for the training data set. In earlier work (Shinoda & Watanabe, 2000), the MDL criterion with a set of assumptions was used in order to decide about the optimal number of hidden states for HMM modelling of phonetic data. We want to construct an HMM for simultaneous recordings of activity of several neurons. The observations for each state are modelled as a mixture of Poisson distributions. Several different models are learned on the same neural data, to choose the most appropriate type of model (e.g. an optimal time window, optimal number of hidden states, optimal number of mixture components for Poissonian mixture). Initialization of learning is described further in the text. The data from every recording session to which HMM was applied were divided into two parts, with odd or even indices of the data files to be used, the order of these indices follows the temporal order of the data record. In every such pair of subsets, each subset was a test set for the model learned on the other data subset. Thus two test subsets and two training subsets correspond to each data set. Now a few words about representing the neural data in terms of the mixture of Poissonian processes. “The Poisson process provides an extremely useful approximation of stochastic neuronal firing” (Dayan & Abbott, 2002). Some authors claim that in spike trains from three areas of the monkey brain (the lateral geniculate nucleus, primary visual cortex (V1), and primary motor cortex) spike times appear to have been thrown down at random, with probabilities determined by the firing rate profile over time, see references in (Wiener & Richmond, 2003). The stochastic description of the temporal spike patterns based on order statistics was used (Wiener & Richmond, 2003) in order to decode the spike trains recorded in the primary visual cortex of monkey. The calculations based on order statistics were substituted by modelling based on the mixture of Poissonian distributions; in this special case, the computations were substantially faster. It was demonstrated that data from neurons in primary visual cortex are well fit by a mixture of Poisson processes. No distribution of spike counts (each distribution corresponds to a single trial) required a mixture of more than five distributions, and 50.8% of the spike count distributions were adequately fit with a single Poisson distribution, 39.4% with a mixture of two Poisson distributions, 7.4% with three, 2.0% with four, and 0.4% with five. Each hidden state in our modelling is characterized by a single Poissonian distribution for each neuron in population, or by a mixture of 3 Poissonian distributions. Below in the text we show the derivation of the iterative procedure for a mixture of arbitrary number of Poissonians. Comparison of the results based on both types of models (1 mixture component or 3 mixture components) led us to the conclusion that


Chapter V

119

the approximation with a single Poissonian distribution is fair. The preprocessing is as follows. The time interval in which all the analyzed units were active simultaneously, was divided into non-overlapping time-bins. For each neuron in the population, spikes are counted in a time window. Mixture of Poissonians is considered for each neuron, and the probabilities for each neuron are multiplied, which results in a probability of observing the population activity in the time-window. Models learned for a set of neural data For every set of neural data we consider 1) different time-windows, from 130 to 30 msec. in which spike counts are observed; 2) different number of hidden states, from 2 to 13; 3) the case of a single Poissonian observation and the case of a mixture of 3 Poissonians are considered. Initialization of learning The output of the K-means algorithm is used on order to initialize the average spike counts and transition probabilities. After the model is learned for the longest timewindow (130 msec.) and for the smallest number of hidden states (2), the algorithm proceeds to shorter time-window and to more hidden states, using the model learned earlier. Such initialization speeds up the process of learning several HMMs on the same data set when the models with finer and finer time resolution are considered and the number of hidden states increases.

5.1.3

EM algorithm for HMM with mixture of Poissonians observations

We want to learn an HMM for simultaneous recordings of activity of several neurons. The observations for each state are modelled as a mixture of Poisson distributions. The update procedure for this model is derived here. The auxiliary function, to which the expectation-maximization (EM) algorithm is applied, is of the form: Q(θ, θ0 ) = E(log f (x1, T , z1, T , y1, T ; θ)| y1, T ; θ0 )) ,

(5.1.1)

where θ0 is the current estimate of the model parameters, y1, T are T observations, and x1, T are the T hidden states. In the case of the observations for a hidden state being produced by a mixture model we have additional latent variables - the mixture


Chapter V

120

components z. Denoting the indicator function by I, we have: f˜(zt , yt | xt ; θ) =

m X

I{zt =s} (cxt s P(yt , µi s )) .

s=1

Let us now write the auxiliary function in more details, using the following notation: x - hidden states, y - observations, z - mixture components. log f (x1, T , z1, T , y1, T ; θ) =

k X

h i I{x1 =i} log π(i) + log f˜(z1 , y1 | x1 ; θ) +

i=1 T X

k X

I{xt−1 =i, xt =j} log p(i|j; θ) +

t=2 i, j=1 k X

T X k X

I{xt =i} log f˜(yt , zt |xt ; θ) =

t=2 i=1

I{x1 =i} log π(i) +

T X

k X

I{xt−1 =i, xt =j} log p(i|j; θ) +

t=2 i, j=1

i=1 T X k X m X

I{xt =i, zt =s} [log ci s + log P(yt , µi s )] .

t=1 i=1 s=1

When evaluating the expectation (5.1.1), only the indicators are random variables, they are conditioned on observations y with parameters θ0 . The expectation step will be to estimate E(I{xt =i} | y1, T ; θ0 ) = p(xt = i| y1, T ; θ0 ) := wti , E(I{xt−1 =i, xt =j} | y1, T ; θ0 ) = p(xt−1 = i, xt = j| y1, T ; θ0 ) := wtij , E(I{xt =i, zt =s} | y1, T ; θ0 ) = p(xt = i, zt = s| y1, T ; θ0 ) := utis . We will have to compute efficiently the posterior probability of a hidden state xt being i: wti , the alpha-beta algorithm can be used for this purpose. In our case this algorithm will be identical to the one for a usual HMM, being endowed with the following equality: X X X p(yt , zt = s| xt ) = cxt s P(yt , µxt s ) = cxt s µxt s yt exp(−µxt s )/yt ! . p(yt | xt ) = s

s

s

Note that the observation yt is the number of spikes in a time interval. 1. Expectation step.

αi (t)βi (t) . wti = Pk α (t)β (t) j j j=1 wtij =

wi (t)aij p(yt |xt )βj (t + 1) . βj (t)

(5.1.2)


Chapter V

121

Finally, we calculate the probability that the s-th component of the i-th mixture generated observation yt as follows: utis = p(zt = s, xt = i| y1, T ; θ0 ) = p(zt = s| xt = i, yt ) p(xt = i, yt | y1, T ; θ0 ) = p(zt = s, yt | xt = i) cis P(yt , µis ) p(xt = i| y1, T ; θ0 ) = wti , p(yt | xt = i) p(yt | xt = i) where p(yt | xt ) is calculated as in (5.1.2). 2. Maximization step. The function Q(θ, θ0 ) is of the form: k X

Q(θ, θ0 ) =

w1i log πi +

T X k X

wtij log p(i| j; θ) +

t−2 i, j=1

i=1 T X k X m X

utis (log cis + log P(yt , µis )) .

t=1 i=1 s=1

The parameters πi , p(i|j), cis , µis are to be estimated as those maximizing Q, which provides maximization w.r.t. θ. At this step, the log likelihood log(y; θ) is non-decreasing. P Noting that πi = 1, and using Lagrange multipliers, we get: " Ã !# X X ∇ w1i log πi + λ πi − 1 = 0. i

i

P This implies the system w /π + λ = 0, thus π = −w /λ, − 1i i i 1i i w1i /λ = 1, P λ = − i w1i , and w1i πi = P . i w1i P Again, noting that s cis = 1, and using Lagrange multipliers, we get: " Ã !# X X ∇ utis log cis + λi cis − 1 = 0. t

s

P

The system Pis t utis /cis +λi = 0, further cis = − 1, λi = − t, s utis , and finally PT

P

utis . cis = PT t=1 Pm u tis s=1 t=1

t utis /λi , −

P P s( t utis )/λi =


Chapter V

122

The values p(i|j) are estimated in exactly the same way as for the usual HMM case: PT −1 wtij pˆ(i|j) = Pt=1 . T −1 w ti t=1 Considering the Poissonian distribution, the probability of observing yt spikes, when the average number in the time window is µis , equals: P(yt , µis ) = exp(−µis ) µis yt /yt ! . ∂ log P(yt , µis )/∂µis = ∂(−µis +yt log µis −log(yt !))/∂µis = −1P + yt /µis . Finding the stationary P point of Q w.r.t. µis , is as follows: ∂ [ tis utisP log P(yt , P µis )] /∂µis = t utis (−1+yt /µis ) = 0, therefore, at stationary point t utis = t utis yt /µis , and PT µis = Pt=1 T

utis yt

t=1

5.1.4

utis

.

Segmentation of the neural data given the model

Assuming that HMM-based labelling of the neural data may reflect encoding of movement primitives, we 1) segment the neural data based on the temporal sequence of hidden states of HMM and 2) investigate the properties of the corresponding temporal segments of behavior, e.g. similarity among the movement trajectories related to the same hidden state. The correspondence between the neural activity and the drawings is inherently related to the problem of estimating the time-lag between these two dynamic processes.

5.1.5

MDL-like estimation of the optimal number of hidden states for HMMs with mixture of Poissonian observations

We use the minimal description length (MDL) like criterion in order to select the number of states in the model. Following (Welling, 2002), we outline briefly the idea of MDL. Let us need to send a message consisting of the samples xN over a communication channel to an imaginary receiver. If both sender and receiver know a probability distribution p(xN ), then the minimum cost of sending the data is − log[p(xN )] bits. This bound can be approximated arbitrarily closely by some coding scheme. Therefore let us assume that this is the actual cost of sending the data. A different approach is to build a model M, which captures the dependencies (structure) in the data, then send: 1) the specifics of the model; 2) The activities of the model when applied to the data; 3) The reconstruction errors which are the differences of the predicted values of the model and the true values. Based on this information, the receiver can losslessly


Chapter V

123

reconstruct the data, but sending fewer bits. The principle behind MDL is the belief that the model for which the sum of those three costs is minimized also has the best generalization performance. Consider, for example, some data which are almost on a straight line in two dimensions. We may send the values {x1 , y1 , . . . , xN ; y1 , . . . , yN } directly or we may recognize that it is cheaper to send 3 parameters determining the straight line (cost 1), then project the data point onto the line and send distances along the line, for instance from the x-intersection (cost 2), and send the errors (cost 3). Instead of sending 2 potentially large values for every data point, we now send one large value (distance along line) and a small one (error) per data point, which may be coded with fewer bits, plus 3 values for the line specifics which are independent of the number of the data points. An error term was derived, which should be added to the maximum likelihood criterium: 1 MDL = ML − (# of parameters) log N , (5.1.3) 2 where ML means maximum likelihood, and N is the number of data points. Let HMM be learned for k hidden states and 1 mixture component for the set of e examples, each consisting of Tj , j = 1, e elements of dimension d. The MDL approach incorporates the number of parameters in the model. We do not consider the transition probabilities in calculation of the number of model parameters, which differs our approach from the standard MDL estimation. There are d k independent parameters, which are Poissonian mean firing rates. The MDL is calculated then according to the following formula: MDL = −

e X

log2

X

p(x1 , . . . , xTj ; Y1 , . . . , YTj ) −

x1 , ..., xTj

j=1

e

X 1 [Estimated Complexity of the Model] log2 Tj = 2 j=1 −

e X j=1

log2

k X i=1

e

X 1 αj, Tj (i) − dk log2 Tj , 2 j=1

with αj, Tj (i) = p(Y1 , . . . , YTj , xTj ) for the exemplar set j.


5.2

Chapter V

124

Time-lag between the neural activity at hidden states and the corresponding pieces of drawings

The segmentation of the neural data results in a corresponding segmentation of the drawings, provided we found the corresponding time-lag between the neural and kinematic data. In a recent work on reconstruction of the drawings based on the multicellular recordings in the arm area of the primary motor cortex (W.Wu et al., 2003) the time-lag −140 msec. between the neural activity and the movement resulted in the smallest reconstruction error. We depicted exemplar drawing segments, corresponding to the states of one of the learned models for the time-lag -140 msec. (the neural activity precedes the drawing by 140 msec.) in figure 5.6A. For 7 states out of 8, the corresponding (e.g. with time-lag -0.14 msec. or with the optimal time-lag) drawing paths look similar (considering the intrastate comparison). The method of quantifying the similarity is described further in the text. Assume that the neural activity from the same hidden state encodes some movement primitive, for which visually observed similarity of the corresponding movement segments gives support. Our goal is to identify the time-lag between the neural activity and the corresponding movement. Identifying the time-lag one can obtain the pieces of drawing directly corresponding to the state-defined segments of the neural activity, supposedly movement primitives. We look for the time-lag, which is optimal from the point of view of the intrastate similarity among the time-lagged movements. The similarity is estimated in the pairwise manner, based on the time-warping approach (Giese & Poggio, 2000).

5.2.1

Estimate of the similarity between two pieces based on the time-warping

The main purpose of the time-warping is to find temporal correspondence between two movements. Following (Giese & Poggio, 2000), we find the correspondence using the temporal and metric (direction of motion) properties of the trajectories by means of the dynamic programming algorithm. Let the first movement be composed of n1 samples and the second be composed of n2 samples; T1 [i] and T2 [j] be the time needed to reach the sample i on the first trajectory and the sample j on the second trajectory. 1. Rescale the duration of both movements to 1 second: T1 [i] = T1 [i]/T1 [n1 ], T2 [j] = T2 [j]/T2 [n2 ], assume that T1 [1] = T2 [1] = 0; 1 ≤ i ≤ n1 , 1 ≤ j ≤ n2 . 2. Resample the movement with smaller number of samples to the number of samples with the spline interpolation. From now on both movements have the same total duration 1 and the same number of samples n + 1. Let the time


Chapter V

125

interval between the neighboring samples be T = 1/n. Having two movements, with rescaled time-courses T1 and T2 , we establish the correspondence between them by minimizing the discretized version of the cost Z £ ¤ (x1 (t) − x2 (t0 (t)))2 + λ(t − t0 (t))2 dt . (5.2.1) We find the trade-off λ between the temporal and metric differences of the two from (5.2.1). The maximal possible value of the metric cost R trajectories 0 (x1 (t) − x2 (t (t)))2 dt, C1 , can be found by applying the dynamic programming to discretized approximation of the integral, accounting R for item 4 from this list. The maximal possible value of the temporal cost (T1 (t) − T2 (t0 (t)))2 dt, R 0.5 C2 = 2 0 (t − 2t)2 dt = 1/12. We take λ = C1 /C2 . 3. Calculate the entries of the matrix 2

E(i, i0 ) = |x1 [i] − x2 [i0 ]| + λ(i − i0 )2 T 2 . Using dynamic programming, find the path i0min [i] and the path i0max [i] minimizing (time-warping) and maximizing (providing us with auxiliary cost) the cost n+1 X Ec = E(i, i0 [i]) i=1 0

respectively, where i [i] specifies the discrete times i0 for the sequence 2 that are corresponding to the times i in the sequence 1. 4. Specify the following constraints on the boundaries: i0 [1] = 1, i0 [n + 1] = n + 1. The dynamic programming algorithm starts with the index pair i = 1 and i0 = 1. Along the path i is increased by one. The set of permitted path transitions for i0 is restricted by the inequality: i0 [i − 1] ≤ i0 [i] ≤ i0 [i − 1] + 2 . Exemplar correspondence between two trajectories is shown in figure 5.2 (a). An example of the change of the similarity between the two pieces w.r.t. the timelag is shown in figure 5.2 (b). We estimate the similarity between two movements from the database, having indices p and q as W (xp , xq ), based on the ratio of the minimal and maximal values of the costs (5.2.1), both obtained my means of dynamic programming. Every matrix W (p, q) was normalized so that the values of its maximal elements would be equal to 1.


Chapter V

126

(a)

(b)

Figure 5.2: (a) An example of the correspondence between two movements, established by means of the time-warping. (b) An example of the changes in similarity w.r.t. the time-lag.


5.2.2

Chapter V

127

Fast estimate of the similarity

The square matrix W (p, q) is not symmetric, because the cost (5.2.1) is not symmetric. Our way of estimation of the pair-wise costs based on dynamic programming is time-consuming, it runs about 2 minutes per pair. We tested whether a much simpler symmetric similarity estimate, based on the direction of motion only, would provide reasonable results. We used mutual information to compare the two methods — time-warping and direct estimation — and found that the mutual information is very similar. More details follow. Direct estimation of similarity Given two trajectories x1 and x2 , we 1. resample each into 20 samples equidistant in time; 2. calculate direction of motion at the samples for each piece, thus obtain two 20-dimensional vectors θ1 and θ2 ; 3. calculate the cost ¯2 P20 ¯¯ iθ1 (k) − eiθ2 (k) ¯ k=1 e C(x1 , x2 ) = P19 , P19 iθ1 (k+1) − eiθ1 (k) | iθ2 (k+1) − eiθ2 (k) | k=1 |e k=1 |e where the denominator is the normalization factor needed for the cases when pieces have high curvature and small shifts of the samples with coinciding indices from the two movements on very similar paths would result in high cost. Possibility for such discrepancy is eliminated in the time-warping approach, due to the flexibility of the temporal courses of the movements; 4. find median value m among the elements of the matrix C(p, q). Set the values in C(p, q) exceeding m to m and normalize C by m so that the value of its maximal element would be equal to 1. The direct estimate of the similarity can be a substitute for the timewarping estimate We applied hierarchical clustering using ’Statistics’ toolbox of Matlab, release 13, to find the clusters of similar movements. Each cluster of higher hierarchy consists of two clusters from a lower hierarchy and its cost is the average of the costs of the two subordinates. The costs are based on the pair-wise costs W obtained by the timewarping or, on the pair-wise costs C obtained by direct computations. Example of clustering of the pieces is shown in figure 5.3. The pieces from state 6, figure 5.6A,


Chapter V

128

were clustered. The dendrogram for the similarity between the pieces is depicted in figure 5.3(a). The pieces, which were included in clusters with the cost lower than the threshold 0.15, are labelled with the index of the cluster they belong to and are depicted in figure 5.3(b). The singleton pieces which do not belong to the clusters with the cost below the threshold are considered as not belonging to any cluster. Given the threshold, one can get clusters of pieces, as described above. We have introduced two different estimates of the pair-wise similarity: 1) based on the timewarping and 2) based on the direct computations. The second one is very attractive for its computational ease, comparing with the first, which is very time-consuming. We used mutual information to compare the results of the similarity estimates obtained by both methods: 1) hierarchical clustering is applied to both matrices W [τ ] and C[τ ] for different time-lags τ ; 2) for every time-lag we construct the joint probability distribution for the pieces, based on the clusters for W [τ ] and on the clusters for C[τ ]; 3) compare the entropy for marginal distributions and the mutual information for the joint probability distributions as a function of the time-lag. We compared the mutual information between the pair-wise costs W (1, 2) and W (2, 1) (upper plot in figure 5.4) with the mutual information between W (1, 2) and C(1, 2) (middle plot in figure 5.4), and the mutual information between W (2, 1) and C(1, 2) (below plot in figure 5.4). We did so for state 4 from figure 5.6A, where the state-wise plots with the paths time-lagged by -0.14 sec. (neural activity precedes the movement) are depicted. As can be seen in figure 5.4 the mutual information between the clusters based on the upper and lower echelon forms of the non-symmetric matrix with the time-warping costs W is similar to the mutual information between each of the two sets of costs and the cost obtained by direct computation. Our conclusion is that the direct method gives fair estimation, compared with the time-warping.

5.3

Results

Here we show the properties of the HMMs learned on the actual and simulated data. The models are used to segment the neural data in time according to the state with the dominant a-posteriori probability. Corresponding movement segments, in some cases, resemble clusters of parabolic pieces.

5.3.1

Properties of the learned models

Number of hidden states and duration of the time bin We remind that the HMM modelling was implemented for the data collected in 8 recording sessions. Every data set was separated into two subsets, each being the training set for the other. Several models that differed in their number of hidden


Chapter V

129

(a) 28jun00. Dendrogram, timelag= −0.14, cost type Angles, states:6; thresh= 0.15 1 0.9 0.8 0.7

Cost

0.6 0.5 0.4 0.3 0.2 0.1 0

111525 3 14 4 6 9 2021272938232431351928343649394246444810133312454050471617

# of the trajectory piece

(b) 28jun00. Paths, timelag= −0.14, cost type: Angles, states: 6, thresh= 0.15 80 60 40 20 0 −20 −40

100 80 60 40 20 −50

0

50

50 0 −50 −50

0

50

−50

0

50

80

60

50

60

40

40 0

20

20

0 0 −20

0

20

40

60

−20 −40 −20

0

20 40 60 80

20

−50 −50

0

50

100

80 60

0

50

−20

0

20

−50

−20

40

0

−40 20

40

60

80

100

−50

0

50

100

−50

0

50

Figure 5.3: An example of hierarchical clustering with the ‘Statistics’ toolbox of Matlab, release 13. Here the clustering was applied to the matrix of pair-wise costs for the segments from the state 6, figure 5.6A. (a) Dendrogram of the pair-wise costs. It enables to visualize the clustering. Horizontal links show how the lower-level elements in the hierarchy are joined. All the pieces that are joined together below the threshold 0.15, indicated by a dashed line, belong to the same cluster. The singletons are not considered as clusters. Hierarchical clustering selected 9 clusters. (b) Pieces that belong to the same cluster are plotted in the same subplot. One can see that indeed the pieces from the same cluster are more similar geometrically to each other than to the pieces from other clusters.


Chapter V

130

State 2 Entropies and Mutual Information for State 2 and threshold 0.15 along timelags X = Timewarp Upper Triangle, Y = Timewarp Lower Triangle 4.5 H(X) H(Y) 4 I(X,Y) 3.5

[bits]

3 2.5 2 1.5 1 0.5 0 −0.2 −0.15 −0.1 −0.05

Upper-lower

0 0.05 Time−lag

0.1

0.15

0.2

Entropies and Mutual Information for State 2 and threshold 0.15 along timelags X = Timewarp Upper Triangle, Y = Direct 4.5 4 3.5

[bits]

3 2.5 2 1.5

H(X) H(Y) I(X,Y)

1 0.5 0 −0.2 −0.15 −0.1 −0.05

Upper-direct

0 0.05 Time−lag

0.1

0.15

0.2

Entropies and Mutual Information for State 2 and threshold 0.15 along timelags X = Timewarp Lower Triangle, Y = Direct

4.5 4 3.5

H(X) H(Y) I(X,Y)

[bits]

3 2.5 2 1.5 1 0.5

Lower-direct

0 −0.2

−0.1

0

0.1

0.2

Time−lag

Figure 5.4: We compare the similarity estimates based on the time-warping and on the direct estimation by means of the mutual information using the hierarchical clustering. The matrix of the pair-wise costs for the time-warping is not symmetric. Therefore we compare both the upper and lower echelon forms of the matrix with the pair-wise costs from direct estimation. The comparison is based on the mutual information between the clusters. Here the results are depicted for state 4 from figure 5.6A. Upper row: comparison between the two non-symmetric parts of the time-warping based costs. Middle row: comparison between the upper echelon form of the time-warping matrix and the matrix of direct estimation. Lower row: comparison between the lower echelon form of the time-warping matrix and the direct estimation. One can see that the mutual information between the upper and lower parts of the time-warping is comparable with their mutual information with the directly estimated costs. Our conclusion is that the direct method gives fair estimation of similarity, comparable with the time-warping.


Chapter V

131

states were learned on every training set. We observed meaningful (curved and not too short) movement strokes corresponding to the segments of neural data obtained based on the models learned on as short time-bins as 30 msec. The decision about how many states should the model contain was based on the MDL criterion. The MDL criterion and not log likelihood (LL) criterion was applied because sometimes the graph of LL contained rather a plateau and not a clear maximum on the test set, still there was a minimum for MDL on the training set. Whenever MDL graph achieved minimum, the corresponding number of states in the LL graph corresponded to the maximum or to the plateau. Log likelihood for the training set is a monotonously increasing function of the number of hidden states. Exemplar graphs for log likelihood and MDL, calculated for the training set (A) and for the test set (B), are depicted in figure 5.5. The graphs with the values of the log likelihood of the data versus number of states in the model are depicted in the upper row. The values of the description length are depicted in the middle row. The MDL reaches minimum at 8 states on the training set. The log likelihood on the test set reaches maximum at 11 states, but there is a plateau in the log likelihood plot starting from 6. Though on the test set the MDL criterion points on 6 and not on 8 states, the MDL on the training set reaches local minimum which is close to the global at 6 states too. We estimate the correlation between the activity of the neuron j and state i of the model as follows. " # ¤ ª PN ©£ fj − f j Isi − p(si ) fj (t) − f j [p(si (t)) − pi ] t=1 cij = E · =r h , ip PN SD(fj ) SD(Isi ) 2 N pi (1 − pi ) t=1 (fj (t) − f j ) where fj (t) is the firing rate of neuron j at time t, and I is the indicator function. The squared correlation coefficient between the activity of the neuron j and the model, c2j , is computed then as follows: c2j =

X

pi c2ij .

(5.3.1)

i

The exemplar (for one day) correlations between the individual units and the models for the training and test data sets versus the number of the states in the model are depicted in figure 5.5, lower row. Two units are correlated with the model stronger than the other ones. The highest correlation corresponds to the unit 70, 28jun00. In particular, this units is responsive to getting reward (see figure H.29). The graphs for the log likelihoods, MDL and correlations for all 8 days are depicted in figures H.49, H.50.


Chapter V

132

A 4

−2.7

x 10

B

Day: 03, 28jun00; set: 1; time bin=0.03[sec.]. Actual data. Training set.

4

−2.41 −2.42

Log lik.

−2.72

Log lik.

x 10

Day: 03, 28jun00; set: 1; time bin=0.03[sec.]. Actual data. Test set.

−2.74 −2.76

−2.43 −2.44 −2.45

−2.78 x 104 4.01

−2.46 x 104 3.55

4 3.54

MDL

MDL

3.99 3.98

3.53

3.97 3.52 3.96

0.4

0.4

0.3

1 2 3 4 5 6

0.2 0.1 0

Correlation

3.51 0.5

Correlation

3.95 0.5

2

4

6

8

10

12

14

0.3

1 2 3 4 5 6

0.2 0.1 0

2

4

6

# of hidden states.

8

10

12

# of hidden states.

Figure 5.5: Graphs of log likelihood of the model (first row) and the description length of the model (second row), versus number of hidden states. A. Estimates for the training set. B. Estimates for the test set. The MDL reaches minimum at 8 states on the training set. The log likelihood on the test set reaches maximum at 11 states, but there is a plateau in the log likelihood plot starting from 6. Though on the test set the MDL criterion points on 6 and not on 8 states, the MDL on the training set reaches local minimum which is close to the global at 6 states too. In the lower row, the correlations (5.3.1) between the units and the models are depicted. The values of the log likelihood are larger for the test set here because the amount of data in the test set was smaller than the amount of data in the training set by about 10%.

HMM states correspond to similar geometric patterns In figures H.47, H.48 we depicted the paths corresponding to the hidden states, timelagged from the neural activity by the fixed time-lags -140 msec. (neural activity precedes the movement). We used the histograms with the time-lag dependent numbers of similar pairs of paths (figure H.51) to decide about the optimal time-lag for a given state in a day. The state-wise plots with the paths of movement segments lagged from the neural data by the optimal time-lags are depicted for each recording session (for the model with the optimal number of the hidden states) in figures H.53 - H.60. Each separate collection of plots corresponds a recording session. As can be seen, the paths of the segments corresponding to the same hidden state usually possess geometric similarity in terms of direction of motion, still the path pieces are not straight lines and they were obtained under a strict segmentation condition, see Methods. An important conclusion from the segmentation of movements based on HMM is

14


Chapter V

133

that one can get geometrically similar pieces of trajectories. Therefore, the decision of how many states to choose does not have to result in precise number. Indeed, some clusters of paths remained similar even when they corresponded to the states from different models. Transitions between the states may be related to decision making, based on times of getting a reward For one day of recordings (28jun00) out of 8 analyzed, we saw evidence for the existence of relationship between the hidden states and rewards; these reward-related states were not defined by the reward only, they also have specific geometric pattern. Such relationships were not observed for the rest 7 recording sessions. In figure 5.6, the model learned on the data from day 28jun00, with eight states, is visualized in different ways. The state-wise paths corresponding to the segments of the neural data are plotted in A. The graph of transition between the states is depicted in B. The wider the arrow, the higher is the transition probability. Note that topologically the graph is similar to a closed loop, with some noise. Note also that although the paths for states 5 and 6 look similar, state 4 is always followed by the state 5 and never by state 6 or any other state. The learned average firing rates for the states from the model are depicted in C. In D and E, the histograms of the occurrence of the first reward before and after the start or end of the segment correspondingly are depicted. Based on the histograms, we conclude that the states 2-8 (all but 1) are in some way related to reward. State 2: the monkey is rewarded before start of the segment and is not rewarded during the segment. States 3 and 4: the monkey is not rewarded during, prior, or after the segment. State 5: no reward was obtained before the segment and only a few rewards were obtained during the segment. State 6: rewards were obtained near the end of the segment or immediately after the end. State 7: similar to state 6, but geometric properties of this state differ from the geometric properties of the state 6. State 8: the monkey is rewarded prior to the segment. The paths of the segments from state 8 are geometrically dissimilar, opposite to other states. Interestingly, many segments from set 11 (very short segments, their duration is not longer than 100 msec.), start briefly after the monkey gets a reward. This might mean simply abortion of the preplanned movement during a short time period, or a short duration of the process of decision making. State 9 corresponds to the segments with the probability of the dominant state being less than 1/2, and state 10 corresponds to the segments with the average of the dominant probability being less than 0.75. Sequential and parallel encoding of movements In order to visualize the results of segmentation with better ’resolution’, we plotted separate segments with the corresponding neural activity, speed, and time-varying


Chapter V

134

A State 2

lag: −0.14 [s.]

State 1

State 4

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

0

−100 100 −100

State 6

State 5 lag: −0.14 [s.]

State 3

100

−100 −100

−100 100 −100

0 State 7

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

State 9

0

−100 100 −100

State 10 100

100

50

50

50

0

0

0

−50

−50

−50

0

−100 100 −100

0

100

−100 100 −100

0

0

100

State 11

100

−100 −100

0 State 8

100

−100 −100

lag: −0.14 [s.]

B

−100 100 −100

0

100

C

D

E 28jun00, set# 1; norm−d=0; 8 states; loop# 4T =30 [msec.]; Poiss: 0 # mixtures: 1; learned set 1 or 2: 1

28jun00, set# 1; norm−d=0; 8 states; loop# 4T =30 [msec.]; Poiss: 0 # mixtures: 1; learned set 1 or 2: 1

State 1

State 2

2

8

1.5

6

1

4

0.5

2

State 3

State 1

State 4

3

State 3

State 2

2

3

State 4

3

3

2

2

6

0 −1

0 State 5

1

2

0 −1

2

2

1

1

1.5 4

1

0 State 6

1

10

1.5

0 −1

0 State 7

1

0 −1

4

5

3

4

0 State 8

1

0 −1

1

2

0.5 0 State 5

1

0 −1

0 State 6

1

1 10

0 −1

1

0 State 7

1

4

5

3

4

5

0.5

2

0 −1

0 State 9

1

0 −1

0 State 10

1

1

0 −1

0 −1

0 State 11

1

1

0

1

2

2 1

0 State 8

3

3 1 0.5

0 −1

2

5 1 0

1

0 −1

0 State 9

1

0 −1

0 State 10

1

0 −1

1 0 State 11

1

0

1

0 −1

6 6

10

20

4

10

2

10

20

5

10

4 5 2 0 −1

0

1

0 −1

0

1

0 −1

0

1

0 −1

0

1

0 −1

0

1

0 −1

Figure 5.6: A. Paths corresponding to the segments for the hidden states (1-8), with time-lag -140 msec. We introduce 3 artificial sets of segments: set 9 consists of the segments with a-posterior probability of a state (states 1-8) below 0.5; set 10 consists of a segments with average a-posterior probability of a state (states 1-8) less than 0.75; set 11 consists of a very short segments (shorter than 100 msec.). The procedure of segmentation of the neural data is described in Methods. B. Graph of transitions between the states. The width of the links corresponds to the probability of transition. C. Average firing rates across states as learned by the model. D. Histograms of the time of occurrence of the first reward before and after start of a segment. E. Histograms of the time of occurrence of the first reward before and after end of a segment. State 11 labels low maximal a-posterior probabilities that cannot be classified to the states 1-8, for which the model was learned. The peak near 0 in the histograms with numbers of rewards corresponding to state 11, D, E, indicates that soon after getting reward the neural activity was altered in many cases. Such a peak was not observed in the histograms for other days, in which the reward-related activity was not revealed.


Chapter V

135

probabilities of the corresponding dominant state. Exemplar plots for state 4 from figure 5.6 are depicted in figure 5.7. For more details one can see eight separate figures H.39 - H.46 that correspond to the 8 respective states from figure 5.6. In the first column we depicted the firing rate in the bins of 30 msec., normalized to one by the maximal activity among the entire data set. The end-points of the segments are designated by vertical white lines. In the second column a long piece of the drawing is depicted. It contains the stroke (bold green) time-lagged from the segment of the neural lagged by the ‘optimal’ time-lag. The following values are the state-wise optimal time-lags (state number / lag): 1 / -0.21; 2 / 0.28; 3 / 0; 4 / -0.14; 5 / 0.07; 6 / 0; 7 / -0.21; 8 / 0.07. We describe what we mean by ‘optimal’ time-lag in the text below. The speeds of drawing are plotted in the third column. Again, the speed profiles that correspond to the segments of the neural data with the optimal time-lag are depicted by bold green. Vertical red lines signify the time of getting a reward. The fourth column contains the a-posterior probabilities of hidden states for which the colorbar is located in the last plot of the first column. The last column contains graphs of the a-posterior probabilities of the hidden states. The values are the same as in the fourth column of part A, but the visualization is different. The legend for colors of the graphs is on the right side of the lowest plot. Numbers above each plot correspond to the dominant hidden state. Vertical black lines indicate the end-points of the segments, thus the posterior probabilities between the vertical black lines are always dominant. Vertical dashed red lines mark the time when the monkey got a reward. The graphs of posterior probabilities for some of the states possess specific features. State 1: the graph is jerky (non-smooth); 2: the graph has tendency to be more stretched to the right; 3: the visible part of the graph (noticeably non-zero) is usually wide; 4: more stretched to the left; 5: is sharper from the right. In these examples one can also see the relation between the time of getting reward and start/end of the segment. For state 2, there are 7 out of 10 plots in which reward precedes initiation of the segment, which can be seen in figure H.40 B, rows 1, 2-5, 9, 10. The graphs of posterior probabilities provide evidence for both sequential and parallel processing of the states. Similarity in shape of the paths from the same state may mean that hidden states indeed correspond to encoding of primitive submovements. Thus, the submovements might be represented in the neural activity both in sequential and parallel manner.

5.3.2

Finding optimal time-lag, based on geometric properties of the paths

As we pointed out earlier, the segments of neural data that belong to the same state correspond to similar geometric shapes of the drawings. We therefore attempted

6

6

6

6

6

probabilities for state 4 from figure 5.6. See more explanations in text on page 133.

2

4

6

2

4

2

4

2

4

2

4

2

4

2

4

0

Path, τ=−0.14

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0

0.5

1

0

0 1 0.5

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2

A−poster. prob., for the states

0.5

0

Speed

0.5

1

0

1

3

4

4

7

6

8

3

3

5

8

1

2

3

56

1

2

2

4

8

1

1

4

4

4

8

4

5

2

5

5

5

6

45

6

5

8

6

3

7

3

3

6

3

6

3

7

3

2

7

4

5

4

4

4

4

5

4

5

5

4 5

4

4

4

5

5

6

6

6

5

6 1

6

6

56

5

8

7

8

7

6

7

8

7

3

6

1

6

18

8

7

8

81

2

3

6

4

2

A−posterior probabilities

8

2

4

5

2

8

6

4

3

3

3

5

6

1 2 3 4 5 6 7 8

State: learneddata=1 4;jit.:0;Poiss:0; 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 State: 4; jit.:0; Poiss:0; 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; Timelag: −0.14 file: plot; dataPieces; diffColors.m Timelag: −0.14 file: plot; dataPieces; diffColors.m

20 40 60 80 100

6

2

4

0.5

6

2

4

1

6

2

4

6

Spke. count, norm−d for each neuron

18; 373−429

17; 1388−1441

39; 2014−2061

15; 2872−2919

17; 3455−3499

15; 3595−3636

48; 2496−2534

16; 1431−1469

17; 989−1021

20; 890−904

PHD thesis by Felix Polyakov Chapter V 136

Figure 5.7: Neural activities, paths, speeds, a-posterior probabilities, and graphs of a-posterior


Chapter V

137

to find the time-lag between the neural activity and the movement as the time-lag providing the highest similarity among the paths corresponding to the same state. Optimal time-lag as maximal number of pairs with pair-wise cost below the threshold We used pair-wise similarity to estimate the optimal time-lag between the neural data and the drawings, separately for each state. An example of the pair-wise costs is depicted in figure 5.8A. Corresponding paths are depicted in figure 5.8B. The A

B 28jun00; state #4; τ=−0.28

28jun00; state=4; lag=−0.28;#=66 18 1 0.8

14

1 0.8

12

12 0.6

10

0.4

6 4

0.2

0.6

10

0.4

6 4

0

5

10

15

0

0

100

100

50

50

50

0

0

0

−50

−50

−50

0.8 0.6

10 8

0.2

2

2

14 12

8

8

1 16

14

0.4

6 4

0.2

−100

−100 −50

0

50

−100 −50

0

5

10

15

0

0

0

5

10

15

0.8

8

14

4

0.2

2 0

5

10

15

0

4

state=4; lag=0.14;#=26

0.2

0

5

10

15

0

0.8 0.6 0.4

6 4

0.2

0

5

10

15

0

14

0.8

0.8

50

50

50

0.6

0

0

0

−50

−50

−50

0.4

6 4

0

0.2

−100 0

5

10

15

−50

0

50

−50

0

50

−50

0

50

0

state #4; τ=0.14

1 14

state #4; τ=0.21

state #4; τ=0.28

6 4

0.2

100

100

50

50

50

0.4

0

0

0

0.2

−50

−50

−50

0.6

10 8

0.4

100 0.8

12 0.6

10

0

100

state=4; lag=0.28;#=28

2

2

state #4; τ=0.07

100

16

8

8

10

1

12

12 10

14

state=4; lag=0.21;#=34 16

14

state #4; τ=0

100

1

2

1

16

50

8 0.4

6

0

0

12 0.6

2

18

0

0.8

10 8

0.4

6

−50

16

12 0.6

10

0

state=4; lag=0.07;#=30 1

16

12

50

0

state #4; τ=−0.07 state=4; lag=0;#=44 1 14

0

2

state=4; lag=−0.07;#=42 16

state #4; τ=−0.14

100 state=4; lag=−0.14;#=68

16

16

0

state=4; lag=−0.21;#=68

state #4; τ=−0.21

6 4 2

0

5

10

15

0

0

0

5

10

15

0

−50

0

50

−50

0

50

−50

0

50

Figure 5.8: The pairwise costs based on direct estimation, for the paths depicted in figure 5.6A, state 4. The costs are depicted for different time-lags. The number in the title indicates the number of cells in the depicted matrix with the cost not higher than the threshold = 0.15, excluding the costs 0 on the diagonal. One can see that the biggest number of pairs with the cost below the threshold corresponds to the time-lags -0.14, -0.21 msec. in this example. This means that our criterion of the optimal time-lag is satisfied for the time-lag in the range from -0.21 to -0.14 sec. The path pieces that correspond to these time-lags resemble parabolic pieces with the downward pointing normal at the point of maximal curvature.

number of pairs not exceeding the threshold 0.15 was calculated for each state in every analyzed day, and the histograms with the number of such pairs as a function of the time-lag plotted. One can see then, for which time-lag such a number of pairs is the highest. As can be seen in figure H.51, different states from the same model may correspond to different time-lags. For example, in 27jun00, the optimal time-lag 0.07 sec. corresponds to the state 2, and the optimal time-lag -0.21 sec. corresponds


Chapter V

138

to the state 3. Same type of histograms for different thresholds (0.1,0.2,0.25) showed very similar optimal time-lags. In the example depicted in figure 5.8A the biggest number of pairs with the cost below the threshold corresponds to the time-lags -0.14, -0.21 sec. This means that our criterion of the optimal time-lag is satisfied for the time-lag in the range from -0.21 to -0.14 sec. The path pieces that correspond to these time-lags resemble parabolic pieces with the downward pointing normal at the point of maximal curvature, as can be seen in figure 5.8B. We observed both negative and positive time-lags, movement after and before the neural activity correspondingly. For some states no clear time-lag could be found. Our observations of parallel processing in representation of the hidden states implies that temporal segmentation of the neural data with respect to states cannot provide precise segmentation of the movements, that is the end-points of the segments are badly defined. This can be a reason of impossibility to find precise time-lag between the neural activity for a certain state and the movement. Different time-lags resulted in relatively similar mean squared errors when inferring hand movements from multicell recordings in the motor cortex in (W.Wu et al., 2002). This may be exactly the effect of the parallel processing we show here. The optimal time-lags were stable with respect to changes in the threshold. 10 pieces,90 pairs

28jun00, state=1; thresh=0.15

7

27 pieces,702 pairs 60

6

50

5

14 pieces,182 pairs

14 pieces,182 pairs

14

35

12

30

10

25

8

20

6

15

4

10

2

5

40 4 30 3 20 2 10

1

0

−0.2

0

0.2


0

−0.2

0

0.2

0

−0.2

0

0.2

59 pieces,3422 pairs 23 pieces,506 pairs 500 140

0

−0.2

0

0.2



450 120

25

400

20 100

350

20

300 15

15

80

250 60

200 10

150

10

40

100

5

5 20

50 0

−0.2

0

0.2

0

−0.2

0

0.2

0

−0.2

0

0.2

0

−0.2

0

0.2

Figure 5.9: Numbers of pairs below the threshold = 0.15, for the day 28jun00 for all accounted time-lags: -0.28:0.07:0.28 sec.


5.3.3

Chapter V

139

Estimation of the optimal time-lag based on separability of the paths

The approach of estimation of the time-lag as the one which provides the largest number of similar pairs is state-wise, that is the optimal time-lag can be estimated separately for every state. As we show in Results, the time-lags can indeed differ among different states within the same day. Here we propose another method of estimation of the time-lag, the same for all states. The optimality is now related to separability of the pieces from different states. That is, we ask the following: for which time-lag, the number of paths classified according to the state they belong is the highest? The classification quality is measured by the mutual information estimated on the following two-dimensional joint probability distribution calculated on the movement segments. One of its margins corresponds to the label of the state, the other margin corresponds to the cluster from hierarchical clustering (the clustering is based on the direct estimation, see page 127), as explained previously. Note again that all pieces, from all states, are put together prior to hierarchical clustering, and the time-lag for which the mutual information is maximal is assumed to provide the best separability among into the states. The mutual information for the threshold = 0.15 for all accounted time-lags: -0.28:0.07:0.28 sec. is depicted in figure H.52. For some days the graph of the mutual information is flatter than for other days, e.g. the graph for the day 28jun00 is flatter than the graph for 27jun00. The mutual information was calculated for different thresholds. We observed that the optimal time-lag for the day 27jun00, -0.07 sec., is not sensitive to changes in the threshold. For the day 28jun00, the optimal time-lag is sensitive to the threshold: the current threshold (0.15) resulted in the time-lag 0 sec, and the threshold 0.05 results in the time-lag 0.07 msec. The state-wise estimation of the time-lag, see figure H.51, is consistent with this result in that there are several states with many pairs in the day 28jun00, for which the optimal time-lag cannot be clearly deduced.

5.4

Discussion

We have implemented an unsupervised state-wise segmentation of the population activity recorded in PMd and M1 areas of scribbling monkey by means of hidden Markov modelling. The analysis was performed on the data collected in 8 recording sessions. In one session, the segmentation was related to the time of getting a reward and the movements corresponding to the identified states of neuronal activation formed clusters of similar geometric shapes. Three clusters resemble parabolic shapes. The observations of the model were the firing rate vectors of several cells recorded simultaneously. Our model assumption was that, given the states, the cells are conditionally independent, Poisson processes. The spikes were counted in the non-overlapping time


Chapter V

140

28jun00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6 7 8, threshold=0.15 X = Metric cost clustering, Y = State clustering 7 H(X) H(Y) I(X,Y) 6

5

4

3

2

1

0

−0.25

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

Figure 5.10: Mutual information for the joint probability distribution of the states labels and hierarchical clusters with the threshold = 0.15 for the day 28jun00 for all accounted time-lags: 0.28:0.07:0.28 sec. The optimal time-lag is sensitive to the threshold. The current threshold resulted in the time-lag 0 sec, and the threshold 0.05 results in the optimal time-lag 70 msec. The state-wise estimation of the time-lag, see figure 5.9, is consistent with this result in that there are several states with many pairs in the day 28jun00, for which the optimal time-lag cannot be clearly deduced. Careful reader will find this type of graphs for all days on to which HMM analysis was applied in figure H.52. bins of 30 msec. We have also implemented learning of the model for the simulated Poissonian spike trains, based on the model of cosine directional tuning, as described on page 55. Segmentation results based on the simulated data were mostly junky — the ratio of the time with small values of the dominant a-posteriori probabilities was very high. Movement segments corresponding to the identified states were short. Following (Gat et al., 1997), “The Markovian assumption behind the HMM should not be taken as an assumption about the actual cortical activity, nor should its states be considered as states of mind or the true cortical activity states. These are merely modelling assumptions which enable a direct glimpse into the actual cortical dynamics.” The HMM states carry significance beyond their place in the model. This significance is manifested in the following way: for one recording session, the model was able, within certain limitations, to relate the neuronal activation states to clusters of parabolic pieces. The state-wise segments were also related to the time of getting a reward. A discussion of the results related to the HMM analysis of the data


Chapter V

is presented in the context of the previous chapters in the “General Discussion”.

141

Chapter 6 General discussion and further questions Invention is not a result of logical thought, even though the end result is intimately bound to the rules of logic

(Albert Einstein)

6.1

Summary

In this work we have applied computational approaches to the simultaneous recordings of monkey scribbling movements and multicellular motor cortical activities. One of the main goals of the data analysis was to reveal plausible movement primitives and their neural representation. Another main goal was to test whether non-Euclidian (in our case equiaffine) metrics are relevant to neural representation of the hand movements. Each chapter contains its own, specific discussion. 1. We have successfully derived candidates for a movement primitive, both theoretically and empirically. The results point to pieces of parabolas. In particular, we present a necessary and sufficient mathematical condition for a common geometric template of the constrained minimum-jerk and the two-thirds power law; prove that a parabolic piece is a unique common geometric template that is invariant under arbitrary equiaffine transformations; describe a simple scheme of the generation of complex movements based on equiaffine transformations of drawing a single parabolic piece. Fitting drawings with elementary shapes like parabolas has enabled us to show the emergence of a low-dimensional description of the behavior in terms of a small number of directionally defined parabolic clusters.

142


Chapter VI

143

2. We suggest a principle of greater parsimony: practicing a motor task leads to more parsimonious movement strategies. 3. We have shown that there exist dorsal premotor and primary motor cortical units whose activities are correlated more strongly with equiaffine velocity rather than with Euclidian speed. Therefore some single neurons in the premotor and primary motor cortices may encode movements rather in terms of equiaffine than in terms of Euclidian geometry. 4. Monkey trajectories are influenced by getting/not getting a reward, and there exist units in M1 and PMd areas, that change their activity as the monkey gets a reward. 5. We propose that the studies of movement primitives have to account for the underlying decision-making. 6. By means of HMM analysis we have segmented the motor cortical activities into several states. In many cases, the movements corresponding to the identified states of neural activities formed clusters of similar geometric shapes, most of them being parabolic. Transitions between some of the states were related to the time of getting a reward. In short, based on the analyses of the behavioral and neurophysiological data, we point to parabolas as our candidates for movement primitives, acquired with practice. Emergence of parabolas may be a result of the development of attractor-based encoding of movements. We have also shown for the first time that an equiaffine metric may be used in neural coding of movements.

6.2

Smoothness and geometry

Parabolas appear to be the meeting point of smoothness (the minimum-jerk model) and geometric (the 2/3 power-law) constraints. The constraints, therefore may be related to a more parsimonious way of generation of movements, since (1) parsimony is the grounding reason in adopting the primitives of motion paradigm and (2) representation of movements in terms of the fitted parabolic pieces indeed becomes more parsimonious following practice.

6.2.1

The principle of greater parsimony

We propose an intuitive principle: the development of the motor control strategies (e.g. during learning) seeks to achieve a greater parsimony of the control process.


Chapter VI

144

The principle is consistent with the development of piece-wise parabolicity and clustering of the parabolic pieces. We propose that the outcome of the objective to move with constant equiaffine velocity along parabolic pieces might ease coordination between the visual and motor systems. It can be argued that through the process of practice the equiaffine velocities of monkey drawings became closer to constant for the predicted and not for the actual trajectories (Appendix G). However, this argument does not imply that the deviations of the actual equiaffine velocities from constancy are so large that they contradict our proposition. Moving with constant equiaffine velocity along a parabolic piece is equivalent to minimization of the jerk. The parsimony principle does not imply that the maximization of smoothness is not a goal of the motor system. Both maximization of smoothness and advancement of parsimony per se may be important for motor control. We have shown that they can also be satisfied simultaneously, nevertheless one is not necessarily the origin of the other.

Rhythmic or discrete? Rhythmic movements, such as walking, chewing or scratching, are phylogenetically old motor behaviors found in many organisms, ranging from insects to primates. In contrast, discrete movements, such as reaching, grasping or kicking, are behaviors that have reached sophistication primarily in younger species, particularly primates. Neurophysiological and computational research on arm motor control has focused almost exclusively on discrete movements, essentially assuming similar neural circuitry for rhythmic tasks. In contrast, many behavioral studies have focused on rhythmic models, subsuming discrete movement as a special case. Using a human functional neuroimaging experiment, Schaal et al. (2004) showed that in addition to areas activated in rhythmic movement, discrete movement involves several higher cortical planning areas, even when both movement conditions are confined to the same single wrist joint. These results provided neuroscientific evidence that rhythmic arm movement cannot be a part of a more general discrete movement system and may require separate neurophysiological and theoretical treatment. The scribbling movements that we have analyzed could turn closer to rhythmic behaviors after a long practice. It should be noted that we did not classify the data as discrete or rhythmic, but should be aware about differences in the neural representation of the rhythmic and discrete movements. The control of rhythmic movements may be simpler than the control of the discrete movements. Therefore, the rhythmic movements can be considered as a type of higher level movement primitives, in our case composed of parabolic primitives. Thus, the convergence of discrete movements to rhythmic-like movements satisfies the greater parsimony principle. Future work with similar types of data may need to test the degree of involvement


Chapter VI

145

of the rhythmic and discrete components in the performance. For example, on the kinematic level the durations of similar drawing loops can be analyzed; and on the neurophysiological level activation in the areas corresponding to the rhythmic movements and exclusively to the discrete movements may be tested during a period of practice.

6.2.2

More about merging kinematic smoothness with geometry

An interesting perspective follows from the finding that the performance of the monkey converged to smooth and well clustered piece-wise parabolic scribblings even though the monkey practiced while its hand was hidden by an opaque plate, and the only visual feedback was provided by the red dot indicating the location of the endeffector. Note that in a recent study involving point-to-point drawing-like practice of humans (Sosnik, 2004) “no geometrical motion elements were acquired while training in the dark”. The monkey might have learned to map proprioceptive feedback from its arm to the location of the end-effector in a way that enabled it to develop drawing of smooth parabolic segments. Researches claim involvement of the two-thirds power-law in the perception of motion (Viviani & Stcucchi, 1992; Levit-Binnun et al., 2005). Thus equiaffine geometry may be relevant for both perception and execution of movements. Concerning the visual system, the primary visual cortex (V1) can be viewed as the bundle of what are called 1-jets of curves in R (Petitot, 2003). The 1-st order jet of a function f , is characterized by three slots: the coordinate x, the value of f at x, y = f (x), and the value of its derivative p = f 0 (x). The latter is the slope of the tangent to the graph of f at the point a = (x, f (x) ) of R. “Jets are feature detectors specialized in the detection of tangents. The fact that V1 can be viewed as a jet space explains why V1 is functionally relevant for contour integration. ... The Frobenius integrability condition ... is an idealized mathematical version of the Gestalt principle of good continuation” (Petitot, 2003). Smooth drawings possess nice integrability properties. Edge completion as the interpolation of gaps between edge segments, which are extracted from an image, can be performed by parabolas (Handzel & Flash, 2001). The formation of the smooth, parabolic-like drawing shapes may be explained by completion of the support direction with parabolas into corresponding integral shape. If we assume involvement of the equiaffine geometry in both perception and generation of movements, parabolic shapes become relevant as they mediate between the invariants of the equiaffine geometry and the smoothness constraint. Smoothness, in turn implies nice integrability. There is evidence that the cells in both primary visual and primary motor cortices possess directional tuning (Hubel & Wiesel, 1962; Georgopoulos et al., 1982) and that


Chapter VI

146

directionally tuned cells of both cortexes are organized in columns (DeAngelis et al., 1999; Amirikian & Georgopoulos, 2003). In case the functional architecture of the motor cortex possesses pinwheel structure, the neurogeometry of pinwheels (Petitot, 2003) may have the same relevance for the motor cortex as for the visual cortex, thus giving similar explanations for emergence of smooth parabolic shapes with practice as the explanations of Gestalt principle of good continuation with the Frobenius integrability condition.

6.3

Synthesis

How does the monkey choose an appropriate parabolic piece from a cluster? Consider that the monkey produced a movement through some part of the workspace and was not rewarded. The chance to get a reward will be higher if the animal continues to search in a different area. Search in the proximal part of the workspace is energetically more efficient. Therefore, after getting a reward, the monkey will save energy if it continues to search in the proximal part of the workspace. We have indeed observed differences between the segments of trajectories that followed a reward obtained in a specific area, and the segments that cross the same area without being rewarded. There are also areas in which such differences have not been observed. Kolmogorov considered the movement planning as the mental process situated in between classical conditioning and logical decisions and termed it synthesis (see Introduction). We propose that the decision-making is an essential parameter in the analysis of the tunable movement primitives.

6.3.1

Rewarded versus non-rewarded trajectories

In some cases, after getting a reward the monkey decreased the speed of drawing or almost stopped. The rewarded pieces of trajectories that showed decrease in their speed were stereotypical and resembled parabolic shapes. Here we propose a definition of the movement primitive as the movement entity which cannot be stopped uncompleted. The definition implies existence of the so called ‘point of no return’. The model describing the double-step paradigm (Flash & Henis, 1991) also assumes existence of the ‘point of no return’. A preliminary hypothesis based on stereotypy of the stopping rewarded trajectories is that the point of no return appears in the curved (even parabolic-like !) and not only straight shapes with practice, implying the existence of the primitives with curved shape. Beside the current and earlier work (Polyakov et al., 2001), the development of curved primitives was also reported by Sosnik et al. (2004). Such primitives may be context dependent (Jax & Rosenbaum, 2003; Sosnik et al., 2004). Drori (2005) proposed existence of motor cortical neurons with context-dependent activity.


Chapter VI

147

The question of which movement generation mechanisms cause deviation of the non-rewarded trajectories from the rewarded trajectories remains open. The existence of neurons with reward-related activity in PMd may reflect the existence of decisionmaking mechanism in this relatively low hierarchical level (this needs to be proved or rejected in future studies). We speculate that the decision-making mechanism may apply the compositionality rules for trajectory formation.

6.4

Does converges to piece-wise parabolic performance reflect an emergence of corresponding attractors in neural networks in the brain?

Abeles et al. (1995) initially supported the idea that neural networks in the brain dwell most of the time in stable configurations of activity (“attractors” or “states”), each having distinct firing rates and neuronal interactions. We have shown that monkey movements converge to piece-wise parabolic representations with the fitted parabolic pieces being from 2-4 directionally defined clusters (chapter 2). Do the results of our HMM analysis of the neural data imply that those parabolic clusters are behavioral output of the organized (through the process of practice) activity of dynamically switching cortical “attractors”? Using voltage sensitive dye imaging, a close link was established between ongoing activity in the visual cortex of anaesthetized cats and the spontaneous firing of a single neuron (Tsodyks et al., 1999). Such activity encompasses a set of dynamically switching cortical states, many of which correspond closely to orientation maps (Kenet et al., 2003). When such an orientation state emerged spontaneously, it spanned several hypercolumns and was often followed by a state corresponding to a proximal orientation. Kenet et al. (2003) suggested that dynamically switching cortical states could represent the brain’s internal context, and therefore reflect or influence memory, perception and behavior. We mention above certain functional properties that exist in both cortices M1 and V1 (e.g. directional tuning). We have also segmented multicellular motor cortical activity in an unsupervised way into several states by means of HMM. For 1 out of 8 recording sessions, 3 identified states correspond to the clusters of similar parabolic-like drawings. The transitions between the states were related to getting a reward. Possible sequential-like performance of the monkey may underly the observed clustering of parabolic pieces. We relate the elements composing the sequences which were acquired through practice to movement primitives. Several recent studies report evidence for the encoding of the sequential behavior in the activity of motor cortical neurons. In (Carpenter et al., 1999) “the serial order of stimuli in a motor task” was


Chapter VI

148

identified “as an important determinant of motor cortical activity during stimulus presentation and in the absence of changes in peripheral motor events, in contrast to the commonly held view of the motor cortex as just an upper motor neuron” that was based on the observation that “the effect on cell activity of the serial order of stimuli during their presentation was at least as strong as the effect of motor direction on cell activity during the execution of the motor response”. As reported in the study of the motor cortical activity associated with a single/double segment movements (Ben-Shaul et al., 2004), the neuronal activity in the motor cortex (PMd and M1) associated with a given motion segment differs between the two contexts. Despite context-related differences on the single-neuron level, the population as a whole still allows a reliable readout of movement direction regardless of the sequential context. “This implies that direction of a movement and the sequential context in which it is embedded may be simultaneously and reliably encoded by neurons in the motor cortex” (Ben-Shaul et al., 2004). Recent analysis of the involvement of the motor cortex in sequence production challenged the role of medial motor areas in the control of well-practiced movement sequences and suggested that the motor cortex contains a complete apparatus for the planning and production of this complex behavior (Lu & Ashe, 2005). Considering the question of what is represented in the motor cortex (see introduction and introduction in chapter 3), we hypothesize that encoding of the sequential behavior in the motor cortex may also be considered in the framework of dynamically switching cortical states. Encoding of the movements from the same cluster may correspond to a specific attractor. Dwelling near attractors is also supported by our results of the HMM-based analysis. Convergence to the attractor behavior is also compatible with the principle of greater parsimony.

6.5 6.5.1

Further questions and directions Preliminary perspective for the hierarchical scheme of movement compositionality; ability to co-articulate

As mentioned earlier, we have shown that in some cases the monkey changes its strategy according to getting/not getting a reward. Considering online decisionmaking, we relate it to the compositional rules, and show here a hierarchical scheme for generating parabolic trajectories (and thus generalizable to piece-wise parabolic trajectories) based on vectorial composition of the short point-to-point minimum-jerk movements. Short point-to-point minimum-jerk movement may be considered as the simplest primitive in the hierarchy. Vectorial summation of the point-to-point minimum-jerk movements (X(τ ) =


Chapter VI

149

Xstart + (Xend − Xstart ) (10τ 3 − 15τ 4 + 6τ 5 ), τ = t/duration) was used successfully to model double step movements (Flash & Henis, 1991; Henis & Flash, 1995). The minimum-jerk movements with one via-point and conditions of zero velocity and acceleration on the boundaries follow approximately parabolic paths, as we show in chapter 2, see also figure 2.16. Our case studies show that the curved minimum-jerk trajectories with one via-point and zero boundary conditions, can be approximated with two to three point-to-point minimum-jerk movements with zero boundary conditions. We tested several examples with obtuse angles between the segments formed by the end-points and the via-point. Comparisons between the minimum-jerk trajectory and its reconstruction based on two point-to-point minimum-jerk trajectories are depicted in figures 6.1 and 6.2. In figure 6.1 the path (blue) with its reconstrucI

II

0.6 Path Reconstructed Point−to−point

0.5

Path Reconstructed Point−to−point

1.2

1

0.8

0.3

y, [units]

y, [units]

0.4

0.2

0.6

0.1 0.4 0 0.2 −0.1

0

0.2

0.4

0.6

x, [units]

0.8

1

0

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

x, [units]

Figure 6.1: The minimum-jerk path (blue) with its reconstruction (magenta) based on vectorial summation of two (for obtuse angle) or three (for acute angle) point-to-point minimum-jerk trajectories (black segments). The via-point is depicted by blue asterisk. The reconstruction of the path is close to the minimum-jerk path (which is close to being parabolic). The velocity components of the minimum-jerk trajectory and its reconstruction are depicted in figure 4.8. (I) Obtuse angle. (II) Acute angle.

tion (magenta) based on vectorial summation of two point-to-point minimum-jerk trajectories (black segments) are plotted. The via-point is depicted by blue asterisk. The reconstruction of the path is close to the minimum-jerk path (which is close to being parabolic). The velocity components of the minimum-jerk trajectory and its reconstruction are depicted in figure 4.8: blue — component of the minimum-jerk profile, green — components of the point-to-point reconstruction, black dashed — sum of the reconstructed components. Comparing the speed of the minimum-jerk


Chapter VI

150

I A. X component of velocity

−3

2

II

x 10

A. X component of velocity

−4

20

Vx

V

x

ΣV

1.5

x 10

ΣV

15

xi

xi

Vxi

V

xi

10 1

5 0.5

0

0

0

0.1

0.2

0.4

0.5

0.6

0.7

0.8

0.9

1

−5

B. Y component of velocity

−3

1.5

0.3

x 10

0

0.1

0.2

0.4

0.5

0.6

0.7

0.8

0.9

x 10

Vy

ΣVyi

1

ΣVyi

Vyi

0.5

1

B. Y component of velocity

−3

5

Vy

0.3

Vyi 0

0 −0.5 −1

0

0.1

0.2

0.3

0.5

0.6

0.7

0.8

0.9

1

−5

C. Speeds

−3

3

0.4

x 10

0

0.1

−3

5

Speed Sp. reconstr. ΣSpi

2.5 2

Spi

x 10

0.2

0.3

0.4

0.5

0.6

0.7

0.9

1

C. Speeds. Costst=0.0034499, costopt=0.00068468

4

Speed Sp. reconstr. ΣSpi

3

Spi

1.5

2

1

1

0.5 0

0.8

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Time

Figure 6.2: The velocity components and the speed of the minimum-jerk trajectory and its reconstruction, corresponding to the path depicted in figure 6.1. Blue — component of the minimum-jerk profile, green — components of the point-to-point reconstruction, black dashed — sum of the reconstructed components. (A) x-component; (B) y-component; (C) corresponding speed. In (C) the speed of the vectorial sum of two point-to-point trajectories is depicted by magenta color. Comparing the speed of the minimum-jerk trajectory and of the reconstructed trajectory, one can see that the reconstruction is close to the minimum-jerk movement. (I) Obtuse angle. (II) Acute angle.


Chapter VI

151

trajectory and of the reconstructed trajectory, one can see that the reconstruction is close to the minimum-jerk movement. The speed profile of the point-to-point movement is defined by two scalars: movement extent Xend − Xstart and movement duration. We found that point-to-point minimum-jerk movement is decomposable into three identical point-to-point movements, as depicted in figure 6.3. Therefore, the formation of parabolic movements can be decomposed hierarchically into short point-to-point minimum-jerk movements. The ratio of the peaks of the decomposed to the decomposing speed profile is approximately 1 : 0.55, meaning that the corresponding magnitude of speed decreases twice while considering a one step lower hierarchical level. Speeds ratio ~0.55 Vx ΣVxi Speed, [units]

Vxi

Time, [units]

Figure 6.3: Point-to-point minimum-jerk movement is decomposable into three identical pointto-point movements. The ratio of the peaks of the decomposed to the decomposing speed profile is approximately 1 : 0.55. Construction of parabolic movements can be decomposed hierarchically into short point-to-point minimum-jerk movements.

Human subjects succeeded to co-articulate the obtuse movements with a viapoint, but not acute ones, as one can see from figure 6.4 (Sosnik et al., 2001). In the experiment described in (Sosnik et al., 2004) in more detail, the human subject had to connect a sequence of rectangles (abcd) “as fast and as accurate as possible”. In figure 6.4, the drawings correspond to the days of practice for the obtuse configuration of the line segments defined by the via-points (a) and the acute configuration (b). In the first day both types of movements were segment-like, though through practice they became more and more co-articulated (days 3 and 5) for obtuse configuration. The duration of the performance of the task also became shorter for the template with the


Chapter VI

152

obtuse configuration compared to the acute configuration. Through practice, the coarticulated drawing movements became well-fitted by the minimum-jerk trajectories with one via-point (Sosnik et al., 2004). The recorded movements could be vectorially decomposed into two or three pointto-point minimum-jerk trajectories (based on personal communication with Ronen Sosnik and Tamar Flash). For example, the obtuse configuration can be well fit with two point-to-point minimum-jerk profiles, and the acute configuration can be represented using three such profiles. The following hypothesis has to be tested then: is the minimum number of point-to-point minimum-jerk movements needed to fit a minimum-jerk profile for a given point-wise configuration a defining factor in the ability of the subject to co-articulate? A positive answer would mean that only two elementary point-to-point submovements have to be considered simultaneously in the decomposition of complex drawing trajectories. Such a decomposition may be applied to the recorded monkey drawing movements, e.g. to the non-rewarded trajectories as in figures 4.3, 4.5. We suggest that piece-wise parabolic trajectories may be executed hierarchically with short point-to-point minimum-jerk movements in the lowest level of the hierarchy. Each level of the hierarchy consists of the concatenated elements from the lower hierarchy.

6.5.2

Is there a syntax of hidden states?

Are movements corresponding to the hidden state n, in the sequence mn different from the movements corresponding to the state n in the sequence kn (k 6= m)? We intended to test the paths corresponding to a specific sequence of the flipped hidden states. For example, which are the paths that correspond to the sequence {3, 6, 2}, and how different are they from the paths corresponding to the sequence {4, 6, 2}? We introduce a function ξ(t) = arg maxi {pi (t)}, which is the label of the hidden state with the highest a-posterior probability at time t. The sets of continuously repeated values of ξ(t) are then replaced by a single value and a trimmed sequence is obtained. For example, the series {1, 1, 1, 6, 3, 3} is to be replaced with the trimmed sequence {1, 6, 3}. Such sequences appear as state labels in figures 5.7, H.39-H.46 above each plot. Further, for a sequence of labels, we can find all corresponding repetitions of the trimmed sequences. Examples of the paths corresponding to the trimmed sequences of the labels are depicted in figure 6.5. Note small loops in the paths 1, 3, 6, in figure 6.5A, which correspond to the sequence of states 834. Other states, beside 8, being before the sequence of states 34 do not correspond to such loops. Note also that the sequence 45 (yellow, green) contains the cusp with the normal oriented downward. The pair 45 may underly the construction of a class of parabolic pieces.


Chapter VI

153

Figure 6.4: Illustration taken from the poster (Sosnik et al., 2001). Briefly, the human subject had to connect a sequence of rectangles (abcd) “as fast and as accurate as possible”. The experiment is described in (Sosnik et al., 2004) in more details. The depicted drawings corresponding to the days of practice for the obtuse configuration of the line segments defined by the via-points (a) and the acute configuration (b). In the first day both types of movements were segment-like, though with practice they became more and more co-articulated (days 3 and 5) for obtuse configuration. The duration of the performance of the task also became shorter for the template with obtuse configuration comparing with the acute configuration.

6.5.3

List of questions

1. The PCCM analysis has been performed on the entire data set from the segments of active motion. It should be carried out separately for the clusters of parabolic pieces, HMM states. This could be a straightforward test of “attractor”-based representation of parabolic clusters/HMM states. 2. Studies of the motor cortical activity similar to (Kenet et al., 2003) may clarify whether motor cortical activity dwells in stable cortical states. It may also shed more light on the similarities and differences in the functional organization of the primary motor and primary visual cortices.


Chapter VI

154

A

B seq: 2 3 4 5 6

seq: 8 3 4 5 6

2

3

4

5

6

7

8

9

10

11

12

13

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Figure 6.5: Examples of the paths corresponding to two sequences of states. A. The sequence is 8 3 4 5 6. B. The sequence is 2 3 4 5 6. Note small loops in the paths 1, 6, in A., which correspond to the sequence of states 834. Other indices, beside 8, being before 34 do not form such loops. Note also that the sequence 45 (yellow, green) contains the cusp with the normal oriented downward. The pair 45 may underly the construction of a class of parabolic pieces.

3. Comparison of the results of the HMM analysis with the analysis based on the jittered neural data may be a test to the validity of the HMM-based segmentation. 4. The results of the HMM-based segmentation for the populations from which certain neurons are removed may lead to the conclusions about the properties of those specific neurons. 5. Involvement of the decision-making into trajectory formation should be studied further. The hierarchical decomposition into primitive, short minimum-jerk


Chapter VI

155

submovements may be considered in such study. It may involve group generators. In particular, one should test whether the drawing of piece-wise parabolic trajectories can be represented hierarchically with short point-to-point minimum-jerk movements in the lowest level of the hierarchy. 6. A related test may consider whether the ability to co-articulate serial point-topoint movements may depend on the angle in the configuration of the line segments connecting the points. Co-articulation was not observed in case the angle is acute (Sosnik et al., 2001). We could reconstruct the exemplary simulated minimum-jerk trajectories based on the obtuse angles with 2 point-to-point movements, and those based on the acute angles only with 3 movements. Is the ability of the subject to co-articulate defined by the minimum number of the point-to-point minimum-jerk movements needed to fit a single minimum-jerk profile with one via-point (e.g. for 2 movements the subject does coarticulate, and for 3 does not)?

Bibliography Abeles M., Bergman H., Gat I., Meilijson I., Seidemann E., Tishby N. & Vaadia E. (1995). Cortical activity flips among quasi-stationary states. PNAS , 92(19):8616– 8620. Alexander G.E. & Crutcher M.D. (1990). Neural representations of the target (goal) of visually guided arm movements in three motor areas of the monkey. J Neurophysiol , 64(1):164–178. Amirikian B. & Georgopoulos A.P. (2003). Modular organization of directionally tuned cells in the motor cortex: Is there a short-range order? PNAS , 100(21):1247412479. B.Averbeck B., V.Chafee M., A.Crowe D. & P.Georgopoulos A. (2002). Parallel processing of serial movements in prefrontal cortex. PNAS , 99:13172–13177. Ben-Shaul Y., Drori R., Asher I., Stark E., Nadasdy Z. & Abeles M. (2004). Neuronal activity in motor cortical areas reflects the sequential context of movement. The Journal of Neurophysiology, 91:1748–1762. Bishop C.M. (1995). Neural networks for pattern recognition. Clarendon Press, Oxford. Bjursten L.M., Norrsell K. & Norrsell U. (1976). Behavioural repertory of cats without cerebral cortex from infancy. Experimental Brain Research, 25(2):115–130. Blaschke W. (1923). Vorlesungen Differentialgeometrie. Springer-Verlag, Berlin. Calabi E., Olver P.J. & Tannenbaum A. (1996). Affine geometry, curve flows, and invariant numerical approximations. Adv. in Math., 124:154–196. Www: http://www.math.umn.edu/ olver/papers.html. Calabi E., Shakiban C., Olver P.J., Tannenbaum A. & Haker S. (1998). Differential and numerically invariant signature curves applied to object recognition. International Journal of computer Vision, 26(2):107–135. 156


Bibliography

157

Carpenter A.F., Georgopoulos A.P. & Pellizzer G. (1999). Motor cortical encoding of serial order in a context-recall task. Science, 283(5408):1752–1757. Cisek P. (2005). Neural representations of motor plans, desired trajectories, and controlled objects. Cognitive Processing, 6(11):15–244. Cisek P. & Kalaska J.F. (2005). Neural correlates of reaching decisions in dorsal premotor cortex: specification of multiple direction choices and final selection of action. Neuron, 45(5):801–814. Coleman T., Branch M.A. & Grace A. (1999). Optimization Toolbox for use with Matlab. MathWorks. Cover T.M. & Thomas J.A. (1991). Elements of Information Theory. Wiley, New York. Dayan P. & Abbott L. (2002). Theoretical Neuroscience. Electronic version. DeAngelis G.C., Ghose G.M., Ohzawa I. & Freeman R.D. (1999). Functional microorganization of primary visual cortex: Receptive field analysis of nearby neurons. The Journal of Neuroscience, 19(9):40464064. Drori R. (2005). Neural representation of drawing movements in the motor cortex . PHD Thesis, Hebrew University. Dum R. & Strick P. (1991). The origin of corticospinal projections from the premotor areas in the frontal lobe. Journal of Neuroscience, 11(3):667–689. Efimov A. (1980). Mathematical analysis (special topics), general functional series and their applications, volume 1. Vysshaya shkola, Moscow. Evarts E.V. (1968). Relation of pyramidal tract activity to force exerted during voluntary movement. J Neurophysiol , 31(1):14–27. Fishbach A., Roy S., Bastianen C., Miller L. & Houk J. (2005). Kinematic properties of on-line error corrections in the monkey. Experimental Brain Research, 164(4):442–457. Flament D. & Hore J. (1988). Relations of motor cortex neural discharge to kinematics of passive and active elbow movements in the monkey. J Neurophysiol , 60(4):1268– 1284.


Bibliography

158

Flash T., Gurevich I. & Henis E. (1996). Characterizing and modelling human arm movements: insights into motor organization. in ”Advances in Processing and Pattern Analysis of Biological Signals”, edited by Isak Gath and Gideon F.Inbar, Plenum Press, New York . Flash T. & Handzel A. (1996). Affine differential geometry analysis of human arm trajectories. Abstracts of the Society of Neuroscience, 22. Flash T. & Henis E. (1991). Arm trajectory modification during reaching towards visual targets. Journal of Cognitive Neuroscience, 3(3):220–230. Flash T. & Hogan N. (1985). The coordination of arm movements: an experimentally confirmed mathematical model. The Journal of Neuroscience, 5(7):1688–1703. Fu Q.G., Flament D., Coltz J.D. & Ebner T.J. (1995). Temporal encoding of movement kinematics in the discharge of primate primary motor and premotor neurons. J Neurophysiol , 73(2):836–854. Gat I. & Tishby N. (1993). Statistical modeling of cell assemblies activities in associative cortex of behaving monkeys. Neural Information Process. Syst., 5:945–953. Gat I., Tishby N. & Abeles M. (1997). Hidden markov modelling of simultaneously recorded cells in the associative cortex of behaving monkey. Network: Computation in Neural System, 8:297–322. Gelfand I. & Fomin S. (1961). Calculus of variations. Nauka, Moscow. Georgopoulos A. & Grillner S. (1989). Visuomotor coordination in reaching and locomotion. Science, 245:1209–1210. Georgopoulos A., Lurito J., Petrides M., Schwartz A. & Massey J. (1989). Mental rotation of the neuronal population vector. Science, 243:234–236. Georgopoulos A.P., Kalaska J.F., Caminiti R. & Massey J.T. (1982). On the relations between the direction of two-dimensional arm movements and cell discharge in primate motor cortex. The Journal of Neuroscience, 2(11):1527–1537. Giese M.A. & Poggio T. (2000). Morphable models for the anlysis and synthesis of complex motion patterns. International Journal of Computer Vision, 38(1):59–73. Golub G.H. & Loan C.F.V. (1983). Matrix computations. John Hopkins University Press, Baltimore. Grillner S., Georgopoulos A.P. & Jordan L.M. (1997). Selection and initiation of motor behavior , chapter 1. Neurons, networks, and motor behavior. MIT Press, .


Bibliography

159

Guggenheimer H.W. (1977). Differential Geometry. Dover, New York. Handzel A. & Flash T. (1999a). Geometric methods in the study of human motor control. Cognitive studies, 6(3):309–321. Handzel A. & Flash T. (2001). Affine invariant edge completion with affine geodesics. IEEE Workshop on Variational and Level Set Methods (VLSM’01). Handzel A.A. & Flash T. (1999b). Geometric methods in the study of human motor control. Cognitive Studies, Bulletin of the Japanese Cognitive Science Society, 6(3):309–321. Hanneton S., Berthoz A., Droulez J. & Slotine J. (1997). Does the brain use sliding variables for the control of movements? Biological Cybernetics, 77(6):381–393. Harris C.M. & Wolpert D.M. (1998). Signal-dependent noise determines motor planning. Nature, 394:780–784. Henis E.A. & Flash T. (1995). Mechanisms underlying the generation of averaged modified trajectories. Biological Cybernetics, 72(5):407–419. Hernandez A., Zainos A. & Romo R. (2002). Temporal evolution of a decision-making process in medial premotor cortex. Neuron, 33(6):959–972. Hocherman S. & Wise S.P. (1991). Effects of hand movement path on motor cortical activity in awake, behaving rhesus-monkeys. Experimental Brain Research, 83(2):285–302. Hubel D. & Wiesel T.N. (1962). Receptive fields, binocular interaction and functional architecture of cat’s visual cortex. J. PhysioL, 160:106–154. Huk A.C. & Shadlen M.N. (2005). Neural activity in macaque parietal cortex reflects temporal integration of visual motion signals during perceptual decision making. J. Neurosci., 25(45):10420–10436. Jax S.A. & Rosenbaum D.A. (2003). Sequential effects in obstacle avoidance: the obstacle-perseveration effect. Poster . Kenet T., Bibitchkov D., Tsodyks M., Grinvald A. & Arieli A. (2003). Spontaneously emerging cortical representations of visual attributes. Nature, 425:954–956. Kettner R., Schwartz A. & Georgopoulos A. (1988). Primate motor cortex and free arm movements to visual targets in three- dimensional space. iii. positional gradients and population coding of movement direction from various movement origins. J. Neurosci., 8(8):2938–2947.


Bibliography

160

Krebs H., Aisen M., Volpe B. & Hogan N. (1999). Quantization of continuous arm movements in humans with brain injury. Proceedings of the National Academy of Science of the USA, 96(8):4645–4649. Lacquaniti F., Terzuolo C. & Viviani P. (1983). The law relating the kinematic and figural aspects of drawing movements. Acta Psychologica, 54:115–130. Levit-Binnun N., Schechtman E. & Flash T. (2005). On the similarities between the perception and production of elliptical trajectories. Experimental Brain Research, online. Liddell E. & Phillips C. (1944). Pyramidal sections in the cat. Brain, 67:1–9. Liu N. & Lovell B.C. (2001). Mmx-accelerated real-time hand tracking system. Proceedings IVCNZ 2001, 26-28 November, Dunedin, New Zealand , pages 381–385. Lu X. & Ashe J. (2005). Anticipatory activity in primary motor cortex codes memorized movement sequences. Neuron, 45:967–973. Maynard E.M., Hatsopoulos N.G., Ojakangas C.L., Acuna B.D., Sanes J.N., Normann R.A. & Donoghue J.P. (1999). Neuronal interactions improve cortical population coding of movement direction. J. Neurosci., 19(18):8083–8093. McLeod R.J.Y. & Baart M.L. (1998). Geometry and Interpolation of Curves and Surfaces. Cambridge University Press, Cambridge. Moran D.W. & Schwartz A.B. (1999). Motor cortical activity representation of speed and direction during reaching. Journal of Neurophysiology, 82:2676–2692. Moran D.W., Schwartz A.B., Georgopoulos A.P., Ashe J., Todorov E. & Scott S.H. (2000). One motor cortex, two different views. Nature Neuroscience, 3(10):963–965. Myers J.L. & Well A.D. (2003). Research design and statistical analysis. Lawrence Erlbaum Associates, Inc. Panzeri S., Treves A., Schultz S. & Rolls E.T. (1999). On decoding the responses of a population of neurons from short time windows. Neural Comput JT - Neural computation., 11(7):1553–1577. Paz R., Boraud T., Natan C., Bergman H. & Vaadia E. (2003). Preparatory activity in motor cortex reflects learning of local visuomotor skills. Nature Neuroscience, 6:882–890. Petitot J. (2003). The neurogeometry of pinwheels as a sub-riemannian contact structure. Journal of Physiology - Paris, 97:265–309.


Bibliography

161

Pollick F.E., Flash T., Giblin P. & Sapiro G. (1997). Three dimensional movements at constant affine velocity. Abstracts of the Society for Neuroscience. Pollick F.E. & Sapiro G. (1997). Constant affine velocity predicts the 1/3 power law of planar motion perception and generation. Vision Research, 37(3):347–353. Polyakov F. (2001). Analysis of monkey scribbles during learning in the framework of models of planar hand motion. MSc Thesis, Department of Computer Science and Applied Mathematics, Weizmann Institute of Science. Polyakov F., Flash T., Abeles M., Ben-Shaul Y., Drori R. & Nadasdy Z. (2001). Analysis of motion planning and learning in monkey scribbling movements. Proceedings of the tenth biennial conference of the International Graphonomics Society, the University of Nijmegen, Nijmegen, The Netherlands:78–83. Polyakov F., Flash T., Abeles M., Ben-Shaul Y., Drori R. & Nadasdy Z. (2003). Analysis and modeling of drawing movements. Abstracts of the Society of Neural Control of Movement, pages Group G – Sequencing and learning. Rabiner L.R. (1989). A tutorial on hidden markov-models and selected applications in speech recognition. Proceedings of the IEEE , 77(2):257–286. Richardson M.J.E. & Flash T. (2002). Comparing smooth arm movements with the two-thirds power law and the related segmented-control hypothesis. Journal of Neuroscience, 22(18):82018211. Romo R., Hernandez A. & Zainos A. (2004). Neuronal correlates of a perceptual decision in ventral premotor cortex. Neuron, 41(1):165–173. Salinas R.R..E. (2003). Flutter discrimination: neural codes, perception, memory and decision making. Nature Reviews Neuroscience, 4(3):203–218. Schaal S., adn Reiko Osu D.S. & Kawato M. (2004). Rhythmic arm movement is not discrete. Nat Neuroscience, 7(10):1136–1143. Schwartz A.B. (1992). Motor cortical activity during drawing movements: single-unit activity during sinusoid tracing. Journal of Neurophysiology, 68(2):528–541. Schwartz A.B. (1994). Direct cortical representation of drawing. Science, 265:540– 542. Schwartz A.B. & Moran D.B. (1999). Motor cortical activity during drawing movements: population representation during lemniscate tracing. Journal of Neurophysiology, 82(5):2705–2718.


Bibliography

162

Scott S.H. & Kalaska J.F. (1995). Changes in motor cortex activity during reaching movements with similar hand paths but different arm postures. J Neurophysiol , 73(6):2563–2567. Shadlen M.N. & Newsome W.T. (2001). Neural basis of a perceptual decision in the parietal cortex (area lip) of the rhesus monkey. Journal of Neurophysiology, 86(4):1916 – 1936. Shinoda K. & Watanabe T. (2000). Mdl-based context-dependent subword modeling for speech recognition. J. Acoust. Soc. Jpn. (E), 21(2):79–86. Shirokov P. & Shirokov A. (1959). Affine differential geometry. GIFML, Moscow. German edition: Affine differentialgeometrie, Teubner, 1962. Shmiel T., Drori R., Shmiel O., Ben-Shaul Y., Nadasdy Z., Shemesh M., Teicher M. & Abeles M. (2005). Neurons of the cerebral cortex exhibit precise interspike timing in correspondence to behavior. PNAS , 102(51):18655–1657. Sosnik R. (2004). The evolution of writing-like sequence movement representation during practice. PHD thesis, Weizmann Institute. Sosnik R., Flash T., Hauptmann B. & Karni A. (2001). The evolution of versatile but effector specific novel motor. Society for Neuroscience. Sosnik R., Hauptmann B., Karni A. & Flash T. (2004). When practice leads to co-articulation: the evolution of geometrically defined movement primitives. Experimental Brain Research, 156. Spivak M.A. (1979). A Comprehensive Introduction to Differential Geometry, volume 1. Publish or Perish Press, Berkeley, CA. Stark E., Drori R. & Abeles M. (2006). Partial cross-correlation analysis resolves ambiguity in the encoding of multiple movement features. J Neurophysiol , 95(3):1966– 1975. Thach W.T. (1978). Correlation of neural discharge with pattern and force of muscular activity, joint position, and direction of intended next movement in motor cortex and cerebellum. J Neurophysiol , 41(3):654–676. Todorov E. (2000). Direct cortical control of muscle activation in voluntary arm movements: a model. Nature Neuroscience, 3(4):391–398. Todorov E. & Jordan M.I. (1998). Smoothness maximization along a predefined path accurately predict the speed profiles of complex arm movements. Journal of Neurophysiology, 80(2):696–714.


Bibliography

163

Tsodyks M., Kenet T., Grinvald A. & Arieli A. (1999). Linking spontaneous activity of single cortical neurons and the underlying functional architecture. Science, 286(5446):1943–1946. Viviani P. (1986). Generation and modulation of action patterns, chapter Do units of motor action really exist?, pages 201–206. Number 15 in Experimental Brain Research. Berlin: Springer-Verlag. Viviani P. & Cenzato M. (1985). Segmentation and coupling in complex movements. Journal of Experimental Psychology: Human Perception and Performance, 11(6):828–845. Viviani P. & Flash T. (1995). Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. Journal of Experimental Psychology: Human Perception and Performance, 21(1):233–242. Viviani P. & Stcucchi N. (1992). Biological movements look uniform: evidence of motor-perceptual interactions. Journal of Experimental Psychology: Human Perception and Performance, 18(3):603–623. Welling M. (2002). Estimation. http://www.ics.uci.edu/ welling/classnotes/papers class/StatEst.ps.gz . Wiener M.C. & Richmond B.J. (2003). Decoding spike trains instant by instant using order statistics and the mixture-of-poissons model. The Journal of Neuroscience, 23(6):2394–2406. W.Wu, Black M., Gao Y., Bienenstock E., Serruya M. & Donoghue J. (2002). Inferring hand motion from multi-cell recordings in motor cortex using a kalman filter. SAB’02-Workshop on Motor Control in Humans and Robots: On the Interplay of Real Brains and Artificial Devices, August 10, 2002 . W.Wu, Black M., Gao Y., Bienenstock E., Serruya M., Shaikhouni A. & Donoghue J. (2003). Neural decoding of cursor motion using a kalman filter. Advances in Neural Information Processing Systems, 15:133–140.

Index rXY , 68 An , 164 σ, 16 (a × b), 16 α, 29 r, 39

jerk, 19 local isochrony, 41 main theorem of equiaffine theory of plain curves, 17 model minimum-jerk — — — with via-points, 19 constrained — —, 19 two-thirds power-law —, 19

affine system of coordinates, 165 arc-length equiaffine —, 16 CC, see correlation coefficient correlation — coefficient, 68 partial —, 67 curvature, 168 equiaffine —, 17, 170 — —, arbitrary parameterization, 17 — —, natural parameterization, 17 Euclidian —, 19

parabola, 18, 20 partial cross-correlation matrix, 69 PCCM, see partial cross correlation matrix PETH, 75 primitive movement —, 109 parabolic —, 110 principle of greater parcimony, 34 propagation of the fitted curve, 21

distribution Poissonian —, 54

quantization, 53

element of arc, 167 ellipse, 18 equiaffine normal, 17 equiaffine tangent, 17 geometry equiaffine —, 16, 166 hyperbola, 18

segment — of the neural activity, 75 movement —, 11 transformation affine —, 165 equiaffine —, 16, 166 velocity equiaffine —, 16, 19 tangential —, 18 164

Appendix A Topics from equiaffine differential geometry The tools of equiaffine differential geometry play a very important role in this research. We therefore present in this Appendix the basic definitions and properties (some of them are followed with derivations) related to the topic and most of the discussion is devoted to the theory of equiaffine differential geometry of plain curves. In the end of this Appendix we included a subsection concerning curves with constant equiaffine curvature - they appear to be the conics. The material of this Appendix can be considered as widening of the section “Methods” from chapter 2.

A.1

Some definitions and properties from affine geometry (Shirokov & Shirokov, 1959)

Definition A.1.1. The set of elements (points) is called n-dimensional affine space −→ An , if for every pair of points (A, B) of this set, there exists a vector a = AB ∈ Rn and the following conditions are satisfied: 1. ∃P ∈ An such that there exists one-to-one correspondence between the vectors from Rn and the pairs (P, A). −→ −−→ −→ 2. If a = P A and b = P B, then the vector AB = b − a corresponds to the pair (A, B). It follows from the definition that ∀A ∈ An and ∀x ∈ Rn ∃! B ∈ An such that −→ −−→ −→ AB = x. This point B is defined by the vector P B = P A + x. Therefore the point 165


Appendix

A

166

P does not play any special role and can be replaced by any point of the affine space. If we choose some point O from affine space and set it to be the origin, we establish a one-to-one correspondence among the points M of the space and the vectors OM . We can decompose the vector OM into linear combination of basis-vectors OM = x1 e +x2 e + . . . + xn e of some basis {e, e, . . . , e}. The aggregate of the point O and 1

2

n

1

2

n

the basis {e, e, . . . , e} is called affine system of coordinates; the point O is called 1

2

n

origin and the vectors {e} are called basis-vectors. Affine system of coordinates is i

denoted as {O; e, e, . . . , e}. 1

2

n

Together with the system of coordinates {O; e, e, . . . , e} we consider a new system 1 2 n n P 0 0 0 0 of coordinates {P ; e , e , . . . , e }, where OP = ai e = ai e, e = Ask0 e; det(Ask0 ) 6= 1

2

n

i

i=1

i

s

k

0. If the point M has coordinates x1 , x2 , . . . , xn in the first system of coordinates and 0 0 0 has coordinates x1 , x2 , . . . , xn in the newer system of coordinates, we then have: 0

0

0

OM = xi e = OP + P M = ai e +xs e = (ai + Ais0 xs ) e , i

s

i

from that follows

i

0

xi = ai + Ais0 xs .

(A.1.1) i0

If we take some vector u(u1 , u2 , . . . , un ), then its old (ui ) and new (u ) coordinates are related as follows: ( 0 ui = Ais0 us , 0

ui

0

= Ais us .

0

Where Asi are elements of the matrix, inverse to the matrix (Aij 0 ). ˆ , which has the same coordinates x1 , x2 , . . . , xn in the Let us now fix the point M 0 0 0 system {P ; e , e , . . . , e } as the point M in the system {O; e, e, . . . , e}. Denoting 1 2 n 1 2 n ˆ w.r.t. the system {O; e, e, . . . , e}, we xˆ1 , xˆ2 , . . . , xˆn the coordinates of the point M obtain:

1

2

n

ˆ = xî e = OP + P M ˆ = ai e +xs e 0 = (ai + Ais0 xs ) e , OM i

i

s

i

from that follows: xî = ai + Ais0 xs .

(A.1.2) ˆ The transformation of affine space which transfers the point M to the point M and is defined by the equation A.1.2, is called affine transformation. The formulas A.1.1 and A.1.2 are completely analogous and we can keep the ˆ interpretation that the formulas A.1.2 express the old coordinates of the point M


Appendix

A

167

through its new coordinates, after the transition from the system {O; e, e, . . . , e} to 0

0

1

0

2

n

the system {P ; e , e , . . . , e }. 1

2

n

Equiaffine geometry The set of affine transformation xî = Bsi xs + ai for which det(Bji ) = 1 forms the subgroup of the general affine group; the transformations belonging to this group are called equiaffine. The ensemble of the properties of geometric forms, which are conserved under the equiaffine transformations, forms the contents of the equiaffine geometry.

A.2

Curvature and element of arc in the geometry of the r-parametric Lie group

In this section we follow the §8 of the introduction of (Shirokov & Shirokov, 1959).

Let local r-parametric Lie group G on the plane be given with the equations # " # " # " x x1 f1 (x, y, a1 , . . . , ar ) , G 3 g = g(a1 , . . . , ar ) . =g· = y f2 (x, y, a1 , . . . , ar ) y1 For a given curve y = y(x), y ∈ C (r−1) we use the notation     x x      y    y         dy 0 = . e= y     dx  .   .   ..    ..     dy (r−3) (r−2) y dx Let  x1   y  1 e1 =   y1 0 ...  y1 (r−2)







       =      

f1 (x, y; a1 , . . . , ar ) f2 (x, y; a1 , . . . , ar ) f3 (x, y, y 0 ; a1 , . . . , ar ) .. . fr (x, y, y 0 , . . . , y (r−2) ; a1 , . . . , ar )

     = pr(r−2) g · e    


Appendix

A

168

be elements of the (r − 2)-th prolonged action of g on e. Let us assume that the prolonged group locally acts transitively, i.e., it can transfer any element e to any element e1 whenever e1 is contained in a small enough neighborhood of e, by the transformation close to identity (values of |aα | are small). These parameters can be uniquely expressed with elements e, e1 (Shirokov & Shirokov, 1959): a1 = a1 (e, e1 ), . . . , ar = ar (e, e1 ) . (A.2.1) Let us write the expressions for differentials of the transformed points: ∂f1 ∂f1 dx + dy, ∂x ∂y ∂f2 ∂f2 dy1 = dx + dy . ∂x ∂y

dx1 =

Substituting expressions (A.2.1) instead of parameters aα , we get: ( dx1 = β1 (e, e1 ) dx + β2 (e, e1 ) dy dy1 = γ1 (e, e1 ) dx + γ2 (e, e1 ) dy .

(A.2.2)

The above equalities stay valid if we fix dx1 , dy1 , e1 and apply the group action on dx, dy, e. Introducing the notation β1 (e, e1 ) = λ1 (e), β2 (e, e1 ) = λ2 (e), γ1 (e, e1 ) = µ1 (e), γ2 (e, e1 ) = µ2 (e) for fixed dx1 , dy1 , e1 we get the differential forms ω1 = λ1 (e) dx + λ2 (e) dy , ω2 = µ1 (e) dx + µ2 (e) dy .

(A.2.3) (A.2.4)

Noting that along the curve dy = y 0 dx, we will rewrite the forms ω1 and ω2 for the curve: ω1 = [λ1 (e) + λ2 (e)y 0 ] dx = λ(e) dx , ω2 = [µ1 (e) + µ2 (e)y 0 ] dx = µ(e) dx .

(A.2.5) (A.2.6)

The ratio of these forms is an invariant of the group, and it is constant because the group is transitive. Thus, for each curve there exists an invariant form ω = ω(e) dx , which is defined up to a constant factor. This invariant form is called an element of an arc in the geometry of the given r-parametric Lie group. If we proceed with the substitution for the differential of the variable y1 (r−2) and fixing again the element e1 , and noting that along the curve dy = y 0 dx, dy 0 = y 00 dx, . . . , dy (r−2) = y (r−1) dx ,


Appendix

A

169

we get from the expression dy1 (r−2) =

∂fr ∂fr ∂fr ∂fr dx + dy + 0 dy 0 + . . . + (r−2) dy (r−2) ∂x ∂y ∂y ∂y

an invariant form on the curve: [α(e) + β(e)y (r−1) ]dx . If we divide this form by the arc element of the curve σ = ω(e) dx, we shall obtain the following differential invariant of the curve of the order (r − 1): κ = α(e) + β(e)y (r−1) . This invariant is called curvature of the curve in the geometry of the given r-parametric Lie group. Another method of calculating κ and dσ, which involves operations on prolongations of the infinitesimal generators of the r-parametric Lie group, is described in (Shirokov & Shirokov, 1959).

A.2.1

Element of arc and curvature of curve in the geometry of the equiaffine group (Shirokov & Shirokov, 1959)

The transformations of the group are of the form: ¯ ¯ α β x1 = αx + βy + a ¯ ∆=¯ ¯ γ δ y1 = γx + δy + b ,

¯ ¯ ¯ ¯ = 1. ¯

For the fourth prolongation we have: y1 0 =

γ + δy 0 , α + βy 0

y1 00 =

y 00 , (α + βy 0 )3 y1 IV =

y1 000 =

y 000 2βy 00 2 − , (α + βy 0 )4 (α + βy 0 )5

y IV 10βy 00 y 000 15β 2 y 00 3 − + . (α + βy 0 )5 (α + βy 0 )6 (α + βy 0 )7

It can be obtained from the above: dx1 = (α + βy 0 )dx = y1 00

(−1/3) 00 1/3

it follows then that element of the arc will be: dσ = y 00

1/3

dx .

y

dx ,


Appendix

A

170

For arbitrary parametrization with the parameter t we get then σ˙ = (r˙ × r¨)1/3 .

(A.2.7)

It can be obtained for the curvature: 5 y 000 2 1 ¡ 00 (−2/3) ¢00 1 y (IV) − = − . κ= y 3 y 00 5/3 9 y 00 8/3 2 This can be rewritten for the arbitrary parameter: IV ... ... 3σ˙ 3 (r˙ × r ) + 12σ˙ 3 (¨ r × r ) − 5(r˙ × r )2 . κ= 9σ˙ 8

(A.2.8)

Exactly the same formulas were obtained with the method of Cartan moving frame in H.W.Guggenheimer ‘‘Differential geometry’’, 1977; M. Spivak ‘‘Differential geometry’’, volume 2, 1979.

A.3

Equiaffine theory of plane curves

Following (Shirokov & Shirokov, 1959), we now consider the arc of the curve from the class C (4) with an additional condition: the arc does not contain rectification (inflection) points, i.e. its curvature is never zero c 6= 0. If the curve is represented by the equation r = r(t) then the value Zt1 p 3 σ= (r˙ × r¨) dt

(A.3.1)

t0

corresponding to the two points r(t0 ), r(t1 ) of the curve, does not depend on the choice of the parameter t, does not change under the equiaffine transformations of the curve and coincides with the equiaffine arc of the curve (see (A.2.7)). Taking the equiaffine arc-length for the parameter (natural parametrization), and denoting now the derivative w.r.t. the natural parameter with prime, we have the equality: (r 0 × r 00 ) = 1

(A.3.2)

for the counter-clockwise revolution along the curve. From this, (r 0 × r 000 ) = 0, i.e. vectors r 0 and r 000 are parallel and hence r 000 + kr 0 = 0

(A.3.3)


Appendix

A

171

multiplying the above equation by r 00 from the left and noting (A.3.2), we obtain the proportionality coefficient k for natural and arbitrary parametrization (Shirokov & Shirokov, 1959): k = (r 00 × r 000 ) = −(r 0 × r IV ) = (A.3.4) IV ... ... 3σ˙ 3 (r˙ × r ) + 12σ˙ 3 (¨ r × r ) − 5(r˙ × r )2 . 9σ˙ 8 The coefficient k is differential invariant of the curve - equiaffine curvature (see (A.2.8)). Let us define the tangent vector t = r 0 and the normal vector n = r 00 (they are tangent and normal in the equiaffine geometric sense, what is in general different from Cartesian case). Then (t × n) = 1 , (A.3.5) and we get formulas analogous to the Frenet formulas: ( t0 = n, n 0 = −kt .

(A.3.6)

Equation (A.3.3) together with equality (A.3.2) imply the main theorem of equiaffine theory of curves (Shirokov & Shirokov, 1959) Theorem A.3.1. The natural equation k = f (σ)

(A.3.7)

defines the plane curve up to an arbitrary equiaffine transformation. Proof. Identifying the curve according to the equation (A.3.7) leads to integration of the equation (A.3.3) under the condition that at initial point corresponding · ¸ to x is σ = σ0 , (r 0 × r 00 ) = 1 ((A.3.3) implies conservation of (r 0 × r 00 )). If r = y · ∗ ¸ x the solution of this equation, then for any other solution r ∗ = we will have y∗ · ¸ a r = A · r∗ + . Let us show now that A, a and b are constants and det(A) = 1. b ∗ ∗ Indeed, let us take A0 such that it transforms ¸ the frame {r˙ 0 , r¨0 } into the frame · a0 = r0 − A0 · r0∗ . We shall obtain from {r˙ 0 , r¨0 } (r˙ 0 = A0 · r˙ 0∗ , r¨0 = A0 · r¨0∗ ) and b0 the theorem about existence and uniqueness of solution of the differential equation (κ is “good” enough) that A = A0 , a = a0 , b = b0 . Then, 1 = (r˙ × r¨) = det(A) (r˙ ∗ × r¨∗ ) = det(A) .


A.3.1

Appendix

A

172

Curves of constant equiaffine curvature (Shirokov & Shirokov, 1959)

1. 0 < κ = c2 , equation (A.3.3) implies: r=

1 (a sin(cσ) + b cos(cσ)) + r0 , c

and 2c(a × b) = −1 by A.3.2. We obtained ellipse with the center at r0 ; all equiaffine normals of the ellipse pass through its center. 2. 0 > κ = −c2 , we get: r=

1 (a sinh(cσ) + b cosh(cσ)) + r0 , c

with the condition 2c(a × b) = −1; this is a hyperbola with the center at r0 ; all equiaffine normals pass through the center. 3. 0 = κ gives r = aσ 2 + bσ + c with the condition 2(a × b) = −1. This is a parabola with the diameters parallel to the vector a; equiaffine normals of the parabola coincide with its diameters.

A.4

Main formulae and the main theorem of curves in space (Shirokov & Shirokov, 1959)

Let the vector of a point on a spatial curve r be parameterized by a parameter t and be continuously differentiable up to order 6 inclusively: r = r(t) . Equiaffine transformations in space establish general affine correspondence between planes; therefore, the theory of planar curves is not different here from the theory of planar curves in the geometry of general affine group. We, thus, consider that ... (r˙ r¨ r ) 6= 0. By triple product we denote the volume defined by the three vectors in space. ... The triple product (r˙ r¨ r ) 6= 0 is invariant under the equiaffine transformations; therefore, the expression Z t ... 1 (A.4.1) σ(t) = |(r˙ r¨ r )| 6 dt t0


Appendix

A

173

will be a function of the point on a curve, will not depend on the choice of a parameter, and will not change under the equiaffine transformations of the curve. This function is called equiaffine arc of the curve. Parameterizing the curve by the equiaffine arc, we have: (r 0 r 00 r 000 ) = ±1 , (A.4.2) where prime denotes differentiation with respect to the equiaffine arc. One can change the orientation of the coordinate frame, so without limitation of generality, let us consider that (r 0 r 00 r 000 ) = 1 . This immediately implies that

¡

¢ r 0 r 00 r IV = 0 ,

and hence Noting (A.4.2), one finds that:

r IV + χr 00 + τ r 0 = 0 .

(A.4.3)

¡ ¢ χ = r 0 r 000 r IV τ = − (r 00 r 000 r 000 ) .

(A.4.4) (A.4.5)

The formulae (A.4.4) and (A.4.5) imply that χ and τ are invariants of a curve both with respect to the equiaffine transformation and with respect to the parameterization on the curve. The invariant χ is called equiaffine curvature, and τ — equiaffine torsion of the curve. We will relate the following three vectors to every point on a curve:  0    t = r − vector of e-tangent , (A.4.6) n = r 00 − vector of e-main normal ,    b = r 000 − vector of e-binormal . These vectors do not belong to the same plane because (t n b) = 1 .

(A.4.7)

They constitute the moving frame of the curve. Corresponding to the standard theory of curves in space, we will call the plane defined by the vectors t and b — e-rectifying plane of the curve, the plane defined by the vectors n and b — e-normal plane. The vectors t and n define an osculating plane. The definition of the vectors t, n, b and (A.4.3) imply that  0    t = n, (A.4.8) n0 = b ,    b0 = −τ t − χn .


Appendix

A

174

These equations are analogous to the formulae by Serre-Frenet. The coordinates of a curve can be decomposed with respect to the frame {t, n, b} into series with respect to σ as follows: τ 4 τ0 5 σ − σ + ... , 4! 5! χ χ0 + τ 5 − σ4 − σ + ... , 4 5! χ − σ5 + . . . . 5!

x = σ− σ2 2 σ3 z = 3!

y =

Similar to the standard theory of curves, the following theorem is true: a curve is defined up to an equiaffine transformation provided functions χ and τ as functions of σ.

Appendix B Properties of a parabolic path w.r.t. the minimum-jerk and the 2/3 power-law models It is shown that the motion along a parabola, that satisfies the 2/3 power law, will have zero cost in the jerk sense (thus being the solution of the constrained minimumjerk model). More concretely, the motion that costs 0 in the jerk sense is equivalent to the motion with constant equiaffine velocity along an arc of a parabola or to the motion along the straight line with constant acceleration.

Derivations We claim that [Zero jerk cost along trajectory] ⇔ [Tracing of a parabolic arc with constant equiaffine velocity or tracing a straight line when x and y are linear or quadratic functions of time]. 1. ⇐ We use the the following time-course parameterization of equiaffine arclength σ = αt + β , (B.0.1) α and β being constants, this is the parameterization that provides motion with constant equiaffine velocity. The equation of a parabola (2.2.13), up to an equiaffine transformation (2.2.2) is: x = σ, y =

σ2 , 2

(B.0.2)

thus the functions x = x(σ) and y = y(σ) are always polynomials of σ of degree not higher than 2 (if we apply to {x, y} the transformation (2.2.2) we 175


Appendix

B

176

get all possible parameterizations of parabolas versus equiaffine arc-length, the parameterizations stay 2-nd order polynomials of σ). Substituting (B.0.1) into (B.0.2), we obtain that the time parameterizations of components x = x(t) and y = y(t) are the polynomials of at most second degree too, from which it follows 1 J= 2

ZT

¡...2 ... ¢ x (t) + y 2 (t) dt = 0 .

0

For tracing a straight line, we have an equation for the path (without limitations of generality) y = αx + β and thus, linear of quadratic parameterization of x with time zero zero jerk. 2. ⇒ Let us consider that the jerk-cost is zero along the trajectory for some parameterization. Obviously, x and y are parameterized, in such a case, as second order polynomials of time: ZT ... ¢ 1 ¡...2 x (t) + y 2 (t) dt = 0 J= 2 0 # " # " a2 t2 + a1 t + a0 x(t) = . ⇒ y(t) b2 t2 + b1 t + b0

(B.0.3)

For this rule of motion we immediately obtain that equiaffine velocity is conserved: p 3

(r˙ × r¨) v Ã ! u u 2a t + a 2a 2 1 2 3 = t det 2b2 t + b1 2b2 v ! Ã u u a a 1 2 3 = const . 2 det = t b1 b2

σ˙ (t) =

A chain of elementary transformation shows that the traced path is an arc of a parabola or a straight line. We show calculations for the case when a2 6= 0. Other cases can be trivially completed. After a shift of coordinates " x˜ = x − a0 y˜ = y − b0


Appendix √

in (B.0.3) we have: t =

±

a21 +4a2 x ˜ 2a2

B

177

and so

³b ´ x˜ − a1 t b2 2 + b1 t = x˜ − · a1 − b1 t , a2 a2 a2 p ³ ´ a21 + 4a2 x˜ b2 b2 y˜ − x˜ = ∓ · a1 − b1 , a2 a2 2a2 ³b ´2 2 4(a2 y˜ − b2 x˜)2 = · a1 − b1 (a21 + 4a2 x˜) . a2

y˜ = b2

Which is an equation of parabola or straight line.

¥

It might be useful to mention the fact that the equiaffine velocity of drawing a straight line is zero, i.e. the straight line has zero equiaffine arc-length and therefore straight lines in equiaffine geometry are analogous to points in Euclidean geometry.

Appendix C Paths satisfying simultaneously the minimum jerk and the 2/3 power-law models C.1

Derivation of the necessary condition for the path

Here we attempt to solve the following problem: “... the constrained minimum-jerk model and the [2/3]{my insertion - F.P.} power law predict similar speed profiles for a given path. There seems to exist a family of paths for which the two are exactly equivalent, and identifying that family may provide further insights ...” (Todorov et al., 1998 (Todorov & Jordan, 1998)). We remind the reader that the motion according to the two-thirds power law along a movement piece at which the gain factor K from (2.2.15) is constant, is equivalent to the motion with a constant equiaffine velocity (σ˙ = const). Let us consider the convex path ξ with equiaffine length Σξ given by the vector function ξ(σ) = {x(σ), y(σ)}, σ ∈ [0, Σξ ] parameterized by the equiaffine arc-length. ZT ¤ ... 1 £... 2 x (σ(t)) + y 2 (σ(t)) dt be the jerk-cost along the path with Let Jσ (ξ) = 2 0

a trajectory defined by the functions: ξ(σ) and strictly increasing function1 σ(t) such that σ(0) = 0, σ(T ) = Σξ . In general, the costs will be different for different trajectories having the same paths (the functions σ(t) are different). 1

For strictly increasing function σ(t), σ(t2 ) > σ(t1 ) whenever t2 > t1 , i.e. the graph of the function is always increasing. For example, for Σξ = 1 and T = 1 the function σ(t) could be σ = t or σ = t2 .

178


Appendix

C

179

Let σ ∗ (t) be the solution of the minimization problem σξ∗ (t) = arg min Jσ (ξ). σ(t)

(C.1.1)

Our goal is to find such paths {ξ} for which the corresponding solutions of the minimization problem σξ∗ (t) are linear functions of time (their derivatives which are equiaffine velocities are constant, equivalently, the motions satisfy the 2/3 power-law model): A = {ξ : σ˙ ξ∗ (t) = const}. (C.1.2) For the function σ(t) ∈ [0, Σξ ] and its inverse function t = τ (σ) ∈ [0, T ] we denote ¯ ¯ d va = va (σ) = σ(t)¯¯ dt t=τ (σ) ¯ 2 ¯ d d wa = wa (σ) = 2 σ(t)¯¯ = va va = va 0 va dt dσ t=τ (σ) ¯ ¯ d3 d ja = ja (σ) = 3 σ(t)¯¯ = va wa = dt dσ t=τ (σ) 2

va 00 va 2 + va 0 va where prime means differentiation w.r.t. σ (we used here the following property: d d f (σ(t)) = σ˙ dσ f (σ) = va f 0 ). dt So, finally, 1 Jσ (ξ) = 2

ZT

... 1 ... ( x 2 + y 2 )dt = 2

0

ZΣξ 0

00 2

00 2

2

2

2

9(x + y )wa va + (x0 + y 0 )ja 2 + 6(x000 x00 + y 000 y 00 )va 4 wa + ¤ 2(x000 x0 + y 000 y 0 )va 3 ja + 6(x00 x0 + y 00 y 0 )va wa ja dσ = 1 2

2

1 £ 000 2 2 (x + y 000 )va 6 + va

ZΣξ I(x0 , x00 , x000 ; y 0 , y 00 , y 000 ; va , va0 , va00 ) dσ .

(C.1.3)

0

Given the path ξ we intend to solve the problem (C.1.1) (find the ¯ optimal trajec∗ ∗ ∗ tory), that is to find σξ (t) or, equivalently, to find va (σ) = σ˙ ξ (t)¯t=τ (σ∗ ) . Note that the feasible equiaffine velocity is constrained: ZΣξ 0

1 dσ = T . va

(C.1.4)


Appendix

C

180

The Euler-Poisson (E-P) equation for the recently obtained functional will be: ¶ ¶ µ µ µ ¶ ∂I d2 ∂ d ∂I ∂I 1 E-P(I) = + 2 +λ − = 0 00 ∂va dσ ∂va dσ ∂va ∂va va = va000 · (. . .) + va00 · (. . .) + va0 · (. . .) + 2 2 va4 · (x000 + y 000 − 2x00 x(4) − 2y 00 y (4) + λ 2x0 x(5) + 2y 0 y (5) ) − 2 = 0 . va The desirable va for our optimal solution is constant, according to (C.1.2). We then " # x(σ) have for the path ξ(σ) = belonging to A that x and y satisfy y(σ) 2

2

x000 + y 000 − 2x00 x(4) − 2y 00 y (4) + 2x0 x(5) + 2y 0 y (5) = const

(C.1.5)

or, assuming that the path is 6 times differentiable, µ = x0 x(6) + y 0 y (6) = 0. The derivations of the necessary condition are identical for the 3D case; therefore, we write the necessary condition in the vectorial form, which can be used to deduce both planar and spatial curves. µ = r 0 (σ) · r (6) (σ) = 0 . (C.1.6) This necessary condition may have many planar solutions. One set of solutions is immediately recognized: this is the set of quadratic functions of σ: ( x = a2 σ 2 + a1 σ + a0 . (C.1.7) y = b2 σ 2 + b1 σ + b0 The only curves that can be described by parameterization of this type are all parabolas. Indeed, whenever ξ is any piece of parabola, va∗ = const (see Methods of Chapter 2). It is important to mention here that the solution of (C.1.6) cannot represent a planar curve if it does not satisfy the following equality: (ξ 0 × ξ 00 ) = x0 y 00 − x00 y 0 = 1 ,

(C.1.8)

which is (A.3.2). This means that no corresponding curves exist for some choices of parameters a and b from (C.1.7). For the case of spatial curves, the equality is as follows: (ξ 0 ξ 00 ξ 000 ) = 1 , (C.1.9) where (ξ η ζ) denotes the volume spanned by the three vectors, or, equivalently, their mixed product. Below we show that no other curves satisfying (C.1.6) exist beside parabolas, under the natural


Appendix

C

181

Assumption C.1.1. Simultaneous solution of (C.1.6) and (C.1.8) (or of (C.1.6 and (C.1.9))) is invariant under the equiaffine transformations, that is, equiaffine transformations of the solution is solution as well. We consider here only the solutions of (C.1.6) which satisfy the assumption, i.e., the proof is not appropriate for the case when some equiaffine transformation of the solution is not a solution any more.

C.1.1

Planar curves, eliminating non-parabolic solutions

It is easy to show that the assumption implies that x and y parts of (C.1.6) are zeros: µx = x0 x(6) , µy = y 0 y (6) and µ x = µy = 0 . (C.1.10) To see this, one should just apply one-parametric equiaffine transformation of the form x → (1/α)x, y → αy which implies µx → (1/α2 )µx , µy → α2 µy destroying µ = µx + µy = 0 and so destroying the assumption, if µx and µy are not both zeros. The following chain of reasonings proves uniqueness of parabolic solutions under the assumption. d From (C.1.10) 0 = µx = 2x0 x(V I) hence x(V I) = 0 implying that x(σ) is 5-th dσ order polynomial, the same for y(σ), so we can write: x(σ) = a0 + a1 σ + a2 σ 2 + a3 σ 3 + a4 σ 4 + a5 σ 5 , y(σ) = b0 + b1 σ + b2 σ 2 + b3 σ 3 + b4 σ 4 + b5 σ 5 . Substituting x and y into (C.1.8), we obtain polynomial in σ of order 6: −2a2 b1 {12(a1 b4 {16(a2 b4 {12(a3 b4 30(a3 b5

+ − − − −

2a1 b2 + 6(a1 b3 − a3 b1 )σ + a4 b1 ) + 6(a2 b3 − a3 b2 )}σ 2 + a4 b2 ) + 20(a1 b5 − a5 b1 )}σ 3 + a4 b3 ) + 30(a2 b5 − a5 b2 )}σ 4 + a5 b3 )σ 5 + 20(a4 b5 − a5 b4 )σ 6 = 1 ,

which implies 7 conditions on its coefficients to satisfy the equality, coefficients near non-zero degrees of σ have to be 0 and the free term to be 1. Let us use the following notation: 0 = [a0 b0 ], 1 = [a1 b1 ], . . . , 5 = [a5 b5 ]. We will show that (C.1.6) has no other solutions beside parabolas under the assumption if we show that 3 = 4 = 5 = [0 0] . (C.1.11)


Appendix

C

182

To show this, let us write the conditions on each of the coefficients of the equiaffine velocity, calculated above. In the left hand side we write the degree of σ to which the coefficient corresponds. σ0 σ1 σ2 σ3 σ4 σ5 σ6

: : : : : : :

2 (1 × 2) = 1 6 (1 × 3) = 0 12 (1 × 4) + 6 (2 × 3) = 0 16 (2 × 4) + 20 (1 × 5) = 0 12 (3 × 4) + 30 (2 × 5) = 0 30 (3 × 5) = 0 20 (4 × 5) = 0 ,

(C.1.12) (C.1.13) (C.1.14) (C.1.15) (C.1.16) (C.1.17) (C.1.18)

the definition of the cross product was introduced in (2.2.1). Let us now consider all the possibilities when (C.1.11) is not satisfied. • 3 6= [0 0], 4 6= [0 0], 5 6= [0 0], from (C.1.13), (C.1.17) and (C.1.18) immediately imply that the vectors 1, 3, 4, 5 are parallel (corresponding determinants are zero; we can use here transitivity rule - 3 is parallel to 1 and is parallel to 5, then 1 is parallel to 5, because all considered vectors are non-zero). Then (C.1.15) or (C.1.16) imply that 2 is also parallel to them, which contradicts (C.1.12). • 3 6= [0 0], 4 6= [0 0], 5 = [0 0], from (C.1.13 - C.1.16) we get that all the vectors are parallel, contradicting (C.1.12). Similar proof for the cases when only 3 or only 4 is zero. • 3 6= [0 0], 4 = [0 0], 5 = [0 0], (C.1.13), (C.1.14) imply again, that all the vectors are parallel, again, contradicting (C.1.12), similar proof for the cases when only 4 6= [0 0] or only 5 6= [0 0]. ¥

C.1.2

Spatial curves, eliminating non-screw-parabolic solutions

In case of a 3-dimensional path, it is again easy to show that the assumption implies that x, y, z parts of (C.1.6) are zeros: µx = x0 x(6) , µy = y 0 y (6) , µz = z 0 z (6) , and therefore µx = µ y = µz = 0


Appendix

C

183

implying x(σ3 ) = a0 + a1 σ3 + a2 σ3 2 + a3 σ3 3 + a4 σ3 4 + a5 σ3 5 y(σ3 ) = b0 + b1 σ3 + b2 σ3 2 + b3 σ3 3 + b4 σ3 4 + b5 σ3 5 z(σ3 ) = c0 + c1 σ3 + c2 σ3 2 + c3 σ3 3 + c4 σ3 4 + c5 σ3 5 . Let us consider the following six 3-dimensional vectors: 0 = [a0 b0 c0 ], 1 = [a1 b1 c1 ], . . ., 5 = [a5 b5 c5 ]. Remembering that by the mixed product (ξ η, ζ) we denote the volume spanned by three 3-dimensional vectors and that we demand (C.1.9), we obtain for x, y, z: 12(1 2 3) + 48(1 2 4)σ3 + [72(1 3 4) + 120(1 2 5)] σ3 2 + [48(2 3 4) + 120(1 3 5)] σ3 3 + [180(2 3 5) + 240(1 4 5)] σ3 4 + 240(2 4 5)σ3 5 + 120(3 4 5)σ3 6 = 1 . This implies immediately that the vectors 3, 4, 5 belong to the same plane, vectors 2, 4, 5 belong to the same plane, and vectors 1, 2, 4 belong to the same plane (in any pair of planes formed based on these 3 planes, the planes do not necessarily coincide); and that vectors 1, 2, 3 do not belong to the same plane, which also means that non of them is zero. Now, we consider several cases: • Vectors 4 and 5 are not parallel. This implies that the vectors 2 and 3 belong to the plane defined by the vectors 4 and 5, from what it follows that 1 belongs to this plane too. There is a contradiction with the equality: the vectors 1, 2, 3 have to span a non-zero volume. • Vectors 4 and 5 are parallel, at least one of them is non-zero. Zero coefficients near all non-zero degrees of σ3 are impossible then. • Vectors 4 and 5 are both zeros. We are left with the curves whose components are described by 3-d order polynomials satisfying (1 2 3) = 1/12. Spatial equiaffine transformations of the curve    x = σ3  y =

   z =

σ3 2 2 σ3 3 6

represent all such equiaffinely invariant solutions. ¥


C.2

Appendix

C

184

A sufficient condition o the path

For which paths, beside parabolic (and circular), does the prediction of the constrained minimum-jerk model satisfy the 2/3 power-law? The formula for the second variation can be useful when certain solutions of the necessary condition (12) are obtained and need to be tested: is it a minimum of the functional or is it just an extremal solution? Here we derive a sufficient condition for the minima of the functional (C.1.3), which can be used in order to show that a candidate path indeed complies with both models. Note that if a candidate path does not satisfy the sufficient condition, this does not imply that the path does not provide minimum to the cost. Sufficient conditions for the minimum of the functional containing only the first derivative of the unknown function, based on the second variation, can be found elsewhere (Gelfand & Fomin, 1961). Here, the second variation is derived for the functional that contains second derivative of the unknown function and is used in order to obtain a sufficient condition for the minimum of the functional in the framework of finding paths that comply with both models. Note that if a candidate path does not satisfy the sufficient condition, this does not imply that the path does not provide minimum of the cost. The derivations are shown for the planar case, because derivations for the spatial case are identical. Let h(σ) be a perturbation of va . The function h and its derivatives up to a certain order are zeros at σ = 0 and at σ = Σξ (have zero boundary conditions). The constraint stating conservation of duration (C.1.4) has to be satisfied for the function va (σ) + h(σ). The formula for the second variation of the functional (C.1.3) is as follows: 1 δ 2 I[h] = 2

ZΣξ ³ 2 Iva va h2 + 2Iva va0 hh0 + Iva0 va0 h0 + 2Iva va00 hh00 + 2Iva0 va00 h0 h00 + 0

I

va00

va00

00 2

h

´ dσ ,

(C.2.1)

where Iva = ∂v∂a I. Using integration by parts and taking into account zero boundary conditions on h, ZΣξ 0

¡

¢ 0

ZΣξ µ

2Iva va0 hh dσ = − 0

¶ d Iva va0 h2 dσ , dσ


ZΣξ

¡

2Iva va00 hh

¢ 00

Appendix

ZΣξ

185

¢ d ¡ Iva va00 h h0 dσ = −2 dσ

dσ = −2

0

C

0

ZΣξ ·

¢ d ¡ Iva va00 hh0 + dσ

0

Iva va00 h

02

ZΣξ ·

i dσ =

¸ ¢ 2 d2 ¡ 02 Iva va00 h − 2Iva va00 h dσ , dσ 2

0

ZΣξ

¡

0

2Iva0 va00 h h

¢ 00

ZΣξ µ dσ = −

0

¶ d 2 Iv0 v00 h0 dσ . dσ a a

0

Finally, the second variation can be written in the form of the sum of squares of h and its derivatives multiplied by their corresponding coefficients that do not depend on h: 1 δ 2 J[h] = 2

¶ ¸ ZΣξ ·µ ¡ ¢ 02 d2 d 2 00 2 Iv v0 + Iv v00 h + Iva0 va0 − 2Iva va00 h + Iva00 va00 h Iva va − dσ . dσ a a dσ 2 a a 0

By setting the derivatives of va to zero, we get: ZΣξ n h i 2 2 δ 2 J[h] = 2 x000 + y 000 − 2x00 x(4) − 2y 00 y (4) + 2x0 x(5) + 2y 0 y (5) h2 + h0 ³ ´ i 2 00 2 00 2 0 000 0 000 00 000 00 000 9 x +y + 2 (x x + y y ) − 24 (x x + y y ) h0 + h i o 2 2 2 x0 + y 0 h00 va 3 dσ (C.2.2) The coefficient near h2 is exactly the left hand side of the necessary condition (C.1.6) and is, therefore, constant (say, c0 ). The coefficient near h0 2 is some continuous function, let c1 be its minimum on the interval [0 Σξ ]. The coefficient near h00 2 is always positive (we assume that the curve does not contain inflection points), and let c2 > 0 be its minimum on the interval [0 Σξ ]. Let us show now that the second variation is strictly positive. First we proof a particular case of Poincaré inequality. Consider some smooth enough function u(σ) defined on the interval σ ∈ [a; b], and such that u(a) = 0. This function has the following properties. Z 2

σ

2

u (σ) = u (a)+ a

¡

¢0 u (t) dt = 2

Z

2

σ

µZ

b

0

u(t)u (t)dt ≤ 2 a

¶1/2 µZ

b

2

u (t)dt a

¶1/2 02

u (t)dt a

.


Appendix

C

186

Which implies that Z

b

µZ

b

2

u (t)dt ≤ 2(b − a) a

Finally,

¶1/2 µZ b ¶1/2 02 u (t)dt u (t)dt . 2

a

µZ

a

¶1/2

b

2

u (t)dt

µZ

b

≤ 2(b − a)

a

¶1/2 u (t)dt . 02

(C.2.3)

a

Now, applying (C.2.3), we get: Z 2

δ J[h] ≥ va va

Σξ

3

³

2

02

c0 h + c1 h + c2 h Z

0

Z

0

3

va 3

00 2

´ dσ ≥

Σξ

¡

Σξ

¡ ¢ h2 c0 + c1 /(4Σξ 2 ) + c2 /(16Σξ 4 ) dσ ,

¢ c0 h2 + c1 h2 /(2Σξ )2 + c2 h2 /(2Σξ )4 dσ =

0

which is positive for small enough Σξ because c2 is positive. Our derivations imply that the condition (C.1.6) is both necessary and sufficient on short enough pieces of any of the solutions of the necessary condition. ¥

Appendix D Derivation of the parabola passing through three points and having prescribed tangent direction at one of the points Given three points in the plane and a tangent direction at one of the points, we derive the equation of parabola, which satisfies these constraints. In some degenerate cases a unique solution is possible. There is no solution for certain configurations of the input. In other cases there are two possible parabolas. The current analysis uses the methods from (Handzel & Flash, 2001) and (McLeod & Baart, 1998).

Let us have the three points A, B, C with the tangent direction t at C. Point C lyes on the curve between the points A and B. We consider arbitrary configurations of the input. Thus we will mark the configurations according to the classes of number of parabolas (0, 1 or 2) satisfying the configuration. 1. No parabolas satisfy the configuration. (a) A, B and C belong to the same line. (b) A, B and C do not belong to the same line, see figure D.1. The line Ct with tangent direction t at C separates A and B, that is A and B belong to different half-planes defined by Ct . No short arc 1 , including parabolic 1

Following (Handzel & Flash, 2001) A locally convex arc Γ(p1 , p2 ), together with its end-points p1 and p2 , is called a short arc if not two tangent lines to it are parallel. The equivalent statement in Euclidean geometry is that the total turning angle of the tangent to the curve from p1 to p2 is less than π.

187


Appendix

D

188

piece, can have a tangent, which intersects it. In our case the intersection will take place either between A and C or between B and C. (a)

(b)

C

C

t

B

t

A

B

A

Figure D.1: Impossible case. (a) No parabola can satisfy such initial configuration. The line Ct with tangent direction t at C separates A and B, that is A and B belong to different half-planes defined by Ct . No short arc, including a parabolic piece, can have a tangent, which intersects it. (b) In our case the intersection would take place either between A and C or between B and C.

2. There is (are) parabola(s), which satisfy the configuration. For this we need Ct not to pass between or through A and B. Noting that A, B and C do not belong to the same line, we can introduce an −→ −−→ affine coordinate system with BA being a unit of x-axis, BC being a unit of yaxis. Let the original coordinates of the three points in the Cartesian coordinate system be (Ax0 , Ay0 ), (Bx0 , By0 ), (Cx0 , Cy0 ) . We apply the transformation T = T2 T1

(D.0.1)

T1 ([x y]T ) = [x − Bx0 y − By0 ]T ,

(D.0.2)

to the three points, with

T2 ([x y]T ) = [T1 (A) T1 (C)]−1 · [x y]T = !−1 Ã Ax0 − Bx0 Cx0 − Bx0 Ay0 − By0

Cy0 − By0

· [x y]T .


Appendix

D

189

Affine transformation, corresponding to T maps our input as follows:   T (A) = [1 0]T      T (B) = [0 0]T  T (C) = [0 1]T      T (t) := τ , 2

(D.0.3)

where we introduced new notation τ for the image of the tangent. One can see that this transformation indeed puts point B to the origin of the new coordinate system, and the points A and C to the unit on the coordinate axes. Inverse transformation to T is T −1 = T1 −1 T2 −1 , where

Ã −1

T

T2 ([x y] ) =

Ax0 − Bx0 Cx0 − Bx0 Ay0 − By0

! · [x y]T ,

Cy0 − By0

T1 −1 ([x y]T ) = [x + Bx0 y + By0 ]T .

−→ t is parallel to BA, see figure D.2 (a). In this case we have a single parabola. Let us prove it and write equation of the parabola. (a)

(b) 6

↑y

5

4

C

H

3

t 2

1

G →x

0

−1

−2

F −3

B

A

−9

−8

−7

−6

−5

−4

−3

−2

−1

0

1

2

Figure D.2: (a) t is parallel to BA. This corresponds to single parabola. (b) If tangent is parallel to the chord then, in canonical coordinate system, y coordinate of the point of touch (G) is average of the y-coordinates of the chord (F H). Let us first show that


Appendix

D

190

Observation D.0.1. If, for three points F , G, H on the parabola, the chord −−→ F H is parallel to the tangent to the parabola at the intermediate point G, then the equiaffine arc along the parabola from F to G equals to the equiaffine arc along the parabola from G to H. Proof. Let parabola be represented in the canonical form, see figure D.2(b), y 2 = −2px . Then, the chord is parallel to the vector [xH − xF yH − yF ] = [−yH 2 /(2p) + yF 2 /(2p) yH − yF ]. The slope of the chord as for the relationship x = x(y) will be (−(yH + yF )) /(2p). Let us find a point on the parabola, at which tangent is parallel to this direction, that is it has the slope −(yH + yF )/(2p). We have for the parabola x = −y 2 /(2p) , x0y = −y/p . Hence, equating −(yH + yF )/(2p) = −y/p, we get that yG = (yH + yF )/2. Noting that for a parabola in its canonical coordinate system, equiaffine arc√ length changes linearly with coordinate y: σ(y) = σ0 + (1/ 3 p)y, and using that yG is average of yF and yH , we prove the observation. Every affine transformation equals equiaffine transformation with certain nonzero factor for area, affine transformations preserve parallel curves parallel. Proportions of the equiaffine lengths before and after affine transformation are the same, which follows directly from the formula for calculation of the equiaffine arc-length. Therefore, equiaffine lengths before C and after C are equal: σBC = σCA , see figure D.2(a) because tangent at C is parallel to the −→ chord BA, both before and after an affine transformation. Let us parameterize our parabola as follows: ( x = a0 + a1 t + a2 t2 . y = b0 + b1 t + b2 t2 Calculating equiaffine speed σ(t) ˙ for such parameterization from the formula ¯1/3 ¯ ¯ x˙ x¨ ¯ p ¯ ¯ ˙ = 3 2(a1 b2 − a2 b1 ) = const. Therefore equiaffine σ(t) ˙ =¯ ¯ , we get: σ(t) ¯ y˙ y¨ ¯ arc-length changes linearly with our parameter t, which implies that if xC = x(t1 ), yC = y(t1 ) ,

(D.0.4)


Appendix

D

191

then xA = x(2t1 ), yA = y(2t1 ) .

(D.0.5)

We can define t up to a scaling factor and shift, before defining the coefficients a and b. Let us take x(0) = 0, y(0) = 0, from which a0 = 0, b0 = 0; and take t1 = 1. We get a system of linear equations, corresponding to (D.0.4):   a1 + a2 = 0      b +b =1 1

2

 2a1 + 4a2 = 1      2b + 4b = 0 . 1 2 This results in the solution: a0 = 0, a1 = −0.5, a2 = 0.5; b0 = 0, b1 = 2, b2 = −1. Therefore x(t) = −0.5t + 0.5t2 , y(t) = 2t − t2 and thus t = y + 2x. Substituting t into the formula for y, we get: y = 2(y + 2x) − (y + 2x)2 . Finally, the equation of parabola in the new coordinate system is 4x2 + 4xy + y 2 − y − 4x = 0 .

(D.0.6)

We got only one equation, which means that there is unique parabola satisfying such input. 3. Two possible parabolas. In case tangent is not parallel to the chord and does not pass between the points A and B, there exists one more point D of the parabola on the line passing through C parallel to the chord. We depicted exemplar case in figure D.3(a). For every such point D, the 4 points A, B, C, D form trapezoid. Therefore, there exist unique parabola containing these 4 points (McLeod & Baart, 1998). We reformulate the problem, assuming that the conditions on tangent (the tangent is not parallel to the chord and does not pass between the points A and B) are satisfied, as follows: find the point D on the line passing through C parallel to the chord, such that corresponding parabola (D would define unique parabola) would have tangent at C as in the input. D has coordinates (Dx , 1) in the new coordinate system. Let us denote by l1 = y = 0, l2 = x − y(Dx − 1) − 1 = 0, l3 = y − 1 = 0, l4 = x = 0 linear equations of lines defined by AB, AC, CD, DB respectively. The equation of pencil of conics passing through these 4 points will be L = λl1 l3 + µl2 l4 = 0 .


Appendix

D

192

(a)

(b) reconstruct_parab_tang_3points.m 1.4

1.2

1

0.8

D

C 0.6

t

0.4

0.2

0

−0.2

B

A −0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

Figure D.3: (a) In case tangent is not parallel to the chord and does not pass between the points A and B, there exists one more point D of the parabola on the line passing through C parallel to the chord. (b) Exemplar drawing of two parabolas for the same input.

Thus, L = λy(y − 1) + µx(x − y(Dx − 1) − 1) = 0 . µ2 (D

(D.0.7)

−1)2

x The condition of this conic to be parabola is − λµ = 0. The case µ = 0 4 relates to two parallel lines, which is not our parabola, therefore µ 6= 0 and µ λ = (Dx − 1)2 . 4

Substituting λ into (D.0.7), we get: (−1 + Dx )2 (−1 + y) y µ + x (−1 + x + y − Dx y) µ = 0 , 4 or, cancelling µ L=

L=

(−1 + Dx )2 (−1 + y) y + x (−1 + x + y − Dx y) = 0 . 4 " ∇L =

#

2x + y − Dx y − 1 x (1 − Dx ) +

(−1+Dx )2 (−1+y) 4

and

" ∇L|C=(0, 1) =

−Dx (1−Dx )2 4

+

# .

(−1+Dx )2 y 4

(D.0.8)


Appendix

D

193

y-component of the tangent is non-zero in the new coordinate system. Therefore, we can write the direction of the tangent in the form τ = [τx − 1]T , after appropriate scaling. Gradient is orthogonal to the tangent, therefore we get an equation w.r.t. Dx : · ¸ (1 − Dx )2 (1 − Dx )2 T T ∇L|C · τ = −Dx · [τx − 1] = −τx Dx − = 0. 4 4 This implies that

p Dx = −2τx + 1 ± 2 τx (τx − 1) .

Unrealistic values 0 < τx < 1 correspond exactly to the cases when the tangent line would be between A and B. τx = 0 or τx = 1 correspond to the degenerate case of two parallel lines. p Note that | − 2τx + 1| > 2 τx (τx − 1). Therefore, the sign of both values of Dx is dictated by the sign of −2τx + 1, that is both options of D correspond either to positive x or to negative x in the new coordinate system. This is reasonable because they can be on one side of the tangent only, which crosses the OY axis. Let us substitute Dx into (D.0.8) to get the equation of parabolas in the new 2 x) = −τx Dx , we get: coordinate system. Noting that (1−D 4 L = −τx Dx (y − 1)y + x(−1 + x + y − Dx y) = Dx y [−τx (y − 1) − 1] + x2 + xy and finally h i p L = y −2τx + 1 ± 2 τx (τx − 1) [−τx (y − 1) − 1] + x2 + xy = 0 .

(D.0.9)

We remind that τx is the x component of the tangent direction in the new coordinate system, scaled so that τy = −1. Equation (D.0.9) can be rewritten in parametric form, setting x to be a parameter:  t2 −t  x = t − 1 (1−D x )−1 2 . 2 −t t  y = 1 1 2 (1−Dx ) − (1−Dx ) 4

2

Exemplar drawing of two parabolas for the same input is shown in figure D.3(b).

Summary


Appendix

D

194

We started our construction of parabolas by affine change of coordinates, which is given in (D.0.1) and showed that for the case of tangent parallel to the chord, there exists unique parabola with equation (D.0.6) in the transformed coordinate system. For the case of tangent not being parallel to the chord and not separating A and B, there are two parabolas with equation (D.0.9) in the new coordinate system, where τx is x component of the transformed (as in (D.0.3)) tangent direction and scaled such that τy = −1.

Appendix E Why the time of moving to the via point τ , in the minimum-jerk model with one via-point, is close to 1/2, which implies isochronous predictions It was observed that in human point-to-point movements with one via-point, the time of passing through the via-point is close to the half-time of the entire movement ((Flash & Hogan, 1985)). The minimum-jerk model (Flash & Hogan, 1985) provides close prediction of this phenomena. Here we analyze why predictions of the minimum-jerk model with one via-point and zero velocity and acceleration at the end-points provide the time of passing through the via-point τ close to 1/2 of the whole movement duration, for a wide range of locations of the via-point. We show why, for the via-point close to the initial point, the predicted time τ is not close to 1/2. We also show that the time of passing through the point of the trajectory, at which tangent is parallel to the chord (OX axis) is always between τ and 1/2, that is closer to 1/2 than τ .

E.1

Relationship between the via-points and τ

Let us first formulate the problem.

195


Appendix

E

196

Find {x(t), y(t), τ }, such that 1 {x, y, τ } = arg min x,y,τ 2

ZT

1 ... ... ( x 2 + y 2 )dt = arg min x,y,τ 2

·Zτ

0

under constraints

... ... ( x − 2 + y − 2 )dt+

0

  x|t=0 = x0 , x| ˙ t=0 = x˙ 0 , x¨|t=0 = x¨0 ;      y|t=0 = y0 , y| ˙ t=0 = y˙ 0 , y¨|t=0 = y¨0 ;      x| t=τ = xv , y|t=τ = yv ;  x|t=T = xf , x| ˙ t=T = x˙ f , x¨|t=T = x¨f ;       ˙ t=T = y˙ f , y¨|t=T = y¨f ;   y|t=T = yf , y|   τ ∈ (0, T ).

ZT

¸ ... 2 ... 2 ( x + + y + )dt

τ

(E.1.1)

This problem was solved by means of the optimal control methods in (Flash & Hogan, 1985) and can be reduced to the system of 25 algebraic equations, or, when τ is known, to the system of 24 linear equations (Polyakov, 2001). We assume here that velocity and acceleration are zero on the boundary. Thus, the predicted trajectory is scalable w.r.t. duration and metric units. We, therefore, use fixed end-points locations: the origin (0, 0) and the point (0, 1) as the start and end-points respectively, and fixed duration: 1 units. Location of the via-point is arbitrary. All the rest configurations (positions of the points and duration) can be obtained from our set of configurations by scaling time, units, rigid translation and rotation. Our constraints (in this case duration equals 1, T = 1) imply constraint on the time of passing through the via-point τ : 0≤τ ≤1. In case the velocity and acceleration on the boundaries are zero, it can be directly shown that τ is solution of the following equation: 6 τ 9 − 27 τ 8 + 40 τ 7 + 4 τ 6 (−5 + 3 xv ) − 36 τ 5 xv + 34 τ 4 xv − 8 τ 3 xv − ¢ ¡ (E.1.2) 2 τ xv 2 + yv 2 + xv 2 + yv 2 = 0 . Our simulations always resulted in a single root of (E.1.2) within the prescribed range of values τ ∈ [0, 1], and thus in unique solution of the minimization problem. A predicted path for our configuration with via-point at (−1.45, 1.005) is depicted by blue in figure E.1. Left-hand side of (E.1.2) is the numerator of the derivative of the cost as function of the input parameters (τ, xv , yv ) w.r.t. τ . A typical form of the cost function, as a


Appendix

E

197

xvm01.45yvp01.005 2

1.5

1

0.5

0

−0.5

−1.5

−1

−0.5

0

0.5

Figure E.1: Blue: prediction of the minimum-jerk model with via-point at (−1.45, 1.005), rest parameters correspond to the set of configurations limited by us. Green dashed line depicts the best fitting parabola for this path. function of τ , for fixed other parameters, is depicted in figure E.2. As it can be seen from the figure, the cost is a nice convex function with single minima. Let us rewrite (E.1.2): xv 2 (1 − 2τ ) + xv (−8 τ 3 + 34 τ 4 − 36 τ 5 + 12τ 6 ) + yv 2 (1 − 2τ ) + (−20τ 6 + 40 τ 7 − 27 τ 8 + 6 τ 9 ) = 0 , which is equivalent, after algebraic manipulations, to µ ¶2 −6τ 6 + 18τ 5 − 17τ 4 + 4τ 3 36(−1 + τ )6 τ 6 xv − , + yv 2 = 1 − 2τ (1 − 2τ )2 or (xv − X(τ ))2 + yv 2 = R(τ )2 ,

(E.1.3)


Appendix

E

198

Figure E.2: A typical form of the cost function, as a function of τ , for fixed other parameters. As it can be seen from the figure, the cost is a nice convex function with single minima.

with

¯ ¯ ¯ 6(−1 + τ )3 τ 3 ¯ −6τ 6 + 18τ 5 − 17τ 4 + 4τ 3 ¯ ¯. X(τ ) = , R(τ ) = ¯ 1 − 2τ (1 − 2τ ) ¯

Dependence (E.1.3) defines locations of the via-points for a given value of τ . Each τ , therefore, corresponds to a circle of via-points with center at (X(τ ), 0) and radius R(τ ). Now we will perform analysis of the dependencies between the location of the via-point (xv , yv ) and the time τ . 1. τ = 1/2. (E.1.2), implies then xv = 1/2. 2. 0 ≤ τ ≤ 1/2. For every τ , its circle of via-points, defined by X(τ ), R(τ ), provides us with the via-points having minimal and maximal values of their x component, which equal X(τ ) − R(τ ) and X(τ ) + R(τ ) respectively. Values of X(τ ), R(τ ), their difference, their sum, and R(τ ) versus X(τ ) are depicted in figure E.3. Values, corresponding to τ ∈ [0, 0.45] are in the left column, and values, corresponding to τ ∈ [0, 0.498] are in the right column. In figure E.4 level lines of τ are depicted. The circles on the left correspond to 0 < τ < 0.5, and the circles on the right correspond to 0.5 < τ < 1. Blue color corresponds to τ = 0.35, τ = 0.65. The circles with largest radius correspond to τ = 0.45, τ = 0.55, that is every via-point outside of the two circles with largest radii corresponds to


Appendix

E

199

τ ∈ (0.45, 0.55). As it can be seen from figures E.3, E.4, small values of τ (say, 0 < τ < 0.35) correspond to the via-points, which are close to the initial point. The relationship via-points ↔ τ is symmetric w.r.t. τ = 0.5. Therefore we depicted only values corresponding to 0 < τ < 0.5 in figure E.3. The symmetry results from invariance of the cost to change of start-point to end-point.

E.2

Time of passing through the point, at which tangent is parallel to Ox axis is between τ and 0.5, isochrony is stronger for this point

Without limitations of generality, due to symmetry, we will consider the case when 0 < τ ≤ 0.5. y-component of velocity is zero (y˙ = 0), at the point with tangent parallel to Ox. In case τ = 0, the via-point coincides with the initial point, and solution is straight point-to-point trajectory with bell-shaped speed profile. The tangent will be parallel to Ox at every point for such trajectory. We assume in all further text that τ 6= 0. Knowing τ , the y-component of the trajectory is as follows:  3 2 2 2  t (t +τ t (−5+3 t)+τ (210−15 t+6 t )) yv , 0 ≤ t ≤ τ 6 (−1+τ ) τ 5 y(t) = 3  (−1+t) (10 t2 −5 τ t (1+3 t)+τ 2 (1+3 t+6 t2 )) yv , τ < t ≤ 1 . 6 (−1+τ )5 τ 2 Taking time derivative of y, one gets that for the first part the solution of the equation y(t) ˙ = 0 is √ √ √ √ 2 τ + 6 τ 2 − 2 τ −1 + 3 τ 2 τ + 6 τ 2 + 2 τ −1 + 3 τ t−, 1 = , t−, 2 = , 1 + 3 τ + 6 τ2 1 + 3 τ + 6 τ2 t−, 3 = t−, 4 = 0. For the second part it is √ √ √ √ 2 − τ + 2 2 − 7 τ + 8 τ2 − 3 τ3 2 − τ − 2 2 − 7 τ + 8 τ2 − 3 τ3 , t+, 2 = t+, 1 = 10 − 15 τ + 6 τ 2 10 − 15 τ + 6 τ 2 t+, 3 = t+, 4 = 1 . In case of the first polynomial, real roots take place for 1/3 ≤ τ ≤ 0.5. As one can see from the upper plot in figure E.5, solutions are greater than τ , and are not relevant for us because y-component of the optimal trajectory consists of the second polynomial for t > τ . In case of the second polynomial, we, conversely, need the solution to be greater than τ . As it can be seen from the middle plot of figure E.5,


Appendix

E

200

(a)

(b)

0.2

10 0

0.1

−10

X(τ)

X(τ)

0 −0.1

−20 −30

−0.2

−40

−0.3

−50

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

−0.4 0

0.05

0.1

0.15

0.2

τ

0.25

0.3

0.35

0.4

0.45 40

R(τ)

−0.5

1

R(τ)

0.8

30 20 10 0

0.6 0.4

τ

0.5

0

0.05

0.1

0.15

0.2

τ

0.25

0.3

0.35

0.4

0.4

X(τ) + R(τ)

0.2 0

τ

50

0.45

0.3 0.2 0.1

0.5

0

0

X(τ) − R(τ)

−0.5

−1

0

0.05

−20 −40 −60 −80

X(τ) + R(τ) X(τ) − R(τ) −1.5

τ

0

0.1

0.15

0.2

τ

0.25

0.3

0.35

0.4

−100

0.45

Here τ ∈ [0, 0.45]

τ

Here τ ∈ [0, 0.498]

0.9

50

0.8

45

40

0.7

35 0.6 30

R(τ)

R(τ)

0.5 25

0.4 20 0.3 15 0.2

10

0.1

0 −0.5

5

−0.4

−0.3

−0.2

−0.1

X(τ)

0

0.1

0.2

0 −50

−40

−30

−20

−10

0

10

X(τ)

Figure E.3: Values of X(τ ), R(τ ), their difference, their sum, and R(τ ) versus X(τ ) are depicted. Values, corresponding to τ ∈ [0, 0.45] are in the left column, and values, corresponding to τ ∈ [0, 0.498] are in the right column. For every τ , its circle of via-points, defined by X(τ ), R(τ ), provides us with the via-points having minimal and maximal values of their x component, which equal X(τ ) − R(τ ) and X(τ ) + R(τ ) respectively.


Appendix

E

201

Level lines of τ(x , y ) v

v

1.5

1

yv

0.5

0

−0.5

−1

−1.5 −1

−0.5

0

0.5

1

1.5

2

xv Figure E.4: Level lines of τ are depicted. The circles on the left correspond to 0 < τ < 0.5, and the circles on the right correspond to 0.5 < τ < 1. Blue color corresponds to τ = 0.35, τ = 0.65. The circles with largest radius correspond to τ = 0.45, τ = 0.55, that is every via-point outside of the two circles with largest radii corresponds to τ ∈ (0.45, 0.55). As it can be seen, from this figure and from figure E.3, small values of τ (say, 0 < τ < 0.35) correspond to the via-points, which are close to the initial point. The relationship via-points ↔ τ is symmetric w.r.t. τ = 0.5.


Appendix

E

202

only second solution satisfies this condition. Graph of the times of passing through the point at which tangent is parallel to Ox axis is depicted in the lower plot of figure E.5. As it can be seen, it is between 0.4 and 0.5, always between τ and 0.5, thus providing stronger isochrony for such points. We saw here that the time of passing through the point at which tangent is parallel to Ox axis belongs to the occurrence of the second polynomial for the case 0 < τ ≤ 0.5, it will belong to the time of the first polynomial for the symmetric case, when 0.5 ≤ τ < 1. One can raise a question: which via-points correspond to which values of t+, 2 . It appears, that τ can be expressed via t+, 2 : µ ¶ q ¡ ¢ 2 − 1 + (2 − 15 t+, 2 ) t+, 2 + − (−1 + t+, 2 ) (−1 + 3 t+, 2 (−2 + 5 t+, 2 )) τ= . 12 t+, 2 2 Its graph is depicted in figure E.6, which is actually inverse of the graph depicted by dashed in the lower plot of the figure E.5. Thus, there is one-to-one correspondence between τ and t+, 2 , which can be used to find corresponding locations of the via-points using the locations for τ .


Appendix

E

203

First polynomial (t ∈ [0, τ]) 0.35 t −τ −, 1 t −τ

0.3

−, 2

0.25 0.2 0.15 0.1 0.05 0 0.32

0.34

0.36

0.38

0.4

0.42

0.44

τ Second polynomial (t ∈ [τ, 1])

0.46

0.48

0.5

0.4 t+, 1 − τ t+, 2 − τ

0.3 0.2 0.1 0 −0.1 −0.2 −0.3

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

τ Second polynomial (t ∈ [τ, 1])

0.4

0.45

0.5

0.5

0.4

0.3

0.2

0.1

t+, 2 τ

0

0

0.05

0.1

0.15

0.2

0.25

τ

0.3

0.35

0.4

0.45

0.5

Figure E.5: Upper plot. Difference between the time of passing through the point at which tangent is parallel to Ox axis and τ for the first polynomial. The times are greater than τ , and are not relevant for us because y-component of the optimal trajectory consists of the second polynomial for t > τ . Middle plot. Difference between the time of passing through the point at which tangent is parallel to Ox axis and τ for the second polynomial. In case of the second polynomial, we, conversely, need the solution to be greater than τ . As it can be seen, only second solution satisfies this condition. Lower plot. Graph of the times of passing through the point at which tangent is parallel to Ox axis is depicted in the lower plot of figure E.5. As it can be seen, it is always between τ and 0.5, thus providing stronger isochrony for such points.


Appendix

E

204

0.5 0.4

τ1

0.3 0.2 0.1 0

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

t+, 2 Figure E.6: Time of passing through the via-point versus time of passing through the point, at which tangent is parallel to Ox axis. This is inverse of the graph depicted by dashed in the lower plot of the figure E.5.

Appendix F Numerical approaches to the constrained minimum-jerk model F.0.1

Optimization procedure

The goal of the optimization procedure is to find a speed profile that minimizes the cost (2.2.14) and is constrained by the path and total duration T of a movement segment. Approximation of the cost with finite differences Given a sequence of recorded samples of the movement segment path {xi , yi }, i = 1, . . . , N , the optimal intervals of travelling between adjacent samples ∆t = P time −1 {∆t1 , . . . , ∆tN −1 } ( N ∆t i = T ) were to be estimated (the sum of the increments 1 was constrained to be equal to the total duration). Time increments were used to estimate the time derivatives (first, second and third) of position coordinates and were predicted by the process of cost optimization. Minimization of the cost can be realized by means of the built-in “matlab” function “lsqnonlin” with the large-scale algorithm Coleman et al. (1999). The initial guess for the time increments was taken from the recorded trajectory (∆ti = 1/(Recording frequency), i = 1, . . . , N − 1). We did not specify the boundary velocity and acceleration (i.e. ∆t1 , ∆t2 , ∆tN −2 ,∆tN −1 were also subject to optimization). The reason is that the predicted trajectories are used for analysis of the drawn paths and thereafter we needed the path to be the only constraint. As an example, optimal trajectory for parabolic path satisfies the two-thirds power law (see Results). However, with certain boundary conditions imposed prior to the fit, the predicted trajectory will not satisfy the two-thirds power-law.

205


Appendix

F

206

Optimization procedure, based on decomposing the drawing speed into Fourier series The geometrical constraint (path) can be included into the cost functional for our minimization problem by introducing the curvature of the path into the cost functional, instead of introducing the path sample-wise Richardson & Flash (2002): 1 J= 2

ZL

¡ ¢2 i 1 h 2 00 2 (v v + vv 0 − c2 v 3 )2 + (v 3 c)0 ds. v

(F.0.1)

0

Prime here means derivative with respect to Euclidian arc-length S, L is the Euclidian length of the path, v(s) = s(t)| ˙ t=t(s) is the speed of drawing parameterized by the arc-length, and c(s) is its curvature. The overall duration T is fixed. The speed, v(s), which minimizes the above functional, together with the given path, defines the optimal trajectory. The speed is well defined on the segments of motion. So the goal now is to approximate the solution of the minimization problem w.r.t. v(s). In case of periodic path, v(s) can be approximated with first n terms of Fourier series and the optimization procedure can be applied to the Fourier coefficients Richardson & Flash (2002). We deal with arbitrary, not periodic paths for which the tangential velocity is not periodic function and so Fourier series approximation cannot be accurate especially near the boundaries. Rate of convergence of the series to a periodic function is defined by the rate of smoothness of the function on the period Efimov (1980). We write Fourier series approximation for “improved” function of velocity u(s) = v(s) − g(s), s ∈ [0; L], which is modification of Maliev’s method Efimov (1980). Here u is the improved function, g is the improving function — polynomial of degree (2m − 1) with prescribed derivatives on the boundaries up to order (m − 1) inclusive: (

g(0) = v(0), g 0 (0) = v 0 (0), . . . , g (m−1) (0) = v (m−1) (0); g(L) = v(L), g 0 (L) = v 0 (L), . . . , g (m−1) (L) = v (m−1) (L).

(F.0.2)

Thus, u is continuous periodic together with its derivatives up to order (m − 1). u(s) is computed as follows (l = L2 ): kπs kπs i a0 Xh + ak cos + bk sin . u(s) = 2 l l k=1 n

Recomputing v from u and g, one can calculate analytically the derivatives of v of all orders needed, and compute the corresponding cost as a numerical approximation to (F.0.1).


Appendix

F

207

Fourier coefficients {~a ∈ Rn+1 , ~b ∈ Rn } together with the boundary conditions on v(s) (F.0.2) are included into the parameter space for optimization. The higher is m the more accurate the Fourier approximation is and the faster the decrease of the Fourier coefficients Efimov (1980). The coefficients ~a and ~b should be scaled after each iteration, so that the predicted tangential velocity would be compatible with the length of the path and the duration of the motion Richardson & Flash (2002). Namely, L T = = v

ZL

ds v

0

(v =

L T

denotes the average tangential velocity) and ZL T =

ds = v(s)

0

ZL

ds . u(s) + g(s)

0

Thus it is necessary that ZL

ZL [v − g(s)]ds .

u(s)ds = 0

0

So, after each iteration of the minimization search we implement the following scaling operation: RL (v − g(s))ds 1. k =

RL n 0

0

a0 2

+

Pn h k=1

ak cos kπs + bk sin kπs l l

io ds

2. Set ~a = k~a and ~b = k~b. This method has a drawback: g(x) might have a large amplitude. One can overcome this drawback by introducing a number of points {xj }, j = 0, n; 0 = x0 < x1 < . . . < xn = L and taking g(x) as a spline of order m − 1 passing through {xj }. Minimization of the cost functional was proceeded by the same means as for the minimum-jerk model with via-points.


Appendix

F

208

Approximation of the cost with 5-th order polynomials The x and y components of the optimal trajectory for the minimum-jerk motion with one via-point are described by two 5-th order polynomials of time each, one polynomial before and the other after the via point Flash & Hogan (1985). We consider more general problem, with several via-points, all the recorded samples being the via-points as ideal case. When the number n of via-points is greater than 1, all the derivations for the sets of equations at the via-point Flash & Hogan (1985) are true. The optimal trajectory is defined then by 12(n + 1) coefficients of 2(n + 1) 5-th order polynomials with n unknown time intervals between n + 2 locations; the last, (n + 1)-th time interval adds up to the known total duration of the movement. Twelve boundary conditions and 13n conditions at via-points correspond to 13n + 12 unknowns. When the optimal trajectory has to be found for prescribed time-intervals between the points, 12(n + 1) unknowns can be uniquely defined from the linear system of 12(n + 1) equations. The equations are obtained by substituting values of time-intervals into the polynomials and equating the expressions at the boundary points and at the via-points. In more detail, let Pi denote the segment of the optimal trajectory between the time ti and ti+1 , it connects the points ri and " intervals # xi ri+1 , i = 1, . . . , n + 2. Pi = , xi and yi are some 5-th order polynomials of yi time. Let xi (t) = ai0 + ai1 t + ai2 t2 + ai3 t3 + ai4 t4 + ai5 t5 . The vectors of coefficients {ai }, ai = [ai0 , ai1 , . . . , ai5 ]T can be found from the following linear system with 6(n + 1) unknowns: 

Γ1

  Γ2,en Γ 2,st    Γ3,en Γ3,st        0  

        ..  · .   Γn,en Γn,st    Γn+1,en Γn+1,st   Γn+2 0

a1 a2 a3 .. . an−1 an an+1





              =            

b1 b2 b3 .. . bn−1 bn

              

bn+1

or, in shorter notation C · a = b.

(F.0.3)

Let us define the components of the 6(n + 1) × 6(n + 1) matrix C and 6(n + 1) × 1 column b.




Appendix

1 T1 T1 2

T1 3

F

T1 4

209

T1 5



  2 3 4  Γ1 =  0 1 2T 3T 4T 5T 1 1 1 1  , 0 0 2 6T1 12T1 2 20T1 3   1 Tn+2 Tn+2 2 Tn+2 3 Tn+2 4 Tn+2 5   4  3 2 Γn+2 =  5T 4T 0 1 2T 3T n+2 n+2 n+2 n+2 ,  0 0 2 6Tn+2 12Tn+2 2 20Tn+2 3   Tj 4 Tj 5 1 Tj Tj 2 Tj 3    0 1 2Tj 3Tj 2 4Tj 3 5Tj 4     3  2  0 0 2 6Tj 12Tj 20Tj  , Γj,en =   2  0 6 24Tj 60Tj   0 0     0 0 0 0 24 120T j   0 0 0 0 0 0   0 0 0 0 0 0    0 −1 −2Tj −3Tj 2 −4Tj 3 −5Tj 4      2 3  0 0 −2 −6T −12T −20T j j j  ; Γj,st =    0 −6Tj −24Tj −60Tj 2   0 0    0 0  0 0 −24 −120T j   2 3 4 5 1 Tj Tj Tj Tj Tj     x1 (T1 ) xn+1 (Tn+2 )      , bn+2 =  x˙ n+1 (Tn+2 )  , b1 =  x ˙ (T ) 1 1     x¨1 (T1 ) x¨n+1 (Tn+2 )   xj (Tj+1 )     0       0  , j = 1, . . . , n .  bj =     0     0   xj+1 (Tj+1 )


Appendix

F

210

The matrix of coefficients C and the right-hand side b are composed of the input of the model: C is composed of the functions of the time instants of passing through the via-points, b is composed of the coordinates of the via-points and of the boundary conditions. The solution of the system (F.0.3), a, defines the coefficients of the polynomials that describe the projection of the optimal trajectory on the x axis; the same calculations lead to the expression of the y component of the optimal trajectory. The cost (2.2.14) can be analytically calculated by differentiation, squaring, summation and integration of xi and yi . Minimization of the cost functional can be performed by means of the “matlab” function “fminsearch”. The time intervals ∆t are subject to change, the initial guess can taken from the data. This method has been tested successfully on a number of examples with a priori known solution. When dealing with a small number of points (e.g. 5), perturbations of the initial guess did not result in changes of the prediction; and the method was stable w.r.t. small perturbations of the boundary conditions and positions of the via-points. When there are more than 11 via-points, the determinant of the matrix of coefficients C rapidly decreases with increase of the number of via-points and reaches very small values (see figure F.1 A). The sensitivity of the system Cx = b to small changes in C or b is quantified by the condition number c(A) = σσn1 (C) times the relative error in (C) C and b, where σ1 (C) and σn (C) are the largest and the smallest singular values of C respectively (Golub & Loan (1983), pp. 24-26). Increase of the number of via-points makes the numerical procedure time-consuming and sensitive to small perturbations of the input data (see figure F.1 B). Comparisons of the methods Given a relatively long record of the movement (e.g. 2 sec. duration), one may need to use relatively many via-points to keep the information about the drawn path. Providing many via-points as an input to the system (F.0.3) will result in high condition numbers and unrealistic solution. We modelled the optimal trajectories for several exemplar movement segments by means of (1) the finite differences approximation of the cost and (2) Fourier approximation of the drawing speed. The results were similar for all cases. The method of approximation with Fourier series is rather time-consuming compared with the finite differences. Therefore we propose that the approximation of the cost with finite differences can be used in modelling of optimal trajectories for large number of movement segments.

Appendix

13

20

12 log10 of the value of the condition number

log

F

30

10

0 −10 −20 −30 −40 −50 −60 0

211

11

10

10

of the value of the determninant


9 8 7 6 5 4

5

10

15

20 25 30 Number of via−points

A

35

40

45

50

3 0

5

10

15

20 25 30 Number of via−points

35

40

45

B

Figure F.1: Properties of the matrix C from (F.0.3) versus number of via-points. A. The determinant of C rapidly decreases with increase of the number of via-points when there are more than 11 via-points, and reaches very small values. B. The condition number, defined in the text, quantifies the sensitivity of the system (F.0.3) to small changes in C or b. It increases fast with increase of the number of the via-points, meaning that our solution of the problem with relatively many via-points (e.g. > 15) would be unreliable.

50

Appendix G Minimum-jerk model and equiaffine geometry in the analysis of drawing movements, example and results The trajectories of both monkeys became smoother and more regular with practice. We characterized the convergence properties of the performance with equiaffine invariants of the trajectories and by comparison between the recorded trajectories and the ones predicted by the constrained minimum-jerk model. Figure G.1 A shows an exemplar path that the monkey O drew after a period of practice.

G.0.2

Example: analyzed parameters on a single movement segment

Shown in Figure G.1 are parameters of a movement segment, scribbled by the monkey. The segment path is depicted in Figure G.1A. This path is smooth and consists of several loops w.r.t. direction of motion. The actual and predicted time-courses versus samples of the path are depicted in figure G.1B. Their difference defines the timewarping needed to make the recorded trajectory obey the constrained minimum-jerk model. Figure G.1C shows the speed profiles of the actual and predicted trajectories. These profiles can be compared as functions of time and as functions of location on the path. For actual trajectories sample numbers on the path are proportional to time due to the constant recording frequency. For predicted trajectories, however, the time intervals between consecutive sampling points are not necessarily constant, i.e. the time which must be spent to pass between the samples in optimal motion 212


Appendix

G

213

would be different from the actual time. So we plotted single profile for the actual speed, and two profiles for the predicted speed: one versus time and the other versus sample point numbers divided by recording frequency. As figure G.1C indicates, the locations on the path - sample numbers divided by the recording frequency - at which the predicted speed profiles took extremal values, closely matched those of the actual trajectories. Nevertheless, the model was less successful in predicting the temporal values of these events. The two predicted profiles visualize the time-warping, it is exactly the time difference between time instances when the two graphs, (b) and (c), take the same values, extremal values are clear landmarks. Had the predicted trajectory obeyed the two-thirds power-law, its velocity gain factor K p (t) = σ˙ p (t) (see (2.2.16)) should be piece-wise constant, and hence the sequences {∆tp i } and {|∆σi |} should be piece-wise proportional. In the example shown in figure G.1D both parameters were scaled in order to put them on the same axis. Our estimation of the predicted equiaffine velocity, which is their ratio, together with the actual equiaffine velocity, were scaled to the same axis and plotted in figure G.1E. The scaling does not change the deviation from constancy of a sequence, but helps in visualization. Scaling constants are displayed in the legend of the figure. As figure G.1D demonstrates, the scaled sequences of ∆tp,c (where index c refers to the regularized data, as explained above) and |∆σ c | follow the same phase of lowfrequency oscillations and have similar depths of modulation on these frequencies, disrupted by some higher frequency noise. This means that the predicted equiaffine velocity σ˙ p,c is closer to constant than the actual one. In the legend of figure G.1E, the deviation from constant estimated by the standard deviation normalized by the average, assures such visual observation. The deviation was less for the predicted trajectory, comparing with the actual one, by more than 38%, and could be less but for the noise. The third graph in figure G.1D demonstrates the values of the equiaffine curvature. Several “continuous” pieces with values very close to zero can be observed in the graph, which hints at possible applicability of the equiaffine curvature for movement data segmentation. Zero equiaffine curvature corresponds to parabolic paths (2.2.13). The above-mentioned pieces indeed correspond to parabolic-like forms on the path (see also Figure G.6). Interestingly, equiaffine curvature oscillates around zero, that is its local maxima are mostly positive and minima - negative.

G.0.3

Quantitative characterization of changes in the drawn trajectories with practice

The main goal of the data analysis was to see the quantitative characteristics of the observed changes in the behavior: after a period of practice the drawn trajectories became smoother and often possessed directional regularity in cycles of motion, see


Appendix

G

214

B. Actual and predicted time courses versus sample points 3

4 2 0 −2 −4 −6 −8

Time−course. [sec.]

Position − y, [cm]

A. Path

−10

−5

0

5

10

2

1

0

Position − x, [cm]

Actual Predicted

0

50

100

150

200

250

300

Sample point number

C. Actual and predicted tangential velocity profles for the recorded segment 1

(a) actual speed versus time (b) predicted speed versus time (c) predicted speed versus sample points

[m/s]

0.8 0.6 0.4 0.2 0

0.5

1

1.5

2

2.5

3

Time for (a, b), (Sample point number) / (Sampling frequency) for (a, c)

D. The equiaffine parameters and the predicted time intervals for the above segment scaled ∆ t predicted (by 0.016 s) scaled |∆ σc| (by 1.5 mm2/3) κ (equiaff. curv.), mm−4/3

κ and scaled |∆σ|, ∆ t

1

0.5

0

0

50

100

150

200

250

300

250

300

|∆σc| / ∆tp,c, |∆σc| / ∆ta,c scaled

Sample point number

E. Scaled actual and predicted equi−affine speeds 1 0.5 |∆σc|/∆tp,c, scaled by 123 mm2/3/s; std/aver.=0.21 |∆σc|/∆ta,c, scaled by 146 mm2/3/s; std/aver.=0.34

0

0

50

100

150

200

Sample point number

Figure G.1: Analysis of one movement segment. A. Segment path. By square and circle we denote start and end of the segment respectively. B. Had the monkey drawn the path in A according to the constrained minimum-jerk model, the time evolution along the segment would look like the presented dashed-dotted graph. C. Drawing speeds. Minima and maxima of the actual and predicted trajectories occur at similar positions of the path, but their time-course is different. The two predicted profiles visualize the time-warping, it is exactly the time difference between time instances when the two graphs, (b) and (c), take the same values, extremal values are clear landmarks. D. Scaled magnitudes of the increments of the equiaffine arc-length on their segments of continuity (with outliers omitted, scaled increments of the predicted time intervals between adjacent samples, and equiaffine curvature. The trajectory fits the two-thirds power-law iff equiaffine velocity is piecewise constant, or, equivalently, |∆σ| and ∆t are piece-wise proportional. One can see similar behavior of ∆tp,c and |∆σ c | (the superscript c refers to the regularized values), which means that the predicted equiaffine velocity is close to constant. Equiaffine curvature is close to zero and resembles segmentation into several “continuous pieces” with values closest to zero within such pieces. These pieces correspond to parabolic-like forms on the path. E. Equiaffine velocities, actual and predicted (superscripts a and p respectively), were scaled to the same axes D. Predicted speed demonstrates smaller deviation from the constant than the actual one.


Appendix

G

215

for example path in figure G.1A. We considered different characteristics of the behavior: geometric and temporal, local — point-wise (equiaffine curvature) and global — segment-wise (estimate of the fit of a segment to a model). The convergence was observed for geometric, but not for temporal characteristics among analyzed ones. Noting the unique equiaffine invariant relationship between the constrained minimum-jerk model and the two-thirds power-law via parabolic paths, predicted trajectories can be used to characterize geometric properties of the drawn paths. Beside geometric properties, we looked at the temporal differences introduced by the prediction. They can be measured by means of the time-warping between the actual and predicted trajectories. We also compared the estimates of the fit to the constrained minimum-jerk model for monkey scribblings and for human tracing data (see also methods). Time-warping between the actual and predicted trajectories We compared the time courses {ta 1 = 0, ta i+1 = i/(Recording frequency)} and {tp 1 = P i p p 0 , tp i+1 = j=1 ∆t j } = t (si ), i = 1, . . . , N − 1, for the actual and predicted trajectories correspondingly. An example of these parameters is presented in figure G.1B. The deviation between the actual and predicted time-courses, measured in percents, for each segment was calculated according to the following formula: " N # X 2 · 100 1 ρt = |wi | + (|w2 | + |wN −1 |) (G.0.1) (N − 1)ta N i=1 8 with wi = tp i − ta i . Note that w1 = ta 1 − tp 1 = 0, wN = ta N − tp N = 0 This formula is composed of two multipliers: the total deviation between the actual and predicted time-courses, as approximation of the area between the two graphs in figure G.1B (appears in the square brackets), and normalization to the percents. Time-course is strictly increasing function. Therefore the possible area between the profiles of the actual and predicted time-courses cannot be larger than area below the graph of the actual time = 21 (N − 1)ta N . We chose this area to be 100%. Weighted averages of ρt , with the weights proportional to number of the time increments {∆t} in the segment are depicted in figure G.2. As figure G.2 shows, over days the monkey movements showed deviations of the actual times from the time predicted by the constrained minimum-jerk model. The daily averages of the deviations were in the same range (4%-9%) for both monkeys. For both monkeys we did not detect any tendencies of convergence or divergence in this estimate of the fit to the constrained minimum-jerk model. The deviation of the recorded human drawings from the trajectories predicted by the constrained minimum-jerk model was estimated in the same way as for monkey


Appendix

G

216

Monkey O Monkey U Weighted average values of the deviation of the predicted time−courses from the actual ones, [%] 8 7 6 5 4



Figure G.2: Weighted averages of the estimated fit of the trajectories to the constrained minimumjerk model. Circles denote the averages for all segments in a day, squares denote the averages for the segments when no reward was obtained and triangles depict the averages for the segments when at least one reward was received within a movement segment. For both monkeys the values of the estimate were in the same range of values. No convergence is observed.

Cloverleaf 1.026

Lemniscate 1.990

Limacon 2.051

Table G.1: Average values of the deviation of the predicted time-course from the actual one for human drawings [%]. The deviation was estimated in the same way as for monkey scribblings, according to (G.0.1)

scribblings, according to (G.0.1). The fit for human drawings (table G.1) was good and superior to that of the monkey scribblings depicted in Figure G.2, although human drawings showed different deviations of the predicted time-courses from the actual ones for different geometric templates. Geometric characteristics of the performance converge to the ones characterizing parabolas: analysis of equiaffine curvature and equiaffine velocity We remind that the prediction of the constrained minimum-jerk model for parabolic paths satisfies the two-thirds power-law, that is predicted equiaffine speed for parabolas is constant. This property is invariant under arbitrary equiaffine transformations only for parabolas. Parabolas have zero equiaffine curvature. We test the changes that predicted equiaffine speed and equiaffine curvature undergo with practice. With practice both parameters converge towards the values which characterize parabolas. More details follow. The results of our numerical estimation of the equiaffine parameters contain outliers. In order to regularize values of equiaffine parameters, we performed our analysis on the strokes of continuity of the trajectories (see methods), briefly, every movement segment may contain several strokes of continuity, which correspond to regular parts


Appendix

G

217

of the movement segment. Values of the time increments corresponding to the samples related to the strokes are denoted by ∆tc (superscript c stands for continuity). In order to observe the changes of the equiaffine curvature of the monkey trajectories with practice, we averaged the magnitude of the equiaffine curvature for the samples belonging to the strokes from the segments. Only values of the equiaffine curvature with the magnitude less than the threshold = 0.5 mm−4/3 were accounted. Most of the samples in the strokes have equiaffine curvature bounded by this threshold. Weighted averages of the magnitude of the equiaffine curvature, with weights proportional to the number of accounted values in a segment, are depicted in Figure G.3. The magnitude of the equiaffine curvature decreased, i.e. became closer to zero, during first 6-7 days of practice and then its value stabilized. Accounting equiaffine curvature on entire movement segments including parts outside the strokes (and still using the threshold), resulted in the same tendency. Zero equiaffine curvature characterizes parabolas. Equiaffine curvature is a local, point-wise estimate of the drawn paths. T o further investigate convergence properties of monkey performance, we used a global estimate, based on the constrained minimum-jerk model. Monkey O Monkey U Weighted average values of the magnitude of equi−affine curvature, [mm−4/3] 0.2 0.15 0.1



Figure G.3: Weighted averages of the regular values of the magnitude of equiaffine curvature. For both monkeys equiaffine curvature became closer to 0 during practice. Circles denote the averages for all segments in a day, squares denote the averages for the segments when no reward was obtained and triangles depict the averages for the segments when at least one reward was received within a movement segment.

Whenever the magnitude of the equiaffine velocity σ˙ p (t) is constant, or equivalently, the time intervals of passing between the neighboring points, ∆tp, c (superscripts p and c stand for predicted and continuous respectively) are proportional to the equiaffine arc-length between those points |∆σ c |, the two-thirds power-law is satisfied for the predicted movement composed of parabolic pieces, because in this case const = K p (ti + ∆tp i /2) = |σ˙ p, c (ti + ∆tp i /2)| ≈ |∆σ c i |/∆tp, c i . This argument is visualized in figure G.1 D. We explain why we considered constancy of the velocity gain factor K, and not piece-wise constancy further in the text. In other words, for every movement segment when the two vectors: |∆σ c | and ∆tc are parallel, the motion is


Appendix

G

218

according to the two-thirds power-law. The closer the angle between them is to zero, the better is the fit. Therefore we estimated the global fit to the two-thirds power-law with the angle between the vectors ∆tc and |∆σ c |: γ(∆ta,c , |∆σ c |) − for the actual trajectories, γ(∆t p,c , |∆σ c |) − for the predicted trajectories , with γ(∆t, ∆σ) = arccos

(∆t · ∆σ) . k∆σk · k∆σk

We remind that the actual and predicted trajectories differ in the time intervals of passing between the neighboring samples: ∆ta and ∆tp for the actual and predicted trajectories respectively. The path and duration of both movements coincide. The values of the divergence between the vector of time increments and the vector of the magnitudes of the increments of equiaffine arc-length γ for each movement segment were weighted (with sum of the weights being one) by the values proportional to the number of elements in ∆σ c for that segment, and then summed. The estimates of the fit for the actual and predicted time-courses, and comparison between them, are depicted in figures G.4A, B, C respectively. The global fit of the predicted trajectories to the two-thirds power-law became noticeably better (especially for the first monkey), after several days of practice, and was superior to the global fit of the actual trajectories (Figures G.4A, B, C). We conclude that the monkeys learned to scribble the paths such that the predicted fit of the two-thirds power-law for such paths improved during practice. We performed linear regression fit of the logarithmic form of the power-law log V = log K+β log R for the parameters log K and β given speed V and radius of curvature R on short pieces of the recorded monkey trajectories, and did not see clear cut evidence confirming or rejecting the two-thirds power-law for the data. Additionally, the twothirds power-law assumes the gain factor K to be piece-wise constant, which cannot be estimated by our measure of its constancy γ. The mainstream of the current work was to use our findings related to the relationship between the constrained minimum-jerk and the two-thirds power-law to quantify geometric regularities that emerge during extensive practice. Therefore in this study we did not analyze the temporal aspect of the recorded monkey trajectories concerning the fit to the two-thirds power-law, the analysis of this issue needs to be performed separately. We only analyzed geometric properties of the paths using the constrained minimum-jerk model and the two-thirds power-law simultaneously.


Appendix

Moneky O

G

219

Moneky U

A. Weighted average values of the angles between the vectors describing the a,c c actual times, ∆t , and the vectors ∆σ for corresponding paths [degrees] 25

20

15 B. Weighted average values of the angles between the vectors describing the predicted times, ∆tp,c, and the vectors ∆σc for corresponding paths [degrees] 22 20 18 16 14 12 C. Weighted average values of angles ratio for predicted and actual trajectories (pred/actual) 1 0.9 0.8 0.7 0.6



Figure G.4: Weighted averages of the estimates of the actual and predicted equiaffine velocities. Circles denote the weighted averages of the depicted parameters for all segments in a day, squares denote the averages for the segments when no reward was obtained and triangles depict the averages for the segments when at least one reward was received within a movement segment. A. Fit of the actual trajectories to the two-thirds power-law. The fit of the trajectories of the first monkey did not change with practice. For the second monkey, improvement of the fit took place at the later stage of practice. B. For both monkeys, from start of the practice, there was a clear improvement in the fit of the predicted trajectories to the two-thirds power-law. C The fit of the predicted trajectories to the two-thirds power-law was superior comparing to the fit of the actual trajectories. This superiority increased with practice, the increase was stronger for monkey O.


Appendix

G

220

The monkeys developed a strategy characterized by a more uniform tangential and equiaffine velocities Changes in the performance presented above evidence for the possibility of developing of a new strategies in monkeys’ performance. A natural way of looking at these changes would be in separating the data into movement segments at which monkey got at least one reward and movement segments at which the monkey did not get any reward. For all estimates presented by now, the averages for the rewarded, non-rewarded segments, and total averages (putting the rewarded and non-rewarded segments together) are similar for monkey O and show small consistent differences for some parameters estimated for monkey U, Figures G.2, G.3, G.4, G.5A. Nevertheless, the normalized standard deviations (standard deviation divided by average) of tangential velocity and equiaffine velocity, both actual and predicted, are noticeably smaller for the rewarded segments than for non-rewarded. As an example, we depict the normalized standard deviation of the predicted equiaffine velocity calculated on the pieces of continuity in Figure G.5B. Corresponding average values show only a small consistent bias for monkey U, figure G.5A. We conclude that the monkey gets a reward with higher probability when it follows a strategy characterized by a more uniform ratio: segment extent (both Euclidian and equiaffine) to segment duration. The inverse reasoning: more uniform velocities are caused by the reward, can only reinforce our conclusion. As figure G.5 B shows, the monkeys developed the new strategy after a few (three) days of practice. Additional tests of the results In the data analyses described above, all the averages were weighted with the weights proportional to the number of considered samples in averaged items. We also considered averages without weighting. The convergence properties were exactly the same. Different movement segments have different durations. Thus the vectors ∆σ c from different movement segments are composed of different number of elements. Therefore, when calculating γ for different movement segments, we calculate angles between vectors of different dimensions. We wanted to check feasibility of our analysis of γ calculated in different dimensions. So we resampled the vectors |∆σ c | and |∆tc | to 300 elements for every movement segment. Evolution of γ with practice on these resampled vectors was very similar to the one presented in Figures G.4A, B, C, with the same convergence properties.


Appendix

G

221

Monkey O Monkey U A. Average values of αp: the averages of equi−affine velocity [mm2/3/s] 80 70 60 50 40 30 B. std(αp) / average(αp) 0.4 0.3 0.2



Figure G.5: In both plots circles denote the values that correspond to all segments in a day, squares denote the averages for the segments when no reward was obtained and triangles depict the averages for the segments when at least one reward was received within a movement segment. A. Estimates of weighted averages of the least squares estimates of the predicted equiaffine velocity α = arg minα ||α∆tp, c − |∆σ c |||. B. Normalized standard deviation of α. For both monkeys, after a few (three) days of practice, deviation for the rewarded pieces is smaller than the deviation for the non-rewarded pieces. We interpret this observation as follows: during practice the monkey developed a strategy of drawing and got more rewards when it was following this strategy. Very similar results were obtained for actual equiaffine velocity and for tangential velocity. Therefore the acquired strategy is characterized by a more uniform ratio: segment extent (both Euclidian and equiaffine) to segment duration. Why we analyzed constancy of the predicted equiaffine speed, and not piece-wise constancy We consider different strokes from the same movement segment together (that is speak about constancy of σ˙ p (t) and not piece-wise constancy) because individual analysis of relatively short strokes would make the whole approach of global (for entire movement segment) optimization senseless as the neighboring values of ∆tp are similar and thus locally (for short strokes) actual and predicted equiaffine velocities are close to being proportional. Our approach makes sense because the global estimates of constancy of the predicted and actual equiaffine speeds are different, when we consider the strokes


Appendix

G

222

from the same movement segment together, as was evidenced by differences in the estimates presented in figure G.4C. The main conjecture based on the analysis of the equiaffine invariants of the data: the monkeys’ performance might have converged to piece-wise parabolic drawings Based on the analysis of monkey trajectories, we got 2 empirical evidence that support our finding of the special role of parabolic paths in planar scribbling movements. • During practice the magnitude of the equiaffine curvature became closer to 0. It is 0 for parabolas. • The global fit of the predicted trajectories to the two-thirds power-law improved with practice (see figure G.4B). The fit is exact for parabolas. The above evidence may imply that the monkeys learned to draw trajectories with near-piece-wise parabolic paths. An example of path approximation with parabolas is presented in figure G.6.


Appendix

G

223

Two parabolas fit the segment path 40

Position − y, [mm]

0

Second parabola → −40

First parabola → −80

−80

−40

0 Position − x, [mm]

40

80

Figure G.6: An example of a path pattern (asterisks) and its approximation with two parabolas (dashed lines), after several days of practice. Start and end of the part of the path to be fitted are depicted by the square and circle respectively.

Appendix H Additional figures We moved to this appendix different figures, which can help in getting more intuition about the data and our analysis to the careful reader. The figures were gathered from all chapters, and are referenced to from the corresponding chapters.

H.1

Chapter 2

224

PHD thesis by Felix Polyakov Day 1, 07jul97. Total#=523

Appendix

H

225



5

5

300

4

4

200

3

3

2

2

1

1

20

Angle

15

100 0

0

5

0


0


Day 6, 14jul97. Total#=2119 25

60

50

300

Angle

10

200

20

40

40

15

30 10

20

20

100

5

10 0

0

0

0


Day 9, 22jul97. Total#=2263 150 100 100 50

50

0

Day 10, 23jul97. Total#=1575

0

Day 11, 25jul97. Total#=5724

Day 12, 29jul97. Total#=7304 150

Angle

300

200

60 40

100

20

100

0

0

20

40

60

80

0

50 50 0

20

p

40

60

80

0

0

20

p

Day 13, 31jul97. Total#=9779

40

60

80

Day 14, 01aug97. Total#=7510

Day 15, 03aug97. Total#=9545 600 400

300

200

200

150

100

300

400

200

100

200 100

50

0

0

0

Day 16, 04aug97. Total#=8280

0

Day 17, 05aug97. Total#=6046

Day 18, 06aug97. Total#=9752

400

300

500

300

Angle

300

400

200

200

300

200 100 0

0

p

250

Angle

100

150

200

0

20

40

p

60

80

0

200

100

100

100 0

20

40

p

60

80

0

0

20

40

60

80

0

p

Figure H.1: Two-dimensional histograms with numbers of parabolic pieces. This figure corresponds to the start of practice of monkey O. Two-dimensional histograms of the focal parameter (x-axis) and direction of the normal at the point of maximal curvature (y-axis). From start of practice, the parabolas were getting clustered into two clusters.


Appendix

H

226


Day 3, u12jun99. Total#=225 1

5

0.8

4

0.6

3

0.4

2

0.2

1

0


0



20 300

20

30

Angle

15 15

200

20

10 10

100

5

0

0

0


0



15

300

Angle

10

5

200

10

100

5

0

0

30

60

20

40

10

20

0


0



Angle

50 300

150

200

100

100 0

Angle

20

30

15

20

10

10

5

50

0

20

40

60

80

0

0

20

40

60

80

0

0

20

40

60

p

p

p



Day 15, u01jul99. Total#=2929

80

0

40

300

30

30

20

20

10

10

0

0

30 200 100 0

20


Day 17, u06jul99. Total#=2203 15

200

10

10

100

5

5

0

0

20

40

p

60

80

0

0

20

40

60

10 0

20

40

60

80

0

p

15

300

Angle

25

40

80

0

p

Figure H.2: Two-dimensional histograms with numbers of parabolic pieces. This figure corresponds to the start of practice of monkey U. Two-dimensional histograms of the focal parameter (x-axis) and direction of the normal at the point of maximal curvature (y-axis). From the start of the practice, the parabolas were getting clustered into two clusters, though after the 10-th day of practice, 24jun99, the accumulation into the third cluster can be observed.

PHD thesis by Felix Polyakov Day 1, 26jun00. Total#=2522

Appendix

H

227

Day 2, 27jun00. Total#=3080

Day 3, 28jun00. Total#=3375 120

60

Angle

300 200

40

100

20

100 100

80 60

50

40 20

0

0

0


Angle

Day 6, 02jul00. Total#=3489 80

80

300

100 60

60 200 40 100

40

50

20

20

0

0

0


0


Day 9, 05jul00. Total#=4769 150

100

300

Angle

0


100

80 200

100

60 40

100

50

50

20 0

0

0

Day 10, 06jul00. Total#=3781 300

Angle

0

Day 11, 09jul00. Total#=3833

Day 12, 10jul00. Total#=2567 80

100

100 60

200 50

40

50

100 0

20 0

20

40

60

80

0

40

60

80

0

0

20

40

60

p

Day 13, 11jul00. Total#=2766

Day 14, 16jul00. Total#=3645

Day 15, 17jul00. Total#=5095

200 100 0

100

100

80

80

60

60

40

40

20

20

0

0

Day 16, 18jul00. Total#=2549

Angle

20

p

300

Angle

0

p

Day 17, 20jul00. Total#=3676

300

60

200

40

100

20

80

0

150 100 50

0

20

40

60

80

0

p 100

50

0

0

20

40

p

60

80

0

0

20

40

60

80

0

p

Figure H.3: Two-dimensional histograms with numbers of parabolic pieces. This figure corresponds to the last days of practice of monkey U, for which we have recordings of neural activity. Two-dimensional histograms of the focal parameter (x-axis) and direction of the normal at the point of maximal curvature (y-axis). There are two relatively stable clusters, corresponding to 1000 and 3000 . The cluster, which corresponds to the direction 2000 is not stable. It clearly appears for some days, e.g. 28jun00, 04jul00, 06jul00, 11jul00, 17jul00.


15

Appendix

25

H

228

60

90 80

50

15

10

40

30

20


5

70


10



20

60 50 40 30 20

5

10 10

0

0

250

90

0

0 150

80

100

70


150



200

60 50 40 30

100

50

20

50

10 0

0

150

0

120

400

500 450

350

80

60

40

300 250 200 150 100


50

400


100



100

50

250 200 150

50

0

0

900

700

700

700

600

600

600

600 500 400 300

01aug97; 8 Gaussians

700

500

400

300

200

200 100

100 0 0

0 600

200

100

0

150

200

250

300

350

300

200

500

400

300

200

100

100

0

0

600


300

100

400

700

500

400

50

500


0


0

800


300

100

20


350

500

400

300

200

100

0

Figure H.4: Data for monkey O, from start of practice. Histograms of orientations of parabolic pieces. The histograms were fitted with the mixtures of Gaussians (GMMs). Gaussians multiplied by their weights are depicted by the dashed lines. The optimal number of mixture components is marked in the y label of each plot.


Appendix

H

229

10

45

9

40

7 6 5 4 3

u13jun99; 3 Gaussians


8

2

40 30 20

40 30 20

10

40

30

20

100

0

0

0

200

300

140

80

60

40

20

10

10

10

50


60


70



80

50

15

0

70

50

20

120

80

60

25

0 90

60

30

5

1

70

35

0 120

180

80 60 40

150

100

50

100

80

60

40

100


100

200


120



140

120

250

160

80

60

40

20

20

20 0 140

70

120

120

60

80

60

40

20

0



100

100

80

60

40

100

80

60

40

0

u05jul99; 5 Gaussians

0 140


0 120

50

40

30

20

20

20

10

0

0

0

70


60

50

40

30

20

10

0

Figure H.5: Data for monkey U, from start of practice. Histograms of orientations of parabolic pieces. The histograms were fitted with GMMs. Gaussians multiplied by their weights are depicted by the dashed lines. The optimal number of mixture components is marked in the y label of each plot.


80

Appendix

100

H

230

150

120

90

70

100

80

40 30

60 50 40

28jun00; 6 Gaussians

50



70

30

100

50


60 80

60

40

20 20 10

20

10

0 140

0

0

0

120

120

180

100

100

80

80

160

120

140

40

60

40


60

60


80



100 120 100 80 60

40 40 20

20

0

0

0

250

150

200

90

180

80

20

20

50

120

70

100 80


100

100

140 09jul00; 5 Gaussians

150

160



200

0

60

40 30

10

20 0

0

0

0

120

120

350

80

100

300

70

100

60

40

60

40

200

150

100 20

20

0

0


60

80


80


250 11jul00; 5 Gaussians

50

20

40

50

60

50 40 30 20

50

10

0

0

120


100

80

60

40

20

0

Figure H.6: Data for monkey U, following extensive practice. Histograms of orientations of parabolic pieces. The histograms were fitted with GMMs. Gaussians multiplied by their weights are depicted by the dashed lines. The optimal number of mixture components is designated in the y label of each plot.


Appendix

H

231







60 40

y, [mm]

20 0 −20 −40 −60 −80 −100

60 40

y, [mm]

20 0 −20 −40 −60 −80 −100 −100

−50

0

50

100 −100

−50

x, [mm]


0

50

100 −100

−50

0

50

x, [mm]

x, [mm]



Day 11, 25jul97. Total#=1145

Day 12, 29jul97. Total#=1461

100

60 40

y, [mm]

20 0 −20 −40 −60 −80 −100 −100

−50

0

50

100 −100

−50

0

50

100 −100

−50

0

50

x, [mm]

x, [mm]

x, [mm]

Day 13, 31jul97. Total#=1956

Day 14, 01aug97. Total#=1502

Day 15, 03aug97. Total#=1909

Day 16, 04aug97. Total#=1656

Day 17, 05aug97. Total#=1210

Day 18, 06aug97. Total#=1951

100

60 40

y, [mm]

20 0 −20 −40 −60 −80 −100

60 40

y, [mm]

20 0 −20 −40 −60 −80 −100 −100

−50

0 x, [mm]

50

100 −100

−50

0 x, [mm]

50

100 −100

−50

0

50

100

x, [mm]

Figure H.7: Data for monkey O, from start of practice. Blue dots - locations of vertices of every fifth parabola, red segments - orientations of those parabolas.



Appendix

H

232





100

y, [mm]

50

0

−50

−150

−100

−50

0

50

100 −150

−100

−50

0

50

100 −150

−100

−50

0

50

x, [mm]

x, [mm]

x, [mm]







100

100

y, [mm]

50

0

−50

100

y, [mm]

50

0

−50

−150

−100

−50

0

50

100 −150

−100

−50

0

50

100 −150

−100

−50

0

50

x, [mm]

x, [mm]

x, [mm]




100

100

y, [mm]

50

0

−50

−150

−100

−50

0

50

100

x, [mm]



100

y, [mm]

50

0

−50

−150

−100

−50 x, [mm]

0

50

100 −150

−100

−50

0

50

100

x, [mm]

Figure H.8: Data for monkey U, from start of practice. Blue dots - locations of vertices of every fifth parabola, red segments - orientations of those parabolas.


Appendix

H

233







100 80 60

y, [mm]

40 20 0 −20 −40 −60 −80 −100 100 80 60

y, [mm]

40 20 0 −20 −40 −60 −80 −100 −100

−50

0

50

100

−100

−50

0

50

100

−100

−50

0

50

x, [mm]

x, [mm]

x, [mm]







100

100 80 60

y, [mm]

40 20 0 −20 −40 −60 −80 −100 100 80 60

y, [mm]

40 20 0 −20 −40 −60 −80 −100 −100

−50

0

50

100

−100

−50

0

50

100

−100

−50

0

50

x, [mm]

x, [mm]

x, [mm]



Day 15, 17jul00. Total#=1020



100

100 80 60

y, [mm]

40 20 0 −20 −40 −60 −80 −100

−100

−50

0

50

100

x, [mm]

100 80 60

y, [mm]

40 20 0 −20 −40 −60 −80 −100 −100

−50

0 x, [mm]

50

100

−100

−50

0

50

100

x, [mm]

Figure H.9: Data for monkey U, overtrained movements. Blue dots - locations of vertices of every fifth parabola, red segments - orientations of those parabolas.

PHD thesis by Felix Polyakov 100

Appendix

H

234

O07JUL7A.032, 4.52−5.02

O08JUL7A.014, 19.41−20.18

O10JUL7A.061, 5.36−6.92

O11JUL7A.203, 2.55−3.97

O12JUL7A.091, 2.36−4.6

O14JUL7A.094, 6.06−8.08

O21JUL7A.028, 6.94−8.45

O22JUL7A.072, 5.33−6.89

O23JUL7A.002, 24.74−26.7

O25JUL7A.153, 12.36−13.54

O29JUL7A.144, 1.33−3.03

O31JUL7A.221, 13.78−15.97

O01AUG7A.027, 8.99−10.12

O03AUG7A.125, 0.48−2.81

O05AUG7A.080, 1.25−3.15

O06AUG7A.326, 6.86−8.38

50

0

−50

−100

100

50

0

−50

−100

100

50

0

−50

−100

100

50

0

−50

−100 O04AUG7A.256, 5.48−7.61 100

50

0

−50

−100 −100

−50

0

50

100

−100

−50

0

50

100

−100

−50

0

50

100

Figure H.10: Monkey O, from start of practice. Exemplar paths fitted with several parabolic pieces. One example for one recording day. Path pieces (blue) and parabolic pieces fitting them. Red - best fitting parabolic piece, green — parabolic piece defined as the best fitting second order polynomial in equi-affine length measured along the path.


Appendix

H

235

U11JUN9A.002, 23.43−24.35

U12JUN9A.046, 25.77−26.66

U13JUN9A.098, 20.58−21.33

U16JUN9A.020, 16.81−18.32

U17JUN9A.046, 19.42−20.43

U18JUN9A.031, 0.1−1.23

U21JUN9A.036, 27.27−28.4

U22JUN9A.118, 9.46−10.4

U23JUN9A.114, 13.05−14.65

U24JUN9A.039, 21.93−23.73

U27JUN9A.083, 33.28−34.77

U28JUN9A.080, 36.54−37.67

U29JUN9A.008, 2.27−3.45

U01JUL9A.073, 15.15−16.29

100

50

0

−50

−100

100

50

0

−50

−100

100

50

0

−50

−100

100

50

0

−50

−150

−100 U05JUL9A.022, 7.58−8.57

−100

−50

0

50

100

U06JUL9A.021, 24.39−25.75

100

50

0

−50

−100 −150

−100

−50

0

50

100

−150

−100

−50

0

50

100

Figure H.11: Monkey U, from start of practice. Exemplar paths fitted with several parabolic pieces. One example for one recording day. Path pieces (blue) and parabolic pieces fitting them. Red - best fitting parabolic piece, green — parabolic piece defined as the best fitting second order polynomial in equi-affine length measured along the path.


Appendix

H

236

U26JUN0A.048, 14.64−16.02

U27JUN0A.064, 0.52−2.27

U29JUN0A.086, 18.54−20.01

U30JUN0A.090, 21.55−23.84

U02JUL0A.022, 5.4−6.99

U03JUL0A.117, 1.53−2.94

U04JUL0A.208, 1.88−3.06

U05JUL0A.113, 10.5−12.62

U06JUL0A.083, 0.1−2.07

U09JUL0A.094, 12.89−14.52

U10JUL0A.018, 0.1−1.2

U11JUL0A.034, 14.2−15.89

U16JUL0A.041, 8.35−9.78

U28JUN0A.033, 4.44−6.51; segm# 178; T=2.07 s ;σ=118 mm2/3; s=483 mm

100

y, [mm]

50

0

−50

−100

x, [mm], square. − segment start, circle − segment end.

100

50

0

−50

−100

100

50

0

−50

−100

100

50

0

−50

−100 2/3

U17JUL0A.134, 4.98−6.8; segm# 513; T=1.82 s ;σ=70 mm

; s=397 mm

100

50

0

−50

−100 U18JUL0A.037, 30.02−31.51

U20JUL0A.018, 7.37−9.2

100

50

0

−50

−100 −100

−50

0

50

100

−100

−50

0

50

100

Figure H.12: Monkey U, following extensive practice. Exemplar paths fitted with several parabolic pieces. One example for one recording day. Path pieces (blue) and parabolic pieces fitting them. Red - best fitting parabolic piece, green — parabolic piece defined as the best fitting second order polynomial in equi-affine length measured along the path.

= 37. 07jul97; segm.: Tgv.

Par. e−a.; Max

bin

= 50.

10 20 30 40 50

10 20 30 40 50

mean

=88.658

100

100

200

Par. eucl.; Maxbin = 130.

0

100

200

=87.6238


mean 100

200

0

100

200


mean

value

=100.2142

Eucl.; Maxbin = 279.

value

10 20 30 40 50

= 40.

value

=89.4728

value

0

mean 0

0

200

=102.2847

medianvalue=24.5732

medianvalue=25.2795 10 20 30 40 50

200


Equi−aff.; Maxbin = 246. 12jul97; segm.: Tgv. Par. e−a.; Maxbin = 295.

0

100

value

0

10 20 30 40 50

200 bin


mean

medianvalue=21.7974

medianvalue=21.8888 0

100 Par. eucl.; Max

mean 0

10 20 30 40 50

= 42.

value

=86.2965

value

10 20 30 40 50

0

= 42.

mean 0

10 20 30 40 50

200 bin

Equi−aff.; Maxbin = 164. 11jul97; segm.: Tgv. Par. e−a.; Maxbin = 203.

0

100 Eucl.; Max

Par. e−a.; Maxbin = 124.

medianvalue=21.8765


0

= 49.

mean 0

Equi−aff.; Maxbin = 91. 10jul97; segm.: Tgv.

bin

=83.2586

bin

Par. eucl.; Max

value

=76.5309 10 20 30 40 50 Par. e−a.; Max


= 40.

value

0

= 36. 08jul97; segm.: Tgv.

medianvalue=11.7237

bin

bin

=86.4064

10 20 30 40 50

Eucl.; Max

value

0

Equi−aff.; Max

237

mean

medianvalue=9.1832

medianvalue=15.9148

bin

H

mean

Equi−aff.; Max

Appendix

=73.9455


0

100

200

0

100

200

Figure H.13: Exemplar histograms of the metric parameters of the fitted parabolic pieces. The data corresponds to 5 days of recordings from the start of practice of monkey O. Each row corresponds to one recording day. First column: equi-affine lengths of the path pieces fitted with parabolas. Second column: equi-affine lengths of the fitted parabolic pieces. Third column: Euclidian length of the path pieces fitted with parabolas. Fourth column: Euclidian length of the fitted parabolic pieces. Vertical red lines designate median values of the parameter. Note that histograms for the equi-affine lengths are different, and the histograms for the Euclidian lengths are very similar. Differences in the equi-affine length mean inconsistency with parabolicity.


asym. t; Max

asym. σ; Max

= 11; 07jul97

H

238

= 12, remove 51.816%

0.5 asym. t; Max

1

−1

asym. σ; Max

= 20; 08jul97

0

1

0.5

1

−1

0

0.5

1

0.5

1

−1

0

−1

0

0.5

1

0

1

0.5

1

median=0.1169

non−parab.; Maxbin = 618

0

1

0

asym. σ; Maxbin = 300, remove 3.7797%

−1

0.5 non−parab.; Maxbin = 355

1

median=0.18012

median=0.34783

asym. t; Maxbin = 155; 12jul97

= 37

0.5

1

non−parab.; Maxbin = 1.38e+003

median=0.10711

0

0


median=0.20895

median=0.38462


1 bin

1

median=0.21026

median=0.40825 0

0.5 non−parab.; Max



= 35

median=0.12388

0

0

= 15, remove 46.374%

median=0.37115

bin

median=0.55848

bin

bin

median=0.39264

0

non−parab.; Max

median=0.4934

median=0.37054

bin

median=0.57143

bin

Appendix

0

1

0

0.5

Figure H.14: Exemplar histograms of the asymmetry and inconsistency of the fitted parabolic pieces. The data corresponds to 5 days of recordings from the start of practice of monkey O. Each row corresponds to one recording day. First column: temporal asymmetry, which estimates deviation from local isochrony for the vertex of parabola. Second column: equi-affine asymmetry. Third column: parabolic inconsistency. Note that parabolic inconsistency decreases at start of practice.

1

PHD thesis by Felix Polyakov #8, 21jul97

#6, 14jul97

H

#5, 12jul97

239

#4, 11jul97

#3, 10jul97

#2, 08jul97

#1, 07jul97

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

(a)

#9, 22jul97

Appendix

−0.1 Med. = −0.038 MAD = 0.03 me / ma = −1.3

Med. = −0.069 MAD = 0.038 me / ma = −1.8

peak=34

Med. = −0.028 MAD = 0.026 me / ma = −1.1

peak=42

Med. = −0.055 MAD = 0.035 me / ma = −1.6

peak=197

Med. = −0.021 MAD = 0.018 me / ma = −1.1

peak=389

0.1

Med. = −0.018 MAD = 0.016 me / ma = −1.1

peak=370

peak=527

peak=554

peak=485

0 min, κ Med. = −0.018 MAD = 0.017 me / ma = −1.1

Med. = −0.07 MAD = 0.037 me / ma = −1.9

−0.1 Med. = 0.034 MAD = 0.033 me / ma = 1

Med. = 0.027 MAD = 0.054 me / ma = 0.5

peak=40

Med. = 0.03 MAD = 0.028 me / ma = 1.1

peak=50

Med. = 0.05 MAD = 0.032 me / ma = 1.6

peak=187

Med. = 0.025 MAD = 0.022 me / ma = 1.1

peak=341

0.1

Med. = 0.018 MAD = 0.021 me / ma = 0.85

peak=447

peak=407

peak=408

peak=376

0 max, κ Med. = 0.017 MAD = 0.023 me / ma = 0.77

Med. = 0.024 MAD = 0.056 me / ma = 0.42

−0.1 Med. = −0.036 MAD = 0.029 me / ma = −1.2

Med. = −0.042 MAD = 0.042 me / ma = −1

peak=28

Med. = −0.029 MAD = 0.024 me / ma = −1.2

peak=38

Med. = −0.051 MAD = 0.033 me / ma = −1.5

peak=239

Med. = −0.019 MAD = 0.016 me / ma = −1.2

peak=490

0.1

Med. = −0.014 MAD = 0.012 me / ma = −1.1

peak=375

peak=687

peak=881

peak=742

0 min, κ’ Med. = −0.013 MAD = 0.013 me / ma = −1

Med. = −0.048 MAD = 0.046 me / ma = −1

−0.1 #14, 01aug97

#13, 31jul97

Med. = 0.044 MAD = 0.042 me / ma = 1

#12, 29jul97

Med. = 0.051 MAD = 0.046 me / ma = 1.1

#11, 25jul97

#10, 23jul97

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

(b)

#15, 03aug97

Med. = 0.033 MAD = 0.028 me / ma = 1.2

peak=30

Med. = 0.025 MAD = 0.021 me / ma = 1.2

peak=40

#16, 04aug97

Med. = 0.048 MAD = 0.032 me / ma = 1.5

peak=252

#17, 05aug97

Med. = 0.019 MAD = 0.016 me / ma = 1.2

peak=509

#18, 06aug97

peak=377

0.1

Med. = 0.014 MAD = 0.013 me / ma = 1.1

peak=723

peak=777

peak=704

0 max, κ’ Med. = 0.014 MAD = 0.014 me / ma = 1

−0.1 Med. = −0.014 MAD = 0.012 me / ma = −1.1

peak=364

Med. = −0.014 MAD = 0.012 me / ma = −1.2

peak=1357

Med. = −0.014 MAD = 0.013 me / ma = −1.1

peak=2405

Med. = −0.014 MAD = 0.013 me / ma = −1.1

peak=2701

Med. = −0.015 MAD = 0.013 me / ma = −1.2

peak=1601

Med. = −0.015 MAD = 0.013 me / ma = −1.2

peak=2827

0.1

Med. = −0.019 MAD = 0.015 me / ma = −1.2

peak=2655

peak=1365

peak=2290

0 min, κ Med. = −0.016 MAD = 0.014 me / ma = −1.1

Med. = −0.015 MAD = 0.014 me / ma = −1

−0.1 Med. = 0.014 MAD = 0.018 me / ma = 0.76

peak=283

Med. = 0.015 MAD = 0.016 me / ma = 0.92

peak=975

Med. = 0.012 MAD = 0.018 me / ma = 0.67

peak=1906

Med. = 0.013 MAD = 0.021 me / ma = 0.63

peak=2095

Med. = 0.016 MAD = 0.019 me / ma = 0.87

peak=1171

Med. = 0.014 MAD = 0.018 me / ma = 0.77

peak=2111

0.1

Med. = 0.011 MAD = 0.021 me / ma = 0.51

peak=1945

peak=1170

peak=1744

0 max, κ Med. = 0.013 MAD = 0.019 me / ma = 0.7

Med. = 0.015 MAD = 0.019 me / ma = 0.8

−0.1 Med. = −0.0094 MAD = 0.0085 me / ma = −1.1

peak=713

Med. = −0.012 MAD = 0.0098 me / ma = −1.2

peak=2576

Med. = −0.0097 MAD = 0.0092 me / ma = −1.1

peak=3717

Med. = −0.01 MAD = 0.0099 me / ma = −1

peak=4220

Med. = −0.013 MAD = 0.011 me / ma = −1.2

peak=2584

Med. = −0.011 MAD = 0.01 me / ma = −1.1

peak=4299

0.1

Med. = −0.012 MAD = 0.011 me / ma = −1

peak=3856

peak=2404

peak=4197

0 min, κ’ Med. = −0.01 MAD = 0.0096 me / ma = −1.1

Med. = −0.01 MAD = 0.0095 me / ma = −1.1

−0.1 Med. = 0.01 MAD = 0.0089 me / ma = 1.1

peak=689

Med. = 0.012 MAD = 0.01 me / ma = 1.2

peak=2466

Med. = 0.01 MAD = 0.0097 me / ma = 1.1

peak=3467

Med. = 0.011 MAD = 0.011 me / ma = 1

peak=4029

Med. = 0.013 MAD = 0.011 me / ma = 1.2

peak=2356

Med. = 0.011 MAD = 0.011 me / ma = 1.1

peak=4152

0.1

Med. = 0.012 MAD = 0.012 me / ma = 0.99

peak=3761

peak=2351

peak=4207

0 max, κ’ Med. = 0.011 MAD = 0.011 me / ma = 1

Med. = 0.011 MAD = 0.011 me / ma = 1

Figure H.15: Monkey O, consecutive days from start of practice. Histograms of the values of minima and maxima of κ and κ0 on the segments of motion (separated by the segments of no motion). Each row in (a) or in (b) corresponds to a day of recordings. First column: minima of κ, second column: maxima of κ, third column: minima of κ0 , fourth column: maxima of κ0 . Green: negative values, red: positive values. The values in the histograms are scaled so that the value at peak equals 1, the scaling factor is shown in the title of each plot. Median value is depicted by the vertical blue line, dashed-dotted vertical lines depict the range of the maximal absolute deviation MAD = median(median(data) − data). The values of the median, MAD and their ratio (median / MAD) are depicted in every plot. By ratio (median / MAD) we estimate the tendency of a function to keep the sign of the corresponding extremum constant, that is to have negative minima or to have positive maxima.

PHD thesis by Felix Polyakov #8, u21jun99

#7, u18jun99

#6, u17jun99

H

240

#5, u16jun99

#4, u13jun99

#3, u12jun99

#2, u11jun99

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

(a)

#9, u22jun99

Appendix

−0.1 Med. = −0.083 MAD = 0.037 me / ma = −2.3

peak=4

Med. = −0.047 MAD = 0.033 me / ma = −1.4

peak=12

Med. = −0.05 MAD = 0.031 me / ma = −1.6

peak=112

Med. = −0.058 MAD = 0.032 me / ma = −1.8

peak=204

Med. = −0.051 MAD = 0.031 me / ma = −1.7

peak=155

0.1

Med. = −0.051 MAD = 0.033 me / ma = −1.5

peak=157

peak=282

peak=384

0 min, κ Med. = −0.039 MAD = 0.028 me / ma = −1.4

Med. = −0.072 MAD = 0.039 me / ma = −1.9

−0.1 Med. = 0.04 MAD = 0.057 me / ma = 0.71

peak=4

Med. = 0.024 MAD = 0.038 me / ma = 0.62

peak=14

Med. = 0.018 MAD = 0.035 me / ma = 0.52

peak=127

Med. = 0.022 MAD = 0.039 me / ma = 0.58

peak=241

Med. = 0.017 MAD = 0.035 me / ma = 0.48

peak=209

0.1

Med. = 0.026 MAD = 0.038 me / ma = 0.69

peak=208

peak=301

peak=442

0 max, κ Med. = 0.018 MAD = 0.031 me / ma = 0.59

Med. = 0.015 MAD = 0.053 me / ma = 0.28

−0.1 Med. = −0.056 MAD = 0.05 me / ma = −1.1

peak=3

Med. = −0.034 MAD = 0.032 me / ma = −1.1

peak=10

Med. = −0.032 MAD = 0.031 me / ma = −1.1

peak=138

Med. = −0.039 MAD = 0.035 me / ma = −1.1

peak=227

Med. = −0.031 MAD = 0.029 me / ma = −1

peak=163

0.1

Med. = −0.037 MAD = 0.033 me / ma = −1.1

peak=188

peak=299

peak=527

0 min, κ’ Med. = −0.027 MAD = 0.025 me / ma = −1.1

Med. = −0.05 MAD = 0.049 me / ma = −1

−0.1 #12, u27jun99

Med. = 0.062 MAD = 0.043 me / ma = 1.5

#11, u24jun99

#10, u23jun99

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

(b)

#13, u28jun99

Med. = 0.046 MAD = 0.054 me / ma = 0.86

peak=3

#14, u29jun99

Med. = 0.035 MAD = 0.032 me / ma = 1.1

peak=9

#15, u01jul99

Med. = 0.034 MAD = 0.031 me / ma = 1.1

peak=135

#16, u05jul99

Med. = 0.04 MAD = 0.033 me / ma = 1.2

peak=232

#17, u06jul99

Med. = 0.031 MAD = 0.029 me / ma = 1.1

peak=180

0.1

Med. = 0.037 MAD = 0.032 me / ma = 1.1

peak=187

peak=297

peak=539

0 max, κ’ Med. = 0.027 MAD = 0.025 me / ma = 1.1

−0.1 Med. = −0.048 MAD = 0.031 me / ma = −1.5

peak=342

Med. = −0.05 MAD = 0.032 me / ma = −1.6

peak=192

Med. = −0.049 MAD = 0.031 me / ma = −1.5

peak=228

Med. = −0.052 MAD = 0.032 me / ma = −1.6

peak=221

Med. = −0.051 MAD = 0.033 me / ma = −1.6

peak=246

0.1

Med. = −0.056 MAD = 0.032 me / ma = −1.7

peak=277

peak=236

peak=229

0 min, κ Med. = −0.052 MAD = 0.031 me / ma = −1.7

Med. = −0.041 MAD = 0.029 me / ma = −1.4

−0.1 Med. = 0.02 MAD = 0.035 me / ma = 0.57

peak=491

Med. = 0.018 MAD = 0.034 me / ma = 0.51

peak=252

Med. = 0.029 MAD = 0.035 me / ma = 0.83

peak=301

Med. = 0.03 MAD = 0.036 me / ma = 0.84

peak=243

Med. = 0.027 MAD = 0.037 me / ma = 0.73

peak=269

0.1

Med. = 0.04 MAD = 0.037 me / ma = 1.1

peak=320

peak=235

peak=205

0 max, κ Med. = 0.032 MAD = 0.038 me / ma = 0.84

Med. = 0.014 MAD = 0.03 me / ma = 0.45

−0.1 Med. = −0.035 MAD = 0.03 me / ma = −1.1

peak=509

Med. = −0.034 MAD = 0.031 me / ma = −1.1

peak=266

Med. = −0.042 MAD = 0.033 me / ma = −1.3

peak=280

Med. = −0.042 MAD = 0.033 me / ma = −1.2

peak=218

Med. = −0.039 MAD = 0.033 me / ma = −1.2

peak=239

0.1

Med. = −0.051 MAD = 0.035 me / ma = −1.5

peak=301

peak=192

peak=198

0 min, κ’ Med. = −0.046 MAD = 0.034 me / ma = −1.4

Med. = −0.027 MAD = 0.026 me / ma = −1

−0.1 Med. = 0.034 MAD = 0.03 me / ma = 1.1

peak=549

Med. = 0.034 MAD = 0.03 me / ma = 1.1

peak=269

Med. = 0.039 MAD = 0.032 me / ma = 1.2

peak=280

Med. = 0.041 MAD = 0.033 me / ma = 1.2

peak=248

Med. = 0.037 MAD = 0.031 me / ma = 1.2

peak=270

0.1

Med. = 0.05 MAD = 0.034 me / ma = 1.5

peak=334

peak=197

peak=183

0 max, κ’ Med. = 0.045 MAD = 0.034 me / ma = 1.3

Med. = 0.027 MAD = 0.025 me / ma = 1.1

Figure H.16: Monkey U, consecutive days from start of practice. Histograms of the values of minima and maxima of κ and κ0 on the segments of motion (separated by the segments of no motion). Each row in (a) or in (b) corresponds to a day of recordings. First column: minima of κ, second column: maxima of κ, third column: minima of κ0 , fourth column: maxima of κ0 . Green: negative values, red: positive values. The values in the histograms are scaled so that the value at peak equals 1, the scaling factor is shown in the title of each plot. Median value is depicted by the vertical blue line, dashed-dotted vertical lines depict the range of the maximal absolute deviation MAD = median(median(data) − data). The values of the median, MAD and their ratio (median / MAD) are depicted in every plot. By ratio (median / MAD) we estimate the tendency of a function to keep the sign of the corresponding extremum constant, that is to have negative minima or to have positive maxima.

PHD thesis by Felix Polyakov #8, 04jul00

#7, 03jul00

#6, 02jul00

H

#5, 30jun00

241

#4, 29jun00

#3, 28jun00

#2, 27jun00

#1, 26jun00

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

(a)

#9, 05jul00

Appendix

−0.1 Med. = −0.031 MAD = 0.026 me / ma = −1.2

Med. = −0.034 MAD = 0.027 me / ma = −1.2

peak=292

Med. = −0.031 MAD = 0.027 me / ma = −1.1

peak=436

Med. = −0.026 MAD = 0.023 me / ma = −1.1

peak=458

Med. = −0.045 MAD = 0.032 me / ma = −1.4

peak=421

Med. = −0.035 MAD = 0.028 me / ma = −1.2

peak=715

0.1

Med. = −0.04 MAD = 0.032 me / ma = −1.3

peak=340

peak=406

peak=530

peak=723

0 min, κ Med. = −0.034 MAD = 0.026 me / ma = −1.3

Med. = −0.034 MAD = 0.028 me / ma = −1.2

−0.1 Med. = 0.026 MAD = 0.024 me / ma = 1.1

Med. = 0.025 MAD = 0.026 me / ma = 0.97

peak=337

Med. = 0.023 MAD = 0.024 me / ma = 0.97

peak=543

Med. = 0.025 MAD = 0.02 me / ma = 1.3

peak=570

Med. = 0.034 MAD = 0.03 me / ma = 1.1

peak=542

Med. = 0.028 MAD = 0.026 me / ma = 1.1

peak=873

0.1

Med. = 0.032 MAD = 0.029 me / ma = 1.1

peak=423

peak=493

peak=613

peak=783

0 max, κ Med. = 0.029 MAD = 0.025 me / ma = 1.2

Med. = 0.026 MAD = 0.027 me / ma = 0.95

−0.1 Med. = −0.024 MAD = 0.021 me / ma = −1.1

Med. = −0.026 MAD = 0.023 me / ma = −1.1

peak=479

Med. = −0.021 MAD = 0.02 me / ma = −1.1

peak=736

Med. = −0.022 MAD = 0.019 me / ma = −1.2

peak=796

Med. = −0.034 MAD = 0.028 me / ma = −1.2

peak=746

Med. = −0.029 MAD = 0.024 me / ma = −1.2

peak=1061

0.1

Med. = −0.033 MAD = 0.027 me / ma = −1.2

peak=451

peak=610

peak=648

peak=929

0 min, κ’ Med. = −0.03 MAD = 0.025 me / ma = −1.2

Med. = −0.024 MAD = 0.021 me / ma = −1.1

−0.1 Med. = 0.026 MAD = 0.023 me / ma = 1.1

#12, 10jul00

Med. = 0.025 MAD = 0.023 me / ma = 1.1

#11, 09jul00

#10, 06jul00

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

1

0.5

0

(b)

#13, 11jul00

Med. = 0.024 MAD = 0.021 me / ma = 1.1

peak=457

#14, 16jul00

Med. = 0.022 MAD = 0.02 me / ma = 1.1

peak=746

#15, 17jul00

Med. = 0.022 MAD = 0.019 me / ma = 1.2

peak=795

#16, 18jul00

Med. = 0.036 MAD = 0.03 me / ma = 1.2

peak=782

#17, 20jul00

Med. = 0.03 MAD = 0.026 me / ma = 1.2

peak=1176

0.1

Med. = 0.034 MAD = 0.027 me / ma = 1.2

peak=402

peak=594

peak=650

peak=978

0 max, κ’ Med. = 0.03 MAD = 0.024 me / ma = 1.2

−0.1 Med. = −0.031 MAD = 0.025 me / ma = −1.3

peak=560

Med. = −0.047 MAD = 0.033 me / ma = −1.4

peak=594

Med. = −0.04 MAD = 0.029 me / ma = −1.4

peak=225

Med. = −0.039 MAD = 0.03 me / ma = −1.3

peak=329

Med. = −0.035 MAD = 0.028 me / ma = −1.2

peak=428

0.1

Med. = −0.031 MAD = 0.025 me / ma = −1.2

peak=736

peak=369

peak=540

0 min, κ Med. = −0.032 MAD = 0.025 me / ma = −1.3

Med. = −0.034 MAD = 0.027 me / ma = −1.3

−0.1 Med. = 0.026 MAD = 0.023 me / ma = 1.1

peak=640

Med. = 0.034 MAD = 0.034 me / ma = 1

peak=709

Med. = 0.032 MAD = 0.028 me / ma = 1.1

peak=241

Med. = 0.03 MAD = 0.028 me / ma = 1.1

peak=408

Med. = 0.028 MAD = 0.026 me / ma = 1.1

peak=496

0.1

Med. = 0.024 MAD = 0.025 me / ma = 0.97

peak=940

peak=420

peak=516

0 max, κ Med. = 0.026 MAD = 0.026 me / ma = 1

Med. = 0.028 MAD = 0.025 me / ma = 1.1

−0.1 Med. = −0.027 MAD = 0.022 me / ma = −1.2

peak=760

Med. = −0.041 MAD = 0.031 me / ma = −1.3

peak=808

Med. = −0.034 MAD = 0.027 me / ma = −1.3

peak=218

Med. = −0.033 MAD = 0.027 me / ma = −1.2

peak=426

Med. = −0.029 MAD = 0.025 me / ma = −1.2

peak=567

0.1

Med. = −0.025 MAD = 0.022 me / ma = −1.1

peak=975

peak=548

peak=732

0 min, κ’ Med. = −0.027 MAD = 0.022 me / ma = −1.2

Med. = −0.029 MAD = 0.024 me / ma = −1.2

−0.1 Med. = 0.041 MAD = 0.032 me / ma = 1.3

Med. = 0.027 MAD = 0.022 me / ma = 1.2

peak=756

Med. = 0.035 MAD = 0.028 me / ma = 1.3

peak=873

Med. = 0.033 MAD = 0.028 me / ma = 1.2

peak=233

Med. = 0.029 MAD = 0.024 me / ma = 1.2

peak=424

0.1

Med. = 0.025 MAD = 0.022 me / ma = 1.2

peak=563

peak=1088

peak=577

peak=740

0 max, κ’ Med. = 0.027 MAD = 0.022 me / ma = 1.2

Med. = 0.03 MAD = 0.025 me / ma = 1.2

Figure H.17: mycommentThe plots were generated in the function dac peaks.m . Monkey U, consecutive days following a period of extensive practice. Histograms of the values of minima and maxima of κ and κ0 on the segments of motion (separated by the segments of no motion). Each row in (a) or in (b) corresponds to a day of recordings. First column: minima of κ, second column: maxima of κ, third column: minima of κ0 , fourth column: maxima of κ0 . Green: negative values, red: positive values. The values in the histograms are scaled so that the value at peak equals 1, the scaling factor is shown in the title of each plot. Median value is depicted by the vertical blue line, dashed-dotted vertical lines depict the range of the maximal absolute deviation MAD = median(median(data) − data). The values of the median, MAD and their ratio (median / MAD) are depicted in every plot. By ratio (median / MAD) we estimate the tendency of a function to keep the sign of the corresponding extremum constant, that is to have negative minima or to have positive maxima.


H.2

Chapter 3

Appendix

H

242

PHD thesis by Felix Polyakov Day 1, 26jun00; min = −0.77; max = 0.79

# of file in a day

45

Appendix

Day 2, 27jun00; min = −0.783; max = 0.77

45

35

40

30

35

50

40

30

25

30

25

20

20 20

15

15

10

10

5

10

5

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 4, 29jun00; min = −0.788; max = 0.78 55

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 5, 30jun00; min = −0.808; max = 0.81 60

45

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 6, 02jul00; min = −0.769; max = 0.83 70

50

# of file in a day

243

60

50

40

H Day 3, 28jun00; min = −0.782; max = 0.78

60

50

40 50

40

35 30

40 30

25

30

20

20

15

20

10

10

10

5 −0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 7, 03jul00; min = −0.782; max = 0.77

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 8, 04jul00; min = −0.786; max = 0.83

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 9, 05jul00; min = −0.842; max = 0.83 90

100

60

80 90

# of file in a day

50

70

80

40

70

60

60

50

50

30

40

40 20

30

30 20

20

10

10

10 −0.3

−0.2

−0.1

0

0.1

0.2

0.3

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 10, 06jul00; min = −0.824; max = 0.85

Day 11, 09jul00; min = −0.847; max = 0.83

60

60

50

50

40

40

30

30

20

20

10

10

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 12, 10jul00; min = −0.752; max = 0.8 45

# of file in a day

40 35 30 25 20 15 10 5 −0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 13, 11jul00; min = −0.763; max = 0.78

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 15, 17jul00; min = −0.809; max = 0.84

Day 14, 16jul00; min = −0.769; max = 0.8 100

60

45

−0.3

90

# of file in a day

40

50

80

35

70

40

30

60 50

25

30 40

20

30

20

15

20

10

10

10

5 −0.3

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

−0.2

−0.1

0

tvel − tacc

0.1

0.2

0.3

Day 16, 18jul00; min = −0.782; max = 0.84

Day 17, 20jul00; min = −0.77; max = 0.81 1

40 60

0.8

# of file in a day

35

0.6

50

30

0.4

25

40

20

30

0.2 0 −0.2

15

−0.4

20

10

−0.6 10

−0.8

5 −0.3

−0.2

−0.1

0

tvel − tacc

0.1

0.2

0.3

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

−1

t

− tacc vel

Figure H.18: Data for monkey U, following extensive practice. Correlation coefficients for the files with scribbling movements within a day. The values of the strongest negative and the strongest positive correlation coefficients are shown in the title of every plot. The abscissa axis corresponds to the time-shift between the cosine of the direction of acceleration signal and the cosine of the direction of velocity signal. The ordinate stands for the index of the file with the data. The time-delay of about 300 msec. between the two signals corresponds to strongest correlation and correlation is quite high, which is compatible with the corresponding delays and correlations in (Stark et al., 2006), figure 1B.

PHD thesis by Felix Polyakov Day 1, 26jun00; min = −0.286; max = 0.8 45

H

244

Day 2, 27jun00; min = −0.202; max = 0.71

45

35

40

30

35

25

30

50

40

30

25

20

Day 3, 28jun00; min = −0.284; max = 0.82 60

50

40

# of file in a day

Appendix

20 20

15 15

10

10

5 −0.3

10

5

−0.2

−0.1

0

0.1

0.2

0.3

Day 4, 29jun00; min = −0.186; max = 0.84

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 5, 30jun00; min = −0.212; max = 0.82

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 6, 02jul00; min = −0.221; max = 0.83

55 60

50

# of file in a day

45

70 60

50

40 35

50

40

30

40 30

25

30

20 20

15

20

10

10

10

5 −0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 7, 03jul00; min = −0.326; max = 0.73

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 8, 04jul00; min = −0.326; max = 0.7

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 9, 05jul00; min = −0.311; max = 0.83 90

100

60

80 90

# of file in a day

50

70

80

40

70

60

60

50

50

30

40

40

20

30

30 20

20

10

10

10

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 10, 06jul00; min = −0.258; max = 0.76 60

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 11, 09jul00; min = −0.242; max = 0.68

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 12, 10jul00; min = −0.304; max = 0.7 45

60

# of file in a day

40

50

50

40

40

30

30

20

20

10

10

35 30 25 20 15 10 5

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 13, 11jul00; min = −0.19; max = 0.7

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 14, 16jul00; min = −0.167; max = 0.72

−0.2

−0.1

0

0.1

0.2

0.3

Day 15, 17jul00; min = −0.54; max = 0.66 100

60

45

−0.3

90

# of file in a day

40

50

80

35

70

40

30

60 50

25

30

40

20

30

20

15

20

10

10

10

5 −0.3

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

Day 16, 18jul00; min = −0.134; max = 0.74

−0.3

−0.2

−0.1

0

0.1

0.2

0.3

−0.2

−0.1

t

tang. vel.

0

− taff.

0.1

0.2

0.3

vel.

Day 17, 20jul00; min = −0.197; max = 0.66 1

40 60

0.8

# of file in a day

35 0.6

50

30

0.4 40

25

0.2 0

20

30 −0.2

15 −0.4

20

10

−0.6 10

5 −0.3

−0.2

−0.1

t

tang.

0

− taff. vel.

0.1

vel.

0.2

0.3

−0.3

−0.8 −0.2

−0.1

t

tang.

0

0.1

− taff. vel.

0.2

0.3

−1

vel.

Figure H.19: Data for monkey U, following extensive practice. Correlation coefficients for the files with scribbling movements within a day. The values of the strongest negative and the strongest positive correlation coefficients are shown in the title of every plot. Abscissa axis corresponds to the time-shift between the tangential and equiaffine velocities. Ordinate stands for the index of the file with the data. The two velocities are stronger correlated for smaller time-shifts. The correlation is a bit weaker than that for the direction of acceleration and direction of velocity, figure H.18 but still is relatively high.

PHD thesis by Felix Polyakov Appendix H (ℵ) 27jun00, day # 2; unit: 10-12 (i) 28jun00, day # 3; unit: 40 27jun;10_13;#1;D:20;m

t(ρ(θ

vel

| . ));t(ρ(speed | . ))

20

0.3

28jun;40;#1;D:22;m

0.2

0.2

0.1

0.1

10

0

−0.1

−0.2

−0.2

−0.2

−0.1

0

0.3

0.1

| . )) τacc acc

t(ρ(θ

0.2

0.3 −0.3

−0.2

−0.1

0

0.1

t(ρ(|acc| τ | . ))

0.2

0.3

acc

0 20

−0.3 −0.3

0.2

0.1

0.1

10

−0.1

−0.2

−0.2

−0.2

−0.1

A

0

τacc

0.1

0.2

0.3 −0.3

−0.2

−0.1

0

τacc

0.1

0.2

0.3

0

−0.3 −0.3

27jun;10_13;#1;D:20;m t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . )) 0.3

−0.2

−0.1

0

0.1

| . )) τacc acc

t(ρ(θ

0.2

0.3 −0.3

−0.2

−0.1

0

0.1

t(ρ(|acc| τ | . ))

0.2

0.3

10 5 −0.2

−0.1

0

τacc

0.1

0.2

0.3 −0.3

−0.2

−0.1

0

τacc

0.1

0.2

0.3

0

t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . ))

0.3

10

0.2

0.1

0

acc

28jun;40;#1;D:22;m

10

0.2

0.1

5

0 −0.1

0

5

−0.1

−0.2

−0.2

−0.3 −0.3

−0.2

B

−0.1

0

τ

0.1

0.2

0.3 −0.3

−0.2

[reg]

∆σ

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0

−0.3 −0.3

−0.2

−0.1

τ

0

0.1

0.2

0.3 −0.3

−0.2

[reg]

∆σ

27jun;10_13;#1;D:20;m t(ρ(θV[r] | . ));t(ρ(speed[r] | . ))

28jun;40;#1;D:22;m

0.3

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

15

0.1

12 10 8 6 4 2

0.2 0.1

10

0 −0.1

5

−0.2 −0.3 −0.3

−0.2

0.3

t(ρ(∆σ[r] |0.1. )) 0.2 −0.1 0

τ

0.3 −0.3

−0.2

[reg]

∆σ

0.2

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0

0 −0.1 −0.2 −0.3 −0.3

−0.2

0.3

t(ρ(∆σ[r] |0.1. )) 0.2 −0.1 0

τ

0.3 −0.3

−0.2

[reg]

∆σ

0.2

0.1

0

t(ρ(θV[r] | . ));t(ρ(speed[r] | . ))

0.3

0.2

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0.1

0

0

−0.1

−0.1

−0.2

−0.2

−0.3 −0.3

−0.2

C

−0.1

0

0.1

τ∆σ[reg]

0.2

−0.3 −0.3

0.3

27jun;10_13;#1;D:20;m

t(ρ(θV[r] | . ));t(ρ(∆σ[r] | . ))

15

0.1

−0.1

5

−0.2

0.1

0.2

0.3

t(ρ(θV[r] | . ));t(ρ(∆σ[r] | . ))

0.2

10

−0.2

−0.1

0

0.1

0.2

| . )) τt(ρ(speed[r] [reg] speed

0.3 −0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3

0

0

−0.2 −0.3 −0.3

0.3 0.2

0.1

0.1

0

0

−0.1

−0.1

−0.2

5

−0.1

0.2

−0.3 −0.3

0

0.1

10

0

0.3

−0.1

τ∆σ[reg]

0.3

0.2

−0.3 −0.3

−0.2

28jun;40;#1;D:22;m

0.3

D

5

0

−0.1

−0.3 −0.3

| . ));t(ρ(speed | . ))

10

0.3

0.2

0

vel

0

−0.1

−0.3 −0.3

t(ρ(θ

0.3

−0.2

−0.1

0

0.1

0.2


0.3 −0.3

−0.2

−0.1

0

0.1

−0.2

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3

−0.3 −0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.2

τspeed[reg]

0.3

Figure H.20: PCCMs, see explanations in the text, page 81.

0.3

0

245

PHD thesis by Felix Polyakov Appendix H 246 (ℵ) 03jul00, day # 7; unit: 10 (i) 03jul00, day # 7; unit: 61-63 03jul;10;#1;D:36;m

t(ρ(θ

vel

| . ));t(ρ(speed | . ))

03jul;61_63;#6;D:31;m

0.3

14 12 10 8 6 4 2

0.2 0.1 0 −0.1 −0.2 −0.3 −0.3

−0.2

−0.1

0

0.3

0.1

| . )) τacc acc

t(ρ(θ

0.2

0.3 −0.3

−0.2

−0.1

0

0.1

t(ρ(|acc| τ | . ))

0.2

acc

14 12 10 8 6 4 2

0.2 0.1 0 −0.1 −0.2 −0.3 −0.3

−0.2

−0.1

A

0

τacc

0.1

0.2

03jul;10;#1;D:36;m

0.3 −0.3

−0.2

−0.1

0

τacc

0.1

0.2

6 4

0

2

−0.2

−0.2

B

−0.1

0

τ

0.1

0.2

0.3 −0.3

−0.2

[reg]

∆σ

03jul;10;#1;D:36;m

−0.1

0

0.1

0.2

τ∆σ[reg]

0.3

0

0.1

0.2

0.3 −0.3

−0.2

−0.1

0

0.1

t(ρ(|acc| τ | . ))

0.2

0.3

0

acc

20

0.1 0

10

−0.1 −0.2

−0.2

−0.1

0

τacc

0.1

0.2

0.3 −0.3

−0.2

−0.1

0

τacc

0.1

0.2

0.3

0

15

0.2

0 −0.1 −0.2

−0.2

0.3

−0.1 0 t(ρ(∆σ[r] |0.1. )) 0.2

τ

0.3 −0.3

−0.2

[reg]

∆σ

0.2

−0.1

0

0.1

τ∆σ[reg]

10

0 −0.1

5

−0.2 −0.3 −0.3

−0.2

−0.1

τ

0

0.1

0.2

0.3 −0.3

−0.2

[reg]

0.2

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0

03jul;61_63;#6;D:31;m t(ρ(θV[r] | . ));t(ρ(speed[r] | . ))

12 10 8 6 4 2

0.1

0.3 0.2

20

0.1 0

10

−0.1 −0.2 −0.3 −0.3

0.3

−0.2

0.3

−0.1 0 t(ρ(∆σ[r] |0.1. )) 0.2

τ

0.3 −0.3

−0.2

[reg]

∆σ

0.2

0.1

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0

0.1

0

0

−0.1

−0.1

−0.2

−0.2

−0.2

C

−0.1

0

0.1

τ∆σ[reg]

0.2

03jul;10;#1;D:36;m

−0.3 −0.3

0.3

t(ρ(θV[r] | . ));t(ρ(∆σ[r] | . ))

0.3

0.1 0 −0.1 −0.2

−0.2

−0.1

0

0.1

0.2


0.3 −0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

t(ρ(θV[r] | . ));t(ρ(∆σ[r] | . ))

0.3

20

0.2 0.1 0

10

−0.1 −0.2 −0.3 −0.3

0.3

0.2

0.2

0.1

0.1

0

0

−0.1

−0.1

−0.2

−0.2

03jul;61_63;#6;D:31;m

12 10 8 6 4 2

0.2

−0.3 −0.3

0

| . )) τacc acc

0.2

t(ρ(θV[r] | . ));t(ρ(speed[r] | . ))

0.2

0.3

−0.1

t(ρ(θ

∆σ

0.3

−0.3 −0.3

−0.2

0.3

0.1

−0.1

−0.3 −0.3

10

−0.2

03jul;61_63;#6;D:31;m t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . ))

0.1

D

0

0.3

0.2

−0.3 −0.3

20

−0.1

t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . ))

0.3

−0.3 −0.3

| . ));t(ρ(speed | . ))

0.1

−0.3 −0.3

0.3

vel

0.2

−0.3 −0.3

0.3

t(ρ(θ

0.3

−0.2

−0.1

0

0.1

0.2


0.3 −0.3

−0.2

−0.1

0

0.1

−0.2

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3

−0.3 −0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.2

τspeed[reg]

0.3


0.3

0

PHD thesis by Felix Polyakov Appendix H 247 (ℵ) 17jul00, day # 15; unit: 51-52 (i) 28jun00, day # 3; unit: 61-63 17jul;5101_5102;#6;D:85;mt(ρ(θvel | . ));t(ρ(speed | . ))

20

0.1 0

10

−0.2

30

0.2

20

0 −0.1

10

−0.2

−0.2

t(ρ(θ −0.1

0 acc

| .0.1))

τacc

0.3 0.2

0.2

0.3 −0.3

−0.2

t(ρ(|acc| | 0.1 . )) −0.1 0

τacc

0.2

0.3

0 20

0.1

−0.3 −0.3

−0.2

10

−0.1 −0.2

−0.2

−0.1

0

τacc

0.1

0.2

0.3 −0.3

−0.2

−0.1

0

τacc

0.1

0.2

0.3

0.2

0

−0.3 −0.3

0.2 0.1

5

−0.1

−0.2

−0.2

−0.2

B

−0.1

0

τ

0.1

0.2

0.3 −0.3

−0.2

[reg]

∆σ

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0

−0.3 −0.3

τacc

0.2

0.3

0 30 20 10

−0.2

−0.1

0

0.1

τacc

0.2

0.3 −0.3

−0.2

−0.1

0

τacc

0.1

0.2

0.3

0

t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . ))

30 20 10 −0.2

−0.1

τ

0

0.1

0.2

0.3 −0.3

−0.2

[reg]

∆σ

t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . )) 17jul;5101_5102;#6;D:85;m

28jun;70;#7;D:24;m

0.3

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0

t(ρ(θV[r] | . ));t(ρ(speed[r] | . ))

0.3

0.2

20

0.1

10

0

30

0.2

15

0.1

0

−0.1

10

−0.1

5

−0.2

−0.2

0.3

t(ρ(θ [r] |0.1. )) 0.2 −0.1 V0

τ

0.3 −0.3

−0.2

[reg]

∆σ

0.2

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

−0.2

0

−0.3 −0.3

−0.2 −0.1 t(ρ(∆σ[r] | 0. )) 0.1

τ

0.3

0.2

0.3 −0.3

−0.2

[reg]

∆σ

0.2

0.1

−0.1

0

0.1

τ∆σ[reg]

0.2

0.3

0.1

0

0

−0.1

−0.1

−0.2

−0.2

−0.2

−0.1

0

0.1

0.2

t(ρ(∆σ[r] | . ));t(ρ(speed[r] | . )) 17jul;5101_5102;#6;D:85;m τ [reg]

∆σ

0.3

−0.3 −0.3

0.3

20

0.2

−0.2

−0.1

0

0.1

0.2

28jun;70;#7;D:24;m τ [reg]

∆σ

0.3

0.3

t(ρ(θV[r] | . ));t(ρ(∆σ[r] | . ))

30

0.2

0.1

20

0.1

10

0

D

t(ρ(|acc| | 0.1 . )) −0.1 0

0

−0.1

0

−0.1

−0.1

−0.2

−0.2

−0.2

−0.1

0

0.1

0.2

t(ρ(θV[r] | . )) τspeed [reg]

0.3 −0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3

0

−0.3 −0.3

0.3

0.2

0.2

0.1

0.1

0

0

−0.1

−0.1

−0.2

−0.2

−0.3 −0.3

−0.2

0.3

0.1

0.3

0.3 −0.3

0

0.2

−0.3 −0.3

0.2

28jun;70;#7;D:24;m

10

0

C

| .0.1))

−0.1

t(ρ(speed[r] | . ));t(ρ(∆σ[r] | . )) 17jul;5101_5102;#6;D:85;m 0.3

−0.3 −0.3

0 acc

τacc

−0.2

A

−0.3 −0.3

t(ρ(θ −0.1

0.3

0.1

0

−0.3 −0.3

| . ));t(ρ(speed | . ))

0.1

−0.1

−0.3 −0.3

vel

0.3

0.2

−0.3 −0.3

t(ρ(θ

28jun;70;#7;D:24;m

0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3

−0.3 −0.3

10 −0.2

−0.1

0

0.1

0.2


−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3 −0.3

−0.2

−0.1

0

0.1

0.2

τspeed[reg]

0.3


0.3

0


Appendix

H

248

27jun00. Unit: 10−13, # 1; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 181.2; Tgv

x 7.83

x 2960

1


Scaled

0.8 0.6 0.4 0.2 0

0

0.2

0.4

0.6

0

0.2

0.4

0.6



90

90

1

120

0.8

0.8

0.6 150

0.4

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 12.7667

0.6 30150

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

27jun00. Unit: 10−13, # 1; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 192.4; AffVel

x 8.559

x 3479

1


Scaled

0.8 0.6 0.4 0.2 0

−0.5

0

0.5

1

1.5 −0.5

0

0.5



90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 21.8449

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270

Figure H.23: Directionally decomposed kinematic parameters - tangential speed (A), and equiaffine speed (B): corresponding spike counts, duration of two-dimensional quanta, firing rates. Circle in the center of the decomposed equiaffine speed in (B) corresponds zero equiaffine speed (straight path or no motion). One-dimensional tuning curves are depicted in the upper row. Black tuning curves for the firing rate correspond to estimation based on the directionally decomposed data: P P S fk = i STkk ii 6= Pi Tki ii . The plots are for the same unit as in figure H.20ℵ. i


Appendix

H

249

28jun00. Unit: 40, # 1; τ=0; ’dir_decompose_f_rate_main.m −>’ x 31.81; Tgv

x 2458

x 1.671

1


Scaled

0.8 0.6 0.4 0.2 0

0

0.2

0.4

0.6

0

0.2

0.4

0.6



90

90

1

120

0.8

0.8

0.6 150

0.4

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 7.8808

0.6 30150

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

28jun00. Unit: 40, # 1; τ=0; ’dir_decompose_f_rate_main.m −>’ x 22.52; AffVel

x 2413

x 2.559

1


Scaled

0.8 0.6 0.4 0.2 0

−0.5

0

0.5

1

1.5 −0.5

0

0.5



90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 14.4149

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270

Figure H.24: Directionally decomposed kinematic parameters - tangential speed (A), and equiaffine speed (B): corresponding spike counts, duration of two-dimensional quanta, firing rates. Circle in the center of the decomposed equiaffine speed in (B) corresponds zero equiaffine speed (straight path or no motion). One-dimensional tuning curves are depicted in the upper row. Black tuning curves for the firing rate correspond to estimation based on the directionally decomposed data: P P S fk = i STkk ii 6= Pi Tki ii . The plots are for the same unit as in figure H.20i. i


Appendix

H

250

03jul00. Unit: 10, # 1; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 91.1; Tgv

x 4756

x 2.895

1

Scaled

0.8 0.6 0.4 0.2 0


0.2

0.4

0.6

0

0.2

0.4

0.6


# of quanta, x 143.2831

90

90

1

120

0.8

0.8

0.6 150

0.4

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 5.7494

0.6 30150

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

03jul00. Unit: 10, # 1; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 94.46; AffVel

x 5181

x 2.547

1

Scaled

0.8 0.6 0.4 0.2 0

Non−decomp. Decomp., no outl. −0.5

0

0.5

1

1.5 −0.5

0

0.5


# of quanta, x 104.2609

90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 7.4151

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270



Appendix

H

251

03jul00. Unit: 61−63, # 6; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 568.9; Tgv

x 18.78

x 3924

1

Scaled

0.8 0.6 0.4 0.2 0


0.2

0.4

0.6

0

0.2

0.4

0.6


# of quanta, x 110.4036

90

90

1

120

0.8

0.8

0.6 150

0.4

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 49.2333

0.6 30150

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

03jul00. Unit: 61−63, # 6; τ=−0.12; ’dir_decompose_f_rate_main.m −>’ x 686.8; AffVel

x 19.29

x 4158

1

Scaled

0.8 0.6 0.4 0.2 0

Non−decomp. Decomp., no outl. −0.5

0

0.5

1

1.5 −0.5

0

0.5



90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 60.7934

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270



Appendix

H

252

17jul00. Unit: 5101−5102, # 6; τ=0; ’dir_decompose_f_rate_main.m −>’ x 2293; Tgv

x 1.406e+04

x 21.79

1

Scaled

0.8 0.6 0.4 0.2 0


0.2

0.4

0.6

0

0.2

0.4

0.6


# of quanta, x 615.6512

90

90

1

120

0.8

0.8

0.6 150

0.4

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 35.6608

0.6 30150

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

17jul00. Unit: 5101−5102, # 6; τ=0; ’dir_decompose_f_rate_main.m −>’ x 2473; AffVel

x 1.463e+04

x 23.28

1


Scaled

0.8 0.6 0.4 0.2 0

−0.5

0

0.5

1

1.5 −0.5

0

0.5


# of quanta, x 523.5217

90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 39.8919

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270



Appendix

H

253

28jun00. Unit: 70, # 7; τ=0.06; ’dir_decompose_f_rate_main.m −>’ x 399.5; Tgv

x 2623

x 51.78

1

Scaled

0.8 0.6 0.4 0.2 0


0.2

0.4

0.6

0

0.2

0.4

0.6



90

90

1

120

0.8

0.8

0.6 150

0.4

0.9

60

90 120 30 150

0.4

0.2

0.6

F. rate, x 55.1109

0.6 30150

0.4

x = 0 + 0.7 * R

0.2

1

120

60

0

0.2

0.8

1 60 0.8 0.6 0.4 0.2

0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

A

270

0.1

300 270

300

240

0.2

270

28jun00. Unit: 70, # 7; τ=0.06; ’dir_decompose_f_rate_main.m −>’ x 524.2; AffVel

x 39.8

x 2610

1

Scaled

0.8 0.6 Non−decomp. Decomp., no outl.

0.4 0.2 0

−0.5

0

0.5

1

1.5 −0.5

0

0.5



90

90

1

120

0.8

150

1.5 −0.5

0.8

0

0.4

60

90 120 30 150

0.4

0.2

1

1.5

0.9

0.6 30150

0.5

F. rate, x 53.3317

1

120

60

0.6

x = −0.75 + 2.25 * R

1

0.2

1 60 0.8 0.6 0.4 0.2

0.8 0.7 30

0.6 0.5 0.4 0.3

330210

210

330

210 330 240

300

240

B

270

300

240

300

0.2 0.1

270

270



Appendix

H

254

We conclude based on the directionally decomposed relationships that • unit 10-13, 27jun00, figures H.20(ℵ) and H.23 is more sensitive to faster clockwise movements (negative equiaffine velocity with larger magnitude); • unit 40, 28jun00, figures H.20(i) and H.24 shows no preferences when tangential velocity is decomposed; • the unit 10, 03jul00, figures H.21(ℵ) and H.25 shows preferences for zero equiaffine velocity, which corresponds to straight movements and no motion cases. When considering the decomposition of the tangential speed, we see exactly preferences for the case on no motion (tangential speed close to zero) and fast movements in direction ≈ 2800 . Interestingly, the directional preferences for equiaffine speed are not so pronounced as for the Euclidian speed. The tuning relationships for this unit show clear evidence for the preference to zero equiaffine speed; • unit 61-63, 03jul00, figures H.21(i) and H.26 shows directional preference. • unit 5101-5102, 17jul00, figures H.22(ℵ) and H.27 is more sensitive to faster movements; • unit 70, 28jun00 figures H.22(i) and H.28 is sensitive to faster movements within a range of directions.

H.3

Chapter 4

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

4.3%


5.7% 6.1% 3.9%

9

3.9%

10

5.2%

11

3.9%

12

4.3%

137

4.3%

13

4.8%

14

7.8%

148 159

170 15 181 16

3%

192 17

5.7%

6.5%

203 18

9.1%

225 19

4.3%

214

2

2

5.2%

82 7 93 8

126

1

5.2%

71 6

115

0

6.5%

0


4

6

0

0.2

0.4

0.6

0.8

1

1.2

1.4


1.8


2

5 1 16 2 27 3 38 4 49 5 60

4.3%

5.7% 6.1%

9

3.9%

10

5.2%

11

3.9%

12

4.3%

13

4.8%

14

7.8%

137

4.3%

148

10

159 12

170 15 181 16

3%

192 17

5.7%

6.5%

203

14

18

9.1%

225 19

4.3%

214 16


2

4

6

8

10

12

14

16

18

20

1

2

5.2%

3.9%

126

0

5.2%

82 7 93 8

115

0

6.5%

71 6

104 8

255

Rasters. 28jun00; # 7, 70; r1. P1


2

5 1 16 2 27 3 38 4 49 5 60

104

H


Rasters. 28jun00; # 7, 70; r1. P0 0

Appendix



18 15

15 10

20


5

Likelihoods of change in firing rate, normed by: 1.7; 14 100

1 y, [mm]

0.8 0.6

0.6 0

0.2

0.4

−50

0.6


1.4

1.6

1.8

2

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

3% 4.4% 1.6% 6%

0

0

0

1

2

2

6.9%

875 7 1020 8

5% 2.7%

1165 9

7.1%

1310 10

4.8%

1455 11

4.7%

1600 12

4.1%

1745 13

4.2%

1890 14

8.1%

2035 15

2.9%

2180 16

7.3%

17

4.4%

18

7.9%

2760 19

8.2%

2470

2905



6.8%

730 6

2615


50

0

0.2

0.4

4

6

8

10

12

14

−50

0.6


1.4

1.6

1.8

2

5 1 156 2 3 3 307 4 4 458 5 5 609

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

3% 3.1% 4.4% 4.6% 1.6% 1.6% 6% 5.9%

0

6.9% 6.8%

911 7 7 8 1062 8 9 1213 9 1364 10 10 11 1515 11 12 1666 12 13 1817 13 14 1968 14 15 2119 15 16 2270 16 17 2421 17 2572 18 18 2723 19 2874 19

5% 5.1% 2.7% 2.6% 7.1% 7.1% 4.8% 4.7% 4.7% 4.7% 4.1% 4.1% 4.2% 4.3% 8.1% 8% 2.9% 2.8% 7.3% 7.3% 4.4% 4.5% 7.9% 7.9% 8.2% 8.2%


1

2

2

6.8% 6.7%

760 6 6

0

4

6

8

10

12

14

16

20

20 18

18 15

15 10

10

20


5

Likelihoods of change in firing rate, normed by: 1.5; 1.8 100 y, [mm]

0.8 0.6

0.6

Max. # of spikes: 3044−> 30 0.6


1.6

1.8

2

−50

0.2 −50

1.4

0 x, [mm]

50

Max. # of spikes: 2890−> 30 0

50 0


0.4

0.2

0.4

100

0.8

50

−50

0.2


1

0


0


0

1

0.4

20

5

y, [mm]

0

0

50


2

3025

16

0 x, [mm]


Rasters. 28jun00; # 6, 62; r0. P1


2

1 150 2 3 295 4 440 5 585

2325

0 x, [mm]


Rasters. 28jun00; # 7, 70; r0. P0 0

5

0 −50

0.2


50


0.4

0.2

0

100

0.8

−50

0


1

50


0.4


0

y, [mm]

0

20

5


10

0

0.2

0.4

0.6


−50 1.4

1.6

1.8

2

0 x, [mm]

50

Figure H.29: PSTH for the time when the monkey got a reward, scribbling movements. See explanations in text on page 104.

PHD thesis by Felix Polyakov −0.5

0

0.5

1

1.5

−1 0

11 1

0

1

−1

−0.5

0

0.5

1



2

2

−1 0

5 11 1

17.1%

17

0

1

2

17.1%

17

2

2

2

29

13.2%

35 41 47 53 3

20.9%

59 65 71 77 4

89 95 5

16.3%

107 113 119 6


47 6

53 3

20.9%

59 65

8

71 77 4

89 95 12

5

16.3%

119 6

15.5%

101 107 113

14

4

6

8

17.1%

83

10

125 16


18

15

13.2%

41

15.5%

125

2

35

4

17.1%

83


23


23

101

256

Rasters. 28jun00; # 7, 70; r1. P1


2

5

29

H


Rasters. 28jun00; # 7, 70; r1. P0 −1

Appendix

10

12

14

16

18

20 15

10 20

20

10

5

5 0



1 y, [mm]

0.8 0.6

100 50 0


0.4



1 0.8

y, [mm]

0

0.6

−200

0 x, [mm]

200

0.2

100 50 0


0.4

−200

Max. # of spikes: 132−> 30 0 −1

−0.5

−1

−0.5

0

1.5

2

0

0.5

1

0 −1

−0.5

−1

−0.5


0

1

0


1.5

2


Rasters. 28jun00; # 7, 70; r0. P1


2

−1 0

0

0.5

1

1.5


2

2

5

−1 0

21.2%

10 1

1

2

21.2%

2

2

2

7.1%

25 30 35 3

22.4%

40 45 4

20%

55 60 65 70 5

20 4

6

8.2%

85


7.1%

30 6

35 3

22.4%

40 8

45 50

4

20%

55

10

60 65

12

21.2%

75

2

25

70 5

80

6

8.2%

85

16

4

6

8

10

12

21.2%

75

14


15


15

80

0

5

10 1

50

200



Rasters. 28jun00; # 7, 70; r0. P0

20

0 x, [mm]

0.2


14

16

15 18

18

20

10

15 20

20

10

5 5 0



1 y, [mm]

0.8 0.6 Aver/1.5 All spikes/10

0.4

100 50 0 −200

0.2



1 0.8

y, [mm]

0

0.6 0 x, [mm]

200


0.4

100 50 0 −200

0 x, [mm]

200

0.2

Max. # of spikes: 79−> 30 0 −1

−0.5

0


Max. # of spikes: 111−> 30 1.5

2

0 −1

−0.5

0


1.5

2

Figure H.30: PSTH for the time when the monkey got a reward, center-out movements. See explanations in text on page 104.

PHD thesis by Felix Polyakov 0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

5%

2

6.4%

35 3

7.8%

45 4 55 5

4.1%

0

0

1

2

3.2% 6.4%

8

5.5%

9

6%

95

115 10

6.4%

11

6%

12

5.5%

155 13

6.9%

165 14 175 15

4.1%

135

0.8

1

1.2

1.4


1.8


2 5%

2

6.4%

35 3

7.8%

45 4 55 5

4.1%

0

0

1

2

4.6%

185 16 195 17 205 18

5.5% 1.4% 6%

215 19

4.1%


4

5%

65 6

3.2%

75 7

6.4%

85

6

2

8

5.5%

9

6%

95 105 8

115 10 125

6.4%

11

6%

135 10

12

5.5%

155 13

6.9%

165 14 175 15

4.1%

185 16 195 17 205 18

5.5% 1.4% 6%

215 19

4.1%

145

12

14

4.6%

16


6

8

10

12

14

16

6

18

4

4

3

18

4 20

2

20


1


100

1 y, [mm]

0.8 0.6

0.6 0

0

0.2

0.4

−50

0.6


1.4

1.6

1.8

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8


50 0

4.4%

2

6.1%

3

4.9%

581 4

7.1%

725 5 869 6

4.5%

7

5.3%

0

0

1

2

4.1%

8

4.6%

1301 9

5.8%

1445 10

4.7%

1589 11 1733 12

5%

13

5%

1157

1877

4.6%

2021 14

8.2%

2165 15

8.3%

2453 16 17 2597 18 2741 19 2885

4.9% 2.1% 5%

2309

5.4%


4


1.4

1.6

1.8

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 x, [mm]

50


1.8


2

1

4.4%

2

6.1%

3

4.9% 7.1%

725 5 869 6

4.5%

7

5.3%

1013 6

8

10

4.1%

8

4.6%

1301 9

5.8%

1445 10

4.7%

1589 11 1733 12

5%

13

5%

1157

1877

4.6%

2021 14

8.2%

2165

12

15

8.3%

2453 16 17 2597 18 2741 19 2885

4.9% 2.1% 5%

2309

14

16

5.4%


0

0

1

2

2

4

6

8

10

12

14

16

18

5

4

4

3

3

20

2


1


0.8 0.6

Max. # of spikes: 948−> 30 0.4

100

0.8

50

0.6

0.6


1.6

1.8

2

−50

0.2 −50

1.4

0 x, [mm]

50

Max. # of spikes: 957−> 30 0

50 0


0.4

0.2

0.2


1

−50

0


1

0


0.4

20

2

0

1

0

−50

0.6

581 4

437

18

5

0

0.4

293 2


1013

0.2

Rasters. 18jul00; # 6, 70; r0. P1 5 149

293 437

0


2

1

149

0 x, [mm]


Rasters. 18jul00; # 6, 70; r0. P0 5

0 −50

0.2


50


0.4

0.2

100

0.8

−50

0


1

50


0.4


0

y, [mm]

0

2


145

0.6

y, [mm]

125

0.4



2

5%

75 7

105

0.2

25

65 6 85

0 5 1 15

25

257

Rasters. 18jul00; # 6, 70; r1. P1


2

5 1 15

H


Rasters. 18jul00; # 6, 70; r1. P0 0

Appendix

0

0.2

0.4

0.6


−50 1.4

1.6

1.8

2

0 x, [mm]

50



0

0.5

1

1.5

−1 0

11 1

0

1

−1

−0.5

0

0.5

1



2

2

−1 0

5 11 1

15.8%

17

0

1

2

15.8%

17 2

29 35 2

29

15.8%

41 47 3

15.8%

59 65 71 4

35 2

4

89 95 101 107 5

19.4%

113 119 125 131 6

47 53

6


3

15.8%

4

18.7%

59 65 71

8

77 83 89

10

95 101 107 5

12

19.4%

113 119 14

125

14.4%

137

15.8%

41

18.7%

83

2

23



23

77

258

Rasters. 18jul00; # 6, 70; r1. P1


2

5

53

H


Rasters. 18jul00; # 6, 70; r1. P0 −1

Appendix

131 6

6

8

10

12

14

14.4%

137 16


2.5

4

16

8 18

2

18 6

1.5 4

20

1

20

2

0.5

0



1 y, [mm]

0.8 0.6

100 50 0


0.4



1 0.8

y, [mm]

0

0.6

−200

0 x, [mm]

200

0.2

100 50 0


0.4

−200

0 x, [mm]

200

0.2

Max. # of spikes: 67−> 30 −0.5

−1

−0.5

0


1.5

2

0.5

1

1.5

1

9.1%

2

16.9%

1

3


25

40 45 4

55 60 5

16.9%

65 70 6

13%

75


3

2

0

0.5

1



2

−1 0 1

9.1%

2

16.9%

25 30

6

3

40 45 4

10

55 60 5

16.9%

65 14

70 6

13%

75 16


18

2

4

6

8

23.4%

50

12

1

20.8%

35 8

0

2

20

4

10

12

14

16

18

6

2 20

20

4

1

2 0

Locations of the events Arrows ~ aver. veloc. y, [mm]


100 50 0 −200

0.2

Likelihoods of change in firing rate, normed by: 1; 7.9


1 0.8

y, [mm]


1

0.4

1.5

Rasters. 18jul00; # 6, 70; r0. P1

15

23.4%

50


10

20.8%

35

0

−0.5

0

2

2

20

30

0

5

10 15

−1


2

−1 0 5

−0.5


Rasters. 18jul00; # 6, 70; r0. P0 0

0 −1


0 −1

0.6 0 x, [mm]

200

Aver/1 All spikes/7.9

0.4

100 50 0 −200

0 x, [mm]

200

0.2

Max. # of spikes: 39−> 30 0 −1

−0.5

0


1.5

2

0 −1

−0.5

0


1.5

2



0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

0

5.5%

0

1

2

3 35 4

0.5% 4.1%

45 5

4.6%

6

7.8%

65 75 7

5%

85 8

6.4%

9

5%

10

5.9%

11

4.6%

135 13

4.1%

145 14

7.8%

105 115 125

0.4

0.6

0.8

1

1.2

1.4


1.6

1.8


2 0

5.5%

155 165 15

8.2%

175 16 185 17 195 18 205 215 19

4.1%

2

8.2%

3 35 4

0.5% 4.1%

45 5

4.6%

25 2


8.2%

95

0.2

0

1

2

15 2

55

0 5 1

7.3% 3.7% 6.4%


55

4

6

2


15

259

Rasters. 27jun00; # 1, 10−13; r1. P1


2

5 1 25

H


Rasters. 27jun00; # 1, 10−13; r1. P0 0

Appendix

7.8%

65

6

75 7

5%

85 8

6.4%

95

9

5%

10

5.9%

11

4.6%

135 13

4.1%

145 14

7.8%

105 8

115 125

10

155 12

14

165 15

8.2%

175 16 185 17 195 18 205 215 19

4.1% 7.3% 3.7% 6.4%

16


10

4

6

8

10

12

14

16

6

18

18

8 4

6 20

4

20 2

Locations of the events Arrows ~ aver. veloc. Likelihoods of change in firing rate, normed by: 1.1; 12 100

1 y, [mm]

0.8 0.6

0.6 0

−50

Max. # of spikes: 138−> 30 0.2

0.4

0.6


1.4

1.6

1.8

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.8

2.8% 3.2% 0.3% 7.2%

5

7.8%

785 6 7 941 8 1097 1253 9

5.5% 1.9%

1409 10

5.3%

1565 11

4.1%

1721 13

6.7%

629

6.9% 6.4%

1877 14 2033 15 2189 16 2345

6%

2501 17

10%

7.6% 2.6%

2657 2813 18

0.2

19

5.2%


0

1

2

0.4

0.6


1.4

1.6

1.8

0

0.2

0.4

0.6

0.8

1

1.2

1.4

473

2

1.6

1.8

6

8

10

12

2.8% 3.2% 0.3% 7.2%

5

7.8%

785 6 7 941 8 1097 1253 9

5.5% 1.9%

1409 10

5.3%

1565 11

4.1%

1721 13

6.7%

1877 14 2033 15 2189 16 2345

6%

2501 17

10%

6.9% 6.4%

7.6% 2.6%

2657 14

2813 18

3125

16

0

19

5.2%


18

0

1

2

2

10.6%

2969

6 5


2

629 4

50


Rasters. 27jun00; # 1, 10−13; r0. P1 5 1 161 2 3

0 x, [mm]

2

317 4

10.6%

2969 3125

0


317 4 473

0


2

−50

Max. # of spikes: 82−> 30 0


1.6

0 −50

0.2

50

2

Rasters. 27jun00; # 1, 10−13; r0. P0 5 1 161 2 3

0 x, [mm]

50


0.4

0.2

0

100

1

−50

0


0.8

50


0.4


0

y, [mm]

0


2

4

6

8

10

12

14

16

18

4

4 3 20


1 0

100

1 y, [mm]

0.6

0.6

0.2

0.4

0.6


1.6

1.8

2

−50

0.2 −50

1.4

0 x, [mm]

−50


50 0

50 0


0.4

Max. # of spikes: 1084−> 30 0

100

1 0.8

50

−50

0.2 0


0


0.4


1 0


0.8

20

2

2

y, [mm]

3

0

0.2

0.4

0.6


1.4

1.6

1.8

0 x, [mm]

50

2



0.5

1

1.5

−1 0

5 10 1

15.1%

40 45 50

3

60 65 70

4

17%

75 80 85 5

95 100 6


1

1.5


2

−1 0

25 4

30

2

15.1%

3

22.6%

40 45 50 55

8

60 65 10

70

4

17%

75 80

12

85 5

2

4

6

8

10

12

13.2%

90 14

1

2

35 6

0

17.9%

20

95 100 6

14.2%

105

0.5

15

13.2%

90

0

5

22.6%

55

−0.5

10 1


2

35

−1

2

2

20

30

1

17.9%

15

25

0


Rasters. 27jun00; # 1, 10−13; r1. P1


2

260


−0.5

H


Rasters. 27jun00; # 1, 10−13; r1. P0 −1

Appendix

14

14.2%

105 16


18

16

18

10

6 4

20

5

20

2 0



1 y, [mm]

0.8 0.6

100 50 0


0.4


0.8 0.6

−200

0 x, [mm]

200

0.2

100 50 0


0.4

−200

0 x, [mm]

200

0.2

Max. # of spikes: 80−> 30 0 −1

−0.5

−1

−0.5

0



1.5

2

0

0.5

1

0 −1

−0.5

−1

−0.5


Rasters. 27jun00; # 1, 10−13; r0. P0 1.5

0

1

0


1.5

2


Rasters. 27jun00; # 1, 10−13; r0. P1


2

−1 0

0

0.5

1

1.5


2

2

−1 0

0

1

2

5 1

22.1%

10

20 2

9.1%

25 30 35 3

26%

40 45 50 4


2

15

60 65 5

6


9.1%

30

6

35 3

26%

40

8

45 10

50 4

20.8%

65 5

10.4%

55 12

60

14

70

11.7%

75

2

25

10.4%

70

22.1%

20 2

4

20.8%

55

1

15


5 10


1 y, [mm]

0

6 16


18

6

8

10

12

14

11.7%

75

8

4

16

18

6

6 4

20

4

20

2

2

0



1 y, [mm]

0.8 0.6 Aver/1.2 All spikes/9.3

0.4

100 50 0 −200

0.2



1 0.8

y, [mm]

0

0.6 0 x, [mm]

200


0.4

100 50 0 −200

0 x, [mm]

200

0.2

Max. # of spikes: 48−> 30 0 −1

−0.5

0


Max. # of spikes: 38−> 30 1.5

2

0 −1

−0.5

0


1.5

2



0.6

0.8

1

1.2

1.4

1.6

1.8

1

0

9.2%

0

1

6.4%

40 6

5

10.6%

2


2.1% 1.4% 3.5%

47 7

5%

8

4.3%

9

5%

68 75 10

9.9%

82 11

5.7%

12

5.7%

89 96

103 13 110 117

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8


2 0

1

9.2%

19 2 3 26 4 5 33

6.4%

40 6

10.6%

0

1

2

12

19 2 3 26 4 5 33

61

0

2

12

54


Rasters. 10jul00; # 4, 50; r1. P1


2

261

8.5%

14

3.5%

15

7.1%

16 131 18

4.3% 2.1%

138 19

5.7%

124


4

2

2.1% 1.4% 3.5%


5

0.2

H


Rasters. 10jul00; # 4, 50; r1. P0 0

Appendix

47 54

6

61

7

5%

8

4.3%

9

5%

68 8

75 10

9.9%

82 11

5.7%

12

5.7%

103 13

8.5%

89

10

96

12

110

14

3.5%

15

7.1%

16 131 18

4.3% 2.1%

138 19

5.7%

117 124

14

16


10

18

4

6

8

10

12

14

16

18

8

8 6

6 20

Locations of the events Arrows ~ aver. veloc. Likelihoods of change in firing rate, normed by: 0.89; 9.4 100

1 y, [mm]

0.8 0.6

0.6 0

0.2

0.4

−50

0.6


1.4

1.6

1.8

2

209 311

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

4.5%

2

5.4%

3

4.5%

4

6.1%

5

6.4%

413 515

617 6

4.4%

719 7

6.2%

821 8

3.8%

923 9

7.7%

1025 1127 1229

10

5.4%

11

4.7%

12

6.4%

13

4.1%

1331 1433 14

6.2%

15

12%

1535 1637 1739 1841 1943


50 0

0

0.2

0.4

16

0

0

1

2

209 311

0.6% 1.2%



1.4

1.6

1.8

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

515

4

6

1.8

4.5%

2

5.4%

3

4.5%

4

6.1%

5

6.4% 4.4%

719 7

6.2%

821 8

3.8%

923 9

7.7%

1127 1229 10

10

5.4%

11

4.7% 6.4% 4.1%

14

6.2%

15

12%

16

10.3%

1535

1739 14

1841 1943

18 19

16

0.6% 1.2%


8 18

2

4

6

8

10

12

14

16

18

4

20

2


0

Likelihoods of change in firing rate, normed by: 0.72; 2.1 100

1 y, [mm]

0.8 0.6

Max. # of spikes: 932−> 30 0.4

100

0.8

50

0.6

0.6


1.6

1.8

2

−50

0.2 −50

1.4

0 x, [mm]


50 0

50 0


0.4

0.2

0.2


1

−50

0


0


0.4

20

2

y, [mm]

4

0

1

6

6

0

0

2

13 1433

1637

0

12

1331

12

50


2

617 6

1025

8

0 x, [mm]


1

413

10.3%

18 19

5 107

2

−50

0.6

Rasters. 10jul00; # 4, 50; r0. P1


2

1


107

0

0 x, [mm]


Rasters. 10jul00; # 4, 50; r0. P0 5

0 −50

0.2


50


0.4

0.2

0

100

0.8

−50

0


1

50


0.4


0

y, [mm]

0

2


2

20

4

4

0

0.2

0.4

0.6


−50 1.4

1.6

1.8

2

0 x, [mm]

50



0

0.5

1

1.5

5 1

0

1

262 Normalized speeds for the targets

Rasters. 10jul00; # 4, 50; r1. P1


2

−1 0

−1

−0.5

0

0.5

1

1.5


2

2

8.9%

−1 0 5 1

10

0

1

2

8.9%

10 2

20 25 30 3

15 2 20

4

25

21.1%

35 40 45 4

17.8%

50 55 60 5

30 3

40 45

8

70 75 80 6

21.1%

85 90


17.8%

5

17.8%

55 10

60 65

12

70 75 80 6

14

21.1%

85 90

16


18

10

4 50

17.8%

4

21.1%

35

6

2

13.3%


13.3%


15 2

6

8

10

12

14

16

18

8

8

6

6

20

20

4

4

2

2 0



1 y, [mm]

0.8 0.6

100 50 0


0.4



1 0.8

y, [mm]

0

0.6

−200

0 x, [mm]

200

0.2

100 50 0


0.4

−200

0 x, [mm]

200

0.2

Max. # of spikes: 60−> 30 0 −1

−0.5

0



1.5

0 −1

2


Rasters. 10jul00; # 4, 50; r0. P0 −1

−0.5

0

0.5

1

−0.5

1.5

−1 0 1

10%

2

17.5%

3

10%

5

0

1

20 4

20%

25 5


15

−1

35 6

25%

40


2

−0.5

0

0.5

1



2

−1 0 1

10%

2

17.5%

3

10%

5

15

6

20 4

8

20%

25

10

5

1

2

4

6

8

10

17.5%

30 12

35 6

14

0

2

10

4

17.5%

30

1.5

2

2

10


Rasters. 10jul00; # 4, 50; r0. P1


2

0


65

H


Rasters. 10jul00; # 4, 50; r1. P0 −1

Appendix

25%

40 16


18

12

14

16

18

8 10 6 20

20

4

5

2 0



1 y, [mm]

0.8 0.6 0.4


100 50 0 −200

0.2



1 0.8

y, [mm]

0

0.6 0 x, [mm]

200


0.4

100 50 0 −200

0 x, [mm]

200

0.2

Max. # of spikes: 35−> 30 0 −1

−0.5

0


1.5

2

0 −1

−0.5

0


1.5

2



0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

1

0

7.9%

0

1

2

0 5

0.2

0.4

0.6

0.8

1

1.2

1.4


1.6

1.8


2

1

7.9%

27 2

5.7%

38 3

6.2%

49 4

4.4%

60 5

4.4%

71 6

5.7%

82 7 93 8 9 104 10 115 11 126

4.4% 3.1% 4% 3.1%

0

0

1

2

16

6.2%

49 4

4.4%

60 5

4.4%

71 6

5.7%

82 7 93 8 9 104 10 115 11 126

4.4% 3.1% 4% 3.1%

2

9.3%

12

8.8%

159 13 14 170 15 181 16 192 17 203 18

4.4% 3.1%

148

5.7% 4.4% 3.1% 4.4%

19

7.9%

225


4

6

8

137 10

2


5.7%

38 3


27 2

214

263

Rasters. 04jul00; # 1, 21−23; r1. P1


2

16

137

H


Rasters. 04jul00; # 1, 21−23; r1. P0 0 5

Appendix

9.3%

12

8.8%

159 13 14 170 15 181 16 192 17 203 18

4.4% 3.1%

148

12

14

214

5.7% 4.4% 3.1% 4.4%

19

7.9%

225 16


4

6

8

10

12

14

16

1.5 18

3

18 1

2 20

20 0.5

1

Locations of the events Arrows ~ aver. veloc. Likelihoods of change in firing rate, normed by: 0.35; 13 100 y, [mm]

0.8 0.6

0.6 0

0.2

0.2

0.4

−50

0.6


1.4

1.6

1.8

2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

0 x, [mm]

50

−50 0

1.6

1.8

3.2%

145 2 285 3 425 4 565 5 705 845 6

6.2%

985 7

6.4%

1125 8

2.1%

1265 9

8.2%

1405 10 1545 11 1685 12 1825 13

5.9%

0

0

1

2

4.4% 2

6.2% 5.5%

4.6% 5.2% 2.5% 9%

2105 2245 15


6.2%

1965 14

4

6

8

10

2525 16 17 2665 18 19

7.1% 0.6% 2.3% 4.2%


1.5

0.4

0.6


1.4

1.6

1.8

2

5

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.8

3.2%

145 2 285 3 425 4 565 5 705 845 6

6.2%

985 7

6.4%

1125 8

2.1%

1265 9

8.2%

1405 10 1545 11 1685 12 1825 13

5.9%

1965 14

9%

0

0

1

2

4.4% 2

6.2% 6.2% 5.5%

4.6% 5.2% 2.5%

4

6

8

10

12

10.2%

2385 2525 16 17 2665 18 19

7.1% 0.6% 2.3% 4.2%

16


18

50


2

2245 15 14

0 x, [mm]


1.6

1

2105

12

10.2%

2385

14

16

18

0.8 0.6

1 20


0.8 0.6

0.6

Max. # of spikes: 271−> 30 0.4

0.6


1.6

1.8

2

−50

0.2 −50

1.4

0 x, [mm]


50 0

50 0


0.4

0.2

0.2

100

0.8

50

−50

0


1

0


0.4


0.2 0

1

20

0.4


y, [mm]

0.5

0

0.2

Rasters. 04jul00; # 1, 21−23; r0. P1


2

1

0

0


Rasters. 04jul00; # 1, 21−23; r0. P0 5

0 −50

0.2


50


0.4 −50

0

100

0.8

50


0


1 y, [mm]

1

0.4

Locations of the events Arrows ~ aver. veloc. 0


0

0

0.2

0.4

0.6


−50 1.4

1.6

1.8

2

0 x, [mm]

50



0.5

1

1.5

−1 0

5 10 1

30 2 35

20.5%

40 45 50 55 3 65 70 75 80 4 90 95 100 5 105

12%

6

8.5%


1.5


2

−1 0

20.5%

40 45

6

50 55 3 65 70 75

10

80 4 12

90

14

100 5 105

95

115

4

6

8

10

24.8%

85

110

2

17.9%

60

8

1

2

30 2 35

4

0

16.2%

25

12%

6

8.5%

16


18

12

14

16

18

1.5 1

2

20

20 0.5

1

0



1 y, [mm]

0.8 0.6

100 50 0


0.4



1 0.8

y, [mm]

0

0.6

−200

0 x, [mm]

200

0.2

100 50 0


0.4

−200

0 x, [mm]

200

0.2

0 −1

−0.5

−1

−0.5

0


1.5

2

0

0.5

1

0 −1

−0.5

−1

−0.5


Rasters. 04jul00; # 1, 21−23; r0. P0 1.5

0

1

0


1.5

2


Rasters. 04jul00; # 1, 21−23; r0. P1


2

−1 0

0

0.5

1

1.5


2

2

5

−1 0

0

1

2

5 18.6%

10 1 2

20 25 2


15

14.4%

30 35 3

50 55 60 65 4

25 2

4

75 80 85 5

8.2%

90 6

9.3%

PETH firing rate, [spikes/sec] 3 2.5

35 6

40

55 60 10

65 4

75 80 85 5

14

6

9.3%

16

PETH firing rate, [spikes/sec] 2.5

18

8

10

12

14

16

18

2 1.5

20

1

20

1 0.5

0.5

0

Locations of the events Arrows ~ aver. veloc. y, [mm]


100 50 0 −200

0.2



1 0.8

y, [mm]


1

0 −1

8.2%

90 95

6

30.9%

70 12

4

18.6%

50

8

2

0.4

3

45

1.5

0

14.4%

30

30.9%

70

2

20

18.6%

45

18.6%

15


10 1

95

1

20

3

40

0.5

15

24.8%

85

0

5

17.9%

60

−0.5

10 1


25

−1

2

2

20

115

1

16.2%

15

110

0


Rasters. 04jul00; # 1, 21−23; r1. P1


2

264


−0.5

H


Rasters. 04jul00; # 1, 21−23; r1. P0 −1

Appendix

0.6 0 x, [mm]

200

0.4


100 50 0 −200

0 x, [mm]

200

0.2

−0.5

0


1.5

2

0 −1

−0.5

0


1.5

2



H.4

Chapter 5

Appendix

H

265

2

Figure H.39: Neural activities, paths, speeds, a-posterior probabilities, and graphs of a-posterior probabilities for state 1 from figure 5.6. See more explanations in text on page 133.

0

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

0

4

6

7

5 6

6

6

8

3

5

6

7

67

7

8

6

5

7

6

1

4

61

5

4 856 7

4

6

3

6

6

2

8

4

6

8

8

8

5

1

1

1

1

1

1

1

2

1

8 1

6

5

1 2

6 1

2

4

8

2

2

4

5

2

3

4

6

8

3

2

5

2

2

5

4

3

6

7 8

8

3

7

1

56 7

3


3

4

8

2

8 1

3

3

4 5

4

2

4 5

2

4

4 5

8 3

4

5

6

5

6

1 2 3 4 5 6 7 8

State: 1;jit.:0;Poiss:0; State: 1; jit.:0; Poiss:0; 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 Timelag: −0.21 file: plot; dataPieces; diffColors.m Timelag: −0.21 file: plot; dataPieces; diffColors.m

20 40 60 80 100

0.5

1

Path, τ=−0.21

H

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6


15; 2966−3052

19; 731−802

37; 1831−1896

17; 2952−3008

12; 1662−1715

21; 1777−1821

40; 2015−2053

35; 342−374

20; 1011−1043

Appendix

17; 789−821


2


0

4

6

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

2

0

2

3

1 2

4 5

2

6 1

8

4 5

3

2

3

67

4

3

2 3

3

4

4

5

1

4

5

5

5

6

5

61

61

61

56

7

1

8

2

61

81

4 5 61

3 45

4

4 5 6

4

3

5

8

6 1

4

2

2

2

2

2

2

2

2

2

2

8

8

3

3

2

3

3

3

3

4

3

4

3

3

3

5

5

576

4

4

5

4

6

5

5


4

6

56

7

7

6 1

4

4

6

5

7

7

5

6

4 85

4

2

61

5

3

2

3

5

6

1 2 3 4 5 6 7 8

State: 2;jit.:0;Poiss:0; State: 2; jit.:0; Poiss:0; 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 Timelag: 0.28 file: plot; dataPieces; diffColors.m Timelag: 0.28 file: plot; dataPieces; diffColors.m

20 40 60 80 100

0.5

1

Path, τ=0.28

H

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6


16; 1065−1133

15; 3004−3072

40; 727−792

20; 518−583

45; 1638−1694

40; 1606−1662

18; 484−540

15; 3244−3297

48; 2352−2402

Appendix

39; 1828−1875


2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

0

Path, τ=0

2

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

0


0.5

Speed

0.5

1

0

8

1

6

4

4

4 5

6

2 3

5

5

5

1

6

5

6

1

8 1

6 1

8

7

1

7 8

7

6

4

4

5

5

2

8

2

4

81

6

1

8

6 1

8

2

2

7

2

5

2

2

8

2

2

3

3

3

3

3

3

3

3

3

3

7

4

5

4 5

4

4 6

4

5

5

8

3

6

5 6

6

2

6

4

6

4 5

4 5

4

6

1

51

7

2

8

7

2

8

18

8 61

7

7

61


6

2

4

2

2

1

3

6

8

8

3

3 4

2

7

2 3

5

6

1 2 3 4 5 6 7 8

H

State: 3;jit.:0;Poiss:0; State: 3; jit.:0; Poiss:0; 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 28jun00; set01; nStates08; nMix1; Twind=30 [msec.]learnedset1or2=1; learneddata=1 Timelag: 0 file: plot; dataPieces; diffColors.m Timelag: 0 file: plot; dataPieces; diffColors.m

20 40 60 80 100

0.5

1


15; 3104−3169

40; 1013−1072

20; 1143−1202

12; 1791−1850

37; 292−345

40; 2201−2251

18; 572−622

12; 2064−2114

17; 930−974

Appendix

21; 585−626



2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

0

Path, τ=−0.14

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

0

1

3

4

4

7

6

8

3

3

5

8

1

2

3

56

1

2

4

8

1

4

4

4

2 1 8

4

5

2

5 6

5

5

6

45

5

8

6

3

7

3

3

6

3

6

3

7

3

2

7

4

5

4

4

4

4

5

4

5

5

4 5

4

4

4

5

5

6

6

6

5

6 1

6

6

56

5

6

8 1

7

8

7

6

7

8

7

3

6

18

8

7

8

81

2

3

6

8

2

4

4 5

2


2

8

6

4

3

3

3

5

6

1 2 3 4 5 6 7 8

H


20 40 60 80 100

0.5

1


18; 373−429

17; 1388−1441

39; 2014−2061

15; 2872−2919

17; 3455−3499

15; 3595−3636

48; 2496−2534

16; 1431−1469

17; 989−1021

Appendix

20; 890−904



2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

0

Path, τ=0

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

4

8

8

5 6

2

8 61

0

7

2

6

6 7 45

2

2

6

2

3

1

1

2

3

2

1

1

76 8

2

4

13

3

3

23

2

3

4

4

4

4

2

5

5

5

5

5

5

5

5

5

4 5

4

3 4

3

4

4

6

6

6 1

6

6 7

61

6 8

6

6

7

3

7

1

2

2

7

3

6

2

8

3

4

5

8 1 4

2

7

3

6

5

7

2

8

4

4 6

6

1

6


4

8

3

5

51

3

4

2

6

5

6

1 2 3 4 5 6 7 8

H


20 40 60 80 100

0.5

1


15; 1095−1127

12; 2391−2423

12; 966−998

20; 507−533

48; 1423−1446

45; 1093−1116

37; 553−576

12; 1419−1442

45; 1375−1395

Appendix

40; 2102−2122



2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

0

Path, τ=0

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

4

0

3

2

3

7

2

5 3

2

4

81

4

5

8

5

4

2

4

5

8 1

56

2

3

6

1

5

2

7

7

6

6

3

4

4

3

7

3

8

4

5

5

5

5

2

7 45

4 5

4

4 5

4 5

6

6

6

6

6

6

6

6

6

6

8

7

8

76 8

7

7

8

7

8

6

6

3

6

23

1

8 1

1

4

2

2

5

2


3

6

4

3

3

7

8

4

4

5

6

1 2 3 4 5 6 7 8

H


20 40 60 80 100

0.5

1


40; 2821−2865

39; 2204−2248

37; 1180−1224

17; 3492−3536

15; 4031−4075

15; 1128−1172

12; 1893−1937

12; 1263−1307

11; 1005−1049

Appendix

18; 665−706



2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

0

Path, τ=−0.21

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

0

1

8

3

2

2

2

5 8 1

2

6

2

3

3

4

3

1

37 4

7 6

3

4

4

4

6

5

5

1 8

4 5

6

6

5

5

7

6

6

6

6

6

6

2

7

7

7

7

7

7

7

7

7

7

8

6

6

4

4 5

6

6

6

6

3

1 2

8 1

1

7

5

1

2

8

2

8 1 21 2

7

8

6

1

1

2

3

6

A−posterior probabilities 6

8

4

8

1

1

3

3

2

4 5 6

4 5

2

2

2

5

6

1 2 3 4 5 6 7 8

H


20 40 60 80 100

0.5

1


15; 624−668

47; 3064−3105

40; 368−409

37; 1246−1287

23; 327−368

18; 728−769

14; 433−474

37; 1720−1755

33; 2376−2411

Appendix

23; 687−722



2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

2

4

6

0

Path, τ=0

4

20

40

60

80 100

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0 1

0

0.5

0

0 1

0

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0.5

0.5

1

0

1

0 1

2


0.5

0

Speed

0.5

1

0

4

5

3

2

5

6

4

6

4 5

3

5

7

1

3

6

6 1

1

45

4

4

5

7

4

2

6

5

6

18

5

6

6

7

2

7

2

7 8

6

7

2

7 8

8

8

8

8

8

8

8

8

1

3

1

2

2

2

1

2

2

2

3

13

45

2

4

2

37 4

61

2 1 8

3 45

6

5

6


3

4

4

7

6

4

8

1

5

1

5

2

6

7

2

6 8

4

5

6

1 2 3 4 5 6 7 8

H


20 40 60 80 100

0.5

1


45; 1534−1590

21; 1906−1962

20; 2271−2327

17; 1161−1217

12; 759−815

48; 487−534

33; 2121−2168

24; 2669−2716

16; 1537−1581

Appendix

20; 1767−1808



PHD thesis by Felix Polyakov (1) 27jun00, st=1, τ=−0.14

27jun00, st=2, τ=−0.14

Appendix

H

274 (2)

27jun00, st=3, τ=−0.14

28jun00, st=1, τ=−0.14

60

40 50

50

50 0

0

0

20

50

0

0

40 0

0 −20

−20

−50

28jun00, st=4, τ=−0.14 50

20

−50 −50 −80−60−40−20 0 20

28jun00, st=3, τ=−0.14

28jun00, st=2, τ=−0.14

−40

−50

−50

−40 −80−60−40−20 0 20 40

−100 −50

0

−40 −20

0

−80−60−40−20 0 20 40

20

28jun00, st=5, τ=−0.14

−50

28jun00, st=6, τ=−0.14

0

50

28jun00, st=7, τ=−0.14

−50

0

50

28jun00, st=8, τ=−0.14

100 50

50

50

50 0 0

−50

−50

−50 −50

0

−50

50

0

50

(3) 03jul00, st=1, τ=−0.14

03jul00, st=2, τ=−0.14 100

50

100

10jul00, st=2, τ=−0.14

50

0

0

50

0

−50

50

03jul00, st=5, τ=−0.14

50

50

0

50

−50

03jul00, st=7, τ=−0.14

0

50

−50

0

−50

50

10jul00, st=5, τ=−0.14

03jul00, st=8, τ=−0.14

0

50

50

50

0

0

−50

−50

50

−50

10jul00, st=6, τ=−0.14

40 100

0

50

10jul00, st=4, τ=−0.14

50

−50 −50

03jul00, st=6, τ=−0.14

100

100

50

50

0

0

−50

0

50

−50

0

50

10jul00, st=7, τ=−0.14 100

20

0

0

−50

−50

−100 −50

0

10jul00, st=3, τ=−0.14

50

0

−50

−50 −50

−50

100

0 −50

−50

50

100

0 0

0

100

50

50

−50 −50

(4) 10jul00, st=1, τ=−0.14

03jul00, st=4, τ=−0.14

03jul00, st=3, τ=−0.14 100

100

0

0

0

50

−100 −50

0

50

0

0

−50

−20

−50

0

50

−100

−50

−50

0

50

50

0

−50

0

50

−50

0

50

Figure H.47: Paths corresponding to the segments of trajectories labelled according to the number of the hidden state, with time-lag -140 msec, neural activity precedes the movement by 140 msec. The plots correspond to the first four days for which HMM were learned. We analyze the neural data for the days that follow extensive practice. No learning is expected to occur within these days, though certain changes in parabolic clusters do occur, as can be seen in figure H.3.

PHD thesis by Felix Polyakov (5) 11jul00, st=1, τ=−0.14

11jul00, st=2, τ=−0.14

11jul00, st=3, τ=−0.14

Appendix

(6)

80

100

0.8

275

16jul00, st=1, τ=−0.14

11jul00, st=4, τ=−0.14 100

1

H

100

100

50

0.4

0

20 0

0.2

50

50

0

0

−50 0.5

−50

1

0

−40 −40−20 0 20 40 60

50

100 100

50

50

50

0

50

−50

11jul00, st=8, τ=−0.14

0

−60−40−20 0 20 40

50

−50

50

16jul00, st=5, τ=−0.14

1

40

0.8

20

17jul00, st=2, τ=−0.14

0 −50

0

50

50

−50

0

50

−50

0

50

16jul00, st=6, τ=−0.14

50 0 0

−20

0.2

−50

0

−50

−40 0

0.5

1

−40 −20

−50

0

0

50

(7) 17jul00, st=1, τ=−0.14

0 −50

100

0 −50

0

0.4

−50

−50

0

0.6

0

0

−50

−50 −50

11jul00, st=7, τ=−0.14

11jul00, st=6, τ=−0.14

11jul00, st=5, τ=−0.14 100

50

50

0

−20

0 0

50

40

16jul00, st=4, τ=−0.14

100

60

0.6

16jul00, st=3, τ=−0.14

16jul00, st=2, τ=−0.14

(8) 17jul00, st=4, τ=−0.14

17jul00, st=3, τ=−0.14

20jul00, st=2, τ=−0.14

20jul00, st=1, τ=−0.14

20jul00, st=4, τ=−0.14

20jul00, st=3, τ=−0.14

100 100 50 0

0 −50

0

−50

50

17jul00, st=5, τ=−0.14

0

50

−50

17jul00, st=6, τ=−0.14

0

50

0

50

−60

17jul00, st=7, τ=−0.14

−40

−20

100

−50

20

−50

0

50

−50

0

50

20jul00, st=6, τ=−0.14

0.6 0.4 0.2

0

−50

−50

0

0.8

50

0

0

−60 −40 −20

20jul00, st=5, τ=−0.14

50 0

−50

1

50

50

−50

−20 −50

100

100

0

−40 −60

−50

−100 −100 −50

50 0

−20

0 0

−50

50

0 20

0 −50

20

40

50

50

50

100

0 −100 −50

0

50

−50

0

50

−100 −50

0

50

−50

0

50

0

0.5

1

Figure H.48: Paths corresponding to the segments of trajectories labelled according to the number of the hidden state, with time-lag -140 msec, neural activity precedes the movement by 140 msec. The plots correspond to the last four days for which HMM were learned. We analyze the neural data for the days that follow extensive practice. No learning is expected to occur within these days, though certain changes in parabolic clusters do occur, as can be seen in figure H.3.


Appendix

H


4

−8100

−2.7

−8200 −8250

−8350 x 104 1.225

−2.74

−2.78 x 104 4.01 4 3.99

MDL

1.215 1.21

3.98 3.97

1.205

3.96

1.2 0.4

3.95 0.5

Correlation

MDL

1.22

Correlation


−2.76

−8300

0.3 1 2 3 4 5 6 7

0.2 0.1 0

x 10

−2.72

Log lik.

Log lik.

−8150

276

2

4

6

8

10

12

0.4 0.3

0.1 0

14

1 2 3 4 5 6

0.2

2

4

6

# of hidden states. 4

−2.6

x 10

8

10

12

14

# of hidden states.

Day: 07, 03jul00; set: 1; time bin=0.03[sec.]. Actual data. Training set.

4

−2.4

x 10


Log lik.

Log lik.

−2.42 −2.65

−2.7

−2.44 −2.46

−2.75 x 104 3.95

−2.48 x 104 3.57 3.56

MDL

3.55

MDL

3.9

3.85

3.54 3.53 3.52 3.51 0.5

Correlation

Correlation

3.8 0.8 0.6 1 2 3 4 5 6

0.4 0.2 0

2

4

6

8

10

12

0.4 0.3

0.1 0

14

1 2 3 4 5

0.2

2

4

6


−1.86

x 10


4

−2.64

Log lik.

Log lik.

−1.9 −1.92

x 10


−2.66 −2.67

2.79

3.88

MDL

2.78

MDL

14

−2.69 x 104 3.89

−1.94 x 10 2.8

2.77 2.76

3.87 3.86 3.85

2.74 0.4

3.84 0.4

Correlation

2.75

Correlation

12

−2.68 4

0.3 0.2

1 2 3 4 5

0.1

2

4

6

8

10

12

0.3 0.2

0

14

1 2 3 4 5

0.1

2

4

6


−4.62

10

−2.65

−1.88

0

8

# of hidden states.

x 10

8

10

12

14

# of hidden states.


4

−1.56

x 10


−4.64 −4.66

Log lik.

Log lik.

−1.57 −4.68 −4.7

−1.58 −1.59

−4.72 −4.74 x 104 6.86

−1.6 x 104 2.305 2.3

6.82

MDL

MDL

6.84

6.8

2.29

6.78 1 2 3 4 5 6 7 8 9 10 11 12

0.4 0.3 0.2 0.1 2

4

6

8

# of hidden states.

10

12

2.285 0.4

Correlation

Correlation

6.76 0.5

0

2.295

14

0.3 0.2

1 2 3 4 5

0.1 0

2

4

6

8

10

12

# of hidden states.

Figure H.49: Graphs of log likelihood of the model (first row) and the description length of the model (second row), versus number of hidden states. A. Estimates for the training set. B. Estimates for the test set. In the lower row, the correlations (5.3.1) between the units and the models are depicted.

14


4

−1.13

x 10

Appendix


4

−2.41

Log lik.

Log lik.

−1.14 −1.145

−2.43 −2.44

−2.46 x 104 3.55

−1.15 x 10 1.7 1.69

3.54

1.68

MDL

MDL


−2.45 4

1.67

3.53 3.52

1.66

3.51 0.5

Correlation

1.65 0.4

Correlation

277

−2.42

−1.135

0.3 1 2 3 4 5 6 7

0.2 0.1 0

x 10

H

2

4

6

8

10

12

0.4 0.3

0.1 0

14

1 2 3 4 5 6

0.2

2

4

6


−2.12

x 10

Day: 07, 03jul00; set: 1; time bin=0.03[sec.]. Actual data. Test set.

4

−1.97

Log lik.

Log lik.

−2.16 −2.18 −2.2

−1.99

−2 x 104 2.89

MDL

MDL


2.885

3.14

2.88 2.875

3.12

2.87 0.5

Correlation

3.1 0.8

Correlation

x 10

−1.995

3.16

0.6 1 2 3 4 5 6

0.4 0.2

2

4

6

8

10

12

0.4 0.3

0.1 0

14

1 2 3 4 5

0.2

2

4

6


x 10


4

−2.34

Log lik.

Log lik.

−1.9

−1.92

12

14

x 10


−2.36 −2.37 −2.38

−1.93 4

−2.39 x 104 3.45

−1.94 x 10 2.8

3.44

2.78

3.43

MDL

2.79

2.77

3.42

2.76

3.41

2.75 0.4

3.4 0.4

Correlation

MDL

10

−2.35

−1.91

Correlation

8

# of hidden states.

−1.89

0.3 0.2

1 2 3 4 5

0.1

2

4

6

8

10

12

0.3 0.2

0

14

1 2 3 4 5

0.1

2

4

6


−6.02

14

−1.98

3.18

0

12

−1.985

−2.22 x 104 3.2

−1.88

10

−1.975

−2.14

0

8

# of hidden states.

x 10

8

10

12

14

# of hidden states.


4

−1.635

x 10


−6.04 −6.06

Log lik.

Log lik.

−1.64 −6.08 −6.1

−1.645 −1.65

−6.12 −6.14 x 104 8.85

−1.655 x 104 2.41

8.84 2.4

MDL

MDL

8.83 8.82

2.39

8.81 2.38 8.8 1 2 3 4 5 6 7 8 9 10 11 12

0.4 0.3 0.2 0.1 0

2

4

6

8

# of hidden states.

10

12

2.37 0.4

Correlation

Correlation

8.79 0.5

14

0.3 0.2

1 2 3 4 5

0.1 0

2

4

6

8

10

12

14

# of hidden states.

Figure H.50: Graphs of log likelihood of the model (first row) and the description length of the model (second row), versus number of hidden states. A. Estimates for the training set. B. Estimates for the test set. In the lower row, the correlations (5.3.1) between the units and the models are depicted.

PHD thesis by Felix Polyakov 30 pieces,870 pairs

Appendix

200 300

20 250

150 15

200 100

150

10 100 50

5

−0.2

0

0.2

0

−0.2

0

0

0.2

0

50

5

14 pieces,182 pairs

14 pieces,182 pairs

14

35

12

30

10

25

8

20

6

15

4

10

40 4 30 3 20 2 10

1

0

−0.2

60

6

50 0

278 27 pieces,702 pairs

7

350 25

H 10 pieces,90 pairs




30

0.2

−0.2

0

0

0.2


2

−0.2

0

0.2

5

0

−0.2

0

0.2


0

−0.2

0

0.2



450 120

25

400

20 100

350

20

300 15

15

80

250 60

200 10

150

10

40

100

5

5 20

50 0

160

160

140

140

120

120

100

100

180

80

60

60

40

40

140

20

15

80

40

−0.2

0

0.2


0

20

−0.2

0

0

0.2

10 pieces,90 pairs

−0.2

0

0.2

16

0

−0.2

0

10 1.5

0.5 0.4

6 1

60

0.3 4

40

0.2

0.5 2

20

−0.2

0

0.2

0 pieces,0 pairs 1

0

−0.2

0

0

0.2


−0.2

0

0.2

8 pieces,56 pairs 2

15 pieces,210 pairs

1.6 500

0.4 0.2

1.4

2

1.2

400

0

1 300

−0.2 −0.4

1.5

0.8 1

0.6

200

−0.6

0.4

0.5

100 −0.8

0

0.2

0


−0.2

0

0.2

0

7 pieces,42 pairs 3

−0.2

0

0.2

120

0


200

140

−0.2

0

150

50

0

0.2

0

−0.2

0

0.2

−0.2

0

0.2

20

0

0.2


17jul00, state=5; thresh=0.15

900

−0.2

0

0.2

60

600 50 500

150 40

400 100

300

30 20

200

50

0

0

−0.2

0

0.2

0

−0.2

0

0.2

0

−0.2

0

0.2


140

40

120

100

100 30

80 60

30

80 20

60

20

40

40

10

10 20

−0.2

0

0.2

2 pieces,2 pairs

−0.6

−1


50

160

40

0

−0.2

0

0.2

0

−0.2

0

0.2

0

−0.2

0

0.2


40 20

−0.2

0

0.2

0

3 pieces,6 pairs 1

−0.2

0

0.2



12

5

0.6


200

10

0.4

4

0.2

150

8

0

3 6

−0.2

100 2

−0.4

4

−0.6

50

1

2 −0.8

−0.2

0

−1

0.2

−0.2

0

0.2

0


−0.2

0

0.2

0

−0.2

0

0.2

0

−0.2

0

0.2

0 pieces,0 pairs 1

50

0.6 0.4

40

0.2 30

0 −0.2

20

−0.4 −0.6

10

10

100

70

0

0.8

70 200

27 pieces,702 pairs

50

120

80

700

0.2

180

140


250

800

0

160

0


40

−0.2

10

−0.2

0.8

60

0

20

60

0.2

50 100

0

0.2

60

120

150

50

0

−0.8

80

0

−0.2

0

80

100

20

30

5

100

100

40

0.2

40

−0.4

80

0

60

4

120

200

−0.2

70

6

−0.2

100

0

50

140

250

0.2

8

10



300

60

80

0

0

0


160

−0.2

−0.2

12

15

140

120

17 pieces,272 pairs

0.2

−1

0

18

180

−0.6

350

140

40

10

56 pieces,3080 pairs 40 pieces,1560 pairs 35 pieces,1190 pairs 21 pieces,420 pairs 180 400 150 160 160

0.2

14

20


−0.8

−0.2

0

20

0.4

−0.4

20 0

−0.2

0.6

−0.2

1

0.2

0

1

0 100

0

0.2

0.8

80

−0.2

0

25

0 pieces,0 pairs

0.2

1.5

0

140

20

−0.2

1

0.6

2

0.5

50

0.8

0.4

40

5

16

0.2

100

60

100

20

160 2.5

10

30

0.2

−0.2

150

180 2.5

0.2

60

1

0.2

1.8

0.6


0

0

15

2

−0.2

−0.2

80

0

3

600

−1

0

0

200

1.5

0.1


0.8

0.2

20

23 pieces,506 pairs


0

0

100

2

35

0.6

8

80

−0.2


2.5

4 pieces,12 pairs

0.7

2

100

0.2

120

3

0

1

0.8

120

0

22 pieces,462 pairs

160

4

0.2

12 140

−0.2

25

3.5

0.9

14 2.5

160

0.2

0.5

12 pieces,132 pairs

3

180


5


0

10

60

20

0

0

4.5 25

160

100 80

−0.2


120

20





55 pieces,2970 pairs 42 pieces,1722 pairs 48 pieces,2256 pairs 180 180 200

0

−0.8

−0.2

0

0.2

0

−0.2

0

0.2

−1

−0.2

0

0.2

Figure H.51: Estimates of the similarity between path pieces, segmented based on HMM. Numbers of pairs below the threshold = 0.15, for all days and for all accounted time-lags: -0.28:0.07:0.28 sec.


Appendix

H

27jun00. Entr., and Mut. Inf. Angles ; States: 1 2 3, threshold=0.15 X = Metric cost clustering, Y = State clustering

279

28jun00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6 7 8, threshold=0.15 X = Metric cost clustering, Y = State clustering

7

7 H(X) H(Y) I(X,Y)

H(X) H(Y) I(X,Y)

6

6

5

5

4

4

3

3

2

2

1

1

0

−0.25

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

0

−0.25

03jul00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6 7 8, threshold=0.15 X = Metric cost clustering, Y = State clustering

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

10jul00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6 7, threshold=0.15 X = Metric cost clustering, Y = State clustering

7

7 H(X) H(Y) I(X,Y)

H(X) H(Y) I(X,Y)

6

6

5

5

4

4

3

3

2

2

1

1

0

−0.25

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

0

−0.25

11jul00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6 7 8, threshold=0.15 X = Metric cost clustering, Y = State clustering

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

7 H(X) H(Y) I(X,Y)

H(X) H(Y) I(X,Y)

6

6

5

5

4

4

3

3

2

2

1

1

−0.25

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

0

−0.25

17jul00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6 7, threshold=0.15 X = Metric cost clustering, Y = State clustering

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

20jul00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6, threshold=0.15 X = Metric cost clustering, Y = State clustering

7

7 H(X) H(Y) I(X,Y)

H(X) H(Y) I(X,Y)

6

6

5

5

4

4

3

3

2

2

1

1

0

0.25

16jul00. Entr., and Mut. Inf. Angles ; States: 1 2 3 4 5 6, threshold=0.15 X = Metric cost clustering, Y = State clustering

7

0

0.25

−0.25

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

0

−0.25

−0.2

−0.15

−0.1

−0.05

0 Time−lag

0.05

0.1

0.15

0.2

0.25

Figure H.52: Mutual information for the joint probability distribution of the states labels and hierarchical clusters with the threshold = 0.15 for all accounted time-lags: -0.28:0.07:0.28 sec. For some days the graph of the mutual information is flatter than for other, e.g. the graph for the day 28jun00 is flatter than the graph for 27jun00. The optimal time-lag for the day 27jun00, -70 msec., is not sensitive to changes in the threshold. For the day 28jun00, the optimal time-lag is sensitive to the threshold. The current threshold resulted in the time-lag 0 sec, and the threshold 0.05 results in the optimal time-lag 70 msec. The state-wise estimation of the time-lag, see figure H.51, is consistent with this result in that there are several states with many pairs in the day 28jun00, for which the optimal time-lag cannot be clearly deduced.


State 1, τ=−0.07 [s.]

Appendix

State 2, τ=0.07 [s.]

H

280

State 3, τ=−0.21 [s.]

State 4, τ=0 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=0 [s.] 100

50

50

0

0

−50

−50 0

−100 100 −100

0

−100 100 −100

0

100

State 6, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100

Figure H.53: Day 27jun00, HMM parameters. Paths are depicted based on the optimal time-lags. See more explanations in text on page 133.

PHD thesis by Felix Polyakov State 1, τ=−0.21 [s.]

Appendix

State 2, τ=0.28 [s.]

H

281

State 3, τ=0 [s.]

State 4, τ=−0.14 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=−0.14 [s.]

0

−100 100 −100

State 6, τ=−0.07 [s.]

0

−100 100 −100

State 7, τ=−0.21 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

State 9, τ=0 [s.]

0

−100 100 −100

State 10, τ=0 [s.] 100

100

50

50

50

0

0

0

−50

−50

−50

0

−100 100 −100

0

−100 100 −100

0

100

State 11, τ=0 [s.]

100

−100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100

Figure H.54: Day 28jun00, HMM parameters. Paths are depicted based on the optimal time-lags. See more explanations in text on page 133.


Appendix

State 2, τ=0.07 [s.]

H

282

State 3, τ=0 [s.]

State 4, τ=−0.14 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=0 [s.]

0

−100 100 −100

State 6, τ=−0.21 [s.]

0

−100 100 −100

State 7, τ=0.07 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

State 9, τ=0 [s.]

0

−100 100 −100

State 10, τ=0 [s.] 100

100

50

50

50

0

0

0

−50

−50

−50

0

−100 100 −100

0

−100 100 −100

0

100

State 11, τ=0 [s.]

100

−100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100

Figure H.55: Day 03jul00, HMM parameters. Paths are depicted based on the optimal time-lags. See more explanations in text on page 133.

PHD thesis by Felix Polyakov State 1, τ=0.028 [s.]

Appendix

State 2, τ=−0.07 [s.]

H

283

State 3, τ=−0.07 [s.]

State 4, τ=−0.28 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=−0.07 [s.]

0

−100 100 −100

State 6, τ=−0.21 [s.]

0

−100 100 −100

State 7, τ=−0.28 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

State 9, τ=0 [s.] 100

50

50

0

0

−50

−50 0

−100 100 −100

0

−100 100 −100

0

100

State 10, τ=0 [s.]

100

−100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100


PHD thesis by Felix Polyakov State 1, τ=0 [s.]

Appendix

State 2, τ=0.028 [s.]

H

284

State 3, τ=0.07 [s.]

State 4, τ=0 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=0 [s.]

0

−100 100 −100

State 6, τ=0.07 [s.]

0

−100 100 −100

State 7, τ=0.21 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

State 9, τ=0 [s.]

0

−100 100 −100

State 10, τ=0 [s.] 100

100

50

50

50

0

0

0

−50

−50

−50

0

−100 100 −100

0

−100 100 −100

0

100

State 11, τ=0 [s.]

100

−100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100



Appendix

State 2, τ=0.14 [s.]

H

285

State 3, τ=−0.21 [s.]

State 4, τ=0.21 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=0 [s.]

0

−100 100 −100

State 6, τ=0.14 [s.]

0

−100 100 −100

State 7, τ=0 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

0

−100 100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100

State 9, τ=0 [s.] 100 50 0 −50 −100 −100

0

100



Appendix

State 2, τ=−0.14 [s.]

H

286

State 3, τ=0 [s.]

State 4, τ=0.14 [s.]

100

100

100

50

50

50

0

0

0

−50

−50

−50

−100 −100 State 5, τ=−0.28 [s.]

0

−100 100 −100

State 6, τ=0 [s.]

0

−100 100 −100

State 7, τ=−0.28 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

State 9, τ=0 [s.] 100

50

50

0

0

−50

−50 0

−100 100 −100

0

−100 100 −100

0

100

State 10, τ=0 [s.]

100

−100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100


PHD thesis by Felix Polyakov State 1, τ=0 [s.]

Appendix

State 2, τ=−0.28 [s.]

H

287

State 3, τ=−0.28 [s.]

State 4, τ=0 [s.]

100

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

−100 −100

0

−100 100 −100

State 5, τ=0 [s.]

0

−100 100 −100

State 6, τ=0 [s.]

0

−100 100 −100

State 7, τ=0 [s.]

100

100

100

50

50

50

50

0

0

0

0

−50

−50

−50

−50

0

−100 100 −100

0

−100 100 −100

0

100

State 8, τ=0 [s.]

100

−100 −100

0

−100 100 −100

0

100

State 9, τ=0 [s.] 100 50 0 −50 −100 −100

0

100


motion primitives and invariants in monkey scribbling

motion primitives and invariants in monkey scribbling

Suggest Documents

motion primitives and invariants in monkey scribbling ...

Planning for Manipulation with Adaptive Motion Primitives

Planning for Manipulation with Adaptive Motion Primitives

Motion Primitives for Action Recognition - CiteSeerX

Finding Motion Primitives in Human Body Gestures - CiteSeerX

Using motion primitives in probabilistic sample-based planning for ...

Scribbling grace cactus printable

scribbling grace world printable

Action Recognition using Motion Primitives and ... - Semantic Scholar

Scribbling Machines - Exploratorium

Primitives and Non-primitives of Numerical ...

Differential invariants of the motion group actions

A Motion Direction Preference Map in Monkey V4

A Motion Direction Preference Map in Monkey V4

Motion Primitives for Human-Inspired Bipedal Robotic ... - Aaron Ames

Conservation laws and invariants of motion for nonlinear ... - EqWorld

Feasiblity of Motion Primitives for Choreographed Quadrocopter Flight

Gait Unsteadiness Analysis from Motion Primitives - DiVA portal

Automated Composition of Motion Primitives for ... - George J. Pappas

Learning Motion Primitives of Object Manipulation Using Mimesis Model

Motion Primitives for Realization of Robust Humanoid ...

Coordinating control of multiple rigid bodies based on motion primitives

Neural Primitives for Motion Control - The Sensory Motor Performance ...

Search-Based Planning for Manipulation with Motion Primitives