Data-driven spectral decomposition of ECoG signal ...

Data-driven spectral decomposition of ECoG signal from an auditory oddball experiment in a marmoset monkey: Implications for EEG data in humans 1st Natasza Marrouch

2nd Heather L. Read

Department of Psychological Sciences University of Connecticut Mansfield, CT, United States [email protected]

Director of Brain Computer Interface Core Department of Psychological Sciences Department of Biomedical Engineering University of Connecticut Mansfield, CT, United States [email protected]

3rd Joanna Slawinska

4th Dimitrios Giannakis

Department of Physics University of Wisconsin-Milwaukee Milwaukee, WI, United States [email protected]

Courant Institute of Mathematical Sciences New York University New York, NY, United States [email protected]

Abstract—This paper presents a data-driven method to extract spatiotemporal dynamics of mismatch negativity in a marmoset monkey. In this, we treat electrocorticographic (ECoG) data as observables of a skew-product dynamical system and extract the patterns of the neural dynamics from the point of view of the operator-theoretic formulation of ergodic theory. We successfully extract time-separable frequencies without bandpass filtering. Second, we examine in more detail the frequency band most commonly associated with MMN – beta-band activity (13–20 Hz) and proceed to cross-validate our results with those obtained by Komatsu, Takaura, and Fuji (2015). Having ensured the compatibility and statistical significance of the results, we then examine the spatiotemporal dynamics, and we find that MMN is in part driven by a synchronization in brain response following a deviation in the auditory stimuli. Index Terms—mismatch negativity, electrophysiological data, Koopman operators, kernel methods, spatiotemporal patterns

I. I NTRODUCTION Mismatch negativity (MMN) is an event-related potential component evoked by an unexpected deviation from previous auditory stimuli. The deviation may entail a change in the frequency of the sound wave, amplitude, duration or intervals between sounds [1]–[3] and results in a negative peak that overlaps with N1 and P2 [4]. In humans, MMN wave peaks occur at approximately 100 to 250 ms following the deviant stimuli [4], [5]. The peak of MMN appears sooner in other animals (30 ms to 70 ms) with faster-appearing peaks for rats [6] and cats [7], and longer time lags for monkeys [8]. MMN is one of the best described neural markers of deviance in the environment [9], but the mechanisms underlying it are explained differently by various theories. Some approaches suggest that the reduction of the excitatory

postsynaptic potentials following the repetition of the same stimuli can account for the characteristics of MMN [10]. Other approaches, originating in predictive coding paradigms, suggest that MMN is triggered by a mismatch between topdown predictions regarding the upcoming stimuli, based on past repetitions, and the sensory input that violates these expectations. This situation results in a prediction error that is then communicated to higher-level layers of the cortex [5]. One source of this lack of agreement might be driven by the spatial breadth and variety of spatial locations of brain areas contributing to MMN potentials. MMN has been linked to various parts of the auditory cortex [4], as well as the frontal cortex [4] and dorsolateral prefrontal cortex [11]. Further, several studies suggest that MMN is comprised of sub-components that appear in different temporal stages of MMN [4], [12]. Though MMN measures can indicate normal attentional states, there is growing appreciation for the use of MMN to identify endophenotypes for neurological disorders including psychiatric conditions [13]. For example, MMN anomalies have been observed among sleep-deprived individuals [14], as well as those with schizophrenia [13], [15] and a subset of individuals with schizophrenia have altered spindle oscillations during slow-wave or deep sleep [16]. Neurological cases such as these would benefit from methods that allow one to isolate intrinsic sleep potentials from those that emerge following unpredictable sound changes associated with MMN. Electrocorticographic (ECoG) recordings from multichannel electrode arrays have the potential to provide a high degree of spatial and temporal resolution for describing the underlying mechanisms for MN; however, current methods

are limited. Traditional sound-evoked averaging and linear models have been successful for determining some properties associated with MMN (e.g., [8]), many properties will not be captured. Commonly used statistical methods for examining these responses across channels and time perform dimension reduction using eigenfunctions of linear operators estimated from the data. Among them, principal component analysis (PCA) recovers temporal patterns through the eigenvectors of the empirical signal covariance matrix. A shortcoming of these classical approaches is that the operators employed are not constrained by the dynamical process underlying the observed data, leading to results of potentially limited physical interpretability. For example, it is well known that PCA has limited ability to recover distinct frequencies from broadband signals and may overlook dynamically important but lowvariance modes [17]. Moreover, the PCA patterns are by construction linear functions of the input data, which places constraints on the generality of patterns that the method can recover, particularly when the observations do not provide access to various states of the system (as is typically the case in neuroscience applications). Here, we apply a recently developed framework [18]–[20] inspired from the operator-theoretic formulation of ergodic theory that addresses many of the limitations of PCA and related approaches outlined above. Instead of estimating a covariance operator, our approach is based on data-driven approximations of the Koopman operator; the fundamental operator governing the evolution of observables (functions of the state) of a dynamical system. It is a remarkable fact, realized in the work of Koopman in the 1930s [21], that the action of a general nonlinear dynamical system can be characterized, without approximation by intrinsically linear operators acting on vector spaces of observables. These operators provide a natural and theoretically rigorous framework for spectral decomposition and statistical prediction in complex systems. In particular, the eigenfunctions of Koopman operators factor the dynamics, which may be chaotic, into periodic or quasiperiodic modes, which can be thought of as analogs of Fourier modes tailored to the dynamical system generating the data. While this operator-theoretic framework has long been known, it has only been employed in data-driven approaches [22], [23] fairly recently. A special feature of the techniques employed here is the use of kernel methods for machine learning [24]– [26] to approximate the eigenvalues and eigenfunctions of the Koopman operator with rigorous convergence guarantees and minimal assumptions on the system and measurement modality. Elsewhere [27], [28], we have applied this approach to identify a multiscale hierarchy of modes of variability from sea surface temperature data, spanning months to decades. There are several important similarities between spatiotemporal climatic data and multi-channel ECoG recording of brain potentials, including a 2D surface for sampling the system, as well as meaningful signals that vary on multiple spatial and temporal scales. In this paper, we show that the Koopman operator framework is capable of decomposing the ECoG data

into theoretically meaningful and interpretable frequencies, without band-pass filtering of the data and providing means to examine the spatiotemporal dynamics of the decomposed frequencies. II. DATA DESCRIPTION We searched for endocortical data that were already analyzed using common linear techniques and showed statistically significant expressions of MMN. The search was dictated by the need to obtain clear recordings, with transparent findings to validate the applicability of spectral decomposition. Thus, we sought to test the capability of our Koopman operator framework to capture earlier findings, and then proceed with an effort to expand insights offered by [8]. Recordings were obtained via an open source data portal linked to the RIKEN Brain Science Institute (http://neurotycho.org/) auditory-oddball-task. This open-access data contains recordings from a study designed and conducted by Komatsu, Takaura, and Fujii [8]. Specifically, Komatsu and colleagues make available data for the complete duration of their experiment, along with sound-evoked ECoG recordings, sound stimulus parameters, and other crucial details for one of two monkeys (we refer readers to [8] for a more detailed description of the studys design and data acquisition protocol). Endocortical recordings from 32 electrodes (Fig. 1) placed in various areas of the left hemispheres cortex (sampling rate 1000 Hz) were obtained from a marmoset monkey during a passive listening condition in an oddball paradigm. The time between the onset and the amplitude of sounds were constant, while two characteristics of the auditory stimuli were varied, namely (1) the frequency of the tone (from 250 Hz to 6727 Hz); (2) the number of repetitions in each train (3, 5, or 11). Stimuli were presented for 64 ms, followed by 439 ms breaks before the onset of the next stimuli. In line with [8], we considered the first tone ‘deviant’ and the last tone ‘standard’.

Fig. 1. Locations of 32 endocortical electrodes mapped onto the left hemisphere of a template marmoset brain. The figure is a reconstruction of ECoG electrode positions described in [8]

III. M ETHODS DESCRIPTION A. Koopman operator formalism We approach the problem of extracting spatiotemporal patterns of neuronal dynamics from the point of view of the operator-theoretic formulation of ergodic theory [29]. In particular, we treat electrophysiological data as observables of a skew-product dynamical system. By that, we mean that there is a dynamical system, with state space X and flow map t : X 7! X, t 2 R, driving another dynamical system on a state space Y . Here, (X, t ) describes the evolution of the auditory stimuli, and Y is the state space for the marmoset brain. The dynamics of the latter is modeled via a map t : X ⇥ Y 7! Y such that yt = t (x, y) corresponds to the state of the marmoset brain at time t, given that at time 0 the external stimulus had the state x 2 X, and the brain the state y 2 Y . Note that in this representation the brain dynamics is manifestly non-autonomous since yt depends on both y and the state x of the driving system. On the other hand, the product dynamical system on M will possess ergodic invariant measures where, M = X ⇥ Y and is autonomous, and can be described by the evolution map ⌦t : M 7! M , where ⌦t (x, y) = ( t (x), t (x, y)). The dynamical system (M, ⌦t ) is said to have a skew-product structure as component X affects the dynamical evolution of component Y , but Y does not influence X. In this description, the ECoG signal h 2 Rd recorded at d sensors on the marmoset cerebral cortex when the brain state is y 2 Y can be described via an observation function F : Y 7! Rd , such that h = F (y). Under mild assumptions on the state spaces X and Y , and the evolution maps t and t , the product dynamical system (M, ⌦t ) will possess ergodic invariant measures. That is, given a dynamical trajectory (xn , yn ), where (xn , yn ) = ⌦n⌧ (x0 , y0 )

(1)

is the state in M reached at time tn = n⌧ for a fixed sampling interval ⌧ > 0 and initial conditions (x0 , y0 ) 2 M , there exists a probability measure µ on M , invariant under the flow map ⌦t , such that for every µ-integrable PN 1 function f : M 7! C the time average, f¯N = N 1 n=0 f (x R n , yn ) converges almost surely to the expectation value f¯ = M f dµ. Note that the system can have multiple ergodic components, but data collected from a given experiment sample a specific ergodic component of the system. Associated with the triplet (M, ⌦t , µ) is a Hilbert space H = L2 (M, µ) of square-integrable observables with respect to µ and a group of unitary Koopman operators U t : H 7! H governing the evolution of observables under ⌦t . That is, given f 2 H, g = U t f is defined as the observable satisfying g(x, y) = f (⌦t (x, y)) for µ-almost every (x, y) 2 M . For our purposes, the space H will be the space of admissible temporal patterns generated by the marmoset neuronal dynamics. In particular, for every f 2 H, the mapping t 7! U t f (x, y) defines a temporal pattern associated with a sampling along

the dynamical trajectory starting at (x, y) 2 M . Our approach is to identify dynamically intrinsic temporal patterns through approximate eigenfunctions of the Koopman operator. This will lead to a decomposition of the observation map F into a corresponding set of spatiotemporal ECoG patterns. B. Koopman eigenfunctions Significant temporal modulation frequencies (i.e., observables) can be identified by computing the Koopman eigenfunctions of a dynamical system and this can provide a better approximation of the system than standard approaches such as discrete Fourier transform and PCA approaches. Koopman operators of dynamical systems lead to a distinguished class of observables through their eigenfunctions. In the setting of interest here, an observable zj 2 H is a Koopman eigenfunction if it satisfies the eigenvalue equation U t zj = ei!j t zj

(2)

for all t 2 R. In the above, !j is a real-valued frequency associated with eigenfunction zj . Moreover, !j is real due to the fact the dynamics ⌦t preserves the measure µ. A key property of Koopman eigenfunctions of measure preserving systems is that, along a given dynamical orbit, they vary according to a multiplication by a periodic phase factor. Specifically, it is a direct consequence of (2) that for µ-a.e. point (x, y) 2 M , U t zj (x, y) = zj (⌦t (x, y)) = ei!j t zj (x, y). This means that, whenever they exist, Koopman eigenfunctions give rise to temporal patterns with a highly coherent and predictable temporal evolution, each recovering a timescale 2⇡/!j intrinsic to the dynamical system. This behavior is markedly different from that of the eigenfunctions of covariance operators approximated by PCA, which in general have no direct connection to the spectral properties of the dynamical system. In addition, compared to spectral estimation techniques based on the discrete Fourier transform, which require identification of the “true” frequency peaks from a set of candidate frequencies, an advantage of pattern extraction via (2) is that an estimate of the eigenfrequencies !j is a direct output of the method. It can further be shown that (1) Koopman eigenfunctions corresponding to distinctR eigenfrequencies are orthogonal on H (that is, hzi , zj iH = M zi⇤ zj dµ = ij , where h·, ·iH denotes the Hilbert space inner product); (2) every Koopman eigenspace is one-dimensional; and (3) the spectrum {!j } of Koopman eigenfrequencies is countable. The closed span of all Koopman eigenfunctions identifies a subspace D = span{zj } ✓ H, invariant under U t , leading to an associated orthogonal decomposition H = D D? . Observables lying in D have an associated Koopman eigenfunction expansion, which can be thought of as a generalization of the Fourier transform of time-periodic signals. P Specifically, every f 2 D admits the decomposition f = j fˆj zj , where fˆj = hzj , f iH are complex-valued expansion coefficients. Moreover, the dynamical evolution of f can be computed in closed form via X U tf = fˆj ei!j zj . (3) j

It should be noted that the evolution of observables in the orthogonal subspace D? cannot be determined on the basis of a countable set of eigenfrequencies as in (3). Instead, the action of Koopman operators on such observables can be determined via an expansion involving its continuous spectrum [23]. In general, systems encountered in real-world applications are of the mixed spectrum type, meaning that they exhibit both eigenvalues and continuous spectrum (i.e., D contains nontrivial observables, but it is a strict subspace of H). Due to their desirable temporal coherence and predictability properties, Koopman eigenfunctions and their corresponding eigenfrequencies are natural objects to identify for the purpose of analyzing complex systems. C. Data-driven approximation In order to compute solutions to the Koopman eigenvalue problem in (2), we first construct a data-driven basis of H using a class of kernel algorithms, called nonlinear Laplacian spectral analysis (NLSA), operating on delay-embedded data [26], [30], [31]. Selecting an integer parameter Q (the number of delays), we first define the augmented observation map FQ : M 7! RQd , where ⇣ ⌘ FQ (x, y) = F (y), F ( ⌧ (x, y)), . . . , F ( (Q 1)⌧ (x, y)) , and then define a kernel function kQ : M ⇥ M 7! R+ , which provides a pairwise measure of similarity of points in M , based on the observation function FQ . For the purposes of this discussion, we consider a radial Gaussian kernel, ✓ ◆ kFQ (x, y) FQ (x0 , y 0 )k2 0 0 kQ ((x, y), (x , y )) = exp , ✏ as a concrete example, but in practice we work with anisotropic Gaussian kernels [26], [30]–[32] that also take into account time tendencies of the data. In the above, ✏ is a positive bandwidth parameter controlling the rate of decay of the kernel, and k·k the canonical Euclidean norm on RQd . Given a finite dataset consisting of N + Q snapshots h Q+1 , . . . , hN 1 , with hn = F (an ) taken at the dynamical states an = (xn , yn ) from (1), the kernel values kQ (am , an ) with m, n 2 {0, . . . , N 1} can be empirically computed by first forming the delay-embedded data vectors hn,Q = (hn , hn 1 , . . . , hn Q+1 ) 2 RQd , and then evaluating 2 kQ (am , an ) = e khQ,m hQ,n k /✏ . Using these pairwise kernel values, we then compute the Markov kernel pQ,N (am , an ) =

kQ (am , an ) , lQ,N am rQ,N an

(4)

introduced in the diffusion maps algorithm [25], where rQ,N and lQ,N are normalization functions given by rQ,N (am ) = lQ,N (an ) =

N X1 n=0 N X1 n=0

kQ (am , an ), kQ (am , an ) . rQ (an )

Note PN 1

that pQ,N has the Markov property p (a , a ) = 1 by construction. We then form the n=0 Q,N m n N ⇥ N ergodic Markov matrix PQ,N = [pQ,N (am , an )], and compute its eigenvalues and eigenvectors, PQ,N ~ j =

~

j j.

For pQ,N defined as in (4), the eigenvalues of PQ,N are real, and thus lie in the interval [ 1, 1] by Markovianity. By convention, we order the eigenvalues in decreasing order, 0 = 0 > 1 · · · . The eigenvector ~ 0 has all 2 components equal to 1 by standard properties of Markov matrices. As is standard practice, we take advantage of the exponential decay of the Gaussian kernel to sparsify the matrix PQ,N , retaining approximately knn ⌧ N nonzero entries per row based on the k-nearest neighborhood of the corresponding data point. This step significantly increases the scalability of the approach to large N , yet apart from data with special structure, it would be difficult to carry out using kernels without a localizing behavior (e.g., covariance and polynomial kernels). Each eigenvector ~ j = ( 0j , . . . , N 1,j )> 2 RN can be thought of as sampling a continuous function ~ j on M , such that j (xn , yn ) = nj . It can be shown that in the limit of large data, N ! 1, these functions converge to eigenfunctions of a Markov operator PQ : H 7! H, where Z PQ f = pQ (·, (x, y))f (x, y) dµ(x, y), M

and pQ is a Markov kernel defined analogously to pQ,N in (4) [20]. Importantly, as Q increases, the j converge to an orthonormal basis for an invariant subspace of D of H spanned by Koopman eigenfunctions [19], [20]. This property makes the eigenfunctions produced by NLSA a highly efficient basis for solving the Koopman eigenvalue problem. Note that the invariance of the subspace spanned by NLSA eigenfunctions in the Q ! 1 limit does not depend on the kernel being Gaussian; it is rather a consequence of delay embedding that would also hold for other kernels, including covariance kernels constructed from delay-embedded data as in singular spectrum analysis (SSA) algorithms [33]. An advantage of the nonlinear Gaussian kernels over linear covariance kernels is that in the former case the subspace spanned by eigenfunctions corresponding to nonzero eigenvalues (which amounts to the subspace that can be consistently approximated from data) is infinite-dimensional, irrespectively of Q being finite or infinite, whereas in the latter case the subspace is finite dimensional (in effect, the Gaussian kernel can be viewed as an “infinitedegree” polynomial kernel). For our purposes, this means that Gaussian kernels can provide a richer set of basis functions to approximate the Koopman operator than covariance kernels Following [18]–[20], we employ the data-driven basis { ~ j } from NLSA to compute approximations ~zj 2 CN , and ! ˆj 2 C to the Koopman eigenfunctions zj and the corresponding eigenfrequencies !j , respectively, using a Galerkin method. In all cases, we order Koopman eigenfunctions in order of increasing Dirichlet energy; a non-negative quantity that

Fig. 2. Real part (left) and its frequency spectrum (right) for Koopman eigenfunctions z5 (blue), z9 (red) and z13 (green).

intuitively measures their “roughness” as functions on M . The rationale for this ordering (which does not necessarily correspond to ordering with respect to frequency) is that the functions with low Dirichlet energy are more amenable to robust approximation from finite datasets, reducing the risk of sampling errors. Details on the Koopman Galerkin framework can be found in the references just cited. By virtue of the spectral convergence results on the ( j , ~ j ) stated above, the solutions of this data-driven Galerkin method can be shown to converge to eigenfunctions of an approximate Koopman operator, with a small amount of diffusion added to render the Koopman spectrum discrete [19], [20]. An aspect of the present analysis, which has not been discussed in earlier introductions of the method e.g. [19] is that we consider a skew-product dynamical system appropriate for non-autonomous neuronal dynamics. The source code used here is available for download at the fourth author’s personal website (http://cims.nyu.edu/⇠dimitris), while pseudocode can be found in [19]). D. Spatiotemporal reconstruction We now describe how the Koopman eigenfunctions ~zj can be used to decompose the observed ECoG signal hn into coherent spatiotemporal patterns. The reconstruction method combines SSA [33] and NLSA [26], [30], [31]. First, we compute the time-lagged projections of the observable F onto the Koopman eigenfunctions ~zj , given by 0

N 1 1 X ⇤ Aj (q) = 0 z hn+q , N n=0 nk

(5)

where N 0 = min{N, N + q}, and znk is the n-th component of eigenvector ~zj . That is, Aj (q) is an ECoG pattern in Rd approximating the projection of F , shifted by q sampling time intervals, onto Koopman eigenfunction zj . In the Koopman

operator literature, Aj (0) is referred to as the Koopman mode associated with the observable F [23], [34], though note that here we will use Aj (q) with several values of q to perform reconstruction. In particular, following [26], [30], [31], [33], we define the spatiotemporal pattern associated with the pair (~zj , Aj ) as the function Fj : M 7! Rd such that Fj (an ) =

Q 1 1 X Aj ( q)zn+q,j . Q q=0

For eigenfunctions forming complex-conjugate pairs (zj , zj⇤0 ) (which is is usually the case), we compute the sum Fj + Fj 0 . The latter is real since the observation map F is real. IV. R ESULTS A. Default setup and robustness of results The experiment in question lasted 15 minutes, during which over 900,000 samples were taken. Here, we focus on the subperiod starting at t0 = 147,862 ms, ending at t1 = 192,122 ms, and sub-sampled every 2 ms. Although this selection is arbitrary, it allows us to achieve optimal balance between a sufficient number of samples and computational costs related to subsequent analysis. That said, we find our results robust against the choice of temporal window for the experiment. We computed Koopman eigenfunctions as described in Section III, using the anisotropic Gaussian kernels introduced in NLSA and the family of cone kernels introduced in [32]. We found our results to be robust against the type of kernel employed. However, we selected the NLSA kernel as our primary focus here to evaluate faster (i.e., corresponding to mismatch negativity duration) dynamics. The choice of the NLSA kernel bandwidth parameter was ✏ = 25, and the knearest neighborhood for each data point has size knn = 5000. The specific value of ✏ is rather large versus other applications [27], [28], and the possible reason is the more intermittent

Fig. 3. High amplitude, short latency oscillations following onset of the deviant in Auditory Cortex for two embedding (emb) time windows. Composite of response to three consecutive sound onsets reconstructed for a single electrode in Auditory Cortex (number 25) using selected Koopman eigenfunctions extracted with 400 ms (a) and 500 ms (b) embedding windows. Dashed lines depict responses to repeated trials of deviant sound presentation (N=13) and the solid black line depicts the mean response over all 13 trials. X-axis: zero corresponds to the time of deviant sound onset (marked with solid blue vertical line). Solid gray vertical lines depict the time for sound onsets prior to the deviant (-503 ms) and after the deviant (503 ms). Y-axis: amplitude of the signal µVolts.

character of the ECoG signal as compared to fluid flows. These, in turn, can be related to under-sampling of the MMN signal over the analyzed period of the experiment. The same logic applies to a small number of nearest neighbors. For large enough ✏ and small enough knn , however, the results presented here seem to sample the local neighborhood robustly and sufficiently when it comes to extraction of modes needed to capture anomalous response to deviant sound over 13 repeats. A range of tests led us to conclude that 250 time-steps (equal to 500 ms) is an optimal choice for the embedding window given the short timescale of the modes of interest (tens of milliseconds). Insufficiently large embedding window does not allow clear separation of different frequencies, as expected theoretically, while an overly long embedding window introduces low-frequency modes (not of interest here, thus not shown). Long embedding windows also split high-frequency modes, introducing redundancy (given the purpose of our study). As such, we show that a theoretically meaningful response to ‘deviant’ sounds appears robust within a suitable range of the embedding window (see Figs. 2 and 3). B. Temporal coordinates Fig. 2 displays three representative Koopman eigenfunctions and their corresponding frequency spectra. These modes emerge among the dominant ones (in the sense of having minimal Dirichlet energy; see Section III-C) in the hierarchy of modes retrieved by Koopman analysis; specifically, they

correspond to the complex conjugate pairs {z5 , z6 }, {z9 , z10 }, and {z12 , z13 } (henceforth abbreviated z5,6 , z9,10 , and z12,13 , respectively). Their frequency peaks differ slightly, yet all of them are within a range of 50–100 Hz. These response frequencies correspond to the Beta band in several primate species including marmoset [35]–[37]. Our result confirms and extends prior studies that find a change in beta frequency associated with sound deviants or natural dynamically modulated sounds [35], [36]. However, here we identify multiple peaks within the beta range without having to select an arbitrarily defined range of modulation frequencies. Among the pairs z5,6 , z9,10 , and z12,13 examined here, eigenfunction pair z5,6 is the most periodic in nature, exhibiting a prominent 13 Hz (77 rad/s) spectral peak. Eigenfunctions z9,10 and z12,13 have frequency peaks at 10 Hz and 16 Hz, respectively. Their temporal patterns are more irregular and with a less prominent low-frequency envelope. A similar decomposition of complex signal into periodic and irregular patterns has been observed in the case of other dynamical systems [27], but it can be achieved with sufficiently long embedding windows (this is also true for traditional methods such as SSA). In [8], the difference between evoked-response potentials following the last and first tone of each train were averaged across all trains for each electrode, and the difference in averages tested for statistical significance. Fig. 3 depicts data aligned similarly for electrode 25 which corresponds to primary auditory cortex, A1, according to [8]. Here, the extracted

Fig. 4. Spatiotemporal composite of response to deviant signal reconstructed using selected Koopman eigenfunctions extracted with a 500 ms embedding window. a) At 10 ms, positive valence peaks in auditory (Aud) and parietal (Par) cortices follow the deviant sound onset and b) At 28ms, there is subsequent development of negative valence peaks in auditory cortex near electrode 24 and in frontal (Fr) cortex near electrode 14. c) At 30ms, the negative valence signal strengthens and expands in core and surrounding auditory cortex (electrodes 22–29) and the negative signal travels towards parietal cortex (electrode 1). d) At 50ms, the negative valence signal in parietal cortex is enhanced and propagates from electrode 1 to the rest of the brain. e) Then at 92ms, a second pair of high amplitdue, positive valence peaks appears in about the same positions as seen for 10 ms. f) A second negative valence peak follows at approximately 130 ms.

pattern was averaged for 13 consecutive onsets of deviant and 13 preceding standard tones. Fig. 3 shows the reconstructed signal for electrode 25, obtained via the procedure described in Section III-D using eigenfunctions z5,6 , z9,10 , and z12,13 . To extract an average response to deviant sound, the reconstructions are averaged over the 13 response trials analyzed here. The average corresponds to the more periodic conjugate pair of the temporal pattern z5,6 after subtracting the overlap with the more chaotic z9,10 and z12,13 . It is worth noting that this projection captures amplification of the signal shortly after the appearance of the deviant. In particular, a large positive peak is observed at approximately 10 ms followed by a large negative peak at 25–50 ms. A second large negative peak is observed around 125 ms after the ‘deviant’ sound, and third follows at approximately 200 ms. The temporal dynamic of the mode suggests distinct onset and terminating responses following deviant sound onset matches well with the dynamics observed in core or primary auditory cortices in other studies [38]–[41]. The reader should also notice that all signs, amplitudes, and time relative to the signal onset of the deviant match well with those depicted in Fig. 2 of Ref. [8]. Subsequent negative and positive events are followed by the particular signal weakening (with slight strengthening observed at t = 503 ms), which corresponds to the standard tone occurrence (Fig. 3, gray line,

503 ms). The successful decomposition of the signal can help address several questions related to the underlying nonlinear dynamics and the puzzling statistics associated with them (asymmetry and skewness of dominant modes [28]). Here, we observe that spatiotemporal reduction of a multidimensional signal into three coordinates permits reconstruction of an intermittent neural response and its temporal emergence following the deviant signal. In summary, our results suggest that MMN can be captured by a reduced number of coordinates (temporal modes) derived from the eigenfunctions of an operator governing the evolution of observables under neuronal dynamics. These patterns describe brain dynamic shifts between quasiperiodic and more chaotic regimes, operating over a wide range of the frequency spectrum. Such intermittent dynamics are known to be poorly captured by Fourier methods and are better handled with wavelet transforms [42], [43]. C. Statistical tests Here, we conduct statistical tests matching those performed by the authors of the experiment to compare the significance values obtained in the primary analysis with the secondary analysis presented here using the extracted signal. Using the guidelines provided in [8] we compare ERPs for standard

and deviant stimuli for each channel for the first 250 ms following the onset of the stimuli. We adjust all p-values with the false discovery rate (FDR). Using Wilcox ranked sum tests, we examine a total of 12 channels with significant p-values reported in [8]. We replicate the findings presented in Fig. 3a of [8]. We should note, that the p-values we present in Table 1 are smaller than the p-values in [8]. This reduction in the probability of type I error is very likely driven by the reduction in the error term when comparing amplitude differences of the extracted frequency band. TABLE I W ILCOX RANK SUM TEST Cortex Area Frontal

Primary Analysis < .1; < .1

p-values

Temporal

< .05; < .05; < .05; < .05

Parietal

< .1; < .1; < .1

Occipital

< .05, < .1; < .1,

Current Results 1.06e 7; .01

1.252e 13; 2.2e 16, 7.66e 10; 8.495e 13 6.258e 08; 1.011e 10; .34 < 2.2e 16; < 2.2e 16; 9.501e 09

D. Spatiotemporal dynamics As derived in Section II, reconstruction of the signal associated with the given Koopman eigenfunctions can be performed for every spatial point. Using this approach, we construct a spatiotemporal composite of temporal patterns like those illustrated in Fig. 2. Specifically, Fig. 4 shows representative snapshots of the reconstructed signal for the full array consisting of 32 electrodes, obtained by averaging 13 deviant sound onset events, and combined from three reconstructed modes for every electrode in the manner described above. If the method is successful in capturing spatiotemporal dynamics, we should expect the following: 1) High-amplitude, short-latency (< 100 ms) response for electrodes located in or near the primary auditory cortex (e.g., electrode 25). 2) A directional and time-delayed propagation of the neural response from auditory cortex to other brain regions. 3) A confirmation that the change in amplitude is evident in the parietal areas [44]. As in Fig. 3, we see the short latency (10 ms), positive response that is maximal in auditory and parietal cortices (Fig. 4a, dark red areas). This short latency peak is evident in much of the temporal and parietal cortices. The large negative peak seen in Fig. 2 at 25–50 ms following the onset of the ‘deviant’, is evident in two subsequent snapshots (28 and 30 ms) of the complete array in Fig. 4b and 4c, respectively, as a negative (cyan color) valence response. In the short time between snapshots, this short latency response expands from a presumed auditory core area (Fig. 4b, electrodes 23 and 24) to encompass the core plus the surrounding auditory areas (Fig. 4c, electrodes 22–29). In these early snapshots, there is also a prominent negative valence potential that starts in

frontal cortex and propagates in the posterior direction (Fig. 4b versus 4c, cyan area). In successive snapshots, the amplitude of these temporal and frontal cortical negative valence responses decreases and a wave propagates from temporal cortex towards the parietal cortex (Fig. 4c, cyan area around electrode 1). At a later time point (50 ms), the negative valence response increases in magnitude and expands within parietal cortex (Fig. 4d, green area around electrodes 1, 2, 6 and 7). At 92 ms, we observe large positive valence responses in posterior auditory cortex and within parietal cortex (Fig. 4e, dark red area). Given that the deviant tone duration is 64ms, it is feasible that the late positive response at 92 ms is what physiologists refer to as a sound offset or terminating response. Though such responses are presumably driven by the sound dynamic they likely also reflect neural delays or other dynamics of the marmoset brain. A second negative response with a similar spatiotemporal dynamic is evident in the response snapshot for 130 ms (Fig. 4f). These results indicate that this is a robust approach for characterizing multiple dynamic spatiotemporal responses to deviant sound onset. V. S UMMARY AND FUTURE DIRECTIONS The results presented above provide support for the applicability of the spectral theory of dynamical systems in computational neuroscience. In particular, our analysis explores the applicability of data-driven Koopman eigenfunctions, as ways to capture neural markers that could expand the tools available for clinicians in diagnosing neurological conditions such as schizophrenia. We find that, given the relatively small spatial dimension of the system (i.e., 32 electrodes), computational resources currently available allow researchers to analyze an astonishingly large number of samples – in our case, almost one million. This provides a unique opportunity for fast progress in the analysis of ECoG data, something that for higher-dimensional datasets (e.g., fully resolved turbulent flows) may not be possible for decades. The capacity of this approach to analyze large samples may be especially helpful given the advancements in, and success of, high-density, highchannel-count multiplexed micro-electrocardiographically arrays [38], which increase the spatial dimension of the observed system. The results of the spectral decomposition align well with the results from [8], as well as other related research. As we expected, similar significance levels were obtained for the decomposed signal with larger effect sizes given the subtraction of more chaotic patterns from the extracted signal. That said, follow-up tests should be performed to ensure the validity of this claim. The results also show that mismatch negativitytype signals can be successfully extracted with a small number of coordinates. Given similarities between animal models of mismatch negativity and their human counterparts, the applications of this method are promising for computational neuroscientific studies of humans. The decomposition of the signal using a large delayembedding window was successful in replicating several characteristics of MNN described in earlier research that held space

or time constant. This approach has proven useful in other applications as well. In particular, prevalent indexes of some coherent structures, known to occur frequently in the Earth’s climate system [45], [46], can be derived from a combination of different modes. Future validation of our findings with other spectral approaches [47], [48] should allow for more detailed examination of the derived patterns and the dynamical nature of its components. The key strengths of the current approach lie in its capacity of extracting meaningful patterns of brain activity, without prior filtering of the data. While the specific frequency this paper focuses on was previously associated with MMN [35], the method allows to index and examine a wide range of frequency bands. Further exploration of eigenfunctions other than those shown here, and the broad range of timescales (from milliseconds to minutes), open an avenue for both nonparametric indexing of many neural phenomena as well as investigation of the underlying spatiotemporal dynamics and corresponding physiological/biological/physical mechanisms. The Koopman operator framework does not rely on variance, which ensures that the dynamics of the system are not lost due to unknown and unaccounted for interactions that can obscure the underlying mechanics. Sensitivity tests examined the robustness of this method under varying levels of noise in the signal and provided computational support to the theoretical rationale behind embedding the data, and the ability for sufficiently large embedding windows to obtain a clear separation of different frequencies. Embedding the data, and using kernels that approximate Koopman eigenspaces, ensures that the computations are stable, efficient, and convergent in the limit of large data. Simultaneously, the separated frequencies can be tested for significance using traditional variance based and linear methods. The results of the spectral decomposition align well with the results from [8], Wilcox rank sum tests suggest that the effects found in [8], are also found using this method. It is worth pointing out that the current results were obtained using a random subset of the data. Additionally, the decomposition suggests that the statistically significant difference between evoked response potential following the deviant versus standard tones, may in part be driven by the synchronization of responses following the onset of the deviant sound. ACKNOWLEDGMENTS We thank Misako Komatsu, Kana Takaura, and Naotaka Fuji, and the RIKEN Brain Science Institute for making the data used in this paper available. N. M. acknowledges support from the CT Institute for the Brain and Cognitive Sciences Graduate Summer Fellowship and the University of Connecticut Department of Psychological Sciences’ Maurice L. Farber Endowment. H. R. acknowledges funding from NSF grant 1355065 and NIH DC015138 01 and the University of Connecticut Brain Computer Interface Core. J. S. acknowledges funding from NSF EAGER grant 1551489. D. G. received support from ONR YIP grant N00014-16-1-2649,

NSF grant DMS-1521775, and DARPA grant HR0011-16-C0116. The authors thank Ian Stevenson and Stephen Herzog for feedback on earlier drafts. R EFERENCES [1] J. M. Ford and S. A. Hillyard, “Event-related potentials (ERPs) to interruptions of a steady rhythm.” Psychophysiology, vol. 18, no. 3, pp. 322–330, 1981. [2] R. N¨aa¨ t¨anen, P. Paavilainen, K. Alho, K. Reinikainen, and M. Sams, “The mismatch negativity to intensity changes in an auditory stimulus sequence,” Electroencephalography and clinical neurophysiology, vol. 40, p. 125, 1987. [3] R. N¨aa¨ t¨anen, P. Paavilainen, and K. Reinikainen, “Do event-related potentials to infrequent decrements in duration of auditory stimuli demonstrate a memory trace in man?” Neuroscience Letters, vol. 107, no. 1, pp. 347–352, 1989. [4] K. Alho, “Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes.” Ear &. Hearing, vol. 16, no. 1, pp. 38–51, 1995. [5] C. Wacongne, “A predictive coding account of MMN reduction in schizophrenia,” Biological Psychology, vol. 116, pp. 68–74, 2016. [6] V. Cs´epe, G. Karmos, and M. Moln´ar, “Evoked potential correlates of stimulus deviance during wakefulness and sleep in cat animal model of mismatch negativity,” Electroencephalography and Clinical Neurophysiology, vol. 66, no. 6, pp. 571–578, 1987. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0013469487901039 [7] L. Harms, P. T. Michie, and R. N¨aa¨ t¨anen, “Criteria for determining whether mismatch responses exist in animal models: Focus on rodents,” Biological Psychology, vol. 116, pp. 28–35, 2016. [8] M. Komatsu, K. Takaura, and N. Fujii, “Mismatch negativity in common marmosets: Whole-cortical recordings with multi-channel electrocorticograms,” Scientific Reports, vol. 5, p. 15006, 2015. [9] S. Grimm, C. Escera, and I. Nelken, “Early indices of deviance detection in humans and animal models,” Biological Psychology, vol. 116, pp. 23– 27, 2016. [10] P. J. C. May and H. Tiitinen, “Mismatch negativity (MMN), the deviance-elicited auditory deflection, explained,” Psychophysiology, vol. 47, no. 1, pp. 66–122, 2010. [Online]. Available: https: //search.proquest.com/docview/213477110 [11] K. Alho, D. L. Woods, A. Algazi, R. T. Knight, and R. N¨aa¨ t¨anen, “Lesions of frontal cortex diminish the auditory mismatch negativity.” Electroencephalography and Clinical Neurophysiology, vol. 91, no. 5, pp. 353–362, 1994. [12] M. H. Giard, F. Perrin, J. Pernier, and P. Bouchet, “Brain generators implicated in the processing of auditory stimulus deviance: a topographic event-related potential study,” Psychophysiology, vol. 27, no. 6, pp. 627–640, 1990. [Online]. Available: http://www.ncbi.nlm.nih. gov/pubmed/2100348 [13] U. Schall, “Is it time to move mismatch negativity into the clinic?” Biological Psychology, vol. 116, pp. 41–46, 2016. [14] V. Gumenyuk, T. Roth, O. Korzyukov, C. Jefferson, A. Kick, L. Spear, N. Tepley, and C. L. Drake, “Shift work sleep disorder is associated with an attenuated brain response of sensory memory and an increased brain response to novelty: An ERP study,” Sleep, vol. 33, pp. 703–713, 2010. [15] D. C. Javitt, “Intracortical mechanisms of mismatch negativity dysfunction in schizophrenia,” Audiology and Neurotology, vol. 5, no. 3-4, pp. 207–215, 2000. [16] D. S. Manoach and R. Stickgold, “Sleep, memory and schizophrenia,” Sleep medicine, vol. 16, pp. 553–554, 2015. [17] N. Aubry, W.-Y. Lian, and E. S. Titi, “Preserving symmetries in the proper orthogonal decomposition,” SIAM J. Sci. Comput., vol. 14, pp. 483–505, 1993. [18] D. Giannakis, J. Slawinska, and Z. Zhao, “Spatiotemporal feature extraction with data-driven Koopman operators,” J. Mach. Learn. Res. Proceedings, vol. 44, pp. 103–115, 2015. [19] D. Giannakis, “Data-driven spectral decomposition and forecasting of ergodic dynamical systems,” Appl. Comput. Harmon. Anal., 2017. [20] S. Das and D. Giannakis, “Delay-coordinate maps and the spectra of Koopman operators,” Nonlinearity, 2017, in review. [Online]. Available: http://arxiv.org/abs/1706.08544 [21] B. O. Koopman, “Hamiltonian systems and transformation in Hilbert space,” Proc. Natl. Acad. Sci., vol. 17, no. 5, pp. 315–318, 1931.

[22] M. Dellnitz and O. Junge, “On the approximation of complicated dynamical behavior,” SIAM J. Numer. Anal., vol. 36, p. 491, 1999. [23] I. Mezi´c, “Spectral properties of dynamical systems, model reduction and decompositions,” Nonlinear Dyn., vol. 41, pp. 309–325, 2005. [24] M. Belkin and P. Niyogi, “Laplacian eigenmaps for dimensionality reduction and data representation,” Neural Comput., vol. 15, pp. 1373– 1396, 2003. [25] R. R. Coifman and S. Lafon, “Diffusion maps,” Appl. Comput. Harmon. Anal., vol. 21, pp. 5–30, 2006. [26] D. Giannakis and A. J. Majda, “Nonlinear Laplacian spectral analysis for time series with intermittency and low-frequency variability,” Proc. Natl. Acad. Sci., vol. 109, no. 7, pp. 2222–2227, 2012. [27] J. Slawinska and D. Giannakis, “Indo-Pacific variability on seasonal to multidecadal time scales. Part I: Intrinsic SST modes in models and observations,” J. Climate, vol. 30, no. 14, pp. 5265–5294, 2017. [28] D. Giannakis and J. Slawinska, “Indo-Pacific variability on seasonal to multidecadal time scales. Part II: Multiscale atmosphere-ocean linkages,” J. Climate, vol. 31, pp. 693–725, 2018. [29] T. Eisner, B. Farkas, M. Haase, and R. Nagel, Operator Theoretic Aspects of Ergodic Theory, ser. Graduate Texts in Mathematics. Springer, 2015, vol. 272. [30] D. Giannakis and A. J. Majda, “Time series reconstruction via machine learning: Revealing decadal variability and intermittency in the North Pacific sector of a coupled climate model,” in Conference on Intelligent Data Understanding 2011, Mountain View, California, 2011. [31] ——, “Nonlinear Laplacian spectral analysis: Capturing intermittent and low-frequency spatiotemporal patterns in high-dimensional data,” Stat. Anal. Data Min., vol. 6, no. 3, pp. 180–194, 2013. [32] D. Giannakis, “Dynamics-adapted cone kernels,” SIAM J. Appl. Dyn. Sys., vol. 14, no. 2, pp. 556–608, 2015. [33] M. Ghil et al., “Advanced spectral methods for climatic time series,” Rev. Geophys., vol. 40, 2002. [34] M. Budisi´c, R. Mohr, and I. Mezi´c, “Applied Koopmanism,” Chaos, vol. 22, p. 047510, 2012. [35] C. Haenschel, T. Baldeweg, R. J. Croft, M. Whittington, and J. Gruzelier, “Gamma and beta frequency oscillations in response to novel auditory stimuli: A comparison of human electroencephalogram (EEG) data with in vitro models components analysis of human telomeric quadruplex multimers,” Proceedings of the National Academy of Sciences of the United States of America, vol. 97, pp. 7645–7650, 2000. [36] C. Chandrasekaran, H. K. Turesson, C. H. Brown, and A. A. Ghazanfar, “The influence of natural scene dynamics on auditory cortical activity,” The Journal of Neuroscience, vol. 30, no. 42, p. 13919, 2010. [37] B. Tia, M. Takemi, A. Kosugi, E. Castagnola, A. Ansaldo, T. Nakamura, D. Ricci, J. Ushiba, L. Fadiga, and A. Iriki, “Cortical control of object-specific grasp relies on adjustments of both activity and effective connectivity:A common marmoset study,” The Journal of Physiology, vol. 595, no. 6, pp. 7203–7221, 2017. [38] M. A. Escabi, H. L. Read, J. Viventi, D.-H. Kim, N. C. Higgins, D. A. Storace, A. S. K. Liu, A. M. Gifford, J. A. Burke, M. Campisi, Y.-S. Kim, A. E. Avrin, J. Van der Spiegel, Y. Huang, M. Li, J. Wu, J. A. Rogers, B. Litt, and Y. E. Cohen, “A high-density, high-channel count, multiplexed µECoG array for auditory cortex,” J Neurophysiol, vol. 112, pp. 1566–1583, 2014. [39] C. M. Lee, A. F. Osman, M. Volgushev, M. A. Escab, and H. L. Read, “Neural spike-timing patterns vary with sound shape and periodicity in three auditory cortical fields,” J Neurophysiol, vol. 115, pp. 1886–1904, 2016. [40] B. Ross, B. Schneider, J. S. Snyder, and C. Alain, “Biological markers of auditory gap detection in young, middle-aged, and older adults,” PLoS ONE, vol. 5, no. 4, p. e10101, 2010. [41] L. S. Hamilton, E. Edwards, and E. F. Chang, “Parallel streams define the temporal dynamics of speech processing across human auditory cortex,” bioRxiv, 2017. [42] F. Cong, T. Sipola, T. Huttunen-Scott, X. Xu, T. Ristaniemi, and H. Lyytinen, “Hilbert-Huang versus Morlet wavelet transformation on mismatch negativity of children in uninterrupted sound paradigm,” Nonlinear Biomedical Physics, vol. 3, no. 1, p. 1, 2009. [Online]. Available: https://doi.org/10.1186/1753-4631-3-1 [43] F. Cong, I. Kalyakin, H. Li, T. Huttunen-Scott, Y. Huang, H. Lyytinen, and T. Ristaniemi, “Answering six questions in extracting childrens mismatch negativity through combining wavelet decomposition and independent component analysis,” Cognitive Neurodynamics, vol. 5,

[44]

[45]

[46]

[47] [48]

no. 4, p. 343, 2011. [Online]. Available: https://doi.org/10.1007/ s11571-011-9161-1 S. E. MacLean, E. G. Blundon, and L. M. Ward, “Brain regional networks active during the mismatch negativity vary with paradigm,” Neuropsychologia, vol. 75, pp. 242–251, 2015. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0028393215300671 J. Slawinska and D. Giannakis, “Spatiotemporal pattern extraction with data-driven Koopman operators for convectively coupled equatorial waves,” in Proceedings of the 6th International Workshop on Climate Informatics, A. Banerjee, W. Ding, J. Dy, V. Lyubchich, and A. Rhines, Eds., Boulder, Colorado, 2016, pp. 49–52. J. Slawinska, E. Sz´ekely, and D. Giannakis, “Data-driven Koopman analysis of tropical climate space-time variability,” in Proceedings of Workshop on Mining Big Data in Climate and Environment (MBDCE 2017), 17th SIAM International Conference on Data Mining (SMD 2017), Houston, Texas, 2017. [Online]. Available: https://arxiv.org/abs/1711.02526 D. Giannakis, A. Ourmazd, J. Slawinska, and Z. Zhao, “Spatiotemporal Pattern Extraction by Spectral Analysis of Vector-Valued Observables,” 2017, submitted. [Online]. Available: http://arxiv.org/abs/1711.02798 D. Giannakis, J. Slawinska, A. Ourmazd, and Z. Zhao, “Vector-Valued Spectral Analysis of Space-Time Data,” in Proceedings of the Time Series Workshop, Neural Information Processing Systems Conference, Long Beach, California, 2017.