Procedia Computer Science Volume 53, 2015, Pages 48–55 2015 INNS Conference on Big Data
Connectomics to Semantomics: Addressing the Brain’s Big Data Challenge∗ Xerxes D. Arsiwalla1 , David Dalmazzo1 , Riccardo Zucca1 , Alberto Betella1 , Santiago Brandi1 , Enrique Martinez1 , Pedro Omedas1 , and Paul Verschure1,2 1
Synthetic Perceptive Emotive and Cognitive Systems (SPECS) Lab, Center of Autonomous Systems and Neurorobotics, Universitat Pompeu Fabra, Barcelona, Spain.
[email protected] 2 Instituci´ o Catalana de Recerca i Estudis Avan¸cats (ICREA), Barcelona, Spain.
Abstract Can semantic corpora be coupled to dynamical simulations in such a way so as to extract new associations from the data that were hitherto unapparent? We attempt to do this within neuroscience as an application domain, by introducing the notion of the semantome and coupling it to the connectome of the human brain network. This is implemented using BrainX3 , a virtual reality simulation cum data mining platform that can be used for visualization, analysis and feature extraction of neuroscience data. We use this system to explore anatomical, functional and symptomatic semantics associated to simulated neuronal activity of a healthy brain, one with stroke and one perturbed by transcranial magnetic stimulation. In particular, we find that parietal and occipital lesions in stroke affect the visual processing pathway leading to symptoms such as visual neglect, depression and photo-sensitivity seizures. Integrating semantomics with connectomics thus generates hypotheses about symptoms, functions and brain activity that supplement existing tools for diagnosis of mental illness. Our results suggest a new approach to big data with potential applications to other domains. Keywords: Brain Connectomics, Data Mining, Virtual Reality
1
Introduction
In attempts to stem the data deluge, traditional approaches to big data usually rely on massive computational resources. Even though this approach to data crunching may often be necessary, it may not suffice to outpace the exploding rate of data production. Take for instance, data generated within neuroscience. The adult human brain has on average 86 billion neurons with about 1014−15 synapses [9], exchanging information continuously, thus giving rise to cognition and conscious behavior. Neuronal signaling is not the only form of communication in the ∗ All
authors contributed equally.
48Selection and peer-review under responsibility of the Scientific Programme Committee of INNS-BigData2015 c The Authors. Published by Elsevier B.V.
doi:10.1016/j.procs.2015.07.278
Connectomics to Semantomics
Arsiwalla et al.
brain. Molecular signaling cascades due to protein-protein interactions and tissue-specific gene expression patterns are also known to play a role in brain function and disease (especially, those originating during development). Precision measurements and simulations of these multi-scale processes are needed to understand causal mechanisms and dynamics of disease. However, this approach generates overwhelming amounts of data in the order of terabytes or greater, which is already at the limits of current computational technology.
This calls for alternative approaches to big data. One such attempt within neuroscience is the BrainX3 platform [1], [5], [2]. BrainX3 is a large-scale simulation of the human brain with real-time interaction, rendered in 3D in a virtual reality environment. The idea is to combine pure machine computational power with human intuition for the exploration and analysis of complex dynamical networks. On this platform, we have reconstructed a large-scale 3D simulation of brain activity in virtual reality [2]. The simulation is grounded on human cerebral structural connectivity data obtained from diffusion spectrum imaging studies [8]. Based on this dataset we model neuronal dynamics to simulate large-scale activity over the cortex. Users can interact with BrainX3 by perturbing brain regions with transient stimulations to observe reverberating network activity, simulate lesion dynamics or implement network analysis functions from a library of graph theoretic measures. Within the immersive mixed/virtual reality space of BrainX3 users can explore and analyze dynamic activity patterns of brain networks, both at rest or during tasks, or they can use the system for discovering signaling pathways associated with brain function/dysfunction, or they could use it as a tool for virtual neurosurgery. The specific simulations discussed in [2] reveal the spatial distribution of activity in the attractor state, how the brain maintains a level of resilience to damage, effects of noise and physiological perturbations. Knowledge of brain activity in these varied states is clinically relevant for assessing levels of consciousness in patients with severe brain injury.
The aim of this paper is to extend the functionality of BrainX3 to operate as a full-fledged hypotheses generator for large neuroscience datasets. As is often the case with complex data, one might not always have a specific hypothesis to start with. Instead, discovering meaningful patterns and associations within the data might be a necessary incubation step for formulating well-defined hypotheses that can then be tested with specific generative models. Hence, we introduce the notion of the semantome, a layer of semantics, conceived as a curated corpora of available information on brain regions, functions, diseases, etc. This layer is coupled to the connectomics of BrainX3 in such a way that new associations in the semantomics layer are driven and shaped by neuronal activity generated in the connectomics layer. How does this work? Every individual brain region (nodes of the connectome) is well-documented in the literature and implicated in multiple cognitive functions and diseases. However, large-scale oscillations in the brain result from collective dynamics of locally coupled oscillators. Hence, stimulating a single sensory area activates related circuits that integrate sensory information. This serves as the basis of multi-modal integration in the brain. It is this integrated functionality that we seek to understand at the level of semantomics. To do this, we couple the anatomical network in BrainX3 with available semantic corpora and develop a data mining engine to search across layers of information so as to generate meaningful associations between the neuronal simulation and semantic corpora. Based on the simulated activity, users can query the system for specific brain functions or diseases associated to activated pathways, while the data mining functionality searches and represents associated information either as text or graphs. 49
Connectomics to Semantomics
Arsiwalla et al.
Figure 1. Schema of BrainX3 within the eXperience Induction Machine (XIM). The 3D connectome network and its simulated dynamics are projected on the frontal screen. The screen on the right displays regional information of selected brain areas from a curated database, while the left screen shows 2D axial slices of the brain and indicates regions of activity. The user can navigate and interact with the model with predefined hand gestures. (Figure courtesy [2])
2
Methods
The virtual reality environment supporting BrainX3 is the eXperience Induction Machine (XIM) [3], [6], [11]. The XIM is a 25m2 human accessible space (schema in fig. 1) equipped with 360 degrees surround screens, an interactive luminous floor with pressure sensors, a marker-free tracking system, a KinectTM , microphones, a sonification system and wearable sensors, that support human-machine interaction in the exploration of complex datasets. Within this virtual reality space, BrainX3 is a data visualization, simulation and interaction system [1], [5], [2].
2.1
Visualization and Simulation of Brain Networks
Data visualization in BrainX3 is rendered using Unity 3D (Unity Technologies, CA, USA). We use human brain connectivity data obtained from diffusion spectrum imaging (DSI) [8], which consists of averaged white matter connectivity across 998 voxels (nodes of the network, also referred to as Regions of Interest or ROIs) spanning 66 cortical regions. Each ROI has an average size of 1.5 cm2 and is given coordinates as per the Talairach atlas [13]. The connectivity matrix is symmetric (consisting about 17,000 bi-directional connections). Connection strengths (weighted edges of the network) denote normalized number of fibre tracts between ROIs. We simulate neural activity using iqr, a large-scale multi-level neural network simulator that is bi-directionally interfaced to Unity. iqr allows users to design complex neuronal models and modify model parameters online [4]. In particular, we use the dynamical mean-field model, obtained as a mean-field approximation of spiking neural networks [14], [7]. This model computes aggregate activity for populations of neurons.
2.2
User Interaction
BrainX3 includes real-time user interaction with features to perturb simulated activity using predefined gestures, rotate the visualization and zoom in/out (fig. 2). BrainX3 is also interfaced with the Brain Connectivity Toolbox for performing graph-theoretic analysis on the simulated 50
Connectomics to Semantomics
Arsiwalla et al.
Figure 2. User immersion and interaction within BrainX3 . (Figure courtesy [2]) activity, such as finding the shortest path between two regions or detecting community structure [12]. Other customized interaction features include highlighting pathways, filtering network complexity and modeling lesions by disabling nodes. Interaction is facilitated via a sensing layer that collects data pertaining to user gestures and body movements (using the Microsoft KinectTM v2) and maps this information back to actions within the system [11].
2.3
The Semantome and Data Mining
BrainX3 features a new graphical user interface to display semantic data. We developed a web application program interface (API) that queries a curated database about all brain regions (as per the Talairach atlas), their images, functions, diseases and related literature. We collected data from peer reviewed scientific literature, NeuroLex [10] and Wikipedia (http://wikipedia. org). Each record in our database consists of attributes such as anatomical name, abbreviation, structural description, functions, associated diseases, images and references. Once queried, the API searches back-end for matching records in the relational database and returns attributes associated to a certain record in JSON format, which are then parsed by BrainX3 and displayed in the GUI.
3
Results
Below we demonstrate results obtained from integration of semantic and connectomic data. The system can be operated in the four modes described below.
3.1
Exploratory Search
In the exploratory mode, the user can scroll over the network, either with a Kinect-based gesture or an external device, and click over a node label. Information corresponding to that node from the three curated databases appears on the right-hand side. Fig. 3 shows results of a simulation of resting-state neural activity in a healthy brain after the dynamics have stabilized at the attractor point. The cursor click over the precuneus region displays anatomical information, cognitive functions and disorders associated to this region on the right-side of the figure.
3.2
Query Search
As shown in fig. 4, this functionality enables a bridge between the connectomics and semantomics of BrainX3 . Query keywords referring to specific anatomical structures, symptoms or functions can be entered in multiple GUI search fields (represented as circular nodes). These 51
Connectomics to Semantomics
Arsiwalla et al.
Figure 3. Exploratory search in BrainX3 . The simulation refers to healthy resting state dynamics. The cursor is shown on the precuneus area, whose database information has been displayed on the right side of the screen.
Figure 4. Query search in BrainX3 . The simulation refers to a stroke patient with a lesion in areas rCUN, rLOCC and rPCUN. User queries are entered in GUI search boxes and BrainX3 associates them to the affected regions.
keywords are displayed on the screen within these nodes that form part of the semantic layer. BrainX3 parses through the databases and associates each query with multiple regions (shown in fig. 4 as dashed white lines) in the connectome network that correspond to that keyword. The simulated network in fig. 4 is that of a stroke patient with a lesion in the right hemisphere over the brain areas: right cuneus (rCUN), right lateral occipital cortex (rLOCC) and right precuneus (rPCUN). All white matter fiber connections from these areas are severed, amounting to about 6.64% of the total white matter connections. The displayed queries: neglect, depression and sensory seizures correspond to coexisting symptoms resulting from the lesion in these areas.
52
Connectomics to Semantomics
Arsiwalla et al.
Figure 5. Efferent mapping for circuit extraction in BrainX3 . The simulation refers to a healthy brain with TMS applied to areas rCUN, rLOCC and rPCUN.
3.3
Efferent Mapping
The simulation in fig. 5 refers to brain activity during an inhibitory transcranial magnetic stimulation (TMS) applied to brain areas rCUN, rLOCC and rPCUN (we choose the same areas which were earlier lesioned to simulate a stroke). The TMS completely shuts down neuronal activity in these areas and consequently, efferent projections from these areas do not transmit any activity. Knowledge of regions affected by these projections and their associated affected functionalities reveals the influence of TMS in the brain beyond the localized foci of perturbation. BrainX3 maps these efferent projections. These projections are shown in the darkened edges in fig. 5, giving a visual impression of the stimulated circuit. Though the stimulation was only applied to areas in the occipital and parietal lobes, its effects are felt as far as the frontal, temporal and limbic lobes. Extracting circuits this way is intuitive and user controllable, compared to automated processes based on statistical correlations.
3.4
Semantic Mapping
In fig. 6 we demonstrate the construction of semantic graphs corresponding to user-specified brain functions or disorders. The example shows the extraction of the visual processing pathway affected by a lesion (in areas rCUN, rLOCC and rPCUN) in this simulation. The user can enter names of functions or disorders in the semantic search box on the top right of the screen. The system identifies the main hubs in and around the lesioned area that are associated to the search query and marks these hubs with red flags and then constructs a graph denoting the affected pathway. Fig. 6 shows that the visual processing pathway, bi-directionally connecting the Lingual Gyrus to the Cuneus to the Lateral Occipital Cortex, is affected when areas rCUN, rLOCC and rPCUN are lesioned.
4
Discussion
In this paper, we have presented BrainX3 as a simulation cum data mining tool that builds associations between semantic corpora and network simulations in order to derive meaning or 53
Connectomics to Semantomics
Arsiwalla et al.
Figure 6. Semantic mapping in BrainX3 . Simulation refers to the same lesion as in fig. 4. The semantic query now extracts functional pathways, in this case, the visual processing pathway in the affected regions. This pathway is displayed on the right as a graph connecting the LING, CUN and LOCC. generate plausible hypotheses from observed patterns in network activity. We have shown that integrating available neuroscience databases with simulations of neuronal network dynamics is a first step in extracting new semantic associations about brain function that would not have been apparent by looking at either one of the two layers, the corpora or the network simulations, independently of each other. The enhanced BrainX3 system can be operated in four modes: (i) exploratory search, (ii) query search, (iii) efferent mapping and (iv) semantic mapping. While the exploratory search and efferent mapping are generally useful for exploration and mapping interesting regions of the network, its anomalous activity or functions, the query search and semantic mapping features of BrainX3 are novel in that they emphasize interactions between semantomics and connectomics. As a concrete example of harnessing these semantomic-connectomic interactions in the case of stroke simulations, we found that lesions in the parietal and occipital areas affect the visual processing pathway and that these patients are prone to symptoms such as visual neglect, depression and photo-sensitivity seizures. As described in this paper, the enhanced BrainX3 system thus generates hypotheses about symptoms, functions and anomalous activity in the brain that can supplement existing imaging and physiological tools for the diagnosis of disease.
4.1
Acknowledgments
We thank Olaf Sporns and Chris Honey for sharing the DSI data. Research supported by CEEDS (FP7-ICT-2009-5) & CDAC (ERC-2013-ADG 341196) projects.
References [1] X D Arsiwalla, Alberto Betella, E. Mart´ınez, P. Omedas, R. Zucca, and PFMJ Verschure. The dynamic connectome: a tool for large scale 3d reconstruction of brain activity in real time. In 27th European Conference on Modeling and Simulation, Alesund (Norway), 2013. ECMS W. Rekdalsbakken, R. Bye, H. Zhang eds., ECMS W. Rekdalsbakken, R. Bye, H. Zhang eds.
54
Connectomics to Semantomics
Arsiwalla et al.
[2] Xerxes D. Arsiwalla, Riccardo Zucca, Alberto Betella, Enrique Martinez, David Dalmazzo, Pedro Omedas, Gustavo Deco, and Paul Verschure. Network dynamics with brainx3: A large-scale simulation of the human brain network with real-time interaction. Frontiers in Neuroinformatics, 9(2), 2015. [3] Ulysses Bernardet, Sergi Berm´ udez i Badia, Armin Duff, Martin Inderbitzin, Sylvain Le Groux, Jˆ onatas Manzolli, Zenon Mathews, Anna Mura, Aleksander V¨ aljam¨ ae, and Paul F.M.J Verschure. The eXperience Induction Machine: A New Paradigm for Mixed-Reality Interaction Design and Psychological Experimentation. In Emmanuel Dubois, Philip Gray, and Laurence Nigay, editors, The Engineering of Mixed Reality Systems, Human-Computer Interaction Series, pages 357–379. Springer London, London, 2010. [4] Ulysses Bernardet and Paul F M J Verschure. iqr: a tool for the construction of multi-level simulations of brain and behaviour. Neuroinformatics, 8(2):113–134, 2010. [5] Alberto Betella, Ryszard Cetnarski, Riccardo Zucca, Xerxes D. Arsiwalla, Enrique Martinez, Pedro Omedas, Anna Mura, and Paul F. M. J. Verschure. BrainX3: embodied exploration of neural data. In Virtual Reality International Conference (VRIC ’14), Laval, France, 2014. [6] Alberto Betella, Riccardo Zucca, Ryszard Cetnarski, Alberto Greco, Antonio Lanat` a, Daniele Mazzei, Alessandro Tognetti, Xerxes D Arsiwalla, Pedro Omedas, Danilo De Rossi, and Paul F M J Verschure. Inference of Human Affective States from Psychophysiological Measurements Extracted under Ecologically Valid Conditions. Frontiers in Neuroscience, 8(286), 2014. [7] Gustavo Deco, Adri´ an Ponce-Alvarez, Dante Mantini, Gian Luca Romani, Patric Hagmann, and Maurizio Corbetta. Resting-state functional connectivity emerges from structurally and dynamically shaped slow linear fluctuations. The Journal of Neuroscience, 33(27):11239–11252, 2013. [8] Patric Hagmann, Leila Cammoun, Xavier Gigandet, Reto Meuli, Christopher J Honey, Van J Wedeen, and Olaf Sporns. Mapping the Structural Core of Human Cerebral Cortex. PLoS Biology, 6(7):15, 2008. [9] Suzana Herculano-Houzel. The human brain in numbers: a linearly scaled-up primate brain. Frontiers in human neuroscience, 3, 2009. [10] Stephen D Larson and Maryann E Martone. Neurolex. org: an online framework for neuroscience knowledge. Frontiers in neuroinformatics, 7, 2013. [11] Pedro Omedas, Alberto Betella, Riccardo Zucca, Xerxes D Arsiwalla, Daniel Pacheco, Johannes Wagner, Florian Lingenfelser, Elisabeth Andre, Daniele Mazzei, Antonio Lanat´ a, Alessandro Tognetti, Danilo de Rossi, Antoni Grau, Alex Goldhoorn, Edmundo Guerra, Rene Alquezar, Alberto Sanfeliu, and Paul F M J Verschure. XIM-engine: A Software Framework to Support the Development of Interactive Applications That Uses Conscious and Unconscious Reactions in Immersive Mixed Reality. In Proceedings of the 2014 Virtual Reality International Conference, VRIC ’14, pages 26:1—-26:4. ACM, 2014. [12] Mikail Rubinov and Olaf Sporns. Complex network measures of brain connectivity: uses and interpretations. Neuroimage, 52(3):1059–1069, 2010. [13] Jean Talairach and Pierre Tournoux. Co-planar stereotaxic atlas of the human brain. 3-dimensional proportional system: an approach to cerebral imaging. 1988. [14] Kong-Fatt Wong and Xiao-Jing Wang. A recurrent network mechanism of time integration in perceptual decisions. The Journal of neuroscience, 26(4):1314–1328, 2006.
55
