Using Brain-Computer Interface to Control an Avatar in a Virtual Reality Environment Berthil Borges Longo
Alessandro Botti Benevides, Javier Castillo, Teodiano Bastos-Filho
Post-Graduate Program in Biotechnology Universidade Federal do Esprito Santo Vitoria, Brazil Email:
[email protected]
Post-Graduate Program in Electrical Engineering Universidade Federal do Esprito Santo Vitoria, Brazil Email:
[email protected];
[email protected]
Abstract—The proposal of this research is to present the development of a tool that might be useful in rehabilitation, for subjects with disability, that suffer from some kind of limbs movement limitation. This tool carries a 3D Virtual Reality Environment (VRE), which emulates the movement of a healthy person, using the immersion of the subject through an avatar. To do so, and test its feasibility, pre acquired motor imagery signals were used to test the VRE as an off-line Brain Computer Interface (BCI) feedback. The subject’s brain waves were captured by an Electroencephalography (EEG) equipment. For training the classifier, 45 trials, 25 seconds long, were used, and 15 trials for its validation. Five mental tasks were tested with the BCI, and the one with the best results (imagination of the manipulation of a cube) was used to move the avatar through a virtual room. Index Terms—Motor Imagery, Brain-Computer Interfaces, EEG, Virtual Reality Environment, 3D Virtual Environment.
I. I NTRODUCTION A brain-computer interface (BCI) is a system for communication between human and computer, which allows people to send messages or commands directly from his/her brain to the external world without the use of peripheral nerves and muscles activities. These commands can be detected and recorded by a measurement equipment, such as Electrocorticogram (ECoG), Electroencephalography (EEG) Magnetoencephalography (MEG), Positron Emission Tomography (PET) or Functional Magnetic Resonance Imaging (fMRI). The first one is used in invasive kinds of BCI. It requires surgery to implant sensors to acquire the brain signals. The four last ones quoted above are used in non-invasive BCIs and there is no need of surgery to acquire the brain signals. After acquired, these signals are processed and can be used right away in the BCI (on-line BCI), or later (off-line BCI). This processing period is divided in three sub-stages: preprocessing, feature extraction, and classification, and the results can be sent as a command to the software/hardware, reflecting the user’s intent, and generating a prompt feedback to the user, in the case of an on-line BCI. Brain oscillations are typically categorized according to specific frequency bands which are named after Greek letters (delta: below 4 Hz, theta: 4-7 Hz, alpha: 8-12 Hz, beta: 12-30 Hz, gamma: above 30 Hz). The decrease of oscillatory activity energy in a specific frequency band
is called event-related desynchronization (ERD). Correspondingly, the increase of oscillatory activity in a specific frequency band is called event-related synchronization (ERS). ERD/ERS patterns can be voluntarily produced by motor imagery, which is the imagination of movement without actually performing the movement. The frequency bands that are most important for motor imagery are alpha and beta in EEG signals [1]. 3D VE is an advanced computer interface that allow it’s user to have a better immersion, using computer graphics to build an third dimension environment, and when used with additional sensory information, such as sound through speakers, it is defined as Virtual Reality Environment(VRE). VRE has a broad field of usage, e.g. advanced military training, virtual classrooms and flight/drive simulators. It has been used since 70’s [2] but only in recent years have research groups been attempting to connect BCIs and virtual worlds. However, several impressive prototypes already exist that enable users to navigate in virtual scenes or manipulate virtual objects solely by means of their cerebral activity, recorded on the scalp via electroencephalography (EEG) electrodes. Meanwhile, VRE technologies provide motivating, safe, and controlled conditions that enable improvement of BCI learning as well as the investigation of the brain responses and neural processes involved [3]. VREs can indeed provide a richer and more motivating feedback for BCI users than traditional feedbacks that are usually in the form of a simple 2-D bar displayed on screen. Therefore, a VRE feedback could enhance the learnability of the system, i.e., reduce the amount of time needed to learn the BCI skill, as well as increase the mental state classification performance [4]. II. M ATERIALS AND M ETHODS Most part of the VRE structure, the avatar and its movements, were build in the open source software BlenderTM 3D version 2.69 [5]. The VRE itself was created using the free c version of Unity game engine (from Unity Technologies ) [6], one of the most adequate software to this task [7]. On c the other hand, Gimp , a GNU GLPv3 free license software was used to manipulate images and create textures [8]. All this c cited software were used on a computer running Windows TM 7 64bits, with a AMD Phenom II X6 2.8GHz processor,
Fig. 1. The VRE simulates a rehabilitation room. The avatar walks a straight line followed by a third person camera.
R GeForceTM 460 video card 8GB of RAM memory, nVidia and a HDD of 250 GB. The VRE simulates a person in a rehabilitation room (figure 1), and it might be used as part of another project involving rehabilitation. The avatar stands still, and he is designed to walk in a straight line thought the room when a command is triggered. In this case the command is set as the ”w” key of the keyboard, and it is triggered using off-line motor imagery signal acquired previously, and not specifically obtained to this end. R Regarding the experimental environment, the BrainNet36 (BNT) equipment was used with a cap of integrated electrodes R from MedCap company. We used 19 electrodes positioned according to the international 10-20 system. The grounding electrode was positioned on the user forehead, monoauricular reference was adopted and all impedances were kept below 5 KΩ [9], [10]. The EEG was acquired at a sampling rate of 200 Hz. R The online processing, was performed on MATLAB 7.11.0 R
TM (R2010b) on a computer with Intel Core i7 processor with a clock speed of 2.10 GHz, 8 GB of RAM and a HDD of 1 TB. Regarding the experimental protocol, forty five trials of 25 s long were used for the training of the classifier and 15 trials were used for validation. Five mental tasks were tested with the BCI, which were the imagination of the movement of right and left hands, imagination of the movement of both feet, imagination of the manipulation of a cube with both hands, and music imagery. Figure 2 shows a block diagram of the used BCI with the VRE as the system feedback being presented to the user. Data were acquired with a right handed male subject (aged 30 years) whom performed the five mental tasks cited above. The subject has normal hearing, corrected-to-normal vision and no history of substance abuse, major medical psychiatric illness, or developmental or neurological disorder. It is worth commenting that the project was approved by the Ethics Committee of the Federal University of Esp´ırito Santo (Brazil),
Fig. 2. Block diagram of a BCI.
recognized by the Ethics Committee of the Research Ethics National Commission (CEP-048/08). Regarding the experimental protocol, which is presented in Figure 3, the subject was instructed to sit with his hands resting on his legs and to observe a cross in the center of the screen. The cross is a fixation point to avoid excessive artifact from eye movement. After 5 s, a black arrow pointing to the left or right replaces the cross indicating the start of the mental task. The mental task lasts 10 s, and then the cross reappears in place of the arrow indicating that the mental task is over. The subject must remain still until the cross is replaced by a circle at time 20 s, indicating the end of the record. The BCI also needs to be trained with EEG from the absence of a specific mental task, therefore, 45 trials are taken with this same protocol, but during this fase subject is instructed just to do not move or perform any of the 5 mental tasks: •
•
•
Imagination of the movement of right and left hands: For the left/right hand task the subject was instructed to imagine the movement of the right arm and hand in order to reach and grab an object placed in a table in front of him at a distance of 30 cm. The subject was instructed to repeatedly perform this imaginary movement, at his own pace, from the 0 to the 10 s mark of the EEG record. Imagination of the movement of both feet: For the foot task the subject was instructed to imagine the movement of extending both legs, and thus raising both feet. In the same way, the subject was instructed to repeatedly perform this imaginary movement, at his own pace, from the 0 to the 10 s mark of the EEG record. Imagination of the manipulation of a cube with both hands: For the imagination of the manipulation of a cube task the subject was instructed to imagine that he was manipulating a Rubik’s cube with both hands. In order to facilitate the imagination, a real Rubiks cube was placed in a table in front of him at a distance of 30 cm. In the same way, the subject was instructed to repeatedly
•
perform this imaginary movement, at his own pace, from the 0 to the 10 s mark of the EEG record. Music imagery: Finally, for the music imagery task the subject was instructed to remember just instrumental music, with no lyrics, to avoid the association of brain areas related to language, from the 0 to the 10 s mark of the EEG record.
in the frontal channels. The EEG topography and the EEG channels images are refreshed at every 0.5 s. The trial is shown with a window of 25 s filled with null values that are replaced by the EEG current value. The bottom field of the display shows the target in the dashed gray line and past classifications in the solid black line. Here, the first trial was incorrectly classified and the following fourteen were correctly classified. The right field of the display shows the feature selection method. The Kullback-Leibler symmetric divergence is calculated for every feature of all channels [11]–[14]. This divergence value is color-coded and it is shown in the right side upper panel of Figure 3. Here, dark blue is related to small divergence values and dark red corresponds to large divergence values. The features with highest divergence value are then selected to perform the classification. The right side bottom panel shows in red the selected features.
Fig. 3. Protocol to mental tasks.
The BCI works as a single-switch BCI (ssBCI) performing the classification of two classes, which are the absence of a specific mental task and the mental task. The ssBCI uses the EEG signal energy of α and β bands of all 19 channels. Then, a feature selection step using the Kullback-Leibler symmetric divergence performs the automatic selection of the channels and frequency bands most important for the classification of the mental task. The classification uses the Linear Discriminant Analysis (LDA). This approach makes the classifier independent of the chosen mental task paradigm and also would adapt the BCI for individual features of the user [11]–[16]. After performing the training step, the BCI classifies the current trial. The arrow turns green if the classification was correct and turns red if the classification was incorrect. An arrow pointing to the left indicates that the subject has not to perform any of the 5 mental tasks and an arrow pointing to the right indicates that the subject has to perform one specific mental task. Figure 3 shows the display of the ssBCI, where the upper part shows the band EEG topography. For the EEG topography, the EEG amplitude is color-coded, where dark blue is related to small amplitude values and dark red corresponds to large amplitude values. The arrow pointing to the right indicates that the subject has to perform one specific mental task, which is in this example was the task to imagine that the manipulation of Rubik’s cube. After 20 s the BCI classifies the trial, and here the arrow is shown green because it was a correct classification. The middle field of the display shows the EEG signal of the 19 channels. Here, it can be seen that there are blink artifacts
Fig. 4. BCI display.
III. R ESULTS The mental task used to trigger the command to the avatar was the motor imagery of the manipulation of a cube, since it has the best results, acquiring a success rate up to 93.75 ± 6.29%. The imagination of the movement of the right hand obtained success rate of 86.75 ± 8.74%. The imagination of the movement of the left hand obtained success rate 80.00±12.25%. The imagination of the movement of both feet obtained 80.00 ± 17.80% of success rate. The music imagery obtained 73.75 ± 15.48% of success rate. The interaction R between the VRE and MATLAB was made by a simple R code to switch from MATLAB window to the VRE window, sending to this last window the triggering signals to make the avatar walk. Therefore, after the classification of each trial the BCI sends control commands for the VRE. The control commands are sent by using the JavaTM Robot Class and the R interpreter. The Robot class is used javascript MATLAB
to take the control of mouse and keyboard and emulate the keypress of the keys configured as commands to the VRE, as the ”w” key. IV. C ONCLUSION This study reports on the implementation of a off-line BCI using pre-acquired motor imagery signals to control an avatar in a 3D VRE walking simulator. As future work, an on-line BCI will be implemented, different environments will be built to simulate several ambientations, directed to rehabilitation proposes. Previous work [17] shows the significance of the usage of VRE as feedback to stimulate the BCI users, and improve their performances. Another important implementation is the use of the motor imagery of the specific target of the body region, e.g. legs motor imagery to make the avatar walks or arms motor imagery to make him move his arms. R EFERENCES [1] B. Graimann, B. Allison, and G. Pfurtscheller. Brain-Computer Interfaces: A Gentle Introduction, Brain-Computer Interfaces: Revolutionizing Human-Computer Interaction, 1-27, 2011. [2] C Kilner and R Tori. Realidade Virtual: conceitos e tendˆencias. 2004. [3] A. L´ecuyer, F. Lotte, R. Reilly, and R. Leeb. Brain-computer interfaces, virtual reality, and videogames, Computer, 41:66-72, 2008. [4] R. Leeb, M. Lancelle, V. Kaiser, D. Fellner, and G. Pfurtscheller. Thinking Penguin: Multimodal Brain-Computer Interface Control of a VR Game, IEEE Transactions on Computational Intelligence and AI in Games, 5(2):117-128, 2013. [5] http://www.blender.org/download/ [6] http://unity3d.com/unity/download [7] R. Silva, A. Silva. Tecnologias para Construc¸a˜ o de Mundos Virtuais: Um Comparativo Entre as Opc¸o˜ es Existentes no Mercado, FAZU em Revista, 211-215, 2012. [8] http://www.gimp.org/downloads/ [9] S. J. Luck. An Introduction to the Event-Related Potential Technique, The Mit Press, Cambridge, USA, 2005. [10] M. R. Ford, S. Sands and H. L. Lew. Overview of artifact reduction and removal in evoked potential and event-related potential recordings, Physical Medicine and Rehabilitation Clinics of North America, 15: 1–17, 2004. [11] A. B. Benevides, T. F. Bastos Filho and M. Sarcinelli Filho. A PseudoOnline Brain-Computer Interface with Automatic Choice for EEG Channel and Frequency, Proceedings of the IEEE International Symposium on Circuits and Systems, 81–44, 2011. [12] A. B. Benevides, T. F. Bastos Filho and M. Sarcinelli Filho. Proposal of a Brain-Computer Interface Architecture to Command a Robotic Wheelchair, Proceedings of the 20th IEEE International Symposium on Industrial Electronics, 1–6, 2011. [13] A. B. Benevides, T. F. Bastos Filho and M. Sarcinelli Filho. Design of a General Brain-Computer Interface, Revista Controle & Automacao, 22(6): 638–646, 2011. [14] A. B. Benevides, T. F. Bastos Filho and M. Sarcinelli Filho. PseudoOnline Classification of Mental Tasks Using Kullback-Leibler Symmetric Divergence, Journal of Medical and Biological Engineering, 32(6): 411– 416, 2012. [15] A. B. Benevides, T. F. Bastos Filho and M. Sarcinelli Filho. Classificao Pseudo-Online de Tarefas Mentais para uma Interface CrebroComputador, Anais do X Simp´osio Brasileiro de Automac¸a˜ o Inteligente, S˜ao Jo˜ao Del-Rei, Brazil, 199–204, 2011. [16] A. B. Benevides, T. F. Bastos Filho, R. C. Garcia, J. L. Mart´ın. Classificac¸a˜ o de Tarefas Mentais em Tempo-Real para Controle de Dispositivos Rob´oticos, Anais do XVIII Congresso Brasileiro de Autom´atica, Bonito, Brazil, 1–6, 2010. [17] R. Leeb, C. Keinrath, D. Friedman, C.Guger, R. Scherer, C. Neuper, G.Pfurtscheller. Walking by Thinking: The Brainwaves Are Crucial, Not the Muscles! Presence: Teleoperators and Virtual Environments, 15(5): 500-514, 2006.
