MindTactics: A Brain Computer Interface Gaming Platform - IEEE Xplore

2010 2nd International IEEE Consumer Electronics Society's Games Innovations Conference

MindTactics: A Brain Computer Interface Gaming Platform

Kenneth Oum1, Hasan Ayaz2, Patricia A. Shewokis2, Paul Diefenbach1 1

Digital Media Program, Drexel University, Philadelphia, PA School of Biomedical Engineering, Science & Health Systems, Drexel University, Philadelphia, PA (ko392, ayaz, pas38, pjdief)@drexel.edu

2

ABSTRACT During the last couple of decades, there has been an exponential improvement in neuroimaging technologies that allowed researchers noninvasive assessment of cognitive workload, short term memory, and spatial/navigational behaviors in humans. Through the use of new experimental paradigms and brain imaging devices, researchers have gained deeper insight into the neural correlates of emotion, cognition and motor control. Brain Computer Interface (BCI) systems utilize neuroimaging tools to detect brain activation evoked by a specific thinking process and convert it to a command. We have developed a game environment called “MindTactics” as a test platform for BCI devices to conceptualize experimental cognitive paradigms in ecologically valid environments as well as test BCI gameplay paradigms. Unlike traditional video games where the challenge to the user is the designed game mechanics, BCI based gameplay also involves mastering the use of the BCI device itself. The purpose of this project is to develop compelling BCI game design methods. MindTactics is capable of integrating data from multiple devices including the optical brain imaging based BCI developed at Drexel University, and it records behavioral log files for further analysis. Keywords:Brain Computer Interface, BCI, fNIR, Optical Brain Monitoring, Game Research, Brain Gaming I.

INTRODUCTION

Brain-Computer Interfaces (BCI) systems have matured significantly from their earlier days and even commercial products are now available or under development for entertainment purposes[1, 2].These new systems provide a novel control mechanism for video games. By utilizing the capability of these BCI devices, video games can be uniquely designed to be challenging and adaptive to the user in ways that current video games are not. However, BCI also presents inherent challenges so that a new gameplay paradigm is required by the devices due to the uncertainty in the functioning of both the devices and the brain signals that they measure. In traditional video games, the challenge to the user is simply the designed game mechanics. BCI based gameplay does not only involve the game mechanics based

978-1-4244-7180-5/10/$26.00 ©2010 IEEE

challenges, but also mastering the use of the BCI device itself. The purpose of this project is to develop compelling game design methods that will also serve as a platform for the testing of BCI devices. This includes creating a 3D game environment that presents challenges such as mental distractions and direct multi-user competition. In addition, BCI games can act as a powerful tool in brain research and development of BCI systems by providing more realistic, engaging and demanding experimental conditions.Game environments can be utilized as a test platform for BCI systems and further allow investigating behavioral and environmental factors that can arise during use and deployment of practical BCI systems. Various immersive 3D environments and necessary tools have been designed and have been utilized for such research[3-6]. Evoked brain responses can be tested with various game mechanics such as distractions, which affect a user’s BCI index, and performance. “MindTactics” is a configurable 3D gaming platform that can be used together with multiple BCI devices. MindTactics is built on the Unity3D game engine and receives a BCI index over a TCP/IP network and updates the game environment at in real-time. We have integrated and tested MindTactics with the functional near infrared (fNIR) based BCI [7] developed at Drexel University. II. BRAIN COMPUTER INTERFACE TECHNOLOGY A. Functional Near Infared (fNIR) Technology Functional near infrared spectroscopy, or fNIR, is a noninvasive optical technique that measures neural activity related hemodynamic response within the cortex[8, 9]. fNIR technology uses light within 700nm to 900nm, introduced at the scalp, to enable the noninvasive and safe measurement of changes in the relative ratios of deoxygenated hemoglobin (deoxy-Hb) and oxygenated hemoglobin (oxyHb) in the capillary beds during brain activity. Both, oxy-Hb and deoxy-Hb are correlates of brain activity through oxygen consumption by neurons[10]. fNIR technology allows the design of portable, safe, affordable, noninvasive, and minimally intrusive monitoring systems. These qualities make fNIR suitable for the study of hemodynamic changes due to cognitive and emotional brain activity under ecologically valid, natural conditions. fNIR has been used to monitor the human brain prefrontal cortex for assessment of


cognitive workload, working memory and brain computer interface research[7-9, 11-14]. B. Functional Near Infared (fNIR) Device The fNIR device developed at the Optical Brain Imaging Lab of Drexel University is a continuous wave system[7-9]. The control box hardware is connected to a flexible sensor pad that contains 4 light emitting diodes (LED) and 10 photo-detectors. This sensor pad is positioned over the forehead of the user and designed to sample cortical areas underlying the forehead at 2Hz. C. Closed-looped, feedback regulated, fNIR based BCI In this study, the fNIR based BCI system incorporates the use of a Protocol-Computer, control box, data acquisition computer, and the fNIR sensor as defined in [7] (See Figure 1). The fNIR sensor is placed on the forehead of the user that sits in front of the Protocol-Computer. fNIR data are acquired and transmitted through the control box to the data acquisition computer. The data acquisition software, Cognitive Optical Brain Imaging (COBI) Studio (Drexel University), collects the raw fNIR data and transmits them back to the Protocol-Computer either by Ethernet or by wireless network via TCP/IP [7]. In a common experiment setup, a stimuli delivery software is used on the Protocol Computer such as MazeSuite [15]. For BCI experiments, the fNIR-BCI client software on the Protocol-Computer receives fNIR signalsover TCP/IP, processes the data to calculate oxygenation changes by using modified Beer Lambert Law. Finally, a number between 0 to 100 called the BCI index is calculated using oxygenation data and transmitted to the Protocol-Computer, which in turns updates the visual feedback thereby completing the closed loop [7]. In this current project, MindTactics software is deployed on the Protocol-Computer and received the BCI Index calculated by the fNIR-BCI client.

Figure 1: fNIR experimental setup (Adapted from [7])

978-1-4244-7180-5/10/$26.00 ©2010 IEEE

III.

GAME PROJECT: MIND TACTICS

A. Game Platform and Structure Mind Tactics was created to investigate using concentration as both a gameplay mechanism as well using the game itself to supplement the clinical studies of concentration levels. Concentration-based controls have been demonstrated in previous studies [7]; therefore the focus of this project is shifted towards how this game type could be intertwined with that BCI protocol. By using the same protocol format as defined in [7], specifically the fNIR tasks, this project promotes the exploration of new BCI game mechanics while simultaneously testing research by following researchers' experimental guidelines. These guidelines refer to adhering to the important elements within an experimental protocol structure. MindTactics consists of two modes of play: single player and multiplayer. The single player mode is meant primarily for researchers to test subjects and for subjects to learn the game. The multiplayer strategy mode is designed more for the "gamer" population, and includes a multiplayer archetype with research elements. These two modes are different in pace, yet they consist of the same structure and share the same basic game principle: territorial control. MindTactics uses one of the most basic multiplayer archetypes as its foundation of play: capture points (CP). Capture point gameplay is a competition of territory control on a game map. Like other multiplier archetypes such as death match, capture the flag (CTF), and king of the hill (KOTH), capture points is meant to be an intense competition between human players that promote both strategy and quick thinking [4]. In terms of design, the game is a third person perspective CP environment. The user must navigate the 3D terrain using a secondary controller (keyboard, gamepad, joystick) and capture the three territories which are represented as flag points. When the user completes the game goal of capturing all three flag points, they win the game and are shown a special win-case screen. There are three maps available; two for single player and one for multiplayer. The single player maps are separated into a short map and a long map which are built to be simple and linear. The multiplayer map is set to be an arena style, where the user must capture three of the five flag points available to win the game. The main reason for picking CP as a design foundation was its scalability from multiplayer to single player while being compatible with past-developed BCI game mechanics. In MindTactics, the basic game mechanic uses player concentration to obtain territory within the game. Territory on the game map is represented by flag points. These flag points are captured when the user, based on vicinity, uses their concentration to raise a flag to the point of capture (see Figure 1). The flag capture mechanic uses the calculated BCI index of the connected device as a method of raising the flag. The flag's progression is dependent on this index which can range from a value of 0-100. By having a BCI index or concentration level of 50 or above the flag progresses


upwards, if not the flag digresses downwards. Because the flag moves downwards as well as upwards, users must develop a personal method of concentration that is effective in raising the flag. This concept of maintaining an optimum concentration is the basic challenge of the game.

Trick Bar: This distraction creates a replica of the concentration bar and places three replicas on the screen.

Timed Items: This distraction creates a time limit in which the user must capture the point.

Figure 1: Concentration being applied to a flag point In addition to the basic challenge of the “using concentration to capture points”, distractions are used to challenge the player. The distraction game mechanic was developed as a type of opposition component. This component is applied to further challenge users that have become proficient in their use of the BCI device. When users master general concentration and the device, the distraction mechanic is utilized to affect their attention. As attention is strongly linked to concentration, the distraction game mechanic disrupts a user’s ability to retain attention and forces them to adapt quickly to a given scenario [16]. These scenarios are concentration event triggers that occur either based on time or based on percentage of capture completion of a flag point. The main goal of distractions is to change the pace of the game and augment the challenge. Available distractions are listed in Table 1. Both pace and cognitive challenge are important in maintaining an optimum game flow [17]. Table 1. Mind Tactics distraction types and descriptions Freeze Bar: This distraction freezes the concentration bar so it neither increases or decreases.

Rearrange: This distraction rearranges the user interface to random rotations and locations on the screen.

978-1-4244-7180-5/10/$26.00 ©2010 IEEE

Sequence: This distraction requires the user to press various letter keys in the correct order while capturing a point.


Land Shift: This distraction moves the flag point around the map forcing the player to chase after it while concentrating.

B. Data Gathering and logging In terms of neuroscience research, MindTactics can be used as an environment for testing protocols. Currently the protocol the game is tailored for is the bar-size control task as defined in [7]. However, the game is made to be expandable for other validated BCI protocols. Research games, like MindTactics, utilize the intrinsic qualities of games which are immersion and game flow. By incorporating a BCI protocol into engaging game mechanics, researchers can observe how users react to cognitive challenges through the concealment of a game. If the game focuses on the strengths of the BCI device, the researchers can use a game environment for training which can be augmented for their own purposes. If the game offers a unique gameplay experience, users will return to the game to further progress device mastery and in turn more data about proposed game mechanics will be generated. The term "replay value" is the concept of users returning to the game because of engaging gameplay experience, and it is very important in the design of Mind Tactics. The repetition of play is important for training with the devices and accumulating data on user progress. For research settings, MindTactics currently captures and logs key behavioral events and received BCI index to separate files in xml format with time tags. After the session, these log files can be parsed and the session can be studied. In terms of customization, the researcher has an option menu available where they can save custom settings and load past settings. The settings consist of control over the game user interface (concentration bar, flag ui, and etc.), the use of a timer, device choice and connection, and the setup for single player sessions (distraction events). This menu can be expanded upon with the inclusion of future BCI devices and new distractions or introduction to other game/research elements. IV.

SUMMARY AND FUTURE WORK

Through the exploration of current BCI protocols and game theory, a game platform was developed for BCI research. The BCI prototype game MindTactics included two methods of play; single and multiplayer. The single player was created for researchers and multiplayer was created for the "gamer". By leveraging elements in both respects, BCI benefits from the intrinsic values that games offer to research. This research will then introduce an

978-1-4244-7180-5/10/$26.00 ©2010 IEEE

engaging and valid game design for BCI which has no background in entertainment. MindTactics is an example of how a platform can create this reciprocal process and was developed for future expandability in brain research and for future BCI devices. These future BCI devices also include commercial devices developed for the consumer market. MindTactics is compatible with any BCI device that utilizes attention or concentration control. While MindTactics demonstrated the use of a BCI game for both entertainment and brain research,the game’s true potential lies in its expandability; meaning additional BCI devices, multiple players, and the introduction of online capabilities. Future brain research could incorporate a larger amount of users with emphasis on team work. User information can then be gathered and analyzed on not just the individual but a group dynamic. This can include players on the same team, on opposing teams, or based on certain moments within the game in which two specific users interacted. These game sessions can be in a local environment (e.g. offline) for practice purposes, but is most effective in an online environment. By using an online environment, researchers have the ability to test a much larger amount of users then within a lab setting. Unlike the previous studies with few participants, using an online game environment makes it easier to have a much larger sample size of users from various backgrounds. MindTactics uses the unique properties of games such as immersion, but also uses protocols as game mechanics which researchers can alter and augment based data acquisition and user feedback. REFERENCES [1] N. Birbaumer, "Brain-computer-interface research: Coming of age," Clinical Neurophysiology, vol. 117, pp. 479-483, 2006. [2] J. Millán, et al., "Combining Brain–Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges," Frontiers in Neuroscience, vol. 4, 2010. [3] A. Lécuyer, et al., "Brain-computer interfaces, virtual reality, and videogames," Computer, vol. 41, pp. 66-72, 2008. [4] E. Lalor, et al., "Steady-state VEP-based brain-computer interface control in an immersive 3D gaming environment," EURASIP journal on applied signal processing, vol. 2005, pp. 3156-3164, 2005. [5] A. Nijholt, "BCI for games: A ‘state of the art’ survey," Entertainment Computing-ICEC 2008 Lecture Notes in Computer Science, vol. 5309, pp. 225-228, 2009. [6] A. Nijholt, et al., "Turning shortcomings into challenges: Braincomputer interfaces for games," Entertainment Computing, vol. 1, pp. 85-94, 2009. [7] H. Ayaz, et al., "Assessment of Cognitive Neural Correlates for a Functional Near Infrared-Based Brain Computer Interface System," in Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience, 2009, pp. 699-708. [8] S. C. Bunce, et al., "Functional near-infrared spectroscopy: An Emerging Neuroimaging Modality," IEEE Eng Med Biol Mag, vol. 25, pp. 54-62, Jul-Aug 2006. [9] M. Izzetoglu, et al., "Functional brain imaging using near-infrared technology," Engineering in Medicine and Biology Magazine, IEEE, vol. 26, pp. 38-46, 2007. [10] D. Heeger and D. Ress, "What does fMRI tell us about neuronal activity?," Nature Reviews Neuroscience, vol. 3, pp. 142-151, 2002. [11] H. Ayaz, et al., "Detecting cognitive activity related hemodynamic signal for brain computer interface using functional near infrared spectroscopy," Conf Proc 3rd IEEE/EMBS on Neural Engineering, pp. 342-345, 2007.


[12] H. Ayaz, et al., "Cognitive Workload Assessment of Air Traffic Controllers Using Optical Brain Imaging Sensors," in Advances in Understanding Human Performance: Neuroergonomics, Human Factors Design, and Special Populations, T. Marek, et al., Eds.: CRC Press Taylor & Francis Group, 2010, pp. 21-31. [13] S. Coyle, et al., "Brain-computer interface using a simplified functional near-infrared spectroscopy system," J Neural Eng, vol. 4, pp. 219-26, Sep 2007. [14] R. Sitaram, et al., "Hemodynamic brain-computer interfaces for communication and rehabilitation," Neural Netw, May 24 2009. [15] H. Ayaz, et al., "Maze Suite 1.0: a complete set of tools to prepare, present, and analyze navigational and spatial cognitive neuroscience experiments," Behav Res Methods, vol. 40, pp. 353-9, 2008. [16] A. Jaimes and N. Sebe, "Multimodal human-computer interaction: A survey," Computer Vision and Image Understanding, vol. 108, pp. 116134, 2007. [17] R. Pedersen, Game design foundations: Wordware Publishing, 2003.

978-1-4244-7180-5/10/$26.00 ©2010 IEEE