Wearable, Wireless Brain Computer Interfaces In Augmented Reality Environments Karla Felix Navarro University of Technology, Sydney
[email protected] Abstract After Pierre Gloor and Hans Berger discovered the human electroencephalogram or EEG produced in the brain in 1969, Brain Computer Interfaces (BCIs) became a reality. However, for more than a couple of decades, besides the common social fascination with such devices, they have not yet been considered as a feasible alternative interface for common daily activities. This is attributed to issues such as response time, costs and long initial user training periods. This paper defines and outlines the current BCI technologies and reviews the current status of BCIs in the context of wearable computers. The use of Augmented Reality environments and the integration of wireless technologies such as Bluetooth are proposed as potential catalysts in the process of incorporating BCIs into daily life.
1. Introduction Brain Computer Interface (BCI) devices have been a reality for twenty eight (28) years since Vidal (1973) built the first BCI in 1973 [1], but the technology has not been incorporated into daily life mainly due to slow user response times, excessive error rates, high cost, actual appearance and long initial training periods. Interest in the area of both harnessing brain power and understanding its workings is evidenced by the emerging field of neuro-economics. Functional Magnetic Resonance Imaging (fMRI) is being used to capture the brain activity the instant a thought occurs, as subjects perform diverse mental tasks such as arithmetical, language and memory tasks [2]. The researcher analyzed the suitability of a BCI as a Wearable computer, including inherent issues such as speed, flexibility and usability. This paper defines and outlines the current BCI technologies and reviews the current status of BCIs in the context of wearable computers. The following study reports on current research into BCIs and proposes strategies to improve their acceptability, namely by (1) by using Human Computer Interaction principles in the design of these devices so that they may become acceptable wearable devices, (2) by employing
Augmented Reality (AR) to improve their mobility capabilities and (3) by incorporating wireless technologies such as Bluetooth, Wi-Fi and GPRS technologies for mobility enhancement and context awareness. In the discussion section the pros and cons of the strategies are debated and the conclusion sums up the research to date and points the way forward to further research.
2. Background A Brain Computer Interface (BCI) is a device that allows human beings to control or communicate with a computer, or any other electronic device, by using a person's brain or “thoughts”. A BCI uses electrophysiological human electroencephalogram (EEG) signals produced by the human brain to communicate or control certain devices. These signals (found at the microvolt level) are amplified and digitalized, manipulated by a computer and “translated” to actions or commands. The manipulation or translation process in the computer is not a simple task. It is compounded by several subtasks that could be functionally divided into three main processes. Filtering of the signal (Preprocessing or Artifact Removal, involves the cancellation of unwanted data, or noise, as with Electro Oculoograms (EOG) and Electromyography (EMG). Once the signal has been “cleaned” then the categorization or Feature Extraction process identifies the correlation of a specific mental activity and electrical changes in the brain (e.g. P300, Evoked Related Potentials (ERP), Steady Visual Evoked Potentials (SVEP)). A more detailed explanation of these techniques is discussed in section five. Once the features have been extracted from the signaling, the Translating Process converts these features into symbols or commands to control electrical devices. The procedural architecture of a BCI is shown below in Figure 1.
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) 0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Fig. 1.- Procedural architecture of a BCI Designing a wearable BCI means taking into account concepts inherent in what is considered a wearable computer. A wearable computer should posses the following features [3], which will be further, discussed in the following sections of this paper. 1. 2.
Portable while operational / part of the user Controllable by the user / needing minimal manual input 3. Aware of its environment 4. Always on and able to attract attention (but not of a derogatory nature). 5. Negligible operational delays To develop a BCI as a wearable system, the researcher proposes that designers should be aware of the design principles described in the Human Computer Interaction (HCI) discipline. HCI is concerned with the design, evaluation and implementation of interactive computing systems for human use and the study of major phenomena surrounding them [4]. The majority of existing developed Brain Computer Interfaces have not yet applied HCI knowledge. HCI provides known proven techniques to reduce the probability of malfunctions due to factors that are independent of the effective core functionality of the interface. In some cases, such techniques may compensate for inevitable, inherent deficiencies of the BCI systems. HCI focuses more on human factors/characteristics, for example, knowledge, skills, experience, education, training, physical attributes and motor and sensory capabilities. The researcher recommends the above characteristics be taken into account in the design and/or evaluation of Brain Computer Interfaces. The researcher further proposes the use of Augmented Reality (AR) environments for the BCI wearer. Unlike most virtual reality environments, whose virtual worlds replace the real world, augmented reality systems enhance the real world by superimposing information onto it. One example is the system developed by Feiner to support procedural maintenance routing, where the technician uses a pair of glasses to see the instructions overlaid on top of the real world [5].
The use of AR environments for the BCI wearer will be further discussed in Section 5 of this paper. Finally in order to be able to perform daily activities, the BCI has to become a transparent device that allows the user to be action focused allowing the freedom required to accomplish the daily activities in a more efficient manner. The researcher believes that implementing the use of wireless technologies on BCIs would help with the process of achieving the need for interface transparency. As a result the Wearable BCI could be used as an alternative input to the control devices on a daily basis.
3. Methodology This paper follows a Framework for BCI Classification (based on Wolpaw’s model) [6] devised by the author during an extensive literature survey. Further work is now being undertaken to devise a standardized taxonomy of BCIs using the structured Mason & Birch Framework at the Neil Squire Foundation Institute in Vancouver [7].
4. Architecture In order to accomplish the mobility required in a wearable computer interface, the use of wireless technology on the interface is proposed. The tri-wireless Wanda PDA [8] which incorporates three communication technologies (Bluetooth, 802.11b and GSM/GPRS) could be used as the key device for communication between the wearable BCI and the outside world. By incorporating these three wireless technologies, the device becomes capable of the adaptability required in a nomadic environment.
Fig. 2.- Wireless BCI connectivity [8]. The main communication channel between the BCI and the PDA to facilitate the communication in the user's Personal Area Network (PAN) is Bluetooth technology. The incorporation of the WLAN 802.11b technology could connect the user to the common LAN environments and resources (e.g. home, office). The inclusion of Global System for Mobile Communication and General Packet
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) 0-7695-2108-8/04 $ 20.00 © 2004 IEEE
Radio Service (GPS/GPRS) will allow the BCI to be location and context aware. It would provide advantages such as using this technology for human-computer-human communication as well as for other mobile services available in the market. The combination of these three wireless technologies, Bluetooth for Personal Area Networks, WiFi for InterLANs and MANs and GSM/GPRS for mobile communication, could bring to the Wearable BCI the nomadic freedom required.
Bluetooth Bluetooth wireless technology, which carries voice, data and video [9], has been designed within the communication industry to address portable communications and computing needs in economic and societal environments. As work is now progressing to turn Bluetooth-enabled PDAs into portable e-cash wallets, remote controlled devices, keys and ticket machines in general, the use of this wireless technology was considered as an adequate communication media to incorporate the use of a wearable BCI into a PAN wearable network. Bluetooth is a low cost, low power radio communication technology that operates on 2.4 GHz frequency [10]. It does not require line of sight and therefore is able to operate in areas with high interference as it uses frequency hopping. The system uses 80 communication channels through which it may hop at the rate of 1600 channels per second. Piconets are made up of Bluetooth networks of two to eight devices whilst scatternets connect two or more separate piconets [11]. The potential value of integrating Bluetooth wireless technology into a BCI as its primary communication channel would automatically expose these types of interfaces to a vast realm of new possibilities. For instance the possibility of obtaining the powerful signal processing required in BCI applications without the need of being physically attached to a large system. This allows the user the possibility of accessing through a BCI all kinds of existing nomadic applications over IP. It will be an important step in uniting wirelessly the wearable BCI and its potential users. Further growth could lead to the development of BCI technologies, through the identification and development of adequate user application-specific tasks. Furthermore, the development of portable web services for specific wearable BCI applications could also act as a catalyst in the spread of usage of this type of wearable interface. As mentioned in the introduction, BCIs have been plagued by high error rates. In the table below the best performance results obtained in the BCI Competition 2003 at Graz University of Technology for each of the Data Sets evaluated are shown. The lowest numbers expressed in terms of error rates from the different types of data sets evaluated (using different neurological techniques) range from 0% as per the P300 technique
(achieved by 5 different institutions), to 11.3% applying Slow Cortical Potentials (SCP) by Justin Werfel and Sebastian Seung (MIT). P300 evoked potentials are commonly provoked by the frequency of stimulus occurrence (the less frequent stimuli, the larger response). Table 1.- Results of the BCI competition 2003 [12]. #. 1. 1. 1. 1. 1. 1.
1.
1. 1.
Contributor Kaper M. Gao X. Bostanov V. Blankertz B.
Error
Research Labs Data Set Univ. of Bielefeld Tsinghua Univ. U. of Tuebingen P300 speller 0% Fraunhofer FIRST paradigm (IDA), Berlin Tax D. MIT SCP (Slow Brett Mensh 11.3% MIT Cortical Potentials) Self-regulation Fraunhofer FIRST of mu- and/or Blanchard G. 28.2% (IDA), Berlin central betarhythm 0.61 MI Fraunhofer FIRST Motor Schäfer C. * (IDA), Berlin Imaginary Tsinghua U., Zhang Z. 16% Self-paced Beijing Note: The performance of the motor imaginary data set was measured by mutual information (MI) divided by time.
As the P300 technique shows a 0% error rate this technology will be suitable for Wearable BCIs, where accuracy is critical for the usability of the interface. It is important to take into account both the type of environment(s) and the applications the user will be performing. Some techniques could be quite accurate for certain type of activities or applications, and inappropriate for some others. This is because they are triggered by different neurological processes that are in many cases directly related to the different actions, movements, feelings or thoughts the user is experiencing at the time of triggering the BCI. For example, in order to read a P300 signal from the user brain, an extraordinary event has to happen. One example would be seeing your name in a list of unknown names on a computer screen. This could be used to trigger the selection of printing a document in the computer. It has to be considered that the user can achieve the same effect when seeing any other unexpected real object that brings remarkable memories [13], and thus this effect can produce an unexpected and unwanted action. With BCI operations, the issue of time is also important. If users have to recognize their name from a list, it would take far too long for an able bodied user who would have to perform such an activity for every single letter to be written down. However, it might be worth doing if the user simply wants to shut down the computer when lying down in bed.
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) 0-7695-2108-8/04 $ 20.00 © 2004 IEEE
An alternative technology to solving this trivial printing problem can be Steady Visual Evoked Potentials (SVEP) [14]. By looking at a strobe light on the screen, that can be limited to a specific icon (e.g. the printer), the EEG over the visual cortex will vary. If this is at the same frequency as the stimulating light, at the desired option, the system would be able to translate the user’s braindirected command into a printing action.
temperature, etc.) could be used to add intelligence to our Wearable BCI by predicting specific, and even personalized, offline and online programmable user behaviors. This additional feature would bring to the user a faster response rate, and potentially allow more freedom and flexibility. The user could potentially chose different weights from the different biological inputs according the specific needs, supported by a context aware environment.
Designing Wearable BCIs When designing Wearable BCIs, it is important to take into account the different type of environments and/or applications. A particular computer interface has to fit in with the user’s existing working patterns, other tools and the context of use [15]. The use of real world metaphors allows users to transfer knowledge, thus intuitively understanding the interface. For instance the user can easily predict the use of a calculator shown on a computer screen if the shape and inputs are similar to a “real” one. The conceptual design and use of metaphors, as many other HCI concepts such as usability, portability and transparency must be incorporated into the design. Taking advantage of some of the wearable devices already in the market that are aware of the environment and are context relevant could solve this problem. One example is the text information retrieval Augmented Reality wearable device built by Bradley Rhodes [16]. The device senses the location of the user via a GPS, and pops up information considered relevant for the user on glasses. Applying this feature to the wearable BCI, one could, for example, switch from using one neurological technique to another, according to the location of the user and the application being performed. Additionally one could combine various techniques to cross-relate information, depending on the type of mental activity performed.
5. Discussion Achieving Transparency with a BCI by reducing Latency and improving Accuracy implies making the interface feel natural. Currently the EEG recording caps or BCIs are not considered as fashion items nor as headgear that would be unnoticeable. Much work needs to done on this cosmetic aspect of the BCI. By correlating EEG findings and additional techniques like fMRI, Electromyography (EMG) and magneto encephalography (MEG) it is possible to identify some user’s key signal characteristics when performing specific daily activities; for instance the maneuverability of industrial machines sample or more simply, the control of house appliances. An analysis of the combined use of the information obtained from different brain imaging technologies (EEG, fMRI, MEG (produced by muscular movement) and other biological inputs (heart rate, blood pressure, body
Real World, Virtual Reality (VR) and Augmented Reality (AR) Evaluation One of the problems when trying to incorporate BCIs as a wearable computer into the real world is being able to accurately trigger a BCI in a highly changing environment as commonly is the case in performing daily life activities. This is in part due to the differences between the real world environment (of nomadic nature), and the experimental environments when identifying and matching brain pattern(s) to trigger the BCI in question. Most of the experiments are mainly performed in laboratories in front of a computer screen with controlled variables. This situation is difficult to recreate when exposing the BCI out into a real world nomadic environment (e.g. outdoors where there is no computer screen, a PDA is an alternative). BCI experiments in Virtual Reality (VR) environments have also been done in order to recreate more precisely the situations when triggering a BCI. Such experiments are able to minimize the uncertainty of the brain inputs originated from the “outside” environment, by being able to recreate an identical wider multi-sensorial experience for the user. This potentially engages the user perceptions in a broader manner giving more similar mental states and brain activity patterns when an identifiable event, which triggers the BCI, occurs. The use of VR is, in a broad sense, more precise when matching a pattern. This approach is effective when the BCI is used for an application in a VR environment (e.g. games, virtual worlds, etc.). The error rate increases dramatically when proving such a BCI in a similar real life situation. So far then a wearable/nomadic BCI to aid in our daily life activities is not possible because the Virtual Reality world is separated from the real world. The use of AR would naturally permit preliminary BCI experiments to partially keep a constant of the exogenous sensorial brain factors independent of the location of the user. Such constancy in the experimental and real environments would allow a more transparent coupling of the BCI into the different changing environments, and even more, it would permit a more dynamic development of the interface and allow design by prototyping. The mixed real and virtual reality environments could give the BCI the desired portability without loosing accuracy. Certain AR virtual objects could trigger the BCI
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) 0-7695-2108-8/04 $ 20.00 © 2004 IEEE
independently of the changing real environments. They could potentially control other electronic devices and enhance with relevant information their real world personal spaces.
6. Conclusion A considerable amount of work still needs to be done on BCIs to create one that is a wearable computer interface but it is now in the realm of possibilities. The development of the first BCI as a wearable computer (hardware standardized and application specific, for example, by using web services) would act as a catalyst for finding many more applications that will be used for the input of a BCI in their systems. The researcher analyzed the suitability of a BCI as a Wearable computer, including inherent issues such as speed, flexibility and usability. One of the visible obstacles that is necessary to overcome is to be able to wear a BCI as transparently as possible. This implies making the interface unnoticed, but also wearable. The computer interfaces require mobility, and the freedom from the need to be connected by wire to an electrical outlet, or communications line [17]. The incorporation of technologies such as Augmented Reality and multimodal personalized BCI techniques to increase accuracy in a changing environment, as well as the adoption of wireless technologies such as Bluetooth, WLAN 802.11b and GSM/GPRS (tri-wireless Wanda PDA) in the interface to allow mobility and humancomputer-human communication are proposed through the wearable-computer based analysis [18].
References [1] Vidal, J. J. (1973) Annual Review of Biophysics and Bioengineering, 2, 157-180. [2] Hirsch, J., Ruge, M.I., Kim, K.H.S., Correa, D.D., Victor, J.D., Relkin, N.R., Labar, D.R., Krol, G., Bilsky, M.H., Souweidane, M.M., DeAngelis, L.M., Gutin, P.H. (2000): An Integrated fMRI Procedure for Preoperative Mapping of Cortical Areas Associated with Tactile, Motor, Language, and Visual Functions, Neurosurgery, 47: 711722. [3] Mann, S. (1998) Wearable Computer Definition taken from Steve Mann's Keynote Address entitled "WEARABLE COMPUTING as means for PERSONAL EMPOWERMENT" presented at the 1998 International Conference on Wearable Computing ICWC-98, Fairfax VA, May 1998. [4] Hewett, T., Baecker, Card, Carey, Gasen, Mantei, Perlman, Strong and Verplank et al., (1992) Curricula for HumanComputer Interaction,. ACM Special Interest Group on Computer-Human Interaction, 1992. http://www.acm.org/sigchi/cdg/
[5] Feiner, S., MacIntyre, B., and Seligmann, D. (1993) Knowledge-based augmented reality. Communications of the ACM, 36(7), July 1993, 52-62. [6] Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M., (2002) "Braincomputer interfaces for communication and control," Clin Neurophysiol, vol. 113, pp. 767-91, 2002. [7] Mason, S. G. and Birch G. E. (2003) A General Framework for Brain-Computer Interface Design, IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 11., No. 1., March 2003. [8] Texas Instruments, WANDA PDA, http://focus.ti.com/pdfs/vf/wireless/wanda_031403.pdf [9] Keighran, B. (2002) Bluetooth Submission – 2002 to House of Representatives Standing Committee on Communications, Information Technology and the Arts. [Online] Available at (http://www.aph.gov.au/committee/cita/wbt/sub/sub004.pdf [10] Kalakota, Ravi & Robinson, 2001, Marcia, “M-Business: The Race to Mobility”, McGraw-Hill. [11] Miorandi, D., and Zanella, A. (2002) On the optimal topology of bluetooth piconets: Roles swapping algorithms. In Proc. Mediterranean Conference on Ad Hoc Networks MedHoc, 2002. [12] Outcome of the BCI-competition 2003 on the Graz data Set, Alois Schlögl, 21. Mai 2003 (corrected results)i http://ida.first.fraunhofer.de/projects/bci/competition/result s/TR_BCI2003_III.pdf [13] Farwell, L. A. and Donchin, E. (1991) The Truth Will Out: Interrogative Polygraphy ("Lie Detection") With EventRelated Brain Potentials. Psychophysiology, 28:531-547 [14] Sutter, E. E., (1992) “The brain response interface: communication through visually-induced electrical brain responses,” J. Microcomput. Appl., vol. 15, pp.31-45, 1992. [15] Beyer, H. and Holtzblatt, K., (1998) “Contextual Design”, San Mateo, CA: Morgan Kaufmann, 1998. [16] Rhodes, B. (2003) Using Physical Context for Just-in-Time Information Retrieval, IEEE Transactions on Computers, Vol 52, No. 8, August 2003 pp. 1011-1014. [17] Mann, S., "Wearable Computing as means for Personal Empowerment," presented at International Conference on Wearable Computing ICWC-98, Fairfax VA, 1998. [18] Starner, T., Mann, S., Rhodes, B., Levine, J., Healey, J., Kirsch, D., Picard, R. W., Pentland, A., MIT Media Laboratory 1997 "Augmented Reality Through Wearable Computing"
Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) 0-7695-2108-8/04 $ 20.00 © 2004 IEEE
