2011 15th Illfernational Conference
011
System TheOl),. COl1frol.
alld Computing
(ICSTCC)
Eye Tracking Based Communication System for Patient with Major
Neoro~locomotor Disabilites
Robert G. Lupu, Radu G. Bozomitu. Florina Ungureanu and Vlad Cehan
Abstract- This paper presents a new technology used for communicating with people with major neuro-Iocomotor disability by determining gaze direction on a monitor screen. Gaze direction is determined by pupil position through image "nalysis. The pruposed COlllmunication technolugy tmsures communication with the patient as follows: (1) on a monitor screen divided in four sections are displayed key words and corresponding ideograms; (2) the patient sees the key words lind focuses his/her gaze on a certain section - a pointer allows the control of gaze direction; (3) an infra-red mini/micro video camera placed on the patient's head sends the Image of the eye to a computer; (4) the focusing of the patient's gaze on a certain key word i.e. a certain section of the screen is Identified by image analysis; (5) voluntary blinking, also objectilied by image analysis, shows the selection of a certain word. The patient's wish or need, represented by the key word, is then sent to a seITer and from there to a caretaker by means of a wire or a wireless transmission s~'Stem. The absence of a selection within a certain t!m~ period causes the displaying of a new set of ideograms. The new technology is useful in the case of the patients who cannot communicate verbally, through signs or in writing and is based on their ability to control eye movement, which is often the case with people with major locomotor disabilities.
1.
I;-':TRODUCTIO;o.;
A
report from 2007 of the World Health Organization (WHO) shows that neurological disorders, ranging from epilepsy to Alzheimer disease. and from stroke to headache affect up to one billion people worldwide. Neurological disorders also include brain lI1Junes, neuroinfections, multiple sclerosis and Parkinson disease [1]. Communicating with this kind of people is a challenge even today. Those with aural or visual disabilities can use sign language. hand writing or Braille alphabet as a way of expressing themselves. People with severe neurological
Manuscript received May 30. 2011. This work was supponed by the Romanian Government under PN·n grant (PN·II, ASISTSYS 12 12212008). R. G. Lupu is with the "Gh. Asachi" Technical University of lasi, Faculty of Automatic Control and Computer Engineering. 700050 Romania (fax; +40232231343; e-mail:
[email protected]. R. G. Bozomim is with the ··Gh. Asachi" Technical University of lasi. Faculty of Electronics. Telecommunications and Information Technology, 700506 Romania (fax:+402322 I 7720; e-mail:
[email protected]. F. Ungureanu is with the "Gh. Asachi" Technical University of lasi, Faculty of Automatic Control and Computer Engineering, 700050 Romania (fax: +40232231343; e-mail:
[email protected]). V. Cehan is with the "Gh. Asachi" Technical Universitv of lasi. Faculty of Electronics, Telecommunications and Infonnation T;chnology. 700506 Romania (fax:+40232217720; c-mail:
[email protected]).
318
disorders and especially those with major neuro-locornot disabilities who can produce only inarticulate sounds or . movements or perf orm uncontrolled museulor make certam contractions. can use a board with ideograms or an eye gill! communication board (e-tran) in order to communicat~ However, this requires constant presence of a caretaker. Taking into account that there are periods of time when !he patient does not want or need to communicate, the caretaker has to wait near the bedside or to check from time to time the needs of the patient. This raises the medical insul'llllCe costs. Using computer-based systems and technological advances in communication systems such as graphical user interface (GUI)-driven computer systems[21. we can facilitate communication with these patients. Human interaction with a computer based Syslem endowed with graphical user interface may be classified in two categories: pointing: moYing the pointer over an object, text or icon on the screen; sclection: left or right click. action interpreted as a selection of that object from the corresponding area. These operations are vcry easy to perform for healthy people but they raise big problems for those with disabilities. There are solutions like tongue movement detection device, touch screens, but these can be used only by those capable of fine movements. However. even patients with neuro-Iocomotor disabilities may have fine eye movements which can be detected and used to point
''',---"",./ Fig. 1. Video cam pOSition for eye tracking
objects on the screen. This can be dohe by measuring and interpreting biopotenlials at the level of eye balls through non-invasive techniques using electrodes placed above and underneath the eyes and to their left and right to deleet up and down and left - right movement. This method is called electro-oculo!!rarn [3]. . Another m;thod, different from electro-oculogram one. II eye tracking method which involves the use of a video carn and image prncessing technique to determine the posinoO of
2011 15th Imel'l1ational Conference on System
77WOI}"
Control, and Computing (ICSTCC)
with Majo~ n
major neuro-Iocomo! '. or )' ma.rtlculate sounds Qr In uncontrolled muscular deogrums or an eye gaze order to Communicate. lresence of a caretaker )eriods of time when ~ nmunicate, the caretaker :heck from time to time s the medical insurance
1
1S and technological , such as graphical User systems[2], we can patients. mputer based system cc may be classified in
over an object. text or action interpreted as a Ie corresponding area. o peliorm for healthy terns for those with ke tongue movement :hese can be used only ents. However, even lilities may have fine led and used to point
ye tracking
ne by measuring and of eye balls througb les placed above and .nd right to detect up [his method is called ro-oculogram one, is :! use of a video cam nmne the position of
Laptop Asis~5ysFa!ie:')~
Fig. 2. Asislsys corrununication system
Fig. 4. Asistsys system - communication flow
the eye. The video cam is placed near the eye in order to take only the image of the eye. as is presented in Fig. I. Corresponding to eye position, a cursor is moved inside a certain area of the screen. The selection is made when eye blink is detected. To eliminate fake selections by involuntary blinks, the patient has two possibilities: close the eyes and keep the eyes closed until a beep is heard, or winking two or three times in a short period of time. The proposed system uses eye tracking technique. facilitating selection of a key word and sends it to caretaker. In this way the patient no longer needs the continuous presence of the caretaker.
received/forwarded: AsistsysCaretaker - identified as a PDA or smartphone announces the caretaker about patient request. 4. Briefly. the The communication flow is shown in system works like this: the patient makes a selection and the keyword is sent via Ethernet to the server. There. based on the information in the database. is selected the caretaker who has to answer and the keyword is sent wireless to hislher PDA. The caretaker's answer is sent back to the server and from there to the patient's device screen. The major advantage of the proposed system is that it replaces the traditional manner of communicating. which relies on the constant presence of a caretaker at the patient's bedside, raising thus the cost of patient care laking [3]. The patient gaze direction is determined by the eye tracking technique. To perform this, a video cam is placed very close to the eye right underneath avoiding to obstruct the patient's frontal visual field. The camera was modified to take images only in infrared light. To track eye movements an open source eye tracking Starburst algorithm was used [4]. The original open source code was written in c++ progranuning language for Linux based operating systems using OpenCV library. The source code was modified to be compiled with ANSI C compilers. more precisely with National Instruments LabWindowslCVI compiler because of already existing project developed in
II. SYSTEM PRINCIPLE
The communication system that offers to the patient the possibility to express his needs whenever he wants even he is alone. is depicted in Fig. 2. In front of the patient's eyes, a computer based unit displays keywords on a screen together with a suggestive icon and background colour. Whenever the cursor floats over a certain group (keyword - image) the area is highlighted and the keyword is heard inlo the headphones. The selection is made detecting a voluntary blink. The selected keyword is sent wirelessly to caretaker's PDAlsmartphone. The caretaker notices de patient request and takes action. The proposed system, named ASISTSYS. has the block diagram presented in Fig. 3: AsistsysPatienl a computer based device assisting the patient in keyword selection process and ensures the communication with the server;
AsistsysServcr manages all the information
As;"sy~
~r!-i:$ys
Potion:
Sorver
Fig. 3. System block diagram
AsI,I$),g Camtaf;or
L Fig. 5. Typical eye pOlitions left column (up, down), right column (left, ri~ht)
2011 151h International Conference on Syslem Theol)" ConlroJ, and Compuling (fCSTCC)
camera
~'Ymm)
monilor
Fig. 6. Willdow-Io-viel·vporllralls/ormolioll/moppillg
n.2. Forword Polier4Requesl 0.1: Res""..... to Paller4Requesl
t
this environment. for Windows operating systems. A conversion function from NT Vision Image data type to OpenCV IplImage data type was also implemented. The typical eye positions for basic movements up, down, left, right, shown in Fig. 5, defines a rectangle «x cm , Ycm), (XcM, YcM)) as it is represenled in Fig. 6. For the monitor, it is assumed that x",•• =O, ),,,,,.=0, X",M=X,,, and )'mM=Y",' The relations (I) express the scaling factors s, and Sy and for the same orientation of the axes the translation terms have the expressions from eq. (2).Relation (3) represents the matrix form for the window-to-viewport mapping.
x
s,
ax, s ). = a)'m
XcM
v
8)"
t.z
.. t,\I
= -S),
f,
'XCIl'J
XCII'
(1 )
Yca,
(2)
-Sy·YC-Ul
['"] [I ,,] [" 0
Ym = 0 I 0
rd
-
I
I)'
0
1
..
0
0
s)'
0
0
1]' [~:1
(3)
The patient needs to send the selected key word! ideogram to Asistsys server and from here to the caretaker. The caretaker identified automatically by searching in the database receives the keyword on his mobile devices. The response is sent back to the patient via server. The TCPIIP protocol is used for communication between system devices, via local computer network in three ways: Ethernet (IEEE 802.3) and Wi-Fi (IEEE 802.11 b, g, n), Ethernet and GPRS, mixed. A more detailed communication diagram between ASistsys system components, when the Asists),sPatient is the initiator is shown in Fig. 7. The keyword selected by the patient is sent to AsislsysServer. The server selects the caretaker from the
320
AsistsY$ C.11 C'taker k
Fig. 7, Communication diagram within Asistsys system
database according to priorities and work load, and a con'esponding keyword is forwarded to herlhis mobile device. The positive or negative response, together with or without an optional sentence is sent back to the server. In case of positive response, the server forwards the caretaker's message to the patient's device. In case of negative or no response another caretaker is selected and the process is repeated. If no one sends a positive response, then the server operator is announced and he can take action corresponding to patient's needs. All the messages sent trough the server, are recorded in log files. This way can be generated statistical reports like caretakers load, the use of the system and so on . The Asistsys system is a client/server 3 tier model [5] system architecture where AsistsysPatient and AsistsysCaretaker are the clients (Fig. 8), The server accesses a database with patients, caretakers and logs and manages all the communication in the system. The caretaker application is written in Java for a beller portability. It is always connected to the server through radio (Wi-Fi!GPRS) receiving keywords from it. The keywords are added to a list on the screen from where the caretaker can select the patient and answer. On the other side, the AsistsysPatient application is constantly connected to the server through EthemetlWi-Fi. As it can be seen in Fig. 8, AsistsysPatient is structured in three major layers. The input layer controls the camera and image acquisition. The images are' preprocessed (horizontally flipped, 640x480 resampling, grayscale conversion) before being added into memory buffers. The middle layer takes images from safe thread memory buffers, detects the position of the pupil and simulates the user (patient) input by moving the cursor on the screen. Ever)' time the cursor enters in a certain area, that area is highlighted and the associated keyword can be heard at the
rcc)
2011 15th International Conference on System Theo/y, Conlrol, and Computing (ICSTCC)
{~;';'H;'d;~~.-----u----nn------u----n_u------u-u-n-n-----u-nunu--------'l I~
I
ii U•• ,
Interl.l~.
l
:
:
:!
:
:I
:
I
1,1 I! =:;c :llF~
I
:
11 f1 f
=
!!
J
l~
Video cafn
t
I
,I
t
:
I
I
" ...
,1
\;1
::
._~!O'tCmory buffoOn;
I
_._~ ___ /
\ ...::;::::::::::-.::----.............. "' ..... ----- .. - .. ---------- ..
'ntrf
I
iii:
:::!
1:
:
:
~f L
1
COIlM'etion
•
:
I
,
: ~:,
Safe thrHd laIJ
~
I
:
I
I
(
!
:
! ',I • ...
SeN..
: :
: I~
.j:
~----.~
I
:
!1
j
: :
J :
t
i
M4dule.
f
I: ::
ASlSlSYS
1...,,,.y.C.,oTok.. I
:{
Outputl~f::
:f !!
n
,JA"".yoSe,,,,,, !,
I(
! l:
Tools
:
~
..._ _ _ _ _ _
t
:
•
-----~ .. --- .. -- .... ------.. - .. ------ ..--- ....-- .... - ... ,.
""----'
I
\0 ...... _ .. ______ .....'
_ _ _ _ _ _ _ _ _ _ _ _..J
Fig. 8, Syslem nowchan
system
d work load. and a d to herlhis mObile Jnst:. together With or Jack to the server, tn 'erVer forwards the device. In case of taker is selected and s a positive response, ,nd he can take action rver, are recorded in
tati.,tical reports like 1d so on. IeI' 3 tier model [5] iistsysPatient and 'ig, 8), The server 'takers and logs and ystem. in Java for a better the server through :Jrds from it The 'een from where 'er. :ient application is 19h EthernetlWi-Fi. ient is structured in rols the camera and are preprocessed mpling. grayscale :mory buffers. The 3d memory buffers. simulates fhe user the screen, Every area, that area is can be heard at the
loudspeakers. Tn that moment, the selection can be made by closing the eyes. Once the keyword has been selected, the third layer sends it to the server. Messages received from the server are displayed on the screen by the same layer. The graphic user interface of AsistsysPatient (PC screen), is divided into four sections, in which keywords are displayed on a corresponding background color. as it is illustrated in Fig, 9. Close to every keyword, an appropriate ideogram is also displayed, The keywords window changes automatically every 11 seconds or can be changed by the patient, selecting left or right pointing arrows. A keyword should symbolize not only a desire, a necessity or a demand but also a SU3set of other keywords, The experiments carried out with some volunteers' help (students and patients) reveal that they need minimal training to learn how to point and select a keyword in order to use the system, Also "navigating" trough keywords in order to select the appropriate one is very easy to perform because it is intuitive and similar to any other contemporary
me
HURlS
communication device, III. SYSTEM TESTING
Tests were carried out in two stages. The first one was necessary to ensure a good functionality of the system. The second one certified the need of this kind of communication system for patient with neurolocomotor disabilities, In the first stage every component was tested in part and in system, The tests took place in laboratory as follow: 1) AistsysServer: two software products were developed in order to simulate a caretaker device (Fake Caretaker Device - FCD) and a patient device (Fake Patient Device FPD), The FPD application was installed on twenty computers and FCD application on five computers. All computers were connected in a network. The FPD computers were configured to send randomly key words, to disconnect from and reconnect to server randomly, to display messages received from server. The FeD computers where configured to answer positive or negative (randomly) to any request sent by the server regarding FPD. to disconnect from and reconnect to server, to display received messages,
;AsiS't;:]YS ;L -server - _.
~~
Fig. 9. Asistsys testing scenario
VOMH
Fig. 9. ASimysPalieol prim screen
The communication protocol, the load balancing of FCD and the login process of FPD and FCD where tested in this test scenario, 2) Asis/s),sCarelaker: a small custom server was built in order to test the communication protocol and functionality of AsistsysCaretaker application. The same server was used in development process. The server was programmed to
lOll 151h 1l1lernational Conference on System The01:v, Control, and Computing (ICSTCC)
automatically/manually send keywords. The graphical user interface functionality was also tested. 3) AsisrsysPatielll: firstly, the possibility to point and select an image from screen was tested. The number of images was varied from one to six. This test revealed that the mapping function allows pointing maximum four images. The arrow images are harder pointed and selected. Therefore an auto play was added. Images were changed if no action were taken in II seconds (from 2 to 6 seconds). The communication protocol was tested using a custom small server configured to answer automatically to any sent key word. The close connection function was added in order to test the reconnection function of the AsislsysPaliellt application. Finally. all three components where tested together. In the second stage, the tests where carried out at "Bagdasar Arseni" hospital from Bucharest, Romania and five patients were involved. After a short presentation of the system, the patients started using the AsistsysPaliellC. Firstly, one image was displayed and selection was made \\lith a switch. After accommodation with the system, the webcam was used and blink detection module was enabled, replacing the switch in selection process. Then, the tracking module was enabled and calibrated and patients were trained to move the pointer on the screen. After that, the number of images from the screen where increased gradually corresponding with the patient's ability to point and select the image. Also for the nurses and caretakers, training was necessary to use to AsistsysCaretaker device. A questionnaire with a five-point Likert scale (from 5 completely satisfied to 1 completely unsatisfied) was used to rate the performance of the system. The questionnaire had ten questions, nine for patients and one for medical staff. The system was rated by the patients with 2101225 points and by the caretakers with 18/25 points. The staff included two females and three males with ages between 28-50 years and with 3-25 years experience in neurodisabled care.
circularly arranged keys, to enable "typing" wilbon · k'mg, b Im
t
A Jower cost can be achieved for AsistsysPatient deVic e by porting the AsistsysPatient application on ARM device with additional DSP core for image processing, HDMI ~ SVIDEO output and Linux operating system. /.
REFERENCES II J Plus One Foundation. http://plusonefoundation.org/statistics.html (last visit April 20(1). 12} Craig A. Chin and Ann.ndo Barreto. "Integrated electromyogram and eye-gaze tracking curSOr control system fOT computer users With mOlor disabilities", Journal of K~habililalion Research & Development, Pages 161-174. Volume 45. Number 1, 2008. 13J R.O. Lupu, R. G. Bozomitu. V. Cchan. D.A. Cchan "A New Computer-Based Technology for Communicating with Poople With Major Neuro-Locomotor Disability Using Ocular Electromyogram", ISSE 2011 14] D. Li. D. Winfield, D.J. Parkhurst, "Stnrburst: A hybrid algOrithm (or video-based eye tracking combining feature-based and model. based approaches", Human Computer Inleraclion Program Iowa Slate University. Ames. Iowa, 50010 [5J Unified Modeling Languagc: Superstructure, version 2.0. p. 488 (2005). Available: http;/I"·\\w.omg~org/
