Removing the Interdependency between Horizontal and ... - MDPI

sensors Article

Removing the Interdependency between Horizontal and Vertical Eye-Movement Components in Electrooculograms Won-Du Chang, Ho-Seung Cha and Chang-Hwan Im * Department of Biomedical Engineering, Hanyang University, Seoul 04763, Korea; [email protected] (W.-D.C.); [email protected] (H.-S.C.) * Correspondence: [email protected]; Tel.: +82-2-2220-2322; Fax: +82-2-2296-5943 Academic Editors: Daniel Teichmann and Steffen Leonhardt Received: 20 December 2015; Accepted: 5 February 2016; Published: 14 February 2016

Abstract: This paper introduces a method to remove the unwanted interdependency between vertical and horizontal eye-movement components in electrooculograms (EOGs). EOGs have been widely used to estimate eye movements without a camera in a variety of human-computer interaction (HCI) applications using pairs of electrodes generally attached either above and below the eye (vertical EOG) or to the left and right of the eyes (horizontal EOG). It has been well documented that the vertical EOG component has less stability than the horizontal EOG one, making accurate estimation of the vertical location of the eyes difficult. To address this issue, an experiment was designed in which ten subjects participated. Visual inspection of the recorded EOG signals showed that the vertical EOG component is highly influenced by horizontal eye movements, whereas the horizontal EOG is rarely affected by vertical eye movements. Moreover, the results showed that this interdependency could be effectively removed by introducing an individual constant value. It is therefore expected that the proposed method can enhance the overall performance of practical EOG-based eye-tracking systems. Keywords: electrooculogram (EOG); bio-signal processing; EOG calibration; eye tracking

1. Introduction An electrooculogram (EOG) is the electric potential measured around the eyes, which is generated by the corneo-retinal standing potential between the front and back of the eye [1]. In recent decades, EOGs have been widely used to measure eye movement without a camera [2–5]. To measure eye movement, pairs of electrodes are generally attached either to the left and right of the eyes (horizontal EOG component) or above and below the eye (vertical EOG component) [2]. The horizontal and vertical EOG components are then obtained by subtracting the signal obtained at one electrode from the signal at the other electrode [2–4]. This bipolar measurement method has been widely utilized in various applications since 1936, when it was first revealed that EOG components reflect the horizontal and vertical movements of eyes [5]. To date, many studies have strived to recognize user intention using both the horizontal and vertical EOG components; however, most of these studies have been limited to only the estimation of momentary eye movements. The most successful application of EOG was as a substitute for the standard four directional (arrow) keys. For example, EOG-based gaze-direction estimation has been applied to simple eyeball input devices [6], a wheelchair controller [3], and a computer game controller [7]. Only a few studies have attempted to classify more than four eye-gaze directions. Bulling et al. [2] sought to recognize the daily activities of users, such as reading books and watching videos, by estimating 16 eye-gaze directions from EOG data. However, the individual classification accuracy was not reported. The most advanced study was recently performed by Yan et al. [8]. It classified 24 directions with Sensors 2016, 16, 227; doi:10.3390/s16020227

www.mdpi.com/journal/sensors

Sensors 2016, 16, 227 Sensors 2016, 16, 227

2 of 11 2 of 11

fairly high accuracy, although the number of subjects tested was only three. Although a number of classified 24 directions with momentary fairly high accuracy, although from the number subjects was onlytrack studies have estimated single eye movements EOG, of studies to tested continuously three. Although a number of studies have estimated single momentary eye movements from EOG, has eye movements and estimate eye-gaze patterns have rarely been performed, and the accuracy studies to continuously track eye movements and estimate eye-gaze patterns have rarely been generally been limited. Tsai et al. attempted to recognize ten eye-written digits and four extra symbols performed, and the accuracy has generally been limited. Tsai et al. attempted to recognize ten from 11 subjects. The overall accuracy was reported to be 72.1% [9]. eye-written digits and four extra symbols from 11 subjects. The overall accuracy was reported to be Many [9]. possible reasons exist for the limited performance of EOG-based eye tracking. Muscular artifacts, 72.1% eye blink Many artifacts, and brain signals (electroencephalogram (EEG)) are the major possible reasons exist for the limited performance of EOG-based eye artifacts tracking.contaminating Muscular EOGartifacts, [10]. Moreover, ambientand light conditions and the subject’s state of alertness can additionally eye blink artifacts, brain signals (electroencephalogram (EEG)) are the major artifacts affectcontaminating the EOG signals [11].Moreover, Studies in the last few decadesand on the thissubject’s issue have to improve EOG [10]. ambient light conditions state sought of alertness can additionallyaccuracy affect theby EOG signals specific [11]. Studies in the last few decades on this issue sought to and the estimation adopting signal processing techniques, such ashave digital filters improve the estimation accuracy by adopting specific signal processing techniques, such as digital wavelet transforms [2,12]. For example, Joyce et al. [10] estimated the propagation matrix between filterssignals and wavelet transformseye [2,12]. For example, Joyce et al. [10] estimated propagation matrix recorded and intended movement. Nevertheless, it has been the reported that the vertical between recorded signals and intended eye movement. Nevertheless, it has been reported that EOG component has lower stability than horizontal EOG, thereby making accurate estimationthe of the vertical EOG component has lower stability than horizontal EOG, thereby making accurate vertical location of the eyes difficult [10]. estimation of the vertical location of the eyes difficult [10]. The main objective of the present work is to introduce a method to eliminate the undesired The main objective of the present work is to introduce a method to eliminate the undesired interdependency between horizontal EOGcomponents. components. best of our knowledge, interdependency between horizontaland and vertical vertical EOG ToTo thethe best of our knowledge, this objective has not been considered in previous studies to date. Through experimental studies, this objective has not been considered in previous studies to date. Through experimental studies, we we investigate whether the proposed method can enhance the performance of EOG-based eye tracking. investigate whether the proposed method can enhance the performance of EOG-based eye tracking. 2. Experimental Environment 2. Experimental Environment 2.1. Recording of EOG Signals 2.1. Recording of EOG Signals signals recorded at a sampling rate of Hz 2048byHz by using an ActiveTwo biosignal EOGEOG signals werewere recorded at a sampling rate of 2048 using an ActiveTwo biosignal recording recording system (BioSemi,The Amsterdam, Thewith Netherlands) with six flat-type active electrodes. Two system (BioSemi, Amsterdam, Netherlands) six flat-type active electrodes. Two pairs of electrodes pairs of electrodes were placed at the most widely used locations to record EOG: to the left and right were placed at the most widely used locations to record EOG: to the left and right of the eyes, and above of the the eyes, andeye. above and below right eye. Reference ground weremastoids. attached Details at and below right Reference andthe ground electrodes wereand attached atelectrodes left and right left and right mastoids. Details of the electrode configurations are illustrated in Figure 1. The of the electrode configurations are illustrated in Figure 1. The horizontal EOG component was obtained horizontal EOG component was obtained by subtracting L from R, and the vertical EOG component by subtracting L from R, and the vertical EOG component was obtained by subtracting D from U, was obtained by subtracting D from U, where L, R, D, and U represent the electric potential recorded where L, R, D, and U represent the electric potential recorded from the corresponding electrodes in the from the corresponding electrodes in the figure. Two electrodes—common mode sense (CMS) and figure. Tworight electrodes—common mode as sense (CMS) and right legplaced (DRL),atwhich as the driven leg (DRL), which operate the reference anddriven ground—were the leftoperate and right reference and respectively ground—were mastoids, [13].placed at the left and right mastoids, respectively [13].

Figure 1. Electrode placementsused usedto torecord record the EOG components. Figure 1. Electrode placements the horizontal horizontaland andvertical vertical EOG components.

When placing the electrodes on the skin, a conductive gel was applied on the electrodes When placing the electrodesadhesive on the skin, a conductive gel wasvariations applied on together together with double-sided disks. This gel reduces of the theelectrodes skin-electrode with capacitance double-sided disks. This gel of reduces variations of the skin-electrode capacitance due to dueadhesive to the irregular surface the skin and thereby enables more reliable bioelectric

the irregular surface of the skin and thereby enables more reliable bioelectric signal measurements. We performed no skin preparation processes (e.g., cleaning skin with alcohol) before attaching the EOG electrodes.

Sensors 2016, 16, 227

3 of 11

signal measurements. We performed no skin preparation processes (e.g., cleaning skin with alcohol) 3 of 11 before attaching the EOG electrodes.

Sensors 2016, 16, 227

Sensors 2016, 16, 227Setups 2.2. Experimental Setups 2.2. Experimental

3 of 11

EOG signals were were acquired acquired fromno tenskin participants (between 18(e.g., and 25 25 years yearsskin of age). age). Prior to the the EOG signals from ten participants (between 18 and of to signal measurements. We performed preparation processes cleaning withPrior alcohol) data acquisition, a comprehensive summary of experimental procedures and protocols were data acquisition, comprehensive summary of experimental procedures and protocols were explained before attachinga the EOG electrodes. explained to each All participant. Allsigned participants signed a consent and received monetary to each participant. participants a consent form and receivedform monetary reimbursement for 2.2. Experimental Setups reimbursement forThe their participation. participants were situated distance a their participation. participants wereThe situated a distance away from aamonitor in away a quietfrom room. monitor in a quiet room. The distance from the eyes to the monitor was set to 62.5 cm in our The distance from the eyes to the monitor was set to 62.5 cm in our experiments. To exclude the EOG signals were acquired from ten participants (between 18 and 25 years of age). Prior to the experiments. To exclude the potential summary influence of head movement, thefixed headusing of each participant potential influence ofa head movement, the head of participant was aprotocols type of chin rest data acquisition, comprehensive ofeach experimental procedures and werewas fixed using atotype of participant. chin restapplications generally used inThe ophthalmic applications The of the generally used in ophthalmic [14]. height the chin rest[14]. could beheight adjusted forchin the explained each All participants signed a of consent form and received monetary rest could be adjusted for the convenience of the participants. On the monitor facing the participants, convenience of the participants. On the monitor facing the participants, six dots were arranged reimbursement for their participation. The participants were situated a distance away from aon dots were on amonitor 3 × 2distance grid. size of monitor 28.5 was cmThe ×set 61distance cm62.5 (height × width). asix 3monitor ˆ 2 grid. size of the wasThe 28.5 cmthe ˆ the 61 cm ˆ width). between inThe aarranged quiet room. The from eyes to(height the was monitor to cm in ourthe The distance and right dots wasofbetween 47.7 cm, and the distance the twowas vertically experiments. To exclude the potential head movement, the headbetween of each participant was left and right between dots wasthe 47.7left cm, and theinfluence distance the two vertically adjacent dots 13 cm. adjacent dotsawas 13ofthe cm. The arrangements ofinthe six dots in height Figureof 2. the chin fixed using typeof chin rest generally used ophthalmic applications [14]. The The arrangements six target dots are depicted intarget Figure 2. are depicted rest could be adjusted for the convenience of the participants. On the monitor facing the participants, six dots were arranged on a 3 × 2 grid. The size of the monitor was 28.5 cm × 61 cm (height × width). The distance between the left and right dots was 47.7 cm, and the distance between the two vertically adjacent dots was 13 cm. The arrangements of the six target dots are depicted in Figure 2.

Figure2.2.Size Sizeofof the monitor six dots arranged × 2 grid. The sixwere dotsused wereasused as Figure the monitor andand six dots arranged in a 3in ˆ a2 3grid. The six dots targets for drawing targets for drawing patterns.patterns. The dots2. were displayed via E-Prime E-Prime (Psychology Software Tools, Inc., Sharpsburg, PA, USA), The dots were via (Psychology Tools, Sharpsburg, Figure Size displayed of the monitor and six dots arranged inSoftware a 3 × 2 grid. TheInc., six dots were usedPA, as USA), which is is a software package for synchronizing various events with biosignal recording systems via which a software package for synchronizing various events with biosignal recording systems viaa targets for drawing patterns. parallel port. a parallel port. The dots were displayed via E-Prime (Psychology Software Tools, Inc., USA), E-Prime was used to visualize graphical instructions and target dots on Sharpsburg, the monitor,PA, as well as to which is a software package for synchronizing various events with biosignal recording systems via a to synchronize the experimental paradigm with the recording device by transmitting trigger signals parallel port. the system (receiver). A snapshot of the overall experimental environment is shown in Figure 3.

Figure 3. Snapshot of overall experiment environment. A chin rest was used to prevent potential head movements. The EOG data, collected by ActiveTwo, were transferred to a laptop for recording. Another computer (under the desk) was used to display dots and instructions to each participant and Figure Snapshot overall experiment environment. AAchin potential Figure 3. 3.Snapshot ofofoverall experiment environment. chinrest restwas wasused usedtotoprevent prevent potential synchronize events with the recorded data. head movements.The TheEOG EOGdata, data, collected collected by toto a laptop forfor recording. head movements. by ActiveTwo, ActiveTwo,were weretransferred transferred a laptop recording. Another computer (under the desk) was used to display dots and instructions to each participant and Another computer (under the desk) was used to display dots and instructions to each participant and synchronize events with the recorded data. synchronize events with the recorded data.

Sensors 2016, 16, 227

4 of 11

3. Methods 3.1. Preprocessing For better estimation of eye movements from EOG, noises and artifacts were removed according to the general steps used for previous EOG-based eye movement estimation studies, which were high-frequency noise removal, baseline removal, eye blink removal, and saccadic detection [2,9,10]. The EOG signals were resampled at a sampling rate of 64 Hz and were median filtered to remove high-frequency noises. The width of the median filter window was empirically set to four. The median filter was employed for the high-frequency noise removal because it was known to preserve features of eye-movement signals well [2]. To remove signal baseline drifts, the median value of the baseline period (a 100-ms signal before drawing a target pattern) was subtracted from the signals being analyzed. Eye blink artifacts are often included in EOG and must be detected in the preprocessing phase because they are easily misrecognized as vertical eye movements. An eye blink detection method [15] utilizing a digital filter—called the maximum summation of the first derivative in a sliding window (MSDW)—was employed for this purpose. This method can detect the extents of eye blink periods accurately from a single-channel signal without applying any machine learning algorithms (e.g., [16]). The MSDW filter is defined as: MSDW ptq “ s ptq ´ s pt ´ Kq (1) K “ argmax t|s ptq ´ s pt ´ kq|u

(2)

k

( ( subject to M pt, t ´ kq “ m pt, t ´ kq and min s1 ptq , s1 pt ´ kq ď @s1 pt ´ oq ď max s1 ptq , s1 pt ´ kq , where s ptq is a source signal at time t, s1 ptq “ s ptq ´ s pt ´ 1q, u ď k ď U, 1 ď o ď k ´ 1, and M pa, bq and m pa, bq respectively denote the number of local maxima and minima within a range ra, bs. Furthermore, ru, Us is the expected range of the slope width for eye-blink artifacts. When multiple K’s are found in Equation (2), the smallest K is selected to calculate the filter output; and u is chosen for K if any k does not satisfy the above conditions. The set of eye blink ranges is determined using the following equation: !” R“

ˇ ` ˘ ˇˇ ˇ T Maxi´ j ´ ˇWMSDW pTp Maxi´j qq ˇ ,

ı) T pMini q

(3)

where Maxi and Mini are the ith local maximum and minimum in the filtered signal, respectively, j is a positive integer that maximizes the Maxi´ j ´ Mini subject to Maxi´ j ´ Mini ą θ, Maxi´ j ą θ¨ α, ` ˘ Mini ă ´θ¨ α, T Maxi´ j ´ T pMini q ď M. In addition, θ is a threshold for determining eye blinking, α is a ratio to discard saccades, and T pMaxi q and T pMini q are the time points of the ith local maximum and minimum, respectively. Any j is rejected if the range related to j partially overlaps with another range. After all the ranges were determined, any range that is fully overlapped by another range is discarded. The data in the detected eye-blink regions are removed and then the missing data are replaced by linearly interpolated data. That is to say, a data point s ptq within the removed region is interpolated as: s pbq ´ s paq s ptq “ s paq ` pt ´ aq (4) b´a To remove the unknown sources of artifacts and extract signals only related to saccadic eye movements, we adopted the continuous wavelet transform-saccade detection (CWT-SD) algorithm of Bulling et al. [2]. This algorithm determines signals in a range as the eye-movement signals if the absolute values of the wavelet coefficients of the signals are greater than a preset threshold. The wavelet coefficient Cba of data s at scale a and position b is defined as: Cba psq “

ż8

1 s ptq ? ψ a ´8

ˆ

˙ t´b dt a

(5)

Sensors 2016, 16, 227

5 of 11

( )=

( )

1

−

(5) 5 of 11 √ where ψ represents a Haar mother wavelet. The wavelet scale used to calculate the coefficients was set to 20, as suggested by Bulling al., and the was used empirically set tothe 0.08. The signals were where ψ represents a Haar motheretwavelet. Thethreshold wavelet scale to calculate coefficients was set normalized to have abywidth of one before detection in order toset apply the same threshold. to 20, as suggested Bulling et al., and the saccade threshold was empirically to 0.08. The signals were normalized to have a width of one before the saccade detection in order to apply the same threshold. 3.2. Study on EOG Dynamics 3.2. Study on EOG Dynamics In this section, we visually investigate the changes in the EOG signals recorded while the In section, we visually the changes in theEOG EOG signals recorded while thefrom subjects subjectsthis moved their eyes toinvestigate draw a specific pattern. were recorded ten moved their eyes to draw in a specific pattern. EOG signals were recorded from ten as participants, as described Section 2.2. The recorded signals were resampled, andparticipants, the noises and described in Section 2.2. The recorded signals were resampled, the noises andvertical artifactsand were removed artifacts were removed as described in the previous section.and Figure 4 shows horizontal as described in the recorded previous section. 4 shows vertical and horizontal components recorded EOG components while a Figure participant drew an angulate U shapeEOG starting from a dot on the while a participant drewsequentially an angulatemoved U shape starting a dot on the top-left.horizontally, The participant top-left. The participant his/her eyesfrom in three steps (vertically, and sequentially moved his/her in of three (vertically, horizontally, and vertically) according the vertically) according to the eyes shape thesteps pattern. Consequently, the amplitude of the vertical to EOG shape of thewas pattern. Consequently, of theand vertical EOGatcomponent to component expected to decrease,the beamplitude unvaried (flat), increase each step, ifwas the expected component decrease, be unvaried (flat), and at each step, if the component not have anyan influence does not have any influence fromincrease the other signal source (Let us call thedoes expected signal “ideal” from theThe other signal source (Let ushorizontal call the expected signalwas an expected “ideal” signal). The ideal increase, amplitude of signal). ideal amplitude of the component to be unvaried, and the horizontal component was expected to be unvaried, increase, and unvaried in a series. unvaried in a series. Sensors 2016, 16, 227

Figure 4. 4. Horizontal Horizontaland and vertical EOG components of a participant. and H graphs Figure vertical EOG components of a participant. V and HVabove theabove graphsthe denote the denote the time periods of the vertical and horizontal eye movements, respectively. The horizontal time periods of the vertical and horizontal eye movements, respectively. The horizontal axis represents time. axis represents time.

As clearly seen from the example of the EOG components in Figure 4, the vertical component had As clearly seen from the example of the EOG components in Figure 4, the vertical component downward phantom movements as the eyes moved from left to right, especially when the horizontal had downward phantom movements as the eyes moved from left to right, especially when the component drastically changed (gray regions in Figure 4). On the contrary, the horizontal EOG horizontal component drastically changed (gray regions in Figure 4). On the contrary, the horizontal component was stable during the vertical eye movements (time periods of 0.3–1.4 s and 3–3.5 s, which EOG component was stable during the vertical eye movements (time periods of 0.3–1.4 s and 3–3.5 s, were determined by a visual inspection of the recorded EOG signals). This apparent influence of the which were determined by a visual inspection of the recorded EOG signals). This apparent influence of horizontal eye movement on the vertical EOG component makes accurate estimation of the vertical the horizontal eye movement on the vertical EOG component makes accurate estimation of the vertical location of the eyes difficult. It can thus degrade the overall accuracy of EOG-based eye tracking. location of the eyes difficult. It can thus degrade the overall accuracy of EOG-based eye tracking. For further analysis of this interdependency, we visualized EOG components of all participants, as For further analysis of this interdependency, we visualized EOG components of all participants, shown in Figure 5. In this figure, the horizontal components well follow the ideal signal trends, showing as shown in Figure 5. In this figure, the horizontal components well follow the ideal signal trends, a gradual increase in the middle region, but minimal signal variations in other regions. The vertical showing a gradual increase in the middle region, but minimal signal variations in other regions. The components, however, significantly differ from the ideal signal. In most cases, it is observed that vertical components, however, significantly differ from the ideal signal. In most cases, it is observed the amplitudes of the vertical EOG components decrease during the horizontal eye movements. that the amplitudes of the vertical EOG components decrease during the horizontal eye movements. This interdependency between the vertical and horizontal EOG components is observed for all subjects This interdependency between the vertical and horizontal EOG components is observed for all except one (Subject 5), although the degrees of the interdependency individually differed. subjects except one (Subject 5), although the degrees of the interdependency individually differed.

Sensors 2016, 16, 227 Sensors 2016, 16, 227

6 of 11 6 of 11

saccade detection. detection. The subject Figure 5. Horizontal and vertical EOG components for all subjects after saccade identification numbers bottom-right of to S10). S10). The The shaded shaded identification numbers are are indicated indicated at at the bottom-right of each plot (from S1 to regions indicate the time periods in which the subjects moved their gazes in the horizontal direction. the time periods in which the subjects moved their gazes in the horizontal direction.

3.3. Removal of Interdependency Interdependency between between Horizontal Horizontal and and Vertical Vertical EOG EOG Components Components 3.3. Removal of In the the previous previous section, section, we we confirmed confirmed that that the the vertical vertical EOG EOG components components are are highly highly influenced influenced In by horizontal eye movements, and that the degrees of the interdependency differ among individuals. by horizontal eye movements, and that the degrees of the interdependency differ among individuals. This interdependency interdependencycan canbe bereadily readilyremoved removedby byestimating estimatingthe theindividual individualdegree degree interdependency This ofof interdependency if if isitassumed is assumed to beover stable over time. this assumption, the vertical component by: is it to be stable time. Under this Under assumption, the vertical component is compensated compensated by: EOGv,α “ pU–Dq –α EOGh (6) EOG ) – α EOG (6) , = ( – where U, D, denote the signals recordedrecorded from the from corresponding electrodes where EOG EOGh h==RR´ −L and L and U, R,D,and R, Land L denote the signals the corresponding shown in Figure is a constant for each individual using the using EOG signals recorded electrodes shown1.inαFigure 1. α is adetermined constant determined for each individual the EOG signals during eye movement from left to right. main objective this compensation process is recorded during eye movement from leftBecause to right.the Because the mainof objective of this compensation process is to the verticalascomponent as much as possible during pure horizontal αeye to stabilize thestabilize vertical component much as possible during pure horizontal eye movement, is movement, to α minimize is determined to minimize the variance in the vertical componenteye during pure determined the variance in the vertical component during pure horizontal movement. horizontal eye movement. In other words: In other words: ( α “ arg min σpEOGv,k q (7) α = arg min σ(EOG , ) k (7) where σ denotes the standard deviation. where σ denotes the standard deviation.

Sensors 2016, 16, 227 Sensors 2016, 16, 227

77 of of 11 11 7 of 11

3.4. Experimental Validation of the Proposed Compensation Method 3.4.3.4. Experimental Validation Experimental Validationofofthe theProposed ProposedCompensation Compensation Method Method An experiment was designed to investigate the enhancement of EOG signals by the experiment designed togiven investigate the enhancement of EOGbysignals by the AnAn experiment waswas designed to investigate enhancement of EOG signals thethe introduction introduction of an individual constant inthe Equation (7). For the validation of proposed introduction of an individual constant given in Equation (7). For the validation of the proposed of an individual constant given in Equation (7). For the validation of the proposed compensation compensation method, we compared the original (uncompensated) EOG components with those compensation method, compared the original (uncompensated) EOG components withafter thosethe method, we compared thewe original (uncompensated) thoseEOG obtained obtained after the proposed compensation process, asEOG wellcomponents as the ideal with (desired) components. obtained after the proposed compensation process, as well as the ideal (desired) EOG components. proposed compensation process, well asparticipants the ideal (desired) EOG in components. signals were EOG signals were recorded fromasthe ten introduced Section 3.2EOG while they were EOG signals were recorded fromintroduced the ten participants introduced in Section 3.2 whiledrawing they were recorded from the ten participants in Section 3.2 while they were visually four visually drawing four different target patterns (denoted as P1–P4, as shown in Figure 6). visually drawing four different target patterns (denoted as P1–P4, as shown in Figure 6). different target patterns (denoted as P1–P4, as shown in Figure 6).

P1 P1

PP2 2

P33 P

P4P4

Figure 6. used for the experiment. Figure Fourdifferent differenteye-gaze eye-gaze patterns patterns used Figure 6.6.Four Four different eye-gaze patterns used for forthe theexperiment. experiment.

In this experiment,the theparticipants participants were were provided provided with a short session totofoster In this experiment, short practice practice foster In this experiment, the participants were provided with awith shortapractice session tosession foster familiarity familiarity with the experimentbefore beforethe thedata data acquisition acquisition (EOG signals were not recorded during familiarity with the experiment (EOG signals were not recorded during with the experiment before the data acquisition (EOG signals were not recorded during this session). this session). The participantswere were askedto togaze gaze at at aa fixation fixation cross at the center the monitor for 3 s.3 s. this The participants crossof atthe themonitor centerofof the monitor The session). participants were asked to gazeasked at a fixation cross at the center for 3 s. Then, afor target Then, a target pattern was displayed together with a red dot marking its starting point (see Figure 7a). Then, a target pattern was displayed together with a red dot marking its starting point (see Figure 7a). pattern was displayed together with a red dot marking its starting point (see Figure 7a). When the When the participants recognized the shape of the target pattern and the location of the starting When the participants recognized the shape of the target pattern and the location of the starting participants recognized the shape of the target pattern and the location of the starting point, they point, they began to draw thetarget targetpattern patternalong along aa guide guide line after aakey ininthe in in point, they began totarget draw the line afterpressing pressing key thekeyboard keyboard began to draw the pattern along a guide line after pressing a key in the keyboard in front front of them. They were instructed to move their gaze at a constant speed along the path that of front of them. They were instructed to gaze move their gaze at a constant speed along the pathonthat them. They were instructed move their a constant along path that appeared appeared on the monitor. to When they finishedatdrawing the speed pattern, theythe pressed the key once againthe appeared on the monitor. When they finished drawing the pattern, they pressed the key once monitor. When the pattern, they pressed thefour key patterns once again to times proceed to again finish to proceed to they finishfinished drawingdrawing the pattern. They practiced drawing three each. to proceed topattern. finish drawing the pattern. Theyfour practiced drawing four patterns drawing the They practiced drawing patterns three times each. three times each.

(a)

(b)

(b) Figure 7. Schematic (a) illustration of the experimental procedures: (a) Session to practice eye drawing of patterns; and (b) Session to record the EOG signals. The image of a keyboard implies that a user’s Figure 7. Schematic experimental procedures: Figure 7. Schematic illustration illustration of of the the experimental procedures: (a) (a) Session Session to to practice practice eye eye drawing drawing response is required to proceed to the next step. of EOG signals. signals. The implies that that aa user’s user’s of patterns; patterns; and and (b) (b) Session Session to to record record the the EOG The image image of of aa keyboard keyboard implies response is required to proceed to the next step. response is requiredoftothe proceed the next step. The procedure data to acquisition was similar to the practice session; however, the guide lines were not provided during the pattern drawing. Instead, the target pattern and starting point The procedure of the data acquisition was similar to the practice session; however, the guide Thedisplayed procedure data acquisition similar to the session; however, theatguide were on of thethe monitor for 3 s beforewas the presentation of practice the fixation cross. After gazing the lines were not provided during the pattern drawing. Instead, the target pattern and starting point lines were not provided during the pattern drawing. Instead, the target pattern and starting point fixation cross for 3 s, the participants started to draw the target pattern without guide lines, and they were displayed on the monitor for 3 s before the presentation of the fixation cross. After gazing at the were displayed the monitor 3 s before of the fixation After for gazing at the pressed a key on to finish the eyefor drawing. Thethe fourpresentation target patterns were drawncross. three times the nine fixation cross for 3 s, the participants started to draw the target pattern without guide lines, and they fixation cross 3 s, the participants started to draw thepressing target pattern guide lines, and they subjects andfor twice for one subject. All events, such as a key orwithout changing pages, were also pressed a key tosynchronized finish the eye drawing. The four target patterns were drawn three times for the nine recorded and EOG signals using E-Prime. pressed a key to finish the eyewith drawing. The four target patterns were drawn three times for the nine subjectsAfter and twice foracquisition one subject. events, such as pressing key orwere changing pages, were also data andAll preprocessing procedures, theaasignals resampled to obtain a subjects and the twice for one subject. All events, such as pressing key or changing pages, were also recorded and synchronized with EOG signals using E-Prime. similar and Euclidean distancewith between points. The resampling process is a recommended recorded synchronized EOGadjacent signals data using E-Prime. After the[17] data acquisition preprocessing procedures, the signals were resampled to obtain a procedure character and recognition because different writing speeds over participants After the datafor acquisition and preprocessing procedures, the signals were resampled to may obtain

similar Euclidean distance between adjacent data points. The resampling process is a recommended a similar Euclidean distance between adjacent data points. The resampling process is a recommended procedure [17] for character recognition because different writing speeds over participants may

Sensors 2016, 16, 227 Sensors 2016, 16, 227

8 of 11 8 of 11

affect the precision of the recognition similarity measure. Because writing the objective of this experiment was to procedure [17] for character because different speeds over participants may affect compare the shape the compensated ideal signals, procedure deemed the precision of theof similarity measure. signals Becauseand thethe objective of this this experiment waswas to compare necessary. The resampling procedure was as follows. We (1) selected the first point of the signal; the shape of the compensated signals and the ideal signals, this procedure was deemed necessary. (2) selectedprocedure a point during thefollows. point-by-point progression from theof first the end its distance Thethen resampling was as We (1) selected the first point thetosignal; (2)ifthen selected to the most recently selected pointprogression was larger from than the criterion; added − 1 to points and a point during the point-by-point the first to theand end(3) if its distance the most interpolated them when the distance was larger than n times that of the criterion points. We set recently selected point was larger than the criterion; and (3) added n ´ 1 points and interpolated them + σ the as the criterion, wherethanand σ are the and standard deviation when distance was larger n times that of mean the criterion points. We set m ` σofasthe the distance, criterion, respectively. The resampled signals were then compensated using the individually estimated degree where m and σ are the mean and standard deviation of the distance, respectively. The resampled of interdependency (α), where the degree value was individually calculated with the signal-trial signals were then compensated using the individually estimated degree of interdependency (α), where EOG data given Figure 5. the degree valuein was individually calculated with the signal-trial EOG data given in Figure 5. Next, the accuracy of the compensation was quantitatively by the comparing the Next, the accuracy of the compensation was quantitatively evaluatedevaluated by comparing compensated compensated signals theFor ideal For the quantitative the similarity signals with the ideal with signals. thesignals. quantitative comparison, thecomparison, similarity between the twobetween vertical the two vertical components was evaluated using Pearson’s coefficient with components was evaluated using Pearson’s correlation coefficientcorrelation together with dynamictogether time warping dynamic time warping (DTW) [18]. DTW is often used to align corresponding points between (DTW) [18]. DTW is often used to align corresponding points between two time-series signals. Ittwo can time-series can helpestimate obtain aofmore accurate estimateofofthe similarity regardless the length help obtainsignals. a more Itaccurate similarity regardless length of the signalofor drawing of the signal or drawing speed. sampling ideal to have thetest same datafor points speed. The sampling of the idealThe signal was setoftothe have thesignal same was dataset points to the signal each to the test signal for each comparison. This procedure guarantees that the length of a signal does not comparison. This procedure guarantees that the length of a signal does not influence the comparison influence the comparison Details of DTW are described in Appendix A. After points aligning the results. Details of DTW areresults. described in Appendix A. After aligning the corresponding using corresponding points using DTW, the correlation coefficient was calculated between two signals for DTW, the correlation coefficient was calculated between two signals for the comparison. The horizontal the comparison. Theevaluated horizontal component was notEOG evaluated because the influenced horizontal by EOG component was not because the horizontal component was not the component was not influenced by the vertical eye movements, as shown in Section 3.2. vertical eye movements, as shown in Section 3.2. 4. 4. Results Results Figure depictsexamples examples of horizontal the horizontal and vertical EOG components while Figure 88depicts of the and vertical EOG components acquired acquired while Subject 10 Subject visually drew patterns P1–P4 shown Figure The “ideal signal”ideal represents ideal EOG visually10 drew patterns P1–P4 shown in Figure 6. in The “ideal6.signal” represents EOG components, components, assuming a constant eye-gaze speed. The “uncompensated EOG” representsgaze the assuming a constant eye-gaze speed. The “uncompensated EOG” represents the estimated estimated gaze without the interdependency the two EOGThe components. The without considering the considering interdependency between the twobetween EOG components. “compensated “compensated EOG” represents the signal and (7). Figure 8 shows EOG” represents the signal compensated by compensated Equations (6) by andEquations (7). Figure(6) 8 shows that the horizontal that the horizontal component is similar to thehowever, ideal waveform; however, the vertical component is component is similar to the ideal waveform; the vertical component is severely distorted severely distorted during horizontal eye applying movement before applying a compensation method. It during the horizontal eye the movement before a compensation method. It is evident that the is evident thatvertical the compensated vertical EOG component corresponds with than the ideal compensated EOG component corresponds much better with themuch ideal better waveform does waveform than does the signal before compensation. the signal before compensation.

Figure Figure 8. 8. Examples Examples of of EOG EOG signals signals acquired acquired when when aa subject subject visually visually drew drew four four different different patterns, patterns, where the values were normalized to [´0.5, [−0.5, 0.5]. 0.5].

Table 1 lists the summary of the average correlation coefficients between the recorded (compensated and uncompensated) and ideal vertical EOG components. The correlation coefficients

Sensors 2016, 16, 227

9 of 11

Table 1 lists the summary of the average correlation coefficients between the recorded (compensated and uncompensated) and ideal vertical EOG components. The correlation coefficients of the compensated EOG (average 0.95) is significantly higher than that of the uncompensated EOG (average 0.90) (paired t-test, p ă 10´ 7 ). Table 1. Pearson’s correlation coefficients between the recorded vertical EOG components and ideal EOG components. Patterns

P1

P2

P3

P4

Average

Compensated EOG Uncompensated EOG

0.97 ˘ 0.05 0.92 ˘ 0.08

0.96 ˘ 0.08 0.93 ˘ 0.08

0.98 ˘ 0.02 0.92 ˘ 0.06

0.91 ˘ 0.09 0.79 ˘ 0.20

0.95 ˘ 0.07 0.90 ˘ 0.13

Table 2 lists the individual compensation constant α and the corresponding correlation coefficients for each individual. The data in the table confirm that the interdependency levels vary across subjects, and this interdependency can be effectively removed by applying the compensation constant. The compensation constant varies from ´0.48 to 0 (mean: ´0.21, standard deviation: 0.18). In this result, six participants showed statistically significant increase in the correlation coefficient after the compensation process, but the other four participants (Subjects 5 and 7–9) showed smaller increases in the correlation coefficient. This table additionally demonstrates that the individual compensation constant α does not need to be updated for a given subject unless the electrode locations change. It would be noteworthy to examine in future studies the cause of these large individual variations in the compensation constant. Furthermore, this individual difference in the interdependency between two EOG components may be potentially used as a new feature for biometric verification [19], although confirmation of this notion requires further research. Table 2. Individual compensation constants and Pearson’s correlation coefficients. The p-values were calculated between the coefficients of the vertical components of the compensated and measured EOGs. The p value for Subject 5 was not calculated because the two EOGs were identical. Subject ID

α

Compensated EOG

Uncompensated EOG

p-Value

S01 S02 S03 S04 S05 S06 S07 S08 S09 S10

´0.12 ´0.25 ´0.35 ´0.5 0 ´0.02 ´0.04 ´0.22 ´0.13 ´0.48

0.93 ˘ 0.05 0.94 ˘ 0.07 0.91 ˘ 0.13 0.92 ˘ 0.12 0.98 ˘ 0.02 0.97 ˘ 0.02 0.97 ˘ 0.03 0.96 ˘ 0.06 0.98 ˘ 0.02 0.97 ˘ 0.02

0.90 ˘ 0.04 0.81 ˘ 0.16 0.80 ˘ 0.18 0.79 ˘ 0.14 0.98 ˘ 0.02 0.97 ˘ 0.02 0.98 ˘ 0.01 0.93 ˘ 0.03 0.97 ˘ 0.02 0.80 ˘ 0.18

0.001 0.007 0.021 0.006 0.022 0.181 0.066 0.099 0.005

5. Conclusions In this study, we investigated the changes of vertical and horizontal EOG components during eye tracing of specific patterns. Experiments conducted with ten participants showed that the horizontal eye movement can influence the vertical EOG component, although the degrees of this interdependency showed large inter-individual variability. Therefore, we proposed a method to eliminate this unwanted interdependency between horizontal and vertical EOG components by introducing an individual constant, which can be readily obtained from a short period of EOG signals recorded during a single “left-to-right” movement. The experimental results showed increases in the correlation coefficient with ideal EOG waveforms (p ă 10´ 7 ), demonstrating that the EOG signals could be significantly enhanced by using the proposed compensation method. It is expected that the proposed method can be utilized for a variety of applications of human–computer interaction (HCI), such as EOG-based

Sensors 2016, 16, 227

10 of 11

wheelchair controller, a new type of game input devices, and eye-writing systems. In addition, one of the possible applications of these eye movement-based HCI systems is a communication platform for patients with amyotrophic lateral sclerosis, generally known as Lou Gehrig’s disease. Since these systems generally require reliable and accurate estimation of eye-movement, the proposed method would be an easy and effective way to enhance the system performance. In the future studies, we intend to further investigate the stability of the individual constant value during a longer term period, of which the results might be used to determine a required frequency of EOG recalibration. Acknowledgments: This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2014R1A2A1A11051796) and in part by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2014R1A1A2A16052334). The authors would like to thank J. Lim and D. Jeon for their assistance in the experiments. Author Contributions: Won-Du Chang contributed to investigating the signals, developing the algorithms and code, and writing the main parts of the paper. Ho-Seung Cha conducted the experiments and wrote the experimental parts of the paper. Chang-Hwan Im supervised the overall study and experiments, and revised the paper. Conflicts of Interest: The authors declare no conflict of interest.

Appendix A This research uses a method called dynamic positional warping to align two signals. This method is a type of dynamic time warping (DTW) algorithm. The alignment is determined to minimize the dissimilarity between two signals, A and B, where the dissimilarity is defined as follows: ddtw “ τ pL A , L B q

(A1)

ˇ ` ˘( ` ˘(ˇ τ pi, jq “ ˇ A piq ´ A i prev ´ B pjq ´ B j prev ˇ ` ω pi, jq

(A2)

i prev “ i ´ C A pcmin pi, jqq

(A3)

j prev “ j ´ CB pcmin pi, jqq

(A4)

cmin pi, jq “ argmin tτ pi ´ C A pcq , j ´ CB pcqqu

(A5)

ω pi, jq “ min tτ pi ´ C A pcq , j ´ CB pcqqu

(A6)

c

c

where L A and L B are the lengths of ideal and test signals, respectively, τ p1, 1q = 0, C A pcq and CB pcq are the cth slope constraints on the pattern axes, which limit the number of skipping data points. The pair of the constraints (C A,B ) can be written as: C A,B “ tp1, mq , pm, 1q|1 ď m ď Mu

(A7)

where M is the maximum branch length of slope constraints. References 1. 2. 3.

4.

Malmivuo, J.; Plonsey, R. Bioelectromagnetism: Principles and Applications of Bioelectric and Biomagnetic Fields; Oxford University Press: North Baldwin, NY, USA, 1995. Bulling, A.; Ward, J.A.; Gellersen, H.; Tröster, G. Eye movement analysis for activity recognition using electrooculography. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 741–753. [CrossRef] [PubMed] Barea, R.; Boquete, L.; López, E.; Mazo, M. Guidance of a wheelchair using electrooculography. In Proceedings of the 3rd IMACS International Multiconf. Circuits, Systems, Commun. and Computers, Piraeus, Greece, 16–19 July 1999; pp. 2421–2426. Venkataramanan, S.; Prabhat, P.; Choudhury, S.R.; Nemade, H.B.; Sahambi, J.S. Biomedical instrumentation based on electrooculogram (EOG) signal processing and application to a hospital alarm system. In Proceedings

Sensors 2016, 16, 227

5. 6.

7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19.

11 of 11

of the 2005 International Conference on Intelligent Sensing and Information Processing, Chennai, India, 4–7 January 2005; pp. 535–540. Young, L.R.; Sheena, D. Survey of eye movement recording methods. Behav. Res. Methods Instrum. 1975, 7, 397–429. [CrossRef] Kherlopian, A.R.; Gerrein, J.P.; Yue, M.; Kim, K.E.; Kim, J.W.; Sukumaran, M.; Sajda, P. Electrooculogram based system for computer control using a multiple feature classification model. IEEE Eng. Med. Biol. Soc. Int. Conf. 2006, 1, 1295–1298. Kim, M.R.; Yoon, G. Control signal from EOG analysis and its application. Int. J. Electr. Comput. Electron. Commun. Eng. 2013, 7, 864–867. Yan, M.; Tamura, H.; Tanno, K. A study on gaze estimation system using cross-channels electrooculogram signals. Int. Multiconf. Eng. Comput. Sci. 2014, 1, 112–116. Tsai, J.-Z.; Lee, C.-K.; Wu, C.-M.; Wu, J.-J.; Kao, K.-P. A feasibility study of an eye-writing system based on electro-oculography. J. Med. Biol. Eng. 2008, 28, 39–46. Joyce, C.A.; Gorodnitsky, I.F.; King, J.W.; Kutas, M. Tracking eye fixations with electroocular and electroencephalographic recordings. Psychophysiology 2002, 39, 607–618. [CrossRef] [PubMed] Young, L.R.; Sheena, D. Eye-movement measurement techniques. Am. Psychol. 1975, 30, 315–330. [CrossRef] [PubMed] Deng, L.Y.; Hsu, C.-L.; Lin, T.-C.; Tuan, J.-S.; Chang, S.-M. EOG-based Human-Computer Interface system development. Expert Syst. Appl. 2010, 37, 3337–3343. [CrossRef] Frequently Asked Questions. Available online: http://www.biosemi.com/faq/cms&drl.htm (accessed on 16 October 2015). Yagi, T.; Kuno, Y.; Koga, K.; Mukai, T. Drifting and blinking compensation in electro-oculography (EOG) eye-gaze interface. IEEE Int. Conf. Syst. Man Cybern. 2007, 4, 3222–3226. Chang, W.-D.; Cha, H.-S.; Kim, K.; Im, C.-H. Detection of eye blink artifacts from single prefrontal channel electroencephalogram. Comput. Methods Programs Biomed. 2016. [CrossRef] [PubMed] Ge, S.; Hong, X. A fully automatic ocular artifact removal from EEG based on fourth-order tensor method. Biomed. Eng. Lett. 2014, 4, 55–63. [CrossRef] Delaye, A.; Anquetil, E. HBF49 feature set: A first unified baseline for online symbol recognition. Pattern Recognit. 2013, 46, 117–130. [CrossRef] Chang, W.-D.; Shin, J. Dynamic positional warping: Dynamic time warping for online handwriting. Int. J. Pattern Recognit. Artif. Intell. 2009, 23, 967–986. [CrossRef] Juhola, M.; Zhang, Y.; Rasku, J. Biometric verification of a subject through eye movements. Comput. Biol. Med. 2013, 43, 42–50. [CrossRef] [PubMed] © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).