Optimization of Checkerboard Spatial Frequencies for ... - IEEE Xplore

IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 25, NO. 6, JUNE 2017

557

Optimization of Checkerboard Spatial Frequencies for Steady-State Visual Evoked Potential Brain–Computer Interfaces Nicholas R. Waytowich, Member, IEEE, Yusuke Yamani, and Dean J. Krusienski, Senior Member, IEEE

Abstract — Steady-state visual evoked potentials (SSVEPs) are oscillations of the electroencephalogram (EEG) which are mainly observed over the occipital area that exhibit a frequency corresponding to a repetitively flashing visual stimulus. SSVEPs have proven to be very consistent and reliable signals for rapid EEG-based brain–computer interface (BCI) control. There is conflicting evidence regarding whether solid or checkerboard-patterned flashing stimuli produce superior BCI performance. Furthermore, the spatial frequency of checkerboard stimuli can be varied for optimal performance. The present study performs an empirical evaluation of performance for a 4-class SSVEP-based BCI when the spatial frequency of the individual checkerboard stimuli is varied over a continuum ranging from a solid background to single-pixel checkerboard patterns. The results indicate that a spatial frequency of 2.4 cycles per degree can maximize the information transfer rate with a reduction in subjective visual irritation compared to lower spatial frequencies. This important finding on stimulus design can lead to improved performance and usability of SSVEP-based BCIs. Index Terms — Brain–computer interfaces (BCIs), canonical correlation analysis, checkerboard stimuli, steady-state visual evoked potentials (SSVEP).

I. I NTRODUCTION

T

HE effect of spatial frequency has been extensively tested in clinical fields with the elicitation of pattern electroretinalgrams (PERGs) and pattern VEPs (PVEPs) [1]. These pattern responses are commonly elicited using a single pattern reversal checkerboard stimulus or a vertical square wave grating stimulus. In 1985, Leguire and Rogers recorded ERGs as a function of spatial frequency and contrast and showed that an increase in either spatial frequency or contrast resulted in an increase in the amplitude of the pattern ERG [2]. These results, however, are not in concurrence with results from previous investigators that have found an increase in spatial frequency produces a decrease in amplitude response [3]. Likewise, other studies have shown that an increase followed by a decrease in pattern ERG amplitude results from increased spatial frequencies from 0.5 arc-min/cycle to 1.5 arc-min/cycle

Manuscript received February 29, 2016; revised July 6, 2016; accepted August 13, 2016. Date of publication August 16, 2016; date of current version June 18, 2017. This work was supported by the National Science Foundation (1064912 and 1421948). N. R. Waytowich and D. J. Krusienski are with Biomedical Engineering Institute, Old Dominion University, Norfolk, VA 23529 USA. Y. Yamani is with the Department of Psychology, Old Dominion University, Norfolk, VA 23529 USA. Digital Object Identifier 10.1109/TNSRE.2016.2601013

EEG and ERG amplitude peaking at 1 arc-min/cycle [4]. In 1991, Tomoda et al. recorded simultaneous ERGs and VEPs and showed that ERGs exhibited a bandpass tuning with a peak amplitude at 1.5 cycles/◦ while VEPs had a bimodal spatial frequency function with peaks at 3 c/◦ and 1 c/◦ [5]. Overall, these inconsistent results suggest that the characteristic visual system response elicited from pattern stimuli at various spatial frequencies has not been established and may be subjectdependent [6]. Similar visual stimuli have been studied in the context of controlling brain-computer interfaces (BCIs). It has been shown that checkerboard stimuli produce stronger steadystate visual evoke potential (SSVEP) responses than solid stimuli [7]. In contrast, other studies have determined that solid flashing stimuli perform better than checkerboard pattern reversal stimuli [8]–[10]. Futhermore, the effect of checkerboard spatial frequency (i.e., the size of the individual checks) on SSVEP performance has not been studied in the context of BCIs with multiple simultaneous stimuli. Similar to how stimuli flashing at high temporal frequencies (30–50 Hz) appear less obtrusive and visually fatiguing to the user without significantly detrimenting BCI performance [11], it is hypothesized that stimuli at higher spatial frequencies may also provide a similar advantageous effect. The present study aims to characterize and optimize the spatial frequency of checkerboard stimuli in a BCI paradigm. The first experiment consisted of a four-class BCI paradigm with all targets having the same spatial frequency and flashing at four different temporal frequencies. In separate trials, the entire spatial frequency range was evaluated from a 1 × 1 (0 c/◦ ) checkerboard (solid stimuli) and doubling in spatial frequency (i.e., 2 × 2 (0.15 c/◦ ), 4 × 4 (0.3 c/◦ ), 8 × 8 0.6 c/◦ , etc) up to a 256 × 256 checkerboard (single pixel check sizes with a spatial frequency of 19.2 c/◦ ), providing nine total spatial frequency conditions. The second experiment used a similar display to test the same conditions in a continuous feedback path-traversal task to quantify practical performance. II. M ETHODOLOGY

A. Data Collection A total of 11 subjects (seven male, four female, ages 24–32) participated in the experiment. Data from the first subject was excluded from the analysis of experiment 2 due to an inconsistency with the experimental setup and recording. Thus, results

1534-4320 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

558

IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 25, NO. 6, JUNE 2017

Fig. 1. Electrode montage used for this study. Data were recorded from all 16 electrodes shown; however, data from only eight channels (Oz, O1, O2, POz, PO3, PO4, PO7, PO8) were utilized in the online and offline analysis.

from 11 subjects is presented for experiment 1 and 10 subjects for experiment 2. Each subject gave informed consent prior to the study and was free of any known neurological conditions. All subjects had normal or corrected-to-normal vision and had varying levels of previous BCI experience, with six subjects having no previous BCI experience. The study was approved by the Old Dominion University Institutional Review Board. Data were collected using a 16-channel g.USBAmp amplifier with active electrodes (Guger Technologies, Austria). Electrodes were placed primarily over the occipital, parietaloccipital and parietal regions of the brain (Fig. 1) according to the International 10–20 System [12]. Electrodes were referenced to the right earlobe and grounded to the right mastoid. All data were collected using a sampling rate of 256 Hz, bandpass filtered from 2–30 Hz and stored on a hard disk. All aspects of the data collection were controlled using BCI2000 general-purpose BCI recording software [13]. An Arduino Uno microcontroller board with an ATMEL ATMEGA328P microcontroller chip was used to synchronize the stimulus onsets with the recorded EEG by sending a synchronization pulse to the recording amplifier triggered by the stimulus software.

B. Stimulus Parameters All stimuli were rendered using DirectX (Microsoft Inc.) and displayed on a 24 in LCD monitor with a 60 Hz refresh rate and a 1920 × 1080 resolution. Subjects sat comfortably in dark room and were centrally seated in front of the monitor at a distance of 55 cm. Nine different spatial frequency stimulus conditions were tested: 0 cycles/◦ (1 × 1 checkerboard), 0.15 c/◦ (2 × 2)), 0.3 c/◦ (4 × 4), 0.6 c/◦ (8 × 8), 1.2 c/◦ (16 × 16), 2.4 c/◦ (32 × 32), 4.8 c/◦ (64 × 64), 9.6 c/◦ (128 × 128), and 19.2 c/◦ (256 × 256). Fig. 2 shows the nine spatial frequency conditions with their respective pattern reversed forms. Each checkerboard condition is an image with the size of 256 × 256 pixels. The 1 × 1 represents a checkerboard that has 1 row and 1 column (i.e., a solid square), and the 256×256 is a checkerboard with 256 rows and

Fig. 2. Checkerboard stimuli used in the study. Each of the nine checkerboard spatial frequency conditions is displayed with its respective pattern reversal shown beneath. The corresponding spatial frequencies in cycles per degree (# of rows and cols) going from top left to bottom right are: 0 c/◦ (1 × 1), 0.15 c/◦ (2 × 2), 0.3 c/◦ (4 × 4), 0.6 c/◦ (8 × 8), 1.2 c/◦ (16 × 16), 2.4 c/◦ (32 × 32), 4.8 c/◦ (64 × 64), 9.6 c/◦ (128 × 128) and 19.2 c/◦ (256 × 256).

256 columns. Therefore, the 256×256 checkerboard condition has checker sizes that are composed of single pixels.

C. Canonical Correlation Analysis CCA is a multi-dimensional statistical analysis technique that finds underlying linear correlations between two sets of data. Given two multi-dimensional data sets X, and Y , linear

WAYTOWICH et al.: OPTIMIZATION OF CHECKERBOARD SPATIAL FREQUENCIES

559

combinations x = X T Wx and y = Y T W y can be found that maximize the correlation between x and y. The CCA finds the weight vectors Wx and W y by solving the following optimization problem: max 

Wx ,W y

E[WxT XY T W y ] E[WxT X X T Wx ]E[W yT Y Y T W y ]

(1)

In practice this can be solved using the singular-value decomposition method to diagonalize the covariance matrices as the maximum canonical correlation corresponds to the square-root of the largest eigenvalue. Lin et al. [14] applied CCA for SSVEP classification. In this case, CCA is used to find linear correlations between multichannel EEG data, X, and a set of reference signals Y f . This reference set consists of sine and cosine signals at the fundamental and harmonic frequencies of each stimulus. The reference signal Y f , shown below, can be derived using Nh harmonics, where f is the fundamental frequency and t is time. ⎞ ⎛ sin(2π f t) ⎜ cos(2π f t) ⎟ ⎟ ⎜ ⎟ ⎜ .. (2) Yf = ⎜ ⎟ . ⎟ ⎜ ⎝ sin(2π Nh f t) ⎠ cos(2π Nh f t) EEG data is canonically correlated with each reference signal and the classification output is determined as f s = maxi ρ( f ), where f = f 1 , f 2 , ... f K and K is the total number of classes (target frequencies) in the SSVEP BCI.

D. Experimental Paradigm 1) Subjective Evaluation:: Before the start of the experi-

mental session, subjects performed a subjective evaluation of the visual stimuli. Subjects were placed 55 cm away from a 24 in LCD monitor where all nine checkerboards flashing at different spatial frequencies were displayed simultaneously on the screen. The checkerboards were randomly arranged in two rows with five items in the top row and four items in the bottom row. Each of the checkerboards were continuously flashing with the same temporal frequency of 6 Hz. At this time subjects were asked to subjectively rank order each checkerboard in terms of visual irritation. Subjects were given a sheet of paper that matched the layout of the screen and were asked to write the ranking corresponding to each frequency based on the instruction: “Please order each condition from 1–9 indicating how visually irritating it is to continuously stare at the stimulus where 1 represents the least irritating and 9 represents the most irritating”. The subjective evaluation was performed prior to the experiment to avoid potential biases in preferences after performance feedback during the experimental sessions. 2) Experiment 1: Discrete Classification:: The first experiment consisted of a 4-target paradigm in which four checkerboards with identical spatial frequencies were presented simultaneously in the top, bottom, left and right portions of the screen and were flashed at four distinct temporal frequencies of 6, 6.66, 7.5, and 8.571 Hz, respectively (shown in Fig. 3). The

Fig. 3. Stimulation paradigm used for experiment 1 with currently the 0.6 c/◦ (8 × 8) spatial frequency condition shown. Each trial starts with a 2-s cue period to indicate the current fixation target, shown in part (a). After a 6-s stimulation period, feedback is given as a box surrounding the predicted target (b). Correct predictions were shown with a green box while incorrect predictions were shown with a red box. The left stimulus flashed at 6 Hz, the right flashed at 7.5 Hz, the top at 8.57 Hz and the bottom at 6.66 Hz.

experiment consisted of 18 runs each with 8 trials. Within each run, one of the 9 spatial frequency conditions was tested. Each trial began with a 2-s cue period where an arrow indicated the current target stimulus from a random sequence as shown in Fig. 3(a). The task for each subject was to centrally fixate gaze and attend to the target stimulus for 6 s during the stimulation portion of the trial. The trial concluded with a 2-s feedback period where the predicted target was encompassed by either a green or red square corresponding to a correct or incorrect classification [Fig. 3(b)]. During the feedback period, text was also displayed to the subject indicating the current condition and trial numbers as well as the cumulative classification accuracy. A single trial lasted 10 s and all trials were presented in immediate succession of one another making the duration of a single run equal to 80 s (10 s ×8 trials). After each spatial frequency condition a short rest period was given to the subjects lasting approximately 30–60 s. Each of the nine spatial frequency conditions was tested twice during the

560

IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 25, NO. 6, JUNE 2017

Fig. 4. Stimulation paradigm used for experiment 2. The four stimuli match the same shape and position as experiment 1. The path used in the path-traversal task was placed in the center of the four stimuli, contained no bifurcations and took 48 moves to complete where each direction had 12 moves. The blue square indicates the start/end position which was randomized for each run. Parts 4(a) and (b) show the two path path variants that were randomly presented. The top panel currently shows the 0.3 c/◦ (4×4) spatial frequency condition and the bottom panel shows the 2.4 c/◦ (32×32) spatial frequency condition. The run time was shown at the top left which counted upwards starting from 0. If the path was not completed after 3 min, the run was terminated.

18 runs and presented in random order to mitigate any order biases. The total length of Experiment 1 was approximately 30 min. After completion, subjects were given a short break before proceeding to experiment 2. 3) Experiment 2: Continuous Path-Traversal:: For experiment 2, subjects used the same 4-target configuration and stimulus frequencies from experiment 1 to complete a path-traversal task using a familiar PacMan avatar (shown in Fig. 4). The four stimulus targets controlled avatar movement in four respective directions: up, down, left, and right. The paths contained no bifurcations to provide a distinct, unambiguous path from the starting point to the ending point. To improve subject engagement throughout the session, two different path layouts were used, each with the same length and required number of movements in each direction. Additionally, the starting/ending point and intended direction of movement were randomly

selected for each trial to mitigate any spatial biases or learning effects. Each path took exactly 48 total moves to complete, where each movement direction was equally represented with 12 moves each. The goal of the traversal task was to move the avatar from the starting point to the coinciding ending point of the path that was indicated by a blue square (Fig. 4). The avatar could not cross the path boundaries and the movement was unconstrained so the avatar could move in any direction along the path boundaries, depending on the predicted classification. This represents a practical use-case for an online BCI as incorrect classifications must subsequently be corrected in order to complete the path. The overall direction of path traversal (clockwise or counterclockwise) was indicated by the starting direction of the PacMan avatar. Fig. 4(a) shows an example of path 1 with a starting location that indicates clockwise traversal, and Fig. 4(b) shows path 2 with a starting location indicating counter-clockwise traversal. Subjects performed the path-traversal task for each of the nine spatial frequency conditions for a total of nine runs of the path-traversal task. The conditions were presented in a random order. The minimum time to complete the path was approximately 48 s. The maximum time alloted for path completion was 180 s. If the subject was unable to complete the path in the allotted time, the run ended, and the next run was presented. For experiment 2, EEG signals were classified in real-time using a continuously updating signal buffer with a fixed buffer length of 2 s of EEG data that was classified using a committee of CCA classifiers. The 2-s buffer was split into three 1-s subwindows with a 0.5 s overlap (i.e., sub-windows were from 0-1 s, 0.5-1.5 s, and 1-2 s). A separate CCA classification was performed on each of the 1 s sub-windows resulting in three predictions of the movement direction. A committee scheme was utilized for final prediction by way of majority voting in which the movement direction was chosen when at least two of the three CCA classifiers agreed on the same direction. If no mutual agreement was reached between the three classifiers then no selection was made representing a null state in which the avatar did not move. This classification scheme continuously analyzed the 2-s-long data buffer which updated every second with a new second of data. Therefore, movement decisions and actions were made every second using the previous 2-s of data. For each subject, each movement decision and the total path completion time was recorded for each of the nine spatial frequency conditions.

E. Data Analysis 1) Experiment 1: Discrete Classification:: For the online classification, data from the 6-s stimulation period were classified using CCA and classification feedback was provided at the end of each trial. A target template was created for each of the four temporal frequencies using two harmonics (Nh = 2) each. The use of the first two harmonics of the stimulus frequency has proven to be sufficient for SSVEP detection in high-performance BCIs [15]. Additional offline analysis

WAYTOWICH et al.: OPTIMIZATION OF CHECKERBOARD SPATIAL FREQUENCIES

561

was performed to compare the different spatial frequency conditions. Data from each of the spatial frequency conditions were extracted providing a total of 96 s of SSVEP data for each spatial frequency condition corresponding to 24 s of data for each of the four target stimuli. To examine potential effects of spatial adaptation of the visual system, CCA classification analysis was performed for different observation lengths varying from 6 s to 1 s. To simulate the smaller observation lengths, data from the full 6-s window was divided into 0.5-s increments, starting from the stimulus onset to better emulate actual online performance using shorter window lengths. The Information-Transfer Rate (ITR) was calculated for each spatial frequency and observation length using

1− P 60 ∗ I T R = log2 N + P log2 P + (1 − P) log2 N −1 T (3) where N is the number of possible targets, P is the probability that the target is accurately classified, and T is the time interval for a selection in seconds. The classification accuracy and ITR as a function of overall spatial frequency were calculated by averaging over all observation lengths tested. In the case of ITR, only the lengths from 1–3 s were used in the averaging as ITR places emphasis on smaller time-windows. The visual irritation index from each subject during the subjective evaluation survey was aggregated and averaged for each spatial frequency condition. 2) Experiment 2: Continuous Path-Traversal:: The average path completion time for each spatial frequency condition was computed across all subjects to give an indication of task performance as a function of the spatial frequency. Subjects that were not able to complete a path in the allotted 180-s limit were assigned a path completion time of 180 s. To further differentiate the task performance for each condition, the proportion of the path that was traversed before time expired was calculated as a ratio of the farthest traversed point divided by the total number of moves. For example, a completion percentage of 50% (24/48) represents a condition where the subject only made it to the halfway point around the path before completion. The average path completion percentage across all subjects was computed for each spatial frequency condition. III. R ESULTS

A. Experiment 1: Discrete Classification Fig. 5(a) and (b) show the average classification accuracies and ITRs, respectively, for each spatial frequency condition and observation length. These results are shown as 2-D heatmaps where the color is mapped to either accuracy or ITR, respectively. Each column represents a different observation length and each row represents a spatial frequency condition. The accuracies in Fig. 5(a) show that, overall, the 0 c/◦ spatial frequency condition (solid stimulus) achieves the highest average classification accuracy of 97.7% amongst all conditions for observation lengths ≥ 3.5 s.

Fig. 5. Average Classification accuracy and ITR (bits/min) for experiment 1 for each spatial frequency condition versus observation length. (a) The average classification accuracy. (b) The average ITR (bits/min).

The accuracy plot in Fig. 5(a) shows two other distinct patterns. First, a bimodal distribution can be seen columnwise where a decrease in accuracy is exhibited for spatial frequency conditions between 0.15–0.6 c/◦ as well as for conditions greater than 9.6 c/◦ . Accuracies are greater for the conditions between 1.2 and 4.8 c/◦ with a secondary peak forming at the 2.4 c/◦ condition. Second, the observation length shows a considerable effect on the accuracy, especially for the conditions from 0.3 c/◦ to 9.6 c/◦ . Typically, an increased observation length ensures an increase in classification accuracy; however, the rows in Fig. 5(a) show an interesting decrease in performance given observation lengths longer than 4.5 s with peak accuracies ranging from 2.5–3.5 s. The average ITR results in Fig. 5(b) show a similar columnwise bimodal distribution across the spatial frequency conditions with a peak in the 0 c/◦ condition and a peak over the 2.4–4.5 c/◦ conditions. Additionally, the 2.4 c/◦ spatial

562

IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 25, NO. 6, JUNE 2017

Fig. 7. Classification accuracy versus time window length averaged across all subjects from experiment 1. The dotted blue line shows the accuracy versus time for the 0 c/◦ spatial frequency condition and the solid red line shows the accuracy for the 2.4 c/◦ condition. The dashed black line is the accuracy versus time averaged over all nine spatial frequency conditions.

Fig. 6. Spatial Frequency tuning curves from experiment 1. (a) The classification accuracy as a function of spatial frequency averaged across all subjects. (b) The ITR as a function of spatial frequency averaged across all subjects. For both (a) and (b) the averages were computed using the maximum value over the range of observation lengths. The error bars represent standard error.

frequency condition achieves the highest average performance among all conditions with an average ITR of 45.3 bits/min which exceeds the 35.7 bits/min ITR achieved by the solid stimulus condition (0 c/◦ ). Fig. 6(a) and (b) show 1-D line plots of accuracy versus spatial frequency and ITR versus spatial frequency, respectively. In each figure, the accuracies and ITRs were averaged across multiple observations lengths. Both figures show a bimodal spatial tuning pattern where the performance peaks can be seen at the 0 c/◦ and 2.4 c/◦ spatial frequency conditions. The level of spatial frequency adaptation over time is shown in the accuracy versus time plots in Fig. 7. The 0 c/◦ condition (solid stimulus) indicated by the dotted blue line, shows the typical accuracy versus time profile where accuracy monotonically increases as the observation length increases

until reaching a horizontal asymptote. The 2.4 c/◦ (the best performing checkerboard stimulus) shows a similar proportional relationship between accuracy and observation length up to 2.5 s, after which, the accuracy begins to decrease as the observation length continues to increase in a proportional fashion. The dashed black line in Fig. 7 shows accuracy versus time averaged over all spatial frequency conditions. With the exception of the 0 c/◦ solid stimulus condition, the remaining spatial frequency (checkerboard) conditions all follow a similar unimodal pattern where the accuracy peaks between observation lengths between 2 and 4 s, and longer observation lengths result in significant decreases in accuracy. Fig. 8 shows average irritation index ranking for each spatial frequency from the subject evaluation questionnaires. The subjective irritation index generally follows the expected trend where the level of visual irritation decreases as the spatial frequency increases, with the exception of a slight increase in the 1.2 c/◦ condition.

B. Experiment 2: Continuous Path-Traversal: Fig. 9 shows the performance results from the continuous path-traversal task. The average run duration is shown in Fig. 9(a) where a shorter duration corresponds to better BCI performance as less time was required to fully navigate the path. Note that run duration time does not always imply completion of the path as run duration lengths were capped to a maximum of 180 s and therefore some subjects were unable to complete the path for some of the spatial frequency conditions. Fig. 9(b) shows the completion rate for each spatial frequency condition averaged over all subjects. The 0 c/◦ and 2.4 c/◦ spatial frequency conditions were the only conditions for which all subjects were able to fully complete the path in the alloted time. These results agree with those obtained from experiment 1, which showed performance peaks at the 0

WAYTOWICH et al.: OPTIMIZATION OF CHECKERBOARD SPATIAL FREQUENCIES

563

Fig. 8. Average subjective evaluation of visual irritation for each spatial frequency condition. The error bars represent standard error.

c/◦ and 2.4 c/◦ conditions. Additionally, both the run-duration and the path-completion performance curves show bimodal distributions with the emergence of two distinct peaks; one at 0 c/◦ and one at 2.4 c/◦ . A paired t-test revealed no statisticallysignificant differences in run duration or path completion percentage between the clockwise or counterclockwise movement trials. IV. D ISCUSSION SSVEP-BCI studies most commonly employ flashing stimuli that are solid or an arbitrary checkerboard pattern with checks in the range of 0.15–0.3 c/◦ . This study demonstrates that spatial frequency can have a dramatic effect on SSVEP performance that is consistent across subjects. The results from both the discrete classification and the continuous path traversal experiments show a distinct bimodal distribution of SSVEP performance across the spatial frequency conditions. Fig. 5(a), (b), and Fig. 6 from the discrete classification experiment all reflect a similar bimodal spatial tuning with performance peaks at the 0 c/◦ and 2.4 c/◦ conditions. The results in Fig. 5(a) show that the solid 0 c/◦ condition produces an average accuracy of 97.7% given observation lengths greater than 3.5 s. The 2.4 c/◦ spatial condition achieves a reasonable accuracy of 85.1% using a shorter observation length of 2.5 s. Although the 2.4 c/◦ accuracy is less than the solid condition, the responses generated from the 2.4 c/◦ spatial frequency condition require a shorter time window for excitation. This is reflected in Fig. 5(b) where the 2.4 c/◦ condition obtains an averaged ITR of 45.3 bits/min which is significantly higher compared to 35.7 bits/min with the solid stimulus condition using a paired t-test (p = 0.02). The continuous traversal experiment confirms the results from experiment 1 as Fig. 9 shows that the performance of the traversal task averaged across all subjects exhibits a similar bimodal distribution across spatial frequency conditions.

Fig. 9. Part (a) shows the average run-duration from experiment 2 across all subjects for each of the spatial frequency conditions. Part (b) shows the average path-completion percentage for each of the conditions. On average, the subjects were able to complete the path the fastest with the 0 c/◦ and 2.4 c/◦ conditions which is reflected in (b) as those were the only two conditions where all subjects were able to completely finish the path. The error bars represent standard error.

The 0 c/◦ and 2.4 c/◦ peaks were the only conditions where all subjects were able to fully complete the path, thus giving a strong indication that these two spatial frequency conditions provide superior response characteristics and can result in optimal SSVEP performance compared to other spatial frequencies. These results concur with other findings in humans [5], [16] where similar bimodal spatial tuning curves were found when measuring VEP amplitude during stimulation from a single stimulus. While these studies did not explore the physiological mechanisms for the preferred spatial tuning, studies have shown that neurons in monkey visual cortex are tuned to a wide range of spatial frequencies, and similarly centered around 3 C/° [17]. The present results indicate that a similar relationship between stimulus frequency and amplitude

564

IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 25, NO. 6, JUNE 2017

extends to BCI performance when multiple stimuli are flashed simultaneously at different temporal frequencies. Interestingly, the 0 c/◦ condition resulted in a faster completion time on average, compared to the 2.4 c/◦ condition despite the fact that the 2.4 c/◦ achieves an overall higher ITR than 0 c/◦ condition. This may be due to the decrease in accuracy exhibited by the 2.4 c/◦ condition for longer observation lengths, as seen in Fig. 7. Additionally, all of the spatial frequency conditions, except for the solid condition, show a steady decrease in accuracy with increasing observation lengths over 2.5 s. This suggests the possibility of spatial adaptation in the visual system at these spatial frequencies. This is likely due to the fact that the checkerboard stimuli require more complex spatial/perceptual processing than the solid stimuli, potentially recruiting a different population of neurons in the visual system. Previous studies have shown a reduction of neural activity when patterned stimuli are continuously repeated [18]; however, the underlying neural mechanisms of this phenomenon are still unknown. Other studies have shown that spatially tuned neuronal populations (or spatial channels) can adapt during stimulation of spatial stimuli in which the strength of the spatial channel response declines throughout adaptation [19]–[21]. For the present study, the differences in accuracy with respect to observation length for different spatial frequencies likely indicate a similar pattern of spatial frequency adaptation over the time-course of stimulation. This agrees with previous EEG studies that use a single stimulus [22], [23]. These findings suggest that BCI designers should consider the VEP characteristics when using spatial frequency stimuli to optimize the BCI performance. The results in Fig. 8 of the subjective evaluation of visual irritation for each spatial frequency condition show that subjects, on average, perceive less overall irritation for higher spatial frequencies compared to lower spatial frequencies. Specifically, for the two top performing conditions, the average irritation index was 8.1 for the 0 c/◦ condition, whereas the irritation index was roughly half that at 4.5 for the 2.4 c/◦ condition. This indicates that practical VEP-based BCIs can be employed with less visually irritating stimuli while achieving comparable performance to traditional solid stimuli. This is an important consideration for BCIs that are intended to be used continuously over long periods of time. Although these results demonstrate that spatial frequency can have a measured effect on the accuracy and performance of SSVEP BCI target detection, additional analysis is needed to further characterize these effects. A longitudinal study is needed to determine the stability of subject-specific spatial frequency tuning and adaptation. Further, the relationship between spatial frequency and the temporal flashing (or pattern reversal) frequency needs to be studied. Tomoda et al., used 4 Hz stimuli and found similar tuning results to the present study [5]. In addition, studies in monkeys have shown visual neurons’ preference for spatial frequency is largely independent of temporal frequency in V1 and V2 [24]. While these studies provide some indication that the spatial tuning may be consistent across frequencies, a systematic study must be conducted that manipulates the spatial and temporal frequencies orthogonally. Additionally, further research

should also examine characteristics of human perceptualcognitive mechanisms in the context of SSVEP-based BCI. An ideal BCI reliably detects which object a user fixates among other irrelevant objects; However, psychophysical data indicate that objects that appear in high spatial proximity can cause the crowding effect [25] or localized attentional interference [26], reducing target detectability. Further, in practice, a visual display for SSVEP-BCI may contain numerous items, presumably with different frequencies, which demand attentional selection of an object of interest among others. Empirical investigation on the impact of spatial proximity of and the number of display icons on VEP-based BCI performance may offer an insight into characteristics of the human visual system important for efficient SSVEP-BCI performance. Overall, these results demonstrate that the clinically-studied mechanisms of spatial frequency tuning and adaptation are present in the context of multi-target stimulation, showing that spatial frequency selection can greatly impact SSVEP-BCI performance. This characterization can potentially be utilized for the development of more practical, ergonomic, and robust BCIs. R EFERENCES [1] J. C. Armington, “Psychophysical applications of human electroretinography,” J. Opt. Soc. Am., vol. 67, no. 11, pp. 1458–1465, 1977. [2] L. E. Leguire and G. L. Rogers, “Pattern electroretinogram: Use of noncorneal skin electrodes,” Vis. Res., vol. 25, no. 6, pp. 867–870, 1985. [3] J. C. Armington, T. R. Corwin, and R. Marsetta, “Simultaneously recorded retinal and cortical responses to patterned stimuli,” J. Opt. Soc. Am., vol. 61, no. 11, pp. 1514–1521, 1971. [4] J. C. Armington, K. Gaarder, and A. M. Schick, “Variation of spontaneous ocular and occipital responses with stimulus patterns,” J. Opt. Soc. Am., vol. 57, no. 12, pp. 1534–1539, 1967. [5] H. Tomoda, G. G. Celesia, and S. C. Toleikis, “Effect of spatial frequency on simultaneous recorded steady-state pattern electroretinograms and visual evoked potentials,” Electroencephalogr. Clin. Neurophysiol., Evoked Potentials, vol. 80, no. 2, pp. 81–88, 1991. [6] S. Sokol and B. Bloom, “Macular ERG’s elicited by checkerboard pattern stimuli,” in ERG, VER and Psychophysics (Documenta Ophthalmologica), vol. 13, T. Lawwill, Ed. The Netherlands: Springer, 1977, pp. 299–305. [Online]. Available: http://dx.doi.org/10.1007/978-94-0101312-3_37 [7] E. C. Lalor et al., “Steady-state VEP-based brain-computer interface control in an immersive 3D gaming environment,” Eurasip J. Appl. Signal Process., vol. 2005, no. 19, pp. 3156–3164, 2005. [8] B. Allison and I. Sugiarto, “Display optimization in SSVEP BCIs,” in Proc. Comput.-Human Interact., 2008, pp. 2–5. http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:No+ Title#0 and http://hmi.ewi.utwente.nl/chi2008/chi2008_files/allison.pdf [9] J. Saetang, Y. Punsawad, and Y. Wongsawat, “On the performance comparison of using checkerboard and flash ball visual stimulators for SSVEP-based BCI system,” in Proc. IFMBE, vol. 39. 2013, pp. 1549–1552. [10] R. Zerafa, T. Camilleri, O. Falzon, and K. P. Camilleri, “Comparison of plain and checkerboard stimuli for brain computer interfaces based on steady state visual evoked potentials,” in Proc. Int. IEEE/EMBS Conf. Neural Eng., 2013, pp. 33–36. [11] S. M. T. Müller, P. F. Diez, T. F. Bastos-Filho, M. Sarcinelli-Filho, V. Mut, and E. Laciar, “SSVEP-BCI implementation for 37–40 Hz frequency range,” in Proc. IEEE Eng. Med. Biol. Soc., Aug./Sep. 2011, pp. 6352–6355. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/22255791 [12] F. Sharbrough, C. E. Chatrain, R. P. Lesser, H. Luders, M. Nuwer, and T. W. Picton, “American electroencephalographic society guidelines for standard electrode position nomenclature,” J. Clin. Neurophysiol., vol. 8, no. 2, pp. 200–202, 1991.

WAYTOWICH et al.: OPTIMIZATION OF CHECKERBOARD SPATIAL FREQUENCIES

[13] G. Schalk, “BCI2000: A general-purpose brain-computer interface (BCI) system,” IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 1034–1043, Jun. 2004. [14] Z. Lin, C. Zhang, W. Wu, and X. Gao, “Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs,” IEEE Trans. Biomed. Eng., vol. 54, no. 6, pp. 1172–1176, Jun. 2007. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/17549911 [15] X. Chen, Y. Wang, M. Nakanishi, X. Gao, T.-P. Jung, and S. Gao, “Highspeed spelling with a noninvasive brain-computer interface,” Proc. Nat. Acad. Sci. USA, vol. 112, no. 44, pp. E6058–E6067, Nov. 2015. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/26483479 [16] S. Tobimatsu, S. Kurita-Tashima, M. Nakayama-Hiromatsu, and M. Kato, “Effect of spatial frequency on transient and steady-state VEPs: Stimulation with checkerboard, square-wave grating and sinusoidal grating patterns,” J. Neurol. Sci., vol. 118, no. 1, pp. 17–24, 1993. [17] R. L. D. Valois, D. G. Albrecht, and L. G. Thorell, “Spatial frequency selectivity of cells in macaque visual cortex,” Vis. Res., vol. 22, no. 5, pp. 545–559, Jan. 1982. [Online]. Available: http://dx.doi.org/10.1016/0042-6989(82)90113-4 [18] K. Grill-Spector, R. Henson, and A. Martin, “Repetition and the brain: Neural models of stimulus-specific effects,” Trends Cognit. Sci., vol. 10, no. 1, pp. 14–23, 2006. [19] C. Blakemore and P. Sutton, “Size adaptation: A new aftereffect,” Science, vol. 166, no. 902, pp. 245–247, 1969. [20] S. Klein, C. F. Stromeyer, and L. Ganz, “The simultaneous spatial frequency shift: A dissociation between the detection and perception of gratings,” Vis. Res., vol. 14, no. 12, pp. 1421–1432, 1974. [21] J. A. Movshon and P. Lennie, “Pattern-selective adaptation in visual cortical neurones,” Nature, vol. 278, no. 5707, pp. 850–852, 1979. [22] S. P. Heinrich and M. Bach, “Adaptation dynamics in pattern-reversal visual evoked potentials,” Documenta Ophthalmol., vol. 102, no. 2, pp. 141–156, 2001. [23] J. M. P. Baas, J. L. Kenemans, and G. R. Mangun, “Selective attention to spatial frequency: An ERP and source localization analysis,” Clin. Neurophysiol., vol. 113, no. 11, pp. 1840–1854, 2002. [24] K. H. Foster, J. P. Gaska, M. Nagler, and D. A. Pollen, “Spatial and temporal frequency selectivity of neurones in visual cortical areas V1 and V2 of the macaque monkey,” J. Physiol., vol. 365, pp. 331–363, Aug. 1985. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/4032318 [25] R. F. Hess, S. C. Dakin, and N. Kapoor, “The foveal ‘crowding’ effect: Physics or physiology?” Vis. Res., vol. 40, no. 4, pp. 365–370, 2000. [26] Y. Yamani, J. S. McCarley, J. R. Mounts, and A. F. Kramer, “Spatial interference between attended items engenders serial visual processing,” Attention, Perception, Psychophys., vol. 75, no. 2, pp. 229–243, 2013.

565

Nicholas R. Waytowich is a joint postdoctoral Research Fellow at the Laboratory for Intelligent Imaging and Neural Computing at Columbia University and the Human Research and Engineering Directorate at the US Army Research Laboratory. Waytowich received a Ph.D. in biomedical engineering from Old Dominion University (ODU), Norfolk, VA, USA. His research interests include machine learning, brain-computer interfaces, human-autonomy integration and adaptive systems. He is a member of IEEE and the IEEE Systems, Man, and Cybernetics Society. Yusuke Yamani received the B.A. degrees in Psychology and Philosophy from the State University of New York at Geneseo, and the M.S. degree in Human Factors and the Ph.D. degree in Psychology from University of Illinois at UrbanaChampaign. He was a Post-Doctoral Research Fellow at Human Performance Laboratory at University of Massachusetts Amherst and at Liberty Mutual Research Institute for Safety, Hopkinton, MA. He is currently an Assistant Professor of Psychology at Old Dominion University, Norfolk, VA, USA, where he directs the Applied Cognitive Performance Laboratory. He is interested in basic and applied aspects of perception, cognition, and attention. Dean J. Krusienski (M’01-SM’14) received the B.S., M.S., and Ph.D. degrees in electrical engineering from The Pennsylvania State University. He conducted his post-doctoral research in the Brain-Computer Interface (BCI) Laboratory, Wadsworth Center of the New York State Department of Health. He is currently a Professor of electrical and computer engineering at Old Dominion University (ODU), Norfolk, VA, USA, where he directs the Advanced Signal Processing in Engineering and Neuroscience (ASPEN) Lab. He is also the Graduate Program Director and a founding member of the biomedical engineering program at ODU. His research interests include biomedical signal processing, pattern recognition, braincomputer interfaces, and neural engineering.