EYE-TRACKER DATA FILTERING USING PULSE COUPLED NEURAL NETWORK Sylvain Chartier & Patrice Renaud Centre de recherche de Département de psychoéducation et psychologie l’Institut Philippe Pinel de Montréal Université du Québec en Outaouais 10,905 Henri-Bourassa Est 283 boul. Alexandre-Taché, Montréal, Québec, H1C 1H1, Canada POB 1250, station Hull, Gatineau, QC, J8X 3X7, Canada
[email protected] [email protected]
ABSTRACT
Noise
Data obtained from eye-tracker are contaminated with noise due to eye blink and hardware failure to detect corneal reflection. One solution is to use a nonlinear filter such as the median. However, median filters modify both noisy and noise free data and they are therefore difficult to use in real time applications. To overcome these limits, a simplified pulse coupled neural network (PCNN) is proposed to correctly detect and remove noisy data while leaving uncorrupted data untouched. Results indicated that a filter based on a PCNN achieved a much better performance than the median filter in peak signal-to-noise ratio (PSNR) and in visual inspection.
Figure 1. 1 minute POR angular deviation data sample Noise degrades the signal and it is usually desirable to remove and replace it before any analysis can be performed. Usually, the noise is simply discarded from the data, resulting in some cases up to 10 to 30% data loss [11]. Obviously, this method cannot be used for real time application like an input device because no data can be discarded and signals must remain continuous. Moreover, signal loss duration due to noise can be quite important; between 1/60th of a second to over a second. Also, sometimes uncorrupted data can have a POR deviation amplitude higher than the one associated to noise. Thus, a simple threshold cannot be used to filter the data. In addition, a standard linear filter cannot be used for it fails to distinguish corrupted data from uncorrupted data. Consequently, signal restoration must be accomplished by a nonlinear filter [12]. These filters are popular for they remove high signal values in relation to their neighbor signal values. The classic example of such a filter is the median. The median filter proved to be robust to impulse type noise. It will eliminate changes in the input signal with duration less than half the size of the filter window. Moreover, median filter is applied to whole signal time series leading to the replacement of good uncorrupted samples by the median value. However, if the signal loss duration is long, a large median window is needed and this leads to more uncorrupted data being replaced [13]. Ideally, only the corrupted sample should be replaced while leaving uncorrupted sample intact. Although the median filter gives a better performance than the linear filter, its performance is far from optimal.
KEY WORDS Eye-tracker, filter, pulse couple neural network, median, noise.
1. Introduction Eye-tracking has been used in a wide variety of applications as a window to cognitive processes [1]-[10]. The eye-tracker system relies on the corneal reflection of an infra-red source that is measured relative to the pupil center location. These particular corneal reflections can be located with video-based eye-trackers collecting infra-red reflections. Generated data files contain information on time and position relative to eye movements. From that information, angular deviation between a given target and the point of regard (POR) can be computed [6], [9]. Data collected are generally not noise free. There is some noise due to eye blink and hardware failure in capturing corneal reflection. Figure 1 illustrates an example of 1 minute POR angular deviation data obtained from a human subject.
530-071
91
To our knowledge, no method has been developed to remove and restore POR angular deviation data in real time. In this paper, it is proposed to use a pulse coupled neural network (PCNN) to denoise eye-tracker data. PCNN has been used in standard image processing [14][16] and one of its applications is the detection of image impulse noise. Although this kind of noise is different from the one observed in eye-tracker data, the generic nature of the PCNN makes it well suited to treat eyetracker noise. Before introducing the filter algorithm in Section 3, the PCNN is briefly described in the context of eye-tracker noise detection (Section 2). In Section 4, simulations results on both artificial and natural noise are reported and a short conclusion follows.
vL
αL
W
Yi Linking input
1+βLi
Σ
Threshold
W
vΘ αΘ
Θi M
Yi Feeding input
Fi
Σ
M
Π
Ui
vF
αF
Yi Output Step Function
Si Input part
Linking part
Pulse generator
Figure 2. Architecture of standard PCNN.
2. PCNN
However, since the model is going to process static data (POR angular deviation), including a leaky integrator in the path of external input, it offers no advantages. Therefore, following [15] the model can be simplified while keeping its distinguishing properties. The feeding input Fi can be set to Si and the linking field does not need to be dynamic. The new resulting equation describing the model thus becomes
From studies of a cat visual cortical neurons, linking field networks were developed by Eckhorn [17]. Later, Johnson and Ritter modified the linking field model to propose PCNN [18]. Standard one dimensional PCNN is described by the following equations:
Fi [n] = Si + Fi [n − 1]e −α F + vF ( M * Y [n − 1])i Li [n] = Li [n − 1]e
−α L
+ vL (W * Y [n − 1])i
(1) (2)
U i [n] = Fi [n](1 + β Li [n])
Fij [n] = Si
(6)
(3)
Θi [n] = Θi [n − 1]e −α Θ + vΘY [n − 1]
Li [n] = (W * Y [n − 1])i
(7)
(4)
U i [n] = Fi [n](1 + β Li [n])
(8)
1 if U i [n] > Θi [n] Yi [n] = 0 otherwise
Θi [n] = Θi [n − 1]e
(5)
−α Θ
+ vΘY [n − 1]
(9)
1 if U i [n] > Θi [n] Yi [n] = 0 otherwise
where Si is the input from the POR angular deviation, Fi is the feeding input, Li is the linking input, β is the linking coefficient, n is the iteration step, Yi is the output, M and W are the weight matrices, * is the convolution operator, vL, vF vΘ are potentials constant, Θi is the dynamic threshold, αF, αL and αΘ are time constant.
(10)
This simplification is reflected in the network architecture as illustrated by the Figure 3.
Yi Linking input
Figure 2 illustrates the architecture of the PCNN. Each neuron contains a feeding and linking part (Equation 3). The neuron receives input from the external stimuli and from given neighbour neurons. The output neuron fires if the internal activity surpasses the dynamic threshold. Outputs modify the threshold, as well as properties of the linking and feeding input through feedback. As a result, PCNN produces a series of pulses.
W
1+βLi
Σ
Threshold
W
vΘ
Θi Feeding input
Fi
Si
Input part
Π
Ui
Linking part
αΘ
Yi Output Step Function
Pulse generator
Figure 3. Architecture of the simplified PCNN. Like in the standard PCNN, this simplified model still has the property that its connections encourage neighbour neurons with similar input to emit synchronous pulse. Thus, if a POR angular deviation is dissimilar enough
92
n = 1 to 200. This is illustrated in Figure 5. It can be seen that the third neuron fires at a much higher frequency compared to the other two ( Y3 = 29 200 >> Y1 ≅ Y2 = 8 200 ). Consequently, S3 is replaced by the median value of the window, which is 4. The resulting denoised series is thus [4, 3, 4].
from the previous one then, its firing frequency will be higher (or lower) compared to its neighbour’s average. If a simple threshold is added, then it should be easy to distinguish noisy data from noise free data.
3. Description of the algorithm In short, a convolution window, F, of size K is created and the data, Si, under consideration lies at the end of the window.
F = [ si − K −1 , si − K − 2 ,..., si −1 , si ]
(11)
This window is input into the PCNN and the frequency for each associated neuron is computed. The output frequency associated with Si is compared with its left neighbour average. If this difference is higher (in absolute value) than a given threshold; T, a noisy data is thus detected. This faulty data is then replaced by the median of the convolution window data. If the difference is lower than the threshold then the data remains unchanged. The algorithm is applied recursively, thus the present denoised data is part of the detection of further noisy data. This procedure is illustrated in Figure 4. PCNN
Figure 5. Firing output of 3 neurons as a function of time. T
Input
Output Median
4. Simulations 4.1 Methodology
Figure 4. PCNN filter algorithm procedure.
Raw data were collected by an ASL 504 series eyetracker combined with a V8 Virtual Research head mounted display. The single eye-tracker returns 2 DOF, i.e. variations in the x and y planes, at 60 Hz with a margin of error of ± 0.5 degrees. From the data file generated with ASL software, the POR angular deviation was computed between the POR and the target, which is the center of the virtual environment scene.
Contrary to standard neural networks, there are no weight modifications through learning in PCNN. The weights, W, are usually Gaussian and center on the data under consideration. However, contrary to image processing, the data under consideration, Si, is not at the center of the convolution window but at the end of it. Thus, for the purpose of eye-tracker data filtering, the coefficients are half Gaussian and it is given by the following
Wi = e
−
( σi ) 2
To evaluate the performance of the PCNN filter on eyetracker data, two approaches were followed: artificial and natural noisy data. The first PCNN filter was compared to a median filter on artificially noisy data. The purpose of this setting is to objectively evaluate the performance difference between the two filters. To accomplish this, a POR noise free data portion was selected. Then, noise was randomly added to the data with an amplitude and duration similar to what it found in a natural setting. To evaluate performance, a peak signal-to-noise ratio (PSNR) [12] [19] was used. If x is the original time series of length n and x’ the restored time series of length n, then the PSNR of x’ is given by:
(12)
For example, a 3-dimensional Gaussian weight vector, with a standard deviation of 4, will be W = [0.37, 0.78, 1]. Next, this procedure is illustrated numerically. For example, the PCNN filter must detect the noise present in this short time series containing solely 3 data; s = [4, 3, 46]. The free parameters are set to β = 0.1, vΘ = 15, αΘ = 0.1, W = [0.37, 0.78, 1]. For this example, a size 3 window is used. Let F = [4, 3, 46], where the third element is noisy and it should be detected by the firing pattern. From this feeding, the output is obtained for time
93
Figure 6. Denoised eye-tracker data with median and PCNN filter compared to original noise free data.
Natural Noisy Data
Median Filter
PCNN Filter
Figure 7. Four examples of natural data filter by the Median and PCNN filter.
94
n 2 ∑180 PSNR ( x ' ) = 10 log10 n i =1 2 ' ∑ ( xi − xi ) i =1
The good performance of the PCNN is also shown with real eye-tracker data. Figure 7 shows the results for median filter and PCNN filter on 4 samples time series. Once again, the PCNN filter was able to effectively denoise the data from both eye-blink and hardware failure, while leaving noise free data untouched. For example, the third time series had noise free data amplitude as high as the noisy ones. Nevertheless, the PCNN filter was able to correctly denoise the data. It was not the case with the median filter that not only changed every data value but also often filtered noise free data as well. For example, in the third time series just before the 10th second, the median filter erroneously filtered good data. On the other hand, the PCNN filter correctly handled the data.
(13)
Where, 180 is the maximum possible angle deviation (in degrees) given by the POR. Three noise free series of 33 seconds each were selected where noise was randomly added to each of them with a duration that varied from 1/60 to 1 sec. The number of signals loss was added so that their total duration was about 10% of the whole time series. More precisely, for the first time series 11.2 % of data were corrupted with noise giving a PSNR of 59.2 dB. For the second time series 12.1 % of the data were corrupted with noise giving a PSNR of 59.7 dB. Finally, 9.0 % of the data of the third series were corrupted with noise giving a PSNR of 59.0 dB.
5. Conclusion
In the second setting, the PCNN and median filter were tested on 4 natural time series obtained from the eyetracker. The objective of this setting is to visually evaluate the filter capacity to detect and filter the noisy data naturally found in experiments. To have a good idea of the performance of these filters, 4 one-minute eye-tracker recordings were selected.
This paper has shown that the PCNN can be used in a procedure to detect and replace noisy eye-tracker data. Results have shown that the PCNN filter is more effective, when compared to a standard median filter, from both PSNR and visual inspection. Furthermore, in order to determine if the current pixel is noisy or not, the PCNN filter procedure takes into account only past data. This property enables the filter to be implemented online, which is not the case with median filtering. The PCNN filter procedure is not limited to time series from eyetracker but could also be used in similar time series like pupil dilation. [20]. This paper thus widens the range of PCNN applications in the signal processing domain. Further studies should investigate how PCNN filters could be used for input devices.
For both conditions, free parameters for the PCNN procedure were set according to [15]: The convolution window size was set to 5, β = 0.1, vΘ = 15, αΘ = 0.1, W = [0.07, 0.22, 0.5, 0.85, 1] and T = 18. For the median filter, the threshold was set to 18. The median filter is effective if its window size is twice the maximum noise duration [13]. In our case the maximum noise duration is 1 second (60 data) thus the window size for the median filter is 2 seconds (121 data).
Acknowledgements
4.2 Results
The authors are grateful to J.-F. Ferry for his help in reviewing this paper.
Figure 6 illustrates the various performances of the median and PCNN filters to denoise the 3 time series with artificial noise. It can be seen that PCNN closely matched the original noise free series. The median filter was also able to correctly remove the noise. However, the median filter also affected noise free data. Table 1 shows the performance of each filter as a function of their PSNR. The median filter was able to achieve a better PSNR compared to no filtering for all 3 time series. However, the PCNN filter achieved a far better performance than the median filter; showing an average of 16.24 dB greater.
Noisy Median PCNN
References [1] R. M. Satava, Medical applications of virtual reality, Journal of medical Systems, 19, 1995, 275-280. [2] M. F. Land, & D. N. Lee, Where we look when we steer, Nature, 369, 1994, 742-744. [3] P. Suppers, Eye-movement models for arithmetic and reading performance, Eye movements and their role in visual and cognitive processes (E. Kowler, New York: Elsevier, 1990, 455-477).
Table 1. PSNR of the 3 time series Series 1 Series 2 Series 3 59.18 dB 59.66 dB 59.00 dB 61.94 dB 66.29 dB 61.03 dB 77.12 dB 77.94 dB 82.92 dB
[4] L. E. Magee, Virtual reality simulator (VRS) for training ship and handling skills, Virtual reality, training’s future? (R. J. Seidel & P. R. Chantelier, New York: Plenum, 1997, 19-29).
95
[5] L. Gamberini, P. Cottone, D. Spagnolli, D. Varotto, D. & G. Mantovani, Responding to a fire emergency in a virtual environment: Different patterns of action for different situations, Ergonomics , 46, 2003,842-858.
[13] C. Chandra, M. S. Moore, S. K. Mitra, An Efficient Method for the Removal of Impulse Noise From Speech and Audio Signals. Proceeding to IEEE International Symposium on Circuits and Systems, 1999, 206-209.
[6] P. Renaud, S. Décarie, L. Gourd, B. Paquin, and S. Bouchard, Eye-tracking in immersive environments: a general methodology to analyze affordance-based interactions from oculomotor dynamics, Cyberpsychology and Behavior, 6, 2003, 519-526.
[14] T. M. Nazmy, Evaluation of the PCNN standard model fro image processing purposes, International Journal of Intelligent Computing and Information Sciences, 4(2), 2004, 101-111. [15] Y. D. Ma, F. Shi, & L. Li, A New Kind of Impulse Noise Filter Based on PCNN. Proceeding of the International Conference on Neural Networks and Signal Processing, 1, 2003, 152-155.
[7] P. Renaud, J.-F. Cusson, S. Bernier, J. Décarie, S.-P. Gourd, & S. Bouchard, S., Extracting perceptual and motor invariants using eye-tracking technologies in virtual immersions. HAVE'2002-IEEE (Institute of Electrical and Electronics Engineers) International Workshop on Haptic Virtual Environments and their Applications, Ottawa (November), 2002, 73-78.
[16] J. L. Johnson, & M. L. Padgett, PCNN Models and Applications, IEEE Transactions on Neural Networks, 10(3), 1999, 480-498. [17] R. Eckorn, H. J. Reitboeck, M. Arndt, & P. W. Dicke, Feature linking via synchronization among distributed assemblies: Simulation of results from cat cortex, Neural Computation, 2, 1990, 293-307.
[8] P. Renaud, J. Proulx,. J.-L. Rouleau, S. Bouchard, G. Madrigrano, J. Bradford, & P. Fedoroff, The recording of observational behaviors in virtual immersion: A new clinical tool to address the problem of sexual preferences with paraphiliacs. Annual review of Cybertherapy and Telemedecine, 3, 2005, 85-92.
[18] J. L. Johnson, & D. Ritter, Observation of periodic waves in a pulse-coupled neural network, Optics Letters, 18(5), 1993, 1253-1255.
[9] A. T. Duchowski, E. Medlin, N. Cournia, H. Murphy, A. Gramopadhye, S. Nair, J. Vorah, B. Melloy, 3D Eye Movement Analysis, Behavior Research Methods, Instruments, & Computers (BRMIC), 34(4), 2002,.573591.
[19] E. Abreu, M. Lightstone, S. K. Mitra, & K. Arakawa, A New Efficient Approach for the Removal of Impulse Noise from Highly Corrupted Images, IEEE Transactions on Image Processing, 5, 1996, 1012-1025.
[10] G. Albert, P. Renaud, S. Chartier, L. Renaud, L. Sauvé, & S. Bouchard, Scene Perception, Gaze Behavior and Perceptual Learning in Virtual Environments, CyberPsychology and Behavior, 8, 2005, 592-600.
[20] M. Nakayama, & Y. Shimizu, An estimation model of pupil size for ‘blink artifact’ and its applications, Proceedings of the European Symposium on Artificial Neural Networks (ESANN’2002), 2002, 251-256
[11] B. Law, Eye movements in a virtual laparoscopic training environment, unpublished master thesis, Simon Fraser University, Canada, 2003. [12] T. Chen, & H. R. Wu, Space Variant Median Filters for the Restoration of Impulse Noise Corrupted Images, IEEE Transactions on Circuits and Systems-II: Analog and Digital Signal Processing, 48, 2001, 784-789.
96
