Using Wiimote for 2D and 3D Pointing Tasks: Gesture Performance Evaluation Georgios Kouroupetroglou1,2, Alexandros Pino2, Athanasios Balmpakakis1, Dimitrios Chalastanis1, Vasileios Golematis1, Nikolaos Ioannou1, and Ioannis Koutsoumpas1 1
Department of Informatics and Telecommunications 2 Accessibility Unit for Students with Disabilities, National and Kapodistrian University of Athens, Panepistimiopolis, GR-15784, Athens, Greece
[email protected]
Abstract. We present two studies to comparatively evaluate the performance of gesture-based 2D and 3D pointing tasks. In both of them, a Wiimote controller and a standard mouse were used by six participants. For the 3D experiments we introduce a novel configuration analogous to the ISO 9241-9 standard methodology. We examine the pointing devices’ conformance to Fitts’ law and we measure eight extra parameters that describe more accurately the cursor movement trajectory. For the 2D tasks using Wiimote, Throughput is 41,2% lower than using the mouse, target re-entry is almost the same, and missed clicks count is three times higher. For the 3D tasks using Wiimote, Throughput is 56,1% lower than using the mouse, target re-entry is increased by almost 50%, and missed clicks count is sixteen times higher. Keywords: Fitts’ law, 3D pointing, Gesture User Interface, Wiimote.
1
Introduction
Nowadays, low-cost hand-held devices, introduced along with widespread game platforms/consoles, can also be used as input devices in general purpose Personal Computers. Thus, during the last years there has been a growing research interest in the domain of device-based gesture user interaction. Nintendo's Wii Remote Control (known as Wiimote) represents a typical example of these devices. Most of them incorporate accelerometer sensors. Accelerometer-based recognition of dynamic gestures has been investigated mainly by applying Hidden Markov Models (HMM) [1-2] and their usability has been evaluated compared to other modalities [3]. Gesture recognition for the Wiimote using either its 3-axis accelerometer or its high-resolution high-speed IR camera [7] has been developed by applying various methods and techniques, such as simple pattern recognition approaches [4], HMM [5], Dynamic Time Warping combined with HMM [10], or Slow Feature Analysis and parametric bootstrap [6]. GesText is an accelerometer-based Wiimote gestural text-entry system [9]. Wiimote can be utilized to uncover the user’s cultural background by analyzing E. Efthimiou, G. Kouroupetroglou, S.-E. Fotinea (Eds.): GW 2011, LNAI 7206, pp. 13–23, 2012. © Springer-Verlag Berlin Heidelberg 2012
14
G. Kouroupetroglou et al.
his patterns of gestural expressivity in a model based on cultural dimensions [8]. SignWiiver, a gesture recognition system which lets the user perform gestures with a Wiimote, uses a language built around the movement parameter of Natural Sign Languages [11]. Usability evaluation based on gesture recognition also revealed the applicability of Wiimote as a musical controller [25]. The point-and-click metaphor (usually referred to as pointing) constitutes a fundamental task for most two-dimensional (2D) and three-dimensional (3D) Graphical User Interfaces (GUI) enabling users to perform an object selection operation. Moreover, typing, resizing, dragging, scrolling, as well as other GUI operations require pointing. In order to develop better pointing techniques we need to understand the human pointing behavior and motor control. Fitts’ Law [12] can be used to: a) model the way users perform target selection, b) measure the user’s performance and c) compare the user’s performance amongst various input devices or the change in performance over time. Fitts’ law has been applied to three-dimensional pointing tasks [13] as well as to the design of the gesture-based pointing interactions [14-15], including the Wiimote [16-17]. The most common evaluation measures of Fitts’ law are speed, accuracy, and Throughput [18]. In this paper we present two experiments to comparatively evaluate the performance of gesture-based 2D and 3D pointing tasks. Beyond testing Fitts’ law, we measure the following eight extra parameters that describe more accurately the real cursor movement trajectory: missed clicks (MCL), target re-entry (TRE), task axis crossing (TAC), movement direction change (MDC), orthogonal direction change (ODC), movement variability (MV), movement error (ME), and movement offset (MO). For the 3D experiments we introduce a novel configuration analogous to the ISO 9241-9 standard [19] methodology.
2
Methodology
Fitts [12] proposed a model for the tradeoff between accuracy and speed in human motor movements. The model, commonly known as Fitts' law, is based on Shannon's information theory. Fitts proposed to quantify a movement task's difficulty using information theory by the metric "bits". Specifically, he introduced the Index of Difficulty (ID): log
2
(1)
D and W are the target’s distance and width respectively and are analogous to signal and noise, in Shannon's original research on electronic communications systems. The following expression for ID is more commonly used today, as it improves the information-theoretic analogy [18]: log
1
(2)
Because D and W are both measures of distance, the term in the parentheses is without units. "Bits" emerges from the choice of base 2 for the logarithm. Fitts' law is
Using Wiimote for 2D and 3D Pointing Tasks: Gesture Performance Evaluation
15
often used to build a prediction model with the Movement Time (MT) to complete point-select tasks as the dependent variable: ·
(3)
The slope (a) and intercept (b) coefficients in the prediction equation are determined through empirical tests, typically using linear regression. In order to evaluate the Wiimote’s conformance to Fitts’ law as an input device, we have designed and implemented a novel software application for our experiments that covers both 2D and 3D gesture-based user interaction. Our methodology is based on the ISO 9241-9 standard [19-20], which describes a standardized procedure to evaluate the performance, comfort, and effort in using computer pointing devices; this procedure offers the ability to understand the experimental results or to undertake comparisons between studies. For the 2D case, in each multi-directional test, 16 circular targets are arranged in an equidistance layout (Fig. 1). The task begins with a click on the centre of the first target; then the participant must move the cursor directly to the opposite target and click on it, and so on clockwise. The target to be clicked is highlighted every time. Each test block ends when all targets have been selected (16 trials) and 5 blocks are run with different combinations of target width and circle radius (with 5 different Indexes of Difficulty) giving a total of 80 trials per user.
(1)
(2)
(3)
Fig. 1. Screenshot of the 2D pointing task
For the 3D case, 8 spherical targets are placed at the corners of a 3-dimensional cube (Fig. 2). Each task begins with a click on the centre of a target. Then the participant must move the cursor directly to the target that is opposite to the center of the cube and click on it. After a successful trial the cursor teleports to another target that will become the beginning of the next route. The next target is highlighted every time. Each test block ends when all 8 equidistance diagonal routes, that connect the 8
16
G. Kouroupetroglou et al.
targets, are successfully done (8 trials) and 5 blocks are run for different target circle radii (in total 5 different Indexes of Difficulty) giving a total of 40 trials peruser.
Fig. 2. Screenshot of 3D pointing task
Fitts proposed to quantify the human rate of information processing in aimed movements using “bits per second” as unit. Fitts called the measure “index of performance”; today it is more commonly called “Throughput” (TP, in bits/s). Although different methods of calculating Throughput exist in the literature, the preferred method is the one proposed by Fitts in 1954 [12]. The calculation involves a direct division of means: dividing ID (bits) by the mean Movement Time, MT (seconds), computed over a block of trials: (4) The subscript e in IDe reflects a small but important adjustment, which Fitts endorsed in a follow-up paper [22]. An “adjustment for accuracy” involves first computing the “effective target width” as 4,133
(5)
where SDx is the observed standard deviation in a participant's selection coordinates over repeated trials with a particular D-W condition. Computed in this manner, We includes the spatial variability, or accuracy, in responses. In essence, it captures what a participant actually did, rather than what he or she was asked to do. This adjustment necessitates a similar adjustment to ID, yielding an “effective Index of Difficulty”: log
1
(6)
Using Wiimote for 2D and 3D Pointing Tasks: Gesture Performance Evaluation
17
Calculated using the adjustment for accuracy, TP is a human performance measure that embeds both the speed and accuracy of responses. TP is most useful as a dependent variable in factorial experiments using pointing devices or pointing techniques as independent variables. Additionally, based on the theory proposed by MacKenzie et al. [21], we measure the following extra parameters of the real cursor movement trajectory monitored by our application: • •
Missed Clicks (MCL) – occurs when an input device button click is registered outside of the target. Target re-entry (TRE) – if this behaviour is recorded twice in a sequence of ten trials, TRE is reported as 0,2 per trial. A task with one target re-entry is shown in Figure 3.
Fig. 3. Target Re-Entry
•
Task axis crossing (TAC) – the task axis is defined as the straight line from the starting point to the target centre (see Figure 4). A task axis crossing occurs when the cursor crosses this line, like it does once in Figure 5.
Fig. 4. A “perfect” target-selection task
Fig. 5. Task Axis Crossing
•
Movement direction change (MDC) – occurs when the tangent to the cursor path becomes parallel to the task axis. In the trajectory of Figure 6, three MDCs are logged.
Fig. 6. Movement Direction Change
•
Orthogonal direction change (ODC) – direction changes that occur along the axis orthogonal to the task axis, as it happens twice in Figure 7.
18
G. Kouroupetroglou et al.
Fig. 7. Movement Direction Change
The five measures above characterize the pointer path by logging discrete events. Three continuous measures complete the set of calculations: •
Movement variability (MV) is a continuous measure computed from the x-y coordinates of the pointer during a movement task. It represents the extent to which the sample cursor points lie in a straight line along an axis parallel to the task axis. Consider Figure 8, which shows a simple left-to-right target selection task, and the path of the pointer with five sample points. Assuming the task axis is y = 0, yi is the distance from a sample point to the task axis, and is the mean distance of the sample points to the task axis. Movement variability is computed as the standard deviation in the distances of the sample points from the mean: ∑ 1
(7)
In a perfectly executed trial, MV = 0. • Movement error (ME) is the average deviation of the sample points from the task axis, irrespective of whether the points are above or below the axis. Assuming the task axis is y = 0 as in Figure 8, then ∑|y | n
(8)
In an ideal task, ME = 0. • Movement offset (MO) is the mean deviation of sample points from the task axis. Unlike movement error, this measure is dependent on whether the points are above or below the axis. y
(9)
Movement offset represents the tendency of the pointer to veer “left” or “right” of the task axis during a movement. In an ideal task, MO = 0.
Fig. 8. Sample coordinates of pointer motion
Using Wiimote for 2D and 3D Pointing Tasks: Gesture Performance Evaluation
19
These extra parameters have been previously applied in a 2D setup for Brain Computer Interface Cursor Measures for Motion-impaired and Able-bodied Users [23]. The experimental application was developed as a Virtual Instrument using the LabVIEW (Laboratory Virtual Instrumentation Engineering Workbench) graphical programming environment by National Instruments [24]. We tested Wiimote (Figure 9a) as a gesture input device getting real time data from both its high-resolution IR camera (in combination with the IR LED array illuminator shown in Figure 9b) and its 3-axis accelerometer. The Wiimote was treated as an HID (Human Interface Device) compliant device connected to a regular PC using Bluetooth communication. The computer used was a Pentium Core 2 Duo 1,8 GHz laptop with 3 GB of RAM and a NVIDIA GTS250 graphics card, running MS-Windows 7 Professional and LabVIEW 10.1. It was connected to an external 24” TFT monitor with 1280x800 pixels resolution, which was used as the main display for the experiments. Six male participants, students of the Department of Informatics, University of Athens, volunteered for the study. Their age range was 22-35 years (mean 25, SD 4,9). All of them had normal or corrected vision and were right-handed. They also reported an average three hour daily usage of a the mouse device. None of these participants had any previous experience with the Wiimote.
(a)
(b)
Fig. 9. (a) The Wii Remote Control1, (b) 4-LED infrared light source (illuminator)2
1 2
© http://www.wiids.co.uk © http://jct-sales.ecrater.com
20
G. Kouroupetroglou et al.
3
Results
During the experiments we discovered that even the slightest amount of sun light in the room was interfering with the Wiimote’s IR camera, making it impossible to get
3.000
y=444,72xͲ 33,762 R²=0,7253
MT:MovementTime(msec)
2Dexperiments 2.500
mouse
2.000
wiimote 1.500
mouse Wiimote
1.000
y=193,62x+255,18 R²=0,6869
500 0 2,0
2,5
3,0
3,5
4,0
4,5
5,0
ID:IndexofDifficulty
(a)
MT:MovementTime(msec)
10.000
3Dexperiments
9.000
y=1426,8x+1048,1 R²=0,3968
8.000 7.000
mouse
6.000
wiimote
5.000
mouse
4.000
Wiimote
3.000 2.000 1.000 0 1,5
2,0
2,5
3,0 3,5 ID:IndexofDifficulty
4,0
4,5
y=248,91x+1586,7 R²=0,1786
(b) Fig. 10. Measurements of Movement Time (mean values for all trials) as a function of Index of Difficulty for all the participants in 2D (a) and 3D (b) experiments using Wiimote and the mouse
Using Wiimote for 2D and 3D Pointing Tasks: Gesture Performance Evaluation
21
decent results when running on IR mode, i.e., on 3-D tests. We had to move to a very dark room (artificial light was not a problem) and run the trials again. In both 2D and 3D experiments users were instructed not to stop on erroneous clicks and an audio feedback was given in that case. Visual and audio feedback was also given on successful clicks. Each task was explained and demonstrated to participants and a warm up set of trials was given. A 100 Hz sampling rate was used for cursor trajectory data acquisition. Measurements of Movement Time (mean values for all trials) as a function of Index of Difficulty for all the participants in 2D (a) and 3D (b) experiments using the Wiimote and the mouse are presented in Fig. 10. After the statistical analysis of all data from all users, we present the results for the additional cursor movement parameters in Table 1. Table 1. Calculated parameters of the cursor trajectory generated by the two gesture input devices in 2D and 3D experiments
movement offset
MO
movement error
ME
movement variability
MV
orthogonal direction change
ODC
movement direction change
MDC
task axis crossing
TAC
target re-entry
TRE
missed clicks
MCL
throughput
TP
mouse
5,05
0,05
0,12
1,48
6,04
0,91
0,32
0,39
0,04
Wiimote
2,97
0,14
0,11
1,56
18,06
2,48
0,55
0,79
0,22
mouse
1,71
0,06
0,10
Wiimote
0,75
0,96
1,46
2D
3D
4
Conclusions
Throughput calculations are consistent with other studies for mice (they range from 5 to 5,9 for 2D tasks). For the 2D tasks using Wiimote, Throughput is 41,2% lower than using the mouse, target re-entry is almost the same, and missed clicks count is 3 times higher. For the 3D tasks using Wiimote, Throughput is 56,1% lower than using the mouse, target re-entry is more than 14 times higher, and missed clicks count is 16 times higher. Furthermore, Fig. 10 shows that the fitting line correlation coefficient (R2), which reflects the reliability of the linear relationship between MT and ID values and, therefore, the compliance to Fitts’ law, is generally slightly higher for the Wiimote controller than the mouse and significantly lower for the 3D than the 2D experiments.
22
G. Kouroupetroglou et al.
We must note that for the 2D experiments we used the mouse in a standard way dragging it on a Goldtouch fabric pad and clicking with the left mouse button; as far as the Wiimote is concerned, for the 2D tests we acquired cursor movement coordinates by the device’s accelerometers, taking into account only x and y axis normalized data. For the 3D experiments the difference was in that we used the mouse’s scrolling wheel in order to move on the z axis, scrolling up to “go inwards” the screen, and scrolling down in order to “come outwards”; regarding the Wiimote, for 3D tests we only considered the IR camera data for movement on all axis, calculating each time the distance difference between a pair of lights that the IR camera was seeing for z-axis movement, and the mean x-y coordinates of the same pair of lights for the other two axis respectively. The algorithm of our application chose a pair of visible lights, among the four available, every 10ms and changed pair when one or both of them were no longer visible. We conclude that the Wiimote was proven to be a much slower and harder to use input device for both 2D and 3D pointing tasks than the mouse. 3D tests show a strong weakness of both the Mouse and Wiimote to work effectively as 3D pointing devices, which is partly justified by the fact that all users had no previous experience of using these devices for such tasks. Future work may include the involvement of more users in the experiments, involvement of disabled users to measure their gesture abilities, research on how performance changes over time (i.e., familiarization with the Wiimote and performance improvement), introduction of new trajectory measures for 3D tasks (also in spherical coordinates), and construction of a larger experimental IR LED grid in order to test Wiimote gesture-based interaction again, anticipating more accurate results.
References 1. Hofmann, F.G., Heyer, P., Hommel, G.: Velocity Profile Based Recognition of Dynamic Gestures with Discrete Hidden Markov Models. In: Wachsmuth, I., Fröhlich, M. (eds.) GW 1997. LNCS (LNAI), vol. 1371, pp. 81–95. Springer, Heidelberg (1998) 2. Mantyjarvi, J., Kela, J., Korpipaa, P., Kallio, S.: Enabling fast and effortless customization in accelerometer based gesture interaction. In: MUM 2004, pp. 25–31. ACM Press (2004) 3. Mantyjarvi, J., Kela, J., Korpipaa, P., Kallio, S., Savino, G., Jozzo, L., Marca, D.: Accelerometer-based gesture control for a design environment. Personal Ubiquitous Computing 10(5), 285–299 (2006) 4. Kratz, S., Rohs, M.: A $3 Gesture Recognizer: Simple Gesture Recognition for Devices Equipped with 3D Acceleration Sensors. In: International Conference on Intelligent User Interfaces (IUI 2010), pp. 341–344. ACM Press (2010) 5. Schlomer, T., Poppinga, B., Henze, N., Boll, S.: Gesture recognition with a Wii controller. In: TEI 2008 - Tangible and Embedded Interaction Conference, pp. 11–14. ACM Press (2008) 6. Koch, P., Konen, W., Hein, K.: Gesture Recognition on Few Training Data using Slow Feature Analysis and Parametric Bootstrap. In: International Joint Conference on Neural Networks, Barcelona, pp. 1–8 (2010) 7. Lee, J.C.: Hacking the Nintendo Wii remote. IEEE Pervasive Computing 7(3), 39–45 (2008)
Using Wiimote for 2D and 3D Pointing Tasks: Gesture Performance Evaluation
23
8. Rehm, M., Bee, N., Andre, E.: Wave Like an Egyptian - Accelerometer Based Gesture Recognition for Culture Specific Interactions. In: HCI 2008: Culture, Creativity, Interaction (2008) 9. Jones, E., Alexander, J., Andreou, A., Irani, P., Subramanian, S.: GesText: AccelerometerBased Gestural Text-Entry Systems. In: CHI 2010, Atlanta, Georgia, USA, April 10-15 (2010) 10. Leong, T., Lai, J., Pong, P., Panza, J., Hong, J.: Wii Want to Write: An Accelerometer Based Gesture Recognition System. In: Intern. Conf. on Recent and Emerging Advanced Technologies in Engineering, Malaysia, pp. 4–7 (2009) 11. Malmestig, P., Sundberg, S.: SignWiiver - implementation of sign language technology. University of Göteborg (2008), http://www.tricomsolutions.se/documents/SCP-T001.pdf 12. Fitts, P.M.: The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology 47(6), 381–391 (1954); reprinted in Journal of Experimental Psychology: General 121(3), 262–269 (1992) 13. Murata, A., Iwase, H.: Extending Fitts’ law to a three-dimensional pointing task. Human Movement Science 20, 791–805 (2001) 14. Chen, R., Wu, F.-G., Chen, K.: Extension of Fitts’ Law for the design of the gesture pointing interaction. In: 3th World Conference on Design Research - IASDR 2009, Korea, pp. 4611–4620 (2009) 15. Foehrenbach, S., König, W., Gerken, J., Reiterer, H.: Natural Interaction with Hand Gestures and Tactile Feedback for large, high-res Displays. In: MITH 2008: Workshop on Multimodal Interaction Through Haptic Feedback, Napoli, Italy (2008) 16. Fikkert, W., van der Vet, P., Nijholt, A.: Hand-held device evaluation in gesture interfaces. In: 8th International Gesture Workshop - GW 2009 (2009) 17. McArthur, V., Castellucci, S.J., MacKenzie, I.S.: An empirical comparison of “Wiimote” gun attachments for pointing tasks. In: ACM SIGCHI Symposium on Engineering Interactive Computing Systems – EICS 2009, pp. 203–209. ACM, New York (2009) 18. MacKenzie, I.S.: Movement time prediction in human-computer interfaces. In: Baecker, R.M., Buxton, W.A.S., Grudin, J., Greenberg, S. (eds.) Readings in Human-Computer Interaction, 2nd edn., pp. 483–493. Kaufmann, San Francisco (1995) 19. ISO: Ergonomic requirements for office work with visual display terminals (vdts)-part 9: Req. for non-keyboard input devices. Technical Report 9241-9 (2000) 20. Soukoreff, W., MacKenzie, S.: Towards a standard for pointing device evaluation, perspectives on 27 years of Fitts’ law research in HCI. International Journal of HumanComputer Studies 61(6), 751–789 (2004) 21. MacKenzie, I.S., Kauppinen, T., Silfverberg, M.: Accuracy measures for evaluating computer pointing devices. In: ACM Conference on Human Factors in Computing Systems – CHI 2001, pp. 9–16. ACM, New York (2001) 22. Fitts, P.M., Peterson, J.R.: Information capacity of discrete motor responses. J. Exp. Psychology 67, 103–112 (1964) 23. Pino, A., Kalogeros, E., Salemis, I., Kouroupetroglou, G.: Brain Computer Interface Cursor Measures for Motion-impaired and Able-bodied Users. In: 10th International Conference on Human-Computer Interaction, vol. 4, pp. 1462–1466. Lawrence Erlbaum Associates, Inc., Mahwah (2003) 24. The LabVIEW Environment, http://www.ni.com/labview/ 25. Kiefer, C., Collins, N., Fitzpatrick, G.: Evaluating the Wiimote as a Musical Controller. In: International Computer Music Conference - ICMC 2008 (2008)
