Perception and Haptics: Towards More Accessible Computers for Motion-Impaired Users Faustina Hwang, Simeon Keates, Patrick Langdon, P John Clarkson
Peter Robinson
University of Cambridge, Dept. of Engineering Trumpington Street, Cambridge. CB2 1PZ United Kingdom Tel: +44 1223 766958
University of Cambridge, Computer Laboratory Pembroke Street, Cambridge. CB2 3QG United Kingdom Tel: +44 1223 334637
{fh220, lsk12, pml24, pjc10}@eng.cam.ac.uk
[email protected]
ABSTRACT
1.1 The need to design for a range of users
For people with motion impairments, access to and independent control of a computer can be essential. Symptoms such as tremor and spasm, however, can make the typical keyboard and mouse arrangement for computer interaction difficult or even impossible to use. This paper describes three approaches to improving computer input effectivness for people with motion impairments. The three approaches are: (1) to increase the number of interaction channels, (2) to enhance commonly existing interaction channels, and (3) to make more effective use of all the available information in an existing input channel. Experiments in multimodal input, haptic feedback, user modelling, and cursor control are discussed in the context of the three approaches. A haptically enhanced keyboard emulator with perceptive capability is proposed, combining approaches in a way that improves computer access for motion impaired users.
For people with disabilities, access to and independent control over a computer system is highly desirable and possibly essential, providing a means of communicating and accessing information [9]. Computers can also play an important role in helping older adults function well in society [25]. By 2020, almost half the adult population in the UK will be over 50, with the over 80’s being the most rapidly growing sector [5]. Similar trends can be found in the United States. Users with physical impairments, however, often cannot cope with most current computer access systems. Conditions such as cerebral palsy, muscular dystrophy, and spinal injuries can give rise to many symptoms, including tremor, spasm, restricted range of motion, and reduced strength. The effects of ageing can include a decline in cognitive, perceptual, and motor abilities. For people with reduced capabilities, the typical keyboard and mouse arrangement for interaction with graphical user interfaces can be difficult or even impossible to use. The consequence in computer interface design is that as computer usage spreads throughout the population, the profile of the user base that must be accommodated is changing [22].
Keywords Interfaces for all, user models, force-feedback, keyboard emulator, Logitech Wingman, cursor control
1. INTRODUCTION
Increasingly, legislation is being enacted that prohibits discrimination on the grounds of disability. This includes requirements for employers to take “reasonable” measures to make employment accessible to all and for service providers to ensure equal access to services and products. Examples include the Americans with Disabilities Act [1], Section 508 of the Workforce Investment Act [26], and the UK Disability Discrimination Act [7]. Research in improving computer access for people with disabilities will affect the interpretation of the term “reasonable” in such a way that opportunities for people with disabilities will be improved.
Current interface design practices are based on user models and descriptions derived exclusively from studies of able-bodied users. However, such users are only one point on a wide and varied scale of physical capabilities. There are important differences between those with motion-impairments, whether elderly or disabled, and able-bodied users when they interact with computers. Without a proper understanding of those differences, interface design will remain an exercise in making interfaces more comfortable for motion-impaired users, rather than really usable.
1.2 Designing for a wider range of users
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Discussions about the concept of Universal Access (UA) have shown that traditional HCI approaches are not the best way to achieve wider accessibility and that more specific studies are required to reach this goal [22]. Two core themes from this work are proactivity – addressing the issue of accessibility at design time, and reactivity – the ability for the interface to be adapted to the user retrospectively [22]. It is generally accepted within the
PUI 2001 Orlando, FL USA Copyright 2001 ACM 1-58113-448-7-11/14/01 ...$5.00.
1
UA community that while retrospective adaptation of interfaces is the easiest option for developers to provide superficial solutions, proactive design offers the most potential for genuinely assisting wider access to computers. Central to the goal of wider access is a good understanding of the user, and a framework is required to achieve a successful description of computer interaction.
For a given input system, more than one of the three approaches may be employed at a time. Combining the approaches can greatly increase the complexity of the overall input system, so it makes sense initially to examine each in turn. Section 3 addresses the idea of increasing the number of input channels (approach 1), Section 4 addresses the idea of enhancing a commonly existing channel (approach 2), and Section 5 addresses the idea of making more effective use of existing input information (approach 3).
However, there is a temptation for designers to jump straight to development of technological solutions without first truly understanding the nature of the problem. This is especially true when a new and exciting form of technology is under consideration. For example, it is known that many assistive devices developed specifically for older users are not meeting the needs of the users [10]. Rehabilitation robotics products for functionally impaired users are also failing in a similar vein [2].
2.1 Special considerations for empirical work with motion-impaired users It is important to note that practical limitations, principally involving the variable availability of participants and the small sample set, restrict the usefulness of detailed statistical analysis. The research reported here draws on the results of five years of work by the authors, typically with weekly experimental sessions.
Consequently, it is essential to fully understand the needs and capabilities of the users when designing products and interfaces. The remainder of this paper discusses three approaches to improving computer input for motion-impaired users. Underlying all three is a focus on the needs and capabilities of the end users. Experiments relating to each of the three approaches and to the development of input systems are discussed. The studies are in the areas of multimodal input, haptic feedback, user modelling, and cursor control. The experimental findings then form the basis for the design of a haptically enhanced keyboard emulator with perceptive capability. The keyboard emulator is proposed as a platform for combining more than one approach in a way that improves computer access for motion impaired users.
It cannot be emphasised enough how the difficult circumstances surrounding this form of experimental work can influence the outcome. The users can experience changes in their capability over time and may require medical treatment during an experimental series. They can be easily fatigued, despite extremely high motivation, and sometimes cannot complete trials or conditions. It is necessary for experimenters to run the trials on a long-term basis, to develop a working relationship with the users and to keep experimental conditions constant. Because of the small number of available users, repeated measures designs were generally employed. Obviously, these practical difficulties can give rise to missing data problems resulting from incomplete conditions, cause the loss of levels and factors from experimental designs, and make the systematic varying of conditions difficult.
2. IMPROVING ACCESSIBILITY FOR MOTION-IMPAIRED USERS Work in improving computer accessibility can address a wide range of issues, from reducing the cognitive demands of software applications to finding alternatives to visual interfaces. This paper shall focus on improving accessibility for users with motionimpairments. Although it is possible that this may be achieved by finding alternatives to existing paradigms such as menus and toolbars, the objective here is to improve computer interaction with standard graphical user interfaces (GUIs). There is an implicit understanding that complementary changes to the software interface will also be required.
In addition, the increased range and skewed variability of data resulting from the range of motor deficits leads to increased noise and violation of the assumptions of statistical tests. Consequently, the experimental evidence is presented as primarily qualitative.
3. MULTIMODAL GESTURAL INPUT FOR MOTION-IMPAIRED USERS Human-computer interaction has traditionally been limited to input via a mouse and/or keyboard and visual feedback via a monitor. The first approach to improving the effectiveness of computer interaction involves increasing the number of interaction channels beyond the mouse, keyboard, and monitor. An experiment was conducted to investigate the interactions involved in a multimodal system, and to assess the potential benefits of this type of approach to motion-impaired users. In this study, a prototype system was used, involving two gestural input channels, the head and the hand, and an in-house gesture recognition system, Jester [21]. A brief summary of a long-term study is presented here. See [15] for a detailed analysis.
Three approaches to improving the effectiveness of interaction with standard GUIs are proposed: 1.
Increasing the number of interaction channels. Additional channels may be used for input or feedback (e.g. multimodal input systems).
2.
Enhancing commonly existing interaction channels. Channels may be enriched by enabling them to transmit more information, either to or from the user (e.g. haptic/force feedback).
3.
Making more effective use of all the available information in a commonly existing input channel (e.g. perceptive user interfaces).
3.1 Hypothesised benefits of gestural input Gestural input systems are gaining popularity in the UA research community owing to their inherent flexibility and ability to be tailored to particular users [14]. The theoretical justification for using gestural input with motion-impaired users is that the freeform nature of gestures should allow users to select movements that are comfortable and repeatable to make [3]. Gestures are also considered to be natural and intuitive as they constitute a component of everyday interpersonal communication
In these cases, the improvement sought is not only quantitative in terms of the data transfer rate, but also qualitative in terms of user acceptance. In some cases, a dramatic quantitative improvement may mean that an input method, previously unacceptable to the user, becomes a viable option.
2
[14]. There is also evidence to suggest that increasing the degreesof-freedom of input devices can improve interaction rates [27]. Extending this principle to multimodal input implies that this should also yield improved information transfer rates.
100
Single Mode : Head
90
Single Mode : Hand
80
Duplicated : Head
70 Score
3.2 Investigating multimodal gestural input
20 10 0 1st
Condition Athetoid Cerebral Palsy Tetraplegia (from head injury) Muscular Dystrophy Spastic Quadriplegia Cerebral Palsy Athetoid Cerebral Palsy Friedrich’s Ataxia Athetoid Cerebral Palsy
2nd
3rd
4th Session
5th
6th
Figure 1. Multimodal input scores averaged across users.
For each of the Single modes and for (Both Modes) Duplicated, the alphabet size was 6 gestures, whilst for Different it was 36 gestures (6*6). For the two combined modes, an optimised score derived from a set of recognition heuristics was used in the calculation. If the two input modes agreed, that was the gesture recognised. If either input was not recognised, then the other mode result was used. Finally, if the two modes disagreed, if either mode recorded an oscillatory gesture, that would be taken as the gesture result. Otherwise, the final result was classified as not recognised.
A Polhemus was used for head gesture input and an analogue joystick, similar in nature to a wheelchair joystick, was used for hand gesture input. The gesture alphabet was chosen to be easy to learn. It consisted of 4 directional gestures, UP, DOWN, LEFT and RIGHT, and 2 oscillatory gestures, YES (UP-DOWN-UP) and NO (LEFT-RIGHT-LEFT).
The Single mode head and hand inputs achieved peak rates of 0.72 and 0.77 bits/s respectively. This compares to the 0.65 bits/s obtained for Duplicated and 0.56 bits/s for Different, both calculated using the optimised score. Thus the highest data transfer rates were achieved with the cognitively easiest task, despite having the smallest alphabet size. Even this rate, however, does not compare well to a simple binary switch input, which can conservatively achieve rates of 1 bit/s. The low data transfer rate for gestural input is most likely explained by the fact that the physical motions required for the gestures, though not particularly complex for an able-bodied individual, are difficult and slow for a motion-impaired user.
The experiment combined the head and hand inputs into a multimodal input system. Tests were completed with each mode individually (Single) and combined with both inputs being either identical gestures (Duplicated) or different ones (Different). A software application prompted the users visually to make gesture(s) with the head and/or the hand. The tests were carried out in the order Single, Duplicated and Different to coincide with increasing cognitive load. For Duplicated and Different, users were encouraged to produce both gestures simultaneously. However, this proved too difficult for Different and all users produced sequential gestures instead, typically the head gesture first, followed by the hand gesture.
3.3 Conclusions of the multimodal study
A scoring system was implemented whereby correct recognition earned +1 point, non-recognition scored 0 points, and an incorrectly recognised gesture received a -1 score. These scores were then scaled to a maximum score of 100 to eliminate dependence on the number of gestures recorded per test. This gave a complete potential score range of -100 to +100. By definition, any score below 0 would be unacceptable, because the user would spend all the time trying to correct errors, and could never produce any useful input beyond the simplest of tasks. Figure 1 shows the scores obtained, averaged across all the users.
Gestures as a means of computer input other than keyboard and mouse, while technically feasible, do not necessarily make a good input system either as a single input mode or as the two channels of a multimodal system. In addition, the use of two gestural input modes yielded poorer data transfer rates than the use of a single mode of gestural input alone. Thus not only was the addition of the other mode of input non-beneficial, but it had a detrimental effect on the interaction. The reason for this was identified as being an increase in cognitive load, rather than an increase in physical load [15].
However, analysing the scores does not fully describe the data transfer rates achieved as no account is made of the relative alphabet sizes involved. To rectify this, the bit rates of useful information transfer were calculated with the following equation from information science theory [24]:
Rate =
Different - Hand
30
Table 1. Participants in the multimodal input trials.
log 2 (Vocabulary_ size ) *
Different - Head
50 40
A prototype multimodal gestural input system was evaluated in user trials at the Papworth Trust, a charitable organisation dedicated to the care of people with a range of impairments. The volunteers in the study are described in Table 1.
User PJ1 PJ3 PJ4 PJ5 PJ6 PJ7 PJ8
Duplicated : Hand
60
3.4 Implications for input systems Alternative modes of input that increase either the cognitive or physical loads on the users are unlikely to be successful, unless they are carefully designed with substantial user feedback and thoroughly tested by the end users throughout development. Multimodal systems that combine several modes of input must minimise the overall load placed on the users if any benefit is to be gained from the extra modalities.
Score 100
Time _ taken
3
The task was a Fitts’ Law task, involving multidirectional pointselect interactions on a computer screen using the Wingman mouse. The participants were presented with 16 “target” circles arranged in a larger circle around a central “home” circle. The aim was for the computer users to select each target circle in a random order determined by the software. A target circle became filled in red to indicate that it was active. After successfully selecting the active target, the participants had to select the home circle before the next target was activated.
4. HAPTIC FEEDBACK FOR MOTIONIMPAIRED USERS The second approach to improving computer input involves enhancing commonly existing interaction channels. This section discusses enriching interactions with a mouse. Traditionally, the mouse has been used strictly for input, with information flowing only in one direction – from the user to the computer. Feedback to the user through this channel, in the past, has been restricted to the physical interaction with the specific input device, such as feeling the mouse or sensing its location in space, but haptic feedback has been under-utilised. With the advent of low-cost force feedback technologies the transfer of haptic information from the computer to the user has become a potential method of improving computer interaction. This section considers haptic feedback via a force feedback mouse to improve interaction for motion impaired users.
This task was performed in two separate studies. In the first study, the task was performed under two conditions – None and Gravity Well. In None, no force-feedback is applied and the Wingman operates as a normal mouse. In Gravity Well, the desired target sits at the centre of a circular gravity well that has a radius four times that of the target. When the cursor is outside the well, no forces are felt. When the cursor enters the gravity well, a spring force pulls the mouse toward the centre of the target. For this study, target selection was achieved with a mouse click on the Wingman.
4.1 Hypothesised benefits of haptic feedback Motion-impaired users often exhibit decreased motor control and muscle strength, but not necessarily decreased sensitivity of touch. Consequently, if haptic feedback can be successfully incorporated into the interaction paradigm, then these users may benefit from cutaneous and kinesthetic sensations. There are two ways in which the use of haptic feedback may enhance the usability of interfaces for motion-impaired users. First, it is possible to use force-feedback to present either constant or time-varying forces to the user. These forces have the capability to boost or aid user input, in the case of muscle weakness or poor co-ordination, and to damp or restrain user input, in the case of muscle spasm or tremor. Second, it is also possible to enrich the standard user interface with haptic textures, bumps and edges in order to signal the location of windows, buttons and regions as the mouse passes over them.
In the second study, the task was performed under 4 damping conditions – None, Acceleration Damping, Velocity Damping, and Combined Damping. Non-directional damping was implemented as a viscous force that increases linearly with mouse acceleration (Acceleration Damping), mouse velocity (Velocity Damping), and a combination of the two (Combined Damping). In this study, selection was achieved by dwelling inside the target for 250ms.
4.4 Force feedback results The use of a gravity well around a target appears to give a considerable improvement in times to target (Figure 2). The times averaged over all users was reduced by over 40% (3.9s compared with 2.3s). This improvement is even greater than that previously reported in [16], most likely due to a modification of the gravity well implementation. The most dramatic improvements were seen for the more severely impaired participants. This can be attributed to the fact that for PI3 and PI7, the occurrence of spasm and tremor often made it difficult to hold the cursor over the target while clicking the mouse button, and a large proportion of the time to target could often be spent trying to click inside the target. The addition of the gravity well helped to prevent the cursor from leaving the target region, making this part of the task easier and faster.
4.2 Force feedback technology Much of the research in haptic feedback for users with disabilities has involved a Phantom (e.g. [8]). However, this is an expensive research tool that is unlikely to be used routinely as a generalpurpose interaction device. The Logitech Wingman forcefeedback mouse, on the other hand, is an affordable input device capable of generating both vibratory and force-feedback effects with a wide range of movement generation capabilities. Studies have shown that this device can improve interaction for ablebodied users in cursor control tasks [6].
4.3 Investigating force-feedback
PI3 PI5 PI6 PI7
Athetoid Cerebral Palsy, spasm, wheelchair user Athetoid Cerebral Palsy, deaf, non-speaking Athetoid Cerebral Palsy Friedrich’s Ataxia, tremor, wheelchair user
8
Gravity Well
Times to target (s)
52%
6 5
48%
4 3 2
Table 2. Participants in the haptic feedback trials. Condition
None
7
Previous studies [12, 16] reported that force-feedback has the potential to provide a significant benefit to motion-impaired users performing on-screen point and click tasks. Continuing this work, experiments were designed as pilot studies to examine the benefits of two types of force-feedback: gravity wells and damping. Participants from the Papworth Trust involved in the study are described in Table 2.
User
9
1 0 PI3
PI5
PI6
PI7
Figure 2. The effect of the gravity well on times to target. Figure 3 shows the average time to target for the various damping conditions (6 sequences per damping condition per participant, 16
4
targets per sequence). It appears that all forms of damping have a favourable effect for PI3, reducing times by over 50%. The damping does not appear to have a great effect on times for the other participants. The noticeable effect for PI3 might be explained by the fact that spasms often result in high-speed and high-acceleration movements. Since damping provides resistance to such movements, the damping is able to reduce the effects of the spasms experienced by PI3, consequently improving overall times to target. For the other users, the nature of their impairments might not necessarily result in high-speed nor high-acceleration movements, and therefore they may not benefit so explicitly from the damping. Alternatively, any benefit may be balanced by the extra motor load required to move the mouse. This requires further study.
haptic interfaces that will assist motion-impaired users in interaction tasks is great. However, results reported in the literature [12, 20] indicate that the way in which the haptic feedback is integrated requires careful consideration. In addition, the results of this experiment indicate that a particular form of haptic assistance will not necessarily have the same effect over all users. The degree of benefit provided by a certain form of haptic assistance will depend in part on the nature of the impairment.
5. PERCEPTUAL USER INTERFACES AND MOTION-IMPAIRED USERS The third approach to improving computer interaction involves making more effective use of all the available information in a commonly existing input channel. One such method is to add perception to the interface.
PI3
7
PI7
Time to target (s)
6
5.1 Hypothesised benefits of perception
PI5
The capabilities of people with motion impairments can vary greatly, depending on many factors including the nature and severity of the impairment. The result in terms of computer interaction is that the tasks that are difficult for one person may differ from the tasks that are difficult for somebody with a different type or severity of impairment. Also, as shown by the previous experiment, a form of assistance that is effective for one person may have no benefit for another. With the addition of perceptive capabilities, an interface may be able to react to the individual needs of a user and provide the most appropriate form of assistance. In this way, a perceptive user interface can be an unobtrusive method of making computer interfaces more accessible to a wider range of users.
PI6
5
Overall 4 3 2 1 0 None
Acc Vel Combined Damping Damping Damping Figure 3. The effect of damping on times to target.
5.2 Adding perception to the user interface In the context of this third approach to improving computer input effectiveness, a perceptive user interface is one that can make more effective use of the data provided by existing input devices. Rather than responding in the usual way to events such as mouse movement, mouse clicks, and key presses, the interface should interpret the input data, “perceive” characteristic behaviours of a user, and respond in a way that is most appropriate to the user’s needs. For example, interfaces often initiate an action only when a mouse button is clicked while the cursor is positioned inside an icon. Frequent mouse clicks in the vicinity of a particular icon could be perceived by the interface to be indicative of a user’s difficulty with accurate positioning of the mouse. The interface could respond, for example, by enlarging the icon.
4.5 Conclusions of the haptic feedback study Haptic force-feedback in the form of a gravity well around a target gave improved times to target for all motion-impaired users in the study. Of particular encouragement was that the force feedback was of greatest benefit to the more severely impaired users, especially those with difficulty performing homing actions. Damping improved performance for the one user exhibiting spasm, but had little effect for the other users. The availability of comparatively inexpensive force feedback devices, such as the Logitech mouse, and programmer development kits for adding force feedback effects to software means that this has the potential to become extremely important technology for enabling universal access to computers. However, while force feedback can be very effective in improving interaction, it needs to be applied in a manner that is truly supportive and complementary to the existing input. The poor results of certain types of force-feedback from other experiments [12, 20] show that inappropriate use of the technology can actually have a negative impact on the interaction. As ever, it is vital to verify and validate the use of force-feedback through user trials before incorporating it into an interface.
To facilitate perception, realistic models of user behaviour in computer interaction tasks are required. The next two sections describe work in modelling the behaviour of motion-impaired users in performing two tasks – switch activation and cursor control. These two tasks are of interest since common interface tasks such as clicking a mouse button or striking a keyboard key are fundamentally switch activation activities and graphical user interfaces are most efficiently used with some method of cursor control.
4.6 Implications for input systems
5.3 Modelling switch activation
Given the availability of affordable force-feedback devices, and given the positive results of initial studies on adding forcefeedback to user interfaces, the potential for the development of
One of the most straightforward user models is the Model Human Processor (MHP) [4]. This is a very simple model that segments the interaction process into three broad function types: the time to
5
perceive an event; the time to process the information and decide upon a course of responsive action; and, finally, the time to perform the appropriate response. Consequently, total response times to stimuli can be described by the following equation:
the colour before activating the button, theoretically requiring a task time of τp + 2τc + 2τm. In the third task, a letter was presented above the OK button. Users were required to recognise the letter before activating the button, theoretically requiring a task time of τp + 3τc + 2τm. The cognitive cycle time was obtained by finding the time differences between completion of the three tasks.
Total time = xτp + yτc + zτm where x, y and z are integers and τp, τc and τm correspond to the times for single occurrences of the perceptual, cognitive and motor functions. Card et al predicted that these times are approximately 100ms, 70ms and 70ms respectively for ablebodied users [4]. On empirical and neurological grounds, the model assumes that it is only possible for each constituent of the cycle to occur in integer multiples of the base time. In other words, it is impossible to have half a cognitive cycle or a third of a motor response. Although this is a simple model, it was selected because it is very easy to understand and to observe deviations from the predicted behaviour. It is well accepted that this model has empirical validity for the analysis of basic interactions such as point and click reaction time tasks [11].
Table 4 shows the perceptual and cognitive cycle times estimated from user data, rounded to the nearest 10ms (cf. 100ms and 70ms from Card et al) [4]. Table 4. Perceptual and cognitive response times.
5.3.1 Calibrating the MHP Trials involving motion impaired users were performed to calibrate the MHP for key-pressing activities (see also Keates et al [13]). The aim of this work was to obtain perceptual, cognitive, and motor cycle times for the motion-impaired users. The tasks were computer-based analogies to Card et al’s original experimental methodology [4]. The participants in this study are described in Table 3.
τp (ms)
τc (ms)
PJ3 PJ4 PJ5 PJ6 PJ7 PJ8 Mean
110 100 100 100 90 80 100
100 120 100 120 110 130 110
The motor function time, τm , was obtained by the measurement of simple, repetitive movements. A large OK button was displayed successively ten times in the same location on the screen and the users asked to clear it as quickly as possible. The time was recorded when the button was cleared and was, according to the theory, equivalent to 2τm . The times obtained for the motor function are shown in Table 5.
Table 3. The Model Human Processor trial users. User PJ3 PJ4 PJ5 PJ6 PJ7 PJ8
User
Table 5. Motor function times: motion-impaired users.
Condition Tetraplegia (from head injury) Muscular Dystrophy Spastic Quadriplegia Cerebral Palsy Athetoid Cerebral Palsy Friedrich’s Ataxia Athetoid Cerebral Palsy
User PJ4 PJ6 PJ3 PJ7 PJ5 PJ8
Firstly, the perceptual cycle time was evaluated using two tasks, both involving a black circle moving in a fixed-location circular pattern on a computer screen. In the first task, the user observed the motion of the black circle whilst time delays were inserted into the motion at random points. The user had to detect the locations of the delays for given delay durations. An initial delay duration of 500ms was incrementally decreased until the user was unable to determine the location of the delay. That delay was recorded as the perceptual response time, τp. In the second task, the black circle was moved in the same circular pattern in discrete segments of motion. The duration of each movement segment varied between 10ms and 150ms. The users were asked to state whether the motion was smooth (continuous), jerky (discrete), or borderline. The perceptual response times were recorded as the delay in the borderline cases. The final perceptual response time was an average of values obtained from the two tasks.
τm (ms) 120 100 220 200 300 300
Mean = 110ms Mean = 210ms Mean = 300ms
Finally, the reaction times were measured. The mean times observed for the reaction task are shown in Table 6. In the table, predicted time for motion impaired users was derived using τm = 110ms as this value was closest to the theoretical predicted value. It can be seen that there is an unexpected delay of approximately 200ms in the motion-impaired reaction time that appears to contradict the predicted MHP time of τp+τc +2τm. Table 6. Reaction times to a simple stimulus.
Cognitive cycle time was evaluated through three tasks. In the first task, users had to clear a large OK button as soon as it appeared on the screen, theoretically requiring a task time of τp + τc + 2τm . In the second task, the OK button was accompanied by either a large green triangle or blue square. Users had to recognise
Reaction to simple stimulus (ms)
Able-bodied: theory
Predicted Observed
310 -
Motionimpaired: recorded 430 620
5.3.2 Conclusions from the MHP It can be seen from Table 4 that the motion-impaired users’ perceptual response times compare well with the 100ms from Card et al [4]. The cognitive response times from the same table are noticeably slower than those for able-bodied users (110ms cf.
6
5.4.1 Cursor Speed
70ms). Further work by the authors has shown that extra cognitive effort may be required to control a motion influenced by an impairment [13]. In addition, the insertion of extra cognitive cycles causes the banding of motor response times seen in Table 5 as well as the 200ms discrepancy between the observed and predicted reaction times for motion-impaired users in Table 6. Effectively, motion-impaired users often insert extra cognitive cycles into actions that the theory predicts should be automatic and not require them [13]. However, as can be seen from Figure 4, which shows the spread of times for one of the users pressing and releasing a key, this offset is not constant.
The instantaneous cursor speed along a path is a measure that can be particularly sensitive to detecting the occurrence of spasm. In a study of cursor data from the task described in Section 4.3, high cursor speeds were observed for PI3 during some large spasms, with speeds ranging from 3 pixels/ms to 8 pixels/ms. Speeds observed in able-bodied individuals and motion-impaired users without spasm were typically less than 2 pixels/ms. Using a user’s speed profile, it may be possible for an interface to perceive and compensate for undesired movements such as spasm.
5.4.2 Radius of Curvature The radius of curvature along a path can provide information about the number and proximity of “twists and turns” in the path. Figure 5(a) shows a representative cursor path for user PI7 using the Wingman mouse without any force-feedback assistance. The path is characterised by a high density of tight “turns” about the target. This behaviour is reflected in Figure 5(b) which shows the reciprocal of radius of curvature (approximated for an arc length of 10 pixels) plotted against displacement from the start point. There is a high occurrence of high values (i.e. small radii) near the target. This behaviour can be contrasted with that of an ablebodied user, where the cursor path is relatively straight and the reciprocal of radius of curvature is consistently near zero. Using this information, it may be possible for the interface to perceive when the user has approached a desired target.
25%
Buttondown
20%
Buttonup
15% 10% 5%
450-475
400-425
350-375
300-325
250-275
200-225
150-175
100-125
50-75
0-25
0%
Time (ms)
5.4.3 Conclusions of cursor control modelling
Figure 4. Motor function times for PJ7.
The study of cursor trajectories throughout a task can reveal differences in the cursor behaviour of users. Certain measures such as the cursor speed along the cursor path and the radius of curvature of the cursor path are particularly suitable for capturing characteristics of cursor movement by motion-impaired users.
5.3.3 Implications for input systems The potential for motion-impaired users to consistently insert extra cognitive cycles into actions that theory predicts should be automatic has implications for input system design. An interface that is designed to be aware of the potential for unintentional extra delays should be able to better accommodate the user. Take as a simple example the pressing of a keyboard key. The pressing and releasing of a key to type a single character is generally calibrated for able-bodied people. For a motion-impaired user who may depress a key for a longer period, the result is the typing of several repeated characters, despite the intention to type a single character. An interface that is capable of perceiving the typical response time of a particular user can self-adjust to better accommodate the user, without requiring any explicit change in behaviour on the part of the user. Furthermore, input systems should aim to minimise both the cognitive and physical loads on the users to prevent extra delays from being inserted into the interaction cycle.
(a) Target circle
Home circle
0.4 -1
1/radius of curvature (pixels )
Frequency of occurence (%)
30%
5.4 Modelling Cursor Control
(b)
Interaction with graphical user interfaces typically involves the use of a method of cursor control. This can prove to be a difficult task for a motion impaired user, but also one that has the potential to be assisted by a perceptive user interface. To this end, an understanding of the user’s cursor behaviour during an interaction is important. A study of cursor trajectories throughout a target selection task can elicit subtle differences in the behaviour of users [18]. Our analysis of the cursor movement of motionimpaired users in point-and-click tasks have suggested two characteristics which can be useful in the development of a perceptive (haptic) user interface – the cursor speed along the cursor path and the radius of curvature of the cursor path.
0.3 0.2 0.1 0 -0.1 -0.2 -0.3 -0.4 0
100
200
300
400
Displacement from start (pixels)
Figure 5. (a) A representative cursor path for PI7 and (b) its plot of reciprocal of radius of curvature against displacement from the start point.
7
5.4.4 Implications for input systems
more responsive to user needs. For example, keyboard emulators operate on the fundamental principle of key selection, a task which requires the user to activate a switch or dwell over a key for a certain period of time. Given that motion-impaired users often insert extra cognitive cycles into “automatic” actions, perception can be used to determine the optimum scan interval or dwell time for an individual. In this way, it may be possible to minimise error rates, maximise interaction rates, and improve productivity levels.
Through cursor control analysis, it may be possible both to detect some types of undesired motion as well as to perceive a user’s intent. With this information, a perceptive user interface may be able to prevent or minimise the impact of the undesired motion, or to provide assistance that will aid the user in carrying out the intended task.
6. TOWARDS A HAPTICALLY ENHANCED KEYBOARD EMULATOR
It is also anticipated that perception will complement haptic techniques by addressing some of their limitations. Two possibilities are described next – perception to improve damping techniques, and perception to improve gravity well techniques.
Three approaches to improving the effectiveness of interaction with standard GUIs have been discussed. In developing an input system, it is possible to combine more than one approach. In this section, a haptically enhanced keyboard emulator is proposed as a platform to investigate a combination of the last two approaches. In particular, the keyboard emulator incorporates haptic feedback (approach 2) and perception (approach 3).
Although damping can be effective in improving interaction for users frequently exhibiting high-speed movements such as spasm, strong damping forces in the absence of spasm has the potential to impede intentional movement. Based on a user’s cursor speed profile, a perceptive user interface could tailor the damping profile to best suit the user. Damping parameters such as damping strength or deadband threshold can be adjusted to create a damping profile that is optimised for a particular user.
6.1 Keyboard emulators Keyboard emulators can be an important means of computer access for people whose physical limitations prevent them from accessing a keyboard. A keyboard emulator displays a representation of a keyboard on the screen and contains keys that, when selected, enter keystrokes into a target application. Keyboard emulators are available in a variety of layouts and may incorporate options such as word prediction. They all operate on the same fundamental principle of key selection.
Gravity wells around targets can be very effective in helping motion-impaired users with point-and-click tasks. This would imply that for a keyboard emulator, putting a gravity well around keyboard keys would improve typing rates. In the experiments, however, gravity wells were activated with prior knowledge of the location of the desired target. In a real application, it is not possible to know in advance which target a user wishes to select. Activating gravity wells around all possible targets could actively hinder a user in the completion of a task [19], and is therefore not a practical option, particularly for a cluttered environment.
Key selection can be accomplished in two ways. In the first, the user moves a cursor directly over the key, then “presses” the key by activating a switch or by dwelling over the key for a certain period of time. The switch can be a mouse button, a single keyboard key, or a special access switch. This method requires the user to be able to use a pointing device effectively. The second method of key selection is known as switch-based scanning. Scanning does not require control of a pointing device. Instead, successive keyboard items (or groups of items) are highlighted. The user selects desired items by activating a switch when that item is highlighted. Four general techniques of scanning are described by Lee and Thomas [17], all requiring a switch action to be made in a given period of time.
A better solution would be a perceptive user interface which is able to predict which target the user desires next. The interface could then activate a gravity well around the appropriate target. According to the model of cursor control, the spatial density of the occurrence of small radii in the cursor trajectory increases when the cursor is near the target. By monitoring this characteristic, the perceptive user interface could predict the location of the desired target. Using this method in conjunction with a language model such as that used in [23], the perceptive keyboard emulator should be able to activate a haptic gravity well in an appropriate location.
6.2 Combining perception and haptics
Prediction methods, however, will not be perfect and some method of error handling should be considered. It would be undesirable to activate a gravity well that is holding a cursor “captive” over the wrong target. Some possible methods of error handling will be investigated. For example, it may be possible to deactivate the gravity well if no selection has occurred after a period of time. The duration of the waiting period will be tailored to the user’s behaviour. Alternatively, it may be possible to implement a manual switch that, when pressed, turns off the haptic effect.
Gravity wells and non-directional damping can be beneficial for motion-impaired users during target selection tasks. However, there remain many issues to consider when designing haptic feedback for a keyboard emulator. Further research is required to find out, for example, whether haptic effects should be activated just around keyboard features (e.g. keyboard keys) or whether the entire surface of the keyboard emulator should have a continuous force-field map. It is necessary to investigate a variety of force profiles and layouts, and to find out whether a constant or timevarying profile is more appropriate. Furthermore, the haptic feedback from the keyboard emulator must complement the visual feedback, and several options for keyboard layout, including key shapes and sizes, should be studied. These research questions will be addressed.
The effectiveness of the keyboard emulator will be determined in extensive trials involving computer users with a range of motion impairments. In light of the importance of including the end users of any product early in the design process, there will be on-going evaluation by motion-impaired users throughout the development of the keyboard emulator.
Some issues cannot be addressed effectively by haptics alone, and the addition of complementary perceptive capabilities will be investigated as a means of allowing the keyboard emulator to be
8
[12] Keates S, Langdon P, Clarkson PJ, Robinson P (2000) Investigating the use of force feedback for motion-impaired users. In Proceedings of the 6th ERCIM Workshop (Florence, Italy), pp. 207-212.
7. CONCLUSION This paper has discussed three approaches to improving the effectiveness of interaction with standard GUIs – increasing the number of interaction channels, enhancing commonly existing interaction channels, and making more effective use of all the available information in an existing input channel. Results of experiments investigating each approach have been described and their implications on input methods discussed. In particular, the areas of multimodal interaction, haptic feedback, and user modelling have been studied.
[13] Keates S, Clarkson PJ, Robinson P (2000) Investigating the applicability of user models for motion-impaired users. In Proc. of ACM ASSETS 2000 (Arlington, VA), pp. 129-136. [14] Keates, S. and Robinson, P. (1999) Gestures and multimodal input. Behaviour and Info. Tech., 18(1), pp. 36-44. [15] Keates, S., Clarkson, P.J., and Robinson, P. (1998) Developing a methodology for the design of accessible interfaces. In Proceedings of the 4th ERCIM Workshop, (Stockholm, Sweden), pp. 1-15.
Encouraging results have suggested that a combined approach may be an effective way to design a novel computer input system. A haptically enhanced keyboard emulator with perception has been proposed as a platform for testing a combined approach. A key lesson learned from the experimental work is that any novel computer input system must be carefully implemented and thoroughly evaluated by the end users during development. This way, the resulting product can aim to minimise the cognitive and physical load placed on the user and thereby improve its usability.
[16] Langdon, P., Keates, S., Clarkson, J, and Robinson, P. (2001) Investigating the Cursor Movement Parameters for Haptic Assistance of Motion-Impaired Users. In Proceedings of ICORR 2001 (Paris, France), pp. 237-242. [17] Lee, K. and Thomas, D. (1990) Control of computer– based technology for people with physical disabilities: An assessment manual. Toronto, Canada: U of T Press.
ACKNOWLEDGEMENTS The authors would like to thank the volunteers and staff of the Papworth Trust. This research is funded in part by the EPSRC, the Canadian Cambridge Trust, and NSERC.
[18] MacKenzie, I.S., Kauppinen, T., and Silfverberg, M. (2001) Accuracy Measures for Evaluating Computer Pointing Devices. In Proc. of CHI 2001 (Seattle, WA), pp. 9-16.
REFERENCES [1] Americans with Disabilities Act of 1990 (1990) US Public Law 101-336.
[19] Munch, S. and Dillmann, R. (1997) Haptic Output in Multimodal User Interfaces. In Proceedings of IUI ’97 (Orlando, FL), pp. 105-112.
[2] Buhler, C. (1998) Robotics for rehabilitation - a European(?) perspective, Robotica 16(5), pp. 487-490.
[20] Oakley, I., McGee, M.R., Brewster, S., and Gray, P. (2000) Putting the feel in ‘Look and Feel’. In: Proceedings of CHI 2000 (The Hague, Netherlands), pp. 415-422.
[3] Cairns, A.Y., Newell, A.F. (1994) Towards gesture recognition for the physically impaired. In Proceedings of RESNA ’94, (Nashville, TN), pp. 414-416.
[21] Perricos, C. Jackson R.D. (1994) A head gesture recognition system for computer access. In Proceedings RESNA 94 (Nashville, TN), pp. 92-94.
[4] Card, S.K., Moran, T.P., Newell, A. (1983) The Psychology of Human-Computer Interaction. Lawrence Erlbaum, Hillsdale, NJ. [5] Coleman, R. (1993) A Demographic Overview of the Ageing of First World Populations. Appl. Erg., 24(1), pp. 5-8.
[22] Stephanidis, C. (2001) User interfaces for all: New perspectives into human-computer interaction. In User Interfaces for All (C. Stephanidis ed.), Lawrence Erlbaum, pp. 3-17.
[6] Dennerlein, J.T., Martin, D.B., and Hasser, C. (2000) Forcefeedback improves performance for steering and combined steering-targeting tasks. In: Proceedings of CHI 2000 (The Hague, Netherlands), ACM Press, pp. 423-429.
[23] Ward, D.J., Blackwell, A.F., and MacKay, D.J.C. (2000) Dasher – a Data Entry Interface Using Continuous Gestures and Language Models. In Proceedings of UIST 2000 (San Diego, CA), pp. 129-137.
[7] Dept. for Education and Employment (DfEE) (1996) The Disability Discrimination Act. UK.
[24] Welsh, D. (1998) Codes and Cryptography, Oxford University Press, Oxford.
[8] Dubey, R.V., Pernalete, N., Zekri, S., Gage, A., Fritz, B. (2001) Development of an Intelligent Robotic Haptic Interface to Perform Vocational Tasks by Individuals with Disabilities. Proc. ICORR 2001 (Paris, France), pp. 262-268.
[25] Worden, A., Walker, N., Bharat, K., and Hudson, S. (1997) Making Computers Easier for Older Adults to Use: Area Cursors and Sticky Icons. In Proceedings of CHI ’97 (Atlanta, GA), ACM Press, pp. 266-271.
[9] Evans, D.G., Drew, R., and Blenkhorn, P. (2000) Controlling Mouse Pointer Position Using an Infrared Head-Operated Joystick. IEEE Trans. on Rehab. Engr., 8(1), pp. 107-117.
[26] Workforce Investment Act of 1998 (1998) US Public Law 105-220, USA.
[10] Gardner, L., Powell, L., Page, M. (1993) An appraisal of a selection of products currently available to older consumers, Applied Ergonomics. 24(1), pp. 35-39.
[27] Zhai, S., Milgram, P., Buxton, W. (1996) The influence of muscle groups on performance of multiple-degree-offreedom input. In Proceedings of CHI ’96 (Vancouver, Canada), pp. 308-315.
[11] John, B.E., Kieras, D.E. (1996) The GOMS family of user interface analysis techniques: Comparison and contrast. ACM Trans. Comp.-Human Int., 3(4), pp. 320 – 351.
9
