IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 2, MARCH 2013
177
Improving Mouse-Based Computer Interaction in Users With Weak Upper Limb Motion Control Using a Haptic Assistive System Nikos G. Tsagarakis and Darwin G. Caldwell
Abstract—Computer-assisted therapy is one of the most promising new techniques for those suffering from physical and neurological dysfunction. Yet, impairments to physical movement arising from a central nervous system dysfunction or from muscle spasms generated through other neurological damage or dysfunction can often make it difficult or impossible for individuals to interact with computer-generated environments using a conventional mouse interface. This paper investigates the use of a 2-D haptic device as an assistive robotic aid to minimize the effects of the pathological absence of motor control in the upper limb in impaired users when using a mouse. The assistive functionality is evaluated in 2-D tracking tasks using a human subject with failure of the gross coordination of the upper limb muscle movements—“Muscle Ataxia.” The results demonstrate that with this system the capability of the impaired subject to track predefined trajectories within a computer generated 2D is significantly improved. The average of the means of the error distance for the trajectories performed under the assistive mode was significantly lower (more than 40%) than that of the trajectories without assistance. In addition, when using the assistive device, the impaired subject was able to complete the tracking tasks in less time. Index Terms—Assistive, ataxia, haptic, impedance.
I. INTRODUCTION N recent years, as part of computer-assisted therapy, there has been a considerable body of work directed toward the development of rehabilitation and power/motion coordination systems based on assistive robotic devices [1], [2]. As a result, a wide and diverse range of interface systems have been developed, or used ranging from manipulandums to simple power orthoses and even exoskeletal systems [3]–[13]. These devices aim to assist in all areas of physical therapy, for instance, to recover from different injuries, to compensate for various disabilities, or to provide motion coordination/assistance and performance evaluation [14]–[24]. Some of these pathological conditions such as Parkinson’s disease, muscular dystrophy, muscle ataxia, and cerebral palsy have symptoms such as reduced strength, restricted or irregular (jerky) movements, poor mo-
I
Manuscript received September 11, 2011; revised February 2, 2012; accepted May 17, 2012. Date of current version February 12, 2013. This paper was recommended by Associate Editor M. Jeng of the former IEEE Transactions on Systems, Man and Cybernetics, Part C: Applications and Reviews (2011 Impact Factor: 2.009). The authors are with the Department of Advanced Robotics, Istituto Italiano di Tecnologia, 16163 Genova, Italy (e-mail:
[email protected];
[email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TSMCC.2012.2204872
tion coordination, and a continuum of impairments involving spasms and tremors. Often these physical impairments can make it difficult or impossible for sufferers to interact with computergenerated environments using conventional mouse type interfaces [25], [26] limiting their scope to take advantage of developments in computer technology for work, educational, entertainment, and social purposes. This has a significant impact on life and work opportunities. Assistive robotic devices may help to ameliorate these difficulties for this group. For interactions with a computer-generated environment, the efficacy of various human–machine interfaces such as force feedback mouse [27], [28] has been evaluated in GUI interaction tasks. Velocity-dependent force feedback has been evaluated in a number of other studies to damp erratic motions [29], [30]. It has been shown that increasing the viscous damping helps reduce the level of sudden motions but at the same time resistance to voluntary movement may occur. Following the same trend, this paper aims to improve the capability and efficiency of people with motion impairments arising from the pathological disorder muscle “Ataxia,” while interacting with computer-generated environments using a conventional mouse interface. Ataxia (from Greek ataxia, meaning failure to put in order) is unsteady and clumsy motion of the limbs or trunk due to a failure of the gross coordination of muscle movements. It is a relatively rare disorder with about 20 000 cases in Europe and North America (1/50 000). While the term ataxia is primarily used to describe this set of symptoms, it is sometimes also used to refer to a family of disorders. It is not, however, a specific diagnosis [31]. Symptoms usually appear between the ages of 5 and 15 and involve poor muscle tone and weak muscles, difficulty making rapid changes in muscle tension, and undershooting or overshooting of a target trajectory. The third symptom is not only due to the failure of the gross motion coordination and control, but is also a physical consequence of the first two symptoms. Weak muscle tone or inability to rapidly change the muscle tension prevents the subjects from perceiving the expected motion effect of their effort, leading to overshooting of the target trajectory. Subjects, consequently, apply greater effort to generate satisfactory motions which often results in overshooting trajectories. These symptoms can make the use of computer keyboard and mouse devices difficult or even impossible. A planar two-degree-of-freedom (DOF) haptic device and an assistive control scheme are proposed for improving 2-D cursor-based motions of ataxia-impaired individuals when interacting with a computer-generated environment. The reduction
2168-2291/$31.00 © 2013 IEEE
178
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 2, MARCH 2013
Fig. 1.
Planar pantograph haptic system integrated with a mouse. TABLE I SPECIFICATIONS OF THE PLANAR HAPTIC DEVICE
Fig. 2.
in the undershooting is achieved by assisting the execution of movement to compensate for the poor muscle tone. This allows the impaired subjects to generate motions with less effort which in consequence reduces the possibility of overshooting. The end-tip of the device has been attached to a traditional mouse interface to form the complete mouse motion assistive interface. The paper is organized as follows. Section II presents the assistive haptic device. Section III introduces the operational principles of the assistive control scheme, while Section IV presents the extension of the controller to the full system. Experimental results are shown in Section V, while Section VI addresses the conclusion and comments on further developments. II. DESCRIPTION OF THE ASSISTIVE HAPTIC SYSTEM A. System Configuration The custom assistive haptic device developed in this paper is shown in Fig. 1. It is a five-bar pantograph system with its dimensions (links) specified to permit the end-tip of the device to cover the field of motion usually found in mouse-based interactions. The device is powered by two precious metal-brushed 40-W DC-motors from Portescap (Model 25GT2R82-222E) through capstan cable transmissions which provide a 26:1 gearing ratio with minimum friction and excellent back drivability (backdrivable friction is below 0.2 N). Incremental encoders at the joints monitor the position changes of the device tip, while torque sensors measure the torque at each motor output shaft. The specifications of the haptic device are reported in Table I. The haptic system end-tip has been interfaced with a mouse to form the assistive interface for the GUI-based interactions. To monitor the exchange of forces between the assistive system
Kinematic geometry of the haptic device.
and user’s hand, a force sensor has been installed between the mouse and the haptic device. The cost of the prototype haptic device is 1300€ excluding the cost of the 6-DOF F/T sensor (5000€ ). This additional cost of the force torque sensor can be significantly reduced by replacing it with a two-axis load cell with a cost of approximately 500€ . Although a 2-DOF load cell is sufficient, the 6-DOF sensor was used in this study as it was already available in our laboratory. B. Device Kinematics The kinematic configuration of the assistive haptic device is presented in Fig. 2. The position vector of the device tip with respect to the base frame O(x0 , y0 , z0 ) is denoted by xt . We can obtain xt as a function of the active joint angles qi , i = 1,2 and the upper link angle qu a + L1 cos(q1 ) + L2 cos(qu ) (1) xt = L1 sin(q1 ) + L2 sin(qu ) where L1 , L2 are the lengths of links and a is the location, along the X-axis, of the proximal joints of the inner links with respect to the base frame. The upper link angle qu can be determined using the geometry of the five-bar mechanism qu = π − qd − qα . The angle qd can be computed from the triangle formed by the outer links and the line joining the distal joints of the inner links qd = arccos(d/2L2 ), while angle qα is given by qα = arctan(Δy/Δx). L2 , d, and Δx, Δy represent the length of the distal link, the length of the line joining the distal joints of the inner links, and the x, y distances between the distal joints of the inner links, respectively. Substituting qu into (1) and directly differentiating the final expression of the forward kinematics yields the device Jacobian which relates the handle tip velocity to the two active joint velocities ˙ x˙ t = jq.
(2)
The dynamic behavior of the haptic device is finally described by D(q)¨ q + C(q, q) ˙ + F(q) ˙ + JT Fd = τ
(3)
TSAGARAKIS AND CALDWELL: IMPROVING MOUSE-BASED COMPUTER INTERACTION IN USERS
179
force Fs as follows: F h = Fs + F d .
Fig. 3. User hand/device interaction diagram (top) and conceptual 1-DOF linear model of the proposed assistive/suppression control law (bottom).
where the newly introduced notations describe τ as the generalized joint torque vector, D(q) as the inertia matrix, C(q, q) ˙ as the coriolis/centripetal vector, F(q) ˙ as the friction vector, Fd as the force that the device generates at the end-tip, and JT as the transpose Jacobian of the haptic device. III. ASSISTIVE CONTROL SCHEME The haptic device and its control scheme aims to aid a subject with motion coordination difficulties to interact with GUIs using a mouse by assisting the movements in the trajectory regions where motion is weak. We introduce the assistive control for the haptic device by initially considering the 1-DOF linear mechanical system shown in Fig. 3. The extension of the concept to the full system will be discussed in the next section. In Fig. 3, mass Mh and damper Bh resemble the mass and damping properties of the human forearm and hand segments (mouse motions are primary generated by the movement of the elbow, wrist, and finger joints). The user mass/damper block is fixed to the device end-tip through a force sensor which monitors the forces exchanged between the user and the haptic device. The device itself is represented by the mass/damper network denoted by Md and Bd . The spring/damper arrangement indicated by Ka , Ba is a virtual spring/damper group which is attached to the device tip at its one end point and represents the assistive control network. The remaining notations introduced in Fig. 3 are as follows: Fh is the force applied by the user, Fd the force provided by the device, Fs the force measured by the sensor, Fa the force generated by the actuator A, Xh the current device endtip position, and Xe the assistive network (Ka , Ba ) grounding position which is dynamically updated according to the user effort and intended motion. A. Assistive Controller Description Let us now assume that the subject’s hand is applying a force Fh through the mouse, while the haptic device is exerting a force Fd . The difference between these two forces is measured as the force sensor Fs = Fh − Fd . To provide the assistive functionality, the amplitude and the direction of the assistive force need to be determined. It is reasonable to use the direction of Fh to determine the direction of the user’s intended motion. The force applied by the user Fh can then be computed from the measured
(4)
To generate an assistive force in the user’s intended direction of movement, the assistive control network, represented by the spring/damper group (Ka , Ba ) is used. One end of this virtual spring/damper system is rigidly linked to the device end-tip, while its second end is free to move; its position Xe is determined using the user’s effort and intended motion. In particular, the desired position Xe of the assistive network grounding end point is updated using the user force Fh . This can be written as Xe = Gf Fh
(5)
where Gf represents the transfer function between the force exerted by the user and the grounding end-point position. A possible transfer function Gf which can be used is the simple integrator Gf = Kf /s. The motivation behind the simple integrator is based on the idea that the accumulation of forces applied by the user can give a good indication/estimation of the user motion intention/direction, and therefore, it can be used to update Xe toward this direction. Other transfer function candidates include double integrators or other higher order controllers which could in addition permit shaping of the frequency response, e.g., preventing any tremor noise from entering the estimation of the intention of motion. As ataxia subjects do not suffer for this kind of tremor noise, a simple integrator was experimentally proved (see Section IV) to be adequate in this case. The perturbation of the free end of assistive spring/damper network will trigger the generation of forces attracting the device end-tip toward the position Xe of the free end-point. In other words, the device will generate a force Fd pulling toward Xe , which from the user’s point of view will be perceived as a force assisting his/her movement (the direction of the perturbation of Xe has been selected to coincide with the direction of the user’s applied force Fh ). The pulling force is determined by the properties of the virtual assistive spring/damper (Ka , Ba ) and is written as Fd = Ga (xe − xh ) with Ga = Ba s + Ka .
(6)
To exert the assistive force Fd at the device tip, we select an actuator control signal as follows: Fa = u = Fd + Gd xh = Ga (xe − xh ) + Gd xh
(7)
where the term Gd xh with Gd = Md s2 + Bd s is added to compensate for the device dynamics. This pulling force Fd together with the force applied by the user Fh drives the position of the pair (haptic device + user’s hand) Xh toward Xe . It can be written as xh =
Fh + Fd Gh
(8)
where Gh = Mh s2 + Bh s resembles the mass and damping properties of the human hand/finger segments. By injecting Fh , Fd from (5) and (6) into (8), the position Xh can be obtained as
180
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 2, MARCH 2013
a function of Xe xh =
(Xe /Gf ) + Ga (xe − xh ) (1/Gf ) + Ga Fh + Fd = = xe Gh Gh Gh + Ga
(9) Similarly, by substituting Xh and Xe from (5) and (8) into (6), the assistive force Fd developed by the device can be derived as a function of the human applied force Fh Fd = Ga (xe − xh ) =
Ga Gf Gh − Ga Fh Gh + Ga
(10)
Substituting Gh , Ga , and Gf into (9) and (10), xh and Fd can be obtained as a function of the device, the human arm, and the assistive network parameters. B. Stability Analysis and Assistive Gain Selection The characteristic equation of the transfer functions in (9) and (10) is Gh (s) + Ga (s) = Mh s2 + (Bh + Ba )s + Ka . This characteristic equation is subject to change due to variations in Gh (s). This is due to the variability in the mass and damping coefficients of the human arm Mh and Bh . If (Bh + Ba ) > 0 and Ka > 0, then Gh (s) is Hurwitz. We may write Bh + Ba Bh + Ba Ka = ωn2 and ζ = = √ Mh 2Mh ωn 2 Mh Ka
(11)
where ω n is the natural frequency for a stable, second-order, closed-loop system and ζ is the damping ratio. Through the control parameters of the assistive spring network (Ka , Ba ), we have independent regulation of the natural √ frequency and damping ratio. We choose a desired ζ = 2/2 such that the system is underdamped. Therefore from (11), we can obtain Bh + Ba ⇒ Ba = 2Mh Ka − Bh . (12) ζ= √ 2 Mh Ka The parameters of the assistive control law (Ka , Ba ) are selected by considering the desired system bandwidth and damping ratio. Since the system and controller are used as a mouse-based motion assistive device, it is sensible to select the (Ka , Ba ) gains such that the bandwidth of the system is greater than the maximum motion frequency during mouse-based interaction movements (1–2 Hz). For this simulation study, a bandwidth of ∼ 5 Hz(ωn = 30 rad/ sec) was considered for the 1-DOF system presented in Fig. 3. The assistive network parameters were determined from (11) and (12) as Ka = 10 N/cm and Ba = 30N · s/cm. The mass and damping properties of the human hand/finger were set as Mh = 1 Kg, Bh = 0. The level of damping at the wrist and finger joints during voluntary movements is greatly affected by many factors, the most important of which are the mean level of muscle activation and the angular velocity of the joint [30]. Therefore, the value of the joint damping is a highly uncertain parameter with its maximum level being of an order greater than the minimum. Various studies [32]–[35] report damping ratios for the forearm joints (elbow, wrist, and finger joints) in the range of 0.1–1.0. By setting Bh = 0, the worst case condition in terms of system stability (hand/finger represented by a pure mass) is applied.
Fig. 4.
Simulated responses of the 1-DOF assistive control system.
C. Assistive Controller Validation To test the assistive control scheme, simulation studies were carried out on the 1-DOF system shown in Fig. 3. The gain parameters used were those reported in the previous section. Here, a step like force input lasting for 0.2 s as shown in Fig. 4 (top graph) was used to trigger the 1-DOF system. The next graph in Fig. 4 depicts the resultant perturbation of the grounded end of the assistive spring/damper group Xe and the dynamic behavior of Xh toward Xe (dashed line). The displacement of Xe triggers the generation of force which pulls the device tip Xh toward the position Xe .
TSAGARAKIS AND CALDWELL: IMPROVING MOUSE-BASED COMPUTER INTERACTION IN USERS
Fig. 6.
Fig. 5. Mh .
Damping ratio of xh /xe dynamics as a function of hand/finger mass
The attracting force Fd plotted against the force applied by the human Fh is shown in the following graph, Fig. 4. It increases with the application of the human force Fh to assist the motion and declines as soon as the human force is removed and the device position Xh reaches the target point Xe . The fourth graph in Fig. 4 presents the ratio Fh /Fd which can be seen as the reciprocal of the assistive ratio. A high value indicates the absence of assistive forces, while low values reveal their existence. It can be seen that the reciprocal of the assistive ratio is high during the application of the human force, but rapidly declines indicating the rise of the assistive force component. The Bode diagram in Fig. 4 presents the effect of the variation of the mass of the human hand/finger (0.5 Kg < Mh < 2 Kg) to the bandwidth of xh /xe dynamics demonstrating a bandwidth of ∼5 Hz (30 rad/s) for a nominal mass of Mh = 1 Kg. In Fig. 5, the variation of the damping ratio of xh /xe dynamics is presented as a function of the mass of the human hand/finger Mh . These simulation results demonstrate the potential of the proposed scheme as an assistive controller when triggered by the application of the human force.
User interacting with the planar haptic device.
(Ma , Ba , Ka ) represent the inertia, damping, and stiffness matrix coefficients of the desired impedance Za while xe is the impedance equilibrium position vector (equivalent to the grounding end point of the assistive network of the 1-DOF system presented in the previous section) and xh is the position of subject’s hand and device end tip. Having specified the desired behavior of the system, the con¨ from (3) and trol law can be derived by eliminating x ¨h and q (13). To do this, the following equations, which relate the velocities and accelerations of the assistive haptic device endpoint with the velocities and accelerations in joint space, are introduced ˙ q + Jq. x ¨h = J¨
The overall dynamic behavior of the haptic device is described by (3). This equation can also be used to describe the interaction between the subject and the haptic device. Considering the scenario shown in Fig. 6, where the subject’s hand is attached to the device tip, Fh denotes the 2-D force vector that the subject exerts on the planar device tip and Fd = –Fh is the planar force vector that the device applies back to the subject as a consequence of the displacements of the device tip. Suppose now that Za is the system desired output mechanical impedance which governs the force Fh felt by the user. To make the coupled pair’s (haptic device + subject’s hand) motion follow the desired impedance dynamics, the following expression must be applied (13)
To be consistent with the 1-DOF system presented in Fig. 3,
(14)
¨, respectively, gives Solving (13) and (14) for x ¨h and q ˙ h − Ka (xh − xe )} x ¨h = M−1 a {Fh − Ba x ˙ q ¨ = J−1 (¨ xh − Jq).
(15) (16)
Combining (3), (15), and (16), q ¨ can be eliminated to give ˙ h − Ka (xh − xe )) − J˙ q) ˙ + G(q) D(q)J−1 (M−1 a (Fh − Ba x = τ − JT Fd .
(17)
To keep the cartesian inertia of the device unchanged, we select Ma = J−1 DJ−T .
IV. EXTENSION OF THE ASSISTIVE CONTROLLER FOR THE HAPTIC DEVICE
¨h + Ba x˙ h + Ka (xh − xe ). Fh = Ma x
181
(18)
The choice of keeping the Cartesian inertia unchanged is based on the fact that the device design itself (low gear reduction which reduces the amplification of the actuator inertia) and the materials used (lightweight carbon fibre structures and Ergal7075 aluminium joints) for the realization result in a very light and small inertia device (apparent mass on the device tip is below 130 g). Furthermore, the combination of the low apparent physical inertia with slow motions and low accelerations which are relevant to this application means the inertial forces felt by the user can be neglected without any significant compromise. Considering also slow movements typical in mouse-based interaction and that Fd = –Fh , (17) gives τ = −JT (Ba x˙ h + Ka (xh − xe )) + G(q).
(19)
Since the haptic system is a planar device, (19) can be further reduced by eliminating the gravity term G(q) τ = −JT {(Ba x˙ h + Ka (xh − xe )).
(20)
182
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 2, MARCH 2013
state of the system (τ , q ¨, q, ˙ q). It is evident that the force Fh used in (21) to derive the new position xe cannot be directly measured. The estimated component e Fh is derived from the approximated force applied by the device e Fd and the force measured by the force sensor Fs as follows: e
Fh = Fs + e Fd .
(23)
V. EXPERIMENTAL EVALUATION AND DISCUSSION A number of experiments were conducted to evaluate the performance of the impedance assistive control scheme. A. Assisting the Execution of Free-Path Trajectories
Fig. 7.
Diagram of the proposed assistive control.
Equation (20) describes the impedance control law for the haptic device where Ka and Ba are 2 × 2 diagonal stiffness and damping matrices. Similar to the 1-DOF system presented in Fig. 3, these matrices depend on the desired assistive dynamics. Under the assistive control mode, the haptic system should apply assistive force signals depending on the subject’s desired motion; therefore, the user intention of motion is required. To detect the user’s intention, a force torque (F/T) sensor is mounted between the mouse and the tip of the planar device. This sensor monitors the force signals applied by the user and based on these force signals the desired position vector of the impedance equilibrium xe is updated in (20). The following formula was used to derive the new desired position vector xe using the measured force signal ⎫ ⎧ i i kf (Fh − ai ) Fhi > αi ⎪ ⎪ ⎪ ⎪ ⎬ ⎨ i i i i α < Fh < α i = 1, 2 (21) Δxe = 0 ⎪ ⎪ ⎪ ⎪ i i i i i ⎭ ⎩ k (F + a ) F < α f
h
xie = xie−1 + Δxie , Fhi ,
kfi ,
i
h
i = 1, 2
(22)
where and a are the i components of the force vector applied by the operator, the sensitivity constant, and the noise dead band constant vectors, respectively. By injecting the desired position vector derived from (22) into (20), assistive forces augmenting the user desired actions/motions can be generated that are governed by the stiffness matrix Ka of the impedance network. The damping matrix Ba of the impedance filter helps damp the erratic movements toward the direction of motion. A block diagram of the control scheme expressed by (20), (21), and (22) is shown in Fig. 7. In this diagram, e Fh represents the estimated force applied by the user, r Fd is the desired force that should exerted by the device, r τ d is the desired torque command, and e Fd is the force generated by the device approximated from (3) given the output
The first experiments aimed to validate the functionality of the assistive control scheme in Fig. 7 and evaluate its ability to provide different assistive force levels. During these trials, one healthy subject manipulated the device without any constraints, and executed a number of free-path trajectories. The execution of these trajectories under the assistive mode of operation required the tuning of the assistive control scheme. The inertial matrix Ma of the desired impedance was set according to (18) to maintain the Cartesian inertia of the haptic system at the original level. The tuning of the assistive network k 0 b 0 matrices Ka = [ ], Ba = [ ] was based on both simu0 k 0 b lations studies and experimental manual tuning procedures. As a starting point, we consider a bandwidth of ∼5 Hz, (ω n ∼30 √ rad/s) and a damping ratio of ζ = 2/2 for the assistive response of the 2-DOF system presented in Fig. 6. Fine tuning of the stiffness and damping matrixes was performed by a series of trials where adjustments of the (Ka , Ba ) were made while the healthy subject manipulated the device in the XY plane to reach randomly indicated target locations. Overshooting of the trajectories was measured at the target locations but it was not visually shown to the subjects (overshooting of the cursor was visually removed by attracting the cursor image to the target location despite the fact that the real-measured trajectory may overshoot the target location). The tuning of the assistive network was then performed manually based on the amplitude of the overshooting at the two target point locations and the level of the assistive forces applied during the motion from one target point to the other (higher assistive forces tended to increase overshooting). Feedback provided from the subject regarding the comfort and feeling during the tuning procedure was also taken into account. The time required for the gain adjusting procedure was 5–10 min approximately. The scalar values of the diagonal stiffness and damping matrixes (Ka , Ba ) were adk 0 b 0 justed to Ka = [ ], Ba = [ ] with k = 7 N/cm and b = 0 k 0 b 35 N·s/cm compromising between overshooting user comfort, and assistance (equivalent bandwidth and damping ratio of the xh /xe dynamics are ∼3.2 Hz, (ω n ∼20 rad/s) and ζ=0.87). Control of the assistive force levels was achieved by modula0 k tion of the Gf in (5) through the matrix Kf = [ f ]. Two 0 kf of the executed paths for kf = 1 and kf = 2 are shown in Figs. 8 and 9. Fig. 8(a) presents the trajectory generated for kf = 1
TSAGARAKIS AND CALDWELL: IMPROVING MOUSE-BASED COMPUTER INTERACTION IN USERS
183
Fig. 8. Unconstraint trajectories executed under the assistive mode for (a) k f =1 and (b) k f =2.
using the force applied by the user Fh [last row of Fig. 9(a)]. The computation of the user forces was achieved using (23) and through the monitoring of force applied to the force sensor Fs [top row of Fig. 9(a)] and the force generated by the device e Fd [last row of Fig. 9(a)] which was computed from the measurement of the device state (τ , q ¨, q, ˙ q). The individual X and Y components of the assistive impedance equilibrium trajectory xe as well as the user hand trajectory xh are presented in the middle row of Fig. 9(a). The assistive functionality of the system and its control scheme can be observed in the last row of Fig. 9(a) which shows the forces applied by the user and the forces generated by the device. The ratio between the forces generated by the device and those applied by the user can be regulated by tuning the matrix Kf . Fig. 8(b) presents a second trajectory performed with kf = 2. Fig. 9(b) shows the corresponding user and device forces generated and the resulting trajectory components. In this second case, as can be seen in the bottom row of Fig. 9(b), the ratio between the device forces generated in response to the user applied forces is higher, indicating greater assistance. For both levels of kf , the resultant trajectories are smooth, as predicted in the simulations used for the tuning of (Ka , Ba ) with no evidence of forced oscillations.
Fig. 9. X, Y trajectories, subject forces, and assistive forces generated by the device for the two trajectories of Fig. 8 for (a) k f = 1 and (b) k f = 2.
B. Improving the Tracking Performance The second set of trials involved a subject suffering from muscle ataxia. The purpose of these experiments was to evaluate if the assistive functionality demonstrated in the first (able bodied) experiment can aid the ataxia impaired subject to interact better with a computer-generated environment by reducing the effect of undershooting/overshooting behavior. The test subject was a regular computer user and possessed the ability to manipulate the mouse in a somewhat good manner but with difficulties in the accurate target tracking, e.g., moving a vertical or horizontal slider or selecting an item from a dropdown menu. During this experiment, the subject, sitting in front of a computer display, manipulated the mouse attached to the tip of the haptic device and performed a number of tracking experiments with and without the use of the assistive control scheme, Fig. 10. A target tracking task required the subject to follow an onscreen target pattern with a cursor which was controlled by manipulating the mouse at the tip of the haptic device in a manner similar to computer mouse input. Two tracking patterns
Fig. 10. device.
Subject manipulating the mouse attached on the assistive haptic
were used: a diagonal line and a 90◦ edge. In the first phase of this experiment, no haptic assistance was provided. To demonstrate the tracking difficulties of the muscle ataxia subject, the same experiment was also repeated with the healthy subject. Both the impaired subject and healthy subject were asked to track the line and edge patterns several times as quickly as they could. Observing the impaired subject trajectory in Fig. 11, there is
184
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 2, MARCH 2013
Fig. 11.
Line and edge tracking performance of the muscle ataxia impaired subject.
Fig. 12.
Line and edge tracking performance of a typical healthy subject.
clear evidence of erratic motions during the execution of the task. Comparing the performance of the muscle ataxia subject with that demonstrated by a typical healthy subject, Fig. 12, the tracking difficulty of the impaired subject can be confirmed. Following these initial without assistance trials, the haptic device was set to assistive mode and the same tracking tasks were executed by the impaired subject. The assistive network parameters were tuned with a procedure similar to that described in Section IV-A. The scalar values of the diagonal stiffness and damping matrixes, matrixes Ka and Ba were adjusted accordingly (k = 7 N/cm and b = 50 N·s/cm) with kf = 1. Fig. 13 presents the trajectory for the line and the edge tracking tasks. It can be seen that the amplitude of the sudden motions has been reduced with the application of the assistive control. Comparing the performance of these trajectories with those in Fig. 11 obtained without (WO) the assistance, it can be seen that in the case of the edge pattern, the mean error distance of the trajectories, Fig. 14(a), has been significantly reduced (by 40%) with (W) the provision of the assistive control. We perform a t-test between the means of the error distances of the tracking
trajectories without assistance and of those with assistance. The t-test indicated significant differences on the mean distance errors under the two modes of operation (p = 0.026, t(4) = 3.43). The average of the means of the error distance for the trajectories performed under the assistive mode was lower than that of ¯ the trajectories without assistance X(W/WO) = 10.92/16.72, Fig. 14(b). In terms of execution time, the impaired subject was, in addition, able to complete the task in less time (19% reduction) compared with the time needed in the case of the unsupported trajectory, t(W/WO) = 14.72/18.13 s. In the case of the line target similar trends were observed. The overall trajectory is also smoother and more regular with fewer and smaller jerks or spasms. Fig. 15(a) shows the means of the distance error for the line pattern in the no assistive and assistive mode demonstrating considerable accuracy improvements. The t-test between the means also revealed noteworthy variations between the means under the two modes of operation, (p = 0.011, t(4) = 4.45). The average of the means of the error distance for the trajectories
TSAGARAKIS AND CALDWELL: IMPROVING MOUSE-BASED COMPUTER INTERACTION IN USERS
185
Fig. 15. Line tracking performance: (a) Mean and standard deviation of the error distance for the different trials without and with assistance. (b) Average of means of the error distance and average execution time with and without assistance.
Fig. 13. Line and edge tracking performance of the impaired subject with assistive control.
time, the subject was able to complete the task in less time (40% reduction) compared with the time needed in the case of the unsupported trajectory, t(W/WO) = 7.71/12.89 s. The previous results indicate that the system and control scheme have the potential to improve the tracking performance of muscle ataxia impaired subjects. By providing assistive forces collinear with the forces applied by the subject, the system helped the impaired user to improve their tracking performance, both in accuracy and execution time. VI. CONCLUSION AND FUTURE WORK
Fig. 14. Edge tracking performance: (a) Mean and standard deviation of the error distance for the different trials without and with assistance. (b) Average of means of the error distance and average execution time with and without assistance.
performed under the assistive mode was also considerably lower (by almost 60%) than those of the trajectories without assistance ¯ X(W/WO) = 5.89/16.08, Fig. 15(b). In terms of execution
Computer-assisted therapies are a significant potential training tool for many conditions, but they do rely on the ability to efficiently interact with the computer. In conditions such as muscle ataxia, this efficient interaction is not possible. A 2-DOF pantograph planar device was developed to assist the impaired subjects with poor muscle tone and weak muscles to interact with a computer-generated environment using a mouse interface. The device control scheme was designed to assist the impaired subject to complete a tracking task with higher accuracy and in less time. The overall system was evaluated with experimental trials performed by a healthy and an impaired subject suffering from muscle ataxia. In particular, the benefits of the motion assistance on the trajectory tracking were evaluated by experiments where the impaired subject performed 2-D tracking tasks. With assistance, the tracking performance of the impaired subject improved in terms of the execution time (19% reduction for edge pattern and 40% reduction for the line pattern). In terms of accuracy, the reduction of the mean of the distance error of the tracking trajectory from the target path was statistically signif¯ icant with X(W/WO) = 10.92/16.72 for the edge pattern and ¯ X(W/WO) = 5.89/16.08 for the line trajectory. These results
186
IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 2, MARCH 2013
indicate that the proposed assistive setup has a good potential in improving the hand/arm motion of the muscle ataxia subjects by reducing the trajectory overshooting trends. Following the previous promising output the current work focuses on improving the system hardware optimizing, size, portability, and cost. Further trials are also foreseen for the near future to assess the application of the system in other pathological conditions including Parkinson’s disease, muscular dystrophy, and cerebral palsy. These disabilities are typically associated with symptoms such as reduced strength, restricted or irregular jerky movements, poor motion coordination, and a continuum of impairments involving spasms and tremors. Further extensions of the assistive controller may include the use of electromyography signal to intrinsically monitor and extract the user intention/direction of motion which will drive the device assistance effort. REFERENCES [1] M. Hillman, “Rehabilitation robotics from past to present—A historical perspective,” in presented at the Int. Conf. Rehabil. Robot., Daejeon, Korea, Apr. 2003. [2] C. G. Burgar, P. S. Lum, P. C. Shor, and H. F. M. Van der Loos, “Development of robots for rehabilitation therapy: The Palo Alto VA/Stanford experience,” J. Rehabil. Res. Develop., vol. 37, no. 6, pp. 663–673, Nov./Dec. 2000. [3] J. Oblak, I. Cikajlo, and Z. Matjacic, “Universal haptic drive: A robot for arm and wrist rehabilitation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 18, no. 3, pp. 293–302, Jun. 2010. [4] D. G. Caldwell, N. G. Tsagarakis, S. Kousidou, N. Costa, and Y. Sarakoglou, “Soft’ exoskeletons for upper and lower body rehabilitation—Design, control and testing,” J. Humanoids Robot.: Special Issue Active Exoskeletons, vol. 4, no. 3, pp. 549–573, 2007. [5] S. J. Spencer, J. Klein, K. Minakata, V. Le, J. E. Bobrow, and D. J. Reinkensmeyer, “A low cost parallel robot and trajectory optimization method for wrist and forearm rehabilitation using the Wii,” in Proc. 2008 IEEE Eng. Med. Biol. Soc. Conf. Biorobot., 2008, pp. 869–874. [6] M. Rosen, A. Arnold, I. Baiges, M. Aisen, and S. Eglowstein, “Design of a controlled-energy-dissipation-orthosis (CEDO) for functional suppression of intention tremors,” J. Rehab. Res. Dev., vol. 32, no. 1, pp. 1–16, 1995. [7] M. Brown, N. G. Tsagarakis, and D. G. Caldwell, “Exoskeletons for human force augmentation,” Ind. Robots, vol. 30, no. 6, pp. 592–602, 2003. [8] C. R. Carignan, M. P. Naylor, and S. N. Roderick, “Controlling shoulder impedance in a rehabilitation arm exoskeleton,” in Proc. IEEE Int. Conf. Robot. Autom., 2008, pp. 2453–2458. [9] N. G. Tsagarakis and D. G. Caldwell, “Development and control of a ‘Soft-actuated’ exoskeleton for use in physiotherapy and training,” Auton. Robots, Spec. Issue Rehabil. Robot., vol. 15, pp. 21–33, 2003. [10] K. Kong and D. Jeon, “Design and control of an exoskeleton for the elderly and patients,” IEEE/ASME Trans. Mechatronics, vol. 11, no. 4, pp. 428–432, Aug. 2006. [11] S. Kousidou, N. G. Tsagarakis, D. G. Caldwell, and C. Smith, “Assistive exoskeleton for task based physiotherapy in 3-dimensional space,” IEEE/RAS-EMBS Int. Conf. Biomed. Robot. Biomechatronics, Pisa, Italy, 2006, pp. 266–271. [12] S. Kousidou, N. G. Tsagarakis, C. Smith, and D. G. Caldwell, “Taskorientated biofeedback system for the rehabilitation of the upper limb,” in Proc. 10th Int. Conf. Rehabil. Robot., 2007, pp. 376–384. [13] A. Gupta and M. K. O’Malley, “Design of a haptic arm exoskeleton for training and rehabilitation,” IEEE/ASME Trans. Mechatronics, vol. 11, no. 3, pp. 280–289, 2006. [14] T. Nef, M. Mihelj, G. Colombo, and R. Riener, “ARMin–robot for rehabilitation of the upper extremities,” in Proc. 2006 IEEE Int. Conf. Robot Autom., Orlando, Florida, May 2006, pp. 3152–3157.
[15] D. J. Kim, R. H-Knudsen, H. Culver-Godfrey, G. Rucks, T. Cunningham, D. Port´ee, J. Bricout, Z. Wang, and A. Behal, “How autonomy impacts performance and satisfaction: Results from a study with spinal cord injured subjects using an assistive robot,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 42, no. 1, pp. 2–14, Jan. 2012. [16] G. Rosati, P. Gallina, and S. Masiero, “Design, implementation and clinical tests of a wire-based robot for neurorehabilitation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 15, no. 4, pp. 560–569, Dec. 2007. [17] L. E. Kahn, M. L. Zygman, W. Z. Rymer, and D. J. Reinkensmeyer, “Robot-assisted reaching exercise promotes arm movement recovery in chronic hempiaretic stroke: A randomized controlled pilot study,” J. Neuroeng. Rehabil., vol. 3, pp. 1–13, Jun. 21, 2006. [18] M. Uemura, K. Kanaoka, and S. Kawamura, “Power assist system for sinusoidal motion by passive element and impedance control,” in Proc. IEEE Int. Conf. Robot. Autom., Orlando, Florida, May 2006, pp. 3935– 3940. [19] B. T. Volpe, H. I. Krebs, N. Hogan, L. Edelstein, C. Diels, and M. Aisen, “A novel approach to stroke rehabilitation: Robot-aided sensorimotor stimulation,” Neurology, vol. 54, pp. 1938–1944, 2000. [20] J. J. Magee, M. Betke, J. Gips, M. R. Scott, and B. N. Waber, “A human– computer interface using symmetry between eyes to detect gaze direction,” IEEE Trans. Syst., Man, Cybern. A, Syst., Humans, vol. 38, no. 6, pp. 1248– 1261, Nov. 2008. [21] B. Cesqui, S. Micera, S. Mazzoleni, M. C. Carrozza, and P. Dario, “Analysis of upper limb performance of elderly people using a mechatronic system,” in Proc. Intl. Conf. Biomed. Robot. Biomechatronics, 2006, pp. 365– 370. [22] E. Rocon, A. F. Ruiz, F. Brunneti, and J. L. Pons, “On the use of an active wearable exoskeleton for tremor suppression via biomechanical loading,” in Proc. 2006 IEEE Int. Conf. Robot. Autom., Orlando, Florida, May2006, pp. 3140–3145. [23] E. J. Koeneman, R. S. Schultz, S. L. Wolf, D. E. Herring, and J. B. Koeneman, “A pneumatic muscle hand therapy device,” in Proc. IEEE Conf. Eng. Med. Biol. Soc., 2004, vol. 4, pp. 2711–2713. [24] H. I. Krebs, B. T. Volpe, D. Williams, J. Celestino, S. K. Charles, D. Lynch, and N. Hogan, “Robot-aided neurorehabilitation: A robot for wrist rehabilitation,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 15, no. 3, pp. 327–335, Sep. 2007. [25] S. Trewin and H. Pain, “Keyboard and mouse errors due to motor disabilities,” Int. J. Human-Comput. Studies, vol. 50, no. 2, pp. 109–144, 1999. [26] J. T. Dennerlein and M. C. Yang, “Haptic force feedback devices for the office computer: Performance and musculoskeletal loading issues,” Human Factors, vol. 43, no. 2, pp. 278–286, 2001. [27] S. Hasser, C. Goldenberg, A. Martin, and K. Rosenberg, “User performance in a GUI pointing task with a low-cost force feedback computer mouse,” in Proc. ASME Dyn. Syst. Control Division, 1998, pp. 151–156. [28] F. Hwang, S. Keates, P. Langdon, and P. J. Clarkson, “Multiple haptic targets for motion impaired users,” in Proc. Conf. Human Factors Comput. Syst., Fort Lauderdale, FL, 2003, pp. 41–48. [29] S. Beringhause, M. Rosen, and S. Haung, “Evaluation of a damped joystick for people disabled by intention tremor,” in Proc. 12th Annu. Conf. Rehabil. Eng. Assistive Technol. Soc. North Am., 1989, pp. 41–42. [30] B. Morrice, W. Becker, J. Hoffer, and R. Lee, “Manual tracking performance in patients with cerebellar incoordination—Effects of mechanical loading,” Canadian J. Neurol. Sci., vol. 17, no. 3, pp. 275–285, 1990. [31] M. B. Delatycki, R. Williamson, and S. M. Forrest, “Friedreich ataxia: An overview,” J. Med. Genet., vol. 37, pp. 1–8, 2000. [32] T. E. Milner and C. Cloutier, “Damping of the wrist joint during voluntary movement,” Exp. Brain Res., vol. 122, pp. 309–317, 1990. [33] J. D. Becker and C. D. Mote, “Identification of a frequency response model of joint rotation,” J. Biomech. Eng., vol. 112, pp. 1–8, 1990. [34] A. Z. Hajian and R. D. Howe, “Identification of the mechanical impedance at the human finger tip,” J. Biomech. Eng., vol. 119, pp. 109–114, 1997. [35] F. Lacquaniti, F. Licata, and J. F. Soechting, “The mechanical behaviour of the human forearm in response to transient perturbations,” Biol. Cybern., vol. 44, pp. 35–46, 1982.
TSAGARAKIS AND CALDWELL: IMPROVING MOUSE-BASED COMPUTER INTERACTION IN USERS
Nikos G. Tsagarakis received the Dip.-Eng. degree in electrical and computer science engineering from the Aristotle University in Greece, in 1995, the M.Sc. degree in control engineering in 1997, and the Ph.D. degree in robotics from the University of Salford, Salford, U.K., in 2000. Before becoming a Senior Researcher at Italian Institution of Technology, Genova, Italy, with overall responsibility for Humanoid and Human Centred Mechatronics Design, he was a Research Fellow and then a Senior Research Fellow in the Centre for Robotics and Automation at the University of Salford, where he worked on haptic systems, wearable exoskeletons, rehabilitation robots, and humanoids robots. He is the author or coauthor of more than 140 papers in research journals and at international conferences and has six patents. Dr. Tsagarakis received the 2009 PE Publishing Award from the Journal of Systems and Control Engineering and prizes for Best Paper at the International Conference on Advanced Robotics (2003). He was also a finalist for Best Entertainment Robots and Systems–20th Anniversary Award at the International Conference on Intelligent Robots and Systems (IROS’07) and finalist for the Best Manipulation paper at the International Conference on Robotics and Automation (ICRA’12). He has been on the Programme Committee of over 40 international conferences including IEEE International Conference on Robotic Automation (ICRA), IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Robotics Science and Systems (RSS), International Conference on Humanoid robots, International Conference on Biomedical Robotics and Biomechatronics (BIOROB), and International Conference on Informatics in Control, Automation and Robotics (ICINCO).
187
Darwin G. Caldwell received the B.S. and Ph.D. degrees in robotics from the University of Hull Hull, U.K., in 1986 and 1990, respectively and the M.Sc. degree in management in 1996 from the University of Salford, Salford, U.K. He is currently the Director of Advanced Robotics at the Italian Institute of Technology, Genova, Italy, and a Visiting/Honorary Professor at the Universities of Sheffield, Manchester, Kings College, Bangor, and Tianjin (China). He is the author or coauthor of more than 300 academic papers, and has 15 patents. His research interests include innovative actuators, force augmentation exoskeletons, dexterous manipulators, humanoid (iCub and COMAN) and quadrupled robots (HyQ), and rehabilitation robotics. Dr. Caldwell received awards from several international conferences and journals including 2009 PE Publishing Award, 2011 Emerald Publishing Award, IEEE International Conference on Robotic Automation, IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE International Conference on Information and Automation, Medicine Meets Virtual Reality (MMVR), IEEE Humanoids, IEEE Conference on Automation Science and Engineering; Advances in Computer-Human Interactions, International Conference on Advanced Robotics, IFAC Intelligent Autonomous Vehicles Symposium, and Virtual Concepts. He is on the Editorial Board of the International Journal of Social Robotics and Industrial Robot and Technical Editor of the IEEE/ASME TRANSACTIONS ON MECHATRONICS.
