The impact on musculoskeletal system during ... - Semantic Scholar

CHI 2011 • Session: Gestures, Body & Touch

May 7–12, 2011 • Vancouver, BC, Canada

The Impact on Musculoskeletal System during Multitouch Tablet Interactions Cecil Lozano1, Devin Jindrich2, and Kanav Kahol1 Arizona State University 1 501 E. Tyler Mall, Tempe, AZ 85287, USA 2 427 E. Tyler Mall, LSE 216, Tempe, AZ 85287-45041, USA {Cecil.Lozano, Devin.Jindrich, Kanav.Kahol}@asu.edu ABSTRACT

turn makes the technology suitable to become the default human computer interaction standard of the near future. However, as we move towards a world where interaction is based on human body movements that are not well documented or studied, we face a serious and a grave risk of creating technology and systems that may lead to musculoskeletal disorders.

HCI researchers and technologists have heralded multitouch interaction as the technology to drive computing systems into the future. However, as we move towards a world where interaction is based on human body movements that are not well documented or studied, we face a serious and a grave risk of creating technology and systems that may lead to musculoskeletal disorders (MSD’s). Designers need to be empowered with objective data on the impact of multitouch interactions on the musculoskeletal system to make informed choices in interaction design. In this paper we present an experiment that documents kinematic (movement) and kinetic measures (EMG) when interacting with a multitouch tablet. Results show that multitouch interaction can induce significant stress that may lead to MSDs and care must be taken when designing multitouch interaction.

Often interfaces such as keyboards, mouse and touch screens are designed without considering the effect of interaction on musculoskeletal system [1]. The increase in workplace computer usage since 1980 has been accompanied by a corresponding increase in incidence rates of chronic musculoskeletal disorders reported in the United States [1]. Conservative estimates of the annual cost to the U.S. economy for all musculoskeletal disorders range from $45 to $54 billion [5]. Poor design strategies can cause biomechanical and physical stress to the musculoskeletal system and are associated with work-related musculoskeletal disorders (MSDs) [1]. Additionally, even when users are allowed to define their gestures for interactions with touch based systems, the lack of knowledge about the impact of gestures may cause users to choose unsafe gestures. Thus, there is the need of generating ergonomic standards of interaction in which a diversity of current and novel gestures can be assessed and can therefore aid designers (and users) in making appropriate choices in terms of ergonomic impact of gestures on the user’s musculoskeletal system (MSS).

Author Keywords

Ergonomic Multitouch tablet interfaces. ACM Classification Keywords

H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6) Ergonomics. General Terms

Experimentation, human performance, design.

factors,

measurement,

INTRODUCTION

The advent of numerous commercial devices and software such as iPhone® (www.apple.com/iphone), Microsoft Surface® (www.microsoft.com/surface/index.html), Windows 7® (www.microsoft.com/windows/windows-7/), HP’s new touch screen PC and CNN’s Multitouch collaboration Wall has heralded a new era of multitouch interaction. Multitouch interaction enables a natural, veridical interface to digital media [2]. It allows for users to experience content through multimodal interaction enabling a high degree of interactivity and impressiveness, which in

This paper presents an initial attempt to define scientific principles to build such standards. To that goal it is required a methodology of analysis where gestures are carefully controlled to evaluate their independent and isolated impact to the MSS. With such information a designer can indeed choose which gestures should have higher frequency and which should be used sparingly. This information would be critical in developing data driven guidelines for interactions. Furthermore, we believe that an important initial element of these investigations needs to focus on generation of gestures, as this is the technology’s novelty and there is no knowledge of their effect on the MSS. Thus in this work we investigated the effect of a set of free stroke gestures that represent one and two finger type of interactions on device

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2011, May 7–12, 2011, Vancouver, BC, Canada. Copyright 2011 ACM 978-1-4503-0267-8/11/05....$10.00.

825



conditions relevant to mobile devices and to building design guidelines (see below). Tapping/typing motions also require investigation but were not included as they involve different confounding variables that will be tested independently.

ulnar styloid process. Amplification for each muscle was adjusted to maximize the information captured. Finger movement (kinematic) information was obtained for the dominant hand with a CyberGlove® 22 sensor system that captures angular excursions of the finger joints. We focused our analysis on the index finger as it has the maximum use in the multitouch interactions.

Through this initial experiment, we have developed objective measures of musculoskeletal stress introduced by gestures on tablet system. The short-term aim of this research is to provide designers with objective measures of the impact of gestures and configurations of tablet on musculoskeletal system. This would empower multitouch interaction designers to make informed choices of interaction design. The long-term aim of this research is to develop a software toolkit that would enable designers to input a target gesture and ascertain its impact on the musculoskeletal system, which would allow for safe and effective designs.

Participants

Eight right-handed healthy participants (6 males and 2 females, ages 20 to 27) volunteered for one experimental session that lasted 3-4 hours. Handedness was selected to represent the majority of the population and to minimize confounding factors that need to be address separately. Participants reported to be frequent users of multitouch technology (more than 40 hours), which eliminated familiarization time. Note that level of experience with the iPad was not assessed since the interaction studied was not task oriented. Subjects also reported to be free of any musculoskeletal disorder of the upper extremities. Procedures were approved by the Institutional Review Board at our University and were in accordance with the declaration of Helsinki.

METHODS Experimental design

The goals of this experiment were to measure physiological impact of using gestures for multitouch technology, specifically the iPad, and to provide a glance of some parameters to consider for designing guidelines. We tested 3 different conditions: gesture type, device position and context.

Experimental protocol

Participants comfortably sat in front of a table with the device in front of them. They performed each of the 8 gestures for each of the device position and contextual conditions a total of 3 times (total of 8x2x2x3 = 96 trials). In each recorded trial, participants first waited 5 s with their arms resting on the device’s side, which was used as starting point and allowed recording of baseline signal of minimal muscle activation. They then were cued to repeat the gesture in turn for 20 s until they were cued to stop. A fixed time allowed for collecting enough data for analysis and for statistical comparison between gestures. Participants were asked to perform the gestures in the middle of the device using only the fingers required (index and thumb for rotating and zooming and index for panning), while the arm wasn’t supported. In the conditions when the device was held, participants were requested to lift the device with their non-dominant arm to the previously chosen comfortable position and hold it throughout the time of performing the gesture without resting the arm. We asked participants to look at the center of the device while doing the gestures even when it was off.

We chose 8 gestures using only index and thumb fingers of participant’s dominant (right) hand: rotating to the right and left, zooming in, zooming out, panning with the index to the right, left, up and down. These gestures represent “free” form strokes that involve dragging using single and multiple fingers and are being used in most devices for common HCI actions. We incorporated also an important usability factor of mobile devices by having gestures being executed in 2 device positions: exocentric or fixed on a table in a 15º angle in front of the participant and egocentric while the participant held the device with their nondominant (left) hand. And finally, we included a factor of context geared for design guidelines. Gestures were performed in 2 contextual scenarios: non-contextual when device was off and contextual as participants interacted with mapping software Google Earth (as this application allowed for the use of all the tested gestures). This factor is particularly important to estimate the difference in the effect of novel gestures at the non-contextual design phase and the applied phase in realistic scenarios.

We ran 3 sets (replications) of 8 gestures for each combination of device position and contextual condition. The order of the gestures was randomized within set and the order of the device position and contextual conditions among participants were presented in a complete counterbalance for immediate sequential effects order. We included 20 s rests between trials to allow recovery of gesture in turn, 1 m rest between sets of trials and at least 5 m between changes of position-contextual device condition change to minimize mental/physical fatigue.

To assess the effects of these factors, we measured muscle activation and finger joint angles. Muscle activation was measured using bipolar surface electromyography (EMG; Bortec AMT-8). We recorded from the dominant forearm extensor digitorum communis (wrist and finger extension) and the anterior deltoid (forward shoulder flexion). We also included the non-dominant biceps (forearm flexion) and anterior deltoid. Placement of electrodes achieved through palpation and signal response to isometric test contractions [3]. For the ground reference we used the non-dominant

826



Figure 1. Muscle activation as percentage of maximum voluntary contraction (% of MVC) for each of the four muscles and all the conditions with error bars representing least significant differences. Axis labels of (a & b) same as (c & d).

The kinematic data from the three joints of the index finger yielded angular changes in gestures presented in degrees. We obtained statistical comparisons using the average of the 3 replications of each condition for each participant (i.e., 8 values per condition). We used the ANOVA output of Design Expert for a general factorial design of the 3 factors with 8, 2 and 2 levels corresponding to gestures, device position and contextual conditions. This analysis allowed the removal of any variation attributed to blocks of data (each participant data assigned to one block). Significance was set at 5% (i.e., α = 0.05 or p-value < 0.05). Figure 2. Average rotation amplitude of the index finger interphalangeal joint during gestures.

RESULTS Gestures

We obtained two maximum voluntary contractions (MVCs) for each muscle at the beginning and at the end of the experimental session. These measures were used for both normalizing the EMG data and measuring possible fatigue of the individual muscles after producing all the tested gestures.

Gesture type had a statistical effect on the dominant arm doing the gesture. The average dominant wrist extensor muscle activity was generally greater for the gestures where both fingers (thumb and index) were involved than when only the index finger was used for the gesture (factor’s p-value < 0.0001; see Figure 1a). In average rotating to the right produced the highest (16.1% of MVC) and panning down the lowest muscle activation (8.5% of MVC). It should be noted that the average muscle activation is higher to reported maximum activation of computer mouse use [4].

Data analysis

We evaluated muscle activation as the percentage of MVC, since each muscle had different EMG amplification. We first calculated the root mean square of the EMG signal for the 10 to 20 s of the 25 s trial to assure we didn’t include the transitions within the trial (e.g., beginning and end of doing the correspondent gesture). We then normalized it with the maximum EMG obtained from the MVCs. The maximum EMG values from both MVCs were extracted after smoothing the signal with a 1 s moving average window.

For the dominant deltoid, rotating to the right and panning up produced the highest muscle activation (9.2 and 8.7 % of MVC, respectively), while panning sideways produced the least activation (5.1% and 5.4% of MVC; factor’s p-value < 0.0001; see Figure 1b).

827



Kinematic data analysis showed similar trends to the EMG. The difference between gestures was statistically significant (Figure 2). Rotation and Zooming gesture produced high levels of rotation up to 40 degrees.

used for training, a sustained muscle activation of even 5% of MVC may cause musculoskeletal disorders [7]. Non-contextual gestures produced slightly higher activation of the arm involved, especially for two-finger gestures. This outcome indicates that context might have a different effect depending on the type of gestures. Greater activation could partially be driven by factors such as force and motion amplitude. Our experimental results showed that at least the motion amplitude partially explains the activation patterns.

Device position

Device position had statistical effect on the non-dominant arm muscle activations. Activation was higher when the device was held (egocentric) than when it was fixed on the table (exocentric; factor’s p-value < 0.0001; see Figure 1c and 1d). Additionally, when holding the device, the activity of the biceps with respect to the maximum voluntary contraction was greater for the biceps (9.5% of MVC) than the deltoid (6.3% of MVC). Device position had no significant impact on the kinematic activations.

CONCLUSIONS

The presented data provides important insights for designers to consider ergonomic factors when designing multitouch interaction. It suggests that gestures that involve two fingers (e.g., rotation), can introduce high levels of muscle activations. Furthermore, certain conditions such as egocentric device positioning (such as the condition in using tablets as book readers and hand documentation units) can introduce additional stress. Thus it is important that HCI designers develop applications and devices that encourage safe practices while limiting unsafe practices. Note that the results of this study represent only a portion of the vast repertoire of factors that may affect interaction with multitouch technology. In future work, we aim to asses additional factors to develop guidelines and provide a toolkit for designers to aid in selection process for gestures. For example assess gestures in contexts such as “opening a web page and finding a recipe” etcetera.

Task context

The task context had an effect on muscle activation of the dominant arm only. The average non-contextual gestures (device off) produced higher wrist muscle activation (12.5% of MVC) than contextual gestures (interacting with Google Earth; 11.2% of MVC; factor’s p-value < 0.05). This difference was more noticeable for the two finger gestures, especially for zooming in (see Figure 1a). Similar to the wrist, the non-contextual gestures produced higher deltoid muscular activation (7.7% of MVC) than contextual gestures (6.6% of MVC; Figure 1b). From a kinematic perspective, task context had a significant impact with contextual interaction providing higher joint rotation amplitudes.

REFERENCES

1. (BLS), B.o.L.S., Reports on survey of occupational injuries and illnesses in 1977-2001. (2002).

Maximum voluntary contractions

The biceps was the muscle that had the largest tendency for lower MVCs at the end of the experiment. This shows that holding the device may produce unsafe fatigue.

2. Buxton, B. Multitouch Systems that I have known and loved. www.billbuxton.com/multitouchOverview.html, (2008).

DISCUSSION

3. Cram, J. and E. Criswell, Introduction to surface electromyography. Second ed. (2010): David Cella. 412.

It was expected that gestures require the activation of local (e.g., wrist muscles) and non-local (e.g., deltoid) muscles. Our experiments show that indeed multitouch interaction has impact on the entire hand shoulder system and in some cases the impact can be at risk levels (higher than the lowest recommended). This is an important experimental result, which seems to suggest that caution must be observed when considering interaction design. Multitouch interaction can indeed be very engaging and users may have the tendency to develop MSDs without conscious knowledge of the developing conditions.

4. Laursen, B., B.R. Jensen, and A. Ratkevicius, Performance and muscle activity during computer mouse tasks in young and elderly adults. European Journal of Applied Physiology. 84(4), (2001), 329-336. 5. NSR/IOM, N.R.C.a.I.o.M.C., Musculoskeletal disorders and the workplace: low back and upper extremities. (2001), Washington, D.C.: National Academy Press. 6. Oliveira, L.F., et al., Effect of the shoulder position on the biceps brachii EMG in different dumbbell curls. Journal of Sports Science and Medicine. 8(1), (2009), 24-29.

As expected we also found that holding the device (egocentric) would have greater muscle activation of the non-dominant arm (both biceps and deltoid). The activation of the biceps was around 10% of the MVC, which is about 50% of the average activation found for a similar posture in an experiment for dumbbell biceps curl exercises [6]. Although these and higher levels of muscle activation are

7. Sommerich, C.M., W.S. Marras, and M. Parnianpour, A method for developing biomechanical profiles of handintensive tasks. Clinical Biomechanics. 13(4-5), (1998), 261-271.

828