measuring user experience with postural sway and ... - SAGE Journals

Proceedings of the Human Factors and Ergonomics Society 2017 Annual Meeting

2062

MEASURING USER EXPERIENCE WITH POSTURAL SWAY AND PERFORMANCE IN A HEAD-MOUNTED DISPLAY 1

1

2

3

1

Amelia Kinsella , Sarah Beadle , Michael Wilson , L. James Smart Jr. and Eric Muth 2 Department of Psychology, Clemson University, Clemson, SC; U.S. Army Aeromedical 3 Research Laboratory, Fort Rucker, AL; Department of Psychology, Miami University, Oxford, OH 1

The purpose of this study was to examine differences in performance and postural sway among individuals performing a task when using and not using a head-mounted display (HMD). Twenty participants completed a repeated measures study in which they performed a simple object location targeting task while using and not using an HMD. Accuracy, time-to-hit, and postural sway were measured. Significant differences in accuracy and time-to-hit were found, indicating participants’ task performance was worse while using the HMD. Significant differences in magnitude of postural sway were found in elliptical area and path length, showing more sway when using the HMD than when not using it. These methods can be used to objectively measure user differences in response to wearing and using an HMD that can result in negative effects for users.

Copyright 2017 by Human Factors and Ergonomics Society. DOI 10.1177/1541931213601999

INTRODUCTION The purpose of this experiment was to examine differences in performance and postural sway within individuals when using and not using a head-mounted display (HMD) to complete a simple task. HMDs and systems such as the Oculus Rift have become extremely popular and accessible to consumers (Lewis, 2015). These HMD and virtual reality (VR) or augmented reality (AR) headsets have the capability to immerse the user into different worlds or enhance their experience with the surrounding environment. They can also be used to advance fields such as medicine, engineering, education, design and training (Stanney, Mourant, & Kennedy, 1998). With the new popularity of these devices, it is important to consider objective measures of usability. One way to examine the usability of an HMD is to observe how it changes a person’s movement. Postural control is a complex skill involving multiple sensorimotor processes that allow us to maintain our balance (Horak & Macpherson, 1996). Postural sway offers an objective way to look at the difference in postural control while wearing an HMD. Postural sway can be used as a proxy for instability of balance experienced in these systems. Postural instability has been linked to motion sickness (Riccio & Stoffregen, 1991). Specifically, there is evidence that an increase in magnitude and spatial complexity, and a decrease in temporal complexity are related to simulator sickness (Smart, Otten, Strang, Littman, & Cook, 2014). Measures of postural sway offer an objective way to look at the impact of HMD use on wearers, as increased instability may result in negative effects such as sickness and higher likelihood of falling (Horak, 2006; Adkin, Frank, & Jog, 2003; Corbeil, Simoneau, Rancourt, Tremblay, & Teasdale, 2001). It has long be suspected that individuals underreport symptoms of sickness (Biocca, 1992), thus it is valuable to focus on objective measures to add to current subjective assessment methods for measuring negative symptoms and usability in a laboratory setting. Another way to look at the usability of an HMD is to observe how it affects users’ performance when wearing it. There is a gap in the literature regarding the direct effects of HMD use on task performance. Studies have shown that

simulator sickness has negative effects on human performance with symptoms such as fatigue and drowsiness (Kennedy, Drexler, & Kennedy, 2010). Other studies with object location tasks have seen decreases in accuracy and reaction time while experiencing visual delay, but in terms of frame rate as opposed to latency (Watson, Spaulding, Walker, & Ribarsky, 1997) or in non-HMD virtual reality systems (MacKenzie & Ware, 1993). It remains unknown how using an HMD might impact performance in an object location task involving realtime feedback compared to performing the same task when not using an HMD. The current study aimed to examine differences in user movement and performance when using and not using an HMD. The experiment used a repeated measures design that allowed participants to perform the same task while using and not using an HMD. It was hypothesized that participants’ performance would be worse when using the HMD than when not using the HMD, with slower reaction times and less accuracy. Additionally, it was hypothesized that participants’ movement would be different when using and not using the HMD; specifically, that magnitude and spatial complexity of movement would increase and temporal complexity would decrease when using the HMD. METHODS Participants Twenty (10 male) Clemson University students were recruited to participate in this study. Participants ranged in age from 18-26 (mean age = 20.3). All participants signed an IRB approved informed consent form, and were compensated $50 for their time. Participants who self reported any history of brain, heart, stomach, eye (other than corrected vision), inner ear problems, or being pregnant, were excluded from the experiment. Participants with corrected vision were required to wear contact lenses to participate due to limitations of the HMD fit. Design and Apparatus


This experiment was part of a larger within-subjects experiment examining performance when using an HMD with no additional latency and with experimentally induced latency. This data collection involved participants coming to the lab for two experimental sessions separated by exactly two weeks. During each experimental session participants were exposed to the experimental paradigm twice, once without the HMD and once with the HMD with one of the latency conditions. The order of latency condition was counter balanced. The current analysis was only interested in the no HMD condition and the base latency condition. The no HMD condition was always taken from the second experimental session, as participants were still learning the experimental task during the first no HMD condition. Because of this and the fact that the latency conditions were counterbalanced, eleven participants experienced the no HMD and HMD condition in the same day, and nine participants experienced these conditions separated by two weeks. The experimental apparatus was first described in Moss and Muth (2011) and involved an HMD coupled with a video camera mounted on top (Figure 1). A frame-grabber integrated into a personal computer captured images of the room from the camera and displayed them directly into the HMD. In the case of this study, the images were displayed with only the base system latency of 70 ms (no additional latency). This set up resulted in augmented reality, not virtual reality. Participants wearing the HMD were seeing real footage of the room around them.

2063

The real-world captured images from the camera were projected on the HMD, as well as the computer monitor for the experimenter to observe. Procedure Each experimental session consisted of five blocks of trials with no HMD, and five blocks of trials with the HMD. Each block contained 40 trials and lasted 2 minutes. The experimental task was a modification of an object location task originally described by Moss and Muth (2011) designed to challenge the participant’s visual-vestibular interaction. This task involved participants listening to a recording that gave them a direction (left or right) and an object. Participants then made a head movement to locate the given object in the HMD. The modification created a performance task by adding physical laser targets to the objects around the room and requiring participants to shoot the target with a hand held laser. Figure 2 shows an overhead view of the room layout during the experimental task. Participants were required to look at eight distinct objects around the laboratory throughout the experiment (Figure 2, A-H). Participants stood in a predetermined location in the lab (Figure 2, X), and remained standing for the duration of the experiment. The experimenter played a recording that gave participants a direction and an object (e.g. “left, clock”). Participants made head movements to find the specified object and pointed a laser at the target. Participants were instructed to pulse the laser until the target was hit or until the next object instruction was given. The target indicated a hit by lighting up and buzzing. Participants were asked to find and shoot at a different object every three seconds. The maximum horizontal movement encompassed by stimulus arrangement was 180°. In between trials, participants were asked to look straight ahead at a target placed directly opposite them (Figure 2, E).

Figure 1. The head-mounted display used for this experiment. A ProView TM XL 50 HMD (Kaiser Electro-Optics, Inc.) was used for this experiment. The XL 50 is a bi-ocular HMD with a resolution of 1024 x 768 and a frame rate of 60 Hz. Rubber-like eye cups made specifically for the XL 50 were used to occlude external light from the environment. The HMD had a 50° field of view diagonally, 30° vertically, and 40° horizontally. It weighed 992 g. A Uniq UC-610CL color digital CCD camera was used to capture images of the real world. It was mounted atop the HMD. Resolution was 659 x 494 active pixels at a frame rate of 110 Hz. It used a lens mount platform C-mount and a 1/3” progressive scan CCD imager with R, G, and B primary color mosaic filters. The camera weighed 200 g. A Dalsa X64 CL Express™ PCI camera link frame grabber for image capture was installed on a Windows XP computer containing a 3.2 Ghz Pentium IV processor and 2 GB of RAM. A 256 Mb PCI Express™ video card was used.

Figure 2. An overhead view of the laboratory set up for the object location task. A custom computer code was created to present the object instructions automatically for the object location and targeting task. Laboratory audio was continuously monitored via a microphone during the experimental sessions. The software parsed the audio file in real-time and computed the amount of time between the announcement of the target and the activation of the buzzer indicating a hit.


RESULTS Accuracy and time-to-hit were averaged by block, resulting in five measurements per condition. The same was done for posture measures, which were calculated at 30second intervals and averaged together by block. Performance

A 2 (condition) x 5 (block) repeated measures ANOVA was performed to examine the relationship between condition and block for accuracy as measured by number of hits. There was a significant interaction between condition and trial (F (4, 19) = 5.91, p < .001). There was a significant main effect of condition, with more hits in the no HMD condition (F (1, 19) = 53.56, p < .001). There was also a significant main effect of trial (F (4, 19) = 8.25, p < .001). These findings are demonstrated in Figure 3. 40

Hits

35

NO HMD HMD

30

25 1

2

3 BLOCK

4

5

Figure 3. The relationship between number of hits (out of 40 possible) over blocks of trials comparing the two conditions. A 2 (condition) x 5 (block) repeated measures ANOVA was performed to examine the relationship between condition and block for time-to-hit in seconds. There was a significant interaction between condition and block (F (4, 19) = 2.76, p = .03). There was a significant main effect of condition, with faster reaction times in the no HMD condition (F (1, 19) = 226.99, p < .001). There was also a significant main effect of trial (F (4, 19) = 17.63, p < .001). These findings are demonstrated in Figure 4. 80 75 70

Time-to-hit (seconds)

Accuracy and time-to-hit were used to measure performance. Accuracy was calculated by counting the number of hits scored during each block of trials. Time-to-hit was calculated by summing the time before each hit, plus three seconds for each miss in each block of trials. Posture data were recorded to measure participant movement using the Polhemus Fastrak with the sensor secured to the center of participants’ backs using a Velcro strap. The Polhemus Fastrak was turned on at the beginning of the experiment and stayed on for the duration. The experimenter used a program created in house to put a marker in the data to represent different parts of the experiment (e.g. start and end of blocks of trails). Data, measured in inches, were collected at 120 Hz in both the X and the Y planes. Calculating postural sway. Four measures of posture data were calculated to examine differences in movement. Sample entropy and normalized path length were used to look at temporal complexity in postural sway and elliptical area and path length were used to look at magnitude of postural sway. Sample entropy is a unit-less, univariate measure that indicates the amount of temporal complexity, or predictability in postural sway (Richman & Moorman, 2000). This variable describes the overall structure of sway. High sample entropy is associated with high temporal complexity of sway. Normalized path length is a unit-less twodimensional measure that describes the amount of twisting and turning in sway coordination (Donker, Roerdink, Greven, & Beek, 2007; Donker, Ledebt, Roerdink, Savelsbergh, & Beek, 2008). The path length variable is normalized by dividing the x and y time series by their respective standard deviations. The variable, like sample entropy, represents the overall structure of sway. However, normalized path length indexes coordination spatial complexity rather than temporal complexity. A larger normalized path length represents more twisting and turning in the structure of postural sway, and therefore higher spatial complexity. Elliptical area is a two-dimensional measure that describes the size of the area in which postural sway takes place. This variable describes the magnitude of the sway with respect to geometric size. Its units are the square of the units in which the data are collected (e.g. inches2). A higher elliptical area indicates more movement from the participant. Path length provides information about the overall amount of postural sway. It is measured in the same units the data are collected (e.g. inches). It represents the actual length of their movement (Donker et al., 2007; Donker et al., 2008). Like elliptical area, path length is a two-dimensional variable that describes the magnitude of sway.

2064

65 60 55 50

NO HMD

45

HMD

40 35 30 25 1

2

3 BLOCK

4

5

Figure 4. The relationship between time-to-hit and trial by condition. Postural Sway A 2 (condition) x 5 (block) repeated measures ANOVA was performed to examine differences in the four postural sway measures. When looking at elliptical area there was a violation of the assumption of sphericity when looking


at an effect of block and interaction, therefore the GreenhouseGeisser statistic was used. There was a significant main effect of condition (F (1, 19) = 38.8, p < .01) and trial (F (1.9, 19) = 7.0, p = .003) for elliptical area (Figure 5). No significant interaction occurred.

Figure 5. The difference between HMD and No HMD by block on average elliptical area in inches2. When looking at path length, there was a violation of the sphericity assumption for the interaction term, therefore the Greenhouse-Geisser statistic was used. There was a significant main effect of condition (F(1, 19) = 15.3, p = .001) and trial (F(4, 19) = 4.9, p = .002) for path length (Figure 6). No significant interaction occurred. There were no significant effects for sample entropy or normalized path length.

Figure 5. The difference between HMD and No HMD by block on path length in inches. DISCUSSION The purpose of this experiment was to examine differences in performance and postural sway between participants both using and not using an HMD. Results showed a significant difference in both performance and postural sway between no HMD and HMD conditions. This supports previous research that HMD usage has negative effects on the user experience, specifically with an increased magnitude of motion and degredation of performance of certain tasks (Kennedy et al., 2010; Smart et al., 2014). During the HMD condition, participants had fewer hits and took longer to hit the targets. This finding implies that

2065

the use of an HMD hinders users’ performance. This could be due to time delays in feedback, which has been shown to be harmful to time-dependent performance tasks involving realtime tracking and feedback (Wickens & Hollands, 2000). The performance difference between the two conditions descreased towards the end of the exposure, suggesting it may take more time than the experiment allowed to adapt to the HMD. However, participants experienced 200 trials while wearing the HMD for 15 minutes, which is not a trivial amount of experience. Even if adaptation is possible, the performance decrement experienced is still relevant to users’ experience in HMDs. The time for adaptation continues to pose a limitation of the implementation of HMDs as a training tool and decreases usability of the device. Therefore, head-tracked HMD systems may not offer stable feedback necessary for targeting applications. If this is true, users and designers need to be careful about the types of tasks in which head-tracked HMDs are used. There was a significant increase in magnitude of postural sway during the HMD condition shown by both elliptical area and path length. These findings indicate that participants moved more over a larger area during the experiment when using the HMD compared to when not using the HMD. This could have implications for participants’ sickness levels as postural instability has been shown to precede sickness in some cases (Riccio & Stoffregen, 1991; Stoffregen & Smart, 1998). Additionally, changes in sway, specifically in the amount of sway may lead to less postural control. Research shows when individuals have difficulty maintaining postural control, they also have decreases in reaction time and performance when completing a cogntivie task (Teasdale & Simoneau, 2001). Other research on older adults, overweight adults, and parkinson’s patients have shown increase in falling due lack of postural control (Horak, 2006; Adkin, et al., 2003; Corbeil, et al., 2001). Due to the nature of the expeirmental task, it is not surprising that there were no significant differences between spatial or temporal complexity of postural sway. Spatial complexity measured by sample entropy examines selfsimilarity in movement (Richman & Mooreman, 2000). Temporal complexity measured by normalized path length examines participants’ twisting and turning in movement (Donker et al., 2007; Donker et al., 2008). Since the experimental task involved a series of guided head movements, it is logical that no difference was found between these measures due to participants moving in a very predictable way regardless of condition. To further examine differences in spatial and temporal complexity of sway, future studies should utilize a task that does not involve guided head movements. One limitation to the design of this study is that some participants received both conditions on the same day and some participants received them on separate days. This is a result of using data from a larger experiment. However, this was roughly counterbalanced, as 11 participants received both conditions in the same session and 9 received them in different sessions. Each session always started with a no HMD condition and ended with an HMD condition. There should be no difference in effect of learning or adaptation based on when


the particpant completed the specific conditions for this analysis because they always only saw one condition with the HMD per session. Additionally, sessions were separated by two weeks to eliminate adaptation affects between sessions. Although the sample of this study was limited, findings were strengthened by within-subjects comparisons. In the future, it could be beneficial to examine a larger group of participants to better understand differences in postural sway. Additionally, examining postural sway outside of the current paradigm may provide insight into sway differences. The predictable nature of the current paradigm may have limited changes in posture between conditions, thus masking effects in temporal and spatial complexity in postural sway. To examine at what point, if any, the gap between the two conditions is negligible, future experiments can use a longer exposure time to ensure performance decrements plateau or can be overcome. While the device used in this experiment was not a new commercial product, the delay created in this experiment was created as an equivalent to that of these devices. This finding offers a platform for measuring experience with HMDs regardless of device and this protocol can be generalized to objectively record negative effects resulting from any HMD system. Another logical next step is to analyze simulator sickness and its relationship between posture and performance in the current paradigm. There is evidence that HMDs cause user discomfort and simulator sickness (Moss, Sisco, & Muth, 2008; Moss & Muth, 2011). However, sickness is generally measured using subjective self-reported measures. Combining subjective self-reported measures of sickness with the objective measures of postural sway and performance may provide a more concrete insight into individuals’ experiences while using an HMD. Another future direction of this research is to examine if transfer of training is affected when training occurs in an HMD. Often, HMDs are used in training procedures when training on the job is too dangerous or costly (Stanney et al., 1998). It is important to understand the effect of variable latency on transfer of training to ensure a smooth and safe switch between practice and real life tasks. A simple way to do this is to have participants perform a simple task like the one described in this experiment with and without an HMD to see if any conflict results. While previous studies of this nature have focused on subjective reports of experience, postural sway, accuracy, and reaction time have objective strengths. While there are advantages to using an HMD for training such as reduced cost and resources, other factors should be taken into consideration before implementing this technology. As HMD and AR/VR devices become more commercially available, considerations such as latency and type of task should be considered early in the design process and testing should be extended to include measures of posture and performance to better understand user experience. Adding an objective measure like performance or postural sway can provide additional insights into the user’s experience while using a device such as an HMD. ACKNOWLEDGEMENTS

2066

We would like to thank the Clemson University Human Factors Institute for the grant that funded the supplies and subject payments for this study. REFERENCES Adkin, A. L., Frank, J. S., & Jog, M. S. (2003). Fear of falling and postural control in Parkinson's disease. Movement Disorders, 18(5), 496-502. Biocca, F. (1992). Will simulation sickness slow down the diffusion of virtual environment technology?. Presence: Teleoperators & Virtual Environments, 1(3), 334-343. Corbeil, P., Simoneau, M., Rancourt, D., Tremblay, A., & Teasdale, N. (2001). Increased risk for falling associated with obesity: mathematical modeling of postural control. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 9(2), 126-136. Donker, S. F., Roerdink, M., Greven, A. J., & Beek, P. J. (2007). Regularity of center-of-pressure trajectories depends on the amount of attention invested in postural control. Experimental Brain Research, 181(1), 1-11. Donker, S. F., Ledebt, A., Roerdink, M., Savelsbergh, G. J., & Beek, P. J. (2008). Children with cerebral palsy exhibit greater and more regular postural sway than typically developing children. Experimental brain research, 184(3), 363-370. Horak, F. B. (2006). Postural orientation and equilibrium: what do we need to know about neural control of balance to prevent falls?. Age and ageing, 35(suppl 2), ii7-ii11. Horak, F. B., & Macpherson, J. M. (1996). Postural equilibrium and orientation. Published for the American Physiology Society by Oxford University Press, New York, 255-292. Kennedy, R. S., Drexler, J., & Kennedy, R. C. (2010). Research in visually induced motion sickness. Applied ergonomics, 41(4), 494-503. Lewis, T. (2015, March 16). When Will Virtual-Reality Headsets Stop Making People Sick? Retrieved February 09, 2017, from http://www.nbcnews.com/id/57120561/ns/technology_and_sciencescience/t/when-will-virtual-reality-headsets-stop-making-people-sick/ MacKenzie, I. S., & Ware, C. (1993, May). Lag as a determinant of human performance in interactive systems. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems (pp. 488-493). ACM. Moss, J. D., & Muth, E. R. (2011). Characteristics of head-mounted displays and their effects on simulator sickness. Human Factors: The Journal of the Human Factors and Ergonomics Society, 53(3), 308-319. Moss, J., Scisco, J., & Muth, E. (2008, September). Simulator sickness during head mounted display (HMD) of real world video captured scenes. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 52, No. 19, pp. 1631-1634). Sage CA: Los Angeles, CA: SAGE Publications. Riccio, G. E., & Stoffregen, T. A. (1991). An ecological theory of motion sickness and postural instability. Ecological psychology, 3(3), 195-240. Richman, J. S., & Moorman, J. R. (2000). Physiological time-series analysis using approximate entropy and sample entropy. American Journal of Physiology-Heart and Circulatory Physiology, 278(6), H2039-H2049. Smart Jr, L. J., Otten, E. W., Strang, A. J., Littman, E. M., & Cook, H. E. (2014). Influence of complexity and coupling of optic flow on visually induced motion sickness. Ecological Psychology, 26(4), 301-324. Stanney, K. M., Mourant, R. R., & Kennedy, R. S. (1998). Human factors issues in virtual environments: A review of the literature. Presence: Teleoperators and Virtual Environments, 7(4), 327-351. Stoffregen, T. A., & Smart, L. J. (1998). Postural instability precedes motion sickness. Brain research bulletin, 47(5), 437-448. Teasdale, N., & Simoneau, M. (2001). Attentional demands for postural control: the effects of aging and sensory reintegration. Gait & posture, 14(3), 203-210. Watson, B., Spaulding, V., Walker, N., & Ribarsky, W. (1997, March). Evaluation of the effects of frame time variation on VR task performance. In Virtual Reality Annual International Symposium, 1997., IEEE 1997 (pp. 38-44). IEEE. Wickens, C. D., & Hollands, J. G. (2000). Attention, time-sharing, and workload. Engineering psychology and human performance, 3, 439-479.