Effects of Stereoscopic Displays and Interaction ...

Effects of Stereoscopic Displays and Interaction Devices on Human Motor Behavior Shih-Ching Yeh*, Belinda Lange**, C.Y. Chang*, Chiao Wang*, Alexander A. Sawchuk*, Albert Rizzo** *Integrated Media Systems Center, Univ. of Southern California, Los Angeles, CA, 90089-2564 **Institute for Creative Technologies, Univ. of Southern California, Marina Del Rey, CA, 90292 ABSTRACT The goal of this research is to compare the performance of different stereoscopic displays and tracking/interaction devices in the context of motor behavior and interaction quality within various Virtual Reality (VR) environments. Participants were given a series of VR tasks requiring motor behaviors with different degrees of freedom. The VR tasks were performed using a monoscopic display and two stereoscopic displays (shutter glasses and autostereoscopic display) and two tracking devices (optical and magnetic). The two 3D tracking/ interaction devices were used to capture continuous 3D spatial hand position with time stamps. Participants completed questionnaires evaluating display comfort and simulation fidelity among the three displays and the efficiency of the two interaction devices. The trajectory of motion was reconstructed from the tracking data to investigate the user’s motor behavior. Results provide information on how stereoscopic displays can affect human motor behavior and interaction modes during VR tasks. These preliminary results suggest that the use of shutter glasses provides a more immersive and user-friendly display than autostereoscopic displays. Results also suggest that the optical tracking device, available at a fraction of the cost of the magnetic tracker, provides similar results for users in terms of functionality and usability features. Keywords: Stereoscopic displays, virtual reality, motor rehabilitation

1. INTRODUCTION Personal computer (PC) and video games have been widely applied to fields of psychological treatment and motor rehabilitation using a range of interaction devices and displays. The primary strength that PC/video games offer rehabilitation is in the creation of simulated realistic environments, within which performance can be trained and measured in a systematic fashion. Games providing intensive practice and unlimited repetition with ongoing feedback have been explored as a therapeutic tool to retrain faulty movement patterns resulting from neurological dysfunction. Promising trends have been established for improving hand function and locomotor activity in individuals with chronic stroke, controlling and coordinating volitional movement for children with cerebral palsy, and decreasing akinesia for individuals with Parkinson’s disease [1,2,3,4,5]. Neurorehabilitation after stroke may include interventions designed to improve functional upper extremity (UE) skills through task-specific practice that targets specific relevant movement, is highly repetitive, and can be intensified in a hierarchical fashion based on patient progress. Virtual reality (VR) is a promising modality for the creation of optimal practice environments for neurorehabilitation [6,7,8,9,10,11]. The integration of VR technology into rehabilitation systems provides many new techniques beyond what is currently available with traditional methods. Early research suggests that the use of VR technology is valuable in improving motor skills for post-stroke rehabilitation of patients with functional deficits including reaching, hand function and walking. By designing virtual environments that not only look like the real world, but incorporate challenges that require real world functional behavior, the usability and validity of rehabilitation methods is enhanced. However, the real human performance or behavior might be biased because of the nature or limitation of various interaction devices or displays. While attempting to build an immersive virtual environment with enabling technologies composed of various display (rendering) systems, sensing systems, haptic devices or game features, the mechanism of interaction between human and computer systems moves into highly sophisticated domains. The separation of actual human performance from behavior imposed by the computer system is significantly important, especially if it is applied to people with disability or motor impairment. The Engineering Reality of Virtual Reality 2008, edited by Ian E. McDowall, Margaret Dolinsky, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 6804, 680405, © 2008 SPIE-IS&T · 0277-786X/08/$18

SPIE-IS&T/ Vol. 6804 680405-1 2008 SPIE Digital Library -- Subscriber Archive Copy

Therefore, our research goal is to compare the performance of different stereoscopic displays as well as a monoscopic displays in the context of motor behavior and interaction quality within various VR environments. Both visual and motor characteristics were studied.

2. METHODOLOGY The human subject experiment was conducted via several variables with different combinations of the input of experiment model. Various measures were extracted from collected data as the output of experiment model. The experiment model was composed of input variables and output measures (Figure 1). These variables and measures are further described in the following sections.

INPUT Display

OUTPUT

1) Autostereo Display Performance of

2) Shutter Glasses 3) Mono Display

Visual/Motor Characteristics

Interaction Device 1) Magnetic Tracking Device 2) Optical Tracking Device

1) Kinematics Human Subject Test (50)

1 DOF

2)

3 DOF

3)

6 DOF

3) Complete Speed 4) Display Comfort 5) Device Functionality

Motor-based VR Task 1)

2) Complete Rate/

6) Immersive/Presence Perception

Figure 1 Experiment model.

2.1 Input Variables The input variables in this study were defined as display type, interaction device and degree of freedom of motion in game task. Three display types were compared: monoscopic display, shutter glasses [12] and autostereoscopic display [13]. Shutter glasses and autostereoscopic displays provide the perception of stereo while the monoscopic display does not. Shutter glasses are worn just as a normal pair of eye glasses, therefore allowing the user to move their head freely without losing the stereo effect. Although autostereoscopic displays do not require the use of glasses, the user must position their head within a limited zone (the "sweet spot") in order to perceive the stereo effect on the screen. Two types of interaction devices were compared: magnetic tracking system [14] and optical tracking device (Figure 2). The magnetic tracking system, a high end motion capturing device, provides information of six degrees of freedom with a sampling rate of 120 Hz. The optical tracking device was developed by the authors and utilizes low-cost dual webcams to track multiple LEDs, providing information of six degrees of freedom with a sampling rate of 60 Hz.

SPIE-IS&T/ Vol. 6804 680405-2

78 Figure 2 Optical tracking device.

Four game tasks, developed by the authors, were used in this study: ball catching, depth test, reaching test and spatial rotation. The depth test game task required participants to compare two virtual objects that appeared on the screen in different geometry (sphere or cube), size and depth. Participants were asked to move one of the objects using the interaction device (optical or magnetic) until it was the same depth as the fixed target object. The ball catching game task required the participant to catch a ball moving towards them using a virtual hand. The trajectory and velocity of the ball was varied. The reaching test game task required participants to reach a series of cubes presented one by one in 3D space at different locations on the screen using a virtual hand. The spatial rotation game task required participants to superimpose two identical blocks that appeared with the same configuration but different orientation. One of the blocks was set as target block that was fixed. Participants moved and rotated the second block (using the optical or magnetic interaction device) to superpose it onto the target block. Each of these game tasks involved upper limb motion within different degrees of freedom (Figure 3). Name

Depth Test

Ball Catching

Reaching Test

Spatial Rotation

1

3

3

6

VR Game

Degree of Freedom

Figure 3 Demonstration of degrees of freedom of motion required for each of the Virtual Reality game tasks.

2.2 Output Measures Three types of outcome measures were collected: performance data, kinematic data and user-perception data. Kinematic data included measures of participant’s motor behavior derived from motion data captured by the interaction devices. This data was collected to quantitatively describe motor features such as efficiency, discontinuity, oscillation or stability. The kinematic data is not included in the scope of this paper. Performance data included measures of participant’s performance while playing each VR game (complete rate, complete time or error rate). User-perception data consisted of a series of questionnaires that measured participant’s perception in regard to display comfort, functionality of interaction devices, presence of VR game and system evaluation. 2.3 Data Collection Procedure Participants were included in the study if they were over 18 years of age, did not have any ongoing visual or motor deficits and consented to participating in the study. The data collection was undertaken in four phases:

SPIE-IS&T/ Vol. 6804 680405-3

2.3.1 Phase I: Preliminary Exam The purpose of the study was explained and participants were provided with the opportunity to ask questions. Participants were then invited to participate in the study. All participants agreeing to take part in the study gave their informed consent. Participants were then screened for potential vision or motor deficits. Participants were then asked to complete a background questionnaire, specifically focusing on demographic information and previous video game experience. Finally, a simple test was conducted with stereo images and anaglyph glasses to determine each participant’s ability to perceive 3D stereo effects. 2.3.2 Phase II: Visual Experiment Participants played two VR games (depth test and ball catching), using each of the displays (monoscopic, shutter glasses, autostereoscopic). A total six trials were completed in a random order for each participant. The magnetic tracking system was used for all test trials in this phase. Performance data was recorded (accuracy and time to complete). Upon completion of the trials, participants were asked to complete a questionnaire comparing each of the displays. 2.3.3 Phase III: Interaction Experiment Participants played three VR games (depth test, reaching cube and spatial rotation) using each of the two interaction devices (magnetic tracking and optical tracking). A total of six trials were completed in a random order. Shutter glasses were applied for all test trials at this phase. Upon completion of the trials, participants were asked to complete a questionnaire comparing each of the interaction devices. 2.3.4 Phase IV: Questionnaires Finally, participants were asked to complete two questionnaires; one questionnaire to determine immersive presence of the four VR games and the other questionnaire to provide a general evaluation of the system as a whole. The entire experiment procedure is presented in Figure 4.

3. RESULTS 3.1 Demographics Fifty participants (31 females, 19 males), aged 28.72 (± 5.92) years, completed the data collection procedure. Forty seven participants were right handed and 47 participants wore glasses . Only 22 percent of participants had never played video games. Fifty-eight percent of participants played video games at least once per week, 10 percent played every two to three days and four percent of participants played video games at least once per day. The average time spent playing video games per week for the whole sample was 3.46 (± 6.18) hours. 3.2 Performance Measures 3.2.1 Phase II-Display Ball Catching Each participant completed ten trials using each of the three displays (total of 30 trials). The average number of trials with successful catching was calculated for each display (Figure 5a). Although on average participants had higher catching rates when using shutter glasses, there were only small differences between the three displays. The results of forward depth at which participants caught the ball (Figure 5(b)) demonstrate that participants had a larger forward movement in depth while using shutter glasses. Participants had similar forward movement in depth while using monoscopic and autostereoscopic displays. Depth Test Each participant completed ten trials using each display. The average number of trials in which participants correctly matched the depth of the two objects was calculated for each display (Figure 5c). Participants were able to match the depth of objects quickly and more accurately using shutter glasses and autostereoscopic display. When using the monoscopic display, participants were less accurate at depth matching. In particular, trials in which participants were matching objects that were the same size and shape were matched more accurately than those objects of different sizes and shapes. Participants took an average of 2.5 seconds per trial longer to match objects when using the

SPIE-IS&T/ Vol. 6804 680405-4

Phase!

User Background Questionnaire

Depth Perception

Test.

Phase

Visual — Psych

Experiment

, Display.

Display.

Display.

Shutter Glasses.

Mono

Task.

Task.

Depth Test

Ball Catchi no.

Task. Ball Catching.

Task Depth Test.

Task. Depth Test.

Task. Ball Catching.

Display Comfort and Simulation Fidelfty Questionnaire.

Phase II!

Motor— Psych. Experiment.

Interaction Device.

lreraction Device.

FO8.

Optical Tracking* _______________

____ ____ Task.

____ ____

J.

Task. Depth Test.

Reaching Test.

Task. Mer al Rotation.

(1-DO F).

(3-DO F).

(6-DO F).

Reaching Test.

Task. Mer al Rotation

(1-DO F)

(3-DO F).

(6-DO F).

lreraction Device Functionalfty

Questionnaire.

Phase jV

Immersive Presence Questionnaire.

1 System Performance Questionnaire.

Figure 4 Experimental procedure.

SPIE-IS&T/ Vol. 6804 680405-5

Task.

S

Task . Depth Test.

Depth of Catching

Number of Successful Catch (out of 10) 3

10 8

2

6 4

1

2

0

0 Mono Display

Autostereo Display

Mono Display

Shutter Glasses

Autostereo Display

Shutter Glasses

(a)

(b)

Number of Successful Depth Matching(out of 10)

Average Time of Each Correct Depth Matching

10

12

8

9 (sec)

6 4

6 3

2

0

0 Mono Display

Autostereo Display

Shutter Glasses

Autostereo Display

Shutter Glasses

(d)

(c ) Figure 5 Results of performance measures at phase II.

autostereoscopic display compared to the shutter glass display. The average time taken for participants to match objects with stereo perception is presented in Figure 5d. 3.2.2 Phase III-Interaction Device Depth Test Each participant completed ten trials using each interaction device. The average number of trials classified as correct for the depth matching test using each interaction device was not significantly different (Figure 6a). Participants spent an average of 2.0 seconds per trial longer to match objects when using the optical tracking device compared to the magnetic tracker. (Figure 6b). Reaching Test The average total time taken for participants to reach 20 cubes successfully was calculated (Figure 6c), and did not differ significantly between optical and magnetic trackers. Spatial Rotation The average time for participants to rotate and superpose eight pairs of spatial blocks using two interaction devices was 10 seconds longer when using the optical tracker compared to the magnetic tracker (Figure 6d). 3.3 User-Perception Measure 3.3.1 Phase II- Display The shutter glasses display had consistently higher positive scores on the user-perception questionnaires (Table 1). Participants scored 3D and depth easier to perceive when using shutter glasses, although the autostereoscopic display provided better perception of these factors than the monoscopic display. The autostereoscopic display was associated with greater reports of eye strain, discomfort and greater difficulty concentrating on the VR task. Participants reported the depth test was easier to complete when using shutter glasses compared to the autostereoscopic display.

SPIE-IS&T/ Vol. 6804 680405-6

Average Time of Each Correct Depth Matching

Number of Successful Depth Matching(out of 10) 10

12

8

9 (sec)

6 4

6 3

2

0

0 Optical Tracker

Optical Tracker

Magnetic Tracker

(a)

(b)

Total Time

Total Time

50

50

40

40

30

30

20

20

10

10

0 Optical Tracker

Magnetic Tracker

0

Magnetic Tracker

Optical Tracker

(c)

Magnetic Tracker

(d) Figure 6 Results of performance measures for phase III

Table 1. Median scores for questionnaire comparing the three display types.

I could easily see or perceive the 3D/stereo effect with this display.

Median Score (1= strongly disagree – 7= strongly agree) Monoscopic Shutter Autostereoscopic Display Glasses Display 4.0 6.0 5.0

I could easily tell the depth difference between objects with this display.

3.0

6.0

5.0

I felt discomfort while using this display.

2.0

2.0

4.0

I felt eye-strain while using this display.

2.0

3.0

5.5

I found it difficult to concentrate while playing the game tasks when using this display.

2.0

2.0

4.5

I could complete the “Depth ” game task satisfactorily when this display was provided.

3.0

5.5

5.0

I could complete the “Ball Catching” game task satisfactorily when this display was provided.

5.0

6.0

5.0

Overall: I was very satisfied with this display while playing each of the game tasks.

5.0

6.0

4.5

SPIE-IS&T/ Vol. 6804 680405-7

4.3.2 Phase III- Interaction device Overall magnetic tracking appeared to score higher than the optical tracking, however both devices scored well. Table 2. Median scores for questionnaire comparing the two interaction types.

How well could you move or manipulate objects in the games?

Median Score (1= not at all – 7= very much) Magnetic Optical 6.0 5.0

How sensitive was the motion or movement while you were manipulating objects in the games?

6.0

5.5

Were you able to anticipate what would happen next in response to the actions that you performed?

6.0

5.0

How much delay did you experience between your actions and expected outcomes?

2.0

3.0

How easily did you adjust to get used to the interaction devices while playing the games?

6.0

5.0

How well did you feel you could move and interact with the games by the end of the experience?

6.0

5.0

I could complete the “Depth ” game task satisfactorily when this device was provided.

5.0

4.0

I could complete the “Reach Cube” game task satisfactorily when this display was provided.

6.0

6.0

I could complete the “Blocks” game task satisfactorily when this display was provided.

6.5

5.0

3.3.3 Phase IV- System evaluation and presence Participants provided positive scores for the system as a whole (all display and interaction devices and VR tasks) in terms of comfort and satisfaction, ease of use and being able to complete the tasks effectively and efficiently. The Reaching test scored the highest in terms of immersion and presence, however, Ball Catching and Spatial Rotation scored consistently well. The Depth test was the least popular VR task (Table 3).

SPIE-IS&T/ Vol. 6804 680405-8

Table 3. Median scores for questionnaire comparing the immersion/ presence of the interaction VR game tasks Median Score (1= strongly disagree – 7= strongly agree) Ball Catching

Depth Test

Spatial Rotation

Reaching Test

How natural did your interactions with the games seem?

6.0

4.0

6.0

6.0

How much did the auditory aspects of the games involve you?

5.0

3.0

4.0

5.0

How natural was the mechanism which you can control object’s movement through each interactive device here?

6.0

5.0

6.0

6.0

How much did your experiences in the games seem consistent with your real world experiences?

5.0

4.0

6.0

6.0

How compelling was your sense of moving around inside the games?

5.0

5.0

6.0

6.0

How involved were you in the game experiences?

6.0

5.5

6.5

6.0

How completely were your senses engaged in the game experiences?

6.0

6.0

6.0

6.0

Overall, how much did you focus on using the display and control devices instead of the experimental game tasks?

4.0

4.0

4.0

4.0

How easy was it to identify objects through physical interaction: like touching an object or bumping into an object?

5.0

4.5

6.0

6.0

4. CONCLUSIONS This study compared three display types and two interaction devices during four VR tasks. Participants were able to complete the ball catching and depth test game tasks using each of the three displays. Participants were able to complete the ball catching and depth test tasks faster when using shutter glasses than the other two displays. This might have been the result of the ability of the participant to see the ball clearly in 3D, allowing them to perceive the depth of the ball and anticipate the trajectory of ball movement. Possible reasons the autostereoscopic display did not appear to provide participants with the same comfort and effectiveness as the shutter glasses could be the result of the physical characteristics of the autostereoscopic display, such as field of view and resolution. In addition, when using the autostereoscopic display, participants could have had difficulty maintaining a good 3D stereo picture when playing VR game tasks due to of the requirement of having to keep head movements within the limited area ("sweet spot") where 3D stereo can be seen. Participants rated the autostereoscopic display highest for discomfort and eye strain and least satisfactory overall. The results suggest that the optical tracker performs as well as magnetic tracking system for game tasks requiring motion within three degrees of freedom. However, participants were slower completing game tasks requiring motion within six degrees of freedom when using the optical tracker. Participants took a longer time and noted more difficulty in completing the depth test task when using the optical tracker. Overall, the optical and magnetic tracking devices scored highly in the user perception questionnaire for all tasks. The results of this study provide information about the use of different display types and interaction devices with four VR games. The shutter glasses display was rated higher than monoscopic and autostereoscopic displays for comfort, ability to perceive 3D, performance and time to complete. The optical tracking device, developed by the authors, is less expensive than the magnetic tracking. The two devices performed well. The results suggest that the use of shutter glasses and magnetic or optical tracking are likely to be the most appropriate display and interaction devices for use in VR environments used as rehabilitation tools, given the current state of technology.

SPIE-IS&T/ Vol. 6804 680405-9

REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

Merians, A.S., D. Jack, et al., "Virtual reality-augmented rehabilitation for patients following stroke," Phys Ther 82(9), 898-915 (2002). Kuttuva, M., R. Boian, et al., "The Rutgers Arm, a rehabilitation system in virtual reality: a pilot study," Cyberpsychol Behav 9(2), 148-51 (2006). Tobis, M.J. and B. Azarbal, "Does patent foramen ovale promote cryptogenic stroke and migraine headache?" Tex Heart Inst J 32(3), 362-365 (2005). Bryanton, C., J. Bosse, et al., "Feasibility, motivation, and selective motor control: virtual reality compared to conventional home exercise in children with cerebral palsy," Cyberpsychol Behav 9(2), 123-128 (2006). Ferrarin, M., M. Brambilla, et al., "Microprocessor-controlled optical stimulating device to improve the gait of patients with Parkinson's disease," Med Biol Eng Comput 42(3), 328-332, (2004). Holden M.K, “Virtual environments for motor rehabilitation: review,” Cyberpsychol Behav, 8, 187-211 (2005). Weiss P.L., Katz N, “The potential of virtual reality for rehabilitation,” J Rehabil Res Dev, 41, vii-x (2004). Holden M., Todorov, E., Callahan J., Bizzi E., “Virtual environment training improves motor performance in two patients with stroke: case report,” Neurol Rep, 23, 57-67 (1999). Merians A.S., Jack D., Boian R., Tremaine M., Burdea G.C., Adamovich SV, Recce M., Poizner H., “Virtual reality-augmented rehabilitation for patients following stroke,” Phys Ther, 82, 898-915 (2002). You S.H., Jang S.H., Kim Y.H., Hallett M., Ahn S.H., Kwon Y.H., Kim J.H., Lee M.Y., "Virtual reality-induced cortical reorganization and associated locomotor recovery in chronic stroke. An experimenter-blind randomized study,” Stroke , 36, 1166-1171 (2005). Deutsch J.E., Latonio J., Burdea G., Boian R., “Post-stroke rehabilitation with the Rutgers Ankle System: a case study,” Presence, 10, 416-430 (2001). http://www.reald-corporate.com/scientific/crystaleyes.asp. http://findarticles.com/p/articles/mi_m0BLL/is_12_21/ai_n9533600. http://www.ascension-tech.com/products/flockofbirds.php.

SPIE-IS&T/ Vol. 6804 680405-10