Virtual Multi-Tools for Hand and Tool-Based Interaction with Life-Size ...

Virtual Multi-Tools for Hand and Tool-Based Interaction with Life-Size Virtual Human Agents Aaron Kotranza

*

†

Kyle Johnsen

University of Florida

*

Juan Cendan

A common approach when simulating face-to-face interpersonal scenarios with virtual humans is to afford users only verbal interaction while providing rich verbal and non-verbal interaction from the virtual human. This is due to the difficulty in providing robust recognition of user non-verbal behavior and interpretation of these behaviors within the context of the verbal interaction between user and virtual human. To afford robust hand and tool-based non-verbal interaction with life-sized virtual humans, we propose virtual multi-tools. A single hand-held, tracked interaction device acts as a surrogate for the virtual multitools: the user's hand, multiple tools, and other objects. By combining six degree-of-freedom, high update rate tracking with extra degrees of freedom provided by buttons and triggers, a commodity device, the Nintendo Wii Remote, provides the kinesthetic and haptic feedback necessary to provide a highfidelity estimation of the natural, unencumbered interaction provided by one’s hands and physical hand-held tools. These qualities allow virtual multi-tools to be a less error-prone interface to social and task-oriented non-verbal interaction with a life-sized virtual human. This paper discusses the implementation of virtual multi-tools for hand and tool-based interaction with life-sized virtual humans, and provides an initial evaluation of the usability of virtual multi-tools in the medical education scenario of conducting a neurological exam of a virtual human. virtual

humans,

3D

interaction,

tool-based

INDEX TERMS: I.3.6 [Computer Graphics]: Methodology and Techniques—Interaction techniques; H.5.2 [Information Interfaces and Presentation]: User Interfaces— Evaluation/methodology 1

Bayard Miller

University of Georgia

ABSTRACT

KEYWORDS: interaction

*

†

*

‡

D. Scott Lind

Benjamin Lok

Medical College of Georgia

*

‡

In addition to enabling complex social and task-oriented interaction with VHs, the VMT technique may mitigate the derogatory effects of noisy-interfaces (e.g. error-prone speech recognition and understanding, which contributes to fractured conversations) typically employed in natural interaction with VHs. User satisfaction with these noisy, error-prone interfaces may be enhanced by providing a less error-prone non-verbal interface that both necessitates the use of the verbal interface and potentially reduces the overall noise perceived by the user. The VMT technique uses a single, high degree-of-freedom, tracked, one-handed interaction device to enable high-fidelity, non-verbal interaction (for bi-manual interaction, two devices may be used in conjunction). This interaction device allows a user to control his avatar's virtual hand and any number of virtual handheld tools (VMTs). In general, the VMT technique provides: (1) Correct kinesthetic information – the tracked device is held and manipulated using the same muscle movements as the physical tool being simulated. (2) Passive haptic feedback – from gripping an interface similar to the shape of many hand-held tools. (3) Force-feedback – by providing vibratory force-feedback when VMTs collide with the virtual world. (4) Tool Control and Hand Gestures – Beyond 6 degree-offreedom rigid pose tracking, the device can control other aspects of the tools (e.g. the articulation of a pair of surgical scissors; turning the light of an ophthalmoscope on or off), and the virtual hand can form a variety of poses (e.g. grip, point, lie flat). (5) Automatic adaptation of user input – by scaling and remapping users' manipulation of the interaction device specific to the VMT being used (e.g. fine movement control for a surgical knife, gross control for the user's hand).

INTRODUCTION

Virtual environments inhabited by virtual human (VH) agents have gained traction as an interpersonal skills training and therapy tool (e.g. doctor-patient interaction training, post-traumatic stress disorder and fear of public speaking therapy) [11][21][22]. However, simulating social scenarios requiring extensive verbal and nonverbal bi-directional communication with a VH agent is still a major challenge. While the expressive power of VHs is fast approaching that of real humans, natural and robust recognition and interpretation of complex user actions has been an elusive goal. This work presents a potential solution to this problem, the virtual multi-tool (VMT) technique. The VMT technique leverages a single, robustly-tracked hand-held interaction device to manipulate multiple virtual tools. This paper describes the application of the VMT technique to provide complex hand and hand-held tool based communication with VHs (Figure 1). email: *{akotranz, lok}@cise.ufl.edu, [email protected], [email protected]; †[email protected]; ‡[email protected]

Figure 1. A medical student examines a virtual human patient’s retina using an ophthalmoscope virtual multi-tool, tracked by an external infrared camera mounted below the plasma and by the internal infrared camera in the wireless hand-held interaction device, the Nintendo Wiimote.

IEEE Symposium on 3D User Interfaces 2009 14-15 March, Lafayette, Louisiana, USA 978-1-4244-3812-9/09/$25.00 ©2009 IEEE

23

Similar techniques to the VMT technique are widely known. Virtual reality systems have included tracked hand-held interaction devices to enable 3D manipulation for many years (e.g. [7]). Also, nearly all of the video games for a commercial video game console, the Nintendo Wii, leverage a similar technique by virtue of the standard, tracked, one-handed controller for the system, the Nintendo Wii Remote (“Wiimote”). In fact, this work utilizes the Nintendo Wiimote, augmented by external infrared tracking, as the hand-held interaction device. The purpose and novelty of this work is the application of the virtual multi-tool technique to human-VH interaction. When used for human-VH interaction, the VMT technique has benefits beyond the simple use of hands and tools. VMTs: (1) Form a common ground of communication between the human and VH. The VMT is a grounding object, which provides an unambiguous source of nonverbal information to the VH. (2) Mitigate the effect of error-prone speech recognition. By providing a less error-prone interface for nonverbal communication, the VMTs may offset user dissatisfaction with the error-prone speech interface – improving overall user satisfaction with the VH experience. (3) Enable virtual interpersonal touch. VMTs provide the user with a virtual hand that can touch the VH to provide virtual interpersonal touch. The tracked interaction device provides force-feedback when these touches occur. The VMT technique can be used to enable virtual interpersonal touch of a life-size VH agent. The VMT interaction device can control a virtual hand that can perform explicit tasks (e.g. shaking hands), or satisfy social norms (e.g. providing a comforting touch on the shoulder). The vibratory force-feedback of the interaction device is expected to provide the sensation of touching the VH with the hand or tool.

Figure 2. A virtual human with physiologically accurate eye movement provides medical students with exposure to abnormal medical conditions such as cranial nerve palsy.

1.1

Motivation: Physical Examinations of Virtual Patients with Abnormal Findings We apply the VMT technique to human-VH interactions to enable physical examinations of VH patients with abnormal findings, for use in medical education. Current medical education curriculums do not provide students with planned hands-on exposure to rare conditions which traditional education approaches do not simulate (i.e. abnormal findings). Student's experiences with a specific abnormal finding are thus limited, as these experiences occur rarely, and are often purely by-chance encounters with real patients in clinical

24

rotations. Missing these experiences negatively impacts medical student education, diagnosis skills, and resulting patient care -students graduate without receiving enough hands-on training necessary to correctly diagnose and treat these abnormal conditions. VH experiences in which medical students examine virtual patients simulating abnormal conditions will help to fulfill this important educational need. One such abnormal condition is a vision disorder due to cranial nerve palsy (Figure 2). We have developed a VH agent to simulate this scenario and have applied the VMT technique to allow medical students to practice neurological examination of this VH patient (Sections 3 and 4). 2

PREVIOUS WORK

2.1

Hand and Tool-Based Interaction Techniques for Virtual Environments Hand and tool-based interaction in virtual environments (VEs) has commonly been provided by encumbered glove-based interfaces (e.g. [4]). As an unencumbered alternative, vision-based tracking of the user's hands has afforded hand and gesture based interaction in VEs [15][20]. The amount and complexity of recognizable gestures (e.g. point, pinch, grasp) is suitable for simple gesturing interfaces such as those used for VE navigation, but limits the applicability of this technique to social and complex task-oriented interaction with VHs. The VMT technique provides an alternative, more robust (less noisy than vision-based tracking) technique for complex hand and tool-based interaction with a VH agent. Tangible user interfaces (TUIs) are another approach to providing hand and tool-based interaction in VEs (e.g. for engineering design [25] and neurosurgery planning [9]). TUI advantages include direct manipulation, intuitiveness, and tactile feedback [9]. One drawback of TUIs is that, because the physical interface is closely bound to the specific tasks which it enables, the tangible interface can not be overloaded for multiple tasks. This motivates the creation of a new TUI each time a new virtual task (or tool) is required. The single hand-held interaction device of VMTs takes a contrasting Swiss-army-knife approach, affording multiple hand-held tools to be manipulated using a single interface which is representative of a wide class of hand-held tools. 2.2 Interaction Techniques for Virtual Humans Interaction between humans and life-sized VH agents has primarily been limited to speech and simple gestures (e.g. pointing, iconic gestures) [11][5]. As many social scenarios involve interpersonal touch, tangible interfaces have augmented VHs to provide communication through touch between human and VH [14]. Non-social haptic interaction with smaller-thanlife-size VHs has been enabled using an active-haptic Phantom Omni [1]. The VMT interface provides vibration force-feedback, enabling virtual interpersonal social and task-oriented touching of a VH agent. In specific scenarios, passive haptic props have been used to interact with VHs in a manner specific to the prop, e.g. to play catch with a VH [10] and play a game of checkers against a VH [2]. Virtual multi-tools extend such prop-based interfaces by allowing a single tracked interaction device to act as a surrogate to the user’s hands and multiple tools. 2.3 Simulation of Medical Physical Examinations The medical simulation community has provided many simulations of physical exams, for the training and evaluation of students' physical exam skills. These often commercially

available simulators take the approach of providing physical human-form mannequins or robotics to simulate, e.g. cardiac conditions [13], eye exams [17], and a variety of pathologies [3]. These physical, mannequin-based simulators do an excellent job of educating physical examination skill, however they do not accurately represent the clinical context of patient interaction. Physical simulators do not afford communication between doctor and patient, and do not provide a dynamic representation of the patient (e.g. the mannequin patient can not change its appearance to express pain, and can not ask the doctor questions). To provide a more complete patient encounter, a hybrid mannequin-virtual human patient simulator, a mixed-reality human, was introduced for training clinical breast examination technique [14]. Applying the VMT technique to a life-size VH simulation capitalizes on the enhanced simulation of clinical context and doctor-patient communication provided by mixed reality humans, while providing advantages of a more virtual-oriented solution (e.g. enhanced portability by limiting the physical interface to a hand-held interaction device).

Figure 3. Mapping of tracked interface manipulation to VMT movement and functionality.

3

VMTS FOR CONDUCTING NEUROLOGICAL EXAMINATIONS

We apply the VMT technique to enable conducting a neurological examination of a VH. A neurological examination consists of conversing with a patient to assess the patient’s medical history and current complaint (a patient interview), followed by performing a physical examination of the patient. The physical examination involves extensive use of speech communication in concert with use of hand gestures and handheld tools (e.g. an ophthalmoscope). Because a neurological examination requires speech, hand, and tool-based interaction, it is ideally suited to serve as a test-bed for the application of the VMT technique to interaction with VHs. The virtual neurological examination requires performing the tasks shown in Table 1 with the VH patient. The patient interview portion provides only speech interaction, and uses a system which has been discussed extensively in work on communication skills training with life-size VHs [11][12]. The physical examination tasks make extensive use of both speech interaction and tool interaction. The goal for VMTs is to enable performing these tasks efficiently and accurately.

Table 1. Tasks of a neurological exam.

User action Conduct patient interview

Test

pupillary

history

reflex

Visually examine fundus (rear of inside of eye, e.g. retina) Hold fingers up and ask how many fingers the patient sees Have the patient to follow a moving object Move index finger in a 180o arc in front of the patient's eyes Have the patient read from an eye chart Shake finger in peripheral vision while patient looks straight forward Ask the patient to blink or wink his eyes

Information gained Determines what present illnesses, medication, social, family, and sexual history may be involved in the current neurological problem Checks for pupil abnormalities Checks for intracranial pressure Tests patient's binocular vision Tests for limitation of movement of one or both eyes Tests the patient's field of view and eye movement Tests the patient's visual acuity Tests for peripheral vision disorders Tests for ptosis, drooping of an eyelid.

3.1 Implementing Virtual Multi-tools The central idea behind the virtual multi-tool is that a single interaction device is used to control multiple different virtual tools (one at a time). To enable this, each VMT specifies a mapping between interaction device input and the tool functionality (Figure 3). When the tool is activated, it uses the data available from the interaction device to control its movement and functionality. For example, an ophthalmoscope (a device for examining the inside of an eye) needs fine grain control in a hemisphere encompassing the front (ventral) half of the head, while a virtual hand tool needs to move in a larger volume relative to the user’s body position. An ophthalmoscope has a light that needs to be turned on and off, and a hand needs to be changed to different postures (e.g. pointing). Implementation of the VMT technique requires an interaction device that supports the desired input dimensions and provides passive haptics and appropriate kinesthetic information. The interaction device used for the neurological examination simulator, the Nintendo Wiimote, is a powerful wireless input device. The Wiimote was chosen because it is shaped as a handheld tool, and because it has high degree of freedom control. It features an array of integrated sensors that are reported at 100Hz update rate over a Bluetooth radio: 11 buttons, 3 orthogonal accelerometers (+/- 3g), and a 45-degree field-of-view infrared camera (128x96) that tracks up to 4 points at 1/8 to 1 pixel resolution depending on the size of the infrared point. In addition, the Wiimote can display information through integrated LEDs, a low-quality speaker, and a vibration motor. Open source software was used to acquire data from the Wiimote across the Bluetooth interface [18]. A summary of the Wiimote's potential as an interaction device and descriptions of many applications can be found in the work of Lee [16].

25

For a virtual neurological examination, each aspect of the VMT that needs to be controlled can be mapped to the following information, updated at 100Hz: • The state {on, off} of each of the 11 buttons on the Wiimote • The value of each onboard accelerometer {x,y,z} • The pitch (about the x-axis) and roll (about the z-axis) of the Wiimote, derived from the onboard accelerometers, in the 3-dimensional Cartesian coordinate space formed by the axis of gravity (y-axis), normal vector to the screen plane (z-axis), and the cross product of the two (x-axis). This information is only updated when the magnitude of the accelerometer vector is approximately 1g, indicating constant velocity. • The yaw (between -PI/2 and PI/2) of the Wiimote, as detected by the angle formed by two detected infrared points relative to the vertical axis of the onboard Wiimote camera. • The center-of-mass {x,y} of the tracked points detected by the onboard Wiimote camera, relative to the camera center, and normalized between -1 and +1. • The center-of-mass {x,y} of the tracked points detected in an external camera, facing the user and placed on the top of the VH display device, normalized between -1 and +1. To facilitate camera tracking for the scenario of a virtual neurological examination, two infrared LEDs were placed on the ceiling directly above the expected position of the user (for detection by the onboard Wiimote camera). In addition, a strip of reflective tape was placed on the back of the Wiimote, near the onboard camera. This allowed the external camera to track the position of the Wiimote. The external camera used for the simulator is a Natural Point Optitrack V100 (640x480 resolution) which automatically performs blob tracking at 100Hz. While most simulation inputs mapped to Wiimote button presses are specific to the current VMT selected, a mechanism needed to be included to switch tools. We provide two methods to switch tools. The user can press the 'A' button on the Wiimote to switch the currently selected VMT. Alternatively, and perhaps more exciting, a "throwing over the shoulder" gesture [19] can be performed with the Wiimote to virtually toss away the old tool and grab the new tool. Note: We recognize that this control scheme could be greatly simplified and likely improved by augmenting the Wiimote with a 6DoF tracking device such as an electromagnetic tracker or designing a new device with more integrated sensors. However, there is "no silver bullet" for tracking, and thus every solution will have its own set of limitations (such as interference for an electromagnetic tracker) [26]. Our tracking approach has merit in that it is easily and inexpensively reproduced using readily available equipment. Widely available wireless sensor bars for the Nintendo Wii can be used for the LEDs, and commodity webcameras (such as those commonly found on laptops) can be combined with simple colored blob tracking for the external camera tracking, or a second Wiimote can be used as the external camera. Thus the total cost of the tracking configuration could be as low as $55 (USD) – $40 for the Wiimote and $15 for the sensor bar. In addition, with only minimal infrastructure additions, this tracking scheme can be deployed to the more than 30 million existing Wii systems [16]. 3.2

VMT Tools for the Neurological Examination Tasks To simulate a neurological examination of a VH patient with a cranial nerve disorder, we have implemented three VMTs: an ophthalmoscope, the user’s right hand and fingers, and an eye chart. The three tools allow a user to conduct all components of a

26

neurological examination of a patient with an eye ailment (Table 1). We discuss how each of the VMTs is mapped to the interaction device, how the VMT technique enables each task.

Figure 4. (Left) The virtual ophthalmoscope held in the user’s virtual hand. (Right) By shining the light of the virtual ophthalmoscope into the patient’s eyes, the right eye constricts, while the damaged left eye remains dilated.

3.2.1 Ophthalmoscope An ophthalmoscope is a hand-held tool equipped with a lens for viewing the back of the inside of the VH patient's eye (the fundus), to determine the health of the patient's retina (Figure 1). It is equipped with a light to illuminate the inside of the eye and for examining the pupillary reflex (Figure 4). The virtual ophthalmoscope is manipulated with 3dof position and 1dof orientation. The position {x,y,z} is mapped to {External_Camera:x, External_Camera:y, Onboard_Camera:y}. The rotation of the ophthalmoscope (about it's long axis) is mapped to Onboard_Camera:Yaw. The ophthalmoscope size, shape, and weight are closely approximated by the Wiimote interaction device, providing passive haptic feedback. Force feedback is provided when the VMT ophthalmoscope makes contact with the VH patient. Pupillary Reflex Test: The "trigger" button on the rear of the Wiimote is used to turn the ophthalmoscope's light on and off. The light direction can be changed by twisting the Wiimote, so that the user can get a clear view of the pupillary response. Fundascopic Test: When the ophthalmoscope is moved close to the patients eye (