Designing and Building the PIT: a Head-Tracked Stereo ... - CiteSeerX

Designing and Building the PIT: a Head-Tracked Stereo Workspace for Two Users Kevin Arthur, Timothy Preston, Russell M. Taylor II, Frederick P. Brooks, Jr., Mary C. Whitton, William V. Wright

Department of Computer Science University of North Carolina at Chapel Hill Sitterson Hall, Chapel Hill, NC 27599-3175 Additional information is available at: http://www.cs.unc.edu/Research/graphics/GRIP/PIT.html

Abstract The PIT (“Protein Interactive Theater”) is a dual-screen, stereo display system and workspace designed specifically for two-person, seated, local collaboration. Each user has a correct head-tracked stereo view of a shared virtual space containing a 3D model under study. The model appears to occupy the same location in lab space for each user, allowing them to augment their verbal communication with gestures. This paper describes our motivation and goals in designing the PIT workspace, the design and fabrication of the hardware and software components, and initial user testing.

Motivation and design goals The PIT display system is the most recent of a series of displays used by the GRIP molecular graphics project at UNC [Brooks et al., 1990]. While we intend for the PIT to be generally applicable to other application domains, our initial use is in molecular graphics applications such as protein fitting. The PIT design is motivated by observations gathered over the years by collaborating with biochemists to develop molecular graphics systems. The following were our primary goals in developing the PIT. High quality 3D display. To provide strong 3D depth cues, we employ the technique of head-tracked stereo display. For each user, a stereo image is displayed and updated in real time according to the perspective projection determined by the positions of the user’s eyes. The stereo and motion parallax cues give users the illusion of a stable 3D scene located in front of them and fixed in laboratory space. The users wear LCD shutter glasses and tracking sensors. High-resolution images (four images per frame, each rendered at 1280 × 492 pixels) are displayed on two large rear-projection screens oriented at 90 degrees to each other, with one screen corresponding to each user. Rendering is performed on a Silicon Graphics Onyx workstation with InfiniteReality™ graphics. We allow for decoupling the application’s display and simulation loops so that they run as separate processes, in order to maintain a high display update rate that is independent of the complexity of computations the application may be performing. The display screens may also be oriented at 120 degrees to each other for applications desiring a highresolution, wide field-of-view panoramic view for a single user. Including a second user. Over the years, we have observed that our users, who are biochemists and other scientists, quite often work in pairs to conduct their experiments. To allow close collaboration between the two users we wanted to provide the second user with a view equal in quality and realism to

Page 1 of 10

that given to the first user. In our past work we found that using a single head-mounted display for the first user and a monitor view for the second user was unsatisfactory; the monitor viewer found it difficult to understand the scene. The situation is improved somewhat with large-screen head-tracked displays such as the CAVE™ [Cruz-Neira et al., 1993], but, because the perspective images are correct for only the first user, the second user will still see an inaccurate view unless he or she makes an effort to follow the other’s movements. More seriously, the second user’s view jumps around when the first user moves. This limits that user’s ability to interact with the virtual scene and to collaborate actively with the other user. For the PIT display, we decided to provide two head-tracked stereo views, so that each user has an independent view of the virtual scene under study, and may optionally apply their own custom viewing and model manipulations. To create two independent stereo views without perceptible flicker on a single screen using LCD shutter glasses requires a refresh rate higher than that available with current commercially available projectors and shutter glasses. This drove our decision to have two projectors and opened the question of whether the two screens should be side-by-side or in some other configuration. We decided to use two projection screens oriented at 90° to each other, and to display one user’s view on each screen. The display gives the illusion of a stable 3D structure fixed in lab space in front of the two seated users. Because they see the displayed structure in the same position in physical space, they may physically point to the same places in the model with their hands. The users may work with their two instances of the virtual scene registered in this way, or they may alter the view according to their own preference. For example, a toggle button is provided for either user to rotate his view by 90° so that both users are viewing the model along the same axis, as if they were both seated at the same virtual position, looking in the same direction. In this case they use virtual icons to point to the scene. We have optimized the display for local collaboration, where the two users are seated next to one another in the same physical place, rather than for distance or tele-collaboration. Additional users can stand behind the two users and view the screens with stereo glasses (without head tracking).

Figure 1. Two users interacting in the PIT.

Page 2 of 10

Access to common devices. Over the years, our chemist collaborators have preferred to use multiple physical input devices such as dials and buttons rather than use modal or overloaded 2D interfaces. We wanted to provide a 3D display that would allow the user access to a lab notebook, a keyboard, a small display for text and GUI elements, physical dials and buttons, and other common devices. This led us away from opaque head-mounted displays, and towards head-tracked stereo in an environment that included space for a table, portable PC’s, and other equipment. Input devices may be shared or user-specific. Our intention is to provide devices that are appropriate and natural for the tasks that need to be performed without restricting the users to particular modes of 3D or 2D interaction. Each user has a laptop PC for accessing 2D graphical user interface elements, for reading experiment plans, and for making lab notebook entries based on observations. Two tracked 6 degree-of-freedom handheld controllers provide pointing, picking, and other scene manipulations. Physical dials and buttons are used for shared interactions such as repositioning the displayed structure and controlling PIT display modes. A SensAble Technologies PHANToM arm is available for force-feedback display. Table-top workspace within arm’s reach. We found in our earlier experiments using head-mounted displays for molecular graphics applications that users did not choose to fly around or walk around “room-filling” molecules. The PIT provides a more comfortable environment where the user can sit down and easily reach and manipulate the model under study. Its size can be made similar to the 2 cm/Å scale of the familiar brass models that chemists have historically used.

Related work Systems employing head-tracked stereo on multiple large screens for a single tracked user, such as the CAVE, have been actively used in recent years [Cruz-Neira et al., 1993; Deering, 1993]. Single-screen head-tracked “workbench” style displays have also been widely reported [Agrawala et al., 1997; Czernuszenko et al, 1997; Grant et al., 1998]. Agrawala and others [1997] extended the Responsive Workbench display to allow for two tracked users viewing a single screen. They made custom modifications to their shutter glasses and video system to multiplex the four views on a single projector. Extensions to CAVE systems to allow for multiple tracked users are reported in on-line CAVE documentation maintained by the Electronic Visualization Laboratory at UIC. Others have used headmounted displays for multi-user immersive virtual reality or augmented reality systems [Blanchard, 1990; Szalavari, 1998].

Hardware components Figure 2 shows a block diagram of the hardware components of the PIT workspace. Image generation. We use a Silicon Graphics Onyx2 workstation with Infinite Reality graphics to generate two video output channels of 1280 × 1024 pixels each. The video is displayed in a rearprojection arrangement using two AmPro 3600 projectors positioned to give their smallest focussed image size. A StereoGraphics CrystalEyes box doubles the vertical refresh rate of the projectors to provide stereo images synchronized to LCD shutter glasses. Each eye’s view is rendered into a region in the frame buffer of 1280 × 492 pixels [StereoGraphics, 1997].

Page 3 of 10

Figure 2. PIT hardware components.

In order to do geometrically correct head-tracked stereo, we must accurately measure the physical positions of the screen corners. We have done this manually, and have also used a theodolite surveying device to obtain more accurate measurements [Grant et al., 1998]. We have found that we also need to retune the projectors frequently to maintain alignment of the displayed video with the physical screen corners and minimize other distortions. We also require accurate measurements of the tracker sensor-tohead position transformation, and of the users’ inter-pupillary distance (IPD). We allow the users to adjust the software’s IPD value at runtime using dials. Screen construction. Figures 3 and 4 show the PIT screen material and frame used for our initial implementation of the system. The screen surfaces are each 3 ft. high by 4 ft. wide. We chose a screen size that would provide an adequate field of view for a seated user with line of sight intersecting the center of the screen, as constrained by the smallest focussed image we could get from the projectors. Screen material. We chose a vinyl screen material according to its brightness and gain properties. We wished to maximize image brightness across the range of normal head positions, without brightness falloff being noticeable, and without serious inter-reflection between the two screens. A single sheet is used across the two screen areas and is tensioned to make both surfaces vertical and flat. A Naugahyde fabric border is stitched around the outside of the screen, and behind the corner seam. The border contains grommet holes for tying the screen to the external frame, as well as loops through the material to contain a rod to help keep the screen material flat. The material is gathered at the corner, between the two

Page 4 of 10

sides of the screen, and re-sealed. A loop in the border behind it is used for support. We desired that the inside seam at the corner be straight and not allow light leakage across the two screens. We also required that it be of minimal thickness so as to give the impression of a continuous display surface with no visible line at the seam when the screens are in a 120° panoramic display configuration. In our first screen the corner seam was heat-sealed. In our second, the seam was stitched. Stitching produced a more durable seam that was less susceptible to tearing, but we found its visual appearance to be worse. The stitched seam wasn’t as straight as the heat-sealed seam, and light leakage was visible at the holes created by the stitching. Screen frame and projectors. We required that the frame that holds the screens be made of a non-ferrous material so that its presence would not interfere with the electromagnetic tracker. We chose to use 4-inch PVC pipe for this reason, and for its light weight, rigidity and convenience. Holes are spaced along the outside of the frame for attaching the screen with adjustable elastic bungee cords passing through the grommets. The projectors are fixed to the floor behind the screens on mobile wooden frames that have adjustable bolts for leveling the projectors and adjusting their height. Screen hinge. The corner of the screen frame is hinged to allow the screens to be placed at angles of 90° and 120°. We fix the screens and projectors in place using pins through holes in the screen frame, projector mounts, and the lab floor. The left screen and projector are fixed at all times and the right screen and projector can be easily moved between the two positions and secured to the floor with the pins. To maintain a straight seam at the corner between the two screens we place a vertical glass rod through a loop in the screen border material. Lessons learned. Were we building the PIT screens again today, we would construct the frame from 3inch T-section fiberglass impregnated extrusions, for looks and ease of manufacture. This material would simplify right angle joints and holes for the bungee cord. The T-section also offers the possibility of a less complex hinge assembly. The glass rod used for tensioning the screen could be made of less fragile material, perhaps as simple as a 1”×4” board. Both rod and bungee cord are needed for tensioning the screen. We find that three pieces of cord on each side are sufficient and that surprisingly little tension is required to ensure a flat screen. Servers for peripheral devices. We use a Dell 200 Pentium Pro PC running the Linux operating system as a server for multiple input devices. The PC runs a server process using the locally developed VRPN library (“Virtual Reality Peripheral Network,” described at http://www.cs.unc.edu/Research/nano/manual/vrpn/). It relays data from trackers, buttons, and other input devices over the local ethernet. Additional PC servers provide optional access to a PHANToM™ force-feedback arm, the UNC ceiling tracker, or a sound server for audio output. We also use an SGI button and dial box connected directly to the serial port of an SGI Onyx. Tracking. For head and hand tracking we have used both a commercial electromagnetic tracker, and a locally developed optical tracker. We use the VRPN server PC to drive an Ascension Flock of Birds tracker with Extended Range Transmitter and four sensor units (two for head tracking and two for hand tracking). The server reads records from the four tracker sensors in parallel at an average rate of 100 Hz. We have observed distortion in tracker readings across the PIT working volume, and intend to apply correction tables using methods developed by Livingston and State [1997].

Page 5 of 10

Figure 3. The PIT workspace in its 120°° configuration.

Figure 4. Detail of screen corner, showing frame hinge, screen material, and glass rod for support.

Page 6 of 10

For more accurate and faster head tracking we have also used the UNC ceiling tracker [Ward et al., 1992; Welch and Bishop, 1997]. We attach a HiBall optical sensor to each pair of stereo glasses (see Figure 5). The sensor views active infrared LEDs on the ceiling above the PIT and processes these views to provide position and orientation records at approximately 1 kHz. The ceiling tracker’s latency is userconfigurable from approximately 1 ms to 100 ms, depending on the level of filtering. We expect that a level of filtering resulting in approximately 30 ms of tracker latency will be adequate. In our initial tests, the infrared LEDs on the ceiling occasionally interfered with the infrared synchronization signal sent to the LCD shutter glasses, triggering the shutters. To solve this problem we plan to investigate methods to filter or shield the glasses from seeing the ceiling signal.

Figure 5. StereoGraphics shutter glasses with the UNC HiBall tracker.

Software structure The PIT software library has been designed to facilitate fast conversion of existing applications, and to allow for the same application code to run on different display devices, with either one or two users. Figure 6 shows the basic software structure. The core PIT code handles all functions required to display head-tracked stereo images and to allow a small set of viewing manipulations (primarily translate, rotate, scale, “grab world,” and adjust IPD). The user performs these manipulations using dials, buttons, hand controllers, and GUI control panels displayed on the laptop computers using the X and Tcl/Tk libraries. Applications link with the PIT API library, which is written in C++ and uses OpenGL and the UNC Vlib, VRPN, and other supporting libraries. We are testing methods for decoupling the simulation and display processes, similar to methods described by Pausch et al. [1994] and Shaw et al. [1992]. Our intent is that at runtime, PIT applications will be split into two main threads of execution: a model simulation loop, executing application-defined code, and a display loop, executing PIT library code and applicationdefined callback functions. An application must supply a display callback function that draws the model in world space using OpenGL. The PIT library sets the necessary viewing projections prior to calling the display function four times per frame (once per user, per eye). The application’s display callback is passed information about which view is being drawn (left or right user, and left or right eye), and may use those parameters to customize its model display. In the simplest mode of operation, the display callback ignores these parameters and draws the same geometry for both users. The PIT will also call other optional application-specified callbacks: a model update callback to coordinate data sharing between the application’s simulation and display code, and a graphics initialization callback, called once during initialization in the display execution thread.

Page 7 of 10

Simulation Code

Update Model API Query Functions

Handle Devices

Update View Draw World

Simulation Process

PIT Library Code

Display Process

Application Code

Figure 6. PIT software structure. Blocks indicate modules of source code. Arrows indicate data passed between the code modules.

The PIT API library also includes functions to modify display parameters and to query the PIT for data from the trackers, hand controllers, and other input devices to perform more complicated interactions such as selecting and moving individual model elements. To provide 3D picking, the application queries the PIT API for the status of the hand controller’s buttons, and for the hand controller’s position and orientation with respect to the application’s model space.

Evaluation and next steps We have ported the crystallography application CORWIN (for “coupled reciprocal windows”) for members of the Biochemistry department at UNC to use in the PIT. CORWIN allows users to manipulate molecules to perform protein-fitting tasks. The proteins can be viewed in real-space or reciprocal-space (Fourier-space) representations. Users manipulate a protein model by moving individual bond residues and rotating about individual bonds to make the model fit an electron density map. The basic PIT display provides controls for changing viewing and world-space transformations. The CORWIN application code handles application-specific interaction, such as selecting and manipulating individual atoms and residues, and provides other interactions through a GUI interface on the laptop computers. We plan to test the effectiveness of the PIT version of CORWIN by having UNC biochemists use the system on real data. We plan to gather additional human-factors results by extracting from CORWIN a simple 3D fitting task and conducting controlled experiments to measure performance under different viewing conditions and using different manipulation techniques. We have also created general polygonal model viewers for the PIT for other applications, such as architectural walkthrough or mechanical part design, and are working with outside users to port other Page 8 of 10

molecular graphics applications to the PIT. For additional information, including a list of material specifications and sources, and additional images and video sequences of the PIT, please see the project web page at http://www.cs.unc.edu/Research/graphics/GRIP/PIT.html.

Acknowledgments We thank the following people for their contributions to this work. Sumedh Barde wrote much of the PIT API and core code and contributed to the design. David Harrison helped configure and support video and hardware systems. John Thomas, Kurtis Keller, and John Poulton helped design and build the PIT screen frame. The screen design was done in collaboration with Fakespace, Inc. Ruigang Yang, Hans Weber, and members of the tracker and nanoManipulator projects wrote tracker drivers and provided VRPN and Vlib library support. Brent Insko and Michael Meehan coded earlier PIT software and contributed to the PIT design. Andy Wilson provided the panoramic version of the Walkthrough viewer application. The color photos were taken by Todd Gaul. Work on the PIT is funded by the NIH Division of Research Resources grant number RR02170 with other significant support from Intel Corporation.

References Agrawala, Maneesh, Andrew C. Beers, Bernd Fröhlich, and Pat Hanrahan. “The two-user responsive workbench: Support for collaboration through individual views of a shared space.” Proceedings of SIGGRAPH 97. Computer Graphics Proceedings, Annual Conference Series, 1996, ACM SIGGRAPH, pp. 327-332. Blanchard, Chuck, Scott Burgess, Young Harvill, Jaron Lanier, Ann Lasko, Mark Oberman, and Michael Teitel. “Reality built for two: A virtual reality tool.” Proceedings of the 1990 Symposium on Interactive 3D Graphics, Computer Graphics Vol. 24, No. 2, March 1990, pp. 35-36. Brooks, F. P., Jr., M. Ouh-Young, J. J. Batter, and P. J. Kilpatrick. “Project GROPE: Haptic displays for scientific visualization,” Proceedings of SIGGRAPH 90, Computer Graphics, August 1990, Dallas, TX, pp. 177-185. Cruz-Neira, Carolina, Daniel J. Sandin, and Thomas A. DeFanti. “Surround-screen projection-based virtual reality: The design and implementation of the CAVE.” Proceedings of SIGGRAPH 93. Computer Graphics Proceedings, Annual Conference Series, 1993, ACM SIGGRAPH, New York, 1993, pp. 135142. Czernuszenko, M. D. Pape, D. Sandin, T. DeFanti, G. Dawe, and M. Brown, “The ImmersaDesk and Infinity Wall Projection-Based Virtual Reality Displays,” Computer Graphics , Vol. 31 Number 2, May 1997. Deering, Michael. “Making virtual reality more real: Experience with the virtual portal.” Proceedings of Graphics Interface 93, Canadian Information Processing Society. pp. 219-226. Grant, Brian, Aron Helser and Russell M. Taylor II, “Adding force display to a stereoscopic head-tracked projection display.” Proceedings of VRAIS '98, Atlanta, Georgia, 1998. Livingston, Mark A. and Andrei State. “Magnetic tracker calibration for improved augmented reality registration.” In Presence: Teleoperators and Virtual Environments, MIT Press, Vol. 6, No. 5, October, 1997, pp. 532-546.

Page 9 of 10

Pausch, Randy, Matthew Conway, Robert DeLine, Rich Gossweiler, Steve Maile, Jonathan Ashton and Richard Stoakley “Alice & DIVER: A Software Architecture for the Rapid Prototyping of Virtual Environments,” Course notes for SIGGRAPH '94 course, “Programming Virtual Worlds,” 1994. Shaw, Chris, Jiandong Liang, Mark Green, and Yunqi Sun. “The decoupled simulation model for virtual reality systems.” Proceedings of the CHI 92 Conference on Human Factors in Computing Systems, 1992, pp. 321-328. StereoGraphics Corporation. 1997. StereoGraphics developers’ handbook. StereoGraphics Corporation, San Rafael, CA. Available at http://www.stereographics.com. Szalavari, Zs., D. Schmalstieg, A. Fuhrmann, and M. Gervautz. “Studierstube - An environment for collaboration in augmented reality.” to appear in Virtual Reality Journal, 1998. Available at http://www.cg.tuwien.ac.at/research/vr/studierstube/

Ward, Mark, Ronald Azuma, Robert Bennett, Stefan Gottschalk, and Henry Fuchs. “A demonstrated optical tracker with scalable work area for head-mounted display systems.” Proceedings of 1992 Symposium on Interactive 3D Graphics, Computer Graphics, Vol 25, No. 2, pp. 43-52. Welch, Gregory and Gary Bishop. “SCAAT: Incremental tracking with incomplete information,” Proceedings of SIGGRAPH 97. Computer Graphics Proceedings, Annual Conference Series, 1997, pp. 333-344.

Page 10 of 10