Timothy Poston and Luis Serra
Dextrous
Virtual Work A system for visualizing and manipulating medical images is detailed, with emphasis on interaction techniques.
T
HE media perception of Virtual Reality (VR) is fixated on one potent image: the
blind mask of inner vision, the head-mounted display (HMD). Indeed, even
when reporting work that has abandoned the HMD, television reporters sometimes insist on wearing one.
While the HMD has already shown its worth in such applications as architectural walk-throughs, there are few working examples where the virtual world is left significantly changed by the user. With current technology, affordable HMDs have much lower resolution than a typical monitor, and the field of view is narrow; the user seems to wear blinders and blurred glasses. The polygons-per-second drawing power of current hardware severely limits what can be redrawn to match the head motion fast enough to avoid simulator sickness. Productive work on intricate objects within arm’s reach is hard to achieve in such an environment. Our own goal is a productive work environment, in which delicate work can be performed for hours on end without strain. Our definition of VR is a merging of the numerically modeled space in the computer with the user’s experiential 3D space. This need not mean dissolving the ‘personal space’ in the user’s reach into a larger volume. Our approach is to bring the computer-
modeled work object (i.e., 3D medical image, engineering design, maquette) into the user’s natural work volume. By this we mean the region into which a craftworker will bring a small workpiece (whenever possible) a foot or so in front of the eyes for precise, comfortable depth perception, and in easy reach of both hands. Our first priority is to deliver an effective 3D interface as part of the Centre for InformationEnhanced Medicine (CIEMED), a collaborative project between our institute and Johns Hopkins University, with its concerns in brain and heart surgery, minimally invasive therapies, and craniofacial repair; this requires a robust, high-resolution system with which the medical user can work dextrously for long periods. The lessons learned are relevant outside the field of medicine. The key to dexterity in this space is hand-eye coordination. In the abstract, a mouse cursor seems far better than a finger, pointing more precisely at a point in the monitor screen. In practice, every screen has fingermarks. We have two high-precision 3D systems, visual and ‘proprioceptive’, the sense of one’s body state, including positions found by integrating joint angles. You can touch your fingertips behind your head without visual guidance. In merging our space with the computer’s, we need to respect both COMMUNICATIONS OF THE ACM
May 1996/Vol. 39, No. 5
37
systems. The user both feels Implementation where something is by hand posiHE basic setup, as tion, and sees it. already described, Ideally, this feeling should consists of a computinclude the sense of touching er screen, a pair of ‘computer objects’. For many stereo glasses, a mirpurposes it is enough to touch ror and one or more grippable one object; a generalized tool 3D sensors. Within the work volhandle. Where that real handle ume is a 3D position sensor, is, the user sees the handle of a moved by the hand. We have virtual tool, whose ‘business end’ experimented with acoustic, eleccan be anything—rotator, cutter, tro-magnetic and mechanicallypencil—the software can create. linked 3D input sensors. The We model the handle, not the acoustic (Logitech 6D mouse) hand; if a craftworker can see and the mechanical (the Immerthe tool and feel the tool, and Figure 1. sion ProbeTM) both had spatial these perceptions match, a view The Virtual Workbench restrictions. The electromagnetic of the hand merely obscures the sensors (the Polhemus FasTrakTM workpiece. The floating hands in many VR environ- and The Bird by Ascension Technology), had no ments rarely serve for more than simple, imprecise such restrictions, and were also lighter than the gestures of selection. Most dextrous work in the real mechanical sensor. They were, however, susceptible world is done through the handle of a scalpel, a to noise. The mouse-like design of both the Logitech and brush, a soldering iron, or some other tool. Dexterity The Bird is far from adequate for precise work in this in the virtual world can be achieved the same way. Our basic setup, which we call the Virtual Work- setting: it is hard to click a button on a mouse lifted bench, is shown diagrammatically in Figure 1. It con- from a supporting surface, especially while trying to sists of a computer screen, a pair of stereo glasses, a keep the device motionless. The acoustic system was mirror and one or more 3D position and orientation too limited in allowed orientation, and severely sensors, each on a handle. The user looks into the affected by echoes from nearby surfaces such as the mirror through the glasses and perceives the virtual required mirror. The Bird responded strongly image within the work volume, where the sensors and enough to monitor emissions to make the Dexterity hands lie. This produces a stereo view of a virtual vol- Game (described later) unplayable. We have had good results with both an Immersion ume in which a hand can move a handle, its position known through the arm, without obscuring the display (left- and right-eye views displayed alternately on the screen, with synchronized CrystalEyesTM glasses separating them for the eyes). Our particular environment has the full resolution of a good graphics monitor (not the miniaturizationcostly low resolution of current HMDs), and straightforward calibration to match real and virtual handles. Problems with delayed drawing are much less when we do not track the head; if turning the head right for a moment shifts the view left, and it takes a tenth of a second to shift back while the image is redrawn, the workpiece is too unstable for precise work. The applications we have developed so far are concentrated in the medical field, since for us these are the immediate problems. The range of eventual uses will be much more extensive.
T
Figure 2. The current implementation of the scheme in Figure 1. The user is interacting with the Brain Bench application using his left hand to move the brain and the right hand to perform delicate work. The virtual tool can be seen reflected in the mirror as a gray pen-like instrument pointing downward.
38
May 1996/Vol. 39, No. 5
COMMUNICATIONS OF THE ACM
ProbeTM and a Polhemus FasTrakTM with a pen-shaped arm support and limited hand stability”, and created receiver as shown in Figure 2. Each uses a built-in accuracy problems when drawing was attempted. (We switch. The function and virtual appearance of the are studying the design of an adjustable elbow supstylus change with each different operation, becom- port.) ing a slice-selector, a point-picker, a rotator, etc. The Schmandt used a half-silvered mirror instead of a stylus interacts also with a 3D widget called the ‘tool full mirror, mixing the view of the hand with that of rack’, which holds the buttons to select the different the graphics. This approach suffered from conflicts virtual tools and activates the different modes of between graphical and physical objects that he tried operation. The Immersion Probe is more stable and to resolve by controlling the brightness of the graphprecise, and costs less, than the FasTrak, but feels ical objects as well as by controlling the illumination awkward in some positions. of the hand. This setup is easily changed to an ‘augmented realMore recently, Iwata [11] placed the hand similarly ity’ environment (see the lower portion of Figure 2); behind a mirror to share space with the displayed scene the reflected image is visible in the same stereoscopic (in a monocular view), but the emphasis there was on depth, though neither real nor virtual objects can force feedback to the fingers, rather than manipulation mask each other. (We use this of a tool. The mirrorless configurasetup for calibration). We intend tion of Deering [5] gives correct to develop this mode for surgical In the abstract, hand-eye coordination but allows a use, but for most purposes we prehand or sensor to obscure a graphifer a fully computer-generated pica mouse cursor cal object it is ‘behind’ by stereo ture. Superimposing an convergence but in front of the seems far better than MRI-derived 3D image on a real physical screen. The same problem patient is useful for surgeons, but with reaching into an open struca finger, pointing in acting on a wholly computer-disture arises with other directly viewed played object (such as an MRI screens, such as the CAVE [2], more precisely at image alone, or a CAD/CAM which surrounds the user with disdesign), a view of the physical senplays, its spin-off the Immersadesk a point in the sor is less useful than a display in [10], which uses a single large disthe same location of a graphically play at a drafting-table slope, and monitor screen. generated tool with a ‘business the horizontal Responsive Workend’ (i.e., grabber, cutting blade, bench [13]. On all of these systems, In practice, every brush) appropriate to the current moreover, the eye must focus on a task. real screen beyond any workpiece screen has The system supports fixed and that is to be ‘touched’. The resulting head-tracked viewpoints. A fixed conflict of depth cues is a strain for fingermarks. viewpoint requires less computasome users. tion and is excellent for objects All of these systems use headwithin the skilled worker’s natural tracking, as does the augmented work volume, where for most people stereo fusion reality image overlay scheme of Blackwell et al. [1], appears to be the dominant cue. Head-tracking can which is similar to Schmandt’s but with modern assist in precision jobs, when a fixed view becomes graphics power and excellent calibration. As a result, insufficient to estimate the proximity of an object even fairly low latency produces unsettling ‘view lag’ almost perpendicular to the user. A slight movement by the high standards of the inner ear motion senof the head would solve this problem, if latency is sors. For dextrous work in an augmented reality very low, using the additional computational adjust- where the double-by-stereo update rate is limited by ments described in [5]. If the workpiece is complex, the interesting objects—often medical—being forcing the user to move to a new head position and drawn, head tracking can lose more precision than it then hold still during redrawing, it is better to change adds. This is particularly true for the users shown the view by rotating the workpiece. gathered around the Responsive Workbench in [13], Currently the Virtual Workbench is based on an where one head is tracked and any other viewers see SGI Reality Engine, but we are developing a version views calculated for other eyes, subjecting them to based on a high-end PC. lurches as the tracked head moves. Whenever possible, the worker’s elbow is supportA similar effect of looking down into a space where ed on a table; this important guard against fatigue the user’s hand can move is achieved, at considerably further restricts the volume in which actual work is lower cost, by the CyberscopeTM [4] 3D device in done. This appears to have been one of the reasons which one looks through a small configuration of Schmandt’s system [22], the closest (and earliest) lenses and mirrors, which reflects a screen level with analogous work we know of, did not lead to substan- the eyes. The resulting effective work volume is sometial follow-up. An illustration of his system (Figure 1 what smaller, and the transparency we use for caliin [22]) makes clear that the workspace correspond- bration is not available, so we have developed our ing to ours was set low, accounting for their “lack of system around the configuration in Figure 1. However, COMMUNICATIONS OF THE ACM
May 1996/Vol. 39, No. 5
39
Figure 3. Calibration: Match the tip of the real handle, seen through glass, to the apparent positions of the colored markers drawn at the stereo coordinates shown. Record the sensor (X,Y,Z) values reported for these locations.
it would not be difficult to achieve compatibility between our software and a Cyberscope-based system, given some small engineering changes. Calibration
T
relative importance of different depth cues depends on the task, and the distance. Stereo is most important in the natural work volume, but is outweighed by head-motion parallax further out, and by whole-observer motion effects at flight simulator distances. The analysis of the problem of matching real and HMD-virtual positions in [7] used test distances beyond arm’s reach; it is pointed out in [12] that “for Boeing’s touch labour applications, the projected image should appear at approximately arm’s length,’’ somewhat beyond the maximum-dexterity volume of concern here. (Interestingly, in that paper calibration, errors are reported in the angular units appropriate to the eye, but not in the spatial distance units appropriate to the hand.) A paper [20] on the importance of the interocular baseline concludes that a false value is often harmless in assessing which of two virtual objects is closer (again, at substantial distances), but did not consider the problem of putting a physical object where the virtual one ‘is’; hand-eye coordination in manipulation was not a factor. For our VR, it is central. The problems of matching real and virtual space in a HMD are remarkably intricate, involving 17 parameters even under minimal distortion assumptions [12]. Taking into account the optical compromises typical in a head-mounted display [7], these problems become more so—even before one considers head-tracking inaccuracies. It is neither easy or cheap to reduce these inaccuracies, or for maintenance to keep them small. For the Virtual Workbench, calibration is far simpler. HE
The handle’s position sensor reports in coordinates that depend on where its mechanical base, or electromagnetic source, is placed; where a display is seen depends on the monitor. Through glass replacing the mirror in the Virtual Workbench, we see the real handle held by the hand. In the glass, we see a reflected image (‘virtual’ as in optics) of the monitor, fixed by the real monitor and the mirror. For the standard position of the eyes, we compute and draw stereo images at convenient points in the usual graphics (x,y,z) coordinate scheme based on the monitor screen. The corresponding sensorreported (X,Y,Z) values (Figure 3) directly give the rule for transforming (x,y,z) to (X,Y,Z); just translate (0,0,0) to the (X0,Y0,Z0) found for it, and take (x,y,z) = x(1,0,0) + y(0,1,0) + z(0,0,1) to (X0,Y0,Z0) plus the corresponding combination of the (X,Y,Z) differences found along the three axes. This is a matrix operation, so to transform reported (X,Y,Z) to (x,y,z) (and so draw a virtual tool visible where the real handle is) we can straightforwardly invert it. A full description is given in [17]. The preceding example uses the bare minimum of calibration information, so that errors in matching the held tool to points seen in stereo show up directly as errors in the matrix coefficients; but the resulting calibration is quite comfortable in use. For more precise adjustments we use the sensor values at a larger set of points, and estimate the matrix by a leastsquares procedure. The Tool Rack
V
R management tools provide access to system functions and tools, status, help and other information [25]. Tools come in two classes: manipulators and selectors. Manipulators, interactive devices that form an integral part of the VR, include such ‘virtual tools’ as scalpel, calipers, hammer, pen, and so on. Selectors show a range of selectable alternatives and/or provide status information. Typically, they include menus and sliders. Selectors tend to be used in combination with manipulators; for example,
Figure 4. The Tool Rack. The selected button “edit” displays the descriptive “Edit the Slice” message.
40
May 1996/Vol. 39, No. 5
COMMUNICATIONS OF THE ACM
selecting from a menu selector involves the use of a button-pusher. The virtual ‘tool rack’ (Figure 4) is the manipulator in use with the Virtual Workbench (after experiments with 3D menus that mimicked traditional 2D menus, and with other geometrical selection tools [25]). The ‘tool rack’ holds the buttons to enable tools and activate different modes of operation. Each button’s icon shows its function, which chooses a manipulator, an action such as ‘undo,’ or a change of tool rack. As the user moves the stylus, the button nearest the tip is highlighted and described. Pressing the hand tool switch invokes the button’s function. Alternatively, to control this 1D row of buttons, a constrained mouse can be used with the other hand. This two-handed approach provides an effective interaction scheme, in which 3D-intensive operations are performed with the 3D stylus while 1D operations are left to the constrained mouse. Choosing ‘rotate’ in Figure 4 enables the rotation tool that controls the orientation of the displayed object. When the handle button is held down, the object rotates to follow a ‘rubber’ string joined from its center C to the stylus tip, each step being a rotation in the plane formed by C and two successive stylus positions. Intuitively, the feel is ‘reach in and turn it about its center’.
and thus almost impossible to complete the game. The experience has increased our respect for endoscopic surgeons, who perform difficult manipulations with only a monoscopic camera view, and strengthened our belief that a VR setting can greatly facilitate the dextrous manipulation of 3D objects, in either simulated or camera displays. Another point that emerged from the implementation of the dexterity game is that head-tracking can be very important in precision jobs: a fixed view becomes insufficient to estimate the proximity between the loop and the wire when the flat loop is almost perpendicular to the user (and the user cannot move it for fear of touching the wire). A slight
A Dexterity Test
For general evaluation of the performance and ergonomics of the Virtual Workbench, we have implemented a dexterity test based on a popular skill game commonly found in fairs, which consists of passing a loop over a sinuous/twisted wire without touching it (Figure 5). When the loop and the wire touch, an electric circuit closes and an alarm sounds. In the context of the Virtual Workbench, this test enables us to assess the degree of skill that users can achieve, the speed at which they adapt to the setup (stereo glasses, input sensor) and the length of time that they can play the game without fatigue. The game is also a good test for virtual interfaces in general since it is easy to introduce to people, and fun to play. A 3D menu allows one to configure the skill level required of the user by modifying parameters of the loop, such as hole radius, thickness, etc. The game is extremely difficult to play using only a monoscopic 3D display, without the setup of the Virtual Workbench, but with the Virtual Workbench most people have no difficulty. This fact was made apparent to us while making a demonstration videotape, which forced us to play the game in monoscopic mode so that we could record it in monoscopic video. It proved very difficult to estimate distances,
Figure 5. The dexterity game: The wire turns red and sounds an alarm when touched
movement of the head would solve this problem, but this need not involve attaching anything to the head. A swivelled binocular eyepiece fitting the upper face could track the necessary small movements, while leaving the user as free to pull away as to put down the virtual tool handle. Medical Imaging Applications
The manipulation of medical image data, which increasingly come in 3D form (or 4D, with the time dimension), as Magnetic Resonance (MR), CAT scan, ultrasound, etc., has been our main concern. It is frequently necessary to revolve the 3D display, to specify a cutting plane exposing (for example) a particular cross section of brain data, to specify a point within a suspected tumor and ask the system to display the connected region of similar data values it appears to belong to, and so on. With the 2+e-dimensional arrangement of monocular screen and 2D mouse, COMMUNICATIONS OF THE ACM
May 1996/Vol. 39, No. 5
41
Figure 6. The heart beating application. The oblique cutting plane follows the movements of the knife blade, slaved to the handle felt by the hand.
Visualization Tools for Diagnostics. Our general interface deals with 3D and 4D medical dynamic data sets, such as cine-loop MRI, to facilitate their exploration. It allows the user to turn an object by reaching in and dragging it around, and to select a point by placing the tip of the hand-held instrument, both felt and seen at the same spatial location. Part of the requirement here is an easily controlled interface, with a convenient selection of functions and of current system response rules. It is also necessary to make the interactions themselves natural and easily remembered, dimensionally matching the data to be manipulated. Two different visualizations were developed on two different data sets: a beating heart rendered in full volume, and a head rendered as three orthogonal slices. They have since been merged in a more general volume visualization toolkit. The full heart visualization consists of dynamic MRI scans of the heart at different stages of the beating cycle (Figure 6). The reconstructed volume is displayed by the ‘3D texture’ feature of the Silicon Graphics Onyx RE2 workstation [16]. In an MR image, different tissues can be identified by different density in the image data, which can be appropriately weighted or blended during volume visualization. Thus, for instance, the blood flow can be highlighted in different colors. The rectangular volume of interest can be changed, and cut by an oblique plane for cross-section views.
Figure 7. The Brain Bench: Planning neurosurgery with a virtual stereotactic frame
the design of such interactions becomes a tortuous problem, their implementation difficult, and their use counterintuitive. Our intended users are not motivated to learn complex command sets; spending 30 hours repairing a gunshot-blasted human face leads not to a willingness to spend equal time on software training, but to a value system that makes 30 minutes excessive. Commands and gestures need to be intuitive, easily learned, and easily remembered. Here we describe three medical applications that have been integrated into the Virtual Workbench. Each aims to provide state-of-the art tools for physicians to use in routine clinical investigations. 42
May 1996/Vol. 39, No. 5
COMMUNICATIONS OF THE ACM
The Brain Bench. MR data for the brain can also be shown using a volume display, but Figure 7 shows intersecting slices, registered with an electronic Brain Atlas [17] developed from paper atlases in common surgical use. The left hand holds a tool to grab, move, and rotate the 3D assembly; the right holds a stylus that can drag any slice plane(s) though the brain, select a brain structure and examine it more closely, and so on. In the figure it is adjusting the arch of the virtual ‘stereotactic frame’ shown; the angle of the arch and the placement on it of the guide box, and the (x,y,z) position of the arch base, can be dragged into a chosen position. The white line shows the corresponding path of a probe (i.e., biopsy needle, resection device, electrode) inserted with the frame at that setting. The Brain Atlas structures encountered by the path of the probe are highlighted, indicating which of them are at risk for the chosen approach, and the MR data can be explored in detail along the approach, revealing features such as blood vessels that are too variable to be predicted by a universal atlas.
Figure 8. The 3D Contour Editor
The Contour Editor. To estimate local contraction from MRI heartbeat images, and deduce the presence of non-contributing muscle, one must map the heart wall contours accurately, in each slice, at each imaged time. Neither an algorithm nor a human has been able to do this on the evidence contained in a single slice; machine estimates must be corrected by humans, using criteria of 4D (space and time) consistency. This has taken a full working day per heartbeat, using only a 2D interface; in our environment this time is reduced to less than an hour [18], making it practical to analyze larger numbers of cases. The task is to track and quantify the heart motion
from MRI data, with ‘tags’ (the dark lines in Figure 8) that move with the heart. Our current starting point is the FindTags program developed at Johns Hopkins [9], which simultaneously locates the tag lines and estimates the inner and outer contours of the heart, in individual slices. Given the resolution and quality of the MRI data, this estimation produces results that are not always correct, and that must be manually corrected. After correction, the contours become input to an analysis program that estimates geometric strains in the heart. In our Contour Editor [23], the user can make fine adjustments in computer-estimated curves, using their neighbors in space or time as a guide to help
Figure 9. Tube finder: Editing a sketched artery curve. (Stereo pair, to be viewed by eye-crossing to merge the left eye’s view of the right with the right’s of the left).
COMMUNICATIONS OF THE ACM
May 1996/Vol. 39, No. 5
43
distinguish fact from artifact. Figure 8 shows a snap- Future Development shot of the complete environment, with the stylus HE Virtual Workbench is an appropriate editing a control point. As it does with all interfaces setting for precise work of many kinds. For in the Virtual Workbench, the stylus interacts with CAD it provides a natural environment in the stack of MRI slices and with the ‘tool rack’. which to place and transform 3D shapes. Each slice is a 2D MRI density map, converted into We will chiefly, however, be developing an image and texture-mapped over a polygon. In applications to neuro-, cardiac, craniofacial, and mineach stack, there is always a ‘working slice’ or slice of imally invasive surgery. interest, in which the contours are edited. Its backFor medicine, the Virtual Workbench offers the ground opacity can be interactively controlled from imaging tools discussed previously, and surgical simtotally opaque (useful to concentrate on that slice) ulation at various levels, from assessing the efficiency to totally transparent (to better see the rest of the and risks of various placements of virtual radioactivestack, for which only the contour estimates are tipped needles in the virtual brain, to dextrous shown). manipulation of lasers to clear a blocked artery or The stylus allows the user to reach into the image lung passage. A change in a virtual skull can be specand drag the contours to the correct position. The ified (not at first by simulating the required cutting working slice can be changed by selecting its handle and reattachment, but by geometric virtual tools), with the stylus, and raising or lowering it through and passed to a biomechanical simulation that prethe stack. The slices are arranged in either spatial dicts the resulting rearrangement of the face surface; or temporal sequence, showing a set of slices at a different skull cuts and reassembly patterns can be particular instant (in geometrically tried out as paths to a planned change. Note that these do not correct spacing), or the time variation of a particular slice. The butrequire force feedback; we are, however, planning the development of ton with the hourglass switches between these stacks, leaving the Our intended devices to achieve this in the setting working slice fixed at the current of the Virtual Workbench’s work users are position and orientation. volume, which will further enlarge its range of application. not Tube Finder. Humans are much To reduce the lag between the better than computers at seeing user’s moving the sensor and the motivated arteries, ducts, etc., in stereo pairs movement of the virtual tool, which for dexterity must be as small as posof X-ray images, pairs of projections to learn of volume data, and so on. sible, we are implementing a predicComputer vision has great diffitive filter, using quaternions to complex make the arithmetic of rotations culty with the ‘correspondence problem’ of deciding which points more robust [15]. This will become command correspond to which, for stereo trieven more crucial as we move into angulation—particularly in a soft, force feedback, since the springisets... noisy medical image, lacking the ness of the user’s wrist provides a faster ‘reaction time’ than the user’s point features like corners that dominate mechanical imagery. nervous system, making delay-loop The human sensory system peroscillation a more serious problem. We have developed a C++ toolkit of standardized ceives depth well, but if the depths seen are to contribute to further quantitative analysis, they interactions, utilities for extracting geometric strucmust be entered into the computer. The Tube ture from 3D data such as tissue density, and so on, so Finder application (Figure 9) allows the user to that what we do for the tissues of our initial interest sketch in space the central curve of the tube seen, can be repeated quickly, by us or by collaborators, for either as a spline curve (adding and editing con- other parts of the body. We are also working on Augmented Reality (AR) trol points) or as a chain of straight segments. It creates and edits multiple curves overlaid on applications in medicine. AR can be roughly subdistereo pairs of maximum intensity images. Visual vided into three groups, in which one adds computer feedback cues tell the user which node of which constructs to: curve is active. Translucency allows the stylus to seem to pass behind elements for which the com- • a direct optical view (through glass, as in Figure 2 puter has no depth information, which cannot or Blackwell et al. [1], or through a microscope as therefore mask it. in Edwards et al. [6]), Given this ‘initial guess’, a variational approach • a live camera image as in Edwards et al. [7] or telemedicine as in Green et al. [8] and Rovetta [14, 18] allows the machine to improve the 3D accuracy of the fit, giving a firm basis for (for instance) and Sala [21], quantitative analysis of blood flow along the artery • or an online 3D scan by (for instance) MR or selected. ultrasound.
T
44
May 1996/Vol. 39, No. 5
COMMUNICATIONS OF THE ACM
The greatest obstacle to success is registration of the very different sensing channels involved; a live 3D scan usually tracks fiducials, which simplifies the problem in principle, but it remains crucial. Surgery (except the very least invasive) disturbs the carefully extracted geometric/anatomical structures that augmentation will display, and real-time tracking of a flexible tissue represents a major challenge both in flexible modeling and in CPU power. Even the tracking of multiple rigid fragments is a non-trivial task [3]. Maintenance of precise registration amplifies the latency difficulties that can occur with head tracking; the monitor-mounted glass reflector of Peuchot et al. [16] avoids this source of error by making the view available only when the eyes are properly placed relative to the (movable) display. Equally important, when the reality to be augmented is medical, is the need for a robust, usertransparent interface. Necessary tasks should be easy to perform and it should be extremely difficult to make an irrevocable error. C References 1. Blackwell, M., O’Toole. R.V., Morgan, F., and Gregor, L. Performance and accuracy experiments with 3D and 2D image overlay systems. In Proceedings of the 2d International Symposium on Medical Robotics and Computer Assisted Surgery (MRCAS 95), (Baltimore, Md., November 5–7, 1995), pp. 312–317. 2. Cruz-Neira, C., Sandin, D.J., and DeFanti, T.A. Surround-screen projection-based virtual reality: The design and implementation of the CAVE, In Proceedings of SIGGRAPH 93, (1993), pp. 135–142. 3. Cutting, C., Taylor, R., Khorramabadi, D., Haddad, B., Optical Tracking of Bone Fragments During Craniofacial Surger. In Proceedings of the 2d. International Symposium on Medical Robotics and Computer Assisted Surgery (MRCAS 95), (Baltimore, Md., November 5–7, 1995), pp. 221–225. 4. Cyberscope. Simsalabim Software, Berkeley, CA. 5. Deering, M. High resolution virtual reality. Computer Graphics 26 (1992), 195–201. 6. Edwards, P.J., Hawkes, D.J., Hill, D.L.G., Jewell, D., Spink, R., Strong, A., Gleeson, M. Augmented reality in the stereo operating microscope for otolaryngology and neurosurgical guidance. In Proceedings of the Second International Symposium on Medical Robotics and Computer Assisted Surgery (MRCAS 95), (Baltimore, Md., November 5–7, 1995), pp. 8–15. 7. Edwards, E.K., Rolland, J.P. and Keller, K.P. Video seethrough design for merging of real and virtual environments. In Proceedings of IEEE VRAIS ‘93, 223–233. 8. Green, P.S., Jensen, J.F., Hill, J.W., and Shah, A., Mobile Telepresence Surgery. In Proceedings of the Second International Symposium on Medical Robotics and Computer Assisted Surgery (MRCAS 95), (Baltimore, Md., November 5–7, 1995), pp. 97–103. 9. Guttman, M., Prince, J.L., and McVeigh, E.R. Contour estimation in tagged cardiac magnetic resonance images. IEEE Trans. on Med. Imag. 13, 1 (1994), 74–88. 10. Immersadesk. Descriptions available from Electronic Visual Lab., University of Illinois, Chicago, USA. 11. Iwata, H. Artificial reality with force-feedback: Development of desktop virtual space with compact master manipulator. Computer Graphics 24 (1990), 165–170. 12. Janin, A.L., Mizell, D.,W. and Caudell, T.P. Calibration of head-mounted displays for augmented reality applications. In Proceedings of IEEE VRAIS ‘93, 246–255. 13. Krueger, W. and Froelich, B. The responsive work-
bench. IEEE Computer Graphics and Applications 4, 14 (1993), 12–15. 14. Lawton, W. Mathematical methods for active geometry. In Proceedings of the International Conference on Computer Aided Geometric Design, (Penang, Malaysia, July 1994 ). 15. Lawton, W., Poston, T. and Serra, L. Time-lag reduction in a medical virtual workbench. In Proceedings of “Applications of Virtual Reality” British Computer Society conference (Leeds, England, June 1994). 16. Peuchot, B., Tanguy, A., and Eude, M. Virtual reality as an operative tool during scoliosis surgery. In Proceedings of CVRMed’95—First International Conference on Computer Vision, Virtual Reality and Robotics in Medicine. (Nice, France, 1995), pp. 549–554. 17. Poston, T. and Serra, L. The Virtual Workbench: Dextrous VR. In Proceedings of VRST’94—Virtual Reality Software and Technology (Singapore, August 23–26, 1994), G. Singh, S.K. Feiner, D. Thalmann, Eds., World Scientific, Singapore, 1994, pp. 111–122. 18. Poston, T., Serra, L., Lawton, W., Chua, B.C. Interactive tube finding on a virtual workbench. In Proceedings of the Second International Symposium on Medical Robotics and Computer Assisted Surgery (MRCAS 95), (Baltimore, Md., November 5–7, 1995), pp. 119–123. 19. Poston, T., Nowinski, W., Serra, L., Chua, B.C., Ng, H., and Pillay, P.K. The Brain Bench: Virtual stereotaxis for rapid neurosurgery training and planning. Visualization in Biomedical Engineering, 1996. To be published. 20. Rosenberg, L.B., The effect of interocular distance upon operator performance using stereoscopic displays to perform virtual depth tasks. In Proceedings of IEEE VRAIS ‘93, pp. 27–32. 21. Rovetta, A., and Sala, R. Robotics and telerobotics. In Proceedings of the Second International Symposium on Medical Robotics and Computer Assisted Surgery (MRCAS 95), (Baltimore, Md., November 5–7, 1995), pp. 104–110. 22. Schmandt, C. Spatial input/display correspondence in a stereoscopic computer graphic work station. Computer Graphics 17, (1983), 253–259. 23. Serra, L., Poston, T., Ng, H., Heng P.A., and Chua B.C. Virtual space editing of tagged MRI heart data. In Proceedings of CVRMed’95—First International Conference on Computer Vision, Virtual Reality and Robotics in Medicine (Nice, France, 1995), pp. 70–76. 24. Solaiyappan, M., Poston, T., Heng, P.A., Zerhouni, E.A., McVeigh, E.R., and Guttman, M.A. Interactive visualization for speedy non-invasive cardiac assessment. IEEE Comput. (special issue on computer applications in surgery), (January 1996), 55–62. 25. Waterworth, J.A. and Serra, L. VR management tools: Beyond spatial presence. In CHI’94 Conference Companion, CHI’94 Conference on Human Factors in Computing Systems (Boston, Mass., April 1994), pp. 319–320.
About the Authors: TIMOTHY POSTON is a senior scientist at the Institute of Systems Science of the National University of Singapore. email:
[email protected] LUIS SERRA is the researcher leading the 3D interface group at the Institute of Systems Science of the National University of Singapore. email:
[email protected] Authors’ Present Address: Centre for Information-Enhanced Medicine, Institute of Systems Science, National University of Singapore, 10 Kent Ridge Crescent, Singapore 0511. Permission to make digital/hard copy of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication and its date appear, and notice is given that copying is by permission of ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists requires prior specific permission and/or a fee. © ACM 0002-0782/96/0500 $3.50
COMMUNICATIONS OF THE ACM
May 1996/Vol. 39, No. 5
45
