Design Principles for Distributed, Interactive, Virtual Environments Kevin Jeffay F. Donelson Smith Russell M. Taylor II Gary Bishop James H. Anderson Department of Computer Science University of North Carolina at Chapel Hill Chapel Hill, NC 27599-3175 {jeffay,smithfd,taylorr,gb,anderson}@cs.unc.edu (919) 962-1938 Voice (919) 962-1799 FAX Richard Superfine Department of Physics and Astronomy University of North Carolina at Chapel Hill Chapel Hill, NC 27599-3255
[email protected] (919) 962-1185 Gail Jones School of Education University of North Carolina at Chapel Hill Chapel Hill, NC 27599-3500
[email protected] (919) 966-3291 June 1997
Project Summary It is becoming increasingly common for scientists to interact with instruments such as microscopes, spectrometers, and medical imaging equipment through computer-based interfaces. For example, a virtual reality interface to a scanned probe microscope (SPM) constructed by UNC-CH computer scientists allows chemists, biologists, and physicists to “see” the surface of a material at nanometer scale and “feel” the properties of the surface through the use of a haptic, force-feedback device. This is accomplished by integrating an SPM with a high-performance 3-D graphics workstation, a tracking/force-feedback haptic device, and graphics displays. This interface, called the nanoManipulator, has enabled new scientific investigations that were otherwise not possible to be conceived and performed. Further, the interface has enabled operation of the instrument by non-specialists, including students of all ages. This project advances the technology for virtual reality interfaces by enabling distribution of the interface across large, shared networks, including the Internet. Such distribution enables remote operation of the instrument and thus allows multiple investigators to share a single (potentially expensive) instrument and its operating costs. The challenges in realizing such a distributed virtual laboratory are formidable. Highly interactive virtual environments that provide true telepresence (as opposed to simple “tele-vision”) require low-latency, real-time communication between system components and real-time computation within components. Realizing these requirements on commodity computing engines, and commonly deployed packet-switched networks such as the Internet, is the essence of this research. Unfortunately, current networks are subject to highly variable latency, available bandwidth, and rates of packet loss, all of which present significant challenges in human factors for immersive environments. This project will produce advances in three areas of Computer Science: •
Interactive Graphics in Distributed Virtual Environments — An investigation of user interface mechanisms (image rendering, tracking, and haptic feedback) to maintain the users’ desired interactions and sense of immersion in the face of unpredictable delays and losses in the network.
•
Real-Time Operating Systems — Operating system support is required to control the real-time interactions with the instrument and make effective use of available real-time communications services
•
Networking — The challenge is to develop and evaluate parsimonious mappings from nanoManipulator data flows to service models proposed in the architecture for integrated services on the Internet. The project will also develop application-specific adaptations (based on the Internet Real-time Transport Protocol (RTP)) that can be applied to ameliorate the effects of congestion in networks without service guarantees.
In this project, design principles for distributed, interactive virtual laboratories for uses such as microscopy will be developed and evaluated. The research will be evaluated by constructing working variants of the nanoManipulator system and using them to support research and education. There are a number of collaborators ready to use this system including physicists, chemists, and educators. An outreach program delivered as part of the LEARN North Carolina project will evaluate the application of the virtual laboratory in secondary education.
1
Project Description 1 . Introduction It is becoming increasingly common for scientists to interact with instruments such as microscopes, spectrometers, and medical imaging equipment through computer-based interfaces. For these systems, it is natural to consider physically separating the data acquisition hardware from the interface. In particular, it is appealing to consider the distribution of system components across small-scale, dedicated, local-area networks to bring more computer processing power to bear on the realization of the user interface. Indeed, this has been done successfully at UNC-CH and elsewhere for investigating the value of virtual reality (VR) interfaces. For example, we have recently constructed a virtual environment interface to a scanned probe microscope, that allows scientists to “see” the surface of a material at nanometer scale and “feel” the properties of the surface through the use of a haptic, force-feedback device. This interface, called the nanoManipulator, has enabled new scientific investigations that were otherwise not possible to be conceived and performed. Further, the interface has enabled operation of the instrument by non-specialists, including students of all ages. This project advances the technology for VR interfaces by enabling distribution of the interface across larger, shared, networks, including the Internet. Such distribution would enable remote operation of the instrument and thus allow multiple investigators to share a single (potentially expensive) instrument and its operating costs. Moreover, because the interface to the instrument can be easily replicated (e.g., through the use of multicast and groupware technology), distribution of the system can foster collaborative work by eliminating the requirement for investigators to be collocated. The challenges in realizing such a distributed virtual laboratory are formidable. Highly interactive virtual environments that provide true telepresence (as opposed to simple “tele-vision”) require lowlatency, real-time communication between system components and real-time computation within components. Realizing these requirements on commodity computing engines, and commonly deployed packet-switched networks such as the Internet, is the essence of our research. In this project we will develop and validate design principles for distributed, interactive, virtual laboratories for uses such as microscopy as described above. Our work will consider the trade-offs among (a) computer-human interfaces, (b) network infrastructure for real-time communications, and (c) costs of replicating special purpose hardware. The challenge is to understand which system structures and user interaction models are required for achieving useful, cost-effective results in a distributed laboratory given constraints introduced by the distribution of system elements, network bandwidth and latency, and the desired interactions with the instrument. To build an effective distributed virtual laboratory we require advances in three areas of Computer Science: • Interactive Graphics in Distributed Virtual Environments —New user interface mechanisms (image-based rendering, tracking, display, and haptic feedback) are required to maintain the users’ desired interactions and sense of immersion in the face of unpredictable delays and loses in the network. • Real-Time Operating Systems — Operating system support is required to control the real-time interactions with the instrument and make effective use of available real-time communications services. We will investigate real-time scheduling and synchronization algorithms for this purpose. • Networking — The challenge here is to develop and evaluate parsimonious mappings from nanoManipulator data flows to service models proposed in the IETF’s architecture for integrated services on the Internet. We will also develop application-specific adaptations that can be applied in the real-time protocols to ameliorate the effects of congestion in networks without service guarantees.
2
We will evaluate our research by constructing working variants of the nanoManipulator system and distributing them in a variety of manners described below. We have collaborators ready to use our system including physicists, chemists, and educators who will apply the system to their areas of research or study and provide swift feedback to the systems developers. We also plan an active outreach program to science education in secondary schools that also has additional monetary support from contributions by UNC-CH and private industry. The project has an intradisciplinary team of computer scientists representing the areas of graphics, virtual environments, networking, and operating systems. The team is also interdisciplinary and includes members from physics and education. This team will develop a unique, state-of-the-art virtual laboratory for nanometer-scale research in diverse fields and advance the understanding of applications for high-performance computing and communications.
2. The nanoManipulator System Scanned-probe microscopes (SPMs), such as the atomic-force microscope (AFM)1, allow the investigation and manipulation of surfaces down to the atomic scale. An AFM is capable of positioning a tip very precisely (x, y positions within a fraction of an atomic diameter) over a surface in a wide variety of environments, including ambient, ultrahigh vacuum, and under water. It is capable of resolving individual atoms on crystalline surfaces, and molecules, proteins and viruses under physiological conditions. The AFM tip can provide quantitative data on a wide range of sample features including the surface topography, friction, adhesion, temperature, compliance, etc. Most important, the surface can be modified through the deposition of material or through the mechanical tip/sample interaction by machining the surface or manipulating surface-bound objects. The nanoManipulator (nM) system provides a virtual-environment interface to SPMs which gives the scientist virtual telepresence on the surface, scaled by a factor of about a million to one [Finch95]. A stereo display (usually combined with head tracking) presents the 3D surface floating in space within arm’s reach. A hand-tracking and force-feedback device allows the scientist to actually feel surface contours and to manipulate objects on the surface, such as the tobaccomosaic virus (TMV) particles in the image shown in Figure 1. This user interface enables new forms of experimentation that augment and amplify human cognition. This direct, natural interface to SPMs has revealed new results from existing data sets and allowed new and fruitful experiments that could not be performed otherwise[Flavo97]. This HPCC multidisciplinary problem-solving environment is a tight collaboration between computer scientists as toolsmiths working with physicists as users to create the ideal interface Figure 1: The nanoManipulator system. for SPMs. The project began in 1991 as a collaboration between the CS department at UNC-CH and R. Stanley Williams at the Chemistry department at UCLA. In 1993, collaboration began with the UNC-CH Physics department. There are now three Computer Science principal and two Physics principal investigators and a Biophysics postgraduate student to investigate DNA/protein interactions is actively being recruited to form a bridge to additional collaborators in Chemistry and Biology. The nM system has been used by chemists, biochemists, physicists and gene therapists who at times flew across the country to use the system. They have investigated the mechanical properties of TMV and adenovirus (a vector for gene therapy), manipulated a 20nm colloidal gold particle into a thin gap formed in a wire, broken and repaired tiny nano-wires, and studied the mechanical and electrical 1
SPM is a general class of microscopes of which an AFM is a member. Our testbed involves an AFM but the system is designed for SPMs in general. In this document we often use the terms interchangeably.
3
Figure 2: NanoManipulator high-level system structure. properties of carbon nanotubes. See our project web page at http://www.cs.unc.edu/Research /nano/hotshots for more detailed descriptions of additional experiments using the system. Our publications appear in both computer science and biophysical journals [Falvo97, Finch95]. The nanoManipulator system couples a graphics supercomputer (the UNC-CH PixelFlow machine [Molnar92] or a high-end SGI), an Intel processor-based microscope/controller (a TopoMetrix, Inc. Explorer) and a hand-tracking/force-feedback controller (a SensAble Devices, Inc. Phantom) via a network (a dedicated switched Ethernet) as shown in Figure 2. The dedicated network is used by the graphics computer to send and receive SPM data/commands and position/force descriptions. The graphics process must maintain an image update rate of better than 20 frames/second to keep the illusion of immersion (of course, a frame buffer is used to update the CRT at 60 frames/second). When the system is in scan mode, the microscope controller sends data asynchronously to the graphics display as the tip rasters across the surface of a sample. This data updates each part of the displayed surface image as the scan data arrives. When the user is in direct control of the tip (touch mode), there are no surface geometry updates (though images still need to be displayed from the user’s current viewpoint). Instead, the microscope tip follows the trajectory of the user’s hand as tracked by the force-feedback device. The surface is scanned at these locations and the force feedback tells the user where the surface is and how it is changing. Ideally, this update would happen at a rate better than 500 Hz. In the current system design, we are limited by the graphics update rate. To avoid force discontinuities and instability, we have introduced an intermediate representation (a local plane approximation) which the force controller uses to provide stiff force feedback at 1kHz; the approximation is updated at closer to 30 Hz. The nanoManipulator has proved to be powerful and useful as a microscope control system; the challenge is now to make the system more widely available. The system as it now stands works very well over a dedicated 10Mbps switched Ethernet between the three computers in the system. This configuration is appropriate to the existing testbed environment, where the microscope, 3D graphics system, and force-feedback device are collocated within reach of a dedicated LAN on the UNC-CH campus. It is unreasonable to expect that high-performance computer graphics equipment and advanced scanned-probe microscopy equipment will be so located at other institutions (it took two major NSF grants to make this possible at UNC-CH). In order to allow the wide availability of this interface, it will, therefore, be necessary to tailor the application for operation over shared networks including the Internet.
4
3. Virtual nanoManipulator Laboratories The nanoManipulator System is composed from several elements, each of which may be a scarce resource for any given user or application. In this section we consider several models for constructing a distributed nM system using networks (both local-area networks and the Internet) to provide access to otherwise unavailable parts of the system. As shown in Figure 3, there are four major processing components in the system, each of which may be distributed over a networked environment. • SPM and Control Processing: In scanning mode the SPM can generate approximately 8,500 updates per second of 12 bytes each (816 Kbps = 8500 updates/sec × 12 bytes/update × 8 bits/byte). In touch mode, (x, y) position values (8 bytes) are sent to the SPM and the corresponding 12 byte data values (e.g., height, lateral forces) are returned. The frequency of interactions required in touch mode depends on the user’s tasks and the tracking device response, but the system should be capable of sustaining a rate of approximately 500 interactions per second (80Kbps = 500 updates/sec × (8 + 12) bytes/update × 8 bits/byte). • Phantom and Control Processing: Inputs from the Phantom’s hand-tracking device arrive at a rate of 500 updates per second (24 bytes each) for a maximum data rate of 96Kbps (500 updates/sec × 24 bytes/update × 8 bits/byte). Force-feedback data flowing to the Phantom has approximately the same size and frequency, thus doubling the bandwidth requirement between the Phantom and the remainder of the system to 192Kbps. • User Interface and Application Processing: This component currently manages all of the user’s interface to the system and is the source and destination of communications with both the SPM and the Phantom. When the user is directly manipulating the surface of the sample in the microscope, hand-position information arriving from the Phantom causes new (x, y) coordinates to be sent to the SPM which then returns new data values for that position. This, in turn, results in an update of force feedback information sent to the Phantom. For both SPM and Phantom processing, this component has an aggregate maximum I/O bandwidth of 272Kbps (80 + 192). In addition to handling the SPM and Phantom communications, the application processing includes updates (based on the SPM output data) to the 3D surface model used to generate the display image. The bandwidth required to update the surface model in most applications corresponds directly to the bandwidth generated by SPM updates (816Kbps in scanning mode, 48Kbps (500 updates/sec × 12 bytes/update × 8 bits/byte) in touch mode). There are, however, a few modes when the maximum bandwidth demand is much higher (e.g. when the user is doing something that would affect the way the whole surface is displayed, like passing it through a Gaussian filter and varying the filter size with a slider). In this case the maximum bandwidth could go as high as 250Mbps (512 × 512 positions × 12 bytes/position × 10 updates/second). Head Tracking and User Input Devices
SPM Control Processing
SPM microscope
User Interface and Application Processing (Surface Model Updates)
3D Graphics Processing (Display List Updates & Image Rendering)
Phantom Control Processing
Phantom tracking/ force feedback Graphics Display
Figure 3: Major components of present nanoManipulator system. 5
•
3D Graphics Processing: Based on the surface model generated by the application, the graphics display list is updated and passed to the graphics rendering engine. The processor and memory bus cycles required to generate these display lists represents a major resource requirement of the system. The rendering hardware uses the updated display list to generate new images (frame buffer contents). Expressed as a maximum bandwidth requirement, image rendering represents a potential output of about 1.05Gbps (1,400,000 pixels/frame × 24 bits/pixel × 30 frames/second). Obviously, a 3D graphics engine capable of rendering at these rates also represents a major resource requirement for the system.
One goal of this project is to create a virtual laboratory so researchers can use an SPM when they do not have immediate or proximate access to one. In our first example, illustrated in Figure 4, we consider the case where researchers have access locally to all necessary components except the SPM itself. By interconnecting the SPM control processor with the application processor(s) over local (campus or regional) networks or the Internet, the remote SPM becomes (virtually) a part of the users’ local environment. Obviously this type of usage requires cooperation from the organization housing the SPM and a key part of the system is to provide collaboration tools, including shared drawing tools, chat windows, and audio/video communications to facilitate coordination of the experiment (e.g., loading surface samples into the SPM, changing probe tips, etc.). The bandwidth demands for this connection are relatively modest, especially in touch mode, but the effects of network latency and latency variation have severe impacts on the user’s ability to have a realistic sense of precise control over the experiment. Dealing with the impacts of latency on human factors will require aggressive strategies that integrate advances in networking for real-time applications, advances in graphics for human-computer interaction, and advances in paradigms for distributing function (described below). A much more demanding situation is created if our requirement is to construct a virtual laboratory when both the SPM and the 3D graphics processing are remote (Figure 5). Such situations may arise when a user has neither an SPM or a workstation capable of handling the 3D graphics processing (typically a high-end SGI machine) and is able to arrange for access to these resources over a network. For certain applications, one may even need to exploit the capabilities of specialized equipment designed for very high-end rendering applications such as the Pixel-Flow machine [Molnar92]. In this configuration, bandwidth is likely to be as serious a constraint as latency. To interconnect a 3D graphics-processing engine to the rest of the system requires bandwidth from the applications processing to the graphics processing with a maximum input bandwidth of 250Mbps to transfer surface model updates from the application processor and a maximum output bandwidth of 1.05Gbps to transfer pixel values back for display. Obviously, these data flows must be dramatically compressed either spatially or temporally. Head Tracking and User Input Devices 80Kbps (direct) 816Kbps (scan) SPM Control Processing
User Interface, Application, and 3D Graphics Processing
Network
Graphics Display
Phantom Control Processing
SPM Phantom
Figure 4: Distribution of nanoManipulator system with a remote SPM. 6
Head Tracking and User Input Devices 48Kbps (direct) 816Kbps (scan) 25Mbps (max)
80Kbps (direct) 816Kbps (scan)
User Interface and Application Processing Monitor
SPM Control Processing
Image Warping Network 1Mbps (min) 50Mbps (max)
Phantom Control Processing
SPM Phantom 3D Graphics Processing (Display List Updates & Image Rendering)
Figure 5: Distribution of nanoManipulator system with remote SPM and graphics engine. One very novel aspect of our approach is to introduce a new processing stage in the graphics pipeline following the rendering stage called an image warper. The details of this approach are described in a later section, but, conceptually, image warping works on a local cache of information about the rendered image sufficient to produce new renderings for a limited (but highly probable) set of new viewpoint perspectives. Using these techniques, the frame update rate can be reduced from 30 frames per second to 2-3 frames per second. When combined with more aggressive compression of the image data to eliminate redundant information between frames, the bandwidth required for data flowing from the 3D graphics engine to the user interface processor can be reduced to the range of 1Mbps (for small changes per image) to 50Mbps (for complex, dynamic interactions). Clearly an aggressive compression strategy is also needed to reduce the bandwidth in modes requiring that complete replacements of the surface model flow from the application processor to the 3D graphics processor. In this case we can combine novel strategies of caching with elimination of redundant data to produce reductions of 1-2 orders of magnitude in bandwidth requirements (under 25Mbps). We also integrate these methods into a framework of adaptive response to dynamic network constraints on capacity and latency (described below). The configuration illustrated in Figure 5 allows a user of the SPM access to a rich laboratory environment with a more modest investment in local equipment (a Phantom and its control processor along with a high-end personal computer with graphics capabilities and high-speed network connection such as a 100 Mbps Ethernet) while utilizing other, more costly, resources over the network. This configuration could also be the basis for science outreach programs bringing access to the SPM to high schools and small college campuses. By accepting some constraints on the range of functions and quality of visualizations required for research applications, we can still deliver exciting laboratory and educational experiences for a wide variety of users. In particular, we will show that any institution having Internet access at T1 speeds (1.5Mbps), which some high schools in the state of North Carolina do have already, can make educational use of these resources. The investment required beyond the T1 Internet connection is a high-end personal computer with a graphics card (total cost around $16,000 in today’s prices and surely much lower by the end of this project). We can also provide some of the haptic feedback required for a sense of touching through force-feedback joysticks which are now becoming available on the consumer market for use with computer games (thus eliminating the Phantom and its control processor from this configuration).
7
SPM
SPM Control Processing Head Tracking and User Input Devices
Graphics Display
User Interface Processing
Network
3D Graphics Processing (Display List Updates & Image Rendering)
Image Warping
Phantom
Phantom Control Processing
Figure 6: Distribution of nanoManipulator system with a remote graphics engine. All the above configurations were based on the implicit assumption that the creator of the virtual laboratory did not have local access to an SPM. We now consider a variation where the owner of an SPM desires to assemble the components necessary for more powerful modes of visualization and direct sample manipulations (see Figure 6). The only additional investment required by the SPM owner is to acquire a Phantom and its control processor and a high-end personal computer (with graphics capability and a 100Mbps ethernet connection; head tracking equipment may be a desirable option for some applications). This assumes, of course, that access to the 3D graphics computing resources can be obtained through network connections. This configuraiton introduces no new bandwidth, latency, or packet loss issues beyond those already considered for the configurations presented earlier.
4. Research Challenges The envisioned uses of a distributed nanoManipulator system will require an effective understanding of the basic trade-offs between communication and local processing in a distributed system. These trade-offs are necessary in order to realize a believable and useful virtual reality experience across a range of distribution and interaction paradigms. At a high level, we characterize our problem domain as a 3-dimensional problem space partitioned by an axis of nM applications and interaction paradigms, an axis of function-distribution paradigms within the distributed system, and an axis of available computer and communications technology. We will study instances of a distributed nM that lie scientifically and educationally interesting regions of this 3-dimensional space. First, we will consider several points in space that describe two instances of the nM capable of supporting remote scientific collaboration and teleoperation. One instance is the remote use of a shared SPM by scientists across the UNC-CH campus. Here the graphics engine and the SPM would be collocated and the remote users possess only a high-end personal computer and a tracking/force-feedback device (see Figure 5). Low-latency, synchronized, high-bandwidth communications for delivering the inputs and outputs of the system are required. Excellent local processing capabilities exist and the system components are interconnected via a high-speed campus intranetwork. There is, however, no support for real-time communications on this network and, thus, the network flows must be managed to both adapt to and ameliorate the effects of congestion in the network. Similarly, the nM user interface may have to be modified to help further mask the effects of non-real-time communications. Scaling up the reach of the system over the Internet in the face of additional network elements, greater latency and jitter, and higher loss probabilities adds to the challenge but our basic solutions should apply in this case as well The second scientific use of the nanoManipulator is by scientists at other universities within the Research Triangle region of North Carolina. The distribution of the system is the same as in the previous instance, however, the distance separating the components is larger and the number of network elements in the path between data sources and sinks is greater. The physical network, however, is a high8
speed ATM network that is capable of supporting integrated services. The challenge here is to map nM data flows to elements of the integrated services model to realize real-time communications. Both of these scenarios can be extended to include situations wherein the tracking device and graphics engine are collocated but the SPM is remote. Our team is well positioned to carry out this research because we have access to excellent networking environments including (1) an in-house Multimedia Networking Laboratory recently upgraded to include ATM and 100Mpbs Ethernet switches through an NSF CISE RI grant, (2) high-speed campuswide FDDI rings, (3) direct connection to the Time Warner VITALnet ATM network2 and (4) access to the larger Internet through a recently acquired connection to the NSF vBNS network. Several groups around the world are working to provide network interfaces to scientific instruments, ranging from telescopes to scanning electron microscopes (SEMs) to scanned-probe microscopes (SPMs). Two examples are the Spectro-Microscopy Collaboratory at the Advanced Light Source (Lawrence Berkeley National Laboratory — http://www-itg.lbl.gov/BL7Collab) and the Tele-Presence Microscopy & the ANL LabSpace (eLab) Project (Argonne National Laboratory — http://146.139.72.10/docs/anl/tpm/tpmexecsumm.htm). These systems are addressing the issues of delivering image and audio data across the network from the controlled devices and sending commands (pan, zoom, parameter settings) to the devices. This provides remote operation of the instruments in a command-at-a-time, image-at-a-time mode and can make use of existing teleconferencing infrastructure and network protocols. The nanoManipulator system, in contrast, is sending the real-time, closed-loop, control from the user’s hand location to control directly and immediately the SPM tip and get direct visual and tactile feedback. Without this real-time interaction, scientists could not perform experiments such as moving viruses, manipulating colloidal gold particles 20nm across, repairing tiny nano-wires, or bending carbon nanotubes. Sending real-time control and real-time image data streams, interleaved with the traditional video and audio streams, requires greater capability at the network layer and close cooperation between the application and the network to determine the most appropriate representations, algorithms, and transmission protocols. 4.1 Challenges in Networking Our initial research effort will be an investigation of the suitability of the multimedia transmission protocols for non-audio and video-based applications such as the nanoManipulator. Specifically we will consider if the IETF Audio/Video Transport Working Group’s transmission protocol RTP is general enough to accommodate the requirements of distributed virtual environment applications. We will develop an RTP profile for the nM system and assess the complexity (and feasibility) of modeling flows within the RTP framework. In addition, as we develop distributed version of the nM, we will identify the important network transmission performance parameters that system components need to know in order to effectively manage application flows to achieve real-time communication. We will attempt to communication these performance parameters using the RTP control protocol (RTCP). As the nanoManipulator’s flows are continuous but have timing and semantic properties that are very different from audio and video suggests that modifications to RTCP will be likely. In parallel with the RTP effort we will pursue three other networking research efforts corresponding to three broad areas along a continuum in our 3-dimensional problem space representing degrees of sophistication of network technology and infrastructure deployment. The first is a consideration of the use of network-provided services for end-to-end real-time communication — specifically the services of guaranteed delay and controlled load that presently make up the IETF’s architecture for integrated services on the Internet. The challenge here is to develop parsimonious mappings from nanoManipulator data flows to service model proposed in the architecture and to evaluate the performance and effectiveness of such mappings. Given our previous work in real-time communications (e.g., [Nee et al. 97, Talley & Jeffay 96, 94, Stone & Jeffay 95, ]) as well as the work we propose in operating systems below, we also propose to contribute to the work on implementations of service 2
VITALnet is an OC-48 (2.4 Gbps) SONET network that spans the Research Triangle Park region of North Carolina and interconnects the three major Triangle area universities (UNC-CH, Duke, and North Carolina State University), with local technology companies such as IBM’s Networking Systems Division and Cisco’s Interworks Business Unit.
9
models. It is our belief that within the various service classes that there will be significant latitude in how each service may be realized and that one can bias or “tune” an implementation to meet specific needs of our and other distributed virtual environment applications. The second effort considers a hybrid network environment wherein support for integrated communications exists in the core of the network but this support does not extend all the way to the end-user systems. That is, guaranteed real-time communications is possible across a network service provider’s backbone but not on the campus network that a user’s packets must traverse before entering the service provider’s network. This is how we believe integrated services will in fact be introduced to users and organizations — from the outside in. In this environment the challenge is to manage real-time flows across the “first mile” from a sending machine to the entrance point of a network that is capable of providing some real-time service (e.g., from a user’s workstation to a router connecting the user’s campus to an ISP whose equipment supports a guaranteed delay service), and across the “last mile” from the exit point of the real-time capable network to the receiver’s workstation. The network links traversed along the first and last mile will suffer from all of the congestion-related pathologies that are experienced on the Internet today (albeit hopefully on a much smaller scale) and hence one must adapt to congestion on the endpoints while making good use of a real-time connection between campuses. This will be achieved by placing soft-state in routers at the entrance/exit points of source and destination campuses and using this state to adapt flows along the first and last mile to the perceived levels of congestion on each campus network. The state information includes an indicator of which adaptations are possible on which flows as well as information on the performance of the connection. Adaptations include network-level adaptations such as bit-rate and packet-rate scaling as well as application-level adaptations such as employing alternate flow encodings or compression schemes, or dynamically modifying the current interaction mode with the instrument. State information will be build up in routers through inter-router exchanges of performance feedback via RTCP or a similar protocol and by header information in the RTP flows. Alternatively, resource reservation protocols such as RSVP may be used for this purpose. The third effort considers a sender and receiver connected via the Internet as it presently exists today. The challenge here is to develop generic application-level media and flow adaptations that can be applied at the sender and receiver in concert to reduce or ameliorate the effects of congestion in the network. In addition, the results of the user interface research into techniques for masking network pathology will be integrated into a general performance monitoring and adaptation service. 4.2 Challenges in Operating Systems Operating system support is required to either aid applications to effectively adapt to network congestion or to make effective use of available real-time communications services (e.g., to inject data into the network in such a manner that a performance contract is not violated). There will be two efforts here. The first will be the design of a middle-ware API that will support an application-operating system dialog about current performance. This is needed for applications to dynamically tailor their operation so as to maximize the interaction experience given an operating system-provided estimate of the levels of performance currently sustainable in the distributed system. The second effort is an investigation for operating system mechanisms for efficient, adaptable realtime computing. This will include a study of flexible real-time resource allocation problems and efficient synchronization techniques for orchestrating and adapting application media flows within the operating system kernel (and in particular within the network communications protocol stack). 4.3 Challenges in 3D Graphics and Human-Computer Interaction Little is understood about how to deal with user interaction in a virtual environment where there is latency and jitter (latency variance) caused by a network connection. The effects of latency are understood, and are devastating to system usability if not compensated for [Holloway95]. Recent work by Conner and Holden pushes beyond dead-reckoning to use the techniques of motion-blur, transparency and defocusing to indicate to the user that there is latency in the system [Conner97]. Our proposed work will address the problem of removing and compensating for latency at all levels of the 10
system. We will also address issues in application-level adaptations to decreased bandwidth or increased latency in the network by adjusting user interfaces, representations, and algorithms. Because the current system architecture relies on low-latency (less than 2ms) communication between the force-feedback, graphics and microscope components, any congestion on the network link would make the system unusable due to instability in the force feedback. This requirement essentially limits the system to configurations in which all components are on the same LAN. Part of the low-latency requirement is imposed by the single-threaded main application that coordinates all traffic from all devices. The team assembled for this proposal has expertise and experience dealing with real-time, multithreaded, network-based computing. We will split the existing monolithic application along functional boundaries based on computing and communications requirements, crafting it to match the equipment distributions (described below) that are likely when the system is widely deployed. We are, in fact, running a control loop from the user’s hand-position tracker, through the microscope controller that moves the tip, and back to the user’s hand as force feedback. The stability and fidelity requirements of this loop form the fundamental limit on the amount of tolerable latency. This must be addressed not only in the user interface/graphics processing, but also in the network and operating system services in order to guarantee stability. The proposed team is well-equipped to investigate these protocols running over LANs, ATM, the Internet, and Internet 2. There are well-known techniques to provide a faster display-update rate when the user is interacting with a virtual environment by trading quality for speed. The application can adjust the model tessellation, display resolution, or surface shading quality when the interaction begins and ends. A “sweet spot” of higher-fidelity information can be inset in the display image where the user indicates interest in the scene (by gaze direction or by touching a certain area). It should be possible to modify many of these techniques to allow the application to respond to changes in network performance in addition to user interaction mode. For system configurations in which the microscope and force display are local and the graphics engine is remote, we will modify the application to communicate directly between the two and remove the dependency on the main application loop. This will give high-fidelity force feedback to the user based on the microscope data. In this case, only the icon representing the user’s hand location needs to be updated with low latency, which involves combining image and polygonal display within the image warping engine. For configurations in which the microscope and force display are on the opposite ends of a network, it is fundamentally impossible to remove the latency from the interaction loop. Our intermediate plane surface representation can be used to handle latencies of up to 30ms round-trip, but beyond this we must develop new approaches to provide a stable, high-fidelity representation of the surface as the microscope tip modifies it. One possible attack is to add artificial friction or viscosity to the surface, making it difficult for the user to move their hand rapidly as an indication of the maximum feedback rate and latency. The addition of post-rendering warping to the system also allows the adjustment of the image update rate from the graphics engine to the image warping engine or the initiation of image compression in response to changes in network performance. Should the application learn that the latency has become unacceptable for direct display, techniques such as those in [Conner97] may be applied to alert the user to this fact. Post-rendering warping can remove the apparent latency from the system so long as the user is not attempting to interact with the scene (changing the viewing direction is fine; touching the surface is not). Post-rendering warping Interactive-graphics applications such as the nanoManipulator that seek to immerse the user in a 3D world, demand the ability to display complex geometric models at high frame rates with low delay. Most progress to date has depended on improvements in semiconductor technology and graphicaldatabase simplification techniques. We are exploring a different but complementary approach by exploiting frame-to-frame coherence to avoid completely the conventional rendering of most frames.
11
In most interactive 3D applications, the viewpoint changes gradually, so that adjacent frames are very similar. Most frames can be generated by using an image warp to extrapolate from nearby conventionally rendered frames. We refer to the frames rendered in the conventional manner as reference frames and the frames produced from image warping as derived frames. Techniques we are developing produce several derived frames for each reference frame. We use McMillan and Bishop’s planar-to-planar, forward-mapped image warping algorithm [McMillan1995] to compute derived frames from reference frames. This warp uses a per-pixel disparity as part of the warp computation. The disparity value is a form of depth information that is easily computed from the 1/Z values in a standard Z-buffer. We refer to this warp as a 3D warp because it relies on both disparity information and image coordinates. The 3D warp is a significant advance over simpler image warps and image-compression methods that have been previously used for the transmission of graphics imagery. Simpler 2D warps and imagecompression and motion-compensation methods such as MPEG only capture the way pixels move within the image plane. This 2D limitation causes large errors in the presence of significant changes in the viewing perspective. In contrast, the 3D warp uses the per-pixel depth values to properly construct new views taking into account the 3D structure of the underlying scene. The use of post-rendering warping should have a dramatic impact on the implementation of distributed virtual reality (VR) systems such as the nanoManipulator. Interpolating intermediate frames from many fewer reference frames will allow the data rate over the network to be dramatically reduced while maintaining the same interactivity for the end user. This allows the user to change the viewpoint onto the sample surface without requiring network bandwidth to transmit new renderings of the image computed by the graphics processor – all warping computations can be done on the user’s local workstation. When head-tracking devices are used with the nM, a post-rendering warp tied to the users current viewpoint will allow the network and rendering-system delay to be eliminated from the critical headmotion-view-change loop. Just as the frame-buffer decoupled screen refresh from image update, postrendering warping decouples image update from viewpoint update. Thus the user is presented with an image that appears to be three-dimensional and stationary because it changes appropriately as he moves his head. The low update rate and high-latencies that are likely to be present without our postrendering warp would destroy the illusion of a stable 3D world. A DARPA/NSF funded research effort is already underway to investigate the use of 3D image warps for creating immersive environments of real places. We propose to complement that research effort by focusing on the use of 3D warps for networked VR. We expect the already-funded effort to develop hardware accelerators that we can also use for our investigations. Challenges in Warping How to compress warp image data for transmission Though image compression methods have been studied for many years, the compression of the perpixel disparity values required for post-rendering warp will require a different approach. The presence of occluding boundaries in images, results in arbitrary discontinuities in the resulting depth maps. Even so, there is considerable coherence in much of the depth data that should allow good compression. The use of 3D warps allows new possibilities for compression algorithms. Rather than searching for 2D motion vectors in images as in many conventional compression algorithms (e.g., MPEG), the 3D warp can take advantage of 3D coherence in the data to potentially allow for greater compression. When the viewpoint translates in a scene, some areas may become exposed that were not visible from previous viewpoints. Thus a reference frame may not contain all of the data necessary to produce a particular derived frame. Mark [Mark 97] has shown that multiple reference images may be fruitfully combined to eliminate most of these exposure events. It may also be possible to transmit only that data from new reference images that is necessary to make up for data missing in previous views, thus allowing for additional very significant compression.
12
What do you send first? In the presence of large network delays, what data from the reference images should be transmitted first? It may be possible for the nanoManipulator application to identify the region of interest for the user in the image based on the location of the user interaction devices. With the area of interest identified, it should be possible to adjust the transmission schedule so that the most relevant image data is sent first and used to begin warping/displaying that part of the image while transmission of the remaining parts continues in the background. This strategy will enhance the user’s perception of system responsiveness. How do you trade compression for latency? Since our system goal is minimum perceived latency for the end user, we have interesting opportunities to trade-off time spent in compression for the timeliness of the data presented to the user. Under some network conditions it may be better to do less computation-intensive compression and transmit more data. Under congested conditions it might be more important to compress images as much as possible to allow the new data to get through. View-dependent lighting in Post-rendering warping. Post-rendering warping, as currently formulated, does not support lighting effects such as specular reflection that vary with view position. The nanoManipulator team has discovered that these light effects can be very effective in helping the user to understand the shape of surfaces. Another challenge is to determine how the intermediate representation produced by the graphics engine and transmitted to the warping engine should be altered to allow view-dependent lighting and other effects to be computed efficiently on the warper. 4.4 Challenges in Education Outreach A collaboration between the School of Education and the Department of Physics and Astronomy will develop the outreach program including the design, implementation and evaluation of pedagogical materials. We will use the technological resources and contacts of LEARN NC (Learners’ and Educators’ Assistance and Resource Network of North Carolina). LEARN NC is a partnership of the North Carolina Public Schools, North Carolina Community Colleges, the University of North Carolina System, and private industry. It is building a one-of-a-kind computer resource that will train teachers to use the latest classroom technology and provide a new on-line support system. Classroom materials will be developed through a joint effort of Russell Rowlett (Director, Center for Mathematics and Science Education), Gail Jones and Richard Superfine. During the early phases of the project (approximately the first 1.5 years) materials will be presented at a variety of school systems without connecting to the nM system at UNC-CH. This will allow for an evaluation of the presentation materials until the networking capabilities developed by this project are operating at an appropriate level. The first anticipated site for actually using the remote nM system in secondary science education will be the Orange County High School which currently has a T1 (1.5Mbps) connection. Other local schools are expected to have such connections during the funding period of this project and will participate as well. Some of the issues faced by the project during this development period are: • Explore how students’ conceptual understandings of atoms and viruses change as result of using the nM system to touch and bend them. This could be done with concept mapping, interviews, and drawings. • What samples are most effective in capturing the imagination of the students? A virus? A bucky tube? A live cell? What samples are straightforward to work with and are robust over the time of the class period? • Are the students convinced that they are operating a microscope live? Do we need a live video link to get this across? • Are the modes of interaction available at relatively low bandwidth (T1) sufficient to deliver a meaningful learning experience?
13
• • • • • • • •
How much time does an individual require at the haptic interface (microscope control) in order to have a fulfilling experience? What kind of activities can the students perform during the period while they are waiting for their turn? Can we develop materials which are informative yet order-independent? How effective is off-line interaction (with a stored data set) versus live control of the remote microscope? This can be implemented during the early development phase of the project. What kinds of written material, with figures and pictures from other microscopes (SEM, optical, etc.), are most effective for conveying the length scales involved? Can we develop a mechanical model for the operation of the SPM (i.e., a pantograph) that will convey how the instrument “sees”? While the initial materials will be developed for a high school class, how can they be adapted for other grade levels? How can we design the equipment so that it will be robust for travel and naive user use? How can we evaluate what the students have learned?
5. Research Plan In year 1 of this project we will develop an RTP profile of the nanoManipulator system and commence the Integrated Services-based implementation of the system by distributing the system (according to Figures 5 and 6) between UNC and North Carolina State University (NCSU) using the VITALnet ATM network. Prof. Don Brenner, a material scientist at NCSU has expressed an interest in using such a distributed nM system for teaching undergraduates and is willing to help us test and evaluate the system. We will also commence work on implementing a post rendering warping system and evaluate its performance across instances of the nM distributed across the UNC campus network (a best-effort network). Finally, we will begin the outreach effort in year 1 by having high-school groups visit the UNC campus and use the existing nM system. Based on the results of the image-warping experiments, in year 2 we will investigate media adaptations to support scientific use of the nM on the UNC campus. We will also commence the work on routerbased support for extending the quality-of-service guarantees provided by VITALnet to nM uses that use VITALnet as a backbone but do not originate or terminate on the ATM network. At this time we hope to be able to take a PHAMToM force-feedback robot arm and graphics display to local highschools for low-bandwidth trials of the system. A curriculum development effort for secondary schools centered around the nanoManipulator system will also begin. Year 3 will continue the works begun in the previous two years with the focus being primarily on extending the use of the nM further out into the Internet. We have active collaborators such as Dr. Ruth Pachter at Wright Patterson Air Force Base in Ohio, Dr. Anshuman Razdan from Arizona State University, and Prof. Harry Ruda from the University of Toronto who have all both expressed an interest in assisting with experimentation and in using the software and technology we develop during the course of this project. Section I contains letters of support from these individuals as well as other collaborators who state their interests in the distributed version of the nanoManipulator.
6. Summary and Impact of Project The challenge proposed by our research team is to produce innovative designs for interactive, distributed, virtual environments. Our work is set apart from related projects by our emphasis on maintaining a truly immersive environment even in networked configurations possibly having highly variable bandwidth, latency, and packet loss. Our use of fast display-update rates and, especially, haptic feedback produces a strong visual and tactile connection with the environment. We will provide a powerful sense of telepresence rather than an experience more like watching television (or even a slide show) that one might obtain in a less ambitious system. This is accomplished by integrating new techniques in graphics, human-computer interfaces, haptic feedback, and real-time support in networking services and operating systems. While we have integrated most of the research around a useful testbed as a proof-of concept for these advances, we should emphasize that all of the designs developed in this project will be transferable as will the virtual laboratory itself. The techniques and paradigms pro14
duced from this project will advance computing and communications by providing the foundations for other researchers to use in building networked versions of their own immersive virtual environments (e.g., teleoperation of instrumentation or telemedicine). National and Local Impact on Science: The nanoManipulator project is already funded to produce a tightly-coupled system that will be transferable and affordable to most researchers in physical and biological sciences who already have an SPM. We now have an opportunity to extend the reach of this tool to a wider community of scientists and educators (including secondary education) who cannot afford the investment in an SPM or the graphics computer and I/O devices, or who need only intermittent access. Within the local UNC-CH campus and the wider Research Triangle Park, NC area, our distributed testbed system will serve to foster broader access and collaborations among microscopy users in the state’s universities and local corporations. For example, at UNC-CH, our virtual laboratory will open the way for improved access to Biochemistry researchers working on DNA/protein interactions. At NCSU, access by Don Brenner to this laboratory will enable new explorations using the nM in Materials Science education. In the RTP area, there is also an unusual concentration of biotechnology and drug development/ manufacturing corporations. We will use our access to these corporations through MCNC (a state-supported focal point for collaboration between corporations and the state’s universities) to encourage application of the nanoManipulator virtual laboratory in applied problems. On the national and international level, we have collaborators at Arizona State University, the University of Toronto, and Wright-Patterson Air Force Base that we can support by extending our testbed across the Internet (and Internet 2). Impact on Graduate and Undergraduate Education This project will provide education and research experience to nine graduate students (seven in Computer Science and one each in Physics and Education). We expect at least half of these to complete a Ph.D. based on dissertation research performed on this project. In addition, many of these graduate students will receive hands-on experience with science education through the outreach aspects of the project. Undergraduate students have had direct involvement in the nanoManipulator project since its inception and we expect this to continue in the future. Five undergraduates (three of whom were supported by programs for providing research experiences to minority students) have used the nM for experiments or built some component of the system. We will continue the active recruiting and mentoring of undergraduate students. Impact on Outreach The nanoManipulator Project provides an intuitive experience for students in understanding the microscopic world. Bump the atoms on a surface, push a gold cluster around, smash a virus, feel the sponginess of a living cell! It is rare that such experiences at the forefront of basic science can be made so accessible to young students. However, all of these exhortations can be given to students without exaggeration or hyperbole. The combination of high performance visualizations and tactile sensations with the atomic force microscope in a real-time control system makes each of these activities a reality. The challenge is to bring this experience to the classroom and to get the students involved. One purpose of the nM project as a testbed is to provide the network capability to ensure a palpable experience for students at locations remote from the microscopy facility at UNC-CH. The delivery of these outreach programs will take place through the LEARN NC network. See section I for letters describing various interests in the outreach aspects of the project.
7. References [Conner 97] Conner, Brook and Loring Holden, “Providing A Low Latency User Experience In a High Latency Application,” Proceedings of the 1997 Symposium on Interactive 3D Graphics, Providence, R.I., April 27-30, 1997. pp. 45-48. [Falvo 97] Falvo, M. R., S. Washburn, R. Superfine, M. Finch, F. P. Brooks, Jr., V. Chi, and R. M. Taylor II, “Manipulation of Individual Viruses: Friction and Mechanical Properties,” Biophysical Journal Vol. 72 No. 3, March 1997, pp. 1396-1403.
15
[Finch 95] Finch, Mark, Vernon Chi, Russell M. Taylor II, Mike Falvo, Sean Washburn and Richard Superfine, “Surface Modification Tools in a Virtual Environment Interface to a Scanning Probe Microscope,” Proceedings of the ACM Symposium on Interactive 3D Graphics (Monterey, CA, April 9-12, 1995), special issue of Computer Graphics, ACM SIGGRAPH, New York, 1995. pp. 13-18. [Holloway 95] Holloway, Richard, “Registration Errors in Augmented Reality Systems,” Ph.D. Dissertation, University of North Carolina at Chapel Hill Department of Computer Science, August 1995. [Jeffay et al. 96] K. Jeffay, M. Parris, T. Talley, F.D. Smith, A Router-Based Congestion Control Scheme For Real-Time Continuous Media, Proceedings of the Sixth International Workshop on Network and Operating System Support for Digital Audio and Video, Zushi, Japan, April 1996, pages 79-86. [Mark 97] William Mark, Gary Bishop, and Leonard McMillan. “Post-Rendering 3D Warping,” Proceedings of 1997 Symposium on Interactive 3D Graphics (Providence, Rhode Island, April 27-30, 1997), pp. 7-16. [McMillan 95] McMillan, Leonard, and Gary Bishop. “Plenoptic Modeling,” Proceedings of SIGGRAPH 95, (Los Angeles, CA), August 6-11, 1995. pp. 39-46. [Molnar 92] Molnar, Steven, John Eyles and John Poulton, “PixelFlow: High-Speed Rendering Using Image Composition,” Proceedings of SIGGRAPH ‘92, a special issue of Computer Graphicss Volume 26 Number 2, July 1992. pp. 231-240. [Nee et al. 97] P. Nee, K. Jeffay, G. Danneels, The Performance of Two-Dimensional Media Scaling for Internet Videoconferencing, Seventh International Workshop on Network and Operating System Support for Digital Audio and Video, St. Louis, MO, May 1997, to appear. [Stone & Jeffay 95] , D.L. Stone and K. Jeffay, An Empirical Study of Delay Jitter Management PoliciesACM Multimedia Systems, Volume 2, Number 6, (January 1995) pages 267-279. [Tally & Jeffay 96] T.M. Talley and K. Jeffay, A General Framework For Continuous Media Transmission Control, Proceedings of the 21 st IEEE Conference on Local Computer Networks, Minneapolis, MN, October 1996, pages 374-383. [Talley & Jeffay 94] T.M. Talley and K. Jeffay, Two-Dimensional Scaling Techniques For Adaptive, Rate-Based Transmission Control of Live Audio and Video Streams, Proceedings of the Second ACM International Conference on Multimedia, San Francisco, CA, October 1994, pages 247-254.
16
