Collaborative Interactive Physical Simulation - CRIStAL

Collaborative Interactive Physical Simulation J´er´emie Dequidt, Laurent Grisoni, and Christophe Chaillou ALCOVE, INRIA Futurs, LIFL, Universit´e Lille I, Cit´e scientifique bˆatiment M3 212, 59655 Villeneuve d’Ascq France.

Figure 1: Three distant users interacting on a hanging cloth.

Abstract This article proposes a flexible solution for collaborative physical simulation. The proposed solution is resistant to low-performance networks. Unlike classical client/server models, it maintains the level of interaction of user in case of network shutdown. For such purpose, we use a loose-synchronized simulation framework that weakens (under some tolerance) the classical condition of synchronization for physical simulation. Such simulation is combined with (obviously non-synchronous) network information. In the case of deterministic simulations, such simulation schemes ensures that all hosts asymptotically share the same, physically consistent, physical configuration. What is more, all hosts converge to this configuration in a smooth way. Examples are provided to show the potential and flexibility of such approach. CR Categories: I.6.8 [Simulation and Modeling]: Types of Simulation—Combined; H.5.3 [Information Interfaces and Presentation]: Group and Organization Interfaces—Collaborative computing

simulation for example, increase the quality of interaction in the virtual environment.

Another natural solution for improving the feeling of immersion of users in virtual environments is to allow people to interact all together in the same physical environment. When trying to provide a solution to this, one is immediately confronted to the difficulty that common workstations are not designed for providing interaction to several users. As a result, most work dealing with collaboration usually consider several workstations, over the network. The main difficulty with such approach is that wide area networks and even local networks have well-known limitations in terms of bandwidth and latency. Characterizing the state of a deformable model often requires hundreds of values, which can rapidly reach bandwidth limitations. Latency causes another kind of difficulties. Coupled with the fact that computer hosts might not share the same computation power, synchronization policies need to be introduced to insure that the state of the simulation is the same on each machine which takes part to the simulation. All these points makes the whole problem of collaborative simulation a difficult one.

Keywords: Physical Simulation, Collaborative Work

1

Introduction

A lot is currently being done for improving plausibility of real-time physical simulation. Modern applications take for example advantages of technical progress on CPU and GPU. Nowadays, complex deformations may be computed in interactive time either on CPU or GPU ([Mosegaard et al. 2005; Harris 2004; GPGPU ]) to build richer and more realistic environments. Sophisticated haptic device with 3 or 6 degrees of freedom [Otaduy and Lin 2005] are now available at reasonable cost. Such device, widely used in surgical

This article proposes a solution to the problem of networkcollaborative real-time physical simulation. The proposed solution is flexible enough to adapt itself, to some given extend, to low performance networks. Within our simulation system, we introduced special objects, we called Ghost object, that are associated to objects manipulated over the network, and asynchronously combine the host simulation with the network information received. Our framework is described in detail in section 4.

The remainder of the paper is organized as follows : section 2 presents previous results related to collaborative simulation, that can be useful for collaborative physical simulation. Section 3 discusses the point to know how to manage network latency for realtime physical simulation, and introduces the ideas that lead us to the proposed framework, that is then described in section 4. In this section, network issues are also discussed: special attention is paid to describing how data is exchanged and how all simulations asymptotically (although quite fast in practice) converge to coherent and synchronized state throughout all hosts.

2

Related works

Simulation and especially physical simulation is confronted to many different technical issues when it relies on networks. Latency and bandwidth limitations due to networks make it more difficult to achieve coherency between all sites in an interactive environment. The following works attempt to achieve synchronization of hosts in collaborative simulation environment. The first collaborative environment (SIMNET [Calvin et al. 1993]) that includes simulation was developed in the eighties by DARPA (Defense Advanced Research Projects Agency). SIMNET focuses on military training and is based on DIS (Distributed Interactive Simulation) (a standardized protocol [Locke 1995; IEEE 1995]), which introduces Dead-reckoning (extrapolation algorithm to deduce position of objects with their last known position and speed), a now well-known technique, commonly used in network video games. Dead-reckoning allows to keep the simulation synchronization below a certain threshold without too many network exchanges. Dead Reckoning is useful especially when the simulation is shared by a great number or participants. Some middlewares [Margery et al. 2002; Allart et al. 2004] provides tools to extend virtual reality applications into collaborative ones. In OpenMask [Margery et al. 2002], each object is simulated on a single host and broadcasts its state to replicated images: as a result, simulation cost is spread among hosts. FlowVR [Allart et al. 2004] is a middleware that eases the design of parrallel/distributed virtual reality applications. FlowVR is based on the data flow paradigm and component programming. FlowVR also introduces synchronization components that insure coherency according to different user-selected policies. In the field of physical simulation, there is a strong application need, especially in surgical simulation for collaborative simulation (e.g. distance diagnostic, and teaching). Some frameworks dedicated to surgery [Biocomp ; Gunn et al. 2005; Morris et al. 2004; Hutchins et al. 2005; Montgomery et al. 2002] proposed basic solutions for collaborative use of haptic. Those frameworks mostly rely on client/server approaches, where the simulation is computed on a single machine, and states modifications are transmitted to client hosts. Clients are only in charge of displaying the simulation, and managing the user haptic device. Because of network latency issue, most frameworks rely on local networks [Biocomp ; Morris et al. 2004], although some other can handle quite significant latencies for physical simulation context (about 200 milliseconds) using specialized physics model [Gunn et al. 2005]. The framework described in [Faure et al. 1999] is dedicated to collaborative animation in which some objects may be simulated. The aim of this framework is to make people from different physical places (mostly laboratories in the mentioned work) collaborate in a single virtual environment. As research sites usually develop their own applications, a common interfacing language is presented (called PaVR) for exchanging data. A client-server architecture is used, as the server keeps an up-to-date version of the global database. The server only updates client when receiving an explicit request from the client. In case of multiple interaction types and simulated objects, the issue of synchronization appears especially sensible, due to the heterogeneity of hardware and applications.

3 How to handle network latency ? The main point when trying to provide solution for collaborative physical simulation is obviously network latency. Indeed, real networks introduce non-negligible latency delay, in regard of simula-

tion frequencies that are usually needed for good simulation behavior, and satisfactory interaction. Client/server architectures present the advantage to remains coherent in regard of classic physical simulation rules, especially guaranteeing physical coherence throughout simulation. Yet, they also present the drawback to remain heavily dependant on the network performances. In the worst case, network shutdown freezes all the client, and no interaction with the simulation is any longer possible. Distributed structures, where each computers hold a part of the global simulation, suffers of the same weaknesses: apart from all the network protocol skill such an approach demands, as long as network performance decrease, interaction is directly stopped. In order to preserve interaction, whatever the network performances are, it is then necessary to use a system that is mostly based on a set of local, classical, simulation. Each user would then have its own simulation. Such an approach presents the important difficulty that collaborative interaction can not be done synchronously: the case where two different users A and B (respectively on hosts hA and hB ) achieve some interaction with a model at the same simulation time t: this can not be properly (i.e. in a classical way) handled by such system. Indeed, when hA will receive interaction information of B from hB , simulation on hA will already, because of network latency, have taken into account interaction by A. Yet, in most cases, one can assume that such information will arrive quite fast from hB after numerical treatment of simulation time t on hA . This assumption tends to induce that, if one could consider physical simulation that relaxes the classical condition that all the involved objects are considered at the same simulation date, it would be then possible to combine state variables coming from different simulation host, that have been calculated at slightly different simulation time. This is the principle that has led to the proposition for collaborative simulation we formulate here, that is being developed in next section.

4 Collaborative Architecture As explained in the previous section, we have, within the collaborative context, to relax the synchronization condition in order to preserve interaction whatever the network performance is. In order to get full flexibility in our architecture, we chose to apply such a condition to all simulated objects. Once settled on a single computer, this loose-synchronization between different objects can be extended in a very simple manner to different objects on different computers. However, our goal is to tend to a global synchronized simulation, which means that the simulation state on one host will have to remain close to the states on other hosts. This section presents our proposition for handling collaborative physical simulation, for arbitrary number of hosts. We first describe the simulation architecture used on each host, then we propose a flexible approach for loose synchronization between hosts.

4.1 Loose-synchronization simulation framework We describe here the structure of physical simulation on a single host. The framework used to implement collaborative simulation is fully described in [Dequidt et al. 2004]. This framework is designed for physical simulation of both rigid and deformable bodies. It is based on a multi-agents architecture where each simulated object is considered as an agent that tends to be as autonomous as possible. This means that most computations (mechanics, collision, geometry updating) is done by the agents themselves and not by a centralized classical simulation loop. Such a structure presents, among other, the advantage that objects gain significant flexibility

(implicit or explicit integration scheme, rigid or deformable mechanics, different simulation frequencies, . . . ), since autonomous objects share a minimal interface to interact with other objects and with the environment. A scheduler divides the system resources among all agents. This scheduler is based on priority and is associated with a set of N + 1 queues Qi where the integer i ∈ [0, N] corresponds to a priority. For a given queue Qi , a FIFO algorithm is applied and some control is done to prevent starvation for tasks with lower priorities. When a task is activated, it can dynamically change its priorities according to the state of the environment. To finely control the priority modification, a special component is added to each autonomous agent. This component is composed of simple decision rules that modify the behavior of the agent (see figure 2): in a physical simulation, it is classically required to have all simulated objects at the same simulation time. The simulation time is computed with the number of simulation iterations and the integration step. The role of behavior tuning is to ensure that all autonomous agent will tend to the same simulation time. When an agent is activated by the scheduler, it also receives information about the environment and especially about the fastest and the slowest agent (e.g. the agent with the highest/smallest simulation time). Decision rules that control the agent will require a smaller priority if the agent is too fast (decreasing priority will give less system resources and thus the agent will be slowed down) and higher priority for too slow agents. The behavior component of each agent and its decision rules tend to reduce dynamically the gap between faster and slower agents to a predefined threshold thus making the simulation consistent. The use of penalty-based method provides robust treatment of the induced error. As a result, agents (objects) are not fully synchronized, but remain, under some given tolerance, at consistent simulation date [Dequidt et al. 2004].

4.2 Collaborative framework Our collaborative framework differs from typical client/server or distributed approaches. Client/server approach only relies on the performance of the network and the server. As the server manages to compute the state of the whole environment, no additional synchronization protocol is required on client side. Distributed approach shares simulated objects among available hosts and simulated objects broadcast their state to other hosts. However, when an host is disconnected or achieve bad performance its simulated objects need to migrate to other hosts. This delay is not disturbing when dealing with animated or position-interpolated objects because their state can be computed easily. However simulated objects involve complex behaviors and many events (collision and interactions) and delaying the simulation of an object breaks the synchronization of the whole simulation. The chosen approach provides a framework that robustly handles network latency, while preserving level of interaction. We choose to extend the framework described above, and as already mentioned, we propose to compute the whole simulation on every hosts. This redundant computation ensures that the simulation will not be disabled on hosts in case of disconnection or network problems. However, for trying to make the simulation consistent, synchronization policies need to be added to control the behaviors of simulated objects. This synchronization policies are implemented by ghost objects. The light-weight transport protocol UDP was preferred to TCP for exchanging data over the network. Packets loss due to UDP is not a problem since the simulation is computed on each sites, thus it only leads to divergence between the simulation state on each hosts. These divergences as well as bad-arranged packets are treated by the ghost objects to tend to a whole synchronized and coherent state (see figure 3 for an overview of the framework).

Scheduler

Scheduler

Scheduler

Behavior

Behavior Simulated Object

Behavior

Behavior

Interface

Interface

Geometry

Interaction Geometry

Mechanics

Mechanics Interaction

Interface

Mechanics

Mechanics Interaction

Simulated Object

Interface

Simulated Object

Simulated Object

Geometry

Geometry Ghost Object Autonomous Agent

Interaction

Ghost Object Autonomous Agent

Network Autonomous Agent

Autonomous Agent

Figure 2: Physical simulation framework with loosesynchronization. Behavior components allow to insure consistency of the simulation.

Figure 3: Overview of our collaborative framework : ghost objects as components to exchange data over the network and to insure simulation consistency.

4.3 Ghost object algorithmic For the targeted purpose, i.e. collaborative physical simulation, this multi-agent approach has the immediate advantage to ease the design of agents that encapsulates our simulation object to exchange data over the network. Moreover, this framework allows precise control of the behavior of simulated objects and, as will be described below, to propose a solution to low-synchronization problems due to network latency.

As mentioned before, simulated objects are computed on every hosts and their state is transmitted on the network. Some treatment is needed to help simulated objects to integrate data they received. A special component called Ghost Object that encapsulates a simulated object is created. Ghost Object role is to control the behavior of a simulated object. It will listen about data on the network and,

at each simulation step, combine host simulation with data received over the network. Ghost object schematic principle is illustrated on figure 4. Ghost objects shares data with two threads : a receiving thread called Gather Thread and a sending one called Scatter Thread. These two threads are in charge of exchanging data over the network. For the sake of clarity, we use the following notations: on host Hi (i ∈ {1, · · · , n}), for a simulated object σ , Sσi (tσi ) is the state vector at simulation time tσi (dependent both of host number and simulated object, see section 4.1). On host Hi , and for object σ , ghost object Gσi shares with the Gather Thread a set of n − 1 vectors i, j Qσ (corresponding to data received for object σ from host j) that the Gather Thread fills with data from the network. Giσ also shares with the Scatter Thread a buffer Bσi that the Scatter Thread will i, j i, j look at to send data. Each message qσ in the vector Qσ is timei, j stamped with its corresponding simulation time (field qσ .time) and contains all the data that characterize the state of a simulated object j i, j i, j (field qσ .data = Sσ (qσ .time)).

Ghost Object Scatter Thread

Buffer (B)

Gather Thread

Queue (Q)

the network informations. Such weighted sum ensure timeshiftinvariance of the correction process. Next subsection shows that is actually possible, through different version of function getWeight, to adapt the global behavior of the simulation, and provides some examples.

4.4 Decision rules The getWeight function, introduced in the algorithm describing ghost object behavior, is the key one can use for adapting synchronization to the collaboration needs. Depending on the chosen weight calculation, one can adapt synchronization to the context. j We use the same notations as in the previous section, i.e. wk stands for the weight corresponding to the k-th data coming from host j. Since k is not significant in the following details, we omit it and only consider w j , weight of data coming from host j to host i, for object σ . Full flexibility is provided for weight definition. So far, we based all our test on simulation time difference. We propose here two weight calculation, that exhibit the flexibility of the proposed framework: • client/server: Consider the following function f defined on host i: j

xmax = max(tσ − tσi ) j

½ f (x) =

Network Figure 4: Ghost object access to the network: shared data and two threads.

1 i f f x = xmax 0 else

Such function, when used as j

In order to make notations as light as possible, when nonambiguous, subscript σ and superscript i will be omitted in the rei, j mainder of this article (e.g. qσ will be written as q j , tσi will be written as t, and so on). The main loop of Gσi is described in the following algorithm: GHOST O BJECT U PDATE S TATE ()

1 2 3 4 5 6 7 8 9

S(t) + 10

S(t) =

∑

(

j∈[1,n], j6=i

1+

∑

∑

wj

j

wk Q j [k].data)

k=1

j∈[1,n], j6=i

size(Q j )

(

∑

enables a client-server like approach. The simulated object with the highest simulation time (for example Si ) will impose its state on the other hosts e.g. the S j6=i will only copy the state of Si as the weight associated for what they have computed is null. • The exponential function also has some interesting properties, as a potential weight function. Using

callSimulationStep() B ← getNewSimulationState() sendNoti f icationEvent() for each j ∈ {1, · · · , i − 1, i + 1, · · · , n} do for k = 1 to size(Q j ) do j wk ← getWeight(Q j [k],t) size(Q j )

w j = f (tσ − tσi )

j

wk )

k=1

The callSimulationStep allows the simulated object to compute its next state. The computed values that describes the state of the object (simulation time, position, speed, momentum) are stored in the buffer B. Using received data, stored in the vectors Q j , a weighted sum is achieved between the locally computed state vector and

=

j

i

e(tσ −tσ )

it allows to reduce influence of slowest simulated objects (as their computed weight will be very small) and to give more importance to fastest ones. Moreover, as the data modification works for all values that characterize the simulation state, this allow slowest hosts to catch up their delay. As all the ghost object uses the same decision rules, and as the weight function tends to decrease the gap between distant temporal (for simulation time) and spatial (for state variables) values, such a scheme ensures global stability, in interaction cases where it is possible. Indeed, when all hosts should, according to user interactions, reach some stable state, dynamic data modification is such that all the hosts will asymptotically reach the same physical state. This ensures that, when no interaction is provided by any user, all the host simulations converge to a global, coherent and physically consistent, situation.

4.5

Simulation global convergence

Our system does not guarantee, in the general case, that all simulation host, at a given human time, provide the same instantaneous simulation snapshot: in the general case, all hosts state vectors are different. In order to understand the main argument targeted in this section, further arguments need to be reminded on classical solutions for handling mechanical constraints between physical systems.

A strong correspondance can be established, from numerical point of view, between the problem of constraint respect between two animated objects (such as problems treated in [Cani and Gascuel 1994]), and the problem of constraining two animated physical systems to be equal over the network. Since numerical tools are of the same type, convergence results are the same. This similarity tends to imply, since post-correction methods for mechanical constraints work well in practice, that the same class of method will work well in the case of collaborative physical simulation: this, in our tests, appears to be true: the provided system makes all the hosts tend to the same, physically consistent, configuration, at least in the case of deterministic simulations (chaos cases such as irregular rock falling on a mountain, i.e. situations where small numerical error induces completely different physical animation, is not taken into account by our system)

5 Tests and applications

Figure 5: Collaborative test scene mixing rigid and deformable objects.

ms. The companion video shows that the differences between the recorded distant simulations are small enough to provide illusion of synchronized simulation for distant users. Obviously, the delay between the simulation times depends of the network that is used for exchanging data. Low latency networks can provide more frequent updates to hosts to make them converge to a global synchronized state. On high latency network or in case of brief loss of connection, simulations may diverge due to higher delays but the synchronization policy will tend to keep the delay small. 0.045

0.04

0.035 Delay between distant simulations

Mechanical constraints can be solved with many approaches. The first family of methods is based on an exact resolution with for example Lagrange multipliers. This kind of method is quite expensive but ensures that constraints are not violated. The second family [Cani and Gascuel 1994] which is based on post-correction methods does not ensure that constraints will be respected but will try to tend to a state where no constraint is violated. Most of those techniques provide lower performance than previous ones, usually induce more intertia within system, but globally provides good constraint respect in most cases [Cani and Gascuel 1994].

0.03

0.025

0.02

0.015

0.01

This section presents the results obtained with this collaborative work. Tests were performed on Intel PIV (whose frequency runs from 2.0 Ghz to 2.4Ghz) with 512 MB or 1 GB Ram and connected on a local ethernet 100 MB/s network. The test scenes were composed of several interacting rigid or mass-spring bodies (see figures 1, 5). Figure 5 shows an example of test scene: a cloth is left hanging by its four corners and rigid spheres fall on it. Each user can interact with the simulated objects via a probe. In our example, two users have been added, and access the scene throughout network using respectively green and purple polyhedra, on figure 5. On the snapshot, as well as in the companion video, the green probe stands for the local probe. the purple one is the distant representation of the other user probe, which explains the color inversion one can see in the video. The key point is to ensure that the simulation is synchronized between the hosts. We tried to estimate the delay between the simulations, which is in itself a difficult thing to do, considering that we have to measure decay between two distant host, and that expected decays are close to network performances, which is also a problem when handling shared clock. The most significant value to measure is the simulation time, which represents the time achieved in the simulation. Samples were taken at regular steps and the difference between the simulation times was computed. The results are presented in figure 6. On the local network we achieved our tests, he simulation date decay is less than 40 ms and most often below 20

0.005

0 0

50

100

150

200 250 Samples

300

350

400

450

Figure 6: Delays between two distant simulations(in seconds).

6 Conclusion We presented in this article a framework that can handle collaborative work on rigid and deformable models. The proposed solution is resistant to low-performance networks. unlike client/server based solutions, it maintains the level of interaction in case of network shutdown. It also ensures that the simulated scene goes to the same (coherent and physically consistent on all hosts) state when no interaction is provided, in the case of deterministic simulation. For improving the proposed solution, it would be nice to combine such an approach to distributed haptic. That way, user could for example get experience of multiple access to the same interaction zone. This approach by providing tools for hosts to control received data can also lead to load balancing of the physical simulation. Indeed if an object notes that the computation it has done is useless because information it has received are more recent, it can decide

to stop for a while its simulation, giving more computation power for the simulation of the other objects.

References

M OSEGAARD , J., H ERBORG , P., AND S RENSEN , T. S. 2005. A GPU accelerated spring mass system for surgical simulation. In Medicine Meets Virtual Reality (MMVR). OTADUY, M. A., AND L IN , M. C. 2005. Stable and responsive six-degree-of-freedom haptic manipulation using implicit integration. In Proc. of World Haptics Conference.

A LLART, J., G OURANTON , V., L ECOINTRE , L., L IMET, S., M ELIN , E., R AFFIN , B., AND ROBERT, S. 2004. Flowvr : a middleware for large scale virtual reality applications. In EuroPar. B IOCOMP. http://biocomp.stanford.edu/tech/tech pages/distrib haptics.html. C ALVIN , J., D ICKEN , A., G AINES , B., M ETZGER , P., M ILLER , D., AND OWEN , D. 1993. The simnet virtual world architecture. In IEEE Virtual Reality Annual International Symposium, 450– 455. C ANI , M.-P., AND G ASCUEL , J.-D. 1994. Displacement constraints for interactive modeling and animation of articulated structures. The Visual Computer 10, 4 (mars). Published under the name Marie-Paule Gascuel. D EQUIDT, J., G RISONI , L., AND C HAILLOU , C. 2004. Asynchronous interactive physical simulation. Tech. Report RR-5338, Institut National de Recherche en Informatique et en Automatique (INRIA). FAURE , F., FAISSTNAUER , C., H ESINA , G., AUBEL , A., E S CHER , M., L ABROSSE , F., N EBEL , J.-C., AND G ASCUEL , J.D. 1999. Collaborative animation over the network. In IEEE Proceedings of Computer Animation. GPGPU. http://www.gpgpu.org. G UNN , C., H UTCHINS , M., S TEVENSON , D., AND A DCOCK , M. 2005. Using collaborative haptics in remote surgical training. In Worldhaptics, 481–482. H ARRIS , M. J. 2004. GPU Gems : Programming Techniques, Tips, and Tricks for Real-Time Graphics. Addison Wesley. H UTCHINS , M., O’L EARY, S., S TEVENSON , D., G UNN , C., AND K RUMPHOLZ , A. 2005. A networked haptic virtual environment for teaching temporal bone surgery. In Medicine Meets Virtual Reality (MMVR). IEEE. 1995. Standard for distributed interactive simulation - application protocols. Tech. Rep. IEEE 1278, Institute for Simulation and Training. L OCKE , J. 1995. An introduction to the internet networking environment and simnet/dis. Tech. report, Computer Science Department, Naval Postgraduate Schhol. M ARGERY, D., A RNALDI , B., C HAUFFAUT, A., D ONIKIAN , S., AND D UVAL , T. 2002. Openmask : Multi-threaded or modular animation and simulation kernel or kit : a general introduction. In Virtual Reality International Conference, 101–110. M ONTGOMERY, K., B RUYNS , C., B ROWN , J., S ORKIN , S., M AZZELLA , F., T HONIER , G., T ELLIER , A., L ERMAN , B., AND M ENON , A. 2002. Spring: A general framework for collaborative, real-time surgical simulation. In Medicine Meets Virtual Reality (MMVR). M ORRIS , D., S EWEEL , C., B LEVINS , N., AND BARBAGLI , F. 2004. A collaborative virtual environment for the simulation of temporal bone surgery. In MICCAI.