Virtual Reality — in virtuo autonomy - ENIB

University of Rennes 1 Accreditation to Direct Research Field: Computer Science — Thesis —

Virtual Reality — in virtuo autonomy — Jacques Tisseau

Defense: December 6, 2001 Herman Bourdon Coiffet Thalmann Abgrall Arnaldi

D. F. P. D. J.F. B.

Chairperson Reporter Reporter Reporter Examiner Examiner

2

Contents

Contents

3

List of Figures

5

1 Introduction 1.1 An Epistemological Obstacle . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 An Original Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7 7 8

2 Concepts 2.1 Introduction . . . . . . . . . . . . . . . . . . 2.2 Historical Context . . . . . . . . . . . . . . 2.2.1 Constructing Images . . . . . . . . . 2.2.2 Animating Images . . . . . . . . . . 2.2.3 Image Appropriation . . . . . . . . . 2.2.4 From Real Images to Virtual Objects 2.3 The Reality of Philosophers . . . . . . . . . 2.3.1 Sensory/Intelligible Duality . . . . . 2.3.2 Virtual Versus Actual . . . . . . . . 2.3.3 Philosophy and Virtual Reality . . . 2.4 The Virtual of Physicists . . . . . . . . . . . 2.4.1 Virtual Images . . . . . . . . . . . . 2.4.2 Image Perception . . . . . . . . . . . 2.4.3 Physics and Virtual Reality . . . . . 2.5 Principle of Autonomy . . . . . . . . . . . . 2.5.1 Operating Models . . . . . . . . . . . 2.5.2 User Modeling . . . . . . . . . . . . . 2.5.3 Autonomizing Models . . . . . . . . 2.5.4 Autonomy and Virtual Reality . . . . 2.6 Conclusion . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . .

11 11 11 12 13 15 16 17 18 19 21 22 22 25 26 26 27 28 29 31 33

3 Models 35 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.2 Autonomous Entities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3

Contents

3.3

3.4

3.5

3.2.1 Multi-Agent Approach . . . . . . . . 3.2.2 Simulation multi-agents participative 3.2.3 Mod´elisation comportementale . . . . Collective Auto-Organization . . . . . . . . 3.3.1 Simulations in Biology . . . . . . . . 3.3.2 Coagulation Example . . . . . . . . . 3.3.3 In Vivo, In Vitro, In Virtuo . . . . . Individual Perception . . . . . . . . . . . . . 3.4.1 Sensation and Perception . . . . . . . 3.4.2 Active Perception . . . . . . . . . . . 3.4.3 The Shepherd, His Dog and His Herd Conclusion . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

4 Tools 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . 4.2 The oRis Language . . . . . . . . . . . . . . . . . 4.2.1 General View . . . . . . . . . . . . . . . . 4.2.2 Objects and Agents . . . . . . . . . . . . . 4.2.3 Interactions between Agents . . . . . . . . 4.3 The oRis Simulator . . . . . . . . . . . . . . . . . 4.3.1 Execution Flows . . . . . . . . . . . . . . 4.3.2 Activation Procedures . . . . . . . . . . . 4.3.3 The Dynamicity of a Multi-Agent System 4.4 Applications . . . . . . . . . . . . . . . . . . . . . 4.4.1 Participatory Multi-Agent Simulations . . 4.4.2 Application Fields . . . . . . . . . . . . . 4.4.3 The AR´ eVi Platform . . . . . . . . . . . . 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

36 37 39 40 40 43 47 49 50 54 55 57

. . . . . . . . . . . . . .

61 61 61 62 64 68 70 71 72 75 77 77 79 80 81

5 Conclusion 83 5.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Bibliography

4

87

List of Figures 1.1

Pinocchio Metaphor and the Autonomization of Models . . . . . . . . . . . . .

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10

9

Reality and the Main Philosophical Doctrines . . . . . . . . . . . The Four Modes of Existence . . . . . . . . . . . . . . . . . . . . The Three Mediations of The Real . . . . . . . . . . . . . . . . . Optical System . . . . . . . . . . . . . . . . . . . . . . . . . . . . Real Image (a) and Virtual Image (b) . . . . . . . . . . . . . . . . Virtual Image (a), Real Image (b), and Virtual Object (c) . . . . Illusions of Rectilinear Propagation and Opposite Return . . . . . The Three Mediations of Models in Virtual Reality . . . . . . . . The Different Levels of Interaction in Virtual Reality . . . . . . . User Status in Scientific Simulation (a), in Interactive Simulation Virtual Reality (c) . . . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Autonomy and Virtual Reality . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . (b) . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . and in . . . . . . . .

. . . . . . . . .

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12

Cell-Agent Design . . . . . . . . . . . . . . . . . . . . . . Interactions of the Coagulation Model . . . . . . . . . . Evolution of the Multi-Agent Simulation of Coagulation Main Results of the Multi-Agent Coagulation Model . . Modeling and Understanding Phenomena . . . . . . . . . Example of a Cognitive Map . . . . . . . . . . . . . . . . Definition of a Fuzzy Cognitive Map . . . . . . . . . . . Flight Behavior . . . . . . . . . . . . . . . . . . . . . . . Flight Speed . . . . . . . . . . . . . . . . . . . . . . . . . Perception of Self and Others . . . . . . . . . . . . . . . Fear and Socialization . . . . . . . . . . . . . . . . . . . The Sheepdog and Its Sheep . . . . . . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

. . . . . . . . . . . .

44 45 46 47 50 51 52 53 53 56 57 58

4.1 4.2 4.3 4.4 4.5 4.6

Execution of a Simple Program in oRis oRis Graphics Environment . . . . . . Localization and Perception in 2D . . . Splitting an Execution Flow in oRis . . Structure of the oRis Virtual Machine Modifying an Instance . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

63 64 66 71 72 76

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

18 20 22 23 23 24 25 27 28

. 30 . 32

5

List of Figures

4.7 4.8

6

An AR´ eVi Application: Training Platform for French Emergency Services . . . 81 Metaphor of Ali Baba and Programming Paradigms . . . . . . . . . . . . . . . 82

Chapter 1 Introduction Created from two words with opposing meanings, the expression virtual realities is absurd. If someone were to talk to you about the color black-white, you would probably think that, because this person was combining a word with its opposite, he was confused. Certainly though, in all accuracy, virtual and reality are not opposites. Virtual, from Latin virtus (vertue, or force), is that which is in effect in the real, that which in itself has all the basic conditions for its realization; but then, if something contains the conditions of its realization in itself, what can truly be a reality? When looked at from this perspective, the expression is inadequate. [Cadoz 94a]

The English expression virtual reality was coined for the first time in July of 1989 during a professional trade show1 by Jaron Lanier, founder and former CEO of the VPL Research company specializing in immersion devices. He created this expression as part of his company’s marketing and advertising strategy, but never gave it a precise definition.

1.1

An Epistemological Obstacle

The emergence of the notion of virtual reality illustrates the vitality of the interdisciplinary exchanges between computer graphics, computer-aided design, simulation, teleoperations, audiovisual, etc. However, as the philosopher Gaston Bachelard points out in his epistemological study on the Formation of the Scientific Mind [Bachelard 38], any advances that are made in science and technology must face numerous epistemological obstacles. One of the obstacles that virtual reality will have to overcome is a verbal obstacle (a false explanation obtained from an explanatory word): the name itself is meaningless a priori, yet at the same time, it refers to the intuitive notion of reality, one of the first notions of the human mind. According to the BBC English Dictionary (HarperCollins Publishers, 1992), Virtual: means that something is so nearly true that for most purposes it can be regarded as true, it also means that something has all the effects and consequences of a particular thing, but is not officially recognized as being that thing. Therefore, virtual reality is a quasi-reality that looks and acts like reality, but is not reality: it is an ersatz, or substitute for reality. In French, the literal translation of the English expression is r´ealit´e virtuelle which, as pointed out in the quote at the beginning of this introduction, is an absurd and inadequate expression. The definition according to the Petit Robert (Editions Le 1

Texpo’89 in San Francisco (USA)

7

Introduction

Robert, 1992) is: Virtuel : [qui] a en soi toutes les conditions essentielles a` sa r´ealisation. Therefore, r´ealit´e virtuelle would be a reality that would have in itself all of the basic conditions for its realization; which is really the least for reality! So, from English to French, the term virtual reality has become ambiguous. It falls under a rhetorical process called oxymoron, which combines two words that seem incompatible or contradictory (expressions like living dead, vaguely clear and eloquent silence are all oxymora). This type of construction creates an original, catchy expression, which is geared more towards the media than science. Other expressions such as cyber-space [Gibson 84], artificial reality [Krueger 83], virtual environment [Ellis 91] and virtual world [Holloway 92], were also coined, but a quick Web search2 proves that the antonymy virtual reality remains widely used. The term’s ambiguity is confusing and creates a real epistemological obstacle to the scientific development of this new discipline. It is up to the scientists and professionals in this field to create a clearer epistemological definition in order to remove any ambiguities and establish its status as a scientific discipline (concepts, models, tools); particularly in regards to areas such as modeling, simulation and animation.

1.2

An Original Approach

This document proposes an original contribution to the idea that this new disciplinary field of virtual reality needs to be clarified. The word autonomy best characterizes our approach and is the common thread that links all of our research in virtual reality. Everyday, we face a reality that is resistant, contradictory and that follows its, and not our, own rules; in one word, a reality that is autonomous. Therefore, we think that virtual reality will become independent of its origins if it autonomizes the digital models that it manipulates in order to populate realistic universes, which are already created through computer-generated digital images for us to see, hear and touch, with autonomous entities. So, like Pinocchio, the famous Italian puppet, models that become autonomous will take part in creating their virtual worlds (Pinocchio Metaphor3 , Figure 1.1). A user, partially free from controlling his model, will also become autonomous and will take part in this virtual reality as a spectator (the user observes the model: Figure 1.1a), actor (the user tries out the model: Figure 1.1b) and creator (the user modifies the model to adapt it to his needs: Figure 1.1c). This document has three main focuses: the concepts that guide our work and are the basis of the principle of autonomy (Chapter 2), the models that we developed to observe this principle (Chapter 3), and the tools that we used to implement them (Chapter 4). In Chapter 2, analyzing the historical context, as well as exploring the real in philosophy and the virtual in physics, bring us to a clearer definition of virtual reality that centers 2

A simple search using the search engine www.google.com gives the following results: cyber(-)space(s) ≈ 25,540 hits, virtual reality(s) ≈ 19,920, virtual world(s) ≈ 15,030, virtual environment(s) ≈ 3,670, artificial reality(s) ≈ 185. In English the classification is the same. 3

This metaphor was inspired by Michel Guivarc’h, official representative of the Communaut´e Urbaine de Brest (Brest Urban Community).

8

An Original Approach

a. user-spectator

b. user-actor

c. user-creator

Figure 1.1: Pinocchio Metaphor and the Autonomization of Models around the principle of autonomy of digital models. In Chapter 3, we recommend a multi-agent approach for creating virtual reality applications based on autonomous entities. We use significant examples to study the principle of autonomy in terms of collective auto-organization (example of blood coagulation) and individual perception (example of a sheepdog). Chapter 4 describes our oRis language — a programming language that uses concurrent active objects, which is dynamically interpreted and has an instance granularity — and its associated simulator; the tool that helped build the autonomous models in the applications that we had already created. Finally, in Chapter 5, the conclusion, we review our initial research and outline the future direction of research in perspectives.

9

Introduction

10

Chapter 2 Concepts 2.1

Introduction

In this chapter, we present the main epistemological concepts that guide our research work in virtual reality. First, we will review the historical conditions surrounding the emergence of virtual reality. We will establish that in the early 90’s — when the term virtual reality appeared — the sensory rendering of computer-generated images had become so realistic that it became necessary to focus more specifically on virtual objects than their images. Secondly, an overview of the philosophical notion of reality will allow us to define the conditions that we feel are necessary for the illusion of the real; in other words, perception, experiments and modification of the real. Next, we will use the notion of virtual in physics to explain our design of a reality that is virtual. Several simple, yet significant, examples borrowed from optics, demonstrate the importance of underlying models in understanding, explaining and perceiving phenomena. Virtual will then definitely emerge as a construction in the model universe. Finally, by analyzing the user’s status while he is using these models, we will postulate an autonomy principle that will guide us in designing our models (Chapter 3) and creating our tools (Chapter 4).

2.2

Historical Context Epistemology is not particularly concerned with establishing a chronology of scientific discovery, it concentrates instead on updating the conceptual structure of a theory or work, which is not always evident, even for its creator. Epistemology must try and figure out what connects concepts and plays a vital role in the architecture of the structure: recognizing what is an obstacle and what remains unclear and unsettled. To this effect, it should focus on proving exactly how the conditions of the science that it is studying is a response, critique, or maturation to previous states. [Granger 86]

Historically, the notion of virtual reality emerged at the intersection of different fields such as computer graphics, computer-aided design, simulation, teleoperations, audiovisual, etc. Consequently, virtual reality was developed within a multidisciplinary environment where computer graphics played an influential role because, ever since Sutherland created the first graphic interface (Sketchpad [Sutherland 63]) virtual reality has endeav-

11

Concepts

ored to make the computer-generated digital images that it displays on computer screens more and more realistic. Therefore, we will focus on the major developments in computer graphics before 1990 — during which the expression virtual reality became popular — by taking a look at image construction, animation and user appropriation.

2.2.1

Constructing Images

From 2D Images to 3D Images Computer graphics first had to undertake 2D problems such as displaying line segments [Bresenham 65], eliminating hidden lines [Jones 71], determining segment intersections for coloring polygons [Bentley 79], smoothing techniques [Pitteway 80] and triangulating polygons [Hertel 83]. Research departments in the aeronautical and automobile industries quickly understood the benefit of computer-graphic techniques and integrated them as support tools in designing their products, which led to the creation of computer-aided design (CAD). It quickly evolved from its initial wireframe representations, and added parametric curves and surfaces to it systems [B´ezier 77]. These patch surfaces were easy to manipulate interactively, and allowed the designer to create free shapes. However, when the object shape had to follow strict physical constraints, it was better to work with implicit surfaces, which are the isosurfaces of a pre-defined space scalar field [Blinn 82]. By triangulating these surfaces, physics problems involving plates and shells could then be solved by using digital simulation (finished elements methods [Zinkiewicz 71]), as well as a realistic display of the results [Sheng 92]. Still, surface modeling proved to be inadequate in ensuring the topological consistency of modeled objects. Therefore, volume modeling was developed from two types of representation: boundary representation and volume representation. Boundary representation models an object through a polyhedral surface made up of vertices, faces and edges. This type of representation requires that adjacency relationships between faces, edges and vertices, in other words, the model’s topology, be taken into account. The topology can be maintained through the edges (winged-edges [Baumgart 75], n-G-maps [Lienhardt 89]) or in a face adjacency graph (Face Adjacency Graph [Ansaldi 85]). Volume representation defines an object as a combination of primitive volumes with set operators (CSG: Constructive Solid Geometry [Requicha 80]). The object’s topology is defined by a tree (CSG tree) whose leaves are basic solid shapes and whose nodes are set operations, translations, rotations or other deformations (torsion [Barr 84], free-form deformations [Sederberg 86]) that are applied to underlying nodes. The elimination of hidden faces then replaced the elimination of hidden lines and split into two large categories [Sutherland 74]: visibility was either processed directly in the image-space, most often with a depth-buffer (z-buffer [Catmull 74]), or in the object-space by partitioning this space (BSP: Binary Space Partitionning [Fuchs 80], quadtree/octree [Samet 84]). Thus, wire-frame images became surface images and then became volume images through the use of geometric and topological models that looked more and more like real solid objects.

12

Historical Context

From Geometric Image to Photometric Image The visual realism of objects needed more than just precise shapes: they needed color, textures [Blinn 76] and shadows [Williams 78]. Therefore, computer graphics had to figure out how to use light sources to illuminate objects. Light needed to be simulated from its emission to its absorption at the point of observation, while taking into account the reflective surfaces of the scene that was being illuminated. The first illuminated models were empiric [Bouknight 70] and used interpolations of colors [Gouraud 71] or surface normals [Phong 75]. Then came the methods that were based on the laws of the physics of reflectivity [Cook 81]. Scenes were illuminated indirectly by shooting rays [Whitted 80], or through a radiosity calculation [Cohen 85]; the two methods could also be combined [Sillion 89]. From the observer’s perspective, in order to improve the quality of visual rendering the effects of the atmosphere on light [Klassen 87], as well as the main characteristics of cameras [Kolb 95], including their defaults [Thomas 86], needed to be taken into account. Hence, from representational art, computer-generated images progressively became more realistic by integrating laws of optics.

2.2.2

Animating Images

From Fixed Image to Animated Image During the early history of the movies, image sequences were manually generated to produce cartoons. When digital images arrived on the scene, these sequences could be generated automatically. At one time, a sequence of digital animation was produced by generating in-between images obtained by interpolating the key images created by the animator [Burtnyk 76]. Interpolation, usually the cubic polynomial type (spline) [Kochanek 84], concerns key positions of the displayed image [Wolberg 90] or the parameters of the 3D model of the represented object [Steketee 85]. In direct kinematics, time parameters are taken into account during interpolation. But, with inverse kinematics, a spacetime path for the components of a kinematic chain can be automatically reconstructed [Girard 85] by giving the furthest positions of the chain and evolution constraints. However, these techniques quickly proved to be limited in reproducing certain complex movements, in particular, human movements [Zeltzer 82]. Therefore, capturing real movements from human subjects equipped with electronically-trackable sensors [Calvert 82] or optical sensors (cameras + reflectors) [Ginsberg 83] emerged as an alternative. Nevertheless, all of these techniques only reproduced effects without any a priori knowledge of their causes. Methods based on dynamic models then operate on causes in order to deduce effects. Usually, this approach leads to models that are regulated by differential equation systems with limit conditions, which then have to be resolved in order to clarify the solution that it has implicitly defined. Different numerical resolution methods were then adapted to animation: direct methods by successive approximation (RungeKutta type) worked better for poly-articulated rigid systems [Arnaldi 89], structural discretization of objects by finite elements was used for deformable systems [Gourret 89],

13

Concepts

and structural and physical discretization of systems by mass-spring networks was used to model a large variety of systems [Luciani 85]. Overall problems of elastic [Terzopoulos 87] or plastic deformation [Terzopoulos 88] were dealt with as well as collision detection between solids [Moore 88]. Particular attention was paid to animating human characters, seen as articulated skeletons and animated by inverse dynamics [Isaacs 87]. In fact, animating virtual actors includes all of these animation techniques [Badler 93] because it involves skeletons as well as shapes [Chadwick 89], walking [Boulic 90] as well as gripping [Magnenat-Thalmann 88], facial expressions [Platt 81] as well as hair [Rosenblum 91], and even clothes [Weil 86]. Through kinematics and dynamics, animation sequences could be automatically generated so that their movements were closer to real behavior.

From The Pre-Calculated Image to The Real-Time Image At the beginning of the 20th century the first flight simulators had instruments, but they had no visibility; computer-generated images introduced vision, first for simulating night flights, (points and segments in black and white), then daytime flights (colored surfaces with the elimination of hidden faces) [Schumacker 69]. However, training simulators required response times that corresponded to the simulated device in order for pilots to learn its reactive behavior. Realistic rendering methods based on the laws of physics resulted in calculation times in flight simulators that were often too long compared with actual response times. Therefore, during the 60’s and 70’s, special equipment such as the real-time display device1 was created to accelerate the different rendering processes. These peripheral devices, which generated real-time computer-generated images, were slowly replaced by graphics stations that would both generate and manage data structures on a general-use processor and render it on a specialized graphics processor (SGI 4D/240 GTX [Akeley 89]). At the same time, someone figured out how to reduce the number of faces to be displayed through processes that were carried out before graphic rendering [Clark 76]. On the one hand, the underlying data structures makes it possible to identify, from the observer’s perspective, objects located outside of the observation field (view-frustum culling), poorly directed faces (backface culling) and concealed objects (occlusion culling) so that these objects are not transmitted to the rendering engine. On the other hand, the farther away that an object is located from an observer, the more its geometry can be simplified while maintaining an appearance, or else a topology, that is satisfactory for the observer. This simplification of manual or automatic meshing that generates discrete or continual levels of detail, proved to be particularly effective for the digital land models that are widely used in flight simulators [Cosman 81]. Thus, thanks to optimization, calculation devices, and better adapted hardware devices, images can be calculated in real-time. 1

The Evans and Sutherland company (www.es.com) created the first commercial display devices connected to a PDP-11 computer. For several million Dollars, they allowed you to display several thousand polygons per second. Today, a GeForce 3 for PC graphics card (www.nvidia.com) reaches performances in the order of several million polygons per second for several thousand French Francs.

14

Historical Context

2.2.3

Image Appropriation

From Looking at an Image to Living an Image Calculating an image in real-time means that a user can navigate within the virtual world that it represents [Brooks 86a], but it was undoubtedly artistic experiments and teleoperations that led users to start appropriating images. In the 70’s, interactive digital art allowed users to experiment with situations where digital images evolved through the free form movements of artists who observed, in real-time and on the big-screen, how their movements influenced images (VideoPlace [Krueger 77]). With the work of ACROE2 , computer-aided artistic expression took a new step with multi-sensory synthesis aided by a force feedback management control [Cadoz 84]. The peripherals, called retroactive gestural transducers, allow users, for example, to feel the friction of a real bow on a virtual violin string, to hear the sound made by the vibration of the cord and to observe the cord vibrate on the screen. A physical modeling system of mass-spring virtual objects pilots these peripherals (CORDIS-ANIMA [Cadoz 93]), and allows users to have a completely original artistic experience. For its part, teleoperations became involved very early in ways to remotely control and pilot robots located in dangerous environments (nuclear, underwater, space) in order to avoid putting human operators at risk. In particular, NASA developed a liquid crystal head mounted display (VIVED: Virtual Visual Environment Display [Fisher 86]) coupled with an electronically-trackable sensor [Raab 79] that allowed users to move around in a 3D image observed in stereovision. This was a real improvement in comparison to the first head mounted display, which had cathode ray tubes, was heavier and was held on top of the head by a mechanical arm fixed to the ceiling [Sutherland 68]. This new display was very quickly combined with a dataglove [Zimmerman 87] and a spatialized sound system based on the laws of acoustics [Wenzel 88]. These new functions enhanced the sensations of tactile feedback systems [Bliss 70] and the force feedback systems (GROPE [Batter 71, Brooks 90]) that were already familiar to teleoperators. The robotics experts were using expressions such as telesymbiosis [Vertut 85], telepresence [Sheridan 87] and tele-existence [Tachi 89] to describe the feeling of immersion that operators can have; the feeling that they are working directly on a robot, for example, even though they are manipulating it remotely. As it has since the beginning of teleoperations, adding an operator to the loop of a robotized system poses the problem of how to include human factors to improve the ergonomics of workstations and their associated peripherals [Smith 89]. By becoming multimodal, the image allows users to see, hear, touch and move or deform virtual objects with sensations that are almost real.

2

ACROE: Association (www-acroe.imag.com)

for

the

Creation

and

Research

on

Expression

Tools,

Grenoble

15

Concepts

From Isolated Image to Shared Image During the 70’s, ARPANET3 , the first long-distance network based on TCP/IP4 [Cerf 74] was developed. Given these new possibilities of exchanging and sharing information, the American Defense Department decided to interconnect simulators (tanks, planes, etc.) through the ARPANET network in order to simulate battle fields. So, the SIMNET (simulator networking) project, which took place between 1983 and 1990, tested group strategies that were freely executed by dozens of users on the network, and not controlled by pre-established scenarios [Miller 95]. The communication protocols established for SIMNET [Pope 87] introduced prediction models of movement (estimate: dead-reckoning) in order to avoid overloading the network with too many synchronizations, and at the same time made sure that there was a time coherence between simulators. These protocols were then summarized and standardized for the needs of the distributed interactive simulation (DIS: Distributed Interactive Simulation [IEEE 93]). Thus, the computer network and its protocols enabled several users to share images and then through these images to share ideas and experiences.

2.2.4

From Real Images to Virtual Objects

Interdisciplinarity This brief overview of the historical context from which the concept of virtual reality emerged illustrates the vitality of computer graphics, as also proven by events such as Siggraph5 in the United States and Eurographics6 or Imagina7 in Europe. This new expression and this new field of investigation, which is based upon computer graphics, simulation, computer-aided design, teleoperations, audiovisual, telecommunications, etc., is being formed within an interdisciplinary melting pot. Virtual reality thus seems like a catch-all field within the engineering sciences. It manipulates images that are interactive, multimodal (3D, sound, tactile, kinesthetic, proprioceptive), realistic, animated in real-time, and shared on computer networks. In this sense, it is a natural evolution of computer graphics. During the last ten years, a special effort has been made to integrate all of the characteristics of these images within the same system8 , and to optimize and improve all of the techniques needed for these systems. Virtual reality is therefore based on interaction in real time with virtual objects and the feeling of immersion in virtual worlds: it is like a multisensory [Burdea 93], instrumental [Cadoz 94b] and behavioral [Fuchs 96] interface. Today, the major works in virtual reality give a more precise and especially operational meaning to the expression virtual

16

3

ARPANET: Advanced Research Projects Agency Network of the American Defense Department.

4

TCP/IP : Transmission Control Protocol/Internet Protocol

5

Siggraph (www.siggraph.org): 29th edition in 2002

6

Eurographics (www.eg.org)

7

Imagina (www.imagina.mc): 21st edition in 2002

8

You will find in [Capin 99] a comparison of a certain number of these platforms.

The Reality of Philosophers

reality, which can be summed up by the definition established by the Virtual Reality Work Group9 : a set of software and hardware tools that can realistically simulate an interaction with virtual objects that are computer models of real objects.

Transdisciplinarity With this definition, we changed our approach from researching real images in computer graphics to researching visualized virtual objects. The objects are not only characterized by their appearances (their images), but also their behavior, which is no longer researched under computer graphics. So, images reveal both behaviors and shapes. Virtual reality is becoming transdisciplinary and is slowly moving away from its origins. In 1992, D. Zelter had already proposed an evaluation of virtual universes based on three basic notions: the autonomy of viewed objects, the interaction with these objects and the feeling of being present in the virtual world [Zelter 92]. A universe is then represented by a point on the AIP (autonomy, interaction, presence) index. Hence, on this index, 3D movies have coordinates (0.0.1), a flight simulator (0.1.1), and Zelter would place virtual reality at point (1.1.1). Still, we must point out that there was very little research done on the problem of virtual object autonomy until now. We can, nevertheless, note that animation, so-called behavior, introduced several autonomous behaviors through particle systems, [Reeves 83], flocks of birds [Reynolds 87], shoals of fish [Terzopoulos 94], and of course virtual humans [Thalmann 99]. An object’s behavior is considered autonomous if it is able to adapt to unknown changes in its environment: so, it must be equipped with ways to perceive, respond and coordinate between its perceptions and actions in order to react realistically to these changes. This notion of autonomy is the root of our problem. With this new perspective, we must return to the notion of virtual reality by exploring the real of philosophers (Section 2.3) and the virtual of physicists (Section 2.4) in order to understand why the autonomy of virtual objects is vital to the realism of the images that they inhabit (Section 2.5).

2.3

The Reality of Philosophers Imagine human beings living in an underground cave, which has a mouth open towards the light and reaching all along the cave; here they have been from their childhood, and have their legs and necks chained so that they cannot move, and can only see before them, being prevented by the chains from turning round their heads. [...] And they see only their own shadows, or the shadows of one another, which the fire throws on the opposite wall of the cave? [...] And if they were able to converse with one another, would they not suppose that they were naming what was actually before them? [...] It is without a doubt then, I said, that in their eyes, reality would be nothing else than the shadows of the images. Plato, The Republic, Book VII, ≈ 375 B.C.

The word real comes from the Latin res, meaning a thing. A reality is a thing that exists, which is defined mainly by its resistance and recurrence. Reality is realized, it resists and it goes on without end. But what is a thing, and in what does it exist? These 9

GT-RV: the French Virtual Reality Group of the National Center of Scientific Research and the French Ministry of Education, Research and Technology (www.inria.fr/epidaure/GT-RV/gt-rv.html).

17

Concepts

questions are without a doubt as old as philosophy itself, and this is why philosophers and scientists revisit the notion of intuitive and complex reality time and time again.

2.3.1

Sensory/Intelligible Duality

Almost 2,500 years ago, Plato, in his famous allegory of the cave, had already placed human beings in a semi-immersive situation in front of a screen (the cave wall in front of them) so that they could only see the projected images of virtual objects (the shadows of images). This simple thought experiment is no doubt the first virtual reality experience and a prelude to the Reality Centers of today [G¨obel 93]. Consequently, from Ren´e Descartes10 I think, therefore I am, to physicist Bernard d’Espagnat’s [D’Espagnat 94] Veiled Reality, all philosophical doctrines revolve around two extreme opinions (Figure 11 2.1). The first doctrine, objectivism, maintains that objects exist independently from us: there is an objective reality that is independent of the sensory, experimental and intellectual conditions of its appearance. The second doctrine, subjectivism, relates everything to subjective thinking in such a way that there is no point in discussing the real nature of objects because reality is only an artifact, an idea. This game of opposites between these two perspectives broadens our thinking: everyone finds their own balance between the object and the consciousness that perceives it [Hamburger 95]. - Materialism (philosophy of the material) is the belief in matter over mind. This philosophy, in which there are only physical-chemical explanations, was the philosophy among many of the atomists of Antiquity, as well as scientists from the 19th and 20th centuries.

objectivism materialism

6

empiricism positivism rationalism idealism

?

subjectivism

- Empiricism (philosophy of induction) makes experience the only source of knowledge; it justifies induction as the main approach for going from a set of observed facts to stating a general law. - Positivism (philosophy of experience), empiric rationalism or rational empiricism is based on the experimental method by emphasizing the importance of hypothesis, which governs reasoning and controls experience: experimentation becomes deliberate, conscious and methodical. It gives deduction the role of justifying laws after their inductive or intuitive formulation. The pragmatism of this school of thought, started by A. Comte, and amended by, among others, G. Bachelard and K. Popper, attracted a large number of scientists from the 19th and 20th centuries. (C. Bernard, N. Bohr, . . . ). - Rationalism (philosophy of deduction) postulates that reason is an organizational faculty of experience; through a priori concepts and principles, it provides the formal framework for all knowledge of phenomena by relying on a logical-deductive approach.

- Idealism (philosophy of the idea) states that reality is not sensory, but ideal. Idealism includes the idealist conceptions of Plato, R. Descartes, E. Kant, and G. Berkeley.

Figure 2.1: Reality and the Main Philosophical Doctrines Without getting involved in this debate, let us note that it is through our senses 10

Descartes R., Discourse on Method, Book IV, 1637.

11

Figure 2.1 is freely inspired from the Baraquin and al. Dictionary of Philosophy [Baraquin 95].

18

The Reality of Philosophers

and mind that we understand the world. Certainly, the senses do not protect us from illusions, mirage or hallucinations, in the same way that ideas can be arbitrary, disembodied or schizophrenic. Nevertheless, our only way to investigate reality is through the sensory and the intelligible. Already ahead of his time, I. Kant 12 introduced the notion of noumenon — the object of understanding, from the intelligible — as opposed to the notion of phenomenon — the object of possible experience that appears in space and time, from the sensory. Recently, Edgar Moris and his paradigm of complexity, have blurred the line between the sensory and the intelligible. He postulates a dialogical principle that can be defined as the necessary association of antagonisms in order for an organized phenomenon to exist, function and develop. In this way, a sensory/intelligible dialogue controls perception from the sensory receptors to the development of a cognitive representation [Morin 86]. In a similar perspective, physicist Jean-Marc Levy-Leblond believes that the antinomial pairs that structure thought are irreducible: for example, continuous/discontinuous, determined/random, elemental/compound, finished/infinite, formal/intuitive, global/local, real/fiction, etc. He then uses these examples, borrowed from physical science, to demonstrate how these pairs of opposites are inseparable, and how they structure our scientific understanding of nature, and consequently our understanding of reality [L´evy-Leblond 96]. Therefore, in a dialogue between the objective and subjective, reality becomes descriptive, not of the thing in itself, but of the object as it is seen by man through the sensory, experimental and intelligible conditions of its appearance. The boundary between the opposing sides of this dialogue is continuously redrawn through interactions between the world and man. The senses can be deceiving and the mind imagines actions to improve its understanding of the world; these actions cause our senses to react, which in return, keeps our ideas from becoming disembodied. Each person then creates their own reality by continuously switching between the senses and the imagination. Science, in turn, is trying to give them a universal representation that is free of individual illusions.

2.3.2

Virtual Versus Actual

While the concept of reality has always haunted philosophers, the concept of virtuality in itself has only been a topic of research for about twenty or thirty years. The word virtual, from the Latin virtus (virtue or strength), describes that which in itself has all the basic conditions for its actualization. In this sense, virtual is the opposite of actual, the here and now, that which is in action, and not in effect. This meaning of virtual, which dates back to the Middle Ages, is analyzed in detail by Pierre Levy. He recognized that the real, the possible, the actual and the virtual were four different modes of existence and that the possible and the real were opposites and the virtual and the actual were opposites (Figure 2.2 [L´evy 95]). The possible is an unproven and latent reality, which is realized without any changes in its determination or nature according to a pre-defined possibility. The actual appears here and now as a creative solution to a problem. The virtual is a dynamic configuration of strengths and purposes. Virtualization transforms the initial 12

Kant E., Critique of Pure Reason, I, Book II, Chapiter III, 1781.

19

Concepts

actuality into a solution for a more widespread problem.

POSSIBLE

REAL realization

all of the predetermined possibilities

persistent and resistant potentialization

VIRTUAL

ACTUAL actualization

all of the problems

Latent pole

virtualization

specific solution to a problem here and now

Visible pole

Figure 2.2: The Four Modes of Existence To illustrate this contrast between virtual and actual, let’s examine the case of a company that decides to have all of its employees telecommute. Currently, all of the employees work during the same hours and in the same place. However, with telecommuting, there is no need to have all of the employees physically present in the same place, because the workplace and employee schedule now depend on the employees themselves. The virtualization of the company is then a radical modification of its space-time references; employees can work from anywhere in the world, 24 hours a day. This takes place through a type of dematerialization in order to strengthen the information communications within the company. For the company, this is a sort of deterritorialization where the coordinates of space and time always pose a problem, instead of a specific solution. In order to idealize this new company, the mind creates a mental representation based on the metaphor of the actual company. It begins by interpreting the function of the virtual company by analogy to that of a typical company; then gradually through its understanding, forms a new idea of the notion of a company. The virtual company is not fictitious: even though it is detached from the here and now, it still provides services and advantages! Today, the adjective virtual has become a noun. We say the virtual as we speak of the

20

The Reality of Philosophers

real, and this evolution is accompanied by a change of the senses that makes the contrast between the virtual and the actual less clear. For epistemologist Gilles-Gaston Granger, the virtual is one the categories of objective knowledge, along with the possible and the probable, that is essential to the creation of science objects from actual phenomena of experience. The virtual is a representation of objects and facts that is detached from the conditions of a complete, singular and actual experience [Granger 95]. For Philippe Qu´eau, however, the virtual is realistically and actually present: it is an art that makes imitations that can be used. It responds to the reality of which it is a part [Qu´eau 95]. In the middle of these two extremes, Dominique No¨el bases his theory on the semantics of S. Kripke’s [Kripke 63] possible worlds in order to arrange possible worlds according to a relationship of accessibility, which says the following: a world M 1 is accessible to another world M2 if what is relatively possible for M1 is also relatively possible for M2 . The relationship of accessibility R makes up the hypothesis of an initial world M 0 , which is interpreted as the actual world. The worlds M1 , M2 , etc., are inferred from M0 by a relationship R and are both interpreted as well as virtual. This provides a formalism for mapping out the main concepts of virtual [No¨el 98].

2.3.3

Philosophy and Virtual Reality

Even though philosophers have been divided over the concept of reality since its origins, these same philosophers all agree that, no matter what the status of an object in itself - i.e. does it exist independently of the person that perceives it or not - there are three types of mediation that occur between the world and man: mediation of the senses (the perceived world), mediation of the action (the experienced world) and mediation of the mind (the imagined world). These three mediations are inseparable and create three perspectives of the same reality (Figure 2.3). In the first stage of our study, we will demonstrate that, as reality, virtual reality must also allow this triple mediation of senses, action and mind. In other words, we must be able to respond affirmatively to the three following questions: 1. Is it accessible to our senses? 2. Does it respond to our prompts? 3. Do we have a modifiable representation? But how would such a reality be virtual? The widespread use of the term virtual in all circles of society, from scientists to philosophers, from artists to the general public, changes its definition, which still appears to be a work in progress, and even renews the concept of reality. For our purposes, we have chosen to refer to the physical science notion of virtual to explain our conception of a reality that is virtual.

21

Concepts

Experiment Action •

Perception Senses • A A

A

A A

A

A

Reality A A

A

A A

A
A•






























- Mediation of the senses enables the perception of reality. - Mediation of the action enables one to experiment on reality. - Mediation of the mind forms a mental representation of reality.

Mind Representation Figure 2.3: The Three Mediations of The Real

2.4

The Virtual of Physicists Geometrical optics are based entirely on the representation of light rays by which light would propogate. By designing reflections and refractions from them, these rays end up being seen as real physical objects. [...] The image of both automobile or boat headlight beams, or those modern, thin laser beams or light-conducting glass fibers, impose the too bright feeling of familiar reality. Nevertheless, are these light rays of theory, these lines without thickness, something else besides beings of reason and constructions, which are powerful, certainly, but purely intellectual? [L´ evy-Leblond 96]

Physicists call a certain number of concepts virtual: for example, virtual images in optics, virtual work in mechanics, virtual processes of interaction in particle physics, etc., and many different models whose explanatory and predictive capabilities are widely used to obtain more knowledge about the phenomena that they are trying to represent. In fact, this notion of virtual seems to go hand in hand with the mathematization of physical sciences that has been taking place since the 17th century. The mathematical formalism used in physics is becoming more and more inaccessible to ordinary language, and is in fact, putting itself out of touch with intuitive representation. One is forced then to use analogies, images and words, which can only be understood through calculations. Even though these intermediate notions are not adequate enough to be fully credible, they nevertheless explain the physicist’s approach [Omn`es 95].

2.4.1

Virtual Images

Geometrical optics deals with light phenomena by studying how light rays work in transparent environments. It is based on the notion of independent light rays that travel in a straight line in transparent, homogeneous environments and whose behavior on the surface of a dioptre or a mirror is explained by the laws of Snell-Descartes (see for example [Bertin 78]).

22

The Virtual of Physicists

To demonstrate these ideas, let us examine the case of an optical system (S) surrounded by an entrance face L1 and an exit face L2 , which separates two homogeneous environments, indicated respectively as n1 and n2 (Figure 2.4). By definition, if the light travels from left to right, the environment located to the left of face L 1 is called the object environment, the environment to the right of face L2 is called the image environment. Let

O

 
 

 


 
 





 
 
 




 








  
 
  
 
 
 
 
 



 


 












 














 
      




 



 



 


 


 


 








  
  
 
 



(S)















  



   


 
 
 





 
 
 

   

  
 
  




 










  



 

 




 

  
  
 
 










 
 

 
 
 

 























 
 












     

  

 
 
 













 


    










     

  


 



 














 



 


  
  
 

 

 




 

 
  
   
 
 



 



  
     

 







 
 



object environment 


   
























     




  
 
 
 
(n1)
























  

 L1

I optical axis image environment (n2)

L2

Figure 2.4: Optical System O be a punctual light source that sends light rays, also known as incident rays, onto L 1 . Point O is an object for (S), and if, after having crossed the system (S), emerging light rays all pass by a same point I, this point I is called image O through the system (S). Two different scenarios are then identified: - Emergent rays actually pass through I: it is said that I is the real image of object O; this is the case for a converging lens as shown in the Figure 2.5a; - Only the extension of these rays pass through I: it is said then that I is the virtual image of O; this is the case for a diverging lens as shown in Figure 2.5b. A real image can be materialized on a screen and placed in the plane where it is located, but this is not the same for a virtual image that is only a construction of the mind. Thus, in the case of a real image, if a powerful laser is placed in O, a small piece of paper will burn in I (concentration of energy), but in the case of a virtual image, the paper will not burn in I (no concentration of energy). L

L

optical axis O

I

a) converging lens

O

I

b) diverging lens

Figure 2.5: Real Image (a) and Virtual Image (b) In a transparent, isotropic environment, which may or may not be homogeneous, the path of light is independent of the direction of travel: if a light ray leaves from point O to go towards point I by following a certain path, another light ray can leave from I and

23

Concepts

follow the same path in the opposite direction to go towards O. Thus, the roles of object and image environments can be interchanged: this property is known as the principle of the reversibility of light. Let’s look again at Figure 2.5. In the case of converging lens, it is easy to position a punctual source in I, whose rays converge in O. In the case of the diverging lens depicted in Figure 2.6a, a device must be used. Let us position a punctual source in O 0 so that it forms, through an auxiliary converging lens L 0 , an image that, if L did not exist, would form in I (Figure 2.6b). Because of the presence of L, the rays converge in O (Figure 2.6c): therefore, we say that I plays the role of a virtual object for lens L. L

a) O

I

b) O’

I

L’

To illustrate the principle of reversibility in a diverging lens that creates a virtual image I (Figure 2.6a), from object O, a device must be used. Let us position a punctual source in O 0 so that it forms, through an auxiliary converging lens L0 , an image that, if L did not exist, would form I (Figure 2.6b). Because of the presence of L, the rays converge in O (Figure 2.6c): therefore, we say that I plays the role of a virtual object for lens L.

L

c) O

O’

I

L’

Figure 2.6: Virtual Image (a), Real Image (b), and Virtual Object (c) These notions of virtual images and virtual objects in geometrical optics correspond to geometric constructions, which are the focal point for light ray extensions. These are intermediary concepts, created as such, which cannot be materialized but can facilitate image construction, and for this reason, are widely used in designing optical devices. The images of virtual reality then, are not virtual. These are essentially computergenerated images, produced by a computer, which like any image, reproduce a reality of the physical world or display a mental abstraction. They materialize on our terminal screens and are, therefore, as defined by opticians, real images. However, they give us virtual objects to look at, and it is the quality of the visual rendering that creates in part, the illusion of reality.

24

The Virtual of Physicists

2.4.2

Image Perception

As in most older disciplines, the fundamentals of geometrical optics are based on concepts derived directly from common sense and acquired from daily experience. The same applies to the notion of light ray, of rectilinear propagation and the principle of reversibility. These notions are so instilled in us13 , so effective in the normal perception of our environment, that they cause us to make mistakes of interpretation that cause some illusions and mirage. These illusions are in fact solutions proposed by the brain after a light follows the simulation of a path: it travels the path in the opposite direction (principle of reversibility), in a straight line (rectilinear propagation), by bypassing the diopter and mirrors (no refraction, no reflection), and by assuming a homogeneous environment!

O

I

O

I

O

Earth I

a) mirror

b) ray bending

c) mirage

Each morning, we see our image behind the mirror, even though there is nothing behind it except a mental image formed in our brain . . . that forgot the laws of reflection (Figure 2.7a). In dry, clear weather, we can see an island behind the horizon. The differences in density with the altitude within the atmosphere create a non-homogeneous environment that causes the light rays to bend; if this variation in density is large enough, the curve imposed on the light rays can effectively make the island appear above the horizon (Figure 2.7b). Mirage is the same type of illusion (Figure 2.7c). The layers of air near the ground can become lighter than the upper layers due to local warming: they then cause a curve of rays traveling towards the ground. The simultaneous observation of an object O in direct vision and from an image I due to the curve of certain rays, can also be interpreted by the presence of a body of water in which the object O would be reflected.

Figure 2.7: Illusions of Rectilinear Propagation and Opposite Return

These illusions are real in the sense that real images make an impression on our retina; and they are virtual in the sense that how we interpret these real images leads us to assume that virtual objects, which are the results of incomplete geometric constructions (Figure 2.7), exist. The interpretation is the result of an image construction simulation, for which the brain, like an inexperienced student, would not have verified the application conditions of the model that it used (here, a rectilinear propagation model of light in a transparent and homogeneous environment). Geometric optics provides us with an interpretation of a certain number of optical illusions, but to better understand them, we must, like the heroine of Lewis Carroll14 , cross the mirror, go to the other side and enter into the model’s universe. 13

Geneticists would say that they are coded, neurophysiologists that they are memorized, electronic technicians that they are wired, computer specialists that they are compiled, etc. . . . 14

Lewis Carroll, Through the Looking Glass, 1872.

25

Concepts

2.4.3

Physics and Virtual Reality

Physicists base their models on the observation of phenomena, and in return, they explain the observed phenomena through these models. It is the comparison between the simulation of the model and the experiment of the real that allows this mutual development between observing and modeling a phenomenon. The brain also makes this kind of comparison as it perceives our environment: it compares its own predictions to the information that it takes from the environment. Thus, the senses act as both a source of information and a way to verify hypotheses; perception problems are caused by unresolved inconsistencies between information and predictions [Berthoz 97]. For physicists, the virtual is a tool for understanding the real; it definitely appears as a construction in the universe of models. Consequently, the same applies to geometric constructions that lead to optical virtual images, or the inclusion of virtual movements in the determination of movement in mechanics. Everything happens as if is indeed the most currently used expression by specialists when they are trying to explain a phenomenon which is part of their field of expertise. Just like the expression once upon a time that begins fairy tales, everything happens as if is the traditional preamble that marks the passage into the universe of the scientific model; it clearly puts the reader on guard: what is going to follow only exists and makes sense within the framework of the model. To continue along our line of thought, we will demonstrate that virtual reality is virtual because it involves the universe of models. A virtual reality is a universe of models where everything happens as if the models were real because they simultaneously offer the triple mediation of senses, action and mind (Figure 2.8). The user can thus observe the activity of the models through all of his sensory channels (mediation of senses), test the reaction time of models in real-time through adapted peripherals (mediation of action), and build the pro-activity of models by modifying them on-line to adapt them to his projects (mediation of the mind). The active participation of the user in this universe of models opens the way to a true model experiment and hence to a better understanding of their complexity.

2.5

Principle of Autonomy Geppetto took his tools and set out to make his puppet. “What name shall I give him? he said to himself. I think I will call him Pinocchio.” [...] Having found a name for his puppet, he began to work in good earnest, and he first made his hair, then his forehead, then his eyes. The eyes being finished, imagine his astonishment when Geppetto noticed that they moved and stared right at him. [...] He then took the puppet under the arms and placed him on the floor to teach him to walk. Pinocchio’s legs were stiff and he could not move, but Geppetto led him by the hand and showed him how to put one foot in front of the other. When his legs became flexible, Pinocchio began to walk by himself and run about the room; until, he went out the door, jumped into the street and ran off. Carlo Collodi, The Adventures of Pinocchio, 1883

A virtual reality application is made up of a certain number of models (the virtual of physics) that users must be able to perceive, experiment and modify in the conditions of the real (the three mediations of the real of philosophers). Thus, users can jump in or out of the system’s control loop, allowing them to operate models on-line. As for modelers,

26

Principle of Autonomy reactivity Experiment Action •

activity Perception Sense • A A

A

A A

A

A

Model A A

A

A A

A
A•






























. The mediation of the senses allows you to perceive the model by observing its activity. . The mediation of the action allows you to experiment on the model and thus test its reactivity. . The mediation of the mind creates a mental representation of the model that defines its pro-activity.

Mind Representation pro-activity Figure 2.8: The Three Mediations of Models in Virtual Reality they must integrate a user model (an avatar) into their system so that the user can be effectively included and participate in the development of this universe of models.

2.5.1

Operating Models

The three main operating modes of models are perception, experiment and modification. Each one corresponds to a different mediation of the real [Tisseau 98b]. The mediation of the senses takes place through model perception: the user observes the activity of the model through all of his sensory channels. The same applies to a spectator in an Omnimax theater who, seated on moving seats in front of a hemispheric screen in a room equipped with a surround-sound system, really feels as if he is a part of the animated film that he is watching, even though he cannot modify its course. Here, the quality of the sensory renderings and their synchronization is essential: this is realtime animation’s specialty. The most current definition of animation is to make something move. In the more specific field of animated films, to animate means to give the impression of movement by scrolling a collection of ordered images (drawings, photographs, computergenerated images, etc.). These images are produced by applying an evolution model of the objects in the represented scene. The mediation of action involves model experiments: the user tests the model’s reactivity through adapted manipulators. The same applies to the fighter pilot at the controls of a flight simulator: his training focuses mainly on learning the reactive behavior of the plane. Based on the principle of action and reaction, the emphasis here is placed on the quality of real-time rendering, which is what interactive simulation does best. The standard meaning of simulate is to make something that is not real seem as if it is. In the field of science, simulation is an experiment on a model; it allows you to test the quality and internal consistency of the model by comparing its results to those of the experiment on the modeled system. Today, it is being used more and more to study complex systems in

27

Concepts

which human beings are involved to both train operators as well as research the reactions of users. In these simulations where man is in the loop, the operator develops his own behavioral model, which interacts with the other models. The mediation of the mind takes place when the user himself modifies the model by using an expressiveness that is equivalent to that of the modeler. The same applies to an operator who partially reconfigures a system while the rest of the system remains operational. In this fast-growing field of interactive prototyping and on-line modeling, it is essential that users be able to easily intervene and express themselves. To obtain this level of expressiveness, the user generally uses the same interface and especially the same language as that of the modeler. The mediation of the mind is thus realized by the mediation of language. So, the closely related fields of real-time animation, interactive simulation and on-line modeling represent three aspects of the operation of models. The three together allow the triple mediation of the real that is needed in virtual reality and define three levels of interactivity (Figure 2.9). . Real-time animation corresponds to a zero level of interactivity between the user and the model being run. The user is under the influence of the model because he cannot act on the parameters of the model: he is simply a spectator of the model. . Interactive simulation corresponds to the first level of interaction because the user can access some of the model’s parameters. The user thus plays the role of the actor in the simulation. . In on-line modeling, the models themselves are the parameters of the system: the interaction reaches the highest level. The user, by modifying himself the model being run, participates in the creation of the model and thus becomes a cre-actor (creator-actor). interactivity

mediation

cre−actor on−line modelization

mind 2

actor

interactive simulation

action 1

spectator

real−time animation

senses 0

Figure 2.9: The Different Levels of Interaction in Virtual Reality

2.5.2

User Modeling

The user can interact with the image through adapted behavioral interfaces. But, what the user is able to observe or do within the universe of models is only what the system controls through device drivers, which are vital intermediaries between man and machine. The user’s sensory-motor mediation is thus managed by the system and then modeled within the system in one manner or another. The only real freedom that the

28

Principle of Autonomy

user has is his decision-making options (mediation of the mind), which are restricted by the system’s limitations in terms of observation and action. It is also necessary to explain the inclusion of the user by representing it through a specific model of an avatar within the system. At the least, this avatar is placed in the virtual environment in order to define a perspective that is needed for different sensory renderings. It has virtual sensors, robot actuators (vision [Renault 90], sound [Noser 95], and hand grips [Kallmann 99]) to interact with the other models. The data gathered by the avatar’s virtual sensors are transmitted to the user in real time by device drivers, while the user’s commands are transmitted in the opposite direction to the avatar’s robot actuators. It also has methods of communication in order to communicate with the other avatars, these methods thus strengthen its sensory-motor capabilities and allow it to receive and emit language data. The avatar display may be non-existent (BrickNet [Singh 94]), reduced to a simple 3D primitive, which is textured but not structured (MASSIVE [Benford 95]), and assimilated into a polyarticulated rigid system (DIVE [Carlsson 93]) or a more realistic representation that includes evolved behaviors such as gestures and facial expressions (VLNET [Capin 97]). This display, when it exists, makes it easier to identify avatars and the non-verbal communication between them. Thus, with this explicit user modeling, three major types of interaction can coexist in the universe of digital models: . model-to-model interactions such as collisions and attachments; . model-to-avatar interactions that allow sensory-motor mediation between a model and a user; . avatar-to-avatar interactions that allow avatars to meet in a virtual environment shared by several users (televirtuality [Qu´eau 93a]). The user’s status in virtual reality is different than his status in scientific simulations of digital calculations or in the interactive simulation of training simulators (Figure 2.10). In scientific simulation, the user sets the model’s parameters beforehand, and interprets the calculation results afterwards. In the case of a scientific display system, the development of the calculations may be observed with virtual reality sensory peripherals [Bryson 96], but it remains the model’s slave. The systems of scientific simulation are model-centered systems because science models want to create universal representations that are separate from individual impressions from reality. In contrast, interactive simulation systems are basically user-centered to give the user all of the ways needed to control and pilot the model: the model must remain the user’s slave. By introducing the notion of avatar, virtual reality places the user at the same conceptual level as the model. The masterslave relationship is thus eliminated for a greater autonomy for models and consequently, a greater autonomy for users.

2.5.3

Autonomizing Models

To autonomize a model you must provide it with ways to perceive and respond to its surrounding environment. It must also be equipped with a decision-making module so that it can adapt its reactions to both external and internal stimuli. We use three lines of thought as a guide to the autonomization of models: autonomy by essence, by necessity and by ignorance.

29

Concepts

avatar model

model

model

Figure 2.10: User Status in Scientific Simulation (a), in Interactive Simulation (b) and in Virtual Reality (c) Autonomy by essence characterizes all living organisms, from a simple cell to a human being. Avatars are not the only models that can perceive and respond to their digital environments: any model that is supposed to represent a living being must have this kind of sensory motor interface. The notion of animat, for example, concerns artificial animals whose rules of behavior are based on real animal behaviors [Wilson 85]. Like an avatar, an animat is situated in an environment; it has sensors to acquire information about its environment and effectors to react within this environment. However, unlike an avatar, which is controlled by a human being, an animat must be self-controlled in order to coordinate its perceptions and actions [Meyer 91]. Control can be innate (pre-programmed) [Beer 90], but in the animat approach, it will more often be acquired in order to stimulate the genesis of adaptive behaviors for survival in changing environments. Therefore, research in this very active field revolves primarily around the study of learning (epigenesis) [Barto 81], development (ontogeny) [Kodjabachian 98] and the evolution (phylogenesis) [Cliff 93] of control architecture [Meyer 94, Guillot 00]15 . Animating virtual creatures through these different approaches provides a very clear example of these adaptive behaviors [Sims 94], and modeling virtual actors falls under the same approach [Thalmann 96]. Thus, autonomizing a model associated with an organism allows us to understand the autonomy observed in this organism more fully. Autonomy by necessity involves the instant recognition of changes in the environment, by both organisms and mechanisms. Physical modeling of mechanisms usually takes place by solving differential equation systems. Solving these systems requires knowledge about the conditions of the limits that restrict movement, but in reality, these conditions can change all the time, whether the causes in themselves are known or not (interactions, disturbances, changes in the environment). The model must then be able to perceive these changes in order to adapt its behavior during execution. This is truer when the user is in the system because, through his avatar, he can make changes that were initially completely undefined. The example of sand flowing through an hourglass can be used to illustrate this concept. The physical simulation of granular medium of sand is usually based on micromechanical interactions between two solid spheres. Such simulations take several hours of calculation to visualize the flow of sand from one sphere to another and 15

From Animals to Animats (Simulation of Adaptive Behavior): bi-annual conference since 1990 (www.adaptive-behavior.org/conf)

30

Principle of Autonomy

are therefore, unsuited for the constraints of virtual reality [Herrmann 98]. Modeling with larger grains (mesoscopic level) based on punctual masses connected to each other by appropriate interactions results in displays that are satisfactory, but not interactive [Luciani 00]. Our approach involves large autonomous grains of sand that, individually, detect collisions (elastic collisions) and are sensitive to gravity (free fall). It allows us to not only simulate the flow of sand in the hourglass, but also to adapt in real-time to the hourglass if it is turned over, or has a hole [Harrouet 00]. Thus, autonomizing any model whatsoever, allows it to react to unplanned situations that come up during execution, and which are the result of changes in the environment due to the activity of other models. Autonomy by ignorance reveals our current inability to explain the behavior of complex systems through the reductionist methods of an analytical approach. A complex system is an open system made up of a heterogeneous group of atomic or composite entities. The behavior of the group is the result of the individual behavior of these entities and of their different interactions in an environment that is also active. Based on schools, the behavior of the group is seen as either organized through a goal, which would be teleological behavior [Le Moigne 77], or as the result of an auto-organization of the system, which would be emergence [Morin 77]. The lack of overall behavior models for complex systems leads us to distribute control over the systems’ components and thus, autonomize the models of these components. The simultaneous evolution of these components enables a better understanding of the behavior of the entire overall system. Hence, a group of autonomous models interacting within the same space has a part in the research and experiments of complex systems. Autonomizing models, whether by essence, necessity or ignorance, plays a part in populating virtual environments with artificial life that strengthens the impression of reality.

2.5.4

Autonomy and Virtual Reality

The user of a virtual reality system is an all-in-one spectator, actor and creator of the universe of digital models with which he interacts. Even better, he participates completely in this universe where he is represented by an avatar, a digital model, that has virtual sensors and robot actuators to perceive and respond to this universe. The true autonomy of the user is found in his capability to coordinate his perceptions and actions, either by accident, to simply wander around in this virtual environment, or by following his own goals. The user is thus placed at the same conceptual level as the digital models that make up this virtual world. Different types of models — such as particles, mechanisms and organisms — coexist within virtual universes. For the organism models, it is essential that they have ways to perceive, act and decide in order to reproduce as best as possible their ability to decide by themselves. This approach is also needed for mechanisms that must be able to react to unforeseen changes in their environment. Moreover, our ignorance in understanding systems once they become too complex, leads us to decentralize the control of the system’s components. The models, whatever they are, must then have investigation methods equal to those of the user and decision methods that match their functions. A sensory

31

Concepts

disability (myopia, deafness, anesthesia) causes an overall loss of the subject’s autonomy. This is also true for a mobility impairment (sprain, tear, strain) and for a mental impairment (amnesia, absent-mindedness, phobia). So, providing models with ways to perceive, act and make decisions, means giving them an autonomy that places them at the same operational level as the user. Continuing along our line of thought, which began with the philosophers and physicists, we have established, in principle, the autonomization of digital models that make up a virtual universe. A virtual reality is a universe of interacting autonomous models, within which everything happens as if models were real because they provide users and other models a triple mediation of senses, action and language all at the same time. (Figure 2.11). To accept this autonomy on belief, is to accept sharing the control over the evolution of virtual universes between human users and the digital models that populate these universes.

senses senses senses

action

action

avatar

model

model

language

language

language

senses senses

action

model language

action senses

model

action

language

language senses

senses

action

model

action

model

language

avatar senses

action

action

model language

language

Figure 2.11: Autonomy and Virtual Reality

This conception of virtual reality thus coincides with Collodi’s old dream, as described in the quote located at the beginning of this section 2.5, of making his famous puppet an autonomous entity that fulfills the life of his creator. Geppetto’s approach for reaching his goal was the same as what we have found in virtual reality. He started by identifying him (I will call him Pinocchio), then he worked on his appearance ([he] started by making his hair, then his forehead), and making him sensors and actuators (then the eyes [...]). He then defined his behaviors (Geppetto led him by the hand and showed him how to put one foot in front of the other) in order to make him autonomous (Pinocchio began to walk by himself) and finally, he could not help but notice that autonomizing a model means that the creator loses control over his model (he jumped into the street and ran off).

32

Conclusion

2.6

Conclusion

As a result of interdisciplinary works on the computer-generated digital image, virtual reality transcends its origins and has established itself today as a new discipline within the engineering sciences. It involves the specification, design and creation of a participatory and realistic virtual universe. A virtual universe is a group of autonomous digital models in interaction in which human beings participate as avatars. Creating these universes is based on an autonomy principle according to which digital models: . are equipped with virtual sensors that enable them to perceive other models, . have robot actuators to act on the other models, . have methods of communication to communicate with other models, . and control their perception-response coordinations through a decision-making module. An avatar is therefore a digital model whose decision-making abilities are delegated to the human operator that it represents. Digital models, located in space and time, evolve autonomously within the virtual universe, of which the evolution of the group is the result of their combined evolution. Man, model among models, is spectator, actor and creator of the virtual universe to which he belongs. He communicates with his avatar through a language and various sensory-motor devices that make the triple mediation of senses, action and language possible. The multi-sensory rendering of his environment is that of, realistic, computergenerated digital images: 3D, sound, touch, kinesthetic, proprioceptive, animated in real time, and shared on computer networks. Therefore, the epistemological line of thought that we took in Chapter 2 places the concept of autonomy at the center of our research problem in virtual reality. In the following chapters we will ask ourselves how to implement these concepts by asking the following: which models (Chapter 3) and which tools (Chapter 4) should be used to autonomize models?

33

Concepts

34

Chapter 3 Models 3.1

Introduction

In this chapter, we present the main models that we have developed in order to observe the principle of autonomy that was introduced in the previous chapter. First, we will adapt the multi-agent approach to the problem of virtual reality by introducing the notion of participatory multi-agent simulation. Secondly, we will use the serious example of blood coagulation to explain the phenomenon of collective auto-organization that takes place in multi-agent simulations. Through this example, we will defend the idea of in virtuo experimentation; a new type of experimentation that is possible today in biology. We will then explain how to model perceptive behavior with the help of fuzzy cognitive maps. These maps allow us to define the behavior of an autonomous entity, control its movement, and simulate its movement through the entity itself so that it can anticipate its behavior through active perception. These models are thus our intermediary representations between the concepts (Chapter 2) that inspire them and the tools (Chapter 4) that implement them.

3.2

Autonomous Entities The unit of analysis is therefore the activity of the person in real life. It does not refer to the individual, nor the environment, but the relationship between the two. The environment does not just modify actions, it is part of the system of action and cognition. [...] A real activity then, is made up of flexibility and opportunism. From this perspective, and contrary to the theory of problem solving, a human being does not engage in action with a series of rationally pre-specified objectives according to an a priori model of the world. He looks for his information in the world. What becomes normal, is the way that he integrates himself to act in an environment that changes and that he can modify, and how he uses and selects available information and resources: social, symbolic and material. [Vacherand-Revel 01]

Despite having become more realistic, virtual universes will lack credibility as long as they are not populated with autonomous entities. The autonomization of entities is divided into three modes: the sensory-motor mode, the decision-making mode and the operational mode. It is actually based upon a sensory-motor autonomy: each entity is provided with sensors and effectors that allows it to get information and respond to its environment. It is also based on a decision-making autonomy: each entity makes decisions

35

Models

according to its own personality (its history, intentions, state and perceptions). Finally, it requires execution autonomy: each entity’s execution controller is independent of the other entities’ controllers. In fact, this notion of autonomous entity is similar to the notion of agent in the individual-centered approach of multi-agent systems.

3.2.1

Multi-Agent Approach

The first works on multi-agent systems date back to the 80’s. They were based upon the asynchronism of language interactions between actors in distributed artificial intelligence [Hewitt 77], the individual-centered approach of artificial life [Langton 86], and the autonomy of moving robots [Brooks 86]. Currently, there are two major trends within this disciplinary field: either attention is focused on agents as they are (intelligent, rational or cognitive agents [Wooldridge 95]), or the focus is on the interactions between agents and collective aspects (reactive agents and multi-agent systems [Ferber 95]). Thus, we can find agents that reason through beliefs, desires and intentions (BDI: Belief-DesireIntention [Georgeff 87]), agents that are more emotional, as in some video games (Creatures [Grand 98]) or stimulus response-type agents that are purely reactive, as in insect societies (MANTA [Drogoul 93]). In any case, these systems differ from the symbolic planning models of classic Artificial Intelligence (STRIPS [Fikes 71]) by recognizing that an intelligent behavior can emerge from interactions between agents that are more reactive placed in an environment that is itself active [Brooks 91]. These works were motivated by the following observation: there are systems in nature, such as insect colonies or the immune system for example, that are capable of accomplishing complex, collective tasks in dynamic environments without external control or central coordination. Thus, research on multi-agent systems is striving towards two major objectives. The first objective is to build distributed systems, which are capable of accomplishing complex tasks through cooperation and interactions. The second objective is to understand and conduct experiments of collective auto-organization mechanisms that appear when many autonomous entities are interacting. In any event, these models favor a local approach where decisions are not made by a central coordinator familiar with each entity, but by each of the entities individually. These autonomous entities, called agents, only have a partial, and therefore incomplete, view of the virtual universe in which they evolve. Each agent can be considered as an engine that is continuously following a triple cycle of perception, decision and action: 1. perception : perception: it perceives its immediate environment through specialized sensors, 2. decision: it decides what it must do, taking into account its internal state, its sensor values and its intentions, 3. action: it acts by modifying its internal state and its immediate environment. Two major types of applications are affected by multi-agent systems: distributed problem-solving and multi-agent system simulation. In distributed problem-solving, there

36

Autonomous Entities

is an overall satisfaction function that allows us to evaluate the solution according to the problem to be solved; while in simulation, we assume that there is at best, a local satisfaction function for each agent [Ferber 97]. Like all modeling, the retained multi-agent approach simplifies the researched phenomenon. And in addition, this approach enables it, for the most part, to maintain its complexity by allowing for diverse elements, structures and associated interactions. Today however, multi-agent modeling does not have the right theoretical tools to deduce the behavior of the entire multi-agent system from the individual behavior of agents (direct problem), or to induce through reasoning the individual behavior of an agent when we know the collective behavior of a multi-agent system (reverse problem). Simulating these models partly makes up for the lack of appropriate theoretical tools by allowing us to observe structures and behaviors that collectively emerge from local, individual interactions. We retain this multi-agent simulation approach for virtual reality.

3.2.2

Participatory Multi-Agent Simulation

Many design methods for multi-agent systems have already been proposed [Wooldridge 01], but none of them have really been established yet, as is the case of the UML method for object design and programming [Rumbaugh 99]. We will retain here the Vowel approach, which analyzes multi-agent systems from four perspectives: Agents, Environments, Interactions, and Organizations (the vowels A,E,I,O), which are all of equal paradigmic importance [Demazeau 95]. In virtual reality, we suggest adding the vowel U (forgotten in AEIO), as in U for User, to include the active participation of the human operator in the simulation (man is in the loop). Thus, virtual reality multi-agent simulations will become participatory.

A as in Agent A virtual universe is a multi-model universe in which all types of autonomous entities can coexist: from the passive object to the cognitive agent, with a pre-programmed behavior or an adaptive, evolutionary behavior. An agent can be atomic or composite, and be the head of another multi-agent system itself. Consequently, a principle of component heterogeneity must prevail in virtual worlds.

E as in Environment An agent’s environment is made up of other agents, users that take part in the simulation (avatars) and objects that occupy the space and define a certain number of space-time constraints. Agents are thus placed in their environment, a real open system in which entities can appear and disappear at any time.

37

Models

I as in Interaction Diverse entities (objects, agents and users) generate diverse interaction among themselves. We find physical phenomena such as attractions, repulsions, collisions and attachments, as well as exchanges of information among agents and/or avatars through gesture and language transmissions (language acts or code exchanges to be analyzed). Destroying and creating objects and/or agents also strengthens these different interaction possibilities. Consequently, new properties, unknown for different entities taken individually, will eventually emerge from interactions between an agent and its environment.

O as in Organization A set of social rules, like a driving rule, can structure all of the entities by defining roles and the constraints between these roles. These roles can be assigned to agents or negotiated among them; they can be known in advance or emerge from conflicting or collaborative situations - the results of interactions among agents. So, both predefined and emergent organizations structure multi-agent systems at the organizational level, a given level which is both an aggregate of low-level entities and a high-level entity.

U as in User The user can take part in the simulation at any time. He is represented by an avataragent of which he controls decisions and uses sensory-motor capabilities. He interfaces with his avatar through different multi-sensory peripherals, and has a language to communicate with agents, modify them, create new ones, or destroy them. The structural identity between an agent and an avatar allows the user, at any time, to take the place of an agent by taking control of his decision-making module. During this substitution, the agent can eventually go into a learning mode through imitation (PerAc [Gaussier 98]) or by example [Del Bimbo 95]. At any time, the user can return the control to the agent that he replaced. This principle of substitution between agents and users can be evaluated by a type of Turing test [Turing 50]: a user interacts with an entity without guessing that it is an agent or another user, and the agents respond to his actions as if he were another agent. The Vowel approach thus extended (AEIO + U) fully involves the user in multi-agent simulation, hence concurring with the approach of participatory design (participatory design [Schuler 93]), which prefers to see users as human actors rather than human factors [Bannon 91]. Such a participatory multi-agent simulation in virtual reality implements different types of models (multi-models) from different fields of expertise (multi-disciplines). It is often complex because its overall behavior depends just as much on the behavior of the models themselves as the interactions between the models. Finally, it must include the free will of the human user who uses the models on-line.

38

Autonomous Entities

3.2.3

Behavioral Modeling

A model is an artificial representation of a phenomenon or an idea; it is an intermediary step between the intelligible and the sensory, between the phenomenon and its idealization, between the idea and its perception. This representation is based on a system of symbols that have a meaning, not just for the designer who composes them, but also for the user who perceives them (a user’s perspective of the semantics may be different from that of the modeler’s). A model has multiple functions. For a modeler, the model allows him to imagine, design, plan and improve his representation of the phenomenon or idea that he is trying to model. The model becomes a communication medium to represent, make people aware of, explain or teach related concepts. At the other end of the spectrum, the model helps the user to understand the represented phenomenon (or idea); he can also evaluate and experiment by simulation.

Three major categories of digital models are usually identified: descriptive, causal and behavioral [Arnaldi 94]. The descriptive model reproduces the effects of the modeled phenomenon without any a priori knowledge about the causes (geometric surfaces, inverse kinematics, etc.). The causal model describes the causes capable of producing an effect (for example, rigid poly-articulated, finished elements, mass-spring networks, etc.). Its execution requires that the solution which is implicitly contained in the model be clarified; the calculation time is then longer than that of a descriptive model for which the solution is given (causal/dynamic versus descriptive/kinematic). The behavioral model imitates how living beings function according to a cycle of perception, decision and action (artificial life, animat, etc.). It involves both the internal and external behaviors of entities [Donikian 94]. Internal behaviors are internal transformations that can lead to perceptible changes on the entity’s exterior (i.e. morphology and movement). These internal behaviors are components of entities and depend little on their environment, unlike external behaviors that reflect the environment’s influence on the entity’s behavior. These external behaviors can be purely reactive or driven (stimulus/response), perceptive or emotional (response to stimulus depends on an internal emotional state), cognitive or intentional (the response is guided by a goal), adaptive or evolutionary (learning mechanisms allowing you to adapt the response over the course of time), social or collective (the response is limited by the rules of a society).

Section 3.3, which follows, presents a serious example of multi-agent simulation based on reactive behaviors in order to observe the collective auto-organizing properties of such systems. In another direction, Section 3.4 focuses on an emotional type of behavior to highlight the difference between individual sensation and perception.

39

Models

3.3

Collective Auto-Organization A protein is more than a translation in another language of one of the books from the gene library. The genes that persist in our cells during our existence, contain — are made up — of information. The real tools — the real actors — of cell life are the proteins, which are more or less quickly destroyed and disappear if they are not renewed. Once the cell puts them together, proteins fold up into complex shapes. Their shape determines their capability to attach themselves to other proteins and interact with them. And it is the nature of these interactions that determine their activity — their effect. [...] But trying to attribute an unequivocal activity — an intrinsic property — to a given protein amounts to an illusion. Its activity, effect and longevity depend on its environment, the collectivity of the other proteins around it, the pre-existing ballet into which it will integrate itself. [Ameisen 99]

Faced with the growing complexity of biological models, biologists need new modeling tools. The advent of the computer and computer science, and in particular virtual reality, now offers new experiment possibilities with digital simulations and introduces a new type of investigation: the in virtuo experiment. The expression in virtuo (in the virtual) is a neologism created here by analogy with adverbial phrases from Latin etymology 1 in vivo (in the living) and in vitro (in glass). An in virtuo experiment is thus an experiment conducted in a virtual universe of digital models in which man participates.

3.3.1

Simulations in Biology

Medicine has used human (in vivo) experimentation since its beginnings. In antiquity, this experiment was influenced by magic and religion. It was the Greek school of Kos that demythologized medical procedure: around 400 B.C., Hippocrates, with his famous oath, established the first rules of ethics by outlining the obligations of a practitioner towards his patients. For a long time, life sciences remained more or less descriptive and were based on observation, classification and comparison. It was not until the development of physics and chemistry that (in vitro) laboratory examinations became a common practice. In particular, the progress in optics and the invention of more and more evolved microscopes made it possible to go beyond the limits of direct observation. Italians Malpighi (1628-1694) and Morgagni (1682-1771) hence became the precursors of histology by demonstrating the importance of studying the correlations between microscopic biological structures and clinical manifestations. And undoubtedly, Frenchman Claude Bernard and his famous Introduction to the Study of Experimental Medicine (1865) were the start of real scientific medicine based on the rationalization of experimental methods. Analogical simulation, or experiments on real models, was used for scientific purposes starting in the 18th century with the biomechanical automata of Frenchman Jacques de Vaucanson (The Flute Player, 1738), and reached its peak in the 20 th century, in aeronautics, with wind tunnel tests on plane models. In medicine, (in vitro) laboratory examinations are based on the same approach. In hematology laboratories, in vitro blood 1

in virtuo is not, however, a Latin expression. Biologists often use the expression in silico to describe computerized calculations; there is for example, a scientific journal called In Silico Biology (www.bioinfo.de/journals.html). However, in silico makes no reference to the participation of man in the universe of digital models in progress; this is why we prefer in virtuo which, by its common root, reflects the experimental conditions of virtual reality.

40

Collective Auto-Organization

samples used during laboratory analysis function as simplified representations of in vivo blood: the rheological constraints are, in particular, very different.

Population Dynamics Digital simulation, or experiments on virtual models, are usually based on solving a mathematical model made up of partial derivative equations such as ∂x/∂t = f (x, t), where t represents the time and x = (x1 , x2 , . . . , xn ) represents the vector of the state variables that are characteristic of the system to be simulated. In molecular biology, the macroscopic behavior of a system is often attained by coupling kinetics equations of chemical reactions with thermomechanical equations of molecular flow, which in first approximation solves equations such as ∂x/∂t = D∇2 x + g(x), where D is the system’s scattering matrix and g is a non-linear function that represents the effects of chemical kinematics. In hematology, this type of approach was applied starting in 1965 to the kinematics of coagulation [Hemker 65], then developed and broadened to take into account new data [Jones 94] or to demonstrate the influence of flow phenomena [Sorensen 99]. The solutions that were obtained in this way strongly depended on the conditions of the system’s limits, as well as the values given to different reaction times and diffusion parameters. Biological systems are complex systems and modeling them requires a large number of parameters. As Hopkins and Leipold [Hopkins 96] demonstrated in their critique of the use of differential equation mathematical models in biology, ignoring certain parameters of a model known for being invalid can lead, despite everything, to valid adjustments with experiment data. To avoid this drawback, some authors encourage the use of statistical methods in order to minimize the influence of specific values of certain parameters [Mounts 97]. Today, software tools such as GEPASI [Mendes 93] and StochSim [Morton-Firth 98] make it possible to use this classic problem-solving approach of chemical kinematics through differential equations at a molecular level, while tools such as vCell [Schaff 97] and eCell [Tomita 99] are used at the cell level. This type of simulation in cell biology is based on models that represent groups of cells, not individual cells. Consequently, these models explicitly describe the behavior of populations that, we hope, implicitly reflect the individual behavior of cells.

Individual Dynamics In contrast, two major methods of digital simulation used in biochemistry, Monte Carlo and molecular dynamics, are based on selecting microscopic theories (structures and molecular interaction potentials) to determine certain macroscopic properties of the system [Gerschel 95]. These methods explore the system’s configuration space through an initial configuration of n particles whose development is followed by sampling. This sampling is time-dependent in molecular dynamics, and random in Monte Carlo methods. The macroscopic properties are then calculated as averages for all of the configurations (microcanonical or canonical ensembles from statistical mechanics). The number n of particles influences calculation times, usually proportional to n 2 , and limits the number of sampled configurations, in other words, the average accuracy in a Monte Carlo method, or the system’s evolution time in a molecular dynamics calculation. These methods require

41

Models

significant calculation methods; today, the size of the largest simulated systems is around 106 atoms for real evolution times less than 10−6 s [Lavery 98]. In addition, they mainly involve systems close to equilibrium and because of this, are less suitable for studying systems far from equilibrium and basically irreversible, as the case may be with the phenomena of hemostasis. Cellular automata [Von Neumann 66], inspired by biology, offer an even more individualized approach. Conway’s Game of Life [Berlekamp 82] popularized cellular automata and Wolfram formalized them [Wolfram 83]. These are dynamic systems in which space and time are discretized. A cellular automata is made up of a finite set of squares (cells), each one containing a finite number of possible states. The squares’ states change synchronously according to the laws of local interaction. Although they are based on cellular metaphor, cell automata are rarely used in cell biology; we can however, refer to the simulation of an immune system through a cellular automata in order to study the collective behavior of lymphocytes [Stewart 89], the use of a cell automata to reproduce immune phenomena within lymphatic ganglions [Celada 92], and the development of a programming environment based on cell automata to model different cell behavior [Agarwal 95]. The heavy constraints of time-space discretization as well as those of action synchronicity are undoubtedly what inhibits the use of this technique in biology.

Multi-Agent Auto-Organization To overcome these methodology limitations, we propose modeling biological systems through multi-agent systems. The biological systems to be modeled are usually very complex. This complexity is mainly due to its diverse elements, structures and interactions. In the example of coagulation, molecules, cells and complexes make blood an environment that is open (appearance/disappearance and element dynamics), heterogenous (different morphology and behaviors) and formed from composite entities (organs, cells, proteins) that are mobile and distributed in space and in variable numbers in time. These elements can be organized into different levels known beforehand (a cell is made up of molecules) or emerge during the simulation due to multiple interactions between elements (formation of complexes). The interactions themselves can be different in nature and operate on different space and time scales (cell-cell interactions, cell-protein interactions and protein-protein interactions). There is currently no theory that is able to formalize this complexity, in fact, there is no formal a priori proof method as there is for highly formalized models. In the absence of formal proof, we must turn to experiments of systems that are being developed in order to be able to validate experiments a posteriori: this is what we have set out to do with multi-agent modeling and the simulation of blood coagulation. As we define it within the framework of virtual reality, multi-agent simulation enables real interaction with the modeled system as suggested by a recent prospective study on the requirements of cell behavior modeling [Endy 01]. In fact, during simulation, it is possible to view coagulation taking place in 3D; this visualization makes it possible to observe the phenomenon as if you had a virtual microscope that was movable and adjustable at will, and capable of different focus adjustments. The biologist can therefore

42

Collective Auto-Organization

focus on the observation of a particular type of behavior and observe the activity of a sub-system, or even the overall activity of the system. At any moment, the biologist can interrupt the coagulation and take a closer look at elements in front of him and at the interactions taking place, and can then restart the coagulation from where he had stopped. At any moment, the biologist can disrupt the system by modifying a cell property (state, behavior) or removing or adding new elements. Thus, he can test an active principle (for example, the active principle of heparin in thrombophilia), and more generally an idea, and immediately observe the consequences on the functioning system. Multi-agent simulation puts the biologist at the center of a real virtual laboratory that brings him closer to experimental science methods, while at the same time giving him access to digital methods. In addition to being able to observe the activity of a digital model being run on a computer, the user can also test the reactivity and adaptability of the model in operation, thus benefitting from the behavioral character of digital models. An in virtuo experiment provides an experience that a simple analysis of digital results cannot. Between formal a priori proof and a posteriori validations, there is room today for a virtual reality experienced by the biologist that can go beyond generally accepted ideas to reach experienced ideas.

3.3.2

Coagulation Example

The analogy between a biological cell and a software agent is quasi-natural. In fact, a cell interacts with its environment through its membrane and its surface receptors. Its nucleus is the head of a genetic program that determines its behavior; consequently it can perceive, decide and respond. By extension, the analogy between a multicellular system and a multi-agent system is immediate.

Cell-Agent Our design of a cell-agent revolves around three main steps. The first step is to choose a 3D geometric shape that represents the cell membrane. The second step is to place the sensors — the cell’s receptors — on this virtual membrane (Figure 3.1a). The third step involves defining the cell-agent’s own behaviors (Figure 3.1b). A behavior can be defined by an algorithm — calculation of a mathematical function, resolution of a system of differential equations, application of production rules — or by another multi-agent system that describes the inside of the virtual cell. The choice between algorithm and multi-agent system depends on the nature of each problem within the cell and how much is known about the problem. Therefore, we have chosen an algorithm to model the humoral response [Ballet 97a] and a multi-agent system to simulate the proliferate response of B lymphocytes to anti CD5 [Ballet 98a], as it was demonstrated experimentally [Jamin 96]. We have a library of predefined behaviors such as receptor expression on the cell’s surface, receptor internalization, cell division (mitosis), programmed cell death (apoptosis) and generation of molecular messengers (Figure 3.1c). New behaviors can be programmed and integrated into a cell-agent so that it can adapt to a specific behavior of other cells.

43

Models

agent

agent

02321

behavior

agent

behaviors

02321 receptors

ln(x)

02321

a

agent

f(x)

g(x)

functions

f(x)

f(x)

02321

b

02321

reproduction activation internalization expression mutation apoptosis ...

c

Designing a cell-agent revolves around three main steps. The first step is to choose a 3D geometric shape that represents the cell membrane. The second step is to place the sensors — the cell’s receptors — on this virtual membrane (a). The third step involves defining the cell-agent’s behaviors (b). A behavior can be defined by an algorithm — calculation of a mathematical function, resolution of a system of differential equations, application of production rules — or by another multi-agent system that describes the inside of the virtual cell. The choice between algorithm and multi-agent system depends on the nature of each problem within the cell and how much is known about the problem. A library of predefined behaviors such as receptor expression on the cell’s surface, receptor internalization, cell division (mitosis), programmed cell death (apoptosis) and generation of molecular messengers makes it possible to express a wide variety of behaviors (c).

Figure 3.1: Cell-Agent Design The multi-agent system is then obtained by placing an initial mixture of cells within a virtual environment (i.e. a virtual test tube or virtual organ), in other words, a group of cell-agents that each have a well-defined behavior. Within this universe, the interactions between cell-agents are local: two cell-agents can only interact if they are within the immediate neighborhood of the other cell-agent and if their receptors are compatible. These interactions take into account the distance between receptors and their relative direction, as well as their affinity [Smith 97]. At each cycle, the basic behavior of each cell-agent determines it movement by taking into account the interactions it had in its sphere of activity. If one of its receptors is activated through contact with a receptor from another cell-agent, then a more complex behavior can be triggered if all of its preconditions are validated.

Virtual Vein We have thus created, in collaboration with the Hematology Laboratory of the Brest University Hospital Center, a multi-agent model of a virtual vein [Ballet 00a]. The blood vessels are lined with a layer of endothelial cells of which one of the roles is to stop the adhesion of platelets. The blood circulates in the vessels carrying cells and proteins that will be activated when a vessel is torn in order to stop the bleeding. The stoppage of bleeding is a local phenomenon that takes place after a cascade of cellular and enzymatic events in a liquid environment made unstable by blood flow. The phenomena of hemostasis and coagulation involve mobile cells (platelets, red blood cells, granulocytes, monocytes), fixed cells (endothelial cells, subendothelial fibroblasts), procoagulant proteins and anticoagulent proteins. Each one of these elements has its own behavior; these different behaviors become auto-organized, which results in the hole being blocked by platelets interwoven with fibrin. Hence, our model takes into account the known characteristics of coagulation: 3 types of cells (platelets, endothelial cells and fibroblasts) and 32 types of proteins are

44

Collective Auto-Organization

included in 41 interactions (Figure 3.2). Some of these reactions take place on the cell surface, and others in the blood. Reagents

Products Description Exogenic Pathway TF + VIIa VII:TF Coagulation VII + VIIa VIIa + VIIa Coagulation VII + VII:TF VIIa + VII:TF Coagulation IX + VII:TF IXa + VII:TF Coagulation X + VII:TF Xa + VII:TF Coagulation TFPI + Xa TFPI:Xa Inhibition TFPI:Xa + VII:TF 0 Inhibition Endogenetic Pathway VII + IXa VIIa + IXa Coagulation IXa + X IXa + Xa Coagulation II + Xa IIa + Xa Coagulation XIa + IX XIa + IXa Coagulation XIa + XI XIa + XIa Coagulation IIa + XI IIa + XIa Retro-activation IIa + VIII IIa + VIIIa Retro-activation IIa + V IIa + Va Retro-activation Xa + V Xa + Va Retro-activation Xa + VIII Xa + VIIIa Retro-activation Xa + Va Va:Xa Prothrombinase VIIIa + IXa VIIIa:IXa Tenase VIIIa:IXa + X VIIIa:IXa + Xa Tenase Va:Xa + II Va:Xa + IIa Prothrombinase

Reagents Products Description AT3 + IXa 0 Inhibition AT3 + Xa 0 Inhibition AT3 + XIa 0 Inhibition AT3 + IIa 0 Inhibition alpha2M + IIa 0 Inhibition TM + IIa IIi Inhibition AT3 + IXa 0 Inhibition AT3 + Xa 0 Inhibition AT3 + XIa 0 Inhibition AT3 + IIa 0 Inhibition ProC + IIi PCa + IIi Inhibition PCa + Va 0 Inhibition PCa + Va:Xa Xa Inhibition PCa + PS PCa:S Inhibition PCa + PCI 0 Inhibition PCa:S + Va 0 Inhibition PCa:S + V PCa:S:V Inhibition PCa:S:V + VIIIa:IXa IXa Inhibition PCa:S:V + VIIIa 0 Inhibition Activation of Platelets P + IIa Pa + IIa Coagulation Formation of Fibrin I + IIa Ia + IIa Coagulation

The coagulation process is initiated on the surface of fibroblasts. It is the tissue factor (TF) on the surface of the fibroblasts that initiates the coagulation process. The so-called exogenous pathway is activated: Factor VIIa binds to the tissue factor and activates Factors VII, IX and X. The first thrombin molecules, key factor of the phenomenon, are generated, which launches the endogenous pathway and the actual coagulation cascade. Thrombin retroactivates the Factors XI, V, and VIII, and most importantly, the platelets. On the surface of the platelets thus activated, the tenase and prothrombinase complexes are then in a position to be created and make their important contribution to the formation of thrombin. Thrombin also activates the fibrinogen in fibrin, which then in turn binds together the activated platelets that are over the hole. The blood clot forms. At the same time, the inhibitors enter in action: the TFP1 binds to the activated Factor X to inhibit the exogenous pathway (Factor VIIa bound to the tissue factor), the alpha2macroglobulin and the antithrombin3 inhibit the thrombin and the activated Factors X, XI, IX. The Protein C is activated by the thrombin-thrombomodulin complex (IIi: thrombin inhibitor) cleaving to the surface of the endothelial cells. The activated Protein C has two courses of action: it directly inhibits the activated Factor V and indirectly, adhering to the Protein S and to the Factor V, inhibits the activated Factor VIII. The PCI (Protein C Inhibitor) prevents the inhibitor action of the activated Protein C.

Figure 3.2: Interactions of the Coagulation Model The wall of the vein is represented by a surface of 400 µm2 lined with endothelial cells. In the middle of this virtual wall, a hole exposes 36 fibroblast cells, Willebrand factors and tissue factors (TF). Platelets, the main factors of coagulation (Factors I (fibrinogen), II (prothrombin), V, VII, VIII, IX, X, XI) and inhibition factors (alpha2macroglobuline (alpha2M) and antithrombin3 (AT3)), TFPI (Tissue Factor Pathway Inhibitor), Protein C (PC), Protein S (PS), thrombomodulin (TM) and the Protein C inhibitor (PCi), which are all initially present in plasma, are placed randomly in the virtual vein.

In Virtuo Experiments Multi-agent simulation can then begin: each agent evolves over time according to its predefined behavior. Figure 3.3 presents six intermediary states of the phenomenon through a 3D viewer. So, we can observe primary hemostasis and plasmatic coagulation

45

Models

in different phases and over time, measure how the quantity of each element evolves. [Ballet 00b].

The 6 figures depict different stages of the evolution of the coagulation simulation; they were captured from different viewing angles of the virtual vessel. Image A shows the geometry of the models at the beginning of the simulation: the yellow spheres represent the endothelial cells. The hole in the vein, in the middle of the endothelial layer, shows the subendothelial fibroblasts (blue spheres). The coaguluation and inhibition factors in the plasma are represented by their Roman numerals, their Arabic numerals and their initials. The unactivated platelets are black spheres. In the beginning, the exogenous pathway is activated, indicated by the Factor VIIa’s bond on the tissue factor of the fibroblast membrane (red pyramid in Image B). The inhibition of these VIIa:TF complexes by the activated TFPI-Factor X complex is very fast (disappearance of red pyramids). Images C and D show the intermediary steps of the generation of thrombin: the Factor XI binds to the platelets and forms a complex with the Factor IX that it activates (Image C); the tenase complex (IXa:VIIIa), which also binds to the platelet surface, activates the Factor X (Image D). Image E shows a prothrombinase complex (activated Factor V complex-activated Factor X) on the activated platelets surface (red spheres), activating the prothrombin in thrombin. Finally, Image F demonstrates the adhesion of platelets to the subendothelium and the formation of blood clots. The generated fibrin molecules are represented by white and orange spheres (Image F).

Figure 3.3: Evolution of the Multi-Agent Simulation of Coagulation Validating models is based on several factors: . the similarities between thrombin generation curves obtained in virtuo and in vitro, . the consistency with pathologies: decrease of thrombin generated in hemophilia

46

Collective Auto-Organization

(Figure 3.4a) and increase in thrombophilia, . the correction of hemophilia through activated Factor VII (Figure 3.4b), . the correction of hypercoagulability through heparin (Figure 3.4c). Hence, several thousand entities2 , representing 3 types of cells and 32 types of proteins involved in 41 types of interactions, auto-organize in a cascade of reactions to fill in a hole that was not planned in advance, and which can be created at any time during the simulation and anywhere in the vein. This significant example reinforces the idea that reactive autonomous entities, through their conflicting and collaborative interactions, can become auto-organized and adopt a group behavior unknown from individual entities.

a b c Figure a compares the total number of thrombin molecules created during the simulation in the cases of normal coagulation, hemophilia A (absence of Factor VIII) and hemophilia B (absence of Factor IX). We can therefore observe a weaker coagulation in the case of hemophilia A and B: hemophilia A generates half the thrombins of a normal coagulation, while hemophilia B generates three times less. Figure b compares the thrombin generation curves in the case of normal coagulation, of hemophilia B (absence of Factor IX) and hemophilia B treated by adding activated Factors VII. The difference in coagulation between the treated hemophilia and normal coagulation is no more than 6.5% here. Figure c compares the total number of thrombin molecules created during the simulation in the case of normal coagulation, the Factor V Leiden mutation and Factor V Leiden treated with heparin. The Factor V Leiden mutation inhibits the deactivation of activated Factor V by activated Protein C. This genetic disease increases the risk of venous thrombosis: the number of thrombins formed is 4 times larger with the Factor V Leiden mutation than without it. Heparin is an anticoagulent that stimulates the action of anti-thrombin III and thus corrects hypercoagulability.

Figure 3.4: Main Results of the Multi-Agent Coagulation Model

3.3.3

In Vivo, In Vitro, In Virtuo

The main qualities of a model — an artificial representation of an object or a phenomenon — are based on its capabilities to describe, suggest, explain, predict and simulate. Model simulation, or model experiment, consists of testing this representation’s behavior under the effect of actions that we can carry out on the model. The results of a simulation then become hypotheses that we try to prove by designing experiments on the singular prototype of a real system. These experiments, rationalized in this way, make up a true in vivo experiment. We distinguish four main types of models (perceptive, formal, analogical, digital) that lead to five major simulation families. 2

Specific data is given in the thesis of Pascal Ballet [Ballet 00b].

47

Models

In Petto Simulation of a perceptive model corresponds to in petto intuitions that come from our imagination and the perception that we have of the system being studied. Thus, it makes it possible to test perceptions on the real system. Unclassified and unreasoned inspirations, idea associations and heuristics form mental images that have the power of suggestion. The scientific approach will try to rationalize these first impressions while artistic creation will make something from these digital or analogical works according to the media used. But it is often the suggestive side of the perceptive model that sparks these creative moments and leads to invention or discovery.

In Abstracto Simulation of a formal model is based on an in abstracto reasoning carried out within the framework of a theory. Reasoning provides predictions that can be tested on the real system. Galle’s discovery of Neptune in 1846, based on theoretical predictions by Adams and Le Verrier, is an illustration of this approach within the framework of the theory of disturbances from two bodies in celestial mechanics. Likewise, in particle physics, the discovery in 1983 of intermediate bosons W + , W − and Z 0 had been predicted a few years before by the theory of electroweak interactions. Hence, from the extremely large to the extremely small, the predictive characteristic of formal models has proven to be very fruitful in many scientific fields.

In Vitro Simulation of an analogical model takes place through in vitro experiments on a sample or model constructed by analogy with the real system. The similarities between the model and the system help us to better understand the system being studied. The wind tunnel tests on plane models allowed aerodynamicists to better characterize the flow of air around obstacles through the study of coefficients of similarity introduced at the end of the 19 th century by Reynolds and Mach. Likewise, in physiology, the analogy of the heart as a pump allowed Harvey (1628) to demonstrate that blood circulation fell under hydraulic laws. Thus, from all times, the explanatory side of analogical models was used, with more or less of an anthropocentric approach, to make what was unknown, known.

In Silico Simulation of a digital model is the execution of a program that is supposed to represent the system to be modeled. In silico calculations provide results that are checked against measurements from the real system. The digital resolution of mathematical equation systems corresponds to the most current use of digital modeling. In fact, analytical determination of solutions often encounters problems that are just as likely to involve the characteristics of the equations to be solved (non-linearity, coupling) as the complexity of the limit conditions and the need to take into account very different space-time scales. Studying the kinematics of chemical reactions, calculating the deformation of a solid under

48

Individual Perception

the effect of thermomechanical constraints, or characterizing the electromagnetic radiation of an antenna are classic examples of implementing differential equation systems on a computer. Consequently, the digital model obtained by discretization from the theoretical model has today become a vital tool for going beyond theoretical limitations, but is still often considered as a last resort.

In Virtuo More recently, the possibility of interacting with a program that is being run has opened the way to real in virtuo experiments of digital models. It is now possible to disturb a model that is being run, to dynamically change the limit condition and to delete or add elements during simulation. This gives digital models the status of being virtual models, infinitely more malleable than the real model used in analogical modeling. Flight simulators and video games were the precursors of virtual reality systems. However, these systems become necessary when it is difficult, or even impossible to use direct experiments for whatever reason: hostile environment, access difficulties, space-time constraints, budget constraints, ethics, etc. The user is able to go beyond simply observing the digital model’s activity while it is being run on a computer, and test the reactivity and adaptability of the model in operation; thus benefitting from the digital model’s behavioral characteristic. The idea of a model representing the real is based on two metaphors, one artistic and the other legal. The legal metaphor of delegation (the elected represents the people, the papal nuncio represents the pope, the ambassador represents the head of state) suggests the idea of replacement: the model takes the place of reality. The artistic metaphor of realization (the play is performed in public, artistic inspiration is represented by a work of art) proposes the idea of presence: the model is a reality. An in virtuo experiment of a digital model makes it truly present and thus opens up new fields of exploration, investigation and understanding of the real.

3.4

Individual Perception From time to time, one runs into a figure from a small part of [such] a network of symbols, in which each symbol is represented by a node where arcs come in and go out. The lines symbolize in a certain manner the links of trigger chains. These figures try to reproduce the intuitive notion of conceptual connectedness. [...] The difficulty is that it is not easy to represent the complex interdependence of a large number of symbols through a couple of lines connecting the dots. The other problem with this type of diagram, is that it is wrong to think that a symbol is necessarily in service or out of service. If it is true of neurons, this bistability will not reflect all of the neurons. From this point of view, the symbols are much more complex that the neurons, which is hardly surprising since they are made up of many neurons. The messages exchanged between the symbols are more complex than a simple I am activated, which is about the content of the neuron messages. Each symbol can be stimulated in many different ways, and the type of stimulation determines which other symbols will be activated. [...] Let’s imagine that there are node configurations united by connections (that can be in several colors in order to highlight the different types of conceptual connectedness) accurately representing the simulation mode of symbols by other symbols. [Hofstadter 79]

To become emotional, behaviors must determine their responses, not only according to external stimuli, but also according to internal emotions such as fear, satisfaction, love

49

Models

theories Formal models ion

tat

n me

ple

im

digitization

programs

m

predictions

formalization

in abstracto reasoning ate

ria

liz

ati

on

analogies

prototypes

Digital models

models

Real phenomena

in virtuo experiments

in vivo experiments

dig

ita

lc

rea

tio

n

in vitro experiments

similarities

n

tio

perception

impressions

behaviors

Analogical models

l

ica

a cre

g alo

an

imaginary Perceptual models in petto intuitions

field

modelization

Model simulation

understanding

Figure 3.5: Modeling and Understanding Phenomena or hate. We propose to describe such behaviors through fuzzy cognitive maps where these internal states will be explicitly represented, and make it possible to clearly draw a line between perception and sensation

3.4.1

Sensation and Perception

Fuzzy Cognitive Maps Cognitive maps are taken from psychologists’ works that presented this concept to describe the complex behaviors of topological memorization in rats (cognitive maps [Tolman 48]). The maps were then formalized as directed graphs and used in the theory of applied decision in economics [Axelrod 76]. Finally, they were combined with a fuzzy logic to become fuzzy cognitive maps (FCM : Fuzzy Cognitive Maps [Kosko 86]). The use of these maps was even considered for the overall modeling of a virtual world [Dickerson 94]. We propose here to delocalize fuzzy cognitive maps at every agent level in order to model their autonomous perceptive behavior within a virtual universe. Just like semantic networks [Sowa 91], cognitive maps are directed graphs where the nodes are the concepts (Ci ) and the arcs are the influence lines (Lij ) between these 50

Individual Perception

concepts (Figure 3.6). An activation degree (ai ) is associated with each concept while the weight Lij of an arc indicates a relationship of inhibition (Lij < 0) or stimulation (Lij > 0) from concept Ci to concept Cj . The dynamics of the map are mathematically calculated by normalized matrix product as specified by the formal definition of fuzzy cognitive maps given in Figure 3.7.

a3

This map is made up of 4 concepts and has 7 arcs. Each concept Ci has one degree of activation ai .

+1 −1



−1 −1

a1

a4 +1

−1

+2

a2

 a1  a2   a=  a3  , L = a4



0  0   −1 0

+2 −1 0 0

−1 0 0 +1

 −1 +1   0  0

A zero in the line matrix Lij = 0 indicates the absence of a concept arc Ci to concept Cj and a non null element of the diagonal Lii 6= 0 corresponds to a concept arc Ci on itself.

Figure 3.6: Example of a Cognitive Map

We use fuzzy cognitive maps to specify the behavior of an agent (graph structure), and to control its movement (map dynamics). Thus, a fuzzy cognitive map has sensory concepts whose activation degrees are obtained by the fuzzification of data coming from the agent’s sensors. It has motor concepts whose activations are defuzzified to be sent to the agent’s effectors. The intermediary concepts reflect the internal state of the agent and intervene in the calculation of the map’s dynamics. The fuzzification and the defuzzification are reached according to the principles of fuzzy logic [Kosko 92]. Hence, a concept represents a fuzzy subset, and its degree of activation represents the membership degree in the fuzzy subset.

Feelings As an example, we want to model an agent that is perceiving its distance from an enemy. According to this distance and its fear, it will decide to flee or stay. The closer the enemy, the more afraid it is, and in turn, the more afraid, the more rapidly it will flee. We are modeling this flight through the fuzzy cognitive map in Figure 3.8a. This map has four concepts: two sensory concepts (enemy is near and enemy is far), a motor concept (flee) and an internal concept (fear). Three lines indicate the influences between the concepts: the proximity of the enemy stimulates fear (enemy is near → fear), fear causes flight (fear → flee), and the enemy moving away inhibits fear (enemy is far → fear). We chose the unforced (fa = 0) continuous mode (V = [0, 1], δ = 0, k = 5). The activation of sensory concepts (enemy is near and enemy is far) is carried out by the fuzzification of the sensor from the distance to the enemy (Figure 3.8c) while the defuzzification of the desire to flee gives this agent a flight speed (Figure 3.8d).

51

Models

K indicates one of the rings ZZ or IR, by δ one of the numbers 0 or 1, by V one of the sets ∗. {0, 1}, {−1, 0, 1}, or [−δ, 1]. Given (n, t0 ) ∈ IN 2 and k ∈ IR+ A fuzzy cognitive map F is a sextuplet (C, A, L, A, fa , R) where: 1. C = {C1 , · · · , Cn } is the set of n concepts forming the nodes of a graph. 2. A ⊂ C × C is the set of arcs (Ci , Cj ) directed from Ci to Cj .

C×C → K 3. L : is a function of C × C to K associating Lij to a pair of concepts (Ci , Cj ), (Ci , Cj ) 7→ Lij with Lij = 0 if (Ci , Cj ) ∈ / A, or with Lij equal to the weight of the arc directed from Ci to Cj if (Ci , Cj ) ∈ A. L(C × C) = (Lij ) ∈ K n×n is a matrix of Mn (K). It is the line matrix of the map F that, to simplify, we observe L unless indicated otherwise. C → V IN 4. A : is a function that at each concept Ci associates the sequence of its activation Ci 7→ ai degrees such as for t ∈ IN, ai (t) ∈ V given its activation degree at the moment t. We will observe a(t) = [(ai (t))i∈[[1,n]] ]T the vector of activations at the moment t. 5. fa ∈ (IRn )IN a sequence of vectors of forced activations such as for i ∈ [[1, n]] and t ≥ t 0 , fai (t) given the forced activation of concept Ci at the moment t. 6. R is a recurring relationship on t ≥ t0 between ai (t + 1), ai (t) and fai (t) for i ∈ [[1, n]] indicating the map dynamics F. 

∀i ∈ [[1, n]], ai (t0 ) = 0 ; ∀i ∈ [[1, n]], ∀t ≥ t0 , ai (t + 1) = σ ◦ g fai (t),

X

j∈[[1,n]]



Lji aj (t)

where g : IR2 → IR is a function of IR2 to IR, for example: g(x, y) = min(x, y) or max(x, y) or αx + βy, and where σ : IR → V is a function of IR to the set of activation degrees V normalizing the activations as follows: continuous mode

1

k (1+δ)/2

binary mode

ternary mode

+1 0

(1−δ)/2

σ

σ

σ

(0,0.5,5)

(0,0.5,5)

−δ a0

a

(1,0,5)

0

b

−1

(a) In continuous mode, V = [−δ, 1], σ is the sigmoid function of σ(δ,a0 ,k) centered in slope k · 1+δ in a0 and from limits in ±∞ respectively 1 and −δ : 2 σ(δ,a0 ,k)

IR : a

→ 7→

c ), (a0 , 1−δ 2

from

[−δ; 1] 1+δ −δ 1 + e−k(a−a0 )

0 if σ(0,0.5,k) (a) ≤ 0.5 (b) In binary mode, V = {0, 1}, σ : a 7→ . 1 if σ(0,0.5,k) (a) > 0.5 −1 if σ(1,0,k) (a) ≤ −0.5 if −0.5 < σ(1,0,k) (a) ≤ 0.5 . (c) In ternary mode, V = {−1, 0, 1}, σ : a 7→ 0 1 if σ(1,0,k) (a) > 0.5

Figure 3.7: Definition of a Fuzzy Cognitive Map

Perceptions We make a distinction between feeling and perception: feeling results from sensors only, perception is the feeling influenced by the internal state. A fuzzy cognitive map 52

ennemi is far sensory map

+λ



0  0  L= 0 0

0 0 0 0

 0 0  a +1  0

+1 −1 0 0

ar e los sc

flee

yi em en

fear −1

1

sf

+1

+1

en em yi

enemy is close

degree of activation

Individual Perception

0 0

20

80

100

distance to the enemy c

+1

+1 fear

−1 enemy is far

−λ

flee

+γ



0  0 L=  +λ 0

0 0 −λ 0

+1 −1 +γ 0

 0 0  b +1  0

speed of flight

perception map

enemy is close

1

0 0

1

activation of flight

d

The sensory concepts in dashed lines are activated by the fuzzification of sensors. The motor concepts in pointed lines activate the effectors by defuzzyfication. In (a), concept C1 = enemy is close stimulates C3 = fear while C2 = enemy is far inhibits it and fear stimulates C4 = flee. This defines a map that is purely sensory. In (b), the map is perceptive: the fear can be auto-maintained (memory) and even influence feelings (perception). In (c), the fuzzification of the distance to an enemy results in two sensory concepts: enemy is close and enemy is far. In (d), the defuzzification of flee linearly regulates the flight speed.

Figure 3.8: Flight Behavior makes it possible to model perception thanks to lines existing between internal concepts and sensory concepts. For example, we add three lines to the previous flight map (Figure 3.8b). One auto-stimulator line (γ ≥ 0) on fear models stress. A second stimulator line (λ ≥ 0) of fear to enemy is close and an inhibitor line (−λ ≤ 0) from fear to enemy is far model the phenomenon of paranoia. A given distance to the enemy will seem quite long according to the activation of the fear concept. The agent then becomes perceptive based on its degree of paranoia λ and its degree of stress γ (Figure 3.9).

a b The perception of the distance from an enemy can be influenced by fear: according to the proximity of an enemy and fear, the dynamics of the map determine a speed obtained here in the 3rd cycle. In (a), λ = γ = 0, the agent is purely sensory and its perception of the distance to the enemy does not depend on its fear. In (b), λ = γ = 0.6, the agent is perceptive: its perception of the distance to an enemy is modified by its fear.

Figure 3.9: Flight Speed

53

Models

3.4.2

Active Perception

Feeling of Movement Neurophysiology associates the feeling of movement to the permanent and coordinated fusion of different sensors such as myoarticular (or proprioceptive) sensors, vestibular receptors and cutaneous receptors [Lestienne 95]. Among the myoarticular receptors, certain muscle and articular sensors measure the relative movements of body segments in relation to each other; other articular sensors act as force sensors and are involved for the purpose of effort. The labyrinthic receptors are situated in the vestibular system of the internal ear. This system is a real center of gravity and inertia that, on the one hand, is sensitive to the angular accelerations of the head around three perpendicular axes (semi-circular channels), and on the other hand, is sensitive to the linear accelerations of the head, including that of gravity (otoliths). Cutaneous receptors measure pressure variations and are responsible for tactile sense (touch). Peripheral vision is also included in the feeling of movement because it provides information about relative movements within a visual scene. In certain circumstances, this contribution to vision can conflict with data received from proprioceptive, vestibular and tactile receptors; this conflict can result in postural disturbances that can be accompanied by nausea. It is what can happen when you read a book in a moving vehicle: vision, absorbed with reading, tells the brain that the head is not moving, while the vestibular receptors detect the movement of the vehicle; this contradiction is the main physiological cause of motion sickness.

Internal Simulation of Movement The feeling of movement thus corresponds to a multi-sensor fusion of different sensory information. But this multi-sensory information is itself combined with signals coming from the brain that control the motor control of muscles. According to Alain Berthoz, neurophysiologist, perception is not only an interpretation of sensory messages, it is also an internal simulation of action and an anticipation of the consequences of this simulated action [Berthoz 97]. This is the case when a skier is going down a hill: he mentally goes down the hill, predicting what will happen, at the same time that he is actually going down the hill, intermittently checking the state of his sensors. Internal simulation of movement is facilitated by a neuronal inhibition mechanism. For vision, the brain is able to imagine eye movements without doing them thanks to the action of inhibitor neurons that shut down the ocular muscles’ command circuit: by staring at a point in front of you and focusing your attention, a sort of interior gaze if you will, you really feel as if your gaze is traveling from one point of the room to another. This virtual movement of the eye was simulated by the brain by activating the same neurons; only the action of the motor neurons was inhibited. Anticipating movement is well illustrated by the Kohnstamn illusion, named after the physiologist who was the first to study this phenomenon. When a person is balancing a tray with a bottle on it, the brain adapts to this situation in which the arm remains motionless through a constant effort made by the muscles. If someone was to suddenly

54

Individual Perception

remove the bottle, the tray would spring up all by itself. In fact, the brain continues to apply this force until the muscle sensors signal a lifting movement. If the person carrying the tray took the bottle off himself, the tray would not move: the brain would have anticipated the movement and signaled for the muscles to relax in response to the change in the weight of the tray. The brain can therefore be considered as a biological simulation that makes predictions by using its memory and forming hypotheses about the phenomenon’s internal model. To understand the role of the brain in perception, we need to begin with the goal pursued by an organism and then study how the brain communicates with sensors by specifying estimated values according to an internal simulation of anticipated consequences of action.

Perception of Self An agent can also use a fuzzy cognitive map in an imaginary place and simulate a behavior. Thus, with its own maps, it can reach a perception of itself by observing itself in an imaginary domain. If it is familiar with the maps of another agent, it will have a perception of the other and will be able to mimic a behavior or a cooperation. In fact, if an agent has a fuzzy cognitive map, it can use it in simulation by forcing the values of sensory concepts and by making motor concepts act in an imaginary place as in a projection of the real world. Such an agent is able to predict its own behavior or that of a deciding agent, according to this map, by going through several cycles of the map in his imaginary place or by determining an attractor (fixed point or limited cycle in discrete mode) of the map. Figure 3.10 illustrates this mechanism with the map from Figure 3.8b used for 20 cycles in fuzzy mode and an initial stress vector accounting for ξ for t ≤ t 0 only on fear, then null the rest of the time: fa (t < t0 ) = (0, 0, ξ, 0)T , fa (t > t0 ) = 0. With the same stress conditions, the binary mode converges towards a fixed point: a(t > t f ) = (0, 1, 0, 0)T or a(t > tf ) = (1, 0, 1, 1)T ; either the enemy is far and it is not scared and does not flee, either the enemy is close, it is scared and flees. In addition, if an agent is familiar with the cognitive maps of another agent, it can anticipate the behavior of this agent by simulating a scenario created by going through the cycles of their maps in an imaginary place. This technique opens the doors to a real cooperation between agents, each one being able to represent the behavior of the other. Hence, the fuzzy cognitive maps make it possible to specify the perceptive behavior of an agent (graph structure), to control the agent’s movement (map dynamics) and to simulate movement internally (simulation within the simulation). We can then implement them within the framework of interactive fiction.

3.4.3

The Shepherd, His Dog and His Herd

The majority of works in virtual reality involve the multi-sensory immersion of the user into virtual universes, but these universes, as realistic as they are, will lose credibility as long as they are not populated with autonomous actors. The first works on modeling virtual actors focused on the physical behavior of avatars [Magnenat-Thalmann 91], and made it possible, for example, to study worksation ergonomics (Jack [Badler 93]). Another

55

Models

a b Simulation and fixed point of flight speed with λ = γ = 0, 6. In (a), the mode is continuous and this behavior is obtained at the end of the map’s twentieth cycle. In (b), the mode is binary and the vector of activations of the map stabilizes on a fixed point such as: either all of the concepts are at 1 except enemy is far, which is at 0 and the speed is at 1, or the opposite and the speed is at 0. We can observe that if the agent is scared at the time of the simulation, no matter what the distance from the enemy, it would perceive it as if he was really close.

Figure 3.10: Perception of Self and Others category of works dealt with the problem of interaction between a virtual actor and a human operator: for example, companion agent (ALIVE [Maes 95]), training agent (Steve [Rickel 99]) and animator agent [Noma 00]. Finally, others became interested in the interaction between virtual actors; in such virtual theaters, the scenario can be controlled at an overall level (Oz [Bates 92]), depend on behavior scripts (Improv [Perlin 95]) or be based on a game of improvisations (Virtual Theater Project [Hayes-Roth 96]). We propose here to use fuzzy cognitive maps to characterize believable agent roles in interactive fictions. We will illustrate this through a story about a mountain pasture. Once upon a time there was a shepherd, his dog and his herd of sheep . . . This example has already been used as a metaphor of complex collective behavior within a group of mobile robots (RoboShepherd [Schultz 96]), as an example of the watchdog robot of real geese (Sheepdog Robot [Vaughan 00]), and as an example of improvisation scenes (Black Sheep [Klesen 00]). The shepherd moves around in his pasture and can talk to his dog and give him information. He wants to round up his sheep in an area which he chooses based on needs. By default, he stays seated: the shepherd is an avatar of a human actor that makes all of the decisions in his place. Each sheep can distinguish an enemy (a dog or a man) from another sheep and from edible grass. It evaluates the distance and the relative direction (left or right) from an agent that it sees in its field of vision. It knows how to identify the closest enemy. It knows how to turn to the left or to the right and run without exceeding a certain maximum speed. It has an energy reserve that it regenerates by eating and spends by running. By default, it moves straight ahead and ends up wearing itself out. We want the sheep to eat grass (random response), be afraid of dogs and humans when they are too close, and in accordance with the Panurge instinct, to try and socialize. So, we chose a main map (Figure 3.11) containing all of the sensory concepts (enemy is close, enemy is far, high energy, low energy), motor concepts (eat, socialize, flee, run) and internal concepts (satisfaction, fear). This map calculates the moving speed by defuzzification of the run

56

Conclusion

concept, and the direction of movement by defuzzification of the three eat, socialize and flee concepts; each activation corresponds to a weight on the relative direction to be followed respectively (to the left or to the right) to go towards the grass, join another sheep or flee from an enemy. 1

+

−

friend right

enemy to the right

turn right

friend left

−

turn right

+

− turn left

5

+

enemy in the middle

−

turn left

− −

+

socialization

shepherd

9 9

enemy to the left

sheep c

flight

sheep a 5

enemy is close

1

run

enemy is far

−1 −1

+1 high energy

+1

9 5

+1

+1 +1

+1

fear +1

satisfaction

flee

−1 low energy

−1 +1

perception

sheep b

+1 eat

socialize

b 1 In (a), the fuzzy cognitive maps define the role of a sheep. The two maps at the top determine its direction towards a goal; the one at the bottom gives the sheep its character. In (b), the sheep try to herd together while avoiding the shepherd. The 3 sheep act by leaving a dated footprints on their paths.

Figure 3.11: Fear and Socialization The dog is able to identify humans, sheep, the pasture area and the watchpoint. It distinguishes by sight and sound its shepherd from another man and knows how to spot the sheep that is the farthest away from the area among a group of sheep. It knows how to turn to the left and to the right and run up to a maximum speed. Initially young and full of energy, its behavior consists of running after the sheep, which quickly scatters the sheep (Figure 3.12a). First, the shepherd wants the dog to obey the order to not move, which will lead the sheep to socialize. This is done by giving the dog a sensory map to the shepherd’s message, which inhibits the dog’s desire to run (Figure 3.12b). The dog’s behavior is decided by the map and the dog stays still when its master asks it to (message dissemination stop). Then, the shepherd gives the dog a structured map based on the concepts associated with the herd’s area, for example whether a sheep is either outside or inside the area. The concepts also make it possible for the dog to bring a sheep back (Figure 3.12c,d,e) and keep the herd in the area by situating itself at the watchpoint, in other words, on the perimeter and across from the shepherd. It is remarkable to observe that, the path in S of the virtual sheepdog emerging from this in virtuo simulation (Figure 3.12c), and not explicitly programmed, is an adapted strategy that is observable in vivo by a real sheepdog that herds a group of sheep.

3.5

Conclusion

The models that we develop in virtual reality must follow the principle of autonomy of the virtual objects that make up virtual universes.

57

Models 1

shepherd

4 4

1

1

4

7

stop

sheep a sheep b

1

watchpoint

−1

7

dog

7 4

run

7

sheep c a b In (a), the shepherd is immobile. A circular area represents the grazing area. A watchpoint is attached to the area that is always located diagonally across from the shepherd: this is where the sheepdog must be when all of the sheep are inside the area. Initially, the behavior of a dog without the fuzzy cognitive map is to run after the sheep, which quickly scatter and go outside the gathering area. In (b), the simple map depicts the obedience of a dog to the message stop of its shepherd by inhibiting the desire to run. stop

1

shepherd

sheep to the left

sheep to the right

sheep a 13 13

+2

9

5

sheep b

1

−1

+2 turn right

+1

+1

−1

sheep c area to the left

9

5

1

−2

turn left

13 1 5 9

5

−2

13

return

dog

9

area to the right

watchpoint

e

c sheep is far

+1

−3 run

−1

sheep is close

+1 +1

−3 +1 −1

−1 −1

follow slowly

sheep close to area

−1

−1

+1

−1 watchpoint is close

sheep in −1

+1

sheep out −1

large area

dog

+1

+1 −1

+1

−1

guard

return

+1

sheep far from area

wait +1

+1

For this in virtuo simulation, the socialization of sheep is inhibitied. In (c), the film of the sheepdog bringing back 3 sheep unfolds with their maps of Figure 3.11 for the sheep and the maps represented in (d) and (e) for the dog. In (d), the main map of the dog depicts the role consisting of bringing back a sheep to the area and maintaining a position near the shepherd when all of the sheep are in the desired area. In (e), the map decides the approach angle of the sheep to bring it back to the area: go towards the sheep, but approach it from the opposite side of the area.

stop

watchpoint is far +1 small area

d

Figure 3.12: The Sheepdog and Its Sheep

We have chosen to design these virtual worlds as multi-agent systems that consider the human user as a participant, and not simply as an observer. The evolution of these systems gives rise to multi-agent simulations in which the human user fully participates by playing his role as a spectator, an actor and a creator. Participatory simulation of virtual universes composed of autonomous entities then allows us to first research phenomena that have been made complex by the diversity of components, structures, interactions,

58

Conclusion

and modeling. Entities are modeled by autonomizing all behavioral models, regardless of whether they are reactive, emotional, intentional, developing or social. The behavior of all of these entities is then indicative of the system’s auto-organization capabilities and is one of the expected results of such simulations. By participating in these simulations, the human operator can, according to a principle of substitution, take control of the entity. Thus, by a sort of digital empathy, he can identify with the entity. In return, the controlled entity can learn behaviors that are more adapted to its environment, from the human operator. The study of blood coagulation through this multi-agent approach implemented several thousand autonomous entities with reactive behavior. The simulations highlight the auto-organizing capabilities of cells and blood proteins to fill in a hole created by the user-troublemaker, anytime and anywhere in the virtual vein. The user-observer, upon observing a coagulation abnormality (hemophelia or thrombosis), becomes user-pharmacist by adding new molecules during the simulation, thus allowing the user-biologist to test new reactives, at a lower cost and in total safety. Within this kind of virtual laboratory, the biologist conducts real in virtuo experiments that involve a real-life experience, which is not the only interpretation of in silico calculations. The biologist then has a new experiment process for understanding biological phenomena. Writing an interactive fiction by modeling the behaviors of virtual actors with the help of fuzzy cognitive maps, demonstrates the triple role of these maps. First, they allow you to specify a perceptive behavior by defining sensory, motor and emotional concepts as nodes on a directed graph; the arches between these nodes translate the exciter or inhibitor relationships between the associated concepts. They then allow you to control the movement of actors during execution, thanks to their dynamics. Finally, they open the way to an active perception when they are simulated by the agents themselves. Simulation in simulation, this active perception, like a kind of inner gaze, takes place through an auto-evaluation of the agent’s behavior and is an outline of self-perception. In the end, these autonomous behavior models must be implemented and executed on a computer. Therefore, we must ask ourselves about the language that is used and the associated simulator. This is what we will discuss next in Chapter 4.

59

Models

60

Chapter 4 Tools 4.1

Introduction

This chapter introduces the generic tools that we developed in order to implement the models that were presented in Chapter 3. The tools are based on an oRis 1 development platform (language + simulator), which was designed to meet the needs of our participatory multi-agent simulation and is the core of AR´ eVi, our distributed virtual reality workshop . The first part of this chapter will focus on the oRis language, a generic object-based concurrent programming language that is dynamically interpreted and has an instance granularity. Next, we will concentrate specifically on the associated simulator in order to demonstrate that it was built so that the activation procedure for autonomous entities does not lead to bias in the simulation, for which the execution platform would be solely responsible. In this environment, execution error robustness will prove to be a particularly significant point, especially since the simulator provides access to the entire on-line language. Finally, we will reconstruct our oRis platform in the context of participatory multiagent simulation environments and explain which ones are the first applications.

4.2

The oRis Language When to say something is to do something. [Austin 62]

In order for an operator to easily play those three roles (spectator, actor, creator), while respecting the autonomy of the agents with whom he cohabits, he must have a language to respond to other models, modify them, and eventually create new classes of models and instantiate them. Therefore, while the simulation is running, he has the same power of expression as the creator of the initial model. The language must then be dynamically 1

The name oRis was freely inspired from the Latin suffix oris, which is found in the plural form of words that are related to language such as cantatio → to sing, cantoris → singers (those who sing), or oratio → speech, oratoris → speakers (those who speak). We kept the triple symbolism of language, the plural and the act: multi-agent language.

61

Tools

interpreted: new code must be able to be introduced (and therefore interpreted) during execution. The user may end up only interacting with or modifying an instance of a model. So, the language must have an instance granularity, which requires a naming service and most importantly, the ability to syntactically distinguish the overall context code from the class and instance codes. It seems then that participatory multi-agent simulation needs to have an implementation language that incorporates, of course, the paradigm of object-based concurrent programming (the interest of this paradigm for multi-agent systems is not new [Gasser 92]) but that also provides additional properties such as a dynamically interpreted language with an instance granularity.

4.2.1

General View

oRis is an interpreted, object-oriented language with strong typing. It also has many similarities with C++ languages and Java, which makes learning it easier. Just like these general-purpose languages, oRis makes it possible to tackle different application-oriented topics; if you integrate components developed with these languages, the applications’ logical framework architecture remains homogeneous, which facilitates the reusability of third-party components and the extensibility of the oRis platform. Figure 4.1 provides an illustration of a very basic, yet complete, program, which defines a class and launches several processes (start blocks). We can see that the class describes active objects, whose behaviors run at the same time as other processes that are initiated in other local contexts. oRis generates a garbage collector. It is therefore possible to choose which instances are to be destroyed automatically (however, this decision can be revoked dynamically); the explicit destruction of an instance by the operator delete is always possible. Consequently, there is a mechanism that lets you know if a reference is still valid. We have an interface with C++ language according to a libraries procedure that can be downloaded on request. Not only does oRis make it possible to combine functions or methods with an implementation in C++, but it allows further use of the object approach by directly creating a language instance that is interpreted by an instance described in C++. Although oRis is implemented in C++, it is possible to integrate work that was done in other languages. A package ensures that oRis code can interface with SWI-Prolog 2 code: a function that makes it possible to call upon the Prolog interpreter in oRis and an oRis method in Prolog. oRis was also interfaced with Java in such a way that any Java class, whether it is part of standard classes or the subject of a personal project, can be used directly from oRis. There is no need to format the script language’s calling conventions, as is often the case to load from C++. Many standard packages are provided in oRis. These mainly include services that are completely routine, but are nevertheless essential, so that the tool can really be used in different conditions. These packages manage topics such as data conversion, mathematics, mutual exclusion semaphores, reflex connections, generic containers, graphic interface components, curve plotting, objects situated in a plane or in space, reflection 2

62

SWI-Prolog: www.swi.psy.uva.nl/projects/SWI-Prolog

The oRis Language

class MyClass { string _txt; void new(string txt) { _txt=txt; } void delete(void) {} void main(void) { println("I say: ",_txt); } } start { println("---MyClass i=new MyClass j=new println("---}

// define the class MyClass // // // //

an attribute constructor destructor behavior

// start a process // in the application block start ----"); MyClass("Hello"); MyClass("World"); block end ----");

start { for(int i=0;i Example.3 is an Example ! Example.2 is an Example ! Example.1 is an Example ! Example.2 is an Example ! 2 --> Example.3 is an Example !

Figure 4.6: Modifying an Instance

76

Applications

Figure 4.6 gives an example of using a parse() function to modify the behavior of an instance. Composing a character string that involves one of the three instances allows you to add an attribute and a method as well as the overdefinition of an existing method. The distinction between class and instance is made naturally by using the range resolution operator (::). Such modifications can of course be made several times during execution. The dynamic functionalities presented here require no special programming technique or device in the sense that there is no difference between the code form that you write off-line and that of the code that you create dynamically on-line. The rules for rewriting are the same in every case. This makes it possible in particular to develop a behavior on an instance so that afterwards it can be applied to an entire class by simply changing the established code range. Access to reference on an object-type constants makes it possible to easily act on any instance. Indeed, if this lexical form did not exist, it would be necessary to memorize a reference on each instance in order to be able to act at the right moment. In oRis, the user does not have to worry about this kind of detail; if by some means (graphics inspector, pointing in a window, etc.) we display the name of an object, we can reuse it to make it execute processes or to modify it.

4.4

Applications Virtual worlds must be realized, in other words, one must endeavor to update what is virtually present in them, to know the intelligible models that structure them and the ideas that develop them. The “fundamental virtue” of virtual worlds is to have been designed with an end in mind. It is this end that must be realized, actualized, whether the application is industrial, space-related, medical, artistic, or philosophical. The images of virtual must help us to reveal the reality of virtual, which is of an intelligible order, and of intelligibility proportional to the pursued end, theoretical or practical, utilitarian or contemplative [Qu´ eau 93b]

4.4.1

Participatory Multi-Agent Simulations

The development of software engineering lead to the definition of the models, methods and methodologies that had been made operational by the various tools that implemented them. However, the expression software development environment is far too vague: no universal set and, a fortiori, no tool covers all of the needs. In the more specialized field of multi-agent system development, this same kind of diversity can be found in methods [Iglesias 98], models (for example, a model for coordinating actions such as the contract network [Smith 80]), languages (for example, MetateM [Fisher 94], ConGolog [De Giacomo 00]), application builders such as ZEUS [Nwana 99], component libraries such as JATLite7 , multi-agent system simulators like SMAS8 , toolkits, which are generally class packages offering basic services (agent life cycle, communication, message transferring, etc.) and classes implementing some components of a multi-agent system ([Boissier 99] describes a group of platforms developed by the French science community 7

JATLite: Java Agent Template Lite (java.stanford.edu/java agent/html)

8

SMAS: Simulation of Multiagent Asynchronous Systems (www.hds.utc.fr/~barthes/SMAS)

77

Tools

and [Parunak 99] makes reference to other tools of this type). According to Shoham, agent-oriented programming system is made up of three basic elements [Shoham 93]. 1. A formal language for describing the mental states of agents, which, according to Shoham, are modalities such as beliefs or commitments. As a result, a modal logic is defined, as the author does in his Agent-0 language; the internal states of a reactive agent can be described as a system of discrete events such as a Petri network [Chevaillier 99a]. 2. A language for programming agent behaviors that must respect the semantics of mental states; thus, Agent-0 defines an interpretation for updating mental states – which is based on the theory of language acts – and executing commitments. It would be conceivable in this framework to use KQML9 and, if heterogeneity or standardization constraints were imposed, a language respecting FIPA 10 specifications. In the case of a discrete events system, it would be a good idea to define the semantics of the states machine (for example, an interpretation of a Petri network). 3. An “agentifier” that can convert a neutral component into a programmable agent; Shoham admits to having very few ideas on this topic. Let us reformulate this requirement by saying that it is necessary to have a language that allows you to make agents operational, i.e., which allows you to make them implementable and executable. We prefer to think of this third element as an implementation language. The first two elements – which are closely related – are open research fields and the types of multi-agent systems that we would like to develop must be selected carefully. To integrate these elements into a general-purpose platform inevitably leads to two pitfalls: platform instability, as the field is still open, and the diktat of a model, which is not necessarily appropriate for all situations. In regard to the third element, there are several possible choices, which are conditioned by the type of considered application and by the technological characteristics of the agents execution environments. The two major types of applications for multi-agent systems — the resolution of problems and simulation — result in the following typology (according to [Nwana 96]). . To solve problems:  on-board agents (e.g. physical agents (process controls) and interface agents): there are heavy efficiency constraints for program execution and dependency in regards to the technology of the target system (many applications are written in C or in C++);  mobile agents: it is better to use a language which can be interpreted on a large community of operating systems; using script languages such as Perl is possible;  rational agents: programming in logic with a language such as Prolog can be perfect for meeting this type of need. 9 10

78

KQML: Knowledge Query and Manipulation Language (www.cs.umbc.edu/kqml) FIPA: Foundation for Intelligent Physical Agents (www.fipa.org)

Applications

. For simulation: it is necessary to have an environment where different agent behaviors can be executed in parallel and which offers a wide range of components to interface with users. The choice depends on the languages being executed on a virtual machine that allows parallel programming: in the past Smalltalk’80 and, more and more, Java (the development of the DIMA platform gives evidence of this trend [Guessoum 99]). We are well aware that this classification, like a lot of others, is somewhat arbitrary and that different languages can be applied to different fields (everything can be done in assembler!) and there are several solutions to the same problem. Its only pretense is to explain the oRis positioning to the reader as a participatory multi-agent simulation platform. oRis was designed to meet both the need for an implementation language for multi-agent systems and these systems’ participatory simulation requirements. The entire oRis environment and language represent more than one hundred thousand lines of code mainly written in C++ but also in Flex++ and Bison++, and in oRis itself. oRis is completely operational and stable and is already being used in many projects.

4.4.2

Application Fields

oRis is used as a teaching tool in certain courses given at the Ecole Nationale d’Ing´enieurs de Brest (National Graduate Engineering School of Brest): for example, programming by concurrent objects, multi-agent systems, distributed virtual reality, adaptive control. It is also used by other educational institutions: Military Schools of Saint-Cyr Co¨etquidan, ENSI (National School of Information Sciences) of Bourges, ENST (National Graduate Engineering School of Telecommunications of Brest) in Brittany, IFSIC (Institution for Fuzzy Systems and Intelligent Control)– DEUG (Diploma of General University Studies) University of Rennes 1, DEA (post-Master’s degree) from the University of Toulouse (IRIT), Institute of Technology in Computing of Bordeaux, and a Master’s degree from the University of Caen. oRis is the platform on which research works from the Laboratoire d’Informatique Industrielle (Laboratory for Industrial Data Processing) (LI2) are conducted, which results in the development of many class packages. Among these are packages that coordinate actions according to Contract Net Protocol, agent distribution [Rodin 99], communication between agents by using KQML [N´ed´elec 00], the use of fuzzy cognitive maps [Maffre 01, Parentho¨en 01], the declaration of collective action plans by using an executable extension of Allen’s temporal logic [De Loor 00] and the definition of agent behaviors as trends [Favier 01]. The platform is also used in image processing [Ballet 97b], medical simulation [Ballet 98b] and in the simulation of vehicle manufacturer systems [Chevaillier 99b]. Other research teams also used oRis for their works: the CREC laboratory (Saint-Cyr Co¨etquidan) for simulating battle fields, conflicts and cyber warfare, the Naval School’s SIGMa team for constructing dynamic digital land models from probe points, the GREYC laboratory (University of Caen) for simulating computer networks, the SMAC and GRIC teams from the IRIT (Toulouse), and the UMR CNRS 6553 of Ecobiology (University of Rennes) for simulations in ethology.

79

Tools

4.4.3

The AR´ eVi Platform

The Laboratoire d’Informatique Industrielle (Laboratory for Industrial Data Processing) has developed an Atelier de R´ ealit´e Virtuelle (Virtual Reality Workshop) also known as AR´ eVi, which is a toolkit for creating distributed virtual reality applications. The first three versions of AR´ eVi were built around the Open Inventor graphics library and did not include the multi-agent approach. In version 4, oRis was used to develop AR´ eVi. The kernel of AR´ eVi is none other than oRis, and thus all of the possibilities that were described previously are available; it is extended by the C++ code offering functionalities that are specific to virtual reality [Duval 97, Reignier 98]. This platform offers a graphics rendering that is completely independent from the one provided by oRis. Graphics objects are loaded directly from files in VRML211 format (you can define animations and manage the levels of detail). Graphics elements, such as transparent or animated textures, light sources, lens flare (sunlight on a lens) and particle systems (water streams) are available. AR´ eVi also uses kinematics notions (speeds and linear and angular accelerations), which adds to the possibilities for expressing agent behaviors in a three-dimensional environment. In the sound area, AR´ eVi offers a space sound system and synthesis, as well as voice recognition functionalities. This platform manages various peripherals, such as a dataglove, a control lever, a steering wheel, localization sensors, a force-feedback arm and a stereoscopic head-mounted display, which widen the options for users to immerse themselves in multi-agent systems. It is important to note that the multiple tools offered by AR´ eVi for multi-sensory rendering of the virtual world are oRis objects similar to the others. It is the instances that can change the application and play an active role. It is therefore possible to adapt them to the considered subject, and to even give them an evolved behavior that meets the application’s specific needs. Thus, no matter what application subject you are dealing with, you can always think of a way to customize rendering tools in order to make it easier to use them in the chosen context. We can, for example, adapt a rendering window so that it automatically degrades the quality of its display when the image refresh frequency becomes too low. We can also associate a geographical area with a scene and give it a behavior that tends to integrate entities entering into this area and stops displaying entities that leave. In regards to the distribution of graphics entities on different machines, the functionalities involving network communication provided by oRis were used to establish communication between remote entities and a dead-reckoning [Rodin 00] mechanism. When an entity is created on a machine, copies that have a degraded kinematic behavior on the other machines are made, and their geometric and kinematic characteristics are updated when there is too much of a divergence between the real situation and the copies. Currently, AR´ eVi has been used within the framework of interactive prototyping of a stamping cell [Chevaillier00] and the development of a training platform for French Emergency Services [Querrec01a] (Figure 4.7 [Querrec 01b]). 11

80

VRML : Virtual Reality Modeling Language (www.vrml.org)

Conclusion

Figure 4.7: An AR´ eVi Application: Training Platform for French Emergency Services

4.5

Conclusion

Choosing a language is to choose a method of thinking [Tisseau 98a]. To prove this point, all one has to do is turn to the famous tale of a thousand and one nights: Ali Baba and the Forty Thieves (Ali Baba Metaphor, Figure 4.8). There are three possible scenarios: procedural programming, object-oriented programming or agent programming. 1. In procedural programming (Figure 4.8a), the cave’s door is passive data (Data Sesame) manipulated by the agent AliBaba: it is AliBaba that intends to open the door, and he is the one who knows how to open it (entrance procedure Open: I go to the door, I grab the doorknob, I turn the doorknob, I push the door). 2. In object-oriented programming (Figure 4.8b), the cave door is now an object (Object Sesame) that has delegated the know-how to open the door (example: an elevator door): it is still AliBaba who intends on opening the door, but now the door knows how to do it (Open method from the Door class). To open the door, AliBaba must send it the correct message12 (Open Sesame!: Sesame->Open()). Consequently, the door opens up in accordance with a master-slave schema. 3. In agent programming (Figure 4.8c), the cave door is an agent whose goal is to open when it detects a passer-by (example: an airport door equipped with cameras): the door has both the intention and the know-how (Agent Sesame). Whether or not AliBaba intends on going through the door or not, the door can open if it 12

Ali Baba already knew programming by objects! This analogy between sending a message in programming by objects and the Ali Baba formula (Open Sesame) is given in [Ferber 90].

81

Tools

execute { Object Sesame = new Data; Agent AliBaba = new Avatar; }

execute { Object Sesame = new Door; Agent AliBaba = new Avatar; }

execute { Agent Sesame = new Door; Agent AliBaba = new Avatar; }

AliBaba::execute { Open(Sesame); }

AliBaba::execute { Sesame->open(); }

a. Procedural Programming

b. Object-Oriented Programming

void Door::main(void) { if(view(anObject)) open(); else close(); } c. Agent Programming

Figure 4.8: Metaphor of Ali Baba and Programming Paradigms detects him: eventually it can even negotiate its opening. The user AliBaba is thus immersed into the universe of agents through an avatar, which can be detected by the door. Hence, choosing a language is important when building an application. Agent programming such as we develop, is best suited, by construction, for autonomizing the entities that make up our virtual universes. This is why we have chosen to make oRis a generic language for implementing multiagent systems that makes it possible to write programs based on the interaction of objects and agents while placed in a space-time environment and controlled by the user’s on-line actions. oRis is also a participatory multi-agent simulation environment, which makes it possible to control agent scheduling and interactive modeling through the language’s dynamicity. oRis is stable and operational13 and has already led to many applications such as AR´ eVi, the virtual reality workshop.

13

oRis can be used for free by downloading it from the Fabrice Harrouet’s homepage (www.enib.fr/~harrouet/oris.html). Documentation, examples, course material and the thesis dissertation [Harrouet 00] are also available on this page.

82

Chapter 5 Conclusion 5.1

Summary Virtual objects, just like the space in which they appear, are actors and agents. Equipped with memory, they have functions for processing information and an autonomy, which is regulated by their programs. A strange, intermediary artificial life continually moves through virtual worlds. Each entity, each object, each agent can be considered as an expert system that has its own behavior rules and applies or adapts them in response to changes in the environment, modifications of the rules and metarules that govern the virtual world. [Qu´ eau 93b]

For a dozen years, our works have taken place in the field of virtual reality, a new discipline of engineering sciences that involves the specification, design and creation of realistic and participatory universes. There is no way that we could completely cover all of the aspects of a synthesis discipline like virtual reality, so we tried to answer the questions of what concepts ?, what models ? and what tools ? for virtual reality. Today, we have answered these questions through a principle of autonomy, a multi-agent approach and a participatory multi-agent simulation environment.

What Concepts? The way that we think about virtual reality epistemologically has made the concept of autonomy the core of our research problem. Our approach is thus based on a principle of autonomy according to which the autonomization of digital models that make up virtual universes is vital to the reality of these universes. Autonomizing models means equipping them with sensory-motor means of communication and ways to coordinate perceptions and actions. This autonomy by conviction proved to be an autonomy by essence for organism models, an autonomy by necessity for mechanism models and an autonomy by ignorance for models of complex systems, which are characterized by a large variety of components, structures and interactions. The human user is represented within the virtual universe by an avatar, a model among models, through which he controls the coordination of perceptions and actions. The human user is connected to his avatar through language and adapted behavioral interfaces, which make it possible to have a triple mediation of senses, action and language. Consequently, by a sort of digital empathy, he feels as if he is actually in the virtual universe whose multi-sensory rendering is that of realistic, computer-generated images: 3D, sound, touch, kinesthetics, proprioceptive, animated in real-time, and shared on

83

Conclusion

computer networks. Thus, according to the Pinocchio metaphor, a human user, more autonomous because he is partially freed from controlling his models, will participate fully in these virtual realities: he will go from being a simple spectator to an actor and even a creator of these evolving virtual worlds.

What Models? Based on this principle of autonomy, we modeled these virtual universes through multi-agent systems that allow human users to participate in simulations. We have highlighted the auto-organizing properties of these autonomous entities-based systems through the significant and original example of blood coagulation. We have hence shown that participatory multi-agent simulations of virtual reality have given today’s biologists real virtual laboratories in which they can conduct a new type of experiment that is cost-effective and safe: in virtuo experiment. This in virtuo experiment adds to the traditional investigations which are in vivo and in vitro experiments, or in silico calculations. To go beyond the realm of reactive behaviors, we described emotional behaviors through fuzzy cognitive maps. We use these maps to both specify the behavior of an agent and control its movement. The agent can also auto-evaluate its behavior by simulating its map itself: this simulation within simulation is therefore an outline of the perception of the self, which is necessary for real autonomy. Implementing these fuzzy cognitive maps makes it possible to characterize credible agent roles in interactive fictions.

What Tools? To implement these models, we have developed a language — the oRis language — which favors autonomy by construction of the entities that make up our virtual universes. Thus, oRis is a generic implementation language for multi-agent systems, which makes it possible to write programs based on objects and agents in interaction, placed in a spacetime environment and controlled by the user’s on-line actions. oRis is also a participatory multi-agent simulation environment that makes it possible to control the schedule of agents and interactively model through the dynamicity of language. We have paid careful attention to the associated simulator in order to demonstrate that it was built so that the activation process for autonomous entities does not lead to bias in the simulation, for which the execution platform would be solely responsible. In this framework, execution error robustness proved to be a particularly sensitive point, especially since the simulator provides access to the entire on-line language. oRis is stable and operational: it has already lead to many applications among which our virtual reality workshop AR´ eVi.

84

Perspectives

5.2

Perspectives Microscope, telescope: these words evoke important scientific breakthroughs towards the infinitely small, and towards the infinitely large. [...] Today, we are confronted with another infinite: the infinitely complex. But this time there is no more instrument. Nothing but the mind, an intelligence and logic ill-equipped in front of the immense complexity of life and society. [...] Therefore, we need a new tool. [...] This tool, I’ll call it the macroscope (macro, big; and skopein, to observe). The macroscope is not a tool like all the others. It is a symbolic instrument, made from a collection of methods and techniques borrowed from a wide range of disciplines. Evidently, it is useless to try and find it in the laboratories or research centers. And yet, it is used by many in very different fields. Because, the macroscope can be considered as the symbol of a new way to see, understand, and act. [De Rosnay 75]

We do not claim to have completely answered the questions of what concepts ?, what models? and what tools? for virtual reality, which have fueled our research for the last ten years: We also intend to vigorously pursue our research in other directions.

What Tools? The participatory multi-agent simulation of virtual universes made up of autonomous entities will allow us to research a phenomenon that has become complex because modeling is so diverse. Therefore, virtual reality that uses numerous on-line models must be given engineering tools for these models. The oRis environment is only a middleware for virtual reality; it is definitely reliable and robust, but its abstraction level is still too low for users. So, we must incorporate tools that have the highest abstraction levels possible to make it easier to implement models, debug them, integrate them within existing universes and share them through computer networks. These tools will automatically generate the corresponding oRis code, which will be interpreted on-line by the simulator.

What Models? We have successfully tested the oRis platform in the case of reactive and emotional behaviors. We now need to tackle intentional, social and adaptive behaviors. We will approach intentional behavior modeling through the notion of objective, which is neither a constraint in the sense of programming by constraints, nor a goal of programming in logic. In these two approaches, the solution takes place according to a synchronous hypothesis of an insignificant execution time: the context is seen as unchanged during the resolution. However, in reality, constraints or goals can change during resolution: time goes by, the context changes, and this needs to be taken into account. The objective notion thus takes into account the irreversibility of the time that goes by. Social behaviors will be researched through the notion of the organization of multiagent systems. An organization defines the roles and gives them objectives; the assignment of roles is then negotiated between autonomous agents that decide to cooperate within this organization. By participating in multi-agent simulations, the human user can, according to a principle of substitution, take control of an entity and identify himself as that entity. The controlled entity can then learn behaviors that are better adapted to its environment from

85

Conclusion

the operator. It is this type of learning, by example or by imitation, that we will introduce for evolving behaviors modeling.

What Concepts? Virtual universes are open and heterogeneous; they are made up of atomic and composite entities that are mobile and distributed in space, and in a variable number in time. The components can be structured in organizations, imposed or emergent because of multiple interactions between components. Interactions between entities are themselves of a different nature and operate on different space and time scales. Unfortunately, today, there is no formalism capable of recognizing this complexity. Only virtual reality makes it possible to live this complexity. We will need to strengthen the relationships between virtual reality and the theories of complexity in order to make virtual reality a tool for investigating complexity, like the macroscope described in the quote at the beginning of this section. The human user becomes a part of these virtual worlds through an avatar. The structural identity between an agent and an avatar allows the user to take the place of an agent at any time by taking control of its decision-making module. And conversely, he can at any time give the control back to the agent whose place he took. An in virtuo autonomy test will be able to evaluate the quality of the substitution, which will be positive if a user interacting with an entity cannot tell if he is interacting with an agent or another user; the agents should be able to react as if they were interacting with another agent. This principle of substitution completes our principle of autonomy, and the consequences will have to be evaluated on an epistemological level as well as an ethical level.

86

Bibliography [Abelson 85]

Abelson H., Sussman G.J., Sussman J., Structure and interpretation of computer programs (1985)

[Agarwal 95]

Agarwal P., The Cell programming Language, Artificial Life 2(1):37-77, 1995

[Akeley 89]

Akeley K., The Silicon Graphics 4D/240 GTX super-workstation, IEEE Computer Graphics and Applications 9(4):239-246, 1989

[Ameisen 99]

Ameisen J.C., La sculpture du vivant : le suicide cellulaire ou la mort cr´eatrice, Editions du Seuil, Paris, 1999

[Ansaldi 85]

Ansaldi S., De Floriani L., Falcidieno B., Geometric modeling of solid objects by using a face adjacency graph representation, Proceedings Siggraph’85:131-139, 1985

[Arnaldi 89]

Arnaldi B., Dumont G., H´egron G., Dynamics and unification of animation control, The Visual Computer 4(5):22-31, 1989

[Arnaldi 94]

Arnaldi B., Mod`eles physiques pour l’animation, Document d’habilitation a` diriger des recherches, Universit´e de Rennes 1 (France), 1994

[Austin 62]

Austin J.L., How to do things with words, Oxford University Press, Oxford, 1962

[Avouris 92]

Avouris N.M., Gasser L. (´editeurs), Distributed Artificial Intelligence : theory and praxis, Kluwer, Dordrecht, 1992

[Axelrod 76]

Axelrod R., Structure of Decision, Princeton University Press, Princeton, 1976

[Bachelard 38]

Bachelard G., La formation de l’esprit scientifique (1938), Vrin, Paris, 1972

[Badler 93]

Badler N.I., Phillips C.B., Webber B.L., Simulating humans : computer graphics animation and control, Oxford University Press, New York, 1993

87

Bibliography

[Ballet 97a]

Ballet P., Rodin V., Tisseau J., A multiagent system to model an human secondary immune response, Proceedings IEEE SMC’97:357-362, 1997

[Ballet 97b]

Ballet P., Rodin V., Tisseau J., A multiagent system to detect concentric strias, Proceedings SPIE’97 3164:659-666, 1997

[Ballet 98a]

Ballet P., Pers J.O., Rodin V., Tisseau J., A multi-agent system to simulate an apoptosis model of B-CD5 cell, Proceedings IEEE SMC’98 2:1-5, 1998

[Ballet 98b]

Ballet P., Rodin V., Tisseau J., Int´erˆets mutuels des syst`emes multiagents et de l’immunologie, Proceedings CID’98:141-153, 1998

[Ballet 00a]

Ballet P., Abgrall J.F., Rodin V., Tisseau J., Simulation of thrombin generation during plasmatic coagulation and primary hemostasis, Proceedings IEEE SMC’00 1:131-136, 2000

[Ballet 00b]

Ballet P., Int´erˆets mutuels des syst`emes multi-agents et de l’immunologie. Application a` l’immunologie, a` l’h´ematologie et au traitement d’images, Th`ese de Doctorat, Universit´e de Bretagne Occidentale, Brest (France), 2000

[Bannon 91]

Bannon L.J., From human factors to human actors : the role of psychology and human-computer interaction studies in system-design, dans [Greenbaum 91]:25-44, 1991

[Baraquin 95]

Baraquin N., Baudart A., Dugu´e J., Laffitte J., Ribes F., Wilfert J., Dictionnaire de philosophie, Armand Colin, Paris, 1995

[Barr 84]

Barr A.H., Global and local deformations of solid primitives, Computer Graphics 18:21-30, 1984

[Barto 81]

Barto A.G., Sutton R.S., Landmark learning : an illustration of associative search, Biological Cybernetics 42:1-8, 1981

[Bates 92]

Bates J., Virtual reality, art, and entertainment, Presence 1(1):133-138, 1992

[Batter 71]

Batter J.J., Brooks F.P., GROPE-1: A computer display to the sense of feel, Proceedings IFIP’71:759-763, 1971

[Baumgart 75]

Baumgart B., A polyhedral representation for computer vision, Proceedings AFIPS’75:589-596, 1975

[Beer 90]

Beer R.D., Intelligence as adaptive behavior : an experiment in computational neuroethology, Academic Press, San Diego, 1990

[Benford 95]

Benford S., Bowers J., Fahlen L.E., Greehalgh C., Mariani J., Rodden T., Networked virtual reality and cooperative work, Presence 4(4):364386, 1995

88

Bibliography

[Bentley 79]

Bentley J.L., Ottmann T.A., Algorithms for reporting and counting geometric intersections, IEEE Transaction on Computers 28(9):643-647, 1979

[Berlekamp 82]

Berlekamp E.R., Conway J.H., Guy R.K., Winning ways for your mathematical plays, Academic Press, New York, 1982

[Berthoz 97]

Berthoz A., Le sens du mouvement, Editions Odile Jacob, Paris, 1997

[Bertin 78]

Bertin M., Faroux J-P., Renault J., Optique g´eom´etrique, Bordas, Paris, 1978

[B´ezier 77]

B´ezier P., Essai de d´efinition num´erique des courbes et des surfaces exp´erimentales, Th`ese d’Etat, Universit´e Paris 6 (France), 1977

[Blinn 76]

Blinn J.F., Newell M.E., Texture and reflection in computer generated images, Communications of the ACM 19(10):542-547, 1976

[Blinn 82]

Blinn J.F., A generalization of algebraic surface drawing, ACM Transactions on Graphics 1(3):235-256, 1982

[Bliss 70]

Bliss J.C., Katcher M.H., Rogers C.H., Shepard R.P., Optical-to-tactile image conversion for the blind, IEEE Transactions on Man-Machine Systems 11(1):58-65, 1970

[Boissier 99]

Boissier O., Guessoum Z., Occello M. (´editeurs), Plateformes de d´eveloppement de syst`emes multi-agents, Groupe ASA, Bulletin de l’AFIA n◦ 39, 1999

[Bouknight 70]

Bouknight W.J., A procedure for generation of three-dimensional halftoned computer graphics presentations, Communications of the ACM 13(9):527-536, 1970

[Boulic 90]

Boulic R., Magnenat-Thalmann N., Thalmann D., A global human walking model with real-time kinematic personification, Visual Computer 6(6):344-358, 1990

[Bresenham 65] Bresenham J., Algorithm for computer control of a digital plotter, IBM System Journal 4:25-30, 1965 [Brooks 86a]

Brooks F.P., Walkthrough : a dynamic graphics system for simulating virtual buildings, Proceedings Interactive 3D Graphics’86:9-21, 1986

[Brooks 90]

Brooks F.P., Ouh-young M., Batter J.J., Kilpatrick P.J., Project GROPE : haptic displays for scientific visualization, Computer Graphics, 24(4):177-185, 1990

[Brooks 86]

Brooks R.A., A robust layered control system for a mobile robot, IEEE Journal of Robotics and Automation 2(1):14-23, 1986

89

Bibliography

[Brooks 91]

Brooks R.A., Intelligence without representation, Artificial Intelligence 47:139-159, 1991

[Bryson 96]

Bryson S., Virtual reality in scientific visualization, Communications of the ACM 39(5):62-71, 1996

[Burdea 93]

Burdea G., Coiffet P., La r´ealit´e virtuelle, Herm`es, Paris, 1993

[Burtnyk 76]

Burtnyk N., Wein M., Interactive skeleton techniques for enhancing motion dynamics in key frame animation, Communications of the ACM 19(10):564-569, 1976

[Cadoz 84]

Cadoz C., Luciani A., Florens J.L., Responsive input devices and sound synthesis by simulation of instrumental mechanisms : the Cordis system, Computer Music Journal 8(3):60-73, 1984

[Cadoz 93]

Cadoz C., Luciani A., Florens J.L., CORDIS-ANIMA: a modeling and simulation system for sound and image synthesis - the general formalism, Computer Music Journal 17(1):19-29, 1993

[Cadoz 94a]

Cadoz C., Les r´ealit´es virtuelles, Collection DOMINOS, Flammarion, Paris, 1994

[Cadoz 94b]

Cadoz C., Le geste, canal de communication homme/machine : la communication instrumentale, Technique et Science Informatiques 13(1):3161, 1994

[Calvert 82]

Calvert T.W., Chapman J., Patla A.E., Aspects of the kinematic simulation of human movement, IEEE Computer Graphics and Applications 2:41-50, 1982

[Capin 97]

Capin T.K., Pandzic I.S., Noser H., Magnenat-Thalmann N., Thalmann D., Virtual human representation and communication in VLNET networked virtual environments, IEEE Computer Graphics and Applications 17(2):42-53, 1997

[Capin 99]

Capin T.K., Pandzic I.S., Magnenat-Thalmann N., Thalmann D., Avatars in networked virtual environments, John Wiley, New York, 1999

[Carlsson 93]

Carlsson C., Hagsand O., DIVE – A platform for multi-user virtual environments, Computers and Graphics 17(6):663-669, 1993

[Catmull 74]

Catmull E., A subdivision algorithm for computer display of curved surfaces, PhD Thesis, University of Utah (USA), 1974

[Celada 92]

Celada F., Seiden P.E., A computer model of cellular interactions in the immune system, Immunology Today 13(2):56-62, 1992

90

Bibliography

[Cerf 74]

Cerf V., Kahn R., A protocol for packet network interconnection, IEEE Transactions on Communication 22:637-648, 1974

[Chadwick 89]

Chadwick J.E., Haumann D.R., Parent R.E., Layered construction for deformable animated characters, Computer Graphics 23(3):243-252, 1989

[Chevaillier 99a] Chevaillier P., Harrouet F., DeLoor P., Application des r´eseaux de Petri a` la mod´elisation des syst`emes multi-agents de contrˆ ole, APII-JESA 33(4):413-437, 1999 [Chevaillier 99b] Chevaillier P., Harrouet F., Reignier P., Tisseau J., oRis : un environnement pour la simulation multi-agents des syst`emes manufacturiers de production, Proceedings MOSIN’99:225-230, 1999 [Chevaillier 00]

Chevaillier P., Harrouet F., Reignier P., Tisseau J., Virtual reality and multi-agent systems for manufacturing system interactive prototyping, International Journal of Design and Innovation Research 2(1):90-101, 2000

[Clark 76]

Clark J.H., Hierarchical geometric models for visible surface algorithm, Communications of the ACM 19(10):547-554, 1976

[Cliff 93]

Cliff D., Harvey I., Husbands P., Explorations in evolutionary robotics, Adaptive Behavior 2(1):73-110, 1993

[Cohen 85]

Cohen M.F., Greenberg D.P., The hemi-cube : a radiosity solution for complex environment, Computer Graphics 19(3):26-35, 1985

[Cohen-Tannoudji 95] Cohen-Tannoudji G. (´edit´e par), Virtualit´e et r´ealit´e dans les sciences, Editions Fronti`eres, Paris, 1995 [Cook 81]

Cook R.L., Torrance K.E., A reflectance model for computer graphics, Computer Graphics 15(3):307-316, 1981

[Cosman 81]

Cosman M.A., Schumacker R.A., System strategies to optimize CIG image content, Proceedings Image II’81:463-480, 1981

[De Giacomo 00] De Giacomo G., L´esperance Y., Levesque H.J., ConGolog : a concurrent programming language based on situation calculus, Artificial Intelligence 121(1-2):109-169, 2000 [Del Bimbo 95]

Del Bimbo A., Vicario E., Specification by example of virtual agents behavior, IEEE Transactions on Visualization and Computer Graphics 1(4):350-360, 1995

[De Loor 00]

De Loor P., Chevaillier P., Generation of agent interactions from temporal logic specifications, Proceedings IMACS’00:, 2000

91

Bibliography

[Demazeau 95]

Demazeau Y., From interactions to collective behaviour in agent-based systems, Proceedings European Conference on Cognitive Science’95:117132, 1995

[De Rosnay 75] De Rosnay J., Le macroscope : vers une vision globale, Editions du Seuil, Paris, 1975 [D’Espagnat 94] D’Espagnat B., Le r´eel voil´e, Fayard, Paris, 1994 [Dickerson 94]

Dickerson J.A., Kosko B., Virtual worlds as fuzzy cognitive maps, Presence 3(2):173-189, 1994

[Donikian 94]

Donikian S., Les mod`eles comportementaux pour la g´en´eration du mouvement d’objets dans une sc`ene, Revue internationale de CFAO et d’infographie 9(6):847-871, 1994

[Drogoul 93]

Drogoul A., De la simulation multi-agents a` la r´esolution collective de probl`emes. Une ´etude de l’´emergence de structures d’organisation dans les syst`emes multi-agents, Th`ese de Doctorat, Universit´e Paris 6 (France), 1993

[Duval 97]

Duval T., Morvan S., Reignier P., Harrouet F., Tisseau J., AR´eVi : une boˆıte a` outils 3D pour des applications coop´eratives, Revue Calculateurs Parall`eles 9(2):239-250, 1997

[Ellis 91]

Ellis S.R., Nature and origin of virtual environments : a bibliographic essay, Computing Systems in Engineering 2(4):321-347, 1991

[Endy 01]

Endy D., Brent R., Modelling cellular behaviour, Nature 409(18):391395, 2001

[Favier 01]

Favier P.A., De Loor P., Tisseau J., Programming agent with purposes : application to autonomous shooting in virtual environments, Lecture Notes in Computer Science 2197:40-43, 2001

[Ferber 90]

Ferber J., Conception et programmation par objets, Herm`es, Paris, 1990

[Ferber 95]

Ferber J., Les syst`emes multi-agents : vers une intelligence collective, InterEditions, Paris, 1995

[Ferber 97]

Ferber J., Les syst`emes multi-agents : un aper¸cu g´en´eral, Technique et Science Informatique 16(8):979-1012, 1997

[Fikes 71]

Fikes R.E., Nilsson N., STRIPS : a new approach to the application of theorem proving to problem solving, Artificial Intelligence 5(2):189-208, 1971

[Fisher 86]

Fisher S.S., MacGreevy M., Humphries J., Robinett W., Virtual environment display system, Proceedings Interactive 3D Graphics’86:77-87, 1986

92

Bibliography

[Fisher 94]

Fisher M., A survey of Concurrent Metatem : the language and its applications, Lecture Notes in Artificial Intelligence 827:480-505, 1994

[Fuchs 80]

Fuchs H., Kedem Z., Naylor B., On visible surface generation by a priori tree structures, Computer Graphics 14(3):124-133, 1980

[Fuchs 96]

Fuchs P., Les interfaces de la r´ealit´e virtuelle, AJIIMD, Presses de l’Ecole des Mines de Paris, 1996

[Fujimoto 98]

Fujimoto R.M., Time management in the High Level Architecture, Simulation 71(6):388-400, 1998

[Gasser 92]

Gasser L., Briot J.P., Object-based concurrent programming and distributed artificial intelligence, dans [Avouris 92]:81-107, 1992

[Gaussier 98]

Gaussier P., Moga S., Quoy M., Banquet J.P., From perception-action loops to imitation process : a bottom-up approach of learning by imitation, Applied Artificial Intelligence 12:701-727, 1998

[Georgeff 87]

Georgeff M.P., Lansky A.L., Reactive reasonning and planning, Proceedings AAAI’87:677-682, 1987

[Gerschel 95]

Gerschel A., Liaisons intermol´eculaires, InterEditions/CNRS Editions, Paris, 1995

[Gibson 84]

Gibson W., Neuromancer, Ace Books, New York, 1984

[Ginsberg 83]

Ginsberg C.M., Maxwell D., Graphical marionette, Proceedings ACM SIGGRAPH/SIGART Workshop on Motion’83:172-179, 1983

[Girard 85]

Girard M., Maciejewski A.A., Computational modeling for the computer animation of legged figures, Computer Graphics 19(3):39-51, 1985

[G¨obel 93]

G¨obel M., Neugebauer J., The virtual reality demonstration centre, Computer and Graphics 17(6):627-631, 1993

[Gouraud 71]

Gouraud H., Continuous shading of curved surface, IEEE Transactions on Computers 20(6):623-629, 1971

[Gourret 89]

Gourret J.P., Magnenat-Thalmann N., Thalmann D., The use of finite element theory for simulating object and human skin deformations and contacts, Proceedings Eurographics’89:477-487, 1989

[Grand 98]

Grand S., Cli D., Creatures : entertainment software agents with articial life, Autonomous Agents and Multi-Agent Systems 1(1):39-57, 1998

[Granger 86]

Granger G.G., Pour une ´epist´emologie du travail scientifique, dans [Hamburger 86]:111-129, 1986

93

Bibliography

[Granger 95]

Granger G.G., Le probable, le possible et le virtuel, Editions Odile Jacob, Paris, 1995

[Greenbaum 91] Greenbaum J., Kyng M. (´editeurs), Design at work : cooperative design of computer systems, Lawrence Erlbaum Associates, Hillsdale, 1991 [Guessoum 99]

Guessoum Z., Briot J.P., From active objects to autonomous agents, IEEE Concurrency 7(3):68-76, 1999

[Guillot 00]

Guillot A., Meyer J.A., From SAB94 to SAB2000 : What’s New, Animat?, Proceedings From Animals to Animats’00 6:1-10, 2000

[Hamburger 86] Hamburger J. (sous la direction de), La philosophie des sciences aujourd’hui, Acad´emie des Sciences, Gauthier-Villars, Paris, 1986 [Hamburger 95] Hamburger J., Concepts de r´ealit´e, dans “ Encyclopædia Universalis ” 19:594-597, Encyclopædia Universalis, Paris, 1995 [Harrouet 00]

Harrouet F., oRis : s’immerger par le langage pour le prototypage d’univers virtuels a` base d’entit´es autonomes, Th`ese de Doctorat, Universit´e de Bretagne Occidentale, Brest (France), 2000

[Hayes-Roth 96] Hayes-Roth B., van Gent R., Story-making with improvisational puppets and actors, Technical Report KSL-96-05, Knowledge Systems Laboratory, Stanford University, 1996 [Hemker 65]

Hemker H.C., Hemker P.W., Loeliger A., Kinetic aspects of the interaction of blood clotting enzymes, Thrombosis et diathesis haemorrhagica 13:155-175, 1965

[Herrmann 98]

Herrmann H.J., Luding S., Review article : Modeling granular media on the computer, Continuum Mechanics and Thermodynamics 10(4):189231, 1998

[Hertel 83]

Hertel S., Mehlhorn K., Fast triangulation of simple polygons, Lecture Notes in Computer Science 158:207-218, 1983

[Hewitt 73]

Hewitt C., Bishop P., and Steiger R., A universal modular actor formalism for artificial intelligence, Proceedings IJCAI’73:235-245, 1973

[Hewitt 77]

Hewitt C., Viewing control structures as patterns of message passing, Artificial Intelligence 8(3):323-364, 1977

[Hofstadter 79]

Hofstadter D., G¨ odel, Escher, Bach : an eternal golden braid (1979), traduction fran¸caise : InterEditions, Paris, 1985

[Holloway 92]

Holloway R., Fuchs H., Robinett W., Virtual-worlds research at the University of North Carolina at Chapel Hill, Computer Graphics’92, 15:1-10, 1992

94

Bibliography

[Hopkins 96]

Hopkins J.C., Leipold R.J., On the dangers of adjusting the parameter values of mechanism-based mathematical models, Journal of Theoretical Biology 183:417-427, 1996

[IEEE 93]

IEEE 1278, Standard for Information Technology : protocols for Distributed Interactive Simulation applications, IEEE Computer Society Press, 1993

[Iglesias 98]

Iglesias C.A., Garijo M., Gonzalez J.C., A survey of agent-oriented methodologies, Proceedings ATAL’98:317-330, 1998

[Isaacs 87]

Isaacs P.M., Cohen M.F., Controlling dynamic simulation with kinematic constraints, behavior functions and inverse dynamics, Computer Graphics 21(4):215-224, 1987

[Jamin 96]

Jamin C., Lydyard P.M., Le Corre R., Youinou P., Anti-CD5 extends the proliferative response of human CD5+B cells activated with anti-IgM and IL2, European Journal of Immunology 26:57-62, 1996

[Jones 71]

Jones C.B., A new approach to the hidden line problem, Computer Journal 14(3):232-237, 1971

[Jones 94]

Jones K.C., Mann K.G., A model for the Tissue Factor Pathway to Thrombin: 2. A Mathematical Simulation, Journal of Biological Chemistry 269(37):23367-23373, 1994

[Kallmann 99]

Kallmann M., Thalmann D., A behavioral interface to simulate agent-object interactions in real time, Proceedings Computer Animation’99:138146, 1999

[Klassen 87]

Klassen R., Modelling the effect of the atmosphere on light, ACM Transactions on Graphics 6(3):215-237, 1987

[Klesen 00]

Klesen M., Szatkowski J., Lehmann N., The black sheep : interactive improvisation in a 3D virtual world, Proceedings I3’00:77-80, 2000

[Kodjabachian 98] Kodjabachian J.,Meyer J.A., Evolution and development of neural controllers for locomotion, gradient-following, and obstacle-avoidance in artificial insects, IEEE Transactions on Neural Networks 9:796-812, 1998 [Kochanek 84]

Kochanek D.H., Bartels R.H., Interpolating splines with local tension, continuity, and bias control, Computer Graphics 18(3):124-132, 1984

[Kolb 95]

Kolb C., Mitchell D., Hanrahan P., A realistic camera model for computer graphics, Computer Graphics 29(3):317-324, 1995

[Kosko 86]

Kosko B., Fuzzy Cognitive Maps, International Journal Man-Machine Studies 24:65-75, 1986

95

Bibliography

[Kosko 92]

Kosko B., Neural networks and fuzzy systems : a dynamical systems approach to machine intelligence, Prentice-Hall, Engelwood Cliffs, 1992

[Kripke 63]

Kripke S., Semantic analysis of modal logic, I : normal propositional calculi, Zeitschrift f¨ ur mathematische Logik und Grundlagen der Mathematik 9:67-96, 1963

[Krueger 77]

Krueger M.W., Responsive environments, Proceedings NCC’77:375-385, 1977

[Krueger 83]

Krueger M.W., Artificial Reality, Addison-Wesley, New York, 1983

[Langton 86]

Langton C.G., Studying artificial life with cellular automata, Physica D22:120-149, 1986

[Lavery 98]

Lavery R., Simulation mol´eculaire, macromol´ecules biologiques et syst`emes complexes, Actes Entretiens de la Physique 3:1-18, 1998

[Le Moigne 77]

Le Moigne J-L., La th´eorie du syst`eme g´en´eral : th´eorie de la mod´elisation, Presses Universitaires de France, Paris, 1977

[Lestienne 95]

Lestienne F., Equilibration, dans Encyclopædia Universalis 8:597-601, Encyclopædia Universalis, Paris, 1995

[L´evy 95]

L´evy P., Qu’est-ce que le virtuel ?, Editions La D´ecouverte, Paris, 1995

[L´evy-Leblond 96] L´evy-Leblond J-M., Aux contraires : l’exercice de la pens´ee et la pratique de la science, Gallimard, Paris, 1996 [Lienhardt 89]

Lienhardt P., Subdivisions of surfaces and generalized maps, Proceedings Eurographics’89:439-452, 1989

[Luciani 85]

Luciani A., Un outil informatique de cr´eation d’images anim´ees : mod`eles d’objets, langage, contrˆ ole gestuel en temps r´eel. Le syst`eme ANIMA, Th`ese de Doctorat, INPG Grenoble (France), 1985

[Luciani 00]

Luciani A., From granular avalanches to fluid turbulences through oozing pastes : a mesoscopic physically-based particle model, Proceedings Graphicon’00 10:282-289, 2000

[Maes 95]

Maes P., Artificial life meets entertainment : interacting with lifelike autonomous agents, Communications of the ACM 38(11):108-114, 1995

[Maffre 01]

Maffre E., Tisseau J., Parentho¨en M., Virtual agents’ self-perception in storytelling, Lecture Notes in Computer Science 2197:155-158, 2001

[Magnenat-Thalmann 88] Magnenat-Thalmann N., Laperriere R., Thalmann D., JointDependent Local Deformations for Hand Animation and Object Grasping, Proceedings Graphics Interface’88:26-33, 1988

96

Bibliography

[Magnenat-Thalmann 91] Magnenat-Thalmann N., Thalmann D., Complex Models for Animating Synthetic Actors, IEEE Computer Graphics and Applications 11(5):32-44, 1991 [Mendes 93]

Mendes P., GEPASI: a software package for modelling the dynamics, steady states and control of biochemical and other systems, Computer Applications in the Biosciences 9(5):563-571, 1993

[Meyer 91]

Meyer J.A., Guillot A., Simulation of adaptive behavior in animats : review and prospect, Proceedings From Animals to Animats’91 1:2-14, 1991

[Meyer 94]

Meyer J.A., Guillot A., From SAB90 to SAB94 : four years of animat research, Proceedings From Animals to Animats’94 3:2-11, 1994

[Miller 95]

Miller D.C., Thorpe J.A., SIMNET : the advent of simulator networking, Proceedings of the IEEE 83(8):1114-1123, 1995

[Minsky 00]

Minsky M., The emotion machine : from pain to suffering, Proceedings IUI’00:187-193, 2000

[Moore 88]

Moore M., Wilhelms J., Collision detection and response for computer animation, Computer Graphics 22(4):289-298, 1988

[Morin 77]

Morin E., La m´ethode, Tome 1 : la nature de la nature, Editions du Seuil, Paris, 1977

[Morin 86]

Morin E., La m´ethode, Tome 3 : la connaissance de la connaissance, Editions du Seuil, Paris, 1986

[Morton-Firth 98] Morton-Firth C.J., Bray D., Predicting temporal fluctuations in an intracellular signalling pathway, Journal of Theoretical Biology 192:117128, 1998 [Mounts 97]

Mounts W.M., Liebman M.N., Qualitative modeling of normal blood coagulation and its pathological states using stochastic activity networks, International Journal of Biological Macromolecules 20:265-281, 1997

[N´ed´elec 00]

N´ed´elec A., Reignier P., Rodin V., Collaborative prototyping in distributed virtual reality using an agent communication language, Proceedings IEEE SMC’00 2:1007-1012, 2000

[Noma 00]

Noma T., Zhao L., Badler N., Design of a Virtual Human Presenter, Journal of Computer Graphics and Applications 20(4):79-85, 2000

[No¨el 98]

No¨el D., Virtuel possible, actuel probable (Combien sont les mo(n)des ?), Proceedings GTRV’98:114-120, 1998

97

Bibliography

[Noser 95]

Noser H., Thalmann D., Synthetic vision and audition for digital actors, Proceedings Eurographics’95:325-336, 1995.

[Nwana 96]

Nwana H.S., Software agents : an overview, Knowledge Engineering Review 11(3):205-244, 1996

[Nwana 99]

Nwana H.S., Ndumu D.T., Lee L.C., Collis J.C., ZEUS : a toolkit for building distributed multiagent systems, Applied Artificial Intelligence 13(1-2):129-185, 1999

[Omn`es 95]

Omn`es R., L’objet quantique et la r´ealit´e, dans [Cohen-Tannoudji 95]: 139–168, 1995

[Parentho¨en 01] Parentho¨en M., Tisseau J., Reignier P., Dory F., Agent’s perception and Charactors in virtual worlds : put Fuzzy Cognitive Maps to work, Proceedings VRIC’01:11-18, 2001 [Parunak 99]

Parunak H.V.D., Industrial and practical applications of DAI, dans [Weiss 99]:377-421, 1999

[Perlin 95]

Perlin K., Goldberg A., Improv : a system for scripting interactive actors in virtual worlds, Computer Graphics 29(3):1-11, 1995

[Perrin 01]

Perrin J. (sous la direction de), Conception entre science et art : regards multiples sur la conception, Presses Polytechniques et Universitaires Romandes, Lausanne, 2001

[Phong 75]

Phong B.T., Illumination for computer generated pictures, Communications of the ACM 18(8):311-317, 1975

[Pitteway 80]

Pitteway M., Watkinson D., Bresenham’s algorithm with grey scale, Communications of the ACM 23(11):625-626, 1980

[Platt 81]

Platt S., Badler N., Animating facial expressions, Computer Graphics 15(3):245-252, 1981

[Pope 87]

Pope A.R., The SIMNET Network and Protocols, Report 6369, BBN Systems and Technologies, 1987

[Qu´eau 93a]

Qu´eau Ph., Televirtuality: the merging of telecommunications and virtual reality, Computers and Graphics 17(6):691-693, 1993

[Qu´eau 93b]

Qu´eau Ph., Le virtuel, vertus et vertiges, Collection Milieux, Champ vallon, Seyssel, 1993

[Qu´eau 95]

Qu´eau Ph., Le virtuel : un ´etat du r´eel, dans [Cohen-Tannoudji 95]:6193, 1995

98

Bibliography

[Querrec 01a]

Querrec R., Reignier P., Chevaillier P., Humans and autonomous agents interactions in a virtual environment for fire fighting training, Proceedings VRIC’01:57-64, 2001

[Querrec 01b]

Querrec R., Chevaillier P., Virtual storytelling for training : an application to fire fighting in industrial environment, Lecture Notes in Computer Science 2197:201-204, 2001

[Raab 79]

Raab F.H., Blood E.B., Steiner T.O., Jones R.J., Magnetic position and orientation tracking system, IEEE Transaction on Aerospace and Electronic Systems 15(5):709-718, 1979

[Reeves 83]

Reeves W.T., Particle systems : a technic for modelling a class of fuzzy objects, Computer Graphics 17(3):359-376, 1983

[Renault 90]

Renault O., Magnenat-Thalmann N., Thalmann D., A vision-based approach to behavioural animation, Journal of Visualization and Computer Animation 1(1):18-21, 1990

[Requicha 80]

Requicha A.A.G., Representations for rigid solids: theory, methods and systems, ACM Computing Surveys 12(4):437-464, 1980

[Reignier 98]

Reignier P., Harrouet F., Morvan S., Tisseau J., Duval T., AR´eVi : a virtual reality multiagent platform, Lecture Notes in Artificial Intelligence 1434:229-240, 1998

[Reynolds 87]

Reynolds C.W., Flocks, Herds, and Schools: A Distributed Behavioral Model, Computer Graphics 21(4):25-34, 1987

[Rickel 99]

Rickel J., Johnson W.L., Animated agents for procedural training in virtual reality : perception, cognition, and motor control, Applied Artificial Intelligence 13:343-382, 1999

[Rodin 99]

Rodin V., N´ed´elec A., oRis : an agent communication language for distributed virtual environment, Proceedings IEEE ROMAN’99:41-46, 1999

[Rodin 00]

Rodin V., N´ed´elec A., A multiagent architecture for distributed virtual environments, Proceedings IEEE SMC’00 2:955-960, 2000

[Rosenblum 91] Rosenblum R.E., Carlson W.E., Tripp E., Simulating the structure and dynamics of human hair : modelling, rendering and animation, Journal of Visualization and Computer Animation 2(4):141-148, 1991 [Rumbaugh 99] Rumbaugh J., Jacobson I., Booch G., The Unified Modelling language Reference Manual, Addison-Wesley, New-York, 1999 [Samet 84]

Samet H., The quadtree and related hierarchical data structures, ACM Computing Surveys 6(2):187-260, 1984

99

Bibliography

[Schaff 97]

Schaff J., Fink C., Slepchenko B., Carson J., Loew L., A general computational framework for modeling cellular structure and function, Biophysical Journal 73:1135-1146, 1997

[Schuler 93]

Schuler D., Namioka A. (´editeurs), Participatory Design : Principles and Practices, Lawrence Erlbaum Associates, Hillsdale, 1993

[Schultz 96]

Schultz C., Grefenstette J.J., Adams W.L., RoboShepherd : learning a complex behavior, Proceedings RoboLearn’96:105-113, 1996

[Schumacker 69] Schumacker R., Brand B., Gilliland M., Sharp W., Study for applying computer-generated images to visual simulation, Technical Report AFHRL-TR-69-14, U.S. Air Force Human Resources Lab., 1969 [Sederberg 86]

Sederberg T.W., Parry S.R., Free-form deformation of solid geometric models, Computer Graphics 20:151-160, 1986

[Sheng 92]

Sheng X., Hirsch B.E., Triangulation of trimmed surfaces in parametric space, Computer Aided Design 24(8):437-444, 1992

[Sheridan 87]

Sheridan T.B., Teleoperation, telepresence, and telerobotics : research needs, Proceedings Human Factors in Automated and Robotic Space Systems’87:279-291, 1987

[Shoham 93]

Shoham Y., Agent-oriented programming, Artificial Intelligence 60(1): 51-92, 1993

[Sillion 89]

Sillion F., Puech C., A general two-pass method integrating specular and diffuse reflection, Computer Graphics 23(3):335-344, 1989

[Sims 94]

Sims K., Evolving 3D morphology and behavior by competition, Artificial Life 4:28-39, 1994

[Singh 94]

Singh G., Serra L., Ping W., Hern N., BrickNet: a software toolkit for network-based virtual worlds, Presence 3(1):19-34, 1994

[Smith 80]

Smith R.G., The Contract Net protocol : high level communication and control in a distributed problem solver, IEEE Transactions on Computers 29(12):1104-1113, 1980

[Smith 89]

Smith T.J., Smith K.U., The human factors of workstation telepresence, Proceedings Space Operations Automation and Robotics’89:235250, 1989

[Smith 97]

Smith D.J., Forrest S., Ackley D.H., Perelson A.S., Using lazy evaluation to simulate realistic-size repertories in models of the immune system, Bulletin of Mathematical Biology 60:647-658, 1997

100

Bibliography

[Sorensen 99]

Sorensen E.N., Burgreen G.W., Wagner W.R., Antaki J.F., Computational simulation of platelet deposition and activation: I. Model development and properties, Annals of Biomedical Engineering 27(4):436-448, 1999

[Sowa 91]

Sowa J.F., Principles of semantic network : exploration in the representation of knowledge, Morgan Kaufmann Publisher Inc., San Mateo, 1991

[Steketee 85]

Steketee S.N., Badler N.I., Parametric keyframe interpolation incorporating kinetic adjustment and phrasing, Computer Graphics 19(3):255262, 1985

[Stewart 89]

Stewart J., Varela F.J., Exploring the connectivity of the immune network, Immunology Review 110:37-61, 1989

[Sutherland 63] Sutherland I.E., Sketchpad, a man-machine graphical communication system, Proceedings AFIPS’63:329-346, 1963 [Sutherland 68] Sutherland I.E., A head-mounted three-dimensional display, Proceedings AFIPS’68, 33(1):757-764, 1968 [Sutherland 74] Sutherland I.E., Sproull R., Schumaker R., A characterization of ten hidden-surface algorithms, Computing Surveys 6(1):1-55, 1974 [Tachi 89]

Tachi S., Arai H., Maeda T., Robotic tele-existence, Proceedings NASA Space Telerobotics’89 3:171-180, 1989

[Terzopoulos 87] Terzopoulos D., Platt J., Barr A., Fleischer K., Elastically deformable models, Computer Graphics 21(4):205-214, 1987 [Terzopoulos 88] Terzopoulos D., Fleischer K., Modeling inelastic deformation: viscoelasticity, plasticity, fracture, Computer Graphics 22(4):269-278, 1988 [Terzopoulos 94] Terzopoulos D., Tu X., Grzeszczuk R., Artificial fishes: autonomous locomotion, perception, behavior, and learning in a simulated physical world, Artificial Life 1(4):327-351, 1994 [Thalmann 96]

Thalmann D., A new generation of synthetic actors : the interactive perceptive actors, Proceedings Pacific Graphics’96:200-219, 1996

[Thalmann 99]

Thalmann D., Noser H., Towards autonomous, perceptive and intelligent virtual actors, Lecture Notes in Artificial Intelligence 1600:457-472, 1999

[Thomas 86]

Thomas S.N., Dispersive refraction in ray tracing, The Visual Computer 2(1):3-8, 1986

[Tisseau 98a]

Tisseau J., Chevaillier P., Harrouet F., N´ed´elec A., Des proc´edures aux agents : application en oRis, Rapport interne du LI2, ENIB, 1998

101

Bibliography

[Tisseau 98b]

Tisseau J., Reignier P., Harrouet F., Exploitation de mod`eles et R´ealit´e Virtuelle, Actes des 6`emes journ´ees du groupe de travail Animation et Simulation 6:13-22, 1998

[Tolman 48]

Tolman E.C., Cognitive maps in rats and men, Psychological Review 42(55):189-208, 1948

[Tomita 99]

Tomita M., Hashimoto K., Takahashi K., Shimizu T.S., Matsuzaki Y., Miyoshi F., Saito K., Tanida S., Yugi K., Venter J.C., Hutchison C.A., E-CELL: software environment for whole-cell simulation, Bioinformatics 15:72-84, 1999

[Turing 50]

Turing A., Computing machinery and intelligence, Mind 59:433-460, 1950

[Vacherand-Revel 01] Vacherand-Revel J., Tarpin-Bernard F., David B., Des mod`eles de l’interaction a` la conception participative des logiciels interactifs, dans [Perrin 01]:239-256, 2001 [Vaughan 00]

Vaughan R., Sumpter N., Henderson J., Frost A., Cameron S., Experiments in automatic flock control, Journal of Robotics and Autonomous Systems 31:109-117, 2000

[Vertut 85]

Vertut J., Coiffet P., Les robots, Tome 3 : t´el´eop´eration, Herm`es, Paris, 1985

[Von Neumann 66] Von Neumann J., Theory of self-reproducing automata, University Of Illinois Press, Chicago, 1966 [Weil 86]

Weil J., The synthesis of cloth objects, Computer Graphics 20(4):49-54, 1986

[Weiss 99]

Weiss G. (´editeur), Multiagent systems : a modern approach to distributed artificial intelligence, MIT Press, Cambridge, 1999

[Wenzel 88]

Wenzel E.M., Wightman F.L., Foster S.H., Development of a three dimensional auditory display system, Proceedings CHI’88:557-564, 1988

[Whitted 80]

Whitted T., An improved illumination model for shaded display, Communications of the ACM 23(6):343-349, 1980

[Williams 78]

Williams L., Casting curved shadows on curved surfaces, Computer Graphics 12(3):270-274, 1978

[Wilson 85]

Wilson S.W., Knowledge growth in an artificial animal, Proceedings Genetic Algorithms and their Applications’85:16-23, 1985

[Wolberg 90]

Wolberg G., Digital image warping, IEEE Computer Society Press, 1990

102

Bibliography

[Wolfram 83]

Wolfram S., Statistical mechanics of cellular automata, Review of Modern Physics 55(3):601-644, 1983

[Wooldridge 95] Wooldridge M., Jennings N.R., Intelligent agents : theory and practice, The Knowledge Engineering Review 10(2):115-152, 1995 [Wooldridge 01] Wooldridge M., Ciancarini P., Agent-oriented software engineering : the state of the art, a` paraˆıtre dans Handbook of Software Engineering and Knowledge Engineering, ISBN:981-02-4514-9, World Scientific Publishing Co., River Edge, 2001 [Zeltzer 82]

Zeltzer D., Motor control techniques for figure animation, IEEE Computer Graphics and Applications 2(9):53-59, 1982

[Zelter 92]

Zelter D., Autonomy, interaction and presence, Presence 1(1):127-132, 1992

[Zimmerman 87] Zimmerman T., Lanier J., Blanchard C., Bryson S., A hand gesture interface device, Proceedings CHI’87:189-192, 1987 [Zinkiewicz 71]

Zinkiewicz O.C., The finite element method in engineering science, Mc Graw Hill, New York, 1971

103