Virtual Reality-Based Simulation of Endoscopic ... - Semantic Scholar

G. Sze´kely Ch. Brechbu ¨ hler J. Dual R. Enzler J. Hug R. Hutter N. Ironmonger M. Kauer V. Meier P. Niederer A. Rhomberg P. Schmid G. Schweitzer M. Thaler V. Vuskovic G. Tro ¨ ster Swiss Federal Institute of Technology, ETH Zentrum, CH-8092 Zu¨rich, Switzerland U. Haller Department of Gynecology University Hospital, CH-8091 Zurich M. Bajka Clinic of Gynecology and Obstetrics Hospital Uster, CH-8610 Uster

Presence, Vol. 9, No. 3, June 2000, 310–333

r 2000 by the Massachusetts Institute of Technology 310

Virtual Reality-Based Simulation of Endoscopic Surgery

Abstract Virtual reality (VR)-based surgical simulator systems offer a very elegant approach to enriching and enhancing traditional training in endoscopic surgery. However, while a number of VR simulator systems have been proposed and realized in the past few years, most of these systems are far from being able to provide a reasonably realistic surgical environment. We explore the current limits for realism and the approaches to reaching and surpassing those limits by describing and analyzing the most important components of VR-based endoscopic simulators. The feasibility of the proposed techniques is demonstrated on a modular prototype system that implements the basic algorithms for VR training in gynaecologic laparoscopy.

1

Introduction

Endoscopic operations have recently become a very popular technique for the diagnosis and treatment of many kinds of human diseases and injuries. The basic aim of endoscopic surgery is to minimize the damage of the surrounding healthy tissue that is normally caused by surgery on internal organs. By employing minimally invasive surgical techniques, the relatively large cuts in open surgery can be replaced by small perforation holes that serve as entry points for optical and surgical instruments through the trocar hall inserted into the abdominal wall for laparoscopic interventions. The small spatial extent of the tissue injury and the careful selection of the entry points result in a major improvement in a patient’s recovery after an operation. The price of achieving these advantages is paid by the surgeon, who loses direct contact with the operation site. The necessary visual information is mediated and limited by a specialized camera (the endoscope) and is presented on a screen, which is counterintuitive to normal hand-eye coordination. While preliminary systems that use stereooptics are already available, today’s surgery is usually performed under monoscopic conditions. Due to the geometrical constraints posed by the external control of the surgical instruments through the trocar hull, not only is the tactile feedback reduced, but the surgeon also loses much of the manipulative freedom that is usually available in open surgery (figure 1). This way even simple tasks such as cutting, suturing, or knot tying pose unusual challenges for the inexperienced surgeon. Performing operations under these conditions demands very specific skills that can be gained only with extensive training. The basic visiospatial and manipulative skills can be learned by using inexpensive, traditional training devices (as the Pelvi-trainer (see Storz, GmbH) or the POP unit (see Optimist, Handelsgesellschaft mbH)), which usually involve synthetic or animal tissue

PRE S E N CE : V O L U ME 9 , N UM B ER 3

Sze´kely et al.

311

tomical models in different applications in the field of medical education and simulation. Virtual reality also holds an enormous potential for supporting medical training using surgical simulator systems. The development of full-scale surgical simulators depends, however, on a much broader technological basis than just the 3-D visualization of organs. Subsequent sections of this paper will analyze in detail the current state of development of the following necessary and basic technological components:

Figure 1. Photograph of a laparoscopic gynecological intervention.

mounted in rigid cases. These cheap and effective units allow the trainee to learn how to navigate under monoscopic visual feedback, as well as to perform basic manipulative components of an intervention. In this way the surgeon develops competence in completing a particular task, but, because the real-life effect is lost, he or she obtains only a limited training in dexterity and surgical problem solving. Most important is the lack of realistic tissue reactivity; the trainee is thus unable to learn the techniques of hemostasis (as the training tissue does not bleed realistically) and cannot experience the complexities of abnormal anatomy or pathologic situations. While experiments on animals are sometimes used for testing new surgical techniques, practical as well as ethical reasons strongly restrict their use for everyday surgical training. VR technology has already made a significant impact on the art of medical education and training. Early applications were exclusively based on simulated 3-D visualization of the anatomy. Comprehensive volumetric visualization systems, like the VOXEL-MAN program (Ho¨hne et al., 1995) allow the interactive exploration of 3-D virtual anatomical models coupled with a wide range of auxiliary information as radiology, pathology, or functional behavior. The availability of high-resolution cryosectional data covering the whole human body, as the Visible Human data set of the U.S. National Library of Medicine (http://www.nlm.nih.gov/research/ visible/visible_human.html), strongly accelerated the widespread development and use of virtual 3-D ana-

x x x x

construction of the underlying anatomical model visualization of the operation site modeling organ behavior force feedback for mediating haptic sensation

Increasing computational power, as well as current achievements in the field of interactive computer graphics and virtual reality, have already led to the rapid development of more- or less-sophisticated surgical simulators during the past years. These systems offer an appealing way to provide adequate training without any risks of direct patient involvement. Although some attempts for the simulation of open surgical procedures have already been made (Reinig et al., 1996a; Ross et al., 1999; Suzuki et al., 1998; Bro-Nielsen et al., 1998; MusculoGraphic’s Limb Trauma Simulator), the major problem of providing realistic interaction with organs in open surgery has prevented the widespread development and use of such systems up to now. Interventions, which do not require direct freehand contact with the operation site, are much more suitable for the currently available devices for human-machine interaction. Besides microsurgical procedures especially for eye surgery (Sagar et al., 1994; Schill et al., 1999), interventional radiology (Hahn et al., 1998) or epidural anesthesia (Stredney et al., 1996), most successful applications have been developed on the field of endoscopic diagnosis and surgery. A wide range of VR simulator systems have been proposed and implemented in this area during the past few years. Some are restricted topurely diagnostic endoscopic investigations (Alyassin & Lorensen, 1998; Vining, 1996; Satava, 1996) while others, for example, allow the training of surgical procedures for laparoscopic (Cover, Ezquerra, & O’Brian, 1993;

312


Ku¨hnapfel et al., 1995; Cotin et al., 1996; Baur, Guzzoni, & Georg, 1998; Downes et al., 1998) or arthroscopic (Ziegler et al., 1995; Gibson et al., 1997) interventions. While most of the developed systems require the use of relatively powerful workstations or even graphics supercomputers, the possibility of using low-end personal computers for surgical simulation has also been investigated (Alyassin & Lorensen, 1998; Tseng et al., 1998; Daane, Constantinou, & Hesselroth, 1995). During the past years, not only have numerous academic research projects been reported, but first industrial products have been successfully launched (MusculoGraphics, HT Medical, Virtual Presence, Voxar, and Boston Dynamics). The basic advantage of VR-based endoscopic simulators is their potential for providing a realistic and configurable training environment that bridges the gap between basic training and performing the actual interventions on patients, without any restriction for repetitive training. They potentially allow the simulated organs to behave in a biomechanically authentic manner in which the tissues deform and react in a realistic fashion. At the same time, due to present hardware development for specialized graphic-rendering engines, nearphotorealistic visualization of the operation site will become possible in the foreseeable future. However, the simulator systems proposed to date do not achieve the necessary level of realism. While this goal cannot be reached by today’s technology, it is of major importance to explore the current limits of realism in endoscopic surgery simulation and to analyze the potential for further development. This paper describes a first attempt to meet these long-term research objectives by a project concentrating on the following major areas. x Construction of a very detailed geometric model of the anatomical site to be simulated. In our case, we describe the generation of the anatomical model of the female abdomen based on the Visible Human female data set (section 2). x Study of the visual appearance of internal organs based on synthetic organ textures resulting from the systematic analysis of intraoperative images and from

x

x x

x

using different surface-visualization techniques such as Phong shading, bump mapping, and texturing (section 3). Development of a framework for full-scale, 3-D, finite element modeling (FEM) techniques for physically based simulation of elastic abdominal tissue deformation (section 4). Systematic study of the elastic material properties of living tissue (section 5). Design of a specialized parallel supercomputer together with appropriate FEM algorithms to speed up the expensive FEM calculations to allow realtime performance (section 6). Integration of force-feedback devices to provide the necessary haptic feedback for the surgeon (section 7).

Besides our theoretical and algorithmic results, a first working prototype of the planned system with modular system architecture running on SGI Onyx2/InfiniteReality hardware is described in section 8. This system allows flexible adaptation to different hardware components with widely varying performance capabilities and serves as a platform for ongoing and future developments.

2

Anatomical Model Building

Realistic simulation of the elastic deformation of abdominal organs and the resulting forces is possible only if based on a detailed anatomical model. The motion and deformation of organs is in many cases determined by morphological structures of small spatial extent (such as ligaments), which poses a major challenge for the generation of the anatomical model. Customary radiological imaging procedures cannot provide the necessary information due to serious constraints in image contrast and resolution (Sze´kely et al., 1998a; Hug, Brechbu¨hler, & Sze´kely, 1999). The Visible Human female data set, however, offers a consistent source of anatomical and morphological information that provides a very high-resolution data set with excellent tissue contrast. Consequently, this data set has been selected to form the basis of the anatomical model making. Existing image-segmentation methods can be

Sze´kely et al.

313

Figure 2. User interface of the segmentation tool.

grouped into the following three major methodical categories (Hug et al., 1999): x Fully automatic procedures that are easy to operate but lack precision and completeness and are verifiable only within poorly defined bounds. They have been extensively used for the segmentation of single objects, whereas their usability for multiobject systems is still an open question (Kelemen, 1998; Kelemen, Sze´kely, & Gerig, 2000). x Manual segmentation, which is highly insensitive to noise, tolerant of missing information, and sufficiently precise. The drawbacks, however, are that it is horribly time-consuming and tedious with limited reproducibility. Nevertheless, in many ambiguous situations, it is an advantage to use manual editing. x Between these two extremes, a whole family of semiautomatic tools have been developed (for example, Fischler, Tenenbaum, & Wolf, 1981; Kass, Witkin, & Terzopoulos, 1988), that combine the advantages of computational support by precise border detection with the benefits of manual manipulation. This man-machine cooperation implies some kind of collaboration between the human operator and the computing engine. While organ definition is inherently a 3-D task, we relied on the traditional slice-by-slice technique using

the familiar 2-D outlining methodology due to the possibility of applying simple and intuitive interfaces for interaction and visualization. To test the efficiency of interactive segmentation techniques, we performed some preliminary studies with MRI data volumes from a female volunteer. For potential candidates for semiautomatic segmentation, we identified edge-tracking algorithms such as snakes (Kass et al., 1988) and F*-based algorithms (Fischler et al., 1981). The results of these tests, however, were unsatisfactory with respect to the requirements on precision and consistency. Whenever edge information is incomplete, the algorithms produce undesirable results, and the contours must be adjusted in a time-consuming, manual-editing process. Therefore, we decided to rely completely on manual outlining based on cubic interpolating B-splines. To establish a powerful segmentation environment, we implemented a complete multiuser segmentation system with an underlying anatomical database. Special attention was paid to developing an intuitive user interface and providing very fast data access. To reduce the requirements on our computational infrastructure, we cropped the original images and reduced the spatial resolution by a factor of two. Figure 2 shows the user interface of the developed segmentation tool. All abdominal organs influencing the elastomechanical behavior and visual appearance of the operation site dur-

314


ing gynecological laparoscopic interventions were segmented using the segmentation environment discussed above. Additionally, the inner surface of the abdominal cavity was included. This multiorgan surface is of primary importance during simulation as it delineates the maximal spatial extent of the intervention. It surrounds the potential space where the gas is insufflated at the beginning of the surgery and wherein the surgeon places the instruments. The Visible Human female data set proved to be very suitable for the detailed definition of the abdominal organ geometry to be applied in a laparoscopic surgery simulator (Sze´kely et al., 1998b). Selected views of the resulting model are shown in figure 3. The excellent visual fidelity and spatial resolution provided by the color cryosections allowed the definition of anatomy in the required quality, which no other data source was able to fulfill. Limitations arised primarily from the postmortem anatomical changes. Even though we have been able to build a suitable geometric model for the simulation of laparoscopic surgery, one must not forget that the usage of the Visible Human female for model generation represents only a first milestone in the ongoing research towards the fast and efficient creation of patient-specific digital anatomy representations.

3

Graphical Modeling of Organ Appearance

Providing correct visual information in laparoscopy simulation is necessary to close the gap between VRbased surgical training and surgery on a real patient. Even though visual feedback is the widest information channel available to the surgeon, current surgical simulation systems often treat the visual aspect in an ad hoc manner. Whereas visualization for laparoscopy simulation involves the treatment of a fairly wide range of topics, we will restrict the following discussion to two aspects: the computation of organ-specific texture and the development of accurate illumination models.

3.1 Generation of Organ-Speci3c Texture All organ surfaces are covered by some microstructure, which provides us information about the type of

Figure 3. Different views of the segmented anatomical model.

the tissue and its relative smoothness or coarseness. Methods to simulate such textures are available in almost all visualization packages. Texturing, however, has uses other than increasing realism. Perceptual psychologists have recognized the importance of surface texture as a cue to space perception (Gibson, 1950; Haber & Henderson, 1980). Texturing is also of fundamental relevance in representing pathological tissue. This is especially important because one of the objectives of training on a laparoscopy simulator is the improvement of diagnostic skills. Today, organ-specific texturing is typically accomplished using interactive painting methods (Sellberg et al., 1995), by mapping real laparoscopic imagery (Ku¨hnapfel et al., 1995), or by interpolation based on the Visible Human data set (Reinig et al., 1996b; Knapp, Kerr, & Sellberg, 1997). Even though all of these methods have been successfully used in specific applications, they have disadvantages that make them unsuitable for sophisticated surgical training systems.

Sze´kely et al.

315

Figure 4. Texture analysis/synthesis: (a) original uterus; (b) arti+cial uterus.

The fundamental problem is that one specific virtual patient is not sufficient for training. What is needed is a system that allows medical experts on their own to define arbitrary anatomies, including pathologies, by overriding default attributes of a generic anatomical model. We therefore try to keep all texture-generation methods independent of a specific data set. Our goal is to provide a large texture database that is grouped into organs and pathologies, and a set of algorithms that allow us to automatically generate texture maps for a selected data set in a reasonable amount of time. Geometry-independent, organ-specific textures without blood vessels can be generated by means of the following automatic texture analysis/synthesis procedure (Gagalowicz & Ma, 1985). As a starting point, a statistical description of a small texture sample taken from real laparoscopic imagery is computed in an analysis phase, based on the second-order statistics of the color distribution. In a subsequent optimization step, a 3-D texture block, initially consisting of white noise, is modified until its statistical description is similar to the description of the sample texture (Meier, 1999). In the absence of strong anisotropy of the visual texture (which was the case for all abdominal anatomical structures we investigated), organs of arbitrary shape can then be textured by carving them out of the solid texture block (figure 4). Whereas analysis/synthesis procedures provide a convenient way to automatically generate fairly complex base textures, they do not take into account the lowfrequency variations that often occur on organ surfaces. In addition, texture patterns may be either too large or

Figure 5. Procedural texturing: (a, b) original and arti+cial liver; (c, d) original and arti+cial ovary.

too sparse to be captured by this method. To simulate these effects, we combine the texture analysis/synthesis approach with procedural textures. Procedural texturing is a method that defines 3-D textures by mathematical functions that can be evaluated on the fly (Peachey, 1985; Perlin, 1985). Even though procedural textures

Figure 6. Merging base texture and arti+cial blood vessels: (a) original peritoneum; (b) arti+cial peritoneum.

316


provide a powerful instrument, they are not widely used for medical applications because the design of functions that simulate natural textures is complicated. However, experience has shown that low-frequency variations of texture properties as well as sparsely distributed patterns can reliably be generated using simple noise functions (Perlin, 1985; Lewis, 1989) (figures 5(a) and 5(b)). To this point in our process, geometry and texture have been completely decoupled, which is not possible in some cases. For example, Figure 5(c) shows an ovary covered by follicles that show up in both texture and geometry. However, because the information about the size and position of these follicles must be provided by the anatomical model, it is still possible to generate texture maps without user interaction. A procedural texture simply combines an automatically generated ovary texture with randomly selected follicle images taking into account their positions and sizes (figure 5(d)). To generate organ-specific blood vessels, we developed a method based on L-systems (Prusinkiewicz & Lindenmayer, 1990), which are widely used in computer graphics to generate artificial plants. Basically, in an L-system, simple structures become more and more complex by simultaneously deriving their individual components according to given stochastic rules. Whereas L-systems are very suitable for generating trees, they have their disadvantages in the case of net-like structures. Our current research is therefore directed towards the simulation of the biological growing process of vessels (Meier, 1999). This procedure is based on simplified models of the angiogenesys, the formation of new vessels directed by the perfusion demand of growing tissue as well as physical laws of blood circulation. Besides the ability to naturally handle vascular networks, physiologically based modeling of vessel growth can naturally adapt to organ-specific vessel formation by varying physically meaningful parameters. Up to this point, characteristic fluctuations of tissue color and the visible blood vessels along the organ surface were modeled independently as the two underlying sources of visible organ texture. Realistic organ appearance will be achieved only if those two are combined to a single texture map, which is needed for real-time visualization. This can be reached by using a distributed ray-

tracing approach. Figure 6 illustrates the result of this process on the example of the peritoneal surface.

3.2 Illumination Models The wide range of rendering strategies that are currently available each use different illumination models. Experience has shown that artificial laparoscopic images of very high quality can be produced by a fairly simplified ray-tracing approach (Meier, 1999) (figure 4(b)). Because in laparoscopy the position of the camera and the light source are always identical, shadow computations are not required. Furthermore, the light source can be represented by an infinitely small point that radially emits energy that does not attenuate with distance.1 Finally, the computation of reflection as well as the evaluation of Snell’s law to determine refraction on transparent surfaces can be neglected without noticeable degradation in image quality. Real-time ray tracing cannot be expected to be available in the near future. However, due to the above simplifications, the resulting illumination model can also be evaluated in real time by means of a sophisticated scanline-based approach such as provided by the PixelFlow system (Molnar, Eyles, & Poulton, 1992). As already mentioned, an SGI Onyx2/InfiniteReality workstation has been used as the visualization platform that, at the lowest level, provides OpenGL as a programming interface. As opposed to the PixelFlow system, this rendering engine does not implement per pixel shading. Instead, it evaluates the illumination equations at the vertices of the geometrical primitives and computes the intermediate values by linear interpolation (Gouraud shading) (Watt, 1989). Whereas the restrictions of OpenGL’s illumination equations allow the assembly of very efficient hardware, the associated degradation in image quality strongly disturbs the realistic effect when applied to laparoscopy simulation. However, by using OpenGL extensions, it is possible to improve image quality, but at the expense of rendering performance.

1. Ignoring light attenuation is a simple way of simulating the automatic regulation of light intensity in laparoscopic illumination systems.

Sze´kely et al.

Figure 7 illustrates a series of OpenGL-based rendering techniques that incrementally approximate the simplified ray-tracer model at the cost of a decreased frame rate. Figure 7(a) shows a patch with arbitrarily defined geometry, covered by an automatically generated liver texture that was rendered using the standard OpenGL illumination equation. In figure 7(b), two separate rendering passes—one for specular and one for diffuse illumination—are used to produce the white specular highlights that are a major visual cue in laparoscopic imagery. In figure 7(c), black-white equalization is simulated at the cost of an additional image copy. The purpose of this calibration step is to adapt the sensitivity of the endoscopic camera to the specific lighting conditions of a scene by spreading a specific interval of energy values across the full color range. In figure 7(d), the shininess of the surface has been raised beyond the available maximum value in OpenGL by using a color look-up table and an additional image copy. In this way, we can remove the plastic-like appearance of the surface. Finally, figure 7(e) implements per pixel shading by using additional texture maps to compensate for the error introduced by Gouraud shading.2 Even though image quality can be increased by using complex OpenGL-based rendering strategies (as illustrated by figure 7), the resulting algorithms are rather cumbersome approximations.

4

(a)

(b)

(c)

(d)

317

Tissue Deformation Modeling

Realistic simulation of tissue behavior during interventions is one of the most challenging research areas in surgical simulation. While first attempts have already been made for the highly simplified modeling of complex interactions with organs, such as clipping, cutting, or suturing (Ku¨hnapfel, Cakmak, & Maaß, 1999; Pflesser, Tiede, & Ho¨hne, 1998; Basdogan, Ho, & Srinivasan, 1999; Baur et al., 1998; Voß et al., 1999), our present project concentrated exclusively on the problem of organ deformations. Real-time simulation of elastic tissue deformation is a major obstacle in developing simulator systems for soft2. This strategy is restricted to static surfaces.

(e) Figure 7. Illumination: (a) standard texturing; (b) separating specular and diffuse illumination; (c) black-white equalization; (d) sharpening of specular highlights; (e) per pixel shading.

tissue surgery. Different methods in use for deformation modeling include: x Freeform deformation techniques from computer graphics (Barr, 1984; Sederberg & Parry, 1986) use parametric interpolative models (as polynomial models, splines, or superquadrics, for example) for deformation estimation of the primitives. While the

318


analogy to physical deformation processes is not always obvious, such techniques have become very popular in surgical simulators (Baur et al., 1998; Basdogan, et al., 1998) due to the resulting fast deformation calculation. x Different, simple, physically inspired approximations have also been used for tissue deformation modeling. Most popular are mass-spring models (Ku¨hnapfel et al., 1995; Downes et al., 1998; BroNielsen et al., 1998; Boux de Casson & Laugier, 1999), but other alternatives like space-filling spheres (Suzuki et al., 1998) or the ChainMail algorithm (Gibson, 1997) have also been implemented. x Elastically deformable surface models used in computer graphics and computer vision (Terzopoulos et al., 1987) calculate surface deformations by solving linear elasticity equations. These methods allow simulation of tissue deformation based on physical principles. Full 3-D extensions of these techniques (Cotin et al., 1996; Bro-Nielsen & Cotin, 1996) represent the first attempts for finite element-based modeling of tissue deformation. Because these are linear finite-element approaches, they are not feasible to applications in which big deformations and rotations appear. Today, the nonlinear finite-element method has been implemented in such simulators only for very special cases (Sagar et al., 1994). Due to its physical background, we use this approach in our work even if it needs a good deal of computational power.

4.1 Finite-Element Modeling The finite-element modeling (FEM) is a very common and accurate way to solve boundary-value problems in mechanics of continua (Bathe, 1996; Zienkiewicz & Taylor, 1994). In the case of biological tissue, we have to deal with large deformations and also with anisotropic, inhomogeneous, and nonlinear materials. Furthermore, as organs and surgical instruments interact, numerical contact problems arise. Nonetheless, provided an adequate formulation is chosen, even in these cases FEM is a very powerful tool because it is physically based and,

therefore, can be adapted for these highly nonlinear problems. Within FEM, a body is subdivided by a finite number of well-defined elements (such as hexahedrons, tetrahedrons, and quadrilaterals). Displacements and positions in an element are interpolated from discrete nodal values. For every element, the partial differential equations governing the motion of material points of a continuum can be formulated, resulting in the following discrete system of differential equations: Mu ¨ 1 CuÇ 1 Kd u 5 f 2 r,

(1)

where M is the mass matrix, C is the damping matrix, K is the incremental stiffness matrix, u is the vector of the nodal displacements, f are the external node forces, and r are the internal node forces. All these matrices and vectors can be time dependent. One approach is to solve equation (1) in a quasistatic manner (Crisfield, 1991; Hinton, 1992). In this case, the dynamic part of the equation is neglected (u¨ 5 uÇ 5 0) and the solution is reached iteratively. Within every iteration, a huge set of linear algebraic equations must be solved. In problems with large deformation and contact interactions, the iteration often does not converge, and a stable simulation over several minutes is nearly impossible to achieve. Otherwise, the dynamic equation has to be integrated with respect to time (Hinton, 1992). The time integration of the equation can, in principle, be performed using implicit or explicit integration schemes. Using the implicit method, the solution has to be calculated iteratively at every discrete time step, like in the quasistatic problem. On the other hand, the explicit time integration can be performed without iteration and without solving a system of linear algebraic equations. This integration scheme is only conditionally stable; that is, only very small time steps lead to a stable solution (Flanagan & Belytschko, 1984). An estimation for the critical time step is made by D tmax 5

DL

c

,

where c is the maximal wave propagation speed in the

Sze´kely et al.

Figure 8. Finite-element model of the uterus containing approximately 2,000 elements.

medium and D L is the smallest element length of the model. In the uterus model shown in figure 8, this equation leads to 10,000 time steps per second (D t 5 100 m s). These short time steps increase the computational effort but lead to a very stable contact formulation. The disadvantage of this method (that many steps have to be taken) is not so significant because each step is much less time-consuming than in an implicit algorithm.

4.2 Element Formulation Because of the considerations discussed above and based on our numerical tests, we decided to solve the problem with an explicit finite-element formulation. Unfortunately, numerical integration schemes lead to accumulated discretization errors. This can be circumvented using an adequate formulation. The most time-consuming part in the explicit formulation is the computation of the internal element forces, including the calculation of stresses and strains. These state variables are related to well-defined reference configurations (Sze´kely et al., 1998c; Sze´kely et al., 2000). Hypoelastic materials lead to a relation between stress rates and strain rates (incremental formulation). Here, stresses must be calculated by a time integration and, therefore, no direct relation between the actual stresses and the initial configuration does exist. This means that a body does not remember its stress-free position, and remaining deformations appear after forces have been applied and released (closed load path) even if the material is assumed to be elastic. This, of course, leads to large errors during simulations of long duration and

319

drastically influences the stability of a calculation. In contrast, hyperelastic materials lead to a direct relation between an absolute strain and an absolute stress form. This results in a conservative formulation, and no remaining deformations appear after a closed load path. To compute the internal forces of an element, a volume integral must be evaluated, which is usually performed by a numerical eight-point quadrature. This calculation is very time-consuming and can usually be replaced by a reduced volume integration (Belytschko & Ong, 1984). These classical methods are based on incremental schemes leading to the same drawback as discussed above for hypoelastic materials. Therefore, reduced volume integration based on absolute strain formulation has been developed (Hutter, Hora, & Niederer, 2000). The following structogram represents recovery of all g elements in one time step (5 meaning a gather operas tion and 2 5 a scatter/decrement). ; elements e g

uE 5 u calculation of the deformation gradient calculation of hourglass modal displacements calculation of the invariants of the right CauchyGreen deformation tensor calculation of pressure calculation of the nominal stress tensor calculation of the generalized hourglass forces projection of the nominal stresses and the generalized hourglass forces onto the internal forces fE s f 2 5 fE The above considerations suggest the use of a hyperelastic material law. The constitutive equations appropriate for usage in biological tissue modeling are discussed in more detail in section 5.2.

4.3 Simulation Experiments The developed methods have been extensively tested by offline simulations of regular meshes (Hutter et al., 2000) and a first model of the uterus. Figure 8 shows the generated FEM mesh of the uterus and its adnexes, containing about 2,000 elements. The abdominal cavity was modeled as a rigid surface.

320


Figure 9. Initial state of the model uterus (a) compared to an interoperative image of the operation site (b).

Figure 9 illustrates the initial state of the uterus within the abdominal cavity (a) compared to an interoperative image of the operation site (b). Figure 10 shows an example for the deformation of the corpus (a) as well as for the manipulation of the fallopian tube (b) of the uterus. Here, gravitational forces were applied in the direction of the graphics plane. Therefore, the organ lies on the abdominal cavity (contact type: rigid-elastic) and the fallopian tubes touch the ovarii (elastic-elastic contact). The simulations have turned out to be stable even in the case of big deformations (Hutter, 1999) and complex 3-D contact interactions as illustrated in figure 10.

and animal tissue, we must perform the measurements in vivo on patients during interventions. Additionally, accurate data cannot be obtained from the usual testing set-ups because they need tissue preparation for experimental testing, which results in significant changes in mechanical tissue properties. A number of experiments have been performed on a variety of tissues under different loading conditions (Fung, 1993), but there exists very little data from in vivo measurements (Moulton, 1995; Maaß & Ku¨hnapfel, 1999; Carter et al., 1999).

5.1 Measuring Method 5

In Vivo Measurement of Tissue Elasticity

Realistic modeling of tissue deformation cannot be performed without a knowledge of the elastic properties of living tissue. Even the best description of the mechanical behavior of tissue is useless if its parameters cannot be determined. The selection of the constitutive equation describing the elastic behavior of a specific organic material (the material law) should therefore be followed by the determination of the actual numerical values of material parameters. Because significant differences are expected between the mechanical properties of dead and living, and human

Measurement of mechanical properties of living human tissue presents several problems. Imaging methods, like MR elastography (Muthupillai et al., 1996) or sonoelastic imaging (Gao, 1995) can be applied for only a very limited strain range, while classical methods of biomechanical testing (Fung, 1967) are difficult to use under in vivo conditions. First of all, the physician must ensure that no damage to the patient is caused by the experiment. Accordingly, the measuring instrument has to be secure, ergonomic for the surgeon, and sterilizable. Furthermore, the instrument has to keep track of the boundary conditions during measurements because the contact of the instrument with the tissue to be measured can lead to global motion of the organ under investigation.

Sze´kely et al.

321

Figure 10. Simulated deformation of the corpus (a) and the fallopian tube (b) of the uterus.

Figure 11. Principle of the tissue aspiration experiment (left) and the pro+le measurement (right).

One technique that provides accurate data and overcomes the abovementioned problems is tissue aspiration (Aoki, 1997) in conjunction with inverse FEM. Tissue aspiration involves placing a tube against the target tissue and producing a weak vacuum in the tube. The vacuum fixes the organ to the tube, specifying well-defined boundary conditions, and causes some small deformation of the tissue inside the tube. The implemented measurement process is based on varying the relative negative pressure in the tube over time and determining the functions z(r, t) and P(t), where t is the time from the start of the measurement, P(t) is the negative relative pressure in the tube at time t, and z(r, t) is the profile of

the deformation. Thus, we track over time the applied negative pressure and the resulting deformation. The measurement method, together with the profile of a deformation, is illustrated in figure 11. Assuming axisymmetry and homogeneous tissue in the portion covered by the aspiration tube, a complete description of the deformation is given by the profile z(r, t) of the aspirated tissue (or even by one-half of the profile). Inverse FEM provides the greatest freedom in the boundary conditions and the geometry of the problem, and, due to the complexity encountered during in vivo experiments, seems to be a suitable way to determine material parameters (Kyriacou, Schwab, & Humphrey, 1996). On the

322


other hand, this method based on force-displacement measurements turned out to be very sensitive to measurement errors (Kyriacou, Shah, & Humphrey, 1997; Moulton, 1995) and the initial state of stress. Because it is virtually impossible to determine the initial state of stress of living tissue and the stresses caused by the experiment are much larger than the initial stresses, a stress-free initial configuration is assumed. The diameter of the aspiration tube determines the spatial extent of the tissue affected by the experiment. This allows the mechanical properties of small portions of the tissue to be identified. By tracking the geometry of deformations at different pressures over the entire loading time, we achieve robustness against small measurement errors and also allow potentially the identification of viscoelastic parameters.

5.2 Material Law The material law is one of the most critical points in modeling tissue deformation. Living tissue is a nonlinear, inhomogeneous, anisotropic material with viscoelastic properties. At the present stage of the project, we do not try to account for all these properties in our material model; instead we try to develop a method that allows the identification of material parameters once a material law has been chosen. As a first approach for the description of the mechanical behavior of soft tissue, we use a strain energy function (W) similar to the one used by Mooney-Rivlin (Rivlin & Saunders, 1951): W 5 m (J1 2 3) 1 a (J2 2 3) 1 1L2 k (J3 2 1)2,

(2)

where J1, J2, and J3 are the reduced invariants of the right Cauchy-Green deformation tensor C 5 FTF, and m and a are material parameters. F5

x

X

is the deformation gradient of the actual position x with respect to the position X of the undeformed configuration. The bulk modulus k allows for slight compressibility. It is assumed that the bulk modulus is several orders of magnitude larger than the material constants m and a .

This makes k a penalty parameter in the finite-element formulation and so is not included in the identification process. The second Piola-Kirchhoff stress tensor S can be obtained from equation (2) by differentiation with respect to the Green-Lagrange strain tensor E: S5

W

E

,

(3)

where E 5 1L2(C 2 I). The reduced invariants are defined as follows: J1 5 I1I 23 1/3, J2 5 I2I 23 2/3, and J3 5 det(F) 5 I3, where I1, I2, and I3 are, respectively, the first, second, and third invariant of the right CauchyGreen deformation tensor C. The use of reduced invariants allows us to interpolate the hydrostatic pressure as a separate degree of freedom in the finite-element formulation (u-p formulation), thus avoiding convergence problems for nearly incompressible material behavior. Preliminary theoretical studies revealed the impossibility of identifying both parameters m and a separately, unless the applied deformations in the identification experiment are very large. Because such large deformations cannot be applied in experiments on living tissue, the identification has to be restricted to only one parameter. Therefore, the second material parameter is set to zero and the material law reduced to a Neo-Hookean description: W 5 m (J1 2 3) 1 1L2 k (J3 2 1)2.

(4)

5.3 Numerical Methods Parameter identification in our approach is performed by a minimization of squared differences between measured load-displacement and simulated loaddisplacement data. The minimization starts with an initial estimate of the material parameters and approaches the minimum of the squared differences. Of the many available methods, we chose the LevenbergMarquardt algorithm, which has proved to be robust for our application and very efficient, because the derivatives of the squared differences with respect to the material parameters are not needed explicitly but can be approximated. To solve the geometrically and materially nonlinear

Sze´kely et al.

323

Figure 13. Photograph of the instrument constructed for tissue aspiration experiments.

Figure 12. Schematic design of the vision-based measurement.

problem of the tissue deformation, we use standard FEM described by Sussman and Bathe (1987). Fournode axisymmetric elements are used to discretize the domain of the part of tissue of interest. The profile of the undeformed tissue surface is used to generate an exact finite-element model of the tissue undergoing the experiment. Contact between the tissue and the aspiration tube is modeled as being deformable-undeformable. An explicit integration technique as described in section 4 is used. The relative pressure p(t) measured over time in the experiment is applied as time-dependent, external surface force on the area of aspirated tissue in the model. This model then gives us the needed theoretical force-displacement data corresponding to the assumed material law for comparison with the measured force-displacement data from the experiment. Further details of the applied procedure can be found in Kauer, Vuskovic, and Dual (1999).

5.4 The Measuring Instrument To perform the measurements, we developed a vision-based device that permits the controlled application of negative relative pressure and tracks the profiles of small deformations caused during the measurement process. With a periscope geometry obtained by placing a small mirror beside the aspiration hole as indicated in figure 12, it is possible to view the silhouette of the deformed tissue with a camera placed at the other end of the tube.

The method allows us to measure the desired profile very accurately, rapidly, and without contact with the tissue. An optical fiber fixed in the tube illuminates the aspirated tissue. The camera with close-up lenses, the tube, the light fiber, the pressure sensor, and the control switches are mounted onto an aluminum body, which also gives maneuverability to the instrument. (A photograph of the instrument is shown in figure 13.) The pneumatic system is connected to the aspiration tube via a flexible silicon tube and is designed to allow easy and secure control of the pressure in the tube and to measure the pressure very accurately. The profiles are extracted from the grabbed images in real time at a rate of 25 Hz and with a resolution of 30 micrometers. Real-time extraction of the profiles avoids storage problems and can lead to online material parameter estimation in the future. The measurement instrument and the applied image-analysis procedure is described in more detail by Vuskovic, Blaser, and Spiga (1999).

6

Design of a Real-Time FEM Computation Engine

The only way to provide the necessary computational power for the real-time solution of complex finiteelement systems is to build a parallel computer that can execute fully parallel algorithms for the explicit timeintegration scheme. The design described below allows us to scale the computation to the necessary speed with the selection of an appropriate number of processor units or processing elements (PE). The following subsections summarize the basic requirements and design principles for implementing special-purpose parallel

324


hardware to perform explicit finite-element calculations. Present algorithmic design considers only interventions in which the neighborhood relationship of the single FEM cells are not changed during the simulation by cutting, as in the case of diagnostic laparoscopy or coagulation.

6.1 Performance Requirements To find appropriate parallel algorithms for the necessary FE calculations, we must analyze the performance requirements to determine what kind of computer is needed. Analysis of optimized explicit finite-element algorithms shows that approximately 650 floating-point operations per element are needed in each time step. Additional computation time has to be reserved for collision detection and handling, and we estimate the total cost as 1,000 floating-point operations per element. This leads to 1 kFLOP/100 m s 5 10 MFLOPS (million floating point operations per second) per finite element for the chosen time step of 100 m s. For 2,000 elements, a total of 20 GFLOPS sustained is needed. This amount of computational power is much too high for even a state-of-the-art workstation. Implicit FE calculation can be implemented on vector-supercomputers, because the main task is to solve huge sparse systems of linear equations (Taylor, 1991; Pommerell, 1992). However, the time steps and the vectors of the explicit method are too short for efficient vectorization; therefore, a high-performance parallel machine is needed. The latest processors are able to deliver more than 300 MFLOPS with optimized programming, whereas 64 computational nodes will meet our demands. For the sake of numerical stability, the usage of double precision floating-point arithmetic may be desirable. During the time steps, data has to be communicated with minimal processor involvement, requiring that very lightweight protocols are used. During calculations, three different types of communication are necessary: exchange of forces, collision detection and handling, and data transfer to the graphics engine. For an FE model with 2,000 elements on a 43 43 4 processor machine, 5,600 force vectors must be exchanged each time step. This results

Figure 14. The 3-D communication architecture of the envisioned parallel FEM computation engine.

in a communication bandwidth of 1.35 GByte/s, but the fact that data has to be transferred only between neighbors eases our requirements. According to the previous algorithmic considerations, a 3-D lattice of processors can be optimally used, in which every PE is connected to its six neighbors (figure 14). We build it up from DEC a processors connected with Myrinet.

6.2 Partitioning the Model for Parallel Computation Explicit FE computation suggests element-wise parallelism, thereby decomposing the spatial domain. Each element is mapped to one processor, and a processor computes the internal forces of several elements. Whenever elements residing on different processors share a node, all those processors must update their copy of the node and need the resulting force to update its position. This force is accumulated in the course of the communication between the processors involved, using local communication where possible. The mapping of elements to processors should balance the computational load and keep the necessary communication bandwidth within limits. This could be solved as a discrete optimization problem by minimizing the overall computation time, using a model for the influence of the mapping. We currently define the mapping manually. The selected 3-D topology of the processor grid defines three communication directions (x, y, and z), and each PE has up to six local communication channels to

Sze´kely et al.

325

Figure 15. A 2-D FE model explaining the principles of force recovery during the FE computation. The elements are distributed to processors. The large numbers are the processor coordinates ix, iy. Between elements, x borders are dashed and y borders dotted.

all its direct neighbors (processors on the border of the grid lack some neighbors). Forces on shared nodes are communicated between involved processors in three phases organized according to the neighborhood directions. The recovery of inner forces for all elements is split into four batches, interlaced with the communication phases. All forces acting on a node on the x border must be computed before sending them in direction x, and all elements with such a node must be computed in the first batch, incidentally called ‘‘x.’’ While the forces are being transferred, elements in the similarly defined ‘‘y’’ batch are recovered. The third batch consists of elements containing nodes on the z border but none on x or y. The remaining elements are local to the PE; they form the fourth, ‘‘inner’’ batch. Figure 15 illustrates the above principles of force recovery on a 2-D example for simplicity. Figure 16 presents the computation and communication phases. Interlacing and overlapping communication with the actual calculations allows a maximal utilization of the computer power available without delays caused by communication latencies. Rhomberg et al. (1999) and Rhomberg et al. (1998) provide further information about this parallel computation scheme.

6.3 Collision Detection The selected explicit integration is ideally suited for collision detection, which uses the global network for data transfer. We adopt the following approximation for collision detection that is commonly used with FEM.

Figure 16. Phases of a time step. The elements on different processors are moved apart. Inertial and gravitational forces act initially on thickly lined nodes. Initial forces of the gray copies of nodes are zero. (a): Batch 0 computes (‘‘recovers’’) reaction forces for elements on the x border (dark shade indicates ongoing computation). (b): Some forces of computed elements on their nodes are now known, indicated by a light shade in the respective corners. Now the processors exchange results from batch 0 on x border nodes; arcs show communication. Concurrently, batch 1 computes elements on the y border. (c): This last batch computes forces from ‘‘inner’’ elements while the last communication ( y, shown as arcs) is in progress. The communicated forces include components received earlier from other processors. (In this 2-D illustration, no z batch and z communication exist). (d): After the last batch of elements is computed and all incoming forces have been added, all nodes have the correct resulting force.

326


This method determines only which nodes of one contact partner (the slave) penetrate any face of the other contact partner (the master). Flipping the roles of master and slave and averaging balances the obvious asymmetry of this approach. Collision detection can be performed selectively due to upper limits for the speed of physically realistic motion during operations. Elements are not expected to move faster than 10 cm/s, so, if at some point, two elements are 2 cm apart, the earliest time they can touch is 100 ms or 1,000 time steps away. A collision between these elements cannot take place within the next 999 time steps and therefore needs no attention during this time interval. In this way, the combinatorical explosion caused pairwise collision checking can be eliminated. Additionally, the remaining collision tests are performed in a hierarchical manner based on a dynamically updated hierarchy of bounding boxes, which further reduces the necessary computations.

7

Force-Feedback Manipulator

Although the tactile information mediated by the surgical instruments during laparoscopic surgery is strongly limited, force feedback is an indispensable component of any realistic simulation environment. Until now, no really satisfactory technical solutions have been presented for providing tactile and force feedback in the simulation of open surgery. However, during minimally invasive operations, haptic information is provided exclusively by mechanical manipulators, making feasible the implementation of simulated surgical instruments that provide realistic force feedback based on the technology available today. Force-reflecting interaction devices have a long history. Traditionally, manipulators with force-feedback ability have been developed for and used in teleoperation environments such as nuclear applications (Goertz, 1964). Due to rapidly growing demands in a wide range of virtual reality applications, many solutions have been proposed for force generation in interactive computer environments, such as the GROPE teleoperator master arm (Brooks et al., 1990), joysticks (Schmult & Jebens,

Figure 17. Manipulative degrees of freedom of a laparoscopic surgical instrument.

1993), 3-D pointers (Massie & Salisbury, 1994), or exoskeletons (Bergamasco & Prisco, 1995), allowing movements according to very different degrees of freedom. A good overview of the different approaches can be found in Baumann (1997). Besides general-purpose devices, specialized instruments for force-feedback manipulators in laparoscopic surgery simulators have been developed, some of which are even commercially available (Immersion, 1995). During laparoscopic surgery, the following four specialized degrees of freedom of the manipulators are required (as illustrated in figure 17): x The pivoting of the trocar providing the entry point of the instrument into the body. This defines a conical workspace described by two angles of tilt (a and b ).

Sze´kely et al.

327

x Translation of the surgical instrument through the trocar into the body (d). x Rotation of the instrument along its longitudinal axis (g ). Additionally, different specialized degrees of freedom for the instrument handling may be required. A more detailed analysis of the force-feedback requirements in laparoscopic surgery can be found in Baumann (1997). In our system, we build the simulated surgical instruments on the basis of the commercially available PHANToM device (Massie & Salisbury, 1994) (Sensable Devices). Of course, in this case, the provided manipulative degrees of freedom have to be converted into a subset of the above-listed ones; no force feedback is provided for the instrument rotation (g ) in the first version of the simulator. A mechanical device translates the degrees of freedom of the PHANToM to the laparoscopic situation. A physical phantom of the female abdomen providing realistic geometry and external control of the endoscope (based on simple position tracking without force feedback) and a force-controlled palpation device is currently under construction.

8

Prototype Simulator for Laparoscopic Gynecology

Although the dedicated hardware is not yet assembled and functional, we have implemented the algorithms both for the preparatory steps of partitioning the mesh and for the actual parallel computation, as well as a complete system, simulating not-yet-existing devices. We can identify the following five units that work together in the simulator (figure 18) as independent programs. 1. The instrument device encodes the position of the virtual instrument controlled by the user and feeds back the resulting contact forces. 2. The optic device encodes the position of the virtual endoscope and, hence, the viewpoint of the camera. 3. The mechanic engine performs the computationally expensive FEM calculations.

Figure 18. Flow of data between the distinct parts of the simulator. The fat arrow symbolizes the most substantial data 9ow (coordinates and normals of all surface nodes). The numbers below the boxes are the repetition rates that our design aims toward. Numbers next to arrows indicate how many real numbers need to be transferred in one cycle of the slower participant.

4. The graphics engine renders the scene as seen through the endoscope. 5. These four parts are connected through a link we chose to call ix; it establishes all communications between the above parts. Figure 18 also sketches the flow of data between these five parts of the surgery simulator. The position of the instrument is transfered both to the mechanic engine, where it may cause contact, and to the graphics engine, which must render the instrument as long as it is visible in the scene. Mechanic reaction forces are propagated back to the force-feedback device of the instrument. The new positions of the surface nodes of the deformed organs must be communicated to the graphics engine for each frame including surface normals. Finally, the graphics engine must know the position of the optic device for defining the camera position. In actual surgery, the endoscope may well collide with an organ, resulting in organ deformation and reaction forces (also calculated by the mechanic engine). At present, we do not yet model these occurrances, and the corresponding arrows are dashed.

328


Figure 19. Simulator control using mouse-based graphical user interfaces.

The modular design of the simulator makes it simple to reconfigure, allowing fast adaptation to user-specific needs. Different versions of the modules have already been developed. Fully functional versions allow the integration and support of different hardware components. The instrument device, for example, has two active versions: one presents a mouse-based graphical interface in an X11 window as illustrated in figure 19, and the other interfaces to the PHANToM force-feedback device. Of course, we cannot compute a realistic FE model in real time with the mechanic engine running on the commercial platforms that are currently available. We use a simplistic model of 21 elements and achieve time steps of approximately 2 ms. However, the prototype demonstrates the interplay of all relevant components of the system and the validity of our parallelization concept.

9

Conclusion

We have summarized the first steps we have taken in the construction of a realistic, VR-based, endoscopic surgery simulator. Experience gained during this phase

of the project has shown that, while many theoretical and practical difficulties must be overcome, sufficiently realistic simulator components that mediate visual and haptic information to the user are within reach and can be expected to provide satisfactory results in the foreseeable future. A fully functional prototype of the simulator has been implemented using commercial hardware, and it demonstrates the feasibility of the developed algorithms. The highly modular architecture of the simulator system will allow the replacement of its building blocks by modules with more-advanced hardware components during future development. More-fundamental difficulties arise in the field of physically based simulation of the behavior of soft tissue. While our initial results allow us the simulation of diagnostic laparoscopic procedures using FEM techniques, even in the case of ‘‘simple’’ deformation modeling, much basic research is still needed until the desired level of realism can be achieved. In regards to measuring the properties of elastic tissue, the results are even more preliminary, and long-term research and development efforts—as well as the establishment of complex anatomical databases containing integrated geometrical, textural,

Sze´kely et al.

anatomical, physiological, and elastomechanical data (including information about pathological changes)— are needed before VR-based simulators can comply with the expectations of the surgeon. Our future work will concentrate on the solution of the following open problems. x In anatomical model making, the development of truly 3-D interactive segmentation tools is of paramount importance if training on patient-specific data is to be achieved. The effort for model generation can thus be kept within clinically acceptable limits even for anatomical regions for which no fully automatic procedures can be expected in the near future. The usability of such interactive systems will basically depend on the availability of new paradigms in man-machine interface technology (including virtual reality), allowing fast and efficient interaction with volumetric radiological data. x While we are very encouraged by the first results in the field of physiologically inspired synthesis of organ-specific vascular structures, substantial research is still needed to support the synthesis of near-photorealistic textures of organ surfaces. Further work will also be necessary for the realistic modeling of the visual effects of the intervention on the anatomical scene, such as bleeding, irrigation, or coagulation. x Several important issues still need to be addressed in regards to modeling tissue behavior: 1. While our present framework already includes first solutions for the detection and handling of elastic tissue contact, substantial work is still needed before simulation with quality similar to our results in organ deformation becomes possible. 2. The current FEM framework has to be fundamentally redesigned to model a variety of surgical actions (like cutting). 3. Based on the developed instrumentation and methods, systematic collection of elastic tissue parameters will be performed. Additionally, the acquisition technique, which currently can handle only solid organs, has to be adapted for

329

measurements on hollow structures like the bladder or the intestines.

Acknowledgement This work has been supported by grant 5480/41-2625.5RH of the Swiss Federal Institute of Technology Zurich and grant 309/1995 of the EMDO Foundation, Switzerland.

References Alyassin, A., & Lorensen, W. (1998). Virtual endoscopy software application on a PC. In Proc. MMVR’98 (pp. 84–89). IOS Press. Aoki, T., Ohashi, T., Matsumoto, T., & Masaaki, S. (1997). The pipette aspiration applied to the local stiffness measurement of soft tissues. Annals of Biomedical Engineering, 25, 581–587. Barr, A. (1984). Global and local deformations of solid primitives. Computer Graphics, 18(3), 21–30. Basdogan, C., Ho, C.-H., & Srinivasan, A. (1999). Simulation of tissue cutting and bleeding for laparoscopic surgery using auxiliary surfaces. In Proc. MMVR’99 (pp. 38–44). IOS Press. Basdogan, C., Ho, C.-H., Srinivasan, M., Small, S., & Dawson, S. (1998). Force interactions in laparoscopic simulations: Haptic rendering of soft tissues. In Proc. MMVR’98 (pp. 385–391). IOS Press. Bathe, K. (1996). Finite Element Procedures. Englewood Cliffs, NJ: Prentice Hall. Baumann, R. (1997). Haptic interface for virtual reality based laparoscopic surgery training environment. Unpublished doctoral dissertation, no. 1734. Swiss Federal Institute of Technology, Lausanne. Baur, C., Guzzoni, D., & Georg, O. (1998). Virgy: A virtual reality and force feedback based endoscopy surgery simulator. In Proc. MMVR’98 (pp. 110–116). IOS Press. Belytschko, T., & Ong, S. (1984). Hourglass control in linear and nonlinear problems. Computer Methods in Applied Mechanics and Engineering, 43, 251–276. Bergamasco, M., & Prisco, G. (1995). An experimental approach to visual surfaces exploration exploiting an arm exoskeleton as haptic interface. Telemanipulator and Telepresence Technologies, 2590, 172–182. SPIE.

330


Boston Dynamics, Inc. http://www.bdi.com. Bro-Nielsen, M., & Cotin, S. (1996). Real-time volumetric deformable models for surgery simulation using finite elements and condensation. Comp. Graphics Forum, 15(3), 57–66. Bro-Nielsen, M., Helfrick, D., Glass, B., Zeng, X., & Connacher, H. (1998). VR simulation of abdominal trauma surgery. In Proc. MMVR’98 (pp. 117–123). IOS Press. Brooks, F., Ouh-Young, M., Blatter, J., & Kilpatrick, P. (1990). Project grope—Haptic displays for scientific visualization. Comp. Graphics, 24(4), 177–185. Carter, F., Frank, T., Davies, P., McLean, D., & Cuschieri, A. (1999). Biomechanical testing in intra-abdominal soft tissues. Medical Image Analysis. (Submitted.) Boux de Casson, F., & Laugier, C. (1999). Modelling the dynamics of a human liver for a minimally invasive surgery simulator. In C. Taylor & A. Colchester (Eds.), Second International Conference on Medical Image Computing and Computer-Assisted Intervention MICCAI’99 (pp. 1156–1165). No. 1679 in Lecture Notes in Computer Science. Springer Verlag. Cotin, S., Delingette, H., Cle´ment, J., Bro-Nielsen, M., Ayache, N., & Marescaux, J. (1996). Geometrical and physical representations for a simulator of hepatic surgery. In Proc. MMVR’96 (pp. 139–151). IOS Press. Cover, S., Ezquerra, N., & O’Brian, J. (1993). Interactively deformable models for surgery simulation. IEEE Comp. Graphics and Appl., 13, 68–75. Crisfield, M. (1991). Non-linear Finite Element Analysis of Solid and Structures (vol. 1: Essentials). Chichester: John Wiley & Sons. Daane, P., Constantinou, P., & Hesselroth, G. (1995). A $100 surgical simulator for the IBM PC. In Proc. MMVR’95 (pp. 79–80). IOS Press. Downes, M., Cavusoglu, M., Gantert, W., Way, L., & Tendick, F. (1998). Virtual environments for training critical skills in laparoscopic surgery. In Proc. MMVR’98 (pp. 316–322). IOS Press. Fischler, M., Tenenbaum, J., & Wolf, H. (1981). Detection of roads and linear structures in low-resolution aerial imagery using a multisource knowledge integration technique. Comp. Graphics Image Proc., 15, 201–223. Flanagan, D., & Belytschko, T. (1984). Eigenvalues and stable time steps for the uniform strain hexahedron and quadrilateral. Journal of Applied Mechanics, March, 51, 35–40. Fung, Y. (1967). Elasticity of soft tissues in simple elongation. Am. J. Physiol., 213(6), 1532–1544.

Fung, Y. (1993). Biomechanics: Mechanical Properties of Living Tissues. New York: Springer-Verlag. Gagalowicz, A., & Ma, S. (1985). Sequential synthesis of natural textures. Computer Vision, Graphics, and Image Processing, 30, 289–315. Gao, L. (1995). Sonoelasticity imaging: Theory and experimental verification. J. Acoust. Soc. Am., 97(6), 3875–3886. Gibson, J. (1950). The Perception of the Visual World. Cambridge, MA: The Riverside Press. Gibson, S. (1997). 3d chainmail: A fast algorithm for deforming volumetric objects. In Proc. Symp. on Interactive 3D Graphics (pp. 149–154). Providence, RI. Gibson, S., Samosky, J., Mor, A., Fyock, C., Grimson, E., Kanade, T., Kikinis, R., Lauer, H., McKenzie, N., Nakajima, S., Ohkami, H., Osborne, R., & Sawada, A. (1997). Simulating arthroscopic knee surgery using volumetric object representations, real-time volume rendering and haptic feedback. In Proc. CVRMed’97 (pp. 369–378). Springer-Verlag. Goertz, R. (1964). Manipulator systems development at ANL. In Proc. 12th Conf. Remote Systems Techn., 12 (pp. 117–136). ANS. Gruttmann, F., & Taylor, R. (1992). Theory and finite element formulation of rubberlike membrane shells using principal stretches. International Journal for Numerical Methods in Engineering, 35, 1111–1126. Haber, R., & Henderson, M. (1980). The Psychology of Visual Perception (2nd ed.). New York: Holt, Rhinehart and Winston. Hahn, J., Kaufman, R., Winick, A., Carleton, T., Park, Y., Lindeman, R., Oh, K.-M., Al-Ghreimil, N., Walsh, R., Loew, M., Gerber, J., & Sankar, S. (1998). Training environment for inferior vena caval filter placement. In Proc. MMVR’98 (pp. 291–297). IOS Press. Hinton, E. (1992). NAFEMS—Introduction to Nonlinear Finite Element Analysis. Glasgow: Bell and Bain, Ltd. Ho¨hne, K., Pflesser, B., Pommert, A., Riemer, M., Schubert, R., & Tiede, U. (1995). A new representation of knowledge concerning human anatomy and function. Nature Med., 1(6), 506–511. HT Medical. http://www.ht.com. Hug, J., Brechbu¨hler, C., & Sze´kely, G. (1999). Tamed snake: A particle system for robust semi-automatic segmentation. In C. Taylor & A. Colchester (Eds.), Second International Conference on Medical Image Computing and ComputerAssisted Intervention MICCAI’99 (pp. 106–115). No. 1679 in Lecture Notes in Computer Science. Springer-Verlag. Hutter, R. (1999). Total hourglass control—Eine robuste FE

Sze´kely et al.

Methode zur Simulation von weichen Geweben. Unpublished doctoral dissertation, Department of Mechanical Engineering, ETH Zu¨rich. Hutter, R., Hora, P., & Niederer, P. (1999). Total hourglass control for hyperelastic materials. Computer Methods in Applied Mechanics and Engineering. (Accepted for publication.) Immersion, I. (1995). Laparoscopic impulse engine. Company brochure. Santa Clara, CA. Kass, M., Witkin, A., & Terzopoulos, D. (1988). Snakes: Active contour models. Int. J. Comp. Vision, 1(4), 321–331. Kauer, M., Vuskovic, V., & Dual, J. (1999). In vivo measurements of elastomechanical properties of organic tissue. In European Medical and Biological Engineering Conference, EMBEC’99. Vienna, Austria. Kelemen, A. (1998). Elastic Model-Based Segmentation of 2-D and 3-D Neuroradiological Data Sets. Unpublished doctoral dissertation No. 12907, Department of Electrical Engineering, ETH Zu¨rich. Kelemen, A., Sze´kely, G., & Gerig, G. (2000). Elastic modelbased segmentation of 3-D neuroradiological data sets. IEEE Trans. Medical Imaging. (In press.) Knapp, D., Kerr, J., & Sellberg, M. (1997). Patient specific color texture mapping of ct-based anatomical surface models utilizing cryosectional data. In Medicine Meets Virtual Reality (pp. 608–617). IOS Press. Ku ¨ hnapfel, U., and C ¸ akmak, H., & Maaß, H. (1999). Modeling for endoscopic surgery. In Proceedings of the IEEE Symposium on Simulation. Delft, The Netherlands. Ku ¨ hnapfel, U., Krumm, H., Kuhn, C., Hu¨bner, M., & Neisius, B. (1995). Endosurgery simulations with kismet: A flexible tool for surgical instrument design, operation room planning and VR technology based abdominal surgery training. In B. Gro ¨ ttrup (Ed.), Proc. Virtual reality World’95, Stuttgart (pp. 165–171). Mu¨nchen: Computerwoche Verlag. Kyriacou, S., Schwab, C., & Humphrey, J. (1996). Finite element analysis of nonlinear orthotropic hyperelastic membranes. Computational Mechanics, 18, 269–278. Kyriacou, S., Shah, A., & Humphrey, J. (1997). Inverse finite element characterization of nonlinear hyperelastic membranes. Journal of Applied Mechanics, 64, 257–262. Lewis, J. (1989). Algorithms for solid noise synthesis. In Proc. SIGGRAPH’89 (pp. 263–270). New York: ACM Press. Maaß, H., & Ku¨hnapfel, U. (1999). Noninvasive measurement of elastic properties of living tissue. In Proceedings of the 13th

331

Internat. Congress on Computer Assisted Radiology and Surgery, CARS ’99 (pp. 865–870). Paris, France. Massie, T., & Salisbury, J. (1994). The PHANToM haptic interface: A device for probing virtual objects. In C. Radcliffe (Ed.), Dynamic Systems and Control, 1 (pp. 295–301). ASME. Meier, V. (1999). Realistic visualization of abdominal organs and its application in laparoscopic surgery simulation. Unpublished doctoral dissertation, Diss. ETH No. 13215, Department of Electrical Engineering, ETH Zu ¨ rich. Miller, G. (1988). The motion dynamics of snakes and worms. Computer Graphics, 22(4), 169–178. Molnar, S., Eyles, J., & Poulton, J. (1992). Pixelflow: Highspeed rendering using image composition. In Proc. SIGGRAPH’92 (pp. 231–240). New York: ACM Press. Moulton, M. (1995). An inverse approach to determining myocardial material properties. J. Biomechanics, 28, 935– 948. MusculoGraphics Inc. http://www.musculographics.com. MusculoGraphic’s Limb Trauma Simulator. MusculoGraphics, Inc. http://www.musculographics.com/lts.htm. Muthupillai, R., Rossman, P., Lomas, D., Greanleaf, J., Riederer, S., & Ehman, R. (1996). Magnetic resonance imaging of transverse acoustic strain waves. Magnetic Resonance in Medicine, 36, 266–274. National Library of Medicine. The Visible Human Project. http://www.nlm.nih.gov/research/visible/visible_human.html. Oden, T., & Key, J. (1970). Numerical analysis of finite axisymmetric deformations of incompressible elastic solids of revolution. Int. J. Solids Structures, 6, 497–518. Ogden, R. (1994). Nonlinear Elastic Deformations. Ellis Horwood Limited. Optimist. (Handelsgesellschaft mbH). Pop trainer. Company prospect, A-6900 Bregenz. Peachey, D. (1985). Solid texturing of complex surfaces. In Proc. SIGGRAPH’85 (pp. 279–286). New York: ACM Press. Perlin, K. (1985). An image synthesizer. In Proc. SIGGRAPH’85 (pp. 287–296). New York: ACM Press. Pflesser, B., Tiede, U., & Ho¨hne, K. (1998). Specification, modelling, and visualization of arbitrarily shaped cut surfaces in the volume model. In W. M. Wells, A. Colchester, & S. Delp (Eds.), First International Conference on Medical Image Computing and Computer-Assisted Intervention MICCAI’98 (pp. 853–860). No 1496 in Lecture Notes in Computer Science). Springer Verlag.

332


Pommerell, C. (1992). Solution of large unsymmetric systems of linear equations. Unpublished doctoral dissertation, Department of Electrical Engineering, ETH, Zu¨rich. Prusinkiewicz, P., & Lindenmayer, A. (1990). The algorithmic beauty of plants. New York: Springer-Verlag. Reinig, K., Rush, C., Pelster, H. L., Spitzer, V. M., & Heath, J.

Adelstein (Eds.), Adv. in Robotics, Mechatronics and Haptic Interfaces, DSC-49 (pp. 47–54). ASME. Sederberg, T., & Parry, S. (1986). Free-form deformation of solid geometric models. Computer Graphics, 20(4), 151– 160. Sellberg, M., Murray, D., Knapp, D., Teske, T., Lattie, K., &

(1996a). Real-time visually and haptically accurate surgical

Vanderploeg, M. (1995). Virtual human: An automated vir-

simulation. In Proc. MMVR’96 (pp. 542–545). IOS Press.

tual environment for computer-aided instruction and bio-

Reinig, K., Rush, C., Pelster, H., Spitzer, V., & Heath, J. (1996b). Real-time visually and haptically accurate surgical simulation. In Health Care in the Information Age (pp. 542–545). IOS Press and Ohmsha. Rhomberg, A., Enzler, R., Thaler, M., & Tro ¨ ster, G. (1998). Design of a FEM computation engine for real-time laparoscopic surgery simulation. In Proceedings of the First Merged International Parallel Processing Symposium and Symposium on Parallel and Distributed Processing (pp. 711–715). Los Alamitos, CA: IEEE Comput. Soc. Rhomberg, A., Brechbu¨hler, C., Sze´kely, G., & Tro¨ster, G. (1999). A parallel architecture for interactive fem computa-

mechanical analysis. In Interactive Technology and the New Paradigm for Healthcare (pp. 340–348). IOS Press and Ohmsha. Sensable. SensAble Technologies. http://www.sensable.com. SensAble Devices, I. (1996). PHANToM. Company brochure, Cambridge, MA. Storz, K. (GmbH). Endoscopy. Company catalog, D-78532 Tuttlingen, Germany. Stredney, D., Sessana, D., McDonald, J., Hiemenz, L., & Rosenberg, L. (1996). A virtual simulation environment for learning epidural anesthesia. In Proc. MMVR’96 (pp. 164– 175). IOS Press.

tions in a surgery simulator. In Proceedings of the Parallel

Sussman, T., & Bathe, K. (1987). A finite element formulation

Computing conference, ParCo99. Delft, The Netherlands:

for nonlinear incompressible elastic and inelastic analysis.

The Non-profit Foundation ParCo Conferences. (Accepted for publication.) Rivlin, R., & Saunders, D. (1951). Large elastic deformations

Computers and Structures, 26(1/2), 357–409. Suzuki, N., Hattori, A., Ezumi, T., Uchiyama, A., Kumano, T., Ikamoto, A., Adachi, Y., & Takatsu, A. (1998). Simulator

of isotropic materials. vii experiments on the deformation of

for virtual surgery using deformable organ models and force

rubber. Philosophical Transactions of the Royal Society, 243,

feedback system. In Proc. MMVR’98 (pp. 227–233). IOS

251–288. Series A. Ross, M., Twombly, A., Lee, A., Cheng, R., & Senger, S.

Press. Sze´kely, G., Bajka, M., Hug, J., Manestar, M., Groscurth, P.,

(1999). New approaches to virtual environment surgery. In

& Haller, U. (1998a). Anatomical model generation for lap-

Proc. MMVR’99 (pp. 297–301). IOS Press.

aroscopic surgery simulation. In Proc. 2nd Visible Human

Sagar, M., Bullivant, D., Mallinson, G., Hunter, P., & Hunter, I. (1994). A virtual environment and model of the eye for surgical simulation. Comp. Graphics, 28, 205–212. Satava, R. (1996). Virtual endoscopy: Diagnosis using 3-D visualisation and virtual representation. Surgical Endoscopy, 10, 173–174. Schill, M., Wagner, C., Hennen, M., Bender, H.-J., & Ma¨nner, R. (1999). Eyesi—a simulator for intra-ocular surgery. In C. Taylor & A. Colchester (Eds.), Second International Conference on Medical Image Computing and Computer-Assisted Intervention MIC-CAI’99 (pp. 1166–1174). No 1679 in Lecture Notes in Computer Science). Springer Verlag. Schmult, B., & Jebens, R. (1993). Application areas for a force-feedback joystick. In H. Kazerooni, J. Colgates, & B.

Project Conf. Bethesda, MA: National Library of Medicine. ———. (1998b). Anatomical model generation for laparoscopic surgery simulation. In Abstracts of the 2nd Visible Human Project Conf. (pp. 45–46). Bethesda, MA: National Library of Medicine. Sze´kely, G., Brechbu ¨ hler, C., Hutter, R., Rhomberg, A., Ironmonger, N., & Schmid, P. (2000). Modelling of soft tissue deformation for laparoscopic surgery simulation. Medical Image Analysis. (In press.) Sze´kely, G., Brechbu ¨ hler, C., Hutter, R., Rhomberg, A., & Schmid, P. (1998c). Modelling of soft tissue deformation for laparoscopic surgery simulation. In W. M. Wells, A. Colchester, & S. Delp (Eds.), First International Conference on Medical Image Computing and Computer-Assisted Interven-

Sze´kely et al.

tion MICCAI’98 (pp. 550–561). No. 1496 in Lecture Notes in Computer Science. Springer-Verlag. Sze´kely, G. & Gerig, G. (2000). Model-based segmentation of radiological images. Kuenstliche Intelligenz, Special Issue on Medical Image Analysis. (In press). Taylor, V. (1991). Application-specific architectures for large finite-element applications. Unpublished doctoral dissertation, Dept. El. Eng. and Comp. Sci., Univ. of California, Berkeley, CA. Terzopoulos, D., Platt, J., Barr, A., & Fleischer, K. (1987). Elastically deformable models. Computer Graphics, 21(4), 205–214. Truesdell, C., & Noll, W. (1965). The nonlinear field theories of mechanics. In Encyclopedia of Physics (pp. 1–579). Berlin: Springer-Verlag. Tseng, T., Lee, Y., Chan, Y., Wu, S., & Chiu, A. (1998). A PCbased surgical simulator for laparoscopic surgery. In Proc. MMVR’98 (pp. 155–160). IOS Press. Vining, D. (1996). Virtual endoscopy: Is it really? Radiology, 200, 30–31.

333

Virtual Presence, Inc. http://www.vrweb.com. Voß, G., Hahn, J., Mu¨ller, W., & Lindeman, R. (1999). Virtual cutting of anatomical structures. In Proc. MMVR’99 (pp. 381–383). IOS Press. Voxar, Ltd. http://www.voxar.com. Vuskovic, V., Blaser, R., & Spiga, A. (1999). A vision-based device for in vivo measurements of elastomechanical properties of soft tissue. In Proc. Workshop on Image Processing in Applied Mechanics. Warsaw, Poland. Watt, A. (1989). Fundamentals of Three-Dimensional Computer Graphics. Workingham, England: Addison-Wesley Publishing Co. Ziegler, R., Mueller, W., Fischer, G., & Goebel, M. (1995). A virtual reality medical training system. In Proc. 1st In. Conf. on Comp. Vision, Virtual Reality and Robotics in Medicine, CVRMed’95, Nice, Lecture Notes in Comp. Sci., (vol. 905, pp. 282–286). Berlin: Springer. Zienkiewicz, O., & Taylor, R. (1994). The Finite Element Method. London: McGraw-Hill Book Company.