A good application design is crucial for uncomplicated cooperation ... platform software framework for the development of medical ..... Reality, IOS Press, 1997.
1 of 4
Integrating Simulation Framework MEDIFRAME 1
S. Seifert1, R. Kussaether1, W. Henrich1, N. Voelzow1, R. Dillmann1
Department of Computer Design and Fault-Tolerance, University of Karlsruhe, Germany
Abstract—In this paper we will give an overview of our crossplatform software framework for the development of medical simulations in the field of surgical planning. We use state-ofthe-art technologies, like XML, CORBA and a scalable, component based architecture to build a highly extensible software development platform which provides the researcher with high-level services to build new simulation applications more easily. Keywords— Biomechanics, Geometric Modeling, Laserscanning, Segmentation, Simulation Framework, Surgery Planning
I. INTRODUCTION Today surgeons have to cope with several software tools to reach their goals. Every tool brings along its own lookand-feel and philosophy. Thus, we developed a software framework called MEDIFRAME which provides a consistent look-and-feel and is capable of integrating a wide range of planning tools - from medical image processing to complex simulations. Collaboration is provided by a shared data repository. Input devices, e.g. the PHANTOM, can be plugged-in. Our target groups are researchers interested in surgical planning tools and surgeons working on cardiac or maxillo-facial surgery. In this field we have been working for years within the German Sonderforschungsbereich 414, an interdisciplinary research initiative, dealing with implant planning, soft tissue modeling, computer-aided diagnosis and augmented reality. Successively we are integrating the respective components in MEDIFRAME. II. METHODOLOGY Generally 80% of total developmental time is used on building graphical user interfaces (GUI), e.g. dialog layout and exchange of data between the GUI and the variables in the program. These are frequent and monotonous tasks which can, however, easily be automated. Applications of any size, especially frameworks, which should support the development of several end-user applications, require a consistent architectural vision. Without this a rapidly growing application becomes soon unmanageable. Furthermore new technologies must be able to be incorporated without complete reimplementation. As a consequence we find it important for the development process to have a software design holding the goals subsequently described. First, code and design reuse accelerates and improves the quality of the development process. Beyond design, best practices have to be established and supported by the software.
A good application design is crucial for uncomplicated cooperation between software developers with different programming skills, such as researchers and students at university, as in our case. That’s why MEDIFRAME features developmental tasks isolated by skill sets. Furthermore parts of the application that change quickly require both greater eases of change and looser coupling to the rest of the system. Others that change more slowly can be coupled more tightly providing efficiency opportunities. Hence in MEDIFRAME classes with differing rates of change are decoupled. In addition, applications must be able to keep up with growing functionality and technology change. This extensibility mostly is the consequence of modularity. Modularity allows developers to work independently; it enhances maintainability and testability and permits to use 3rd party components. Last mentioned applications are easier to handle if the user is accustomed to a known presentation of information and handling. Thus we attached much importance to a straight look & feel. These requirements listed above are driven by the development process and are to the cooperation inside teams. Powerful medical applications make great demands on technological aspects. In MEDIFRAME we focused on the following technologies. 1) Heavy calculation load. Simulations are often very time-consuming. Therefore support for distributed computing is available. The physical location of the calculation is transparently hidden and is easily configurable. 2) Multi-Threading is necessary to take full advantage of multiple-processor boards, but it is often difficult to use for novices in parallel computing so we provided an easy-touse, but flexible API for parallelism. 3) Data Management. Saving and loading of simulation data should be extensible with regard to different file formats and additional components. In MEDIFRAME we realized an Import API which allows components to use import/export functionality and use import/export classes from other components. Furthermore the physical storage location is transparent for the developer, thus it is easy for administrators to change these data repositories or use repositories over WAN. Whole projects and sessions can be stored in files via an object-to- XML mapping. 4) Multiple devices and multiple data representation. Modern medical simulations must be able to cope with multiple devices and front-ends to meet all the users’ expecta-
2 of 4 tions. Therefore a simulation framework must supply a good infrastructure to plug-in devices and to present data in different manners. So we adopted the most modern software design principles from state-of-the art applications to build a very flexible software basis [8]. MEDIFRAME is written in C++ and uses the QT library for the user interface and the Visualization Toolkit (VTK) for all visualization tasks. To show all aspects of the architecture, it’s good practice to have a look at the architecture subdivided horizontally and vertically. Horizontal Subdivision The architecture is horizontally subdivided in components dynamic components (or plug-ins) based on dynamic libraries. We realized lazy binding to use a component only at runtime, i.e. not until a specific functionality is used. Dynamic Components are loaded by the Component-Manager on application start-up. A dedicated directory is scanned for these component files.
types [13]. Furthermore the delegation concept allows us to use the QT-designer intensively for faster development. However, the use of the designer and our own window class hierarchy requires a new data exchange mechanism. As a new data exchange mechanism between the presentation and the model layer we developed the bindings and actions concept to fulfil the requirements that came along with the use of the designer and our own class hierarchy. Bindings and actions externalize the internal state and the functionality of a MEDIFRAME object. Herewith views and models can plug-in automatically [9]. The DataModel class is derived from our model interface class and is specialized on VTK pipeline structures. RemoteModels can be called on other machines transparently via the MDCP1.
MainWindow, ApplicationModel, View, WindowView, ActorView,ViewProperty
MDCP
Application Mediframe
Other (WWW,PDA)
GUI
DataModel, PatientModel
RemoteModel
Core
MediBase, MediTools, MediApp, MediIO, MediDataTypes Persistence Manager
SimSurgery
OctreeMeshing
Corba
MirrorTools
MinoltaVivid
DownSample
Segmentation
ImageRegist
SimMandible
Dynamic
OO2XML
Repositories Fig. 2. The layered architecture of MEDIFRAME
III. RESULTS
Components
Fig. 1. The component architecture
Vertical Subdivision – Multi-Tiered Application The Model-View-Controller (MVC) design pattern provides a host of design benefits. MVC separates design concerns (data persistence and behavior, presentation, and control), decreasing code duplication, centralizing control, and making the application more easily modifiable. MVC also helps developers with different skill sets to focus on their core skills and collaborate through clearly defined interfaces. In MEDIFRAME we extended the MVC pattern to meet our needs as follows. Each component contains one or more application objects (ApplicationModel) which control a specific application context, i.e. it knows the process workflow of a medical task, it creates new windows and new data and it informs MEDIFRAME about the component’s potential actions. This class is the main entrance of any component. We built up our own window class hierarchy (WindowView, ActorView), with a shared base class (View) by using the delegation principle. Herewith we standardized QT widgets and VTK actors so that scene graphs can consist of both
Import/Export
In the following we want to present some examples of already implemented components. A) Laserscanning Component As an example for a data acquisition component we will have a look at the Laserscanning Component (Fig. 3) which directly interacts with the Minolta VIVID 900 digitizer. For studies in the field of reconstructive surgery, for validation of biomechanical models and for creating models in Virtual Reality scenes, the use of laser scanners has been established as a new technology for geometric modeling. With Minolta’s VIVID 900 digitizer a very powerful device came to the market. Since its best accuracy is 0.039 mm it is suitable for the smallest anatomical structures. However, one of the main difficulties in creating 3D models is the registration process of single scans and the filling of holes. For small objects a turntable improves this situation but you have to calibrate every change of the digitizer–turntable configura1
the MEDIFRAME Distributed Computing Platform
3 of 4 tion. If humans are scanned, the scans have to be merged by hand. With our software automatic laser scanning is possible without a-priori knowledge by using registration. Because the single scans overlap only in parts we use a variant of the classic ICP, the so-called Trimmed ICP (TrICP) [1]. This algorithm considers partially overlapping scans. Experiments with the classical ICP have shown that performance and robustness are not enough. After having registered the single scans they are further processed by a meshing algorithm which combines these two scans to one polygonal surface mesh. For further information concerning this component see [10].
segmented. To speed-up this method we use the ITK region growing method Confidence-Connected for a rough segmentation. Subsequently a snake with an initial mesh (provided by the classical marching cubes from the pre-segmented image) is started to optimize this segmentation. There are difficulties with snake-type methods: it is hard to keep track of contour if the contour self-intersects during its evolution or there are changes in topology. So we also integrated Sethian’s Level-Set methods, in details the FastMarching method (Fig. 4) and its extension the ShapeDetection method [4]. The Fast Marching method [12] introduced by Sethian (1996) and the Level Set method introduced by Osher and Sethian (1988) [6] are numerical techniques designed to track the evolution of interfaces. To use this algorithm you first have to set seed-points which define an intensity and homogeneity range, then these are used to compute a speed function to move the contour. Easily spoken, Level-Set (1) segmentation transforms the 3D segmentation problem to a 4D cut of a cone surface which is manipulated by a speed-function derived from the gradient of the original image: ∂Φ r + F ⋅ ∇Φ = 0 ∂t
(1)
Φ: level set function F: speed function; specifies how contour points move
Fig. 3. Front-end of the laserscanning component
B) Segmentation Component A frequent task in medical applications is to segment anatomical structures from CT or MRI images. The MEDIFRAME component for segmentation is based on the NLM Insight Segmentation and Registration Toolkit (ITK). Up to now several Region-Growing algorithms have been integrated e.g. Confidence-Connected Region Growing (Fig.4) which uses the mean value and standard deviation as homogeneity criterion to segment a region of the image. Another very powerful algorithm is called Isolated-Connected Region Growing with the possibility to define additional seedpoints where a certain distance should be kept. All Region Growing methods suffer from the fact that noisy images with interrupted boundaries often lead to a flow out. An improvement is to use the provided edge-preserving smoothing filter in advance called Anisotropic Diffusion Filter[7]. The Deformable Mesh Segmentation based on snakes first proposed by M. Kass [3] is another very useful method for the segmentation of very complex anatomical structures. This method starts with a closed contour which evolves to the anatomical boundaries. Inner forces preserve the contour from bending too much. Thus a well-shaped region can be
Fig. 4. Segmented heart; original image (left), Confidence Connected (middle), Fast Marching (right)
C) Three-Dimensional Meshing for Finite-Element-Method To built up three-dimensional models of human soft tissue and organs for the use in biomechanical calculations, we implemented a 3D octree-based meshing algorithm which is an extension of the “modified-octree approach” from Yerri and Shepard [14][5]. This robust and fast algorithm uses image-data or triangulated surfaces, e.g. from laserscans, as input and produces meshes of tetrahedrons for the use in the Finite Element Analysis (FEA). First, a hulling octree is created with typed octants (inner and boundary) which afterwards are meshed by templates. In total we use 15 base templates filled with tetrahedrons for the boundary. The inner octants are assembled by several base templates constrained by the octant’s neighborhood. Because of this template-based manner the algorithm is guaranteed to terminate and the mesh quality is determined by the optimized templates. This meshing algorithm is capable of remeshing after a cut of the model with a bilinear shape which we use as the representation of a surgical incision [11].
4 of 4
Fig. 6. The meshing kit of octants (left); remeshing the surgical incision areas (right)
modeled organ. For real-time interaction with the model for surgery planning we also implemented the mass-tensor approach [2]. With this we are able to adapt our biomechanical model quickly to topological changes invoked by virtual surgical incisions. The main idea of the mass-tensor model is to determine the stiffness tensor portion of each geometric primitive e.g. nodes, lines, triangles and tetrahedrons to the global stiffness matrix. Thus adaptation of the biomechanics means to add or to subtract the geometric primitives’ stiffness tensor portion to and from the global matrix. V. CONCLUSION
D) Biomechanical Modeling MEDIFRAME provides a component for real-time deformation based on a FEM-model (Fig. 7). We use tetrahedron based meshes and the St. Venant-Kirchhoff material to model the nonlinear, viscous behavior of soft tissue. The elastic energy W (2) of this nonlinear material is: W =
λ 2
(trE ) + µ ⋅ trE 2
With MEDIFRAME we present a reliable software basis for medical simulations. For developers it is very easy to integrate their code and to use the already available code. Many powerful features have been integrated so far which ease the creation of complex simulation scenarios. REFERENCES
2
)
1 E = (∇U + ∇U t + ∇U t ∇U ) 2
(2)
[1]
(3) [2]
With the Green-St.Venant strain tensor E and the nodedisplacement vector U. [3] [4]
[5] [6]
[7] [8] [9]
Fig. 7. Deformed test object consisting of St.Venant Kirchoff material
By the use of an integration scheme we model the dynamic deformation to show its viscous behavior, too. Other improvements are anisotropy (more exactly we are using transversal isotropy which provides different behavior in a given direction) and incompressibility, supported by additional energy terms for each tetrahedron to prevent the model from volume reduction. In case of organs with a stiff skin (like the liver capsule) this should be modeled at any rate. For this we provide a mass-spring layer around a FEM-
[10] [11] [12] [13] [14]
D. Chetverikov, D. Svirko, D. Stepanov, and P. Krsek, “The trimmed iterative closest point algorithm”, Proc. International Conference on Pattern Recognition, Quebec City, Canada, vol. 3, pp. 545-548, 2002. S. Cotin, H. Delingette, N. Ayache, “Real-time elastic deformations of soft tissues for surgery simulation”, IEEE Transactions on Visualization and Computer Graphics, vol. 5, no.1, pp. 6273, 1999. M. Kass, A. Witkin, and D. Terzopoulos, "Snake: active contour model", Int. J. Computer Vision, vol. 1, pp. 321-331, 1987. R. Malladi, J. A. Sethian, B. C. Vermuri., “Shape modeling with front propagation: A level set approach”, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 17, no. 2, pp. 158– 174, 1995. A. Mazura, S. Seifert, “Virtual cutting in medical data” Extract from the book: K. S. Morgan, Medicine Meets Virtual Reality, IOS Press, 1997. S. Osher, and J. Sethian, “Fronts propagating with curvaturedependent speed: Algorithms based on Hamilton-Jacobi equations”, Jour. Comp. Phys, vol. 79, pp. 12--49, 1988. P. Perona, and J. Malik, “Scale-space and edge detection using anisotropic diffusion”, IEEE Transactions on Pattern Analysis Machine Intelligence, vol. 12, pp. 629–639, 1990. S. Seifert, O. Burgert, and R. Dillmann, “MEDIFRAME - an extendable software framework for medical applications”, Surgetica, Grenoble, France, 2002. S. Seifert, R. Unterhinninghofen, and R. Dillmann, “MEDIFRAME – a rapid software development platform for medical simulations” unpublished S. Seifert, N. Voelzow, and R. Dillmann,, “Automatic 3D laser scanning for medical simulations” unpublished S. Seifert, R. Kussäther, R. Unterhinninghofen, O. Burgert, and R. Dillmann, „Smooth cutting in nonlinear soft tissue model“, CARS, UK, 2003, in press J.A. Sethian, “Level Set methods and Fast Marching Methods.”, Cambridge University Press, 1996. R. Unterhinninghofen, S. Seifert, and R. Dillmann, “A new architecture for scene-visualization in MEDIFRAME” unpublished M. A. Yerry, and M. S. Shepard, “Automatic three_dimensional mesh generation by the modified_octree technique”, International Journal of Numerical Methods in Engineering, vol 20, 1984.
