An Extendable Application Framework for - Semantic Scholar

Medical Imaging 2001

San Diego, CA, February 17-23, 2001

An Extendable Application Framework for Medical Visualization and Surgical Planning Thomas Jansen, Bartosz von Rymon-Lipinski, Zdzislaw Krol, Lutz Ritter, Erwin Keeve Surgical Simulation and Navigation Group, research center c a e s a r - center of advanced european studies and research {jansen, lipinski, krol, ritter, keeve}@caesar.de, Friedensplatz 16, 53111 Bonn, Germany phone +49-228-9656-0; fax +49-228-9656-111, http://www.caesar.de/surgicalsimulationandnavigation

ABSTRACT This paper introduces an extendable cross-platform software framework Julius for medical visualization and surgical planning, consisting of two conceptual layers: the Julius Software Development Kit (JSDK) and the Julius Graphical User Interface (JGUI). The JSDK can be used stand-alone to speed up development of research tools. While the JGUI acts like a frontend for the JSDK and offers easy handling combined with time-saving functionality to increase performance and productivity. Julius features a modular, cross-platform design and comes with a full set of components, like semi-automatic segmentation, registration, visualization and navigation. Keywords: surgical planning, intra-operative guidance, segmentation, registration, visualization, navigation

1. INTRODUCTION Image-guided surgery became indispensably over the last years. The vision of reality is enhanced, using information from CT, MR and other medical scans. The surgeon is helped in his every-day work and new intervention techniques are opened. Introduction of new hardware and break-throughs in algorithmic work reduces limitations of conventional image-guided surgery, such as time-consuming calculations during the operation. One major challenge is to bring the various facets of medical visualization together into one single environment. This is required for seamless integration of pre-operative and intra-operative data which enables the surgeon to work with the same system before and during the operation. Furthermore the researcher should be able to extend the framework with new algorithms and techniques, even if they are in an experimental state. The environment should offer this expandability in an easy way to speed up the software process. It should also be easy to use the system for non-experts and simplify tasks even in critical situations during surgery. Finally the system has to be platform independent, and should run on a mid-range configuration as stable as on a high-end graphics workstation. The software framework Julius reaches these goals. It can be used as a general software framework for rapid application development, in visualization and in testing new research results. Furthermore the experimental evaluation of such research outcome, Julius can be used as an application for pre-operative planning and intra-operative guidance. Therefore the software package can be seen from a programmers or a users point of view. This separation reflects the internal structure of Julius, where the frontend gives the power on the users fingertips, while the developer can extend the core of the framework. Plug-ins implement cutting edge data manipulation, visualization or navigation techniques and will be added to Julius without recompilation. This can be done at the same time; new developed modules can be used by novice users in the well-known graphical user interface. Because of a well documented plug-in concept, third-party components will be available soon, which let the user choose from a variety of modules. Jansen et al.

An Extendable Application Framework for Medical Visualization and Surgical Planning



2. RELATED WORK Several other software systems similar to Julius exist. ANALYZE has been developed at the Mayo Clinic since the early 1970's, and offers a vast set of tools.1 3D Slicer from Brigham and Women's Hospital, Boston and software projects from Carnegie Mellon University and others are available.2, 3, 6-8, 10 However, Julius provides the whole medical imaging pipeline including semi-automatic segmentation, automatic registration, generation of 3D surface models, 3D visualization as well as biomechanical modeling. Therefore Julius can be used as a teaching, pre-operative planning and intra-operative guidance tool. This makes Julius a system, which covers numerous aspects of image-guided surgery and integrates them into a single environment.

3. METHODS Researchers can use Julius to implement or validate new algorithms, surgeons can use it for planning and intra-operative guidance. All elements of the application are represented by C++ classes. This modular approach ensures simple extendability and platform-independence. Therefore Julius is a software framework, rather than a static software system, which offers many plug-in-points. New modules can be easily incorporated into the system via well documented interfaces even by third-party developers.

b

a c

Figure 1: Screenshot of Julius. (a) the Global Bar, (b) the dataset visualized using volume rendering, (c) slice view and reconstructed standard cross-sections.

Jansen et al.

An Extendable Application Framework for Medical Visualization and Surgical Planning 2/9



Julius consists of two conceptual layers: The Julius Software Development Kit (JSDK) and the Julius Graphical User Interface (JGUI). The JSDK is the core of the application and controls the data processing pipeline. The JGUI is a front-end based on JSDK and provides an easy and full-featured interaction for the user (see Figure 1). Both layers are implemented on Windows and SGI workstations using the object-oriented graphics library “Visualization Toolkit” (VTK) and the GUI toolkit Qt for visualization and user interaction respectively.15-17

3.1. Julius Software Development Kit – JSDK The JSDK contains the central subset of classes, which control the data processing pipeline. Two types of classes coexist: The utility classes form simple tools and basic abstract data types (i.e. arrays and stacks). The core classes are the main building blocks of the JSDK, used to create the medical processing pipeline. The pipeline consists mainly of the following parts: data import / data export, semi-automatic segmentation and registration. In the following two subsections we present the data pipeline in detail and discuss how to manage and access the data. Some examples at the end of each subsection illustrate the presented concept. The last subsection in this chapter deals with information exchange between objects within the application framework. 3.1.1. Data modules and the data pipeline The main elements of the JSDK are the data modules. Each module is a C++ object that encapsulates volume and/or polygon data and provides interfaces for communication with other modules as well as for interaction within the data processing pipeline. The modules and the data access between them are organized in a distributed system, where each module has read-only-access to the data of any other module, and therefore can only modify its own data. The processing order is organized through a module stack. It contains references to modules and describes the current state of the data process. In our concept, each module can be seen as a “basic stack operation” executed within the data pipeline comprising of the following steps: •

The new module gets the reference pointing to the old module on the top position of the stack.

•

The new module reads the data of the old module.

•

The old module is removed from the stack (“pop” stack operation, optional).

•

The new module modifies the data or creates a new one.

•

The reference to the new module is stored on the stack (“push” stack operation).

A module, referencing multiple other data modules, can be realized with several “pop”- and “exchange”-stackoperations. Each single module is responsible for the correct data processing and describes the protocol of the whole data pipeline. Example 1: Medical data may be imported in various format, such as the DICOM 3.0 file format.12 The current version of this codec reads complete CT and MRI datasets from various manufacturers and transforms them into volume data for further processing. This is the first stage for segmentation, registration, visualization and navigation. One challenge was taking gantry angle of a CT scan into account. A “gantry shifting transformation” of individual slices has to be performed coupled with a filtering operation to achieve a correct 3D volume without distortion. Another related feature will be the support of the DICOM communication protocol. Users have access to a DICOM server via network and can request DICOM datasets by a simple query to the associated database. This tool will be integrated completely into the JGUI and will provide an easy daily access to volume data. 3.1.2. Accessing the data pipeline little endian Programming and automation of the pipeline organization can be easily realized by a simple script language, called the Julius Pipeline Script – JPS. Run-time loading of scripts enables the advanced users to configure Julius in a flexible way.

Jansen et al.




#NEW(“DATAIMPORT:DICOM”); Load(“DATA\SAMPLE.DCM”);

// Create DICOM codec // Load the volume data

#NEW(“SEGMENTATION:THRESHOLD”); ThresholdBetween(200.0, 4000.0); Execute();

// Create threshold segmentation module // Set threshold parameters // Perform segmentation

#NEW(“TRIANGULATION:MCUBE”); Execute();

// Create a Marching Cubes module // Perform triangulation

Figure 2:

Example JPS code representing a simple data processing pipeline. Medical volume data is imported, a threshold segmentation performed and finally a polygon mesh generated.

Optional processing can be done by simple interaction with the JGUI – in particular with the Global Bar (to be explained later). The script language is based on the module concept and offers the following basic operations: •

Creation of a new module reference, which will be placed on the stack.

•

Configuration of the current module via module functions.

•

Execution of the encapsulated algorithms of the current module.

Figure 2 illustrates the JPS concept and shows an example code that loads a DICOM 3.0 file, performs a threshold segmentation and generates a polygon mesh via the Marching Cubes algorithm. This short fragment represents a simple data processing pipeline, where volume data is imported and translated into polygon data. After execution of this script the resulting pipeline stack contains three elements, represented in Figure 3. Simple functions of the JSDK enable developers to easily acces the current volume or polygon data on the stack. Additionally, modules can be accessed via the pipeline class identifier. As shown in Figure 2, each module is associated with unique identifier, which describes the category of a specific module. The threshold module belongs to the segmentation class and has therefore the “SEGMENTATION:THRESHOLD” identifier. Data pipeline Triangulation module Polygon mesh Module stack Marching Cubes triangulation Threshold module Threshold segmentation

Segmented volume data

DICOM 3.0 codec DICOM 3.0 codec module Imported volume data

Figure 3: Module stack containing references to data modules (represented by thin arrows). Each module encapsulates data. Thick arrows represent the data access and therefore the inverse processing order of the data pipeline.

Jansen et al.




The deallocation of the module data will be realized using “garbage collection”. For example, if the data of a module is no longer needed during the pipeline process or is not referenced by the pipeline stack anymore, it is released by the JSDK. This system can be extended by an intelligent cache system, swapping data from main memory to the hard disk and therefore providing efficient memory allocation. Example 2: Another feature of Julius is the triangulation, which is implemented in the following way: The process begins with segmented volume data containing different labels. These labels are simple numeric values, that are generated during the previous segmentation process and are used for identification of different anatomical structures. The triangulation process is based on the paper from Bill Lorensen and can be implemented using two data modules:11 • Tissue extraction module: This module uses volume data as input and output. In the first step we use an image threshold filter. This filter extracts a tissue with a specific label. At this point our volume contains only binary values, which associate voxels with certain tissue type. In the second step, we use a Gaussian smooth filter to blur the images and to prevent stepped surfaces. • Triangulation module: This module uses volume data as input and generates polygon data for output. The basic step is the application of the Marching Cubes algorithm. Triangle decimation is done afterwards, since many triangles are generated by the algorithm but are not needed for rendering. Smoothing of the generated surface can be achieved by employing Laplacian filtering. Most of the preceding steps are implemented using VTK classes. Users have the possibility to implement the same triangulation process with a different number of modules. This feature is a result of the principle of generality as an implication from the modular concept of the JSDK. 3.1.3. Julius Inter-Module Communication – JIMC The JIMC is a system for information exchange between distributed objects (such as modules) within Julius. Through messages, developers can attach data to each message and select between different addressing modes. Sending messages is done by accessing the global information node from a Julius object (see Figure 4). The following addressing modes are supported: •

Direct addressing: The target object is addressed directly via C++ pointer. This mode ensures best performance and has it application in time-critical operations, i.e. submission of navigation data from a haptic interface.

Julius modules

Julius objects

External information source

Global information node Message distribution

Message listener

Figure 4: Julius Inter-Module Communication. Arrows represent the access order between objects.

Jansen et al.




•

Multicast addressing: This mode has several target objects, whereas the addressing is realized via class names. It is useful i.e. for the information exchange between the Global Bar und view modules.

•

Broadcast addressing: No target object or module has to be specified. The appropriate message is sent to all valid targets and is convenient for system or status information.

Each object within Julius is able to send messages, but it can receive them only when the listener interface is implemented. Therefore, no redundant targets have to be considered during the information distribution process. An important example for the fast direct addressing is a visual tracking system module (i.e. Flashpoint 5000 18), which generates navigation data in real-time. For this type of data transmission, the sender – in this case the tracking system – is implemented as an external information-source, which is connected to the global information node. All Julius objects can additionally deactivate the data transmission for a certain period of time to prevent message overflow.

3.2. The Julius Graphical User Interface – JGUI Using the JSDK as a stand-alone solution will not be the common case for the end-user. Therefore a fully featured graphical user interface – called JGUI – is available. Elements of the GUI are implemented using the Qt library from Trolltech, while 3D visualization will be done using the “Visualization Toolkit” (VTK) from KitWare. 15-17 Because both software kits are platform-independent, this results in Julius being a cross-platform software framework. Every aspect of the user interface is extendable using the plug-in concept of Julius. Common “add-ins” will be shipped with the framework and third-party modules can be “hot plugged” afterwards without recompilation. Plug-in interfaces are provided for the following categories: •

File format codec for data import and export of volume or polygon objects (JSDK).

•

Data operations and algorithms for any kind of volume or polygon manipulation (JSDK).

•

File format dialog for import / export formats (JGUI).

•

Global Bar modules (JGUI).

•

Visualization techniques (JGUI).

•

Navigation and tracking systems (JGUI).

3.2.1. File format dialog extensions Every file format dialog extension is associated with an import/export data module as mentioned in chapter 3.1.1. The dialog is shown when the user requests loading new data in Julius. Example 3: An example is the DICOM 3.0 file format dialog add-in (see Figure 5), which is connected to the DICOM 3.0 import/export module. The dialog shows information of the selected dataset. Thumbnails of selected files will be shown beside in-depth information, such as patient name or modality. File format dialog add-ins are also available for the VTK and JPS file format. They can be easily developed with little knowledge of the Qt library.

Jansen et al.

Figure 5: The file dialog shows detailed information about the selected file. This feature is extendable to support other formats than the basic ones, such as VTK, DICOM3.0 or script files.




3.2.2. Global Bar extensions A central part of the JGUI is the Global Bar, which is visible all the time. Every main task – such as semi-automatic segmentation or registration – will correspond to a Global Bar add-in which handles most of the interaction. Example 4: A fundamental add-in for the Global Bar will be the anatomy browser (which can be seen on the left side of Figure 1). Several anatomy datasets will be available in a database, representing different parts of the human body and different modalities. The browser displays the anatomical structures in a hierarchical view with additional medical information. If a segmented volume is available, selection of a structure name will highlight the corresponding part of the volume with the appropriate color, and respectively selecting a part in the volume data will highlight the name in the hierarchical view. The main application of the anatomy browser will be segmentation. The user has to select the name of a structure, which toggles the segmentation module into the segmentation mode. Next, the part of the volume dataset – that is associated with this anatomical structure – will be segmented, using techniques such as region growing, snakes or freehand lasso. 3.2.3. Visualization extensions The observer design pattern is used in Julius:5 Several data types (volume or polygon) are visualized, using different addins. Obvious examples for this concept are volume rendering and mesh visualization of the same dataset. Even histogram computation fits into this pattern (see Figure 6). To preserve the platform-independence of Julius, hardware accelerated rendering will be done using VTK. The thin Julius Visualization Layer (JVL) is conceptually on top of the native VTK classes, providing additional functionality. Add-ins call commands of all available visualization packages (see Figure 7). Example 5: One of the visualization add-ins is the volume rendering view module. On Microsoft Windows, Solaris, Irix and Linux, the complete volume rendering is done using available graphics hardware – such as the VolumePro from RTViz.18 This hardware provides visualization of a 256x256x256 voxel set with full gradient calculation in real-time (30 frames per second) using the optimized CUBE4 raycasting algorithm:13 Other hardware acceleration, like the InfiniteReality engine from SGI will also be supported. If no hardware support is available, all volume data visualization will be done in software using a raycasting algorithm.

3.2.4. Navigation extensions Navigation is important for pre-operative simulation and intra-operative guidance. Several modules implementing haptic interfaces and optical tracking systems will be available soon, such as add-ins for the PHANToM haptic interface by SensAble Technologies or the FlashPoint 5000 Optical Tracking system by Image Guided Technologies.4, 14 Julius will support force feedback functionality (like the one provided by the PHANToM).

(a)

(b)

(c)

Figure 6: Different views of the same volume dataset; (a) volume rendering using the VolumePro graphics hardware, (b) surface rendering of the polygon mesh generated by the Marching Cubes algorithm, (c) the full main view of Julius, including the volume slices and the histogram window.

Jansen et al.




Julius Graphical User Interface – JGUI

Add-In

Julius Visualization Layer – JVL Visualization Toolkit – VTK Open Graphics Library - OpenGL

Julius Software Development Kit JSDK

Figure 7: Hierarchy of visualization packages that can be accessed by an add-in.

3.2.5. Slice rendering Visualization of slices through a volume dataset is crucial in Julius. These views corresponds to the classic transversal, coronal and sagittal views, as well as for arbitrary “cut planes” in the 3D view. All positions and orientations will be handled, which enables visual feedback during navigation, e.g. with a haptic interface. The slice view will support multiple cuts through multiple volumes in parallel, overlayed segmented information and merged volume data of several sources and modalities (as used in registration). Speed is mandatory to provide real-time slice visualization, especially for many slices and large volume datasets. VTK does not support hardware accelerated 3D texture mapping at the moment. Therefore the JVL adds this functionality. It calls native OpenGL commands to use available 3D texture mapping extensions. Modern graphics hardware (e.g. Wildcat graphic board from 3Dlabs Inc.19) provides this feature. If hardware acceleration is not available, real-time resampling of the volume data is done using software rendering.

4. RESULTS Julius was developed to achieve best results in both areas: application and research. This reflects the separation between the JSDK for developers and the JGUI for the end-users. The most important reasons for faster development and integration of new algorithms into Julius via JSDK are: • • • •

Standard algorithms are available in Julius and can be reused by new modules. Results of new algorithms can be verified fast, which makes their development easier. The JSDK offers built-in debugging tools, such as bounds checking and dump of pipeline states. The scalability of Julius enables modules to run on every configuration using all its benefits.

The most important reasons for increased productivity while using Julius via JGUI are: • • • •

Julius can be handled the same way on all supported operating systems. One environment for pre-operative planning and intra-operative guidance. Wizards and scripts help starting common tasks with a single mouse click. The Global Bar acts as a “headquarter” and shows useful hints and shortcuts in daily work.

Many other advantages arise from Julius, such as easy implementation of navigation hardware by reusing navigation modules. Experiments have shown that development of new algorithms within the Julius software framework give a noticeable speed up. An example for this fact is the area of computer aided osteotomy design.9

Jansen et al.




The purpose in Julius development was to create a powerful, intuitive and easy-to-use software product for novice and advanced users. Important elements of this concept are the developer-friendly JSDK and the user-friendly JGUI. This design makes Julius both, an expandable experimenting platform for new medical algorithms and a software system for pre- and intra-operative application.

5. CONCLUSION In this paper, we have described an extendable software framework for medical visualization. The system will undergo clinical evaluation by summer of 2001. It features a modular and portable design and will come with a full set of modules for semi-automatic segmentation and registration, visualization and navigation. Its extendability will allow the system to increase its value for pre-operative planning and intra-operative guidance and will make this software suite a complete solution for fully automated processing of medical data.

6. REFERENCES 1. 2.

3.

4. 5. 6. 7.

8.

9.

10. 11. 12. 13. 14. 15. 16. 17. 18. 19.

ANALYZE Software System from Mayo Clinic, http://www.mayo.edu/bir/Analyze_Pages/Analyze_Main_TOC.htm Birkfellner W., Solar P., Gahleitner A., Huber K., Kainberger F., Kettenbach H., Homolka P., Diemling M., Watzek G., Bergmann H., "Computer - Aided Implant Dentistry - An Early Report," Proceedings Medical Image Computing and Computer-Assisted Intervention; MICCAI'99, Cambridge, England, p. 883,1999. Blackwell M., Simon D., Rao S., and DiGioia A., "Design And Evaluation Of 3-D Pre-Operative Planning Software: Application To Acetabular Implant Alignment," Technical Report 44, Carnegie Mellon University, The Robotics Institute, 1996. Flashpoint 5000 3D Optical Localizer from Image Guided, http://www.imageguided.com/flashpoint.htm. Gamma E., Helm R., Johnson R., Vlissides J., “Design Patterns – Elements of Reusable Object-Oriented Software,” Addison Wesley Longman, 19th Printing, pp. 293-303, 2000. Gering D.T., "A System for Surgical Planning and Guidance using Image Fusion and Interventional MR," Master’s Thesis, Massachusetts Institute of Technology, 1999. Golland P., Kikinis R., Umans C., Halle M., Shenton M.E., Richolt J.A., "AnatomyBrowser: A Framework for Integration of Medical Information," In Proc. First International Conference on Medical Image Computing and Computer-Assisted Intervention; MICCAI'98, Cambridge, MA, pp. 720-731, 1998. Kansy K., Wisskirchen P., Behrens U., Berlage Th., Grunst G., Jahnke M., Ratering R., Schwarzmaier H.-J., Ulrich F., “LOCALITE - a Frameless Neuronavigation System for Interventional Magnetic Resonance Imaging Systems,” Proceedings Medical Image Computing and Computer-Assisted Intervention; MICCAI'99, Cambridge, England, pp. 832-841, 1999. Krol Z., Zerfass P., von Rymon-Lipinski B., Jansen T., Zeilhofer H., Sader R., Keeve E., "Computer Aided Osteotomy Design for Harvesting Autologous Bone Grafts in Reconstructive Surgery," Proceedings Medical Imaging, MI’00, San Diego, CA, 2000. Leventon M.E., "A Registration, Tracking, and Visualization System for Image-Guided Surgery," Master’s Thesis, Massachusetts Institute of Technology, 1997. Lorensen B., “Generating Triangle Files From Segmented Medical Images Using VTK 2.0,” http://www.crd.ge.com/~lorensen/seg12/. National Electrical Manufacturers Association, “Digital Imaging and Communications in Medicine (DICOM),” Rosslyn, VA 22209, PS 3.1-14, 1999. Pfister H., “Architectures for Real-Time Volume Rendering,” Master’s Thesis, State University of New York at Stony Brook, 1997. PHANToM haptic interface, SensAble Technologies, http://www.sensable.com/products/phantom.htm. Qt graphical user interface, Trolltech; http://www.trolltech.com/products/index.html. Schröder W., Martin K., Lorensen B., "The Visualization Toolkit - 2nd Edition," Prentice Hall PTR, New Jersey, 1998. Visualization Toolkit, KitWare Inc.; http://www.kitware.com/vtk.html. VolumePro 500 volume graphics accelerator from RTViz, http://www.rtviz.com/products/vp500.html. Wildcat graphics accelerator, Intense3D/3Dlabs, http://www.intense3d.com/brochure.html.

Jansen et al.
