An extensible imaging platform for optical imaging ...

An extensible imaging platform for optical imaging applications Gianluca Paladini*, Fred Azar Siemens Corporate Research, 755 College Rd East, Princeton, NJ, USA 08540 ABSTRACT The National Institutes of Health (NIH) has recently developed an extensible imaging platform (XIP), a new open-source software development platform. XIP can be used to rapidly develop imaging applications designed to meet the needs of the optical imaging community. XIP is a state-of-the-art set of visual ‘drag and drop’ programming tools and associated libraries for rapid prototyping and application development. The tools include modules tailored for medical imaging, many of which are GPU hardware accelerated. They also provide a friendlier environment for utilizing popular toolkits such as ITK and VTK, and enable the visualization and processing of optical imaging data and standard DICOM data. XIP has built-in functionality for multidimensional data visualization and processing, and enables the development of independently optimized and re-usable software modules, which can be seamlessly added and interconnected to build advanced applications. XIP applications can run “stand alone”, including in client/server mode for remote access. XIP also supports the DICOM WG23 “Application Hosting” standard, which will enable plug-in XIP applications to run on any DICOM host workstation. Such interoperability will enable the optical imaging community to develop and deploy modular applications across all academic/clinical/industry partners with WG23 compliant imaging workstations. Keywords: extensible imaging platform, visual programming, optical, medical, visualization, open source

1. INTRODUCTION The Cancer Bioinformatics Grid (caBIG™) program was created in 2004 by the National Cancer Institute in order to develop a broad suite of software tools, IT infrastructure and applications to support and advance cancer research. As many as 900 individuals from academia, industry and more than 50 NCI-designated cancer centers have joined the caBIG™ initiative. In 2005, the In Vivo Imaging Workspace was added to promote the use of imaging in cancer clinical trials, with particular focus on open-source projects for image visualization and analysis. One of such projects is the eXtensible Imaging Platform (XIP) [1]. XIP is a modular application development environment that researchers can use to rapidly create research prototypes of medical imaging applications, and to easily transfer such prototypes to commercial imaging workstations thanks to an emerging DICOM standard for plug-in application hosting. It is open source and released under a very permissive BSD-like caBIG license that allows free academic and commercial use. A principal goal for XIP is to promote and facilitate interoperability between different software libraries and toolkits. Industry, academia and government have made considerable progress in providing the research community with widely available open source libraries and components from which to construct medical imaging applications. Notable examples are the Insight Segmentation and Registration Toolkit (ITK) [2][3], Visualization Toolkit (VTK) [4][5], and Offis DICOM Toolkit (DCMTK) [6][7]. However, each library has its own unique Application Programming Interface (API) and data structures that are not necessarily compatible with each other. To fully benefit from these advanced libraries and toolkits, what XIP provides for the research community is an extensible architecture that supports the easy integration of features from multiple libraries and provides interoperability between the large variety of data types and structures defined by each different library, with automatic format conversions if needed. Another important goal of the XIP project is to introduce a standardized approach for deploying imaging applications onto commercial medical imaging workstations from different vendors. Commercial clinical workstations, while generally based on common PC-based hardware, have vendor-proprietary libraries, databases and architectures that prevent a straightforward migration of research prototypes onto a commercial platform. Such research software typically requires to be re-engineered and re-implemented with the assistance of a specific vendor’s software development group, increasing costs and delaying the deployment of useful applications into clinical practice. The ability to distribute processing and analysis tools across diverse platforms in a timely manner would provide a uniformly high standard of *[email protected]; phone 1 609 734-6500; fax 1 609 734-6565; http://www.usa.siemens.com/en/research/

image quality to support a common approach to image-based diagnosis, staging and treatment. XIP provides the first reference implementation of an emerging DICOM standard for Application Hosting. In addition to providing a software framework for the development of imaging applications that facilitates extensibility and interoperability between software libraries and allows multi-vendor platform deployment, XIP also provides a modular Visual Programming integrated development environment (IDE) for rapid prototyping without the need to write and compile any code. The current implementation of XIP and its advantages are detailed in the following sections.

2. APPLICATION HOSTING STANDARD A major milestone toward interoperable systems for the medical imaging community was the creation in 1993 of DICOM version 3.0, which defined standardized formats and protocols to produce, store, display, process, send, retrieve, query and print medical images and derived structured documents in a multi-vendor environment. This was a key step for imaging vendors, and it continues to be extended and refined in order to fulfill the evolving needs of healthcare enterprises and their workflows. The DICOM Standard is updated each year through the work of hundreds of technical and medical experts, organized in more than 20 active DICOM working groups. In September 2003 the DICOM Strategic Advisory Working Group of the DICOM Standards Committee was approached with the proposal to extend the DICOM standard beyond the ability to exchange multi-vendor compatible medical data, in order to also exchange modular and multi-vendor compatible plug-in software applications capable of analyzing and visualizing such data. The proposal was received favorably, and over the next few months it was refined into a more detailed proposal that outlined a phased rollout of standardized plug-in interfaces, starting with basic interfaces, and expanding functionality and capabilities in later versions of the proposed standard. The DICOM Standards Committee chartered the “Application Hosting” Working Group N. 23 (WG-23) in December of 2004 to develop this proposed standard, which has the potential to revolutionize how the imaging software industry develops and deploys software applications [8][9]. Since then, various WG-23 participants from industry and academia have been defining interfaces that a host workstation would use in order to launch, exchange data with, and provide services to a plug-in application, as well as defining formats for the metadata being exchanged between a host and a plug-in. The NIH/NCI took notice of WG-23 shortly after it was formed, and when the caBIG™ XIP project was launched in November 2006 one of its requirements was to create a reference implementation of the emerging DICOM WG-23 host/plug-in interface. A draft of the first phase (basic interfaces) of the API and metadata WG-23 standard is detailed in DICOM Supplement 118. After ratification, the process for industry adoption and support will begin. The availability of the open-source XIP reference implementation for authoring DICOM compliant plug-ins should greatly speed its adoption process.

3. XIP ARCHITECTURE Put some additional info here 3.1 XIP Libraries XIP Libraries are based on Open Inventor® [10], an open source object-oriented C++ toolkit developed by Silicon Graphics Inc. (Sgi®) which encapsulates OpenGL, thereby allowing users to develop computer graphics applications without having to program lower-level aspects of rendering. Open Inventor provides a set of modifiable and extensible objects which can be used to describe and control a three-dimensional scene. These objects include basic primitives like polygons, text, materials, cameras, lights, viewers, manipulators used for interactive visualization and editing. The following are a few of the reasons for using Open Inventor in XIP: •

Open Inventor’s is very well documented by Sgi®. There are very detailed books that describe in detail how to use [11][12] as well as how to extend [13] Open Inventor, thereby reducing training time for XIP developers.

•

It is based upon the concept of a Scene Graph, where objects called Nodes are connected together in a hierarchical tree. This concept is well known and understood by software developers with a computer graphics background. Open Inventor also supports objects called Engines, which be concatenated together in order to

*[email protected]; phone 1 609 734-6500; fax 1 609 734-6565; http://www.usa.siemens.com/en/research/

form processing pipelines. This is another concept well known and understood by software developers with an image processing and computer vision background. By leveraging both such Open Inventor concepts, XIP can therefore appeal to a broad audience of software developers involved with visualization and image analysis. •

Beginning graphics programmers who would find it difficult to write programs by manipulating down to the graphics primitive level of OpenGL, find they can quickly create graphics applications using the Open Inventor Toolkit.

•

It fulfills XIP’s extensibility and interoperability requirements. New objects can be easily created by deriving them from existing ones in an object-oriented way, with minimal effort. Since basic Open Inventor objects such as Nodes and Engines require very few lines of code, other existing software libraries can be wrapped using simple scripts that generate code automatically. Open Inventor objects can exchange data through Fields, which support automatic data format conversions through the use of Field Converter Engines – allowing wrapped objects from different libraries to become interoperable.

•

It is a de facto cross-platform standard for 3D graphics. It is also strong in academic and research environments, where it is used to teach graphics concepts.

•

It defines a standard file format for complete serialization/retrieval of scene graph designs, allowing users to save and restore their work.

•

Scene graphs can be used to store and manage the entire state of an imaging/graphics software application, thereby reducing the overall application complexity. Typically only a simple GUI layer on top of one or more scene graphs is sufficient in order to control an application’s workflow.

Since the standard set of objects available in the Open Inventor library is limited to basic graphics primitives, XIP extends it with hundreds of additional new objects in order to provide features that are more specific to medical imaging. Such extensions are grouped into the following set of libraries: XipIvCore:

A collection of extended base classes, core objects and utilities for window/viewport management, data conversion and coordinate transforms.

XipIvCoreGL: A collection of objects implementing OpenGL features not supported in Sgi®’s Open Inventor, such as programmable GPU processing using GLSL shaders and off-screen rendering. XipIvOverlay:

A collection of objects implementing 2D and 3D overlay/markup graphic primitives to annotate medical images interactively, perform measurements and plot graphs.

XipIvRenderer: A collection of objects supporting hardware-accelerated Direct Volume Rendering (DVR) and Multi-Planar reformatting (MPR) of 3D medical datasets. XipIvDicom:

A library that can load and display DICOM medical images, with Open Inventor objects built on top of the open source DCMTK library.

XipIvITK:

A collection of script-generated Open Inventor objects that wrap hundreds of C++ classes from the open source Insight Segmentation and Registration Toolkit (ITK) for image processing, segmentation and registration.

XipIvVTK:

A collection of script-generated Open Inventor objects that wrap hundreds of C++ classes from the open source Visualization Toolkit (VTK) for data visualization and processing.

XipIvRemote:

A collection of objects supporting client/server remote visualization of 3D MPR and volume rendered images.

XipCanvas:

A library containing an extended Java awt Canvas and a Java applet which make it possible to use XIP in Web-based Java applications or within any standard Web Browser.


3.2 XIP Builder Existing Integrated Development Environments (IDEs) for software development, such as MS Visual Studio, Delphi, and Eclipse, employ visually intuitive interfaces for managing software objects and for building graphical user interfaces (GUIs) and applications, but have limited built-in functionality when it comes to developing advanced image analysis applications, and require time consuming programming, compilation and debugging effort. Some IDEs are explicitly oriented towards rapid development for imaging and visualization using a modular Visual Programming environment, where a set of modular component can be visually connected together to form advanced image processing pipelines. For example IDL [14], OpenDX [15], SCIRun [16], MeVisLab [17], AVS/Express [18], and amira® [19] implement various approaches to visual programming. Many of such visual programming environments, however, suffer from one or more of the following limitations: •

Applications can only run within the visual programming IDE or have strong dependencies on such IDE or on a proprietary runtime environment, which would prevent easy deployment and integration into vendor-specific commercial imaging workstations

•

Modules are based on proprietary architectures that make it difficult to incorporate the use of existing external libraries and use them in an interoperable way

•

They use modules that are closed-source and/or require the purchase of a commercial license for product development

XIP provides a new visual programming environment that circumvents such limitations, called XIP Builder. The main objective in implementing the XIP Builder tool was to ensure that there would be no dependencies between the tool itself and the scene graphs created with it, essentially making the use of XIP Builder entirely optional: software developers can decide to use XIP Libraries within the XIP Builder tool in order to rapidly prototype some imaging functionality without writing any code, or they can opt to write C++ code that uses XIP Libraries directly without using a visual programming approach. Developers can also use a combination of both: they can design some scene graphs with XIP Builder, then save their work to an Open Inventor file or to a C++ header file and then load it back into a C++ programming environment for more advanced control of the application’s logic and to implement a GUI around XIP’s imaging functionality.

Fig. 1Error! Bookmark not defined.. Figure captions usually should be adjusted to be left-aligned, but fit centered under the image they pertain to, using a hanging indent to emphasize the figure number, and smaller width than the body text. In this case, the hanging indent is .5” and the left and right margins have been increased by .25” each. In addition, the *[email protected]; phone 1 609 734-6500; fax 1 609 734-6565; http://www.usa.siemens.com/en/research/

figure numbering is accomplished by inserting a field code called Seq figure. This automates the process of numbering all of your figures for you.

The XIP Builder environment can dynamically load any library conforming to the Open Inventor standard, i.e. XIP Libraries as well as any other library containing user-created objects. All C++ classes contained in such libraries are automatically parsed and displayed in XIP Builder’s list of available objects, without any additional overhead. The developer is able to use an object by selecting it and dragging it onto the workspace. The developer can fill out as well as connect fields, inputs and outputs for nodes and engines. The developer can also construct complex scene graphs by dragging and dropping, making nodes children of other nodes. An example XIP Builder scene graph is shown in figure . [???]. Write more here about separation of scene graphs and GUIs

3.3 XIP Host Applications developed with XIP tools and libraries can either run stand-alone, or as “hosted” plug-in applications utilizing the DICOM Application Hosting interfaces being defined by DICOM WG-23. The XIP Host project includes an open source reference implementations of WG-23 interfaces, as well as a prototype of a clinical research oriented hosting system for running plug-in applications. A typical WG-23 host system is responsible for the following: •

Authenticates users

•

Manages installation, launching, and termination of XIP Applications

•

Provides data and services to XIP Applications

•

Accepts status information and results back from XIP Applications

•

Deals with auditing and controls access to services and data

•

Isolates the XIP application from the nature of databases, archives, networks, and possibly image data formats

•

Manages access to DICOM networks, objects, and services

•

Creates Abstract Models from input data

In addition to traditional DICOM network interfaces, the XIP Host is caGrid capable, making it possible for users to choose between remotely hosted grid based components and data sources as well as locally available sources. These caGrid capabilities come through the use of caBIG’s Imaging Middleware tools [???]. Since the host is responsible for providing a great deal of the platform infrastructure used by plug-ins, the development of a WG-23 compliant plug-in application can focus primarily on data processing and visualization features (where XIP Libraries are of great help)

4. OPTICAL IMAGING USING XIP This section is for Fred Azar.

5. CONCLUSIONS AND FUTURE WORK The goals of NCI caBIG™ XIP and DICOM WG-23 initiatives is to produce the standards and tools needed to reduce the time and cost of developing and utilizing advanced radiological imaging software in clinical research, clinical trials and clinical operations. Much work is still needed to facilitate adoption of these innovations. There is an ongoing effort to provide the necessary infrastructure to support an open-source development community to continue to advance the tools to meet evolving needs. The existing XIP Wiki is being completely redesigned, and a roll-out of a main web site at


www.OpenXIP.org is planned for mid-2009. The first DICOM WG-23 Supplement 118 is scheduled to move for public comment at the beginning of 2009. Once ratified, the vendors of imaging scanners, workstations and information technology systems, in addition to supporting the plug-in standard in new products, will need to develop new systems for managing the distribution and licensing of plug-in applications. Imaging software developers will also need to work with the FDA to develop new streamlined processes for the validation and approval of such plug-ins, and will need to consider new models for partnering with academia and the imaging vendors in order to best exploit the vendor interoperability features of the Host/Plug-in standard.

6. AKNOWLEDGEMENTS The work on this paper was supported by the caBIG grant number ???get number from Pearson (NCI&TATRC). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation there on. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NIH/NCI or the U.S. Government. Many colleagues have contributed to the work described here. In particular we would like to thank Thomas Moeller, Daphne Yu and John Pearson; the contributors to DICOM WG-23, especially David Clunie; the contributors to the NCI caBIG Imaging Workspace, especially Lawrence Tarbox, Fred Prior, Bradley Erickson and Eliot Siegel. Add more from Frank’s program

REFERENCES [1]

[2]

[3] [4]

[5] [6]

[7] [8]

[9]

[10] [11]

[12]

[13]

[14] [15] [16] [17] [18]

Prior, F., Bradley J, E., Tarbox, L. "Open Source Software Projects of the caBIG™ In Vivo Imaging Workspace Software Special Interest Group", Journal of Digital Imaging, Vol. 20, Supp. 1, Springer New York, 94-100 (2007) Yoo, T.S. (Ed.), "Insight Into Images: Principles and Practice for Segmentation, Registration and Image Analysis", A K Peters Ltd. (2004) The Insight segmentation and registration toolkit (ITK) web site, http://www.itk.org Schroeder, W., Martin, K. and Lorensen, B., "The Visualization Toolkit: An Object-Oriented Approach to 3D Graphics", Prentice Hall, Upper Saddle River, NJ (1997) The Visualization Toolkit (VTK) web site, http://www.vtk.org Eichelberg, M., Riesmeier, J. et al., "Ten years of medical imaging standardization and prototypical implementation: the DICOM standard and the OFFIS DICOM Toolkit (DCMTK) ", in: Ratib, O. M., Huang, H. K. (Ed.), "Medical Imaging 2004: PACS and Imaging Informatics", Proc. SPIE 5371, 57-68 (2004) The Offis DICOM Toolkit (DCMTK) web site, http://dicom.offis.de/dcmtk.php.en Pearson, J., Tarbox, L., Paladini G. et al., "Emerging Radiological Software Standards and Development Technologies: Impact on Clinical Translation and Trials", in: Mulshine, J. and Baer, M. (Ed.), "Quantitative Imaging Tools for Lung Cancer Drug Assessment", Wiley-VHC Publishers, chapter 5, 67-94 (2008) Siegel, E., "Progress Report for the National Cancer Institute's caBIG Imaging Workspace and NCIA", National Cancer Institute’s caBIG™ Imaging Program Demonstration, SIIM Society for Imaging Informatics in Medicine, Annual Meeting, Scientific Session 10 (2008) The Sgi® Open Inventor web site, http://oss.sgi.com/projects/inventor/index.html Wernecke, J., " The Inventor Mentor : Programming Object-Oriented 3D Graphics with Open Inventor", Release 2, Addison-Wesley Longman Publishing Co., Inc., Boston, MA (1994) Sgi® Open Inventor Architecture Group, "Open Inventor C++ Reference Manual - The Official Reference Document for Open Inventor", Release 2, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, available online at http://techpubs.sgi.com (1994) Wernecke, J., "The Inventor Toolmaker: Extending Open Inventor", Release 2, Addison-Wesley Longman Publishing Co., Inc., Boston, MA (1994) ITT Visual Information Solutions IDL web site, http://www.ittvis.com IBM® Open Visualization Data Explorer (OpenDX) web site, http://www.research.ibm.com/dx/ Scientific Computing and Imaging Institute SCIRun web site, http://software.sci.utah.edu/scirun.html MeVis Research GmbH MeVisLAB web site, http://www.mevislab.de/ Advanced Visual Systems AVS/Express web site, http://www.avs.com/software/soft_t/avsxps.html


[19]

Visage Imaging amira® web site, http://www.amiravis.com/
