LNCS 9474
George Bebis et al. (Eds.)
Advances in Visual Computing 11th International Symposium, ISVC 2015 Las Vegas, NV, USA, December 14–16, 2015 Proceedings, Part I
123
Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen
Editorial Board David Hutchison Lancaster University, Lancaster, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Zürich, Switzerland John C. Mitchell Stanford University, Stanford, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen TU Dortmund University, Dortmund, Germany Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max Planck Institute for Informatics, Saarbrücken, Germany
9474
More information about this series at http://www.springer.com/series/7412
George Bebis Richard Boyle Bahram Parvin Darko Koracin Ioannis Pavlidis Rogerio Feris Tim McGraw Mark Elendt Regis Kopper Eric Ragan Zhao Ye Gunther Weber (Eds.) •
•
•
•
•
•
Advances in Visual Computing 11th International Symposium, ISVC 2015 Las Vegas, NV, USA, December 14–16, 2015 Proceedings, Part I
123
Editors George Bebis University of Nevada Reno, NV, USA
Tim McGraw Purdue University West Lafayette, IN, USA
Richard Boyle NASA Ames Research Center Moffett Field, CA, USA
Mark Elendt Side Effects Software Santa Monica, CA, USA
Bahram Parvin Lawrence Berkeley National Laboratory Berkeley, CA, USA
Regis Kopper The DiVE Durham, NC, USA
Darko Koracin Desert Research Institute Reno, NV, USA
Eric Ragan Texas A&M University College Station, TX, USA
Ioannis Pavlidis University of Houston Houston, TX, USA
Zhao Ye Kent State University Kent, OH, USA
Rogerio Feris IBM T.J. Watson Research Center Yorktown Heights, NY, USA
Gunther Weber Lawrence Berkeley National Laboratory Berkeley, CA, USA
ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-319-27856-8 ISBN 978-3-319-27857-5 (eBook) DOI 10.1007/978-3-319-27857-5 Library of Congress Control Number: 2015957779 LNCS Sublibrary: SL6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics © Springer International Publishing Switzerland 2015 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. Printed on acid-free paper This Springer imprint is published by SpringerNature The registered company is Springer International Publishing AG Switzerland
InVesalius: An Interactive Rendering Framework for Health Care Support Paulo Amorim1 , Thiago Moraes1 , Jorge Silva1 , and Helio Pedrini2(B) 1
Tridimensional Technology Division, Center for Information Technology Renato Archer, Campinas, SP 13069-901, Brazil 2 Institute of Computing, University of Campinas, Campinas, SP 13083-852, Brazil
[email protected]
Abstract. This work presents InVesalius, an open-source software for analysis and visualization of medical images. The tool has supported several surgeries in hospitals and has been downloaded from more than a hundred countries around the world. Its main characteristics, aspects of implementation, and applications in areas such as image segmentation, mesh generation, volume rendering, and 3D printing of anatomic models are described. Keywords: Medical image software
1
· Health care · 3D printing
Introduction
Medical diagnostic imaging procedures have been used since the early twentieth century with the emergence of X-ray technique. There are nowadays a variety of medical imaging modalities, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasonography (US), among others. With the continuous advance of technology and the popularity of these medical imaging techniques, it is now common to find facilities for diagnostic imaging in major cities, even in developing countries. The imaging techniques are highly dependent on software tools for processing, analysis and visualization; however, they are not always available to interested users due to cost issues or support for their end-user computing platform. In the late 90’s, one of the first 3D printers of Latin America was installed at the Renato Archer Information Technology Center in Campinas - CTI, S˜ ao Paulo State, Brazil, whose main purpose was the 3D printing of anatomical models to assist Brazilian surgeons in the surgical planning process and design of patient-specific prostheses. At that time, only a few proprietary and expensive software packages for medical image processing were available worldwide becoming unaffordable for the majority of Brazilian public hospitals. In such scenario, it arose the need to develop a free software for processing, analysis, visualization and 3D printing support for medical images. The developed software was called c Springer International Publishing Switzerland 2015 � G. Bebis et al. (Eds.): ISVC 2015, Part I, LNCS 9474, pp. 45–54, 2015. DOI: 10.1007/978-3-319-27857-5 5
46
P. Amorim et al.
InVesalius, as a tribute to the Belgian anatomist Andreas Vesalius (1514–1564), considered the “father of modern anatomy”. This paper presents the InVesalius, an open-source medical framework, including its main applications, relevant concepts used in its development, potential applications, as well as some statistics of its worldwide use. The text is organized as follows: Sect. 2 briefly presents some concepts and works related to the topic under investigation. Section 3 describes some computer graphics and image processing techniques used to develop the tool. Section 4 illustrates some applications of InVesalius and its use around the world. Finally, Sect. 5 outlines new directions for the development of InVesalius software.
2
Background
This section presents some of the key concepts and related works on medical imaging, 3D printing, image processing and computer graphics techniques associated with InVesalius software development. 2.1
Medical Images
Medical imaging is used in the diagnosis of several diseases. There are currently various types of medical images, such that each modality or a combination of them is appropriate for a certain kind of tissue, organ or, in some cases, for a particular type of disease. The intrinsic physical principles used in a set of image acquisition differentiate the imaging modalities. Computed tomography using X-rays measures the attenuation of X-rays in various directions in the tissue study, forming a threedimensional image using reconstruction mathematical algorithms. Denser materials attenuate a larger amount of X-ray and therefore appear more prominently in the resulting image, which makes this modality more suitable to visualize hard tissues. Magnetic resonance imaging measures the spin relaxation time of the water molecule hydrogen atoms after undergoing radio frequency pulses and a high magnetic field highlights soft tissues more clearly. Ultrasonography, on the other hand, measures the sound turnaround time (echo) produced by the sound emitted by focusing on a tissue. Distinct types of tissues produce different echoes and consequently differences in the images. These devices store images in DICOM (Digital Imaging and Communication in Medicine) [1] format. DICOM is an international standard protocol for file storage and network transmission of a set of medical images. This format allows the interoperability among medical imaging equipments and tools from different manufacturers. Due to its advantageous features and to be compliant with international standardization, DICOM is the image format supported by InVesalius.
InVesalius: An Interactive Rendering Framework for Health Care Support
2.2
47
Medical Image Software
There are several software packages available for the processing, analysis and visualization of medical images, some of them with support for exporting files for 3D printing anatomical parts, such as InVesalius. In this section, we give a brief description of some tools with similar functionalities to InVesalius. The Slicer 3D is an open-source software available under BSD (Berkeley Source Distribution) licensing modality and has several tools for processing, analysis and visualization of images. However, it has a complex graphical user interface (GUI) for users in the health care field, consequently mainly used in the academic environment. OsiriX software is widely used by surgeons and radiologists around the world, has a friendly interface and is an open-source code licensed under the GNU GPL 3. It has only a version for OS X operating system and is dependent on Apple Inc. hardware, which makes it expensive when compared to hardware from other manufacturers with similar processing power. Moreover, its version for 64-bit architecture, which is required to process highresolution images, is not for free. The FreeSurfer software, designed for segmentation and analysis of brain images, runs on Linux and OS X platforms and has a complex graphical interface, requiring the use of Unix command lines to perform some tasks. Despite having open-source-code, it has a proprietary license. The Volview software (Kitware Inc.) is an open-source, BSD license, with a userfriendly graphical interface that runs on Windows, Linux and OS X platforms; however, its main purpose is for visualizing of DICOM images and does not support triangular mesh generation for 3D printing. Among the main proprietary software packages, Analyze (Mayo Clinic) has various tools, but a complex graphical user interface. Mimics software (Materialise Corp.) and 3D Doctor (Able Software Corp.) have friendly interfaces with a large number of tools. In Mimics software, some modules need to be purchased separately. Amira 3D, developed by FEI, contains many features but has a complex graphical user interface. ScanIP, developed by Simpleware, has a simple interface with a smaller number of tools compared with the previously mentioned software packages. Vitrea (Vital Images) and Vizua are widely used mainly in radiology field. These tools are normally quite expensive for public hospitals in developing countries, since it is often necessary to have a separate license for each computer intended to run the application. Table 1 summarizes the main characteristics of the previously mentioned software tools. 2.3
3D Printing
3D printing [2], also known as rapid prototyping or, more recently, standardized (ASTM F2792) as additive manufacturing (AM), is nowadays considered a groundbreaking technology for the production of high added-value products. 3D printing began as a way to produce higher quality prototypes more quickly, with minimum human intervention. However, what happened during the almost 30 years during which this technology has existed is that, beyond a myriad of
48
P. Amorim et al. Table 1. Main characteristics of similar tools to InVesalius. Name
License
3D Doctor Proprietary 3D Slicer
BSD
Operating system
Country
Windows
USA
Windows/Linux/OS X USA
Amira 3D Proprietary
Windows/Linux
Analyze
Windows/Linux
Proprietary
FreeSurfer Other/Open Source Linux/OS X
USA USA USA
InVesalius GNU GPL 2
Windows/Linux/OS X Brazil
Mimics
Proprietary
Windows
Belgium
OsIrix
GNU GPL 3
OS X
Switzerland
ScanIP
Proprietary
Windows
UK
Vitrea
Proprietary
Windows
USA
Vizua
Proprietary
Windows/Linux/OS X France
Volview
BSD
Windows/Linux/OS X USA
available processes today, they largely migrate to various sectors of the industry, thanks to the evolution of the AM processes and their associated materials. Therefore, one of the more promising area is the health care sector, where customization is a quantum leap beyond traditional way to produce prostheses and medical devices. The 3D printing process starts with a 3D virtual model, normally represented by means of a simple triangle mesh called STL (Stereolithography) file format. Every single triangle in a STL file is composed of a Cartesian three spatial coordinates (x, y, z) and a normal vector pointing out of each triangle as a way to inform the 3D printing machines where the structures material to be printed are. The STL format is one of the simplest ways to represent a solid and is used as a 3D printing de facto standard since the first machines available in the market; however, STL is not accurate and carries several redundancies. Today, there is a better representation called AMF (Additive Manufacturing File Format) or ASTM F2915 but still not widely used. The STL file can be originated from many sources such as a CAD modeling system, a 3D laser or light scanner, or even medical scanners. The former is the way a medical imaging software integrates to a 3D printing machine. The 3D STL file is then sliced by the 3D printing machine that, by means of a layer-by-layer paradigm, can print objects in many different materials depending upon the 3D printing process used.
3
InVesalius Resources
The development of InVesalius aimed at including the following features: a multilanguage user-friendly graphical interface, cross-platform, and open-source coding. The code was implemented using Python and C++ programming languages, VTK library (Visualization Toolkit from Kitware) for 2D and 3D visualization,
InVesalius: An Interactive Rendering Framework for Health Care Support
49
Fig. 1. Main components of InVesalius.
Numpy (Numeric Python), Scipy (Scientific Python), PIL (Python Imaging Library) and GDCM (Grassroots DICOM). The graphical user interface was built through the wxPython library, which keeps the same look and feel on any operating systems. Figure 1 illustrates the main components of Invesalius. The following sections briefly describe the main software modules and theoretical aspects associated with InVesalius. 3.1
DICOM Import
InVesalius imports DICOM files using GDCM (Grassroots DICOM), which supports JPEG 2000 compression. In addition, the library is composed of methods for verifying the orientation of image volumes (axial, coronal, sagittal or oblique) and sorting the slices. A class has been implemented to classify DICOM files taking into account patient and series information. It is common for an examination of a unique patient to contain several series. For instance, in the case of computed tomography, acquisition can be used for displaying more clearly bones and other acquisition using contrast agents for better visualization of the vascular system.
50
P. Amorim et al.
After performing a sequence of DICOM file importing, InVesalius stacks the images and applies an interpolation algorithm according to the spacing that is indicated in the appropriate field of DICOM file, considering the spacing in x and y axes. It is important to keep the real dimensions when making measurements or exporting the model as an STL mesh file for 3D printing. The volume is saved on hard disk and is used as a file mapping technique of memory for accessing it. The advantage is the lower use of memory (RAM), enabling 32-bit architecture operating systems to work with larger sets of images. With the importation of DICOM files, a multiplanar reconstruction (MPR) is performed. This kind of reconstruction provides the visualization of structures under different anatomical orientations, allowing more precise image segmentation process and anatomical structure measurement and visualization. 3.2
Image Segmentation
Segmentation techniques allows for the “digital dissection” of different tissues or region of interest. Among those techniques, the manual segmentation, thresholding and watershed techniques are available in InVesalius. A mask is employed to select and represent the region of interest. Furthermore, each mask has a level of transparency, preserving the original image in the background. During manual edition, the user can delineate the region of interest in each image (slice) of the dataset using a brush tool or erase the selections performed by other manual editions tools. In the thresholding-based segmentation, the user can select an initial and final grayscale level to be kept in the segmentation mask. Additionally, there are some presets for computed tomography images based on the Hounsfield scale. The watershed segmentation method requires for the user to enter markers to indicate portions of the image that represent objects and background. This method considers the image as a drainage basin, where the graylevel intensities correspond to altitude values, forming valleys and mountains, whereas background and object markers correspond to water sources. These water sources will flood the relief and construct barriers when different water sources meet together, resulting the image segmented into background and objects. The image segmentation methods were implemented with Numpy and Scipy libraries. The interaction and visualization of the mask were implemented with VTK. 3.3
Triangle Mesh
After the segmentation process, it is possible to generate a triangle mesh (Fig. 2) by means of Marching Cubes [3]. This technique considers the segmented volume as an input, analyzes each voxel and its neighborhood, then searches a 256-triangle-layout table in order to determine the most suitable triangle. Binary segmented volumes can generate triangle meshes with certain staircase artifacts (Fig. 3(b)), mainly in regions with high curvature. These artifacts are not natural to the patient anatomy, such that the resulting model is not suitable for 3D printing. Thus, a mesh smoothing is required. The selection of the
InVesalius: An Interactive Rendering Framework for Health Care Support
(a) brain
(b) lung
51
(c) feet
Fig. 2. Examples of triangle meshes.
smoothing algorithm is crucial since it may oversmooth the resulting mesh and cause the loss of fine details. In Fig. 3(c), for instance, it is possible to observe some holes in the eyeball area due to the mesh extraction from the grayscale image (Fig. 3(a)). To address this problem, a context-aware smoothing algorithm [4] was implemented in InVesalius, where higher weights are assigned to artifact regions and lower weights to non-artifact regions. These weights determine the degree of smoothing to be applied to the images with respect to their original dimensions. A sample of a mesh smoothed through this algorithm is shown in Fig. 3(d). By using the Hausdorff distance and taking the grayscale mesh as reference, the difference from the context-aware mesh to the grayscale mesh is on average 0.000239 in units of the bounding box diagonal, whereas the difference from the Gaussian smoothing mesh to the grayscale mesh is on average 0.000431 in units of the bounding box diagonal. A lower Hausdorff distance indicates a more reliable method.
(a) grayscale
(c) Gaussian
(b) binary
(d) context-aware smoothing
Fig. 3. Comparison among meshes extracted from a (a) grayscale image, (b) binary image, (c) binary image smoothed through a Gaussian filter and (d) binary extracted mesh smoothed through context-aware smoothing.
52
3.4
P. Amorim et al.
Volume Rendering
Volume rendering is a type of scientific visualization of medical imaging data. Direct Volume Rendering (DVR) [5] employs presets that highlight certain anatomic structures and is widely used by radiologists. From each pixel that belongs to the view window, rays are emitted in the direction of the volume. The voxels intersected by these rays have their graylevel intensities represented by a color and transparency value. A transfer function is employed to assign color to the voxels. This module of the InVesalius tool is implemented through VTK library. InVesalius also offers projection techniques, such as MIP (Maximum Intensity Projection) and MIDA (Maximum Intensity Difference Accumulation). Similarly to direct volume rendering, MIP and MIDA emit rays to the direction of the volume. In MIP, each ray intersects the voxel with highest value, which is useful to highlight high contrast regions, such as bone regions and nodules; however, it presents some visual drawbacks. In MIDA, on the other hand, rays capture and accumulate changes from low to high values. Thus, regions with large variations of intensity are highlighted in the resulting image. MIDA provides visual information of depth, allowing for the user to better comprehend the image. Both techniques, MIP and MIDA, are less sensitive to occlusion, because they focus on high contrast areas at the expense of low contrast.
4
Distribution and Applications
The InVesalius software was a milestone for the use of three-dimensional technologies (virtual and physical through 3D printing) in Brazilian public hospitals. In this section, we present some usage statistics and applications of InVesalius in the health care field. InVesalius checks on the server if there is a new version of the software every time a user initializes it and, according to the IP (Internet Protocol), it is possible to identify the country that requests the license and count as an installation. To avoid counting the same installation twice, a unique identifier is sent to the InVesalius server. Since it is an open-source software, new features can be incorporated into the tool as needed. From March 2013 to April 2015, there were 7945 installations, distributed in 115 countries. In Brazil, 1794 installations were downloaded from our servers. We can see a map with the InVesalius distribution in Fig. 4. Although developed countries have the highest number of installations along with Brazil, countries in development, BRICS and some of the Asian continent have a significant amount of installations, which demonstrates the popularization of computed tomography and magnetic resonance imaging for medical diagnoses. Unfortunately, use of InVesalus in Africa is low. One of the reasons for high dissemination of InVesalius software is its support for seven languages (English, Portuguese, German, Spanish, French, Greek and Korean). Since its first version started to be developed in January 2001, as the pioneer open-source solution integrating medical scanners and 3D printing, until
InVesalius: An Interactive Rendering Framework for Health Care Support
53
Fig. 4. Installation distribution of InVesalius software between March 2013 to April 2015. A legend is shown on the top right of the figure.
April 2015, InVesalius enabled 3D printing (Fig. 5) of 3808 anatomical models, in the CTI’s ProMED Program - 3D Technologies for Healthcare - that were used for surgical planning. In the context of ProMED, approximately 284 Brazilian hospitals were supported, between the years 2011 and 2014. Surgeons from all over Brazil use models to simulate surgical procedures, fixation plates modeling, and patient-specific prosthesis design and production. A few Latin America hospitals outside Brazil were also supported in countries such as Argentina, Chile, Paraguay, Ecuador, Peru, Uruguay, Colombia and Mexico.
(a) InVesalius
(b) anatomic model
Fig. 5. Virtual and physical model for surgical planning.
InVesalius has also been a basis for new research and developments, such as a ogico of free laparoscopic surgery simulator [6] under development by the Tecnol´ Monterrey, in Mexico, and a neuronavigator [7] for applying transcranial magnetic stimulation (TMS) that supports different types of spatial trackers. In addition to improvements on surgery planning, as mentioned previously, InVesalius has contributed for Brazil to be on par with developed countries in
54
P. Amorim et al.
regard to the development of software for processing, analysis and visualization of medical images.
5
Conclusions
This paper described the open-source software InVesalius, its main resources and some applications related to health care. The software has contributed to the scientific and technological development of Brazil in terms of infrastructure and medical facilities, promoting more efficient procedures in health care with cost-effectiveness in public hospitals. Several new functionalities such as generation of triangle meshes using graphics processing units (GPUs), the integration with mobile devices, touchless interfaces for using in operating rooms, among several other resources already tested as stand alone tools are under development to be incorporated into InVesalius. Acknowledgements. The authors are grateful to FAPESP for the Brazilian Research Institute for Neuroscience and Neurotechnology - BRAINN (CEPID process 2013/07559-3) and for the Thematic Project (Grant 2011/22749-8) for the financial support.
References 1. DICOM: Digital Imaging and Communications in Medicine (2015). http://dicom. nema.org/ 2. Schubert, C., van Langeveld, M., Donoso, L.: Innovations in 3D printing: a 3D overview from optics to organs. Br. J. Ophthalmol. 98, 159–161 (2014) 3. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Comput. Graph. 21, 163–169 (1987). ACM 4. Moench, T., Gasteiger, R., Janiga, G., Theisel, H., Preim, B.: Context-aware mesh smoothing for biomedical applications. Comput. Graph. 35, 755–767 (2011) 5. Rogers, D.F., Earnshaw, R.: State of the Art in Computer Graphics: Aspects of Visualization, 1st edn. Springer, New York (1994) 6. Guerra, M.R.M., Amorim, P.H.J., Moraes, T.F., Silva, J.V.L., Quinones, K.L.B., Villalba, E.F., Zuniga, A.E., Rodriguez, C.A.: Soft tissue modeling for virtual surgery simulation. In: XXIV Brazilian Congress of Biomedical Engineering, pp. 813–816 (2014) 7. Rondinoni, C., Souza, V., Matsuda, R., Salles, A., Santos, M., Baffa Filho, O., dos Santos, A., Machado, H., Noritomi, P., Silva, J.: Inter-institutional protocol describing the use of three-dimensional printing for surgical planning in a patient with childhood epilepsy: from 3D modeling to neuronavigation. In: IEEE 16th International Conference on e-Health Networking, Applications and Services (Healthcom), pp. 347–349 (2014)
