be used for the definition of 2D TFs and the visualization of medical image data. Furthermore, we compare this method to approaches that employ gradient ...
3D Visualization of Medical Image Data Employing 2D Histograms S. Wesarg and M. Kirschner Interactive Graphics Systems Group (GRIS) Technische Universit¨at Darmstadt, Germany {stefan.wesarg, matthias.kirschner}@gris.informatik.tu-darmstadt.de
Abstract—Transfer functions (TF) are a means for improving the visualization of 3D medical image data. If in addition to intensity another property is employed, two-dimensional TFs can be specified. For this, 2D histograms are helpful. In this work we investigate how the property feature size can be used for the definition of 2D TFs and the visualization of medical image data. Furthermore, we compare this method to approaches that employ gradient magnitude as second property. From our experiments with several medical image data we conclude, that structure size enhanced 2D histograms are more intuitive. This is especially true in the clinical area, where physicians are much more familiar with the meaning of the size of anatomical structures than with the concept of gradient magnitude.
easy to grasp for them. The efficient employment of a 2D histogram that combines intensity and gradient magnitude for selecting regions of interest and defining TFs is even much more difficult. Consequently, it is desirable to replace gradient magnitude as second dimension used in a 2D histogram by an other feature that is more intuitive. In this work we investigate how an approach that estimates the size of anatomical structures can be employed for the 3D visualization based on a 2D histogram. This structure size enhanced (SSE) histogram [3] can be used for interactively selecting regions of interest and defining 2D TFs.
Keywords-Volume rendering, transfer functions, histogram, feature size
II. R ELATED WORK
I. I NTRODUCTION Considering the visualization of medical image data, approaches dedicated to a particular group of users have to be designed. In the clinical routine, image-based diagnosis is mainly done by radiologists, who also provide any image data that is required by surgeons to be guided prior and during an intervention. While the first group mainly uses two-dimensional slices for diagnosis purposes and mentally creates 3D volumes, the latter group is used to have a three-dimensional view onto the operating field and therefore prefers meaningful 3D visualizations. Transfer functions (TF) [1] are widely used for assigning optical properties to data sets to be visualized. Onedimensional TFs only use the intensity values contained in the data for defining opacity and color. However, additional properties can be derived from the data sets and used for TF specification. Gradient magnitude is such an example, that emphasizes the boundaries between regions of different intensities [2]. If for all voxels of a data set the frequency of how often a certain combination of intensity and gradient magnitude occurs is plotted as a two-dimensional distribution, a 2D histogram is obtained. There, the x axis represents intensity wheras the y axis stands for the gradient magnitude values. If we focus on supporting the user in creating 3D visualizations in the medical context, any approach has to be adapted to the clinical user. Radiologists are in most cases non-experts in the field of Computer Graphics and Image Processing. Thus, the concept of gradient magnitude is not
Bajaj et al. [4] have introduced the contour spectrum that represents several measures for describing isosurfaces (2D contour length, 3D contour area, and gradient integral) and a user interface for interactively setting parameters for the isovalue definition. A similar approach for isosurface visualization has been proposed by Pekar et al. [5] with the Laplacian-weighted intensity histogram. Several early approaches for assigning TFs are discussed in [1]. The conclusion was that a semi-automatic approach combining intensity with first and second order derivatives into the histogram volume [6] is the most promising one. Kniss et al. [2] extended this approach to interactive volume rendering based on multi-dimensional TFs. There, they put the main focus on interaction widgets for effectively setting the parameters for opacity and color TFs in the 2D histogram domain. Later work is mostly focusing on two-dimensional TFs that are defined employing intensity and gradient magnitude information – called IGM histogram in the following. Sereda et al. [7] derive the so-called LH histogram that shows lower and higher intensities forming the boundaries. TF specification is then done in the LH domain, where homogeneous regions are represented by the diagonal of the rectangular histogram, and boundaries form clusters above that line. This approach overcomes the difficulties that occur when trying to select a certain boundary in a IGM histogram: there, boundaries are represented by arches that often intersect or overlap. An alternative for picking these arch structures directly in the 2D histogram are spatialized transfer functions intro-
Figure 1. The aneurysma data set. From left to right – Three IGM histogram based and three SSE histogram based visualizations with 1, 2 and 5 classes, respectively. The 2D bins have been spatially classified.
duced by Roettger et al. [8]. They incorporate the spatial location of each voxel by computing the mean position of all voxels of one single bin and its variance and classifying thereon the 2D bins. This work provides a solution for enhancing a 2D histogram with spatial information that is normally lost. An opacity TF is automatically defined based on the gradient magnitude value, the color of each cluster is randomly assigned and also used for the volume rendering. Recently, feature size has been introduced by different groups [9], [10], [3] as an alternative property that can be derived from image data. Correa et al. [9] employ a scale space approach for assigning size information to each voxel. In addition to the intensity values that are used for defining a 1D opacity TF, size information is used for specifying a 1D color TF and providing a second means for 1D opacity TF definition. Hadwiger et al. [10] present in their work a region growing (RG) based approach for deriving feature size information. During an unattended preprocessing stage, seeds are automatically placed and a RG with various parameters is performed. This information is used for generating a 3D histogram with a time dimension standing for the different parameter settings for the RG. Each 2D sub-space for a given time value represents a 2D histogram combining intensity and feature size. Finally, our structure size enhanced (SSE) histogram approach [3] computes feature size based on a user given tolerance value. That value is used for defining a lower and an upper intensity threshold. Instead of performing a RG, we search for voxels with a similar gray value along the directions given by the 26 adjacent 3D neighbors. The derived feature size information is used as second property for the 2D SSE histogram. III. M ATERIAL AND METHODS We want to compare the specification of TFs based on the scalar values of the data set and an additional property – gradient magnitude and feature size, respectively. In this section we describe the necessary steps: computation of the second property, generation of the 2D histogram, spatial
classification of the 2D bins, and interaction for TF specification and visualization. The structure size estimation used in this paper is an improved version of our method introduced previously [3]. The 2D histogram generation and its spatial classification is mainly based on the work by Roettger et al. [8]. Differences to that approach are named explicitely. A. Preprocessing The preprocessing step creates the two additional properties for the 2D histogram. Gradient magnitude calculation is straightforward. We use a 3D implementation based on central differences. The output image data contains high scalar values at positions where boundaries are located in the input data set. For estimating the size of the structures we use a threshold based approach. The user has to provide a tolerance value τ , a neighborhood size σ, and the number of accepted invalid voxels η. For each voxel a 3D star search is performed in a breadth-first search manner, along the 26 directions defined by its 26 adjacent neighbors, and the number of valid steps is counted. A step along one of these directions is valid, if the intensity Iv of the visited voxel is within the range [Iref − τ, Iref + τ ] with Iref representing the intensity of the reference voxel. The search continues until either the number of invalid voxels reaches η, or σ steps have been done in each direction. The number of valid steps represents the estimated structure size. In section IV, examples for the structure size estimator’s output are given. Typical values used in this work for τ, η, and σ are • τ : 2.5% of the maximum intensity in the data set 1 • σ: 8 of the maximum extent of the data set • η: 8 voxels. B. 2D histogram computation Next, the 2D histogram can be generated. There, the x axis represents the intensity values and the y axis gradient magnitude or structure size – for the IGM histogram and the SSE histogram, respectively. At each 2D bin position
Table I T HE PARAMETERS USED IN THIS WORK FOR DEFINING THE TRANSFER FUNCTIONS FOR OPACITY AND COLOR .
Opacity
Hue
IGM histogram
SSE histogram
fop (0, 0) = 0.0
fop (0, 0) = 0.25
fop (1, 0) = 0.25
fop (1, 0) = 1.0
fop (0, 1) = 0.75
fop (0, 1) = 0.0
fop (1, 1) = 1.0
fop (1, 1) = 0.25
(0.0, 0.7)
(0.0, 0.7)
1.0
1.0
1.0 − 0.75 · |gradn |
0.25 + 0.75 · sstruct,n
Saturation Value
the number of voxels with the corresponding combination of intensity and the second property is stored. In order to reduce noise in the 2D histogram, we use a k-neighborhood with k = 2 as it has been proposed by Roettger et al. [8]. This avoids to some extent under-populated bins. The supersampling however, that is also used in that work, is not employed here due to the huge memory consumption and extended computing time.
If the value of N (T, T0 ) falls below a user defined value – the feature radius rF – the two bins T and T0 are considered to belong to the same feature. Thus, the bins in the 2D histogram are classified, and a unique random color is assigned to each cluster. See reference [8] for details.
We decided to implement an automatic opacity assignment based on both properties. Opacity fop (u, v) at position (u, v) of a 2D bin always increases with intensity. In addition, it increases with gradient magnitude and decreases with structure size, respectively. Consequently, boundary voxels and voxels belonging to large structures are located in the upper part of the IGM and the SSE histogram, respectively. The user has to set the desired opacity values for all four corners of the SSE histogram, and the values in between are linearly interpolated (table I). For both, IGM and SSE histogram, a global opacity factor ω ∈ [0, 1) can be set by the user. Hence, all voxels that are represented by the bin at position (u, v) will be rendered with an opacity ω ·fop (u, v). Colors are defined in HSV space. We have implemented several methods for assigning color to each 2D bin: 1) bin population is mapped to hue value 2) value of the second property is mapped to hue value 3) bins are classified by their barycenter, hue value is randomly assigned Saturation and value are both set to 1.0 for the display of the 2D histogram. The color TF for rendering of the image data uses the same hue range as is used in the IGM and SSE histogram. Saturation S is set to 1.0, too. Value V decreases with increasing gradient magnitude |gradn | and decreasing structure size sstruct,n , respectively. The subscript n indicates that normalized values are used. Boundaries that represent changes in intensity are either regions of high gradient or of smaller size. Thus, a pseudo shading effect can be reached. Table I summarizes the above described parameters and shows the settings that we have used in this work.
D. TF specification
E. Interaction
Opacity definition methods that solely use image intensity conventionally assign a high opacity to high intensity values and vice versa. If in addition gradient information is considered, high gradient magnitude values are rendered more opaque than low gradient values, thus emphasizing object boundaries [2]. It is also possible to simply take gradient magnitude into account for opacity TF definition [8].
Clusters can be selected and deselected globally for the whole 2D histogram and locally by pointing and clicking on the corresponding cluster. In extension to the work of Roettger et al. [8], the individual selection of a cluster can be controlled: in global mode all 2D bins with the same label are selected, in local mode only those 2D bins that are connected (over bins of the same label) to the 2D bin, that
C. Histogram classification One severe limitation of histograms is the fact that they represent mainly statistical information – a probability distribution – and that any spatial information is lost. However, this information can be kept if the bins are weighted by the mean and the covariance of the spatial locations of all voxels that contribute to that bin. This concept, called spatiogram, has been introduced by Birchfield et al. [11]. Roettger et al. [8] propose a quite similar approach for enhancing an IGM histogram with spatial information by computing the barycenter b(T ) and the variance v(T ) of all voxels that fall into the same 2D bin T . The spatial relationship between an arbitrary bin T and a reference bin T0 can be evaluated by considering the distance norm N (T, T0 ) = kb(T ) − b(T0 )k + |v(T ) − v(T0 )| .
(1)
Figure 2. The heart data set. Classes representing the large cardiac cavities as well as lung tissue have been selected. Left: IGM histogram based visualization, middle: axial slice showing the result of the structure size estimation, and right: SSE histogram based visualization.
was clicked on, are selected. The reason for that is the fact, that 2D bins of the same label may form disjoint clusters, and the local selection mode allows for a refined picking. A possibility for selecting and deselecting arbitrary regions in the 2D histogram has been implemented by means of closed splines with control points adjustable in number and position. IV. R ESULTS For comparing the two 2D histogram approaches we selected several 3D image data sets – own data as well as data from http://www.volvis.org. There, we were focusing on how well the 2D histogram discriminates between different image features. In addition, we investigated how easily several structures of interest could be selected for being included in the direct volume rendering of the data sets. For the case where the spatial classification has been used, we adjusted the feature radius so that IGM and SSE histogram both contained the same number of classes. Our general experience was, that the selection of an arbitrary region in the 2D histograms is difficult, when color coding is based on bin population. If the bins had been classified by their location, the selection of structures was much easier. Comparing IGM and SSE histogram, it turned out, that there was less trial-and-error work if the SSE histogram had been used. This is not surprising, since feature size is a property that is easier to grasp than gradient magnitude. A. Aneurysma A vascular aneurysma is represented by a dilatation of the vessel. Consequently, the size in the corresponding region is larger than elsewhere. Thus, taking the structure size into account, highlighting of that pathology should be easy. IGM and SSE histogram have both been used for visualizing a contrast-enhanced rotational C-arm data set of a head (fig. 1). First, we had to state that the spatially classified IGM histogram was difficult to read. Classified 2D bins were
little compact and spread over the whole 2D histogram. We concluded that feature size as second property leads to a better separation of the differently sized vascular structures. In contrast to the IGM histogram, the SSE histogram showed a better concentration of the different classes. There, we first selected the class representing the largest structure and added afterwards those classes belonging to the next smaller structures and so on. The first class already highlighted the aneurysma, and by adding the next classes, structures with decreasing size were added to the visualization – exactly what we expected. Knowing that the vascular structure is the brightest region in the data set and that the aneurysma has a rather homogeneous region inside, we started with the selection of the bins at the lower right of the IGM histogram and added then classes with a comparable intensity but higher gradient. In this case, the first class represented a frayed out region inside the aneurysma and the largest vessel. By adding the next classes, several regions of the vascular structure became colored. B. Cardiac CT In a contrast-enhanced CT data set of the heart, blood filled cavities appear bright, wheras the lungs are much less intense. For this data set, we wanted to select the large cardiac cavities and lung structures for visualization (fig. 2). It turned out, that IGM and SSE histogram both separated well the different structures. The ventricles and atria appeared as low gradient regions at the bottom of the IGM histogram, but as large structure in the upper part of the SSE histogram. For the lungs it was the other way round – a high gradient region in the upper part of the IGM histogram, but a small structure in the lower part of the SSE histogram. C. Head MRI In this data set we investigated how well the eyes and brain structures could be visualized using both approaches.
Figure 3. The head MRI data set. Each approach has been used for highlighting eyes and brain. Left: two IGM histogram based visualizations, middle: axial slice showing the result of the structure size estimation, and right: two SSE histogram based visualizations.
The eyes are compact structures of medium intensity, whereas the brain is quite diffused and of low intensity. Comparing IGM and SSE histogram based visualizations, we could determine significant differences. A fundamental problem with IGM histograms is the fact, that homogeneous regions are always located at the bottom. That is why they are neither easy to perceive nor easy to select. However, after several trials we could select the correct classes in the IGM histogram. In the SSE histogram, the eyes as well as the brain are the largest structures in their respective intensity range. Thus, selecting the corresponding classified bins in the 2D histogram highlighted these structures. Concerning visualization quality, we could notice, that neither the eyes nor the brain could correctly be selected in the IGM histogram. Here, always other additional structures had been highlighted. The reason for that seems to be the fact, that all homogeneous regions with a similar intensity are located at the IGM histogram’s bottom – no matter to which structure they belong to. The SSE histogram however, better separates these structures, since they belong with a high probability to differently sized anatomies. D. Head CT Our final example shows a data set acquired during a high dose rate brachytherapy of a brain tumor patient. In the CT data set, the tumor is slightly visible – a little brighter than the surrounding healthy brain tissue. Again, the whole brain was represented by only a small number of bins in the IGM histogram. Selecting this one, highlighted the brain as well as some additional structures. As already discussed before, the SSE histogram provided a much better separation of different parts of the brain. Especially, the excluded tumor region was clearly visible in the visualization. V. C ONCLUSIONS In this work we have compared two types of 2D histograms for the definition of TFs for opacity and color –
IGM and SSE histogram. The property of the SSE histogram that the location of a 2D bin can be connected with a graspable meaning – voxels of a certain intensity that belong to a structure of a particular size – is its main benefit. In our opinion, this is better adapted for non-expert users – e.g. clinicians – than conventional 2D histograms that combine intensity and gradient information. In addition, in IGM histograms homogeneous regions of the same intensity fall into the same 2D bin, even if they belong to different structures. But, in the SSE histogram they are more likely to be located at different bin positions. Thus, the overall separation of relevant structures seems to be better in the SSE histogram than in the IGM histogram. Furthermore, we have shown how the 2D histograms can be applied to the definition of TFs for direct volume rendering. An opacity TF can be automatically set up based on the combined parameters intensity on one side and gradient magnitude and feature size, respectively, on the other side. For our visualization examples we used simple ramp functions for opacity and color definition. Employing more complex definitions may deliver even better visual results. We have also shown how spatial information can be used for an automatic definition of color TFs. Real world examples using medical image data coming from different modalities have shown the usefulness of 2D histograms for generating 3D visualizations. R EFERENCES [1] H. Pfister, B. Lorensen, C. Bajaj, G. Kindlmann, W. Schroeder, L. Avila, K. Raghu, R. Machiraju, and J. Lee, “The transfer function bake-off,” Computer Graphics and Applications, IEEE, vol. 21, no. 3, pp. 16–22, May/Jun 2001. [2] J. Kniss, G. Kindlmann, and C. Hansen, “Multidimensional transfer functions for interactive volume rendering,” Visualization and Computer Graphics, IEEE Transactions on, vol. 8, no. 3, pp. 270–285, Jul-Sep 2002.
Figure 4. The head CT data set. The brain contains a tumor that is treated with high dose brachytherapy. Left: IGM histogram based visualization of the brain, the tumor is not visible at all; middle: axial slice showing the result of the structure size estimation; and right: SSE histogram based visualization where the tumor appears as a hole.
[3] S. Wesarg and M. Kirschner, “Structure Size Enhanced Histogram: A Transfer Function for 3D Volume Visualization,” in Bildverarbeitung f¨ur die Medizin, Proceedings, ser. Informatik aktuell, H.-P. Meinzer and et al. Eds. Springer, 2009, pp. 16–20. [4] C. Bajaj, V. Pascucci, and D. Schikore, “The contour spectrum,” Visualization ’97., Proceedings, pp. 167–173, Oct. 1997. [5] V. Pekar, R. Wiemker, and D. Hempel, “Fast detection of meaningful isosurfaces for volume data visualization,” in VIS ’01: Proceedings of the conference on Visualization ’01. Washington, DC, USA: IEEE Computer Society, 2001, pp. 223–230. [6] G. Kindlmann and J. Durkin, “Semi-automatic generation of transfer functions for direct volume rendering,” Volume Visualization, 1998. IEEE Symposium on, pp. 79–86, Oct. 1998. [7] P. Sereda, A. Bartroli, I. Serlie, and F. Gerritsen, “Visualization of boundaries in volumetric data sets using LH histograms,” Visualization and Computer Graphics, IEEE Transactions on, vol. 12, no. 2, pp. 208–218, March-April 2006. [8] S. Roettger, M. Bauer, and M. Stamminger, “Spatialized Transfer Functions,” in Proc. of EUROVIS 2005: Eurographics / IEEE VGTC Symposium on Visualization, 2005, pp. 271–278. [Online]. Available: http://www.eg.org/ EG/DL/WS/VisSym/EuroVis05/271-278.pdf [9] C. Correa and K.-L. Ma, “Size-based Transfer Functions: A New Volume Exploration Technique,” IEEE TVCG, vol. 14, no. 6, pp. 1380–1387, 2008.
[10] M. Hadwiger, F. Laura, C. Rezk-Salama, T. Hllt, G. Geier, and T. Pabel, “Interactive Volume Exploration for Feature Detection and Quantification in Industrial CT Data,” IEEE TVCG, vol. 14, no. 6, pp. 1507–1514, 2008. [11] S. T. Birchfield and S. Rangarajan, “Spatiograms versus histograms for region-based tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Proc. of the, vol. 2, 2005, pp. 1158–1163.
