Multidimensional Volume Visualization for PC-based Microsurgical Simulation System Zhenlan Wang1*∆
Chee-Kong Chui2#∆
Yiyu Cai3+
Chuan-Heng Ang4*
*
School of Computing, National University of Singapore, Singapore Graduate School of Frontier Sciences, University of Tokyo, Japan + School of MPE, Nanyang Technological University, Singapore ∆ Biomedical Imaging Lab, Institute for Infocomm Research, Singapore #
Microsurgical techniques are being increasingly applied in many surgical disciplines because of its superiorities over conventional surgeries in terms of small incision, fast recovery and special capabilities in complex and delicate surgeries such as cataract removal. However, rare opportunities are available for trainees to learn and exercise these skills before practice in clinical situations. With the rapid advances of virtual reality technology, a microsurgical simulation system is becoming possible by using the state-of-the-art virtual/augmented reality techniques for the purpose of training and even pretreatment planning based on patient-specific medical images.
Abstract Microsurgery is a highly complex surgical procedure on small body parts performed by a dedicated surgical team. An operating microscope is typically used to obtain a precise view of the soft tissues. The complexity of the microsurgical procedure makes it a suitable application of virtual/augmented reality technology for training purpose. In this paper, we present an overview of our simulator and then describe the visualization work that reconstructs the magnified view of the operating area from medical images. Our 3D linear level octree(LLO) based volume rendering algorithm, and newly proposed dynamic linear level octree (DLLO) based 4D volume rendering algorithm are integrated with a texture-based renderer to form the visualization component. In contrast to the conventional algorithms, our method achieves fast rendering speed, reduced requirement for texture bandwidth and combined surface-volume rendering for realistic and interactive PCbased medical simulations.
Superior visualization, sufficient force feedback and interactive manipulation of surgical tools are regarded as main elements of creating a realistic virtual microsurgical environment and thereof visualization is critical as surgeons perform operations and make decisions mostly based on visual cues and the magnified view of operating area is inevitable in most microsurgeries as well [Salisbury et al. 1998]. In [Wang et al. 2002b], we have demonstrated a prototype of our microsurgical simulator applied for craniotomy simulation, which involves expensive rendering of perspective views of brain inner structures. Extended from previous work, the detailed visualization solutions are presented in this paper.
CR Categories: I.3.3 [Computer Graphics]: Picture/Image Generation; I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism; J.3 [Computer Applications]: Life and Medical Sciences. Keywords: visualization, volume rendering, LLO/DLLO, microsurgery, simulation 1
Due to the nature of medical imaging applications, volume rendering is usually superior over other visualization methods such as 2D imaging and surface rendering. In [Hata et al. 2003], authors reported their study of using threedimensional (3D) volume rendering for differentiating congenital cystic adenomatoid malformation from congenital diaphragmatic hernia, which is noted by authors that it is hard to be diagnosed with only 2D MR images. Volume rendering is widely used in medical simulations. Ra et al. presented a spine needle biopsy simulator, in which volume rendering is adopted for effectively examining relative locations of spine and organs and tracking the needle path in terms of both accuracy and reality [Ra et al. 2002]. In the simulation of interventional neuroradiology procedures, volume rendering is used to produce both 3D and fluoroscopic representations of human vasculature system [Chui et al. 2002].
Introduction 1
e-mail:
[email protected] e-mail:
[email protected] 3 e-mail:
[email protected] 4 e-mail:
[email protected] 2
Copyright © 2004 by the Association for Computing Machinery, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions Dept, ACM Inc., fax +1 (212) 869-0481 or e-mail
[email protected]. © 2004 ACM 1-58113-884-9/04/0006 $5.00
Long Paper
309
Typical volume rendering methods include ray-casting [Levoy 1988] and splatting [Westover 1990]. They are characterized as production of high quality images. The shear-warp factorization algorithm is regarded as one of the fastest volume rendering methods [Lacroute 1994]. Incorporated with supplementary data structures, it is improved to support extensive applications [Kim et al. 2002; Wang et al. 2002a]. With the development of graphics hardware, texture mapping comes mature and is widely used in hardware-accelerated volume rendering. In [Boada et al. 2003], 3D texture mapping is used for the visualization of the volume data integrated with the surface polygons based on a hybrid octree. Wilson et al. reported a hybrid point-based volume rendering technique by using texture mapping [Wilson et al. 2002]. Since Levoy first utilized octree in ray-casting algorithm [Levoy 1990], hierarchical data structures are widely used to speed up intersection calculations for volume rendering acceleration [Yagel 1999]. The VolumePro graphics card has also developed to the second-generation for custom hardwareaccelerated real-time volume rendering [Pfister et al. 1999; Wu et al 2003].
2
Microsurgical Simulation System
An overview of our microsurgical simulator for craniotomy is shown in Figure 1. Craniotomy is a typical example of a microsurgical procedure. It involves a surgical removal of a portion of the skull to operate on a targeted region in the brain. The simulator is designed as a low cost system that can be comfortably placed on the desktop of a neurosurgeon. We aim to design and develop the software executed on a normal notebook computer with dual displays. The user will see the microscopic views with the stereoscope hanging in front of a computer monitor connected to a graphics port of the notebook computer. In the current set-up, we used a mid-range graphic acceleration board (32M RAM GeForce II). Microsurgical operations, such as tissue-cutting with micro-scissors, are simulated using a force feedback probe. This set-up and the stereoscope resemble the type of microsurgical environment with high-resolution microscope typical in craniotomy. Our proposed volume rendering technique is the primary method of computer graphics rendering in this simulation.
One of the significant usages of microsurgery is performed to connect/repair vessels, termed microvascular anastomoses. For example, in case of hand, the surgeon needs to align the vessels from a cut finger with that of the main hand. A smooth blood flow implies that the vessel has connected well. Multiple scans of the hand while contrast injection is used to capture the blood flow. Fourdimensional (4D) or time-varying visualization is meaningful for surgeons to observe such procedure. Shen et al. proposed a 4D volume rendering algorithm based on time-space partitioning (TSP) tree [Shen et al. 1999], and the algorithm is improved by using new color-based error metrics [Ellsworth et al. 2000]. The shear-warp factorisation algorithm is also extended to support timevarying volume rendering [Anagnostou et al. 2000, 2001]. Texture mapping is active in the research of 4D volume rendering as well. Lum et al provide users interactive 4D data exploration by fully utilizing the texturing capability of a commodity 3D graphics card [Lum et al. 2001]. In [Sohn et al. 2002], a wavelet-based codec is introduced for time-varying volume compression, and a combined rendering of isosurface-volume is presented with geometric primitives and texture mappings.
Figure 1. Overview of the microsurgical Simulation System For surgical training, the trainee will first load a set of volume images comprising a clinical scan of a human head. In the planning mode, he/she can selectively render and inspect the various structures within the skull. Both perspective and parallel renderings are available for him/her during the study. The main purpose of this study was to allow the trainee to identify the target area and plan his/her approach. For pre-surgical planning, over 80% of the plan can be achieved with realistic 3D graphics. Only perspective rendering is available to the trainee when he/she is performing the "surgery" using a combination of virtual tools. These tools are modelled as simple geometrical objects.
In this paper, after an overview of our microsurgical simulation system given in Section 2, the 3D and 4D volume visualization algorithms are presented in Section 3 and Section 4 respectively, followed by a discussion of the preliminary results in Section 5. Finally, Section 6 concludes this paper.
Instructor can monitor progress of the training via display on the notebook computer - supervisor console. The supervisor console does not have to mirror the display on
310
the stereoscope. The instructor may view a wider version of the rendered image the trainee is using during virtual surgery. The capability of the trainee neurosurgeons to reach the target region and removed the tumour tissue without hurting the surrounding healthy tissue is a good quantitative performance indication. However, we have yet to implement a scoring module to help track the trainee's performance.
Either of the criteria satisfied, an octant is a leaf node. If all the voxels contained in an octant are transparent, the octant is a “white” node and it will not be saved. Otherwise, it is a “black” node, and saved. To output an LLO, together with the code key, if the octant variance is below ν, only mean value of the octant is saved, otherwise, all the voxels are saved with the octant.
Our priority of the simulation system was to duplicate the high power microscopic views during microsurgery. Proficiency of a neurosurgeon depends on how well a user can relay the stereoscopic view with his/her hands. It is unclear to us on the effectiveness of force feedback for this simulation. Hence, our emphasis in microsurgery simulation and particularly, training of microsurgery procedure is on realistic visual effects. In addition to the provision of accurate and high-speed perspective rendering, our rendering engine is capable of handling the emerging 4D medical imaging data. In the following sections, we will describe in details the underlying data structure and algorithms of the rendering engine in our simulation system.
Smaller size of leaf node allows smaller regions of coherence to be exploited without sacrificing image quality. However, due to the limit of graphics hardware, the performance of the texture-mapped volume rendering will be greatly reduced if too many texture polygons are used. Therefore, l is selected to ensure that there will not be too many black leaf nodes generated in an LLO. Normally, it is selected greater than 4. More description on advantages of LLO as the data structure for interactive, low cost volume rendering can be found in [Wang et al. 2002a][Chui et al 1991].
3
3.2 Texture-based Octant Rendering The overview of the rendering algorithm is given in Figure 2. An LLO is either encoded from a volume dataset for the first time or reloaded from storage media. During the rendering, the LLO is traversed and octants are output one by one according to current view direction. In order to utilize the texture mapping method for hardwareaccelerated volume rendering, octants must be traversed in back to front order for proper texture mapping blending.
Data Structure and 3D Volume Visualization
In [Wang et al. 2002a], we proposed the LLO-based shearwarp algorithm for 3D volume rendering. To take advantage of the graphics card for hardware-accelerated volume rendering and achieve integrated surface-volume rendering for flexible and interactive simulations, the algorithm is revised and a texture-based renderer is incorporated.
Volume dataset
3.1 Linear Level Octree LLO
Octree, a hierarchical data structure, is an efficient representation of bounded 3D space by recursively subdividing it into eight uniform partitions so that the occupancy of 3D objects can be identified very fast. Octree can be labeled with different schemes. Linear Level Octree (LLO) [Chui et al. 1991], which is extended from linear octree scheme, is a label scheme with many advantages such as reduction of memory consumption, simple representation of octants, fast tree traversal and so on. In LLO, each octant is labeled with a unique code key: (Li, xi, yi, zi), where Li is the level of the node, and xi, yi, and zi are the x, y, z level-coordinates of the node respectively. With a code key, the location, the size and key codes of parent and children of an octant all can be computed very fast.
LLO Traversal
Display Order Arrays
Octant
Texture-based Renderer Volume rendering output
Figure 2. Pipeline of the LLO-based 3D Volume Rendering In parallel projection, octants can then be processed in specific order without affecting the result image and it has been determined that there are only eight distinct sets of such viewing directions. An eight-entry node-display-order table is already identified [Wang et al. 2002a]. Therefore, the LLO can be pre-traversed and the pointers to octants that need to be rendered are saved in arrays. Thus, traversal of the LLO during the rendering can be avoided.
To take advantage of the spatial-coherence, volume datasets are converted into the representation of LLO. Without loss of generality, a volume dataset is assumed to have 2n voxels in each dimension. Two leaf node criteria are adopted for LLO generation. First, the minimum edge length of an octant is 2l, i.e. each dimension of the octant contains 2l voxels at least, where l is an integer between 1 and n. Second, the variance of voxel intensities contained in an octant is less than ν, a user predefined threshold.
311
For perspective projection, the LLO traversal order is not only dependent on the view direction but also the viewpoint and the field of view (FOV). Therefore, the traversal orders of different intermediate nodes will vary and should be carefully considered. The field of view is divided into regions in the object coordinate system. In the same FOV region, all the view directions are conformable and they belong to the same entry of the node-display-order table as suggested by the parallel projection. The intermediate node that resides in the same FOV region adopts the same traversal order as the node-display-order table as well. An example of perspective projection is given in Figure 3 in 2D. The field of view is divided into two regions, where all the view direction vectors (vx, vy) in Region I are conformed to vx < 0 and vy > 0 and similarly for the region II. Therefore, volume octants in the two regions are traversed in different orders, and the octants in the same region are traversed in the same order.
3.3 Surgical Instruments Rendering In the simulation of interactive microsurgical procedures, the integrated rendering of virtual surgical instruments with the volume data is necessary for users to provide a realistic scene of virtual surgical environment. For simplicity, surgical instruments such as scalpels, forceps and needles are normally rendered with opaque geometric polygon primitives. Wang et al. achieved the volume embedded geometric rendering by cutting-plane technique based on hardware-assisted volume rendering [Wang et al. 2001]. Incorporated with texture-mapped volume rendering, the integrated rendering can be achieved more easily. Opaque geometric surgical instruments must be rendered before the texture mapped volume rendering so that depth buffer test is enabled. Afterwards, the graphics hardware is able to blend the geometric polygons with the texture images correctly and efficiently.
Region II
Volume LLO
4
Region I
Extended from the 3D LLO-based volume rendering algorithm introduced in the previous section, the 4D volume rendering is performed with a time-dynamic LLO. During the rendering time, a working LLO is updated by new datasets. A change detection algorithm is employed to exploit the temporal-coherence between datasets, and 3D volume rendering is only applied to the sub-LLOs changing over time. This is the essential idea about the dynamic linear level octree (DLLO).
y x
FOV Viewer
Dynamic Linear Level Octree (DLLO)
Image Plane
Figure 3. FOV Regions in Perspective Projection Time-varying volume datasets VD1, VD2, …, VDN LLO Conversion Time-varying LLO datasets LLO1, LLO2, …, LLON LLO Differencing
DLLO Renderer
Rendering stage
Differencing LLO datasets LLO1, dLLO2, …, dLLON
Data Processing stage
Texture mapping technique is employed for octant rendering. Depending on the type of graphics hardware available, leaf octants with low variance are required to be reconstructed from their mean values in different representations. If 3D texture hardware is available, fast 3D interpolation can be performed by the hardware, so a reconstructed octant is a cube with mean values at eight corners. But commonly, only 2D texture mapping is available on personal computers (PC). The reconstructed octant will contain multiple duplicated 2D texture images along the view direction, and each of the texture images contains 2×2 mean-value pixels. Especially, in the case of 2D texture-mapped octant rendering, software sampling is required for all the octants to create the texture images along the view direction so that texture images are mapped in back to front order. To minimize the software sampling cost, texture images can be pre-computed and saved at the expense of system memory. After octant data are downloaded to the hardware texture memory, texture polygons are transformed to the correct locations corresponding to the octant in the volume according its code key. Then texture mapping is applied. Special care may be taken to ensure that boundaries between octants do not result in image artefacts.
4D volume rendering output
Figure 4. Pipeline of the DLLO-based 4D Volume Rendering Based on the idea of the DLLO, the 4D volume rendering is divided into two steps, data processing stage and
312
rendering stage as shown in Figure 4. In the first stage, a sequence of time-varying volume datasets is converted into a more effective representation so that a time-dynamic LLO can be updated efficiently over time. In the next stage, 4D volume rendering images are updated over the change of time. These two stages will be introduced in the following subsections.
algorithm, two LLOs are traversed and compared simultaneously. The second LLO is clipped into a differencing LLO (dLLO), while the first one remains intact. During the differencing procedure, only changed nodes are maintained while the homogeneous leaf nodes between two datasets are cropped. The details of LLO differencing algorithm is given in Figure 5.
4.1 Data Representation
The two parameters of the differencing algorithm are either empty, leaf or intermediate octants from the same location of two LLOs. LLO A and LLO B are two successive datasets. Initially, the LLO differencing algorithm will be invoked with their roots as parameters:
A 4D medical volume dataset is normally composed of a set of 3D volume datasets, which are acquired at a sequence of time steps, so the 4D volume is often referred to as time-varying volume data as well. We denote the N sets of time-varying volume data as (VD1, VD2, …, VDN), which are inputs to our 4D volume rendering system. In the data processing stage, first, they are converted into the representations of LLO so that the spatial-coherence is exploited inside each of the datasets. We denote them as (LLO1, LLO2, …, LLON). Second, an LLO differencing algorithm is employed to exploit the temporal-coherence between datasets. After this process, these time-varying LLO datasets are converted into the representations, named differencing LLO datasets (LLO1, dLLO2, …, dLLON). Finally, they are passed to the rendering stage.
Differencing(LLO_A_root, LLO_B_root);
The algorithm clips LLO B into dLLO B without change of LLO A. If octant B either is an empty node or is removed from LLO B, the algorithm will return 1, otherwise, return 0. Such attribute is used to remove intermediate nodes that have no descendants. It is a recursive algorithm that is utilized in the LLO differencing algorithm, and the recursive termination conditions are given in Table 1, where E, L and I stand for empty, leaf and intermediate node respectively. The first two columns are possible states of octant A and B, and the third and fourth columns suggest the return value and additional actions applied in the differencing algorithm.
int Differencing(Octant_A, Octant_B) { Check the termination conditions in Table 1. If octant A and B satisfy an entry, the value is returned and additional actions are applied as suggested by the table. Then this function is terminated.
E E E
E L I
Return Value 1 0 0
L
E
0
Octant A Octant B
// Both Octant A and B are // intermediate nodes. int EmptyChildNodeCount = 0; for (int i=0; i
