ical domain for surgical training, planning, diagnosis, and in particular for image guided surgery ... topics such as displays, hardware, calibration and registration. ... for IGS that allows the comparison of visualization techniques in a controlled.
A Realistic Test and Development Environment for Mixed Reality in Neurosurgery Simon Drouin, Marta Kersten-Oertel, Sean Jy-Shyang Chen, and D. Louis Collins McConnell Brain Imaging Center, MNI, McGill University, Montreal, Canada
Abstract. In a mixed reality visualization, physical and virtual environments are merged to produce new visualizations where both real and virtual objects are displayed together. In image guided surgery (IGS), surgical tools and data sets are fused into a mixed reality visualization providing the surgeon with a view beyond the visible anatomical surface of the patient, thereby reducing patient trauma, and potentially improving clinical outcomes. To date few mixed reality systems are used on a regular basis for surgery. One possible reason for this is that little research on which visualization methods and techniques are best and how they should be incorporated into the surgical workflow has been done. There is a strong need for evaluation of different visualization methods that may show the clinical usefulness of such systems. In this work we present a test and development environment for augmented reality visualization techniques and provide an example of the system use for image guided neurovascular surgery. The system was developed using open source software and off-the-shelf hardware.
1
Introduction
Mixed reality is considered the area on the reality-virtuality continuum between reality, the unmodeled real environment, and virtual reality (VR), a purely virtual and modelled environment [1]. The point on the continuum at which an environment lies will correspond to the extent to which the environment is modelled and whether real or virtual objects are introduced into this environment. Upon this continuum lie augmented reality (AR) and augmented virtuality (AV). In AR the environment is real and virtual objects are added to it. In AV the perceptual environment is virtual and live objects are introduced into it. In our work, we present an AR test environment for image-guided neurosurgery. Live images of a patient phantom (the real environment) are captured by a video camera and pre-operative models (virtual objects) of the patient are superimposed on the patient phantom. The end-user views the AR scene on a standard computer monitor. Augmented reality visualizations have become a focus for research in the medical domain for surgical training, planning, diagnosis, and in particular for image guided surgery (IGS). Their purpose in IGS is to overcome the surgeon’s limited visual perception and their restricted view of the region of interest. This is C.A. Linte et al.(Eds.): AE-CAI 2011, LNCS 7264, pp. 13–23, 2012. c Springer-Verlag Berlin Heidelberg 2012
14
S. Drouin et al.
achieved by merging pre- and intra- operative data sets with images of the patient in the operating room (OR), and displaying surgical tools and instruments within the visualization. By providing the surgeon with a more extensive view, beyond that of the visible anatomical surface of the patient, such visualization may contribute to reducing patient trauma by increasing surgical precision and decreasing surgical time, thus potentially improving clinical outcomes. Although augmented reality visualizations are increasingly studied for use in IGS, few such systems are introduced for daily use into the OR. One reason for this may be the common focus on technical aspects of such systems, including topics such as displays, hardware, calibration and registration. In the majority of publications on this topic, the focus is not on visualization and the evaluation of these systems within a clinical context [2]. The evaluation of new visualization methods, however, is crucial. The lack of such evaluation that would demonstrate the clinical usefulness of these systems may explain the absence of these systems in the OR. Without rigorous evaluations of new visualization methods to show whether there is an improvement in patient outcomes, surgery times, etc. there is no motivation to introduce new visualization methods and systems into the OR. Ultimately, new AR visualization techniques should be evaluated in the OR. However, in order to do so one must have ethical approval, access to surgeons, access to the OR, and a statistically significant way to compare different visualization techniques across surgeries. Such studies are challenging and are not always feasible. In this paper, we propose a simple AR visualization platform for IGS that allows the comparison of visualization techniques in a controlled lab environment. Although such testing cannot replace OR studies, we believe it is a valuable first step. The proposed system is based on off-the-shelf hardware and publicly available software and tools.
2
Related Work
The requirement made on neurosurgeons to precisely resect the smallest possible volumes of tissue while avoiding eloquent areas has been a strong motivation for effective visualization techniques such as those which may be provided by augmented reality. Such navigation systems have been developed, and include those that use head-mounted displays (HMD) [3], light projection onto the patient [4], images injected into the surgical microscope [5] and those where a monitor is used as the perception location, which we focus on. In Lorensen et al. [6] pre-operative 3D surface models of the patient’s anatomy are mixed with video images for neurosurgery. In the work of Pandya et al. [7] a live video camera, mounted on a passive articulated and tracked arm, is superimposed in real-time with segmented objects of interest from pre-operative CT scans. In the work of Paul et al. [8], both AV and AR are presented as visualization solutions for neurosurgery. In the case of AV, a 3D surface of the operating field is generated intraoperatively using computer vision techniques and the stereo images from the operating microscope. The microscope image is
Mixed Reality Environment for Neurosurgery
15
then mapped onto the generated model. Low et al. [9] used augmented reality with the Dextroscope 3–D planning and system. In their work, real-time live video from the OR was enhanced with planning data, including tumours and veins, on the guidance station to enhance the visualization of the tumour in 3–D space.
3
Materials and Methods
In IGS, pre-operative plans, patient models and graphical representations of surgical tools and instruments are localized in real-time and displayed to guide the surgeon in their task. The incorporation of the real surgical scene with the surgical tools and pre-operative models facilitates the determination of the spatial relationship between the patient and the 3–D models of the patient. An overview of how our augmented virtuality system works is shown in Fig. 1. 3.1
System Overview
In neurosurgery, a surgical microscope is used to have a magnified, more precise view of the region of interest within the craniotomy. Instead of using a surgical microscope, we produce an equivalent, but less expensive “microscope image” using a Sony HDR-XR150 video camera. We constructed a 4-point passive tracker (Traxtal Technologies) which fits into the camera shoe. A Polaris Tracking System from Northern Digital Technologies is used for tracking the camera and the passive trackers. In Fig. 2 we show the laboratory set-up of our AR system. 3.2
Calibration
In order to render the virtual objects (e.g. the pre-operative models of the patient) with the real-world image from the camera, the transformation between camera image space and world space (i.e., extrinsic camera parameters), as well as focal length and principal point (i.e., intrinsic camera parameters) must be computed (Fig. 3 (a)). The intrinsic camera parameters are computed for a fixed zoom and focus using the Camera Calibration Toolbox for Matlab1 . Note that a C implementation of this toolbox is also available in the Open Source Computer Vision library2 . Calibration is done by taking photos of a calibration grid and choosing the extreme corners of the grid points. All edges within the grid are then determined by the toolbox and, based on the knowledge of the size of the squares on the calibration grid, the transformation from image space to real world space is found. The calibration remains valid as long as the zoom or focus are not changed. We found that there was no significant amount of distortion so this parameter was not considered. 1 2
http://www.vision.caltech.edu/bouguetj/calib_doc/ http://opencv.willowgarage.com/wiki/
16
S. Drouin et al.
Input volumes
Segmentation + carving
Augmented virtuality
3D printing Phantom
Phantom registration
Camera Parameters
Camera calibration
Grid registration
Fig. 1. The AV system within an IGS framework
Once the intrinsic camera parameters are found, the mapping from the passive tracker on the camera to the camera’s focal point, Ttc , must be determined. This transformation is computed in two steps: (1) the transform from calibration grid and tracker, Tgt , is found and (2) the transform from calibration grid to camera, Tgc , is found. Thus, we compute the transform as follows: −1 Ttc = Tgt ∗ Tgc
(1)
where the transformation from calibration grid to camera, Tgc , is computed using the Matlab Calibration toolbox. The transformation from calibration grid to tracker, Tgt , is determined by capturing 4 landmark points (in tracker space) on the calibration grid using an optically tracked pointer tool. Since the grid space coordinates of those landmark points are known, we can then compute the transform that corresponds to the best fit mapping between the two sets of points
Mixed Reality Environment for Neurosurgery
17
Fig. 2. Our laboratory system set-up. A Polaris Tracking System is used to track the camera and the reference attached near the target. The neuronavigation system displays the augmented reality visualization.
using least squares. We use the publicly available Visualization Toolkit (VTK) class vtkLandmarkTransform (based on the work of Horn [10]) for computing the transform. Once Ttc is known, the world-to-image space transform can be updated using the coordinate information from the Polaris, allowing us to be able to move the camera around the scene to get different views of the real world which remain registered with the pre-operative models. 3.3
Rendering of AR View
We render video images as semi-transparent planes in the 3D window as illustrated in Fig. 3 (b). The position and orientation of the cameras focal point is updated in real-time using input from the optical tracking system: Tgc = Tgt ∗ Ttc
(2)
where Tgt and Ttc are defined as above. The image plane is positioned at a user-specified distance from the focal point along the negative z axis of camera space. The full height of the image plane can easily be determined from the focal distance of the camera and the full height of the charge couple device (CCD):
18
S. Drouin et al.
(a)
(b)
Fig. 3. (a) The mapping from the passive tracker on the camera to the image space, Ttc is computed in two steps: (1) the transform from calibration grid and tracker, Tgt and (2) the transform from calibration grid to camera, Tgc . (b) The focal point of the camera is positioned and oriented in 3D using Tgc . The video image plane is then placed at a user-specified distance from the focal point along the negative z-axis of the camera.
h=
dhccd f
(3)
where hccd is the height of the CCD of the camera, d is the user-defined plane distance and f is the focal length of the camera. To produce the augmented reality view, we apply the camera parameters to the virtual camera model, used to render the 3–D view. The vtkCamera class is used to compute the model view and projection matrices of the rendering system. The transformation Tgc is used to position and orient the camera (vtkCamera::ApplyTransform) and the cameras view angle (vtkCamera::SetViewAngle) is computed from the cameras focal distance as follows: hccd θ = 2 · tan−1 (4) 2f 3.4
Experimental Set-Up
In our first implementation we look at how well the calibrations worked to align the real and rendered objects using a Lego (Lego Group, Denmark) phantom. We chose Lego as they provide a low cost and easy to use framework for building simple models and testing the accuracy and visual results of our set-up. In Fig. 4 we show a 3–D model of Lego and the unregistered projection of the camera image in (a), in (b) the Lego phantom is shown, in (c) an image of the 3–D model with the camera image registered and in (d) an image of the 3–D model with the camera image and overlayed real and virtual pointer tool.
Mixed Reality Environment for Neurosurgery
(a)
(c)
19
(b)
(d)
Fig. 4. (a) The 3–D model of the Lego and the unregistered projection of the camera image. (b) The Lego phantom. (c) The 3–D model with the camera image after calibration and registration. (d) The 3–D model with the camera image and overlayed real and virtual tracked pointer tool represented as a light grey hexagonal prism extending from the tip of the real pointer to the first IR reflective ball.
4
AR for Neurovascular Surgery
We applied our system to look at how AR could be used for neurovascular surgery and in particular arteriovenous malformations (AVMs), which are abnormal collections of blood vessels in the brain. AVMs are fed by one or more arteries (feeders), and are drained by one or more major draining veins (drainers). Neurosurgery for AVMs involves identifying the margins of the malformation and tying off or clipping the feeder vessels, obliterating the draining veins and removing the nidus. During surgery the localization of these feeding and draining vessels from the pre-operative plans to the patient lying on the table is necessary. This often difficult task may be facilitated by using mixed reality visualizations. Most surgical navigation systems display three 2–D views (coronal, axial and sagittal) and one 3–D rendered view of the patient data. However, the burden
20
S. Drouin et al.
remains with the surgeon to transform the 2– and 3–D images from the screen to the patient lying on the table, a task that is sometimes non-trivial and often time-consuming. This is especially the case with respect to: (1) locating vessels beyond the visible surface of the patient, (2) determining the topology of the vessels below the visible surface, and (3) locating and identifying the feeding and draining veins. Normal patient variability, as well as vascular anomalies make this task even more difficult. Registration of the live microscope image or a camera image within the OR with the pre-operative plans and data sets should help in understanding the layout of the vessels beyond the surface. 4.1
Patient Phantom
To examine the use of AR in neurovascular surgery we created a 3–D phantom from real patient data. The acquired datasets of the patient included a computed tomography digital subtraction angiography (CT DSA) at the late arterial and early venous stage, a specific CT DSA with injection into the carotid artery showing the feeding arteries and draining veins of the malformation, a CT DSA with injection into the vertebral artery, and a magnetic resonance angiography (MRA).
Fig. 5. The 3–D phantom printed using SLS and nylon plastic material
The vessel datasets were segmented from the CT DSA volume using region growing with a pre-defined threshold. The cerebral cortex surface was extracted using FACE [11] on the MRA data (with vessels masked out). The skin surface of the patient was extracted using marching cubes from MINC tools3 . To create the 3–D phantom, each of the mesh objects (vessels, cortex and skin surface) were imported into the open source 3D modelling and animation 3
http://en.wikibooks.org/wiki/MINC/Tools/
Mixed Reality Environment for Neurosurgery
21
program Blender (GPL, Blender foundation). As all of the datasets were already in the same space they did not have to be registered. Some post-processing to fill in holes in the meshes was done. Furthermore, two holes (simulated craniotomies) were cut into the skin surface exposing the cortex and vessels. A 3–D model was exported as STL (a CAD Software Solid Modeling/Prototyping File) and was printed using the online ZoomRP printing service4 with Selective Laser Sintering (SLS) in nylon plastic material (Fig. 5).
4.2
Visualization
For visualization we use IBIS (Interactive Brain Imaging System) NeuroNav [12], a custom developed neuronavigation prototype from the Image Processing Lab at the McConnell Brain Imaging Center (Montreal, QC). In our current implementation, surface models of the pre-operative data are colour-coded and registered to the live camera image and merged using alpha blending, Fig. 6.
Fig. 6. Top: stereo screenshot of IBIS with the AR visualization for the healthy craniotomy. Bottom: stereo screenshot of IBIS with the AR visualization where the AVM malformation is. Colours represent vessel from different data sets. For both sets of images, left and center images provide divergent eyes stereo while center and right provide convergent eyes stereo.
4
http://www.zoomrp.com/
22
5
S. Drouin et al.
Discussion and Future Work
By using AR in neurovascular surgery, we believe we can aid the surgeon to understand the topology and location of important vessels below the surface of the patient, potentially reducing surgical time and increasing surgical precision. The preliminary work presented in this paper is a necessary first step for comparing perceptual aspects of AR visualization methods in the context of neurosurgery. For this reason, we have deliberately left out many technical problems inherent in the application of augmented reality in the operating room. Among other things, our system does not consider the impact of brain shift on the accuracy of images. This phenomenon may introduce an important offset between the video images and the virtual models. Also, the system presented here assumes that the video images represent the surface of the operating field while the virtual objects are below the surface. In practice, resection performed by the surgeon during an operation will cause some elements of the virtual model to be on top of the image or will have been removed. There is currently no mechanism in our system to adapt the virtual models during the operation. One of the important tasks in our future research will be to make the phantom more realistic in order to ensure that the developed visualization methods remain valid in the OR. Another important step will be to incorporate different visualization techniques such as volume rendering and different depth and perspective cues such as fog, edges and chromadepth. We will also need to define meaningful tasks that can be accomplish by surgeons in the lab in order to statistically compare the effectiveness of different visualization techniques. It is not obvious whether new visualization methods or systems, which provide alternative and novel views of existing information, have added value, given that they do not provide new information. However, even though developing realistic tests, studies and scenarios for new visualization methods is challenging, it is extremely important to show whether new methods are useful. In our work, we have developed a laboratory augmented reality environment for testing of novel view and visualization methods for neurosurgery that is inexpensive and based on open source software. Although we believe the system could be eventually used pre-operatively for planning or intra-operatively for guidance, the current focus has been on developing a simple test-bed with publicly available tools for testing AR visualization techniques.
References 1. Milgram, P., Colquhoun, H.: A Taxonomy of Real and Virtual World Display Integration. In: Mixed Reality - Merging Real and Virtual Worlds, pp. 1–16 (1999) 2. Kersten-Oertel, M., Jannin, P., Collins, D.L.: DVV: A Taxonomy for Mixed Reality Visualization in Image Guided Surgery. IEEE TVCG (March 2011) (preprint) 3. Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hummel, J., Hanel, R., Greimel, W., Homolka, P., Ewers, R., Bergmann, H.: A head-mounted operating binocular for augmented reality visualization in medicine-design and initial evaluation. IEEE Trans. Med. Imag. 21(8), 991–997 (2002)
Mixed Reality Environment for Neurosurgery
23
4. Marmulla, R., Hoppe, H., Muhling, J., Eggers, G.: An augmented reality system for image-guided surgery. Int. J. Oral Maxillofac. Surg. 34, 594–596 (2005) 5. Edwards, P.J., King, A.P., Maurer Jr., C.R., de Cunha, D.A., Hawkes, D.J., Hill, D.L., Gaston, R.P., Fenlon, M.R., Jusczyzck, A., Strong, A.J., Chandler, C.L., Gleeson, M.J.: Design and Evaluation of a Aystem for Microscope-assisted Guided Interventions (MAGI). IEEE Trans. Med. Imag. 19(11), 1082–1093 (2000) 6. Lorensen, W., Cline, H., Nafis, C.: Enhancing Reality in the Operating Room. In: Proceedings of IEEE Vis. Conf. (VIS), pp. 410–415 (1993) 7. Pandya, A., Siadat, M.R., Auner, G.: Design, implementation and accuracy of a prototype for medical augmented reality. Comput. Aided Surg. 10(1), 23–35 (2005) 8. Paul, P., Fleig, O., Jannin, P.: Augmented Virtuality Based on Stereoscopic Reconstruction in Multimodal Image-Guided Neurosurgery: Methods and Performance Evaluation. IEEE Trans. Med. Imaging 24(11), 1500–1511 (2005) 9. Low, D., Lee, C.K., Tay Dip, L.L., Ng, W.H., Ang, B.T., Ng, I.: Augmented reality neurosurgical planning and navigation for surgical excision of parasagittal, falcine and convexity meningiomas. Brit. J. Neurosurg. 24(1), 69–74 (2010) 10. Horn, B.: Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A 4, 629–642 (1987) 11. Eskildsen, S.F., Østergaard, L.R.: Active Surface Approach for Extraction of the Human Cerebral Cortex from MRI. In: Larsen, R., Nielsen, M., Sporring, J. (eds.) MICCAI 2006, Part II. LNCS, vol. 4191, pp. 823–830. Springer, Heidelberg (2006) 12. Mercier, L., Del Maestro, R.F., Petrecca, K., Kochanowska, A., Drouin, S., Yan, C., Janke, A.L., Chen, S., Collins, D.L.: New prototype neuronavigation system based on preoperative imaging and intraoperative freehand ultrasound: system description and validation. Int. J. Comput. Assist. Radiol. Surg. 6(4), 507–522 (2011)
