A Virtual Reality Patient and Environments for Image ...

3 downloads 238 Views 1MB Size Report
stick device so that patient, if needed, can move freely and navigate in 3D virtual ... This information is transmitted to a small computer (Apple macMini), .... governed by a number of parameters, called constants all stored in registers (baseR-.

A Virtual Reality Patient and Environments for Image Guided Diagnosis Barnabas Takacs1, David Hanak1, and Kirby G. Voshburg2 1 MTA SZTAKI / VHI Group, 1111 Kende u 13-17, Budapest. HUNGARY 2 Harvard CIMIT 165 Cambridge St. Suite 702, Boston, MA, USA

Abstract. We describe a real-time virtual reality platform and a novel visualization technique designed to deliver constant rendering speed of highly detailed anatomical structures at interactive rates using portable, and low-cost computers. Our solution represents the torso section of the human body as a volumetric data set and employs label maps as the prime data format of storing anatomical structures. Multi-channel 3D textures are uploaded to the GPU and a simple pixel shader algorithm allows operators to select structures of interest in real-time. The visualization module described herein was successfully integrated into a virtual-reality 3D Anatomical Guidance System for Ultrasound Operators and its rendering performance tested on a “backpack” system. Keywords: Anatomical Guidance, FAST Exam, Augmented Reality, Virtual Human Interface.

1 Introduction Anatomical Guidance (AG) combines generic 3D models of the human body with medical real-time imaging devices, such as laparascopic or hand-held ultrasound [1]. The purpose of image-guided interventions is to visually aid medics and operators in correctly identifying anatomical structures for the purposes of examination and planning of subsequent surgical interventions. It has been shown, that such tools doubled the performance of novice users, during the in vivo examination of anesthetized pigs, effectively boosting their skills to reach the accuracy of experts without the AG support. Furthermore experts also performed nearly 150% better when using the AG system in comparison to performing diagnostics without it. These studies indicate that a portable, anatomically guided ultrasound system would find many applications in the field of patient care. For many decades researchers and medics has been seeking for safe and effective casualty care technologies to address situations where initial emergency life-saving surgery. In such circumstances prompt, aggressive resuscitation may have to be performed within a limited time in which medical care can be effective in saving life and to render a patient transportable. A leading cause of death in these situations is hemorrhage. To address these difficulties Ultrasound offers a fast method to detect internal T. Dohi, I. Sakuma, and H. Liao (Eds.): MIAR 2008, LNCS 5128, pp. 279–288, 2008. © Springer-Verlag Berlin Heidelberg 2008


B. Takacs, D. Hanak, and K.G. Voshburg

bleedings and help save the life of people in the field. It is currently the imaging solution available in a portable, low power consumption form and therefore it has very high potential for being the first such technology successfully deployed to help Medical Aid Stations (MAS). This paper describes the architecture and applications of a compact anatomically guided 3D ultrasound solution that employs virtual- and augmented reality to successfully increase the diagnostic capabilities of novice users and experts alike.

2 Internal Bleeding and the FAST Exam When a medic first reaches a casualty, his or her priorities are to determine signs of life, then to ensure breathing. Among the possible, and necessary, treatments are the insertion of a chest dart or tube to relieve tension pneumothorax. Following these initial steps, the medic’s next priorities are to search for bleeding. If there is bleeding from an extremity wound, compresses, active compresses, or, if necessary, a tourniquet are applied. Then, he may attempt to determine whether there is a treatable site of internal hemorrhage. It is unfortunately the case that vital signs, as currently defined and measured, are of minimal use in determining, in the early phase, whether there is excessive internal bleeding. It is well accepted, most notably in recent studies at the Army’s Institute of Surgical Research, that conventional vital signs such as arterial blood pressure, body temperature, or blood oxygen saturation, that is, the parameters that are easy to measure in a field setting, are not good prognostic indicators. On the other hand, it is not possible or even desirable to immediately begin resuscitation with intravenous fluids, for both tactical and medical reasons. The medic is then faced with a significant diagnostic challenge. He or she must determine if there is internal bleeding, quantify the rate of such bleeding, identify the sources of bleeding, and guide appropriate treatment to the site. If the source is visible and accessible, as it is for the extremity injuries mentioned above, the obvious first choice is the use of hemostatic agents. If this therapy fails, the direct application of coagulative thermal therapies, such as focused ultrasound or, more directly, thermal cautery, might be attempted. To address internal hemorrhage, whether from penetrating or concussive insults, the diagnostic instrument of choice that may be deployed in Medical Aid Stations (MAS) for the present as well as for the foreseeable near future is transcutaneous |ultrasound imaging. This contrasts with the situation in major medical center emergency departments, where Computer Tomography (CT) is the easiest and most definitive choice for detecting pooled blood. Several high quality, low cost, portable (battery powered or low power) instruments are now commercially available, creating the opportunity to design such a system for portable casualty care. However, since ultrasound imaging requires a level of care and sophistication that makes it unlikely that a minimally trained, unsupported operator could perform a diagnostic examination under field conditions, tools for 3D visualization, guidance and telemedicine using the toolset of Image Guided Surgery become a necessity [2]. The Focused Assessment with Sonography for Trauma (FAST) [3] examination was originally developed for the assessment of blunt abdominal trauma, but has been shown to be effective for assessing penetrating trauma as well. FAST examinations

A Virtual Reality Patient and Environments for Image Guided Diagnosis


have shown a sensitivity near 85% and a specificity of better than 95% (op. cit.). The primary indicator for a positive reading is the observation of a pool of intraperitoneal blood in regions around the liver, spleen or the pelvic organs. The standard treatment after a positive FAST examination is an exploratory laparotomy to identify and treat the injury. Before reaching the second echelon of care, however, this is not possible. Rather, the non-incisional approach using focused ultrasound therapy (High Intensity Focused Ultrasound, or HIFU [4]) is a promising, practical alternative. The effective use of HIFU depends on identifying the site to be coagulated and then establishing an acoustic path to the bleeding site. The most challenging problem is site identification in which a minimally trained medic requires the assistance of doctors from remote sites. For HIFU (or any coagulative therapy) to be effective, it must be applied to the site of injury. While it is sometimes very difficult to use the ultrasound imager to detect a region of diffuse bleeding, it is possible to identify a ruptured artery or vein, particularly if Doppler capability is available. In general, these lesions are the most serious and desirable to treat, since they lead to the most rapid exsanguination. In order to be effective in a mobile environment, the emergency instruments carried on the field must be portable, self contained, easy to operate, and provide clear guidance to personnel who had only limited training. Using ultrasound for imaging the body of an injured person, has many advantages. First, it is the only imaging solution that is available in a portable, low power consumption form that has the potential of successfully being deployed in MAS or with paramedics in civilian sector. Second, it offers a low-cost, yet powerful method to detect internal bleedings; and third, it can be used virtually without any restrictions on exposure, i.e. as many times without any risk to the patients’ health (unlike X-rays, CT, etc.). However, a critical difficulty when using US is that images are noisy and features are often rather difficult to recognize even for trained personnel. In fact, without knowing the context where the US probe is located, it is almost impossible even for doctors to recognize whet they are looking at. On the other hand several studies have shown that 3D guidance can be of significant help both for novice users and expert users in laparascopic surgical procedures. Based on this premise, in a previous study we have successfully built and demonstrated the power of 3D anatomical guidance and validated its usability. Specifically, using a clinical setup at Harvard we assessed the effectiveness and efficiency of anatomically guided ultrasound vs. traditional ultrasound for a standard EUS exam in a porcine model. Novices carrying out EUS reached only 30% in correctly identifying anatomical structures, however their performance increased to 59% when using the AGU setup. Note that this is almost the same (only 2% lower) then the performance of experts using traditional EUS (61%), which is a significant result in alone itself from the perspective of the major goals of this project. Furthermore, expert users also benefit from using AGU, as their performance too increased above 90%.

3 The CyberCare Portable Virtual Reality System The power of virtual- and augmented reality (CyberCare), systems stems from is its ability to offer flexibility, repeatability and a controlled environment that aids medical personnel to carry out certain tasks. The CyberCare VR system we developed is comprised of multiple configurable hardware software components, each addressing key aspects of the


B. Takacs, D. Hanak, and K.G. Voshburg

goals of specific rehabilitative needs. The doctor may wear a head-mounted display system (HMD) that delivers visual and auditory input directly to his or her eyes and ears. Attached to the HMD is a motion tracker that measures the rotation parameters (yaw, pitch and roll) of the head. Translation (i.e. x,y,z motion in 3D space) is controlled via a joystick device so that patient, if needed, can move freely and navigate in 3D virtual space. The head motion and joystick information are relayed to a computer, which in turn generates the appropriate visual representation of the patient model and the 3D augmented environment. This virtual reality computer is responsible for rendering the high-resolution scene in photo-realistic quality based on the patient’s movement information as received from the sensors. Along with the visual stimulus sent directly back to the HMD, the VR computer is also responsible for recording live video data, physiological measurements and biological feedback from the patient, all displayed in readily accessible form. The video input is used to support real-time interaction with the virtual environment. This virtual environment itself is comprised of a high fidelity digital human model, 3D environments combined with photo-realistic elements, such as a 360o virtual panorama as background, and overlaid animated synthetic objects. The CyberCare system we describe here was built upon our core technology, called the Virtual Human Interface or VHI [5,6] shown in Figure 1. Each of these modules play a vital part in the process of rehabilitation. As shown in the figure we devised a general architecture that can be freely and easily configured to address the various needs of a large variety of virtual-reality-based applications. Its main features are as follows: Portable Virtual-Reality Platform, High Fidelity Digital Humans, Real-Time Panoramic Video, Compositing and Augmented Reality, Physical Simulation, Low-Cost Input/Output Devices, Motion Capture (MOCAP) Support. One of the key focuses of the CyberCare system is the ability to produce virtual scenarios that are not only highly realistic, but also run on portable platforms that may be deployed in the field. Therefore the CyberCare system combines the benefits of high quality 3D virtual models with panoramic spherical video, both demonstrated in Figure 2. These environments maybe used to show our virtual patient during operation or as lying on the table.

Fig. 1. Core modules of the Virtual Human Interface (VHI) System

A Virtual Reality Patient and Environments for Image Guided Diagnosis


Fig. 2. We developed a virtual OR model complete with a fully detailed Virtual Patient that may be combined with Panoramic Video imagery obtained in the OR

4 Anatomically Guided Ultrasound System Using AR Using the CyberCare system introduced above we have developed a prototype application for medical guidance and diagnosis. The basic setup of the compact anatomically guided Ultrasound system is shown in Figure 3. On the left, a medic is using the system to detect internal bleeding of a patient lying on the ground. The entire system is carried in a backpack on his back, his hands hold three outside components, namely the US probe, the reference cube for the tracking system, and the touch-sensitive computer screen which serves as the main interface to access the system’s functions. On the right the HW elements packed tightly inside the backpack are shown. From left to right a Portable Ultrasound device (Sonosite C180 [7) is combined with a 6DOF, high precision tracker (Ascension Microbird [8]) to create a tracked sequence of US images. This information is transmitted to a small computer (Apple macMini), the heart of our Anatomical 3D Guidance solution, that combines this information with generic models of the human body, called virtual humans, and outputs guidance information to the medic’s hand-held touch screen (or HMD if the field conditions permit). Figure 4shows the hand-held touch screen and its use during a field test, respectively. Specifically, The control interface allows the operator to: • • •

Calibrate the US probe to the injured body lying on the floor, Switch between views (in order all three visualization modes together, or 3D guidance, CT cross-section, or Ultrasound only) and control visualization parameters, such as guidance info for FAST exam. The navigation buttons on the left allow rotations of the 3D models up, down, left or right, as well as zooming in or out.

One of the key technical elements of our backpack Ultrasound guidance solution is a highly detailed digital and animatable representation of the human body and its anatomy. This representation comprises a generic 3D virtual human model as well as models of internal organs segmented from volumetric data representations. The purpose of this generic 3D model is to provide guidance to the operator showing major anatomical areas of interest when patient-specific data is not available. Guidance herein refers to showing regions inside the body in order to help medics properly move the US probe and thus obtain high quality US images. Guidance information is


B. Takacs, D. Hanak, and K.G. Voshburg

Fig. 3. Overall architecture of a “backpack” anatomical guidance solution

Fig. 4. Portable Augmented reality US guidance system to support FAST

also very important to the doctors reviewing the recorded US scans, as it provides 3dimensional context information without which interpreting the US images would practically be impossible. The visualization system was optimized for portable performance and it offers different methodologies for showing regions of interest to the operator. They all rely on the notion of creating Anatomical Atlases and use them later as reference for segmenting patient-specific CT’s and visualizing target information. Instead of representing structures as polygonal objects, however, our atlas contains colored image slices combined with detailed label maps. Specifically, we focused our efforts on creating a complete model of the inside the torso section of the body and optimize it for realtime viewing and manipulation on a portable computational platform. As the first step in this process the torso segment from the volumetric data scans of the Visible Human Male Data set [9] was processed with an open-source segmentation and labeling tool, called 3D Slicer [10]. Slicer provides a set of automated techniques to specify regions of interest in volumetric data sets and construct 3D polygonal surfaces. These organ models typically contain up to one million polygons each, thereby making them difficult to use in a real-time system. Figure 5 / Left shows examples of the

A Virtual Reality Patient and Environments for Image Guided Diagnosis


named geometries including muscles, bone structure of the chest and spine, vascular structure of veins and arteries, and internal organs, (the entire model contains 478 identified anatomical structures in total.), while on the right, the same data structure is demonstrated encoded with the help of label maps.

Fig. 5. Torso model of 478 elements shown as 3D geometry and labelmap-based visualization

Fig. 6. Examples of visualizing internal anatomical structures at a constant rendering rate with the help of label maps and dynamic shading technology

The volumetric data set (MRI/CT) at the input of the visualization pipeline is defined as a stack of dynamic images (gray scale, color and/or label maps attached). In the first stage real-time image processing operators are used to automatically process all slices within a given volumetric object and implement the required preprocessing steps. These steps comprise of removing unwanted parts, image operators for noise reduction, contrast enhancement or substructure masking. Using the processed slices, a volumetric object is created and linked to an anatomical skeletal structure by an object linking mechanism. The resulting volume object then enters the rendering pipeline where it first passes through a vertex shader and subsequently in a pixel shader. The vertex shader implements local shape deformations (not used in this paper) while the pixel shader implements the algorithms required to highlight different internal structures inside the volume to help guide the US scanning process. The pixel shader can use gray scale images or color scheme for best viewing. More specifically, it can be programmed to color the pixels within a volume according to local image density, gray scale color or other algorithmically extractable features and its


B. Takacs, D. Hanak, and K.G. Voshburg

operation maybe directly controlled via label maps as well. The unit thereby offers an extended set of parameters to visualize anatomically important information a medic needs to scan. The key advantage of this approach is that it delivers high performance and constant visualization speed. Figure 6 demonstrates the power of shader-based visualization pipeline by showing the same structures with different settings. The fragment code used in the system to create the above figures is shown below. The operation of the pixel shader which takes a 3D texture (volumeTex) as input is governed by a number of parameters, called constants all stored in registers (baseRange, highlightColor, highlightDelta, massColor in c0 – c3). These constants are four element vectors with x,y,z,w coordinates. For each voxel’s output (color) the red, green, blue and transparency or alpha (r,g,b, and a) values are computed via the local algorithm and uploaded to the graphics card. When using this shader various constant settings (c0 through c3) refer to different visibility and transparency values for each pixel. The user interface of the system than allows for changing these constants to best fit the needs for showing internal structures. To take this concept even further these shader parameters can be dynamically uploaded and changed in real-time to best suite the needs of guidance and visualization. Different sets of parameters may be grouped together, interpolation schemed allow medics to interpolate and navigate these settings by a simple interface called the Disc Controller [5]. struct vertout { float4 Pos : POSITION; float4 Col : COLOR0; uvw : TEXCOORD0; }; sampler3D volumeTex : register( s0); float4 baseRange : register( c0); float4 highlightColor : register( c1); float4 highlightDelta : register( c2); float4 massaColor : register( c3); float4 main( vertout IN) : COLOR { float4 color = (float4)1; float4 texCol = tex3D(volumeTex, IN.uvw.xyz); color.rgb = texCol.rgb; if ((texCol.a > baseRange.x) && (texCol.a < baseRange.y)) color.a = highlightColor.w; else color.a = 0; return color; }

As stated above one of the key advantages of our proposed approach is to use volumetric representation in combination with label maps, a technique that delivers high performance and constant visualization speed even on portable computers. To test this assumption we measured the overall performance of the rendering algorithm on an Apple macMini computer (1.66GHz Intel Core Duo, 2GB 667 DDR2 SDRAM) we used in our “backpack” ultrasound guidance system. The speed evaluation comprised of two steps. First we used the polygonal models of increasing complexity in terms of polygon counts and recorded the overall rendering speed in frames-per-second (fps). Real-time performance requires a minimum of 15 fps update rate for the ultrasound operator to see smooth motion and perceive the system’s reaction time as seamless. As shown in the figure below even with relatively small 3D polygonal models the Apple macMini did not reach this performance level. This is largely due to the

A Virtual Reality Patient and Environments for Image Guided Diagnosis


relatively slow performance of the built-in graphics chip that is quite significantly slower than a high-end graphics card would be. In the second test the same anatomical structures are visualized using the label-map volume algorithm to compare the two methodologies. Our findings are summarized in Figure 7. The graph shows the rendering speed (i.e. how fast the computer reacts and updates images on the screen during anatomical guidance) as a function of scene complexity (i.e. how many visual elements are shown). The blue line is the performance curve for 3D models. A typical guidance system shows a few major landmarks, bones, lungs, liver, kidney, vascular structure, etc. The blue curve demonstrates that more structures are shown the slower frame rate becomes, eventually reaching only two frames per seconds (fps). On the other hand the labelmap-based shading algorithm delivers much higher speeds (30fps) on the same hardware and a constant performance even when hundreds of internal structures are shown.

Fig. 7. Comparisons of rendering performance of polygon-based with our new. labelmap-based shading algorithms, the later providing constant visualization rate.

5 Conclusion In this paper we described a compact anatomically ultrasound system that employs high fidelity 3D digital human models in compination of virtual- and augmentedreality techniques to provide medics with minimal training the capability to take highly accurate ultrasound scans. We discussed the benefits of using ultrasound as the only currently available imaging technology to detect internal hemorrhage, and showed that 3D guidance significantly increases the accuracy of diagnostic capabilities of novice and expert users alike. To fit the needs of creating a portable and lowcost AR visualization engine we introduced a novel methodology to display large numbers of complex anatomical structures for 3D guidance purposes. Specifically, our solution replaced polygonal models with colored volumetric representation combined with pre-segmented label maps. To test our approach we used a detailed 3D models segmented from the torso section of the NIH Visible Male data set and


B. Takacs, D. Hanak, and K.G. Voshburg

experimentally showed that our algorithm delivers high performance and constant speed visualization outperforming traditional polygon-based methods on the same portable computer hardware. The visualization method described herein was successfully integrated into an advanced ultrasound guidance framework and used to aid operators to find anatomical structures and obtain high quality ultrasound image sequences. Future work involves constructing reference data sets for the entire volume data made available and generalization of our algorithms to handle multiple volumes in parallel. We argue that these advanced capabilities in the future will allow for better patient care and medical response, as well as will help save the lives of many future accident victims or patients.

References 1. Elsmere, J., Stoll, J., Rattner, D., Brooks, D., Kane, R., Wells III, W., Kikinis, R., Vosburgh, K.: A Navigation System for Augmenting Laparoscopic Ultrasound. In: Ellis, R.E., Peters, T.M. (eds.) MICCAI 2003. LNCS, vol. 2879, pp. 184–191. Springer, Heidelberg (2003) 2. Boctor, E.M., Viswanathan, A., Pieper, S., Choti, M.A., Taylor, R.H., Kikinis, R., Fichtinger, G.: CISUS: An Integrated 3D Ultrasound System for IGT Using a Modular Tracking API. In: SPIE Proceedings, vol. 5367, p. 27 (2004) 3. Jones, R.A., Welch, R.D.: Ultrasonography in Trauma: The FAST Exam. ACEP Publishing (2003) 4. Kennedy, J.E.: High-intensity Focused Ultrasound in the Treatment of Solid Tumours. Nat. Rev. Cancer 5(4), 321–327 (2005) 5. Takacs, B., Kiss, B.: Virtual Human Interface: a Photo-realistic Digital Human. IEEE Computer Graphics and Applications 23(5), 38–45 (2003) 6. Takacs, B.: Special Education and Rehabilitation: Teaching and Healing with Interactive Graphics. IEEE Computer Graphics and Applications 25(5), 40–48 (2005) 7. Sonosite Inc. (2008), http://www.sonosite.com 8. Ascension Technology Corp. (2008), http://www.ascension-tech.com 9. National Library of Medicine, Visible Human Data Set, http://www.nlm.nih.gov/research/visible/getting_data.html 10. 3D Slicer, http://www.slicer.org

Suggest Documents