Haptic Exploration Laboratory Technical Report 04-1, The Johns Hopkins University, June 1, 2004. This work was supported by NSF grants EIA-0312551 and EEC-9731478, and Whitaker Foundation grant RG-02-911.
Modeling and Measuring the Dynamic 3D Effects of Needle Insertion in Soft Tissue Chad Schneider1 , Jessica R. Crouch2 , and Allison M. Okamura1 1
2
Department of Mechanical Engineering Engineering Research Center for Computer Integrated Surgical Systems and Technology The Johns Hopkins University, Baltimore, MD, USA {cschneider, aokamura}@jhu.edu,
[email protected]
Abstract. Models that predict soft tissue deformation during needle insertion could improve the accuracy of needle based medical procedures. In an effort to establish a method for defining and validating such models, we present an experimental setup including hardware and software for recording and measuring the time-dependent, 3D deformation that occurs as a needle is inserted into soft material. Algorithms are described for tracking a needle and fiducials in 3D based on stereo video recordings. The results of the initial experiment clearly demonstrate the non-linear, viscoelastic nature of deformations caused by needle insertion.
1
Introduction
A major source of error in image guided procedures such as needle biopsy and brachytherapy is the soft tissue deformation that occurs as a needle is inserted. The guidance accuracy provided by a pre-operative planning image is limited by the difference between the location of a tissue target pre-operatively and the location of the same target intra-operatively, when the target and surrounding tissues are deformed by needle insertion forces. If the deformation caused by needle insertion could be accurately modeled and predicted, then treatment plans could be adapted to compensate for expected tissue deformation, thereby improving needle procedure accuracy. A primary impediment to the development of realistic deformable tissue models is the scarcity of data about soft tissue mechanical properties. Very limited data is available for prostate and other soft pelvic and abdominal organs. Especially lacking from the literature are well-controlled experiments investigating the dynamic response of living soft tissue to interaction with needles and surgical tools. A number of factors make the design of such experiments challenging, including acquisition speed limits for 3D imaging modalities and the difficulty of accurately extracting tissue motion from images. Our approach to this problem has been to conduct needle insertion experiments in a highly controlled, synthetic tissue environment in order to validate an experimental methodology for recording, measuring, and modeling the dynamic, three dimensional deformation that occurs as a needle is inserted into
2
C. Schneider, J. Crouch, A. Okamura
soft material. Special attention is paid to capturing the time-dependent data necessary for creating and validating non-linear finite element tissue models and needle models that explicitly include cutting and friction forces. Consideration is also given to the process of evaluating and comparing the fidelity of different needle and tissue models. The goal is to establish a methodology which could be applied in future experiments with real soft tissues. While our work focuses on dynamic, 3D deformations, much of the previous work in modeling the interaction of tools with soft tissue has been limited to 2D models. DiMaio and Salcudean fit a 2D finite element model to needle insertion force recordings and estimated a 3D correction term based on a 3D finite element model [3]. In contrast to our methodology, they measured displacements on the surface of the material instead of inside the volume where force was applied. Alterovitz, et al. developed a simulation of needle insertion in the prostate, modeling the motion of a target within the prostate for use in sensorless planning [1]. Their model accounts for target motion in a 2D plane through the prostate. Kerdok, et al. created the Truth Cube with fiducials inside a phantom tissue volume and developed a 3D model of its material properties by measuring fiducial motion while applying various compressive forces [5]. However, this work did not address tool-tissue interaction as our work does.
2
Methods
The goal of the experiment was to collect data about cutting and friction forces on the needle and to determine the deformation of the phantom tissue as the needle was inserted. For each trial, the needle was inserted at a constant velocity into a phantom tissue mold that was impregnated with a grid of tiny fiducials. By tracking the displacement of these fiducials, the deformation of the phantom tissue could be determined at a number of time steps during needle insertion. The experimental setup consisted of the following components: – Hardware to provide nearly frictionless motion of the needle and to measure forces on the needle – Software to control the needle insertion device and to record needle force, position, and other parameters detected by the device – Calibrated stereo video cameras for recording the experiment visually – Algorithms for automatic segmentation and tracking of the needle and fiducials in the recorded video – Computer vision software to reconstruct the 3D coordinates of tracked points – An algorithm for temporally and spatially registering the video data and needle data from multiple insertion trials 2.1
Needle Insertion Device
Hardware Design The setup consists of a linear stage to translate the needle as much as 152.4mm along a linear path. A modified 18 ga. brachytherapy needle
Modeling and Measuring the Effects of Needle Insertion
3
is mounted to an ATI Nano-17 6-axis force sensor through a machined aluminum port. During insertion, friction along the needle shaft and cutting forces at the head of the needle are measured along the z-axis of the force sensor. The motion of this stage is controlled by an digitally encoded Maxon A-max 22 dc motor configured as a capstan drive. The motor is connected to a 128:1 GP 22-A metal planetary gearhead, greatly reducing the shaft output speed and enhancing the resolution of the encoder. A threaded output shaft is mounted to the motor shaft to provide a stable connection to the braided wire of the capstan drive and limiting backlash. Software Control The needle stage is controlled through a C++ graphical user interface. The program allows the user to start and stop the insertion, reset the insertion, and change several parameters of the motion. Velocity is kept constant by feeding a simple trajectory generation algorithm, Posdesired = Poscurrent + Veldesired 4t,
(1)
into a Proportional Derivative controller that applies a voltage based on the difference between the current position and the desired position. 4t is updated every time-step of the program with milli-second resolution. A hardware limit was designed into the needle stage to prevent the needle from driving into the back of the mold, bending the needle, and ruining the silicone gel. With all systems ready to collect measurements, both the needle stage and the vision system were manually prompted to begin their automated processes. Precise coordination of these actions was unnecessary because the needle was initially set 5 to 10mm from the front of the gel to allow for the ramp to constant velocity. 2.2
Phantom Tissue
The phantom tissue is constructed from GE RTV-6166 2-part silicone dielectric gel, the same homogeneous silicone material used by Kerdok [5] to create the Truth Cube. The mold is a five-sided box created from 6.35mm cast-acrylic pieces and left open on the side facing the needle, as shown in Fig. 1. The phantom is made in several layers at a ratio of 30 parts A to 70 parts B by weight. Each layer is poured into the mold to a depth of approximately 8mm and immediately placed into a 130mmHg vacuum chamber to remove any air bubbles. At room temperature, the RTV has set enough to support the weight of the fiducials in four to five hours. Once the RTV had partially set, a row of 26 fiducials were dropped onto the surface of the RTV through a template that spaced them 5mm apart along the width of the mold, pictured in Fig. 2. The fiducials are 0.8mm diameter alloy 102 copper balls. They were chosen because their small size minimizes their influence on the material properties of the phantom tissue. By only allowing the material to partially set before the new layer was added, strict borders were prevented from forming between the layers because layer separation could harm the needle insertion results. A total of 15 layers were stacked,
4
C. Schneider, J. Crouch, A. Okamura
filling the entire mold with a grid of 390 fiducial markers evenly spaced in two dimensions with no more than 3mm out of plane variation. The phantom tissue mold is screw-mounted to an acrylic support block to align the needle in the plane of the fiducials. This block and the needle stage are both mounted to a main platform to rigidly mount the needle to the phantom tissue. To allow multiple trials, the mold could move laterally in 5mm increments. As shown in Fig. 3, the hardware setup aligned the needle to travel directly between the columns of fiducials. 2.3
Motion Tracking
Each trial was recorded by two Sony DFW-X700 digital cameras at a 15Hz frame capture rate and 640 x 480 frame resolution. This resulted in 150 1MB frames from each camera for each 10 second trial. Before initiating experiment trials, a camera calibration procedure was performed to determine the position and orientation of the left camera relative to the right camera [2]. This calibration procedure indicated an average registration error of 0.2 pixels, which equates to approximately 0.031mm error in reconstructed world space coordinates. This is desirable performance because the error from the computer vision system is two orders of magnitude smaller than the range of fiducial and needle motion recorded in the experiment. In the following sections, algorithms for segmenting and tracking the fiducials and the needle tip in the video recordings are described. These algorithms provide the 2D image coordinates of the objects of interest in corresponding right and left camera frames. From the right and left image coordinates, the 3D world space coordinates of the fiducials and needles are computed using a standard computer vision algorithm to reconstruct 3D points from stereo images [4]. The camera calibration procedure provided the parameter values necessary for the reconstruction.
Fig. 1. CAD model showing the setup of the needle stage and phantom tissue.
Fig. 2. Building phantom tissue. Fiducials, circled here, were dropped through hollow tubes onto a partially cured layer of silicone.
Modeling and Measuring the Effects of Needle Insertion
5
Fiducial Tracking A 9x9 grid of fiducials was segmented around the needle track in each trial. Three rows of fiducials lay on one side of the needle track and six rows lay on the other side. This configuration provided dual-sided information about the deformation in the region the closest to the needle and single-sided information about the deformation further away. This approach was designed to use the constructed phantom tissue as efficiently as possible. The fiducials are clearly visible because the gel material is transparent and the entire apparatus is backlit during the experiment. The fiducials show up as shadows in the phantom tissue. For the first frame of each video sequence, the approximate image coordinates of the 81 fiducials of interest are manually identified using an interactive software program that allows the user to click near each of the fiducials with the mouse. An automated algorithm then finds the center of the selected seeds in each of the video frames. The fiducial tracking algorithm upsamples each video frame by a factor of 3 using bicubic interpolation. Then the images are convolved with a rotationally symmetric Laplacian of Gaussian kernel that has the same width as the fiducial shadows. A region of interest near each selected fiducial is examined, and the location of the maximum match value in the convolved image is identified as the fiducial’s location. This approach identifies fiducials with subpixel accuracy. The result of this tracking algorithm for one trial can be seen in Fig. 3.
Needle Tracking Needle tracking is more complicated than fiducial tracking because the visible length of the needle in each frame is not known before segmentation. Also, needle orientation can change slightly between trials, complicating a tracking algorithm based on convolution. The approach taken here is based on an analysis of difference images computed between successive video frames.
Fig. 3. In left and right camera views untracked fiducials appear as small dark spots, and automatically segmented fiducials are marked by white dots. Along the bottom of the images needle tracks from previous insertion trials are visible. The larger dark spot in the upper center portion of the image is the shadow of a mounting screw.
6
C. Schneider, J. Crouch, A. Okamura
In the first frame of each video sequence the insertion point of the needle into the phantom tissue and another point along the shaft of the needle are manually selected by mouse click. This allows the origin and slope of the needle path to be computed. Then the position of the needle tip in each frame is automatically computed using the following procedure: 1. Mask the images so only pixels in line with the needle path are considered. 2. Compute the difference image between successive frames as shown in Fig. 4, and sum across the direction perpendicular to the needle. 3. Square the distribution. The leading peak indicates the needle tip. (Fig. 5)
3
Data Registration
The video recording was temporally registered with the force recording by matching the video frame showing puncture with the peak in the axial needle force indicating tissue puncture. Successive trials were not inherently registered spatially because the phantom tissue was shifted before each trial to give the needle a new, clear path to traverse. The first step for spatial registration was to reconstruct the 3D coordinates of the needle tip at each time step and to fit a line to the 3D needle data, as shown in Fig. 6. The needle insertion point was taken as the origin, and the needle and fiducials were rotated so that the needle inserted in the positive z direction and the fiducial grid lay in the (x,z) plane. This registration allows direct comparisons between corresponding fiducials in different trials.
4
Results and Discussion
Needle insertion trials were performed at three insertion speeds: 9mm/s, 12mm/s, and 15mm/s. There were three repetitions of each insertion speed. The correspondence between same-speed trials showed good reproduceablity of the results.
Fig. 4. The difference image, masked to show the region of interest along the needle’s path. The bright bands indicate a shift in the position of the needle.
Fig. 5. This distribution is generated by summing the squared difference image intensities in the direction perpendicular to the needle shaft.
Modeling and Measuring the Effects of Needle Insertion
7
Fig. 6. The 3D needle path reconstructed from the tracked tip positions is graphed with a dashed line. The needle trajectory line calculated from this data is drawn as a solid line. The needle data is shown with a 3D rendering of the fiducials segmented from the same 15mm/sec insertion trial.
Table 1 shows the average maximum displacement and standard deviation across the three same-speed trials at 7 locations along the needle. Figure 7 shows the displacement and velocity at a single fiducial position over all 150 frames of recording. These graphs clearly show that the time-dependence of the deformation has been captured. The deformation is non-linear, and material relaxation occurs after the needle halts. Table 1. Max. displacements and std. dev. for 7 positions near the needle. 12mm/sec distance along needle (mm) 18.9 maximum displacment (mm) 4.6 standard deviation 0.45
5
27.0 4.8 0.54
35.8 4.6 0.24
44.3 4.4 0.51
52.2 4.0 0.28
60.1 3.8 0.10
66.9 3.5 0.05
Conclusions and Future Work
The experiment has shown that our methodology is suitable for capturing the dynamic, 3D aspects of deformation caused by needle insertion. Fitting linear and non-linear finite element models to the acquired data remains as future work. The fitness of a model can be evaluated in our framework by 1. Providing a needle force recording to the model as input, and predicting tissue deformation using the model. 2. Comparing the predicted deformation with the actual deformation, as measured by the motion of tracked fiducials. The next experiment will have holes in the back wall of the phantom tissue mold so the needle can be inserted all the way through the tissue block and come out the other side. This experiment will allow us to differentiate between the needle forces due to tissue cutting and the forces due to sliding friction.
8
C. Schneider, J. Crouch, A. Okamura
Fig. 7. In the graphs, the motion of one fiducial position near the needle’s path is averaged over trials performed at the same insertion speed. The dependence of the fiducial’s motion on the needle velocity is evident, as is the non-linear material relaxation that occurs after the needle halts.
A similar methodology could also be applied to real soft tissue samples to determine deformation due to needle insertion. Bi-plane x-ray could replace stereo cameras if radio-opaque fiducials were implanted in the tissue.
Acknowledgment The authors thank Darius Burschka and Panadda Marayong for their help acquiring and setting up the stereo-vision system critical to collecting our data.
References 1. R Alterovitz, J Pouliot, R Taschereau, I Hsu, and K Goldberg. Sensorless planning for medical needle insertion procedures. IEEE International Conference on Intelligent Robots and Systems (IROS 2003), Oct. 2003. 2. Jean-Yves Bouguet. Camera calibration toolbox for matlab. Website, 2004. http: //www.vision.caltech.edu/bouguetj/calib_doc/. 3. S P DiMaio and S E Salcudean. Needle insertion modelling and simulation. IEEE Trans. on Robotics and Automation: Special Issue on Medical Robotics, Oct. 2003. 4. D A Forsyth and J Ponce. Computer Vision, A Modern Approach. Prentice Hall, Upper Saddle River, New Jersey, 2003. 5. A E Kerdok, S M Cotin, M P Ottensmeyer, A Galea, R D Howe, and S L Dawson. Truth cube: Establishing physical standards for real time soft tissue simulation. Medical Image Analysis, 7, 2003. 6. A. J. Harrison T. A. Thomas P. N. Brett, T. J. Parker and A. Carr. Simulation of resistance forces acting on surgical needles. J. of Eng. in Medicine, Sept. 1997.
