A Single Camera Tracking System for 3D Position ... - Springer Link

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 15, No. 10, pp. 2155-2160

OCTOBER 2014 / 2155

DOI: 10.1007/s12541-014-0576-6

A Single Camera Tracking System for 3D Position, Grasper Angle, and Rolling Angle of Laparoscopic Instruments Sangkyun Shin1,2, Youngjun Kim1, Hyunchul Cho1, Deukhee Lee1, Sehyung Park1, Gerard Jounghyun Kim2, and Laehyun Kim1,# 1 Center for Bionics, Korea Institute of Science and Technology, 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul, South Korea, 136-791 2 Digital Experience Laboratory, Korea University, 145, Anam-ro, Seongbuk-gu, Seoul, South Korea, 136-701 # Corresponding Author / E-mail: [email protected], TEL: +82-2-958-6726, FAX: +82-2-958-5649 KEYWORDS: Computer vision, Laparoscopic surgery, Medical simulation, Tool tracking, Virtual reality

Advances in computer vision technology and virtual reality technology have been achieved through improvements in camera performance and computability. Cost-effective simulation systems for laparoscopic surgery are proposed by combining such technologies. In this study, we propose a novel 3D instrument tracking technique using a single camera to obtain the 3D positions, roll angle, and grasper angle of a laparoscopic instrument. The 3D tracking of the instrument requires that all of the markers are collinear and that the actual distances between markers are known The results of our study show that the minimum uncertainty in the position was 0.237 mm and the maximum uncertainty in the position was 1.476 mm. Manuscript received: February 5, 2014 / Revised: May 26, 2014 / Accepted: June 17, 2014

1. Introduction The use of minimally invasive surgery (MIS) has been increased in order to minimize wounds and reduce patient recovery times. In particular, the demand for laparoscopic surgery, which is one form of MIS, has increased.1 Laparoscopic surgery means using surgical instruments in the form of a long bar and using a laparoscope. Laparoscopic surgery requires much training in the control of laparoscopic instruments due to the movement on a pivot, the narrow field of view, and the lack of perspective. In order to solve this problem, virtual laparoscopic surgery training systems have gradually been introduced.2-4,5 The MIST-VR simulator6 and LapSim trainer7 provide haptic functionality training in a virtual environment. However, system cost is increased by the complex configuration problems of using an encoder or magnetic sensors to estimate the position of the surgical instruments. To solve this problem, we provide the surgeon with a simple low-cost system of computer vision techniques applied to the simulation of laparoscopic surgery. The proposed system consists of a computer vision part and a virtual reality part, as shown in Fig. 1. The computer vision part finds laparoscopic instruments in the real world by using a single camera.

© KSPE and Springer 2014

The 3D tracking data obtained from laparoscopic instruments give realism to virtual instruments in virtual space. In this paper, an algorithm for estimating 3D position, an algorithm for estimating grasper angle and roll angle, and noise reduction are discussed.

2. Method 2.1 Finding marker, tracking condition and noise reduction The grasper angle and roll angle of a laparoscopic instrument are estimated from the positions of markers in real-time tracking video. The positions of markers are known more exactly once the resolution is high. A high frame rate is required for providing real-time movement in the virtual space. A camera that provided 30 frames/s with a resolution of 640×480 was used in a previous study.8 However, the Flea3 camera by the Point Grey Co. is now mounted and yields more excellent performance than the former camera. The Flea3 provides images at 60 frames/s with a resolution of 1920×1080. Hence, the 3D tracking of laparoscopic instruments is more accurate and faster than in the previous study. Markers are attached to laparoscopic instruments for tracking the

2156 / OCTOBER 2014

INTERNATIONAL JOURNAL OF PRECISION ENGINEERING AND MANUFACTURING Vol. 15, No. 10

Fig. 1 Schematic of laparoscopic simulator using computer vision

Fig. 2 Markers attached to laparoscopic instrument: P0, P1, P2 are static markers; P3 is a dynamic marker for calculating the grasper angle; P4 is a dynamic marker for calculating roll angle; and distance elements are λ0=0, λ1=P1–P0, λ2=P2–P0, λ3=P3–P0, λ4=P4–P0; P0=t+λ0r, P1=t+ λ1r, P2=t+λ2r, P3=t+λ3r, P4=t+λ4r;λi=Pi–P0, Pi=t+λir

positions of the instruments as shown in Fig. 2. The 3D position, the roll angle, and the grasper angle are calculated from the information of the attached markers. The laparoscopic instruments are tracked through the color separation when there are more than two instruments. The following conditions are required for obtaining the information: • All markers are collinear. • The actual distances between markers are known. • The actual dynamic displacements of the markers are known for estimating the roll angle and the grasper angle. • The markers on different laparoscopic instruments do not cross or meet. • Each marker on the laparoscopic instruments is indexed (from top to bottom). The main tracking technology for laparoscopic instruments is used in finding the markers attached to the laparoscopic instruments and reducing the jitter error in digital images. A high-level tracking algorithm9 is used for extracting the laparoscopic instruments from the images taken by the camera. A Kalman filter is used for reducing the jitter error of images10,11. Normally, jitter error is encountered as the recognition error of a boundary condition in the image processing. The tracking speed is influenced by parameters of the Kalman filter, so the jitter error is minimized through predicting the next position of the

Fig. 3 Applying high-level tracking algorithm and Kalman filter: (a) sequential color image, (b) (ui, vi) are determined as the center points (circles) of markers in binary image

laparoscopic instrument as the controlling parameter of the processing speed (Fig. 3).

2.2 Finding 3D position With Haralick’s algorithm, if more than three points lie on a common line and the distances between n collinear points are given, the position vector t and the relative orientation vector r of the n collinear points can be recovered.8,12-16 The input parameters for the estimation are 1) the intrinsic parameters of the camera, 2) the actual distances between the three static fiducial markers, and 3) the image coordinates of the markers. The actual gaps between the markers are measured with a caliper. The image coordinates (u, v) of the markers are taken from the image pixel coordinates (Fig. 3). From the input parameters, the vectors t and r are obtained as described below. ui c 0 0 1 [ t + λi r ] vi = K ( t + λi r)

(1)

1 c

where K is a 3×3 upper diagonal matrix whose components are the camera parameters. From Eq. (1), with a univariate matrix Kc = diag(f, f,1), Haralick proposed a homogeneous linear system, Ar A t

r =0 t

(2)

where A is a 2n×6 matrix. This linear system can be solved with n ≥ 3 distinct points. Eq. (2) is reformulated as a classical optimization problem,


min Ar r + At t

T

r r=1

subject to

OCTOBER 2014 / 2157

(3)

where Ar and At are two 2n×3 matrices containing the camera parameters and values of λi, The solution to Eq. (3) is obtained through singular value decomposition of the following symmetric matrix E: T

T

–1 T

E = Ar ( I – At ( At At ) At )Ar

(4)

The eigenvector corresponding to the smallest eigenvalue of E is the relative orientation vector r, and the position vector t is given by t = – (AtTAt)–1AtTArr. The Ar and At are defined as follows: λ0 0 … … 0

0

0 λ0 … … 0

0

Ar = … … … … … … At 0 0 … … λn – 1 0 0 0 … … 0 λn – 1

At =

⎛ ⎞ υ0 ⎟ 1 0 0 ⎜⎜ KC – v0 0 0 1 ⎟ ⎟ 010⎜ ⎝ ⎠ 1 ⎛ ⎞ υn – 1 ⎟ 1 0 0 ⎜⎜ KC – vn – 1 0 0 1 ⎟ ⎟ 010⎜ ⎝ ⎠ 1

Fig. 4 Experimental setup for position accuracy test: (a) the optical tracker and computer vision tracking system, (b) the laparoscopic surgical instrument with grasper removed and probe attached

2.3 Grasper angle and rolling angle The angle of the grasper is correlated with the distance between the static marker (P0) and the dynamic marker (P3), because the angle and the distance are represented by a straight line. The distance is given by Eq. (2), which is then reinterpreted as

⎛ λ3 f 0 −λ3 u3 ⎜ ⎝ 0 λ3 f −λ3 v3

⎛ rx⎞ ⎜r ⎟ ⎜ y⎟ f 0 −u3⎞ ⎜⎜ rz⎟⎟ =0 ⎟ 0 f −v3 ⎠ ⎜ tx ⎟ ⎜ ⎟ ⎜t ⎟ ⎜ y⎟ ⎝t ⎠

(5)

linear interpolation using the predicted λ3 and the measured λ3. 85° × λ3 AngleGrasper = ------------------------------------------------MAX ( λ3 ) – MIN ( λ3 )

The maximum open angle of grasper is 85o, The measured maximum λ3 open angle of grasper is MAX(λ3). The measured minimum λ3 open angle of grasper is MIN(λ3).The approximation to exchange from the distance to the rolling angle is made in the same way (Eq. (9)). 360° × λ4 AngleRolling = ------------------------------------------------MAX ( λ4 ) – MIN ( λ4 )

z

where λ3 is the distance between the static marker and the dynamic marker, (rx, ry, rz) is the vector r, and (tx, ty, tz) is the vector t. Since Eq. (5) is homogeneous, we can deduce λ3 from one of the following equations: u3 tz – ftx λ3 = ------------------frx – u3 rz

(6)

v3 tz – fty λ3 = -----------------fry – v3 rz

(7)

The value of the distance ë3 between the static marker (P0) and the dynamic marker (P3) is obtained from Eqs. (6) or (7) using the image coordinates (u3, v3), the vector t, and the vector r. The vectors t and r are calculated from Section 2.1.1. The grasper angle is calculated by

(8)

(9)

3. Experiments 3.1 Experimental environment and methods 3.1.1 Accuracy test of position An optical tracker made by the NDI Co. is used for accuracy testing as shown in Fig. 4. The accuracy of the NDI tracking system is 0.3 mm. A new surgical instrument is used for accuracy measurements, to which are attached the markers for computer vision tracking and a probe for optical tracking, as shown in Fig. 4. The size of each marker is 2 mm and the distance between markers is 10 mm. As illustrated by Fig. 4, the positions of the tool-tip are compared. The location of the tool-tip is measured to apply the offset with the value of λi based on the

2158 / OCTOBER 2014


Fig. 5 Preparing laparoscopic surgical instrument for accuracy test: (a) setting grasper angle using the semicircular protractor, (b) setting roll angle on arranged line of laparoscopic surgical instrument

standard of the original value from the computer vision tracking. The optical tracker measures the location of the tool-tip on the standard of the tool-tip after the pivot. To improve the accuracy of the location measured by the pivot of the optical tracker when using the new surgical instruments, the graspers on the original surgical instruments are removed. Setting the ground truth for comparisons as the location data acquired by the optical tracker, the accuracy is measured for the location values calculated through the computer vision tracking. The values of 100 samples are acquired for the same position as under the stationary condition, and then the averages of these values are compared by the iterative closest point (ICP) method.17

3.1.2 Accuracy test of grasper angle and rolling angle Approximate angles and actual angles are compared. The approximate grasper angles from the computer vision tracking are compared with the actual grasper angles, which are measured 10 times at 15o, 30o, 45o, 60o, and 85o, respectively, as shown in Fig. 5. The experiment is repeated in four different positions (center, left, right, and bottom of the camera image). Moreover, the approximate roll angles from the computer vision tracking are compared with the actual roll angles, which are measured 10 times at 0o, 72o, 144o, 216o, and 288o, respectively (Fig. 5). This experiment is also repeated in four different positions (center, left, right, and bottom of the camera image). 3.2 Results As shown in Fig. 6, the position accuracy of vision tracking system is 0.438±0.231 mm. The accuracy of grasper angle and rolling angle are shown in Figs. 7 and 8.

Fig. 6 Position accuracy results: (a) positions measured by the optical tracker. The standard deviations were calculated with about 100 values acquired from among the stopping positions. The minimum value is 0.034 mm and the maximum value is 0.062 mm, (b) positions measured by the computer vision tracking system. The standard deviations were calculated like those in the method of the optical tracker. The minimum value is 0.007 mm and the maximum value is 0.289 mm, (c) average values of positions measured by the optical tracker (red squares) and the vision tracking system (green squares) in 3D space. The distance between 3D coordinates that are measured with these two systems is minimized by the iterative closest point (ICP) method. As a result, the minimum value is 0.237 mm and the maximum value is 1.476 mm

4. Discussion In the case of the middle of the image, the uncertainty in the position attained the minimum value of 0.237 mm. In the case of the verge of the image, the uncertainty in the position attained the maximum value of 1.476 mm. The accuracy at the middle defines the confidence level. Conversely, the accuracy at the verge is within the allowable limits of error. Hence, the error level at the verge is acceptable for the laparoscopic surgery simulator. The methods for improving the accuracy at the verge of the image are to upgrade the calibration of the camera and provide an effective undistortion of the camera. As shown in Fig. 6, the confidence level of the accuracy of the position depends on the


OCTOBER 2014 / 2159

Fig. 9 Integrated simulation system for laparoscopic training in gallbladder removal surgery

Fig. 7 Results for grasper angle accuracy: (a) The average error of the center image is 2.236o, the minimum value is 0.061o, and the maximum value is 0.233o, (b) The average error of the bottom image is 3.589o, the minimum value is 0.121o, and the maximum value is 5.125o, (c) The average error of the left image is 7.434o, the minimum value is 2.375o, and the maximum value is 8.517o, (d) The average error of the right image is 18.675o, the minimum value is 7.153o, and the maximum value is 21.619o

Fig. 8 Results for roll angle accuracy: (a) The average error of the center image is 2.080o, the minimum value is 0.033o, and the maximum value is 2.312o, (b) The average error of the bottom image is 3.273o, the minimum value is 2.412o, and the maximum value is 4.195o, (c) The average error of the left image is 5.134o, the minimum value is 2.362o, and the maximum value is 11.083o, (d) The average error of the right image is 13.932o, the minimum value is 9.097o, and the maximum value is 15.148o

depth at the middle of the image. As a result of this study, the position acquired is accurate to less than 1 mm. Even if the space of the laparoscopic surgery is determined as 100 mm × 100 mm × 100 mm, the position acquired is still accurate to less than 1 mm. Therefore, the proposed laparoscopic surgical instrument tracking system is able to track movements like those in actual laparoscopic surgery.

The large size of the point at the upper-right in Fig. 6 indicates that the jitter error there is more than at other positions. The jitter error is possible because the image processing is heterogeneous with respect to position. The angle of the grasper and the angle of roll are dependent on the accuracy of the position. When the grasper angle is 0o, the length from P0 to P3 at the marker of the laparoscopic surgical instrument is 11.93 mm. When the grasper angle is 85o, the length from P0 to P3 at the marker of the laparoscopic surgical instrument is 18.12 mm. The accuracy is superior because the orientation of the grasper is expressed as the length (from 0 mm to 6.19 mm) instead of the angle (from 0o to 85o). Similarly, when the roll angle is 0o, the length from P0 to P4 at the marker of the laparoscopic surgical instrument is 30 mm, and when the roll angle is 360o, the length from P0 to P4 at the marker of the laparoscopic surgical instrument is 60 mm. Hence, the difference in distance is 30 mm. In this respect, the effect that the wide range of changes in the grasper angle has on the error is relatively small. Although the results of the experimental data were analyzed, the roll error still existed. However, the error is discontinuous at the coordinate singularity of 0o and 360o, so additional research is needed. We integrated the proposed instrument tracking method into laparoscopic training simulation software developed for gallbladder removal surgery18,19 (Fig. 9). The virtual instruments of the software are manipulated in real-time while the positions and status of the instruments are obtained with the proposed tracking method. The actual movement of laparoscopic instruments was naturally reconstructed by the simulation software. The speed of the augmented simulation software was checked at rates above 25 frames/s.

5. Conclusion Currently, high-definition images and high frame rates are required for processing 3D information in real time. For this reason, the test was performed with a high-performance camera. Averaged over all images, the accuracy was improved by 26% compared with that in a previous study. The instrument of actual laparoscopic surgery is tracked with a single endoscopic camera. An improved image input instrument, noise reduction technology, and a 3D information tracking algorithm are used with the proposed single camera. The movement of surgical instruments in the virtual environment is naturally rendered. Our laparoscopic surgical simulation system implements the simulation with virtual

2160 / OCTOBER 2014


instruments using the information gathered by the tracking method. Using the virtual reality technology, the activity of internal organs can be displayed in the simulation and the separation of internal organs can be practiced. Furthermore, some effects such as bleeding and smoke enable the simulation to mimic real operating environments. The technology of 3D tracking with a single camera can be used in real laparoscopic surgery, and low-cost implementation of the system is feasible due to its simple structure.

ACKNOWLEDGEMENT This Research is supported by the Multimodal Image Guided Micro Surgical System Project of the Korea Institute of Science and Technology (2E24551).

REFERENCES 1. Iribarne, A., Easterwood, R., Chan, E. Y., Yang, J., Soni, L., et al., “The Golden Age of Minimally Invasive Cardiothoracic Surgery: Current and Future Perspectives,” Future Cardiology, Vol. 7, No. 3, pp. 333-346, 2011. 2. Schijven, M. P., Jakimowicz, J. J., Broeders, I., and Tseng, L. N. L., “The Eindhoven Laparoscopic Cholecystectomy Training Course Improving Operating Room Performance using Virtual Reality Training: Results from the First Eaes Accredited Virtual Reality Trainings Curriculum,” Surgical Endoscopy and other Interventional Techniques, Vol. 19, No. 9, pp. 1220-1226, 2005. 3. Gauger, P. G., Hauge, L. S., Andreatta, P. B., Hamstra, S. J., Hillard, M. L., et al., “Laparoscopic Simulation Training with Proficiency Targets Improves Practice and Performance of Novice Surgeons,” The American Journal of Surgery, Vol. 199, No. 1, pp. 72-80, 2010. 4. Zhang, A., Hünerbein, M., Dai, Y., Schlag, P. M., and Beller, S., “Construct Validity Testing of a Laparoscopic Surgery Simulator (Lap Mentor®),” Surgical Endoscopy, Vol. 22, No. 6, pp. 1440-1444, 2008. 5. Ahn, B. and Kim, J., “Efficient Soft Tissue Characterization under Large Deformations in Medical Simulations,” Int. J. Precis. Eng. Manuf., Vol. 10, No. 4, pp. 115-121, 2009. 6. Wilson, M., Middlebrook, A., Sutton, C., Stone, R., and McCloy, R., “MIST VR: A Virtual Reality Trainer for Laparoscopic Surgery Assesses Performance,” Annals of the Royal College of Surgeons of England, Vol. 79, No. 6, pp. 403, 1997. 7. Kovac, E., Azhar, R. A., Quirouet, A., Delisle, J., and Anidjar, M., “Construct Validity of the LapSim Virtual Reality Laparoscopic Simulator within a Urology Residency Program,” Canadian Urological Association Journal, Vol. 6, No. 4, pp. 253-259, 2012. 8. Shin, S., Kim, Y., Kwak, H., Lee, D., and Park, S., “3D Tracking of Surgical Instruments using a Single Camera for Laparoscopic Surgery Simulation,” Studies in Health Technology and Informatics, Vol. 163,

pp. 581-587, 2010. 9. Marselli, C., Daudet, D., Amann, H. P., and Pellandini, F., “Application of Kalman Filtering to Noise Reduction on Microsensor Signals,” Proc. of the Colloque Interdisciplinaire en Instrumentation, Vol. 2, pp. 443-450, 1998. 10. Senior, A., Hampapur, A., Tian, Y. L., Brown, L., Pankanti, S., and Bolle, R., “Appearance Models for Occlusion Handling,” Image and Vision Computing, Vol. 24, No. 11, pp. 1233-1243, 2006. 11. An, L., Xue-mei, X., Qiaoyun, G., and Qin, M., “Robust Segmentation using Marked Regular Pyramid and Mean Shift,” Proc. of International Symposium on Intelligence Information Processing and Trusted Computing (IPTC), pp. 341-344, 2010. 12. Haralick, R. M., “Monocular Vision using Inverse Perspective Projection Geometry: Analytic Relations,” Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 370-378, 1989. 13. Haralick, R. M., Lee, D., Ottenburg, K., and Nolle, M., “Analysis and Solutions of the Three Point Perspective Pose Estimation Problem,” Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 592-598, 1991. 14. Haralick, R. M. and Shapiro, L. G., “Computer and Robot Vision,” Addison-Wesley Pub. Co., pp. 44-59, 1992. 15. Doignon, C., “An Introduction to Model-Based Pose Estimation and 3-D Tracking Techniques,” in: Scene Reconstruction Pose Estimation and Tracking, Stolkin, R., (Ed.), InTech, pp. 359-383, 2007. 16. Doignon, C., Nageotte, F., Maurin, B., and Krupa, A., “Pose Estimation and Feature Tracking for Robot Assisted Surgery with Medical Imaging,” in: Unifying Perspectives in Computational and Robot Vision, Kragic, D. and Kyrkl, V., (Ed.), Springer, pp. 79-101, 2008. 17. Kitware Inc., “The VTK User’s Guide,” 2010. 18. Kim, Y., Lee, S., Roy, F., Lee, D., Kim, L., And Park, S., “Carving Mesh with Deformation for Soft Tissue Removal Simulation,” in: Mesh Processing in Medical Image Analysis 2012, Levine, J., Paulsen, R. R., and Zhang, Y. Z., (Ed.), Springer, pp. 70-79, 2012. 19. Jun, Y., and Park, S., “Polygon-based 3D Surgical Planning System for Hip Operation,” Int. J. Precis. Eng. Manuf., Vol. 12, No. 1, pp. 157-160, 2011.