sensors Article
An Augmented Reality System Using Improved-Iterative Closest Point Algorithm for On-Patient Medical Image Visualization Ming-Long Wu 1 1 2 3 4
*
ID
, Jong-Chih Chien 2 , Chieh-Tsai Wu 3 and Jiann-Der Lee 1,3,4, *
Department of Electrical Engineering, Chang Gung University, Taoyuan 333, Taiwan;
[email protected] Degree Program of Digital Space and Product Design, Kainan University, Taoyuan 333, Taiwan;
[email protected] Department of Neurosurgery, Chang Gung Memorial Hospital, LinKou, Taoyuan 333, Taiwan;
[email protected] Department of Electrical Engineering, Ming Chi University of Technology, New Taipei City 24301, Taiwan Correspondence:
[email protected]; Tel.: +886-3-211-8800
Received: 30 May 2018; Accepted: 17 July 2018; Published: 1 August 2018
Abstract: In many surgery assistance systems, cumbersome equipment or complicated algorithms are often introduced to build the whole system. To build a system without cumbersome equipment or complicated algorithms, and to provide physicians the ability to observe the location of the lesion in the course of surgery, an augmented reality approach using an improved alignment method to image-guided surgery (IGS) is proposed. The system uses RGB-Depth sensor in conjunction with the Point Cloud Library (PCL) to build and establish the patient’s head surface information, and, through the use of the improved alignment algorithm proposed in this study, the preoperative medical imaging information obtained can be placed in the same world-coordinates system as the patient’s head surface information. The traditional alignment method, Iterative Closest Point (ICP), has the disadvantage that an ill-chosen starting position will result only in a locally optimal solution. The proposed improved para-alignment algorithm, named improved-ICP (I-ICP), uses a stochastic perturbation technique to escape from locally optimal solutions and reach the globally optimal solution. After the alignment, the results will be merged and displayed using Microsoft’s HoloLens Head-Mounted Display (HMD), and allows the surgeon to view the patient’s head at the same time as the patient’s medical images. In this study, experiments were performed using spatial reference points with known positions. The experimental results show that the proposed improved alignment algorithm has errors bounded within 3 mm, which is highly accurate. Keywords: image alignment; improved-ICP algorithm; head-mounted display
1. Introduction With the rapid development of computer and image processing technologies, Augmented Reality (AR) has become more widely used in many different areas, such as education [1], entertainment [2], medicine [3], etc., and it also adds the feelings of reality for the user compared to Virtual Reality (VR) [4]. In AR applications, images of the real world are captured with a camera and virtual objects are drawn on top of them according in the user’s designated locations. This can be useful in medical applications such as Image-Guided Surgery (IGS). Nowadays, the use of IGS system [5] plays a very significant role in the healthcare industry. The motivation for introducing IGS is to reduce invasiveness and to implement non-contact alignment. IGS will improve surgical safety and help to avoid contact with the patient with the physical alignment instrument, which would pose a risk of infection. In 2015, Sensors 2018, 18, 2505; doi:10.3390/s18082505
www.mdpi.com/journal/sensors
Sensors 2018, 18, 2505
2 of 15
Kersten-Oertel et al. [6] applied augmented reality to Image-Guided Neurosurgery System (IGNS) and used expensive commercial spatial alignment instruments that require physical contacts, which our proposed system seeks to avoid. In 2014, Deng et al. [7] proposed using a tablet computer to achieve image-guided neurosurgery. However, his proposal included the use of multiple markers which need to be placed on the tablet computer and by the dummy head to correctly align and display the head on the tablet computer, so it is not actually superimposing the medical image on top of the skull’s surface. In this study, the preoperative medical imaging data obtained of the patient would help the medical staff to evaluate the lesion and to confirm the location of the lesion using AR. The preoperative surgical image-assisted navigation system can also be used to plan the surgery routine and help locate the tumor. Various studies such as by Macedo et al. [8] intensified the research in the use of medical imaging in augmented reality. In other words, in the use of AR in surgery, the patients’ medical images must be obtained using various instruments before the operation, according to the needs of the surgeries and the conditions of the patients, medical images such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI) and Ultrasound Image can be used to analyze the location of the lesion through these preoperative data [9]. Today’s surgical medical imaging technology still requires surgeons to look up from time to time to the images on the monitor to aid the progress of the surgery [10]. Microsoft has developed several Hololens applications for medical treatment [11,12], but most of them have not been applied to the alignment of the virtual images with the human body; instead, the virtual objects are placed in an empty space environment for educational guidance purposes. However, if it is possible to display the virtual data derived from medical images at the same time as the patient [13], then this technology is bound to be the trend of the future. This goal requires accurate alignment of the positions of the virtual data with the patients. Thus, how to correctly align medical images with patients’ positions is very crucial. The traditional alignment method, the Iteration Closet Point (ICP) algorithm, which was introduced by Besl and McKay [14], and Zhang [15], uses rigid transformation and continuous iterative calculation, to find the best coordinate transformation for a group of 3D-points from one coordinate system to another coordinate system via translations and rotations [16]. In 2001, Penney et al. [17] proposed the StochastICP algorithm, which uses a random perturbation at each iteration to escape from the local minimum, and to converge on the globally best solution. However, their results show that the total number of iterations will become much larger than the traditional ICP method and would be time-consuming. In 2010, Andriy et al. [18] proposed the Coherent Point Drift (CPD) alignment algorithm. Its main idea is to initialize a Gaussian Mixture Model (GMM) and use the Estimation Maximization (EM) calculation to move the centroid of the mixture as a whole. This algorithm has three versions, which are rigid, affine, and non-rigid methods. The rigid method requires that the rotation matrix must be an orthogonal and positive definite matrix. The affine method does not require the rotation matrix to be an orthogonal and positive definite matrix, but it does not consider the scaling factor, so there would still be a small possibility of cloud deformation. The above mentioned ICP-based alignment methods, including the traditional ICP, Stochastic-ICP, and CPD-ICP, all must have a good initial position to converge to desired solutions. In addition, in 2015, Yang et al. [19] proposed the Go-ICP (Globally Optimal-ICP) algorithm, where rigid Euclidean alignment are performed for two sets of points in the L2 space under the amount of error as defined by the ICP. This algorithm incorporates a Branch and Bound (BnB) method that effectively searches in the 3D motion space SE(3), and, by using the special structure of the SE(3) geometry, the upper and lower limits of the alignment error function can be deduced. Because of this, the authors claimed that a good initial location is not required for the algorithm to get the best global solution. This claim is examined in the Experimental Results Section 3.
Sensors 2018, 18, 2505
3 of 15
Sensors 2018, 18, x FOR PEER REVIEW
3 of 15
2. Method Method 2. 2.1. Modeling Modeling from from Medical Medical Images Images 2.1. In our our experiment, dummy model model was was used used to to simulate simulate the the head head of of aa patient. First, aa CT In experiment, aa dummy patient. First, CT scan scan of the dummy model was performed and the resulting stack of images were processed using of the dummy model was performed and the resulting stack of images were processed using image image processing regions of of interest (ROI) andand a threshold waswas set on processing methods methods[20] [20]ininorder ordertotocapture capturethe the regions interest (ROI) a threshold set the on image stack to obtain the 3D modeling data, as shown in Figure 1. the image stack to obtain the 3D modeling data, as shown in Figure 1.
(a)
(b)
Figure vascular tissue. tissue. Figure 1. 1. (a) (a) A A dummy dummy head head model; model; and and (b) (b) model model of of intracranial intracranial vascular
2.2. Surface Data Alignment 2.2. Surface Data Alignment Currently, the most widely used surface alignment algorithm for three-dimensional objects is Currently, the most widely used surface alignment algorithm for three-dimensional objects is the the Point Cloud Library-Iteration Closest Point (PCL-ICP) algorithm [21]. This algorithm aligns two Point Cloud Library-Iteration Closest Point (PCL-ICP) algorithm [21]. This algorithm aligns two sets of sets of point cloud data and its processes are divided into four main steps: (1) sample from the original point cloud data and its processes are divided into four main steps: (1) sample from the original point point cloud data; (2) determine the initial correspondence point data; (3) remove the erroneous cloud data; (2) determine the initial correspondence point data; (3) remove the erroneous corresponding corresponding points; and (4) derive the coordinate transformation. First, it assumes two points; and (4) derive the coordinate transformation. First, it assumes two corresponding point sets corresponding point sets P and Q, or reference and floating, are determined, and the number of P and Q, or reference and floating, are determined, and the number of corresponding points is N. corresponding points is N. The optimal coordinate transformation between P and Q is iteratively The optimal coordinate transformation between P and Q is iteratively calculated by the least square calculated by the least square method to obtain a rotation matrix R and a displacement matrix T, method to obtain a rotation matrix R and a displacement matrix T, set in the same coordinate system. set in the same coordinate system. The main method aligns the floating point group to the reference The main method aligns the floating point group to the reference point group by means of iteration. point group by means of iteration. In this study, a RGB-D sensor [22] and Point Cloud Library (PCL) [23] are used to reconstruct 3D In this study, a RGB-D sensor [22] and Point Cloud Library (PCL) [23] are used to reconstruct information of the surface of real objects. The traditional ICP alignment algorithm is not considered 3D information of the surface of real objects. The traditional ICP alignment algorithm is not for our system. It is because there are two distinct disadvantages in the traditional ICP algorithm: The considered for our system. It is because there are two distinct disadvantages in the traditional ICP first disadvantage is that an arbitrarily different starting position may cause the algorithm to ends up algorithm: The first disadvantage is that an arbitrarily different starting position may cause the in a locally optimal solution instead of the globally optimal solution. The second disadvantage is that algorithm to ends up in a locally optimal solution instead of the globally optimal solution. The second grossly different points or noise in the non-overlapped area may affect the result of the alignment. Thus, disadvantage is that grossly different points or noise in the non-overlapped area may affect the result to solve these two problems, a perturbation mechanism is added in each iteration to facilitate escaping of the alignment. Thus, to solve these two problems, a perturbation mechanism is added in each from locally optimum solutions. For further improvements, the improved alignment algorithm, i.e., iteration to facilitate escaping from locally optimum solutions. For further improvements, the I-ICP, adds the concept of weights into the strategy: According to the distance between the two points improved alignment algorithm, i.e., I-ICP, adds the concept of weights into the strategy: According of the correspondence in each group, different weight values are assigned to each point to reduce the to the distance between the two points of the correspondence in each group, different weight values error caused by the matching wrong points. This weighting scheme is so designed because, when are assigned to each point to reduce the error caused by the matching wrong points. This weighting a point in the floating point group searches for its corresponding point in the reference point group, scheme is so designed because, when a point in the floating point group searches for its it may be possible that multiple floating points find correspondences in the same reference point. corresponding point in the reference point group, it may be possible that multiple floating points find Therefore, the weighting values are assigned to the floating point group to prevent the accumulation correspondences in the same reference point. Therefore, the weighting values are assigned to the of multiple alignment errors. The median value is used in order to avoid choosing the extreme values floating point group to prevent the accumulation of multiple alignment errors. The median value is thus avoiding misalignments. used in order to avoid choosing the extreme values thus avoiding misalignments. The procedure of the I-ICP alignment algorithm can be divided into four parts: data input, ICP The procedure of the I-ICP alignment algorithm can be divided into four parts: data input, ICP registration, the stop condition and the perturbation mechanism. The flowchart of the algorithm is registration, the stop condition and the perturbation mechanism. The flowchart of the algorithm is shown in Figure 2. shown in Figure 2.
Sensors 2018, 18, 2505
4 of 15
Sensors 2018, 18, x FOR PEER REVIEW
4 of 15
Figure 2. The flowchart of I-ICP alignment algorithm.
The I-ICP Procedure Procedure The I-ICP ••
Step 1: 1: Data Data Input Input Step
Assuming Assuming that that the the surface surface of of the the preoperative preoperative CT CT image image is is taken taken as as aa set set of of reference reference points points R, R, ( , , ), 1 and the facial feature points captured by the RGB-D senor and then = , then R = r j ( x, y, z), 1 ≤ j ≤ NR , and the facial feature points captured by the RGB-D senor and calculated with the the PCL PCL library library and and the the KAZE KAZE [24] [24] algorithm are to to be be taken calculated with algorithm are taken as as aa set set of of floating floating points points ( ), F, then = , , 1 . F, then F = { f ( x, y, z), 1 ≤ i ≤ N }. i
••
f
Step 2: 2: ICP ICP Alignment Alignment Step
For any given given point point ff in in the the set set floating floatingdata dataset setF,F,find finditsitsnearest nearestpoint pointr j in the in the reference For any reference datadata set set R, by calculating the minimum distance between f and each , as shown in Equation (1): R, by calculating the minimum distance between f and each r j , as shown in Equation (1): ( , )= ( , ) ∈( ,… d) f , r d( f , R) = min j j∈(1,...NR )
(1) (1)
where d is the minimum distance calculated by the corresponding points, and median is the middle numberd in eachminimum group after the corresponding are sorted from the smallest to theislargest as in where is the distance calculated bypoints the corresponding points, and median the middle Equation number in(2): each group after the corresponding points are sorted from the smallest to the largest as in Equation (2): NF is even (2) = d j = NF 2 median d j = if (2) d NF + 1 NF is odd j=
2
Then, according to the distance calculated by each corresponding point, different weight values are Then, according to the distance calculated by each corresponding point, different weight values given in Equation (3): are given in Equation (3): ( 1 1 i f d
