A new automatic landmarks extraction framework on ultrasound images of the femoral condyles. Agn`es Masson-Sibutab , Amir Nakiba , Eric Petita and Fran¸cois Leitnerb a Laboratoire
Images, Signaux et Syst`emes Intelligents (EA3956), 122 rue Paul Armangot, 94400 Vitry-Sur-Seine, France; b Aesculap SAS,1 place du Verseau, 38130 Echirolles, France ABSTRACT
In Computer Assisted Orthopaedic Surgery (CAOS), the surgeon has to acquire some anatomical landmarks as inputs to the system. To do so, he uses manual pointers that are localized in the Operating Room (OR) space using an infrared camera. When the needed landmark is not reachable trough an opening, the palpation is percutaneous and less precise. In this paper, we propose a new framework to extract automatically anatomical landmarks with an ultrasound probe based on three main steps to register the bone surface. This framework is based on the calculation of an oriented gradient, a simulated-compound and a contour closure using a graph representation. The oriented gradient allows extracting the set of pixels that probably belong to the bone surface. The simulated-compound step allows using ultrasound images properties to define a set of small segments which may belong to the bone surface, and the graph representation allows eliminating the false positive among the segments. The proposed method has been validated on a database of 230 ultrasound images of the anterior femoral condyles (on the knee). The average computation time is 0.11 sec per image, and the average errors are: 0.54 mm for the bone surface extraction, 0.31 mm for the condylar line, and 1.4 mm for the middle of the trochlea. Keywords: Orthopedics, Ultrasound imaging, Landmarks extraction
1. INTRODUCTION Ultrasound (US) imaging became commonly used in some medical fields, for example to image structures like the prostate, the heart, or the liver, or to heal some cancer using High Intensity Focused Ultrasound (HIFU). Yet, it is hardly used for orthopedic interventions. Barkmann et al. proposed to use Quantitative Ultrasound (QUS) instead of X-rays to determine the risk of osteoporotic fractures.1 The device used was made of focused US transducers with low frequencies (500 kHz center frequency). Even though the results were less accurate than those with X-ray images, the wish of using more non-invasive imaging modalities strongly exists, decreasing the amount of radiation patients and surgeons receive. Also, some techniques have been developed in order to extract the bone position on ultrasound images: using manually digitized bone contour points,2 studying the local phase features on 3D ultrasound images,3 proposing to use dynamic programming,4 or using a two step method combining the identification of a Region of Interest (ROI) with a fully automatic segmentation of the bone contour.5 Though, the manual segmentation by the surgeon is highly time consuming, and can also lead to important error if the user is not correctly trained to read ultrasound images. Regarding to other methods, they cannot be used under real-time constraints as the computation time is at least 0.5 second per image. In order to allow the surgeon to do non-invasive pre-operative measures, such as laxity, on the knee, we propose to use ultrasound images. Then, the purpose consist in assisting the surgeon, not at ease with this imaging modality, to extract anatomical landmarks from these images. The proposed framework allows to extract the bone contour from ultrasound images, under real-time constraints, and then, to extract the landmarks targeted. The extraction of the contour is based on three main steps: the application of an oriented gradient, a simulated compound to get potential segments of bone contour, and the use of an oriented graph to differentiate true Further author information: (Send correspondence to Agn`es Masson-Sibut.) Agn`es Masson-Sibut: E-mail:
[email protected], Telephone: +33 (0)4 38 12 46 74 Amir Nakib: E-mail:
[email protected], Telephone: +33 (0)1 45 17 14 91
US probe
(a)
(b)
Figure 1: Illustration of the protocol for the acquisition of anatomical landmarks using the Orthopilot station. (a) The Orthopilot station. (b) The acquisition of an ultrasound image of the anterior femoral condyles.
positive and false positive among these segments, and in order to close the contour. Then, the landmarks are extracted based on a geometrical description of the contour. In this paper, we focus on the extraction of the condylar line, and the middle of trochlea, used to calculate the knee centre. The protocol for the ultrasound acquisition is described in Section 2.1 and the different steps of the framework are developed in the Sections 2.2 and 2.3. Then, some results are exposed and discussed in Section 3, before future work and conclusions in Section 4
2. METHOD The proposed framework consists in extracting the bone contour in ultrasound images under real-time constraints, and then, extracting the landmarks from the images. In Section 2.1 we develop the acquisition protocol and the materials we use for the study. Then, the framework is developed in Sections 2.2 and 2.3.
2.1 Materials and ultrasound acquisition The materials used for this study are: an Orthopilot station (Aesculap, Germany) (Figure 1a) with a Polaris infrared camera (Northern Digital Inc., Canada). The ultrasonic device is a linear probe (Telemed, Lituania) containing 128 crystals distributed on 80mm. The range of frequencies is between 3 MHz and 7 MHz. The patient has to be supine, and the surgeon locks the ultrasound probe under the patella, directed to the distal part of the femur (Figure 1b). The targeted landmarks are the middle of the trochlea which is a point, and the condylar line, which is represented by two points: the two condyles. To extract these targeted landmarks, the system assists the surgeon to find the image where the quantity of ultrasonic energy reflected by the bone surface is maximized, and regarding to ultrasonic properties, it occurs when the image plane is orthogonal to the bone surface. Consequently, the surgeon scans the condylar area and the bone contour is extracted in each frame. Then, on each extracted bone contour, the system calculates the sum of the intensities, representative of the quantity of energy in the image, and fixes the range of images where the correct image can be found. However, the choice of the image is supervised by the surgeon. Once the image is chosen, the landmarks are extracted and proposed to surgeon who has to validate the selection.
(a)
(b)
Figure 2: (a) Binary image after the thresholding. (b) Image after the simulation of the compound acquisition.
2.2 Bone contour segmentation The proposed framework allows to extract the bone contour in ultrasound images. The extraction method is based on three main steps: the application of an oriented gradient, a simulated compound to get potential segments of bone contour, and the use of an oriented graph to differentiate true positive and false positive among these segments, and in order to close the contour. The first step consists in using a gradient filter to highlight the high frequencies in the image. Before the application of the oriented gradient, a pillbox filter is used to smooth the noise in the ultrasound image (Figure 2a). Thus, interesting features (high gradients) are strengthened. The used gradient filter is a vertical filter. This choice is based on the assumption that the contour to be extracted is mainly horizontal. Jain and Taylor6 showed that the bone surface more likely lies on the top of the tick interface visible on ultrasound images. Thus, the gradient filter is defined to emphasize the upper interface of the bone surface in the ultrasound image. The resulting image IG is then thresholded using the cumulative histogram to only keep the highest values of the gradient representing horizontal features as the bone contour. The threshold value was fixed empirically so that only 5% of the highest values are considered. An illustration of the resultant binary image is presented Figure 2b. Then, to eliminate small features the binary image is processed by a simulated-compound. A compound-mode probe allows doing broad bandwidth acquisition that combines multiple coplanar images captured from different beam angles and from multiple ultrasound frequency spectra to form a single image in real time. The ultrasound image of a bone surface presents a shadow area under the bone surface because ultrasonic waves with a 7 MHz frequency cannot go through the bone structure. A bottom-up is applied to the binary image, with 3 different angles: -45◦ , 0◦ , and 45◦ (Figures 3a-3c). Then, points encountered with more than two angles are considered, and the resultant image is named IGC (Figure 3d) Using this image (IGC ), a set of segments is determined by contiguous points. The goal is to separate those that are part of the bone contour, and the false positive that are noise. Then, an oriented graph, denoted G, is built using the segments as nodes. For each segment Si , a weight w(si ) is calculated, taking into account the distance between Si and its closer neighbors (Si−1 and Si+1 ) and the sum of the intensities for the pixels on Si : w(Si ) =
i+1 X k=i−1
1 1 + kSi − Sk k
X
I(x, y)
(1)
(x,y)∈Si
where Si is the ith segment from the left side, and I represents the original image. The segments are then split in two classes, maximizing the inter-class variance and the strongest segments are defined to be fixed in the graph G so that they cannot be eliminated. The oriented graph G is presented Figure 4. The weight of an edge
(a)
(b)
(c)
(d)
Figure 3: Illustration of the simulated compound calculation. (a) Bottom-up calculated on IG with the angle -45◦ . (b) Bottom-up calculated on IG with the angle 0◦ . (c) Bottom-up calculated on IG with the angle 45◦ . (d) Image resulting of the combination of the three previous images, denoted IGC .
S1
S2
S3
S4
S5
S6
Sn
Figure 4: Illustration of the oriented graph G. The full circles represent the fixed nodes, and the dotted circles represent the nodes to be determined if they belong or not to the bone contour.
between two nodes is equal to the distance between the two nodes if they are linked, infinity elsewhere. The nodes strictly behind two fixed nodes are linked together, and if there is a fixed node between two nodes, they cannot be linked together. To eliminate the segments that do not belong to the bone surface, the Dijkstra’s algorithm7 is used to find the shortest path between the first node and the last node. These nodes are defined as the first and last nodes, from the left to the right, to be fixed. The result is shown Figure 5a. Afterwards, the closure of the contour is performed using a polynomial interpolation.8 The result is shown Figure 5b without the original image, and the overlap with the original image is showed Figure 6a.
(a)
(b)
Figure 5: Illustration of the result after the selection of the segments by the Dijkstra’s algorithm, and the polynomial interpolation. (a) The image after the elimination of non-bone segments using the Dijkstra’s algorithm. (b) The polynomial interpolation, defining the bone contour.
(a)
(b)
Figure 6: Illustration of the extraction of landmarks. (a) Original image with the bone contour extracted. (b) Original image, with the bone contour and the landmarks extracted. The line represents the condylar line, and the round is the middle of the trochlea.
2.3 Landmarks extraction Once the bone contour is extracted, the landmarks are extracted using a geometrical description of the contour. In this paper, the targeted landmarks are the two condyles representing the condylar line, and the center of the trochlea. The extraction is performed calculating the first derivative of the contour, and extracting the sign changes. The result is shown Figure 6b.
3. RESULTS AND DISCUSSION 3.1 Ultrasound images To test the validation of the method, a database with 230 ultrasonic images of anterior femoral condyles was built. The acquisition of these images was conducted according to the protocol described in Section 2.1. The population used for the study is made of 9 volunteers, men and women, from 25 to 51 years old, covering an as large as possible panel of Body Mass Index (BMI) values. The US device used for the study has a large range of parameters tunable. To build the database, the parameters were the same on every subjects: the focus of the beam was 34 mm deep, the frequency was 7 MHz, and the Speed Of Sound (SOS) was set to 1540 m/s, as an average value of the SOS in soft tissues, and bones.
3.2 Validation To evaluate the accuracy, the Root Mean Square Error (RMSE) and the Misclassification Error (ME)9 has been calculated between the result of the automatic extraction, and a manual extraction performed by an expert. These measures are defined as follows: n
RM SE(ys )2 M Ep (ys )
1X i (y − y¯s )2 n i=1 s |Bm ∩ Bc | + |Fm ∩ Fc | = 1− × 100 |Bm | + |Fm |
=
(2) (3)
where ys is the automatic segmentation of the bone contour for the image s and y¯s is the result of the manual segmentation, and the unity is millimeters for the RMSE measure. The ME measure corresponds to the ratio of background pixels wrongly assigned to the foreground and vice versa. The unity is percent, and Bm and Fm are, respectively, the background and the foreground of the manually segmented image, while Bc and Fc are, respectively, the background and the foreground of the automatically segmented image.
(a)
(b)
(c)
(d)
Figure 7: Comparison between the proposed method and a deformable contour based method. (a) Original image. (b) Result of the manual segmentation. (c) Result of the deformable contour based method. (d) Result of the proposed method.
For the extraction of the contour, the average RMSE value is 0.52 mm with a range between 0.08 mm and 3.09 mm and the average ME value is 0.18% with a range between 0.10% and 0.25%. The system was able to extract anatomical landmarks in 75% of cases, and the corresponding RMSE values are: for the extraction of the condylar line, the average RMSE is 0.30 mm, with a range between 0.00 mm and 4.86 mm. Regarding to the extraction of the trochlea, the average RMSE value is 1.49 mm, with a range between 0.15 mm and 4.97 mm. The average computation time is 0.11sec with a Matlab implementation. As an indication, these results can be compared to those with the same method but without using the Dijkstra’s algorithm to select the segments. Only the segments fixed by the weight classification are considered to be part of the bone contour. Then, the average RMSE is 1 mm with a range between 0.15 mm and 4.82 mm and the average of the ME is 0.19% with a range between 0.12% and 0.25%. Regarding to the extraction of the landmarks, it is processed in 69% of the cases and the average RMSE for the condylar line is 0.28 mm with a range between 0.00 mm and 2.72 mm and the average for the extraction of the trochlea is 3.38 mm with a range between 0.00 mm and 19.13 mm. The main advantage when using the Dijkstra’s algorithm to select segments part of the bone contour is that the maximal error is significantly lower than without using the Dijkstra’s algorithm. Also, a comparison with the results obtained using deformable model based methods10 was done. For the same database of 230 images, the average RMSE value was 0.83 mm in a range between 0.38 mm and 2.84 mm for the extraction of the bone contour. Regarding to the ME, the average value was 0.29% in a range between 0.26% and 0.33%. The difference is not significant on the entire database. Although, the differences for the ME values
show that the results obtained with the Dijkstra based method have a better correspondence with the manual segmentation than the deformable contour based method. To illustrate the difference, the two methods where used to calculate the bone contour on an ultrasonic image: Figure 7a shows the original image, Figure 7b shows the result of the manual segmentation, and Figures 7c and 7d show the comparison between the result obtained using the proposed method, and the result obtained by the deformable contour based method. On the contrary of the proposed method, the deformable contour based method extract the bone contour with difficulties when some parts are not clearly defined. In the presented image, the left condyle is not completely defined and the deformable contour based method pass through the bone contour.
4. CONCLUSION AND FUTURE WORK In this paper, we proposed a new framework to assist the orthopaedic surgeon during the acquisition of ultrasound images, and to extract automatically the bone contour, and anatomical landmarks in order to register the bone surface in the OR. It has been tested on 230 ultrasound images and the accuracy fits with the standards required in the OR. In future work, we plan to introduce machine learning in order to focus on a region of interest to calculate the contour from frame to frame. We should also do some tests in the OR context.
ACKNOWLEDGMENTS Funding for this research is provided by Aesculap SAS (Chaumont, FRANCE).
REFERENCES [1] Barkmann, R., Laugier, P., Moser, U., Dencks, S., Padilla, F., Haiat, G., Heller, M., and Gl¨ uer, C. C., “A method for the estimation of femoral bone mineral density from variables of ultrasound transmission through the human femur.,” Bone 40, 37–44 (Jan. 2007). [2] Chan, C., Barratt, D., Edwards, P., Penney, G., Slomczykowski, M., Carter, T., and Hawkes, D., “Cadaver validation of the use of ultrasound for 3D model instantiation of bony anatomy in image guided orthopaedic surgery,” in [Proceedings of the 7th International Conference on MICCAI], Lecture Notes in Computer Science 3217, 397–404, SPRINGER-VERLAG BERLIN (2004). [3] Hacihaliloglu, I., Abugharbieh, R., Hodgson, A., and Rohling, R., “Bone segmentation and fracture detection in ultrasound using 3D local phase features,” in [Proceedings of the 11th International Conference on MICCAI], 287–295 (2008). [4] Foroughi, P., Boctor, E., Swartz, M. J., Taylor, R. H., and Fichtinger, G., “Ultrasound Bone Segmentation Using Dynamic Programming,” in [2007 IEEE Ultrasonics Symposium Proceedings ], 2523–2526 (Oct. 2007). [5] Kowal, J., Amstutz, C., Langlotz, F., Talib, H., and Ballester, M. G., “Automated bone contour detection in ultrasound B-mode images for minimally invasive registration in computer-assisted surgery-an in vitro evaluation.,” The international journal of Medical Robotics and Computer Assisted Surgery : MRCAS 3, 341–8 (Dec. 2007). [6] Jain, A. K. and Taylor, R. H., “Understanding bone responses in B-mode ultrasound images and automatic bone surface extraction using a Bayesian probabilistic framework,” in [Proceedings of International Conference SPIE Medical Imaging], 5373, 131–142, Spie, Bellingham, WA, USA (2004). [7] Dijkstra, E. W., “A note on two problems in connexion with graphs,” Numerische mathematik 1(1), 269–271 (1959). [8] Clenshaw, C. W. and Hayes, J. G., “Curve and Surface Fitting,” IMA Journal of Applied Mathematics 1(2), 164–183 (1965). [9] Sezgin, M. and Sankur, B., “Survey over image thresholding techniques and quantitative performance evaluation,” Journal of Electronic Imaging 13(1), 146–165 (2004). [10] Kass, M., Witkin, A., and Terzopoulos, D., “Snakes: Active contour models,” International Journal of Computer Vision 1, 321–331 (Jan. 1988).
