3D Statistical Shape Models to Embed Spatial Relationship Information Jurgen Fripp1,2 , Pierrick Bourgeat1 , Andrea J.U. Mewes3 , Simon K. Warfield3 , Stuart Crozier2 and S´ebastien Ourselin1 1
BioMedIALab CSIRO ICT Centre, Australia. {jurgen.fripp, pierrick.bourgeat, sebastien.ourselin}@csiro.au 2 School of ITEE, University of Queensland, Australia.
[email protected] 3 Computational Radiology Laboratory Harvard Medical School, Departments of Radiology, Brigham and Women’s Hospital and Children’s Hospital, 75 Francis St, Boston MA 02115 {mewes, warfield}@bwh.harvard.edu
Abstract. This paper presents the creation of 3D statistical shape models of the knee bones and their use to embed information into a segmentation system for MRIs of the knee. We propose utilising the strong spatial relationship between the cartilages and the bones in the knee by embedding this information into the created models. This information can then be used to automate the initialisation of segmentation algorithms for the cartilages. The approach used to automatically generate the 3D statistical shape models of the bones is based on the point distribution model optimisation framework of Davies. Our implementation of this scheme uses a parameterized surface extraction algorithm, which is used as the basis for the optimisation scheme that automatically creates the 3D statistical shape models. The current approach is illustrated by generating 3D statistical shape models of the patella, tibia and femoral bones from a segmented database of the knee. The use of these models to embed spatial relationship information to aid in the automation of segmentation algorithms for the cartilages is then illustrated.
1
Introduction
Osteoarthritis (OA) of the knee is usually characterized by the degeneration of the articular cartilage. However, the progression of OA is much more complicated, usually consisting of changes in all the cartilages, bones and other tissues in the knee. Nevertheless, the loss of the few millimeters of thickness in the articular cartilage is still regarded as the most important feature to monitor OA progression. As OA develops, the changes undergone by the articular cartilage are not simply degenerative, rather it usually consists of localised thinning or thickening that develops slowly over time. The ability to accurately measure this
II
degeneration is essential to the development of pharmaceuticals and therapies for OA. Several MR sequences now exist that provide a non-invasive way to obtain high contrast accurate images of the knee cartilage morphology [1]. The potential of MRI to more accurately diagnose and monitor OA compared to other imaging modalities has now been shown in numerous studies. These studies usually monitored the progression of OA using measurements like thickness [2][3], volume [4] and surface area [5]. To obtain these measures from 3D MR images requires the articular cartilage to be segmented, preferably in an accurate and repeatable way. Unfortunately due to the low contrast in several areas particularly in the joint contact areas, tendons and ligaments, fully automated segmentation of the cartilage has not been achieved. Current clinical studies have favored the use of supervised semiautomated 2D algorithms like region growing [6], active shape models [7], active contours [8] and live wire [9]. These algorithms all require various degrees of user interaction, verification and correction of the segmentations results on a slice by slice basis. These approaches have significantly reduced the time taken to segment a dataset and provide provide higher intra- and inter-observer consistency than fully manual methods. However, for routine clinical use they are still too time consuming, so a fully automated approach is desirable. To fully automate the segmentation of the cartilages requires the segmentation to be performed directly in 3D. There have only been a few attempts to do this, including a model based approach [10], immersion based watershed [11] and statistical classification [12]. The most promising work uses a modified watershed metric that utilises prior information to perform the segmentation of the cartilages [13]. This algorithm can be combined with an atlas registration to fully automate the segmentations, however this has yet to be performed with the cartilages. Segmentation System for the Knee The philosophy behind our segmentation system for the knee is similar to the thoughts of both Kapur [10] and Hamarneh [14] and is focused on the effective but flexible incorporation of a priori knowledge, which is utilised intelligently to obtain automated, accurate and robust segmentations of the knee. Different types of knowledge needs to be used in order to attain the required automation, accuracy and the robustness. Shape information is the most important piece of knowledge for the knee as it allows us to compensate for the missing or poor delineation seen in the cartilages. This primarily is used to improve the robustness of the segmentation algorithm, which is essential if full automation is desired. In 2D, active shape models (ASMs) have been shown to produce accurate and robust results for the articular cartilage [7]. The primary problem with ASM type approaches is that they are sensitive to initialisation, and for many providing a good automatic initialisation is difficult. Spatial relationship information is one of the primary
III
piece of knowledge that can be used to aid in automating the initialisation of segmentation algorithms. The core of the segmentation system will be a statistical map of the knee based around 3D statistical shape models (SSMs). Besides the use of the SSM to incorporate shape information we propose to use it as a framework to include a priori knowledge by utilising the corresponding landmarks to extract information from the set of training images. This type of approach is used in grey level modeling of ASMs; however, our segmentation system incorporates various other types of information including texture and spatial relationship information. This information is used either by embedding it inside the SSM or modeling the associated information separately. The incorporation of image derived information will aid the accuracy and robustness of the segmentations, while spatial relationship information will aid in automating and self correcting the segmentation system. The first part of this paper presents the automatic creation of 3D SSMs of the patella, tibia and femoral bones using an implementation based on the Point Distribution Model optimisation scheme of Davies [15]. The second part of the paper presents how these models are then extended to include associated or embedded spatial relationship information. The strong spatial relationship between the bones and the cartilages is used to illustrate how this information could be used to automatically initialise segmentation algorithms of the cartilages.
2
Methodology
Subjects and Imaging This work used a knee database provided by Brigham and Women’s Hospital that consisted of 15 normal adults scanned using 1.5 and 3 T G.E. MR scanners with a fat suppressed 3D SPGR MR sequence. The sequence parameters were te = 5 or 7 msec, tr = 60 msec and a flip angle of 40o . The FOV was 120×120 and the acquisition matrix was either 512×512 or 256×256. These were reconstructed to images with dimension of 0.23×0.23 or 0.46×0.46 and slice thickness of 1.5mm. The bones and cartilages in the images were then interactively segmented by experts. To retain the variability of the shaft lengths in the SSM, the femoral and tibia shafts were not concatenated to be proportional to the head widths. This database is sufficient to evaluate the feasibility and effectiveness of 3D SSM for the knee segmentation system. However, the final knee segmentation system will either require the use of a much larger database or the use of a more sensitive modeling technique than principal component analysis (ie wavelets). Statistical Shape Models for the Bones SSMs provide a compact representation of the shape variability from a training set [16]. A SSM is built from a set of N training shapes si (i = 1, . . . , N ). Each shape si has M points sampled on its surface. These shapes are aligned using
IV
generalised Procrustes alignment. Principal Component Analysis (PCA) allows each shape to be written as X si = s˜ + P bi = s˜ + pk bki (1) k
where s˜ is the mean shape and P = pk contains the k eigenvectors of the covariance matrix. The corresponding eigenvalues (λk ) describe the amount of variation expressed by each eigenvector. The shape parameters b = bk are used to control the modes of variation. To obtain a valid SSM it is necessary that the coordinates are in a common frame of reference and all the points on each surface correspond in an anatomically meaningful way. Several approaches can be used to obtain point correspondence including non-rigid registration [17], deformable models [18] and Point Distribution Model optimisation [15]. The primary problem with using non-rigid registration is that the correspondence obtained is not unique and dependent on the accuracy of the registration, which for objects like cartilage can be highly inaccurate. Deformable models do not generate any true correspondence and Davies Point Distribution Model framework is restricted to genus 0 surfaces and requires already segmented training sets. For the knee segmentation system we have implemented an automated scheme to generate 3D SSMs based on the Point Distribution Model optimisation framework of Davies et al [15]. The primary reason for this is the flexibility the optimisation scheme allows, where the target results obtained can be altered simply by changing the objective function or the type of model we are optimising4 . This flexibility as well as the known improvement in the correspondence obtained compared to other techniques are the reason we chose this approach.
3
Implementation
The implementation we used for the Point Distribution Model optimisation framework of Davies can be broken down into 3 stages: pre-processing, generation of initial landmarking and the optimisation of the landmarking. Pre-processing and Initial Landmarking The 15 manually segmented knee datasets are pre-processed as outlined in Figure 1. In the literature the surfaces are usually extracted before being decimated (or remeshed) and then smoothed before being parameterized onto a unit sphere. In this work we have used a subdivision based parameterized surface extraction algorithm to guarantee we obtain genus 0 surfaces and parameterization. The 4
eg. inclusion of curvature, thickness and other derived measures into the model or objective function
V
full algorithm is similar to shrink wrap algorithms [19] however it is still under development and beyond the scope of this paper. The bone surfaces and parameterization that is used consisted of 4098 vertices for the patella and 16,386 vertices for the tibia and femoral bones. These surfaces are then rigidly ICP aligned, centroid matching and RMS normalisation. The resulting surfaces are then used to align the set of parameterizations.
Fig. 1. Overview of Pre-processing
Initial Landmarking The initial SSM is created by quasi-uniformly sampling5 the parameter space of each ICP aligned surface as outlined and illustrated in Figure 2. The set of re-parameterized surfaces are then used to generate a 3D SSM (see Section 2). To inverse map each vertex in parameter space to the surface, an efficient intersection algorithm using a partitioned parameter space and barycentric coordinates has been utilised.
Fig. 2. A surface is re-parameterized by sampling its parameter space and inverse mapping the sampled landmarks onto the surface
5
The patella consisted of 1026 vertices and the tibia and femur 4,098 vertices.
VI
Optimisation of Landmarking Although the parameter spaces are rotationally aligned there is no true correspondence from the parameterization; the correspondence is obtained by using an implementation of the Point Distribution Model optimisation scheme of Davies [15] (See Figure 3). The Nelder Mead P simplex algorithm was used to optimise an eigenspace objective function, F = i log(λi + ²) with ² = 0.01.
Fig. 3. 3 Patella bones pairs where the left pair is held fixed during the optimisation process. Top Initial landmarking Bottom Optimised landmarking Note: Colouring of surface based on corresponding triangles
Associating or Embedding Information into 3D SSM The creation of corresponding point distribution models allows the extraction of accurate information from the training datasets. The simplest approach is to associate with each point in the model the set of extracted or derived information. An example of this is the expected thickness of the cartilage above a point on the tibia, or the variance of the thickness across the training set (see Figure 4 and 5). Ideally, rather than having a static representation of the information at each point in the model, it is often preferable that the information varies based on the parameters of the model. One approach to this is to embed the information extracted from the training sets directly into the model by simply extending the dimensionality from R3 to R3+m , where m is the number of different types of information to be included in the model. This type of approach has been previously used to incorporate thickness variability into the optimisation of a model for the Levator Ani [20]. However, spatial relationship information between the cartilages and bone is disassociated from the whole shape, because the thickness
VII
is only defined for a subset of the model. As a result, by directly incorporating the information into the model, the expected thickness values obtained as the model is varied extend beyond the range that is expected anatomically (see Figure 6). As we are still examining how best to combine several types of disassociated information into one model, the current models are optimised in the standard way and from the corresponding points the information of interest is extracted. The information is associated with each point and kept constant, rather than being modeled in a way that would allow its value to vary based on the parameters of the shape model. Several examples of the type of spatial relation information that we are interested in are illustrated in Figure 4, 5 and 8.
4
Results and Discussion
The initial landmarking obtained for the patella, tibia and femur provide a satisfactory initial approximation for correspondence landmarks. The optimisation scheme improves this correspondence significantly as can be seen in Figure 3 and by the improvement in compactness shown in Table 1. The primary mode of variation of the optimised models can be seen in Figure 7. Table 1. Compactness and the first four primary modes of variation of the bone models: Optimised (Initial) Mode 0 1 2 3 Compactness
Patella 5.77 (12.98) 1.92 (2.62) 1.71 (2.16) 1.27 (1.72) 14.78 (25.00)
Tibia 13.32 (14.01) 9.34 (9.44) 2.31 (2.93) 1.68 (2.38) 33.11 (36.51)
Femur 13.37 (22.17) 10.60 (12.53) 4.06 (4.87) 2.20 (2.80) 37.03 (51.86)
Associating or embedding information into the SSM will aid the automation of the initialisation of the cartilage models. After segmenting the bones using these models, the information embedded in these models (like cartilage thickness)
Fig. 4. The mean thickness of the cartilage above each landmark in the SSM
VIII
Fig. 5. The standard deviation of the thickness of the cartilage above each landmark in the SSM
Fig. 6. The primary√mode of the combined tibia shape and thickness (of the meniscus) model varied by ± 3 standard deviations Note: The cartilage thickness extending beyond 5mm
√ Fig. 7. The primary mode of the bone models varied by ± 3 standard deviations
Fig. 8. The probability that cartilage is above each landmark in the SSM
IX
will be used to determine the placement and parameters used to initialise them. The type of information includes the probability that the cartilage exists above a point, its expected thickness and the variation of the thickness (see Figures 4, 5 and 8).
5
Conclusion
The optimisation process presented has allowed the automatic creation of 3D SSMs of the knee bones. We propose the use of these models as a basis to associate or embed a priori knowledge like spatial relationship information. The initial use of this information is to aid in the automatic initialisation of 3D segmentation models of the cartilages using the probability and expected thickness. This information can also be utilised by each model to improve their segmentation process and allow for the development of cooperative or competitive segmentations of difficult regions like the meniscus/articular cartilage boundary. The incorporation of information such as thickness into the models will become even more important with models created from a database of osteoarthritis sufferers. This approach extends naturally to the embedding of other a priori knowledge and is an ideal way to aid in the development of 3D intelligent deformable models [14] for use in an automated segmentation system.
Acknowledgements The authors wish to thank Johannes Pauser for help in acquiring and interactively segmenting the scans. This investigation was supported in part by a research grant from the National Multiple Sclerosis Society and by NIH grants R21 MH67054, R01 LM007861, P41 RR13218 and P01 CA67165 and NSF grant 0426558.
References 1. Hargreaves, B.A., Gold, G.E., Beaulieu, C.F., Vasanawala, S.S., Nishimura, D.G., Pauly, J.M.: Comparison of new sequences for high resolution cartilage imaging. Magnetic Resonance in Medicine 49 (2003) 700–709 2. Cohen, Z.A., McCarthy, D.M., Kwak, S.D., Legrand, P., Fogarasi, F., Ciaccio, E.J., Ateshian, G.A.: Knee cartilage topology, thickness and contact areas from MRI: in-vitro calibration and in-vivo measurements. Osteoarthritis Cartilage 7 (1999) 95–109 3. Williams, T.G., Taylor, C.J., Gao, Z., Waterton, J.C.: Corresponding articular cartilage thickness measurements in the knee joint by modelling the underlying bone. In: MICCAI’03. Volume 2879. (2003) 480–487 4. Wluka, A.E., Stuckey, A., Snaddon, J., Cicuttini, F.: The determinants of change in tibial cartilage volume in osteoarthritic knees. Arthritis and Rheumatism 46 (2002) 2065–2072
X 5. Hohe, J., Ateshian, G., Reiser, M., Englmeier, K.H., Eckson, F.: Surface size, curvature analysis and assessment of knee joint incongruity with MRI in vivo. Magnetic Resonance in Medicine 47 (2002) 554–561 6. Eckstein, F., Schnier, M., Haubner, M., Priebsch, J., Glaser, C., Englmeier, K.H., Reiser, M.: Accuracy of cartilage volume and thickness measurements with magnetic resonance imaging. Clinical Orthopaedics and related research 352 (1998) 137–148 7. Solloway, S., Hutchinson, C., Waterton, J., Taylor, C.: The use of active shape models for making thickness measurements of articular cartilage from MR images. Magnetic Resonance in Medicine 37 (1997) 943–952 8. Lynch, J., Zaim, S., Zhao, J., Stork, A., Peterfly, C.G., Genant, H.: Cartilage segmentation of 3D MRI scans of the osteoarthritic knee combining user knowledge and active contours. SPIE 3979 (2000) 925–935 9. Gougoutas, A., Wheaton, A., Borthakur, A., Shapiro, E., Kneeland, J., Udupa, J., Reddy, R.: Cartilage volume quantification via live wire segmentation. Academic Radiology 11 (2004) 1389–1395 10. Kapur, T., Beardsley, P., Gibson, S., Grimson, W., Wells, W.: Model-based segmentation of clinical knee MRI. In Proc. IEEE Int’l Workshop on Model-Based 3D Image Analysis (1998) 97–106 11. Ghosh, S., Beuf, O., Ries, M., Lane, N., Steinbach, L.S., Majumdar, S.: Watershed segmentation of high resolution magnetic resonance images of articular cartilage of the knee. Engineering in Medicine and Biology Society 4 (2000) 3174–3176 12. Warfield, S.K., Kaus, M., Jolesz, F.A., Kikinis, R.: Adaptive, template moderated, spatially varying statistical classification. Medical Image Analysis 4 (2000) 43–55 13. Grau, V., Mewes, A., Alcaniz, M., Kikinis, R., Warfield, S.: Improved watershed transform for medical image segmentation using prior information. IEEE Transactions on Medical Imaging 23 (2004) 447–458 14. Hamarneh, G., McInerney, T., Terpzopoulos, D.: Intelligent deformable organisms: An artificial life approach to medical image analysis. Technical report CSRG-432 (2001) 15. Davies, R., Twining, C., Cootes, T., Waterton, J., Taylor, C.: 3D statistical shape models using direct optimisation of description length. In: 7th European Conference on Computer Vision. Volume 3. (2002) 3–21 16. Cootes, T., Taylor, C., Cooper, D., Graham, J.: Active shape models - their training and application. Computer Vision and Image Undertanding 61 (1995) 38–59 17. Rueckert, D., Frangi, A.F., Schnabel, J.A.: Automatic construction of 3d statistical shape models using non-rigid registration. IEEE Transactions on Medical Imaging 22 (2003) 1014–1025 18. Siers, M., Frangi, A., Kaus, M., Niessen, W.: Comparison of two 3D automatic landmarking methods for a large training set of cardiac MR images. (citeseer) citeseer.ist.psu.edu/529161.html. 19. Kobbeltm, L., Vorsatz, J., Labsik, U., Seidel, H.: A Shrink Wrapping Approach to Remeshing Polygonal Surfaces. In: Proceedings of the 20th Annual Conference ot the European Association of Computer Graphics (Eurographics-99). (1999) 20. Lee, S.L., Horkaew, P., Darzi, A., Yang, G.Z.: Statistical Shape Modelling of the Levator Ani with Thickness Variation. In: MICCAI’04. Volume 3216. (2004) 258– 265
