Bone segmentation using multiple communicating snakes Lucia Ballerinia and Leonardo Bocchib a Dept. of Technology, Orebro ¨ ¨ University, Orebro, Sweden b Dept. Electronics and Telecommunications, University of Florence, Firenze, Italy ABSTRACT Skeletal age assessment is a frequently performed procedure which requires high expertise and a considerable amount of time. Several methods are being developed to assist radiologists in this task by automating the various steps of the process. In this work we describe a method to perform the segmentation step, by means of a modified active contour approach. A set of separate active contours models each bone in a portion of the radiogram. Due to the complexity of the contour, and to the presence of multiple adjacent contours, we add to the commonly used energy terms a first-order derivative energy which allow to take into account the direction of the contour. Moreover, anatomical relationships among bones are modeled as additional internal elastic forces which couple together the contours. Contour energy is optimized using a genetic algorithm. Chromosomes are used to encode positions of snake points, using a polar representation. The genetic optimization overcomes the difficulties related to local minima and to the initialization criterium, and conveniently allows the addition of new energy terms. Experimental results show the method allows to achieve an accurate segmentation of the bone complexes in the region of interest. Keywords: Skeletal images, multiple active contours, snakes, genetic algorithms
1. INTRODUCTION Bone age assessment is a procedure frequently performed in pediatric radiology. A discrepancy between bone age and chronological age indicates the presence of some abnormality in skeletal growth. The assessment of bone age is almost universally performed by examination of the left-hand radiogram. This procedure requires only a minimal exposure, with an high degree of simplicity. Moreover, the hand presents a large number of ossification centers which can be analyzed in order to obtain an accurate evaluation of the skeletal age. In contrast with such advantages, inspection of the resulting radiogram is a quite complex task. A correct evaluation of the degree of maturation of the bones, requires an high degree of expertise. Several methods have been proposed to perform this evaluation. The most commonly used method is the atlas matching method by Greulich and Pyle.1 The hand radiogram is visually compared with a series of images reproduced in the atlas, grouped by age and sex. The pattern which appears to be the most similar to the clinical image is selected, and the corresponding age is indicated to assess the skeletal age. The major drawback in this method is its subjectivity, which produces an high degree of variability in the outcoming results, both inter-observer, and intra-observer. A more complex approach uses the Tanner and Whitehouse (TW2) method.2 This method involves a detailed analysis of a group of about 20 bones of hand and wrist. Each bone complex is assigned to one of eight classes reflecting the various development stages, depending on the degree of calcification and the shape of the complex. In this way, a maturation score is assigned to each bone. A weighted sum of all scores is the used to evaluate the skeletal age. This method yields the most reliable results, but, due to its complexity, it does not present a high application rate (less than 20%). Several research groups are working to develop automatic methods which can speed up the evaluation process. A complete method can roughly be subdivided in a segmentation step, where the bones are identified and labelled, Further author information: Lucia Ballerini: E-mail:
[email protected], Telephone: +46 (0)19 30 1359, Fax: +46 (0)19 30 3463, Address: Fakultets¨ gatan 1, 70182 Orebro, Sweden Leonardo Bocchi: E-mail:
[email protected], Telephone: +39 055 4796 443, fax: +39 055 494 569, Address: Via S.Marta 3, 50139 Firenze, Italy.
a feature and shape analysis step which assesses the bone age of each region, and a classification stage which summarizes the partial data to produce the final age assessment. In this work, we focus on the first stage, the segmentation procedure. The segmentation stage presents an high degree of complexity due to several factors; among those, we face with the presence of several overlapping regions of interest (ROI), the presence of ROI having completely different degree of calcification and overlapping of soft tissue. We propose the use of Genetic Snakes,3 that are active contour models, also known as snakes,4 with an energy minimization procedure based on Genetic Algorithms (GA).5 Snakes optimization through Genetic Algorithms proved to be particularly useful in order to overcome problems of the classical snakes related to initialization, parameter selection and local minima. New internal and external energy functionals have been proposed in our previous works and they have been successfully applied to a variety of images from different domains. The purpose of this paper is to extend the Genetic Snakes model to handle complex contours, composed of distinct regions, allowing introduction of external knowledge expressed by additional energy terms. Each bone contour is associated to an independent snake, while the anatomical knowledge about relative placement of hand bones is modelled by means of a binding energy which couples together the contours. The organization of the paper is as follow: in Section 2 we briefly review active contours, the basic notions, their limitations and some improvements proposed in literature. In Section 3 we describe the Genetic Snakes model. In Section 4 we extend our previous formulation by the introduction of the multiple snakes structure and of the binding energy. Experimental results are reported in Section 5.
2. ACTIVE CONTOURS Snakes are planar deformable contours that are useful in several image analysis tasks. They are often used to approximate the locations and shapes of object boundaries on the basis of the reasonable assumption that boundaries are piecewise continuous or smooth. Representing the position of a snake parametrically by v(s) = (x(s), y(s)) with s ∈ [0, 1], its energy can be written as: Z 1 Z 1 Esnake = Eint [v(s)] ds + Eext [v(s)] ds (1) 0
0
where Eint represents the internal energy of the snake due to bending and it is associated with a priori constraints, Eext is an external potential energy which depends on the image and accounts for a posteriori information. The final shape of the contour corresponds to the minimum of this energy. In the original technique of Kass et al.4 the internal energy is defined as: " 2 # ∂v(s) 2 ∂ v(s) 2 1 + β(s) α(s) Eint [v(s)] = ∂s2 . 2 ∂s
(2)
This energy is composed of a first order term controlled by α(s) and a second order term controlled by β(s). The two parameters α(s) and β(s) dictate the simulated physical characteristics of the contour: α(s) controls the tension of the contour while β(s) controls its rigidity. The external energy couples the snake to the image. It is defined as a scalar potential function whose local minima coincide with intensity extrema, edges, and other image features of interest. The external energy, which is commonly used to attract the snake towards edges, is defined as: Eext [v(s)] = −γ|∇Gσ ∗ I(x, y)|2
(3)
where I(x, y) is the image intensity, Gσ is a Gaussian of standard deviation σ, ∇ is the gradient operator and γ a weight associated with image energies. Due to the wide and successful application of deformable models, there exist survey papers focusing on different aspects of the model and its variants proposed in the literature.6–9 The application of snakes and other similar deformable contour models to segment structures is, however, not without limitations. For example, snakes were designed as interactive models. In non-interactive applications,
they must be initialized close to the structure of interest to guarantee good performance. The internal energy constraints of snakes can limit their geometric flexibility and prevent a snake from representing long tube-like shapes or shapes with significant protrusions or bifurcations. Furthermore, the topology of the structure of interest must be known in advance since classical deformable contour models are parametric and are incapable of topological transformations without additional machinery. Due to its own internal energy, the snake tends to shrink in case of lack of image forces, i.e. constant image backgrounds or disconnected object boundaries, and not to move towards the object. Various methods have been proposed to improve and further automate the deformable contour segmentation process. See the above mentioned surveys for a review of some of them. As concerns the energy minimization, the original model employs the variational calculus to iteratively minimize the energy. There may be a number of problems associated with this approach such as algorithm initialization, existence of local minima, and selection of model parameters. Simulated annealing,10, 11 dynamic programming12, 13 and greedy algorithm14, 15 have been also proposed for minimization. However they are restricted either by the exhaustive searches of the admissible solutions either by the required accurate initialization. Few authors propose the application of GA to active contours. Among those, MacEachern and Manku16 introduce the concept of active contour state and encode the variants of the state in the chromosome of the genetic algorithm. Tanatipanond and Covavisaruch17 apply GA to contour optimization with a multiscale approach. The fitness function is “trained” from a previously segmented contour. Ooi and Liatsis18 propose the use of coevolutionary genetic algorithms. They decompose the contour into subcontours and optimize each subcontour by separate GA working in parallel and co-operating. However, in these approaches the optimization is done in the neighborhood of the snake control points. In other existing GA-based active contours the optimization is done indirectly, i.e. optimizing the parameters of the contour such as encoding the polygon,19 Point Distribution Models,20, 21 Fourier Descriptors,22 Probability Density Functions23 or edge detector and elastic model parameters.24
3. GENETIC SNAKES In this section we review the genetic snake model, i.e. our model of active contours, where the energy minimization procedure is based on genetic algorithms.3 The parameters that undergo genetic optimization are the positions of the snake in the image plane vi = (xi , yi ), for i = 0, ...N where N is the total number of snake points. To simplify the implementation we used polar coordinates vi = (ri , θi ) with the origin in the center of the contour. Actually, this point must lie inside the object, but its position may be arbitrary. The magnitudes ri are codified in the chromosomes, while θi = 2πi/N . The polar representation introduces ordering of the contour points and prevents the snake elements from crossing each other during evolution. The genetic operators can be implemented straightforward on this representation, no additional check is required to ensure that mutation and crossover produce valid individuals. The fitness function is the total snake energy as previously defined in Eq. (1), where Eint and Eext are defined in Eqs. (2) and (3). The initial population is randomly chosen in a region of interest defined by the user, and each solution lies in this region. This replaces the original initialization with a region-based version, enabling a robust solution to be found by searching the region for a global solution. The region of interest can be the image itself, so the solution can be searched in the whole image, making the initialization fully automatic. An accurate description of implementation details along with a discussion on the choice of the model coefficients can be found in.3 The genetic search strategy works against constant image background and overcomes difficulties related to spurious edge-points that can drive the snake to a local minima. To reach an optimal minimum while avoiding local minima, some approaches suggest to consider the whole set of admissible curves and choose the best one. Other methods are for local optimization, where only sub-optimal solutions can be guaranteed. The genetic algorithms are particularly useful in simultaneously handling possible solutions and achieving a global minimum, while avoiding an exhaustive search.
v left
∆
v
v i+1
i
∆
v
right
vi−1 Figure 1. Points used in the evaluation of derivative energy
4. MULTIPLE GENETIC SNAKES A straightforward extension from a single contour to multiple contours poses a few questions which could prevent convergence. First of all, the spatial relations between facing bones produce sets of parallel edges. Each contour need to be univocally associated to the correct edge to achieve a correct segmentation, while Eq. (3) does not allow to discriminate between the edges, because it is sensitive to the gradient modulus, but not to its orientation. We solve this problem with the introduction of a first-order derivative energy, which introduces a directionality in the contours. For each point vi belonging to the snake (see Fig. 1), we define vleft and vright , which are placed on a line orthogonal to vi−1 vi , and spaced of a small distance ∆ from vi . To allow a faster computation, the orientation of the line has been constrained to be multiple of π/4, and the distance ∆ is assumed to be one pixel. The derivative energy used can then be expressed as: Eder [vi ] = δ [(F ∗ I)(xleft , yleft ) − (F ∗ I)(xright , yright )]
(4)
where δ is a weight used to balance the derivative energy with the other terms on external energy and F is a smoothing filter. This definition introduces an energy term that presents a minimum point when the snake is positioned on the image edge, having the brighter region on the left side of the snake, and the darker region on the right side. In our implementation, snakes are running counterclockwise around their center, so the left side corresponds to the internal region, and the right side to the external region. With a positive value of δ, we reach the minimum energy when the snake encloses a bright region on a darker background. As introduced before, we also add an additional term to the internal energy, which we call binding energy. This term models the anatomical relationships between adjacent bones, by introducing and elastic force that connects together appropriate points of adjacent snakes, as shown in Fig. 2. The energy associated to the elastic force is assumed to be represented by the relation: Ebind [u(s), v(t)] = µ|u(s) − v(t)|2
(5)
where µ represents the elastic constant of the spring. The application points of the elastic forces are selected accordingly to the physical relationships which exist between the anatomical regions.
5. APPLICATION The image data set is composed of radiographic images of the left hand and wrists, acquired by means of a conventional radiographic system, and digitized with a spatial resolution of 300dpi, and a pixel depth of 12 bits. Patient age ranges from 0 to 12 years. Afterward, the images have been downsampled by a factor of three to speed up the segmentation process (see Fig. 3). In this work we evaluate the application of genetic snakes to the segmentation of a subpart of the hand, the first finger. This allows to develop and test a simpler model, although the task includes most of the segmentation
v u ui
vj
Figure 2. Binding between adjacent snakes
Figure 3. Radiographic image of the hand
Figure 4. Gradient of Gaussian images (σ = 0.3, σ = 3, σ = 10).
Figure 5. Difference of Gaussian images (σ1 = 0.5; σ2 = 1.5; σ1 = 1, σ2 = 3; σ1 = 5, σ2 = 8)
problems. In particular, the three phalanx present a different contrast and mean gray level, due to their different thickness, have different size, and may have an incomplete boundary due to the presence of cartilaginous tissue. The situation has been modelled using three snakes, which represent the three bones, chained together by means of the binding force. Each snake is composed of 36 points, and the binding energy acts on five couples of consecutive points in each junction. The internal energy of the model is given by a weighted sum of Eint and Ebind , defined respectively in Eqs. (2) and (5). In our implementation it is possible to use different α and β for each snake. The image energy is computed as an appropriate combination of Eext and Eder ((Eqs. 3) and (4)). Equation (3) has been applied by incorporating three gradient of Gaussians with different σ (see Fig. 4). The filter F used in Eq. (4) is a Diffence of Gaussians, computed for three different couples of σ to obtain three smoothed versions of the image (see Fig. 5) The GA implementation adopted in this work is GAucsd-1.4.25 We used most of the default options proposed by the GAucsd package, i.e. Gray-code, fitness sigma scaling, two point crossover, roulette wheel selection. The parameters of the GA were: length of the genome = 324, population size = 100000, maximum number of generations = 1000, crossover rate = 0.59, mutation rate = 0.00001. We performed several experiments varying the snake energy weighting coefficients. As more internal and external energy terms are considered in our model than in the classical snake, it is hard to determine the appropriate ratio between different forces in the total energy function of the model, due to lack of understanding on the effect of each force on the energy function.
Figure 6. Segmentation results of the three phalanx
Moreover we can not compare experiments based on the fitness value, because it depends on the snake coefficients and a smaller fitness value does not imply a better solution if different weights are used; as extreme case, with all the weights set to zero, the fitness is zero, but the desired contour can not be achieved. For this reason statistics on fitness values are not reported here. The results showed in Fig. 6 have been obtained with following weights: α1 = 0.9, β1 = 6, α2 = 0.5, β2 = 5, α3 = 0.7, β3 = 10, µ = 1.25, δ = −1.2, γ1 = 2.5, γ2 = 1. γ1 = 0.5. For this setting of weights we did 25 GA runs. Fitness values of the best individual evolved in each run range from about 3.8 · 104 to about 1.4 · 104 .
6. CONCLUSIONS In this paper a method for automatic segmentation of hand radiograms is described along with some results obtained with the proposed approach on a subpart of the hand. The complexity of the skeletal structure in the hand and the variability between different subjects makes very difficult to realize an automatic segmentation of the bones. The proposed method allow to combine the a-priori knowledge on the hand structure to the adaptative behaviour of active contours and genetic algorithm. Binding and derivative energy allow to introduce adequate constraints on the geometry of the snake to obtain a satisfactory segmentation. It is known that the snake model requires either a local minimizer with good initialization or otherwise a global minimizer. Genetic snakes confront and overcome at the same time the two primary problems of initialization and optimization, and provide a global optimization with an automatic initialization. The encouraging results reported prompt us that the method could be extended and applied to other bone structures as well as to other images. Other extensions could consider the study of the parameters and the functionals governing the snake behaviour. A method to reduce user interaction by automatically assigning snake energy weights is still an open problem. Therefore the evolution of weights could be considered for future studies.
REFERENCES 1. W. W. Greulich and S. I. Pyle, Radiographic atlas of skeletal development of the hand and wrist, Stanford University Press, Palo Alto, CA, 2nd ed., 1959.
2. J. M. Tanner, R. H. Whitehouse, W. A. Marshall, and M. J. R. Healy, Assessment of skeletal maturity and prediction of adult height (TW2 method), Academic Press, London, 2nd ed., 1983. 3. L. Ballerini, “Genetic snakes for medical images segmentation,” in Evolutionary Image Analysis, Signal Processing and Telecommunications, Lectures Notes in Computer Science 1596, pp. 59–73, Springer, 1999. 4. M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contour models,” International Journal of Computer Vision 1(4), pp. 321–331, 1988. 5. D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley, Reading, MA, 1989. 6. T. McInerney and D. Terzopoulos, “Deformable models in medical image analysis: A survey,” Medical Image Analysis 1(2), pp. 91–108, 1996. 7. A. K. Jain, Y. Zhong, and M.-P. Dubuisson-Jolly, “Deformable template models: A review,” Signal Processing 71(2), pp. 109–129, 1998. 8. C. Xu, D. L. Pham, and J. L. Prince, “Image segmentation using deformable models,” in Handbook of Medical Imaging, M. Sonka and J. M. Fitzpatrick, eds., 2, ch. 3, pp. 129–174, SPIE Press, 2000. 9. K.-W. Cheung, D.-Y. Yeung, and R. T. Chin, “On deformable models for visual patter recognition,” Patter Recognition 35(7), pp. 1507–1526, 2002. 10. G. Storvik, “A bayesian approach to dynamic contours through stochastic sampling and simulated annealing,” IEEE Transactions on Pattern Analysis and Machine Intelligence 16, pp. 976 –986, October 1994. 11. R. P. Grzeszczuk and D. N. Levin, “Brownian strings: Segmenting images with stochastically deformable contours,” IEEE Transactions on Pattern Analysis and Machine Intelligence 19, pp. 110–1114, October 1997. 12. A. Amini, T. Weymouth, and R. Jain, “Using dynamic programming for solving variational problems in vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence 12(9), pp. 855–867, 1990. 13. D. Geiger, A. Gupta, L. Costa, and J. Vlontzos, “Dynamic programming for detecting, tracking and matching deformable contours,” IEEE Transactions on Pattern Analysis and Machine Intelligence 17, pp. 294–302, March 1995. 14. D. J. Williams and M. Shah, “A fast algorithms for active contours and curvature estimation,” CVGIP: Image Understanding 55, pp. 14–26, January 1992. 15. L. Ji and H. Yan, “Attractable snakes based on the greedy algorithm for contour extraction,” Pattern Recognition 33(4), pp. 791–806, 2002. 16. L. A. MacEachern and T. Manku, “Genetic algorithms for active contour optimization,” in Proc. IEEE International Symposium on Circuits and Systems, 4, pp. 229–232, 1998. 17. T. Tanatipanond and N. Covavisaruch, “An improvement of multiscale approach to deformable contour for brain MR images by genetic algorithms,” in Proc. IEEE International Symposium on Intelligent Signal Processing and Communication Systems, pp. 677–680, (Phucket, Thailand), December 1999. 18. C. Ooi and P. Liatsis, “Co-evolutionary-based active contour models in tracking of moving obstacles,” in Proc. International Conference on Advanced Driver Assistance Systems, pp. 58–62, 2001. 19. A. Toet and W. P. Hajema, “Genetic contour matching,” Pattern Recognition Letters 16, pp. 849–856, 1995. 20. T. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, “Active shape models - their training and application,” Computer Vision and Image Understanding 61, pp. 38–59, January 1995. 21. C. F. Ruff, S. W. Hughes, and D. J. Hawkes, “Volume estimation from sparse planar images using deformable models,” Image and Vision Computing 17(8), pp. 559–565, 1999. 22. P. E. Undrill, K. Delibasis, and G. G. Cameron, “An application of genetic algorithms to geometric modelguided interpretation of brain anatomy,” Pattern Recognition 30(2), pp. 217–227, 1997. 23. M. Mignotte, C. Collet, P. P`erez, and P. Bouthemy, “Hybrid genetic optimization and statistical modelbased approach for the classification of shadow shapes in sonar images,” IEEE Transactions on Pattern Analysis and Machine Intelligence 22, pp. 129–141, February 2000. 24. S. Cagnoni, A. B. Dobrzeniecki, R. Poli, and J. C. Yanch, “Genetic algorithm-based interactive segmentation of 3D medical images,” Image and Vision Computing 17(12), pp. 881–895, 1999. 25. N. N. Schraudolph and J. J. Grefenstette, “A user’s guide to GAucsd 1.4,” Tech. Rep. CS92-249, Computer Science and Engineering Department, University of California, San Diego, La Jolla, CA, 1992.
