INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN BIOMEDICAL ENGINEERING Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 Published online 30 August 2010 in Wiley Online Library (wileyonlinelibrary.com). DOI: 10.1002/cnm.1410
Adaptive B-Snake model using shape and appearance information for object segmentation Yue Wang1, ∗, † , ZuJun Hou1 , XuLei Yang2 and KartLeong Lim1 1 Institute
for Infocomm Research, A*Star (Agency for Science, Technology and Research), Singapore, Singapore of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Singapore
2 School
SUMMARY A novel adaptive B-Spline deformable model is presented in this paper for object segmentation. Comparing with other B-Spline models, the proposed model has the following advantages. First, a fully automatic and affine-invariant strategy is proposed for landmark point assignment. Second, contrary to other B-Spline models that rely on predetermined number of control points, an automatic scheme for control point insertion is designed to enhance the adaptivity and the flexibility of B-Spline model for segmenting shapes with high complexity. Thirdly, a statistical framework is embedded for modeling the shape distribution and appearance characteristics of landmark points in the training samples. Fine deformation can be achieved through the minimum mean square error approach that allows the model to accurately adapt to the desired object boundaries in the image. Experiments on medical image segmentation are carried out to validate the performance, and comparison has been made with respect to the traditional Snake and ASM. It turns out that the proposed adaptive B-Spline model can attain more accurate object segmentation. Copyright 䉷 2010 John Wiley & Sons, Ltd. Received 5 February 2009; Revised 6 May 2010; Accepted 23 June 2010 KEY WORDS:
deformable model; active contour; active shape model; B-Spline; B-Snake; object segmentation; shape; appearance; principal component analysis
1. INTRODUCTION Over the past decade, there has been an increasing research focus on deriving a mathematical description of object boundaries from images (also known as object segmentation or boundary extraction), which is a fundamental step for many active research areas in image analysis, computer vision, and medical imaging. Boundary extraction aims to augment our understanding on various object properties of interest in images, and can be utilized for applications such as motion tracking, shape matching and identification, object recognition, image registration and warping. Object segmentation from images remains a difficult problem due to the variability of object shapes and diverse image sources. A number of techniques have been proposed to address this issue. Among them, the Active Contour Model, or Snake, as pioneered by Kass et al. [1] has been one of the most popular methods for object segmentation. The model defines a curve within the image domain that is subjected to the influence of internal forces from the curve itself and external forces from the image data. Once internal and external forces have been determined, the curve can move within the image like a Snake and detect object boundaries automatically. Since then, many approaches
∗ Correspondence
to: Yue Wang, Institute for Infocomm Research, A*Star (Agency for Science, Technology and Research), Singapore, Singapore. † E-mail:
[email protected] Copyright 䉷 2010 John Wiley & Sons, Ltd.
634
Y. WANG ET AL.
have been presented for curve modeling such as Fourier descriptors [2], B-Splines [3–7], autoregressive model [8], moments [9], HMM models [10] and wavelets [11]. Recent research works on active contour in capturing complex geometries and dealing with difficult initializations [12, 13], improving the speed [7, 14, 15], affine-invariant shape descriptor [16, 17] and global minimization approach [18], etc. In this study, the B-Spline model is of particular interest. B-Spline stands as one of the most efficient curve representation and possesses some very attractive properties such as compactness, continuity and local shape controllability. It has been extensively used in computer-aided design, computer graphics and computer vision. For deformable models based on B-Splines, the number of control points is usually fixed [4, 19–21] or set to a ratio of the curve length [3, 22], which may not be convenient for realistic applications because it is non-trivial to determine the criteria for various object shapes. Stammberger et al. [23] proposed a B-Spline Snake algorithm for the segmentation of the knee joint cartilage from MR images through a multi-resolution approach. Mário presented an approach to unsupervised contour representations and estimations using B-Spline [24]. Wang et al. [25] presented a B-Snake-based lane model for lane detection. Although the results are good in lane detection, the number of control points is fixed to three, which limits the capability to describe complex shapes. In their later papers [5, 21, 26], a structure-adaptive B-Snake model was proposed for segmenting the complex structures in real images. When a priori knowledge on the object of interest is available, it is possible to integrate into the Snake model. Cootes et al. presented a point distribution model (PDM) [27] for building flexible shape models, where the shape is represented by a set of landmark points. The shapes are aligned and the deviations from the mean are analyzed using principal component analysis (PCA). As an extension of the PDM, Baumberg et al. proposed a cubic B-Spline model [20] for detecting and tracking the walking pedestrians. A possible issue with the PDM lies in the fact that the B-Spline curve is not uniquely described by a single set of control points, which means that different sets of control points may describe the same curve. The active shape models (ASM) introduced by Cootes et al. [28] is a statistical technique for building deformable shape templates and segmenting various organs from 2D medical images. In this model, the shape models represent objects using sets of landmark points with fixed number which are manually placed on the object boundary in each image. For different shapes in a training set, after compensating for the differences of the object posed by affine transformation, the PDM is constructed to model the landmark points’ distribution of these shapes. The ASM segmentation method involves finding the new landmark points that are similar to the shapes in the training set constructed from the PDM. The method has been successfully applied to several segmentation problems in medical images [29–34]. Nevertheless, the limitations of ASM can be observed in the following aspects. First, the landmark points in ASM are usually chosen manually for describing the whole shape. Second, given that the method models how different landmark points move together as the shape varies, if any labeling error occurs (e.g. a particular point placed at different sites on each training shape) the method would fail to capture the shape variability. Thirdly, a mechanism is not present to smoothen the formed curve based on these individual landmark points. Fourth, appearance information of region around landmark point has not been used. As shown in the active appearance model (AAM) [35], the appearance information can drive the shape model toward target boundary more efficiently. As aforementioned, the landmark points in ASM are manually chosen so that they will include corners of object boundaries, ‘T’ junctions and equally spaced intermediate points between them. However, these points are not preserved under affine transformation. Therefore, the parameters of the affine transformation calculated from these points are not precise. An algorithm is proposed in [36] for landmark point assignment that minimizes the description length of the training set. Here, the correspondence problem is converted to find the parameterization for each shape in the training set that builds the ‘best’ model. This algorithm requires a large number of function evaluations that could take several hours. Therefore, it becomes impractical when a larger set of training shapes is used. Hill and Taylor [37, 38] have proposed an algorithm for automatic generation of landmark points, which requires that the contour of the object is segmented prior to the landmark generation. However, the landmark points remain non-affine invariant. Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
635
ADAPTIVE B-SNAKE MODEL
In this paper, we will address the problems associated with the existing B-Snake models and present a B-Snake model that utilizes both the shape and the appearance information for object segmentation. The problem of landmark points and the problem of curve representation are dealt with separately. The former only represents the key points of shape, whereas the B-Spline is utilized to describe the whole shape. It is combined with the robustness of B-Spline, ASM and AAM for object segmentation and can be regarded as an extension to our previous research work [5, 21]. The new model will include: (1) a mechanism for control point insertion to handle shapes with high complexity, (2) a fully automatic landmark point assignment to extract the feature of shapes and appearance of training set, and (3) a minimum mean square error (MMSE) method to capture object boundaries more precisely. Some of the results have been highlighted in [39]. The paper is organized as follows. For completeness, a brief account on the B-Snake model, MMSE and control point insertion strategy are introduced in Section 2. Then, Section 3 details the method for new landmark point assignment using affine invariant features. In Section 4, the statistic model is described to guide the B-Snake deformation. The whole algorithm for object segmentation is given in Section 5. After that, simulation results are presented in Section 6. Finally, this paper is concluded in Section 7.
2. B-SPLINE SNAKE MODEL B-Spline Snake employs B-Spline for curve description and it can be opened or closed with any order. In this paper, we will investigate the close cubic B-Spline in B-Snake modeling. 2.1. Close cubic B-Splines A close cubic B-Spline has n +1 control points {Q i = [xi y]T , i = 0, 1, . . . , n}, and n +1 connected curve segments {gi (s) = (xi (s), yi (s)), i = 0, 1, 2, . . . , n}. Each curve segment is a linear combination of four cubic polynomials by the parameter s, where s is normalized between 0 and 1 (0s1). That is, ⎡
⎤
Q (i)
⎥ ⎢ ⎢ Q (i+1) mod (n+1) ⎥ ⎥, gi (s) = M R (s) ⎢ ⎥ ⎢ ⎣ Q (i+2) mod (n+1) ⎦
i = 0, 1, 2, . . . , n
(1)
Q (i+3) mod (n+1) where ⎡
− 16
⎢ 1 ⎢ ⎢ 2 3 2 M R (s) = [s s s 1] ⎢ ⎢− 1 ⎢ 2 ⎣ 1 6
1 2
− 12
−1
1 2 1 2 1 6
0 2 3
1⎤ 6
⎥ 0⎥ ⎥ ⎥. 0⎥ ⎥ ⎦ 0
The points connecting the neighboring segments are called knot points Hi (i = 0, 1, 2, . . . , n), where the B-Spline bases are tied together. Given the set of knot points H = (H0 , H1 , . . . , Hn ) of a uniform cubic B-Spline curve, we can uniquely determine control points Q = (Q 0 , Q 1 , . . . , Q n ), by substituting s = 0 into Equation (1). The relationship between the control points and the knot points is given as: Q = A−1 H, Copyright 䉷 2010 John Wiley & Sons, Ltd.
(2)
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
636
Y. WANG ET AL.
where A is an (n +1)×(n +1) matrix: ⎡1
2 3 1 6
1 6 2 3
0
...
...
1 6
0
...
.
.
.
.
.
.
.
.
.
.
...
0
1 6
...
...
0
2 3 1 6
1 6 2 3
6
⎢ ⎢0 ⎢ ⎢ ⎢. ⎢ A=⎢ ⎢. ⎢ ⎢ ⎢0 ⎣ 0
0
⎤
⎥ 0⎥ ⎥ ⎥ .⎥ ⎥ ⎥ .⎥ ⎥ ⎥ 0⎥ ⎦
(3)
1 6
2.2. The B-Snake model The cubic B-Snake is defined as follows [19]: r (s) =
gi (s) where 0s1.
(4)
i
The external energy term on r (s) is defined as E ext (r (s)). No internal force is required in B-Snake since the B-Spline representation maintains smoothness implicitly. Therefore, the total energy function of the B-Snake E B-snake can be defined by integrating E ext (r (s)) along the B-Snake. That is,
1
E B -snake =
E ext (r (s)) ds.
(5)
0
To identify the edge features in the image, the above expression must be minimized along the B-Spline curve such that, at equilibrium, r (s) stabilizes closely to the contour of object; the details are presented in next section. 2.3. Approaching object boundary using MMSE Gradient vector flow (GVF) [40] is selected here as the external forces for B-Snake, since GVF has a large capture range. It is computed as a diffusion of the gradient vector of a gray-level or binary edge map derived from the image. GVF filed is generated before conducting B-Snake deformation. When the B-Snake is on the boundaries of object, its external force f ext (r (s)) sampled from GVF should be zero. That is, f ext (r (s)) = 0.
(6)
Then there will be no change for the position of B-Snake. We assume that, at time t, the localized external force f ext (r t (s)) is connected with the position change r t (s)−r t−1 (s) in the following way: f ext (r t (s)) = (r t (s)−r t−1 (s)),
(7)
where is step-size. The position change of the B-Snake can be mapped to the adjustment of the control points Q at time t. For example, Q(t) is defined as the adjustment at time t, then we have Q t = Q t−1 +Q t .
(8)
The localized external force f ext (r t (s)) can be sampled along the B-Snake with a predefined distance (3 to 5 pixels in our experiment). In this way, we can solve Equation (7) numerically. Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
637
ADAPTIVE B-SNAKE MODEL
Here, the MMSE solution for the discretized version of Equation (7) is given in a matrix form [26]: Q t = −1 [M T M]−1 M T F,
(9)
where ⎡
M1
⎢ ⎢ 0 ⎢ ⎢ ⎢ · ⎢ ⎢ ⎢ 0 ⎢ M =⎢ ⎢ 0 ⎢ ⎢ M ⎢ n−1 ⎢ ⎢ M ⎢ n ⎣ Mn+1 ⎡
3 si,1
⎡
...
...
M2
0
...
·
·
·
...
0
Mn−3
...
...
0
0
...
0
0
...
0
0
...
0
2 si,1
si,1
2 si,2
si,2
·
·
2 si,m
si,m
⎢ 3 ⎢s ⎢ i,2 Mi = ⎢ ⎢ · ⎣ 3 si,m
0
3 sn−1,1
2 sn−1,2
sn−1,2
·
·
3 sn−1,m
2 sn−1,m
sn−1,m
1
3 sn−1,1
2 sn−1,1
sn−1,1
1
2 sn−1,2
sn−1,2
·
·
2 sn−1,m
sn−1,m
3 sn−1,m
⎡
3 sn,1
2 sn,1
sn,1
2 sn,2
sn,2
·
·
3 sn,m
2 sn,m
sn,m
3 sn,1
2 sn,1
sn,1
2 sn,2
sn,2
·
·
2 sn,m
sn,m
⎢ 3 ⎢s ⎢ n,2 Mn = ⎢ ⎢ · ⎣ ⎡ Mn
⎢ 3 ⎢s ⎢ n,2 =⎢ ⎢ · ⎣ 3 sn,m
Copyright 䉷 2010 John Wiley & Sons, Ltd.
1
−1
1 2 1 2 1 6
⎤⎡
1⎤ 6
⎥ 0⎥ ⎥ ⎥, 0⎥ ⎦
0 − 16
0 − 16 1 2
0 − 12 1 6
0
⎤⎡ 1 −2 1 ⎥⎢ 1 ⎢ 1⎥ ⎥⎢ 2 ⎥⎢ ⎢ 1 ·⎥ ⎦⎣ 2 1 6
1 6
− 12
−1
1 2 1 2 1 6
0 2 3
⎤ 0 0 0 ⎥ 0 0 0⎥ ⎥ ⎥ 0 0 0⎥ ⎥ ⎦ 0 0 0
⎤⎡ 1 6 ⎥⎢ ⎢ 0 1⎥ ⎥⎢ ⎥⎢ ⎢0 ·⎥ ⎦⎢ ⎣ 0 1
0
1 2
1 6
0
1 2
⎤ ⎥ ⎥ ⎥ ⎥, ⎥ ⎦
⎤
⎥ −1⎥ ⎥ ⎥, 0⎥ ⎥ ⎦ 2 3
0 0
⎤
0
⎥ 0 0⎥ ⎥ ⎥, 0 0⎥ ⎦
0
0 0
0
i = 1, 2, . . . , n −2.
0
⎥⎢ 1 ⎢ 1⎥ ⎥ ⎢0 2 ⎥⎢ 1 ⎢ ·⎥ ⎦ ⎣0 − 2
⎤ ⎡0 1 ⎥⎢ ⎢0 1⎥ ⎥⎢ ⎥⎢ ⎢0 ·⎥ ⎦⎢ ⎣ 0 1
1
− 12
2 3
1 6
sn−1,1
⎢ 3 ⎢s ⎢ n−1,2 Mn−1 =⎢ ⎢ · ⎣
1 2
0
2 sn−1,1
⎢ 3 ⎢s ⎢ n−1,2 Mn−1 = ⎢ ⎢ · ⎣ ⎡
⎥ 0 ⎥ ⎥ ⎥ · ⎥ ⎥ ⎥ 0 ⎥ ⎥ ⎥, Mn−2 ⎥ ⎥ Mn−1 ⎥ ⎥ ⎥ Mn ⎥ ⎥ ⎦ Mn+1
⎤⎡ 1 1 −6 ⎥⎢ 1 ⎢ 1⎥ ⎥⎢ 2 ⎥⎢ ⎢ 1 ·⎥ ⎦ ⎣− 2 1
⎤
0
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
638
Y. WANG ET AL.
Three new knot points will be inserted at these locations.
Figure 1. The locations of new knot points inserted.
⎡
3 sn+1,1
2 sn+1,1
sn+1,1
2 sn+1,2
sn+1,2
·
·
3 sn+1,m
2 sn+1,m
sn+1,m
3 sn+1,1
2 sn+1,1
sn+1,1
2 sn+1,2
sn+1,2
·
·
2 sn+1,m
sn+1,m
⎢ 3 ⎢s ⎢ n+1,2 Mn+1 = ⎢ ⎢ · ⎣ ⎡
⎢ 3 ⎢s ⎢ n+1,2 Mn+1 =⎢ ⎢ · ⎣ 3 sn+1,m
1
⎤⎡
0 0 0 − 16
⎤
⎥ ⎥⎢ ⎢0 0 0 1 ⎥ 1⎥ ⎥⎢ 2 ⎥ ⎥, ⎥⎢ ⎢0 0 0 − 1 ⎥ ·⎥ ⎦⎣ 2⎦ 1
0 0 0 ⎤ ⎡ 1 1 − 12 2 ⎥ ⎢ 1 ⎢ 1⎥ ⎥ ⎢ −1 2 ⎥ ⎢ 1 ⎢ ·⎥ ⎦ ⎣ 0 2 1
2 3
1 6
1 6 1 6
0
⎤
0
⎥ 0⎥ ⎥ ⎥, 0⎥ ⎦
0
0
0
where si,m is the parameter s in Equation (1) to form the sampling point m in the ith segment, and F = [... f ext (r t (si,m ))...]. 2.4. Control point insertion strategy To adapt the B-Snake to a complex contour, a structure-adaptive knot point insertion strategy is developed in [21, 41], which is adopted here with a modification to insert more control points rather than only one at the same time. The location of new knot point is determined by checking the external force on the B-Snake. And a new knot point is inserted into the location of B-Spline curve where maximal local external force exceeds a threshold, referring to Figure 1 for an example. Note that in the GVF field, the magnitude of the force reflects the distance to the nearest significant edge. The stronger external force means the farther distance to the nearest object boundary. Thus, the knot point should be inserted to adapt the B-Spline curve to the object boundary. Compared with other knot point insertion algorithms, this method is simple and fast.
3. LANDMARK ASSIGNMENT FOR B-SNAKE MODEL In this section, a novel method for fully automatic landmark point assignment using inflection point and affine invariant features will be introduced. 3.1. Inflection points of B-Snake In order to perform statistical analysis on the object properties over a set of images, one needs to first register the images. As pointed out in [42], a major problem associated with the B-Spline representation is the non-uniqueness of the control points, which makes the curve comparison very Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
639
ADAPTIVE B-SNAKE MODEL
difficult based on the control points. To overcome this problem, the affine invariant properties of the B-Spline curve can be used. Since the inflection point is a curvature zero point and affine invariant, it is an ideal candidate to solve the registration problem as aforementioned. It is noted that the inflection point has been used in [22] for the construction of absolute invariants in contour registration. Here the inflection point is applied to landmark point assignment for guiding the deformation of B-Snake model as detailed below. For each cubic B-Spline curve segment i, which is formed by control points {Q i , Q i+1 , Q i+2 , Q i+3 }, the parameter s associated with an inflection point can be calculated as follows [22]: √ √ −b + b2 −4ac −b − b2 −4ac s1 = , s2 = , (10) 2a 2a where a = (Q i × Q i+1 −2Q i × Q i+2 + Q i × Q i+3 +3Q i+1 × Q i+2 −2Q i+1 × Q i+3 + Q i+2 × Q i+3 ), b = (−3Q i × Q i+1 +4Q i × Q i+2 − Q i × Q i+3 −3Q i+1 × Q i+2 + Q i+2 × Q i+3 ), c = (2Q i × Q i+1 −2Q i × Q i+2 +2Q i+1 × Q i+2 ), s is real and confined to [0, 1). 3.2. Landmark points of B-Snake model Owing to the limited number of inflection points in each object shape, it may not be sufficient to construct the main shape structure. To get more landmark points, we present a method based on the derived inflection point and affine invariant feature as described in the following: (1) Smooth the B-Snake curve using the method in [6]. This method first smoothes the B-Spline curve by increasing the degree of the curve. It is followed by a reduction of the curve degree using the least-square error (LSE) approach to get a smoothed curve; (2) Construct the inflection points of B-Spline by Equation (10); (3) Calculate and add new points that are affine invariant into the inflection points and form the full set of landmark points. To get the new landmark point, we search the point with tangent equal to the line connecting the adjacent two landmark points. For a point located on the curve between the two inflection points, if its tangent is equal to the tangent of line that connects the two inflection points, the parallel is reserved under affine transform and this point will be considered as a new landmark point. For illustration, Figure 2 shows an example. In Figure 2(a), the two adjacent •
•
P new1
P new1 Pi+1
•
•
•P new3
• P i+1
P new2
Pi •
Pi •
(a)
(b)
Figure 2. Method used in identifying new landmark points, where (a) selects Pnew1 whose tangent is −→
parallel to Pi Pi+1 as a new landmark point, and (b) repeats the process after (a). Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
640
Y. WANG ET AL.
inflection points are Pi and Pi+1 . In the first-round landmark point insertion, the new landmark point Pnew1 is inserted. It is a point located on the curve between Pi and Pi+1 , and its tangent is parallel to the line that connects Pi and Pi+1 . Figure 2(b) shows the second-round landmark point insertion. Through this way, we can easily create as many landmark points as desired by the shape complexity. In our application, the exact number of landmark points is depended on the number of inflection point and two rounds landmark insertion. In general, one convex in a closed curve would create two inflection points. For the case of ventricle segmentation showed in Figure 5, there are four concaves in the shape and they generate eight inflection points. Here we only consider the significant concave as small ripples on the shape have been removed in Step 1. After two rounds landmark insertion, there are 24 landmark points inserted; therefore, a full set of landmark points for B-Snake model includes 32 points (8 inflection points +24 inserted landmark points). The main characters of our method are summarized as follows: (1) Only those points located on the B-Spline curve with affine invariant feature are chosen as the landmark points. This guarantees that the following process will not be affected under affine transformation. It avoids the problem that exist in ASM and AAM that corresponding landmark points may not be preserved under affine transformation, and it will largely improve the shape alignment and estimation of affine transform. (2) These landmark points can be derived directly from B-Spline without any manual operation. This is different from ASM and AAM where landmark points are manually assigned. (3) These landmark points are only used to extract the statistical information of primary appearance and structure of object, but not to describe the whole shape of object. They will be applied to roughly locate the proper location of B-Snake model in the process of deformation.
4. STATISTICAL APPEARANCE AND GEOMETRIC INFORMATION EXTRACTION FOR LANDMARK POINTS The statistical appearance and geometric information are very important prior knowledge of the object being studied. It makes sense to extract and implement them to guide the deformation of B-Snake to a proper location. To use the statistical information of an object, we need a set of training samples. One sample in the training set is chosen as the reference shape and others are aligned to it. After that, a variety of object statistics, e.g. intensity, geometry and appearance, can be extracted for further implementation.
4.1. Shape alignment strategy To determine the correspondence between two B-Snakes, the shape algorithm in [43], which uses an affine-invariant feature for shape alignment, is employed. In this algorithm, for every landmark point Pi obtained in the last section, an attribute vector Fi is calculated by the areas formed by adjacent landmark points. As the area is affine invariant, the attribute vector is affine-invariant accordingly. Hence, the affine transformation can be estimated subsequently. The triangles’ area in Figure 3 shows an example on the attribute vector.
4.2. Modeling the structure of landmark points PCA is applied to model the distribution of landmark points. The data form a cloud of points in the nd space. PCA computes the main axes of this cloud, allowing one to approximate any of the original points using a model with less than n-d parameters. After modeling the distribution of landmark points, new shapes can be generated whose landmark points are similar to those in the original training set. The new shapes can also be used to decide whether they are plausible shapes. We briefly Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
641
ADAPTIVE B-SNAKE MODEL
Figure 3. Illustration of the concept of the ‘attribute vector’ on the ith landmark point. The area of a triangle P[i−vs] P[i] P[i+vs] is used as the vsth element of the ith attribute vector Fi . Here, 1vsm/2, m is the number of landmark point. align
introduce this below. The m+1 aligned landmark points {Pi presented as a 2(m+1)-element vector: align
P = [x0
align
, x1
align
, . . . , xm
align
, y0
align
, y1
align
= (xi
align
, yi
align
, ..., ym
), i = 0, 1, . . . , m} are
].
Given N sets of aligned landmark points, the approach is as follows: ¯ and the covariance of the data, C. 1. Compute the mean of data, P, 2. Compute the eigenvectors, i , and the corresponding eigenvalues i of C. 3. In order to approach any of the training set, we can use: P ≈ P¯ +bT ,
(11)
where contains the k eigenvectors corresponding to the largest eigenvalues and b is a k-d vector given by ¯ b = (P − P)∗
(12)
The number of eigenvectors to retain, k, is chosen so that the model represents 98% of total variance of the training data. 4.3. Appearance information extraction In order to extract the appearance information around each landmark point of aligned shapes, the original image of the aligned shape has to be registered to the reference one. This is done by applying the affine transform calculated in the shape alignment stage in the last section. Figure 4 shows an example, where (a) is the original image with the extracted shape. The reference shape is shown in (b) and (c) is the shape alignment with the reference one. The affine transformed image of (a) based on (c) is shown in (d). For every landmark point in each affine transformed image, the pixel intensities of a neighborhood are taken as a vector ti . The gradient of ti , qi is used to extract the appearance information. Then qi is normalized by: qi qˆi =
j qij
Copyright 䉷 2010 John Wiley & Sons, Ltd.
(13)
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
642
Y. WANG ET AL.
Figure 4. Aligned and affine transformed images: (a) shape to be aligned; (b) reference shape image; (c) aligned shape; and (d) affine transformed image of (a).
We repeat this for each training image and get a set of normalized sampled {qˆi } for the given landmark points. The average values of each landmark point of B-Snake model, q, ¯ are used to measure the appearance. The quality of fitting a new sample ql to the model is given by: f (ql ) = (ql − q)∗(q ¯ ¯ T l − q)
(14)
Minimizing f (ql ) is equivalent to maximizing the similarity of appearance to the training data. During the search stage, we sample the intensities in a profile around the current landmark point and calculate the similarity, and the location with the best match is chosen as the new position for the landmark point.
5. ALGORITHM FOR OBJECT SEGMENTATION USING APPEARANCE AND GEOMETRIC STATISTIC INFORMATION Based on the rough location of the B-Snake by searching the similar appearance and structure in the training set, the B-Snake is deformed to the studied object more precisely with the external forces and an adaptive strategy of inserting control points. The algorithm using both appearance and geometric statistical information is as follows: Get a reference model {Q imodel , i = 0, 1, . . . , N } as mentioned in Section 4. Initialize the control points of B-Snake model as {Q i = Q imodel , i = 0, 1, . . . , N }. Calculate the inflection points and define the landmark points P. For each landmark point, search the region nearby and get the best appearance fitting location (by minimizing Equation (14)), then form a new landmark vector P1 . 5. Map the landmark vector P1 to the training set using PCA (refer to Section 4), and get the updated landmark vector P2 . 6. Deform the B-Snake model using MMSE to landmark P2 (here use the vectors from P1 to P2 to replace the external force in MMSE), and get the new control points Q i of B-Snake. If the changes of Q i are greater than a threshold1 , go to step 3. 7. Deform the B-Snake using MMSE to minimize external force. If the iteration number exceeds a predefined number or external force among B-Snake is below a predefined value threshold2 , go to step 9. Otherwise, if the movement of control points is less than a threshold3 , but the maximum of external force has exceeded threshold4 , a new control point will be inserted, repeat step 7. 1. 2. 3. 4.
Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
643
ADAPTIVE B-SNAKE MODEL
8. Re-calculate the landmark points, align the current Snake configuration with the standard model contour and get the affine-transformation matrix Aalign (as detailed in Section 4.1). Then, stack the aligned landmarks as a vector align
P = [x0
align
, x1
align
, . . . , xm
align
, y0
align
, y1
align
, ..., ym
]
Go to step 4. 9. Stop.
6. EXPERIMENTS A database including 150 brain MR images has been set up to evaluate the performance of the proposed B-Snake Model. Three experiments have been carried out: (a) an experiment that demonstrates the performance of landmark point assignment algorithm; (b) an experiment for ventricle segmentation; and (c) a comparison of the proposed method with respect to the traditional Snake and ASM. 6.1. Landmark point assignment and shape alignment The method presented in Section 3.2 for landmark point assignment has been implemented to a set of MR images with extracted ventricle shapes. Some of the shapes are shown as contours in Figure 5 for describing the ventricle boundaries. Figure 5 shows a few examples where the inflection points and new landmark points are inserted in real brain MR images. Points are the inflection points, whereas the ‘x’ marks are the new inserted landmark points. Note that the inflection points are also regarded as landmark points in the following procedures. In these images, it can be observed that most of the landmark points are located on the key positions. The CPU time for an image is less than 20 ms. These landmark points will be used in shape alignment and statistical information extraction in the following stages.
Figure 5. Examples on new landmark points insertion. Points are inflection points, whereas points with ‘x’ mark are new landmark points inserted. Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
644
Y. WANG ET AL.
Figure 6. Some aligned results of the B-Snake model.
For shape alignment based on assigned landmark points, one of the shapes in training set was chosen as the reference (see the first sample of the shapes in Figure 5), and the rest are aligned to it. Some results of shape alignment are shown in Figure 6, where the gray color shape is the reference and the black ones are the aligned shapes in the training set. The obtained results show that the proposed method has good performance in shape alignment. The assumed affine transformation between each alignment is estimated as well, and the affine transformed images are then processed to extract the statistical appearance and geometric information for each landmark point. 6.2. Experimental results for object segmentation A set of 105 ventricle contours is used as the training set and other 45 images for testing in this experiment. One of the shapes in the training set was chosen as the initialization for B-Snake model. Figure 7 shows an example on applying our B-Snake model to object segmentation. The initialization of the B-Snake model is shown in (a). After first iteration, the landmark points have been assigned new locations with a similar appearance in the training set, as shown in (b). The locations of landmark point after second iteration is shown in (c), and the corresponding B-Snake model shape is shown in (d). There are a total of six iterations before applying the MMSE and knot insertion strategy. The results of third to sixth iterations are shown in Figure 7(e) to (j), respectively. As we can see, the shape of B-Snake has been deformed to the ventricle boundary more and more closely. The MMSE and knot insertion strategy have been involved after termination of search with a similar appearance and statistical information for landmark points. From Figure 7(j), it can be observed that MMSE steered the B-Snake model toward the object edges, whereas the knot point insertion strategy aligned the B-Snake to the object shape. The final result is shown in Figure 7(k). A demo of the proposed B-Snake model deformation can be accessed at http://www1.i2r.a-star.edu.sg/∼ywang/demo.htm. Figure 8 shows more results of ventricle segmentation in different MR brain images using the proposed B-Snake model. The gray contours are the initial shapes of B-Snake, whereas the white contours denote the final results. Note that some of the deformations from the initialization to the final result are significant. As shown in Figure 8, the B-Snakes approached ventricle boundaries accurately regardless of the variety of shapes and sizes of ventricles in the brain MR images. 6.3. Comparison with the traditional Snake and ASM In order to demonstrate the robustness of our B-Snake model, a comparative study of our BSnake model with the traditional Snake and ASM has been carried out. Figure 9 qualitatively compares the performance of traditional Snake, ASM and our B-Snake model in the ventricle segmentation. Initialization for four different MR images are shown in Figure 9(a1)–(a4); the three models are initialized with the same location and contour. Figure 9(b1)–(b4) show the final results of the traditional Snake for the four MR images, respectively. From there we can observe that the traditional Snake lacks the capability to deform to the ventricle edges. In addition, it does not preserve any anatomical homology during the deformation. ASM segmentation results are much better than the traditional Snake in preserving the model shape in the deformation process (see Figure 9(c1)–(c4)). However, due to the fact that ASM is only modeling the points sampled along the lines perpendicular to the landmark points of contour in the training set, it only describes the texture of the lines, but not the appearance of regions. Therefore, it may fail to locate an acceptable result if initialized not close to the object. This may explain why some parts of ASM model curve Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
ADAPTIVE B-SNAKE MODEL
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
645
Figure 7. B-Snake model in ventricle segmentation. In (b)–(c), (e)–(f) and (h)–(i), white points denote the landmark points before deforming whereas ‘x’ marks denote the landmark points after deforming: (a) initialization; (b) after one iteration; (c) after two iterations; (d) ABM after two iterations; (e) after three iterations; (f) after four iterations; (g) ABM after four iterations; (h) after five iterations; (i) after six iterations; (j) ABM after six iterations; (k) final result with MMSE and knot point insertion strategy and (l) final result with control points.
in Figure 9(c1)–(c4)) are either trapped in local minimum or are not approaching the desired object boundaries precisely. Hence, its final results in Figure 9(c1) to (c4) are not accurate. On the contrary, the proposed B-Snake model can attain good results as shown in Figure 9(d1)–(d4). In addition to preserve the model shape, our model can match the ventricles’ boundaries more Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
646
Y. WANG ET AL.
Figure 8. More segmentation results of different MR brain images using the proposed B-Snake model. In (a)–(i), the gray contour in each image is the initialization of B-Snake whereas the bright one is the final result.
precisely. Table I gives the quantitative comparison of our B-Snake model with the traditional Snake and ASM, using the results presented in Figure 9. The average and the maximum distance with respect to the ground truth as marked by experts are shown in Table I. The proposed model has the least average distances at 1.7–2.7 pixels and the least maximal distance at 2.6–6.5 pixels for all the four examples. In view of the above comparison, our B-Snake model clearly demonstrates its superiority over the two other deformable models. Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
647
ADAPTIVE B-SNAKE MODEL
Image 1
(a1) Initialization Image 1
(b1) result of Snake Image 1
(c1) result of ASM Image 1
(d1) result of ABM
Image 3
Image 2
(a2)
Initialization
Image 3
Image 2
(b2) result of Snake
result of ASM
(c3) result of ASM Image 3
Image 2
(d2)
(b3) result of Snake Image 3
Image 2
(c2)
(a3) Initialization
result of ABM
(d3) result of ABM
Image 4
(a4) Initialization Image 4
(b4) result of Snake Image 4
(c4) result of ASM Image 4
(d4) result of ABM
Figure 9. Qualitative comparison of our B-Snake model with the traditional Snake and ASM. (a1)–(a4) indicate the initialized contours; (b1)–(b4) indicate the results of traditional Snake; (c1)–(c4) indicate the results of ASM; and (d1)–(d4) indicate the results of our B-Snake model.
7. CONCLUSION In summary, this study presents a B-Snake Model for object segmentation. A fully automatic landmark point assignment strategy is developed for the B-Snake to extract the geometric structure and appearance information. It uses affine invariant features and greatly reduces the time for assigning the landmark point. Furthermore, a statistical framework is embedded into the B-Snake model in order to integrate the prior knowledge of the object under study. Constrained by prior geometric knowledge that reflects shape and appearance characteristics of the model around each landmark point, our B-Snake model can keep its geometric shape and search for similar appearance during the process of deformation. The structure-adaptive capability of the proposed B-Snake Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
648
Y. WANG ET AL.
Table I. Quantitative comparison of our B-Snake model with respect to the traditional Snake and ASM based on the results in Figure 9: the ground truth is manually marked by experts and the distance (average and maximum) is in units of pixels. Image 1
Traditional Snake ASM Our B-Snake model
Image 2
Image 3
Image 4
Avg.
Max.
Avg.
Max.
Avg.
Max.
Avg.
Max.
5.1 3.7 2.1
19.3 16.7 4.9
6.2 3.1 2.7
20.1 8.9 6.5
4.2 5.4 1.9
18.9 10.2 3.1
4.8 3.6 1.7
22.8 8.8 2.6
model is achieved by the strategy of adaptively inserting control points and the MMSE approach during the B-Snake deformation process. It provides fine deformation and allows the B-Snake to accurately adapt to the desired object boundaries in the image. This method overcomes the problems that existed in other B-Spline based models that have to predetermine the number of control points of B-Spline; hence, the proposed method is more flexible to describe un-known complex shapes. The simulation results show that our B-Snake model has a robust capability to describe more complex object contours and it can attain a more accurate object contour segmentation. In this paper, we mainly focus on ventricle segmentation in medical images. However, this method obviously can be used for any object segmentation, such as face and human body segmentation in surveillance system, etc. In addition, it is also suitable for object tracking. Although our model can be employed to segment 3-D objects by slice-to-slice segmentation, the results may be worse than those 3-D B-Surface models. Using B-Surface to construct and describe the object surface remains a challenge. Two problems need to be solved. First, how to define the network of B-Surface patches that are continuously connected at their boundaries without any overlap or leakage to form a close object surface is difficult. The technique of multiple control points may be used to form a particular shape of patch or using different orders of B-Surface patched to form one surface. Second, although the MMSE method in this paper is only meant for 2-D purpose, it is also possible to be extended to 3-D B-Surface model for estimating the 3-D object. Here, the principle for deforming the B-Surface is the same as the B-Snake model, but more calculation is expected for the B-Surface deformation.
REFERENCES 1. Kass M, Witkin A, Terzopoulos D. Snakes: active contour models. International Journal of Computer Vision 1987; 1(4):321–331. 2. Person E, Fu KS. Shape discrimination using Fourier descriptors. IEEE Transactions on Systems, Man and Cybernetics 1977; 7:170–179. 3. Huang Z, Cohen FS. Affine-invariant B-Spline moments for curve matching. IEEE Transactions on Image Processing 1996; 5(10):1473–1480. 4. Klein AK, Lee F, Amini AA. Quantitative coronary angiography with deformable spline models. IEEE Transactions on Medical Imaging 1997; 16(5):463–477. 5. Wang Y, Teoh EK. Dynamic B-Snake model for complex objects segmentation. Image and Vision Computing 2005; 23(12):1029–1040. 6. Wang Y, Teoh EK. 2D affine-invariant contour matching using B-Spline model. IEEE Transactions on Pattern Analysis and Machine Intelligence 2007; 29(10):1853–1858. 7. Bernard O, Friboulet D, Thevenaz P, Unser M. Variational B-Spline level-set: a linear filtering approach for fast deformable model evolution. IEEE Transactions on Image Processing 2009; 18(6):1179–1191. 8. Paulik MJ, Das M, Loh NK. Nonstationary autoregressive modeling of object contours. IEEE Transactions on Signal Processing 1992; 40:660–675. 9. Flusser J, Suk T. Pattern recognition by an affine moment invariant. Pattern Recognition 1993; 26:167–174. 10. Fred ALN, Marques JS, Jorge PM. Hidden Markov models vs syntactic modeling in object recognition. Proceedings of International Conference on Image Processing (ICIP’97), Washington, DC, U.S.A., 1997; 893–896. 11. Tueng QM, Boles WW. Wavelet-based affine invariant representation: a tool for recognizing planar objects in 3D space. IEEE Transactions on PAMI 1997; 19:846–857. 12. Xie X, Mirmehdi M. MAC: magnetosatic active contour model. IEEE Transactions on Pattern Analysis and Machine Intelligence 2008; 30(4):632–646. Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
ADAPTIVE B-SNAKE MODEL
649
13. Xie X. Active contouring based on gradient vector interaction and constrained level set diffusion. IEEE Transactions on Image Processing 2010; 19(1):154–164. 14. Krinidis S, Chatzis V. Fuzzy energy-based active contours. IEEE Transactions on Image Processing 2009; 18(12):2747–2755. 15. Paragios N, Mellina-Gottardo O, Rameshm V. Gradient vector flow fast geometric active contour. IEEE Transactions on Pattern Analysis and Machine Intelligence 2004; 26(3):402–407. 16. Foulonneau A, Charbonnier P, Heitz F. Affine-invariant geometric shape priors for region-based active contours. IEEE Transactions on Pattern Analysis and Machine Intelligence 2006; 28(8):1352–1357. 17. Foulonneau A, Charbonnier P, Heitz F. Multi-reference shape priors for active contours. Internal Journal on Computer Vision 2009; 81:68–81. 18. Bresson X, Esedoglu S, Vanderheynst P, Thiran J, Osher S. Fast global minimization of the active contour/Snake model. Journal of Mathematical Imaging and Vision 2007; 28:151–167. 19. Menet S, Saint-Marc P, Medioni G. B-Snake: implementation and application to stereo. Proceedings of the DARPA, Pittsburgh, PA, U.S.A., 1990; 720–726. 20. Baumberg AM, Hogg DC. Learning flexible models from image sequences. European Conference on Computer Vision’94, Stockholm, Sweden, vol. 1, 1994; 299–308. 21. Wang Y, Teoh EK, Shen DG. Structure-adaptive B-Snake for segmenting complex objects. International Conference on Image Processing, Thessaloniki, Greece, 2001; 769–772. 22. Ali WSI, Cohen FS. Registering coronal histological 2-D sections of a rat brain with coronal sections of a 3-D brain atlas using geometric curve invariants and B-Spline representation. IEEE Transactions on Medical Imaging 1998; 17(6):957–966. 23. Stammberger T, Rudert S, Michaelis M, Reiser M, Englmeier KH. Segmentation of MR Images with B-Spline Snakes: a multi-resolution approach using the distance transformation for model forces. CEUR Workshop Proceedings, Trento, Italy, vol. 12, 1998. 24. Figueiredo MAT, Leitão JMN, Jain AK. Unsupervised contour representation and estimation using B-Splines and a minimum description length criterion. IEEE Transactions on Image Processing 2000; 9(6):1075–1087. 25. Wang Y, Shen DG, Teoh EK. Lane detection using spline model. Pattern Recognition Letters 2000; 21(8):677–689. 26. Wang Y, Teoh EK, Shen DG. Lane detection and tracking using B-Snake. Image and Vision Computing 2004; 22(4):269–280. 27. Cootes TF, Taylor CJ, Cooper DH, Graham J. Training models of shape from sets of examples. Proceedings of the British Machine Vision Conference, Leeds, U.K., 1992; 9–18. 28. Cootes TF, Tayor CJ, Cooper DH, Graham J. Active shape models—their training and application. Computer Vision and Image Understanding 1995; 61(1):38–59. 29. Cootes TF, Hill A, Taylor CJ, Haslam J. The use of active shape models for locating structures in medical images. Image and Vision Computing 1994; 12(6):355–366. 30. Smyth PP, Taylor CJ, Adams JE. Automatic measurement of vertebral shape using active shape models. Proceedings of the British Machine Vision Association Conference, Edinburgh, U.K., 1996; 705–714. 31. Solloway S, Taylor CJ, Hutchinson CE, Waterton JC. The use of active shape models for making thickness measurements from MR images. Proceedings of the Fourth European Conference on Computer Vision, Cambridge, U.K., 1996; 400–412. 32. Behiels G, Vandermeulen D, Maes F, Suetens P, Dewaele P. Active shape model-based segmentation of digital X-ray images. MICCAI ’99. Lecture Notes in Computer Science. Springer: Berlin, Germany, 1999; 128–137. 33. Vogelsang F, Kohnen M, Mahlke J, Weiler F, Kilbinger MW, Wein B, Günther RW. Model based analysis of chest radiographs. Proceedings of the SPIE, San Diego, CA, U.S.A., vol. 3979, 2000; 1040–1052. 34. Hamarneh G, Gustavsson T. Combining Snakes and active shape models for segmenting the human left ventricle in echocardiographic images. Computers in Cardiology, vol. 27. IEEE: Piscataway, NJ, 2000; 115–118. 35. Cootes TF, Edwards GJ, Taylor CJ. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001; 23(6):681–685. 36. Davies RH, Twining CJ, Cootes TF, Waterton JC, Taylor CJ. A minimum description length approach to statistical shape modeling. IEEE Transactions on Medical Imaging 2002; 21(5):525–537. 37. Hill A, Taylor CJ. Automatic landmark generation for point distribution models. British Machine Vision Conference, University of York, U.K., 1994; 429–438. 38. Hill A, Taylor CJ. A method for non-rigid correspondence for automatic landmark identification. Proceedings of the BMVC, Edinburth, U.K., 1996; 323–332. 39. Wang Y, Teoh EK, Hou Z, Wang J. Object boundary extraction using active B-Snake model. International Conference on Pattern Recognition, Tampa, U.S.A., December 2008; 1–4. 40. Xu C, Prince JL. Snakes: shapes and gradient vector flow. IEEE Transactions on Image Processing 1998; 7(3):359–369. 41. Wang Y, Teoh EK. Object contour extraction using B-Snake model. Journal of Mathematical Imaging and Vision 2006; 24(3):295–306. 42. De Boor C. A Practical Guide to Splines. Springer: New York, 1978. 43. Horace HS, Shen DG. An affine-invariant active contour model (AI-Snake) for model-based segmentation. Image and Vision Computing 1998; 16(2):135–146.
Copyright 䉷 2010 John Wiley & Sons, Ltd.
Int. J. Numer. Meth. Biomed. Engng. 2011; 27:633–649 DOI: 10.1002/cnm
