Active Shape Models and the Shape Approximation ... - CiteSeerX

Active Shape Models and the Shape Approximation Problem A. Hill, T. F. Cootes and C. J. Taylor Department of Medical Bio-Physics University of Manchester [email protected]

Abstract The Active Shape Model(ASM) is an iterative algorithm for image interpretation based upon a Point Distribution Model. Each iteration of the ASM has two steps: Image data interrogation followed by shape approximation. Here we consider the shape approximation step in detail. We present a new method of shape approximation which uses directional constraints. We show how the error term for the shape approximation problem can be extended to cope with directional constraints and present iterative solutions to the 2D and 3D problems. We also show how the error term can be modi ed to allow a closed solution in the 2D case.

Keywords

Statistical Shape Models, Flexible Templates, Active Shape Models.

1 Introduction We have previously described a generic approach to interpreting both 2D and 3D images [2{5]. The essential components of the system are a compact model representing the shape of a set of variable objects - a Point Distribution Model (PDM) - and an iterative method of image search. The combination is known as an Active Shape Model (ASM). A PDM is a statistical shape model which is generated from a set of annotated examples of the objects(s) to be modelled. The ASM is an iterative procedure which locates an instance of a PDM in a given image. Each iteration of the ASM has two steps: image data interrogation followed by shape approximation. In the rst step a new instance of the model in the image is proposed. In the second step the proposed shape is approximated as closely as possible whilst applying shape constraints captured by the PDM. Here, we consider the shape approximation step of the ASM in detail. We present a new approach to the problem which uses directional constraints to reduce the number of iterations required for the ASM to converge and increases the accuracy of the interpretation. We show how the representation of the errors we wish to minimise when approximating a given image shape can be expressed in a more formal way than previously and can be extended to cope with the case of

Hill,Cootes,Taylor(2)

British Machine Vision Conference

directional constraints. Iterative solutions for the most general case are presented for both the 2D and 3D problems. We also show that, by modifying the error term slightly, a closed, rather than iterative, solution to the shape approximation problem can be achieved in 2D.

2 Background We will brie y review how a PDM is generated and how the ASM image search procedure locates an instance of the PDM within an image. For a detailed description the reader is referred to [2{4].

2.1 Point Distribution Models

A PDM is generated via a principal components analysis of a training set of N object descriptions fy ; (1  i  N )g. An object description, y , is simply a labelled set of points fy ; (1  j  n)g which we will call landmarks. The analysis involves; aligning the set of examples into a common frame of reference, fx = aligned(y ); (1  i  N )g; calculating the mean of the aligned examples, x, and the deviation from the mean of each aligned example x = x ? x; calculating P the eigensystem of the the co-variance matrix of the deviations, C = (1=N ) =1 x x . The t principal eigenvectors of the eigensystem are then used to generate examples of the modelled objects via the expression : i

i

i;j

i

i

i

N i

i

i

T i

x = x + Pb

(1)

where b is a t-element vector of shape parameters and P is a (2n  t) in 2D or (3n  t) in 3D matrix of t eigenvectors. By selecting b from a pre-de ned shape vector space, established from the set of training examples, new instances of the modelled object(s) can be generated. This enables the PDM to represent previously unseen examples and forms the basis for locating examples of the modelled object(s) in unseen images via the ASM.

2.2 Active Shape Models

The ASM uses an iterative algorithm for locating an instance of a PDM in a given image assuming some initial guess of the shape, b, and pose, Q, of the object(s) in the image is available. At each iteration the current instance of the PDM is projected into the image via Q and the image data in the vicinity of the model instance is interrogated, usually along pro les normal to the boundary/surface see gure 1. This leads to a new proposed set of landmarks within the image. The ASM approximates this proposed model instance as closely as possible while remaining within the shape vector space. This approximation then provides the starting point for the ASM at the next iteration.

Hill,Cootes,Taylor(3)

British Machine Vision Conference

. . . . . . ........ .... . . . . . . . . . . . .

1 f .... ... .... ...... .... ....

........

c

. . . . . . ........ ..... . . . . . . . . . . . .

2

c

current landmarks proposed landmarks edge in image model boundary search pro le

u

.. .. .. .. ....... .... .. .. .. .. . . .. .. .. ..

c

3

c

....... ............................ ...

4

c

...... ............................ ....

5

e

JJ T ] v  6 3   T J T  . . . . . . . . . . . . . .d J . ....  J...d ? ..... . . . .. K ? ... . A A eu .... A . . . ev . . .

. . . . . . . . . . . . . . . . .

model boundary

d

Figure 1: Searching for New Landmarks Figure 2: Re-de ning the Error Vector

3 The Shape Approximation Problem The shape appriximation problem we consider is simply stated : nd a set of values, fQ,bg, which best approximate a set of landmarks de ned in the image frame, fxI ; (1  i  n)g. We use a least squares measure of the errors between the landmarks proposed in the image and the landmarks represented by fQ,bg as the criterion for the best approximation. In this section we show how to generate the simplest form of the error term to be minimised. We then consider how the introduction of directional constraints re-de nes the error vectors and extend the error term accordingly. i

3.1 The Standard Error Term

The i model point is de ned by : th

x = x + P b M i

i

(2)

i

where P is formed from the appropriate rows of P in equation(1) associated with the i point. If b is a (t  1) vector, then P is a (2  t) matrix in 2D and a (3  t) matrix in 3D. The pose, Q, can be used to transform the image shape into the model frame or vice versa. We must decide which frame the errors are to be measured in. This is important because, as Horn[6] points out, if we nd the pose which minimises the errors resulting from transforming the image points into the model frame, Q1 say, and the corresponding pose which minimises the errors resulting from transforming the model points into the image frame, Q2 say, then Q1 6= Q?2 1 . We wish to approximate the shape in the image frame as closely as possible since the aim of the ASM is to locate the instance of the model in the image. The error term we wish to minimise is thus : i

th

i

E2 =

X



2 e ; e = x ? Q(x n

I

=1

i

i

I i

I i

M i

)

(3)

Hill,Cootes,Taylor(4)

British Machine Vision Conference

3.2 Directional Constraints

Consider the situation shown in gure 1. If we use a standard least-squares approach to minimise equation(3) we nd that model points f1,2g are prevented from moving closer to their target points because of the increased vertical error in points f4,5g. Similarly, points f4,5g are prevented from moving closer to their target points because of the increase in the horizontal error of points f1,2g. However, points f1,2g lie on a horizontal edge and points f4,5g lie on a vertical edge. We would be willing, therefore, to allow points f1,2g/f4,5g to move freely in a horizontal/vertical direction, as long as the distance to their target boundaries were reduced. This is the so called aperture problem discussed in [1]. Below, we describe a general framework for measuring the errors along pre-de ned unit vectors and weighting these modi ed errors in such a way that points are allowed to move more freely around the boundary/surface of the modelled object(s).

3.2.1 Directional Constraints in 2D

For the i point we have an error between the current model point and its corresponding target point of e and a pre-de ned unit vector, u , normal to the boundary along which we want to measure the error i.e e = u e (see gure 2). To measure the error along a direction tangential to the boundary we use e = v e where v = [?u ; u ] . The total weighted error term for the i point is now given by : th

i

i

T i

u;i

v;i

T i

i

T

i

i

i;y

 e2 + e2 =  e i

i

u;i

v;i

th

i;x

u u e + e v v e = e W e ; where W =  u u + v v i

T i

T i

T i

i

i

i

i

T i

i

i

T i

i

i

i

i

T i

i

i

(4)

T i

By varying  ; we can change the relative importance of the errors normal and tangential to the boundary at the i point. i

i

th

3.2.2 Directional Constraints in 3D

Here, we consider measuring the error along both a unit normal to the surface, u , and in the plane tangential to the surface. The error measured along the unit normal is e = u e . The component of the error vector in the tangential plane is given by : i

T i

u;i

i

e = e ? ?u e
u = V e where V = ?I ? u u
v;i

i

T i

i

i

i

i

i

i

(5)

T i

The total weighted error term for the i point is given by : th

 e2 + e i

u;i

i

T v;i

e = e u u e + e V V e ?
= e W e ; where W =  u u + I ? u u 2 v;i

i

T i

T i

T i

i

i

i

i

3.3 Generalised Error Term

i

T i

T i

i

i

i

i

i

T i

i

i

T i

(6)

In the previous section we showed that directional constraints result in a symmetric weight matrix associated with each error vector. In both the 2D and 3D directional

Hill,Cootes,Taylor(5)

British Machine Vision Conference

weighting schemes the weight matrices are derived from a set of unit normals. These normals are de ned by the current instance of the model which is itself de ned by the shape vector, b. We will use fWM ; (1  i  n)g to denote the weight matrices de ned in the model frame. In order to measure the errors in the image frame the unit normals must be rotated into the image frame. It can be shown that, if R is the rotation matrix associated with the pose Q, then the weight matrices de ned in the image frame are given by fWI = RWM R ; (1  i  n)g. The error term we wish to minimise in the general case is now given by : i

T

i

E2 =

X n

=1

i

(xI ? Q (xM )) WI (xI ? Q (xM ))

(7)

T

i

i

i

i

i

i

where xM is de ned by equation(2) and Q(x) = sRx + t where s is the scale factor, R is the rotation matrix and t is the translation. i

4 A Two Stage Iterative Solution We can minimise equation(7) using a two stage iterative scheme in which we rst assume we have a known approximation to b and solve for Q; then, given an approximation for Q, solve for b and so on. This is the method used in the shape approximation step of the ASM as described previously[3, 4]. The dierence here is that we are going to solve for fQ,bg with respect to equation(7), thus guaranteeing the minimisation of the errors in the image frame for a combined set of fQ,bg values in the general case of full (2  2) or (3  3) weight matrices.

4.1 Solving for Q given b - the Alignment Problem Here, we must solve :

Min E 2 =

X n

=1

(xI ? fsRxM + tg) WI (xI ? fsRxM + tg)

(8)

T

i

i

i

i

i

i

where xM is de ned by equation(2). We consider the 2D and 3D cases separately. i

4.1.1 Alignment in 2D   In 2D we use sR = dc ?dc where c = s cos(); d = s sin() and  is the angle of

rotation. In order to minimise equation(8) we set the partial derivatives to zero :



X @Q(xM ) @E 2 = 2 @v @v =1 n

j

i

j

i



T

8 (1  j  4) < W (x ? Q(x )) = 0; : v1 = c; v2 = d; v3 = t v4 = t I

i

I i

M i

x

(9)

y

The partial derivatives of Q are given by :  ?xM  @Q(xM )  1  @Q(xM )  0  @Q(xM ) @Q(xM ) M = x ; = xM ; @v3 = 0 ; @v4 = 1 (10) @v1 @v2 i

i

i

i;y

i;x

i

i

Hill,Cootes,Taylor(6)

British Machine Vision Conference

We can express equation(9) as follows :

X  @Q(x )  n

=1

i

T

M

@v

i

j

W

i

k



) v = X @Q(xM ) @v =1 =1 @v

4 @Q(x X

I

n

M i

i

k

k



T

W x (1  j  4) I

i

I

i

j

i

(11) These 4 equations in the 4 unknowns fc; d; t ; t g can be solved using standard methods. For scalar weights, WI = w , this solution can be shown to be the same as that described for the original ASM method [4]. x

y

i

i

4.1.2 Alignment in 3D

To minimise equation(8) in 3D when using scalar weights, WI = w , we can use Horn's method of alignment [6] which uses unit quaternions to represent the rotation. Horn's method relies on being able to isolate the rotational element of the alignment problem. The scale and translation are computed once the rotation is known. By using quaternions the alignment of two 3D pointsets can be solved as a linear problem via an eigenvector analysis. Here, the coupling of the components of the error vectors, eI , via the weight matrices, WI , means that the rotation cannot be isolated in this fashion. We have used an alternative, iterative approach. We represent the rotation using quaternions but allow the quaternion to also perform a scaling (i.e. we no longer consider unit quaternions). We will use Horn's notation for quaternions : q = q0 + q i + q j + q k. The complex conjugate is     2 2 2 2 2

de ned by q = q0 ? q i ? q j ? q k and q
q = q0 + q + q + q = q . For the de nition of multiplication etc. see Horn[6]. A 3D point, p, is represented by the purely imaginary quaternion p = 0 + p i + p j + p k and a rotation of p is de ned by q p q  with q
q  = 1. To enable q to perform a scaling as well as a rotation we do not require a unit quaternion but set the scale using s = q
q  . We can write equation(8) as : i

i

i

i

x

x

y

y

x

x

Min E 2 =

X n

=1

z

z

y

y

z

z

(xI ? =fq x M q  g ? t) WI (xI ? =fq x M q  g ? t) T

i

i

i

i

i

(12)

i

 where =fq g is the imaginary part of the quaternion q. To solve equation(12) we use an iterative approach as follows: Assume we have some estimate of q available at the r iteration, q [ ] say. We generate a new  estimate at the (r + 1) iteration using q [ +1] = q [ ] + . We can now write : th

r

th

r

r









=fq [ +1]x q [ +1]g = =fq [ ]x q [ ] + x q [ ] + q [ ]x  + x  g M

r

i

r

r

M i

M i

r

r

r

M

M

i

i



We assume that  is small with respect to q [ ] and approximate the previous expression using : r







=fq [ ] x q [ ] + x q [ ] + q [ ] x  g = s[ ]R[ ]x + G  2 g g g ?g 3 0 G = 4 g ?g g 0 g 5 ; g = 2x q [ ] g g ?g g 0 r

i

M i

M i

r

r

r

i;x

i;

i;z

i;y

i;y

i;z

i;

i;x

i;z

i;y

i;x

i;

M

r

i

i

M

r

i

M i

r

i

(13) (14)

Hill,Cootes,Taylor(7)

British Machine Vision Conference





     where we have used =f a x b g = =f b x a g for the purely imaginary quaternion x.   In equation(13)  is regarded as a 4-element vector in the term G  . Equation(12) now becomes : i

Min E 2 =

X n

=1





(z ? G  ? t) WI (z ? G  ? t); z = xI ? s[ ] R[ ]xM i

T

i

i

i

i

i

r

i

r

(15)

i

i



Equation(15) is linear with respect to  ,t and we can write :

7 @ eI ! 7 @ eI ! X X X 2 I z ? @v v W z ? @v v Min E = (16) =1 =1 =1 with v1 = 0 ; v2 =  ; v3 =  ; v4 =  ; v5 = t ; v6 = t ; v7 = t and the partial T

n

i

i

i

k

x

y

derivatives de ned by :

k

i

i

i

k

k

z

x

k

k

y

z

@ eI @ eI @ eI @ eI = g ; = g ; = g ; =g 4 1 2 3 @v1 @v2 @v3 @v4 @ eI @ eI @ eI = [1 ; 0; 0] ; = [0 ; 1; 0] ; = [0; 0; 1] @v5 @v6 @v7 is the j column of G . We can set the partial dierentials of E 2 to i

i

i;

i

where g zero and obtain : i;j

T

n

I

=1 @v

i

i

i;

i

th

X  @e 

i

i;

T

i;

i

T

i

T

i

j

W

I

i

7 @e X k

=1 @v

I i

k

v = k

X  @e  n

I

=1 @v

i

T

i

j

W z ; (1  j  7) I

(17)

i

i



which gives 7 equations in the 7 unknowns. Once  has been determined the new estimate of the quaternion de ning the rotation and scaling is given by q[ +1] = 

 p q [ ] + . The iterative scheme is deemed to have converged when  = s < . We have used  = 10?6 . r

r

4.2 Solving for b given Q

Here, we substitute equation(2) into equation(7) to give : E2 =

X n

=1

(xI ? Q(x + P b)) WI (xI ? Q(x + P b)) T

i

i

i

i

i

i

i

i

which can be written as : E2 =

X n

=1

(xI ? sRx ? t ? sRP b) WI (xI ? sRx ? t ? sRP b) T

i

i

i

i

i

i

i

i

We minimise as before, by setting the partial derivatives to zero :

X @E 2 = 2 @b =1 n

j

i



 @b ?sRP @b W (x ? sRx ? t ? sRP b) = 0; (1  j  t) T

I

i

j

i

I i

i

i

Hill,Cootes,Taylor(8)

British Machine Vision Conference

This expression is rearranged to give : @b @b

T

X?

@b X P R W 1s (x ? sRx ? t) P R W RP
b = @b

n

=1

j

i

T i

n

T

I

T

i

i

T i

=1

j

I

T

I

i

i

i

i

We can use the relationship WI = R WM R to simplify the previous equation :   X? M
X P W P b = P WM 1 R (xI ? t) ? x T

i

i

n

n

T i

=1

i

i

T i

=1

i

i

T

s

i

i

i

This equation can be expressed simply as : ?P WM P
b = P WM ?Q?1(xI ) ? x
(18) where WM = diagfWM g. For scalar weights, WM = w , WM simpli es to a diagonal matrix and equation(18) represents the same solution as that used in [3]. T

T

i

i

i

5 A Closed Solution for the 2D Problem

The non-linear relationship between fQ,bg in equation(7) can be handled using the 2-stage scheme described above. To obtain a closed solution we linearise the expression by measuring the errors in the model frame rather than the image frame. In section 6 we will see that, for a real image interpretation problem, the increased errors resulting from measuring the errors in the model frame are statistically insigni cant. We measure the errors in the model frame as follows : eM = Q?1(xI ) ? (x + P b) (19) The error term we wish to minimise becomes : i

E2 =

X? n

=1


Q?1 (xI ) ? (x + P b)

T

X 1

i

i

i

n

=1

I

n

d c

=1

i

i

i

X  c ?d  i

=

I

M

x

I i

i

T



M

i

1

i

?P b W i

i



R (x ? t) ? x ? P b s

T

+ t0 ? x

i

i

i

R (x ? t) ? x ? P b W s T

i

W ?Q?1(x ) ? (x + P b)




i

=

i

i

M

i

I

T

i

i

 c ?d  d c

x

I i

(20)

+ t0 ? x

i

i

?P b



i

where c = cos(?)=s, d = sin(?)=s, t0 = ?R t=s. Equation(20) can be written as : T

E2 =

0 +4 X @X @e n

t

1 0 +4 X ? x A W @ @@ve T

M

t

M v @v =1 =1 =1 where v1 = c; v2 = d; v3 = t0 ; v4 = t0 ; v +4 = b (1 i

j

i

j

j

i

i

j

x

y

derivatives in equation(21) are given by : @ eM @ eM I = x ; = @v1 @v2 i

i

i

 ?x  @ e ; I

i;y

xI

i;x

j

 

j

M i

j

v

j

1 ? x A

 j  t). The partial

 

1 ; @ eM = 0 ; @ eM = ?p = 0 @v3 1 @v +4 @v3 M i

i

(21)

i

i

j

i;j

(22)

Hill,Cootes,Taylor(9)

British Machine Vision Conference

8

Average Error of Landmarks

7 6

Unweighted

5 4 3 2

Directional Weights

1 0 0

5

10

15

20

25

30

Number of Iterations

Figure 3: Improved Search with Directional Constraints where p is the j column of P . To minimise E 2 in equation(21) we set the partial derivatives to zero, @E 2 =@v = 0; (1  k  t + 4), and obtain : th

i;j

X  @e  n

M

T

i

=1 @v

i

k

i

k

W

+4 @ e X t

M

i

j

M i

=1 @v

j

v = j

X  @e  n

M i

=1 @v

i

k

T

W x (1  k  t + 4) M

i

i

(23)

which form (t + 4) equations in the (t + 4) unknowns. In the special case of unit weights, WM = I, we note that the system of equations(23) takes the form : i

 B B  q   y  c; d; t0 ; t0  1 2 ; q = = B2 I b y

T

Q

T

which we can solve as :

x

b

y

(24)

?B ? B B
q = y ? B y 2 1 2 2 T

Q

b = y ? B2 q b

b

T

6 Results To see how using directional constraints improves the performance of the ASM we used a set of images and accompanying PDM of lumbar vertebrae [5] - see gure 3. We selected a subset of the training images on which to test the method, the correct labelling of the landmarks being known for each image. The ASM was applied to each of the 25 selected images 8 times. The initial set of shape parameters was set to b = 0 (i.e. the mean shape). The initial pose was a random perturbation of the pose required to align the mean shape with the known landmarks for the given image. At each iteration of the ASM the average distance of each landmark from its annotated position was computed for all landmarks and all images. The consolidated results are shown in gure 3 for the ASM both with and without

Hill,Cootes,Taylor(10)

British Machine Vision Conference

directional weights. For these experiments we used  = 1; = 0:09 8 i. It can be seen clearly that directional weighting improves both the rate of convergence and the accuracy of the nal interpretation. In order to test how the modi cation of the error term used in section 5 to achieve a closed form solution for fQ,bg aects the interpretation generated by the ASM we have performed a similar set of experiments. In this case the ASM using the standard 2-stage iterative algorithm for the errors de ned in the image frame was compared with the ASM using the closed solution for errors de ned in the model frame. For the vertebra location problem described above we found the increased errors resulting from calculating the errors in the model frame were statistically insigni cant. i

i

7 Conclusions We have presented a detailed and extended analysis of the shape approximation problem associated with the ASM algorithm. We have shown how to establish error terms relevant to both the image and model frames for the general case of symmetric weight matrices. This extension is necessary to support a new approach to the shape approximation problem which uses directional constraints to allow the points more freedom of movement in speci ed directions during ASM search. We have presented iterative solutions for the most general case in both 2D and 3D. We have shown how directional weighting results in an ASM which requires fewer image iterations than an un-weighted ASM and also generates more accurate interpretations. We have also shown that it is possible to obtain a closed solution to the shape approximation problem in 2D which, for a real image interpretation problem, generates interpretations which are statistically no less accurate than those generated by the corresponding 2-stage iterative solution.

References

[1] A. Blake, R. Curwen, and A. Zisserman. A framework for saptiotemporal control in the tracking of visual contours. International Journal of Computer Vision, 11(2):127{ 145, 1993. [2] T. F. Cootes, D. H. Cooper, C. J. Taylor, and J. Graham. A trainable method of parametric shape description. Image and Vision Computing, 10(5):289{294, June 1992. [3] T. F. Cootes, A. Hill, C. J. Taylor, and J. Haslam. The use of active shape models for locating structures in medical images. Image and Vision Computing, 12(6):276{285, July 1994. [4] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Active shape models - their training and application. Computer Vision and Image Understanding, 61(1):38{59, Jan. 1995. [5] A. Hill, T. F. Cootes, C. J. Taylor, and K. Lindley. Medical image interpretation: A generic approach using deformable templates. Journal of Medical Informatics, 19(1):47{59, 1994. [6] B. K. P. Horn. Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, 4(4):629{642, Apr. 1987.