Sobolev Active Contours - CiteSeerX

Sobolev Active Contours Ganesh Sundaramoorthi

∗

Anthony Yezzi

Andrea Mennucci

†

‡

December 31, 2005

Abstract All previous geometric active contour models that have been formulated as gradient flows of various energies use the same L2 -type inner product to define the notion of gradient. Recent work has shown that this inner product induces a pathological Riemannian metric on the space of smooth curves. However, there are also undesirable features associated with the gradient flows that this inner product induces. In this paper, we reformulate the generic geometric active contour model by redefining the notion of gradient in accordance with Sobolev-type inner products. We call the resulting flows Sobolev active contours. Sobolev metrics induce favorable regularity properties in their gradient flows. In addition, Sobolev active contours favor global translations, but are not restricted to such motions; moreover, they gradually move from coarse to finer-scale motions. These properties are particularly useful in tracking applications. We demonstrate the general methodology by reformulating some standard edge-based and region-based active contour models as Sobolev active contours and show the substantial improvements gained in segmentation and tracking applications.

Contents 1 Introduction 1.1 Background on Active Contours . . . . . . . . . . . . . . . . . . . 1.2 Problems with Gradient Flows for Active Contours . . . . . . . . 2 General Theory 2.1 Structure on the Space of Curves . . . . . 2.2 Inner Products on Tc M . . . . . . . . . . 2.3 Gradient of Functionals on M . . . . . . . ñ . . . . . . . 2.4 Relation Between H n and H 2.5 Sobolev Inner Products in Fourier Domain ∗ School

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

of Electrical Engineering, Georgia Institute of Technology, Atlanta, USA of Electrical Engineering, Georgia Institute of Technology, Atlanta, USA ‡ Scuola Normale Superiore, Pisa, Italy † School

1

. . . . .

. . . . .

2 2 4 6 6 7 8 10 11

1

INTRODUCTION

3 H1 3.1 3.2 3.3 3.4

˜ 1 Gradients and H Solving the ODEs . . . . . . . Properties of the Kernels . . . Properties of Sobolev Gradients ˜ 1 over H 1 . . . Advantages of H

2

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

14 14 15 16 16

4 Sobolev Gradients in Frequency Domain 17 4.1 Coarse-to-Fine Motion of Sobolev Contours . . . . . . . . . . . . 18 5 Some Sobolev Gradient Flows 5.1 Length and Weighted Length . . . . . . 5.1.1 Variation of the Standard Model 5.2 Area and Weighted Area . . . . . . . . . 5.3 Elastic Energy . . . . . . . . . . . . . . ˜1 . . . . . 5.4 Comparison of H 0 and H 1 , H

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

19 19 21 22 23 25

6 Numerical Implementation 25 6.1 Multiple Curves Evolution . . . . . . . . . . . . . . . . . . . . . . 25 6.2 Level Set Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 6.3 Level Set Method Without Polygon Extraction . . . . . . . . . . 31 7 Experiments 7.1 Merging and Splitting . . . . . . . . . . . 7.2 Noisy Square Segmentation . . . . . . . . 7.3 Variation of Standard Edge-Based Energy 7.4 Segmentation of Real Images . . . . . . . 7.5 Tracking: Simple Image Sequences . . . . 8 Conclusion

1 1.1

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

33 33 33 34 36 36 40

Introduction Background on Active Contours

Active contours, introduced by Kass et al. [1], have been widely used for the segmentation problem. In their work, a PDE, derived by minimizing an energy, is used to drive an initial contour toward the edges of the desired object in an image. The energy is a combination of a potential energy, which is low when the curve is aligned along the edges, and internal regularity terms that are low when the curve is smooth. Many modifications have been proposed to the original parametric model (see for example, [2], and [3] for a survey). One undesirable feature of Kass’s original model and some subsequent works is that the energy functional used to derive the PDE flow is not invariant with respect to different parameterizations of the same curve; thus, the flow depends on how the curve is parameterized.

1

INTRODUCTION

3

A geometric PDE flow, which is a flow not dependent on curve parameterization, was considered by [4, 5]. The flows that the authors considered have an edge stopping and smoothing terms similar to [1], but the flows are geometric. In these papers, a level set technique, introduced by [6], is used to deal with topological changes; in addition, the level set method provides several other benefits over a particle based implementation. Thereafter, [7, 8] considered minimizing a geometric energy, which is a generalization of Euclidean arclength, defined on curves. The authors derived a gradient descent flow in order to minimize the geometric. A common feature of the works mentioned above from the work of [1] is that all are edge-based ; that is, the evolution of the curve is driven by an edge detector of the image. The main disadvantages of the edge-based approaches is that they are limited by the performance of the edge-detector, which is dependent on differential operators of the image and hence sensitive to noise, and the initial contour of these methods typically needs to be initialized close to the desired object. In [9], the authors introduce a conformal area integral energy based on the edge-map in an attempt to have greater flexibility of the initial contour and better handle concavities. Another method alleviating problems of the original edge-based formulations for parametric active contours was proposed in [10] by deriving a more global force from the edge-map that attracts the contour from locations far away from the desired object. In contrast to the edge-based approaches for active contours, region-based energies [11, 12, 13, 14, 15] for active contours have provided many desirable features, for example, less sensitivity to noise, better ability to capture concavities of objects, more dependence on global features of the image, and less sensitivity to the initial contour placement. In these approaches, an energy is designed to be minimized when the curve partitions the image into statistically distinct regions. In [16, 17], the authors introduce and rigorously study a region-based energy that is designed to both extract the boundary of distinct regions while also smoothing the image within these regions. Subsequently, [18, 19] gave a curve evolution implementation of minimizing the energy functional considered by [16, 17] in a level set framework [6]. Another work that makes use of a region-based energy is [20, 21], which also combines edge-based information. More recent works that use active contours for segmentation are not only based on information from the image to be segmented (edge-based or regionbased), but also a-priori information known about the shape of the desired object to be segmented. The work of [22] showed how to incorporate prior information into the active contour paradigm. Subsequently, there have been a number of works, for example [23, 24, 25, 26, 27], which design energy functionals that incorporate prior shape information of the desired object. In these works, the main idea is designing a prior term of the energy that is small when the curve is close, in some sense, to a pre-specified shape. The need for this type of prior term arose from several factors including the fact that some images contain limited information, the energies functions considered previously could not incorporate complex information, the energies had too many extraneous local minima, and the gradient flows to minimize these energies allowed for arbitrary deformations that gave rise to unlikely shapes.

1

INTRODUCTION

1.2

4

Problems with Gradient Flows for Active Contours

The works on shape priors have led to a fundamental question on how to define a distance between two curves or shapes. Many works, for example [28, 29, 30, 31, 32, 33], in the shape analysis literature have proposed different ways of define this distance. However, in [33, 34], the authors observe that all previous works on geometric active contours that derive gradient flows to minimize energies, which were described in the previous section, imply a natural notion of distance induced by a Riemannian structure. Indeed, the notion of gradient of such energies relies on an inner product defined the set of perturbations of a curve. All of these previous works on geometric active contours use the same geometric L2 -type inner product, which we refer to as H 0 , to define a gradient. However, [32, 35] have shown that the Riemannian metric on the space of curves induced by the H 0 inner product is pathological. In addition to the pathologies of the Riemannian structure induced by H 0 , there are also undesirable features of H 0 gradient flows as we will demonstrate in the paper. Some of these features are listed below. First, there are no regularity terms in the definition of the H 0 inner product. That is, there is nothing in the definition of H 0 that discourages flows that are not smooth in the space of curves. By smooth in the spaces of curves, we mean that the surface formed by plotting the evolving curve as a function of time is smooth. Thus, when energies are designed to depend on the image that is to be segmented, as in the case for the geometric active contours mentioned above, the H 0 gradient is very sensitive to noise in the image. As a result, the curve becomes unsmooth instantly. Therefore, in geometric active contours models, a penalty on the curve’s length is added to keep the curve smooth in addition to keeping the variational problem well-posed. However, this changes the energy that is being optimized. Moreover, as we shall demonstrate, the length penalty is sometimes insufficient, and leads to other problems. Second, H 0 gradients, evaluated at a particular point on the curve, depend locally on derivatives of the curve. Therefore, as the curve becomes unsmooth, as mentioned above, the derivative estimates become inaccurate, and thus, the curve evolution becomes inaccurate. Moreover, for regionbased and edge-based active contours, the H 0 gradient at a particular point on the curve depends locally on image data at the particular point. Although region-based energies may depend on global statistics, such as mean values of regions, the H 0 gradient still depends on local image data. These facts imply that the H 0 is sensitive to noise and local features. Third, the H 0 norm gives non-preferential treatment to arbitrary deformations regardless of whether the deformations are global low order motions such as translations, rotations, and scales or more local diffeomorphisms. This implies, as we shall show, that the H 0 -gradient of image dependent energies “encourages” points on the evolving curve to move “independently”

1

INTRODUCTION

5

to decrease energy rather than encouraging the points to move more collectively. This restricts the gradient at a particular point from “seeing” information located at other points of the curve. Therefore, the curve evolving according to the H 0 gradient flow is suspticle to local minima of the energy. Fourth, all geometric active contours require that the unit normal to the evolving curve be defined. As such, the evolution does not make sense for polygons. Moreover, since in general, a H 0 active contour does not remain smooth, one needs viscosity theory to define the flow. Fifth, if the energy depends on n derivatives of the curve, then the H 0 gradient has 2n derivatives of the curve. Since the corresponding level set flows with higher than two derivatives are not known to have a maximum principle, level set methods [6] cannot be used. This forces one to use particle methods to implement the flow. However, flows with higher than two derivatives are generally difficult to implement because of numerical artifacts. Lastly, there are cases when the gradient descent or ascent of an energy in the H 0 -metric is ill-posed; and therefore the flow is unstable. In some of these cases, this ill-posedness is simply a property of the metric and not inherent to the energy. As a specific example, the H 0 gradient ascent for arclength, i.e., backward heat flow, is not stable. This is quite odd in an intuitive manner because there is nothing in the definition of length itself that indicates that a flow to increase length should be unstable. In this paper, we consider using inner products arising from Sobolev spaces to define gradients. Unlike the research done in active contours for the past 20 years, which has focused on finding various new energies to deal with local minima and other problems, we present a new way of doing active contours by giving an alternative way of minimizing any energy. As we shall see, this new approach to active contours gives a remedy to many of the undesirable properties with the current approach for active contours. Moreover, any existing active contour method can get the added benefits of this new approach with minimal changes to its existing implementation and with little extra computational time. A preliminary version of this paper appeared in [36]. We have learned of a concurrent, independent work [37] with similar ideas as ours. The authors of [37] also consider an H 1 inner product, and go on to consider other “coherent” motions resulting from various inner products. They consider inner products that favor tranlations, rotations, and affine motions. The procedure they consider amounts to projecting the usual H 0 gradient onto to a subspace of these motions. The disadvantage of this approach, like the work of [29], is that one must specify the group to be favored. This also presents a pratical problem, namely, that groups can be used beyond the affine group are difficult to define. As we shall see, Sobolev active contours address this problem since there is an automatic and continuous transition through a countably infinite number of

2

GENERAL THEORY

6

group motions, starting from coarse scale motions to increasingly finer scale motions in the curve evolutions. In this paper, we also give a more careful analysis of the Sobolev active contour in comparison with [37].

2

General Theory

2.1

Structure on the Space of Curves

Let M denote the set of smooth immersed curves in Rd , where d ≥ 2. We show how to make M a differentiable manifold. Let us give a precise definition of M : M := {c ∈ C ∞ (S 1 , Rd ) : c0 (θ) 6= 0 ∀θ ∈ S 1 }, where S 1 denotes the circle. Denote by F the Fréchet space C ∞ (S 1 , Rd ) of smooth vector fields on S 1 whose topology is defined by the following seminorms: pn (h) = sup |h(n) (θ)|,

n = 0, 1, . . .

θ∈S 1

where h ∈ F . We now show Proposition 2.1. M is a differentiable manifold whose model space is the infinite dimensional space, F . Proof. 1. We first define charts from M to F . Let c ∈ M , then denote by ˜c ⊂ F → Uc ⊂ M a chart at c ∈ M defined by Φ−1 Φ−1 :U c c (h) = c + h. ˜c , and show that c + h ∈ M where We must define the neighborhood U ˜c . Choose h∈U 0 < < inf1 |c0 (θ)|. θ∈S

Note the choice of is possible since |c0 (θ)| 6= 0 (c ∈ M ), and the infimum cannot be zero since |c0 | is continuous on the compact set S 1 . Now define ˜c := {h ∈ F : p1 (h) < }, which by definition of F is open. It is now U ˜c . It is also clear that Φc is a bijection. clear that c + h ∈ M for h ∈ U 2. We now show that the charts defined above are compatible. Let c, c˜ ∈ M , c 6= c˜ and Uc ∩ Uc˜ 6= ∅. First note that Φc (Uc ∩ Uc˜) = {h ∈ F : p1 (h) < c , p1 (h − (˜ c − c)) < c˜}, which is clearly open. Next, we see that Φc˜ ◦ Φ−1 c : Φc (Uc ∩ Uc˜) → Φc˜(Uc ∩ Uc˜) is given by (Φc˜ ◦ Φ−1 ˜) + h, c )(h) = (c − c which is clearly a C ∞ diffeomorphism. Remark 2.1. Note, we may consider M = {c ∈ C 1 (S 1 , Rd ) : c0 (θ) 6= 0 ∀θ ∈ S 1 } and F to be the Banach space C 1 (S 1 , Rd ) with topology defined by the norm khk = supθ∈S 1 (|h(θ)| + |h0 (θ)|). In this case, M is also a differentiable manifold.

2

GENERAL THEORY

7

Remark 2.2. Note that in our definition of the manifold M , the space of immersed curves, a curve with two different parameterizations (i.e., c1 , c2 ∈ M such that c1 (θ) = c2 (φ(θ)) for θ ∈ S 1 and φ 6= idS 1 is a smooth automorphism of S 1 ) are considered two distinct elements of M . Therefore, one may consider the quotient space M/Diff(S 1 ) where Diff(S 1 ) is the set of smooth automorphisms of S 1 . This space, however, is not a manifold [32]; but a slightly smaller space Immf of free immersed curves is a manifold. In our work, we will use M . The results we obtain, however, are geometric in that they do not depend on a particular parameterization of a curve. For c ∈ M , we denote by Tc M the tangent space of M at c. By definition, Tc M = {[C] : C : (−δ, δ) → M, C(0) = c} where δ > 0, and C˜ ∈ [C] iff d d ˜ = , ∀f ∈ Cc∞ (M, R). f (C(t)) f (C(t)) dt dt t=0 t=0 Notice that for any C : (−δ, δ) → M with C(0) = c, we may write C(t) = c + th + o(t) for some h ∈ F and so we see that d d f (C(t)) = D(f ◦ Φ−1 )(Φ (C(t)) ◦ Φ (C(t)) = D(f ◦ Φ−1 c c c c )(h) ◦ h, dt dt t=0 t=0

which means that [C] depends only on h and c. Therefore, we see that Tc M ∼ = C ∞ (S 1 , Rd ).

2.2

Inner Products on Tc M

We now define inner products on Tc M . Definition 2.1. Let c ∈ M , L be the length of c, and h, k ∈ Tc M . Let λ > 0, and n ∈ N. We assume h and k are parameterized by the arclength parameter of c. Z 1 L h(s) · k(s) ds 1. hh, kiH 0 := L 0 E D 2. hh, kiH n := hh, kiH 0 + λL2n h(n) , k (n) 0 D

H

3. hh, kiH˜ n := avg(h) · avg(k) + λL2n h(n) , k (n)

where avg(h) :=

1 L

Z

L

E

H0

h(s) ds, and the derivatives are with respect to arclength. 0

It is easy to verify that the above definitions are inner products. Note that we have introduced length dependent scale factors so that the above inner products (and the corresponding norms) are independent of curve rescaling; moreover they add some convenience in later computations.

2

GENERAL THEORY

8

Remark 2.3. The most general scale invariant definition of H n is hh, kiH n = hh, kiH 0 +

n X j=1

E D λj L2j h(j) , k (j)

H0

where λj ≥ 0 for j = 1, . . . , n − 1, and λn > 0. We will see in a later section that it is unnecessary for our purposes to use this general definition. ˜ n are possible [35]. We illustrate Remark 2.4. Other definitions of H n and H 1 the ideas with H . First, we will need to consider M with another model space. This leads to the tangent space Tc M/ ∼ where if h, k ∈ Tc M then h ∼ k iff h · N = k · N where N is the inward normal of c. The other definitions are as follows: Let α, β : S 1 → R be the normal components of two perturbations h, k (i.e., α = h · N and β = k · N ) then Z 1 α(s)β(s) + λL2 α0 (s)β 0 (s) ds (2.1) h[h], [k]iH 1 ∗ := L c Z 1 h[h], [k]iH 1 + := α(s)β(s) + λL2 (α(s)N (s))0 · (β(s)N (s))0 ds. (2.2) L c

The inner product in (2.2) can be simplified as follows Z 1 h[h], [k]iH 1 + = (1 + λL2 κ2 (s))α(s)β(s) + λL2 α0 (s)β 0 (s) ds. L c

(2.3)

Ignoring the α0 β 0 term and the length factors, (2.3) becomes the same inner product considered in [32] for shape analysis. One of the important consequences of defining an inner product on Tc M is that, with some other smoothness and compatibility conditions, it produces a Riemannian structure on M . This means that specifying an inner product on Tc M induces a distance between points on M . The authors in [32, 35] consider the Riemannian structure induced from the H 0 inner product and show that it is pathological; they also propose alternative inner products to achieve a nontrivial Riemannian distance. We do not explore this idea further in this paper; however, we consider the Riemannian structure induced from the Sobolev inner products in a future paper.

2.3

Gradient of Functionals on M

We now define the notion of gradient of a functional E : M → R. Definition 2.2. Let E : M → R.

1. If c ∈ M and h ∈ Tc M , then the variation of E in the direction h is d dE(c) · h = , E(c + th) dt t=0

where (c + th)(θ) := c(θ) + th(θ) and θ ∈ S 1 . Note dE(c) ∈ Tc∗ M is a linear operator on Tc M , dE(c) is called the differential at c.

2

GENERAL THEORY

9

2. Assume h, ic is an inner product on Tc M . The gradient of E is a vector field ∇E(c) ∈ Tc M that satisfies dE(c) · h = hh, ∇E(c)ic for all h ∈ Tc M . We now give an intuitive interpretation of the gradient, which tells us the significance of this perturbation. We show that the gradient is the most efficient perturbation; that is, the gradient maximizes the change in energy per “cost” of perturbing the curve. Proposition 2.2. Let k · kc be the norm induced from the inner product h·, ·ic on Tc M . Suppose dE(c) 6= 0, and the gradient with respect to h·, ·ic exists; then the problem sup {h∈Tc M,khkc =1}

dE(c) · h =

sup {k∈Tc M,k6=0}

dE(c) · k kkkc

(2.4)

has a unique solution, k = ∇E(c) ∈ Tc M, h = k/kkk. Proof. Note dE(c) · h = h∇E(c), hic ≤ k∇E(c)kc khkc by Cauchy-Schwartz; moreover, dE(c) · (∇E(c)) = k∇E(c)k2c . If the gradient of E exists, then by the proposition above, we have that h = ∇E(c) attains the supremum on the right hand side of (2.4). Note for λ → +∞, translations have negligible norm w.r.t. other directions in the tangent space, that is, if h is a translation (i.e. it is constant w.r.t. s) and k is not a translation, then limλ→+∞ khk/kkk = 0 , both when k·k is the norm induced by ˜ n . In light of the interpretation of the gradient as H n and when it comes from H the perturbation that attains the supremum in (2.4), it follows that translations ˜ n when λ is large (if they reduce energy). are favored for gradients in H n and H Remark 2.5 (Comment on H n for n ≥ 2). Translations are favored for H n ˜ n gradients when λ → +∞. This can be quite important for tracking and H applications where the object to be tracked is usually translating. One may ˜ n for wonder whether using higher order Sobolev inner products, H n and H n ≥ 2, will favor higher order polynomial motions of degree n. Note however, that any polynomial perturbation defined on S 1 , the circle, must be constant to conform to periodic boundary conditions. Thus, higher than order n = 1 Sobolev gradients also favor just translations. In this sense, there is not an advantage, with respect to gaining higher dimensional prefered motions, in using higher order Sobolev gradients. However, one gains added regularity of the gradient flow in using higher order Sobolev gradients. Remark 2.6. For the inner product on Tc M/ ∼ defined in (2.1), it follows that dilations (i.e., perturbations of the form h = ±N ) are favored for gradients for λ large.

2

GENERAL THEORY

2.4

10

ñ Relation Between H n and H

˜ n , i.e., We show that the norms associated with the inner products H n and H s Z L 1 n |h(s)|2 + λL2n−1 |h(n) (s)|2 ds (2.5) khkH = L 0 s Z L |avg(h)|2 + λL2n−1 khkH˜ n = |h(n) (s)|2 ds (2.6) 0

are equivalent. ˜ n ). The norms defined by (2.5) Proposition 2.3 (Equivalence on H n and H and (2.6) on Tc M are topologically equivalent; that is, αkhkH˜ n ≤ khkH n ≤ βkhkH˜ n for all h ∈ Tc M , and α, β > 0 not a function of h ∈ Tc M . Proof. We first derive a simple Poincaré inequality. For convenience, we will replace h : S 1 → Rd with its periodic extension h : R → Rd . We have Z L Z u 0 h0 (s) ds, h(u) − h(0) = h (s) ds = − 0

u

since h is periodic. Now, h(u) − h(0) =

1 2

avg(h) − h(0) =

1 2L

|avg(h) − h(0)| ≤

1 2L

=

1 2L

Z

Z Z Z

u 0 L

0 L 0 L 0

0

h (s) ds − Z Z Z

u

L

0

h (s) ds u

0

h (s) ds −

0 u

Z

0

|h (s)| ds +

0 L 0

Z

0

|h (s)| ds

!

L

!

⇒

0

h (s) ds u

Z

L u

!

0

|h (s)| ds

du =

1 2

Z

L 0

du ⇒

!

|h0 (s)| ds

so that by replacing 0 with an arbitrary point, we have Z 1 L 0 sup |h(u) − avg(h)| ≤ |h (s)| ds. 2 0 u Now, using (2.7) and Hölder’s inequality, we see s Z L √ L sup |h(u) − avg(h)| |h(s) − avg(h)|2 ds ≤ 0

u

√ Z L 0 L |h (s)| ds ≤ 0 s Z L ≤ L |h0 (s)|2 ds, 0

du

(2.7)

2

GENERAL THEORY

11

which is the Poincaré inequality. We now prove the equivalence of the two norms. By Hölder’s inequality, we have that Z 1 L |h(s)|2 ds |avg(h)|2 ≤ L 0

so that khkH˜ 1 ≤ khkH 1 . On the other hand, note that 1 L

Z

L 0

1 |h(s) − avg(h)| ds = L 2

Z

L 0

|h(s)|2 ds − |avg(h)|2 ,

so that khk2H 1

Z

L

1 |h(s)|2 + λL|h0 (s)|2 ds 0 L Z L Z 1 L = λL|h0 (s)|2 ds + |avg(h)|2 |h(s) − avg(h)|2 ds + L 0 0 Z L |h0 (s)|2 ds ≤ ((1 + λ)/λ)khk2H˜ 1 ≤ |avg(h)|2 + L(1 + λ)

=

0

One can iterate Poincaré’s inequality to demonstrate the equivalence of H n and ˜ n. H Note that we have not established any relation between the geometry of the ˜ n ; however, in the next sections, we show that the inner products H n and H ˜ n have similar properties. gradients with respect to H n and H

2.5

Sobolev Inner Products in Fourier Domain

Notice that since any h ∈ Tc M is smooth on S 1 , it follows that h ∈ L2 (S 1 ). Thus, we may write h as a Fourier series, i.e., X 2πi b h(l) exp h(s) = ls (2.8) L l∈Z

with convergence in L2 (S 1 ) (and in fact pointwise since h is smooth) where b h ∈ `2 (Z) is defined by 1 b h(l) = L

Z

L

0

2πi ls ds. h(s) exp − L

(2.9)

It should be noted that (2.8) decomposes the perturbation into the orthonormal basis of exponentials. This allows us to write Definition 2.1 in the frequency domain. Note that X 2πi n 2πi (n) b h (s) = l h(l) exp ls . L L l∈Z

2

GENERAL THEORY

By Parseval’s theorem, Z

12

L 0

h(s) · k(s) ds = L

X l∈Z

b k(l). h(l) · b

where · denotes complex conjugation. We also have that Z L X 2π 2n b k(l); h(n) (s) · k (n) (s) ds = L h(l) · b l L 0 l∈Z

therefore,

Proposition 2.4. If h, k ∈ Tc M , L is the length of c, and b h, b k : Z → C are defined by (2.9). Then, hh, kiH n =

X l∈Z

k(l) (1 + λ(2πl)2n )b h(l) · b

hh, kiH˜ n = b h(0) · b k(0) +

X l∈Z

k(l). λ(2πl)2nb h(l) · b

and the corresponding norms are X khk2H n = (1 + λ(2πl)2n )|b h(l)|2 l∈Z

khk2H˜ n = |b h(0)|2 +

X l∈Z

λ(2πl)2n |b h(l)|2 .

(2.10) (2.11)

(2.12) (2.13)

˜ n inner products Notice that Proposition 2.4 allows us to define the H n and H for n that is any real number greater than 0. These inner products are defined the same way as in (2.10) and (2.11). It is easy to verify in this case too, the definitions are indeed inner products. For example, we may define the inner ˜ 12 for which we comment on the usefulness to computer vision briefly product H in a later section. Unfortunately for n that is not an integer, the inner products (therefore, norms) are not local, that is, they cannot be written as integrals of derivatives of the curves; but, given r ∈ R+ we can represent them, for n integer n > r + 1/4, as Z Z h(n) (s)K(s − s˜)k (n) (˜ s) ds d˜ s hh, kiH˜ r = c

c

for a kernel K with Fourier coefficients ( 1 b K(l) = λ(2πl)2r−2n

if l = 0 if l = 6 0.

The norms shown in (2.12) and (2.13) measure the perturbation magnitude in terms of its Fourier coefficients, which are the weights of its corresponding

2

GENERAL THEORY

13

˜ n norms, high frequency frequency components. We see that for both H n and H components of the perturbation contribute increasingly to the norm of the perturbation. Indeed, the norm of a frequency component increases with frequency, and the growth rate of the weights of the frequency coefficients for both H n and ˜ n are the same. We now give a simple proof of Proposition 2.3. H Proposition 2.5. The norms defined by (2.12) and (2.13) on Tc M are equivalent; that is, αkhkH˜ n ≤ khkH n ≤ βkhkH˜ n for all h ∈ Tc M , and α, β > 0 not a function of h ∈ Tc M . Proof. Clearly, khkH˜ n ≤ khkH n . We also have that 1 1 + λ(2πl)2n ≤ 1 + λ(2πl)2n λ(2π)2n for |l| ≥ 1. Thus, khkH n ≤

s

1+

1 khkH˜ n . λ(2π)2n

Therefore, k · kH n and k · kH˜ n induce the same the topologies on Tc M . Based on Proposition 2.4, we define the following new inner products Definition 2.3. Let m ≥ 1, define X X b h(l) · b k(l) + λ (2πl)2nb h(l) · b k(l); hh, kiH˜ n,m =

(2.14)

|l|>m

|l|≤m

the corresponding norm is

khk2H˜ n,m =

X

|l|≤m

|b h(l)|2 + λ

X

|l|>m

(2πl)2n |b h(l)|2 .

(2.15)

We use Proposition 2.2 to discuss the behavior of the gradient as λ → +∞. For the norms k · kH n and k · kH˜ n , translations τ are favored for gradients since kτ kH 1 and kτ kH˜ 1 are close to zero when compared to other motions. Let us consider the behavior for k · kH˜ 1,m . For this, define the following finite dimensional subspaces of Tc M    X 2πi l· : al ∈ C, a−l = al . (2.16) Gm = al exp   L |l|≤m

Then it is clear that motions in this group are favored as λ → +∞ when using the norm k · kH˜ n,m .

3

˜ 1 GRADIENTS H 1 AND H

3

14

˜ 1 Gradients H 1 and H

In this section, we describe how to compute first order Sobolev gradients from the H 0 gradient. Denote by f = ∇H 0 E(c) the gradient of E with respect to the H 0 inner product at c. We would like to compute first the H 1 gradient at c. Assuming g = ∇H 1 E(c) exists, we have for all h ∈ Tc M ,

dE(c) · h = hh, giH 0 + λL2 hh0 , g 0 iH 0 = h, g − λL2 g 00 H 0

where we have integrated by parts and noted that we have periodic boundary conditions. Since gradients are unique (if they exist), we must have that f (s) = g(s) − λL2 g 00 (s) where s ∈ [0, L].

(3.1)

Note that this is an ODE defined on [0, L] with periodic boundary conditions, that is, all derivatives match on the boundary of [0, L]. ˜ 1 gradient. Assuming Now we take a similar approach to compute the H g = ∇H˜ 1 E(c) exists, we have

dE(c) · h = avg(h) · avg(g) + λL2 hh0 , g 0 iH 0 = h, avg(g) − λL2 g 00 H 0 .

Again by uniqueness, we have that f = avg(g) − λL2 g 00 . Noting periodic boundary conditions, we have that avg(g) = avg(f ), and so f (s) = avg(f ) − λL2 g 00 (s) where s ∈ [0, L]

(3.2)

and we have periodic boundary conditions.

3.1

Solving the ODEs

We want to first solve the ODE (3.1) for g. It suffices to solve (3.1) with the boundary conditions g(0) = g(L) and g 0 (0) = g 0 (L). One can show that RL g(s) = 0 kλ (s, sˆ)f (ˆ s) dˆ s, where kλ : [0, L] × [0, L] → R satisfies the following conditions for all s, sˆ ∈ (0, L) ∂ 2 kλ (s, sˆ) = δ(s − sˆ) ∂s2 kλ (0, sˆ) = kλ (L, sˆ); ∂s kλ (0, sˆ) = ∂s kλ (L, sˆ) kλ (s, sˆ) − λL2

(3.3a) (3.3b)

and δ denotes the Dirac distribution. It can be shown that the solution to the previous system is kλ (s, sˆ) = Kλ (|s − sˆ|), where Kλ : R → R is given by L s− 2 cosh √λL , for s ∈ [0, L], Kλ (s) = (3.4) √ 2L λ sinh 2√1 λ

and Kλ is periodically extended to all of R. We may write Z s − s)∇H 0 E(ˆ s) dˆ s = (Kλ ∗ ∇H 0 E)(s) ∇H 1 E(s) = Kλ (ˆ c

(3.5)

3


15

where the integral over C denotes any range of sˆ that corresponds to one full period around the curve C (e.g. [0,L], [–L,0], [–L/2,L/2], etc.). We now solve the second ODE (3.2). It suffices to solve (3.2) with the boundary conditions g(0) = g(L), g 0 (0) = g 0 (L), and the relation avg(f ) = avg(g). Integrating (3.2) twice yields Z s 1 g(s) = g(0) + sg 0 (0) − (s − sˆ)(f (ˆ s) − avg(f )) dˆ s. (3.6) λL2 0 Using (3.6), applying the boundary conditions, and noting that avg(g) = avg(f ), after some manipulation, yields g 0 (0) = −

1 λL3

Z

L 0

s(f (s) − avg(f )) ds and

g(0) =

Z

L

˜ λ (s) ds (3.7) f (s)K

0

˜ λ is given by where the kernel function K (s/L)2 − (s/L) + 1/6 1+ 2λ

˜ λ (s) = 1 K L

!

,

s ∈ [0, L].

(3.8)

˜ λ (0) = K ˜ λ (L) and thus we may periodically extend K ˜ λ as before. Note that K R ˜ In this case, we may rewrite, g(0) = c f (ˆ s)Kλ (ˆ s) dˆ s, where, again, the integral over c denotes any range of sˆ that corresponds to one full period over c. Now if we shift the arclength parameterization of the curve, we obtain a convolution formula for g at any point s. Therefore, Z ˜ λ (ˆ ˜ λ ∗ ∇H 0 E)(s). ∇H˜ 1 E(s) = K s − s)∇H 0 E(ˆ s) dˆ s = (K (3.9) c

3.2

Properties of the Kernels

˜ λ: Note the following formal properties of Kλ and K 1 1 1 00 ˜ (K − δ(s)) and K (s) = − δ(s) , Kλ00 (s) = λ λ λL2 λL2 L

s ∈ [0, L).

(3.10) The first property is just the relation in (3.3a), and the second is obtained ˜ λ . Using these relations, it is easy to see that Kλ ∗ f through differentiation of K ˜ and Kλ ∗ f formally solve (3.1) and (3.2), respectively. Next, note that Z Z ˜ λ (ˆ Kλ (ˆ s) dˆ s = 1 and K s) dˆ s=1 (3.11) c

c

˜ λ ≥ 0 only for all λ > 0. Also observe that Kλ ≥ 0 for all λ > 0, and that K when λ ≥ 1/24. Finally, it is easy to verify that as λ → +∞, Kλ → 1/L and ˜ λ → 1/L. See Fig. 3.1 for plots of Kλ and K ˜ λ. K


3 6

16 10

λ=0.01 λ=0.04 λ=0.08

5

λ=0.01 λ=0.04 λ=0.08

8 6

4

4 3 2 2

0

1

0 −0.5

−2

0

0.5

−4 −0.5

0

0.5

˜ λ (right) for various λ with L = 1. The Figure 3.1: Plots of Kλ (left) and K plots show the kernels over one period.

3.3

Properties of Sobolev Gradients

˜ 1 gradients are First note, from formulas (3.5) and (3.9), that the H 1 and H geometric, i.e., they do not depend on a particular parameterization chosen for the curve. This is also evident from the definition of these inner products. The formulas (3.5) and (3.9) show that there may be a tangential component of the gradients; but these tangential components may be ignored when considering gradient flows. This is different from H 0 where if the energy is geometric, then the gradient will have only a normal component. ˜ 1 gradients are given by integrals of H 0 , given in formulas Because H 1 and H (3.5) and (3.9), integration by parts and the relations in (3.10) imply that two derivatives of the curve can be moved to derivatives on the kernels. This means ˜ 1 gradients involve two fewer derivatives of the curve than H 0 that H 1 and H gradients involve. Note that H 0 gradients have twice the number of derivatives of the curve as is defined in the energy E to be optimized. Thus, fourth order evolution equations of curves in H 0 may reduce to second order equations in ˜ 1 . A similar remark can be made for H n and H ˜ n gradients; these H 1 and H gradients require 2n less derivatives of the curve than the H 0 gradient requires. The property that the integral of both the kernels is unity (3.11) implies that the H 1 gradient can be interpreted as a weighted average of the H 0 gra˜ 1 when λ > 1/24. In light of this dient; the same interpretation holds for H weighted average interpretation, we see that Sobolev gradients are less sensitive to noise and local features than H 0 gradients are. Moreover, the property that the kernels approach 1/L as λ → +∞ shows that, in this case, the H 1 and ˜ 1 gradients approach pure translations equal to the average value of the H 0 H gradient, as expected from the interpretation of gradient noted in Section 2.1.

3.4

˜ 1 over H 1 Advantages of H

˜ 1 gradient as opposed to There is a computational advantage of using the H ˜ 1 gradient as a single the H 1 gradient since the formulas (3.6), (3.7) give the H

4

SOBOLEV GRADIENTS IN FREQUENCY DOMAIN

17

integral without convolution, as opposed to the necessary convolution for H 1 . ˜ 1 over H 1 is that we can eliminate the dependence Another advantage of H ˜ 1 gradient flows. Observe from the on the parameter λ when implementing H ˜ λ is a sum of two terms: one that depends on λ kernel definition (3.8) that K ˜ 1 gradient is a sum of two components: and another that does not. Thus, the H one that depends on λ by a simple scale factor, and another that is independent of λ. The component that does not depend on λ is avg(∇H 0 E), which is a just a translation. The other component is a complex deformation. An algorithm to ˜ 1 gradient flow is to first evolve the curve by the translation implement the H component until this component becomes zero, then to evolve the curve by the deformation component, and the process is repeated until convergence. Note that λ does not need to be chosen for evolving the deformation component because λ only changes the speed of the curve, not the geometry. Therefore, this algorithm also gives a way of separating the (rigid) motion of the curve from the deformation. Separating the motion from deformation has particular importance in tracking applications [29].

4

Sobolev Gradients in Frequency Domain

We now calculate Sobolev gradients of an arbitrary energy in the frequency domain. By Definition 2.2, if the H 0 and H n gradients of an energy E : M → R exist, then it follows that

dE(c) · h = ∇H 0 E(c), h H 0 = h∇H n E(c), hiH n for all h ∈ Tc M . Using Parseval’s Theorem, the last expression becomes X X b b \ \ (1 + λ(2πl)2n )∇ ∇ H n E(l) · h(l) = H 0 E(l) · h(l). l∈Z

l∈Z

Since the last expression holds for all h ∈ Tc M , we have \ ∇ H n E(l) =

\ ∇ H 0 E(l) 1 + λ(2πl)2n

for l ∈ Z

(4.1)

Using a similar argument, we see that \ \ ∇ ˜ n E(0) = ∇ H H 0 E(0) \ ∇ H 0 E(l) \ ∇ ˜ n E(l) = H λ(2πl)2n

(4.2) for l ∈ Z\{0},

(4.3)

and more generally, \ ∇\ ˜ n,m E(l) = ∇ H H 0 E(l) for |l| ≤ m \ ∇ H 0 E(l) ∇\ ˜ n,m E(l) = H λ(2πl)2n

for |l| > m.

(4.4) (4.5)

4

SOBOLEV GRADIENTS IN FREQUENCY DOMAIN

18

It is clear from the previous expressions that high frequency components of ∇H 0 E are less pronounced in the various forms of the H n gradients, with higher order Sobolev gradients damping high frequency components with faster decay rates. We should also remark that these expressions give a decomposition of the gradients onto an orthogonal basis of simple motions starting from translations to higher order trigonometric motions. We see that with the Sobolev gradients, these high order motions do not contribute to the motion of the curve as much as the H 0 gradient. In fact, these high order motions decay at a much quicker \ rate than in H 0 . It should be noted that ∇ H 0 E(l) decays to zero as |l| → +∞, \ but can there can be an arbitrary large frequency component of ∇ H 0 E. The Sobolev gradients’ frequency components decay rate at a much faster rate than H 0 , and larger the frequency, the higher the killing factor.

4.1

Coarse-to-Fine Motion of Sobolev Contours

We now discuss the implications of the analysis of Sobolev active contours in the Fourier domain. We note that the Fourier basis of the perturbations of a curve decomposes Tc M from global perturbations (low frequency perturbations) to increasingly more local perturbations (high frequency perturbations). Indeed the zero frequency perturbation is a simple translation of the curve, which is completely global. See Figure 4.1. Therefore, by (4.1), and comments in the previous section, it is apparent that Sobolev gradients yield perturbations with more pronounced global components than the standard H 0 gradient. While H 0 gradients give equal weighting across all scales, Sobolev gradients give less weight to finer scales. However, this does not mean that very local (fine scale) deformations of the curve are restricted from Sobolev gradient flows. It just means that if there exists a low order perturbation (a more global motion) that increases the given energy just as would a higher order perturbation (a more local motion), then the low order perturbation will be preferred in the Sobolev gradient, as shown by Proposition (2.2). Also, if no perturbations in Gm can increase the energy, E, that is dE(c) · h ≤ 0 for all h ∈ Gm , then by \ Definition 2.2, we must have that ∇ H n E(l) = 0 for l ≤ m, and therefore, we can write ( 0, |l| ≤ m 1 \ . ∇ ˜ n E(l) = 1 H \ 0 ∇ E(l), |l| >m λ(2π(m + 1))2n (2π(l−m−1))2n H Thus, we see that since the gradient flow does not geometrically depend on a scale factor, the Sobolev gradient automatically has the weights on high order perturbations of the gradient readjusted. This means the Sobolev gradient flow at this particular instant of the evolution changes the fine scale structure of the curve. Thus, with Sobolev active contours, we see a progression from coarse scale motion to finer scale motion, much more so than the standard H 0 active contour. Figure 4.2 shows the tracking of a noisy square image using both H 0 and H 1 active contours, which illustrates the ideas of the previous comments. Notice that with the H 0 active contour, the fine structure of the curve is changed

5

SOME SOBOLEV GRADIENT FLOWS

19

Figure 4.1: Increasingly higher frequency perturbations applied to a circle (left to right, l = 0, 2, 5, 10). immediately, while the H 1 active contour gradually changes finer scale features of the curve after changing coarse-scale features. We now discuss a possible application of using a fractional Sobolev inner ˜ 21 . This inner product may be useful because the cost (norm) of product, H ˜ 12 than H ˜ 1 . One reason to weight frequency components is weighted less in H high frequency perturbations less is because they may arise from noise, which ˜ 21 punishes these components highly are very high frequency components. H ˜ 1 . But when we are segmenting an object or shape that has bends, as does H and our initial curve is a coarse representation of the object, we want high ˜ 12 frequency components in the perturbation to capture the bends. Using H ˜ 1 , while still not reacting to may lead to capturing these bends faster than H noise. We should note that any Sobolev gradient will capture these bends if that ˜ 12 . Moreover, is energetically favorable, but it will be faster numerically with H we still have the coarse to fine behavior, as discussed in the previous paragraph, ˜ 12 gradient. Another approach to capturing bends and fine details of for the H an object faster is to adaptively decrease the parameter λ with the H n gradient. Notice that as λ → 0, we have that ∇H n E → ∇H 0 E. Thus, one can start with λ high, corresponding to a more global motion, then as the contour moves more slowly, the parameter λ is lowered so that the motion is more similar to the H 0 flow.

5

Some Sobolev Gradient Flows

In this section, we simplify the formulas (3.5) and (3.9) for some common geometric energies, note some interesting properties, and compare these with the usual H 0 gradients. In what follows, we use K to denote either the kernel (3.4) ˜ 1 gradient; when the distinction or (3.8), and ∇1 will denote either the H 1 or H ˜ 1. is needed, we will use the subscript λ on the kernels, and write H 1 or H

5.1

Length and Weighted Length

We consider the geodesic active contour model [8, 7]. The energy is Z E(c) = φ(c(s)) ds c

5


20

(a)

(b)

Figure 4.2: (a) Standard (H 0 ) active contour alters fine structure of the curve immediately. (b) Sobolev (H 1 ) active contour moves from coarse to finer scale motions. Both use same energy.

5


21

where φ : R2 → R+ . Then the gradient with respect to H 0 is ∇H 0 E = L(∇φ · N )N − LφκN where N is the unit inward normal, and κ is the curvature. Let us first note that ∇H 0 E = L∇φ − L(φc0 )0 . Integrating by parts we find that 1 ∇1 E = ∇φ ∗ K − (φc0 )0 ∗ K = ∇φ ∗ K − (φs c) ∗ K 0 − (φc) ∗ K 00 , L where φs (ˆ s) := d/ dˆ sφ(c(ˆ s)). Using the relations in (3.10), we find that ∇H˜ 1 E =

φc − avg(φc) ˜ λ0 + L∇φ ∗ K ˜ λ. − L(φs c) ∗ K λL

(5.1)

Of particular interest is when φ = 1, that is E = L, the length of the curve. We see that c − avg(c) . ∇H˜ 1 L = λL ˜ 1 gradient flows are stable for It is interesting to notice that the H 1 and H 0 both ascent and descent while the H gradient flow is only stable for descent. ˜ 1 gradient flow constitutes a simple rescaling of the curve about Note that the H ˜ 1 gradiits centroid. While the H 0 gradient descent smooths the curve, the H ent descent (or ascent) has neither a beneficial nor a detrimental effect on the regularity of the curve. 5.1.1

Variation of the Standard Model

We now consider a variation of the geodesic active contour model [8, 7] that can be implemented with the use of Sobolev active contours, and cannot be implemented with the standard H 0 -active contour. Consider the energy Z 1 φ(c(s)) ds, (5.2) E(c) = L c which is simply the standard energy scaled by a length factor. In the model of [8, 7], φ is chosen to be high near edges of the image and low otherwise; for example, φ = (1+k∇(G∗I)k2 )−1 , where G is a Gaussian smoothing kernel. One of the properties of the gradient descent of the standard energy is that it has the tendency to shrink the length of the curve; indeed, the term φκN shrinks the length of the curve. Thus, where φ is constant, the traditional model shrinks the length of the curve. This preference of shrinking the curve is artificial, and has nothing to do with the edges of objects in the image. Indeed, there are many cases where this preference has a detrimental effect on the result of the segmentation; for example in segmenting concave objects (see the experiment in Section 7.3). Therefore, we consider the energy in (5.2), which is invariant under scale when φ is constant. Hence there is no preference to shrink the curve unless the edges of the object in the image require that the curve shrink.

5


22

We now compute the H 0 gradient. ∇H 0 L 1 E(c) + ∇H 0 (LE(c)) L L = EκN + (∇φ · N )N − φκN

∇H 0 E(c) = −

= (∇φ · N )N − (φ − E)κN .

(5.3)

Since φ − E is not strickly positive, the gradient descent is ill-posed, and the ˜ 1 gradient. corresponding flow is not stable. We now compute the H ∇H˜ 1 L 1 E(c) + ∇H˜ 1 (LE(c)) L L c − avg(c) φc − avg(φc) ˜ λ0 + ∇φ ∗ K ˜ λ .(5.4) + − (φs c) ∗ K = −E λL2 λL2

∇H˜ 1 E(c) = −

The corresponding flow is stable for both gradient descent and ascent. Thus, we see that the fact that the H 0 gradient descent is ill-posed has nothing to with the energy; it is a property of the H 0 metric.

5.2

Area and Weighted Area

We consider region-based active contour models; for example, [13, 15]. The energy is Z E(c) = φ dA, cin

where cin denotes the region enclosed by the closed curve c, φ : R2 → R and dA is the area form; this energy is defined only for embedded curves. The gradient with respect to H 0 is ∇H 0 E = −LφN = −LφJc0

and J is a rotation by 90o matrix. Integrating by parts we find that 1 ∇1 E = −(φJc0 ) ∗ K = (φs Jc) ∗ K + (φJc) ∗ K 0 . L

(5.5)

˜ 1 gradient, this simplifies to For the H ∇H˜1 E =

J λL2

Z

L 0

˜ λ. φc(· + sˆ) − avg(φc) sˆ dˆ s + (φs Jc) ∗ K

(5.6)

Of particular interest is when φ = 1, that is E = A, the area enclosed by the curve. We see that ∇1 A = (Jc) ∗ K 0 . This simplifies to the gradient ascent/descent Ct (s) = ± ˜ 1 gradient case. in the H

J λL2

Z

L 0

C(s + sˆ) − avg(C) sˆ dˆ s

(5.7)

5


5.3

23

Elastic Energy

Consider the scale invariant elastic energy defined by Z E(c) = L κ2 (s) ds, c

where L is the length of c, and κ is the signed curvature. This may serve as a regularizer that does not favor smaller length curves, unlike the standard curvature curve shortening term. Notice that if we consider the energy E/L, then this energy can be made arbitrarily small by a circle of appropriately large radius. Thus, we expect the gradient flow of E/L to move the curve to an increasingly large circle. ˜ 1 flow by first calculating the H 0 - gradient. Let us We know derive the H write a family of curves as C : R+ × S 1 → R2 , then we may write Z Css · Css ds. E(t) = L C

Let us first compute some intermediate formulas. Let f : R+ × S 1 → R, then ∂ ∂ f ∂t ∂s

∂ 1 ∂ f ∂t kCp k ∂p 1 ∂ ∂ Ctp · Cp ∂ f+ f = − kCp k3 ∂p kCp k ∂p ∂t ∂ ∂ ∂ f, = −Cts · Cs f + ∂s ∂s ∂t

=

or more conveniently, fst = fts − (Cts · Cs )fs .

(5.8)

We now compute

∂ (Css · Css ) = 2Csst · Css , ∂t but using (5.8) twice, we see Csst

= Csts − (Cts · Cs )Css ∂ (Cts − (Cts · Cs )Cs ) − (Cts · Cs )Css = ∂s = Ctss − (Ctss · Cs + Cts · Css )Cs − (Cts · Cs )Css − (Cts · Cs )Css = Ctss − (Ctss · Cs + Cts · Css )Cs − 2(Cts · Cs )Css ,

and since Cs · Css = 0, we have ∂ (Css · Css ) = 2(Ctss · Css ) − 4(Cts · Cs )(Css · Css ). ∂t

(5.9)

5


24

Now, E 0 (t) = =

d dt

Z L

1 0

E Lt + L L

Css · Css kCp k dp

Z

1 0

∂ (kCp k)Css · Css dp + L ∂t

Z

C

∂ (Css · Css ) ds. ∂t

By substituting (5.9), we have Z Z E Css ds + L (Cts · Cs )(Css · Css ) ds Ct · E 0 (t) = − L C ZC + 2L (Ctss · Css ) − 2(Cts · Cs )(Css · Css ) ds Z C Z E Ct · = − Css ds + L (−3(Cts · Cs ))(Css · Css ) ds L C ZC φ(Ctss · Css ) ds + 2L Z C E Ct · − Css ds = L C Z + L (2(Ctss · Css ) − 3(Cts · Cs )(Css · Css )) ds. C

Integrating by parts, we find Z 1 E 0 (t) = Ct · −ECss + 2L2 ∂ss (css ) + 3L2 ∂s ((css · css )cs ) ds. L C

Hence,

∇H 0 E(c) = −Ecss + 2L2 cssss + 3L2 ∂s ((css · css )cs ).

(5.10)

Computing the Sobolev gradient from (5.10), we have ∇H˜ 1 E

= K ∗ ∇H 0 E = −EK 00 ∗ c + 2L2 K 00 ∗ (φcss ) − 3L2 K 0 ∗ (φ(css · css )cs ) + L2 K ∗ ((css · css )∇φ).

˜ λ kernel we have Hence, for the K ∇H˜ 1 E = −

2 E (avg(c) − c) − κN − 3L2 K 0 ∗ (κ2 T ) λL2 λ

(5.11)

since avg(κN ) = 0. Note that from the computations above, we see that the gradient of the energy E/L, the scale varying elastic energy, is ∇H˜ 1 (E/L) = −

2 κN − 3LK 0 ∗ (κ2 T ). λL

(5.12)

6

NUMERICAL IMPLEMENTATION

5.4

25

˜1 Comparison of H 0 and H 1 , H

˜ 1 gradients We notice several advantages of the gradients flows for H 1 and H 0 as compared with H gradients. First note that both the expressions for edge˜1 based and region-based active contour gradients with respect to H 1 and H (5.1), (5.5) do not involve any derivatives of the curve. This is in contrast to H 0 , which requires two derivatives for geodesic active contours and one derivative for region-based active contours. Hence, the Sobolev flows are defined for polygons, without the use of viscosity theory. Note that the expression in (5.1) does not require any more derivatives of φ than the expression for H 0 does. This is not the case for (5.5), which requires a derivative of φ. However, since φs is contained within a convolution, the possible noise generated by φs is mitigated. Alternatively, the original expressions (3.5) and (3.9) may be used if a derivative of φ is not desired to be computed. Notice the expressions of Sobolev gradients for the elastic energy (5.11) only require two derivatives of the curve; this is in contrast to the H 0 gradient, which requires four derivatives of the curve. Since there is no maximum principle for fourth order equations, the H 0 gradient of elastic energy cannot be implemented using level set methods. Thus, a particle method must be used; however, this is prone to numerical problems.

6

Numerical Implementation

In this section, we describe the numerical scheme used to simulate Sobolev active contours. Sobolev active contours are naturally suited for a parametric or a marker particle based implementation. This is because computing Sobolev gradients requires computing an integral around the curve, which is straightforward to compute if one has an ordered set of sample points of the curve.

6.1

Multiple Curves Evolution

In this section, we consider evolving multiple curves according to gradient flows for the metrics defined in Section 2.2. We consider defining the flows for curves that undergo topological changes. Consider a family of embedded curves c = (ci ) in the plane and of displacements h = (hi ), where ci , hi : S 1 → Rd and i = 1 . . . N = N (c). We may define a metric on multiple curves h, ic using a choice of some metric h, ici that was defined for a single curve in Section 2.2. We define the inner product on multiple curves to be X hh, kic := λ(ci )hh, kici i

where λ(ci ) is a conformal factor. In the following, unless explicitly stated otherwise, we will choose λ(ci ) ≡ 1. Consider an energy, E, on multiple curves defined by N (c)

E(c) :=

X i=0

E(ci ),

6


26

Figure 6.1: Some illustrations of topological changes. then, the variation of E for several embedded curves is N (c)

dE(c) · h =

X i=0

h∇E(ci ), hi ici ,

for some given inner product. Thus, we consider evolving the curves according to the curve evolution ∂t ci = −∇E(ci ) i = 1, . . . , N (c).

(6.1)

Clearly, if the curves are moved according to the above equation, and the curves remain embedded (that is they do not self-intersect) then the evolution reduces the energy, E. We now consider the possibility of topological changes of c, that is merging and splitting of curves in c. Two ways of topological changes are pictorially shown in Figure 6.1. On the left, the curves develop kinks; on the right, they develop cusps. Note that other topological changes may be possible. We consider whether the evolution of (6.1) reduces the energy, E of the curve c, even when topology changes occur. First, we note a problem for actually defining the evolution in (6.1) for Sobolev metrics. Note that at a topological change of the curve, c, there is an ambiguity in defining the Sobolev inner product, and hence there is an ambiguity in defining the gradient and the curve evolution. For example, on the right side of Figure 6.1, at the topology change, there is an ambiguity of whether to treat c as a single curve or two separate curves. In the case of the H 0 -gradient, this ambiguity is meaningless since either way the curve is treated (as one or two curves), the H 0 gradient is the same in both cases as it does not depend on the curve globally as the Sobolev gradients do through a convolution as in (3.5) and (3.9). Note that one obtains a different Sobolev gradient when treating the topologically changing curve as one curve versus two curves. Thus, mathematically speaking, the Sobolev curve evolution is not well defined for curves which change topology; it is only well-defined for single immersed curves. However, as a numerical scheme using level set methods (as described in Section 6.2), the curve configuration is never at the topology

6


27

change, it is either in a configuration before the topology change or after the topology change due to discrete time steps. Thus, there is never a need to actually define the Sobolev gradient of the curve at a topological change. We note that there is a discontinuity in the flow for Sobolev metrics before and after topology change, but it is still possible for the numerical scheme to reduce the energy. As noted above, (6.1) will decrease E if the evolution preserves topology; thus, we need to only consider whether there may be an increase in energy just before and just after a topological change. If there is continuity1 of the energy through the topology change, then the scheme (6.1) reduces the energy. Otherwise, there may be a possibility that the evolution does not reduce the energy. In this case, there is a possibility that the curve c oscillates between previous states of the curve, which is undesirable. The weighted length and area energies defined in Section 5 are an example of energies that satisfy this continuity condition across a topology change. Indeed on right-hand side of Figure 6.1, these energies remain constant for the curve just before and just after the topology change. On the otherhand, energies that depend on local derivatives of the curve such as the elastic energies defined in Section 5 may not be continuous through the topology change. Both the elastic energy and the scale-invariant elastic energy have different energy values just before and after the topology change on the right side of Figure 6.1. Therefore, it is not guaranteed that the numerical scheme that implements (6.1) will reduce the elastic energies if a topology change occurs. Also, there is possibility of oscillations if a topology change occurs. Note that these elastic energies not only pose a problem for the Sobolev metrics, but they also pose a problem for the standard H 0 metric. As noted, the energy is not continuous across topology changes, and therefore, the scheme of (6.1) may not reduce energy. Moreover, if the H 0 evolution is converted to a level set evolution, the equation is fourth order, and has no maximum principle. Thus, the zero level set evolution may not be modeling the true evolution of the curve. Despite our numerical approach described in the previous paragraph that reduces the energy of certain energies for multiple curves, there is still no mathematical theory to address this situation. Indeed it is quite clear that when curves do split or merge, a singularity occurs in the metric.

6.2

Level Set Method

We first describe a straightforward numerical method to extend curve evolutions that depend on integrals around the curve to level set evolutions, introduced in [6]. These ideas are seen in [38, 39, 40], where the energies considered are double integrals around the curve, and thus the curve evolutions are single integrals around the curve. The algorithm for updating the level set function, Ψ : R2 × R → R, for most Sobolev active contours is 1 Lower semicontinuity is enough when considering the continuous evolution. Also, we are being quite informal, but intuitive, when we talk of continuiuty or lower-semicontinuity since we do not specify a topology on the space of multiple curves.

6


{Ψ = 0}

+

+

+

+

+

+

+

−

−

−

−

−

−

+

+

+

+

−

−

−

−

−

−

+

+

−

−

−

+

+

+

+

PSfrag replacements

28

−

−

−

−

+

+

+

+

+

+ −

+

−

+

−

−

+

+

+

−

Figure 6.2: Zero level set of a some level set function along with the vertices of the polygon estimate. 1. Compute polygon(s) estimate of zero level set of Ψ. 2. Calculate and interpolate H 0 -gradient to polygon(s) estimate 3. Compute Sobolev gradient on polygon(s) estimate by using one of the ˜ 1 -gradients in terms of the H 0 -gradient shown formulas for the H 1 or H in Section 3.1. 4. Extend polygon(s) forces to narrowband band of level set function domain. 5. Evolve Ψ by the transport equation Ψt = −∇Ψ · F~ , where F~ : Un ⊂ R2 → R is the extended Sobolev gradient to the narrowband region, Un . Note for some energies (such as the elastic energy) the H 0 -gradient will not need to be computed, only some intermediate expressions. We now describe the details of the implementing the first and part of the fourth step of the algorithm above. The idea is that the polygon estimate of Ψ will have vertices that match with the points of the zero level set of Ψ that intersect the grid that is produced by the discretization of the domain of Ψ. Figure 6.2 shows a diagram representing a level set function along with the vertices of the polygon estimate (black dots). The algorithm is similar to marching cubes [41], but much simpler. The procedure is to march through the points of the narrowband, which is defined to be the set Un = {(i, j) ∈ Z × Z : sgn(Ψ(i, j)) 6= sgn(Ψ(i + 1, j)) or sgn(Ψ(i, j)) 6= sgn(Ψ(i − 1, j)) or sgn(Ψ(i, j)) 6= sgn(Ψ(i, j + 1)) or sgn(Ψ(i, j)) 6= sgn(Ψ(i, j − 1))}, where sgn(x) = + if x ≥ 0 and sgn(x) = − if x < 0. The marching through Un is done in such a way so that the vertices of the polygon(s) are obtained

6


29

for ( q ∈ Un ) { if (!visited[q] && validPixel(q)) { ps.addPolygon(1); // create new polygon p=q; prevp= (Psi[p]>=0) !=(Psi[p+Y]>=0) ? p-X : ( (Psi[p]>=0)!=(Psi[p+X]>=0) ? p-Y : p+Y ); do { // find all vertices of the current polygon visited[p]=1; recordPt(prevp, p, nextp, polynumb); prevp=p; p=nextp; } while (p!=q); polynumb++; } } Figure 6.3: Algorithm that does polygon extraction from Ψ, and calculates locations of force extensions. in an ordered fashion, so that the result is an ordered set of vertices belonging to each of the polygons. We need the following data structures for an ordered extraction of the polygons: • visited - an N × N char array, which marks the visited points of U n • ps - a dynamic list of polygons, which are 2×MAX POINTS float arrays • polygonnumb - an N × N int array; polygonnumb[i,j] is a number representing the polygon of where the extended force F (i, j) is derived from. • vertexnumb - an N ×N int array; vertexnumb[i,j] is the vertex number of the appropriate polygon which the extended force F (i, j) is derived from. The algorithm is shown in Fig. 6.2. In the algorithm, the main procedure is recordPt, which takes in the previously visited point of Un , prevp, the current point, p of Un that is being visited, the current number of the polygon being processed, and returns nextp, the next point of Un that should be visited. A vertex is recorded in the structure ps with sub-pixel accuracy. The procedure also updates polygonnumb and vertexnumb at points of U n near p if the currently added vertex’s force should be extended to these points close to p. Fig. 4(a) shows all the possible configurations of a particular grid box in the grid of Ψ. Once the particular configuration of p is determined, then based on prevp, the procedure can determine a unique vertex to record and the next grid box to visit. An example is given in Fig. 4(b). Given the previous grid box visited prevp, the current grid box p and the current configuration, nextp

6


30

nextp

prevp

(a)

p

(b)

Figure 6.4: (a) Distinct configurations for the zero level set of Ψ passing through a particular grid box of the domain grid. The last configuration is by convention assumed since this configuration of the values of the level set function is ambiguous. is uniquely determined. The computational complexity of the entire algorithm for extracting the polygon(s), and computing the extensions is linear in the size of the narrowband, Un . In most of the cases of various energy functionals, the H 0 -gradient of the energy must be computed on the narrowband Un in the usual fashion; or in our case, we directly compute the H 0 -gradient to the polygon estimate of {Ψ = 0}. In other cases (such as for the elastic energy), the H 0 does not need to be computed, but only certain expressions that do not involve the convolutions. For example, in the elastic energy, the term κ2 T should be computed at every point of the polygon estimate. Note that in computing quantities such as the normal vector, and curvature of {Ψ = 0}, we compute them directly from Ψ using standard formulas and interpolate them to the polygon estimate. The Sobolev gradient is then computed using a convolution or simple integral with the formulas in Section 3.1. Note that we must compute separate convolutions or integrals on each polygon extracted from Ψ (or connected component of Ψ) in the fashion described in Section 6.1. We use a standard Riemann sum to compute the curve integrals, and the polygon estimate is used to determine the arclength measure, ds. After the Sobolev gradient is computed on the polygon estimate, it can be extended to Un with the use of the arrays polygonnumb and vertexnumb. We call the extended Sobolev gradient F~ . Note that we are extending the gradient along the curve so that ∇Ψ · ∇Fi = 0 where i = 1, 2 and F~ = (F1 , F2 ). That is, we assume that all the level sets of Ψ are moving by the same speed function, which means that the level sets of Ψ do not collide. Now the level set evolution can be computed by Ψt (x) = −∇Ψ(x) · F~ (x).

6


6.3

31

Level Set Method Without Polygon Extraction

Although the algorithm described in the previous section is both fast and accurate, we would like to eliminate the polygon extraction step. One reason for eliminating this step is that it becomes much more difficult for surfaces, although we have not considered Sobolev active surfaces in this paper. In this section, we describe an approach where we can directly compute (without polygon extraction) an approximation to the Sobolev gradient from the H 0 gradient that is defined on the narrowband, Un , of the level set function. Our approach is to approximate the convolution integrals (3.5) and (3.9) with region integrals evaluated within the narrowband Un . The arc-distance between points on the curve (i.e., s − sˆ) that is required by the formulas (3.5) and (3.9) can be approximated by the use of the eikonal equation [42]. Consider an embedded curve, c ∈ C 2 (S 1 , R2 ), and the function F : R2 → R+ defined as ( +∞ for x ∈ / {c} F (x) = 1 for x ∈ {c} where {c} = {c(θ) : θ ∈ S 1 }. Define u : R2 → R+ by u(c(s)) := inf

γ∈Γ

Z

1

F (γ(θ))kγ 0 (θ)k dθ,

0

where Γ := {γ : [0, 1] → R2 , γ(0) = c(ˆ s), γ(1) = c(s)}; then clearly, we see that u(c(s)) equals the minimum distance between the points c(s) and c(ˆ s) along the curve c. Because of the symmetry of the kernels in (3.4) and (3.8), it suffices to use the quantity u(c(s)) as a substitute for s − sˆ in the convolutions formulas (3.5) and (3.9). As in [42], we may solve for the viscosity solution of the eikonal equation k∇u(x)k = F (x), u(c(ˆ s)) = 0 (6.2) to obtain the desired solution of u at all points along the curve, c. As we will solve this equation numerically on a grid, we consider the following approximation to (6.2), k∇u(x)k = 1 + |Ψ(x)|/, u(c(ˆ s)) = 0 (6.3) where > 0 is chosen small enough, and Ψ is the level set function with Ψ(c(s)) = 0 for all s. Next, note the co-area formula for a Borel measurable function f : R2 → R, a Borel set A ⊂ R2 such that ∇Ψ(x) 6= 0 for x ∈ A, Z Z Z f (x) dH 1 (x) dt f (x)k∇Ψ(x)k dx = A

Ψ(A)

Ψ−1 (t)

where dH 1 is the one-dimensional Hausdorff measure, that is, arc-length measure. We use this formula to convert the contours integrals of interest to integrals

6


32

over the domain of the level set (or the narrowband region). Thus, we find by the co-area formula and the Lebesgue differentiation theorem, Z Z H(x)K(u(x))δa (Ψ(x))k∇Ψ(x)k dx H(c(s))K(ˆ s − s) ds ≈ A

c

where δa : R → R is a smooth approximation to the Dirac distribution, H : R2 → R is some function, u is the solution to (6.3), and A ⊂ R2 , such that {c} ⊂ Ao , the interior of A. According to the previous comments, we propose the following theorem: Theorem 6.1. Suppose that c : S 1 → R2 is a C 2 curve embedded in the plane, Ψ is zero only on the image of c, and ∇Ψ(c(θ)) 6= 0 for θ ∈ S 1 ; then, Z 1 lim |∇Ψ(x)|Z(|Ψ(x)|/) dx = length of the curve (6.4) →0 2 A

+ + where R ∞ Z : R → R is a positive continuous decreasing functions s.t. Z(0) = Z = 1 and lim x→∞ Z(x) = 0 (for example Z(x) = exp(−x)); the integral 0 on the left hand side is on a compact set A such that the curve is contained in Ao , the interior of A. Fix a continuous H : R2 → R, a kernel K : R+ → R, and suppose that limx→∞ K(x)/x = 1. Let > 0, and c(¯ s) be a point in the curve and let u : R2 → R be the viscosity solution of

|∇u(x)| − (1 + |Ψ(x)|/) = 0

then 1 lim →0 22

Z

,

2

A

u(c(ˆ s)) = 0

|∇Ψ(x)| H(x) K(u(x)) Z(|Ψ(x)|/ ) dx =

Z

c

(6.5)

H(c(s)) K(|s − sˆ|) ds

(6.6) where |s − sˆ| is the shortest arclength distance between c(s) and c(ˆ s) along c.

To solve the eikonal equation (6.5) numerically, we use the fast marching method [43]. We discretize on a N × N grid, then the integrals above are numerically substituted by a sum; if → 0 while N is kept constant, then clearly the numerical versions of (6.4) and (6.6) would converge to zero rather than to the desired result; so we propose the following numerical schemes for the approximation of (6.4) and (6.4): N 1 X |∇Ψ(xi,j )|Z(|Ψ(xi,j )|/) ≈ length of the curve 2N 2 i,j

(6.7)

where xi,j are points of the discretization of the domain of Ψ to an N × N grid, and → 0, N → +∞ and N → +∞. Similarly, we use the method by [43] to solve for a numerical approximation uN i,j to (6.5); then (6.6) becomes N 1 X 2 |∇Ψ(xi,j )| H(xi,j ) K(uN i,j ) Z(|Ψ(xi,j )|/ ) dx ≈ 22 N 2 i,j Z H(c(s)) K(|s − s¯|) ds. c

(6.8)

7

EXPERIMENTS

33

Figure 7.1: Segmentation of various shapes using a Sobolev active contour for a region-based energy. This illustrates the ability of Sobolev active contours to cope with topological changes: merging and splitting.

7

Experiments

In this section, we show some simulations of Sobolev active contours used for segmentation and tracking. In all the simulations done below, the results for ˜ 1 inner product. Using the H 1 the Sobolev active contours are done with the H inner product gives the same results as to what are shown.

7.1

Merging and Splitting

In Figure 7.1, we demonstrate the experimental evidence for the ideas presented in Section 6.1. In this experiment, we segment an image using a region-based energy [13, 15], and a Sobolev active contour. We see that the active contour can change topology to achieve the global minimum.

7.2

Noisy Square Segmentation

In Figure 7.2, we show the results of an experiment in which we segment a noisy square image with the region-based energy of [15], which we refer to as Er . In this experiment, we compare the results obtained from using the usual H 0 active contour with the result obtained from using a Sobolev active contour. Note to prevent the H 0 active contour from becoming stuck at a local minimum of the region-based energy (as in Figure 4.2), we add a regularization term to the evolution. The typical regularization term (see [15, 13] for example) is curvature flow (ct = κN ), which we use in this experiment. The noise that we added to the binary image is salt and pepper noise with densities of ρ = 0.5, 0.6, 0.7. First, we explore the effects of various weights on the curvature term that is added to the H 0 active contour. In Figure 7.2, we see that with using only a small weight on the curvature item, the contour becomes stuck in the noise, at an intermediate local minimum of Er + αL, where α is the weighting on the curvature term and L is the length of the curve. The image second from the right in each row is the result of the H 0 -active contour with a minimum weighting of curvature just high enough so that the contour is not stuck in noise at a local minimum. In Figure 2(a), we see that with relatively low noise, the result of the H 0 -active contour with just enough curvature weighting to overcome the local minimum captures the desired square accurately. As the noise increases, a higher curvature weighting is needed to overcome local minima; however, because of

7

EXPERIMENTS

34

such a high curvature weighting, the active contour is unable to capture the fine-scale structure of the desired object. In Figure 2(c), we see that using the minimal amount of curvature to overcome the noise results in the contour failing to capture the corners of the desired square object. Notice that for each of the noise levels, the Sobolev active contour without any curvature term moves in such a way as to avoid any local minimum of the region-based energy, as it moves from coarse to finer scale motions. The end result captures the square accurately and looks to be the global minimum of the region-based energy. The result is independent of λ, the weighting on the derivative component since the contour’s centroid does not need to move. Notice that as the converged Sobolev contour becomes more rugged as the noise increases. This is because the image becomes corrupted by the noise, and the original square is no longer the global minimum of energy. We should point out that the Sobolev active contour gives smoothness in the contour flow ; it does not guarantee smoothness in the final contour. The smoothness of the final contour will be determined by the energy that is being minimized. In the case of the H 0 -active contour, there is no smoothness in the flow; the curvature term provides some smoothness in the flow and provides smoothness of the final contour. Besides obtaining a more accurate segmentation, the Sobolev active contour converges much faster than the H 0 active contour with just enough weighting on the curvature term to get passed local minima. The reason is because of the small step size needed for stability of the curvature term in the level set implementation. A crude estimate for the Sobolev active contour shows that the step size is less than 0.5/2552 = 130 050−1 , whereas for the H 0 -active contour with a curvature penalty of 300 000, as in Figure 2(c), a step size of less than min{0.5/300 000 = 600 000−1 , 0.5/2552 = 130 050−1 } = 600 000−1 is needed. This is a factor of about 5 for this crude estimate.

7.3

Variation of Standard Edge-Based Energy

In this experiment, we illustrate the advantage of using the edge-based energy in (5.2), which is not preferential to shrinking the curve. Because the standard edge-based energy prefers to shrink the curve in the absence of edge information, the corresponding flow (either in the H 0 or the Sobolev metrics) is not suitable for capturing concavities, where it is desirable to increase the length of the curve. Since the average value of the standard geodesic energy is not preferential to shrinking nor increasing the length of the curve and it is solely influenced by the edge information, this model has a better ability to capture concavities. Figure 7.3 illustrates the advantage on a simple synthetic image. The H 0 active contour on the standard energy is stuck in a local minimum (note that the Sobolev active contour on the standard energy gives a similar result). The Sobolev contour minimizing (5.2) successfully captures the object boundary.

7

EXPERIMENTS

35

(a) ρ = 0.5. Left to right: Initialization, H 0 : κ = 20 000, 50 000, 90 000, Sobolev.

(b) ρ = 0.6. Left to right: Initialization, H 0 : κ = 50 000, 120 000, 160 000, Sobolev.

(c) ρ = 0.7. Left to right: Initialization, H 0 : κ = 50 000, 100 000, 300 000, Sobolev.

Figure 7.2: Segmentation of square binary image with salt and pepper noise of various densities. The experiment shows the results with H 0 (of various curvature weightings) and Sobolev active contours.

7

EXPERIMENTS

36

Figure 7.3: Segmentation of concave object. Standard H 0 geodesic active con˜ 1 (bottom). tour (top), and variation of geodesic energy using H

7.4

Segmentation of Real Images

We illustrate the advantages of using Sobolev active contours over H 0 with the same energy on real medical images in Figures 7.4 and 7.5. In Figure 7.4, we use an edge-based energy [8, 7] to segment a cardiac image. We show the results with two different initializations using both H 0 and Sobolev active contours. Because of the complicated texture of the image, the edge-based energy is riddled with local minima. Because the Sobolev active contour moves from the coarse scale motion to finer scale motions, it is less likely than the H 0 active contour to become stuck in local minima, as shown in Figure 7.4. In Figure 7.5, we segment an ultrasound image using a region-based energy [13, 15]. In Figure 5(a), the results are shown using an H 0 active contour with two different weightings on a curvature regularization term. In both cases, extraneous features of the image are detected. In Figure 5(b), we see that the contour moves first according to a translation and dilation (lower order motions), and finally detects fine scale structure of the desired object. Thus, the contour is able to skip over irrelevant local features.

7.5

Tracking: Simple Image Sequences

We illustrate some advantages of using Sobolev active contours rather than H 0 (with the same energy) in tracking applications with some simple image sequences. See [44, 45, 46] for classical works on tracking with parametric contours, and [47, 48, 49] for more recent work for geometric active contours. Tracking is usually a two-step process: prediction followed by detection. In the prediction step, a prediction of the object’s motion is made from the past history of the contour; this prediction is used to give an estimate of the contour in the next frame. In the detection step, the contour from the previous step is used to detect a more accurate representation of the object in the current frame. Some of the advantages are as follows: • Continuity of Flow: It is typically assumed that the object being tracked is moving continuously and deforming continuously. Otherwise a prediction of the contour does not make sense. Since the convolution formulas imply that the Sobolev active contour is a smooth flow, the Sobolev active

7

EXPERIMENTS

37

(a)

(b)

Figure 7.4: Segmentation of a cardiac image using an edge based energy. (a) Top row shows evolution using H 0 active contour; bottom shows evolution with a different initialization. (b) Same as (a) using Sobolev active contours.

7

EXPERIMENTS

38

(a)

(b)

Figure 7.5: Segmentation of an ultrasound image using region based energy. Top row shows the evolution using H 0 active contours with small regularization; the center row is H 0 with higher regularization; the bottom row shows the evolution using Sobolev active contours with no regularization item.

7

EXPERIMENTS

39

contour is ideal for tracking an object that is moving smoothly. Moreover, in many applications, it is assumed that the object moves first according to a global translation before slightly deforming [29]. The Sobolev active contour favors global translations, and hence it is suited for these tracking applications. • Coarse-to-Fine Motion: One reason for the prediction step is that the H 0 active contour is too sensitive to fine features of the image, and hence it becomes stuck in an intermediate local minima. The rationale behind the prediction step is that if the predicted contour is closer to the desired object, the contour will be less likely to become stuck in intermediate local minima. As the Fourier analysis showed, the Sobolev active contour moves from coarse to finer scale motions, and therefore is less likely to get stuck in local minima, and less dependent on the prediction step. ˜ n gradient is slightly • Speed of Convergence: While computing the H more computationally costly than computing the H 0 gradient, though both have the same complexity, we point out that without accurate prediction, the number of iterations in typical contour tracking applications required to update the active contour from frame to frame is usually much smaller with Sobolev active contours. Therefore the total computational time for processing between frames is significantly lower with Sobolev active contours. The reason is that the frame-to-frame motion of the object to be tracked is, as mentioned previously, usually dominated by more global motions: translations, scaling, and coarse scale deformations. Accordingly, a Sobolev active contour needs only a few iterations to lock onto the object in the next frame because the Sobolev gradient moves globally at first, preferring coarse scale motions in the first few iterations before proceeding to fine scale motions in later iterations. In contrast, standard H 0 active contours requires many more iterations since they immediately deform by local motions, significantly changing their initial shape (often to meaningless intermediate shapes), before deforming back to only slightly deformed, translated and scaled versions of their initial shape, and that is assuming they don’t first get trapped into intermediate local minima! We now illustrate the advantages discussed in the previous paragraph with simple synthetic image sequences (Figures 7.6 and 7.7) in which we employ the naive tracker. The prediction step in the naive tracker, assumes no motion so the result of the previous frame is used as the initialization for the current frame. We use the energy functional for geodesic active contours [8, 7] for the detection part of the tracking. Figure 7.6 shows tracking a square that translates using both H 0 active contours (in red) and Sobolev active contours (in blue). The segmentation result from the previous frame is used as the initial contour for the current frame being processed. The segmentation evolutions are run until convergence of both contours. Notice the H 0 active contour becomes stuck in a undesirable local minima after the initial movement of the object and soon lose

8

CONCLUSION

40

Figure 7.6: Left to right, top to bottom: Tracking a moving square in a noisy environment using edge-based H 0 active contour (red) and Sobolev active contour (blue). track of the object. The Sobolev active contour does not have this problem and it successfully tracks the object. Figure 7.7 shows the tracking of a simple concave object for both the H 0 ˜ 1 gradient flow. The flows are run until convergence in gradient flow and the H each frame. Note that the H 0 active contour deforms its initial shape greatly to react to local information. Hence the contour changes shape and must re-deform back to its initial shape. However, the contour gets trapped in an undesirable local minimum. The Sobolev active contour, on the other hand, only changes shape slightly while moving by an overall translation. This means that the number of iterations until convergence for the H 0 active contour is much greater than the Sobolev active contour, and therefore the computational time is also much greater. See Figure 7.8 for a simple quantitative analysis of the number of iterations and computational times. In this experiment, we segment the object shown in Figure 7.7 when the initial contour is a translated and a slightly deformed version of the object. We quantify the difference by using the set symmetric difference between the desired object and the region enclosed by the initial contour. From the graph in Figure 7.8, we see that the number of ˜ 1 active iterations and the computational time is significantly lower for the H contour.

8

Conclusion

We have introduced using Sobolev inner products on the set of perturbations of a curve rather than the traditional H 0 inner product used in all previous works on geometric active contours. We have demonstrated the general methodology for

8

CONCLUSION

41

(a)

(b)

Figure 7.7: Simple tracking using geodesic active contours: (a) Standard (H 0 ) active contour deforms the initialized contour greatly and is stuck in local minima, and (b) the Sobolev active contour moves in a global manner only slightly changing its shape. In each frame, the initial curve (given by the contour detected in the previous frame) is blue, the intermediate curve is green, and the final detected curve is red (see video concaveH0.mov and concaveH1.mov).

4

2.5

100

x 10

H0 H1

0

H H1

90 80

2

Running Time (seconds)

70

Iterations

1.5

1

60 50 40 30 20

0.5

10 0 2000

2500

3000

3500

4000

SSD

(a)

4500

5000

5500

6000

6500

0 2000

2500

3000

3500

4000

SSD

4500

5000

5500

6000

6500

(b)

Figure 7.8: (a) Graph showing number of iterations to converge versus set symmetric difference (SSD) of initial region and desired object. (b) Graph of running time versus SSD.

REFERENCES

42

computing Sobolev gradients, and derived various flows with respect to Sobolev inner products. We have shown that Sobolev flows are smooth in the space of curves, are not as dependent on local image information as H 0 flows, are global motions which deform locally after moving globally, and do not require derivatives of the curve to be defined for region-based and edge-based energies.

References [1] Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. International Journal of Computer Vision 1 (1987) 321–331 [2] Cohen, L.D.: On active contour models and ballons. Computer Vision, Graphics, and Image Processing: Image Processing 53 (1991) [3] Blake, A., Isard, M.: Active Contours. Springer Verlag (1998) [4] Caselles, V., Catte, F., Coll, T., Dibos, F.: A geometric model for edge detection. Num. Mathematik 66 (1993) 1–31 [5] Malladi, R., Sethian, J., Vemuri, B.: Shape modeling with front propagation: a level set approach. IEEE Transactions on Pattern Analysis and Machine Intelligence (1995) 158–175 [6] Osher, S., Sethian, J.: Fronts propagating with curvature-dependent speed: algorithms based on the Hamilton-Jacobi equations. J. Comp. Physics 79 (1988) 12–49 [7] Kichenassamy, S., Kumar, A., Olver, P., Tannenbaum, A., Yezzi, A.: Gradient flows and geometric active contour models. In: Proceedings of the IEEE Int. Conf. on Computer Vision. (1995) 810–815 [8] Caselles, V., Kimmel, R., Sapiro, G.: Geodesic active contours. In: Proceedings of the IEEE Int. Conf. on Computer Vision, Cambridge, MA, USA (1995) 694–699 [9] Siddiqi, K., Lauzière, Y.B., Tannenbaum, A., Zucker, S.: Area and length minimizing flows for shape segmentation. IEEE Transactions on Image Processing 3 (1998) 433–443 [10] Xu, C., Prince, J.L.: Snakes, shapes, and gradient vector flow. IEEE Transactions on Image Processing 7 (1998) 359–369 [11] Ronfard, R.: Region based strategies for active contour models. International Journal of Computer Vision 13 (1994) 229–251 [12] Zhu, S.C., Lee, T.S., Yuille, A.L.: Region competition: Unifying snakes, region growing, energy/bayes/MDL for multi-band image segmentation. In: ICCV. (1995) 416–

REFERENCES

43

[13] Yezzi, A., Tsai, A., Willsky, A.: A statistical approach to snakes for bimodal and trimodal imagery. In: Int. Conf. on Computer Vision. (1999) 898–903 [14] Yezzi, A., Tsai, A., Willsky, A.: Binary and ternary flows for image segmentation. In: IEEE International Conference on Image Processing. Volume 2. (1999) 1–5 [15] Chan, T., Vese, L.: Active contours without edges. IEEE Transactions on Image Processing 10 (2001) 266–277 [16] Mumford, D., Shah, J.: Boundary detection by minimizing functionals. In: Proc. IEEE Conf. Computer Vision Pattern Recognition. (1985) [17] Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math. 42 (1989) 577–685 [18] Tsai, A., Yezzi, A., Willsky, A.S.: Curve evolution implementation of the mumford-shah functional for image segmentation, denoising, interpolation, and magnification. IEEE Transactions on Image Processing 10 (2001) 1169–1186 [19] Vese, L.A., Chan, T.F.: A multiphase level set framework for image segmentation using the mumford and shah model. Int. J. Computer Vision 50 (2002) 271–293 [20] Paragios, N., Deriche, R.: Geodesic active regions: a new paradigm to deal with frame partition problems in computer vision. International Journal of Visual Communication and Image Representation, Special Issue on Partial Differential Equations in Image Processing, Computer Vision and Computer Graphics 13 (2002) 249–268 [21] Paragios, N., Deriche, R.: Geodesic active regions and level set methods for supervised texture segmentation. International Journal of Computer Vision 46 (2002) 223 [22] Leventon, M., Grimson, E., Faugeras, O.: Statistical shape influence in geodesic active contours. In: IEEE Conf. on Comp. Vision and Patt. Recog. Volume 1. (2000) 316–323 [23] Tsai, A., Yezzi, A.J., III, W.M.W., Tempany, C., Tucker, D., Fan, A., Grimson, W.E.L., Willsky, A.S.: Model-based curve evolution technique for image segmentation. In: CVPR (1). (2001) 463–468 [24] Rousson, M., Paragios, N.: Shape priors for level set representations. In: Proc. European Conf. Computer Vision. Volume 2. (2002) 78–93

REFERENCES

44

[25] Chen, Y., Tagare, H., Thiruvenkadam, S., Huang, F., Wilson, D., Gopinath, K., Briggs, R., Geiser, E.: Using prior shapes in geometric active contours in a variational framework. International Journal of Computer Vision 50 (2002) 315–328 [26] Cremers, D., Soatto, S.: A pseuso distance for shape priors in level set segmentation. In: IEEE Int. Workshop on Variational, Geometric and Level Set Methods. (2003) 169–176 [27] Raviv, T.R., Kiryati, N., Sochen, N.: Unlevel-set: Geometry and priorbased segmentation. In: Proc. European Conf. on Computer Vision. (2004) [28] Younes, L.: Computable elastic distances between shapes. SIAM J. Appl. Math. 58 (1998) 565–586 [29] Soatto, S., Yezzi, A.J.: DEFORMOTION: Deforming motion, shape average and the joint registration and segmentation of images. In: ECCV (3). (2002) 32–57 [30] Mio, W., Srivastava, A.: Elastic-string models for representation and analysis of planar shapes. In: CVPR (2). (2004) 10–15 [31] Charpiat, G., Faugeras, O.D., Keriven, R.: Approximations of shape metrics and application to shape warping and empirical shape statistics. Foundations of Computational Mathematics 5 (2005) 1–58 [32] Michor, P., Mumford, D.: Riemannian geometries on the space of plane curves. ESI Preprint 1425, arXiv:math.DG/0312384 (2003) [33] Yezzi, A., Mennucci, A.: Conformal riemannian metrics in the space of curves. In: MIA, EUSIPCO04 (2004) [34] Yezzi, A.J., Mennucci, A.: Conformal metrics and true ”gradient flows” for curves. In: ICCV. (2005) 913–919 [35] Yezzi, A., Mennucci, A.: Metrics in the space of curves. arXiv:math.DG /0412454 (2005)

Preprint,

[36] Sundaramoorthi, G., Yezzi, A., Mennucci, A.: Sobolev active contours. In: VLSM. (2005) 109–120 [37] Charpiat, G., Keriven, R., Pons, J., Faugeras, O.: Designing spatially coherent minimizing flows for variational problems baseed on active contours. In: ICCV. (2005) [38] Sundaramoorthi, G., Yezzi, A.J.: More-than-topology-preserving flows for active contours and polygons. In: ICCV. (2005) 1276–1283 [39] Sundaramoorthi, G., Yezzi, A.: Global geometric flows for topology preservation of active contours and polygons. Technical report, Georgia Institute of Technology, http://users.ece.gatech.edu/∼ayezzi (2005)

REFERENCES

45

[40] Rochery, M., Jermyn, I., Zerubia, J.: Higher order active contours and their application to the detection of line networks in satellite imagery. In: IEEE Workshop on VLSM. (2003) [41] Lorenson, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. ACM Computer Graphics 21 (1987) 163–170 [42] Rouy, E., Tourin, A.: A viscosity solutions approach to shape-fromshading. SIAM Journal on Numerical Analysis 29 (1992) 867–884 [43] Adalsteinsson, D., Sethian, J.: A fast level set method for propagating interfaces. J. of Comp. Phys. 118 (1995) 269–277 [44] Blake, A., Brockett, R.: On snakes and estimation theory. In: IEEE CDC. (1994) [45] Terzopoulos, D., Szeliski, R.: Tracking with Kalman Snakes. In: Active Vision. MIT Press (1992) [46] Isard, M., Blake, A.: Condensation – conditional density propagation for visual tracking. IJCV 1 (1998) 5–28 [47] Niethammer, M., Tannenbaum, A.: Dynamic geodesic snakes for visual tracking. In: CVPR. Volume 1. (2004) 660–667 [48] Jackson, J., Yezzi, A., Soatto, S.: Tracking deformable moving objects under severe occulsions. In: IEEE Conference on Decision and Control. (2004) [49] Rathi, Y., Vaswani, N., Tannenbaum, A., Yezzi, A.: Particle filtering for geometric active contours and application to tracking deforming objects. In: IEEE CVPR. (2005)