Discrete calculus optimization methods - and applications in ...

I - Standard graph-based methods II - Flow based methods III - Power watershed, a unifying framework IV - Semantic segmentation

Discrete calculus optimization methods and applications in computer vision

Camille Couprie IFP Energies Nouvelles Joint work with Laurent Najman, Hugues Talbot, Leo Grady, Clément Farabet, Yann Lecun, and Jean-Christophe Pesquet

13 Juin 2013

Camille Couprie, IFPEN

Discrete calculus optimization methods

1 / 57


Contexte de recherche x ∗ = arg min x∈Rn

X

R(xi , xj ) +

eij ∈E

| {z } Régularisation

X

D(xi , xj )

eij ∈E

{z } | Fidelité aux données

Segmentation d’images

→ ? Restauration d’images

Différentes fonctions de coût : → coupes minimales, → marcheur aléatoire, → variation totale, → critères de parcimonie, ...

→ ? Camille Couprie, IFPEN

Différentes méthodes d’optimisation Discrete calculus optimization methods

2 / 57


Contexte de recherche x ∗ = arg min x∈Rn

X

R(xi , xj ) +

eij ∈E

| {z } Régularisation

X

D(xi , xj )

eij ∈E

{z } | Fidelité aux données

Classification → ? Filtrage de maillages

Différentes fonctions de coût : → coupes minimales, → marcheur aléatoire, → variation totale,

→ ?


→ critères de parcimonie, ...

Différentes méthodes d’optimisation Discrete calculus optimization methods

2 / 57


Outline I - Standard graph-based methods II - Flow based methods 1 2

Segmentation : Combinatorial Continuous Maximum Flow Restoration : Dual constrained TV-based regularization

III - Unifying Framework 1 2 3 4

A new graph-based optimization framework Image segmentation Image filtering (nonconvex optimization) Surface reconstruction

IV - Semantic segmentation V - New problems



3 / 57


Outline I - Standard graph-based methods



3 / 57


1) Watershed method 2) Classical Max Flow / Graph cut method 3) Random Walker method

Some graph-based segmentation tools Watershed [Beucher-Lantuéjoul 1979, Vincent-Soille 1991] Advantages Fast



4 / 57



Some graph-based segmentation tools Watershed [Beucher-Lantuéjoul 1979, Vincent-Soille 1991] Advantages Fast Multilabel



4 / 57



Some graph-based segmentation tools Watershed [Beucher-Lantuéjoul 1979, Vincent-Soille 1991] Advantages Fast Multilabel Robust to markers size



4 / 57



Some graph-based segmentation tools Watershed [Beucher-Lantuéjoul 1979, Vincent-Soille 1991] Advantages Fast Multilabel Robust to markers size Drawbacks Leaking effect



4 / 57



Some graph-based segmentation tools Watershed [Beucher-Lantuéjoul 1979, Vincent-Soille 1991] Advantages Fast Multilabel Robust to markers size Drawbacks Leaking effect Non unique solution (difficult to get a non algorithmically dependent result) Camille Couprie, IFPEN


4 / 57



Watershed and Maximum Spanning Forest equivalence

MSF : set of trees spanning all nodes not connecting different seeds such that the total sum of their weights is maximum. If seeds are the maxima of the weight function, every MSF cut on the weight function is a watershed cut [Cousty et al 07, the drop of water principle]



4

3 F

1

3 1

4

4 3

2

2

B 3

2

5 / 57



Watershed and Maximum Spanning Forest equivalence

MSF : set of trees spanning all nodes not connecting different seeds such that the total sum of their weights is maximum. If seeds are the maxima of the weight function, every MSF cut on the weight function is a watershed cut [Cousty et al 07, the drop of water principle]



4

3 F

1

3 1

4

4 3

2

2

B 3

2

5 / 57



Some graph-based segmentation tools : Graph Cuts Graph cuts / Max flow Advantages Energy formulation → extends to a large class of problems

[Ford & Fulkerson 60s, Boykov-Joly 1998] S ∞ 3 11

1 1

4

4 3

4 3

22

22

3 ∞



2 T

6 / 57





1

4

4 3

4 3

2

2

3 ∞



2 T

6 / 57





3

1

1 3

0

∞



3

2 0

3

1 4

1

4

4

1 2

0

2

0

T

6 / 57




[Ford & Fulkerson 60s, Boykov-Joly 1998]

Robust to markers placement



6 / 57





Robust to markers placement Drawbacks Bias toward small contours



6 / 57





Robust to markers placement Drawbacks Bias toward small contours Block artifacts



6 / 57





Robust to markers placement Drawbacks Bias toward small contours Block artifacts Super-linear complexity



6 / 57





Robust to markers placement Drawbacks Bias toward small contours Block artifacts Super-linear complexity Limited to binary (2 labels) segmentation Camille Couprie, IFPEN


6 / 57



Some graph-based segmentation tools : Random Walker Combinatorial Dirichlet problem. Seeded segmentation [Grady 2006] Resolution of system of linear equations. Advantages 0.7 1 0.2

3

1

0


3 0.7

4

3

2

4


0.8

3

0.2

1 4 0.8 2

2

0.3

7 / 57



Some graph-based segmentation tools : Random Walker Combinatorial Dirichlet problem. Seeded segmentation [Grady 2006] Resolution of system of linear equations. Advantages Energy formulation → extends to a large class of problems

0.7 1 0.2

3

1

0


3 0.7

4

3

2

4


0.8

3

0.2

1 4 0.8 2

2

0.3

7 / 57



Some graph-based segmentation tools : Random Walker Combinatorial Dirichlet problem. Seeded segmentation [Grady 2006] Resolution of system of linear equations. Advantages Energy formulation → extends to a large class of problems No blocking artefacts

0.7 1 0.2

3

1

0


3 0.7

4

3

2

4


0.8

3

0.2

1 4 0.8 2

2

0.3

7 / 57



Some graph-based segmentation tools : Random Walker Combinatorial Dirichlet problem. Seeded segmentation [Grady 2006] Resolution of system of linear equations. Advantages Energy formulation → extends to a large class of problems No blocking artefacts Drawbacks Requires a more centered markers placement



7 / 57



Some graph-based segmentation tools : Random Walker Combinatorial Dirichlet problem. Seeded segmentation [Grady 2006] Resolution of system of linear equations. Advantages Energy formulation → extends to a large class of problems No blocking artefacts Drawbacks Requires a more centered markers placement



7 / 57



Some graph-based segmentation tools : Random Walker Combinatorial Dirichlet problem. Seeded segmentation [Grady 2006] Resolution of system of linear equations. Advantages Energy formulation → extends to a large class of problems No blocking artefacts Drawbacks Requires a more centered markers placement Super-linear complexity Camille Couprie, IFPEN


7 / 57



Outline I - Standard graph-based methods II - Flow based methods 1 2


III - Power watershed 1 2 3 4

A new graph-based optimization framework Image segmentation Image filtering (nonconvex optimization) Surface reconstruction




8 / 57



Outline

II - Flow based methods 1 2




8 / 57



Outline

II - Flow based methods 1 2




8 / 57


1) Flow constrained image segmentation 2) Flow constrained image restoration

Minimal surfaces



9 / 57



Motivation

In the continuum : Minimal cut (surface in 3D) is dual of continuous maximum flow [Strang 1983] In the classic discrete case min-cut (= “Graph cuts”)/ max flow duality but grid bias in the solution Recent trend : employ a spatially continuous maximum flow to produce solutions with no grid bias


Max Flow (Graph Cuts)

Continuous Max Flow [Appleton-Talbot 2006]


10 / 57



Motivation [Appleton-Talbot 2006, generalized by Unger-Pock-Bishof 2008] Fastest known continuous max-flow algorithm has no stopping criteria and no converge proof.

Our contribution : Combinatorial Continuous Maximum Flow a new discrete isotropic formulation avoids blockiness artifacts is proved to converge, is fast generalizes to arbitrary graphs [In SIAM Journal on Imaging Sciences, 2011] Camille Couprie, IFPEN


11 / 57



Combinatorial Continuous Maximum Flow (CCMF) Incidence matrix of a graph noted A Continuous MaxFlow max − → F

→ − F st

→ − s.t. ∇ · F = 0, → − || F || ≤ g.

MaxFlow, GraphCuts

Combinatorial formulation

max max F

s.t.

F

Fst AT F

s.t. AT F = 0,

= 0,

|AT |F 2

Fst

≤ g2

|F | ≤ g g defined on edges

g defined on nodes

CCMF : convex problem Resolution by an interior point method. Camille Couprie, IFPEN


12 / 57




→ − F st

→ − s.t. ∇ · F = 0, → − || F || ≤ g.

MaxFlow, GraphCuts

Combinatorial formulation

max max

Fst

F

s.t.

T

A F = 0, |AT |F 2 ≤ g 2

F

Fst

s.t. AT F = 0, |F | ≤ g g defined on edges

g defined on nodes



12 / 57



Discrete formulation on graphs - notations Graph of N vertices, M edges

A gradient operator Incidence matrix A ∈ RM×N e1 e A= 2 e3 e4 e4 Camille Couprie, IFPEN

p1 p2 p3 p4 −1 1 0 0 −1 0 1 0 0 −1 1 0 0 −1 0 1 0 0 −1 1

A> divergence operator allows general formulation of problems on arbitrary graphs


13 / 57




Fst

→ − s.t. ∇ · F = 0, → − || F || ≤ g.

Combinatorial formulation max F

s.t.

Fst AT F

MaxFlow, GraphCuts

max F

= 0,

|AT |F 2

≤

g2

Fst


g defined on nodes



14 / 57




Fst

→ − s.t. ∇ · F = 0, → − || F || ≤ g.


s.t.

Fst AT F

MaxFlow, GraphCuts

max F

= 0,

|AT |F 2

≤

g2

Fst


g defined on nodes



14 / 57




Fst

→ − s.t. ∇ · F = 0, → − || F || ≤ g.


s.t.

Fst AT F

MaxFlow, GraphCuts

max F

= 0,

|AT |F 2

≤

g2

Fst


g defined on nodes



14 / 57



Graph Cuts vs CCMF S

T

S

minimal cut on saturated edges

Scale of weight intensity :


T 1

...


minimal cut on saturated vertices

∞

15 / 57



CCMF dual problem The dual of the CCMF problem is 1 X (νi − νj )2 1 (νs − νt − 1)2 + + λi gi2 |{z} 4 λi + λj 4 λs + λt λ≥0,ν eij ∈E \{s,t} {z } | vi ∈V weighted cut | {z } source-sink smoothness term enforcement

min

X

Image with seeds Camille Couprie, IFPEN

λ

ν


Threshold of ν at .5 16 / 57



Minimal surfaces Catenoid test problem : source constituted by two full circles sink by the remaining boundary of the image, constant metric g

analytic minimal surface

CCMF result isosurface of ν

Root Mean Square Error between the surfaces : 0.75 (Appleton-Talbot error : 1.98) Camille Couprie, IFPEN


17 / 57



Comparison with Graph cuts

Graph cuts result

GC


CCMF

GC

CCMF result

CCMF


GC

CCMF

18 / 57



Genericity of the method S

Unseeded segmentation

Classification

T



19 / 57



Genericity of the method S

Unseeded segmentation

Classification

T



19 / 57



Dual constrained TV based formulation min x

1 F > (Ax) + kHx − f k22 2λ |A> |F 2 ≤g 2 {z } | {z } | data fidelity regularization max

f ∈ RQ observed image x ∈ RN restored image F ∈ RM flow, projection vector H ∈ RQ×N linear operator (e.g. degradation matrix)

Joint work with Jean-Christophe Pesquet [Couprie-Talbot-Pesquet-Najman-Grady, ICASSP 2011] Camille Couprie, IFPEN


20 / 57




1 F > (Ax) + kHx − f k22 2λ |A> |F 2 ≤g 2 {z } | {z } | data fidelity regularization max




20 / 57




1 F> (Ax) + kHx − f k22 2λ |A> |F 2 ≤g 2 {z } | {z } | data fidelity regularization max




20 / 57




1 F > (Ax) + kHx − f k22 2λ |A> |F2 ≤g2 {z } | {z } | data fidelity regularization max




20 / 57



Dual constrained TV based formulation 1 min max F > (Ax) + kHx − f k22 x F∈C 2λ {z } | {z } | data fidelity regularization f ∈ RQ observed image x ∈ RN restored image F ∈ RM flow, projection vector H ∈ RQ×N linear operator (e.g. degradation matrix)



20 / 57



Dual constrained TV based formulation 1 min max F > (Ax) + kHx − f k22 x F∈C 2λ {z } | {z } | data fidelity regularization f ∈ RQ observed image x ∈ RN restored image F ∈ RM flow, projection vector H ∈ RQ×N linear operator (e.g. degradation matrix) Combinatorial variant of TV with flexible choice for C



20 / 57



Dual constrained TV based formulation 1 min max F > (Ax) + kHx − f k22 x F∈C 2λ {z } | {z } | data fidelity regularization f ∈ RQ observed image x ∈ RN restored image F ∈ RM flow, projection vector H ∈ RQ×N linear operator (e.g. degradation matrix) Combinatorial variant of TV with flexible choice for C C = ∩si=1 Ci decomposed in an intersection of convex sets Joint work with Jean-Christophe Pesquet [Couprie-Talbot-Pesquet-Najman-Grady, ICASSP 2011] Camille Couprie, IFPEN


20 / 57



Results Applications in data restoration



21 / 57



Results Applications in data restoration Image denoising and deblurring

Original image


Noisy, blurry

DCTV

SNR=24.3dB

SNR=27.7dB


21 / 57



Results Applications in data restoration Image fusion

Original image


Noisy

blurry

DCTV

SNR=17.3dB

SNR=23.9dB

SNR=26.5dB


21 / 57



Results Applications in data restoration Mesh denoising

Original mesh


Noisy mesh

DCTV regularization


21 / 57



Results Applications in data restoration Image denoising using image patches

Nonlocal graph Figure from P. Coupé et al.


Original image

Noisy image PSNR=28.1dB


Nonlocal DCTV PSNR=35 dB

21 / 57






Original image




21 / 57






Original image




21 / 57



Conclusion on the flow-based methods

Discrete isotropic formulation of the max flow problem that avoids blockiness artifacts Guaranteed convergence of the Interior Point method Works on arbitrary graphs Extension to multi-label problems in a new framework : "dual constrained total variation"



22 / 57


1) 2) 3) 4)

Image segmentation Nonconvex Image filtering Stereo-vision Surface reconstruction

Outline I - Introduction II - Flow based methods 1 2



A new graph-based optimization framework Image segmentation Image filtering Surface reconstruction




23 / 57


1) 2) 3) 4)


Outline






23 / 57


1) 2) 3) 4)


Outline






23 / 57


1) 2) 3) 4)


What does all those algorithms have in common ? Graph cuts

Random walker


Shortest paths

Watersheds


24 / 57


1) 2) 3) 4)


Previously established links limq→∞ arg min x

X

wij q |xi − xj |q +

X

wi q |xi − li |q

v ∈V

eij ∈E

|i

{z } | Smoothness term

{z } Data term

l

x



25 / 57


1) 2) 3) 4)



X


X

wi q |xi − li |q

v ∈V

eij ∈E

|i


{z } Data term

l

x

q = 1 : Graph cuts [Boykov-Joly 2001 (only for 2 labels l)] Camille Couprie, IFPEN


25 / 57


1) 2) 3) 4)



X

wij 2 |xi − xj |2 +

X

wi 2 |xi − li |2

v ∈V

eij ∈E

|i


{z } Data term

l

x

q = 2 : Random walker [Grady 2006] Camille Couprie, IFPEN


25 / 57


1) 2) 3) 4)



X

X


wi q |xi − li |q

v ∈V

eij ∈E

|i


{z } Data term

l

x

q → ∞ : Shortest paths [Sinop et al 2007] Camille Couprie, IFPEN


25 / 57


1) 2) 3) 4)


Previously established links limp→∞ arg min x

X

wij p |xi − xj |q +

X

wi p |xi − li |q

v ∈V

eij ∈E

|i


{z } Data term

l

x

p → ∞ : MSF (Watershed) [Allène et al. 2007] Camille Couprie, IFPEN


25 / 57


1) 2) 3) 4)


Power watershed framework ∗ xp,q = arg min x

X


eij ∈E

X

wi p |xi − li |q

v ∈V

|i

| {z } Smoothness term

{z } Data term

l

x



26 / 57


1) 2) 3) 4)



X


eij ∈E

HHp H

0

finite

1

Reduction to seeds

Graph cuts

2

`2 -norm Voronoi

Random walker

∞

`1 -norm Voronoi

`1 -norm Voronoi

q


wi p |xi − li |q

v ∈V

| {z } Smoothness term H

X |i


{z } Data term ∞ Max Spanning Forest (watershed) [Allène et al. 07]

Shortest Path [Sinop et al. 07]

26 / 57


1) 2) 3) 4)



X


eij ∈E

HHp H

0

finite

1

Reduction to seeds

Graph cuts

2

`2 -norm Voronoi

Random walker

∞

`1 -norm Voronoi

`1 -norm Voronoi

q


wi p |xi − li |q

v ∈V


X |i




26 / 57


1) 2) 3) 4)



X


eij ∈E

HHp H

0

finite

1

Reduction to seeds

Graph cuts

2

`2 -norm Voronoi

Random walker

∞

`1 -norm Voronoi

`1 -norm Voronoi

q


wi p |xi − li |q

v ∈V


X |i




26 / 57


1) 2) 3) 4)



X


eij ∈E

HHp H

0

finite

1

Reduction to seeds

Graph cuts

2

`2 -norm Voronoi

Random walker

∞

`1 -norm Voronoi

`1 -norm Voronoi

q


wi p |xi − li |q

v ∈V


X |i




26 / 57


1) 2) 3) 4)



X


eij ∈E

HHp H

0

finite

1

Reduction to seeds

Graph cuts

2

`2 -norm Voronoi

Random walker

∞

`1 -norm Voronoi

`1 -norm Voronoi

q


wi p |xi − li |q

v ∈V


X |i




26 / 57


1) 2) 3) 4)



X


eij ∈E

HHp H

0

finite

1

Reduction to seeds

Graph cuts

2

`2 -norm Voronoi

Random walker

∞

`1 -norm Voronoi

`1 -norm Voronoi

q

wi p |xi − li |q

v ∈V


X |i



[Couprie-Grady-Najman-Talbot, ICCV 2009, PAMI 2011] Camille Couprie, IFPEN


26 / 57


1) 2) 3) 4)



X


eij ∈E

X

wi p |xi − li |q

v ∈V


|i

{z } Data term

∗ x¯ = lim xp,q p→∞



26 / 57


1) 2) 3) 4)


Convergence of RW when p → ∞ toward MSF cut Input seeds ∗ x01 = arg min x

X

wij 01 |xi − xj |2 +

eij ∈E


∗ solution x01 Camille Couprie, IFPEN

D(x) | {z } Data fidelity

∗ cut : threshold of x01


27 / 57


1) 2) 3) 4)



X

wij 02 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)



X

wij 03 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)



X

wij 04 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)



X

wij 06 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)



X

wij 09 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)


Convergence of RW when p → ∞ toward MSF cut Input seeds x13 ∗ = arg min x

X

wij 13 |xi − xj |2 +

eij ∈E


solution x13 ∗ Camille Couprie, IFPEN


cut : threshold of x13 ∗


27 / 57


1) 2) 3) 4)



X

wij 18 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)



X

wij 24 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)



X

wij 30 |xi − xj |2 +

eij ∈E






27 / 57


1) 2) 3) 4)


Convergence of RW when p → ∞ toward MSF cut Input seeds xp ∗ = arg min x

X


eij ∈E



Theorems When p → ∞, the obtained cut is an MSF cut. x¯ = limp→∞ xp∗


cut : threshold of x¯

when q > 1, the solution x¯ is unique.


28 / 57


1) 2) 3) 4)


Power watershed algorithm

1

2

3

Choose an edge with maximal weight emax . Let S the set of edges connected to emax with the same weight as emax . If S does not contain vertices that have different labels, merge the nodes of S into one node, otherwise minimize E1,q on S. Repeat steps 1 and 2 until all vertices are labeled.


2 5

1 3

4 8

5 5

5 7

9

5

9

x¯ = limp→∞ arg minx


9

5 5

4 4

P

1

5 5

7 0

9

4 4

eij ∈E

wijp |xi − xj |q

29 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

4 8

5 5

5 7

9

5

9



9

5 5

4 4 P

1

5 5

7 0

9

4 4

eij ∈E

wijp |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3 5 5

0

5 7

0

9



9

5

4 P

1 5

4 0

1

5 5

7

9

9

5

4 8

1

4 4

eij ∈E

wijp |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

0

5 5

0

5 7

0

9



9

5

4 P

1 5

4 0

1

5 5

7

9

9

5

4

8

1

4 4

eij ∈E

wijp |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

0

5 5

0 9

5

7 0

9



9

5

4 P

1 5

4 0

1

5 5

0

7

9

5

4

8

1

4 4

eij ∈E

wijp |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

0

5 5

0 9

5

7 0

9



9

5

4 P

1 5

4 0

1

5 5

0

7

9

5

4

8

1

4 4

eij ∈E

wijp |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

0

5 5

0 9

5

7 0

9



9

5

4 P

1 5

4 0

1

5 5

0

7

9

5

4

8

1

4 4

eij ∈E

wijp |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

0

5 5

0 9

5

7 0

x¯ = arg minx


P

9 1 5 5

4 0

9

1

5 5

0

7

9

5

4

8

1

4

eij ∈plateau

4 4 |xi − xj |q

30 / 57


1) 2) 3) 4)



1

2

3



2 5

1 3

0

4

8

5

0.35

5

5 0

9

5

7 0

9



9

0.71

0.48

4

P

1 5

5

4 0

1

5

5 0

7

9

1

0.74 4

4

eij ∈E

wijp |xi − xj |q

31 / 57


1) 2) 3) 4)



1

2

3



0

2

5

1 3

0

4

8

5

0.35

5

5 0

9

5

7 0

9



9

0.71

0.48

4

P

1 5

5

4 0

1

5

5 0

7

9

1

0.74 4

4

eij ∈E

wijp |xi − xj |q

31 / 57


1) 2) 3) 4)



1

2

3



0

2

5

1 3

0

4

8

5

0.35

5

5 0

9

5

7 0

9



9

0.71

0.48

4

P

1 5

5

4 0

1

5

5 0

7

9

1

0.74 4

4

eij ∈E

wijp |xi − xj |q

31 / 57


1) 2) 3) 4)



1

2

3

Choose an edge with maximal weight emax . Let S the set of edges connected to emax with the same weight as emax . If S does not contain vertices that have different labels, merge the nodes of S into one node, otherwise minimize E1,q on S.

0

2

5

1 3

0

4

8

5

0.35

0

7

9

5

7 0

9

x¯ = arg minx limp→∞ Discrete calculus optimization methods

9

0.71

4

P

0.48

5

0.74 4

0.32

eij ∈E

1 5

4 0

1

5

5

Repeat steps 1 and 2 until all vertices are labeled.


5

5 0

9

1

4

0.55

wijp |xi − xj |q 32 / 57


1) 2) 3) 4)



1

2

3


0

2

5

0.35 3

0

4

8

0.35

5

5 0

9

5

7 0

9


9

0.71

4

P

0.48

5

0.74 4

0.32

eij ∈E

1 5

4 0

1

5

5 0

7

9

1 5



1

4

0.55

wijp |xi − xj |q 32 / 57


1) 2) 3) 4)



1

2

3


0

2

5

0.35

1

3 0

4

8

5

0.35

5

5 0

9

5

7 0

9

0.71

4


P

5

0.74 4

0.32

eij ∈E

1 5

0.48



9

4 0

1

5

5 0

7

9

1

4

0.55

wijp |xi − xj |q 32 / 57


1) 2) 3) 4)


Comparison of results

Input seeds

GraphCut

Input seeds

GraphCut

RandWalk

ShtPath

RandWalk

ShtPath

MaxSF

PW q = 2

MaxSF

PW q = 2



33 / 57


1) 2) 3) 4)


Comparison of results

Input seeds

GraphCut

Input seeds

GraphCut

RandWalk

ShtPath

RandWalk

ShtPath

MaxSF

PW q = 2

MaxSF

PW q = 2



33 / 57


1) 2) 3) 4)


Algorithms comparison Evaluation on GrabCut database 2 sets of seeds to study robustness to seeds centering 1

2

seeds well centered around boundaries : Best performer : Shrt path, worst performer : GraphCuts seeds less centered around boundaries : From best to worst : GraphCuts, PWshed, Random Walker, MaxSF, Shrt path

Algorithms behavior on plateaus Seeded image


Graph Cuts

Shortest Paths, Watershed


Random Walker, PW q = 2

34 / 57


1) 2) 3) 4)



2




Graph Cuts




34 / 57


1) 2) 3) 4)



2




Graph Cuts




34 / 57


1) 2) 3) 4)


Computation time



35 / 57


1) 2) 3) 4)


Optimal multilabels segmentation l solutions x 1 , x 2 , ...x l computed k k x (l ) = 1 x k computed by enforcing x k (l q ) = 0 for all q 6= k. Each node i is affected to the label for which xik is maximum : si = arg max xik k

Input seeds


Segmentation by PowerWatershed (q = 2)


36 / 57


1) 2) 3) 4)


Unseeded segmentation F

wF

x¯ = lim arg min p→∞

x

X

wijp |xi

eij ∈E

q

− xj | +

X

p

q

wFi |xi −1| +

vi

X

p

wBi |xi |

q

vi wB B

Image


Graph Cuts


Watershed

37 / 57


1) 2) 3) 4)


Unseeded segmentation F

wF

x¯ = lim arg min p→∞

x

X

wijp |xi

eij ∈E

q

− xj | +

X

p

q

wFi |xi −1| +

vi

X

p

wBi |xi |

q

vi wB B

Image

Graph Cuts

Watershed

This is the first time that it is shown how to incorporate data unary terms into watershed computation. Camille Couprie, IFPEN


37 / 57


1) 2) 3) 4)


Semantic Segmentation Power Watershed Result

Unary weights

Image l

→

x

→

Pairwise weights



background bookshelf cabinet floor ceiling wall picture

Ground truth

38 / 57


1) 2) 3) 4)


Semantic Segmentation

Image

Legend : background bed


Graph Cuts

blind bookshelf cabinet

Random Walker

Power watershed

ceiling floor picture


sofa table television

Ground truth

wall window

39 / 57


1) 2) 3) 4)


Non-convex diffusion using power watersheds Anisotropic diffusion [Perona-Malik 1990]

Image

100 iterations

200 iterations

Goals of this work : perform anisotropic diffusion using an `0 norm to avoid the blurring effect optimize a non convex energy using Power Watershed [Couprie-Grady-Najman-Talbot, ICIP 2010] Camille Couprie, IFPEN


40 / 57


1) 2) 3) 4)


Anisotropic diffusion and `0 norm X

x ∗ = arg min x

σ(xi − xj ) + λ

eij ∈E

σ(xi − fi )

vi ∈V

{z } | smoothness term σ(x) = 1 − e−αx

X

{z } | data fidelity term

Leads to piecewise constant results Original image PW result

2

α = 100 α=1

α = 10

x



41 / 57


1) 2) 3) 4)



min x

σ(xi − xj ) + λ

eij ∈E

σ(xi − fi )

vi ∈V

| {z } smoothness term σ(x) = 1 − e−αx

X


2

α = 100 α=1

α = 10

x



42 / 57


1) 2) 3) 4)



min x

σ(xi − xj ) + λ

eij ∈E

X

σ(xi − fi )

vi ∈V

| {z } smoothness term


2

High gradient xi − xj ⇒ σ = 1

σ(x) = 1 − e−αx

α = 100 α=1

α = 10

x



42 / 57


1) 2) 3) 4)



min x

σ(xi − xj ) + λ

eij ∈E

X

σ(xi − fi )

vi ∈V



2


σ(x) = 1 − e−αx

No gradient ⇒ σ = 0 α = 100 α=1

α = 10

x



42 / 57


1) 2) 3) 4)



min x

σ(xi − xj ) + λ

eij ∈E

X

σ(xi − fi )

vi ∈V



2


σ(x) = 1 − e−αx


α = 10

Finite α, low gradient ⇒ 0 < σ < 1 Piecewise smooth result

x



42 / 57


1) 2) 3) 4)



min x

σ(xi − xj ) + λ

eij ∈E

σ(xi − fi )

vi ∈V



2


σ(x) = 1 − e−αx


α = 10

x


X

Finite α, low gradient ⇒ 0 < σ < 1 Piecewise smooth result α → ∞, approximation of `0 norm low gradient ⇒ σ = 1 Piecewise constant result


42 / 57


1) 2) 3) 4)


Anisotropic diffusion using power watershed min x

X

σ(xi − xj ) + λ

eij ∈E

{z } | smoothness term

X

σ(xi − fi )

vi ∈V


Nonconvex energy Set the gradient of this energy to zero Fixed point iteration scheme with energy at step k :



43 / 57


1) 2) 3) 4)


Anisotropic diffusion using power watershed X

min x

σ(xi − xj ) + λ

eij ∈E

X

σ(xi − fi )

vi ∈V



Nonconvex energy

f

Set the gradient of this energy to zero Fixed point iteration scheme with energy at step k : X

Ek+1 =

eij ∈E Camille Couprie, IFPEN

k −x k )2 j

e −α(xi

(xik+1 − xjk+1 )2 + λ

xk+1

X

k −f

e −α(xi

2 i)

(xik+1 − fi )2

vi ∈V Discrete calculus optimization methods

43 / 57


1) 2) 3) 4)


Anisotropic diffusion using power watershed X

min x

σ(xi − xj ) + λ

eij ∈E

X

σ(xi − fi )

vi ∈V



Nonconvex energy

f

Set the gradient of this energy to zero Fixed point iteration scheme with energy at step k : Ek+1 =

X eij ∈E


k −x k )2 j

e −(xi

α

(xik+1 −xjk+1 )2 +λ

xk+1

X

k −f

e −(xi

2 i)

α

(xik+1 −fi )2

vi ∈V Discrete calculus optimization methods

43 / 57


1) 2) 3) 4)


Stereovision using power watershed Compute the disparity map from two aligned images

Labels correspond to the disparities, weights to similarity coefficients between blocks



44 / 57


1) 2) 3) 4)


Surface reconstruction from a noisy set of dots

⇒ Goal : given a noisy set of dots, find an explicit surface fitting the dots. Joint work with Xavier Bresson Camille Couprie, IFPEN


45 / 57


1) 2) 3) 4)





45 / 57


1) 2) 3) 4)





46 / 57


1) 2) 3) 4)





46 / 57


1) 2) 3) 4)


How to solve this problem Graph : 3D grid Here x represents the object indicator to recover. X x¯ = lim arg min wij p |xi − xj |q p→∞

x

eij ∈E

s.t. x(F ) = 1, x(B) = 0 weights : distance function from the set of dots to fit

Why PW are a good fit for this problem ? numerous plateaus around the dots to fit → smooth isosurface is obtained Camille Couprie, IFPEN


Power watershed solution 47 / 57


1) 2) 3) 4)


Comparisons

Total variation

Graph cuts

Power watershed

Size of required seeds



surface normals estimation required Camille Couprie, IFPEN


48 / 57


1) 2) 3) 4)


Comparisons

Total variation

Graph cuts

Power watershed

Fast, accurate, globally optimal surface reconstruction from noisy set of dots Robust to markers placement No normal estimation information required No post-processing smoothing step Camille Couprie, IFPEN


49 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Convergence issues AT-CMF Filtering using Graph cuts expensive



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Filtering using Graph cuts expensive



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive Extention of the CCMF framework leads to a flexible filtering framework on graphs



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive Extention of the CCMF framework leads to a flexible filtering framework on graphs Power watersheds answers several problems of standard methods Non unique solution Leaking effect Random Walker, Graph cuts : super-linear complexity



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive Extention of the CCMF framework leads to a flexible filtering framework on graphs Power watersheds answers several problems of standard methods Non unique solution Unique solution Leaking effect Random Walker, Graph cuts : super-linear complexity



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive Extention of the CCMF framework leads to a flexible filtering framework on graphs Power watersheds answers several problems of standard methods Non unique solution Unique solution Leaking effect Reduction of the leaks Random Walker, Graph cuts : super-linear complexity



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive Extention of the CCMF framework leads to a flexible filtering framework on graphs Power watersheds answers several problems of standard methods Non unique solution Unique solution Leaking effect Reduction of the leaks Random Walker, Graph cuts : super-linear complexity Quasi-linear experimentally. Worst case : RW complexity.



50 / 57


1) 2) 3) 4)


Conclusion Reformulation of classical max flow method Block artifacts of classical max flow Not present with CCMF Convergence issues AT-CMF Convergence guaranteed Filtering using Graph cuts expensive Extention of the CCMF framework leads to a flexible filtering framework on graphs Power watersheds answers several problems of standard methods Non unique solution Unique solution Leaking effect Reduction of the leaks Random Walker, Graph cuts : super-linear complexity Quasi-linear experimentally. Worst case : RW complexity. More importantly : use of unary terms and multi labels opens the way to large field of applications Camille Couprie, IFPEN


50 / 57









51 / 57


Outline




51 / 57


Feature Learning with Convolutional Networks Feature learning : not new [LeCun et al. 89, 98]

Visual cortex organizes recognition in a hierarchical way Convnet : Layered convolutions and downsampling steps Applications : classification, object recognition. Semantic segmentation ? Camille Couprie, IFPEN


52 / 57


Multiscale feature learning for scene labeling Full  image labeling implies joint  Recognition Localization of objects  Delineation

f1 (X1;𝛉1) X1

pyramid I

g (I)

X2 X3

F1

convnet

labeling

f2 (X2;𝛉2) f3 (X3;𝛉3)

F2

l (F, h (I)) F

F3

[Clément Farabet, C. Couprie, L. Najman, Y. LeCun ICML,PAMI 2012] Camille Couprie, IFPEN


53 / 57


Segmentation sémantique – près de Broadway

Play Farabet, Couprie, Najman, LeCun PAMI 2013 Camille Couprie, IFPEN


54 / 57


Segmentation sémantique à Washington Square Park

Play Farabet, Couprie, Najman, LeCun PAMI 2013 Camille Couprie, IFPEN


55 / 57


Vers de la segmentation video temps réel en intérieur Input depth image

f1 (X1;𝛉1)

RGBD Laplacian pyramid

F1

convnet

g (I)

f2 (X2;𝛉2) f3 (X3;𝛉3)

F2

F

F3

labeling

l (F, h (I))

pict.

wall

chair

ceiling pict.

chair table

superpixels Input RGB image

segmentation

h (I)

Utilisation d’un nouveau canal “profondeur” dans le réseau multi-échelle [Couprie, Farabet, LeCun, Najman ICLR 2013] Segmentation en superpixels de flux videos causaux [Couprie, Farabet, LeCun, Najman ICIP 2013] Camille Couprie, IFPEN


56 / 57









57 / 57


Outline




57 / 57


L’IFPEN en bref

“Institut Francais du Pétrole, énergies nouvelles” Environ 1700 employés, une majorité de docteurs Financement actuel : 50 % Etat, 50 % resources propres Deux sites : Rueil Malmaison, et Solaize



58 / 57


Activités à l’IFPEN

Bio-informatique Analyse de materiaux Traitement de données sismiques



59 / 57






59 / 57






59 / 57


Perspectives / travaux futurs Segmentation / détection d’horizons et failles dans images sismiques

Co-segmentation image / maillages, recalage de graphes irréguliers Compression de données sismique Segmentation interactive de gros volumes Camille Couprie, IFPEN


60 / 57


Questions

Source code for segmentation available from: http ://sourceforge.net/projects/powerwatershed/



61 / 57


References C. Farabet, C. Couprie, L. Najman, and Y. Lecun : Hierarchical Feature Learning for Scene Parsing to appear in PAMI 2012. C. Couprie, L. Grady, L. Najman, and H. Talbot : Power Watersheds : A unifying graph-based optimization framework. In IEEE Trans. on PAMI 2011. C. Couprie, X. Bresson, L. Najman, H. Talbot and L. Grady : Surface reconstruction using Power watersheds. In Proc. of ISMM 2011. C. Couprie, L. Grady, L. Najman, and H. Talbot : Anisotropic diffusion using power watersheds. In Proc. of ICIP 2010. C. Couprie, L. Grady, L. Najman, and H. Talbot : Power watersheds : A new image segmentation framework extending graph cuts, random walker and optimal spanning forest. In Proc. of ICCV 2009.



62 / 57

Discrete calculus optimization methods - and applications in ...

Discrete calculus optimization methods - and applications in ...

Suggest Documents

Discrete Optimization Methods and their Role in

Discrete Optimization Methods and their Role in the ... - CiteSeerX

Calculus, Applications and Theory

Calculus, Applications and Theory

Editorial Optimization Theory, Methods, and Applications in ... - Hindawi

Editorial Optimization Theory, Methods, and Applications in ... - Hindawi

Applications of Malliavin calculus to Monte Carlo methods in finance

Calculus with applications for Scientists Discrete Mathematics for ...

Fractional Calculus - Modern Methods, Problems and Applications of ...

Calculus with applications for Scientists Discrete Mathematics for ...

Calculus with applications for Scientists Discrete Mathematics for ...

discrete size and discrete-continuous configuration optimization ...

pdf-1862\data-mining-methods-and-applications-discrete ... - Google

Discrete Mathematics, Discrete Physics and Numerical Methods

Engineering Applications in Differential and Integral Calculus*

Noncommutative Geometry and Stochastic Calculus: Applications in ...

Noncommutative Geometry and Stochastic Calculus: Applications in ...

Editorial Fractional Calculus and Its Applications in

Multivariable Calculus, Applications and Theory

Multivariable Calculus, Applications and Theory

Differential Evolution in Discrete and Combinatorial Optimization

Linear and Discrete Optimization - GitHub

Notes on Discrete Exterior Calculus

MA6010 DISCRETE MATHEMATICS Propositional Calculus ...