Interactive Segmentation

5/22/2006

SIGGRAPH 2004 Lazy Snapping

Interactive Image Cutout

Yin Li Jian Sun Chi-Keung Tang Heung-Yeung Shum

z Separate an object from its background z Compose the object on another image

Microsoft Research Asia Hong Kong University

Carsten Rother Vladimir Kolmogorov Andrew Blake

GrabCut

Microsoft Research CambridgeCambridge-UK

Interactive Graph Cut

Interactive Graph Cut

z (Boykov & Jolly. ICCV’01) z Optimized by s-t min-cut algorithm Draw Drawforeground foreground

and andbackground background

Graph Graph Cut Cut Segmentation Segmentation

Interactive Graph Cut

Image

z(Boykov & Jolly. ICCV’01)

Hard Constraints z X : Segmentation.

xi ∈ {" obj" , " bkg"} z Hard Constraint:

∀i ∈ O

xi = " obj"

∀i ∈ B

xi = " bkg"

z(Boykov et al. ICCV’01)

1

5/22/2006

Image as a Weighted Graph

Soft Constraints

Foreground (source S)

Image

Min Cut

z Minimize the Energy:

E ( X ) = ∑ E1 ( xi ) + λ ∑ E2 ( xi , x j ) i∈V

i , j∈E xi ≠ x j

z E1 : Region: Color difference to user marks z E2 : Boundary: Color similarity between pixels

Background (sink T) Graph:

source & sink, n-links & t-links

Cut=Segmentation: Separate ‘source‘ & ‘sink‘ Energy of cut: sum wieghts of edges Min-Cut Max-Flow: Global minimal enegry in polynomial time

Weights t-links

i ∈ B ⇒ {i, T } : ∞ {i, S } : 0 i ∈ U ⇒ E1 ( xi ) = hxi ( I i )

Lazy Snapping Li et al. SIGGRAPH’04

n-links E2 ( xi , x j ) ∝ exp( - (I 1 – I 2 )2 ) Min Cut = Minimize Soft Constraints keeping Hard Constraints

Lazy Snapping z Lazy Snapping for

Lazy Users z 2 Steps UI:

1. Coarse Step: Obj/Bkg Marking => Graph Cut

Lazy Snapping 2. Fine Step: a. Border Brush b. Pixel Editing

=> Graph-Cut on border

2

5/22/2006

Weights

Per-Pix Graph Cut

z E1 : Color difference to user marks Intensities -> Colors Histogram -> “K-means” clustering

E1 ( xi =" obj" ) ∝ RGB_dist to closest cluster centroid

z E2 : Color similarity between pixels For neighboring pixels of different xi

E2 ( xi , x j ) =

1 1 + Ci - C j

2

Pre-Segmentation

Graph Cut on Regions

Graph Cut on Regions

Graph Cut on Regions

3

5/22/2006

Graph Cut Algorithm

Region-based Graph Cut

Per-pixel method

Region based method

Pixels

Small regions

Neighbors

Region connection

Pixel color

Region mean color

Color difference

Region color difference

First Step: Object Marking Second Steps: Boundary Editing Coarse Boundary

z Pre-processing overhead ‹ 2~3 seconds background processing

Polygon Fitting

Divide and Conquer

Input Image

z Advantages ‹ More than 10 times fewer nodes ‹ Instant feedback of cutout result

Refined Boundary

z First vertex – border pixel with highest curvature z Next vertices: furthest boundary pixel z Stop when distance < thresh

Border Editing z Brush - Replace polygon segment z Vertex Editing: Move/Add/Delete

Quickly identify the object

Control the detail boundary

Band of Uncertainty

=> Graph Cut on border pixels

Optimization in the Band

Pixel Based Graph Cut Segmentation

4

5/22/2006

Edit the Polygon Vertices

Edit the Polygon Vertices

Low Contrast Example

Boundary Editing

Boundary Editing

Video Demo (Left boy)

z For Low Contrast case: ‹ In E2 - Add a term to reflect distance from polygon z Hard Vertex constraint ‹ Adjust graph so cut passes through vertex

5

5/22/2006

Summary: Two Steps

Video Demo (Right Boy)

First Step: Object Marking Second Steps: Boundary Editing Input Image

Small Regions

Coarse Boundary

Region Based PrePre-Segmentation Graph Cut

Editable Polygon

Polygon Fitting

Refined Boundary

Band Pixels Graph Cut

Photomontage

GrabCut Interactive Foreground Extraction using Iterated Graph Cuts

Iterated Graph Cut

User Initialization

Gaussian Mixture Models (GMMs (GMMs)) z GMM instead of Histogram (Color model)

?

z Assume distribution is a mixture of Gaussians

Gμ ,Σ ( x) − Gaussian K

GMM(x) = ∑ wk Gμ k ,Σ k ( x) GMM estimation for learning colour distributions

Graph cuts to infer the segmentation

∑w

k

k =1

=1

z EM algorithm – find best

wk , μ k , Σ k

for the given set of samples z GrabCut – Different approach

6

5/22/2006

Incomplete Labeling

Iterated Graph Cuts z

z

z User specifies border => B, U = B, F = φ z F populates through iterations z Some F pixels can be retracted. B cannot

E1 – GMMs (E2 – No change)

Algorithm: 1. Initialize B, U = B, F = φ Initialize GMMs wk , μ k , Σ k 2. Repeat (until constant energy) a. ∀p ∈ U assign best Gk => 2K clusters

Editing (In case of error):

b. For each cluster calculate wk , μ k , Σ k => 2 GMMs c. Find Min Cut => U decreases

z User adds F, B (brush) z Re-compute z Graph Cut can be reused.

3. Apply border matting 4. Enable user editing & repeat

Iterated Graph Cuts

Gaussian Separation Gu

ar co ante n v ed er g e to

R

Foreground & Background

Background

1

Result

2

3

G

R Foreground

Background

4

Energy after each Iteration

Moderately straightforward examples

Iterated graph cut

G

Gaussian Mixture Model (typically K=5)

Difficult Examples Camouflage & Low Contrast

Fine structure

No telepathy

Initial Rectangle

Initial Result

7

5/22/2006

Comparison

Evaluation – Labelled Database

Boykov and Jolly (2001)

GrabCut

User Input

Result

Error Rate: 1.87% 1.81% 1.32% 1.25% 0.72%

Border Matting

Error Rate: 0.72%

Bayes Matting - Chuang et. al. (2001)

Extract α-values along border

z Create U band ± w z Local rectangle z Estimate GF , GB z U: μα = α μ F + (1 − α ) μ B

α

F

z

U B Hard Segmentation

Band of Uncertainty

GU (α ) = G( μα , Σα )

z Find α that maximizes GU

with respect to pixels in U

Soft Segmentation

Border Matting - GrabCut

Dynamic Programming

Foreground Noisy alpha-profile 1 Mix

t+1 t

σ

Background

DP 0 Foreground

Δ Mix

Result using DP Border Matting Background

Max : G ( μα , Σα )

Fit a smooth alpha-profile with parameters Δ, σ Noisy alpha-profile

T

Min : ∑ (Δ t − Δ t −1 ) 2 + (σ t − σ t −1 ) 2 t =1

Regularisation

8

5/22/2006

Summary

Matting Results

z GU (α ) should match U pixels z α should change like a soft step function z Step function should change smoothly

along contour

Input

Bayes Matting

GrabCut

(no regularization)

(with regularization)

Lazy Snapping vs. Grab Cut

Lazy Snapping

GrabCut

User Interface

Marking brush – FG + BG Overriding brush Vertex editing

Rectangle/lasso – BG only Marking brush - [optional]

Algorithm

Region-based Graph Cut Border pixel Graph Cut

Iterative Graph Cut

Performance

Fully interactive Includes Pre-Processing

Fast

Border

Border Editing

Border Matting

Thank You

9