Regularized Estimation of Occluded Displacement ... - Semantic Scholar

REGULARIZED

ESTIMATION

OF OCCLUDED

Damon

L. Tull

DISPLACEMENT

Aggelos

VECTOR

FIELDS

K. Katsaggeios

Department

of Electrical and Computer Engineering Northwestern University Evanston, Illinois 60208-3118 [email protected]. edu, aggkfleecs.nwu.edu

ABSTRACT Occluded

regions

and

placement

vector

reconciled

to accurately

+A)z: ;; [@([vu(i, j)l) + W7qi.i)l)]

motion

field (DVF)

boundaries

introduce

discontinuities

estimate

image

flow.

dis-

that must be In this work,

the robust regularized estimation of the DVF is considered in the presence of these discontinuities. A robust convex estimation criterion is presented that preserves motion boundaries and allows for a globally A new class edge

of robust

preserving

gradient sequences

convex

and

of the DVF.

is introduced

an occlusion

as mechanism

due to occlusion.

estimate

measures

regularization

is proposed

continuities

optimal

for

weighted

for managing

Results

DVF

using synthetic

dis-

image

are presented.

where A is the regularization parameter, and lVv(i, j) I and IVU(Z, j)l represent the magnitude of the gradient of the vertical and horizontal components of the DVF, respectively. The addition of the (~) terms in Eq. (2) imposes a level of smoothness

on the motion

In traditional

(Tikhonov)

rj and ~ are chosen 5., unfortunately tion estimators ous features

(unweighed)

applications

including

cal imaging.

Discontinuities

used to estimate image scene. essential

image sequence

and filtering),

The effective

(depth)

Most traditional formulations izontal and vertical components U(Z, j), respectively ference (DFD), DFD(i,

j)

and/or

management estimation

(compres-

vision

convey information

the structure

for the accurate

processing

machine

and medithat can be

motion

of an

of discontinuities

is

of these features.

attempt to find the horof the DVF, V(Z, j) and

that minimize

the displaced

frame dif-

=

p(i,j) - p-’(i -

=

f:(i.m(i.o + f:(~jj)u(i~) + ff!

U(i, j), j -

v(i,j)) (1)

Dubois

measures

bet ween their robust

mulations proved

are obtained

rion to be minimized

authors

considered

iterative ited

performance

Although

the

and M RF forsignificantly

formulation

im-

the crite[1, 2] and in

non-convex approach

of ~ which

of the globally work,

~ (@ quadratic)

regulariz at ion algorithm.

of their

selection

8Q(u,

was attributed only

optimal

Rouchouze

et.

allowed

in an

The limfor

to the an ap-

DVF.

al. [5] and later Tull and

[6] suggested robust, convex@ functions which of edge preserving regularization. Selecting @ allows one to obtain a globally optiThe mechanism for edge preserving

regularization in this framework can be seen by taking the derivative of Eq. (2) with respect to U(Z, j) which gives,

criterion,

v, A)

Zirl(i, j)

NM

j=]

approach

in both formulations,

robust

minimization

C=l

and

[2] a ‘line process’

# and demonstrated

in the MRF

half-quadratic

and 4 to be convex mal DVF estimate.

~

is an

In Konrad

the robust formulation of [3] is non-convex, and have many local minima requiring costly minimization techniques that only approximate e a globally optimum solution. In [4] the

the DVF estimate accounts for differences in the frames of interest. The unconstrained global minimization of Eq. (1) is an ill-posed problem, requiring techniques such as regularization to obtain a unique (and meaningful) solution. A well-posed estimate of the DVF can be obtained from the

Q(u, r),A) = ~

for rj and

with a ‘line process’.

results

Katsaggelos are capable

DFD

and Katsaggelos

relationship

izontal and vertical) and temporal first derivatives of j~, respectively. The DFD is a non-linear measure of how well

of the regularized

mo-

discontinu-

of discontinuities

estimation.

non-convex

In recent

are the spatial

regularized

important

formulation. Black and Anandan in [3] first introduced the use of robust measures in Eq. (2). They proposed robust

proximation

of the pixel at location

and ~~, $$ and j$

both

in Section

was used to adapt their motion models in the presence of abrupt motion boundaries in a Markov random field (MRF)

(z, j)

is the intensity

Tikhonov

the management

(hor-

where jk (z, j),

As shown

of the DVF.

[1] and Brailean

non-convex in the kth frame

the DFD.

DVF estimation,

to be quadratic.

active area of research in motion

INTRODUCTION

The management, detection and estimation of discontinuous features in an image scene is vital to a wide variety of sion, interpolation

field that minimizes

regularized

tend to over-smooth

For this reason, 1.

(2)

i=l J=l

= #( DFD(i,

j))fj

+A

~

W(i,.i)

aprt(i,j)l ~u(z, ~,

,

e,j~e

rJ(DFD(i, j))

(3) ., where,

Copyright (C) 1996 IEEE.

@ is the support

over which

the derivative

All Rights Reserved.

is non-

trivial

and W(Z, j) is defined

as,

LJ(i,j) = d’(lv4i.ol) glvu(i,j)l ‘ where

+’ denotes

ferences

the first derivative

for spatiaJ derivatives,

the second

summation

(4)

of ~.

Using

the authors

Although

somewhat

functions inst ante,

for # successfully

some

adhere

previously

to these

first dif-

#GRN(Z)

=

log(cosh(~))

in [5] show that

#ENT(~)

=

(1~1+e-’)ln(lxl

proposed

conditions

for

(lo)

+ e-’).

becomes, The

(5) where Z(Z, j) is defined

from

authors

in [5] used 4GRN

j) = W(i, j – l)u(i,

+W(i, j)u(i,

+1, j)

j – 1) + W(i, j)u(i

j + 1)) +U(i

– l,j)u(i–

robust

and convex

ularization

motion

esti-

twice differentiable convex

measure

can be derived

for all z.

for edge

from

preserving

the error function

reg-

defined

as,

(6)

I,j),

for regularized

was introduced , an “entropic” class mation. In [6] #ENT of functional [8] to robust motion estimation. Both are

A new robust WU(i,.i)ti(i,

restrictive,

z

and W(Z, j) is defined

It can

erf(z)

as,

/ ..6

Wu(i, j) = W(i, j – 1) +Zu(i,j)

+W(i

be

LOU(Z,j)it(i,

weighted

seen

from

Eq.

(6)

sum of the motion

neighborhood

of (z, j).

not want motion

that

j)

is the

field at pixel locations

in the

To preserve

at pixels

(7)

– 1,.i)

across

edges in the DVF,

we do

the edge to influence

A closed form for this integral, tive distribution

function

does not exist.

Further

expressed

to the smoothness

this it should

the edge preserving

nature

In the following for edge preserving convex

measures

ularization.

penalty

at the center.

be clear that the structure

of@

Section

are suitable

3. describes

for edge preserving

a robust

iterative

regions

where the DVF is undefined

manage

occluded

gradient

is incorporated.

to synthetic

regions

sequences

in the DVF

es-

in Section

algorithm

is applied

motion

in Section

CLASS

OF

ROBUST

selection

of @ is non-trivial. gradient,

must

gument,

the spatial

removing the local

the influence of an edge pixel in the surround of computation of the smoothness in Eq. (6). In [7]

vanish,

several requirements for convex edge preserving measures are present ed. The first constraint is placed on the asymptotic behavior of ~’ in the form,

A sufficient

be convex

x-m

4’(Z)_ ~ —_ x

lim $+0

$$’(~) = (-J> —

().

z

for edge preserving

the minimum

selection

condition

From

this fact,

its second shown

regularization

(12) of

of Q4is convex

f(z)

for a function

f“(a)> O,vz6 Y?. Eq.

derivative

that

@IERF

(11)

shows

4TERF

is the normal also complies

(8) and (9) for convex the first integral

to

to be convex

distribution. with

robustness of CDFS

(13) for

It can be

the constraints

in

in this formulation.

as a family

of robust Due na-

ture of I#IERF. Figure 1 illustrates the robust structure of this proposed functional. In comparison to the quadratic, @IERF has a sknificanW srnfler P-h for large values of the argument while approximating the quadratic for smaller values. The first derivative (i$IERF achieves its asymptotic values faster than the previously proposed @ ENT and d GRN functions. The weighting functions (# ’(z) /z) in Figure l(r.)

and #GRN

(8)

are

considered in the context of anisotropic diffusion. In their analysis, an additional (weak) constraint was imposed on the second derivative of ~ to avoid smoothing across edges,


The

to Vanish in a similar manner.

ratio of Eq. (9) for dIERF

for much smaller considered

1The

(9)

seem

The difference can be seen in Figure 2 were the derivative constraint of Eq. (9) is plotted against gument.

In [7], the conditions

be

is,

for @IERF ]im

to make

convex functional is of significant theoretic int crest. to space limitations we only consider the qualitative

For large values of its arthe # function

~–1 12 is included

zero at z = O. This

Eqs.

CONVEX

can only

of the er~ exists

+7r-1’2f3-’2 – rr-]’z.

x*erf(z)

and robust. ]

5.

FUNCTIONAL The

erf(z)dz

constant

Clearly, A NEW

2.

the cumuladistribution,

of the erf

The integral

this function

4. To

weighted

processing

r =

motion for oc-

an occlusion

The proposed of occluded

reg-

DVF

timation algorithm that successfully manages DVF boundaries. We extend this approach to account cluded

The

section, we discuss the conditions on ~ regularization and present a new class of that

41ERF(~)=

From

solution.

of the normal

and takes the form,

the

will determine

of the regularized

which represents

(CDF)

in terms of the erf.

cost of smoothness at the present location (i, j). That is, if the gradient of a pixel in the surround is large, it should not contribute

(11)

~e-’’dt.

=

—er.f (z)

values of the argument

second its ar-

is near w4@le

than for previously

~.

erj(z)

function

is

allowing for negative

odd

symmetric,

e~~(–z)

values of the argument

modification.


=

without

4. The

AN OCCLUSION WEIGHTED SMOOTHNESS CONSTRAINT

algorithm

of the previous

discontinuities

in the DVF.

section

However

effectively like most

manages motion

es-

timation algorithms, motion is assumed to be defined for all pixels in an image. In practice, this is not the case. Occluded

Figure

1.

su;es.

(r.)

(1.)

Robust

tions.

Quadratic

Robust

ted), IERF

quadratic and

and

quadratic

(solid),

Entropic

(dash),

such as uncovered

background

or new ar-

meafunc-

sion weighted

robust

weighting

regions

eas unrelated to motion are common in (video) image sequences. Such areas should not impact the DVF estimate at locations where the motion is defined. To manage the discontinuities due to occlusions using the edge preserving mechanism described in Section 1. we introduce an occlu-

LogCosh

gradient,

(dot-

(dot- da;h).

Ivzlo(i,’j)l’ = K(o(i,

j+

l))uz(i,

j)’

+K(o(i+

l,j))%(~,j

)’> (16)

where

the function

~(o(i, j))

the value of the occlusion

O(Z,j) 6 [0, 1], is

= [1, ~~~~],

field at pixel location

(z, j)

and

~ ~~~ is a large positive constant. If a neighborhood point is found to belong to an occluded region (where O(Z,j) tends to zero) K(O(Z, j)) tends to ~~~., removing its effect on the local smoothness calculation. We estimate the occlusion field from an initial

O(i, j)

Figure for

2. Robust

Quadratic

IERF

3.

and

quadratic

(solid),

weighting

Entropic(dash),

(~_ ~ _

2Qrag(i,

j)

)

s@(24er~(eedge(~) 2Wge(i,

(

field using,

d)

w4f+~f(~tTag(i

functions

Log f30sh (dotted),

of the motion

~)) j)

(17) )

(dot-dash).

A ROBUST

In [3], Black terms

ITERATIVE DVF ALGORITHM

and Anandan

bust functional

for both

IL allows

assumptions purposes

Letting 4= vertical

component

the DFD

that

are often

of the DVF

Using

the Horn

gives the vertical

Wg.(i.i)

optimal

For the

estimate

is obtained

of the

by setting

tensity

and Schunk

=

Aqi,

of occlusion

horizontal

component

approximation

To

is obtained

manage

xl(i,

(.f:(i, j).f$(i, J + ft(idf:(i d) Xh(i,.i) + f$(i,.i)z (15)

Equations

(14)

the occlusion 5.

on a uniform j) –

and (15)

are evaluated

the entire image until the change predetermined threshold.

for each pixel

in the DVF

of ~(.)

simple

defined

per-

measure

as the linear

(19)

= 1 + O(i, j) * (Kmax – 1).

over

is below

occlusion

j)l in Eq. (3).

a


regions Replace

weighted

Vu(i,

j)

AND

a uniform

background.

is

replaced

with

u with v in Eq. (16) to obtain

the horizontal

RESULTS

Figure 3 illustrates

from,

?J(2,j) =

This

of

of the DVF,

of the DVF

pair.

in in-

function, K(o(i, j))

j) – (f:(i,.i).ft(i, j) + ft(i.o.f:(i.i)) AJAJ(i, j) + R(i, j)’ motion

is the argument

the deviation

and the temporal

the

(14) similarly

of motion

sist ence of edges in the image

Vuo(i, ‘u(z’,j)

- Ivfk-’(i - ZL(i, j),j - V(i,j))l.

ct,~g and ~.dg. measure

in the direction

(3) to zero and solving

component

Ivfk(i,j)l

=

The quantities

required

of noise.

we select

of Q(M, v, A) in Equation

for U(Z, j).

note that US-

in the underlying

in the presence

The globally

(18)

~)

~ quadratic as in [5, 6]. r#lERFmakes the regularized DVF criterion of

(2) convex.

derivative

They

deviations

constraint

the use of ro-

(~) and motion

formulation.

flow estimation of this work

Equation

first considered

for slight

of motion

for optical

ESTIMATION

the smoothness

in their regularized

ing robust

The

=

estimate

motion

component.

DISCUSSION

object

translating

In this image

pair,

to the right the uncov-

ered background is identical to the (previous) surround, in a sense, masking the occlusion effect. In theory, the DFD can be minimized to zero for this image pair. Figure 4 illustrates the DVF estimates obtained using ~(z) = X2 (left) and O(X) = q51ERF (right)

the over smooth This is expected

without

estimate because

occlusion

management.

Note

obtained in the case of @(z) = Z2. the quadratic equally weights the


smooth areas and edges in the estimate as shown in Figure 1. The robust convex selection of @ = CJ51ERF allows for

segmentation

DVF edges while smoothing the motion field by discounting the penalty for DVF edges. Although the robust estimate results in a DVF more useful for higher order processing (segmentation, region coding), in a strict sense, both approaches make errors in their estimates in that motion is

sion region detect ion [9, 10] is also under investigation.

imposed

in areas where motion

was experiment

ally chosen

is undefined.

to be equal

In all tests,

is expected

information

by further

into the robust

embedding

formulation.

occlusion

Enhanced

occlu-

-. —. —: —. --.__=

A

1.1.

Figure 5 shows the same object moving with anomalous background exposed due to motion. Clearly, motion for the bright region exposed in the second frame of the image pair is undefined.

This

region

ance is irreconcilable

is occluded

because

its appear-

frame.

The robust

using the previous

estimate of the motion field with and without weighted gradient is shown in Figure 6.

the occlusion

Robust estimate with exposed backFigure 6. ground without occlusion management (L), estimate with occlusion management (r.). REFERENCES [1] J. Konrad

and E. Dubois.

tion vector

fields”.

“Bayesian

IEEE

Trans.

estimation

on PAMI,

of mo-

14(9):910-

927, September 1992. [2] J.C. Brailean

Figure

3.

right).

Note:

Horizontal

motion

Uniform

(block

moving

left

backgroundmasks occlusion

to

effects.

[3]

and A.K.

Katsaggelos.

stationary

map displacement

gorithm”.

IEEE

429,

1995.

April

Trans. on Image

M.J Black and P. Anandan. of motion

Vision,

Processing,

“A model

In Proc.

over time”.

“A recursive

Int.

al-

4(4):416-

for the detection

Conf.

on Computer

pages 33-37, Osaka, Japan, December 1990.

[4] L. Blanc-F6raud, M. Barlaud, and T. Gaidon. tion

non-

vector field estimation

estimation

resolution

involving

scheme”.

discontinuities Optical Engineeringj

“Mo-

in a multi7(32):1475-

1482, June 1993.

[5] B. Rouchouze, Figure 4. (1.) Horn and Schunk estimate @ and ~ quadratic. (r.) Robust regularized estimate ~ = @E~F, @ quadratic. Uniform object, uniform background.

laud.

P. Mathieu,

“Motion

estimation

T. Gaidon, based

and

on markov

M. Barrandom

fields”. In International Conference on Image Processing, volume 2, pages 270–274, November 1994.

[6] D.L. Tull and A.K. estimation

Katsaggelos. using robust entropic

ternational

Conference

pages 212-215,

on Image

October

“Regularized functional”. Processing,

motion In involume

3,

1995.

[7] L. Blanc-F6raud, M.

P. Charbonnier, G. Aubert, and “Non-linear image processing: model-

Barlaud.

ing and fast algorithm tection”.

In IEEE

age Processing,

Figure 5. Horizontal motion (block moving left to right) exposing anomalous background. CONCLUSION

6. Occluded defined,

regions, should

where the motion

not impact

is (instantaneously)

the motion

estimate

un-

where mo-

tion is well-defined. Most motion estimation algorithms, assumes that the DVF exists everywhere in a scene. This is assumption

often violated

for incorporatin~ bust

convex

DVF

new class of robust liminary

results

in practice.

In this work, a method

knowledge

of occluded

estimation

criterion

convex

measures

are promising

regions

into

was presented was derived.

but limited.

a ro-

and a

Our pre-

Improved

DVF


D. C., USA,

for regularization International

volume

October

with edge de-

Conference

1, pages 474–477,

on Im-

Washington,

1995.

[8] M.E. Zervakis,

A.K. Katsaggelos, and T.K. Kwon. “A class of robust functional for the enhancement of imIEEE Trans. on Image Processing, 3(14):752ages”. 773, June 1995.

[9] R. Depommier with detection

and

E. Dubois.

of occlusion

areas”.

“Motion estimation In Proceedings of the

Conference on Acoustics, Speech IEEE International and Signal Processing, pages 269–272, March 1992.

[10] S.L. Iu. “Robust

estimation

continuity y and occlusion

of motion

vector

with dis-

using out lier reject ion “. Jozw-

and Image nal of Visual Communication tion, 6(2):132-141, June 1995.


Representa-

REGULARIZED

ESTIMATION

PLACEMENT Damon

VECTOR

L. Tull

Department Northwestern Evanston,

and Aggelos

of Electrical

OCCLUDED

DIS-

K. Katsaggelos

and Computer

Engineering

University

Illinois

60208-3118

[email protected]. Occluded

OF

FIELDS

edu, [email protected]

regions

and

motion

boundaries

introduce

dis-

placement vector field (DVF) discontinuities that must be reconciled to accurately estimate image flow. In this work, the robust regularized estimation of the DVF is considered in the presence timation

of these discontinuities.

criterion

aries andallows A new class edge

for a globally of robust

preserving

gradient

is presented convex

regularization

is proposed

optimal

convex

motion

estimate

measures

of the DVF.

for managing

Results

for

weighted DVFdis-

using synthetic


es-

bound-

is introduced

and an occlusion

asmechanism

continuities due to occlusion. sequences are presented.

A robust

that preserves

image
