spread field, a motion field and a restored image in an ... of the blur in order to restore the scene. ... train in a direction dictated by the displacement vector.
REGULARIZED
BLUR-ASSISTED Damon
DISPLACEMENT
L. Tull and Aggelos Northwestern
Department
of Electrical Evanston,
ABSTRACT
ESTIMATION
K. Katsaggelos
University and Computer
Illinois
damonQece.nwu.edu,
FIELD
Engineering
60208-3118 aggk@ece.nwu.edu
a regularized
solution
approach.
tion required some knowledge
The problem
formula-
of the point spread sup-
Due to the finite acquisition time of practical cameras, objects can move during image acquisition, therefore introducing motion blur degradations. Traditionally, these degradations are treated as undesirable artifacts that should be removed before fllrther processing. In this work, we consider the use of motion blur as an indication of scene motion. JVe present two robust regularized motion estimation algorithms that consider the use of (motion) blur in their formulation. The first algorithm uses motion blur as prior knowledge for the estimation of the motion field. The second algorithm
complete knowledge of the blur point spread function in order to restore the scene. The blur was derived from
considers the joint estimation of the motion and motion blur. Each approach results in a motion blur point
an estimate of the image motion of that frame using a constant velocity assumption. The (motion) blur point
spread field, a motion field and a restored image in an approach that is different from previous work. Prelim-
spread field was modeled
inary results are presented.
field (DVF)
port.
In their most recent work, You and Kaveh in [2]
estimated the support of the blur function using a pruning algorithm which gave promising results in practical image scenes. In [1], we considered fication
and restoration
the spatially
algorithm
train in a direction
identi-
of blurs caused by object
tion during image acquisition. age restoration
variant
A robust regularized
was proposed
impulse
by the displacement
at each pixel.
im-
which required
as a two dimensional
dictated
mo-
The support
vector
of the motion
blur PSF at each pixel was assumed to be proportional 1.
to the extent of the motion at each site. This constant velocity (CV) assumption proved to be useful in a wide
INTRODUCTION
It is common to observe blur due to motion in the frames of an image sequence. If any point on an ob,ject
variety of degraded
passes over two or more photoreceptors
tivation.
acquisition, a blur will occur. single class in a host of possible
during
image
}Iotion blur is only a image distortions. The
In both between
the work of You image sequences
treated
as an undesirable
artifact
moved before further processing. a spatially
varying
degradation
blur has been
that should Motion
be re-
blur is often
as in the case of fast
was the mo-
restoratio~) and propose the use of blur in the estimate of image motion. We propose an extension of
ple degradations.
motion
[1, 3].
[2], image restoration
In this work, we consider the relationship motion and motion blur (identification and
ability to classify degradations is an important step towards the restoration of image sequences with multiTraditionally,
image sequences
[1] and
and Kaveh
to the case of dynamic
where motion
is defined.
CV motion assumption developed DVF to a parameterized (motion) the scene.
A bilinearly
interpolated
Using the
in [1], we relate the blur field (PBF) for motion blur model
moving objects [1]. Most image restoration algorithms require knowledge of the blur in order to restore the
is used to provide a discrete approximation tinuous PBF. Two regularized displacement
scene. The identification varying case is a difficult
of the blur in the spatially problem addressed by only a
algorithms are proposed. Each algorithm is based on a robust regularized formulation that allows disconti-
few researchers. In recent work by You and Kaveh [2], blur identification is considered in the context of a blind deconvolution approach where a restored image and the blur
nuities in both the DVF and the PBF. In the first algorithm, motion information, obtained from the PBF, is incorporated as “cue” or a priori knowledge of the motion
field using a recently
point spread function
robust
convex
are simultaneously
estimated
in
stabilizers.
proposed
of the conestimation
entropic class of
The entropic
class of stabi-
lizers allows
the incorporation
of prior
knowledge
of the
structure of the DVF into the estimation procedure, in a non-linear manner similar to the maximum informa[4]. The second algorithm
tion principle
combines
mo-
tion estimation, blur identification and image restoration in an approach that represents a joint estimation of image motion. In the following section, we discuss a robust regularized motion estimation formulation and an iterative
solution
approach.
Section
3 describes
a
data fidelity energy while ~D measures the smoothness of the DVF estimate. Selecting dD and #D to be convex measures allows the optimal regularized estimate of the DVF to be obtained
v~+l(m,
n) =
from the iteration, ~QD(ub,
v~(m, n) – v
vb,
~)
dv~(m, n)
where v~+l (m, n) is the updated
estimate
‘
(3)
of the hori-
simultaneous blur identification and image restoration algorithm following [2]. Section 4 present the first of
zontal component of the motion field at iteration i and q is the relaxation parameter to ensure proper conver-
two algorithms
which utilize
gence of the iteration.
tion estimation
criterion.
algorithm
in Section
the DVF.
Section
blur in the mo-
the joint
image restoration
mation problem.
Our proposed
by some preliminary
from the
3 is used as prior knowledge
5 describes
blur identification.
tion 6. Section
motion
The PBF obtained
formulation
and motion
approach
esti-
in motivated
results which are presented
7 concludes
of of
The
solution
to obtain
motion
fields in the presence of motion
in Sec-
our discussion.
3.
A REGULARIZED MOTION ESTIMATION ALGORITHM
The goal of most DVF estimation
DFDk(m.
(coding
problem
based)
A REGULARIZED
In [2], a solution
formulations
of the
is to minimize,
BLUR ALGORITHM
to the identification
problem
the restored
n) = - UD(Tmn), n
- VD(m,
in the
case of spatially variant blur is proposed. The formulation incorporates piecewise smoothness of the PBF and image in the identification
anisotropic diffusion. The minimization ing functional is proposed,
fk(m, n) - f~-l(?n
field
discontinuities via a robust convex selection of $. We adopt this approach for the subsequent analysis.
IDENTIFICATION 2.
for the vertical
component of the motion is obtained by replacing v with u in Eq. (3). This iteration was used in [5, 6, 7]
process
using
of the follow-
n))(l) MN
where f~ (m, n) is the pixel intensity at discrete location (rn,n) at image frame k. components
of the DVF
respectively.
Finding
The vertical
and horizontal
are uD (m, n.) and ~D(m, n),
the motion
field that minimizes
the DFD at each pixel location is an ill-posed problem in that there are many motion fields that minimize the DFD.
To make the minimization
posed problem, smoothness,
prior knowledge,
is imposed
of Eq.
(1) a well-
often a desired level of
on the solution.
For an LI x N
image, a well-posed or regularized estimate of the DVF can be obtained through the minimization of the cost function, h[ QD(UD,
IID, A) =
where.
AF, ~}, Afi are the regularization
parameters
and uE, uE are the vertical and horizontal components of the PBF. The functions @F, OF, QB are the identification
fidelity, restored
smoothnem
image smoothness
measures respectively,
bust and convex (i.e., log(cosh(x) of the current estimate
denoted
and PBF
all selected to be ro) ). The residual error
by e(m, n), is given by,
N
~
~
rn=l
71=1
?bD(DFD~(m.
e(m, n) = g(m, n) – h(m, n) * f(m, n)
n))
where g(m, n) is the observed
MN
(5)
image, * denotes convo-
lution and h(m, n) is the response of the degradation filter at (m,n). In this work, h(m, n) is chosen to model where V~D(m,
n) and V~D(m.
dient of the horizontal
7L) are the Spatial gra-
and vertical
components
of the
motion field, respectively. The constant A E [0, m) is the regularization parameter which controls the tradeoff between original
smoothness
(A = m)
data (~ = O). The function
and fidelity
CV motion blur such that the pixel location tains the value given by,
to the
@D measures
the
h(m, n) * f(m, n) =
(m, n) ob-
where
~1 = l/L and L is the
mined
by
length
of the
blur
cleter-
This criterion is convex for fixed prior knowledge, allowing for efficient optimization. The optimal solution of this criterion
L = /UB(~.~)2
+ uB(~..
n)2.
(7)
The robust selection of the measures @B in Eq. (4) allows for edge preserving regularization similar to the anisotropic diffusion mechanism described in [8]. The vertical and horizontal components of the PBF (UB, ~B )
the DVF
structure
cedure is detailed
MOTION
in [2].
A COOPERATIVE
Incorporating
AS A MOTION
prior knowledge
ill-posed problem
is common
forming
the problem
ditional
regularization
a way of making for trans-
explicitly
through
problem.
incorporates
the addition
Tra-
smooth-
of a regular-
taneous
spatially
constraint.
varying
The result is a simul-
motion
and blur estimation image.
By selecting
@B = @D to be a robust, convex functional, and using the CV constraint to express the PBF in terms of the DVF, we obtain t,he robust PBF-DVF criterion, QBD(UD, hf
can introduce prior knowledge about the solution in a non-linear manner. It is called the robust minimum (R.MIAI) measure
ALGORITHM
scheme that results in a restored
ization term like 4D (.) in Equation (2). In [4], Zervakis et al. proposed a robust, convex functional that
absolute-information},
of is,
of the DVF and the PBF may be combined
using the CV motion
about the solution in an
into a well-posed
ness of the solution
PRIOR
That
BLUR-ASSISTED
ESTIMATION
Estimation
BLUR
the PBF.
proach to incorporating prior knowledge has been recently shown to be effective in image restoration [4, 3]. 5.
MOTION
by prior knowledge
from
features such as discontinuities and edges in the PBF will be imposed on the solution of the DVF. This ap-
are obtained from the alternating minimization of Eq. (4) between the restored image j and the PBF. This pro-
4.
is guided obtained
~JD,f,~)
=
N
~~~(e(m,n))
~
+ @D(DFD(m,n)) +
m=l n=l
defined by,
+~-1)
~B(l
[@D (l~UD(m,n)/)
+ @D (l~vD(m,~)l)]
(11)
+~F#F([~f(m,n)l)
Since @ and d are convex where c is a small constant zero and T represents
used to avoid division
prior or target knowledge
by
of the
estimate Z. The exponential constant e is used to scale the functional so that the optimal solution would be Z. This functional was shown to be a powerful means of introducing
prior knowledge.
rate the PBF as a motion the DVF by invoking straint [l],that
that this criterion
is also a function
6.
velocity
(CV)
RESULTS
JVe are concerned ~’UD
=
of a, which relates
~uD,
AND
DISCUSSION
contion.
UB
(9)
with degradations
The assumptions
motion
be of constant
proper
idefitification,
to this point velocity
time.
Physically,
represents
related to the camera
a is a global
the fraction
image
of the DVF
by an intensity point during practical cameras a ~ [0, 1).
image
Replacing ~ with of in Eq. (2)gives prior criterion,
exposure
parameter
distance
that
traveled
acquisition.
For
the robust PBF-
of blur is a common quences.
occurance
Figure 1 illustrates
related impose
to mothat the
and translational.
via motion
estimation,
must persist for at least two frames. where a is a constant
of
and ~. Note
In order to incorpo-
the constant
=
the DVF,
the motion to the PBF. This constant can be found iteratively, solving the motion blur identification problem in the process.
prior, we relate the PBF to
is. ‘UB
in Eq. (11) the iteration
Eq. (3) can be used to obtain
For
the blur
The persistence
in practical
image se-
the degradation
caused by
a moving block over a uniform background. The length of the DVF is 10 pixels and the length of the PBF is 5, which corresponds to a = 0.5. Figure 2 shows spatially variant restoration this object
using the algorithm
using the DVF
of Section
3 of
and using the CV assump-
tion. k~ QPRIOR(UD,
VD, ~,~)
=
+~D@I(l~uD(m,
+ADdI(luvD(m,
N
~ ~ 4(DFD~(m,n)) ln=l rz=l n)l, ~-ll~UB(m,
n)l, a-11vVB(m,7L)[).
7. ~)l)
(lo)
CLOSING
REMARKS
In this work, we consider the use of blur information in the regularized formulation of the motion estimation
Figure 2: Spatially a by product
variant restoration
of using the algorithm
In ICASSP,
variant blurs. 2610. IEEE,
of motion
blur.
of Section 3
volume
4. pages 2607–
May 1995.
[3] D .L. Tull and A.K. Katsaggelos. Iterative restoration of fast, moving objects in dynamic image sequences. Figure
1:
Synthetic
example
moving left to right.
of motion
blur.
Optical
Engineering,
To appear.
1997.
Block
DVF length = 10, PBF length =
5. effective a = 0.5
[4] LM.E. Zervakis, A.K. Katsaggelos. and T.K. Kwon. A class of robust functional for image restoration.
IEEE Trans. on Image Process i?t,q,3(14] :752-773, June 1995.
problem.
We proposed
t~vo algorithms
for this purpose.
The first approach models the role of motion blur as being simply a cue or a “stimulus”. providing prior structural information for motion estimation. Our approach differs from previous work for example, Chen et al. [9], in that we consider tion approach explicitly second
accounts
proposed
or cooperative counts
a robust regularized
to the motion
estimation
for the source of motion algorithm
estimate
for (motion)
represents
that
blur.
Our
a simultaneous
of the image motion
along with additional
fields.
Conference on Image ProIn International volume 2, pages 270–27’4. November 1994.
cessing,
[6] D .L. Tull and A.K. tion estimation In International
Conference
[7] D .L. Tull and A.K.
Results
In IEEE
International
Systems,
Atlanta,
Regularized
entropic on
volume 3, pages 212–215,
timation
results will
Kat,saggelos,
using robust
that ac-
blur in the formulation.
for both approaches
solu-
problem
[5] B. Rouchouze, P. Mathieu, T. Gaidon, and \I. Barlaud. Motion estimation based on markov random
Image
October
Regularized
displacement Conference
Proce,ssi~ig,
1995.
Katsaggelos.
of occluded
mo-
functional.
vector on
es-
fields.
Circuits
and
GA, May 1996.
be shown during our presentation. [8] .J. Malik and P. Perona. tection using anisotropic 8.
REFERENCES
actions gence,
[1] D .L. Tull and A .1
