recursive map displacement field estimation and ... - Semantic Scholar

RECURSIVE MAP DISPLACEMENT FIELD ESTIMATION AND ITS APPLICATIONS James C. Brailean1 1

Motorola, Chicago Corporate Research Laboratories Schaumburg, IL 60196-1078, USA. 2

Northwestern University, Dept. of ECE Evanston, IL, 60208-3118, USA.

ABSTRACT In this paper we briefly describe some of our work on the use of stochastic models to describe the displacement vector field (DVF) in an image sequence. Specifically, autoregressive models are used which describe the abrupt transitions in the DVF with the use of a line process, but also result in spatio-temporally recursive structures. The use of such models in developing maximum a posteriori estimators for the DVF and the line process is subsequently described. Finally, the extension and application of the resulting estimator to the problems of object tracking, video compression and restoration of video sequences is briefly reviewed. 1.

Aggelos K. Katsaggelos2

INTRODUCTION

In the past several years, applications utilizing digital video have increased dramatically. For instance, the pervasive use of video conferencing seen today is a direct result of the advances in video processing as well as the development of video coding standards. By utilizing digital video, compression algorithms enable a reduction in required bandwidth over analog video systems. In addition, through the use of application specific integrated circuits, the cost of hardware required to implement these algorithms is greatly reduced. Other applications that have benefited by the use of digital video include, but are not limited to, multimedia services, autonomous navigation, motion analysis, object tracking, surveillance, astronomical and medical imaging. In conjunction with this increased use of digital video has been the increased demand for advanced motion estimation algorithms. These algorithms are crucial for most if not all of the video processing applications. For instance, the effectiveness of a video compression algorithm is dependent upon its ability to reduce the amount of temporal redundancy present in a video sequence. Accurate motion estimation greatly enhances a compression algorithm’s ability to reduce this temporal redundancy. Another obvious example of an application dependent on accurate motion estimation is object tracking. In this case, the movement of a particular object or objects is tracked by the estimation of their apparent motion through the video scene. Currently, there are many different approaches to estimating the motion within a video sequence. These approaches can be classified into two distinct categories. The first category, called feature-based motion estimation, consists of approaches which attempt to extract 3-D motion parameters from a sparse set of 2-D features. Generally, due to the complexity of the problem, these approaches are utilized in application in which the objects are predetermined (computer vision). The second category, called optical flow-based motion estimation, consists of approaches which compute the instantaneous velocities of the intensity values on the image plane. A relatively dense flow field or displacement vector field (DVF) is estimated. The DVF represents the apparent motion of 3-D objects on the 2-D image plane. Generally, it is the optical-flow based motion estimators which are used in applications such as compression and image sequence restoration.

In this paper, we review a particular optical flowbased displacement field estimation algorithm, which we have recently developed, called the recursive coupled linearized maximum a posteriori (RCLMAP) estimator. The RCLMAP estimator is a recursive model-based algorithm which locates the MAP estimate of the DVF from two consecutive frames of a noise-free, noisy, or noisy-blurred image sequence [1, 2]. To model the DVF we use a nonstationary vector field model called the Vector Coupled Gauss-Markov (VCGM) model. The VCGM model consists of two levels: an upper level, which is made up of several submodels with various characteristics, and a lower level, or structure model, which governs the transitions between the submodels. This model-based MAP estimator simultaneously segments and estimates the DVF. This segmentation combined with accurate estimates of the DVF, allows the use of this algorithm in different applications. Specifically, in this paper we discuss the usefulness of the RCLMAP algorithm in applications such as, object tracking, motion compensated video compression, and image sequence restoration. 2. PROBLEM FORMULATION The problem of estimating the motion between consecutive frames of video can be formulated in essentially the same way for the applications discussed in this paper. That is, let fk (~ r ) denote the image intensity of the k-th frame at the spatial location ~r, where ~r = [m, n]T , 1 ≤ m ≤ M, 1 ≤ n ≤ N , and T denotes the transpose of a vector or matrix. The displacement field {d(~r) = [dx (~r), dy (~r)]T } is a vector field which maps points in the current frame fk , to their corresponding locations in the previous frame fk−1 . We impose the commonly used assumption that the image intensity is constant along the motion trajectory [3], which translates to fk (~ r ) = fk−1 (~ r − d(~ r )).

(1)

That is, knowledge of the true displacement vector allows for the perfect prediction of the next frame from the current one. It is apparent from Eq. (1) that the relationship between the DVF and the intensity field is a nonlinear one. Generally, in the development of a recursive motion estimator a linear relationship between the observed intensity values and the DVF is required. Using a Taylor series expansion of fk−1 (~r−d(~r)) about the location (~r−d◦ (~r)), where d◦ (~r) denotes an initial estimate of d(~r), we obtain fk−1 (~ r − d(~ r ))

=

fk−1 (~ r − d◦ (~ r ))

+

∇T fk−1 (~ r − d◦ (~ r ))(d(~ r ) − d◦ (~ r )) − eL (~ r ), (2)



T

∂ ∂ where ∇ = ∂x , ∂y represents the spatial gradient operator and eL is the truncation or linearization error term. Substituting fk−1 (~r − d(~r)) from Eq. (1) into Eq. (2) we obtain the following expression for the displaced frame difference (DFD)

∆(~ r, ~ r − d◦ (~ r ))

=

fk (~ r ) − fk−1 (~ r − d◦ (~ r ))

=

∇T fk−1 (~ r − d◦ (~ r ))(d(~ r ) − d◦ (~ r )) − eL (~ r ). (3)

We can interpret the DFD as the error due to the nonlinear temporal prediction of the intensity field. It is this error that motion estimation algorithms attempt to minimize. The direct minimization of the mean square error results in the steepest-descent class of PR algorithms [3]. Minimizing the linearized form of the DFD with respect to (d(~r) − d◦ (~r)) results in an updated estimate of the displacement vector. Therefore, an estimate of d(~r) results from knowledge of the DFD (temporal gradient) and the spatial gradients. The truncation error was ignored in earlier algorithms, while it was treated as a sample of a random field in [4, 5]. In the MAP estimator we propose, we use both the linearized and the original expression for the DFD and assume that the error term eL (~r) is a Gaussian random variable with variance γ . However, with the incorporation of the VCGM model and MAP criterion, the resulting estimator has a significantly different form from the above mentioned algorithms. RECURSIVE MAP DISPLACEMENT FIELD ESTIMATOR 3.1. Vector Coupled Gauss-Markov (VCGM) Model The modeling of random fields potentially improves the robustness of the estimation process. By incorporating a model that utilizes the correlations of the random field, an initial estimate can be obtained which minimizes a prediction error function. Autoregressive (AR) models have been used for such a reason in the estimation of both the intensity as well as the displacement fields [5]. However, neither the image intensity nor the DVF are modeled properly by a linear shift invariant (LSI) model. The major problem with using an LSI model for the DVF estimation process, is that the boundaries separating homogeneous regions become over-smoothed or blurred. The VCGM model is based on a multi-level Markov random field approach, where the lower layer or line process is used to represent the boundaries or structure of the random field and thus prevent the errors associated with the oversmoothing of these boundaries. In the development of our stochastic characterization of the line process, we incorporated many of the a priori assumptions presented in [6, 8]. However, we further expanded upon these characterizations in order to accommodate a causal region of support required by the recursive estimator, while also increasing the level of sophistication of the line processes. Also in developing these characterizations, we utilize the fact that there exists a correspondence between the line process and the edges in the intensity field. For instance, from the rigid body (perfect registration) assumption we know that boundaries in the displacement field should correspond to edges in the intensity field. However, the converse may not necessarily be true. The VCGM model [1], as the name implies, results from the coupling of several Vector Gauss-Markov (VGM) models with a line process. It is given by

that indicate the presence or absence of a boundary. With the knowledge of ld (~r), a model is chosen that excludes the points that are separated by the boundary. The resolution of the orientation is limited to 45, 90, 135, and 180 degrees, due to the use of