Frame rate up conversion via Bayesian motion ... - Semantic Scholar

Frame Rate Up Conversion Via Bayesian Motion Estimation Yue Wang*a, Siwei Ma b, Wen Gao b a Graduate University of Chinese Academy of Sciences, Beijing,100080, China; b Institute of Digital Media, Peking University, Beijing, 100871,China {wangyue,swma,wgao}@jdl.ac.cn

ABSTRACT In this paper, a novel block-based motion compensated frame interpolation (MCI) algorithm is proposed to enhance the temporal resolution of video sequences. We formulated motion estimation into MAP framework, and solved it via Bayesian belief propagation. By effectively incorporating a priori knowledge of the motion field and optimizing the whole motion field synchronously, it could derive more accurate motion vectors than traditional methods. Finally, adaptive overlapped block motion compensation (OBMC) is used to reduce blocking artifacts. Experimental results show that the proposed method outperforms other methods in both objective and subjective quality. Keywords: frame interpolation, frame rate up conversion, motion estimation

1. INTRODUCTION Frame interpolation, which is also referred to as frame rate up conversion (FRUC), is a video processing technique used in various applications. The most practical one is video format conversion [1]. For example, old television is typically filmed at 30 or 60 frames/s, but high definition television (HDTV) has a refresh rate of 60Hz or 120Hz. To play the old videos on the new display devices, we need to convert the lower frame rates to higher ones. Besides, frame interpolation can also be used in low bit rate video coding [2].With low bandwidth limits, one can send the full frame rate video at the cost of introducing annoying artifacts. Alternatively, we could reduce the frame rate by half so that each frame has better quality. In the latter case, we need to perform FRUC at the decoder to display the video at a full frame rate. Conventional FRUC methods usually used simple frame repetition or frame averaging. However, since these methods didn’t take motion information into account, the interpolated frame would have annoying artifacts such as motion jerkiness and blurring. To reduce these artifacts, motion compensated frame interpolation (MCFI) is widely adopted in recent FRUC methods. The most important thing to be considered in MCFI is the accuracy of estimated motion vectors, since it determines the quality of interpolated frames. Most MCFI methods utilize the block matching algorithm (BMA) to perform motion estimation (ME), since it’s simple and easy to implement. Although BMA has been used successfully in video compression, however, in FRUC the task of ME is to get the true motion field rather than minimizing the residue energy. So a lot of efforts have been made to improve the block-based motion estimation methods. Most of them are imposing a smoothness constraint on the motion filed, since it’s widely accepted that true motion field should be smooth except along motion boundaries. The most famous and successful method may be 3D-RS block matcher proposed by De Haan[3]. Rather than taking all the possible candidate vectors into account, 3DRS algorithm calculates on a small number of spatial and temporal prediction vectors from a 3-D neighborhood, which makes it efficient in yielding coherent vector fields. In [4], Zhai proposed to use overlapped motion estimation (OBME) to avoid falling into local optimal solution. Besides, many methods adopt various post-processing to correct the unreliable motion vectors after BMA. For example, MV-median filtering is widely used in many FRUC schemes, since it is effective in suppressing single outliers and preserving edges. In [5], Huang proposed to detect and correct unreliable motion vectors according to residual energy and MV correlation. However, motion vectors obtained by these methods are still unreliable in many cases. Although 3DRS or OBME takes motion coherence into account, however, the cost functions are still on a finite support around current block and they may be easily trapped into sub-optimal solutions when multiple motion vectors may lead to small SADs. Furthermore, in

Visual Communications and Image Processing 2010, edited by Pascal Frossard, Houqiang Li, Feng Wu, Bernd Girod, Shipeng Li, Guo Wei, Proc. of SPIE Vol. 7744, 77442L · © 2010 SPIE · CCC code: 0277-786X/10/$18 · doi: 10.1117/12.863363 Proc. of SPIE Vol. 7744 77442L-1

the area where complicated motion or deforming happens, there is usually a group of wrong MVs. In this case, neighboring candidates or median filtering will cause error propagation and make the case worse. In this paper, we present a true motion estimation method, which formulates motion estimation into MAP framework. We set up a global cost function for the whole motion field. A priori knowledge of the motion field is effectively incorporated into the cost function. Then the cost function is minimized by belief propagation. Finally, adaptive OBMC is used to reduce blocking artifacts of the area which contains complicated motions or deforming objects. It differentiates from previous methods by optimizing the whole motion field synchronously, thus local optimum could be avoided. Experimental results show that the proposed method outperforms others in both objective and subjective quality. The rest of the paper is organized as follows. In Section 2, the proposed algorithm for FRUC is presented. Experimental results and analysis are in section 3. Finally, section 4 concludes this paper.

2. PROPOSED METHOD 2.1 Bayesian formulating BMA usually takes sum of absolute difference (SAD) as the matching criterion. For the (i, j ) th block Bt 1/ 2 (i, j ) in the interpolated frame I t 1/ 2 , the bidirectional SAD of motion vector V is computed as:

SAD( Bt 1/ 2 (i, j ),V )

¦

I t ( X s  V )  I t 1 ( X s  V )

(1)

S Bt 1/ 2 ( i , j )

X s is the coordinates of pixel S , which is located within block Bt 1/ 2 (i, j ) . I t (