Phase Correlation-Based Motion Estimation Using Variable Block Sizes for Frame Rate Up-Conversion Suk Ju Kang1, Dong Gon Yu2, and Young Hwan Kim3 Division of Electrical and Computer Engineering, Pohang University of Science and Technology Nam-Gu, Pohang, Kyungbuk 790-784, Republic of Korea Tel: +82-54-279-2878, Fax: +82-54-279-5933 E-mail:
[email protected],
[email protected],
[email protected]
Abstract: Consisting of motion estimation and motion compensated interpolation, the performance of frame rate upconversion highly depends on the motion estimation method it uses. This paper evaluates the performance-complexity trade-offs of the phase correlation method for motion estimation, obtained for various ratio of the number of candidate peaks to the number of total peaks. In addition, this paper presents a phase correlation method that uses variable block sizes. In experimental results, the proposed method outperformed other benchmarks in terms of the performance over complexity.
correlation method has difficulty in recognizing the movement of fine objects within a block. In this paper, Section 2 describes the proposed method. Section 3 illustrates the trade-offs between performance and complexity by the change of C, and presents the effectiveness of the proposed method. Finally, Section 4 concludes the paper.
2. Variable Block-Size Phase Correlation Method The proposed method, variable block-size phase correlation method (VBSPCM), is a combination of the phase correlation method and the variable block-size method.
1. Introduction Motion blur is a serious problem in hold-type displays such as Liquid Crystal Displays (LCDs). Two of the effective methods to reduce motion blur are scanning backlight and black data insertion, which imitate the driving method of the impulse-type display [1]. These methods, however, reduce luminance significantly and are impractical in many applications. Another effective method for reducing motion blur is displaying more frames than the original one during the same time period. This is called frame rate upconversion [2]. Compared to the previous methods, this method has the advantage of maintaining luminance, while reducing motion blur. For the frame rate up-conversion, the additional frame images are created using motion estimation and motion compensated interpolation, as shown in Figure 1. The performance of the frame rate up-conversion considerably depends on the motion estimation method it uses.
Figure 2. Correlation window in the phase correlation method
Figure 1. Motion estimation and motion compensated interpolation The phase correlation method became one of the popular motion estimation methods because of its high efficiency in terms of the performance over complexity [3]. The performance and complexity of the phase correlation method greatly depend on the ratio (C) of the number of candidate peaks to the number of total peaks. In this paper, we present the trade-off relationship between the performance and complexity when C is changed. We also propose a new phase correlation method which uses variable block sizes. Note that the conventional fixed block-size phase
In the phase correlation method, cross correlation refers to the process to find a correlation window, which is a set of peaks related with motion vectors between blocks of two successive frame images. Figure 2 shows an example of the cross correlation result. The motion vector of the target block is obtained from the position of the high peak, shown in Figure 2. But practically there are many high peaks within a block of a real image. In that case, the number of candidate peaks affects the performance and complexity in the phase correlation method. In this paper, we evaluate the relationship between the above two factors when the ratio (C) of the number of candidate high peaks to the number of total peaks is changed. The proposed VBSPCM also uses the variable block sizes instead of the fixed block size, which is used by the conventional phase correlation method. The variable block-size method can recognize the movement of fine objects within a block, which the fixed block-size mehtod may miss [4]. Specifically, the proposed method uses the quad-tree structure and the bottom-up approach; Four blocks at the current level are merged if the sum of absolute difference (SAD) of four blocks is higher than the SAD of one block at the upper level.
the average PSNR, the complexity of the proposed method is higher than that of PCM as expected, but it is lower than that of BMM. Conditions
Complexity
BMM
Block size=NⅹN Search parameter=N
4N4
PCM
Block size = NⅹN
24N2log2(2N)+4CN4
VBSCPM
Block size = 1/2Nⅹ1/2N, NⅹN
24N2(2log2N+1) +5CN4
Table 1. Complexity comparison
29.7
4
1
0
0.5
1
x 10
5
Block Size=16
3 2 1 0
0
C 6
x 10
6
0.5
1
C
Block Size=32
4
BMM PCM VBSPCM
2
0
0.5
1
C
29.5
PSNR(dB)
Block Size=8
2
0
29.6
4
3
0
Complexity value
Foreman 29.8
x 10
Complexity value
4
Complexity value
3. Experimental Results To evaluate the performance and complexity of the proposed method, we used two popular motion estimation methods as benchmarks: block matching method (BMM) and phase correlation method (PCM). In BMM, we used the block size of 16ⅹ16 pixels and the search parameter of 16 pixels for the experiment [5]. In PCM, we used the block size of 16ⅹ16 pixels. Finally, we used two levels for the proposed method, and their block sizes were 8ⅹ8 and 16ⅹ16 pixels. Foreman and News were used for test sequences. The experiment was performed for the frame rate up-conversion, and we examined the relationship between the performance and complexity when C was changed in the phase corelation method. We also evaluated the performance and complexity of the proposed method and benchmark methods. For 71 consecutive frame images for a moving image, we removed 35 odd frame images and constructed 35 new odd frame images from 36 even frame images. Then, we computed the 35 PSNRs of the constructed odd frame images with respect to the original odd frame images. And, we obtained the average of these 35 PSNRs for various C values. Figure 3 indicates that the average PSNR is almost saturated when C is 0.5. At this point, the average PSNR of the proposed method is 0.56 dB and 0.49 dB higher than those of the PCM and BMM.
Figure 4. Variation of complexity with respect to C
29.4 29.3
4. Conclusion
29.2 29.1 29
BMM PCM VBSPCM
28.9 28.8 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
C
News 34.2 34 33.8
This paper investigated a new motion estimation, based on phase correlation, which uses variable block sizes. First, we investigated the trade-off relationship between the performance and complexity for various values of C, the ratio of the number of candidate peaks to the number of total peaks for phase correlation. We then evaluated the performance and the complexity of the proposed method and benchmarks. In experimental results, the average PSNR is saturated at C=0.5. At this point, the average PSNR of the proposed method is 0.56 dB and 0.49 dB higher than those of the PCM and BMM respectively, and the complexity of the proposed method is lower than that of BMM.
PSNR(dB)
33.6
Acknowledgement
33.4
This work was supported by IDEC, BK21 and LG PHILIPS LCD Company.
33.2 33 32.8
References
32.6
BMM
32.4
PCM VBSPCM
32.2 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
C
Figure 3. Average PSNRs for Foreman, News sequences We computed the complexity of the proposed and benchmark methods for various C values. Table 1 shows the complexity of each method and Figure 4 compares the complexity for the fixed block size (NⅹN). In Figure 4, the complexity of the proposed method is lower than that of BMM when C is under 0.6 and the block size is 16ⅹ16 pixels, which is the value used in the above experiment. Specifically, when C is 0.5, the saturation point for
[1] K. Sekiya and H. Nakamura, “Eye-trace integration effect on the perception of moving pictures and a new possibility for reducing blur on hold-type displays,” SID ’02 Digest, pp. 930-933, 2002 [2] T. Kurita, "Moving picture quality improvement for hold-type AM-LCDs," SID'01 Digest, pp.986–989, 2001. [3] C. Stiller and J. Konrad, “Estimating motion in image sequences,” IEEE Signal Proc., Vol. 16, pp. 70–91, July 1999. [4] I. Rhee, G.R. Martin, S. Muthkrishnan, R. A. Packwood, “Quadtree-structured variable-size block-matching motion estimation with minimal error,” IEEE Trans. on Circuits and Systems for Video Technology, Vol. 10, pp. 42-50, Feb. 2000. [5] F. Dufaux and F. Moscheni, “Motion estimation techniques for digital TV: A review and a new contribution,” Proc. IEEE, Vol. 83, pp. 858-876, June 1995.
