Status Update: Adaptive Motion Estimation Search Range - Google Sites

Motion Vector Search Window Prediction in MemoryConstrained Systems Chung-Cheng (Roy) Lou Szu-Wei (Wesley) Lee C.-C. Jay Kuo 8/3/2009

Outline     

Introduction Review of Previous Work Proposed Search Window Decision Algorithm Experimental Results

Conclusion

8/21/2009

Media Communication Lab

2

Complexity of Video Encoding  Motion estimation accounts for the most complexity • 33% of CPU time • 90% of memory access DCT 3.3% Others 5.4%

Entropy Coding 4% Quant 7% Other 7% Recon 7%

Entropy Coding Deblocking 0.8% 0.1%

Motion Estimation 33%

Rate Control 8% Interpolation Motion Comp 13% DCT 9% 12%

CPU usage*

Motion Estimation 90%

Memory Access**

*Z. He, Y. Liang, L. Chen, I. Ahmad, and D. Wu, “Power-rate-distortion analysis for wireless video communication under energy constraints,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 15, 2005, pp. 645-658. **K. Denolf, C. Blanch, G. Lafruit, and A. Bormans, “Initial memory complexity analysis of the AVC codec,” IEEE Workshop on Signal Processing Systems, 2002.(SIPS'02), 2002, pp. 222227.

8/21/2009

Media Communication Lab

3

Search Window Decision  Directly achieve the memory access reduction  Two factors need to be considered jointly • Motion Vector Predictor • Search Range

8/21/2009

Media Communication Lab

4

Previous Work: Motion Vector Predictor  Spatial* • Median (mva, mvb, mvc or mvd )

 Temporal* • mv0, mv1, … mv8

* Chalidabhongse, J. and Kuo, C. C. J., “Fast motion vector estimation using multiresolution-spatio-temporal correlations,” Circuits and Systems for Video Technology, IEEE Transactions on 7(3), 477–488 (1997).

8/21/2009

Media Communication Lab

5

Previous Work: Motion Search Range  Motion Vector * • SR = Maximum of neighbor motion vectors

 MV Predictor ** • SR = Default SR / 4 + |Spatial MVP – Temporal MVP|

 Rate-Distortion Cost *** • Cost ↑, search range ↑ • SR by comparison between R-D cost of coded MB and pre-setup thresholds

*Chen, Z., Liu, Q., Ikenaga, T., and Goto, S., “A motion vector difference based self-incremental adaptive search range algorithm for variable block size motion estimation,” in [Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on], 1988–1991 (2008). **Chen, Z., Song, Y., Ikenaga, T., and Goto, S., “A macroblock level adaptive search range algorithm for variable block size motion estimation in H.264/AVC,” in [Intelligent Signal Processing and Communication Systems, 2007. ISPACS 2007. International Symposium on], 598–601 (2007). ***Paul, A., Wang, J., and Yang, J., “Adaptive search range selection for scalable video coding extension of H.264/AVC,” in [TENCON 2008 2008, TENCON 2008. IEEE Region 10 Conference], 1–4 (2008).

8/21/2009

Media Communication Lab

6

Problems of Previous Works  MV Predictor and Search Range need to be considered jointly. • Precise MV Predictor → Smaller search range

 Search range decision is not controllable.

Histogram of MV centered with Spatial MVP 8/21/2009

Histogram of MV centered with (0,0) MV

Media Communication Lab

7

MV vs. Spatial MVP vs. Temporal MVP  MVDspat = MV – Spatial MVP  MVDtemp = MV – Temporal MVP X-Component in 3D

Y-Component in 3D

X-Component in 2D logarithmic magnitude MVDtemp

MVDtemp

Y-Component in 2D logarithmic magnitude

MVDspat

MVDspat

Histogram of MVDspat vs. MVDtemp, “Football” test sequence, CIF, 260 frames 8/21/2009

Media Communication Lab

8

Analysis of Histogram Between MVPs (MV – MVP ) – (MV – MVPspat) = 0 Small variance →temp small SR → MVPtemp = MVPspat

Histogram along (MVPspat = MVPtemp)

MVDtemp

(MV – MVPtemp) – (MV - MVPspat) = -15 → MVPspat – MVPtemp = -15

X-Component

MVDspat

Large variance → Large SR High correlation between - deviation of predictor candidates - search range

Histogram along (MVPspat - MVPtemp = -15) 8/21/2009

Media Communication Lab

Variance under different (MVPspat - MVPtemp) 9

Proposed MV Predictor Selection  Spatial Candidate • Ns = {mva~mvd}

 Temporal Candidate • Nt = {mv0~mv8}

 MVP = Mean(Ns∪Nt)  Var(Ns∪Nt) is also calculated. • Deviation among candidates

8/21/2009

Media Communication Lab

10

Search Range Decision  By modeling as distribution.  Connect search range and probability of reaching optimal MV.

 Distributions: • Cauchy, Laplacian, Gaussian • Modified Cauchy* f MC ( x, y )  f MC , X ( x)  f MC ,Y ( y ) f MC , X ( x)  f MC ,Y ( y ) 

Cx x x

5/3

1

Cy y y

5/3

1 Histogram of MV centered with Spatial MVP

*Tsai, J. and Hang, H., “Modeling of Pattern-Based block motion estimation and its application, Circuits and Systems for Video Technology, IEEE Transactions on 19(1), 108–113 (2009).

8/21/2009

Media Communication Lab

11

Search Window Prediction Algorithm 1. Decide MV Predictor • Mean(Ns∪Nt)

2. Find the deviation among candidates • Var(Ns∪Nt)

3. Decide distribution parameter 4. Decide search range f MC , X ( x) 

• SR = min (SRmin + SRdyn, SRdefault) • Probability target • Search window size target

Cx x x

5/ 3

1

Var(Ns∪Nt)

0

Mean(Ns∪Nt) Search Range

8/21/2009

Media Communication Lab

12

Experimental Setups

8/21/2009

Standard

H.264 / AVC Baseline Profile

Ref. Software

JM 15.1

Frame Size

QCIF

Type of Frame

IPPP…

QP values

22, 28, 32, 36, 40

# of Ref-frames

1

Inter modes

16x16

ME Strategy

fast full search

Default Search Range

16x16

RD optimization

Off

Media Communication Lab

13

Result: Rate / Distortion / Complexity

PSNR drop < 0.5 dB 92.2% reduction of search window size

Rate - Distortion Curve

Search Window Size - Distortion Curve

“Foreman” test sequence, QCIF, 300 frames Fixed SR: Fixed search range with spatial predictor. Modified Cauchy + Probability = 80%: SR decision with spatial predictor Mean + Modified Cauchy + Probability = 10%: MVP and SR decision jointly. 8/21/2009

Media Communication Lab

14

Result: Previous Works

“Foreman” test sequence, QCIF, 300 frames [A] Chen, Z., Liu, Q., Ikenaga, T., and Goto, S., “A motion vector difference based self-incremental adaptive search range algorithm for variable block size motion estimation,” in [Image Processing, 2008. ICIP 2008. 15th IEEE International Conference on], 1988–1991 (2008). [B] Chen, Z., Song, Y., Ikenaga, T., and Goto, S., “A macroblock level adaptive search range algorithm for variable block size motion estimation in H.264/AVC,” in [Intelligent Signal Processing and Communication Systems, 2007. ISPACS 2007. International Symposium on], 598–601 (2007). [C] Paul, A., Wang, J., and Yang, J., “Adaptive search range selection for scalable video coding extension of H.264/AVC,” in [TENCON 2008 2008, TENCON 2008. IEEE Region 10 Conference], 1–4 (2008). [D] Li, G. L. and Chen, M. J., “Adaptive search range decision and early termination for multiple reference frame motion estimation for h. 264,” IEICE Transactions on Communications , 250–253 (2006).

8/21/2009

Media Communication Lab

15

Summary  A search window decision algorithm considering MV predictor and search range jointly.

 A search range decision algorithm with controllable complexity. • Probability of reaching optimal motion vector • Search window size

 92.2% of search window size reduction with the same quality.

8/21/2009

Media Communication Lab

16

8/21/2009

Media Communication Lab

17

Backup Slides

8/21/2009

Media Communication Lab

18

Search Range Decision Algorithm 1. 2. 3. 4.

Model MVD histogram as distribution function. Find sample variance from previous MVD. Compute distribution parameter. Decide search range by complexity targets. • Probability of reaching ideal MV • Search point f

MC , X ( x) 

Cx x x

5/ 3

1

Sample Variance

Origin

Spatial MV Predictor Search Range

8/21/2009

Media Communication Lab

21

Probability based Search Range Decision  Object • Decide search range based on probability of reaching optimal MV.

 FMC ( x, y )  FMC , X ( x)  FMC ,Y ( y ) FMC , X ( x)  

SRx  0.5

 SRx  0.5

FMC ,Y ( y )  

SR y  0.5

 SR y  0.5

f MC , X ( x)dx f MC ,Y ( y )dy

 X / Y decided individually

illustration of SR decision based on probability 8/21/2009

Media Communication Lab

22

Search Point based Search Range Decision  Object • Maximize cumulative probability under search point constrain.

 Ideal Solution • Search point boundary is tangent to contour of cumulative probability.

 Approximate Solution

ideal search range

• By steepest descent in X/Y

search range (7,6)

direction.

illustration of SR decision based on search pts 8/21/2009

Media Communication Lab

23

Result: SR Decision Constrains

“Foreman” test sequence, QCIF, 300 frames

8/21/2009

Media Communication Lab

24

Result: More Sequences

“Crew” test sequence, QCIF, 150 frames

“Football” test sequence, QCIF, 130 frames 8/21/2009

Media Communication Lab

25