Rate-Distortion Optimized Motion Estimation for ... - Semantic Scholar

Rate-Distortion Optimized Motion Estimation for Error Resilient Video Coding Hua Yang and Kenneth Rose Signal Compression Lab ECE Department University of California Santa Barbara, USA Mar. 2005

Outline ‰ Motion estimation (ME) for coding efficiency – Conventional ME – Rate-constrained ME & rate-distortion (RD) optimized ME

‰ Motion estimation for error resilience ‰ Proposed end-to-end distortion based RDME – Intuition behind – End-to-end distortion analysis

‰ Simulation results ‰ Conclusions Mar. 2005

ICASSP 2005

2

Motion Estimation for Coding Efficiency ‰ Motion compensated prediction (MCP) – To remove inherent temporal redundancy of video signal – Both the motion vector and the prediction residue are encoded.

Coded frame n-1

Mar. 2005

Original frame n

ICASSP 2005

3

Motion Estimation for Coding Efficiency ‰Conventional motion estimation – ME Criterion: minimize prediction residue min D res = min mv

mv

∑ (f

i∈ MB

i n

− fˆni−+1mv

)

2

• Ignoring the motion vector bit-rate cost

Mar. 2005

ICASSP 2005

4

Motion Estimation for Coding Efficiency ‰ Motion estimation in low bit rate video coding – In low bit rate video coding, motion vectors may occupy a significant portion of total bit rate. – Efficient bit allocation between motion vector and prediction residue coding is necessary for better overall coding efficiency. – Rate-constrained motion estimation min [D res + λ ⋅ R mv ] mv

λ: Lagrange multiplier

• However, not yet the ultimate rate-distortion optimization for the best overall coding performance.

Mar. 2005

ICASSP 2005

5

Motion Estimation for Coding Efficiency ‰ Motion estimation for low bit rate video coding (cont’d) – Rate-distortion optimized motion estimation (RDME) min

{ mv , QP }

D =

[D + λ ⋅ (R mv

∑ (f

i∈ MB

i n

− fˆni

+ R res + R header

) ≅ ∑ (e 2

i∈ MB

i n

− eˆ ni

)

2

)]

= DQ

– Some references • • • • Mar. 2005

[Girod `94] Theoretical analysis of rate-constrained ME [Sullivan `98] Summary of rate-constrained ME [Chung `96] Low complexity RDME for each MB using RD modeling [Schuster `97] Joint RDME for multiple MB’s ICASSP 2005

6

Motion Estimation for Error Resilience ‰ In the presence of packet loss: – Packet loss & error propagation • Internet – no QoS guarantee Wireless – inherent error-prone channel • Error propagation due to MCP

– Error resilient video coding • RD optimization with end-to-end distortion • Coding mode selection: {Intra/Inter, QP}

No mv for Inter-mode!

‰ Error resilience via motion compensation – Multi-frame motion compensation (MFMC) [Budagavi `01] – Reference picture selection (RPS) [H.263+] – Error resilient rate-constrained ME [Wiegand `00] Not comprehensively attack the RD optimization problem! Mar. 2005

ICASSP 2005

7

Motion Estimation for Error Resilience ‰We propose end-to-end distortion based RDME [accounting for packet loss]

&The exact RD optimal ME solution for error resilience &Critical: accurate pixel-level end-to-end distortion estimation • Build on: recursive optimal per-pixel estimate (ROPE) [R. Zhang, S. Regunathan, and K. Rose `00]

Mar. 2005

ICASSP 2005

8

Proposed RDME ‰Intuition for “error resilience via ME” P1

I

P2

P3 P1 P2 P4 P1 For coding efficiency

I

For error resilience

Best trade-off

' Conventional motion estimation completely ignores the error resilience information. – This error resilience information should be exactly considered for each pixel.

Mar. 2005

ICASSP 2005

9

Proposed RDME ‰ROPE-based end-to-end distortion analysis E{D} =

∑

i∈ MB

=

~ E {( f ni − f ni ) 2 } =

∑ [(1 − p ) ⋅ E {( f

i∈ MB

i n

~i ~i 2 i 2 i [( f ) − 2 f ⋅ E { f } + E {( f n ) }] ∑ n n nError concealment

i∈ MB

~ ~ − f ni−+1mv − eˆ ni ) 2 } + p ⋅ E {( ROPE f ni − f ni−1 ) 2 }

~  ≅ (1 − p ) ⋅  ∑ E {( fˆni−+1mv − f ni−+1mv ) 2 } +  i∈ MB = (1 − p ) ⋅ D EP + (1 − p ) ⋅ D Q + p ⋅ D EC

∑ (e

i∈ MB

i n

− eˆ ni

]

)  + p ⋅ D 2



EC

Error propagated distortion – DEP is explicitly affected by mv, whose minimization favors mv’s that point to reference areas with less encoder-decoder mismatch.

Mar. 2005

ICASSP 2005

10

Proposed RDME ‰ The proposed RDME solution min

{ mv , QP }

[E { D } + λ ⋅ (R mv

+ R res + R header

)]

  λ ≅ min  DEP + DQ + ⋅ (Rmv + Rres + Rheader ) { mv ,QP} 1− p   Packet loss impact

– Comparing with existent RDME • Source coding distortion ⇒ end-to-end distortion • mv affects not only the Rmv vs. Rres trade-off, but also more importantly, the coding efficiency vs. error resilience trade-off.

– Comparing with existent RD optimized coding mode selection • Extended Inter mode with the mv parameter • Further optimize the Inter-mode performance Mar. 2005

ICASSP 2005

11

Simulations ‰Objective: to check upper-bound performance – Joint {mv, QP} optimization – RD calculation via actual encoding

‰Simulation settings – – – – –

UBC H.263+ Encoding: I-P-P-…… Transmission: independent packet loss, with a uniform p Decoding: 50 different packet loss realizations for each p Performance: average luminance PSNR

Mar. 2005

ICASSP 2005

12

Simulations ‰Simulation settings (cont’d) – Testing methods • Conventional ME (cME) • The proposed RDME (RDME)

– Testing scenarios • Random Intra updating (rI): arbitrarily assigns MB’s to 1/p groups, and cycles through them updating one group per frame.

• Optimal Intra updating (oI): RD optimized Intra/Inter mode selection.

Mar. 2005

ICASSP 2005

13

Simulation Results  Random Intra 40

30

cME RDME

39 38

28 27

36

PSNR (dB)

PSNR (dB)

37

35 34

26 25 24

33

23

32

22

31 30

cME RDME

29

0

5

10

15

20

25

30

21

0

5

10

15

20

Packet loss rate (%)

Packet loss rate (%)

Miss_am

Foreman

25

30

PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s]

Mar. 2005

ICASSP 2005

14

Simulation Results  Optimal Intra 40

30 RDME-rI cME-oI RDME-oI

38

28

37

27

36 35

26 25

34

24

33

23

32

22

31

0

5

10

15

20

25

RDME-rI cME-oI RDME-oI

29

PSNR (dB)

PSNR (dB)

39

30

21

0

5

10

15

20

Packet loss rate (%)

Packet loss rate (%)

Miss_am

Foreman

25

30

PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s]

Mar. 2005

ICASSP 2005

15

Simulation Results  Random Intra 38

26.5 cME RDME

37

cME RDME

26 25.5 25

PSNR (dB)

PSNR (dB)

36

35

34

24.5 24 23.5 23

33

22.5 32

31 30

22 40

50

60

70

80

90

100

110

120

21.5 30

40

Total bit rate (kb/s)

50

60

70

80

90

100

110

120

Total bit rate (kb/s)

Miss_am

Foreman

PSNR vs. Total bit rate [QCIF, 10f/s, p=10%]

Mar. 2005

ICASSP 2005

16

38

27.5

37.5

27

37

26.5

36.5

26

PSNR (dB)

PSNR (dB)

Simulation Results  Optimal Intra

36 35.5 35

25 24.5

34.5

24

RDME-rI cME-oI RDME-oI

34 33.5 30

25.5

40

50

60

70

80

90

100

110

RDME-rI cME-oI RDME-oI

23.5

120

23 30

40

Total bit rate (kb/s)

50

60

70

80

90

100

110

120

Total bit rate (kb/s)

Miss_am

Foreman

PSNR vs. Total bit rate [QCIF, 10f/s, p=10%]

Mar. 2005

ICASSP 2005

17

Simulation Results

Conventional ME [29.58dB]

RDME [33.83dB]

Miss_am: QCIF, 10f/s, 48kb/s, p=10%, random Intra

Mar. 2005

ICASSP 2005

18

Simulation Results

RDME [26.92dB]

Conventional ME [23.92dB]

Foreman: 1st 200f, QCIF, 10f/s, 112kb/s, p=10%, random Intra

Mar. 2005

ICASSP 2005

19

Conclusions ‰Identify the new opportunity of achieving error resilience via motion estimation. ‰Propose an RD optimal ME solution, which further optimizes the Inter-mode performance. ‰Investigate the upper-bound performance. – With random Intra: substantial gain – With optimal Intra: significant gain at low bit rates.

‰Besides Intra updating, RDME presents another good alternative for error resilience. Mar. 2005

ICASSP 2005

20

Conclusions ‰Originally, the power of Intra coded MB’s is only recognized as stopping past error propagation, while the proposed RDME reveals their new potential on reducing future error propagation. ‰Future work I: more comprehensive tests – Inaccurate p, bursty loss, or over actual networks, etc.

‰Future work II: complexity reduction – RD modeling, separate mv and QP optimization, sophisticated ME strategies, etc.

Mar. 2005

ICASSP 2005

21

References ‰ [Girod `94] B. Girod, ``Rate-constrained motion estimation,'' Nov. 1994. ‰ [Sullivan `98] G. J. Sullivan and T. Wiegand, ``Rate-distortion optimization for video compression,’’ Nov. 1998. ‰ [Chung `96] W. C. Chung, F. Kossentini, and M. J. T. Smith, ``An efficient motion estimation technique based on a rate-distortion criterion,'' May 1996. ‰ [Schuster `97] G. M. Schuster and A. K. Katsaggeslos, ``A theory for the optimal bit allocation between displacement vector field and displaced frame difference,'' Dec. 1997. ‰ [Budagavi `01] M. Budagavi and J. D. Gibson, ``Multiframe video coding for improved performance over wireless channels,'' Feb. 2001. ‰ [H.263+] ITU-T, Rec. H,263, ``Video codeing for low bitrate communications'', version 2 (H.263+), Jan. 1998. ‰ [Wiegand `00] T. Wiegand, N. Farber, K. Stuhlmuller and B. Girod, ``Error-resilient video transmission using long-term memory motion-compensated prediction,'' June 2000. ‰ [Zhang `00] R. Zhang, S. L. Regunathan, and K. Rose, ``Video coding with optimal intra/inter mode switching for packet loss resilience,'' June 2000.

Mar. 2005

ICASSP 2005

22

The End

Mar. 2005

ICASSP 2005

23