spatial error concealment based on directional decision and intra ...

Spatial Error Concealment Based on Directional Decision and Intra Prediction Yan Zhao

Dong Tian

Institute of Communication Engineering Jilin University Chanchun, China [email protected]

Tampere International Center for Signal Processing Tampere, Finland [email protected]

Miska M. Hannukasela

Institute of Signal Processing Tampere University of Technology Tampere, Finland

Moncef Gabbouj

Nokia Research Center Tampere, Finland Abstract—The paper presents a novel spatial error concealment algorithm based on directional decision and intra prediction. Unlike previous approaches that simultaneously recover the pixels inside a missing macroblock (MB), we propose to recover them 4×4 block by 4×4 block. Each missing 16×16 MB in an intra frame is divided into 16 blocks each with size 4×4 first, then recovered block by block using Intra_4×4 prediction. The previously-recovered blocks can be used in the recovery process afterwards. The principle advantage of this approach is the improved capability of recovering MB with edges and the lower computational complexity. The proposed algorithm has been tested on the H.264/AVC reference software JM7.2. Experimental results demonstrate the advantage of the proposed method.

I.

INTRODUCTION

Techniques for error control and error concealment must be employed when video communication on unreliable transmission environments such as wireless network and Internet. Error concealment by post-processing at the decoder is one of the important strategies for combating transmission errors in video communication. It can be applied in almost any application [1]. The latest international video coding standard, H.264/AVC provides a compression efficiency of 1.5 to 2 times over previous standards such as H.263+ and MPEG-4 Part 2 [2][3]. The coding algorithm of H.264/AVC is based on the same block-based motion compensation and transform-based spatial coding framework as prior video coding standards. Therefore, we focus on the spatial domain block-based error concealment by post-processing in this paper. Many spatial block-based error concealment algorithms have been proposed in literatures. The method of project-

0-7803-8834-8/05/$20.00 ©2005 IEEE.

onto-convex-sets (POCS) estimates the edge information by spatial and spectral domain constraint-satisfying iterative process firstly. Then, directional filtering is used for recovery of the missing MB [4]. Since the iterative process required by POCS is computationally expensive and difficult to implement in hardware, the multi-directional interpolation (MDI) algorithm is proposed further. MDI performs pixel domain interpolation along eight possible edge directions and considers the cases of both single edge and multiple edges [5]. The spatial error concealment algorithm for intra frame in current H.264/AVC reference software is based on weighted pixel averaging. The weight used for averaging is simply the inverse distance between the source and destination pixels [6]. It utilizes the smoothness property only, while ignores edge continuity to recover the missing MBs, therefore, it can not restore MBs with edges efficiently. In this paper we propose an error concealment approach, which can recover more detail in the missing MB when the surrounding pixels strongly suggest that one edge or multiple edges exists. Moreover, it is not computationally complex. II.

PROPOSED ALGORITHM

The missing 16×16 MB is recovered 4×4 block by 4×4 block in the proposed error concealment approach. The order of 16 blocks recovered in an MB is from the blocks with dependable available neighbours to the blocks without dependable available neighbours. There are mainly two steps to recover each 4×4 block. In the first step, the block is determined if it is a flat block or a block with one of the eight directional edges according to the estimation of surrounding available pixels. In the second step, the flat block is recovered using weighted pixel averaging and the block with

2899

missing MB. And the angle θ is rounded to the nearest 22.5 degree. Therefore, there are 8 different directions between 0 and 180 degree as shown in Fig.2. In order to determine if a 4×4 block is flat or with some direction of edge, eight counters C k , k = 0,1, " 7 , are used. The initial value of C k is set zero. For every pixel x(i,j) and all pixels (we use y(u,v) to represent any one of these pixels) in the block to be recovered, if y(u,v) satisfy:

Available Neighbor MB

x(i,j)

Missing MB

k ≤ ((arctan(( v − j ) /( u − i )) ∗ 180 / π ) / 22 .5) + 0 .5 < k + 1

(1)

k ≤ ((arctan(( v − j ) /( u − i )) ∗ 180 / π + 180 ) / 22 .5 )

(2)

or + 0 .5 < k + 1

Figure 1. Pixels to compute the gradient components

then C k plus 1/D, where D is given by:

135°

112.5

°

90°

67.5

D = (u − i ) 2 + ( v − j ) 2

°

45°

157.5°

(3)

If the largest counter value is below a certain threshold, there is no discernible edge and the block is classified to a flat block. Otherwise, the most likely direction of the edge in the block is the one with the largest counter value and the corresponding prediction mode of the block is determined by this direction. The value of threshold is different according to the number of available neighbor MBs.

22.5° 0° Figure 2. Eight edge directions

edge is recovered using Intra_4×4 prediction. A. Block Classification and Directional Decision The absence or presence of edges and the most likely direction of the edge, if present, of each 4×4 block in the missing MB are obtained by the use of gradient measures to three layers of pixels of the available adjacent MB as shown in Fig. 1. The local edge gradient components of the pixel x(i,j) in the second layer of the available adjacent MB is computed by: gx = xi +1, j −1 − xi −1, j −1 + 2 xi +1, j − 2 xi −1, j + xi +1, j +1 − xi −1, j +1

B. Block Concealment Each 4×4 block with edge in the missing MB is recovered by Intra_4×4 prediction which is similar to that used in H.264/AVC [2]. There are totally nine Intra_4×4 prediction modes in H.264/AVC. In addition to DC prediction mode, eight directional prediction modes correspond to the eight directions in Fig. 2. In our proposed method, we use eight directional Intra_4×4 prediction modes to recover each 4×4 block with edge according to directional decision in the first step. Each flat block is recovered by using the method of weighted pixel averaging. The number of each block in the missing MB is shown in Fig.3. Since the

g y = x i −1, j −1 − x i −1, j +1 + 2 x i , j −1 − 2 x i , j +1 + x i +1, j −1 − x i +1, j +1

which is equivalent to applying 3×3 Sobel operators: − 1 0 1  Sx = - 2 0 2  and Sy =   - 1 0 1 

1 2 1 0 0 0    - 1 - 2 - 1

4 5 6 7 12 8 9 13 10 11 14 15 0 2

to three layers of pixels in available adjacent MB as shown in Fig.1.

1 3

The angular of gradient at coordinate (i, j) is: θ = arctan ( g y /g x ).

The angular of gradient is computed for every pixel x(i, j) at the second layer of the neighbourhood surrounding the

2900

Figure 3. 16 blocks in the missing MB

X I J K L

A B a b e f i j m n

C c g k o

Fig.4 (a); the pixels in block 4, 5, 6, 7 will be recovered using pixels X, A, B, C, …, G, H as shown in Fig. 4 (b); the pixels in block 8, 9, 10, 11 will be recovered using pixels X, A, B, C, …, G, H as shown in Fig.4 (c); the pixels in block 12, 13, 14, 15 will be recovered using pixels X, A, B, C, …, G, H as shown in Fig. 4 (d).

D E F G H d h l p

(a) H G F E D a e i m

III.

C b f j n

B c g k o

AX d I hJ lK pL

(b) L K J I X

a b e f i j m n A B

c g k o C

d h l p D E F G H

(c ) a b c d L e f g hK i j k l J m n o pI H G F E D C B AX (d) Figure 4. Different scenarios of Intra_4×4 prediction: (a) up and left, (b) up and right, (c) left and down, (d) right and down

order of 16 blocks recovered in a MB is from blocks with dependable available neighbours to those without dependable available neighbours, the order of 16 blocks to be recovered will be changed according to the scenario of available neighbour MBs. For examples, if a missing MB has four available edge neighbour MBs, the order of 16 blocks to be recovered is 0, 5, 1, 4, 10, 15, 11, 14, 2, 7, 3, 6, 8, 13, 9, 12; if a missing MB only has up and right available edge neighbour MBs, the order of 16 blocks to be recovered is 5, 4, 1, 0, 7, 6, 3, 2, 13, 12, 9, 8, 15, 14, 11, 10. Furthermore, for each block from 0 to 15, the 16 pixels a, b, …, n, o, p should also be recovered by Intra_4×4 prediction from more reliable pixels X, A, B, C, …, G, H. In H.264/AVC, Intra_4×4 prediction is performed using pixels X, A, B, C, …, G, H, which locations are on the up and left side of the block to be recovered as shown in Fig.4 (a). In order to use more reliable pixels X, A, B, C, …, G, H to recover the pixels a, b, …, n, o, p in a block, we use other three scenarios: up and right, left and down, right and down in Fig.4 (b)-(d) to get them. For example, if a missing MB has four available edge neighbour MBs, the pixels in block 0, 1, 2, 3 will be recovered using pixels X, A, B, C, …, G, H as shown in

EXPERIMENTAL RESULTS

The proposed error concealment algorithm is implemented based on the H.264/AVC reference software JM7.2. Flexible macroblock ordering (FMO) is a new error resilient tool which is available in H.264/AVC baseline and extended profiles [7]. FMO allows assigning MBs to slices in an order other than the raster scan order. FMO is especially powerful in conjunction with post-processing error concealment when the samples of a missing slice are surrounded by many samples of correctly decoded slices. Therefore, we use FMO with two slice groups and three slice groups respectively in our simulations, and the FMO type is scattered slices [8]. The proposed algorithm is tested on frame 0 and frame 10 of foreman QCIF sequence. For U and V frames of the color image, the size of MB is 8×8, therefore, there are 4 blocks in each MB. The reconstruction of each block in the missing 8×8 MB of U or V frame is same as that of Y frame. In order to compare the proposed algorithm with the weighted pixel averaging (WPA) and multi-directional interpolation (MDI) methods, simulation results are also reported when using the two existed methods, which are also based on the H.264/AVC reference software JM7.2. Table I gives the comparison PSNR of Y frame of frame 0 and frame 10 for foreman sequence both in the cases of two slice groups and four slice groups. As can be seen, for foreman with more edges in the missing MBs, the proposed algorithm is able to obtain an average gain of 0.55 dB over MDI and 0.42 dB over WPA. Fig. 5 presents the visual quality of frame 0 of foreman image in the case of two slice groups: (a) is the original image, (b) is the recovered image without losses, (c) is the damaged image with lost MBs, (d) is the recovered image using our proposed algorithm, (e) is the recovered image using MDI, and (f) is the recovered image using WPA. Fig. 6 presents the visual quality of the first frame of foreman image in the case of four slice groups: (a) is the damaged image with lost MBs, (b) is the recovered image using our proposed algorithm, (c) is the recovered image using MDI, and (d) is the recovered image using WPA. As for visual quality, it can be seen that, besides providing a better PSNR, the proposed method also yields better subjective quality than MDI and WPA. As for computational complexity, we compare the decoding speed of using different error concealment methods (see Table II). The speed is evaluated on a 1.2GHz Pentium-4 PC. As can be seen, the proposed method has less computation complexity than MDI.

2901

TABLE I.

PSNR(dB) RESULTS OF THE PROPOSED ALGORITHM MDI AND WPA METHODS

COMPARING WITH

(a)

(c)

(e)

(b)

FMO

Frame

2 Slice Groups 4 Slice Groups

Frame0 Frame10 Frame0 Frame10

TABLE II.

Proposed method 22.72 23.34 22.83 22.05

WPA method 22.26 22.77 22.62 21.61

DECODING SPEED OF USING DIFFERENT METHODS

FMO

Frame

2 Slice Groups 4 Slice Groups

Frame0 Frame10 Frame0 Frame10

Proposed method 0.32s 0.36s 0.30s 0.45s

IV.

(d)

MDI method 22.48 22.88 21.68 21.72

MDI method 0.76s 0.73s 0.70s 0.70s

WPA method 0.17s 0.26s 0.18s 0.24s

CONCLUSIONS

In this work, we proposed a new spatial error concealment algorithm. It is based on block classification and directional decision as well as Intra_4×4 prediction which is similar to that used in the latest international video coding standard H.264/AVC. The algorithm is capable of reconstructing the missing MBs with edges more efficiently than weighted pixel averaging method and multi-directional interpolation algorithm in the case of using FMO with two and four slice groups. In order to be used in real time system, the computational complexity is also an important factor for an error concealment algorithm. It has been shown that the proposed approach has appropriate computational complexity.

(f)

Figure 5. Images in the case of two slice groups: (a) original, (b) recovered without losses, (c) damaged with lost MBs, (d) recovered by proposed method, (e) recovered by MDI, (f) recovered by WPA.

REFERENCES [1] Y. Wang and Q-F Zhu, “Error Control and Concealment for Video [2]

(a)

[3]

(b)

[4] [5] [6]

(c)

[7]

(d)

Figure 6. Images in the case of three slice groups: (a) damaged with lost MBs, (b) recovered by proposed method (c) recovered by MDI, (d) recovered by WPA.

[8]

2902

Communication: A Review,” Proc. of the IEEE , vol.86, no.5, pp974991, 1998. “Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T Rec. H.264/ISO/IEC 14496-10 AVC,” in Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VCEG, JVT-G050, 2003. T. Wiegand and G. Sullivan, “The emerging JVT/H.26L video coding standard,” presented at the Tutorial at ICIP 2002, Rochester, NY, Sept.2002. Huifang Sun and Wilson Kwok, “Concealment of Damaged Block Transform Coded Images Using Projections onto Convex Sets,” IEEE Trans. Image Processing.,vol.4, no.4, pp470-477, Apr. 1995. Wilson Kwok and Huifang Sun, “Multi-directional interpolation for spatial error concealment,” IEEE Trans. Consumer Electron., vol. 39, pp455-460, Aug. 1993. Ye-Kui Wang, M. M. Hannuksela, Viktor Varsa, Ari Hourunranta and Moncef Gabbouj, “The Error Concealment Feature in the H.26L Test Model”, Proc. of ICIP 2002, pp. II-729 - II-732. S. Wenger, “H.264/AVC over IP,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, pp. 645-656, July 2003. S.Wenger and M. Horowitz. (2002) Scattered Slices: A New Error Resilience Tool for H.264. [Online]. Available: ftp://ftp.imtcfiles.org/jvt-experts/0202_Gen/JVT-B027.doc