Recovering Connected Error Region Based on Adaptive Error ...

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

683

Recovering Connected Error Region Based on Adaptive Error Concealment Order Determination Xueming Qian, Guizhong Liu, and Huan Wang

Abstract—Parts of compressed video streams may be lost or corrupted when being transmitted over bandwidth limited networks and wireless communication networks with error-prone channels. Error concealment (EC) techniques are often adopted at the decoder side to improve the quality of the reconstructed video. Under the conditions of a high rate of data packets that arrives at the decoder corrupted, it is likely that the incorrectly decoded macro-blocks (MBs) are concentrated in a connected region, where important spatial reference information is lost. The conventional EC methods usually carry out the block concealment following a lexicographic scan (from top to bottom and from left to right of the image), which would make the methods ineffective for the case that the corrupted blocks are grouped in a connected region. In this paper, a temporal error concealment method, adaptive error concealment order determination (AECOD), is proposed to recover connected corrupted regions. The processing order of an MB in a connected corrupted region is adaptively determined by analyzing the external boundary patterns of the MBs in its neighborhood. The performances, on several video sequences, of the proposed EC scheme have been compared with those obtained by using other error concealment methods reported in the literature. Experimental results show that the AECOD algorithm can improve the recovery performance with respect to the other considered EC methods. Index Terms—Boundary matching algorithm, error concealment, error concealment order determination, external boundary pattern, H.264, packet loss, video transmission.

I. INTRODUCTION HE basic idea of video coding standards, e.g., MPEG-1/2/4, H.261, H.263, H.26L, and H.264/AVC, is to reduce the spatial and temporal redundancy in video sequences, and the discrete cosine transform (DCT) is often adopted to transform spatial domain residual video signals into frequency domain coefficients, where much more coefficients are zeros after quantization. Entropy encoding methods, e.g., the variable length coding (VLC), are further adopted to remove the statistical redundancy [1]. The coded bit streams are sensitive to errors and losses. Video decoder may not parse the

T

Manuscript received February 20, 2008; revised December 05, 2008. First published April 28, 2009; current version published May 15, 2009. This work was supported in part by the National 973 Project No. 2007CB311002, in part by the National Natural Science Foundation of China (NSFC) under Project No. 60572045, and in part by the Ministry of Education of China Ph.D. Program Foundation under Project No. 20050698033. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Ling Guan. The authors are with the School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an 710049, China (e-mail: [email protected]; [email protected]; [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TMM.2009.2017609

corrupted code words even though only a few bits are damaged until the next resynchronization codeword is identified. Parts of bit streams may be corrupted by noise over the wireless communication networks with error-prone channels and some packets may be lost over the bandwidth limited networks when congestion happens. These situations would make the visual quality of the reconstructed video seriously degraded at the receiver side if no postprocessing techniques are adopted [2]. In order to enhance the robustness of video bit streams against errors, the error resilience (ER) and the error concealment (EC) techniques are often adopted. The ER techniques are carried out at the encoder side by adding some redundant information to the coded bit streams. They are preventative approaches designed to improve the compressed bit streams against packet losses and errors at the cost of reducing the coding efficiency [3]–[6]. The EC techniques are adopted at the decoder side to recover the corrupted MBs and improve the reconstructed video quality. As a video sequence has usually strong spatial and temporal correlation, the corrupted MBs can be approximated from the information of the neighboring MBs in spatio-temporal domain [7]–[11]. The spatial error concealment is adopted for the erroneous macro-blocks (EMBs) of the frames located at shot boundaries, according to the usual smoothness characteristics of adjacent spatial pixels, where no temporal information is available. The spatial domain methods usually replace the missing pixels by weighted averages of the pixels of the respective adjacent MBs [12], [46] or interpolate the missing pixels according to the estimated edge orientations [13], [14], [33]. Temporal EC methods estimate lost motion vectors (MVs) for the corrupted MBs according to the temporal redundant information [2], [15]–[23], [43]. The EMB can be recovered by replacing the corrupted MB with spatially corresponding MBs in the reference frames according to the estimated MV. The MV for each EMB can be substituted by the average (or medium) MVs of its neighborhood [15], [44]. Kim et al. estimated MVs for the EMB according to the motion vector and block mode information for H.264 compressed video [23]. A global motion based error concealment method is proposed by Su et al. [24], [25]. The MVs of the corrupted MBs are generated from the global motion parameters which are estimated from motion vector field. Better recovery results and visual qualities are achieved for the EMBs undergoing global motion. However, if a corrupted MB is located in local motion regions, the recovered result would not be so satisfactory. So, in our previous works, an adaptive global motion and local motion determination based error concealment method was proposed [48]. Boundary matching algorithms are used to find the best matched regions in the reference frame to conceal EMBs due to the stronger correlation of the block border pixels belonging to

1520-9210/$25.00 © 2009 IEEE Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

684

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

Fig. 1. Slice group pattern information. (a) Box-out, (b) wipe scan, (c) raster scan, (d) foreground and background scan, (e) interleaved FMO, and (f) dispersed FMO.

external MBs [26]. Moreover, the external boundary matching methods [38], [39] have been proposed to estimate the lost motion vector of each EMB by using its external boundary pixels in the top, bottom, left and right adjacent MBs as reference and searching for the best match within a given search range in the temporal reference frames [40], [41], [49]. In video coding standards, each picture is usually partitioned into several slice groups as shown in Fig. 1, such as the arbitrary slice ordering (ASO) [35]–[37] and the flexible macroblock ordering (FMO) [37]. From Fig. 1, we find that macroblocks in each slice in the box-out, wipe, raster, foreground and background, and interleaved FMO are connected. In the flexible macro-block ordering pattern, MBs in each slice group are dispersed throughout the picture. In H.264/AVC the dispersed FMO technique is advised to protect the video bit stream from transmission errors [1], [34], [37]. The dispersed FMO technique aims at maximizing the external MB number in its 4-connected neighborhood. The available external MBs in the 4-connected neighborhood of an EMB as shown in Fig. 1(f) will provide valuable information for its recovery. However, the excellent error resilience performances achieved by the dispersed FMO technique are at the cost of decreasing the video coding efficiency. In the previous video coding standards, such as MPEG-1/2 and MPEG-4 part 2, dispersed FMO technique is not adopted. In these cases as shown in Fig. 1(a)–(e), the corrupted macro-blocks are connected. The recovery results for the EMBs are not satisfactory due to the fact that there are only a small amount of correctly received macro-blocks available. In error concealment, the corrupted macro-blocks are usually recovered in raster scan pattern. It is likely that the left and top boundary blocks of the current macro-block are not original macro-blocks but recovered macro-blocks [17], [27]. It is obvious that the recovery of one MB is influenced by the previous

recovered macro-blocks in its neighborhood. If the neighboring erroneous macro-blocks are recovered with larger matching errors, then this will do harm to recover the succeeding ones. This phenomenon is called as a recovery dependency problem in the slice layer [28]. In order to overcome the drawbacks of the recovery dependency problem, some iterative boundary estimation methods have been proposed [28], [29]. Chen et al. proposed a recursive block matching (RBM) technique to recover the EMBs [29]. They made full use of the information of the correctly received MBs and/or of the recovered erroneous MBs in the neighborhood of a corrupted MB. The bottom 8 16 block in the top-adjacent MB and the upper 8 16 block in the bottom-adjacent MB are used as references. The recovery dependence is removed due to the fact that the left and right side blocks are not used. However, the error concealment results are not satisfactory when the upper and bottom-adjacent 8 16 blocks are located in a region with periodically appearing texture pattern. In H.26L, the error concealment orders of the EMBs in a corrupted region are from the external to inner [45]. It is included within the non-normative H.264/AVC decoder as the temporal error concealment method which is specified by JVT [46]. These processing orders are very different from a raster scan orders and help to prevent the error propagation. The method prevents a typical concealment error that is made in the usually “difficult” center part of the frame from propagating to the “easy” parts of the frame, because more correctly received and recovered MBs provide valuable information for recovering these EMBs. The processing starts with MB columns at the frame boundaries and then moves inwards column-by-column, which can prevent error propagation under the assumption that the external EMBs can be recovered “easily” with small matching errors. This is not true in many real video sequences. What would happen if

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

QIAN et al.: RECOVERING CONNECTED ERROR REGION

the left-most and right-most columns are localized in smooth regions while the center columns are localized in texture rich region? In our opinion, if an error appears in the external columns, this error would have higher probabilities to be propagated to the central column. Under this circumstance, the central column rather than the external columns can be “accurately” or “easily” recovered. The reason is that the “easy” or “difficult” should be determined by the external patterns (i.e., texture intensity) but not by the spatial layout [49]. In this paper, we propose an adaptive error concealment order determination for the error macro-blocks in a connected region. The error concealment orders for the macro-blocks are determined according to the external macro-blocks pattern information rather than the conventional raster scan mode. The rest of this paper is organized as follows: In Section II, the proposed error concealment method with adaptive error concealment order determination is described in detail. In Section III, comparisons with other existing EC methods and some discussions are given. And finally conclusions are drawn in Section IV. II. PROPOSED ERROR CONCEALMENT METHOD WITH ADAPTIVE ERROR CONCEALMENT ORDER DETERMINATION In this paper, the external boundary matching (EBM) criterion is used to estimate the optimal motion vector for recovering a corrupted macro-block [33], [38], [39]. Fig. 2 shows a corrupted MB and its 4-connected neighboring MBs. Let and denote the size of an MB and the external boundary width, respectively, usually. The external boundary blocks in the top, where bottom, left and right MBs are denoted Rt, Rb (with sizes ), Rl and Rr (with sizes ), respectively. Let denote the map of the MB located at , as is defined by if MB if MB if MB

is an un-recovered EMB is a correctly received MB is a recovered EMB (1) where is a weighting factor. It represents the reliability of the external boundary block provided for recovering the EMB in its neighborhood. In the following of this paper we call it conmeans that the th MB is fidence factor. an un-recovered EMB which does not provide any information

685

Fig. 2. Corrupted MB and its 4-connected neighboring blocks.

for recovering the other erroneous macro-blocks in its neighbormeans that the hood at the current time. recovered EMB provides the same reference information as the implies that the recorrectly received MB. A with covered EMB does provide some reference information which is not as reliable as the correct MBs. In the following part of this paper, detailed discussion of the effect of this parameter to the recovery performance will be given (see Section III for detail). and with denote Let the radius of search region and the corresponding displacement with the top-left coordinate , refrom the MB spectively. The best match of external boundary is determined by the criterion of mean absolute difference (MAD) of the corresponding blocks in its 4-connected neighborhood between , the current and the reference frames. Let , , and denote the MADs of the top-, bottom-, left-, and right-adjacent blocks between the reference and current frames in the pixel domain respectively, i.e., (2)–(5) at the bottom of the page, and are the frame indices of the current and refwhere as well as denote erence frames, and

(2)

(3)

(4)

(5)

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

686

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

Fig. 3. Error concealment orders of the EMBs in a connected region and the corresponding recovery results: (a) a corrupted frame with a connected EMB region, (b) EMB region and its neighboring MBs, (c) error free region, (d)–(g) different error concealment orders, and (h)–(k) recovery results of the external boundary matching based error concealment according to the error concealment orders given in (d)–(g).

the pixel values at of the current frame and the reference denotes the boundary width as shown frame. Hereinafter, in Fig. 2. The total MAD of the 4-connected neighborhoods, , is calculated as follows: denoted

(6) Correspondingly the best match with displacement is determined by (7) According to , error concealment is carried out. In the following of this section, detailed description of how to determine the error concealment order of an MB in a connected region is given by analyzing the patterns of its external boundary blocks Rt, Rb, Rl, and Rr. Fig. 3 shows an example which illustrates the importance of the processing order of the erroneous macro-blocks in a connected error region. Fig. 3(a) shows a video frame with connected error region, where 15 MBs are corrupted. The corrupted local region as shown in Fig. 3(a)

is enlarged and shown in Fig. 3(b). The corresponding error free region of Fig. 3(b) is shown in Fig. 3(c). The error concealment orders for the EMBs with raster scan order, randomly determined EC order, H26L order [45], [46], and the AECOD order are shown in Fig. 3(d)–(g), respectively, where the numerals represent their relative error concealment order. The external boundary matching method with boundary pixel width is used to estimate the motion vector for each EMB. The corresponding recovery results in terms of the above error concealment orders are shown in Fig. 3(h)–(k), respectively. Obvious artifacts are produced in the inclined edge region with those error concealment orders shown in Fig. 3(d)–(f), where the PSNR values are 34.34, 34.83, and 36.49, respectively. However, the EC result with the error concealment order in Fig. 3(g) outperforms the others, where edges are recovered with higher accuracy and the PSNR value is 37.31. By comparing the recovery results of those error concealment orders, we find that more accurate matching can be achieved and the recovery dependency problem can be removed by recovering the EMBs with larger external boundary gradient intensity or more texture information with certain priority. The adaptively determined EC orders are not only valuable for finding the optimal motion vector for the EMB but also helpful for decreasing the influence of recovery dependency problem. In the following part of this section we will describe how to determine the error concealment order by analyzing the external boundary patterns. We use

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

QIAN et al.: RECOVERING CONNECTED ERROR REGION

687

the standard deviation, gradient intensity and gray-level co-occurrence matrix features to represent the external block patterns. The EC performances of different external block patterns representation methods are compared and analyzed in detail in Section III. Now, we use the standard deviation of a block to represent denote the texture intensity of a its texture intensity. Let with sizes or , which is calculated block as follows:

(8)

(9) and denote the pixel value at the point where and the average pixel value of the block, respectively. In the proposed AECOD based error concealment, the pro, is determined by its excessing order of a corrupted MB of the blocks in its 4-conternal boundary pattern nected neighborhood, as is defined by

(10) It is likely that the larger the the more accurate the MB is recovered and the more reliable reference is provided to recover the other erroneous macro-blocks in its neighborhood, as shown in Fig. 3. Hence we adaptively determine the processing orders of the EMBs in a connected error region. Assume that the error concealment order has been determined for each of the EMBs in a given connected error region. For the current EMB, error concealment is carried out by using the external boundary matching criteria [33], [38], [39]. It is assumed that the reference frames are error free or already recovered before error concealment of the current frame. The detailed steps of the proposed AECOD based error concealment method consists of the following steps as shown in Fig. 4(a). In the first step the bit stream is decoded and the EMBs are determined. We assign the map information of a correctly received MB “1” and that of an EMB “0” according to (1). In the second step, we determine the connected error region by morphological labeling (in 4-connected) in the map image. Then, the error concealment is carried out for the connected error regions. For each connected error region, the detailed block diagram of the proposed AECOD based error concealment is shown in Fig. 4(b), which consists of the following steps. Step 1) Determine the EMB number in the connected error region. for the un-recovered EMBs by (10). Step 2) Calculate of the EMBs in descending order. Step 3) Sort the Step 4) Carry out error concealment for the EMB with the by finding an optimal replacement in largest

Fig. 4. Block diagram of the proposed error concealment method based on adaptive error concealment order determination. (a) Block diagram of the proposed AECOD based error concealment for recovering a corrupted frame. (b) Block diagram of the AECOD based error concealment for recovering a connected error region.

a specified search range from the reference frame using the external boundary matching criterion as shown in (6). Step 5) Update the map information of the recovered EMB from “0” to “ ”. This operation has the results that this EMB is recovered now and it will provide some clues for recovering the remaining EMBs in its neighborhood. Step 6) The Steps 2–4 are repeated until all the erroneous macro-blocks in this connected error region is recovered. For the connected error region shown in Fig. 3(b), the processing order of the proposed AECOD is shown in Fig. 3(g) and the corresponding recovery result is shown in Fig. 3(k). III. EXPERIMENTAL RESULTS AND DISCUSSION AECOD is a decoder independent method like the BMA [17], EBM [33], [38], [39], RBM [29], and H26LOD [45], [46] methods. They can be embedded into the decoders of MPEG-1/2/4, H.26l, H.26L, and H.264/AVC. In this section, eight video sequences, namely football, carphone, coastguard, mobile, flower, mother and doctor (denoted as M&D), foreman, and table tennis are used to evaluate the performance of the proposed AECOD based error concealment. The test video sequences are encoded by the MPEG-4 Momusys [30] main profile with the coding structure IBBP, and global motion compensation (GMC) with the sprite number 4, the rate control method is MPEG-2 TM5, the group of pictures (GOP) size is 12, and the quantization parameters for I, P and B frames are all the same with . The detailed information, including frame sizes, target bit-rates, and average PSNR values of the

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

688

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

TABLE I DETAILED INFORMATION OF THE TEST VIDEO SEQUENCES

TABLE II COMPARISON OF AVERAGE PSNR VALUES FOR THE TEST VIDEO SEQUENCES WITH DIFFERENT EC METHODS AT PER 10% AND 15%

eight test video sequences are listed in Table I. Moreover, these video sequences are encoded by H.264/AVC encoder [47] and error concealment are carried out to the corrupted video streams to show the effectiveness of the proposed EC method and its decoders independence. In our experiment, we consider a system in which each slice (group of macro-blocks) is coded and packed into one source packet [31], [32], and every packet is independently decoded. We assume that the packets (slices) get lost or corrupted independently with a certain packet error ratio (PER) and then error concealment is carried out for the corrupted video frames. Five error concealment methods are compared with each other to elucidate the advantages of AECOD: 1) BMA, the boundary matching algorithm used in [17]; 2) EBM, the external boundary matching method used in [33], [38], and [39] with boundary pixel width ; 3) H26LOD used in H.26L [45] and the H.264/AVC [46]; 4) RBM, the recursive block matching method in [29]; and 5) the proposed AECOD based error concealment with , where the matching criterion is in terms of EBM with boundary pixel width . Since the main contribution of this paper is based on the concealment order determination, for simplicity, we set the value of each of the recovered block to be a constant instead of a variable value. The search ranges are 15 for all the BMA, EBM, RBM, H26LOD, and AECOD. Note that the frame copy technique is embedded in all the compared EC methods. Therefore, when all the packets of a frame are lost or corrupted, all the above EC methods carry out directly by copying the previous frame to recover the lost frame.

A. Performance Comparison With Other Error Concealment Methods In the experiments the 10% and 15% packet errors are randomly generated. The proposed AECOD achieves better recovery result than the other error concealment methods. The average PSNR value for each of the video sequences is determined by running the respective error concealment method over 20 times under a specified PER, as is shown in Table II, from which we find that the proposed AECOD outperforms the BMA, EMB, H26LOD, and RBM about 4.79, 1.78, 1.58, and 0.88 dB, respectively, at PER 10% in average. At PER 15%, AECOD outperforms the BMA, EMB, H26LOD, and RBM about 6.01, 3.78, 2.13, and 0.66 dB, respectively. A bar graph of the performances on the eight test video sequences at PER 18% is shown in Fig. 5. From the experimental results it is obvious that determining the error concealment order according to the external blocks texture information can improve the reconstructed video quality at the decoder side. In order to show the effectiveness of the proposed AECOD method, it is also embedded in the H.264/AVC decoder [47]. Correspondingly, the performances of different EC methods are shown in Table III. By comparing Tables II and III, we find that AECOD is decoder-independent. It outperforms the BMA, EBM, RBM, and H26LOD. Fig. 6(a) and (b) compares the average PSNR values of different EC methods, applied to the test sequences flower and football at the PER 10%. From these figures we find that the proposed AECOD based EC method usually gives better recovery results. By comparing the PSNR values of EBM and AECOD,

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

QIAN et al.: RECOVERING CONNECTED ERROR REGION

689

TABLE III COMPARISON OF AVERAGE PSNR VALUES FOR THE TEST VIDEO SEQUENCES WITH DIFFERENT EC METHODS AT PER 10% AND 15%

Fig. 5. Performance comparison for BMA, EBM, H26LOD, RBM, and for the eight AECOD based error concealment methods under PER test video sequences.

= 18%

we can conclude that the processing order determined by analyzing the external boundary patterns, rather than in the raster scan order, indeed improves the recovery performance. The subjective quality of the recovery results by the BMA, EBM, H26LOD, RBM, and the proposed AECOD error concealment methods are also given based on the three corrupted frames as shown in Fig. 7(a)–(c), respectively, which correspond to the frame #22 of mobile, frame #52 of football and frame #31 of flower. The erroneous slices are manually generated and their recovery results using different error concealment methods are shown in Figs. 8–10 respectively. Fig. 8 illustrates a comparison for recovering the corrupted slices shown in Fig. 7(a) where some digits in the mobile calendar are corrupted. Obvious artifacts are generated by the BMA, EBM, H26LOD, and RBM methods for the digits on the right side of the fifth erroneous slice. Since the information of the left and right sides is not used in recovering each vertical EMBs by RBM, the recovery results are usually not satisfactory when the top- and bottom- adjacent external blocks are in smooth regions. This accounts for the strange recovery results in Fig. 8(d) where the digits “4”–“6” are all recovered by digit “5”. From the above comparisons it is obvious that the recovery dependency problem makes the BMA and EBM based error concealment results not very satisfactory. However, the recovery results by AECOD achieve both better visual results and higher PSNR values. Fig. 9 shows the recovery results for the corrupted slices shown in Fig. 7(b). It is obvious that the stripes on the trousers of the players in the erroneous slices are recovered more ac-

Fig. 6. Performance comparison of the BMA, EBM, H26LOD, RBM, and AECOD for two video sequences. (a) flower at PER 10%; (b) football at PER 10%.

curately by the AECOD than by the other methods. Fig. 10 shows the recovery results of BMA, EBM, H26LOD, RBM, and AECOD for the erroneous frames as shown in Fig. 7(c). By comparing the recovery results with the error-free frames as shown in Fig. 10(f), we find that the proposed AECOD method outperforms the other EC methods. B. Discussion on Computation Cost and Memory Requirements for Carrying Out AECOD The proposed AECOD based error concealment outperforms other MV estimation methods using a conventional concealment ordering. As each coin has two sides, computational intensity of AECOD is comparatively higher than BMA, EBM, and

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

690

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

Fig. 7. Three corrupted frames with several erroneous slices. (a) Frame #22 of mobile; (b) frame #52 of football; (c) frame #31 of flower.

Fig. 8. Recovery performances for Fig. 7(a) with different error concealment methods. (a) BMA with PSNR 19.23; (b) EBM with PSNR 21.32; (c) H26LOD with PSNR 20.56; (d) RBM with PSNR 22.91; (e) AECOD with PSNR 23.40; (f) error free with PSNR 31.72.

H26LOD. Compared to EBM, the proposed AECOD needs more CPU time and memory to calculate the external boundary texture information and to determine the processing order in a connected EMB region by sorting and updating the texture intensities. However, the computational cost of the external boundary texture calculation and error concealment order determination in AECOD is comparatively very low compared to the exhaustive MV estimation in a specified search range. In our experiment, the average computational costs of EBM 3 are obtained by and AECOD with boundary width running the programs on a P4 2.8-G PC with 1-G RAM. The average computational costs of recovering a corrupted frame (with image size 352 288) by EBM and AECOD based EC methods are, respectively, 0.04896 and 0.04770 seconds per

frame at PER 10%. From the above comparisons, it is obvious that the proposed AECOD needs about extra 3% computations over EBM to determine EC order. In order to carry out decoding and error concealment for recovering the corrupted I-, P- and B- frames, their temporal reference frames, the corrupted frames, and the erroneous map information are required and must be stored in buffer. Let denote the basic memory requirement for carrying out decoding and error concealment. Moreover, the non-sequential ) EMB method also needs some extra memory (denoted for carrying out exhaustive based motion vector search. Compared to the non-sequential EBM based error concealment method, the proposed AECOD method requires more memory for carrying out the calculation of texture information of the

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

QIAN et al.: RECOVERING CONNECTED ERROR REGION

691

Fig. 9. Recovery performances for Fig. 7(b) with different error concealment methods. (a) BMA with PSNR 22.30; (b) EBM with PSNR 21.18; (c) H26LOD with PSNR 22.55; (d) RBM with PSNR 23.99; (e) AECOD with PSNR 24.28; (f) error free with PSNR 34.89.

Fig. 10. Recovery performances for Fig. 7(c) with different error concealment methods. (a) BMA with PSNR 15.79; (b) EBM with PSNR 17.22; (c) H26LOD with PSNR 18.50; (d) RBM with PSNR 19.76; (e) AECOD with PSNR 20.56; (f) error free with PSNR 31.43.

external boundary blocks (denoted ) and error concealment order determination (denoted ). Let and denote the memory requirements of the EBM and AECOD. Then they have following relationships:

and From (11) and (12), the extra memory requirement the extra ratio of the AECOD over EBM method are as follows:

(11) (12) Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

(13) (14)

692

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

It is obvious that is a constant. In AECOD, the erroneous slices are recovered one by one. In this circumstance, denotes the amount of memory to record the texture information of the external blocks. For the CIF format video seis less than 10 kbits and less than 1% quences, usually in our experiments. C. Discussion on the Selection of the Parameters the Performance of AECOD

and

to

The above objective and subjective comparison results show the effectiveness of the proposed AECOD. In Section II, the external boundary information is represented by the standard deviations of the external blocks with the boundary width , as shown in (8). The recovery results may be affected by the parameter . Moreover, the confidence factor , which stands for the contributions of the external boundaries of the recovered EMB for recovering the remaining EMBs in its neighborhood, may also influence the final recovery result. Here more detailed discussions are given in terms of the effect of the boundary width and confidence factor on the final performance of the proposed AECOD. We compare the performance of AECOD at , 0.5, 0.75, and 1.0 with the EBM method against the different boundary widths , taking the integers in the range 1–16. Fig. 11(a) shows the objective comparison of the error concealment performance on the test video sequence flower at PER 10%, and Fig. 11(d) shows the performance for all the test video sequences at PER 12.5%, respectively. From Fig. 11 we gives better recovery results find that a with and . It shows that the recovered EMBs in than the connected error region will make certain contributions for recovering the un-recovered EMBs. However, the confidence factors provided by the recovered EMBs should not be as reliable as those of the correctly received macro-blocks. Comparat atively, better recovery results are achieved with the same boundary width. In addition, we find that the recovery results are not improved when the boundary width is larger than and the 4. Comparatively, under the boundary width a better error concealment perforconfidence factor mances is achieved, which outperforms the error concealment and by about 1 dB. The comperformance at ) at PER 10% and parisons of EBM and AECOD (with 15% with boundary widths 1, 3, and 5 are listed in Table IV. Under the same conditions, AECOD outperforms EBM by at least 1 dB. This further shows that the determination of the error concealment order for EMBs in a connected error region improves the final recovery performance and reduces the influence of recovery dependency problem effectively. D. Discussion on the Influence of the Texture Information Representation on the Performance of AECOD The external boundary pattern is represented by the standard deviation as is shown in (8). Now we discuss the influence of different patterns on the performance of the proposed AECOD based error concealment method. The representations of patterns including the standard deviation, the gradient intensity, and the gray-level co-occurrence matrix (GLCM), are compared with each other in AECOD. Let and

Fig. 11. Performances of the proposed method with different boundary width. (a) flower at PER 10%; (b) average for all the test video sequences.

denote the gradient image and the gradient intensity of a block with sizes or , which are calculated as follows:

(15) (16) The gray-level co-occurrence matrix for the top and bottom boundary images and for the transpositions of the left and right ones is also used to represent the boundary pattern. GLCM calculates how often a pixel with a gray-level (grayscale intensity) value occurs horizontally adjacent to a pixel with another value . Each element in GLCM specifies the number of times that the pixel with value occurs horizontally adjacent to a pixel denote the GLCM of the top/bottom with value . Let or the transposition of the left/right block with sizes block with sizes . The gray-level co-occurrence matrix has sizes , where is the number of gray scale shades [44]. Three texture features, namely the texture energy (denoted

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

QIAN et al.: RECOVERING CONNECTED ERROR REGION

693

TABLE IV COMPARISON OF AVERAGE PSNR VALUES FOR THE TEST VIDEO SEQUENCES WITH THE EBM AND PER AND THE AECOD BASED EC METHODS AT PER

= 10%

= 15%

TABLE V COMPARISON OF AVERAGE PSNR VALUES FOR THE TEST VIDEO SEQUENCES FOR DIFFERENT EXTERNAL BOUNDARY PATTERNS AT PER 10%

), the contrast (denoted ), and the homo), are extracted from GLCM geneity (denoted

(17) (18) (19) Those boundary patterns in the standard deviation (denoted as STD), gradient intensity (denoted as GRAD), and GLCM features (denoted as GLCM_EN, GLCM_CT, and GLCM_HO) are used to represent the external boundary pattern described by (10), which is used to determine the processing order of the proposed AECOD. Their corresponding EC performances at PER 10%, different confidence factor and boundary widths

and are compared in Table V. From those data we observed that the result differences among the different patterns are relatively small. The maximum difference of the PSNR . Compared values is smaller than 0.05 dB at with the performances obtained by EBM, better recovery performances are achieved when the patterns of the external boundary blocks are analyzed and the error concealment order of the erroneous macro-blocks in the connected error regions are adaptively determined. IV. CONCLUSION Video frames are often corrupted by the packet loss when being transmitted over bandwidth limited networks, which makes the error region connected when the dispersed flexible macro-block ordering technique is not adopted. To recover the connected erroneous region, it is inevitable to take account of the recovery dependency problem. The commonly used error concealment methods are carried out in the raster scan mode. In this paper, the processing order of each erroneous macro-block

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

694

IEEE TRANSACTIONS ON MULTIMEDIA, VOL. 11, NO. 4, JUNE 2009

is adaptively determined by the proposed AECOD based method by analyzing the textural patterns of external blocks. The proposed error concealment method can improve the recovery performance and deal with the influence of the recovery dependency problem successfully. A confidence factor is introduced for each of the external blocks which measure their contributions to the recovery of the corrupted macro-block. Experimental results show that the recovered macro-blocks can provide important clues to the recovery of the remaining erroneous macro-blocks in a connected erroneous region. ACKNOWLEDGMENT The authors would like to thank the associate editor and the anonymous reviewers for providing several constructive suggestions during the reviews and showing their patience by giving helpful comments to improve the quality of this paper. The authors also would like to thank Ms. Y. Gao in their laboratory and a staff member in the Department of Foreign Languages of Xi’an Jiaotong University for correction to some grammatical errors existing in the previous version of this paper. REFERENCES [1] Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU-T Rec. H.264/ISO/IEC 14486-10 AVC), Joint Video Team (JVT) of ISO/IEC MPEG and ITU-T VECG, JVT-G050, 2003. [2] M. Chi, M. Chen, J. Liu, and C. Hsu, “High performance error concealment algorithm by motion vector refinement for MPEG-4 video,” in Proc. Int. Conf. Circuits and Systems, May 2005, vol. 3, pp. 2895–2898. [3] Y. Wang, G. Wen, S. Wenger, and A. K. Katsaggelos, “Review of error resilient techniques for video communications,” IEEE Signal Process. Mag., vol. 17, no. 4, pp. 61–82, Jul. 2000. [4] A. Vetro, J. Xin, and H. Sun, “Error resilience video transcoding for wireless communications,” IEEE Wireless Commun., vol. 12, no. 4, pp. 14–21, Aug. 2005. [5] S. Wenger, “H.264/AVC over IP,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 645–656, Jul. 2003. [6] B. Yan and K. W. Ng, “A survey on the techniques for the transport of MPEG-4 video over wireless networks,” IEEE Trans. Consum. Electron., vol. 48, no. 4, pp. 863–873, Nov. 2002. [7] Z. Wang, Y. Yu, and D. Zhang, “Best neighborhood matching: An information loss restoration technique for block-based image coding systems,” IEEE Trans. Image Process., vol. 7, no. 7, pp. 1056–1061, Jul. 1998. [8] G. Yu, M. M. K. Liu, and M. W. Marcellin, “POCS-based error concealment for packet video using multi-frame overlap information,” IEEE Trans. Circuits Syst. Video Technol., vol. 8, no. 8, pp. 422–434, Aug. 1998. [9] W. Zeng and B. Liu, “Geometric-structure-based error concealment with novel applications in block-based low-bit-rate coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 6, pp. 648–665, Jun. 1999. [10] W. Y. Kung, C. S. Kim, and C. C. J. Kuo, “A spatial-domain error concealment method with edge recovery and selective directional interpolation,” in Proc. IEEE Int. Conf. Multimedia and Expo, Jul. 2003, vol. 2, pp. 145–148. [11] O. Nemethova, A. Al-Moghrabi, and M. Rupp, “Flexible error concealment for H.264 based on directional interpolation,” in Proc. Int. Conf. Wireless Networks, Communications and Mobile Computing, Jun. 2005, vol. 2, pp. 1255–1260. [12] Y. Wang, M. Hannuksela, V. Varsa, A. Hourunranta, and M. Gabbouj, “The error concealment feature in the H.26L test model,” in Proc. Int. Conf. Image Processing (ICIP), 2002, pp. 729–732. [13] S. Hsia, “An edge-oriented spatial interpolation for consecutive block error concealment,” IEEE Signal Process. Lett., vol. 11, no. 6, pp. 577–580, Jun. 2004.

[14] H. Sun and W. Kwok, “Concealment of damaged block transform coded images using projection onto convex set,” IEEE Trans. Image Process., vol. 4, no. 4, pp. 470–477, Apr. 1995. [15] J. Sub and Y. Ho, “Error concealment based on directional interpolation,” IEEE Trans. Consum. Electron., vol. 43, no. 3, pp. 295–302, Aug. 1997. [16] E. T. Kim, S.-J. Choi, and H.-M. Kim, “Weighted boundary matching algorithm for error concealment in the MPEG-2 video bit stream,” Signal Process., vol. 73, pp. 291–295, Jan. 1999. [17] W.-M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost or erroneously received motion vectors,” in Proc. IEEE Int. Conf. Acoustic, Speech and Signal Processing, Apr. 1993, vol. 5, pp. 417–420. [18] T. H. Tsai, Y. X. Lee, and Y. F. Lin, “Video error concealment techniques using progressive interpolation and boundary matching algorithm,” in Proc. IEEE Int. Symp. Circuits and System, May 2004, vol. 5, pp. 433–436. [19] T. Chen, X. Zhang, and Y.-Q. Shi, “Error concealment using refined boundary matching algorithm,” in Proc. IEEE Int. Conf. Information Technology, Research and Education, Aug. 2003, pp. 55–59. [20] W. Lie and Z. Gao, “Video error concealment by integrating greedy suboptimization and Kalman filtering techniques,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 8, pp. 982–992, Apr. 2006. [21] Q. Peng, T. Yang, and C. Zhu, “Block-based temporal error concealment for video packet using motion vector extrapolation,” in Proc. Int. Conf. Communications, Circuits and Systems and West Sino Expositions, 2002, vol. 1, pp. 10–14. [22] Y. Xu and Y. Zhou, “H.264 video communication based refined error concealment schemes,” IEEE Trans. Consum. Electron., vol. 50, no. 4, pp. 1135–1141, Nov. 2004. [23] D. Kim, S. Yang, and J. Jeong, “A new temporal error concealment method for H.264 using adaptive block sizes,” in Proc. Int. Conf. Image Processing, Sep. 2005, vol. 3, pp. 28–31. [24] Y. P. Su, M. T. Sun, and V. Hsu, “Global motion estimation from coarsely sampled motion vector field and the applications,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 2, pp. 232–242, Feb. 2005. [25] Y. P. Su and M. T. Sun, “A non-iterative motion vector based global motion estimation algorithm,” in Proc. Int. Conf. Multimedia and Expo., Jun. 2004, vol. 1, pp. 703–706. [26] Y. Wang, Q.-F. Zhu, and L. Shaw, “Maximally smoothness image recovery in transform coding,” IEEE Trans. Commun., vol. 41, no. 10, pp. 1544–1551, Oct. 1993. [27] J. Feng, K. Lo, and H. Mehrpour, “Error concealment for MPEG video transmissions,” IEEE Trans. Consum. Electron., vol. 43, no. 2, pp. 183–187, May 1997. [28] Y. Jung, Y. Kim, and Y. Choe, “Robust error concealment algorithm using iterative weighted boundary matching criterion,” in Proc. IEEE Int. Conf. Image Processing, Sep. 2000, vol. 3, pp. 384–387. [29] M. Chen, C. Chen, and M. Chi, “Temporal error concealment algorithm by recursive block-matching principle,” IEEE Trans. Circuits Syst. Video Technol., vol. 15, no. 11, pp. 1385–1393, Nov. 2005. [30] ISO/IEC JTC1/SC29/WG11/MEPG/M2915 MPEG-4 VM 8.0. Momusys implementation of VM (VM8-971021). Fribourg, 1997. [31] F. Zhai, Y. Eisenberg, T. N. Pappas, R. Berry, and A. K. Katsaggelos, “Rate-distortion optimized hybrid error control for real-time packetized video transmission,” IEEE Trans. Image Process., vol. 15, no. 1, pp. 40–53, Jan. 2006. [32] M. Gallant and F. Kossentini, “Rate-distortion optimized layered coding with unequal error protection for robust Internet video,” IEEE Trans. Circuits Syst. Video Technol., vol. 11, no. 3, pp. 357–372, Mar. 2001. [33] D. Agrafiotis, D. Bull, and C. Canagarajah, “Enhanced error concealment with mode selection,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 8, pp. 960–973, Aug. 2006. [34] S. Wenger and M. Horowitz, “FMO: Flexible macroblock ordering,” JVT-C089, May 2002. [35] Coding of Audio-Visual Objects, Part 2: Visual, ISO/IEC 14496-2, 1999, MPEG-4 visual version 1. [36] Video Coding for Low Bit Rate Communication, Version 1, ITU-T Recommendation H.263, 1998. [37] I. E. G. Richardson, H.264 and MPEG-4 Video Compression: Video Coding for Next-Generation Multimedia. New York: Wiley, pp. 167–172.

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.

QIAN et al.: RECOVERING CONNECTED ERROR REGION

[38] J. Zhang, J. F. Arnold, and M. R. Frater, “A cell-loss concealment technique for MPEG-2 coded video,” IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 4, pp. 659–665, Jun. 2000. [39] T. S. Valente, C. Dufour, F. Groliere, and D. Snook, “An efficient error concealment implementation for MPEG4 video streams,” IEEE Trans. Consum. Electron., vol. 47, no. 3, pp. 568–578, Aug. 2001. [40] S. Tsekeridou and I. Pitas, “MPEG-2 error concealment based on block-matching principles,” IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 4, pp. 646–658, Jun. 2000. [41] L.-W. Kang and J.-J. Leou, “A hybrid error concealment scheme for MPEG-2 video transmission based on best neighborhood matching algorithm,” J. Vis. Commun. Image Represent., vol. 16, no. 3, pp. 288–310, Jun. 2005. [42] S. K. Im and A. J. Pearmain, “An efficient data classification scheme with the H.264 flexible marcoblock ordering,” in Proc. IEEE Int. Conf. Information, Communications and Signal Processing, 2005, pp. 1135–1139. [43] J. Suh and Y. Ho, “Error concealment techniques for digital TV,” IEEE Trans. Broadcast., vol. 48, no. 4, pp. 299–306, Apr. 2004. [44] K. R. Castleman, Digital Image Processing. Englewood Cliffs, NJ: Prentice-Hall, 1999, pp. 499–501. [45] Y.-K. Wang, M. M. Hannuksela, V. Varsa, A. Hourunranta, and M. Gabbouj, “The error concealment feature in the H.26L test model,” in Proc. IEEE Int. Conf. Image Processing, 2002, vol. 2, pp. 729–732. [46] V. Varsa, M. M. Hannuksela, and Y. K. Wang, Non-Normative Error Concealment Algorithms, ITU-T VCEG-N62, 2001. [47] JVT Reference Software Version 10.2. [Online]. Available: ftp://ftp. imtc-files.org/jvt-experts/reference_software/. [48] X. Qian and G. Liu, “Global motion estimation from randomly selected motion vector groups and GM/LM based applications,” Signal, Image, Video Process., vol. 1, no. 3, pp. 179–189, 2007. [49] X. Qian, G. Liu, and H. Wang, “Texture based selective block matching algorithm for error concealment,” in Proc. IEEE Int. Conf. Multimedia and Expo., Jun. 2007, pp. 739–742.

695

Xueming Qian received the B.S. and M.S. degrees in Xi’an University of Technology, Xi’an, China, in 1999 and 2004, respectively, and the Ph.D. degree in the School of Electronics and Information Engineering, Xi’an Jiaotong University, Xi’an, China, in 2008. From 1999 to 2001, he was an Assistant Engineer at Shannxi Daily. Now he is a faculty member of the School of Electronics and Information Engineering, Xi’an Jiaotong University. His research interests include video/image communication and transmission, video analysis, processing and compression, video transmission and error concealment techniques, and semantic-based video analysis, indexing, and retrieval.

Guizhong Liu received the B.S. and M.S. degrees in computational mathematics from Xi’an Jiaotong University, Xi’an, China, in 1982 and 1985, respectively, and the Ph.D. degree in mathematics and computing science from Eindhoven University of Technology, Eindhoven, The Netherlands, in 1989. He is currently a Full Professor with the School of Electronic and Information Engineering, Xi’an Jiaotong University. His research interests include nonstationary signal analysis and processing, image processing, audio and video compression, and inversion problems.

Huan Wang received the B.S. degree from Xi’an University of Technology, Xi’an, China, in 2005. Now she is pursuing the M.S. degree in the School of Electronic and Information Engineering, Xi’an Jiaotong University.

Authorized licensed use limited to: Xian Jiaotong University. Downloaded on June 27, 2009 at 01:57 from IEEE Xplore. Restrictions apply.