Spatial Error Concealment With an Adaptive ... - Semantic Scholar

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 25, NO. 3, MARCH 2015

353

Spatial Error Concealment With an Adaptive Linear Predictor Jing Liu, Guangtao Zhai, Member, IEEE, Xiaokang Yang, Senior Member, IEEE, Bing Yang, and Li Chen, Member, IEEE Abstract— In this paper, a novel spatial error concealment (EC) algorithm is proposed. Under the sequential recovery framework, pixels in missing blocks are successively reconstructed based on adaptive linear predictor. The predictor automatically tunes its order and support shape according to local contexts. The predictor order and support shape are determined using Bayesian information criterion, which is able to strike a balance between the bias and variance of the prediction errors. The flexibility of the order-adaptive predictor is able to recover more important features or structures. A novel scan order based on the uncertainty of each pixel is also proposed to alleviate error propagation problem. Compared with the state-of-the-art EC algorithms, experimental results show that the proposed method gives better reconstruction performance in terms of objective and subjective evaluations. Index Terms— Adaptive linear predictor, Bayesian information criterion (BIC), error concealment (EC).

I. I NTRODUCTION LOCK-BASED image/video coding standards, such as Joint Photographic Experts Group (JPEG) [1], Moving Picture Experts Group (MPEG) [2], and H.264/Advanced Video Coding (AVC) [3], are widely used in recent multimedia applications. An image/frame is split into nonoverlapped blocks that are coded separately. The loss of one bit often causes the loss of the whole block. When transmitted over error-prone channels, packet loss will lead to severe quality reduction. A simple method to correct error is to retransmit the lost data. However, in many cases, retransmission is impossible due to real-time constraints or lack of bandwidth. Therefore, it is crucial to study error resilience techniques to guarantee the quality of the received videos within limited transmission conditions [4]–[6]. Error concealment (EC), as a postprocessing method, recovers the missing blocks without modifying the encoder or channel

B

Manuscript received January 21, 2014; revised May 28, 2014 and July 28, 2014; accepted September 15, 2014. Date of publication September 19, 2014; date of current version March 3, 2015. This work was supported in part by the National Natural Science Foundation of China under Grants 61025005, 61129001, 61371146, 61422112, 61331014, and 61102098; in part by 973 Program under Grant 2010CB731401; in part by the Shanghai Municipality Science and Technology Commission under Grant 13511504500; in part by the 111 Program under Grant B07022; and in part by the Foundation for the Author of National Excellent Doctoral Dissertation of China under Grant 201339. This paper was recommended by Associate Editor J. Zhang. The authors are with the School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TCSVT.2014.2359145

coding schemes [7], [8]. The basic idea of EC is to estimate the corrupted pixels using correctly received ones in current image or adjacent frames based on strong correlations within images or videos. Depending on which kind of correlation is exploited, EC techniques can be categorized into three classes: 1) spatial EC (SEC); 2) temporal EC (TEC); and 3) spatial– temporal EC (STEC). SEC relies on the information within current frame [8]–[17], while TEC takes advantage of the temporal correlations [18]–[21]. STEC is a combination of the previous two methods [8], [22], [23]. Although temporal correlation tends to be higher than the spatial one, there are situations where it is difficult to access temporal information, such as the recovery of still images and intracoded frames [24]. We focus on the first class of EC techniques, SEC, in this paper. SEC approaches reconstruct the corrupted blocks by utilizing successfully received surroundings under local continuity assumption. Most of existing SEC approaches are block-wise algorithms, where the corrupted pixels inside a block are filled in simultaneously. Seiler and Kaup [25] filled in the missing pixels with orthogonality deficiency compensated frequency selective extrapolation scheme. Sun and Kwok [10] recovered lost blocks based on projections onto convex set (POCS). Apart from spectral information, spatial correlation is also utilized for recovery. Shirani et al. [26] modeled natural images as Markov random fields (MRF) and produced a visually comfortable but sometimes over-smoothed recovery. A series of edge-directed interpolation-based methods is suggested in [9], [12], and [15], where different kinds of edge models are used to restore the missing blocks. In addition, block-based bilateral filter (BBF) was suggested in [27] to preserve edges in lost blocks. Even though a lot of attempts have been made to preserve edges in the corrupted blocks, these block-wise algorithms often fail when more than one edge is involved or a successive sequence of blocks are corrupted. To deal with multiple edges in the missing block, Asheri et al. [28] proposed an algorithm called novel adaptive Gaussian process (NAGP). The missing regions are separated into several subregions according to multiple hypothesized edges. Then, the subregions are extrapolated individually with adaptive kernel functions. Although the blurred edges caused by stationary kernel functions are avoided, the hard division of missing block may introduce false borders. Multiple edges are also addressed in [29], where several directional interpolations are combined according to the visual clearness (VC) of the edges. In spite of its capability to reconstruct complicate edges, it is hard to accurately determine the location and the VC of the edges.

1051-8215 © 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

354


To overcome the drawbacks of block-wise schemes, Li and Orchard [14] proposed a sequential framework based on orientation adaptive interpolation (OAI). The missing pixels are successively interpolated in a pixel-wise manner. Linear prediction model is utilized and the model parameters adapt to local contexts on a pixel-by-pixel basis. The sequential framework enjoys more flexibility of capturing important edge features from local available information compared with the block-wise framework. However, the support vector of OAI is fixed throughout the prediction process and includes all available pixels in the local window. The underlying assumption is that two pixels are more correlated if they are spatially closer to each other. Although it is reasonable for many 1-D sequences of datum, such as temperature curve, this assumption no longer holds for 2-D images since pixel dependencies are often anisotropic. Therefore, the predictor with predetermined support vector may fail to capture some important features, especially for fine details. Koloda et al. [24] sequentially reconstructed the missing blocks with adaptive predictor where support samples are implicitly selected by assuming the linear coefficients to be sparse. An enhanced version using multivariate kernel density estimation (MKDE) was suggested in [30]. However, these methods do not fully solve the problems of how to determine the model order and support shape. Recently, adaptive predictors have been researched in many other image processing applications, such as lossless predictive coding and image interpolation. Takeda et al. [31] denoised and interpolated images using an adaptive ellipse shaped support, with the major axis aligned with the direction of local edge. Kervrann and Boulanger [32] adaptively chose the size of support vector based on local image statistics for image regularization and representation. Wu et al. [33] proposed an MDL-based sequential predictor for lossless predictive coding. The predictor adapts to changing statistics with a locally tuned support shape and training set. In spite of the above mentioned flexibilities and advantages of adaptive predictors, if these predictors are directly utilized for EC, they will cause severe artifacts, such as error propagation. OAI addresses error propagation problem by estimating the missing block from eight directions in raster scan order and merging them with weighted combination. The fusion procedure alleviates propagated errors at the expense of blurred details, which are undesired for high visual quality. In this paper, we propose an EC algorithm to sequentially estimate missing pixels in a novel scan order using a pixel-wise adaptive predictor. Bayesian information criterion (BIC), or the Schwarz criterion [34], is adopted to explicitly determine the order of the predictor. More specifically, a sequence of linear predictors with different orders is derived. The shape of model support and training samples are uniquely determined for each model order. Correlation instead of Euclidean distance is utilized to form spatially nested predictor supports for different prediction models. Consequently, support shape can be arbitrary and unnecessary connected. Then, BIC is applied to the sequence of predictors and the most appropriate linear model is determined. Such an order-adaptive predictor is capable of dealing with arbitrary contexts and significantly

Fig. 1. Typical block loss. Each square represents a block of size 8 × 8 or 16 × 16. White squares are successfully received blocks, while black ones denote corrupted blocks. (a) Isolated loss. (b) Consecutive loss. (c) Slice loss.

improves the restoration performance of fixed-support one. Unlike OAI, we estimate each missing pixel only once, hence, requiring a more carefully designed scan order than the fixed raster pattern. Motivated by the fact that prediction tends to be more accurate with more reliable support vector, we propose a scan order based on the self-designed uncertainty of each pixel. We start from the pixel with lowest uncertainty and proceed according to the uncertainty in a nondecreasing manner. This scan order significantly reduces the propagation errors of raster scan order used in OAI. This paper is organized as follows. Section II formulates the EC problem and introduces the linear predictor. The proposed algorithm is described in Section III. Extensive experimental results and comparisons with other state-of-the-art SEC techniques are presented in Section IV. The difference between the proposed algorithm and the image inpainting algorithms is discussed in Section V. Section VI concludes this paper. II. P ROBLEM F ORMULATION In practice, the locations of missing blocks can be obtained at the decoder [21] and are assumed already known. We consider both 8 × 8 and 16 × 16 block loss in this paper. According to the availability of the surrounding blocks, the corrupted situation can be generally categorized into two types: isolated loss and consecutive loss. Isolated loss represents the case where all eight-connected surrounding blocks of a corrupted block are correctly received. The rest situations belong to consecutive loss. A special case of consecutive loss is slice loss. It is a typical situation of packet loss where a whole row of blocks are lost. For convenience, if the lost pattern repeats throughout the image, we call the loss regular. Fig. 1 shows three common cases of block loss: regular isolated loss, regular consecutive loss, and slice loss. Each square represents a block of pixels. The correctly received blocks are marked in white and the corrupted blocks are in black. The SEC is designed to estimate some corrupted blocks from correctly received parts of images or videos. Under a sequential framework, the recovery of image data relies on not only successfully received pixels but also previously recovered ones. For convenience, we use available pixels to represent either successfully received pixels or previously filled-in pixels in the rest of this paper. Under the assumption that a globally nonstationary image can still be locally modeled as a stationary Gaussian process, each pixel x is estimated as a linear combination of its neighborhoods u K (x) =

LIU et al.: SPATIAL EC WITH AN ADAPTIVE LINEAR PREDICTOR

355

variance of estimation errors. To solve the critical problems, we derive a sequence of predictors with different orders. A unique support shape u K and a unique training set SK are derived for each model order K . Then, the determination of prediction model is regarded as a model selection problem. The BIC has been proposed as an asymptotically consistent selection criterion. It attempts to compensate for the overfitting problem of more complex models by adding a penalty term to the likelihood function of training set [34], [35]. Specifically, the objective function of our BIC-based model selection problem is K · log |SK | 1 loglik + (4) |SK | 2|SK | where |SK | stands for the number of elements in SK . The loglik in the first term is the log-likelihood of training samples. The second term penalizes the model complexity. We seek for the predictor which produces minimum L in (4). The loglik is evaluated using the optimized coefficients aˆ L(u K , S K ) = −

Fig. 2. 2-D linear predictor. Each square stands for one pixel. Current missing pixel x is on the top-left corner of 8 × 8 isolated corrupted block. 2-D support u13 (x) is comprised of 13 available neighborhoods, denoted by squares with crosses. Pixels in blue window are training sample set S13 (x).

[u 1 (x), u 2 (x), . . . , u K (x)]T . The corresponding linear model with additive noise term is defined as x = u K (x)T · a +

(1)

wherein u K (x) is 2-D model support of size K and a = [a1 , a2 , . . . , a K ]T is linear coefficients. Existing EC algorithms generally assume the 2-D support u K (x) to be the K available pixels that are closest to x in Euclidean distance. Fig. 2 shows the 2-D support u13 (x) with x being the top-left corner of an 8 × 8 isolated missing block. Given the spatial configuration of the 2-D support u K (x) = [u 1 (x), u 2 (x), . . . , u K (x)]T , the linear coefficient a is estimated by solving the least squares problem aˆ = argmina y − u K (y)T · a2 (2) y∈S K (x)

where u K (y) is the 2-D support of y with the same support shape as u K (x) and S K (x) is a training set of pixels, as shown in Fig. 2. Therefore, the estimated value for current pixel x is xˆ = u K (x)T · aˆ .

(3)

III. P ROPOSED A LGORITHM To achieve better prediction performance, the predictor needs to be adaptive to local image contexts. Two critical issues are associated with the design of linear predictor. One is the choice of model order K and support shape, the other being the estimation of linear coefficients a. In [14], the linear coefficients are adaptively determined for each pixel. However, the model supports of OAI are predetermined rather than adaptive to local contexts. In [24], although the sparse linear prediction (SLP) technique implicitly selects a small set of support samples by assuming a sparse constraint on linear coefficients, it does not fully solve the issues mentioned above. These questions are closely related to the tradeoff between the bias and

loglik =

|S K |

log Pr(y j − u K (y j )T · aˆ ), y j ∈ SK .

(5)

j =1

As pointed out by natural image statistics, the prediction residual e j = yj − uk (y j )T · aˆ can be modeled by Laplacian distribution [36]. We integrate the Laplacian distribution into integer coordinate and get the following probability of e j : ⎧ − √1 ⎪ 1 − exp 2σˆ , ⎪ ekj = 0 ⎪ ⎪ k k ⎪ |e |−0.5 |e |+0.5

⎪ − j √ − j √ k ⎨1 σˆ / 2 −exp σˆ / 2 , 0 < |ekj | < V (6) exp Pr e j = 2 ⎪ ⎪ ⎪ |ekj |−0.5 ⎪ ⎪ ⎪ ⎩1 exp− σˆ /√2 , |ekj | = V 2 wherein variance σˆ 2 is estimated from prediction residuals and V denotes the maximum absolute residual value, which is 255 for 8-bit images. The details of our algorithm are described as follows. Sections III-A and III-B show how to derive 2-D support u K and training set SK for model order K , respectively. Section III-C gives a novel scan order based on uncertainty. A. 2-D Support We suppose that 2-D support u K (x) of current pixel x is selected from a support pixel set U(x), which is comprised of all available pixels within a region around x. To get the unique 2-D support of K -order predictor, we need to sequentialize the pixels in U(x). A simple method is to sort pixels according to their Euclidean distances to x, which is adopted by [14] and [37]. However, Euclidean distance is not a reasonable criterion to sort pixels of natural images, especially for edges and texture regions. We sequentialize the pixels in U(x) based on their correlations to x, leading to the following sorted support set U(x): U(x) = {u 1 (x), u 2 (x), . . . , | |ρ 1 (x)| ≥ |ρ 2 (x)| ≥ · · · } u i (x)

ρ i (x)

(7)

where is an available support pixel and is the correlation between x and u i (x). The support vector u K (x)

356


of K -order linear predictor simply selects the first K pixels from U(x). Since the support pixels are ordered by correlation, the shape of 2-D support can be arbitrary and unnecessarily connected. As the model order increases from K to K +1, 2-D support u K (x) will grow and include one more pixel u K +1 (x) to form u K +1 (x). Consequently, a nested sequence of 2-D support is produced, i.e., u K 1 (x) ⊂ u K 2 (x) if K 1 < K 2 . One trivial problem here is that it is impossible to calculate the correlation between u i (x) and x, especially when x is unknown. To estimate ρ i (x), we derive a training set T that is comprised of available training samples whose support set are sufficiently similar to that of x, that is T = {y|U(y) − U(x) ≤ τT }

(8)

where τT is a threshold. Given the selected training set T , ρ i (x) is defined as the correlation coefficient between samples of u i (x) and samples of x in T i 1 i u (y)y − y u (y) |T | (9) ρ i (x) = u i (y)2 − ( u i (y))2 y 2 − ( y)2 wherein all are with respect to y ∈ T . Consequently, the sorted support set in (7) is completely defined and the 2-D support for each order K can be easily determined. B. Training Set Having derived the unique 2-D support u K (x) for K -order predictor, we need to select an appropriate training set SK (x) to estimate linear coefficients a. A training sample is included in SK (x) if its 2-D support pixels are close enough to those of current pixel x. Putting it into mathematic language, we have

1 (10) S K = y | u K (y) − u K (x) ≤ τ S K where τ S is a threshold. Since local contexts vary from each other, τ S is defined adaptively as 1 uks (y) − uks (x). (11) y ks Here, we assume that ks support pixels are required to represent the local contexts. Then, 1/ks uks (y) − uks (x) denotes the similarity between local contexts with neighboring samples. Therefore, the right-hand side of (11) gives a threshold for similar training sample selection. ks is empirically set to 3 in the experiments. Recall that 2-D support of each order is nested. As the model order K increases, less relevant support pixels are added into 2-D support u K (x) and then the size of training set SK (x) tends to shrink. One rough heuristic is that the number of training samples should be no less than a multiple (say 3 or 5) of the number of adaptive parameters in the model. We set a hard constraint as |SK | ≥ 3 × K . An example of the proposed adaptive predictor is given in Fig. 3. We plot the 2-D support and the training set during the reconstruction of an edge pixel on an image of 8 × 8 isolated loss. Current missing pixel x is marked in red and unrecovered pixels are in black. The left subfigure is the 2-D support u9 (x). Color intensity denotes the correlation coefficients in (9). Thirty training samples with minimum norm of τ S = max

Fig. 3. Support and training set of adaptive linear predictor. The pixel being estimated is marked in red. The black pixels are lost pixels.

Fig. 4.

Diagram for scan order determination.

difference in (10) are shown in the right subfigure. Brighter pixels represent samples that are similar to u9 (x) in the left figure. We can see clearly that the proposed predictor automatically adapts to image contexts, e.g., the hat boundary here. C. Scan Order The last issue associated with sequential reconstruction is the choice of scan order, which is critical to a high quality sequential EC algorithm. The scan order determines the available context of each missing pixel, hence, influencing the derivation of model support and training set. Moreover, as Li and Orchard [14] have pointed out, error propagation is inevitable and the pattern of propagated error is dependent on scan order. OAI operates from eight different directions and merges eight estimates together. It is time consuming and the merging strategy may blur important details, as shown in Fig. 8. In this paper, we estimate each missing pixel only once and propose a novel method to determine scan order based on the self-designed uncertainty of each pixel. The pixels to be estimated are sequentially determined, predicted, and enqueued into a sorted queue Q. When all the corrupted pixels are recovered, Q will completely represent the scan order of the whole image. The diagram for scan order determination is given in Fig. 4. Without loss of generality, missing blocks are filled in one by one in raster scan order. Suppose that the T th recovered pixel in the whole image is the tth recovered pixel in the missing block. Let p T (x) stands for the uncertainty of pixel x before recovering the T th pixel. The smaller uncertainty a pixel has, the more credible it is. The uncertainty of pixels are initialized as c, x is correctly received (12) p1 (x) = 1, x is corrupted where c is a negative constant. In the experiments, we empirically set a default c of −100. Missing pixels are given the positive uncertainty value to represent that they are uncertain, while successfully received pixels are assigned the lowest uncertainty value of c to denote that they are the most credible.


357

Fig. 5. Sequence of uncertainty maps during the recovery of an isolated 8 × 8 lost block. The uncertainty values are normalized to [0, 255] here to facilitate the display. One square stands for one pixel. The red squares are the pixels to be recovered, whose uncertainties are updated based on their available support pixels in yellow windows.

The first subfigure of Fig. 5 shows the initialized uncertainty map around an isolated missing block, where the uncertainty values are normalized to [0, 255] to facilitate the display. Inspired by the fact that prediction tends to be more accurate with more credible support vector, the unrecovered pixel with the most credible support vector is selected as the pixel to be recovered. It is a natural idea to represent the credibility of support vector using the sum of uncertainty of available T support pixels, denoted by sp T (x) = u i ∈U (x) p (u i ). T Obviously, low sp (x) represents the set of support pixels with high credibility. Then, the T th pixel to be recovered is determined as x T = argminx sp T (x),

p T (x) = 1.

(13)

xT

is marked in red in Fig. 5 and its support pixels are surrounded by yellow window. After x T is determined, it is estimated using the algorithm mentioned in previous sections and enqueued into Q. The next step is to update the uncertainty of x T . One intuitive approach to do it is to set the uncertainty of x T as the average uncertainty of its support set, that is ⎧ T ⎨ sp (x) , x = xT p T +1 (x) = (14) |U(x)| ⎩ T p (x), otherwise however, if the uncertainty is updated like this, all the pixels will have uncertainty of c in the end, which is in contrary to our intuitions. Therefore, we modify (14) and make the updated uncertainty relevant to its processing order. The modified update formula is defined as ⎧ sp T (x) ⎨ , x = xT T +1 p (x) = |U(x)| + α · t (15) ⎩ T p (x), otherwise where α is set to 0.5 empirically. Since the scan order inside the corrupted blocks is more important than between the blocks, t instead of T is involved here. Fig. 5 gives a sequence of uncertainty maps during the recovery of an 8 × 8 isolated missing block. The pixels to be

Algorithm 1 Algorithm to Determine Scan Order Input: Corrupted image, parameters c and α. Output: The scan order Q. 1: Initialize the uncertainty for every pixel. The successively received pixels have uncertainty value of c while the uncertainties of missing pixels are initially set to 1. 2: T = 1. 3: for each missing block B do 4: t = 1. 5: while ∃x ∈ B, p T (x) = 1 do 6: For all the x ∈ B, p T (x) = 1, sum up the uncertainty of available support pixels to form sp T (x). 7: Find the T th pixel to be recovered x T by (13). 8: Enqueue x T into Q. 9: Update the uncertainty by (15). 10: t ← t + 1, T ← T + 1. 11: end while 12: end for Algorithm 2 BIC-Based EC Algorithm Input: A corrupted image. Output: An error concealed image. 1: Determine the scan order Q as described in Algorithm 1. 2: while Dequeue a pixel x from Q do 3: Get the sorted support pixel set U(x) based on the correlations by (7). 4: for each model order K do 5: Derive the support vector u K (x) as the first K pixels in U(x). 6: Determine the training set SK by (10). Estimate the coefficients of linear predictor by (2). 7: end for 8: Determine model order by minimizing BIC in (4). 9: Predict x by (3). 10: end while

recovered are marked in red and their available support pixels are in yellow windows. Compared with raster scan order, the proposed uncertainty-based out-to-inner scan order avoids severe block artifacts at the bottom right corner and produces more continuous transition between corruption boundaries. Since the determination of scan order is irrelevant to the estimated value of missing pixels, scan order can be completely derived before recovery. Algorithm 1 describes how to determine scan order and Algorithm 2 gives the overall algorithm. IV. E XPERIMENTAL R ESULTS To evaluate the performance of the proposed algorithm, extensive experiments are conducted. We compare with other state-of-the-art methods, such as POCS [10], MRF [26], nonnormative SEC for H.264 (AVC) [38], OAI [14], content adaptive technique (CAD) [39], exemplar-based inpainting (INP) [40], BBF [27], combined first- and secondorder total variation (FSTV) inpainting [41], SLP [24],

358


TABLE I PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 8 × 8 R EGULAR Isolated L OSS . B EST M ETHOD I S IN B OLDFACE

TABLE II PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 16 × 16 R EGULAR Isolated L OSS . B EST M ETHOD I S IN B OLDFACE

VC [29], and MKDE [30].1 Traditional Peak Signal-to-Noise Ratio (PSNR) is chosen as one of the objective measurement in the experiments. The PSNR for the whole image is used throughout our paper. Since PSNR may fail to capture visually important perception cues that are critical to EC performance, the structural similarity (SSIM) index [43] is also considered.

A. Test on Images To evaluate the proposed algorithm, a variety of the experiments are carried out on images, involving both 8 × 8 and 16 × 16 block loss. Four types of block loss are considered: regular isolated loss (≈25% loss), regular consecutive loss (≈50% loss), random consecutive loss (≈10% loss), and random slice loss (≈10% loss). The test images are shown in Fig. 6, including Lena (512×512), Car (512×512), Baboon 1 The implementations of POCS, AVC, OAI, CAD, MRF, SLP, VC, and MKDE are downloaded from http://dtstc.ugr.es/~jkoloda/research.html. The code of FSTV is downloaded from http://dx.doi.org/10.5201/ipol.2013.40. The code of INP is based on a third-party implementation [42].

Fig. 6. Test images used in our experiments. From left to right and top to bottom: Lena, Car, Baboon, Man, Elaine, Beach, House, and Lock.

(512 × 512), Man (1024 × 1024), Elai ne (512 × 512), Beach (512 × 768), House (512 × 768), and Lock (512 × 768). The PSNR and SSIM results of recovered images are given in Tables I–VII. As can be observed from the tables, the proposed EC algorithm achieves the best performance for all seven types of loss. In particular, the advantages are clearly demonstrated for 16 × 16 regular consecutive loss. The average gain over the second best method is over 0.5 dB in terms of PSNR and 0.0164 in terms of SSIM. Moreover, our approach achieves up to 2.07 dB higher PSNR and 0.0271


359

TABLE III PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 8 × 8 R EGULAR Consecutive L OSS . B EST M ETHOD I S IN B OLDFACE

TABLE IV PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 16 × 16 R EGULAR Consecutive L OSS . B EST M ETHOD I S IN B OLDFACE

TABLE V PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 8 × 8 R ANDOM Consecutive L OSS . B EST M ETHOD I S IN B OLDFACE

higher SSIM than the well-known OAI method. When compared with the recently proposed FSTV, SLP, VC, and MKDE, our algorithm obtains a maximum gain of 2.09, 2.08, 1.50, and 1.63 dB in terms of PSNR and a maximum gain of 0.0393,

0.0458, 0.0372, and 0.0243 in terms of SSIM. We attribute this remarkable performance to the flexibility of BIC-based adaptive predictor and well-designed scan order based on uncertainty.

360


TABLE VI PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 16 × 16 R ANDOM Consecutive L OSS . B EST M ETHOD I S IN B OLDFACE

TABLE VII PSNR AND SSIM C OMPARISON OF EC A LGORITHMS W ITH 16 × 16 R ANDOM Slice L OSS . B EST M ETHOD I S IN B OLDFACE

To better represent the superiority performance of the proposed algorithm, subjective quality comparisons are given in Figs. 7–11. Fig. 7 compares the performance of the proposed algorithm with some classical EC algorithms, including POCS, MRF, AVC, CAD, and BBF. Severe blocking artifacts are observed using POCS, AVC, and CAD. Although the blocking artifacts are weaken in the filled-in images using MRF and BBF, they produce blurred and lumpy boundaries, as shown around the shoulder of Lena. Fig. 8 presents the restored images by the proposed method and edge-directed methods, including OAI and VC. It is observed that OAI produces a few blurred sparkles throughout the image, which heavily degrade visual quality. VC over-emphasizes the edges and produces many pseudoedges. In Fig. 9, the visual comparisons with kernel-based SLP and MKDE are provided. SLP incorrectly estimates the direction of eye boundaries and MKDE introduces blocking artifacts. In addition, only the proposed algorithm produces a natural transition between the eye and cheek. The visual comparisons with the image inpainting

algorithms INP and FSTV are also provided in Fig. 10. As shown in the regions marked in green ellipses, INP produces mussy restored images due to incorrect exemplars and total-variation based FSTV heavily blurs the corrupted blocks. However, the restored image using our algorithm is visually more plausible and coherent. A detailed comparison between the proposed method and the image inpainting algorithms will be given in Section V later. To give more general visual comparison, we present zoomed-in results of all 12 EC algorithms in Fig. 11. It is evident that the proposed approach is the only one algorithm that completely recovers the wheel and baffle. Subjective performance is also evaluated by human viewing test. Degradation category rating (DCR) [44] is used here. The subjective test is carried out on Dell XPS ONE 2710 with the monitor resolution of 2560 × 1440. The graphics card is NVIDIA GeForce GT 640 M. The test is conducted in a workspace environment under formal indoor illustration levels. The source images and the recovered images are presented in


361

Fig. 7.

Subjective comparison of different EC algorithms on Lena with 16 × 16 random consecutive loss.

Fig. 8.

Subjective comparison of different EC algorithms on Baboon with 16 × 16 regular consecutive loss.

Fig. 9.

Subjective comparison of different EC algorithms on Man with 16 × 16 random slice loss. Zoomed-in details are in the lower right corners.

pairs and displayed side-by-side on the same monitor. Fifteen subjects are asked to evaluate the impairment of the recovered images with respect to the source reference. The 21 grade scale is used, where five verbally defined quality categories (i.e., Imperceptible, Perceptible but not annoying, Slightly annoying, Annoying, and Very annoying) are used as labels

for every five grades on the scale. When the subject finishes voting for one pair, a gray image is presented for 2 seconds. Then, the next pair is displayed. The averaged DCR results for regular isolated, regular consecutive, random consecutive, and slice loss are shown in Fig. 12. Nine competitive EC algorithms including the proposed algorithm are involved here.

362


Fig. 10.

Subjective comparison of different EC algorithms on Lock with 8 × 8 regular isolated loss. Zoomed-in details are in the lower right corners.

Fig. 11.

Subjective comparison of different EC algorithms on zoomed-in Car with 16 × 16 regular isolated loss.

Fig. 12. Averaged DCR results for different types of loss. The scores are normalized to [0, 1] here.

It is observed that the proposed algorithm obtains higher scores than the others as expected. B. Test on Videos The proposed algorithm is also utilized to recover corrupted videos to demonstrate its effectiveness when blocking artifacts and blurring caused by compression are present. Three sequences with different resolutions and different levels of

activities are tested: Aki yo (352 × 288), Blowi ng Bubbles (416×240), and RaceH or ses (832×480). The first 30 frames of each sequences are encoded using H.264/AVC reference software JM18.5. Since SEC is mainly employed for intracoded frames, all frames are intracoded. Each 16 × 16 block is transmitted in a separate packet. Packet loss are randomly generated at the packet loss rates (PLRs) of 5%, 10%, and 15%. Quantization parameter (QP) is set to 22, 27, 32, and 37, respectively. Table IX presents the averaged PSNR and SSIM results of each test sequence using different SEC algorithms. Fig. 13 gives the PSNR and SSIM results of Aki yo frame-by-frame. The video is encoded using QP = 32 and the PLR is 10%. Four most recently proposed EC algorithms are involved and the proposed algorithm outperforms them on average in terms of both PSNR and SSIM. Subjective results are further presented for visual comparison in Fig. 14. The 22nd frame of Aki yo with QP = 32 and PLR = 10% is presented here. Five competitive EC methods including the proposed one are compared. As shown in the regions surrounded by the green ellipses, the continuity


363

TABLE VIII AVERAGE RUN T IME ( IN S ECONDS ) C OMPARISON FOR D IFFERENT EC A LGORITHMS AND D IFFERENT T YPES OF L OSS . F OUR 512 × 512 I MAGES , I NCLUDING Lena, Car, Baboon, AND Elaine A RE T ESTED

Fig. 13. Frame-by-frame PSNR and SSIM results of Akiyo with QP = 32 and PLR = 10%.

of hair boundary is broken using FSTV, SLP, and MKDE. VC preserves the continuity but creates wrong curves. For the proposed algorithm, the broken curve is restored with imperceptible artifacts. C. Run-Time Comparison To compare the run time of different EC algorithms, four 512 × 512 images (Lena, Car, Baboon, and Elaine) are tested. The averaged run time for different loss pattern is presented in Table VIII. The computation time reported in the table is obtained with nonoptimized MATLAB implementations with Intel CORE i7-3770S 3.1 GHz CPU and 8 GB memory.2 We can see that our algorithm is much faster than the recently proposed VC and MKDE algorithms. It also outperforms SLP when dealing with 16 × 16 block loss. Although the proposed algorithm requires longer time than some methods (e.g., OAI and BBF), its advantages over other methods are obvious in terms of objective and subjective evaluations, as shown in Sections IV-A and IV-B. Moreover, our algorithm is highly parallel-friendly: the sequence of linear predictors can be derived independently. One of our future works is certainly to exploit parallel-programming to speedup our program. V. D ISCUSSION Recently, an image restoration algorithm called inpainting is widely researched. It is designed to restore lost regions of arbitrary shape, hence, being applicable to the EC problem. Image inpainting algorithms can be generally separated into two types: local and nonlocal, depending on the location of information utilized. Local inpainting algorithms usually employ the gradients [41], [45] and perform well when filling in the narrow gaps, such as corruption caused by characters. 2 FSTV is excluded here since it is implemented in C language.

Fig. 14. Subjective comparison for the 22nd frame of Akiyo with QP = 32 and PLR = 10%. Zoomed-in details are in the lower right corners.

However, it fails when recovering 8 × 8 or 16 × 16 blocks, as shown in Fig. 10. For nonlocal inpainting algorithms, the major disadvantage is the high computational complexity, which is a fatal flaw for EC algorithms. It costs around 10 min to inpaint a 320 × 480 image with 30% loss using [46] or inpaint a 512 × 512 images with 25% loss using INP. The high-computational complexity is often caused by searching for the desired information (e.g., searching for exemplars in INP). In addition, nonlocal inpainting algorithms often employ intricately determined inpainting order, which is time consuming. Moreover, hierarchical or multiscale processes are always involved, such as [46] and [47]. Reference [46] even inpaints the image several times using different parameter settings. All these procedures are designed for large patch loss, which is commonly seen in inpainting problems. However, as for the EC, the block loss is relatively small for most of the time. Therefore, the latent flaw associated with the proposed algorithm, i.e., error propagation, can be well alleviated or avoided by a carefully designed scan order. In addition, the adaptive predictor designed within limited complexity enjoys the flexibility to capture local contexts and gives smart prediction. In other words, our EC algorithm strikes a balance between the computational complexity and recovery performance. VI. C ONCLUSION In this paper, we develop a novel SEC algorithm based on BIC. The missing pixels are sequentially recovered using adaptive linear predictor, whose order and support shape are automatically tuned according to image contexts. The predictor order and support shape are determined by solving

364


TABLE IX AVERAGED PSNR AND SSIM R ESULTS OF V IDEO S EQUENCES W ITH D IFFERENT PLR AND QP. B EST M ETHODS A RE IN B OLDFACE

a model selection problem with BIC. A novel scan order is also suggested to alleviate error propagation problem. Our proposed algorithm achieves better performance than other state-of-the-art techniques in terms of objective and subjective evaluations.

R EFERENCES [1] W. B. Pennebaker and J. L. Mitchell, JPEG: Still Image Data Compression Standard. New York, NY, USA: Springer-Verlag, 1992. [2] J. L. Mitchell, C. Fogg, D. J. LeGall, and W. B. Pennebaker, MPEG Video Compression Standard. New York, NY, USA: Springer-Verlag, 1996.


[3] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, “Overview of the H.264/AVC video coding standard,” IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 560–576, Jul. 2003. [4] Y. Wang, S. Wenger, J. Wen, and A. K. Katsaggelos, “Error resilient video coding techniques,” IEEE Signal Process. Mag., vol. 17, no. 4, pp. 61–82, Jul. 2000. [5] L. Zhang, Q. Peng, X. Wu, and Q. Wang, “SSIM-based error resilient video coding over packet-switched networks,” J. Signal Process. Syst., vol. 74, no. 1, pp. 103–113, Jan. 2014. [6] J. Xiao, T. Tillo, and Y. Zhao, “Real-time video streaming using randomized expanding Reed–Solomon code,” IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 11, pp. 1825–1836, Nov. 2013. [7] A. Peinado, A. Gomez, and V. Sanchez, “Error concealment based on MMSE estimation for multimedia wireless and IP applications,” in IEEE Int. Symp. Pers., Indoor Mobile Radio Commun., 2008, pp. 1–5. [8] Y. Zhang, X. Xiang, D. Zhao, S. Ma, and W. Gao, “Packet video error concealment with auto regressive model,” IEEE Trans. Circuits Syst. Video Technol., vol. 22, no. 1, pp. 12–27, Jan. 2012. [9] P. Salama, N. B. Shroff, E. J. Coyle, and E. J. Delp, “Error concealment techniques for encoded video streams,” in Proc. IEEE Int. Conf. Image Process. (ICIP), vol. 1. Oct. 1995, pp. 9–12. [10] H. Sun and W. Kwok, “Concealment of damaged block transform coded images using projections onto convex sets,” IEEE Trans. Image Process., vol. 4, no. 4, pp. 470–477, Apr. 1995. [11] X. Lee, Y.-Q. Zhang, and A. Leon-Garcia, “Information loss recovery for block-based image coding techniques-a fuzzy logic approach,” IEEE Trans. Image Process., vol. 4, no. 3, pp. 259–273, Mar. 1995. [12] W. Zeng and B. Liu, “Geometric-structure-based error concealment with novel applications in block-based low-bit-rate coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 4, pp. 648–665, Jun. 1999. [13] S.-C. Hsia, “An edge-oriented spatial interpolation for consecutive block error concealment,” IEEE Signal Process. Lett., vol. 11, no. 6, pp. 577–580, Jun. 2004. [14] X. Li and M. T. Orchard, “Novel sequential error-concealment techniques using orientation adaptive interpolation,” IEEE Trans. Circuits Syst. Video Technol., vol. 12, no. 10, pp. 857–864, Oct. 2002. [15] J. W. Park and S. U. Lee, “Recovery of corrupted image data based on the NURBS interpolation,” IEEE Trans. Circuits Syst. Video Technol., vol. 9, no. 7, pp. 1003–1008, Oct. 1999. [16] J. Park, D.-C. Park, R. J. Marks, II, and M. A. El-Sharkawi, “Recovery of image blocks using the method of alternating projections,” IEEE Trans. Image Process., vol. 14, no. 4, pp. 461–474, Apr. 2005. [17] G. Zhai, X. Yang, W. Lin, and W. Zhang, “Bayesian error concealment with DCT pyramid for images,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 9, pp. 1224–1232, Sep. 2010. [18] W.-M. Lam, A. R. Reibman, and B. Liu, “Recovery of lost or erroneously received motion vectors,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 5. Apr. 1993, pp. 417–420. [19] K. Song et al., “Efficient multi-hypothesis error concealment technique for H.264,” in Proc. IEEE Int. Symp. Circuits Syst. (ISCAS), May 2007, pp. 973–976. [20] J. Zhang, J. F. Arnold, and M. R. Frater, “A cell-loss concealment technique for MPEG-2 coded video,” IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 4, pp. 659–665, Jun. 2000. [21] X. Qian, G. Liu, and H. Wang, “Recovering connected error region based on adaptive error concealment order determination,” IEEE Trans. Multimedia, vol. 11, no. 4, pp. 683–695, Jun. 2009. [22] W.-Y. Kung, C.-S. Kim, and C.-C. J. Kuo, “Spatial and temporal error concealment techniques for video transmission over noisy channels,” IEEE Trans. Circuits Syst. Video Technol., vol. 16, no. 7, pp. 789–803, Jul. 2006. [23] M. Ma, O. C. Au, S.-H. G. Chan, and M.-T. Sun, “Edge-directed error concealment,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 3, pp. 382–395, Mar. 2010. [24] J. Koloda, J. Ostergaard, S. H. Jensen, V. Sanchez, and A. M. Peinado, “Sequential error concealment for video/images by sparse linear prediction,” IEEE Trans. Multimedia, vol. 15, no. 4, pp. 957–969, Jun. 2013. [25] J. Seiler and A. Kaup, “Fast orthogonality deficiency compensation for improved frequency selective image extrapolation,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Mar./Apr. 2008, pp. 781–784. [26] S. Shirani, F. Kossentini, and R. Ward, “An adaptive Markov random field based error concealment method for video communication in an error prone environment,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 6. Mar. 1999, pp. 3117–3120.

365

[27] G. Zhai, J. Cai, W. Lin, X. Yang, and W. Zhang, “Image errorconcealment via block-based bilateral filtering,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Apr. 2008, pp. 621–624. [28] H. Asheri, H. R. Rabiee, and M. H. Rohban, “Signal extrapolation for image and video error concealment using Gaussian processes with adaptive nonstationary kernels,” IEEE Signal Process. Lett., vol. 19, no. 10, pp. 700–703, Oct. 2012. [29] J. Koloda, V. Sánchez, and A. M. Peinado, “Spatial error concealment based on edge visual clearness for image/video communication,” Circuits, Syst., Signal Process., vol. 32, no. 2, pp. 815–824, Apr. 2013. [30] J. Koloda, A. M. Peinado, and V. Sánchez, “On the application of multivariate kernel density estimation to image error concealment,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2013, pp. 1330–1334. [31] H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Trans. Image Process., vol. 16, no. 2, pp. 349–366, Feb. 2007. [32] C. Kervrann and J. Boulanger, “Local adaptivity to variable smoothness for exemplar-based image regularization and representation,” Int. J. Comput. Vis., vol. 79, no. 1, pp. 45–69, Aug. 2008. [33] X. Wu, G. Zhai, X. Yang, and W. Zhang, “Adaptive sequential prediction of multidimensional signals with applications to lossless image coding,” IEEE Trans. Image Process., vol. 20, no. 1, pp. 36–42, Jan. 2011. [34] G. Schwarz, “Estimating the dimension of a model,” Ann. Statist., vol. 6, no. 2, pp. 461–464, 1978. [35] C. M. Bishop, Pattern Recognition and Machine Learning, vol. 1. New York, NY, USA: Springer-Verlag, 2006. [36] E. Y. Lam and J. W. Goodman, “A mathematical analysis of the DCT coefficient distributions for images,” IEEE Trans. Image Process., vol. 9, no. 10, pp. 1661–1666, Oct. 2000. [37] N. Memon, D. L. Neuhoff, and S. Shende, “An analysis of some common scanning techniques for lossless image coding,” IEEE Trans. Image Process., vol. 9, no. 11, pp. 1837–1848, Nov. 2000. [38] V. Varsa, M. M. Hannuksela, and Y.-K. Wang, Non-Normative Error Concealment Algorithms, document Rec. ITU-T VCEGN, Geneva, Switzerland, 2001. [39] Z. Rongfu, Z. Yuanhua, and H. Xiaodong, “Content-adaptive spatial error concealment for video communication,” IEEE Trans. Consum. Electron., vol. 50, no. 1, pp. 335–341, Feb. 2004. [40] A. Criminisi, P. Perez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200–1212, Sep. 2004. [41] K. Papafitsoros, C. B. Schoenlieb, and B. Sengul, “Combined first and second order total variation inpainting using split Bregman,” Image Process. On Line, vol. 2013, pp. 112–136, Jan. 2013. [Online]. Available: http://dx.doi.org/10.5201/ipol.2013.40 [42] S. Bhat. (2004). Object Removal by Exemplar-Based Inpainting. [Online]. Available: http://www.cc.gatech.edu/~sooraj/inpainting/, accessed Sep. 14, 2013. [43] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004. [44] Subjective Video Quality Assessment Methods for Multimedia Applications, document Rec. ITU-T, Geneva, Switzerland, 1999. [45] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” in Proc. 27th Annu. Conf. Comput. Graph. Interact. Techn., 2000, pp. 417–424. [46] O. Le Meur, M. Ebdelli, and C. Guillemot, “Hierarchical superresolution-based inpainting,” IEEE Trans. Image Process., vol. 22, no. 10, pp. 3779–3790, Oct. 2013. [47] Y. Wexler, E. Shechtman, and M. Irani, “Space-time completion of video,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 3, pp. 463–476, Mar. 2007.

Jing Liu received the B.E. degree from Shanghai Jiao Tong University, Shanghai, China, in 2011, where she is currently working toward the Ph.D. degree with the Institute of Image Communication and Network Engineering. Her research interests include image and video processing.

366


Guangtao Zhai (M’10) received the B.E. and M.E. degrees from Shandong University, Jinan, China, in 2001 and 2004, respectively, and the Ph.D. degree from Shanghai Jiao Tong University, Shanghai, China, in 2009. He was a Student Intern with the Institute for Infocomm Research, Singapore, from 2006 to 2007; a Visiting Student with the School of Computer Engineering, Nanyang Technological University, Singapore, from 2007 to 2008; a Visiting Student with the Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON, Canada, from 2008 to 2009, where he was a Post-Doctoral Fellow from 2010 to 2012; and a Humboldt Research Fellow with the Institute of Multimedia Communication and Signal Processing, University of Erlangen-Nuremberg, Erlangen, Germany, from 2012 to 2013. He is currently a Research Professor with the Institute of Image Communication and Information Processing, Shanghai Jiao Tong University. His research interests include multimedia signal processing and perceptual signal processing. Dr. Zhai received the National Excellent Ph.D. Thesis Award from the Ministry of Education of China in 2012.

Engineering, Shanghai Jiao Tong University. He has authored over 80 refereed papers and holds six patents. His research interests include video processing and communication, media analysis and retrieval, perceptual visual processing, and pattern recognition. Dr. Yang is a member of the Visual Signal Processing and Communications Technical Committee of the IEEE Circuits and Systems Society. He received the Microsoft Young Professorship Award in 2006, the Best Young Investigator Paper Award at the IS&T/SPIE International Conference on Video Communication and Image Processing in 2003, and several awards from the Agency for Science Technology and Research, Singapore, and the Tan Kah Kee Foundation. He was the Special Session Chair of Perceptual Visual Processing of the IEEE International Conference on Multimedia & Expo in 2006. He was the Local Co-Chair of the ChinaCom 2007 and the Technical Program Co-Chair of the IEEE Workshop on Signal Processing Systems 2007. He actively participates in the International Standards such as MPEG-4, JVT, and MPEG-21.

Bing Yang received the B.E. degree from Wuhan University, Wuhan, China, in 2011. He is currently working toward the Ph.D. degree with the Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China. His research interests include image and video processing, video coding, and parallel realization with embedded system.

Xiaokang Yang (A’00–SM’04) received the B.S. degree from Xiamen University, Xiamen, China, in 1994; the M.S. degree from Chinese Academy of Sciences, Shanghai, China, in 1997; and the Ph.D. degree from Shanghai Jiao Tong University, Shanghai, in 2000. He was a Research Fellow with the Center for Signal Processing, Nanyang Technological University, Singapore, from 2000 to 2002 and a Research Scientist with the Institute for Infocomm Research, Singapore, from 2002 to 2004. He is currently a Full Professor and the Deputy Director of the Institute of Image Communication and Information Processing with the Department of Electronic

Li Chen (M’13) received the B.S. and M.S. degrees from Northwestern Polytechnical University, Xi’an, China, and the Ph.D. degree from Shanghai Jiao Tong University, Shanghai, China, in 2006, all in electrical engineering. He has been devoted to image completion and inpainting, video frame rate conversion, image deshake, and deblur, under grants from the National Natural Science Foundation of China. He currently focuses on very-large-scale integration (VLSI) for image and video processing. His research interest includes image and video processing, DSP, and VLSI for image and video processing.