Reconstruction of Missing Blocks in JPEG Picture Transmission. M. Ancis and D. D. Giusto. CNlT Lab of Multimedia Communications, DEE, University of Cagliari.
Reconstruction of Missing Blocks in JPEG Picture Transmission M. Ancis and D.D.Giusto CNlT Lab of Multimedia Communications, DEE, University of Cagliari P.zza d’Armi, 09123 Cagliari, Italy { ancis, giusto} @diee.unica.it
ABSTRACT: The paper deals with error concealment in blockcoded image transmission over noisy channels; in particular, it proposes a novel algorithm for missing block reconstruction in the frequency domain. In fact, damaged blocks are recovered by interpolating the DCT coefficients of available neighboring blocks. Coefficient interpolation is investigated in four different variants; median and edge-based interpolation are chosen for their capabilities in high-quality reconstruction. Experimental results show a good performance in homogeneous and textured regions, as well as in blocks containing edges.
The algorithm exploits the information contained in blocks adjacent to the spoiled ones, by interpolating the relevant DCT Coefficients. This work has been carried out by increasing refinements of a same basic approach, under the constraint of low computational complexity and consequently high processing speed for real-time applications. Results obtained by simulating the loss of blocks in different significant zones of an image are presented and show a good performance in different situations (blocks containing edges, different gray levels shades, homogeneous areas, and so on).
Introduction to error concealment DCT-based imagehide0 transmission over noisy channels is subject to block damages that may cause significant picture degradation, also for several frames in video sequences. The only solution to this problem, as retransmission is usually not allowed (real-time applications), consists in the concealment of these damages; to this aim, different interpolation techniques are used for reconstruction of missing data and simulate lost information. The state-of-the-art offers several different approaches, even though researchers firstly faced this problem just recently. Usually, post-processing techniques that perform error concealment can be applied in the spatial or in the DCT domain for intra-coded or still pictures. Scanning the literature, a popular approach that can be found to error concealment is the one based on smoothness constraints on image intensity on the borders of lost block to estimate the DC and low-frequencies coefficients of lost blocks; a basic technique was proposed in [Wang91], and elaborated in successive works [Wang93, Zhu931. Based on this idea are also more recent papers like [Park97, Chung981. [Hemami951 proposed a different technique that takes advantage of the correlation between transformed blocks in an image. Another method to estimate the missing block coefficients is through projections onto convex set, as in [Sun95], or by reconstruction of missing edges [Kwok93, Atzori991. [Salama96] proposes a statistic model of the image as a MRF and estimate the missing coefficients from their neighbours according to the MAP criterion; a sub-optimal solution based on median filtering, which is simpler and computationally less expensive is proposed in [Salama97]. The present paper is devoted to introduce a novel error concealment algorithm aimed at reconstructing a picture block under the assumption of a full damage, that is no a-priori knowledge about lost information is available.
0-7803-5582-2/99/$10.00 01999 IEEE
288
Basic block interpolation The basic idea is to interpolate the values of coefficients of lost blocks yi by averaging the values xi of DCT coefficients in neighboring blocks. The first possibility is to consider those coefficients belonging to the 4-connected blocks (4-connected interpolation), that gives
for i=1,2,...,64; w”’)=l forj=2,4,5,7 (see Figure 1).
Figure 1. The Scheme of 4-connected interpolation. This results in a good error concealment in homogeneous areas and textured regions, while notable artifacts arise along damaged blocks edges (see Figure 2). Moreover notable errors (step-migration effects) appear when one of the 4-connected blocks contains a marked step-edge, while the others contain homogeneous gray levels values. In this case an absolutely out of place step-edge arises in the reconstructed block (see Figure 3). An improvement on this simple interpolation is obtained by exploiting also the information contained in the remaining 4 near blocks (i.e., 8-connected interpolation).
Figure 2. Details of the 4-connected interpolation results for Lenna image: A good reconstruction quality on smooth (leji) and textured areas (middle) can be appreciated; anyway, some artifacts are generated along edges (right).
Figure 3. Details of the 4-connected interpolation results for Lenna image: The step migration effect. The coefficient estimate is achieved by a weighted average of the values of the Coefficients belonging to the 8-connected blocks near the damaged one (see Figure 4). The 4-connected blocks coefficients have unitary weights while the weights associated to
Figure 5. Details of the 8-connected interpolation results for Lema image: Compared to 4-connected interpolation results, a better reconstruction quality can be appreciated on smooth (le$) and textured areas (middle); on the other end, annoying artifacts are still generated along edges (right).
Figure 6. Details of the 4-connected interpolation results for Lenna image: The step migration effect is less evident. The resulting expression for the missing coefficients is:
the other values are equal to 1/ fi.
As result, the step edge migration effect is absent in the processed picture (see Figure 8), while a loss of details is still present in blocks containing edges (see Figure 7), due to homogeneous gray levels shading off. The reconstruction of homogeneous regions is quite accurate.
11=1,2 ,...,64
w”’)=lforj=2,4,5,7 and w ” ’ = l / f i forj=1,3,6,8. The introduction in the estimation of these four more values results in a picture quality improvement by reducing the step edge migration effect (see Figure 3,even if blocks containing edges are still interpolated in an homogeneous way, so introducing an annoying tessellation effect (see Figure 6). Figure 7. Details of the 8-connected hybrid average-median interpolation for Lenna image: Accurate reconstruction on smooth (leji) and textured areas (middle); detail loss in edge blocks (homogeneously shaded offgray level values) (right).
1 1 1 1 ;
~
__ . i ______ I. .._i
t. _. i
Figure 4. The scheme of 8-connected interpolation.
Hybrid average-medianinterpolation The main drawback in linear interpolation lies in the step migration effect; to avoid this kind of distortion, a 3x3 hybrid average-median filter is applied for estimation, by taking into account the corresponding coefficients into each of the 8connected blocks (see Figure 4). For each coefficient, the 8 values of the corresponding coefficients in the 8-connected blocks are ordered and the average of the 4-th and the 5 t h value is taken as estimation of the unknown value.
289
Figure 8. Details of the 8-connected hybrid average-median interpolationfor Lenna image: The step migration effect is missing.
Edge-based interpolation To take into major account the possible presence of edges, a selection of coefficients, belonging to block containing edges, is applied before averaging (edge-based interpolation). The four
possible edge directions are investigated for each missing blocks, using a simple differential operator on the pixel in the half block near the missing one; a thresholding process is used to identify possible edges. Figure 11. Details of the edge-based interpolation results of Lenna image: The step migration effect is missing.
Experimental results Figure 9. Vertical edge identijkation. In particular, vertical edges ( v = l ) are searched in the two blocks above and under the damaged one (see Figure 9). Horizontal edges ( h = l ) are searched in the two blocks on the left and on the right side of the damaged one; descendant diagonal edges ( d = l ) are searched in the two blocks on the left side above and on the right side under the damaged one; ascendant diagonal edges ( a = l ) are searched in the two blocks on the left side under and on the right side above the damaged one. If one or more edges are found around the missing block, the DCT coefficients of the blocks containing edges are taken into account in estimation, by a weighted average, as in the following espression:
This section is devoted to present some global results of the four missing block coefficient interpolation algorithms proposed; a JPEG-coded version of Lema image (presented in Figure 12) is used, and a loss of 13 blocks located in areas with different characteristics was simulated. In the following pictures, as well in the previous one devoted to illustrate details of the effect of local interpolation, reconstructed blocks are indicated by small arrows.
Ii=l,2,...,64 where the weights are set as wj=l forj=4,5 if v=l; w,=l forj=2,7 if h=l; wj=l forj=3,6 if a = l ; w,=l forj=1,8 if d = l . In the case no edges are found for a missing block, the corresponding coefficients are estimated through the hybrid average-median interpolation described above. Relevant results, presented in Figures 10 and 1 1 , show that significant edges are reconstructed with a nice visual quality, while other kinds of artifacts remain concealed by the hybrid average-median filtering.
Figure 10. Details of the edge-based interpolation results of Lenna image: Compared to previous results, a slightly better reconstruction quality on smooth (lef) and textured areas (middle) can be appreciated, while a far better pegormanee is obtained on blocks containing edges (right).
290
Figure 12. A JPEG-coded version of Lenna image. The first algorithm achieves the result presented in Figure 13; one may notice some reconstruction defects on the shoulder, and in the right part of the hat, where some edges arise, in expected homogeneous areas. Otherwise, visual quality for reconstructed blocks is good. By extending interpolation to 8-connected blocks (see Figure 14), previous defects remain visible, even that less enhanced; in particular, step edges on shoulder and hat are less evident. The median-interpolatedpicture presented in Figure 15 shows an almost perfect reconstruction of shoulder and hat blocks, with a good visual quality in other blocks. Finally, result obtained by edge-based interpolation, presented in Figure 16, show an overall very good reconstruction quality, of edges and textured regions, as well as of homogeneous ones. In particular, among all reconstructed blocks, one may notice the quality of those ones on the eyebrow and the lip. For the block located in the plume, the reconstruction result is more visually pleasant that in former cases, as the technique is able to reconstruct the descending behavior. The annoying step-edge effects on the hat and the shoulder are also avoided by the median interpolation used there.
Acknowledgments This work has been supported by the Italian Ministry for University and Research (project: Multimedia Communications).
References [Wang91] Y.Wang, Q.Zhu, Proc. SPIE-VCIP'91, vol. 1605, pp. 667-678, November 1991 [Zhu93] Q.Zhu, Y.Wang, L.Shaw, IEEE Trans. CAS-VT, vol. 3, pp. 248-258, June 1993 [Kwok93] W.Kwok, H.Sun, IEEE Trans. CE, vol. 3, pp. 455460, August 1993 [Wang93] Y.Wang, Q.Zhu, LShaw, IEEE Trans. COM, vol. 41, pp. 1544-1551, October 1993
[Sun951 H.Sun, W.Kwok, IEEE Trans. IP, vol. 4, pp. 470-477, April 1995 [Hemami951 S.S.Hem-9 T.H.-Y.Meng, IEEE Trans. Ip, vol. 43 _pp. _ 1023-1027, July 1995 [Salama96] P.Salama, N.B.Shroff, E.J.Delp, Proc. IEEEICIP'96, pp. 49-52, September 1996 [Salama97] P.Salama, N.B.Shroff, E.J.Delp, Proc. IEEEICIP'97, pp. 11.101-104, October 1997 [Park971 J.W.Park, J.W.Kim, S.U.Lee, IEEE Trans. CAS-VT, pp. 845-854, December 1997 [Chung98] Y.J.Chung, J.W.Kim, C.-C. J.Kuo, Proc. IEEEISCAS'98 [Atzori99] L.Atzori, F.G.B.DeNatale, Signal Processing: Image Communication, 1999 (in press)
Figure 13. Result by 4-connected interpolation.
Figure 15. Result by 8-connected median interpolation.
Figure 14. Result by 8-connected interpolation.
Figure 16. Result by edge-based interpolation
29 1
