Concealment of Visual Effects of Image Transmission Errors by a Sketch-Based Recovery Approach Luigi Atzori, Francesco G.B. De Natale
Dept. of Electrical and Electronic Engineering, University of Cagliari Piazza d’Armi, 1-09123 Cagliari, Italy e-mail: gigi@ -
[email protected] The layered coding schemes belong to the first category, and are typical solutions to ensure a minimum guaranteed level of quality. They are based on partitioning the picture information into a base layer, that conveys coded video at low bit rate, and in higher layers that provide additional information to progressively increase the quality to the desired maximum. To achieve this goal, some schemes make use of subband coding approaches [ 2 ] , while others are targeted to specific applications and standards (e.g., MPEG video stream transmission over packet based networks [3]). In conjunction or substitution of robust coding scheme, concealment techniques at the reception side can be applied. In [4], spatially correlated information from a large region surrounding the damaged area is exploited by means of an iterative process in the frequency domain, based on the method of projections onto convex sets. In [ 5 ] , high-frequency information is recovered by using the theory of fuzzy logic reasoning. A common approach found in proposed concealment algorithms is based on the recovery of the lost information by means of a maximally smooth procedure, most of times integrated by a procedure aimed at enhance the visual quality introducing predicted high frequency information. According to the assumption that for image edges are proven to be fundamental for the preservation of both the semantics and the perceived quality of an image, in this work the problem of recovering as accurately as possible the contour geometry and relative gray level information is addressed. The basis of the proposed approach can be found in the theory of sketch coding, which provides a very efficient image representation methodology. Based on the recovered edge data procedure a low-frequency content recovery is accomplished.
Abstract In this paper, the problem of masking unpleasant visual effects caused by data loss during image transmission over packet network is addressed. Due to the widespread use of block-based algorithm in image coding, such loss results in a block or sequence of blocks missing in the decoded image. In order to improve the quality of the image subject to cell or packet loss, appropriate concealment algorithms can be satisfactorily applied. The technique proposed is based on a sketch representation of the visual information associated to the high-frequency content. The proposed method has been applied with good results in the transmission of both video and still pictures.
1. Introduction Transmission of visual data (still pictures and video sequences) over fast packet networks, such as the Internet and the ATM networks, introduces heavy fidelity problems, produced mainly by packet or cell losses. In high-speed packet switching, such a issue depends often on electrical/mechanic problems at the physical layer (e.g., noise, interference) and on link congestion problems at the network layer (e.g., traffic peaks) [l]. Since the signal transmitted is highly compressed, a cell loss can produce catastrophic effects in the decoded data. Besides, in video transmission the loss of coded data implies not only the unsuccessful reconstruction of the relevant visual data but often temporal error propagation effects. It is so a matter of fact the need of techniques aimed at making the transmission more robust or at minimizing the effects of errors or cell losses. The possible solutions can be grouped into two categories: packet-loss resilient coding techniques, which make possible to decode the stream also in the presence of errors, and error concealment algorithms, which mask the effect of corrupted blocks by means of an appropriate algorithm applied during the decoding phase.
0-8186-8821-1/98 $10.00 0 1998 IEEE
2. Proposed error concealment approach The proposed error concealment algorithm is based on the theory of sketch image representation, that has been successfully employed in low bit-rate image coding, and
482
- detection of the sketch information in a region of
exploits these concepts to extend high frequency information from the available visual data to the missing area. Belonging to the class of second-generation image coding techniques, the sketch-based technique aims at describing the visual data making use of basic primitives that can be extracted from an image, such as contours, regions or textures. With this end only the most significant information for the semantics and perceived quality are retained during the coding process. In particular the sketch based algorithm codes the edge geometry information combined with the relevant gray level information across them. The ensemble of these two data constitutes the image sketch, and is a perceptually very rich representation of the original image. In [6] Carlsson proposed to code only the gray level intensities of the one-pixel wide strip along the right and left side of each edge chain, representing in such a way the sharp transitions in the gray level intensity. Once the contour gray level intensities are provided, the reconstruction of the smooth area should be accomplished by means of a maximally smooth criterion. Being sketch information the basic content in visual data, the approach of the above-mentioned image coding technique can be successfully exploited in recovering the high-frequency content in the lost area. Actually, if such information is recovered in a way, the whole lost image block can be reconstructed with good accuracy. Once the sketch information in the lost area is available then the gray level information in the area between the edges is easily recovered by means of an interpolation procedure.
interest and selection of a subset of the sketch elements that potentially extend into the missing area; - the candidate sketch elements are conveniently coupled considering several contiguity criteria, in order to recover the contour geometry information in the lost area; - each couple is connected across the lost block, and the gray level information of the interpolated curves is reconstructed to complete the sketch approximation.
3.1. Extraction of significant sketch elements The extraction of the surrounding sketch information is accomplished by means of an edge detection procedure that constitutes the basic task for the recovering procedure, influencing heavily the subsequent steps. It is well known that in real images gradient peaks are not always related to object boundaries, but are often caused by noise and textures. Such peaks can generate false edges, which can seriously compromise the effectiveness of the following processing steps. In our scheme, the Canny edge extractor [7] was chosen for several reasons: it allows to detect one pixel wide contour lines (providing a good localization of the edge), the extracted edge lines show a sufficient connectivity, and smoothed edges are provided. The filter is applied in a narrow area surrounding the lost blocks in order to detect only the edge segments that are of potential interest. Among the edges detected in the surrounding area, the segments that end on the border of the missing image area are the selected candidates for the following interpolation procedures. The final set of contour segments is then integrated with the gray level information that is easily obtained.
3. High-frequency content recovery Lost Area
3.2. Prediction of the lost sketch geometry J
Once the sketch information have been extracted in the area surrounding the missing block the lost edge geometry has to be predicted, localizing the possible connections among the edges selected in the previous operation. The most likely pairs of sketch segments are identified (or, in rare cases, the links among three or more segments) making use of several parameters, including shape, distance, gray level intensities, edge strength, and so on. Some typical properties of natural images have been identified that proved to be useful in sketch prediction: i) smoothness of the area encompassed by edges: the intensity of the areas on each side of the edge shows a slow luminance variation;
J
Figure 1. Sketch recovery procedure.
As already mentioned the recovery of the highfrequency content in the missing blocks is accomplished by means of a sketch recovering procedure that exploits only spatial correlation with the surrounding available information. The method we propose is applied through three main steps (see Fig. 1):
483
ii) smoothness of the edges: natural object’s contours show a slow direction variation; iii) closure of the edges: in natural images, rarely contour segments terminate abruptly.
where
v k is the standard deviation of
fi5 computed
inside the set of extracted sketch elements. The features that after the normalization still exceed the value 1.0 are clipped. Then for each couple the two cost functions should be added in order to obtain the coupling cost function F]:,j :
Having defined the cost function, the coupling strategy can be easily outlined based on property (iii) of natural edges. Since the abrupt termination of an edge is highly improbable and the intersections of more than two edges are less frequent, the coupling procedure consists in progressively linking couples of edges, searching for “minimum cost couples”. Thus, the procedure is carried out by choosing the couple with the lowest values of F,,j between the sketch set, iteratively until none or one edge is left:
> J
get the lowest Fi,jwithin the sketch set eliminate the couple i - j from the set if remaining elements > 1 start again else exit
Figure 2. Sketch elements coupling configuration.
The above procedure provides the couples of edges that potentially belong to a shape and in some cases one single edge is provided. In this case this edge represents a contour that naturally ends whiting the lost area without being connected with other edge or represents an edge broken by another contour.
Properties (i) and (ii) take into account the luminance and the geometrical information contained in the set of sketch elements extracted, respectively. They are quantified with local measures that are used to evaluate their similarity. Property (iii) is a more global parameter, and is used to formulate the coupling strategy. The cost term relevant to the smoothness of the area encompassed by edges (property (i)) is expressed as follow:
3.3. Sketch recovery The recovery of the lost edge connections is accomplished by a polynomial interpolation applied to the segment couples defined in the previous step. Furthermore, single segments remaining after the coupling procedure are extended into the missing area. The maximum number of points available for the interpolation is equal to 2xm, where m is the maximum number of edge points considered for each segment (12, in our tests). Since this set of points is overabundant for an efficient interpolation, the edge segments are previously under-sampled, choosing a subset of n points (typically, n=6). An effective solution is to use a cubic spline, which accomplishes the interpolation of successive couples of points by a third degree polynomial, so as to avoid critical oscillations. A further problem, is the presence of unmatched edge segment. A possible strategy is to manage these cases at the end of the recovery, by simply extending the curve within the lost area until another edge is found. The broken segment can be extrapolated by propagating the last spline segment close to the lost block.
where mi,l and mi,,represent the mean luminance intensity along the left and right sides respectively of the i-th sketch segment (see Fig. 2). The term introduced to take into account the property (ii) is a measure of the total curvature of the recovered edge segment, that is expressed as follows:
where Siis the angle between the tangent of the edge segment i at the point A and the rectilinear segment AB (see Fig. 2), and analogously for 4. Since the two measures fi:j and fi:i have different ranges and variance, a normalization is applied to make them homogeneous:
484
Once the edge geometry has been recovered, the gray level values associated to the edges should be predicted to complete the reconstruction of the sketch. In particular, the two pixel-wide strips along the recovered edge are again interpolated by means of a cubic spline, this time applied to the gray level as a function of the contour length.
4. Low-frequency content recovery The recovered sketch data allow to easily recover the visual low-frequency content. A fast procedure that gives sufficiently accurate results consists in applying a oneshot interpolation. To each non-contour pixel the weighted mean of the nearest known pixels in the four main directions (N,W,S,E) is assigned, as shown in Fig. 3. The weight associated to each direction simply depends on the distance between the pixel to be recovered and the nearest available pixel in that direction (either a recovered sketch point or a border point), namely:
Block loss rate
PSNR
Maskerade
14.2%
30,61
Lena
14.06%
32,41
Salesman
14.14%
30,85
Image
5. Conclusions A concealment technique was presented, aimed at restoring the lost visual information in the transmission of coded images and video over unreliable networks. The technique is based on the interpolation of the sketch information recovered from the available data surrounding the damaged blocks. The computational complexity of the method is mainly due to the edge extraction operation, which should be appropriately simplified in a possible hardware or DSP implementation. Nevertheless, the image area involved in the concealment operation is very restricted, thus allowing a very fast computation, even in more complex cases. Further studies are currently being developed in order to extend the validity of the method in particular situations like the loss of large block sequences, or the unavailability of part of the surrounding area.
Figure 3. Low-frequency visual content recovery.
5. Experiments The proposed method has been tested with several pictures and video frames, characterized by different resolutions and complexity. Simulations were performed on gray level images, while the extension to color images can be simply achieved by extracting and interpolating the contour information on the luminance plane and then extracting and interpolating the color sketch (luminance + two chrominance components). The cell loss has been simulated by a regular distribution of single lost blocks
485
Acknowledgments L.Atzori acknowledges the European Social Fund for supporting his Ph.D. fellowship.
[4]
References [5]
Hong, T.D.Suda, “Congestion control and prevention in ATM networks,” IEEE Network Magazine, vol. 3, no. 4, pp. 10-16, July 1991. L.H. Kieu, K.N. Ngan, “Cell-loss concealment techniques for layered video codecs in an ATM network,” IEEE Trans. Image Processing, vol. 3, no. 5, pp. 666-677, Sept.
[6] [7]
1994.
R. Aravind, M.R. Civanlar, A.R. Reibman, “Packet loss resilience of MPEG-2 scalable video coding algorithms,”
IEEE Trans. Circuits and Syst. Video Technol., vol. 6. no 5, pp. 426-435, Oct. 1996. H. Sun, W. Kwok, “Concealment of damaged block transform coded images using projections onto convex sets,” IEEE Trans. Image Processing, vol. 4, no. 4, pp. 470-477, Apr. 1995. X. Lee, Y.-Q. Zhang, A. Leon-Garcia, “Information loss recovery for block-based image coding techniques-A fuzzy logic approach,” IEEE Trans. Image Processing, vol. 4, no. 3, pp. 259-273, Mar. 1995. S. Carlsson, “Sketch based coding of grey level images,” Signal Processing, vol. 15, pp. 57-83, July 1988. J. Canny, “A Computational Approach to Edge Detection,” IEEE Trans. Patt. Anal. Machine Intell., vol. 8 , no. 6, pp. 679-698, NOV.1986.
Figure 4. Results on “Maskerade” test image: (a) enlarged area of the original image, (b) damaged with 15%block loss, (c) recovered sketch information, (d) sketch-based recovered image.
486
