data recovery approach, in which a content-associative signature of a block in ... At the decoder, the embedded data for each corrupted block are extracted and.
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
A Survey of Error Resilient Coding Schemes for Image/Video Transmission Based on Data Embedding Li-Wei Kang*(康立威) and Jin-Jang Leou#(柳金章) *Institute of Information Science Academia Sinica, Taipei 115, Taiwan # Department of Computer Science and Information Engineering National Chung Cheng University Chiayi 621, Taiwan
Abstract For entropy-coded images/video, a transmission error in a codeword will result in a great degradation of the received images/video. Recently, several error resilient coding schemes based on data embedding are proposed, in which some important data for error resiliency can be embedded into the compressed bitstream at the encoder. At the decoder, the important data can be extracted for error resiliency so that high-quality images/video can be recovered. In this article, a survey of error resilient coding schemes for image/video transmission based on data embedding is presented.
1.
Introduction For entropy-coded images/video, a transmission error in a codeword will not
only affect the underlying codeword but also may affect subsequent codewords, resulting in a great degradation of the received images/video. To cope with the synchronization problem, many image/video coding standards [1]-[2] insert synchronization codewords (start codes) into the compressed image/video bitstream. For example, each of the two top layers of the H.264/AVC hierarchical structure, namely, picture and slice, is ahead with a fixed-length start code. After the decoder receives any start code, the decoder resynchronizes regardless of the preceding slippage. However, a transmission error may affect the underlying codeword and its subsequent codewords within the corrupted slice, as three illustrated examples shown in Figs. 1-3 Moreover, because of the use of motion-compensated interframe coding,
-4-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
the effect of a transmission error may be propagated to the subsequent frames, as an illustrated example shown in Fig. 4.
(a)
(b)
Fig. 1. The error-free and corrupted JPEG images (the Y component) of the “Lenna” image with the block loss rate = 10%: (a) the error-free image and (b) the corresponding corrupted image.
(a)
(b)
Fig. 2. The error-free and corrupted JPEG-2000 images of the “Lenna” image with bit rate = 0.4bpp and bit error rate = 0.1%: (a) the error-free image; (b) the corresponding corrupted image.
(a)
(b)
Fig. 3. The error-free and corrupted MPEG-2 video frames (the Y component) of the first frame of the Flower sequence with the bit error rate = 0.1%: (a) the error-free video frame and (b) the corresponding corrupted video frame.
-5-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
(a)
(b)
Fig. 4. The error-free and corrupted H.263 video frames (the Y component) of the sixth frame of the Carphone sequence with the video packet loss rate = 20%: (a) the error-free video frame and (b) the corresponding corrupted video frame.
In general, error resilient approaches include three categories [1]-[2], namely, (1) the error resilient encoding approach, (2) the error concealment approach [3]-[4], and (3) the encoder-decoder interactive error control approach. Recently, several error resilient coding approaches based on data embedding are proposed [5]-[29], in which some important data useful for error resiliency (error detection, correction, and/or concealment) performed at the decoder can be embedded into the compressed image/video bitstream. The embedded data should be “almost” invisible and cannot degrade the image/video quality greatly. At the decoder, the important data will be extracted for error resiliency so that high-quality images/video can be recovered from the corresponding corrupted ones. In this study, a survey of error resilient coding schemes based on data embedding is presented. The following issues should be addressed: at the encoder (1) what kind of important data in an image/video should be extracted and embedded, (2) where should the important data be embedded, (3) how to embed the important data, and (4) at the decoder, how to extract and use the important data for error resiliency. This paper is organized as follows. The error resilient coding schemes for image transmission based on data embedding is addressed in Section 2. The error resilient coding schemes for video transmission based on data embedding is addressed in Section 3, followed by concluding remarks.
2.
Error Resilient Coding Schemes For Image Transmission For a still image, the important data for the image should be embedded into the -6-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
same image [5]-[12]. For example, in [8], the important data for a block in a JPEG image will be embedded into the DCT coefficients of another (remote) block, called the masking block, in the same image. The major design criteria for data embedding in an image can be described as follows. (1) A block and its masking block should be as far as possible so that the two corresponding blocks will be seldom corrupted simultaneously. (2) A block and its masking block should not be in the same synchronization unit. (3) The masking blocks of the blocks in a synchronization unit should not be completely in the same another synchronization unit due to the two synchronization units may be corrupted simultaneously. For DCT-based or JPEG images [5]-[8], Yu and Yin [5] proposed a multimedia data recovery approach, in which a content-associative signature of a block in an image is generated and inserted imperceptibly into another block of the image. At the decoder, the embedded content-associative signature for each corrupted block is extracted and employed to conceal the corrupted block. Yin, Liu, and Yu [6] embedded the block type and edge direction index of a block in an image into the DCT coefficients of another block in the image by the odd-even data embedding scheme. At the decoder, the embedded data for each corrupted block are extracted and employed to conceal the corrupted block by the bilinear interpolation according to its edge direction. In [6], an effective data embedding scheme, called the odd-even data embedding scheme is proposed, which is widely employed by several error resilient coding schemes [6], [8], [13], [23], [25]-[28] and described as follows. If a data bit to be embedded is “0,” the selected quantized DCT coefficient for data embedding will be forced to be an even number. If a data bit to be embedded is “1,” the selected quantized DCT coefficient will be forced to be an odd number. Additionally, only the quantized DCT coefficients larger than a predefined positive integer threshold, T, are used to embed data bits. That is, if the data bit to be embedded is bj and the selected quantized DCT coefficient is Ci, the odd-even data embedding scheme operates as
⎧C i + 1 ⎪ C i = ⎨C i − 1 ⎪ C ⎩ i
if |Ci| > T, Ci mod 2 ≠ bj, and Ci > 0, if |Ci| > T, Ci mod 2 ≠ bj, and Ci < 0, otherwise.
-7-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
The bilinear interpolation is effective for a corrupted block with regular edge direction, but not for a complex corrupted block. To solve this problem, Lin, Sow, and Chang [7] proposed a self-authentication-and-recovery image (SARI) system, in which a watermarked SARI image can detect manipulations, resynchronize the remaining bitstream following the corruption, and recover an approximated original image on the corrupted area by embedding two kinds of information watermarks (authentication information and recovery information), where the recovery information is generated from the down-scaled version of the original image. Additionally, Kang and Leou [8] proposed an error resilient coding scheme for JPEG image based on data embedding and side-match vector quantization (VQ). At the encoder, the important data (the VQ codebook index) for each block in a JPEG image are extracted and embedded into another “masking” block in the image by the odd-even data embedding scheme. At the decoder, after all the corrupted blocks in a JPEG image are detected, if the important data for a corrupted block can be extracted correctly from the corresponding “masking” block, the extracted important data will be used to conceal the corrupted block by the corresponding closest codeword in the codebook; otherwise, the side-match VQ technique is employed to conceal the corrupted block. If the employed codebook is designed well enough, the corrupted complex block will be also well concealed. An illustrated example is shown in Fig. 5, where BLR denotes the block loss rate and M denotes the quality factor for scaling quantization in JEPG. For wavelet-based or JPEG-2000 images [9]-[12], Kurosaki, Munadi, and Kiya [10] proposed an error concealment approach using the layer structure of JPEG-2000, in which the most significant layer is embedded into the least significant layer of the JPEG-2000 bitstream. When the most significant layer is affected by transmission errors, the corrupted data are concealed by the embedded data in the least significant layer. Lu [11] proposed a data embedding technique to offer error resiliency for wavelet-based image transmission. The authentication information generated from the structural digital signature is used for error detection, whereas the recovery information (an approximated version of the original image) is used for error concealment. Kang and Leou [12] proposed two error resilient coding schemes for
-8-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
wavelet-based image transmission based on data embedding and genetic algorithms (GA’s), in which, as JPEG-2000, an image is decomposed into 6 wavelet levels (levels 0-5). For level 0, an error resilient coding scheme based on data embedding and side-match VQ [8] is employed. For level 1, the proposed basic scheme uses the embedded data (the mean value for each code block) to conceal the corrupted code block, whereas the proposed enhanced scheme uses a GA-based error concealment scheme to conceal the corrupted code block. For level 2, the embedded data for a corrupted code block are used to conceal the corrupted code block. For levels 3-5, the wavelet coefficients of each corrupted code block are simply replaced by zeros. An illustrated example is shown in Fig. 6.
(a)
(e)
(b)
(c)
(f)
(g)
(d)
Fig. 5. The error-free and concealed JPEG images (the Y component) of the “Lenna” image with BLR = 10% and M = 75: (a) the error-free image, (b) the error-free image with data embedding; (c)-(g) the concealed images by Zero-S, BNM, H.264, ERDE, and the proposed scheme, respectively.
-9-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i) Fig. 6. The error-free and concealed JPEG-2000 images of the “Lenna” image with bit rate = 0.4bpp, BER = 0.1%, and Nave = 3: (a) the error-free image; (b)-(c) the error-free images with data embedding by the proposed basic and enhanced schemes, respectively; (d) the corrupted image; (e)-(i) the concealed images by Zero-S, Mean-S, Inter, and the two proposed (basic and enhanced) schemes, respectively.
3.
Error Resilient Coding Schemes For Video Transmission For a video, the important data for a video frame can be embedded into the same
frame and/or the next frame(s) [13]-[29]. For example, in [25], for an H.263 I frame, the important data for each MB will be embedded into another MB(s) in the same I frame and/or the next P frame. For a P frame, the important data for all the MBs in a GOB will be embedded into the two corresponding “masking” GOBs in the next frame. The major design criteria for data embedding in a P frame can be described as follows. (1) The important data for the MBs in a GOB should be separately embedded into different masking GOBs in the next frame because a GOB and its masking GOBs may be corrupted simultaneously. (2) The masking GOBs for a GOB should not be consecutive ones because burst GOB losses may occur. For error detection [13]-[16], Gao and Lie [14] proposed an error detection scheme for MPEG-4 video transmission by embedding useful information (e.g., the -10-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
types of motion vector information in MPEG-4 data-partitioned packet) at the encoder so that transmission errors can be detected by consistency-checking at the decoder. For error detection and/or correction/concealment [17]-[29], Song and Liu [20] proposed a data embedding scheme for H.263 video transmission in error-prone channels, in which some redundant information used to protect motion vectors and coding modes of MBs in one frame is embedded into the motion vectors in the next frame. Yilmaz and Alatan [23] proposed an error resilient H.263 video transmission codec
utilizing
imperceptible
embedded
information
for
error
detection,
resynchronization, and reconstruction. For an I frame, both a spatial error recovery technique embedding the edge orientation information of each MB and a resynchronization technique embedding the bit-length of each MB are proposed to recover the corrupted MBs. For a P frame, the embedded motion vector information is used to recover the corrupted MBs. Zeng [24] embedded the best suitable error concealment scheme for each MB at the encoder for H.263 video transmission. At the decoder, the best error concealment scheme for each corrupted MB is used to conceal it. In some recent error resilient coding schemes based on data embedding, a case that both an MB and its important data (embedded into another place) are corrupted simultaneously is not well treated. For example, in [20], a frame-based embedding scheme is proposed, based on the two assumptions that the next frame of a corrupted frame will be correctly received and at most one GOB will be corrupted in a frame. If any of the two assumptions is not valid, the important data for the corrupted frame cannot be correctly extracted. Note that the two assumptions are usually not valid in a noisy channel. In [23], the important data for a GOB is completely embedded into one masking GOB in the next frame. If the GOB and its masking GOB are corrupted simultaneously, the important data for the corrupted GOB cannot be correctly extracted. To cope with the above-mentioned problems, Kang and Leou [25] proposed an error resilient coding scheme for H.263 video transmission. At the encoder, for a P frame, the important data (the coding mode and motion vector information) for each GOB are extracted and embedded into the next frame by the MB-interleaving
-11-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
GOB-based data embedding scheme. If two non-adjacent GOBs in a frame are denoted by GOB A and GOB B, respectively, the important data extracted from the even-number MBs of GOB A and those extracted from the odd-number MBs of GOB B are interleaved by the even-odd order and concatenated to a mixed bitstream, which is embedded into its masking GOB in the next frame. On the other hand, the other corresponding bitstream is also similarly concatenated, which is embedded into another masking GOB in the next frame. At the decoder, for a corrupted GOB, if only one of its masking GOB is corrupted, the even (or odd) MBs of the corrupted GOB can be concealed by using the important data extracted from the “good” masking GOB first. Then the odd (or even) MBs can be concealed by the employed error concealment scheme [3] with more neighboring MB information. Because the corresponding two masking GOBs of a pair of GOBs are usually far apart, i.e., the corresponding two masking GOBs are seldom corrupted simultaneously. Hence, the important data for a corrupted GOB are usually “completely” available or at least “partially” available. Cooperating with the employed error detection and concealment scheme, the better concealed results will be obtained. To obtain more reliable error resiliency, Kang and Leou [27]-[28] proposed an error resilient coding scheme for H.264/AVC video transmission. At the encoder, for a P frame, two types of important data with different error recovery capabilities for each MB are extracted and embedded into the next frame by the proposed MB-interleaving slice-based data embedding scheme, in which the important data for all the MBs in a slice will be embedded into the four corresponding “masking” slices in the next frame. Here, the Type-I data for an MB contains the coding mode, the reference frame(s), and the motion vector(s) for the MB, whereas the Type-II data for the MB includes the best error concealment scheme among 15 “pre-evaluated” error concealment schemes for the MB. At the decoder, if the important data for a corrupted MB can be correctly extracted, the extracted important data will facilitate the employed error concealment schemes [3]-[4] to conceal the corrupted MB; otherwise, the employed error concealment schemes are simply used to conceal the corrupted MB. Finally, the approach that the important data are transmitted as extra side information (e.g., extra packets) and the approach with data embedding can be
-12-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
compared as follows. (1) If the important data are transmitted as extra packets, it may contain more extra overhead in maintaining the inter-packet synchronization between the important data packets and the image/video data packets. Furthermore, the delay jitter of the important data packets may also introduce extra delay in error concealment performed at the decoder, since it may need to wait for the important data packets when errors occur. However, if the important data are embedded, less extra overheads are needed. (2) If the important data transmitted as extra packets are variable-length coded, extra synchronization codewords will be required to protect these data. On the other hand, if the important data are embedded into the compressed image/video bitstream, they are protected explicitly without extra synchronization codewords. (3) Based on [27]-[28], the bit rate increments induced by transmitting the important data as extra packets are more significant than those induced by the approach with data embedding when image/video qualities are similar. However, when perceptibly invisible image/video quality degradation is allowed, the bit rate will be only slightly increased. An illustrated example is shown in Fig. 7.
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 7. The error-free and concealed H.263 video frames (the Y component) of a P-frame (the fourteenth frame) within the “Foreman” sequence at frame rate = 10 fps with VPLR = 30% and bit rate = 48kbps: (a) the error-free frame; (b)-(f) the concealed frames by Zero-S, TMN-11, DEVCS, ERDE, and the proposed scheme, respectively.
4.
Concluding Remarks In this study, a survey of error resilient coding schemes based on data embedding -13-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
is presented. For future researches, new important data helpful for image/video error resiliency, new data embedding schemes, and new error detection, correction, and concealment schemes can be investigated.
References [1]
Y. Wang and Q. F. Zhu, “Error control and concealment for video communication: a review,” Proceedings of the IEEE, vol. 86, no. 5, pp. 974-997, May 1998.
[2]
Y. Wang, S. Wenger, J. Wen, and A. K. Katsaggelos, “Error resilient video coding techniques,” IEEE Signal Processing Magazine, vol. 17, no. 4, pp. 61-82, July 2000.
[3]
L. W. Kang and J. J. Leou, “A new hybrid error concealment scheme for MPEG-2 video transmission,” in Proc. of IEEE Int. Workshop on Multimedia Signal Processing, St. Thomas, US Virgin Islands, Dec. 2002, pp. 29-32.
[4]
L. W. Kang and J. J. Leou, “A hybrid error concealment scheme for MPEG-2 video transmission based on best neighborhood matching algorithm,” in Proc. of IEEE Int. Conf. on Multimedia and Expo, Taipei, Taiwan, June 2004.
[5]
H. H. Yu and P. Yin, “Multimedia data recovery using information hiding,” in Proc. of IEEE Int. Global Telecommunications Conference, San Francisco, CA, USA, Nov. 2000, vol. 3, pp. 1344-1348.
[6]
P. Yin, B. Liu, and H. H. Yu, “Error concealment using data hiding,” in Proc. of IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, May 2001, vol. 3, pp. 1453-1456.
[7]
C. Y. Lin, D. Sow, and S. F. Chang, “Using self-authentication-and-recovery images for error concealment in wireless environments,” in Proc. of SPIE, vol. 4518, Aug. 2001, pp. 267-274.
[8]
L. W. Kang and J. J. Leou, “A new error resilient coding scheme for JPEG image transmission based on data embedding and vector quantization,” in Proc. of IEEE Int. Symposium on Circuits and Systems, Bangkok, Thailand, May 2003, vol. 2, pp. 532-535. -14-
影像與識別 2005 Vol. 11 No. 4
[9]
A Survey of Error Resilient Coding Schemes for Image…
J. Wang and L. Ji, “A region and data hiding based error concealment scheme for images,” IEEE Trans. on Consumer Electronics, vol. 47, no. 2, pp. 257-262, May 2001.
[10] M. Kurosaki, K. Munadi, and H. Kiya, “Error concealment using layer structure for JPEG2000 images,” in Proc. of IEEE Asia-Pacific Conf. on Circuits and Systems, 2002, pp. 529-534. [11] C. S. Lu, “Wireless multimedia error resilience via a data hiding technique,” in Proc. of IEEE Int. Workshop on Multimedia Signal Processing, St. Thomas, US Virgin Islands, Dec. 2002, pp. 316-319. [12] L. W. Kang and J. J. Leou, “Two error resilient coding schemes for wavelet-based image transmission based on data embedding and genetic algorithms,” in Proc. of IEEE Int. Conf. on Image Processing, Barcelona, Spain, Sept. 2003, pp. 461-464. [13] T. H. Wang, T. H. Lee, and P. C. Chang, “Error Detection by Parity Checks for H.263 Compatible Video Coding,” in Proc. of IEEE Data Compression Conference, Snowbird, Utah, March 2001, p. 522. [14] Z. W. Gao and W. N. Lie, “MPEG-4 video error detection by using data hiding techniques,” in Proc. of IEEE Int. Symposium on Circuits and Systems, May 2002, vol. 3, pp. 397-400. [15] P. Zhou and Y. He, “A fragile watermark error detection scheme for JVT,” in Proc. of IEEE Int. Symposium on Circuits and Systems, Bangkok, Thailand, May 2003, vol. 2, pp. 956-958. [16] F. Bartolini, A. Manetti, A. Piva, and M. Barni, “A data hiding approach for correcting errors in H.263 video transmitted over a noisy channel,” in Proc. of IEEE Int. Workshop on Multimedia Signal Processing, Cannes, France, Oct. 2001, pp. 65-70. [17] D. L. Robie and R. M. Mersereau, “Video error correction using data hiding techniques,” in Proc. of IEEE Int. Workshop on Multimedia Signal Processing, Cannes, France, Oct. 2001, pp. 59-64. [18] D. L. Robie and R. M. Mersereau, “Video error correction using steganography,” EURASIP Journal on Applied Signal Processing, vol. 2002, no.
-15-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
2, pp. 164-173, 2002. [19] D. L. Robie, N. Wu, and R. M. Mersereau, “The use of steganography to enhance error detection and correction in MPEG-2 video,” in Proc. of IEEE Asilomar Conf. on Signals, Systems and Computers, Nov. 2002, vol. 2, pp. 1204-1209. [20] J. Song, and K. J. R. Liu, “A data embedded video coding scheme for error-prone channels,” IEEE Trans. on Multimedia, vol. 3, no. 4, pp. 415-423, 2001. [21] M. Carli, D. Bailey, M. Farias, and S. K. Mitra, “Error control and concealment for video transmission using data hiding,” in Proc. of IEEE Int. Symposium on Wireless Personal Multimedia Communications, Oct. 2002, vol. 2, pp. 812-815. [22] C. B. Adsumilli, M. Farias, M. Carli, and S. K. Mitra, “A hybrid constrained unequal error protection and data hiding scheme for packet video transmission,” in Proc. of IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, April 2003, vol. 5, pp. 680-683. [23] A. Yilmaz and A. A. Alatan, “Error concealment of video sequences by data hiding,” in Proc. of IEEE Int. Conf. on Image Processing, Barcelona, Spain, Sept. 2003, pp. 679-682. [24] W. Zeng, “Spatial-temporal error concealment with side information for standard video codecs,” in Proc. of IEEE Int. Conf. on Multimedia and Expo, Baltimore, Maryland, USA, July 2003, pp. 113-116. [25] L. W. Kang and J. J. Leou, “A new error resilient coding scheme for H.263 video transmission,” in Proc. of IEEE Pacific-Rim Conf. on Multimedia, Hsinchu, Taiwan, Dec. 2002, pp. 814-822. [26] S. W. Lin, J. J. Leou, and L. W. Kang, “An error resilient coding scheme for H.26L video transmission based on data embedding,” Journal of Visual Communication and Image Representation, vol. 15, no. 2, pp. 214-240, 2004. [27] L. W. Kang and J. J. Leou, “An error resilient coding scheme for H.264 video transmission based on data embedding,” in Proc. of 2004 IEEE Int. Conf. on Acoustics, Speech, and Signal Processing, Montreal, Quebec, Canada, May 2004, vol. 3, pp. 257-260.
-16-
影像與識別 2005 Vol. 11 No. 4
A Survey of Error Resilient Coding Schemes for Image…
[28] L. W. Kang and J. J. Leou, “An error resilient coding scheme for H.264/AVC video transmission based on data embedding,” Journal of Visual Communication and Image Representation, vol. 16, no. 1, pp. 93-114, 2005. [29] J. R. Chen, C. S. Lu, and K. C. Fan, “A significant motion vector protection-based error-resilient scheme in H.264,” to appear in Proc. of 2004 IEEE Int. Workshop on Multimedia Signal Processing, Siena, Italy, Sept. 2004.
-17-
