Reconstruction of JPEG Coded Images in Lossy ... - Semantic Scholar

Reconstruction of JPEG Coded Images in Lossy Packet Networks  Shahram Shirani, Faouzi Kossentini, Samir Kallel, and Rabab Ward Department of Electrical and Computer Engineering, University of British Columbia, Vancouver BC V6T 1Z4, Canada

Abstract

In this paper packetization and reconstruction algorithms in transmission of JPEG coded images over lossy channels are proposed. An interleaving method is devised that maximizes the minimum distance between the 8  8 blocks, used in JPEG, that are in the same packet. Then a method for estimating the missing DC coecients of a JPEG coded image which is required for decoding the compressed image, is suggested and evaluated. As eects of errors in estimating the missing DC value will appear as stripes across the image, a postprocessing technique for removing such stripes is also developed. Finally the missing data is reconstructed by exploiting the correlation between adjacent blocks. A novel reconstruction technique which has a good performance in reconstruction of edges is proposed. Simulation results indicate that our algorithms perform very well, even when more than one packet is lost during transmission of the JPEG coded image. Keywords: JPEG, image reconstruction, packetization, post-processing.



This work was supported by MDSI, BC ASI and NSERC.

1

1 Introduction The JPEG (Joint Photographic Expert Group) standard is intended to compress still, continuous tone, monochrome and color images. JPEG has two classes of encoding and decoding processes: lossy and lossless. In this work only the baseline sequential encoding mode, which belongs to the lossy class, will be considered. In this coding mode the input source image is partitioned into 8  8 blocks along the conventional raster scan order (i.e. left to right / top to bottom) for the 2-D DCT operation. Each of the 64 resulting DCT coecients is quantized using a dierent uniform quantizer. The DC coecient is treated separately from the 63 AC coecients. It is dierentially coded, that is, the dierence coecient DIFF given by

DIFF = d , d ,1 ; i

i

where d and d ,1 are the current block and the previous block DC coecient values (respectively), is coded. The quantized 63 AC coecients are formatted in a zigzag scan in preparation for variable-length coding. Compression is achieved by utilization of both quantization and entropy coding[1]. In this work, we will address the problem of packet based transmission of JPEG coded images over a lossy network. It is assumed that the compressed image data is packetized spatially. That is, each packet contains an integer number of entire 8  8 blocks of DCT coecients. It is also assumed that the decoder knows which blocks belong to each packet, specifying uniquely the location of missing blocks. Because of dierential encoding of DC coecients, loss of a block aects all the blocks after it. To the best of our knowledge, in previously published work in this area, it has always been assumed that the DC coecients of 8  8 blocks are coded non-dierentially, thus constraining the impact of a block loss to only the subject block[2][3]. Such an assumption leads to a coder that is not compliant with JPEG[4]. As a key contribution of this work, we here extend the reconstruction method to baseline JPEG coded images. The structure of this paper is as follows. In Section 2, an interleaving method for packetization of JPEG coded images is proposed. In Section 3, estimation of DC value of lost blocks is presented. Section 4 describes our post-processing technique. Section 5 discusses the details of our nal reconstruction method and section 6 presents some conclusions. i

i

2

2 Packetization of JPEG coded images by interleaving Suppose that a JPEG coded image is packetized spatially as mentioned in the previous section. While transmitting the JPEG coded images over lossy packet networks, it is possible that one or more of the packets will not reach the destination. This type of loss of coded data can aect the decoded image to a large extent, making correction of errors caused by data loss an important issue. Fortunately, redundancies in JPEG coded images can be exploited to improve the quality of an image corrupted by packet loss. Loss of blocks (due to a packet loss) causes two problems at the decoder. First, because the DC coecient of the DCT transform is dierentially coded, the decoder is unable to decode the DC coecient of the blocks which follow the missing one. Thus it is necessary to estimate the missing DC coecient by a proper method. Second, the data of missing blocks should be reconstructed. The DC coecient of the missing block is estimated using the DC value coecients of adjacent blocks. The data of adjacent blocks are usually highly correlated, with correlation decreasing as the distance between the blocks increases. Thus the best blocks for estimating the DC value of the missing block are those adjacent to it. To avoid large contiguous missing blocks in the image, and to facilitate the estimation of lost DC coecients, it is better to assign adjacent blocks to dierent packets (i.e. the blocks should be interleaved such that blocks in the same packet are not spatially close to each other). The best way to make sure that blocks in the same packet are not close to each other is to maximize the minimum distance between the blocks in the same packet (distance is considered as Euclidean distance between centers of blocks). The problem, in general, can be stated as follows: Given an NN block structure, assign each block to a packet enumerated from 1 to P, where P is the number of packets, such that packets have equal number of blocks and the minimum distance between blocks in the same packet is maximized 1 . An algorithm is here developed for the special case of P=N. Instead of providing a mathematical solution to the problem, we follow an alternative method that originated from the following two observations. First, the maximum value for the minimum distance is achieved when the packets are distributed as uniformly as possible over the NN structure. Second, the distribution is uniform when every rectangle containing N contiguous blocks has exactly one member of each of the packets. The rectangle should be 1

It is assumed that P is a factor of NN

3

a square or as close as possible to it, because a square can generate the maximum distance in a 2-D sense. Therefore, the rst and second steps of our proposed algorithm should generate an a2  a1 rectangle, that contains N blocks. The third and fourth steps should guarantee that every set of N contiguous blocks selected in an a2  a1 rectangle from the NN structure represents exactly N dierent packets. The steps of our algorithm are given below: 1. Factor N into product of two integer numbers a1 and a2 such that a1 and a2 be as close as possible to each other and a1  a2 . 2. Form an a2  a1 rectangular block pattern and enumerate the blocks in it from 1 to N. Figure 1(a) shows the rectangle for the case N=8. For N=8, a1 = 4; a2 = 2. 3. Use this rectangle pattern repeatedly to assign all the blocks in rows 1 to a2 and in columns 1 to N in the NN block structure to one of the packets. 4. Shift the columns of the rectangular pattern circularly b 21 c steps (bxc is the largest integer not greater than x). This generates another rectangular pattern. Figure 1(b) shows this pattern for N=8. a

5. Use the rectangular pattern obtained in step 4 to assign all the blocks in rows a2 + 1 to 2a2 , and in columns 1 to N in the NN structure to one of the packets. 6. Repeat steps 4 and 5 until all the blocks in the NN structure are assigned to a packet. Figure 1(c) shows the result for case N=8. Computer simulations done for various values of N show that the obtained packetization patterns, maximize the minimum distance, con rming that the above rule-based algorithm works well.

3 Estimation of DC values As mentioned previously, the decoder knows the position of missing data blocks. It starts decoding the data, and when it reaches an area representing a missing block, it estimates the DC value of the missing block using the DC value of adjacent blocks. The DC value of the missing block is estimated by a linear, causal estimator d^ given by

d^ = w1d1 + w2d2 + w3d3 + w4 d4; 4

(1)

where d1 , d2, d3, d4 are the DC values of the previously coded blocks shown in Figure 2. The coecients of the estimator are obtained by using well known MMSE (minimum mean square error) techniques[5][6]. They are w1 = 0:1, w2 = 0:4, w3 = 0:1, w4 = 0:4. Figure 3 shows the 512  512, 8 bits/pixel, Lena and Pepper images coded and decoded using baseline JPEG. The size of compressed images are about 21 KByte. It is assumed that the coded image data is packetized into 64 packets each containing 64 blocks. Each packet is about 13 KByte. Figure 4 shows the same images when one packet is missing and Figure5 shows them missing two randomly selected packets. The distance between blocks in the same packet is maximized using the suggested algorithm, and the missing DC value required for decoding succeeding blocks is estimated using equation (1). The stripes appearing in Figures 4 and 5 are due to error in estimating the DC value of missing blocks.

4 Post-processing of stripes Error in estimating the DC value of a missing block causes a level shift in all the pixels between that block and the next missing block. This level shift appears as a stripe. To remove the stripe, the amount of the level shift should be determined. Adjacent pixels in an image are highly correlated and their values are close to each other. Therefore the dierence between the average of pixel values in the stripe and their corresponding pixels outside the stripe is an estimation of the level shift. To use the pixels that are as close as possible to each other, the average of pixel values along the upper line segment connecting two consecutive missing blocks is compared to the corresponding line segment in the previous row, as shown in Figure 6. Therefore using the a priori information about the position of missing blocks, we compute
= l +1 , m +1 ; i

i;i

i;i

(2)

where l +1 is the average value of pixels along the upper line segment connecting missing blocks i and i + 1 and m +1 is the average value of pixels in the corresponding line segment in the previous row (See Figure 6). If j
j > T , where T is a threshold, a stripe is detected and removed by subtracting the
from all the pixels between missing blocks i and i+1. The procedure is repeated for all the missing blocks, 1 < i < M , where M is the number of missing i;i

i;i

i

i

5

blocks. To determine the threshold value, the distribution of the dierences between the average pixel values of consecutive rows in an image was obtained for a set of images. More speci cally Figure 7 shows the histogram of the dierence values for the original Lena and Pepper images. As it can be seen, the histograms have a peak around zero, indicating that most of the rows have almost the same average pixel values. Nevertheless with a small threshold value ( i.e. around zero), there is the possibility of exceeding the threshold level because of an edge at l +1 in Figure 6, despite that there is not a visible stripe. Taking this fact into account and considering the dierence value at which stripes become visible to the human eye (especially in the smooth regions of the image), and to maintain the probability of false alarm [7](i.e. there is no stripe but the algorithm detects one) to less than 1%, a threshold value of 4 was selected. Figure 8 shows the images of Figure 5 after applying the post-processing algorithm. Clearly most of the error eects in estimating the DC values are removed. i;i

5 Reconstruction of missing data As a nal stage, the missing data should be reconstructed. Our method for estimating the missing data is based on the technique suggested in [2]. The method proposed in [2] estimates the data of a missing block as a linear combination of its top, bottom, right and left blocks, i.e., the estimate matrix p^ is given by Z

p^

Z

= w p +w p +w p +w p T

T

B

B

L

L

R

R

(3)

where p ,p ,p , and p are 8  8 matrices of pixel values on top, bottom, left and right of the missing block. The weights w ,w ,w ,w are selected so that the total squared edge error T

B

L

R

T

B

L

R

2 = 2 + 2 + 2 + 2 T

B

L

R

is minimized. Each edge error is de ned in terms of pixels by

2 = k(^p , p )k2 ; 2 = k(^p , p )k2 ; T

Zt

Tb

B

Zb

Bt

6

(4)

2 = k(^p , p )k2 ; 2 = k(^p , p )k2 ; L

Zl

R

Zr

Lr

(5)

Rl

where the matrix p^ is generated using (5). The parameter vectors p^ , p^ , p^ and p^ are those elements of p^ that correspond to the top, bottom, left and right pixels in the missing block respectively. Similarly, the vector p consists of the bottom line pixels of the top block, p consists of the top line pixels of the bottom block, p consists of the right line pixels of the left block and p consists of the left pixels of the right block. This technique cannot reconstruct strong diagonal edges. The reason is that a diagonal edge cannot be constructed as a linear combination of shifted diagonal edges[2]. In Figure 9, the poor performance of the above algorithm is apparent around diagonal edges. Figure 10 shows a magni ed picture of the shoulder area of the Lena image. What yields a poor reproduction quality around the edges is the fact that only one weight for each of the top, below, right, and left blocks is used. In this work, these blocks are partitioned into two parts, and a dierent weight is assigned to each part. Thus the estimation relation can be modi ed to Z

Zt

Zb

Zl

Zr

Z

Tb

Bt

Lr

Rl

"

#

"

#

w 1I44 0 ^ = w 10I44 w 0I p p + 0 w 2I44 p 2 44 " # " # w I44 0 w I44 0 1 1 +p 0 w 2I44 + p 0 w 2I44 ; T

B

Z

T

B

T

B

L

(6)

R

L

R

L

R

where I44 is the 4  4 identity matrix, 0 is the 4  4 zero matrix and w 1, w 2, w 1, w 2, w 1, w 2, w 1 , and w 2 are the weighting coecients. Values for the latter coecients are selected by minimizing the cost function described in (5) and (6). Figure 11 shows the Lena and Pepper images after reconstruction using our proposed algorithm. Figure 12 shows a magni ed picture of the same area shown in Figure 10. By comparing Figures 10 and 12, it is clear that our algorithm improves the reconstruction of edges. Finally we evaluate the whole algorithm which involves interleaving, estimation of DC values, post-processing of the decoded image to remove the stripes and reconstruction of missing pixel values. Reconstructed images, illustrated in Figure 11, show a substantial improvement in visual quality. Comparing Figure 4, which shows the Lena and Pepper images missing about 3% of compressed image data packets before applying the algorithm, to Figure 11, which shows the T

L

R

R

7

T

B

B

L

same images after removing the stripes and reconstructing the missing data, one can see that all visible artifacts have been removed. To quantify (objectively) the amount of improvement, the PSNR measure is also used. Table 1 provides the PSNRs for the Lena and Pepper images before and after post-processing (removal of the stripes and reconstruction of the missing data). Figure 11 and Table 1 illustrate that our algorithms perform very well, both subjectively and objectively.

6 Conclusions In this work, packetization and reconstruction of missing data in transmission of baseline JPEG coded images were investigated. We devised an algorithm for interleaving the DCT blocks that maximizes the minimum distance between blocks in the same packet. Moreover a postprocessing algorithm was suggested to remove the eects of the missing DC value on the rest of the image. A key contribution of the above algorithm is that, unlike previously published algorithms, dierential encoding of the DC values of the DCT coecients is assumed. Finally, to estimate the missing data, a reconstruction algorithm that has a good performance in reconstruction of edges was suggested. The performance of the proposed reconstruction algorithm was tested using several images. The quality of the reconstructed images is relatively good both subjectively and objectively.

References [1] K. R. Rao and J. J. Hwang, Techniques and Standards for Image, Video and Audio Coding. Englewood Clis, NJ: Prentice-Hall, 1996. [2] S. S. Hemami and T. H. Y. Meng, \Transform coded image reconstruction exploiting interblock correlations," IEEE Transactions on Image processing, vol. 4, pp. 1023{1027, July. 1995. [3] Y. Wang and Q. F. Zhu, \Signal loss recovery in dct-based image and video codecs," SPIE Conf. Visual Commun. Image Processing, pp. 667{678, Nov. 1991. [4] ISO/IEC, Digital compression and coding of continuous tone still images: requirements and guidlines. International Standard Organization, 1994. 8

[5] A. K. Jain, Fundamentals of Digital Image Processing. Englewood Clis, New Jersey: Prentice-Hall, 1989. [6] S. M. Kay, Fundamentals of Statistical Signal Processing. Englewood Clis, New Jersey: Prentice-Hall, 1993. [7] H. L. Van-Trees, Detection, Estimation, and Modulation Theory. USA: John Wiley and Sons, Inc., 1968.

9

List of Figures 1

Interleaving technique: (a) First rectangular pattern; (b) Second pattern; (c) The interleaved 8  8 structure. . . . . . . . . . . . . . 2 Estimation of DC value. . . . . . . . . . . . . . . . . . . . . . . . . 3 Original images: (a) Lena; (b) Pepper. . . . . . . . . . . . . . . . . 4 Decoded images with one packet missing: (a) Lena; (b) Pepper. . . 5 Decoded images with two packets missing: (a) Lena; (b) Pepper. . 6 Post-processing for removing the stripes. . . . . . . . . . . . . . . . 7 Histogram of the dierence value: (a) Lena; (b) Pepper. . . . . . . 8 Pictures after removing the stripes: (a) Lena; (b) Pepper. . . . . . 9 Reconstructed images using the method proposed in [2]. . . . . . . 10 Magni ed picture of part of the Lena image. . . . . . . . . . . . . . 11 Reconstructed images using the proposed method. . . . . . . . . . 12 Magni ed picture of the same area shown in Figure10. . . . . . . .

10

rectangular ....... ....... ....... ....... ....... ....... ....... ....... ....... ....... ....... .......

. . . . . . . . . . . .

12 12 13 13 14 14 15 15 16 16 17 17

List of Tables 1

PSNR results for the 512  512 Lena and Pepper images. Post-processing involves removing the stripes and reconstruction of missing data. . . . . . . . . . . . . . . 18

11

1 2 3 4 5 6 7 8 (a)

3 4 1 2 7 8 5 6 (b)

1 5 3 7 1 5 3 7

2 6 4 8 2 6 4 8

3 7 1 5 3 7 1 5

4 1 8 5 2 3 6 7 4 1 8 5 2 3 6 7 (c)

2 6 4 8 2 6 4 8

3 7 1 5 3 7 1 5

4 8 2 6 4 8 2 6

Figure 1: Interleaving technique: (a) First rectangular pattern; (b) Second rectangular pattern; (c) The interleaved 8  8 structure.

d1 d2 d3 d4 d^ Figure 2: Estimation of DC value.

12

(a)

(b) Figure 3: Original images: (a) Lena; (b) Pepper.

(a)

(b)

Figure 4: Decoded images with one packet missing: (a) Lena; (b) Pepper.

13

(a)

(b)

Figure 5: Decoded images with two packets missing: (a) Lena; (b) Pepper.

m

i,i+1

l i,i+1

i

i+1

Figure 6: Post-processing for removing the stripes.

14

180

200

160

180

160

140

140 120

Number

Number

120 100

80

100

80 60 60 40

40

20

0 −2

20

−1.5

−1

−0.5 0 0.5 1 1.5 Difference of average pixel values of consecutive lines

2

0 −6

2.5

(a)

−5

−4

−3 −2 −1 0 1 2 Difference of average pixel values of consecutive lines

(b)

Figure 7: Histogram of the dierence value: (a) Lena; (b) Pepper.

(a)

(b)

Figure 8: Pictures after removing the stripes: (a) Lena; (b) Pepper.

15

3

4

(a)

(b)

Figure 9: Reconstructed images using the method proposed in [2].

Figure 10: Magni ed picture of part of the Lena image.

16

(a)

(b)

Figure 11: Reconstructed images using the proposed method.

Figure 12: Magni ed picture of the same area shown in Figure10.

17

Table 1: PSNR results for the 512  512 Lena and Pepper images. Post-processing involves removing the stripes and reconstruction of missing data. Lena Pepper

PSNR before post-processing PSNR after post-processing 25.66 34.85 17.89 30.07

18