Combined error protection and compression using ...

COMBINED ERROR PROTECTION AND COMPRESSION USING TURBO CODES FOR ERROR RESILIENT IMAGE TRANSMISSION Anna N. Kim? , Stefania Sesia† , Tor A. Ramstad? and Giuseppe Caire† ?

†

Norwegian University of Science and Technology, Dpt of Telecommunications O.S. Bragstads Plass 2B, N-7491 Trondheim, Norway Institut Eur´ecom, Dpt of Mobile Communications F-06904 Sophie-Antipolis,France annak,[email protected],stefania.sesia,[email protected]

ABSTRACT A joint source channel coding scheme for error resilient image transmission is proposed. A practical image coder was introduced in [1] using modified Differential Pulse Coded Modulation (DPCM) codec with multi-rate processing and adaptive entropy coding. In this paper the residual redundancy of the prediction error image is exploited by using turbo codes for both data compression and error protection. In the paper we deal with robust transmission of the source over a BSC channel, but the results can be easily extended for non binary channels. Note also that simple modification of the quantizer allows for progressive transmission and successive refinement of information. With properly chosen rate and puncturing, the system is able to approach the limit theoretically attainable and to outperform the separated approach that consists on the concatenation of the system in [1] and the best turbo codes for the same spectral efficiency. 1. INTRODUCTION Shannon’s source channel separation theorem sets the theoretical limit for information transmission. However it requires infinite complexity and delay for achieving optimality. Joint Source Channel Coding (JSCC) on the other hand can provide more robust and less complex systems by linking the source and channel coder through real integration, joint-optimization or information exchange. A well known JSCC technique consists on exploiting the residual redundancy of the source coder output. Sayood et al. [2] suggested to use this redundancy for error protection. Hagenauer et al. in [3] proposed to use the redundancy to modify the soft information processed by the decoder. Another increasingly popular scheme involves dual-functional channel codes. It was shown in [4] that fixed-to-fixed length data compression of a discrete source using linear codes is strongly related to transmission via linear codes on a discrete additive noise channel where the noise has the same statistic of the source. This analogy can be exploited by using linear error correction channel codes such as Low-Density Parity-Check codes [4] or turbo codes for data compression. In [5] a scheme for data compression was proposed for both single memoryless source and correlated sources where the desired compression ratio can be achieved by properly puncturing turbo codes, in particular by puncturing the information bits and the parity bits. A priori probability of the source is used to modify the extrinsic information in the iterative decoding process. When channel error is present, the rate of the turbo codes has to be selected such that it complies with Shannon’s source and channel coding theorem η < C(Γ)/H(s), where η is the total transmission rate in

source symbol per channel use, C(Γ) is the channel capacity-cost and H(s) is the source entropy. This approach has been adopted in robust video coders for iterative decoding of correlated sources [6], where the authors proposed a practical coding scheme that achieves the Wyner-Ziv limit for lossy source coding with side information. In [1] Kim et al. introduced a low bit-rate predictive image coder, which consists of a modified Differential Pulse Code Modulation (DPCM) coder using multi-rate processing and the Wiener filter. Further rate reduction is achieved through allocating different entropy coders to different areas of the image. The main contribution of our paper is the concatenation of the source coder in [1] with powerful turbo codes to jointly perform data compression and error protection and to present a scheme that can be easily generalized to allow for progressive transmission and successive refinement of information. The joint decoder exploits the residual redundancy of the channel encoder input bits and produces softbits for the reconstruction to be used in the predictive source decoder. This scheme is to be compared with the conventional Separate Source Channel Coding (SSCC) setup where the source coder output (the output of the entropy coder) is protected from channel noise by conventional turbo codes. The rest of the paper is organized as follows: section 2 gives details about the proposed scheme and explains how turbo codes are designed, section 3 shows some simulation results and in particular the comparison between the JSCC and the SSCC scheme and finally section 4 discuss some interesting extensions and concludes the paper. 2. SYSTEM STRUCTURE 2.1. Encoder Figure 1 illustrates the proposed system. In the predictive encoder, N the source image s ∈ R is first low-pass filtered and downsampled two dimensionally. It is then fed through the DPCM encoder, which outputs the difference between the down-sampled N0 image and its prediction. The prediction error image U ∈ R is quantized using a uniform quantizer and fed back through the prediction loop. The prediction error image still contains residual redundancy and this can be exploited through a classification process based on it varying local statistics. The prediction error image is first divided into blocks of fixed size, typically 4x4 pixels. The block variance σ 2 is used as classification criterion. The image can then be classified into M sub-sources based on the probability distribution of the block variances. A Lloyd-Max like algorithm is used to find the optimal variance representation values of each

2.3. JSCC via Turbo Codes

Fig. 1. JSCC Using Turbo Compression and Error Protection

class such that the over-all average description length is minimized at a given distortion. More details can be found in [7]. For the SSCC approach it is possible to exploit the non-stationarity of the source and to achieve rate reduction by using the adaptive entropy coding (AEC) method [7], i.e by designing M different entropy codes, one for each class. The output of the M entropy encoders can be concatenated and fed into punctured turbo codes to obtain rate RT such that the spectral efficiency is η. Instead of applying arithmetic code as entropy code to the quantization indices of each class followed by channel coding, the JSCC scheme provides compression and error protection via turbo codes. The detailed structure is shown in Figure 2 with the examN0 ple of M = 3. Call q ∈ N the quantization indices vector. Without loss of generality, the indices that belongs to the same class are grouped together, i.e. q = (q1 , . . . , qM ) and q ∈ {0, ...Qm − 1} for m = 1, . . . , M , where Qm = 2Lm . The indices are mapped into binary bitstream by using a one-to-one mapL ping µ : ZQm → F2 m , where µ(q) = (µ1 (q), . . . , µLm (q)) is the binary codeword for an index belonging to class m. Applying the mapping µ` to the sequence q componentwise, we have B` = µ` (q), for ` = 1, . . . , L where B` is the `-th bitplane and L = maxm (Lm ) is the maximum number of bitplanes necessary to represent all the quantization indices. After interleaving, each bitplane is mapped onto a distinct channel codeword x` , such that the composition of all the codewords (x1 , . . . , xL ) represents the transmitted codeword that is sent by using a time sharing approach. The total codeword is then passed through the BSC channel and observation is given by (y1 , . . . , yL ).

2.2. Decoder The decoder consists of standard turbo decoders, inverse binary mapping ϕ(·) and the predictive decoder. The quantization indices are decoded through multi-stage decoding of the bitplanes. The reconstructed quantization values are then passed through the predictive decoder for obtaining the decoded image ˆs. It includes a prediction filter which is same as the one used in the encoder, a Wiener filter for further noise reduction, up-sampling and finally the same low-pass filter as in the encoder for removing ‘mirrored’ spectral components from interpolation, as shown in Figure 1.

We now take a closer look at how binary data can be compressed via linear channel codes. In this section we limit our attention to the case of BSC channel but the rationale and results can be extended to other binary input symmetric output channels and to non binary input channels with slight modifications. Consider a binary linear systematic code of rate R = k/n k defined by a generator matrix G, and let c ∈ F2 denote the information vector. The corresponding codeword is obtained as x = cG = (c, cP), which can be split into the systematic part c and parity part u = cP. Examples of powerful systematic binary linear codes are turbo parallel concatenated codes [8], and irregular repeat-accumulate codes [9, 10]. In [11, 4, 12], linear codes and iterative Belief Propagation decoding are shown to be able to provide data compression. The idea is as follows. Let c be sequence of i.i.d. symbols such that Pr(ci = 1) = p. By retaining only the parity part u of the codeword x, we have the compressed sequence of length n − k. Let compression ratio be Rc = 1 − k/n = 1 − R. For a BSC with parameter p and capacity C = 1 − h(p), the compression rate can be as close as desired to 1 − R = 1 − 1 + h(p) = h(p), i.e., to the source entropy, if the channel code is powerful enough to perform close to capacity. For successful recovering of the source sequence c, let us suppose that the parity part of the code is transmitted via another BSC with crossover probability ρ, then it can be shown (see [12]) that the decoder is fully equivalent to a channel decoder that observes u via the BSC channel with parameter ρ and c via a BSC with crossover probability p. In other words, the statistics of the source yields an “equivalent” noise statistics. The achievable transmission rate in terms of source symbols per channel use is given by (1 − h(ρ))/h(p). Now suppose that we have a collection of linear binary channel codes with systematic encoders, with information length k = N 0 and rate R1 , . . . , RL . Let u` denote the parity part of code at level `, of length m` = N 0 ( R1` − 1). Suppose that level-` code can recover with high probability the bitplane B` from the output y` and using the a priori probability P`,m (1|b1:`−1 ) defined in (3) and the knowledge of the previous bitplanes b1 , . . . , b`−1 , that has already been recovered at the previous decoding stages. This yields a necessary condition on the coding rate R` for successful decoding with high probability, i.e 1 − h(ρ) ¯ ` ⇒ R` ≤ N 0 /m` ≤ (1 − h(ρ))/H ¯ ` + 1 − h(ρ) H

(1)

The overall coding rate of the scheme is upperbounded by η

=

N PL `=1

≤

m`

N (1 − h(ρ)) N = 0 (1 − h(ρ)) P ¯` N H(q) N0 L H `=1

(2)

¯ ` is the entropy of `where N is the dimension of the source and H th bitplane and H(q) is the entropy of the prediction error image. Note that for a perfect channel and an ideal entropy compressor applied to the sequence of quantization indices q, we obtain a rate of N/(N 0 H(q)) source symbols per coded bit, which is exactly the best achievable by the lossy source encoder alone, without the channel. The entropy of the source coder output H(q) can be determined as follows: The quantization index sequence q is a piecewise independently identically distributed sequence where each segment has a probability mass function (pmf) PQm (q). Define

B1

b1,1 ..bl,1

q1 q2 . . . , q i . . . q N 0 m=1

m=2

bl+1,1 ..bj,l

bj+1,1 ..bN 0 ,1

`-th bit plane

µ

B`

b1,L1 ..bl,L1

m=3

0

π1

TC 1

π`

TC `

πL

TC L

U X

bl+1,L2 ..bj,L2

0

M

bj+1,L3 ..bN 0 ,L3 BL

m=1

m=2

m=3

Fig. 2. Transmission scheme based on bit-planes. In this example there are 3 classes, the transmission is done by bit-plane, so each bit-plane represents the information bits for a turbo code. ∆

bi∈1:N 0 ,1:` = (µ1 (q), . . . , µ` (q)) = µ`1 (q). For each bit plane and for a class m we define the conditional marginal probability at level ` = 1, . . . , Lm as ∆

P`,m (1|b1:`−1 ) =P (µ` (q) = 1|µ`−1 (q) = b`−1 , . . . , µ1 (q) = b1 ) P PQm (q) q∈ZQm :µ` (q)=1,µ`−1 (q)=b1:`−1 1 P = PQm (q) q∈Z :µ`−1 (q)=b Qm

1:`−1

1

(3)

By applying the chain rule, we can express the entropy of the samples that belong to class m as: Em (q) =

Lm X

Hm (µ` (q)|µ`−1 (q), . . . , µ1 (q))

`=1

=

Lm X X

Pm (b1:`−1 )Hm (µ` (q)|µ1`−1 (q) = b1:`−1 )

`=1 b`−1 1

=

Lm X X `=1 b1:`−1

X

PQm (q)h(P`,m (1|b1:`−1 ))

ZQm :µ`−1 (q)=b1:`−1 1

q∈

(4) where the second summation is done over all the possible vector realizations of the vector b1:`−1 , Pm (b1:`−1 ) is the probability that µ`−1 = b1:`−1 and h(p) = −p log(p) − (1 − p) log(1 − p) is the 1 binary entropy function. The total entropy of the source is then computed averaging over all the classes M X

H(q) =

Em (q)Πm

m=1

=

L X M X

Hm (µ` (q)|µ`−1 (q), . . . , µ1 (q))Πm

`=1 m=1

=

L X

¯` H

(5)

`=1

where we have defined ∆ ¯` = H

M X

Hm (µ` (q)|µ`−1 (q), . . . , µ1 (q))Πm

(6)

m=1

and Πm is the probability of the m-th class. Note that Hm (µ` (q)|µ`−1 (q), . . . , µ1 (q)) = 0 for ` > Lm 1 . 1 This

is because Pm (bLm +1,L = O) = 1.

3. SIMULATION AND RESULTS In this section we discuss some results in terms of entropy and bound on the rate and in terms of peak signal to noise ratio (PSNR) 2 for 8 bit gray scale source image, defined as 10 log10 255 where D D is the mean squared distortion per pixel, vs channel capacity. The test image is the monochrome 512 × 512, “Lena” image and the JSCC system has been design to work at a nominal channel parameter ρ = 0.05. The first column of table 1 shows the results in terms of average entropy per bit-plane while the second column gives the bound on the rate of the turbo code, obtained from equation (6) and (1). These rates can be obtained by puncturing a standard rate 1/3 turbo codes. Note that the puncturing patterns are additional side information to be transmitted errorfree for decoding. In [13], the authors show that a priori probability-based polynomial generator selection gives better thresholds. For almost equiprobable input and compression scheme the code (5, 7) (octal notation) is shown [14] to outperforms Berrou-Glavieux (37, 21) turbo codes [8], while for highly skewed a priori probability the BerrouGlavieux Turbo codes yields better performance. Moreover symmetric random puncturing pattern have been shown to provide some gains over asymmetric puncturing pattern and structured pattern, especially for high rate. Note that exact rate cannot be found by puncturing rate 1/3 turbo codes. The results are given by conservatively choose the rate of the codes. These are given in the third column of table 1. For the SSCC setup we use arithmetic codes as entropy coders and the coder outputs are coded by same turbo codes. Figure 3 shows the comparison between the JSCC with the rates shown in table 1 and the SSCC scheme when the rate RT = 0.72 and generator polynomial (37, 21) and (5, 7). The true spectral efficiency (by considering the second column of table 1) is η = 1.25 source sample per channel use, while the actual spectral efficiency (due to quantization of the rate) is equal to η ∼ 1.1 source sample per channel use. We can clearly see the difference between the separated scheme and the joint approach. The separated scheme needs a much higher input mutual information in order to achieve high PSNR performances. This is due to the fact that the performances of the arithmetic code are strongly dependent on the Frame Error Rate (FER) performances of turbo codes and not on Bit Error Rate (BER). The entropy decoder can recover the indices when the output of the turbo codes is error free, so it needs very low FER performances in order to achieve close to optimal PSNR. Turbo Codes are known to have very good performances in terms of BER but it is not the case for FER. The predictive coding scheme generally has the problem of error propagation. Our bitplane setup ease the problem by relying on low BER performance of turbo codes at

¯` H 0.071 0.374 0.734 0.805 0.174 0.079

R` 0.9 0.65 0.49 0.46 0.8 0.9

¯` R 0.88 0.64 0.46 0.42 0.78 0.88

Table 1. Average entropy per bit-level, nominal channel coding 0 rate ( N 0N+m` ) and quantized values. the same time avoid spending much synchronization bits as arithmetic codes need. And thus we can achieve a PSNR equal to the noiseless case for lower input mutual information. Results shown here are with hard decision decoding. Further improvement can be obtained through using so-called ‘soft-bits’ where the soft-outputs of the turbo codes are exploited in reconstruction of the quantized values.

35 G(37,21) G(5,7)

30

PSNR in (dB)

` 1 2 3 4 5 6

JSCC

25

20

SSCC

15

1−h(0.05)

10

5 0.7

0.75

0.8

0.85 1−h(p)

0.9

0.95

1

Fig. 3. Comparison between JSCC and SSCC. Here the JSCC is with natural binary code as binary mapping. The SSCC are arithmetic codes with turbo codes using the corresponding generator polynomials

4. CONCLUSIONS AND FUTURE WORK [5] J.Garcia-Frias and Y. Zhao, “Data compression of unknown In this paper we have given results for BSC channel. This scheme single and correlated binary sources using punctured turbo and rationale can be easily extended non channels such as QAM/PSK codes,” in Proceedings of the 39th Annual Allerton Confermodulation and AWGN channel. In that case good performances ence on Communication, Control, and Computing, 2001. can be achieved by resorting powerful method based on Bit-Interleaved [6] A. Aaron; E. Setton and B. Girod, “Towards practical wynerCoded Modulation (BICM) schemes with Gray or SP labelling ziv coding of video,” in Proceedings of International Con[15] and by mapping binary symbols onto modulation symbols. ference on Image Processing, 2003. One other interesting generalization is to extend this scheme to [7] J.M. Lervik and T.A. Ramstad, “Optimality of multiple enachieve progressive image transmission through embedded quantropy coder systems for nonstationary sources modelled by a tization. The proposed scheme encodes bit-plane by bit-plane and mixture distribution,” in Proceedings of Int. Conf. on Acousdecodes them in sequence, such that bit-plane ` can be recovered tics, Speech and Signal Proc. (ICASSP), 1996. after having received the corresponding channel output y` and af[8] C. Berrou; A. Glavieux and P. Thitimajshima, “Near shannon ter having decoded the previous bit-planes at levels 1, . . . , ` − 1. limit error-correcting coding and decoding,” in Proceedings Hence, the scheme is suitable for progressive transmission. By of Int. Conf. on Commun. (ICC), 1993. choosing the Q-ary to binary mapping µ such that it is embedded, the source can be reconstructed at different levels of distor[9] A. Roumy; S. Guemghar; G. Caire; and S. Verdu, “Design tion from 1, 2, . . . , ` bit-planes. If the reconstruction operation methods for irregular repeat-accumulate codes,” IEEE Transˆ to b (from q s) is linear, then the reconstruction of the bitplanes can actions on Information Theory, vol. 50, no. 8, pp. 1711– be simply added after interpolation for finer resolution. This fea1727, Aug. 2004. ture might be interesting in certain broadcast applications where [10] S. ten Brink and G. Kramer, “Design of repeat-accumulate the same image is sent to terminals of different capability, both in codes for iteratve detection and decoding,” IEEE Transacterms of processing power and in terms of screen resolution. tions on Signal Processing, vol. 51, no. 11, pp. 2762–2772, Nov. 2003. 5. REFERENCES [11] J. Garcia-Frias and Y. Zhao, “Compression of binary memoryless sources using punctured turbo codes,” IEEE Communication Letters, vol. 6, no. 9, pp. 394–396, Sept. 2002. [1] A.N. Kim and T.A. Ramstad, “Practical low bit rate predictive image coder using multi-rate processing and adaptive [12] G. Caire; S. Shamai and S. Verdu, “Noiseless data compresentropy coding,” in Proceedings of International Conference sion with low-density parity check codes,” in Procceedings on Image Processing (ICIP), 2004. of the DIMACS workshop, 2004. [13] G.-C. Zhu and F. Alajaji, “Design of turbo codes for [2] K. Sayood and J.C. Borkenhagen, “Use of residual redunnonequiprobable memoryless sources,” in Proceedings of the dancy in the design of joint source/channel coders,” IEEE 39th Annual Allerton Conference on Communication, ConTransactions on Communications, vol. 39, no. 6, pp. 838– trol, and Computing, 2001. 846, June 1991. [14] Stefania Sesia, Advanced Coding Techniques for Multicas[3] W. Xu.; J. Hagenauer and J. Hollmann, “Joint source-channel ting and Broadcasting in Wireless Communications, PhD decoding using the residual redundancy in compressed imDissertation, Department of Mobile Communications, Instiages,” in Proceedings of the IEEE International Conference tut Eur´ecom, Sophie-Antipolis, France, 2005. on Communications, 1996. [15] G. Caire; G. Taricco and E. Biglieri, “Capacity of bit[4] G. Caire; S. Shamai and S. Verdu, “Universal data compresinterleaved channels,” IEEE Electronic Letters, vol. 32, no. sion with ldpc codes,” in Proceedings of 3rd International 12, pp. 1060–1061, June 1996. Symposium on Turbo-Codes, 2003.