Iterative Decoding of Serially Concatenated Arithmetic ... - CiteSeerX

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 16, NO. 6, JUNE 2007

1557

Iterative Decoding of Serially Concatenated Arithmetic and Channel Codes With JPEG 2000 Applications Marco Grangetto, Member, IEEE, Bartolo Scanavino, Member, IEEE, Gabriella Olmo, Senior Member, IEEE, and Sergio Benedetto, Fellow, IEEE

Abstract—In this paper, an innovative joint-source channel coding scheme is presented. The proposed approach enables iterative soft decoding of arithmetic codes by means of a soft-in soft- out decoder based on suboptimal search and pruning of a binary tree. An error-resilient arithmetic coder with a forbidden symbol is used in order to improve the performance of the joint source/channel scheme. The performance in the case of transmission across the AWGN channel is evaluated in terms of word error probability and compared to a traditional separated approach. The interleaver gain, the convergence property of the system, and the optimal source/channel rate allocation are investigated. Finally, the practical relevance of the proposed joint decoding approach is demonstrated within the JPEG 2000 coding standard. In particular, an iterative channel and JPEG 2000 decoder is designed and tested in the case of image transmission across the AWGN channel. Index Terms—Arithmetic coding, iterative decoding, jointsource channel coding (JSCC), JPEG 2000, MAP decoding.

I. INTRODUCTION HE provision of reliable multimedia communications over radio links represents a crucial objective for the deployment of future generation wireless networks. In such a context, cross-layer design and joint-source channel coding (JSCC) are emerging as promising strategies in order to provide wideband wireless services [1]–[3]. Both of the aforementioned strategies aim at improving the performance of a communication system by means of a joint optimization of traditionally separated layers. In this paper, joint-source and channel decoding is performed with an iterative approach, based on extrinsic information exchange between the source and channel modules. The related literature mainly refers to the case of robust decoding of variable length codes (VLC), such as Huffman. The possibility of exploiting the residual redundancy of VLC as a form of implicit error protection has been studied in [4] and [5], where maximum a posteriori (MAP) decoding of Huffman codes in presence of transmission errors is proposed. A code construction technique for VLC with a given Hamming distance is presented in [6], where VLC MAP decoding performance is

T

Manuscript received March 13, 2006; revised February 1, 2007. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Ricardo L. de Queiroz. The authors are with the Center for Multimedia Radio Communications (CERCOM), Department of Electronics, Politecnico di Torino, 10129 Torino, Italy (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/TIP.2007.896612

compared with conventional FEC. A soft-in soft-out (SISO) decoder for VLC is presented in [7], whereas in [2], [3], [8], and [9], iterative decoding of VLC, when serially concatenated with convolutional codes (CC), is proposed. On the other hand, the most recent coding standards for media applications, such as JPEG 2000 [10] for still images and H.264/AVC [11] for video, are adopting arithmetic codes (AC) [12], because of their excellent performance on binary alphabet and the availability of powerful and low complexity adaptive algorithms. In this view, the development of JSCC approaches involving AC is of paramount interest. In fact AC, being the final entropy coding stage of modern standards for multimedia compression, represents the natural interface between source and channel coding. The techniques adopted for VLC error correction cannot be extended straightforwardly to the AC case. In fact, after AC, very little residual redundancy is left in the coded sequence. Moreover, AC exhibits poor resynchronization properties, and it is, thus, very sensitive to bit errors. In [13], an error recovery technique based on automatic repeat request (ARQ) is proposed by means of AC with proper insertion of markers, whereas an error detection technique using AC with markers is proposed in [14]. In [15] and [16], a minimum Hamming distance constraint is enforced among AC codewords so as to allow error correction at the decoder side. AC with error detection, based on the insertion of a forbidden symbol in the input alphabet, was initially proposed in [17], and further studied in [18] and [19]. The forbidden symbol allows one to adjust the amount of coding redundancy to be embedded in the coded stream. In [18], an ARQ strategy is validated in the case of lossless image transmission across the binary symmetric channel (BSC). In [19], it is shown that the forbidden symbol provides an efficient form of continuous error detection. The coding redundancy conveyed by the forbidden symbol can be recognized by the decoder in order to attempt not only error detection but also error correction. In [20], error correction is performed in the case of transmission across the additive white gaussian noise (AWGN) channel. The error correction algorithms are based on sequential search techniques, and the performance is evaluated in terms of packet recovery rate for differentially encoded images. MAP decoding of AC with forbidden symbol is studied in [21] and [22], where sequential decoding of AC transmitted across BSC and AWGN channels is presented. In [23], the MAP decoding of AC is applied to the case of reliable video transmission, whereas in [24], a JSCC concatenated scheme based on AC and trellis coded modulation is applied to image transmission. In [25] and [26], the forbidden

1057-7149/$25.00 © 2007 IEEE

1558

symbol and MAP decoding is exploited to design an error correcting JPEG 2000 codec. In particular, in [26], the performance of JPEG 2000 decoding based on MAP and maximum likelihood (ML) metrics is analyzed in the case of transmission across the BSC and the AWGN channel. Soft decoding of arithmetic codes with synchronization marker is studied in [27] and applied to the case of image transmission. It is worth pointing out that the techniques introduced in [25]–[27] are being included in the upcoming Part 11 of the JPEG 2000 standard, which addresses the wireless applications of the novel image coding standard. This paper is devoted to the design of a novel SISO module [28] for soft AC decoding. The objective of this work is the development of an iterative joint-source channel decoder that is able to improve the performance with respect to the traditionally separated channel and AC decoding. The proposed technique can be used to exchange extrinsic information between the AC decoder and any channel code, typically present in a multimedia communication system. The same concept has been already introduced in [27]. However, the latter paper is mainly focused on soft decoding of AC by means of a symbol clock or bit clock AC decoding model, and does not treat in detail the case of iterative source and channel decoding. As apposite to [27], the SISO module proposed in this paper is obtained following the standard approach of the BCJR algorithm [29]. This choice is important from the theoretical point of view, since it helps exploiting the analogies with a well known channel decoding algorithm. From the point of view of implementation, this aspect translates into the possibility of reusing some computational modules already developed for the channel decoding case. Experimental results have been worked out employing AC with forbidden symbol and a first order Markov source model. The error correction and convergence property of the proposed approach are evaluated. Preliminary results have already been presented in [30], where an AC SISO working in the linear probability domain is tested in the case of a memoryless binary source. In this paper, the proposed AC SISO is implemented in the logarithmic domain, and its performance is deeply analyzed. The optimal partitioning between source and channel coding rate is investigated as well. Finally, the extrinsic information transfer (EXIT) chart technique [31] is used to evaluate the convergence behavior of the proposed iterative JSCC scheme. The same approach has been recently proposed in [32] to study the convergence behavior of a JSCC system based on a quantizer followed by proper bit mapping. The remaining major contribution of the paper is the implementation of the proposed AC SISO module within the JPEG 2000 decoder and the analysis of the improvements yielded by iterative joint-source channel decoding. In the case, the JSCC approach is applied to the JPEG 2000 entropy coding stage, i.e., the adaptive binary AC, known as MQ coder. The paper is organized as follows. In Section II, AC is briefly reviewed. In Section III, the derivation of the proposed AC SISO module is described, along with its implementation details. Section IV reports the experimental results obtained with the proposed SISO module in the case of transmission across the AWGN channel. Section V is devoted to the iterative decoding of serially concatenated AC and CC. In Section VI, the application of the proposed iterative decoding approach to


a JPEG 2000 based image communication system is presented. Finally, in Section VII, conclusions are drawn. II. OVERVIEW OF ARITHMETIC CODES AC represents the entropy coding stage in many emerging standards for image and video communications. For example, the popular JPEG 2000 [10] and H.264/AVC [11] standards employ the MQ [33], [34], and the context adaptive binary AC (CABAC) codec, respectively. They are binary, adaptive, and context-based AC. For the sake of simplicity, in the following we address a binary and nonadaptive AC, assuming that the source model is static and known by both the AC encoder and decoder. We consider the encoding of a binary sequence of fixed length . The source is modeled as a first order Markov chain, with transition probabilities and . The steady-state probabilities for this model are and , whereas the source correlation is . The AC encoding procedure [12] is based on the Elias onto the coding principle. It maps the input sequence probability interval , represented by the codeword of variable length . The encoding is based on the iterative selection of the probability , which is progressively refined according to the interval input source symbols , . At the first iter, i.e., the whole ation, the interval is initialized to , the interval probability space. For is progressively selected according to the following rules: if if (1) where represents the interval amplitude at iteration . After all source bits have been processed, the interval is identified along with the codeword , which corresponds to the binary representation of the value , chosen so as to minimize the number of bits in . In Fig. 1, a coding example is shown, in the case of a three bit input sequence. The decoding task is performed by the dual procedure which, given the sequence , progressively selects the interval , . In order to guarantee the decoding uniqueness, either the number of input symbols is provided to the decoder, or an end-of-file symbol is used to terminate the decoding process. The coding procedure described above is made feasible in practice by means of finite precision arithmetic, with proper in order to avoid the underflow conscaling of the interval dition. Furthermore, the interval scaling makes the sequential , the encoding easier [12]. For example, as soon as most significant bit of is fully determined and can be transmitted to the decoder, and the interval can be scaled up by a factor of two. The results reported in this paper are obtained with a 32-bit fixed point implementation. Sequential decoding is obtained analogously. In particular, let us assume that only out of the codeword bits are available. From the partially

GRANGETTO et al.: ITERATIVE DECODING OF SERIALLY CONCATENATED ARITHMETIC AND CHANNEL CODES

1559

Fig. 2. Arithmetic code decoding tree.

error detection capability depends on the value of . The interested reader is referred to [17]–[19] for a detailed description of the forbidden symbol technique.

Fig. 1. Example of arithmetic coding.

received codeword, denoted in the following as , the de, which is a truncated coder gets an approximated value binary representation of . Depending on the numerical pre, a certain number of decoding iterations cision given by can be performed and a variable number of source symbols can be decoded. Of course, the decoding stops as far as the numerand does not allow to unamical error between biguously select a probability interval. In the following, a bit level sequential decoder will be employed, which consumes the . Given the th input codeword bit by bit, , codeword bit, the value of is progressively refined and of source symbols is decoded, the maximum number . At each iteration, the interval is updated , where , and represents the chosen from to interval at the th decoding step. The AC decoder state can be represented as for

(2)

It is well known that the decoding iterations are very sensitive to transmission errors. In fact, even a single flipped bit in the sequence can heavily modify the corresponding value , thus causing irreversible desynchronization of the decoder. In [17] and [18], the use of a forbidden symbol is proposed. This symbol, whose probability is fixed to an arbitrary value , is never encoded. The probability interval is partitioned according to a ternary source with symbols “0,” “1, ” and and conditional probabili, , and , reties spectively. The introduction of prevents from using a fraction of the probability interval and, thus, corresponds to an amount of coding redundancy per encoded symbol [18]. At the decoder side, the presence of permits error detection at the expenses of compression efficiency. In fact, decoding the symbol identifies the presence of transmission errors. The

III. SISO MODULE FOR AC The decoding procedure sketched in the previous section uses as input to the decoder the hard quantization of the output of a matched filter (or a channel decoder, when present). In other words, it is a hard decoding procedure. In this section, the BCJR algorithm [29] is modified in order to design a SISO module for soft decoding of AC. Let us assume that the decoder observes of the encoded sea noisy version quence , transmitted through a memoryless AWGN channel (channel coding is not considered here). The objective of the SISO module is to compute the a posteriori probabilities (APP) , , . The modified BCJR algorithm is employed to compute the APPs, considering the binary tree [21] depicted in Fig. 2, which . represents all possible compressed sequences of length the subset of all codeIn the following, we will denote with . It is worth pointing out that only a subset words of length of all possible binary paths in the tree corresponds to valid . The AC decoding tree, which takes the codewords role of the classical trellis representation, can be obtained by the sequential decoder sketched in Section II. Each node in the tree represents an arithmetic decoder state , defined in (2), where identifies the decoding step. Each transition and is associated to a bit label , between two states and represents a codeword bit. Depending on , the arithmetic decoder updates the state from to and releases the variable number of decoded source symbols . A. APP Computation In this section, the details of the APP evaluation are presented. The objective of the decoder is to estimate the APP . Since, given the reis just a scaling factor, in the following ceived sequence , we derive the joint probabilities ,

1560


. The can be expressed averaging over all possible paths in the tree having (3)

the APP. Second, only one quantity needs to be computed and propagated through the algorithm [35]. The LLRs are defined as follows:

The probability can be evaluated by separating the contributions from the past and future with respect to the considered transition at depth . As a consequence, we and can write (4)

(5)

(9) Dividing numerator and denominator by introducing the quantities ,

and and , the LLR can be

expressed as (10)

In the previous expression probability along the branch . It can be evaluated as follows:

represents the transition in the tree with bit label

Finally, and can be computed with a forward and backward recursion, respectively for

(6) Given that AC is close to ideal, so that the codeword bits are almost independent and equally likely, we can assume . Consequently, depends only on the , which represents the input of the a priori probability AC SISO decoder. and , can be The remaining terms in (4), computed with a forward and backward recursion respectively. In the following equations, we use a shorthand notation where , , and

(11) for (12)

In conclusion, the AC SISO decoder goal is the evaluation of (10) through a forward and backward recursion defined by (11) and (12), respectively. The inputs of the module are the a and the ratio priori source probabilities fed by the channel. C. Practical Decoder Implementation

for

(7) for (8)

In the forward recursion (7), the term is obtained to , i.e., the by summing the branch metric forward metric of the departing state . In the backward recursion, two states can merge back into a single node of the tree, as shown in Fig. 2; as a consequence, in (8), the backis evaluated by summing over all the contriward metric butions along the branches merging into represents the a priori the state . The term probability of the sequence corresponding to the decoding path . Each term is evaluated by terminating in the state multiplying the transition probabilities of the source symbols decoded along a given tree branch. B. Logarithmic APP Evaluation It is well known that the use of log-likelihood ratios (LLR) is generally preferred for instead of symbol probabilities implementation purposes. In fact, the logarithmic approach allows one to obtain an additive algorithm for the computation of

The remaining issue to be solved in order to implement the proposed AC SISO is the construction of the decoding tree (Fig. 2). As already stated, the tree is used to represent all . The explicit enumeration of all possible codewords sequences is clearly unfeasible for practical input lengths . As a consequence, it is essential to resort to suboptimal search techniques to limit the exponential growth of the paths in the tree. We propose to use the so called M-algorithm (MA) [36] to sequentially prune the decoding tree. MA is a breadth first technique, which limits the storage to the best nodes at each depth in the tree. At each depth in the tree, the available states are extended forward. Each branch is characterized by its transition , which is used to update the forward metric metric of the new node. The metric is employed to select the nodes at level . If the forbidden symbol is used, i.e., best at the encoder side, a supplementary pruning criterion can be found. In fact, sequences that are not AC codewords are guaranteed to cause error detection after a certain delay, thus permitis not ting to prune their corresponding nodes. If the value of correctly set, it is possible to end-up with all the nodes pruned. In such a case, the decoding procedure is aborted. Finally, when the last decoding step has been performed at level in the tree, only the nodes corresponding to input sequences of


symbols are kept. At this point, a suboptimal decoding tree, , has been containing only the most probable paths can be computed with a backward determined and the LLR step. In fact, moving from the leaves towards the tree root, the and the LLR can be evaluated acbackward metric cording to (10) and (12), respectively. The previously described algorithm requires that the debefore coding tree is completed up to the level performing the backward step. This approach can be very cumbersome from the point of view of memory requirements and does not allow sequential decoding. In order to overcome these disadvantages, the so called sliding windowed SISO can be used as described in [28] and [37]. The forward and backward passes are performed on a certain number of . The forward evaloverlapped windows of length uation of is performed for , is the window index. When where , the backward recursion the tree depth reaches are is initiated and the , evaluated. At the same time the LLRs are computed by (10) for , where represents the window overlap. This approach requires the storage only of a and yields sequential section of the decoding tree of width decoding. The computational complexity of this sequential strategy is the same as the standard BCJR algorithm applied to states [38], [39]. Furthermore, the construction a trellis with AC of the decoding tree requires to perform about decoding steps. On the contrary, standard AC decoding would steps. It is apparent that the computational take only cost of the proposed algorithm is mainly related to the selected value of . IV. AC SISO SIMULATION RESULTS The performance of the proposed AC SISO module has been evaluated by simulation. Transmission of the AC codewords across the AWGN channel with BPSK modulation has been considered. In the following, the channel signal-to-noise ratio is de, where is the energy per source bit (before fined as compression) and is the two-sided noise power spectral density. The use of the forbidden symbol in the AC encoder cor, where responds to an overall coding rate is the source entropy and . Alternatively, the redundancy introduced by the forbidden symbol can be interpreted as a separated channel coding stage, whose rate . is In Fig. 3, the performance of the AC SISO in terms of word (i.e., the probability to have decoding ererror probability for rors in an AC coded block) is reported as a function of , several source models. The experimental settings are bps, , SISO module with yielding and without sliding window. The choice of the parameter is driven by complexity reasons; the interested reader can find more details on the sequential search algorithms in [36]. of standard AC decoding is also reported for comThe parison. As previously stated, standard AC is extremely sensitive to transmission errors and turns out to be almost useless in . On the contrary, the use of forthe simulated range of bidden symbol and soft decoding significantly improves the AC

1561

Fig. 3. P (e) of standard AC and proposed AC SISO in the case = 0:05 for several source correlation values.

error resilience. In Fig. 3, the performance obtained with three different source models is shown: 1) uncorrelated source with , corresponding to entropy bps and (circle); 2) correlated Markov source with bps (star); 3) correlated Markov source with and , corresponding to bps (squares). Besides observing the large coding gain obtained with respect to standard AC, the reported results enable some general considerations. In particular, the comparison between cases 1) and 2) shows that the proposed decoder guarantees a performance that is almost independent of the correlation , as far as the source entropy and the coding redundancy are kept constant. If one considers the two sources 1) and 3) characterized by the same stationary probability but different correlation, a considerable gain can be appreciated in the case . in terms of These results are not surprising, since it is well known that, according to Shannon channel coding theorem, the error probability only depends on the source entropy . The simulations assume perfect knowledge of the source statistical properties. As a final consideration we note that the use of AC with forbidden symbol permits to freely adjust the coding redundancy , thus yielding a fully rate-compatible code. The performance of sequential decoding of similar systems based on error detecting AC have already been studied in the literature [20]–[23]. Here, it is shown that it is possible to obtain superior error correction performance with respect to separated channel coding approaches based on conventional FEC codes. The performance of such decoding schemes are similar to those obtained using the SISO decoder. However, being soft-input hard-output decoders, they do not permit iterative decoding with channel decoders. The sequential decoding obtained by means of the SISO approach, besides confirming the error correction performance in the case of single AC decoding, paves the way to joint AC and channel decoding strategies. The topic represents the main contribution of the present paper and will be thoroughly studied in the following.

1562


Fig. 4. Iterative decoding scheme.

V. SERIALLY CONCATENATED SCHEME WITH INTERLEAVER As mentioned, modifications of the AC in order to achieve error resilience, which encompasses the insertion of the forbidden symbol, are being considered for inclusion in Part 11 of JPEG 2000. Therefore, it makes sense to consider the serial concatenation of AC with an inner channel code, as these modules may be already present in a future generation multimedia communication system. Let us consider the concatenation of an AC module with a convolutional encoder. The source symbols are mapped onto the variable length sequence and then permuted by a random interleaver . The CC uses the output of the interleaver as information bits and generates the codeword . The latter is transmitted across the AWGN channel by means of BPSK modulation. The decoding of such a serial source-channel coding scheme can be performed with an iterative approach, employing the designed AC SISO and a standard CC SISO module [28]. In Fig. 4, the proposed iterative decoding strategy and the probability flow between the source and channel modules are shown. The CC SISO module is a standard BCJR decoder, which is able to com, being the pute the LLR’s codeword received from the channel. The inputs are the channel , and the a priori term LLR’s . The a priori probability represents the extrinsic information produced by the AC SISO module, and, therefore, it is available only after the first iteration. The decoding chain is completed by the AC SISO, which receives the CC extrinsic information and evaluates , as described in Section III. The CC extrinsic information in logarithmic form can be easily obtained by subtracting the a priori CC SISO input from its output. The inverse permutation is used to feed the AC module properly. Another input of the AC SISO is represented by the adopted source model, which allows to inject the a priori knowledge on the source in the iterative decoder. The AC module output is properly interfaced to the CC SISO by means of a subtraction and direct permutation . As already mentioned in the introduction, in [27] an example of iterative AC-CC decoding was already described. In fact, the symbol clock or bit clock AC decoding model introduced in [27] is equivalent to the decoding tree proposed in [21] and in this paper. Both approaches allow one to evaluate the APP for iterative decoding. However, in [27], iterative decoding does not represent the main

Fig. 5. Performance of the JSCC with global rate R = 1:07, = 0:0, R =

1=2, and L = 250.

focus of the paper and little implementations details are provided. On the other hand, in this paper a solution based on the standard BCJR approach (i.e., forward and backward recursions for LLRs evaluation) is adopted. This leads to the proposal of a practical logarithmic solution. Moreover, investigation on the convergence of the iterative algorithm and the interleaver gain is provided. Finally, in this work, AC with forbidden symbol is employed, whereas in [27] resynchronization markers are used for AC error detection; a comparison between the two approaches has already been presented in [21], showing that the forbidden symbol offers a better error protection than resynchronization markers. A. Simulation Results In this section, we present the simulation results showing the behavior of the proposed JSCC scheme in terms of word error probability assuming a memoryless source model. In fact, according to the results reported in Section IV, the AC SISO word error probability depends only on the source entropy. In particular, a memoryless source with , corresponding to an entropy bps, is used. The baseline configuration is obtained concatenating a standard AC (without forbidden symbol) and a channel code with ,1 corresponding to an overall coding rate rate . The interleaver, used in all the simulations, is selected from the S-random class [40]. In Fig. 5, results are , obtained with the iterative decoder reported in terms of proposed in Section V. These results are worked out with the , systematic recursive convolutional enbest 8-state, coder listed in [41], six iterations of the proposed decoder, and bits. In all the following simulations, the source length AC SISO module employs a sliding window of bits, overlapped by 25%. The simulations are worked out with a max. In Fig. 5, imum value of the memory parameter the performance of the proposed JSCC approach is compared 1In

the following, the i subscript identifies the inner code.


1563

Fig. 6. Performance of the JSCC with different values of and L = 250. Fig. 7. Comparison of the JSCC with two values of L (250,1000), R = 4=7 and = 0:05.

with separated soft channel decoding and conventional arithmetic decoding (dashed line). The is representative of the traditional separated approach where the source decoder cannot take advantage of the extrinsic soft information from the channel. The JSCC approach yields a decoding gain of about 0.5 dB with respect to the traditional separated system. The improvement obtained with the second iteration is apparent; unlike the Turbo codes, further iterations after the fourth yield no significant gain. This behavior will also be discussed in Section V-B. The performance obtained with the baseline configuration is particularly significant because it demonstrates that the proposed iterative joint-source/channel decoder does not necessarily require the forbidden symbol and can be applied to already existing source coding standards employing AC [10], [11]. In order to investigate the benefit of the error correction capability provided by the use of the forbidden symbol, three configand urations with the same total coding rate are analyzed. In particular, we chose , , and , yielding , and , respectively. The rate 2/3 and 3/4 constituent encoders are 8-state recursive convolutional encoder listed in [41]. The rate 4/7 constituent encoder is obtained puncturing 1 bit every 8 bits at the output of the rate 1/2 encoder. The results obtained with the separated approach and the iterative decoding technique (sixth iteration) with several values of are compared in Fig. 6 in terms . For a given a value of , it is possible to select of the best , identifying the best trade-off between the inner and the outer coding rate. Clearly, if rate compatible CC are used, all possible coding rates can be achieved, and the optimal tradeoff , the cases with can be selected. In the simulated range of and exhibit the best performance, showing with an improvement of more than 1.0 dB at respect to the separated decoding. Moreover, the gain obtained with respect to the configuration without forwith bidden symbol and iterative decoding is about 0.5 dB. It is well known that, in the serial concatenation of two convolutional encoders, the use of a recursive convolutional inner encoder yields an interleaver gain increasing with the block

Fig. 8. P (e) in the case R = 4=7 for different values of M .

size [42]. The experimental results confirm that this property holds for our system, given that outer AC behaves as a channel for code, i.e., the forbidden symbol is used. In Fig. 7, (solid line) and two different source lengths, (dashed), are reported for the case and . As expected, an increased block length impairs the perobtained with separate decoding formance in terms of (circle markers). This degradation is due to the impaired perfor. mance of the inner convolutional encoder in the case Vice versa, by means of the proposed iterative decoder, one can improve the performance of the system by increasing the block length. In Fig. 7, the iterative decoding in the case outperforms the case after the second iteration (diamond marker). No interleaver gain is observed when . As stated in Section III-C, the computational complexity of the AC-SISO decoder is mainly determined by the selected , obtained with value of parameter . In Fig. 8, the AC

1564


Fig. 9. Joint AC and turbo decoding performance; R = 1=2, = 0:0.

6 iterations in the case , (the configuration which yields the best performance in Fig. 6), is reported for 64, 128, 256, 1024. These results show several values of that using larger than 256 does not improve the performance . Moreover, it is important to in the selected range of observe that a reduced complexity decoder with exhibits a penalty limited to 0.12 dB at . Note that has turned out to be sufficient in most simulated and small situations, except for the cases of very low values of (i.e., poor AC error detection capability). Finally, given the flexibility of the proposed scheme, we tested the iterative decoder using the standard UMTS turbo code as the inner code. In fact, as already remarked, the designed AC SISO can be interfaced to every channel coding scheme. In Fig. 9, the obtained with the concatenated standard AC and rate turbo code is shown for a block length . The dashed line represents the performance of the separated system using ten iterations in the standard turbo decoder. In the joint decoding case, the CC SISO in Fig. 4 is replaced by the turbo decoder; in such a case, for each external iteration of the joint system, five internal iterations of the turbo decoder are performed. In particular, in Fig. 9, the performance obtained with four external iterations is reported. The joint approach outperforms the separated system already at the second iteration. B. Convergence Behavior In this section, we analyze the convergence properties of the iterative decoding process using the EXIT chart technique [31], [32]. Since an analytical treatment of the iterative decoder behavior is not available, we can predict it by looking at the input/output relations of the individual constituent decoders. The EXIT chart technique is based on the assumption, supported by simulations with large interleavers, that the extrinsic information passed from one SISO to the other is a Gaussian random variable. Based on this assumption, the log-likelihood version of the SISO input/output can be modeled as (13)

Fig. 10. EXIT chart for the JSCC scheme, E =N =

01:7 dB.

where is a binary antipodal symbol ( 1), is a Gaussian random variable with zero mean and variance , and is the mean of . The mutual information between the random variable and the information symbol is given by

(14) At the CC-SISO output, it is possible to evaluate the mutual information , as a function of the input mutual informaand the channel log-likelihood ratio tion , which depends on the value. At can be obthe AC-SISO output, the mutual information . tained as a function of the input information Through independent simulations of the two SISO modules, can the input/output mutual information with parameter be estimated [31]. In Fig. 10, the EXIT chart, constituted by the as a function of output mutual information of the CC-SISO for , and the input mutual information versus are plotted. These results are of the AC-SISO obtained with the same simulation settings of Section V-A and . The two configurations , and , with the same total rate are reported. For , the configuration is the assumed not able to converge to unit mutual information, due to a closed tunnel in the EXIT charts. On the contrary, for the configuration and the same value of , the mutual inforwith mation characteristics of the two SISOs show a fully open tunnel, thus allowing convergence. This remark is confirmed by the experimental results in Fig. 6, where the rate 2/3 choice yields better results, while the rate 3/4 shows the beginning of an error floor. Moreover, the EXIT chart analysis show the fast convergence of the proposed JSCC, which reaches the unit mutual information in a few number of iterations. This analysis also confirms the simulation results, reported in Section V-A, which show no significant gain after three iterations.


1565

VI. JPEG 2000 ITERATIVE DECODING In previous sections, the proposed AC SISO, has been used in the case of nonadaptive binary AC with first order Markov source model. This allowed us to analyze the behavior of the proposed AC SISO and its performance within an iterative source channel decoding scheme. Here, we employ it within the JPEG 2000 image coding standard [10], and demonstrate its advantages within a practical image communication application. The SISO decoder is implemented using the JPEG 2000 context based, binary and adaptive AC, known as the MQ coder. The latter is a low complexity AC which, through proper approximations, avoids the use of multiplications during the interval partitioning. Moreover, the symbol probabilities are estimated from the data by means of a finite state machine which progressively refines the source statistics. The MQ coder represents the final JPEG 2000 entropy coding stage and is used to compress the image in the wavelet transformed domain. In the JPEG 2000 coder, the wavelet domain is divided into nonoverlapping regions, called codeblocks (CB), which are independently entropy coded. Recently, in [25] and [26], the feasibility of MAP MQ decoding has been demonstrated. Moreover, the possibility to use a modified MQ with forbidden symbol for error resilience purposes is foreseen in the Part 11 of the standard, devoted to JPEG 2000 wireless applications. As a consequence, the SISO module presented in this work can be applied to the JPEG 2000-Part 11 entropy decoding without modifications. The main difference with respect to previous results is that, in the following, the a priori source probability used to initialize the backward recursion is no longer known exactly. The approximated statistics, estimated according to the MQ coder contexts, are used instead. The experimental setup is as follows. One frame of the standard QCIF Foreman sequence (luminance component) is JPEG 2000 encoded at 1 bit per pixel (bpp), using MQ with and four resolution levels for forbidden symbol the wavelet transform. A single quality layer at maximum rate is constructed. The value of is selected as a trade-off between coding efficiency and error detection. Given a certain channel condition, one could look for the value of yielding the best performance; since analytical performance bounds for the MQ error correction performance are currently unavailable, this goal can be solved only through simulations. The selected coding options yield a decoded image with peak signal-to-noise ratio (PSNR) equal to 33.69 dB in case of error free transmission. The obtained bitstream is constituted by a sequence of packets containing the compressed data, preceded by headers describing the coding parameters. In order to protect such critical headers, the Packed Packet Headers option is employed. This feature allows to pack all the headers of the individual packets in a single marker segment in the main or tile-part header, which can be easily located in the codestream and protected properly. In the following experiments, this header is considered to be transmitted error free. The remaining part of the JPEG 2000 compressed stream is constituted by MQ coded bits, and is transmitted across the AWGN channel after channel coding. To this end, the MQ bitstream is divided into chunks of 150 bytes. Each chunk is scrambled by means of an interleaver of the S-random class and coded by the 8-state , already used recursive convolutional code with rate

Fig. 11. Average PSNR versus the number of iterations employed for JPEG 2000 iterative decoding.

in Section V-A. Finally, the image is transmitted across the , where AWGN channel with signal to noise ratio represents the energy per MQ compressed bit. At the receiver side, the channel and JPEG 2000 decoders are allowed to interact yielding iterative decoding as described in Section V. In particular, the JPEG 2000 decoder receives the CC extrinsic information and updates the a priori term for next CC iteration. As opposed to the system presented in Section V-A, the CC and AC coding blocks no longer overlap, a fact that has no impact on the iterative decoding procedure. The modified JPEG 2000 decoder scans the image CBs and attempts standard decoding. The forbidden symbol error detection is employed to recognize the bitstream portions which contain residual errors. In order to limit the computational complexity, the MQ SISO module is activated and no only on those erroneous portions, employing windowing. The LLRs of (10), corresponding to the sections of the JPEG 2000 bitstream that do not cause error detection, are fixed to arbitrarily high values in magnitude (the sign depends on the decoded bit value). As a consequence, these bits will be recognized as very reliable during the subsequent CC SISO iteration. The same saturation approach has been proposed in [43], where LDPC codes and a JPEG 2000-Part 1 decoder are used iteratively. However, in [43], MAP decoding of the MQ compressed bitstream is not attempted and JPEG 2000-Part 1 error detection is used to recognize erroneous portions of the bitstream. In Fig. 11, the average PSNR is reported versus the iteration 1.8, 2.0, 2.5 dB. As common practice, the number for decoded image quality is measured in terms of average PSNR, computed as the linear average of the MSE over 100 independent transmission trials. Fig. 11 shows that the improvements yielded by iterative JPEG 2000 decoding can be significant. In order to appreciate the gain with respect to a traditional approach, where source and channel decoding are carried on separately, a standard Part 1 JPEG 2000 decoder has been used to retrieve the image under the same transmissive conditions. In this case, the compressed image has been protected with the same CC and transmitted across the AWGN channel. Then, separated CC decoding and JPEG 2000 decoding with error detection and concealment,

1566


performance in terms of image quality. Finally, in order to appreciate the obtained improvements in terms of visual quality, in dB is reported. Fig. 13 a decoding example for On the right side the difference images, computed with respect to the image decoded in the error free case, are shown so as to better visualize the reduction of the decoding errors. VII. CONCLUSION

Fig. 12. Residual BER after channel decoding versus the number of iterations.

In this paper, a novel iterative joint-source channel decoder has been proposed. An innovative SISO module for soft decoding of AC has been designed. The technique is based on the construction of a decoding tree, which allowed us to adapt the BCJR algorithm to the AC case. The AC SISO has been tested in a serially concatenated scheme based on CC, exhibiting promising performance. Moreover, the introduction of redundancy in the source encoder by means of a forbidden symbol has proved to be an effective error correction tool. Finally, the proposed technique has been successfully applied to JPEG 2000. In particular, the feasibility of a SISO JPEG 2000 decoder has been shown and the gain in terms of image quality has been investigated. REFERENCES

Fig. 13. JPEG 2000 iterative decoding example in the case R = 1=2, = 0:11, and M = 256 when E =N = 2:0 dB. (Left) Decoded images for three iterations (PSNR = 25.82, 32.61, 33.69 dB) and (right) difference images with respect to the error-free decoder.

as defined in the Part 1 of the International standard, are performed. This separated decoding system yields a PSNR of about 12 dB, i.e., an almost useless image, in the same range of values used in Fig. 11; these observations allows one to appreciate the tremendous gain offered by the first iteration of the joint decoding approach. Moreover, the subsequent iterations, and in particular the second one, still improve the image quality by a significant extent. Fig. 12 shows the residual bit error rate (BER) after the convolutional decoding for the first four iterations. The iterative decoder reduces the residual errors by one order of magnitude in the first two iterations, thus enabling the reported good

[1] A. Maharshi, L. Tong, and A. Swami, “Cross-layer design of multichannel reservation mac under Rayleigh fading,” IEEE Trans. Signal Process., vol. 51, no. 8, pp. 2054–2067, Aug. 2003. [2] J. Hagenauer and R. Bauer, “The turbo principle in joint source channel decoding of variable length codes,” in Proc. Information Theory Workshop, 2001, pp. 33–35. [3] A. Guyader, E. Fabre, C. Guillemot, and M. Robert, “Joint sourcechannel turbo decoding of entropy-coded sources,” IEEE J. Sel. Areas Commun., vol. 19, no. 9, pp. 1680–1696, Sep. 2001. [4] K. Sayood, H. Otu, and N. Demir, “Joint source/channel coding for variable length codes,” IEEE Trans. Commun., vol. 48, no. 5, pp. 787–794, May 2000. [5] M. Park and D. Miller, “Joint source-channel decoding for variablelength encoded data by exact and approximate MAP source estimation,” IEEE Trans. Commun., vol. 48, no. 1, pp. 1–6, Jan. 2000. [6] M. Bystrom, S. Kaiser, and A. Kopansky, “Soft source decoding with applications,” IEEE Trans. Circuits Syst. Video Technol., vol. 11, no. 10, pp. 1108–1120, Oct. 2001. [7] J. Wen and J. Villasenor, “Soft-input soft-output decoding of variable length codes,” IEEE Trans. Commun., vol. 50, no. 5, pp. 689–692, May 2002. [8] J. Hagenauer and R. Bauer, “On variable length codes for iterative source/channel decoding,” in Proc. Data Compression Conf., 2001, pp. 273–282. [9] K. Lakovic and J. Villasenor, “On reversible variable length codes with turbo codes, and iterative source-channel decoding,” in Proc. Int. Symp. Information Thery, 2002, p. 170. [10] JPEG2000 Image Compression Standard, ISO/IEC 15444-1, 2000. [11] Joint Video Team JVT of ISO/IEC MPEG and ITU-T VCEG, Joint Committee Draft (CD) May 2002. [12] I. H. Witten, J. G. Cleary, and R. Neal, “Arithmetic coding for data compression,” Commun. ACM, no. 6, pp. 520–540, Jun. 1987. [13] G. F. Elmasry, “Joint lossless-source and channel coding using automatic repeat request,” IEEE Trans. Commun., vol. 47, no. 7, pp. 953–955, Jul. 1999. [14] S. Chokchaitam and P. Teekaput, “Embedded error detection in arithmetic coding using markers,” in Proc. ITCC, Apr. 2004, pp. 747–750. [15] G. F. Elmasry, “Embedding channel coding in arithmetic coding,” in IEE Proc. Communications, Apr. 1999, vol. 146, pp. 73–78. [16] G. F. Elmasry and Y. Q. Shi, “MAP symbol decoding of arithmetic coding with embedded channel coding,” in Proc. Wireless Communications and Networking Conf., 1999, no. 2, pp. 988–992. [17] C. Boyd, J. Cleary, S. Irvine, I. Rinsma-Melchert, and I. Witten, “Integrating error detection into arithmetic coding,” IEEE Trans. Commun., vol. 45, no. 1, pp. 1–3, Jan. 1997. [18] J. Chou and K. Ramchandran, “Arithmetic coding-based continuous error detection for efficient ARQ-based image transmission,” IEEE J. Sel. Areas Commun., vol. 18, no. 6, pp. 861–867, Jun. 2000.


[19] R. Anand, K. Ramchandran, and I. V. Kozintsev, “Continuous error detection (CED) for reliable communication,” IEEE Trans. Commun., vol. 49, no. 9, pp. 1540–1549, Sep. 2001. [20] B. D. Pettijohn, M. W. Hoffman, and K. Sayood, “Joint source/channel coding using arithmetic codes,” IEEE Trans. Commun., vol. 49, no. 9, pp. 1540–1548, Sep. 2001. [21] M. Grangetto, P. Cosman, and G. Olmo, “Joint source/channel coding and MAP decoding of arithmetic codes,” IEEE Trans. Commun., vol. 53, no. 6, pp. 1007–1016, Jun. 2005. [22] M. Grangetto, G. Olmo, and P. Cosman, “Image transmission by means of arithmetic codes with forbidden symbol,” presented at the Int. Conf. Acoustics, Speech, Signal Processing, Hong Kong, Apr. 2003. [23] M. Grangetto, E. Magli, and G. Olmo, “Robust video transmission over error-prone channels via error correcting arithmetic codes,” IEEE Commun. Lett., vol. 51, no. 12, pp. 596–598, Dec. 2003. [24] C. Demiroglu, M. W. Hoffman, and K. Sayood, “Joint source channel coding using arithmetic codes and trellis coded modulation,” in Proc. DCC, Mar. 2001, pp. 302–311. [25] M. Grangetto, E. Magli, and G. Olmo, “Error resilient MQ coder and MAP JPEG 2000 decoding,” presented at the ICASSP, Hong Kong, Apr. 2003. [26] M. Grangetto, E. Magli, and G. Olmo, “A syntax preserving error resilience tool for JPEG2000 based on error correcting arithmetic coding,” IEEE Tran. Image Process., vol. 15, no. 4, pp. 807–818, Apr. 2006. [27] T. Guionnet and C. Guillemot, “Soft decoding and synchronization of arithmetic codes: Application to image transmission over noisy channel,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1599–1609, Dec. 2003. [28] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “Soft-input soft-output modules for the construction and distributed iterative decoding of code networks,” Eur. Trans. Telecommun., vol. 9, pp. 155–172, 1998. [29] L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for mimimizing symbol error rate,” IEEE Trans. Inf. Theory, vol. 20, no. 2, pp. 284–287, Mar. 1974. [30] M. Grangetto, B. Scanavino, and G. Olmo, “Joint source-channel iterative decoding of arithmetic codes,” presented at the ICC, Paris, France, Jun. 2004. [31] S. Ten Brink, “Convergence behavior of iteratively decoded parallel concatenated codes,” IEEE Trans. Commun., vol. 49, no. 10, pp. 1727–1737, Oct 2001. [32] M. Adrat, U. von Agris, and P. Vary, “Convergence behavior of iterative source-channel decoding,” in Proc. ICASSP, Hong Kong, Apr. 2003. [33] D. Taubman, “High performance scalable image compression with EBCOT,” IEEE Trans. Image Process., vol. 9, no. 7, pp. 1158–1170, Jul. 2000. [34] C. Christopoulos, A. Skodras, and T. Ebrahimi, “The JPEG2000 still image coding system: An overview,” IEEE Trans. Consum. Electron., vol. 46, no. 4, pp. 1103–1127, Nov. 2000. [35] S. Benedetto and E. Biglieri, Principles of Digital Transmissions With Wireless Applications. New York: Kluwer, 1999. [36] J. B. Anderson and S. Mohan, Source and Channel Coding. New York: Kluwer, 1991. [37] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “Soft-input softoutput modules for iterative decoding of turbo codes,” TDA Progress Rep. Feb. 1996, pp. 42–124. [38] G. Montorsi and S. Benedetto, “Design of fixed-point iterative decoders for concatenated codes with interleavers,” IEEE J. Sel. Areas Commun., vol. 19, pp. 871–882, May 2001. [39] G. Masera, G. Piccinini, M. R. Roch, and M. Zamboni, “ architectures for turbo codes,” IEEE Trans. Very Large Scale () Integr. Syst., vol. 7, no. 3, pp. 369–379, Sep. 1999. [40] D. Divsalar and F. Pollara, “Turbo codes for pcs applications,” in Proc. ICC, Seattle, WA, Jun. 1995, pp. 54–59. [41] S. Benedetto, R. Garello, and G. Montorsi, “A search for good convolutional codes to be used in the construction of turbo codes,” IEEE Trans. Commun., vol. 46, no. 9, pp. 1101–1105, Sep. 1998. [42] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “Serial concatenation of interleaved codes: Performance analysis, design, and iterative decoding,” IEEE Trans. Inf. Theory, vol. 44, no. 5, pp. 909–926, May 1998. [43] L. Pu, Z. Wu, A. Bilgin, M. W. Marcellin, and B. Vasic, “Iterative joint source/channel decoding for JPEG2000,” in Proc. 37th Asilomar Conf. Signals, Systems, Computers, Nov. 2003, vol. 2, pp. 1961–1965.

1567

Marco Grangetto (S’99–M’03) received the degree (summa cum laude) in electrical engineering and the Ph.D. degree from the Politecnico di Torino, Torino, Italy, in 1999 and 2003, respectively. He is currently a Postdoctoral Researcher at the Image Processing Laboratory, Politecnico di Torino. His research interests are in the fields of multimedia signal processing and communications. In particular, his expertise includes wavelets, image and video coding, data compression, video error concealment, error resilient video coding, unequal error protection, and joint source channel coding. He is participating to the ISO standardization activities on JPEG 2000, where he is a contributor to Part 11 of the standard. Dr. Grangetto has been member of the Technical Program Committees for several international conferences, including IEEE MMSP, ICIP, and ISCAS. He was awarded the premio Optime by Unione Industriale di Torino in September 2000 and a Fulbright grant in 2001 for a research period with the Department of Electrical and Computer Engineering, University of California, San Diego. Bartolo Scanavino (S’00–M’03) was born in 1974. He received the Laurea degree in electronics engineering and the Ph.D. degree in communications engineering from Politecnico di Torino, Torino, Italy, in 1999 and 2003, respectively. From April 2002 to October 2002, he was a Visiting Researcher at the University of California, San Diego, and a Consulting Engineer with Titan Corporation, San Diego. From January 2003 to December 2005, he was Young Researcher at the Politecnico di Torino. He is currently with Motorola, Inc. His research is mainly in wireless communications, especially modulation and coding, including turbo codes and, more recently, radio resource management. Gabriella Olmo (S’89–M’91–SM’06) received the Laurea degree (summa cum laude) and the Ph.D. in degree in electronic engineering from the Politecnico di Torino, Torino, Italy. She is currently an Associate Professor at the Politecnico di Torino. She has coordinated several national and international research programs in the fields of wireless multimedia communications, under contracts by the European Community and the Italian Ministry of Education. She has coauthored more than 140 papers in international technical journals and conference proceedings and is part of the Editorial Board of the Springer journal Signal, Image, and Video Processing. Her main recent interests are in the fields of wavelets, remote sensing, image and video coding, resilient multimedia transmission, joint source-channel coding, and distributed source coding. Dr. Olmo is a Member of the IEEE Communications Society and IEEE Signal Processing Society. She has been a member of the technical program committees and the Session Chair for several international conferences. Sergio Benedetto (M’76–SM’90–F’97) has been a Full Professor of digital communications with the Politecnico di Torino, Torino, Italy, since 1981. He has been a Visiting Professor at the University of California, Los Angeles (UCLA), and the University of Canterbury, New Zealand. He has coauthored two books on probability and signal theory (in Italian), as well as Digital Transmission Theory (Prentice-Hall, 1987), Optical Fiber Communications (Artech House, 1996), and Principles of Digital Communications with Wireless Applications (Plenum-Kluwer, 1999), and over 250 papers in leading journals and conferences. He has taught several continuing education courses on the subject of channel coding for the UCLA Extension Program and for the CEI organization. Dr. Benedetto received the Italgas Prize for Scientific Research and Innovation in 1998 and the Cristoforo Colombo International Communications Award in 2006. He was Chairman of the Communications Theory Symposium of the ICC 2001 and ICC 2006, and he has organized numerous sessions in major conferences worldwide. He has been the Area Editor for Modulation and Signal Design of the IEEE TRANSACTIONS ON COMMUNICATIONS and a Distinguished Lecturer of the IEEE Communications Society. He has also been the Chairman of the Communication Theory Committee of IEEE. He is presently Vice-President of the Communications Society for Technical Activities.