Abstract- Several recent publications have shown that joint source- channel decoding (JSCD) could be a powerful technique for taking ad- vantage of residual ...
Source and Joint Source-Channel Decoding of Variable Length Codes M. Jeanne, J. C. Carlach, P. Siohan1 and L. Guivarch2 France Télécom R&D, DMR/DDH, Site de Rennes 4, rue du Clos Courtel, B.P. 59, 35512 Cesson-Sévigné Cedex, France Abstract- Several recent publications have shown that joint sourcechannel decoding (JSCD) could be a powerful technique for taking advantage of residual source redundancy. Furthermore, it appeared that the principle could also be successfully applied to variable length codes (VLCs). This paper gives an in-depth analysis of a new method recently proposed in [1], [2] to provide a low complexity soft decoding of VLCs with and without channel coding. Furthermore we also present new results obtained in a video coding context, using a VLC table from the MPEG4 video standard. Compared to separated decoding systems, the gain in signal to noise ratio varies from 0.3 dB up to 2 dB.
I. I NTRODUCTION Source coding and channel coding are essential functions in any communication system. The source coding stage is designed to remove as much redundancy as possible from the source while the channel coding stage is bound to add control redundancy to the compressed source. For practically all existing systems these two blocks are separately optimized, which can be theoretically justified by Shannon’s theorems related to source and channel coding [3]. However, Shannon’s source and channel theorems are only relevant for codes of infinite lengths which, therefore, can be infinitely complex. In practice these assumptions are not satisfied, then it can be more efficient to jointly optimize source and channel coders/decoders rather than trying to improve source and channel coder design separately. And, indeed, we have seen that in recent years the interest in joint source-channel coding (JSCC) and decoding (JSCD) has considerably increased. In this paper we limit our study to JSCD. Until now, in order to limit the propagation error phenomenon, most of the proposed JSCD techniques deal with cases where the source is encoded with fixed length codes. However, because of the high compression capacity of VLCs, attention is shifted to this technique. Our work deals with such a situation; it gives an in-depth analysis of a new method recently proposed by Guivarch et al. in [1], [2] to provide a low complexity soft decoding of VLCs with and without channel coding. The soft decoding technique, without explicit channel coding, uses the source residual redundancy as a protection technique against channel errors. Our analysis also includes the case of channel encoding/decoding with This work was supported in part by the French ministry of research under the contract named COSOCATI (http://www.telecom.gouv.fr/ rnrt/index_net.html). 1 P. Siohan is now with IRISA-INRIA, Campus de Beaulieu 35042 Rennes Cedex, France. 2 L. Guivarch is now with Agreen Tech SA, 2, rue de la Chataigneraie, 35510 Cesson Sévigné, France.
turbo-codes, here we decode with a joint source-channel approach. Furthermore we also present new results obtained in a video coding context, using a VLC table from the MPEG4 standard [4]. In section II we present our technique to obtain the a priori bit probabilities from the a priori symbol probabilities of the source. Our method to use these a priori bit probabilities in a soft decoding procedure of an entropy-coded source is presented in section III. Channel coding using turbo-codes is introduced afterwards in section IV. Several simulation results are presented using two different models for the source of symbols: in the first, all successive symbols are assumed to be independent while in the second, they correspond to a first order Markov model. II. A priori BIT PROBABILITY IN VLC S In the decoding of entropy-coded sources, it may be assumed that the symbol probabilities are known. These symbol probabilities come either directly from the source or from an estimation algorithm. This knowledge can naturally lead to improve performances in source decoding. Based on the same basic assumption, i.e. a priori knowledge at the decoder, we show that it is also possible to derive a priori bit probabilities and use them in a soft decoding process. The bit probabilities are directly derived from the tree representation of VLCs. Let each symbol, say cj , of C be such that cj = [cj (1), ...., cj (i), ...], with j and i the symbol and bit index, respectively, and let P (cj ) denote its probability. Let cj (i < k) = [cj (1)...cj (k − 1)] and Ijk = {n ∈ N |∀i < k, cn (i) = cj (i)} be the set of indices of all codewords with the first k − 1 bits being equal to the first k − 1 bits of the codeword cj and the bit value being equal to bit, then we have: P p p∈Ijk |cp (k)=bit P (c ) j j P . (1) P (c (k) = bit|c (i < k)) = p p∈I k P (c ) j
These equations are relevant for the independent symbols of a VLC. If instead we have a first order Markov relation between two successive symbols, the calculations must be done considering each symbol as a possible previous symbol. So, we have ∀cm ∈ C: P p m p∈Ijk |cp (k)=bit P (c |c ) j j m P . P (c (k) = bit|c (i < k), c ) = p m p∈Ijk P (c |c ) (2)
The probabilities given in equations (1) and (2) are the a priori bit probabilities, we will use them in a soft decoding scheme. However, to alleviate the notation we will replace P (cj (k) = bit|cj (i < k)) with P (cj (k)|V LCT ree) for the independent source, and P (cj (k) = bit|cj (i < k), cm ) with P (cj (k)|V LCT ree, cm ) for the Markov source. In these latter two notations the condition VLCTree must be understood as the computation of a probability for which we have an exact knowledge of the location in the tree of each bit being processed. III. S OFT DECODING OF ENTROPY- CODED SOURCES In this section we are going to deal with a new method to decode entropy-coded sources that are corrupted by additive white Gaussian noise (AWGN). A. Trellis representation of VLCs A key element in our method is the representation of a VLC as a trellis that is named “reduced binary trellis”. Indeed, as the root and the terminal leaves of a VLC tree can be merged by a single state, we can get a binary trellis with a number of states which is less than the number of nodes in the VLC tree. The building algorithm to get a reduced binary trellis is presented next. Trellis initialization • first state of the trellis, state 0: origin of the tree, i = 1 Building loop { – For each codeword of the VLC table, – For each bit of this codeword, Building of the states ∗ If the end state of the edge is not associated to a state of the trellis · if it corresponds to a terminal node of the tree, add a state to the trellis, state j (j==i), i++ · otherwise associate this state to state j (j=0) ∗ If the starting state of the edge labeled by this bit is not associated to a state of the trellis, add a state to the trellis, state i, i++ Labeling of the edges ∗ If it does not already exist, create an edge between the state i and the state j labeled by the bit being processed } It is important to note that the above algorithm creates a one to one relation between edges in the VLC tree and in the trellis.
cp
Independent Source
P symbols
Huffman Coding
dl L bits
Additive White Gaussian Noise Channel
yl
a priori
yl L bits
Decoding on the reduced binary trellis
dˆl L bits
Huffman Decoding
cˆp P symbols
Fig. 1. Transmission model for an independent entropycoded source. which can cope with this particular feature. Thus, we propose a decoding method based on Benedetto’s modification [5] of the BCJR algorithm [6]. It is assumed that the source symbols are independent.We limit the error propagation encapsulating the sequence of symbols into packets. First, our notations are introduced. l ∈ {1, .., L} and p ∈ {1, .., P } are the time for the bit and symbol scale, respectively. Let cp be a symbol at time p; they are P in each packet. Then C1P is a sequence of P symbols. Let dl be the encoded bit at time l. D1L is the sequence of bits. There are L bits by packet with L variables. Let yl be the noisy bit. Y1L is the noisy bits sequence. An estimation of dl after decoding is denoted by dˆl . Similarly cˆp is an estimation of cp . Sl is a state of the trellis at time l. They are M states at each trellis stage. Finally, El is an edge of the trellis that starts at state Sl−1 , ends at state Sl and labels with dl : Sl−1 = Sl−1 (El ), dl = dl (El ). Our notation is illustrated in Fig. 1 with the transmission model we use. We will now present the equations of the decoding algorithm in our case. For each transmitted bit, dl , we have to compute the a posteriori probability (APP) denoted by AP P (dl ). However, as AP P (dl ) = P (dl |Y1L ), and P (Y1L ) is constant at each stage of the decoding algorithm, it is in fact sufficient to compute P (dl , Y1L ). The forward and backward probabilities defined in [6] are given by L |Sl ), respectively. α(Sl ) = P (Sl , Y1l ) and β(Sl ) = P (Yl+1 Hence, denoting for all l, by El → Sl , an edge El ending at state Sl and by Sl−1 → El , an edge El starting at state Sl−1 , the decoding equations are: X P (dl , Y1L ) = α(Sl−1 )β(Sl )P (yl |dl )P (dl |V LCT ree), El |dl
(3) with α(Sl ) =
X
α(Sl−1 )P (yl |dl )P (dl |V LCT ree),
(4)
El →Sl
B. Soft decoding with the reduced binary trellis The reduced trellis contains parallel edges. Therefore we have chosen to adapt to our problem a decoding algorithm
β(Sl ) =
X
β(Sl+1 )P (yl+1 |dl+1 )P (dl+1 |V LCT ree).
Sl →El+1
(5)
VLC 0 10 11
Symbol a b 2
Probability 0.75 0.125 0.125
Y ↓ |X → a b c
a 0.94 0.18 0.18
b 0.03 0.712 0.108
TABLE II VLC S ASSOCIATED TO THE SYMBOLS OF THE DC’ S LUMINANCE AFTER DCT AND PROBABILITIES FOR THE SEQUENCE ”S TEFAN ”. VLC 011 11 10 010 001 0001 00001 000001 0000001 00000001 000000001 0000000001 00000000001
c 0.03 0.108 0.712
The probability denoted P (dl |V LCT ree) corresponds to the a priori source probability computed in section II. In these latter and fundamental equations the forward and backward probabilities have to be initialized. As usual [6], we impose that α(S0 = 0) = 1, α(S0 = m) = 0 for m 6= 0, β(SL−1 = 0) = 1 and β(SL−1 = m) = 0 for m 6= 0. P (yl |dl ) depends upon the channel and upon the modulation used. In the case of an AWGN channel and with a binary phase shift keying (BPSK) modulation we get (6)
Independent source.
0
Symbol error rate Symbol error rate
−1
10
To limit the error propagation at a certain point, we have used resynchronization, packetizing data into blocks of 256 symbols. The end of the transmission occurs when either the maximum bit error number fixed to 1000, or the maximum transmitted bits equal to 10 000 000 is reached. In our simulations we compute the bit error rate (BER), symbol error rate (SER) and the Levenshtein symbol error rate (LSER) as a function of the signal to noise ratio. The LSER is based on the Levenshtein metric [7] and gives a measure of the capacity of the code to resynchronize itself. In our simulations we compare the decoding scheme with a priori and without a priori on the source. If we have no a priori knowledge, the probability P (dl |V LCT ree) is set to 1/2 for all l. Two different VLC tables have been used for the simulation: • The first is the one used by Murad and Fuja [8]. The associated probabilities are shown on the left in Table I. Let us denote R as the residual source redundancy, i.e. the difference between the VLC encoder’s code rate and the entropy rate on the source. For the source of independent symbols R = 0.18874. • The second table (cf. Table II) is taken from the MPEG4 standard [4, page 344]. We compute the probabilities values by training on the sequence named “Stefan”. The residual redundancy is R = 0.2864. The error rate curves for the VLCs, are depicted in Fig. 2. The improvement (in dB) obtained with the a priori bit prob-
−1
10
Levenshtein symbol error rate Levenshtein symbol error rate −2
10
Bit error rate
−4
10
0
1
2
−2
10
Bit error rate
without a priori with a priori without a priori with a priori without a priori with a priori
10
C. Simulation results
Independent source.
10
−3
where N0 denotes the unilateral noise density and Eb is the energy per transmitted bit. These equations (1) and (3)-(6) provide a way to compute optimal maximum a posteriori (MAP) decoding.
Symbol probability 0.172 0.154 0.114 0.131 0.160 0.169 0.092 0.008 0 0 0 0 0
0
10
error rate
√ 2 1 P (yl |dl ) = √ e−(yl −dl Eb ) /N0 , πN0
DCT Symbol 0 1 2 3 4 5 6 7 8 9 10 11 12
error rate
TABLE I VLC S AND THEIR PROBABILITIES ON THE LEFT, TRANSITION PROBABILITIES BETWEEN SYMBOLS I . E . P (X|Y ) ON THE RIGHT, FOR THE M URAD AND F UJA CASE .
without a priori with a priori without a priori with a priori without a priori with a priori
−3
10
−4
3
4 Eb/N0
5
6
7
8
10
0
1
2
3
4 Eb/N0
5
6
7
8
Fig. 2. Bit and symbol error rate with and without a priori information for the VLC table I on the left and for the VLC table II on the right. ability for an error rate equal to 10−2 are reported in Table III. First of all, it is worthwhile mentioning that the results obtained without a priori source information are equivalent to the ones obtained with a hard decoding procedure, e.g. a symbol-by-symbol decoding of the VLC as proposed for a standard MPEG4 decoder. It is clear from Table III that soft decoding using source information always provides improved decoding performances compared to hard decoding. This method can be used with little change if we have a first order Markov source. IV. JSCD OF ENTROPY- CODED SOURCE WITH TURBO - CODES In this section we use, at the source level, two different models: an independent source and a Markov source. Compared to the transmission scheme studied in the previous section, we add a turbo-code to protect the transmitted data TABLE III I MPROVEMENT ( IN D B)
OBTAINED WHEN USING THE A PRIORI SOURCE INFORMATION
BER = 10−2 SER = 10−2 (TABLE I), 10−1 (TABLE II) LSER = 10−2
VLC table from Murad and Fuja 0.3 0.25
VLC table from MPEG4 standard 0.45 0.6
0.2
0.35
ul Independent Source or first order Markov Source
E*
a priori
cp P symbols
dl L bits
Huffman Coding
Turbo Coding parallel concatenation
DEC 1 Ext1 (dl ) yl1 AP P (dl ) × SUBMAP P (xl |dl )
a priori ul
DEC 2
vl
E
yl2
SUBMAP
AP P (dl )
×
Ext2 (dl )
E
xl Additive White Gaussian Noise Channel
vl
Turbo decoding utilization of the SUBMAP
yl
dˆl L bits
Huffman Decoding
1/x
cˆp P symbols
1/x P (xl |dl ) binary hard output
Fig. 3. Transmission scheme of a first order Markov source with turbo-code.
Fig. 4. Turbo-decoding with a priori probability from the source.
against channel errors. A. JSCD with turbo-code Fig. 3 gives the transmission scheme for an independent, or a first order Markov source. The turbo-code used is a parallel concatenation of two systematic recursive coders followed by a 1/2 puncturing. Let us denote the two coded bits by ul and vl and their noisy represention by xl and yl , respectively. The couple (xl , yl ) is represented by rl . The sequence of L inputs is denoted by R1L . We can note that dl = ul because the coders are systematic, and that vl comes from one of the two coders depending on the puncturing. The turbo-decoding requires computation of an extrinsic information, Ext(dl ), for each transmitted bit. The SUBMAP: Max-Log-MAP has been chosen because it offers a good trade-off between complexity and efficiency. A.1 Independent source The equations hereafter present how to introduce the a priori probability of the source in the MAP algorithm when using turbo-codes. They will, later on, help to present the SUBMAP as is it used in our simulation. Again, what we need is the AP P of the transmitted bits which, after some computation, is found to be: AP P (dl ) = P (dl |R1L ) ∼
M X
M X
β(Sl )α(Sl−1 )
Sl =1 Sl−1 =1
× P (rl |dl , Sl , Sl−1 )P (dl |Sl , Sl−1 )P (Sl |Sl−1 ), (7) with α(Sl ) =
P (Sl , R1l )
=
M 1 X X
α(Sl−1 )P (dl , Sl , rl |Sl−1 ),
Sl−1 =1 dl =0
(8) and
L β(Sl ) = P (Rl+1 |Sl ) =
M X
1 X
Sl+1 =1 dl+1 =0
× P (dl+1 , Sl+1 , rl+1 |Sl ).
At this point we recover a result from Berrou [9], with P (dl |Sl , Sl−1 ) equal to 0 or 1 (depending on whether the branch exists or not) with P (rl |dl , Sl , Sl−1 ) the error probability of the memoryless (AWGN) channel and with P (Sl |Sl−1 ) the a priori information on the source. We also have P (rl |dl , Sl , Sl−1 ) = P (xl |dl , Sl , Sl−1 )P (yl |dl , Sl , Sl−1 ), (10) and as in our case P (Sl |Sl−1 ) corresponds to the VLC source statistics, P (Sl |Sl−1 ) = P (dl |V LCT ree). Furthermore, as the encoders are systematic, the bit AP P is equal to the transition probability of the bit on the channel, times the extrinsic information. Indeed we have P (xl |dl , Sl , Sl−1 ) = P (xl |dl ), so AP P (dl ) ∼ P (xl |dl )
M M X X
β(Sl )P (yl |dl , Sl , Sl−1 )
Sl =1 Sl−1 =1
× P (dl |Sl , Sl−1 )P (dl |V LCT ree)α(Sl−1 ) (11) ∼ P (xl |dl )Ext(dl ). (12) This last equation (12) is the basis of the turbo-decoding. The decoding scheme is shown in Fig. 4. The addition of the a priori information, defined above, is only possible into the first decoder, because the interleaver breaks the one to one relation between the VLC tree and the decoding trellis in the second decoder. So, the probability P (Sl |Sl−1 ) is set to 1/2, which is equivalent to consider the source as binary symmetric in the second decoder. In our study a Max-LogMAP is used. Denoting a set of n positive real numbers by δi , ∈ {1...n}, the central approximation of this algorithm is given by: ln(eδ1 + ... + eδn ) ≈ max δi . i∈{1..n}
β(Sl+1 )
(9)
(13)
Using this latter relation we get approximations of the different probabilities that are used in the Max-Log-MAP decoding.
Markov source
0
Markov source
0
10
As in the case of soft source decoding it can be seen again from Table IV, that when the a priori on the source defined in section II is added, the decoding process is improved. In the best case a gain of 2.1 dB is obtained.
10
−1
10
−1
Symbol error rate
Bit error rate
10
−2
10
−3
10
−4
10
−5
10
−2
iteration 1, with a priori iteration 1, without a priori iteration 2, with a priori iteration 2, without a priori iteration 3, with a priori iteration 3, without a priori iteration 4, with a priori iteration 4, without a priori −1.5
−1
−0.5
−2
10
V. C ONCLUSION iteration 1, with a priori iteration 1, without a priori iteration 2, with a priori iteration 2, without a priori iteration 3, with a priori iteration 3, without a priori iteration 4, with a priori iteration 4, without a priori
−3
10
−4
0 Eb/N0 (DB)
0.5
1
1.5
10
2
−2
−1.5
−1
−0.5
0 Eb/N0 (DB)
0.5
1
1.5
2
Markov source
0
10
−1
Levenshtein symbol error rate
10
−2
10
−3
10
−4
10
−5
10
−2
iteration 1, with a priori iteration 1, without a priori iteration 2, with a priori iteration 2, without a priori iteration 3, with a priori iteration 3, without a priori iteration 4, with a priori iteration 4, without a priori −1.5
−1
−0.5
0 Eb/N0 (DB)
0.5
1
1.5
2
Fig. 5. Bit and symbol error rate with and without a priori information for a Markov source (cf. Table I). TABLE IV I MPROVEMENT ( IN D B) AFTER FOUR ITERATIONS , OBTAINED WHEN USING THE A PRIORI SOURCE INFORMATION .
−2
BER = 10 SER = 10−2 LSER = 10−2
VLC table from Murad and Fuja Independent source Markov source 0.4 2.1 0.35 2 0.45 2
A.2 First order Markov source We need to know, for each state of each stage, which was the previous symbol being transmitted. Compared to the processing involved for an independent source, it does not increase the complexity of the SUBMAP too much, which only keeps one survivor at each state of each stage. Therefore it is possible to keep track of the survivor in the VLC tree. Then the only change compared to the independent source is that now P (Sl |Sl−1 ) = P (dl |V LCT ree, cp−1 ). B. Simulation results The generators used in the turbo-code are equal to (1 + D + D2 + D4 )/(1 + D3 + D4 ). The rate is equal to 1/2 and the size of the interleaver is 64 × 64. Contrary to the previous cases, the packets are given in bits and not in symbols. As we work with VLCs, it means that a packet does not obviously correspond to an integer number of symbols, so some extra bits are added. Results are given for Murad and Fuja’s table in the Markov case in Fig. 5. The transition probabilities used are shown in Table I and the residual redundancy is R = 0.6742.
In summary we have proposed a new method to split the symbol probabilities of a source into bit probabilities, which will be used to decode VLC encoded sources with or without channel coding. At the expense of a slight increase in complexity, these decoding methods improve performances, compared to those without the a priori on the source. To evaluate these algorithms, three error rates are computed: the BER, the SER and the LSER. If we consider a transmission scheme without channel coding, our algorithm leads to a gain in signal to noise ratio up to 1.2 dB for a Markov source. If a turbo-code is added, the improvement is up to 2.1 dB, when the a priori bit probabilities are used. This approach has a lot of potential applications since VLCs are used in many transmission systems as, for example, in the MPEG4 video standard for intraframe compression, and for the coding of motion vectors [4]. VI. ACKNOWLEDGEMENT We would like to thank Dr. A. Murad and Prof. E. Fuja for providing further information on reference [8]. R EFERENCES [1] L. Guivarch, J.C. Carlach, and P. Siohan, “Joint source-channel soft decoding of Huffman sources with Turbocodes,” in Data Compression Conference (DCC), Snowbird, Utah, USA, Mar. 2000, pp. 83–92. [2] L. Guivarch, P. Siohan, and J.C. Carlach, “Low complexity soft decoding of Huffman encoded Markov sources using Turbocodes,” in International Communication Conference (ICT), Acapulco, Mexico, May 2000, pp. 872–876. [3] C. E. Shannon, A Mathematical Theory of Communication, vol. 27, Bell System Tech. J., 1948. [4] MPEG-4 video verification model version 11.0, Coding of moving pictures and associated audio information, vol. ISO/IEC JTC1/SC29/WG11, Touradj Ebrahimi, 1998. [5] S. Benedetto and G. Montorsi, “Generalized concatenated codes with interleavers,” in International Symposium on Turbocodes, Brest, France, Sept. 1997, pp. 32–39. [6] L. R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, “Optimal decoding of linear codes for minimizing symbol error rate,” IEEE Transactions on Information Theory, pp. 284–287, Mar. 1974. [7] T. Okuda, E. Tanaka, and T. Kasai, “A method for the correction of garbled words based on the Levenshtein metric,” IEEE Transactions on Computers, vol. C-25, pp. 1064–1070, Feb. 1976. [8] A. H. Murad and T. E. Fuja, “Joint source-channel decoding of variable length encoded sources,” in Information Theory Workshop (ITW), Killarney, Ireland, June 1998, pp. 94–95. [9] C. Berrou and A. Glavieux, “Near optimum error-correcting coding and decoding: Turbo-Codes,” IEEE Transactions on communications, vol. 44, pp. 1261–1271, Oct. 1996.
