On Error Bounds and Turbo Codes Christian Schlegel and Lance Perez Department of Electrical Engineering University of Utah Salt Lake City, UT 84112
[email protected] Department of Electrical Engineering 229 N Walter Scott Engineering Building University of Nebraska-Lincoln Lincoln, NE 68588-0511
[email protected] March 3, 1999
Abstract: Turbo codes have been hailed as the ultimate step towards achieving the capacity limit Shannon established some 50 years ago. In this letter we look at the performance of Turbo codes with respect to various information theoretic error bounds. This comparison suggests that, if (block, or) frame error rates are considered, careful interleaver design is necessary to ensure an error performance within a fraction of a dB of the theoretical limit for large block sizes, while random interleavers perform well for block sizes smaller than about 2K. If bit error performance is considered, interleaver design seems to have only a minor e ect, and the codes perform close to the limit for all block sizes considered.
Keywords: Turbo codes, error bounds, sphere-packing bounds
Manuscript submitted December 10, 1997, revised February 28, 1998. This work was supported by a grant from the Nokia Research Center, Irving, TX and the Center for Communication and Information Science at the University of Nebraska-Lincoln.
1
I. Introduction
In 1993, a new coding and decoding method, called Turbo coding for its iterative decoding mechanism, was reported to achieve low error rates at unprecedentedly low signal-to-noise ratios [1, 2]. This new method achieved its performance through the use of two or more parallel concatenated convolutional codes, which were individually decoded by maximum-a-posteriori (MAP) decoders [9]. The MAP decoders exchange information by providing estimates of the a posteriori probabilities of the information bits to each other. This process is iterated serveral times to achieve the low error rates. Although both encoders operate on the same block of information bits, these bits are interleaved in a pseudorandom (non-structured) fashion between the two encoders. It is indeed this pseudorandom interleaving which is now recognized as providing the excellent performance results [3]. Bit error rates of Pb = 10, were achieved at SNR's of 0:7dB, merely 0:5dB from Shannon's capacity limit for binary-phase shift keying (BPSK) signalling. However, in order to achieve this \near-Shannon limit" performance, very large block lengths are required. In the original example, a block length of N = 131; 072 encoded bits was used. Suggestions have been made that these codes are the \Shannon codes" promised by the existence argument of the capacity theorem. We would like to take a closer look at this proposition and relate the performance of Turbo codes to various bounds from information theory. 5
II. Error Bounds and the Capacity Theorem
One common way of proving the capacity theorem is via the Gallager bounds on the block error probability of block codes. Let N be the length of a block, and PB the probability of making a block error at the decoder, i.e., decoding the wrong code word. Then we can upper bound the block error probability of the best code [5] by PB < 2,N E R ; E (R) = max max [E (; q) , R] ; (1) q ( )
0
0
1
where R is the code rate in bits/symbol, q is the distribution of the input symbols, and # Z "X = E (; q) = , log q(x)p(yjx) : (2) 1+
0
2
1 (1+ )
y
x
In the above equations, x is an input signal to the channel, q(x) is its probability of being chosen by the encoder, y is the channel output signal, and p(yjx) is the conditional probability of the channel output signal, given the input. E (R) in (1) is known as the Gallager exponent. For additive white Gaussian noise (AWGN) channels, y = x + n, where n is uncorrelated zero-mean Gaussian noise with variance N =2 and ! ( y , x) 1 exp , : (3) p(yjx) = p 0
2
N0
N0
p
In the case of BPSK modulation, we also have x = Es and it is known that the maximizing distribution q(x) is given by q(x) = 1=2, for all x, and !" p !# Z 1 2 y Es y + Es pN exp , N E (; q) = , log cosh : (4) N (1 + ) y From (1) we can conclude that as long as E (R) > 0, increasing N will decrease PB exponentially. It is well known that E (R) > 0 for all rates R < C , where C is the channel capacity. This then implies that the best code achieves arbitrarily low block error probabilities with increasing N . This in turn implies that the bit error probability, Pb, also decreases exponentially, since Pb PB . (That is, every block error implies at least one bit error.) Thus, these bounds provide limits on what can be achieved with coding. We will call codes for which PB ! 0 as N ! 1 \Shannon codes". Since PB =N Pb PB ; (5) the bit error probability Pb ! 0 also for Shannon codes due to the right inequality above. 1+
2
0
2
0
0
0
III. Converses
There are also several results from information theory which dictate what cannot be achieved with coding. Fano's converse [4, 5] to the channel coding theorem states that at rates larger than C , the bit error probability is bounded away from zero, but gives no functional dependence on N , regardless of the encoding mechanism. Fano's converse is sometimes refered to as the weak converse to the channel coding theorem [5]. Furthermore, there exist lower bounds on the block error probability showing that PB ! 1, as N ! 1, for R > C . These bounds [5, 6, 7] are refered to as the strong converses of the channel coding theorem. For rates below the channel capacity, Shannon, et. al. [8], showed that PB (of the best codes) is lower bounded by PB > 2,N Esp R o N ; Esp (R) = max max [E (; q) , R] : (6) q (
( )+ (
))
0
0
The expression in (6) is known as the sphere-packing bound. The only di erence in the exponents in (6) and (1) is the range of maximization of . This means that as long as the maximizing value of is less than or equal to 1, the two bounds are asymptotically identical. In addition, 1 holds for R ! C , i.e., for the interesting case of operation close to capacity. Thus, the sphere-packing bound (6) gives us a tool to evaluate the best possible performance of a block code of length N , in terms of block error rates. We have plotted the bound of (6) for R = 1=2 in Figure 1 (neglecting the o(N ) term), together with the performance of a variety of Turbo codes of various lengths using iterative MAP decoding algorithms [9, 10]. The term o(N ) in (6) plays an appreciable role only for short block length. It is dicult to calculate and we have dashed the curve representing the bound where it becomes loose.
From the gure it becomes apparent that PB of Turbo codes using random interleavers increases again after reaching a minimum at around N = 2000. While this may seem surprising at rst glance, the error performance analysis of Turbo codes [3] shows that Turbo codes with long blocklengths have very small free distances, and, as the block length increases, so does the probability that a code word with a very close neighbor is generated. In order to remedy this situation and generate Turbo codes with larger free distance, JPL have used spread interleavers, which guarantee a minimum spread of information symbols in the interleaved data stream at the encoder. We have used their interleavers [11] to generate the data points (8) and (9). These codes perform within less than 1dB of the sphere packing bound. Also shown in Figure 1 are the block error rates of various concatenated coding schemes, illustrating that they are close competitors only to Turbo codes using random interleavers.
Eb/N0 [dB] (Block Error Rates 10-4)
4
N=448 4-state
(1) N=2040 concatenated (2,1,6) CC RS (255,239) code (2) N=2040 concatenated (2,1,6) CC RS (255,223) code (3) N=2040 concatenated (2,1,8) CC RS (255,239) code (4) N=2040 concatenated (2,1,8) CC RS (255,223) code
(7)
N=360 16-state N=448 16-state
3
N=1334 4-state (1) (2)
N=1334 16-state
2
(5)
(3) (4) N=2048 16-state
Unachievable Region
(6)
(5) N=4599 concatenated (2,1,6) CC RS (511,479) code (6) N=4599 concatenated (2,1,8) CC RS (511,479) code (7) N=1024 block Turbo code using (32,26,4) BCH codes (8) N=10200 (2,1,4) Turbo code using a spread interleaver
N=4096 16-state
(9) N=16384 (2,1,4) Turbo code using a spread interleaver N=10200 16-state N=16384 16-state (8) (9)
1
0
N=8192 16-state
Shannon Capacity
10
100
1000 104 Blocklength N
105
106
Figure 1: The sphere-packing bound and the block error performance of various complex coding schemes as a function of the block length N . The interesting point, however, is that the bit error rate continues to decrease for Turbo codes using random interleavers, even as PB increases again. It is this remarkable bit error performance which lead the inventors of Turbo codes to conclude that they had found the \Shannon codes". This bit error rate performance for these codes, as well as for the Turbo codes using spread interleavers, and concatenated codes is shown in Figure 2. The sphere packing bound is drawn as a dashed line, since there are no meaningful lower bounds on the bit error probability, other than the Shannon capacity limit.
The situation seems that for Turbo codes using random interleavers the left inequality in (5) is tighter, and hence as N grows, low Pb 's can be achieved without PB ! 0. VI. Conclusions
We have taken a look at what information theory has to say about the excellent performance of Turbo codes, and have come to the conclusion that in order to obtain block error rates close to what is promised by Shannon's capacity theorem, careful interleaver design is required for long block lengths, while for short block lengths and bit error rates, random interleavers are fully sucient, in achieving extremely small bit error rates even if operated very close to capacity. 4 Eb/N0 [dB] (Bit Error Rates 10-5)
N=360 16-state
N=448 4-state (7)
(9)
3
(1) N=2040 concatenated (2,1,6) CC RS (255,239) code (2) N=2040 concatenated (2,1,6) CC RS (255,223) code (3) N=2040 concatenated (2,1,8) CC RS (255,239) code (4) N=2040 concatenated (2,1,8) CC RS (255,223) code
N=448 16-state
Block Error Rate Sphere Packing Bound
(1) (2)
N=1334 4-state
(5) (3) (4)
2
(8) (6)
(8) N=4599 concatenated (2,1,8) CC RS (511,479) code (9) N=10200 (2,1,4) Turbo code using a spread interleaver
N=1334 16-state N=2048 16-state N=4096 16-state
1
N=10200 16-state
Shannon Capacity
(5) N=4599 concatenated (2,1,6) CC RS (511,479) code (6) N=4599 concatenated (2,1,8) CC RS (511,479) code (7) (8) (9) BCH based Block Turbo Codes
N=8192 16-state (9)
(10) N=16384 (2,1,4) Turbo code using a spread interleaver
N=32768 16-state N=131072 16-state (10) N=16384 16-state N=229376 16-state
0 Unachievable Region
10
100
1000 104 Blocklength N
105
106
Figure 2: The bit error performance of the same coding schemes as a function of the block length N .
References [1] C. Berrou, A. Glavieux, and P. Thitimajshima, \Near Shannon limit errorcorrecting coding and decoding: Turbo codes", Proc. International Conference on Communications, ICC'93, Geneva Switzerland, May 1993. [2] C. Berrou and A. Glavieux, \Near optimum error-correcting coding and decoding: Turbo codes", IEEE Trans. Commun., Vol. 44, No. 10, pp. 1261{1271, October 1996.
[3] L.C. Perez, J. Seghers and D.J. Costello, Jr., \A Distance Spectrum Interpretation of TURBO Codes," IEEE Trans. Inform. Theory, special issue on codes and complexity, Vol. IT-42, pp. 1698{1709, November 1996. [4] R.M Fano, \Class Notes for Transmission of Information," Course 6.574, MIT, Cambridge, Mass. [5] A.J. Viterbi and J.K. Omura, Principles of Digital Communication and Coding, McGraw-Hill Inc. 1979. [6] J. Wolfowitz, \The Coding of Messages Subject to Chance Errors", Ill. Journal of Math., vol. 1, pp. 591{606, 1957. [7] S. Arimoto, \On the Converse to the Coding Theorem for Discrete Memoryless Channels," IEEE Trans. Inform. Theory, vol. IT-19, pp. 357{359, 1973. [8] C.E. Shannon, R.G. Gallager, and E.R. Berlekamp, \Lower Bounds to Error Probability for Coding on Discrete Memoryless Channels," Inform. Contr., vol. 10, pt. I, pp. 65{103, pt. II, pp. 522{552, 1967. [9] L.R. Bahl, J. Cocke, F. Jelinek, and J. Raviv, \Optimal decoding of linear codes for minimizing symbol error rate," IEEE Trans. Inform. Theory, vol. IT-20,, pp. 284{287, 1974. [10] C. Schlegel, Trellis Coding, IEEE Press, Piscataway, NJ, 1997. [11] D. Divsalar, private communication about spread interleavers.
