Compression of Binary Sources with Side

Compression of Binary Sources with Side Information Using Low-Density Parity-Check Codes Angelos D. Liveris, Zixiang Xiong and Costas N. Georghiades Electrical Engineering Department, Texas A&M University College Station, Texas 77843

Abstract— In this paper, it is shown how low-density paritycheck (LDPC) codes can be used as an application of the SlepianWolf theorem for correlated binary sources. We focus on the asymmetric case of compression with side information. The approach is based on viewing the correlation as a channel and applying the syndrome concept. The encoding and decoding procedures, i.e. the compression and decompression, are explained in detail. The simulated performance results are better than most of the existing turbo-code results available in the literature and very close to the Slepian-Wolf limit.

I. I NTRODUCTION The Slepian-Wolf theorem [1] states that the lossless compression of the output of two correlated sources that do not communicate their output with each other, can be as efficient as if they communicated their output. This is true when their compressed outputs are jointly decompressed at a decoder. A sys tem exploiting this property is shown in Fig. 1, where are the compressed versions of the source output andand and are sent (without any distortion, perfect respectively. channel) to the joint decoder which yields and with negligible probability of and . The achievable rates for this system are shown in Fig. 2 of [1], [2]. Practical schemes exploiting the potential of the Slepian-Wolf theorem were only recently introduced based on channel codes, like block and trellis codes [3], [4]. The main idea in [3], [4] was to consider that each output sequence belongs to a coset, which is described by a unique syndrome of a channel code. So instead of transmitting the entire sequence, the shorter length syndrome can be sent and thus the source output is compressed. At the decoder, the information coming from the sources together with the correlation model are used to estimate the original sources’ output with negligible probability of error, i.e. practically without loss. This scheme yields some gains compared to the case where the correlation is not taken into account, but cannot approach the theoretical limits set by the Slepian-Wolf theorem. More advanced schemes were then proposed based on the more powerful turbo codes [5], [6], [7], [2]. Although much better results were presented, it is still not clear how turbo-codes should be employed. Different ways have been proposed so far, like structured lattices [5], puncturing of conventional turbo codes [6], [7] and the syndrome concept [2]. The problem lies in the fact that, apart from the need for the usual performance analysis, there is also an issue with the way to split the compressed output between the component codes.

Source X

Source Y

X

Z

Encoder 1

Z

Y

1

2

Joint Decoder

Xˆ , Yˆ

Encoder 2

Fig. 1. System with two correlated sources not communicating their output and a joint decoder.

Low-density parity-check (LDPC) codes seem to be more suited for such an application. Their application to the SlepianWolf problem was first suggested in [8] in a more general and theoretical context. No component code issues arise and nowadays they appear to be the most powerful channel codes [9]. Furthermore, in distributed source coding based on the syndrome concept, no “conventional” channel encoding takes place. Instead, from each sequence of source output bits the corresponding syndrome is determined using the sparse parity check matrix as proposed in [8]. This syndrome is the compressed information sent to the joint decoder. But the symmetric case considered in [8] could not be linked to the already available LDPC code design results in a straightforward way. Our main contribution in this paper is combining a different way of viewing the problem using an equivalent channel and the syndrome approach, in the case where one of the two correlated sources is available losslessly at the joint decoder (asymmetric case). The simplicity with which LDPC codes can then be used for this compression problem with side information becomes clear and results from two points. First, it is straightforward to modify the conventional message passing LDPC decoder to take the syndrome information into account in the binary case considered here. Secondly, all LDPC code design techniques can be applied to distributed source coding producing simulation results clearly better than any turbo code scheme suggested so far. And further optimization is expected to yield even more compression. This paper is outlined as follows; Section II describes the overall system and introduces the equivalent way to view it through a correlation channel and the syndromes. In Section III the encoding and decoding procedures with LDPC codes are explained and in Section IV regular and irregular LDPC codes are simulated and compared with turbo codes. The conclusion sums up the paper.

II. P ROBLEM S ETUP, C ORRELATION C HANNEL AND S YNDROMES We consider the system of Fig. 1 with the following assumptions and are memoryless discrete random variables, i.e. knowing the previous values of either of them does not provide any information for their current value. The same instant and are correlated. is available losslessly at the joint decoder and we try to compress as efficiently as possible. Since the rate used , the theoretical limit for for is its entropy lossless compression of is from the Slepian-Wolf theo . rem [1] Imposing the above three assumptions to the system of Fig. 1 we end up with a compression with side information problem. The resulting system is shown in Fig. 2. There is an equivalent way one can view the system of Fig. 2 in order to allow the use of channel codes. The correlation between and can be modeled with a channel; will be the input to the channel and its distorted output as shown in Fig. 3. Then the compressed version of , i.e. , can be used to make look like a codeword of a channel code. A partition of all the typical sequences of is made into disjoint channel codes with the same properties, i.e. they all have is the inthe same performance over the correlation channel. dex of the codebook belongs to. This approach was actually the basis of the proof of the Slepian-Wolf theorem [1], but so far it has not been made clear in the existing literature for the design of good codes. If we also introduce the assumption that all the values of and are equiprobable, the disjoint channel codes can be formed using syndromes. This is very easily done in the binary case where the previous assumptions are modified as follows: and are equiprobable memoryless binary random variables. The same instant and are correlated with Pr $ & # ( % * ' , ) + . This will be called binary symmetric chan"! nel correlation from now on. is available losslessly at the joint decoder and we try to compress as efficiently as possible. Since the rate used -. / bit, the theoretical for is its entropy limit for lossless compression of is from the Slepian0 12# 43#057698 #:3 Wolf theorem [1] ;/3/3?# . The rest of the paper is going to be based on these three assumptions. Using a linear A@CB>D binary block code in this binary case, there are EGF*HJI distinct syndromes, each indexing a set of E9I binary words of length @ . We call the linear block code (all-zeros syndrome set) original code. All the EKFLHJI sets are disjoint, because the binary summation of any two binary words in the same set (same syndrome) equals a codeword of the original code, while the binary summation of any two binary words in distinct sets (different syndromes) does not equal an codeword of the original code. This also means that in each set the Hamming distance properties of the original code are preserved, i.e. all codes have the same performance over the binary symmetric

MSource X

X Rate RX

Q

M

R P

Q

N

Z1 Z1 Source Perfect Joint Encoder 1 Rate R ≤ R Channel Rate R ≤ R Decoder 1 X 1 X

OXˆ

MSide information S Y

Fig. 2. System for compression with side information.

V WChannel Y Decoder T(sourceXoutput) T(correlation) T(side information) T(code defined U Xˆ by Z ) 1

T(compressedZ version of X) 1

Fig. 3. Equivalent correlation channel for compression with side information.

correlation channel. So the compression ratio achieved with this scheme is @YXZA@[3D . Since low-density parity-check (LDPC) codes have been proven to be very powerful linear binary block codes, we will next explain how they could be used for compress and decompress with side information. III. E NCODING

AND

D ECODING

WITH

LDPC CODES

Low-density parity-check (LDPC) codes are best described by their parity-check matrix \ and the associated bipartite graph. The A@]3^D C_ @ parity-check matrix \ of a binary LDPC code is like that of an ordinary binary block code, but it is sparse, i.e. it has a small number of ones. The way these ones are spread a in \ is described by the degree distribution polynomials ` Ab and c ` db , which indicate the percentage of columns and rows of \ respectively, with different Hamming weights (number of ones). gf a 'L) h=+9bLi , means that E +9j of all For instance ` Ab e'L) E +9b the columns in \ have Hamming weight equal toa three and the rest h9+9j have Hamming weight four. When both ` db and c ` db have only a single term, the LDPC code is regular, otherwise it is irregular. In general, an optimized irregular LDPC code is expected to be more powerful than a regular one of the same codeword length and code rate. a Given both ` Ab and c ` db , the code rate is exactly determined, but there are several \ ’s that can be formed. Usually one is picked at random, sometimes satisfying some constraints that can be more easily associated with the bipartite graph. The bipartite graph of an LDPC code is an equivalent representation of the parity-check matrix \ . Each column is represented with a variable or left node and each row with a check or right node. All variable nodes (circles in Fig. 4) are put in one column, all check nodes (squares in Fig. 4) in a parallel column and then wherever there is an one in \ , there is an edge connecting the corresponding row and column. a The reason for using the tilde in ` Ab and c ` Ab , is that they refer to the node profile, i.e. they give the percentage of nodes a with various degrees. In the LDPC literature, the edge profiles db and c Ab are more frequently used, which indicate how many edges in the bipartite graph are connected to different degree nodes. The bipartite graph is used in the decoding procedure, allowing the application of the message-passing algorithm [10].

. qi,in1 . q out i,1 . q out vi i, m q in i, m out . q .. ¢qi,in¡l i,l

1

y 0

0

. . .

i

. . .

i

1

kuncompressed binary lsource output 0

1

mcompressed output n(syndrome)

1

Fig. 4. Encoding with an LDPC code using its bipartite graph: at each right (check) node all the connected left (variable) nodes are added modulo 2. Here the codeword length is only 6, , and the rate of the original (conventional) LDPC code is , yielding a compression . ratio of

oZp7q9rYsutvwqyx{zvwqK| w} p7q9r~suzvwqK|x{tvKq z >

A. Encoding The encoding procedure for our system is quite straightforward. We first fixa the codeword length @ and determine the degree distribution Ab and c db . Then we generate a random realization of \ . To encode, i.e. compress, an arbitrary binary input sequence, we form a vector with @ successive bits multiply them with \ [8] and find a corresponding syndrome. Equivalently in the bipartite graph this can be viewed as binary addition of all the left node values that are connected to the same right node, as shown in Fig. 4. The left side in Fig. 4 shows 6 successive values of the binary output of the source of Fig. 1 and 2 and the right of the first encoder in side the corresponding 4 output values Fig. 1 and 2. This encoding procedure has low complexity, linear with the codeword length, since \ is a sparse matrix. For instance, with the code of Fig. 4, :9E compression is achieved, and for all inputs, the distance properties of the original LDPC code are preserved. B. Decoding

tin j,π (i,m,j) c j tout j,π (i,m,j) © ¦ § . j, r . ©t¨ªin § t¨ out . j, r

Fig. 5. Notation for the decoder operation at the check node .

²´³

s

. . .

j

. . .

j

j

«7¬7

variable node

®°¯

and

±;¬7

0

1

Cs&w

i

1

¥ £in out t¤ j,1 .. t j,1 .

The decoder has to determine an @ -length sequence of successive ’s from its A@3D -long syndrome and the corresponding @ -length sequence of successive ’s. Given the ’s alone it would be like transmitting over a binary symmetric channel with crossover probability # . With the syndrome, it is like these @ -successive ’s form an LDPC-like codeword. For instance if the syndrome is an all-zeros vector, the @ ’s are indeed an LDPC codeword and can be decoded with the message-passing algorithm [10] using the bipartite graph. The modification needed for the case of a nonzero syndrome is quite simple to think. Before starting the decoding algorithm for each

binary word of length @ , all the nonzero positions in the syndrome vector are marked in the bipartite graph, i.e. the corresponding check nodes. Again the same message-passing decoding algorithm is used, but whenever we have to operate over a marked check node (applying the “tanh rule” [9], [10]), an extra change of sign is performed at each output value. This is because the inverse parity has to be satisfied and in the log-domain the ratio of probabilities has to be inverted, which is equivalent to an extra change in the sign at the ouput of the check node. To present the decoding algorithm in more detail, we use the notation of Fig. 5 where: 3µbL¶·B¹¸w¶1º¼»½'*Bw/K¾ are the values of and respectively, corresponding to the ¿´ÀÂÁ variable (left) node Ã ¶ , 3µÄÅ¶:ºÆ» E B Bw)w))Ç¾ZB ¿ º»È/9B E Bw)w))*B>@É¾ÈB is the degree of the ¿´ÀÂÁ variable node Ã ¶ , i.e. the number of edges connected to Ã ¶ in the biparite graph, 3µÊw¶ÎË>Í Ì Ï À dÊw¶ÇÐ Ñ Í Ï ºÒ , ¿ º(»Z/9B E Bw))w)ZB;@É¾ZB*ÓÔºÕ»È/9B E Bw)w))ZB>Ä ¶ ¾ÈB is the log-likelihood ratio (LLR) sent along the Ó ÀÂÁ edge from (to) the ¿´ÀÂÁ variable node Ã ¶ , 3µÖØ×-ºÙ»K'*Bw/K¾ is the value of corresponding to the ÚKÀÂÁ check (right) node Û × , i.e. the value of the Ú½À´Á syndrome component, 3µÜ × ºÙ» E B B)w)w)Ç¾ is the degree of the ÚKÀÂÁ check node Û × , i.e. the number of edges connected to Û × in the biparite graph, 3ÞÝ·×Ë>ÍÌ Ï À ´Ý·×°Ð Ñ Í Ï º Ò , Ú º »È/9B E Bw)w))ZB>@43ßDJ¾ÈB*Ó º »È/=B E B)w)w)ÈB>ÜØ×9¾ZB is the LLR sent along the Ó À´Á edge from (to) the ÚKÀÂÁ check node Û × . With this notation in the sum-product decoding algorithm, the edges originating from the variable nodes are initialized with

' Ê ¶ÇÍ à u57698áãâ bb*¶¶~1 4 / áãâ

¸ ¶ ! ä>/3 E ¸ ¶ 57698 /3?# B (1) # ¸w¶ ! ¿ 4/9B E Bw))w)*B>@CB ¸w¶ ! áCâ Þ ]! %æ'*),+ . where #å áCâ bL¶ e In a single iteration, first the variable nodes are processed. The LLR sent from the ¿ÂÀ´Á variable node Ã ¶ along the Ó ÀÂÁ edge is determined from

é Åç è Ê ¶ÎÐ Ñ Í × BíÓî4/9B Bw))w)*B>Äë¶ØB E × ê Íë×9êgì Ï ¿ 4/9B E Bw))w)*B>@CB where initially Êw¶ÇÐ Ñ Í × e' . Ê Ç¶Ë>Í Ì Ï À 1Ê¶ÎÍ à

f

(2)

The values Êw¶ÇË;Í Ì Ï À are assigned to the corresponding Ý·×Ð Ñ Í ïKð7¶ÇÍ ÏñÍ ×wò according ó to the connections in the bipartite graph and are then used to do the processing at the check nodes. Following the “tanh rule” and taking into account the syndrome information, we determine the LLR sent from the ÚKÀÂÁ check node Û × along the Ó À´Á edge from

ô;õKöL÷õKö*÷^þ E ÿ B B Ú /=B E B)w)w)*B;@ 3{D ) ý

B. Improved Irregular Code Design (3)

This is actually the only point that is modified in the “conventional” sum-product decoding algorithm, in order to account for the syndrome information. Now Êw¶ÇÐ Ñ Í Ï ÔÝ·×Ë>ÍÌ ïKÀ ð7¶ÇÍ Ï Í ×ò for all edges in the bipartite graph, which can be used to start a new iteration and estimate b*¶ from

Lb ¶~

'LBÊ ¶ÇÍ à f f /=BÊ¶ÇÍ à

ÏçÅè ê wÊ ¶ÇÐ Ñ Í Ï ' ) y ÏyçÅè ê wÊ ¶ÇÐ Ñ Í Ï %('

(4)

IV. S IMULATION R ESULTS AND C OMPARISON T URBO -C ODES A. Simple Regular and Left Regular Codes

WITH

a

We first simulated regular and left regular (only db has a single term) LDPC codes in order to compare the results with those of turbo-codes [6] using approximately the same parameters. So here the codeword length is @ÆÙ/ , i.e. equal to the interleaver length of [6], and more than 2000 blocks were transmitted without a single error after 40 iterations of the message algorithm. As for the LDPCfcode distribution, a Ab passing c b H ,f where b ç H and c db >/3 c f b H Ä and Üä >/3 c AÜ 3î/ û c û ÜäíÜ[û 3îû / c is varying so that the code rate û changes. Weû should note that a û approximations were needed in some cases so that @ and Ab b ç H remained fixed. For example, f for @ / , Ä c Ü h A b * ' , ) 9 ' = ' * ' / b '*) b , or equivalently and , f E c ` Ab e'L) '='9'L/ b '*) 9b . The results are given in the following table together with those of [6]. and are the rates used for and respectively, in compressed bits per information bit and so in our case µ/ . For each of the results in the table two difalways ferent random realizations of the LDPC code with this distribution were implemented and both could correct all errors in 2000 blocks of length 16384. Of course, as mentioned in [6], this does not guarantee that the compression is completely lossless, but it shows that there are good LDPC codes with negligible loss.

#

0.025 1.169

0.05 1.286

0.1 1.469

0.2 1.722

1.310

1.435

1.630

1.890

1.300 (3,10)

1.429 (3,7)

1.600 (3,5)

-

1.276 (3,10.9)

1.402 (3,7.5)

-

1.875 (3,3.5)

f B >/ f .2# Ø [6] regular f / ( ÄdB Ü ) left regular f / ( ÄdB Ü )

First the regular LDPC code results are shown (c / ) and then some simple irregular code results ( ' % c %Ùû / ), for the cases that further compression could be achieved. û From the table, it is clear that these very simple LDPC codes could improve the turbo-code performance.

Much better results are expected from LDPC codes with the improved irregular code design as done in [10], [11], [12], which can directly be applied here. In the literature, a few good irregular codes have already been suggested and next we are going to use the simplest good irregular code of [12],1 i.e. the rate /wX E code with the degree distribution:

/ 'LE ) ÈE ' +=b ' f b 'L )2/ b / +=b

E

f '*) E9 ' E 9b 'L) E f f 'L)2/w' E f=' b '*),h*/ h=+9b L' ) E

a A b c A b

(5) (6)

This irregular code was simulated together with the regular (3,6) code and the results are shown in Fig. 6. The codeword (frame) length was @$/w' for the regular code and @Æ¼/' and @å4/' for the irregular code (marked “awgn”). The probability Pr ! , bit error rate (BER) for , was measured after 100 iterations of the decoding algorithm. The Slepian-Wolf theoretical limit at 'L) + bits as well as the best turbo code performance reported for this code rate [7] (about 'L) + bits for codeword length /' ) are also shown in Fig. 6. The second irregular code, marked “bsc” in Fig. 6, is the rate 1/2 LDPC code given in example 2 of [11], which has been optimized for the binary symmetric channel (BSC) and has the distribution

a d b e 'L)2/w+9h=+ f f 'L),' +=+ f 'L),'*/ =' f 'L),'9'9' E 'L),' E ' f f 'L),' / f 'L),'*/9/ / f 'L),'*/' 'L),' ' f c ) Ab æ'L) E +9b

f f w/ b '*)7/ 9+ b '*),' */wb i f f E=E ÈE /b f 'L),'*/9/ + +9b 'Lf ),' 9' 9b 9h9b '*f ),'9' E + 9E h=h9b · 'Lf ) '=' + h 9h=b · h =h9 b f à '*),'L/ = L/b f '*),' =E h9h=b 9h9b i 'L) '='9'=h '=+*/ 9 b '*),' 9 9' b f f b > f 'L) '=' E= /9/wh9b f '*),'='*/ /w+9h9b à b f 'L) 'L/9/ 9b f '*),' +9'L/ = b i à b '*f ),'*/ ' b '*),' E +9+ b 9b à 'L) ' + 9b (7) '*),h=+9b à (8)

The threshold for this code, i.e. its best possible performance for very large codeword length and number of iterations, is .2# 1'*) bits [11], also shown in Fig. 6. It is clear from Fig. 6 that the regular (3,6) code slightly outperforms the turbo coding schemes of [5], [7], which use greater or equal codeword length. As for the irregular code of (5) and (6), it almost halves the gap to the Slepian-Wolf limit from what was achieved before with the best turbo coding scheme [7], even with a codeword length of @æ /' , two orders less than that

"!

#

The irregular codes in [12] have been optimized for the additive white Gaussian noise (AWGN) channel, but we expect them to perform better than regular or nonoptimized irregular codes over the binary symmetric channel (BSC) as well. This was verified with the density evolution on-line demo at http://truth.mit.edu/ sychung/.

$

a A b

−2

10

c A b

5

irregular 10 (awgn)

-

(11) (12)

4

irregular 10 (awgn)

4

regular 10

−3

e'L)7/ E + results is f f '*f ),' 9E b 'L ) E f h9' ÈE h=b 'L) ' E h=h*/ =b à 'L) E ' ÈE h=b '*) +=h +9b B b K)

and the rate 7/8 one used to get the

Probability of error for regular and irregular rate 1/2 LDPC codes (100 iterations)

−1

10

Optimization for the binary symmetric channel (BSC) is expected to increase the gains over turbo codes.

BER for X

10

5

irregular 10 (bsc) −4

10

V. C ONCLUSION threshold for irregular (bsc)

−5

10

Slepian−Wolf limit

best turbo [7] −6

10

0.34

%

0.36

0.38

0.4

0.42 0.44 H(X|Y)=H(p) (bits)

%'&

0.46

0.48

0.5

0.52

°

Fig. 6. Simulation results for a regular and three irregular rate LDPC codes of length and for 100 iterations in the decoder. The regular code was the (3,6) one and the irregular codes marked “awgn” both have the distribution of (5) and (6), while the irregular marked “bsc” has the distribution of (7) and (8). For each point either 100 frame errors or bits were simulated. The best turbo-code performance, the threshold of the irregular bsc code and the SlepianWolf limit are also shown.

t

)( %'* >

of [7]. Further irregular LDPC code design for the binary symmetric channel yields slightly better results as the difference between the “awgn” and “bsc” curves of Fig. 6 shows. However, regular codes could not outperform the codes of [7] for higher compression, i.e. 4:1 and 8:1. Even the advantage of good irregular codes was becoming less noticable the higher the compression. This is shown in the following table comparing our results for length /' irregular LDPC codes with those of [7] (length /' ). For the rate 1/2 codes we considered the results of the LDPC code of (7) and (8), whereas for the other two cases we designed a rate 3/4 and a rate 7/8 LDPC code respectively using the method of [10], i.e. the latter two codes were optimized for the additive white Gaussian noise (AWGN) channel. The irregular LDPC results shown assume a probability of error smaller than /w' H for + =/' simulated bits.

,+

: [7] (irreg. LDPC)

0.125 0.089 0.091

0.250 0.185 0.204

0.500 0.346 0.466

For the design of the rate 3/4 and 7/8 LDPC codes we considered right regular codes and allowed a maximum left node degree of 50. To easily avoid length 4 cycles of degree-2 left nodes, we also upper-bounded the percentage of degree-2 left a nodes, i.e. the coefficient of b in ` Ab , with 1/4 and 1/8 respec tively. The resulting rate 3/4 distribution used to get the '*) E +9' results is

a A b c A b

f b '* à) E9f '*'L/=) '=/ h b f /wb '* ),'9 h9+9b b B 9E

'Lf ),'9h*/ ÈE b '*)7/ h9' È f '*) =h b KB

(9) (10)

We presented a way of doing distributed compression with side information using binary LDPC codes. We showed how the encoding and decoding procedures should be modified based on the correlation channel and the syndrome approach. The simulated results show that in the binary case, the practically lossless compression achieved by LDPC codes, especially irregular ones, is clearly higher than that of the currently available turbo schemes and very close to the theoretical limit. R EFERENCES [1] D. Slepian and J.K. Wolf, “Noiseless coding of correlated information sources,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 471–480, July 1973. [2] A.D. Liveris, Z. Xiong and C.N. Georghiades, “A distributed source coding technique for highly correlated images using turbo-codes,” to appear in ICASSP ’02. [3] S.S. Pradhan and K. Ramchandran, “Distributed source coding using syndromes (DISCUS): Design and construction,” in Proc. IEEE DCC, Mar. 1999, pp. 158–167. [4] S.S. Pradhan and K. Ramchandran, “Distributed source coding: Symmetric rates and applications to sensor networks,” in Proc. IEEE DCC, Mar. 2000, pp. 363–372. [5] J. Bajcsy and P. Mitran, “Coding for the Slepian-Wolf problem with turbo codes,” in Proc. IEEE Globecom ’01, Nov. 2001, vol. 2, pp. 1400–1404. [6] J. Garcia-Frias and Y. Zhao, “Compression of correlated binary sources using turbo codes,” IEEE Comm. Letters, vol. 5, pp. 417–419, Oct. 2001. [7] A. Aaron and B. Girod, “Compression with side information using turbo codes,” to appear in DCC ’02. [8] T. Murayama, “Statistical mechanics of linear compression codes in network communication,” Europhysics Letters, preprint, 2001. [9] S.-Y. Chung, G.D. Forney, T.J. Richardson and R. Urbanke, “On the design of low-density parity-check codes within 0.0045 db of the Shannon limit,” IEEE Comm. Letters, vol. 5, pp. 58–60, Feb. 2001. [10] S.-Y. Chung, T.J. Richardson and R. Urbanke, “Analysis of sum-product decoding of low-density parity-check codes using a Gaussian approximation,” IEEE Trans. Inform. Theory, vol. IT-47, pp. 657–670, Feb. 2001. [11] T.J. Richardson, M.A. Shokrollahi and R.L. Urbanke, “Design of capacity-approaching irregular low-density parity-check codes,” IEEE Trans. Inform. Theory, vol. IT-47, pp. 619–637, Feb. 2001. [12] S.-Y. Chung, On the Construction of Some Capacity-Approaching Coding Schemes, Ph.D. thesis, Massachusetts Institute of Technology, 2000, Chapter 8.