IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 12, DECEMBER 2004
2087
Transactions Papers Threshold Values and Convergence Properties of Majority-Based Algorithms for Decoding Regular Low-Density Parity-Check Codes Pirouz Zarrinkhat, Student Member, IEEE, and Amir H. Banihashemi, Member, IEEE
Abstract—This paper presents a detailed study of a family of binary message-passing decoding algorithms for low-density paritycheck (LDPC) codes, referred to as “majority-based algorithms.” Both Gallager’s algorithm A ( ) and the standard majority decoding algorithm belong to this family. These algorithms, which are, in fact, the building blocks of Gallager’s algorithm B ( ), work based on a generalized majority-decision rule and are particularly attractive for their remarkably simple implementation. We investigate the dynamics of these algorithms using density evolution and compute their (noise) threshold values for regular LDPC codes over the binary symmetric channel. For certain ensembles of codes and certain orders of majority-based algorithms, we show that the threshold value can be characterized as the smallest positive root of a polynomial, and thus can be determined analytically. We also study the convergence properties of majority-based algorithms, including their (convergence) speed. Our analysis shows that the stand-alone version of some of these algorithms provides significantly better performance and/or convergence speed compared with . In particular, it is shown that for channel paramethe error probability converges ters below threshold, while for to zero exponentially with iteration number, this convergence for other majority-based algorithms is super-exponential. Index Terms—Density evolution, hard-decision iterative decoding algorithms, low-density parity-check (LDPC) codes, majority-based iterative decoding algorithms, message-passing decoders.
I. INTRODUCTION
G
ALLAGER introduced low-density parity-check (LDPC) codes in his thesis in the early 1960s [1]. They were soon forgotten, however, due to limited computational resources available at the time, which made the corresponding iterative decoding algorithms look impractical. It was not until the introduction of turbo codes [2] that researchers realized the
Paper approved by P. Hoeher, the Editor for Coding and Communication Theory of the IEEE Communications Society. Manuscript received December 17, 2003; revised July 19, 2004. This work was supported in part by an Ontario Graduate Scholarship in Science and Technology (OGSST) and in part by an Ontario Graduate Scholarship (OGS). This paper was presented in part at the 40th Annual Allerton Conference on Communications, Control, and Computing, Allerton House, IL, October 2002. The authors are with Broadband Communications and Wireless Systems (BCWS) Centre, Department of Systems and Computer Engineering, Carleton University, Ottawa, ON K1S 5B6, Canada (e-mail:
[email protected];
[email protected]). Digital Object Identifier 10.1109/TCOMM.2004.838738
strength of iterative coding schemes and the attractive performance/complexity tradeoff that they offer. Since then, turbo codes and LDPC codes have been the subject of much research. In particular, there has been a great amount of research devoted to LDPC codes and the performance and the complexity of associated iterative decoding algorithms [3]–[5]. LDPC codes are linear block codes, and therefore, can be fully described by their parity-check matrices. In particular, a -regular LDPC code has a parity-check matrix binary with and ones in each column and row, respectively. In irregular LDPC codes, column and row weights are not constant and are chosen according to some distribution. Iterative decoding algorithms for LDPC codes can be naturally described using bipartite graphs constructed based on [6]. Decoding process is performed by exchanging messages between variable nodes and check nodes along the edges of these graphs, iteratively. The corresponding decoding algorithms are thus referred to as message-passing algorithms [7]. There is a wide range of iterative message-passing algorithms for decoding LDPC codes, each offering a particular tradeoff between error performance and decoding complexity. The algorithms discussed in this paper, which we call majority-based algorithms, also belong to the category of message-passing algorithms. In a message-passing decoder, messages are generated at the nodes of the graph according to certain message maps and based on extrinsic incoming messages, and possibly the channel message. The channel, variable-node, and check-node message maps are denoted by , and , respectively, where , and represent the channel alphabet, the message alphabet, and the iteration number, respectively. Majority-based algorithms work with hard information (bits) (i.e., the message alphabet is binary), and thus have a very low complexity (per iteration). Gallager used these algorithms as the components of his algorithm B [1, p. 50], and showed that by properly switching between them, a better threshold, compared with algorithm A [1, p. 47], is achievable. To the best of our knowledge, however, he did not study the behavior of the stand-alone version of these algorithms in detail. Subsequently, other researchers also mainly focused on as a benchmark for hard-decision iterative algorithms (see, e.g., [8]–[10]). In this paper, by a detailed study of majority-based algorithms,
0090-6778/04$20.00 © 2004 IEEE
2088
we show the great advantages that some of these algorithms . In particular, are capable of offering over the well-known we demonstrate that the error probability for these algorithms converges to zero very fast; in many cases, much faster than it does for . This makes such algorithms good candidates for very-high-speed applications, such as optical communication systems, where the processing delay has to be minimized. Moreover, our study shows that some of the majority-based algorithms appear to outperform by a large margin, in the sense that they offer much better (noise) threshold values. The faster convergence and the better performance offered by majority-based algorithms, other than , suggests a serious role for these algorithms in the future coding schemes where the system constraints require the use of binary (hard-decision) decoders. The outline of the paper is as follows. In Section II, we give a formal definition of majority-based algorithms, and the recursive equation for average error probability of these algorithms over ensembles of regular LDPC codes. (This section does not contain any new results, and is just given to make the paper self-contained and to set out the stage for the following discussions.) We will then present the analysis based on density evolution [7] in Section III. This includes discussions on the fixed points of the majority-based algorithms and on the stability of the fixed point at zero (corresponding to average error probability of zero). We also investigate the speed of convergence and convergence properties of these algorithms in this section, and provide characterizations and tight upper bounds for the threshold. Section IV contains numerical results on threshold values of majority-based algorithms, and discussions on the convergence speed of these algorithms. Some concluding remarks are given in Section V. II. MAJORITY-BASED ALGORITHMS AND DENSITY EVOLUTION In the following, after a brief explanation of the terms, notations, and assumptions,1 we formally define majority-based algorithms and provide an analysis based on density evolution. A. Ensembles of LDPC Codes and Graphs -regular LDPC code of length , described Consider a by a parity-check matrix , which has columns and rows, of weights and , respectively. The ensemble of -regular LDPC codes of length is considered as the set of all parity-check matrices with columns and rows, fulfilling the above column and row weight constraints. We also assume that parity-check matrices from this set are selected equally likely. For a given , we can construct a bipartite graph with variable nodes and check nodes [6]. Each variable node corresponds to one column of , and each check node corresponds to one row of . An edge in the graph connects variable node and check node if and only if a 1 is located at position of . Such a graph has edges, edges incident to each variable node and edges incident to each check node. Conversely, we can construct a parity-check matrix, and therefore, a -regular LDPC code of length 1For
definitions and notations, we mainly follow [7], to which the reader can refer for a more detailed treatment.
IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 12, DECEMBER 2004
, based on a bipartite graph with variable nodes, each of degree , and check nodes, each of degree , if the graph does not contain parallel edges. The set of all such graphs, when equipped with a uniform probability distribution, is called the ensemble of -regular graphs. Note that the ensembles -regular LDPC codes and -regular graphs are of essentially equivalent and can be used interchangeably. B. Symmetry Conditions In [7], it is shown that under certain symmetry conditions on the channel, and on the variable-node and the check-node message mappings, the conditional (bit or block) probability of error, given that a specific codeword has been transmitted, is independent of the codeword. The channel of interest in this paper is the memoryless binary symmetric channel (BSC) which satisfies the channel-symmetry condition of [7]. Moreover, all majority-based algorithms, defined in the following subsection, satisfy the symmetry conditions required for message mappings. As such, in our study, the bit- or the block-error performance does not depend on the transmitted codeword. C. Majority-Based Algorithms: Formal Definition Consider the ensemble of -regular graphs and let the channel be binary symmetric with output alphabet . For a nonnegative integer , the majority-based algorithm of order , denoted by , is defined by message maps null m m m
m m
m m m m m if m m otherwise
m
m
where all messages, including the channel message m , have an alphabet space of . In words, a variable node passes its channel message m to a neighboring check node , unless and at least of its extrinsic incoming messages disagree with m , in which case m is sent. A check node always passes the product of its extrinsic incoming messages. The variable-node operation of majority-based algorithms is based on the well-known majority-decision rule, hence, the name majority-based. Note that the value of determines the strength of the disagreement required for the change. Extreme cases are standard majority-decoding algorithm , and . (For the latter, a unanimous vote is required for the change.) Majority-based algorithms are especially attractive for their remarkably simple implementation (per iteration). D. Majority-Based Algorithms: Density Evolution We assume that majority-based algorithms are used to decode the ensemble of -regular LDPC codes over a BSC with crossover probability . We use density evolution to
ZARRINKHAT AND BANIHASHEMI: THRESHOLD VALUES AND CONVERGENCE PROPERTIES OF MAJORITY-BASED ALGORITHMS
2089
track the evolution of the average fraction of erroneous messages, i.e., messages whose signs do not agree with the signs of their corresponding transmitted bits. In order to apply density evolution, we assume a tree-like decoding neighborhood [7] to guarantee statistical independence among messages throughout the iterative process. Since the symmetry conditions are fulfilled for this channel and for these algorithms, the decoding performance is independent of the transmitted codeword. Therefore, we assume that the all-one codeword is transmitted to simplify the analysis. be the average probability that the Let message traversing an edge of the ensemble of -regular graphs in iteration , from a variable node to a check node, is and . It can be shown equal to . Clearly, that under , the recursion
where
(1)
Fig. 1. Graphical interpretation of density evolution for the ensemble of (3, 6) regular LDPC codes, decoded by MB , for " = 0:0350 and 0:0450.
(2)
(3) describes the evolution of the average probability of error with [1, p. 50]. The iteration number , if initiated by first term in (1) represents the cases where the channel message is in error and less than of extrinsic messages are correct. The second term, however, represents the cases where the channel message is correct, but at least of extrinsic messages are in error. For a given ensemble of -regular codes and a given order , the threshold , or simply , is defined as the supremum of all values of in the interval [0, 0.5], such that . We will see in the next section that is an increasing function of . It then follows that for all values . The threshold, in general, can be computed numerically. However, as we will show later, in some cases, the threshold can be determined analytically. (Similar results for are also presented in [8] and [9].) Note that although the density-evolution analysis is based on the assumption of random cycle-free graphs with given degrees for variable nodes and check nodes, concentration results of [7] will guarantee that for sufficiently large block lengths, almost every randomly generated code has a bit-error rate (BER) waterfall region very close to the threshold. III. DYNAMICS OF MAJORITY-BASED ALGORITHMS In this section, we analyze the dynamics of majority-based algorithms via density evolution. We use a graphical approach to visualize the behavior of iterative algorithms. This approach is particularly helpful in discussing the threshold values and speed
of convergence. The graphical approach, introduced originally by Gallager [1, p. 48], has been used in similar contexts by others, including ten Brink [11] and Divsalar et al. [12]. One of the main results of this paper is Theorem 3, which provides a tight upper bound on the threshold of majority-based algorithms. It should be noted that the approach taken in this paper to derive the bound is closely related to the approach of [9] in . deriving a similar bound for A. Fixed Points, Threshold Characterizations, and Bounds To investigate the evolution of (1) throughout the iteration process and study the behavior of majority-based algorithms, and defined as (2) and with polynomial functions (3), respectively, we define the following function:2
We are interested in the properties of for , as , and therefore the sequences and are identical. We use this relationship to give a graphical interpretation for density evolution. For a given order , and for a given in the channel parameter , consider the curve plane. We start from the point on the curve. From this starting point, we move horizontally toward the line to reach the point on the line. We then to reach the point move vertically toward the curve on the curve. By repeating these horizontal and vertical steps consecutively, we obtain all the values in the . Fig. 1 illustrates this graphical sequence interpretation for the ensemble of (3, 6) regular LDPC codes, , and for two channel parameters decoded by 2Clearly, the functions h (x); f (x), and g (x) also depend on node degrees d and d . In most cases, however, we deal with these dependencies implicitly, as our focus is mainly on the dependency of these functions on parameter ! (and in the case of h (x), on parameter ", as well).
2090
IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 12, DECEMBER 2004
Fig. 2. Ensemble of (4, 6) regular LDPC codes, decoded by MB , for " = 0:0800 and 0:1000. (a) Example of a decreasing trend of average error probability that does not converge to zero. (b) Example of an increasing trend.
and . As can be seen, depending on the channel paramor an increasing eter, we can have a decreasing trend of average error probability with iterations. , the algorithm is stuck For a starting point on the line right from the beginning, and there will be no decreasing or increasing trend. In the following, we discuss some of the properties of that are useful in understanding the dynamics of majority-based algorithms. and any given Lemma 1: For any given order is an increasing function of the channel for all .3 parameter , i.e., Proof is provided in Appendix I. and any given channel Lemma 2: For any given order is increasing with respect to parameter , i.e., for all . Proof is provided in Appendix II. The next proposition follows from Lemmas 1 and 2 by finite induction. Proposition 1: For any given order is an increasing function of . In other words, majority-based algorithms respect physical degradation ordering of BSCs. Also, the next two propositions follow from Lemma 2. and for a given channel Proposition 2: For a given , the sequence , or equivalently, the parameter sequence , is strictly decreasing, constant, on the or strictly increasing if the starting point is below, on, or above the line , respeccurve tively. It is noteworthy that having the starting point below the line , as the next proposition implies, is no guarantee for the decreasing average probability of error to converge to zero. Proposition 3 (Convergence to Fixed Points): For a given and a given channel parameter , the sequence 3At the end points of a closed interval, derivative is defined as right-hand or left-hand derivative, accordingly.
converges to a crossing point (a point at which the curve and the line intersect). A crossing point is, in fact, a fixed point of the recursion given in (1). Fig. 2(a) depicts a decreasing trend that does not converge to zero and is trapped at a crossing point, while Fig. 2(b) demonstrates an increasing trend, which also converges to a crossing point. The following theorem describes the condition for the convergence of average error probability to zero. and a given channel paTheorem 1: For a given order is a decreasing sequence which conrameter is below the verges to zero if and only if the curve line . is below Proof: For sufficiency, if the curve for all , then combining Propositions 2 the line and 3, one can see that the probability of error decreases and will , which converge to the only crossing point in the interval . For necessity, we show that if the curve is is not below the line for all , then the sedoes not converge to zero. Suppose that the curve quence is not below the line for all . the curve is either Then for some . If is amongst such , then acabove or on the line cording to Proposition 2, the trend is not decreasing. Otherwise, is below the line , and the starting point is continuous, there must exist at since the curve . In this case, acleast one crossing point in the interval cording to Propositions 2 and 3, the rightmost of such crossing points entraps the decreasing average probability of error and prevents it from converging to zero. Combining Proposition 1 with the definition of threshold , one can see that for all values . In fact, to provide a slightly different perspective, one can comand bine Theorem 1 with Lemma 1, to see if for a given , the sequence is for a given channel parameter decreasing and converges to zero, then for every channel param-
ZARRINKHAT AND BANIHASHEMI: THRESHOLD VALUES AND CONVERGENCE PROPERTIES OF MAJORITY-BASED ALGORITHMS
2091
eter , the sequence is decreasing and converges to zero. One can then provide an alternative definition for the of an as the supremum of all , threshold is below the line for every such that the curve . A more precise statement is given in the following. Proposition 4 (Characterization of Threshold): For any with threshold , we have
It is noteworthy that the above characterization closely resemon BSC bles the alternative description of the threshold for given in [9], and that of binary erasure channel (BEC) and its associated iterative decoding algorithm presented in [13]. is always a fixed point of (1). The As following theorem describes the stability of this important fixed point. Located Theorem 2 (Stability of the Fixed Point of -regular LDPC at Zero): Consider the ensemble of , and for codes. For any , there exists a constant any channel parameter , such that for all , i.e., the fixed point is stable.4 For , if , or provided that , then there exists a constant , such that for all , is stable. On the other hand, if i.e., the fixed point , or and , , such that then there exists a constant , i.e., the fixed point is for unstable.5 , and based on (II.1) in Proof: We know that Appendix II, we have
if otherwise. For sufficiently small values of , the convergence behavior is determined by the linear term in the Taylor series around zero. with , we thus have For any for sufficiently small . For , this is only , or the case if and . The following corollary, which is the same as the upper bound given in [9] for the special case of regular on the threshold of codes, follows from Theorems 1 and 2. , the threshold satisfies Corollary 1: For
It should be clear to the reader by now that the starting point has a major role in the convergence properties of majority-based algorithms. In the following, we provide a concise discussion on this role. Consider the function
4The stability condition can equivalently be expressed as the existence of a positive constant , such that if for some `; p < , then p converges to zero as ` tends to infinity. 5Similar results for belief propagation can be found in [13]–[16].
Fig. 3. Starting-point loci for two ensembles of regular LDPC codes with (d ; d ) = (5; 10) and (3; 4), decoded by MB and MB , respectively.
in [0, 0.5], where the polynomial functions and are given by (2) and (3), respectively. We notice that . The curve is, in fact, the locus of the for different channel parameters . We starting points of and the line call any crossing point of the curve a steady point. Obviously, the sequence remains constant if the starting point is a steady point. The points (0, 0) and (0.5, 0.5) are two steady points of all majority-based -regular LDPC codes. algorithms for any ensemble of As the two coordinates for a steady point are equal, in the rest of the paper, we identify a steady point with a single value. Fig. 3 depicts the loci of starting points for the ensembles of regular and , decoded by LDPC codes with and , respectively. The figure shows that in general, in addition to the points 0 and 0.5, there may exist other steady points. The smallest positive steady point is denoted by , and as we will see later, is an upper bound on the threshold. and the We continue this section with a few results on steady points of majority-based algorithms. We start with the following corollary, which follows from Lemmas 1 and 2, and , and thus the fact that
Corollary 2: For any given order is increasing . with , i.e., We note that based on the discussions in Appendixes I and is a strictly increasing function of in the interval II, . The only exception is when (0, 0.5], i.e., is an even number, and . In this case, the derivative is zero. Combining (I.1) and (II.1), it is easy to see that and . The also crosses the line at curve
2092
IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 12, DECEMBER 2004
and . Putting these together, we conclude that the must cross the line (continuous) curve at least once more in the interval (0, 0.5), and therefore, in and , there exists at least one other addition to . steady point. Thus, , the starting point is below the line . For all The next theorem then follows from Proposition 4. Theorem 3: The smallest positive steady point is an upper . bound for the threshold , i.e., As we will see later, the upper bound of Theorem 3 is achieved for many majority-based algorithms applied to a wide range of . This proregular LDPC code ensembles, i.e., we have vides an alternative characterization for threshold and an easier method to compute it. When the bound is not achieved, i.e., , then for , the trend of the sequence when is decreasing, but . [Refer to Fig. 2(a).] It is also worth mentioning that one can combine Corollary 1 and Theorem 3 to conclude the following. satisfies Corollary 3 [9]: For Fig. 4. Comparison of convergence speed of different orders of majority-based decoding algorithm for the ensemble of (8, 10) regular LDPC codes and for " = 1=63, which is the threshold for MB .
B. Speed of Convergence It is easy to see that for a given channel parameter , the at any iteration depends on how convergence speed of an is from the line at that iteration. far the curve The farther the curve, the faster the trend. This explains the slow in decoding the ensemble of -regular convergence of and close LDPC codes over a channel with a parameter . In this case, there is only a narrow to . tunnel between the curve and the line in the vicinity of (Refer to Table II.) The following theorem partially orders majority-based algorithms in terms of their convergence speed. , channel Theorem 4: Suppose that for a given order . Then parameter , and , we have for all and for all . To prove this theorem, we first state the following lemma. Lemma 3: Given order , channel parameter , and , we have the inequality
faster at the later stages of decoding. Fig. 4 illustrates Theorem and the 4. In this figure, we have depicted the line , for , curves (which is and , and , i.e., ). As can be seen, for the threshold for , and in accordance instance, with Theorem 4, for all . In fact, in the intervals ,
and
,
the
partial ordering of the curves are
(4) . if and only if Proof is provided in Appendix III. , acProof of Theorem 4: Since cording to Lemma 3, the order , the channel parameter , satisfy (4). This inequality remains satisand the parameter is increased or is decreased. Hence, according to fied if for all and Lemma 3 again, . It is noteworthy that for any given order and for any given channel parameter , (4) is satisfied for sufficiently small . In other words, between two majority-based algorithms used to decode the same ensemble of regular LDPC codes over the same channel, the one with smaller order eventually converges
respectively. Notice that the ordering of the curves in the vicinity is , while it is exactly the other way of . It is also interesting to note around in the vicinity of that over a channel with this parameter, the average error probis achieved after 3, 2, 4, and 28 567 iterations ability of , and , respectively. This enormous diffor orders ference is expected, since unlike the other curves, the tunnel beand the line is very narrow tween the curve . (Table II in Section IV presents similar in the vicinity of comparisons for more codes.) It is worth mentioning that in a hybrid decoding scenario [17]–[21], where, for example, switching among different
ZARRINKHAT AND BANIHASHEMI: THRESHOLD VALUES AND CONVERGENCE PROPERTIES OF MAJORITY-BASED ALGORITHMS
2093
the initial condition , it can then be seen that for sufficiently small values of , we have
(5) We can rewrite the right-hand side of (5) as
Fig. 5. G as an example of hybrid decoding. The ensemble of (5, 10) regular LDPC codes is used over a BSC channel with parameter " = 0:0400. Switching occurs from MB to MB at iteration 7.
majority-based algorithms is allowed throughout the iteration process, curves such as those presented in Fig. 4 can be used to decide which algorithm should be chosen at each iteration for the fastest convergence. The proper choice is the that has the smallest value of , or in other words, the algorithm whose associated curve is the furthest from the [18]. In fact, this is exactly what does by finding line is replaced the smallest value of that satisfies (4) when by [1, p. 50]. Fig. 5 illustrates an example of . The ensemble of (5, 10) regular LDPC codes is used over a BSC . It is easy to see that the channel with parameter . One should then switch fastest algorithm to start with is at iteration 7. Notice that using this simple hybrid to scheme has resulted in the convergence of the decoder over a channel which is considerably worse than the worst channel that either one of the two algorithms can handle. (The threshold and are 0.0248 and 0.0277, respectively.) values for More results on hybrid schemes can be found in [20] and [21]. We finish this section with the following theorem which discusses the convergence speed of majority-based algorithms in the vicinity of the fixed point located at zero. , the probability of error goes to Theorem 5: For , and zero exponentially with the number of iterations for super-exponentially for other majority-based algorithms, provided that is large enough.6 Proof: Considering only the most dominant term in the Taylor series of (1), it is easily verifiable that for suffican be approximated by ciently small values of
for , and for . It is now , the probability of error goes to zero clear that for exponentially with the number of iterations, while for the other majority-based algorithms, the convergence to zero is super-exponential.7 This is, of course, with the condition that the channel parameter is less than the threshold, and is also small enough to satisfy
. One can now see that
by the above conditions on can be relaxed to only noticing the fact that if , which implies on that if is large enough, we will be in the vicinity of the curve, and thus the probability of error goes to zero with the same exponent as in (5). It should be noted that for lower order majority-based algorithms, the asymptotic convergence of error probability to zero is faster as the parameter in the exponent of (5) is larger. IV. SIMULATION RESULTS AND SPECIAL CASES
. (Note that , recursively with
In this section, we compute the threshold values of majority-regular LDPC based algorithms for some ensembles of , and thus, codes. We also show that in many cases the threshold can be analytically computed. Moreover, we provide simulation results to compare majority-based algorithms in terms of their speed of convergence. As an example for analytical computation of the threshold, consider the ensemble of (4, 5) regular LDPC codes decoded . For this case, as we will prove in Theorem 6, . by in the The threshold is thus the smallest real root of , interval (0, 0.5). Applying the change of variable as . Computing is we rewrite of this new then equivalent to finding the largest real root . We polynomial in the interval (0, 1), i.e., , and thus obtain . and for some pairs and for Table I contains different orders of majority-based algorithms. It also lists the for . As can be seen, for many values pairs, is not the best choice in terms of the threshold value. As a matter of fact, in some cases, this algorithm has the smallest threshold value amongst all the orders of majority-based decoding algorithm (refer to the last
6A preliminary version of this result was brought to our attention by one of the reviewers.
7Similar results for the special case of ! = 0 are given in [17]. Our results generalize those of [17] to the other orders of majority-based algorithms.
, where and
, otherwise, and .) Using
if
or
is odd,
2094
IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 12, DECEMBER 2004
TABLE I THRESHOLD AND SMALLEST STEADY-POINT VALUES FOR BSC AND DIFFERENT ORDERS OF MAJORITY-BASED DECODING ALGORITHMS
six rows of Table I). Moreover, as we will see later, other orders . also appear to be, in many cases, considerably faster than Table I also shows that for a large percentage of ensembles, the bounds on the threshold value (given in Theorem 3 and , for Corollary 3) are tight, i.e., we usually have , and , . It should be noted that although for for many cases the bounds are tight, there are also examples where they cannot be achieved. (Refer to the underlined entries in the last three rows of Table I. For these cases, the curve touches the line somewhere in the interval .) The following theorems cover some of the cases for which the bounds are achieved. -regular Theorem 6: Consider the ensemble of LDPC codes, and suppose that this ensemble is decoded by . If the variable-node degree is an even number, then . Proof is provided in Appendix IV. For the ensembles of regular LDPC codes with odd variablenode degrees, we could neither prove a similar general result, nor find any counterexample. In the next theorem, however, we for the special case of . prove that -regular LDPC Theorem 7: For the ensemble of . codes, decoded by Proof is provided in Appendix V. , i.e., In Table I, we observe that for is used, for all pairs with is equal when and is less than . It should be noted that to this is not the case in general. A counterexample is the ensemble
. In this case, of (4, 11) regular LDPC codes, decoded by . To compare the convergence speed of majority-based algoensemble, rithms, we proceed as follows. For a given we compute the smallest number of iterations that each order of the algorithm needs to achieve an average fraction of incorrect when the channel parameter messages equal to or less than is 90% of the smallest value amongst the thresholds of different orders. (We choose 90% arbitrarily, with the consideration that it results in both a relatively large channel parameter and a relatively fast convergence, even for the algorithm with the smallest threshold.) The results are listed in Table II. It can be seen that enjoy a very fast convergence, in those cases in which general. The convergence speed of some of these cases is more . On the than 20 times faster than the cases for which usually has a slow convergence. This contrary, we see that for touches the line happens when the curve at zero, and the channel parameter is also close to . and is very In this case, the tunnel between , resulting in the slow decrease narrow in the vicinity of of error probability with iterations.8 Fig. 6 shows BER curves for a randomly constructed regular LDPC code of length 100 000, with no cycle , of length four in the Tanner graph, decoded by . For all simulation results, the maximum and 8In Table II, it is observed that for cases with d = 4; MB , equivalent to G algorithm, is faster than MB . The reason is that in these cases, the threshold for MB , and thus the selected channel parameter, is much smaller than the threshold of G . This means that for G , the tunnel between y = h (x) and y = x is wide open in the whole interval (0; "), including the vicinity of 0.
ZARRINKHAT AND BANIHASHEMI: THRESHOLD VALUES AND CONVERGENCE PROPERTIES OF MAJORITY-BASED ALGORITHMS
TABLE II NUMBER OF ITERATIONS REQUIRED TO ACHIEVE AN AVERAGE FRACTION OF ERRONEOUS MESSAGES EQUAL TO OR LESS THAN 10 . CHANNEL PARAMETER IS 90% OF THE SMALLEST THRESHOLD VALUE AMONGST DIFFERENT ORDERS OF MAJORITY-BASED ALGORITHMS
2095
dB), and dB), respectively. Fig. 6 shows that the waterfall regions of BER curves associated with lower order majority-based algorithms are very sharp, and occur at close proximity to the theoretical threshold however, the waterfall region is for these algorithms. For and not nearly as sharp. At a BER of about outperform by more than 2.7 and 0.5 dB, respectively. This is even worse than is while the theoretical threshold for by more than 0.6 dB. This, which suggests that for that of , the convergence of the ensemble performance to its average by increasing the block length, predicted by concentration results of [7], happens much slower, compared with the other two algorithms, can be loosely explained as follows. Amongst curve, majority-based algorithms, due to the shape of its has a relatively slow convergence speed and needs more iterations to achieve a certain (low) error probability. For a low error probability, therefore, the (cycle-free) supporting tree has to be deeper. This exacerbates [7] in decoding under the performance deviation from the asymptotic limit under finite block lengths, where the graph inevitably has cycles with relatively short lengths. It is worth mentioning that in some applications, having a gradual slope in the waterfall region can be considered an advantage. This is particularly the case for could change over a rather wireless applications, where wide range. , the average number of iterations for conAt BER , and are about 5.60, 9.21, and verged cases for 13.29, respectively. These numbers are increased to about 6.19, . These 10.79, and 37.95, respectively, for a BER of about compared with the results show the slower convergence of other two algorithms. (
dB),10 0.0453 (
V. CONCLUDING REMARKS
number of iterations is set to 500, and for each point, enough codewords are simulated to generate at least ten codeword errors.9 The threshold values for these algorithms are 0.0155
Majority-based algorithms, which include Gallager’s algoand standard majority decoding algorithm as sperithm A cial cases, are of interest mainly due to their low complexity (per iteration). By performing a detailed study of these algorithms through density evolution, we show that many of majority-based and algorithms have larger noise thresholds, compared with the standard majority decoding algorithm. We present tight analytical upper bounds on these thresholds and prove that in many cases, these bounds are achieved. We also highlight another attractive feature of majority-based algorithms, i.e., their generally fast convergence. We show that a majority-based algorithm’s convergence speed is determined by its order, and exsubstanplain why many majority-based algorithms outpace tially. In particular, we prove that while the asymptotic conis exponential with iteration number, for vergence speed of the other majority-based algorithms, it is super-exponential. As a result of this work, we expect that more attention would be paid to majority-based algorithms other than the well-known . and widely popular The fast convergence of majority-based algorithms, along with their relatively large noise thresholds, make them good candidates for hybrid algorithms in which different decoding
9For many simulation points at higher BERs, at least 200 codeword errors are generated. Reducing this to ten for just a few points at the bottom of the curves was due to the limitation in computational resources.
10This is the equivalent E =N assuming that the BSC is a model for an additive white Gaussian noise channel with binary phase-shift keying and a 1-bit quantizer.
Fig. 6. BER curves for a randomly constructed (6, 10) regular LDPC code of length 100 000, decoded by different orders of majority-based algorithms. (Note that MB is the same as G .)
2096
IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 52, NO. 12, DECEMBER 2004
algorithms are combined to achieve better performance/complexity tradeoffs, compared with those achievable by individual algorithms [17]–[21]. An example is Gallager’s algorithm , where majority-based algorithms are combined by B switching between them through iterations. Another possibility is to combine these algorithms in each iteration and in a time-invariant fashion [20], [21]. This is shown to provide more robustness to the changes of channel conditions, compared with [20], [21].
APPENDIX II Proof of Lemma 2: It can be shown that with defined as (I.2) and (I.3), respectively, we have
and
APPENDIX I Proof of Lemma 1: It can be seen that
(II.1) (I.1)
which is only a function of nonnegative, we first define
and
. To show that (I.1) is
(I.2) (I.3) Clearly,
which can be seen to be nonnegative in the interval of interest, (0, 0.5). It should be noted that the derivative for all . It is equal to zero at for any value of and , and also at for any value of and any other than Gallager’s algorithm A . For is only zero for Gallager’s algorithm A, the derivative at .
, and
APPENDIX III
. Thus Proof of Lemma 3:
(I.4) The above summation can be decomposed to three terms as
APPENDIX IV Proof of Theorem 6: Using (2) and (3), it is easy to see that for even (IV.1) (I.5) and . Based on must be equal to
Therefore, in which the second term can be zero. (This only happens when or when the upper limit of the second summation is strictly smaller than its lower limit.) The first and the third terms and , rein (I.5), according to (2) and (3), are spectively, and the second term is nonnegative. As such, we have , or equivalently
It can be seen that is a strictly increasing function , i.e., , if the second term in of and the upper (I.5) is not equal to zero, or equivalently, if limit of this summation is not smaller than its lower limit, i.e., is an odd number or . either
the definitions of and , this means that . It is worth noting that for even values, uses the channel messages only once at the beginning. When the initialization is completed, these messages play no further role, and the variable nodes only pass the majority decision of incoming extrinsic messages. This independence of the decoding process ( curves) from , is, in fact, the underlying cause of the equality and . of APPENDIX V Proof of Theorem 7: It is easily verifiable that for , and , we have . In the following, we prove the . Assuming that and by using equality for
ZARRINKHAT AND BANIHASHEMI: THRESHOLD VALUES AND CONVERGENCE PROPERTIES OF MAJORITY-BASED ALGORITHMS
Descartes’ rule of signs,11 one can show that for any , has at most three real zeros in [0, the polynomial , then, according to Proposition 4, 0.5].12 If touches the line in at least once, and thus, will have at least two zeros in . Moreover, for crosses the line at two different points in . It can also be shown that for all is increasing with . Being increasing with , and , having , and having a value less than a derivative of zero at at , the curve , for , has no choice but at two different points in , in adto cross the line dition to the touch-induced zeros of . Therefore, the number of zeros will add up to at least four, which contradicts Descartes’ rule of signs. The only way to avoid this contradic. tion is to have ACKNOWLEDGMENT The authors would like to thank the anonymous reviewers for their helpful comments and suggestions, and Mr. H. Xiao for preparing the simulation results presented in Fig. 6. REFERENCES [1] R. G. Gallager, Low-Density Parity-Check Codes. Cambridge, MA: MIT Press, 1963. [2] C. Berrou, A. Glavieux, and P. Thitimajshima, “Near Shannon limit error-correcting coding and decoding: Turbo codes (I),” in Proc. IEEE Int. Conf. Communications, 1993, pp. 1064–1070. [3] IEEE Trans. Inform. Theory Special Issue, Codes on Graphs, Iterative Algorithms, vol. 47, pp. 493–853, Feb. 2001. [4] P. H. Siegel, D. Divsalar, E. Eleftheriou, J. Hagenauer, D. Rowitch, and W. H. Tranter, “The turbo principle: From theory to practice,” IEEE J. Select. Areas Commun., vol. 19, pp. 793–799, May 2001. [5] P. H. Siegel, D. Divsalar, E. Eleftheriou, J. Hagenauer, and D. Rowitch, “The turbo principle: From theory to practice II,” IEEE J. Select. Areas Commun., vol. 19, pp. 1657–1661, Sept. 2001. [6] R. M. Tanner, “A recursive approach to low complexity codes,” IEEE Trans. Inform. Theory, vol. IT-27, pp. 533–547, Sept. 1981. [7] T. J. Richardson and R. L. Urbanke, “The capacity of low-density parity-check codes under message-passing decoding,” IEEE Trans. Inform. Theory, vol. 47, pp. 599–618, Feb. 2001. [8] L. Bazzi, T. Richardson, and R. Urbanke, “Exact thresholds and optimal codes for the binary symmetric channel and Gallager’s decoding algorithm A,” in Proc. Int. Symp. Information Theory, 2000, p. 203. [9] L. Bazzi, T. Richardson, and R. Urbanke, “Exact thresholds and optimal codes for the binary symmetric channel and Gallager’s decoding algorithm A,” IEEE Trans. Inform. Theory, vol. 50, pp. 2010–2021, Sept. 2004. [10] N. Miladinovic and M. Fossorier, “Improved bit flipping decoding of low-density parity check codes,” in Proc. Int. Symp. Information Theory, 2002, p. 229. [11] S. ten Brink, “Convergence behavior of iteratively decoded parallel concatenated codes,” IEEE Trans. Commun., vol. 49, pp. 1727–1737, Oct. 2001. [12] D. Divsalar, S. Dolinar, and F. Pollara, “Low complexity turbo-like codes,” in Proc. Int. Symp Turbo Codes, 2000, pp. 73–80. 11Descartes’ rule of signs states that if (x) = c x with 0 m < m < 1 1 1 < m and with all coefficients c 6= 0, then the number of positive real zeros of is upper bounded by the number of sign changes between consecutive coefficients c [22]. 12By defining a new variable X = 1 0 2x; (x) = h (x) 0 x can be + (1 0 2")X . rewritten as (X ) = 0(1 0 2") + 2X 0 2X Applying Descartes’ rule to (X ), it is easy to see that (X ) has at most three positive real zeros for " < 0:5, and at most one positive real zero for " = 0:5. This is equivalent to saying that (x) has at most three real zeros in (01; 0:5), and therefore, in [0; 0:5). As X = 0( x = 0:5) is a simple zero of (X ) for " = 0:5, and is not a zero of (X ) otherwise, the interval [0; 0:5) can be extended to [0; 0:5].
2097
[13] T. J. Richardson, M. A. Shokrollahi, and R. Urbanke, “Design of capacity-approaching irregular low-density parity-check codes,” IEEE Trans. Inform. Theory, vol. 47, pp. 619–637, Feb. 2001. [14] D. Burshtein and G. Miller, “Bounds on the performance of belief propagation decoding,” IEEE Trans. Inform. Theory, vol. 48, pp. 112–122, Jan. 2002. [15] A. Khandekar and R. McEliece. A lower bound on the iterative decoding threshold of irregular LDPC code ensembles. presented at Proc. CISS. [CD-ROM] [16] C.-C. Wang, S. R. Kulkarni, and H. V. Poor, “Density evolution for asymmetric memoryless channels,” in Proc. Int. Symp Turbo Codes, 2003, pp. 121–124. [17] K. Sh. Zigangirov and M. Lentmaier, “On the asymptotic iterative decoding performance of low-density parity-check codes,” in Proc. Int. Symp Turbo Codes, 2000, pp. 39–42. [18] M. Ardakani and F. R. Kschischang, “Gear shift decoding,” in Proc. 21st Biennial Symp. Communications, Kingston, ON, Canada, June 2002, pp. 86–90. [19] A. H. Banihashemi, A. Nouh, and P. Zarrinkhat, “Hybrid (multistage) decoding of low-density parity-check (LDPC) codes,” in Proc. 40th Allerton Conf. Communication, Control, Computing, Allerton, IL, Oct. 2002, pp. 1437–1438. [20] P. Zarrinkhat and A. H. Banihashemi, “Hybrid hard-decision iterative decoding of regular low-density parity-check codes,” IEEE Commun. Lett., vol. 8, pp. 250–252, Apr. 2004. [21] P. Zarrinkhat, A. H. Banihashemi, and H. Xiao, “Time-invariant and switch-type hybrid iterative decoding of low-density parity-check codes,” Ann. Télécommun., to be published. [22] P. Henrici, Applied and Computational Complex Analysis. New York: Wiley, 1974, vol. 1.
Pirouz Zarrinkhat (S’01) was born in Esfahan, Iran, in 1971. He received the B.Sc. degree from Isfahan University of Technology, Esfahan, Iran, in 1994, and the M.Sc. degree from Sharif University of Technology, Tehran, Iran, in 1997, both in electrical engineering. He is currently working toward the Ph.D. degree in electrical engineering with the Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada. His research interests include information theory, wireless communications, and quantum computing.
Amir H. Banihashemi (S’90–A’98–M’03) was born in Isfahan, Iran. He received the B.A.Sc. degree in electrical engineering from Isfahan University of Technology (IUT), Isfahan, Iran, in 1988, and the M.A.Sc. degree in communication engineering from Tehran Univesity, Tehran, Iran, in 1991, with the highest academic rank in both classes. He received the Ph.D. degree from the Department of Electrical and Computer Engineering, University of Waterloo, Waterloo, ON, Canada in 1997. From 1991 to 1994, he was with the Electrical Engineering Research Center and the Department of Electrical and Computer Engineering, IUT. In 1997, he joined the Department of Electrical and Computer Engineering, University of Toronto, Toronto, ON, Canada, where he was a Natural Sciences and Engineering Research Council of Canada (NSERC) Postdoctoral Fellow. He joined the Faculty of Engineering at Carleton University, Ottawa, ON, Canada, in 1998, where he is currently an Associate Professor in the Department of Systems and Computer Engineering. His research interests are in the general area of digital and wireless communications and include coding, information theory, and theory and implementation of iterative coding schemes. Dr. Banihashemi has served as an Editor for the IEEE TRANSACTIONS ON COMMUNICATIONS since May 2003. He is also a member of the Board of Directors for the Canadian Society of Information Theory, and a member of the Advisory Committee for the Broadband Communications and Wireless Systems (BCWS) Centre at Carleton University. In 1995 and 1996, he was awarded two Ontario Graduate Scholarships for international students.
