Polymer Expansions for Cycle LDPC Codes
arXiv:1202.2778v1 [cs.IT] 13 Feb 2012
Nicolas Macris and Marc Vuffray LTHC, IC, EPFL, CH-1015 Lausanne, Switzerland
[email protected],
[email protected] Abstract—We prove that the Bethe expression for the conditional input-output entropy of cycle LDPC codes on binary symmetric channels above the MAP threshold is exact in the large block length limit. The analysis relies on methods from statistical physics. The finite size corrections to the Bethe expression are expressed through a polymer expansion which is controlled thanks to expander and counting arguments.
spins σab ∈ {−1, +1} attached to each edge. At each function node a ∈ V we attach a non-negative function fa (σ∂a ) depending only on neighboring variables σ∂a ≡ (σab )b∈∂a . We study probability distributions which can be factorized as XY 1 Y fa (~σ∂a ) , ZΓ = fa (~σ∂a ) , (1) µΓ (~σ ) = ZΓ a∈V
I. I NTRODUCTION A few years ago Cherktov and Chernyak [1] devised a loop series which represents the partition function of a general vertex model as the product of the Bethe mean field expression and a residual partition function over a system of loops. In this representation all quantities are entirely expressible in terms of Belief Propagation (BP) marginals or messages. However it has not been clear so far if this representation leads to a controlled series expansions for the log-partition, in other words the free energy. If this is the case it should hopefully allow to control the difference between the true free energy and the Bethe free energy. The loop expansion has a potential interest in coding theory since Low-Density-Parity-Check (LDPC) and Low-DensityGenerator-Matrix (LDGM) codes on general binary-input memoryless symmetric (BMS) channels fit in the framework of (generalized) vertex models. In this context free energy is just another name for conditional input-output Shannon entropy. For these models it is believed that the Bethe formula for the conditional entropy/free energy is exact. However there is no general proof, except for the cases of the binary erasure channel [2], LDGM codes for high noise, and in special situations for LDPC codes at low noise [3]. We consider cycle LDPC codes for high noise (above the MAP threshold) on the binary symmetric channel (BSC). We show that, under the assumption that there exists a fixed point for the BP equations, the average conditional entropy/free energy is given by the Bethe expression. The novelty of the approach is to turn the loop expansion into a rigorous tool allowing to derive provably convergent polymer expansions [4]. Controlling the loop expansion is a non-trivial task because in most situations of interest the number of loops proliferates. For example, this is the case (for the system of fundamental cycles) in capacity approaching codes even under MAP decoding [5]. II. L OOP
AND POLYMER REPRESENTATIONS
Let Γ = (V, E) be a graph with vertices a ∈ V of regular degree d and edges ab ∈ E. The symbol ∂a denotes the set of d neighbors of a. In vertex models the degrees of freedom are
~ σ a∈V
1 n
and their associated free energy fn ≡ ln ZΓ . For each edge ab ∈ E we introduce two directed “messages” ηa→b and ηb→a . For the moment these variables are arbitrary and are collectively denoted by ~η. One has the identity [1] 1 1 ln ZBethe (~η ) + ln Zcorr (~η ). (2) n n The first term is the Bethe free energy functional, X Y X ln ZBethe (~η ) = ln fa (σ∂a ) eηb→a σab fn =
a∈V
−
b∈∂a
~ σa
X
ln (2 cosh (ηa→b + ηb→a )) .
(3)
ab∈E
The “partition function” in the second term can be expressed as a sum over all subgraphs (not necessarily connected) g ⊂ Γ X Zcorr (~η ) = K(g) (4) g⊂Γ
and K(g) = Ka =
Q
a∈g
X
Ka with
pa (σ∂a )
Y
σab e−σab (ηa→b +ηb→a )
b∈∂a∩g
~ σa
Q fa (~σ∂a ) b∈∂a eηb→a σab Q . σ∂a ) b∈∂a eηb→a σab ~ σa fa (~
pa (σ∂a ) = P
It is well known that the stationary points of (3) satisfy the BP fixed point equations, Qb6=c P σ∂a ) b∈∂a eηb→a σab ~ σa σac fa (~ ηa→c = P . (5) Q =c η σ σ∂a ) b6b∈∂a e b→a ab ~ σa fa (~
Remarkably, for any solution of (5), K(g) = 0 if g contains a degree one node. Thus if ~η is a fixed point of the BP equations then Zcorr (~η ) is given by the sum in (4) over g ⊂ Γ with no degree one nodes. Such graphs are called loops (see figure 1). One can recognize that Zcorr can be interpreted as the partition function of a system of polymers. A loop g ⊂ Γ can be decomposed into its connected parts in a unique way as
illustrated on figure 1. Connected loops are called polymers and are generically denoted by the letter γ. The important point is that by definition the polymers do not intersect. For
uniformly in system size n (and thus ensures convergence in the infinite size limit) is +∞ X X 1 |γ|t |K(γ)| < 1 sup t! a∈V γ∋a t=0
(8)
To illustrate the use of the polymer expansion in a simple case, consider a vertex model at high temperature defined by Y 1 1 σab )e 2 hab σab fa (σ∂a ) = (1 + tanh Ja 2 b∈∂a
Fig. 1. Left: an example of a loop graph g with no dangling edge. Right: decomposition of g into its connected parts γi .
each polymer γ we define a weight (also called activity), Q K(γ) = a∈γ Ka . Let g = ∪M i=1 γi . Since the γi are disjoint, QM Q K = K(γ ). Thus equation (4) can be cast in the i a a∈g i=1 form M X 1 X Y Y I (γi ∩ γj = ∅) . K (γi ) M ! γ ,..,γ i=1 in/2 X C d 2 P |K (g) | ≥ δ ≤ e−nαd 2 h . (16) δ g⊂Γ
This inequality is a fortiori valid for g’s replaced by γ’s in the sum. Sketch of Proof: We denote by Kn the complete graph with n vertices. By Markov’s inequality, |g|>n/2 |g|>n/2 X 1 X P |K (g) | ≥ δ ≤ E[|K (g)| I (g ⊂ Γ)] δ g⊂Γ
≤
g⊂Kn
|g|>n/2 d−1 Y d 1 X (αi hd−i )ni (g) P[g ⊂ Γ] (1 − αd h2 )nd (g) δ 2 i=2 g⊂Kn
(17) Consider graphs g with ni (g), i = 2, ..., d fixed. Mackay [9] provides a bound for the probability2 P[g ⊂ Γ] of finding aP particular subgraph into a regular graph Γ. Namely for d 1 nd 2 i=2 ini (g) + 2d ≤ 2 , 2 Qd ni (g) i=2 [d]i . (18) P[g ⊂ Γ] ≤ 1 Pd 2 Pd 2 2 i=2 ini (g) nd 2 − 2d 1 ini (g) 2
i=2
The number of subgraphs of Kn with given ni (g) is n! (n − ×
( 12
Pd
i=2
Pd
i=2
Qd ni (g))! i=2 ni (g)! Pd ( i=2 ini (g))! 1
ini (g))!2 2
Pd
i=2
ini (g)
Qd
ni (g) i=2 (i!)
.
(19)
Replacing (18) in (17), using (19), setting xi = nni , and performing an asymptotic calculation for Pdn large, we show (here ~x ≡ (x2 , ..., xd ) and ∆ ≡ {~x| 21 ≤ i=2 xi ≤ 1}) Z |g|>n/2 X 1 P |K (g) | ≥ δ ≤ dd ~x gn (~x) exp n{fn (~x) δ ∆ g⊂Γ
d−1 X d + xd ln(1 − αd h2 ) + xi ln(αi hd−i )} 2 i=2
2 Here
[m]i = m(m − 1)...(m − i + 1).
(20)
The large n behavior of the integral asymptotic is controlled by fn (~x), and gn (~x) gives sub-dominant contributions that do not concern us here. We have d d d X xi 1 X 1 X xi ln r ixi ) ln ( ixi ) − fn (~x) = ( 2 i=2 2 i=2 i i=2 d X
d X
2r2 r 2r2 r ) ln( − ) xi ) − ( − xi ) ln(1 − (1 − 2 n 2 n i=2 i=2 d
d
r 2r2 2r2 1X r 1X +( − ixi − ixi − ) − ln( − ) (21) 2 2 i=2 n 2 2 i=2 n For h small enough, in the domain ∆, the exponent in (20) is
Now suppose for a moment that there exist a positive constant independent of n such that |γ|≥n/2 X 1 Zp (~η | γ) ′′ ln 1 + (23) K(γ) ≤C n Zp (~η ) γ⊂Γ
Then B ≤ C ′′ P[Iζc ]. Setting ζ = δe2nǫ , the above arguments imply |γ|≥n/2 X Zp (~η | γ) 1 E ln 1 + K(γ) ≡ A+B n Zp (~η ) γ⊂Γ
1 1 C d 2 ≤ | ln(1 − δe2nǫ ) + e−nαd 2 h + on (1). n δ ǫ d
2
d 2 We are free to choose δ = e−nαd 4 h and ǫ = αd 16 h (lemmas 4.3 4.4 hold) and this choice A + B = on (1), which proves (22). It remains to justify (23). From the convergence of the polymer expansion we deduce that n1 ln Zp (~ηn∗ ) is bounded uniformly in n. From (1), (10) we easily show that n1 ln ZΓ ≤ ln 2 + d2 h. In the high noise regime the BP messages are bounded so that from (3) we deduce that n1 ln ZBethe (~ηn∗ ) is bounded by a constant independent of n. Finally the triangle inequality implies that n1 | ln ZΓ −ln ZBethe (~ηn∗ )−ln Zp (~ηn∗ )| is bounded uniformly in n. This is precisely the statement (23).
Fig. 2. The exponent in (20) for d = 3, for h small enough, is strictly negative in the domain ∆. Its maximum at x2 = 0, x3 = 1 is O(h2 ).
strictly negative and attains its maximum at the corner point x2 = · · · = xd−1 = 0, xd = 1. At this point it is equal to ln(1 − αd d2 h2 ) which allows to conclude (16). We are now in a position to prove proposition 4.1. Proof of proposition 4.1: In view of (12), we must show that for h small enough, |γ|≥n/2 X Zp (~ η | γ) 1 (22) E ln 1 + K(γ) = on (1). n Zp (~ η) γ⊂Γ
Call Iζ the event
|γ|≥n/2
X
|K(γ)|
γ⊂Γ
Zp (~ η | γ) n/2 Z (~ η |γ {∀γ ⊂ Γ : e−2nǫ ≤ Zpp (~η) ≤ e2nǫ } and { γ⊂Γ |K(γ)| ≤ c δ} imply Iδe2nǫ . Therefore Iδe 2nǫ implies the union of the complementary events, so that applying the union bound together with lemmas 4.3 and 4.4, c P[Iδe 2nǫ ] ≤
C −nαd d h2 1 2 e + on (1). δ ǫ
V. C ONCLUSION The approach is quite general and can hopefully be generalized to standard irregular LDPC codes with bounded degrees and binary-input memoryless output-symmetric channels with bounded log-likelihood variables. This will be the subject of future work. ACKNOWLEDGMENT M.V acknowledges supported by the Swiss National Foundation for Scientific Research, Grant no 2000-121903. N.M benefited from discussions with M. Chertkov and R. Urbanke. R EFERENCES [1] M. Cherktov, V. Chernyak, Loop series for discrete statistical models on graphs, J. Stat. Mech., pp. 1-28, P06009, (2006). [2] C. Measson, A. Montanari, R. Urbanke, Asymptotic Rate versus Design Rate, ISIT (2007) pp. 1541-1545. [3] S. Kudekar, N. Macris, Decay of Correlations for Sparse Graph Error Correcting Codes, SIAM J. Discrete Math. 25, pp. 956-988 (2011). [4] D. Brydges, A short course on cluster expansions, in Ph´enomen`es critiques, syst`emes al´eatoires et th´eories de jauge, K. Osterwalder and R. Stora ed. Les Houches, session XLIII, Part I, (1984). [5] I. Sason, On universal properties of capacity-approaching LDPC code ensembles, IEEE Trans. on Information Theory, 55 pp. 2956-2990 (2009). [6] S. C. Tatikonda and M. I. Jordan, Loopy belief propagation and Gibbs measures, in Proc. 18th Annu. Conf. Uncertainty in Artificial Intelligence (UAI-02), San Francisco, pp. 493-500 (2002). [7] J. S. Yedida, W. T. Freeman, and Y. Weiss, Understanding belief propagation and its generalizations, in Exploring artificial intelligence in the new millennium, Morgan Kaufmann Publishers (2003). [8] B. Bollobas, The isoperimetric number of random regular graphs, Euro. J. Combinatorics 9 241244 (1988). [9] B.D. McKay, Subgraphs of random graphs with specified degrees, Congressus Numerantium 33, 213-223 (1981).
