Decoding Turbo Codes and LDPC Codes via Linear Programming

Decoding Turbo Codes and LDPC Codes via Linear Programming Jon Feldman

David Karger

[email protected]

[email protected]

MIT LCS

MIT LCS

Martin Wainwright [email protected] UC Berkeley

J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.1/22

Binary Error-Correcting Code

cements

Encoder code word

010011

110011101001

Noisy Channel

information word

decoded code word

010011

110011101001

corrupt code word

1100101010 11

decoded info

Binary Symmetric Channel (BSC): each bit flipped independently with probability (small constant).

Decoder


Turbo Codes + LDPC Codes Low-Density Parity-Check (LDPC) codes [Gal ’62] . Turbo Codes introduced [BGT ’93], unprecedented error-correcting performance. Ensuing LDPC “Renaissance” [SS ’94, MN ’95, Wib ’96, MMC ’98, Yed ’02, ... ]. Simple encoder, “belief-propagation” decoder. Theoretical understanding of good performance: - “Threshold” as [LMSS ’01, RU ’01]; - Decoder unpredictable with cycles. Finite-length analysis: combinatorial error conditions known only for the binary erasure channel [DPRTU ’02]. J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.3/22

Our contributions Decodes: binary linear codes turbo codes.

LDPC codes

“Pseudocodewords:” exact characterization of error patterns causing failure. “Fractional distance” : - LP decoding corrects up to errors. - Computable efficiently for turbo, LDPC codes.

[FK, FOCS ’02] [FKW, Allerton ’02] [FKW, CISS ’03] Poly-time decoder using LP relaxation.

Error rate bounds based on high-girth graphs. Closely related to iterative approaches, other notions of “pseudocodewords.” J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.4/22

Outline Error correcting codes. Using LP relaxation for decoding. Details of LP relaxation for binary linear codes. Pseudocodewords. Fractional Distance. Girth-based bounds.


if

,

.

takes on arbitrary “soft values.”

Other channels:

if

BSC:

.

: negative log-likelihood ratio of

Cost function

.

Code

Maximum-Likelihood Decoding

.

Linear Programming formulation: - Variables for each code bit, - Linear Program:

Given: Corrupt code word . Find: such that is minimized.


.

: relaxation,

Polytope

No noise:

].

.

optimal.

Noise: perturbation of objective function.

noisy

110

:

[

no noise

acements101

011

Want low word error rate (WER) :=

ML certificate property: all outputs ML codewords.

Decoder: Solve LP using simplex/ellipsoid. If , output , else output “error.”

Linear Programming Relaxation

000

Design code, relaxation accordingly.


Tanner Graph

: n’hood of check

.

.

s.t.:

Code words:

“Check Nodes”

.

“Variable nodes”

The Tanner Graph of a linear code is a bipartite graph modeling the parity check matrix of the code.

acements

Codewords: 0000000, 1110000, 1011001, etc. J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.8/22

.

.

.

LP:

.

,

.

,

, ,

,

,

,

,

Vars:

IP:

for each check node

acements

Variables

= valid configurations of

,

For check bit

.

LP:

.

IP:

for each code bit

Variables

IP/LP Formulation of ML Decoding


be the relaxed polytope.

Let

, subject to:

Minimize

IP/LP Formulation of ML Decoding

IP: formulation of ML decoding. What do fractional solutions look like? J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.10/22

Suppose:

Fractional Solutions

ML codeword:

acements

.

Frac. sol cost:

.

Frac. sol:

ML codeword cost: . ——————————————


Pr[ Decoding Success ] = Pr[

is the unique OPT ].

Assume - Common asssumption for linear codes. - OK in this case due to symmetry of polytope. Pr[

LP Decoding Success Conditions

is the unique OPT ] = Pr[ All other solutionss have cost > 0].

Theorem [FKW, CISS ’03]: Assume the allzeros codeword was sent. Then, the LP decodes correctly i˜ff all non-zero points in P have positive cost. J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.12/22

Pseudocodewords are scaled points in

.

.

.

Scaled to integers:

Previous example:

Pseudocodewords

Natural combinatorial definition of pseudocodeword (independent of LP relaxation).

Theorem [FKW, CISS ’03]: LP decodes correctly i˜ff all pseudocodewords have cost . J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.13/22

Classical distance: - = min Hamming dist. of codewords in Adversarial performance bound: - ML decoding can correct

errors.

Fractional distance: = min ( ) dist. between an integral and a fractional vertex of . = min wt. fractional vertex of . - Lower bound on classical distance: . - LP Decoding can correct errors.

.

Another way to define minimum distance: = min ( ) dist. between two integral verts of .

Fractional Distance


Therefore

Since

if flipped, +1 o.w.; So,

,

.

.

When

,

errors occur.

be a vertex of

Let

errors

Suppose fewer than

LP Decoding corrects

. J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.15/22

Computing for linear/LDPC codes is NP-hard. If the polytope has small size (LDPC), the fractional distance is easily computed. - More general problem: Given an LP, find the two best vertices . - Algorithm: * Find . * Guess the facet on which sits but does not. * Set facet to equality, obtaining . * Minimize over .

Computing the Fractional Distance

Good approximation to the classical distance? Good prediction of relative classical distance? J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.16/22

For rate-1/2 RA (cycle) codes: If has large girth, neg-cost pseudocodewords (promenades) are rare.

Erdös (or [BMMS ’02]): Hamiltonian 3-regular graph with girth .

Using Girth for Error Bounds

, as

.

, girth , all var. nodes have degree

:

Can achieve . Stronger graph properties (expansion?) are needed for stronger results.

Theorem [FKW, CISS ’03]:

Arbitrary

Theorem [FK, FOCS ’02]: For any , WER long as


Other “pseudocodewords” BEC: Iterative decoding successful iff no zero-cost “stopping sets.” [DPRTU ’02] - In the BEC, pseudocodewords = stopping sets. - Iterative/LP decoding: same performance in BEC. Tail-Biting trellisses (TBT): Iterative decoding successful iff “dominant pseudocodeword” has negative cost [FKMT ’98]. - TBT: need LP along lines of [FK, FOCS ’02]. - Iterative/LP decoding: same performance on TBT. “Min-sum” decoding successful iff no neg-cost “deviation sets” in the computation tree [Wib ’96]. - Pseudocodewords are natural “closed” analog of deviation sets. J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.18/22

For “high-density” binary linear codes, need representation of without exponential dependence on check node degree. - Use “parity polytope” of Yannakakis [’91]. - Orig. representation: . - Using parity polytopes: .

Other Results

New iterative methods [FKW, Allerton ’02]: - Iterative “tree-reweighted max-product” [WJW ’02] tries to solve dual of our LP. - Subgradient method for solving LP gives provably convergent iterative algorithm. Experiments on performance, distance bounds. J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.19/22

Performance Comparison Random rate-1/2 (3,6) LDPC Code

Word Error Rate

100

Min-Sum Decoder (100 iterations) LP Decoder Both Error

10-1 -2

10

-3

10

10-4 10-5

10-0.8 10-1 10-1.2 10-1.4 10-1.6 10-1.8 10-2 10-2.2 10-2.4

BSC Crossover Probability

Length 200, left degree , right degree . J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.20/22

Growth of Average Fractional Distance Rate 1/4 Gallager Ensemble Fractional Distance

Average Fractional Distance

10 9 8 7 6 5 0

50

100

150

200 250 300 Code Length

350

400

450

500

“Gallager” distribution, left degree , right degree . J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.21/22

Future Work New WER, fractional distance bounds: - Lower rate turbo codes (rate-1/3 RA). - Other LDPC codes, including * Expander codes, irregular LDPC codes, other constructible families. - Random LDPC, linear codes? ML Decoding using IP, branch-and-bound? Using generic “lifting” procedures to tighten relaxation? Deeper connections to “sum-product” belief-propagation? LP decoding of other code families, channel models? J. Feldman, Decoding Turbo Codes and LDPC Codes via Linear Programming – p.22/22