1740
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 7, JULY 2006
Source-Optimized Irregular Repeat Accumulate Codes With Inherent Unequal Error Protection Capabilities and Their Application to Scalable Image Transmission Ching-Fu Lan, Zixiang Xiong, Senior Member, IEEE, and Krishna R. Narayanan
Abstract—The common practice for achieving unequal error protection (UEP) in scalable multimedia communication systems is to design rate-compatible punctured channel codes before computing the UEP rate assignments. This paper proposes a new approach to designing powerful irregular repeat accumulate (IRA) codes that are optimized for the multimedia source and to exploiting the inherent irregularity in IRA codes for UEP. Using the end-to-end distortion due to the first error bit in channel decoding as the cost function, which is readily given by the operational distortion-rate function of embedded source codes, we incorporate this cost function into the channel code design process via density evolution and obtain IRA codes that minimize the average cost function instead of the usual probability of error. Because the resulting IRA codes have inherent UEP capabilities due to irregularity, the new IRA code design effectively integrates channel code optimization and UEP rate assignments, resulting in source-optimized channel coding or joint source-channel coding. We simulate our source-optimized IRA codes for transporting SPIHT-coded images over a binary symmetric channel with crossover probability . When = 0 03 and the channel code length is long (e.g., with one codeword for the whole 512 512 image), we are able to operate at only 9.38% away from the channel capacity with code length 132380 bits, achieving the best published results in terms of average peak signal-to-noise ratio (PSNR). Compared to conventional IRA code design (that minimizes the probability of error) with the same code rate, the performance gain in average PSNR from using our proposed source-optimized IRA code design is 0.8759 dB when = 0 1 and the code length is 12800 bits. As predicted by Shannon’s separation principle, we observe that this performance gain diminishes as the code length increases. Index Terms—Irregular repeat accumulate (IRA) codes, scalable source coding, source-channel coding, source-optimized channel coding, unequal error protection (UEP).
I. INTRODUCTION
S
HANNON’S separation principle [1] states that, asymptotically, source coding and channel coding can be done separately without performance loss. However, it was shown
Manuscript received November 12, 2004; revised September 20, 2005. This work was presented in part at the 2003 IEEE Communication Theory Workshop, Mesa, AZ, April 2003, and the 37th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, November 2003. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Aria Nosratinia. The authors are with the Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843 USA (e-mail:
[email protected];
[email protected];
[email protected]). Digital Object Identifier 10.1109/TIP.2006.877074
by Massey [2] that separation does not necessarily lead to less complex solution or is always applicable. Indeed, for the separation principle to hold, one has to tolerate infinite delay and complexity. Therefore, in practical multimedia communication systems with delay and complexity constraints, joint sourcechannel coding (JSCC) [3] should be employed to provide acceptable performance. Since the early 1990s, great progresses have been made in the design of practical source code and channel codes. On one hand, several wavelet-based image/video coding algorithms [4]–[9] have been developed. These algorithms have the distinct feature of being able to generate scalable (or embedded) bitstreams while offering superb coding performance. Scalability induces a sequential dependency among the compressed multimedia source bits, calling for unequal error protection (UEP) in the form of more protection for the beginning part of the bitstream, less for the middle part, yet still less or even no protection at all for the last part. This is different from conventional data (e.g., satellite) communications where all bits are equally important and, hence, equal error protection (EEP) suffices. On the other hand, many significant advances have been made in channel coding. It has been shown that turbo [10] and low-density parity check (LDPC) codes [11] are capable of near Shannon limit error performance. Of particular interest to this paper is a class of codes related to LDPC codes introduced by Jin et al. [12] called irregular repeat accumulate (IRA) codes. IRA codes combine the advantages of both turbo and LDPC codes—they have a simple encoding algorithm and are amenable to analysis using density evolution [13] and the irregularity can be carefully optimized [12]. For JSCC, the SPIHT image coder [5] and rate-compatible punctured convolution (RCPC) codes [14] were concatenated in [15] for image transmission over a binary symmetric channel (BSC). Despite the sequential dependency induced by the scalable SPIHT bitstream that calls for UEP, EEP was employed in [15]. For example, when the crossover probability of the BSC is , a rate 2/7 RCPC code was chosen. Note that the channel capacity of the BSC is , and . Using a rate 2/7 RCPC code operates 46.19% away from the channel capacity in this case. Nevertheless, the scheme in [15] gave the best performance at its time because it took advantage of the high performance of the SPIHT source coder. The impressive results in [15] inspired many new approaches to JSCC (in the form of UEP
1057-7149/$20.00 © 2006 IEEE
LAN et al.: SOURCE-OPTIMIZED IRREGULAR REPEAT ACCUMULATE CODES
of embedded multimedia bitstreams) for improved performance [16]–[18]. These approaches generally follow one of the two directions: channel-optimized source coding or source-optimized channel coding. The aim of channel-optimized source coding is to design robust quantization schemes against errors introduced by the channel (see [19] and [20]). Other earlier related works are channel-optimized scalar quantization (e.g., [21]), channel-optimized vector quantization (e.g., [22]), trellis waveform coding [23], and channel-optimized trellis-coded quantization [24]. Source-optimized channel coding tries to match the amount of introduced redundancy to the significance of the source bits. For scalable multimedia transmission, this approach usually involves two steps: rate-compatible channel code design [14], [25]–[27] and UEP optimization algorithms [28]–[31]. The former provides different levels of error protection for source bits by puncturing the same mother code at different degrees; the latter performs optimal channel code rate assignments. To enable UEP, rate-compatible channel codes are usually designed beforehand and the best code rate assignments chosen algorithmically using dynamic programming [29], the Viterbi algorithm [30], or a local search algorithm [31]. This paper proposes a new approach towards source-optimized channel code design. Realizing the inherent UEP capabilities provided by irregularity in the degree profiles of IRA codes, we optimize the irregularity (or UEP) by explicitly taking into account the distortion-rate (D-R) performance of the source code [32]. Specifically, we use the distortion determined by the location of the first error bit at the scalable source decoder as the corresponding cost function when channel decoding fails, and use the average cost as the criterion in the IRA code design. Thus, in contrast to conventional code designs that minimize the probability of error, the IRA codes are designed to minimize the average distortion in our new approach. By combining the channel code design and UEP rate assignments in one single irregularity optimization step that also integrates the D-R characteristics of the source code, our proposed source-optimized IRA code design effectively realizes true JSCC. A scalable image transmission system is simulated with source-optimized IRA codes for transporting SPIHT-coded images over the BSC. Under the assumption that one codeword is sent for an entire image, we design both long and short IRA codes for transmitting different size SPIHT-coded images. In the former case, we further assume that events corresponding to information bits at different positions being flipped after channel decoding are independent and, consequently, use the distortion determined by the position of the first error bit at the SPIHT decoder as the cost function. In the latter case, when it is known that a similar independence assumption does not hold, we employ a modified cost function. In both cases, a model based on [33] for the operational D-R function of the SPIHT coder is adopted and incorporated into density evolution, with the objective of minimizing the average cost function in the IRA code design. For error detection, cyclic redundancy check (CRC) bits are embedded into the SPIHT-coded bitstream in our simulations. Our simulation results are uniformly better than those reported in the literature [15], [27], [30]. For example, using long
1741
source-optimized IRA codes for transmitting 512 512 images , we are able to over the BSC with crossover probability operate at 13.37% away from the channel capacity with code length 66060 bits. In contrast, the rate 2/7 RCPC code used in [15] operates at 46.19% away from the capacity with code length 222 bits for the same channel. Compared to conventional IRA code design that minimizes the probability of error, the performance gain in average PSNR from using our proposed source-optimized IRA code design with the same code rate is 0.8579 dB when the code length is 12800 bits. The performance gain diminishes as the code length increases. This reconfirms Shannon’s separation principle. Only a few works in the literature are related to ours. Huebner et al. [34] considered the serial concatenation of an array of repetition codes, an interleaver and turbo codes. With this scheme, different levels of protection are achieved by different numbers of repetitions. Caire and Biglieri [35] suggested the use of turbo codes to achieve UEP. Vasic et al. [36], [37] proposed algebraic/combinatorial constructions of LDPC codes based on balanced incomplete block designs. Very recently, Poulliat et al. [38] proposed techniques to construct LDPC codes that provide UEP. However, the methods in [34]–[38] still only provide UEP (in terms of BER) and the codes do not directly optimize a cost function directly related to the distortion. Further, these methods still need an algorithm to determine the best code rate and classes. The rest of the paper is organized as follows. In Section II, we briefly review IRA codes and the density evolution technique used in the code design. Section III establishes the connection between UEP and irregularity in IRA codes. We formulate the source-optimized IRA code design problem and present design results in Section IV. Section V addresses application of these codes to transmission of SPIHT-coded images over BSCs and Section VI concludes the paper. II. IRA CODES The recent rediscovery of LDPC codes [11] has caught a lot of attention and stimulated extensive research efforts in this area. Richardson et al. [39] proposed a numerical technique called density evolution to analyze LDPC codes. Density evolution can be used along with an optimization procedure to design good LDPC code ensembles that perform very close to the Shannon limit on AWGN channels [40]. Jin et al. [12] introduced a new class of codes called IRA codes which combine advantages of both turbo and LDPC codes. First, unlike LDPC codes, IRA codes can be easily encoded. Second, unlike turbo codes, IRA codes can be analyzed [41], [42] and carefully designed [43]. A. System Model and Density Evolution IRA codes can be represented with Tanner graphs as shown Fig. 1. The information bits of different degrees are connected to check nodes with different numbers of edges. An IRA code ensemble is specified by two degree profiles and , where and are the fraction of edges incident on the information and check nodes of degree , respectively. In this paper, we assume a concenfor some , to simplify trated right degree, i.e.,
1742
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 7, JULY 2006
denotes the log-likelihood ratio (LLR) provided by is its correthe channel on the parity bits, and sponding pdf; denotes the LLR provided by the channel on the in• is its corresponding pdf. formation bits, and With (2) and (3) and the above notations, a detailed decoding algorithm is as follows. • Step 1) Initialize and with a given pair of and . and according to (3). • Step 2) Compute and according to (2). • Step 3) Compute • Step 4) Repeat Steps 2 and 3 until the maximum number of iterations is reached. Assuming no cycles in the corresponding Tanner graph of the IRA codes, all the incoming messages at information or check nodes are then statistically independent and the pdfs of outgoing messages at information and parity nodes can be obtained by conventional convolution. Computation of the pdfs of the outgoing messages at check nodes requires a change of measure on the pdfs of the incoming messages and taking the Fourier transforms of the pdfs. For the sake of simplicity, we define an as operator •
Fig. 1.
Graph representation of systematic IRA codes.
the code design. The sufficiency of concentrated right degree to achieve capacity on the erasure channels has been shown by (or simply ) and , the rate of Shokrollahi [44]. Given the IRA codes is
(4) s are the pdfs of the incoming messages and where is the pdf of outgoing message. With this setup, density evolution for IRA codes can be described as follows:
(1) From Fig. 1, one can see that IRA codes are a special class of LDPC codes where the encoding can be easily implemented in a top-down and zig-zag fashion. The decoding can be implemented with an iterative sum-product algorithm as used in decoding LDPC codes. The updates for extrinsic messages at information and check nodes along the th edge are given as (2)
(5) (6) (7) (8) denotes the time index of the correwhere superscript or means applying conventional convolution sponding pdf and times. The probability of error after the th iteration is given by [39]
and
(9) (3) III. UEP BASED ON IRREGULARITY
respectively. It can be seen that the sum in (2) and the product in (3) are over the edges adjacent to the th edge. The behavior of the iterative decoder based on the above message update rules can be described with density evolution. To give a detailed decoding algorithm for IRA codes, we adopt the following notations: and denote the messages passed between • and check and information nodes, and are the corresponding pdfs; and denote messages passed between check • and are the corand parity nodes, and responding pdfs;
In multimedia transmission systems, UEP is necessitated by the sequential dependency in the source bitstream induced by scalable source coding. It is known that scalable coding is optimal in the D-R sense for Gaussian sources because they are successively refinable [45]. Lastras and Berger [46] further showed that all sources are almost successively refinable. Thus, we can treat scalability in source coding (hence, UEP for the source bitstream) as an almost universal concept and expect a well-designed scalable source coder to deliver near-optimal performance over a variety of sources. Indeed, the fact that the JPEG2000 image coder is scalable and MPEG is currently exploring a scalable video coding standard based on
LAN et al.: SOURCE-OPTIMIZED IRREGULAR REPEAT ACCUMULATE CODES
1743
subband/wavelet interframe coding reflects the trend towards scalable source coding. that inOn the other hand, irregularity in degree profile herently comes with IRA codes means UEP. This is because different bit nodes have different number of neighboring check nodes in IRA codes, and bit nodes with more neighboring check nodes are more protected than those with less neighboring check nodes. Thus, there is a natural connection between the current practice of scalability in source coding, which calls for UEP, and the latest development in graph-based channel coding, where irregularity in IRA codes inherently provides the needed UEP capability. This observation motivates us to optimize the inherent UEP in the IRA code design while taking into account the performance of the scalable source code. Although the UEP capability of IRA codes can be experimentally verified in the density evolution process, we attempt to make the connection between UEP and irregularity formally in a theoretical setting. The main idea is borrowed from [39], whose authors use it to show that the probability of error is nonincreasing with increasing number of iterations. Since IRA codes is a class of LDPC codes, without loss of generality, we treat information nodes and parity nodes as variable nodes collectively. For the sake of completeness, we give a detailed [39, Theorem 7] using our notations before stating our result in a corollary. Theorem 3.1: For LDPC codes with given degree profiles, the probability of error in density evolution is nonincreasing with the number of iterations. Proof: See Appendix A. Corollary 3.1: For LDPC codes with given degree profiles and the number of iterations , the probability of error for variof degree is no greater than that for able nodes of degree . variable nodes Proof: See Appendix B. IV. SOURCE-OPTIMIZED IRA CODES A. Problem Formulation One approach to designing good IRA codes involves picking and for a fixed-rate so that the probability of error in (9) is minimized for given and . For a given , the problem is formulated as
the th bit being flipped at the IRA decoder is independent of . the one corresponding to the th bit being flipped for This assumption is needed for the derivation of density evolution (see [13]). We further assume that the source bitstream is encoded into a single channel codeword and that there exists a cost model at the source decoder for different outcomes at the channel decoder. For scalable source codes, the cost due to channel decoding errors is determined by the location of the first erroneous bit and the cost model can be the operational D-R function of the source code. With this model and the independence assumption of error events, we establish the average cost as (11) is the cost (PSNR) associated with the event that bits where 1 through are decoded correctly and the first channel decoding error happens at the th bit position, is the probability is the length of the source that the th bit is in error and bitstream (or information bits). is computed assuming a D-R model for the source (please see Section IV-B for the specific for model used). According to [13], the error probability the th information bit of degree is given by (12) Then, the new IRA code design problem can be formulated as (13) The above minimization is performed for a fixed number of iterations (120 in our work) used in the density evolution. The problem formulation in (13) assumes independence of error events in density evolution, which is always true with infinite code length (or infinite graph). For finite code length (due to finite ), it is necessary to validate this assumption. Deas the binary value of the th bit after decoding. For a fine specific error event, is assigned a value of one if the th bit is flipped; otherwise, it is zero. Generally, to validate the indepenfor , we need to verified that dence between and (14)
(10)
where
.
Other approaches in the past have included optimizing the degree profiles so that the area between the EXIT charts of the bit nodes and the check nodes is minimized, number of iterations to obtain a given probability of error is minimized or the decoding threshold (in terms of SNR) is minimized. In this paper, we consider designing IRA codes with optimal irregularity/UEP for a given source bitstream. This is done by minimizing a different cost function that also involves the source code. Before formulating our source-optimized IRA code design problem, we shall assume that the event corresponding to
where is the joint pdf of and , and and are the pdf of and , respectively. Note that (14) is true if and only if and are independent, i.e., (15) Since no analytic approach is available to prove the above independence assumption about and , we resort to actually simulating a particular IRA code of length 10000 bits. Fig. 2 deand averaged over 10000 error events. picts , which means for fiWe can see that nite-length IRA codes, the independence assumption is not true and (11) needs to be modified. One solution is to compute the average distortion only for error events occurring at information
1744
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 7, JULY 2006
Fig. 2.
Top: P (V = 1; V = 1). Bottom: P (V = 1)P (V = 1) for i = 0, j = 1; . . . ; 5919.
bits with the highest degree and approximate the corresponding distortion as (16)
TABLE I PROFILES OF THE FOUR SOURCE-OPTIMIZED IRA CODES DESIGNED FOR BSCS WITH TWO DIFFERENT CROSSOVER PROBABILITIES p AND TWO DIFFERENT CODE LENGTHS N FOR EACH p. NOTE: IS THE FRACTION OF EDGES CONNECTED TO DEGREE i NODES
where is the number of information bits with the highest degree (or strongest protection). The reasons for this approximation are twofold. First, the distortion due to this type of error events should be as small as possible. Second, intuitively, the error-correcting performance of the source-optimized IRA codes should not be very different from that of the conventionally designed codes using (10). Hence, even after taking correthe dependence into account, the error probability sponding to the event that the first error at the th bit has high deserves gree is still much smaller than one, and as a good approximation to the probability that the first bits have no error. B. Design Results The new IRA code design assumes a D-R model for the source code. Although several models are available (e.g., the classic exponential decay model and the Weibull model [47]), we use results of [33], which show that the D-R performance of embedded wavelet image coders at low bit rates roughly , where is a constant. follows the rule of With this D-R model of the source code, we optimize for s and s using (11) as the cost criterion for BSCs with two crossover probabilities ( and ). We use the constrained nonlinear optimization function fmincon in Matlab with several randomly chosen initial values of s for a fixed to perform the optimization. Once the degree distribution is determined, we construct finite length codes of two lengths ( and bits for each ), resulting four sourceoptimized IRA codes. The profiles of these IRA codes are shown in Table I. For the sake of comparison with the source-optimized Code in Table I, we separately design a conventional IRA Code using density evolution and (10) as the cost criterion for the same BSC with the same code rate of
and code length of bits. In constructing both information bits have decodes, we assume that the creasing levels of importance and assign higher degrees to the more important parts. S-random interleavers [48] with the same are used in both cases to connect the spreading factor of edges in the IRA code graph. To better illustrate UEP, Table II lists the degree profiles of both codes in terms of node perspective. That is, instead of listing s as in Table I, we give (17) in Table II. For example, with Code , the first of information bits have the most protection, while the last 20637 bits have the least protection. The block error rate is 8.06 . The block error rate of Code , of Code which is expected to be lower than that of Code due to (10),
LAN et al.: SOURCE-OPTIMIZED IRREGULAR REPEAT ACCUMULATE CODES
TABLE II PROFILES OF CODE A (SOURCE-OPTIMIZED) AND CODE A (CONVENTIONAL) =i)K WITH IN TERMS OF NODE PERSPECTIVE K = ( =i= K = 30147 BITS. BOTH CODES ARE DESIGNED FOR THE SAME p = 0:1 BSC WITH THE SAME CODE RATE OF R = 0:46 AND CODE LENGTH OF N = 66060 BITS. THE ONLY DIFFERENCE LIES IN THE DESIGN CRITERION
1745
TABLE III AVERAGE PSNR IN DECIBELS FOR 512 512 IMAGES OVER BSC WITH CROSSOVER PROBABILITY 0.1 AND 0.03 AND TOTAL TRANSMISSION RATE 0.252 BPP
2
TABLE IV AVERAGE PSNR IN DECIBELS FOR 512 512 IMAGES OVER BSC WITH CROSSOVER PROBABILITY 0.1 AND 0.03 AND TOTAL TRANSMISSION RATE 0.505 BPP
2
is 4.23 . We also record the first error bits for Code and Code and calculate the average cost according to (11) , where is the for both codes with number of correct information bits before the first decoding error (if any). We obtain an average cost of 8.7322 for Code and 8.8527 for Code . Although Code has a lower probability of error, the number of correct information bits before the first error bit is smaller than that of code . These smaller s in the rare error events contribute to the slightly larger average cost for Code . V. APPLICATIONS TO SCALABLE IMAGE TRANSMISSION We use the four IRA codes in Table I to simulate scalable image transmission over BSCs with the SPIHT coder. The images considered are the well-known 512 512 Lena, Goldhill, and Barbara. The code lengths of bits and bits are chosen so that the transmission rates are 0.252 and 0.505 bpp, respectively, for easy comparison with results published in the literature (e.g., [15], [27], and [30]). In our simulations, 16 CRC bits are added to each 4120-bit SPIHT bitstream packet, in accordance with [30], for the purpose of error detection. We assume that erroneous packets (if any) can be detected perfectly by the CRC bits and the SPIHT decoder uses only correctly decoded bits before the first erroneous packet. Tables III and IV show image transmission results in terms of average PSNR in decibels from our simulations using two and ) and two different BSC conditions ( transmission rates (0.252 and 0.505 bpp). Each reported result is averaged over 2000 runs of transmitting the same image. Note in Table I are chosen in our design that our IRA code rate such that at most one decoding error occurs in every 1000 runs. The results from [15], [27], and [30] are also included in Tables III and IV for comparison. Note that JPEG2000 used in [30] usually gives higher PSNR than SPIHT at the same rate, especially for Barbara. We also give in the tables the limiting performances achievable by capacity-approaching IRA codes dB, where is the operaas 10 is the transmission tional D-R function of the SPIHT coder,
is the channel capacity of BSC with crossover rate, and probability . It is seen from Tables III and IV that our results are uniformly better than those in [15] and [27] when the same SPIHT coder is used and they are only 0.40–0.73 dB away from the theoretical limits. These superb performances are due to the , powerful IRA codes we employ. For example, when ; using Code with , we are able to operate at only 9.38% away from the channel capacity with code length bits. When , ; using Code with , we are able to operate at 13.37% away from the channel capacity with code length bits; in contrast, the rate 2/7 RCPC code used in with code [15] operates at 46.19% away from the capacity length bits for the same channel. However, the price paid for the excellent performance of the IRA codes are longer code lengths (or latency in image transmissions). When Code is replaced by Code for the same BSC with , our simulations give roughly the same average PSNR for each of the three test images, indicating almost no performance gain from our proposed source-optimized channel code design. This is because Shannon’s separation principle holds in the regime with long code length. Another reason is because the block error rates for both Code and Code are too small for the rare error events to impact the average PSNR over 2000 runs of image transmissions. We thus consider employing shorter IRA codes for transmission of smaller (e.g., 176 144 QCIF) images. Again, we limit ourselves to the case when each SPIHT-coded source bitstream is encoded into one IRA codeword. When the transmission rate is low, the code length is relatively short. Since Shannon’s separation principle does not hold in such scenarios, our source-optimized IRA code design is expected to outperform the con-
1746
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 7, JULY 2006
Fig. 3. Cumulative distribution functions of the first error bits of Code E and Code E .
ventional code design. To highlight the advantage of our JSCC methodology, we keep the same channel code rate for the corresponding BSC channel as before; this way, as the channel code length is shorter than before, we can observe more channel decoding failures to assess their impact. Note that, in scalable image communication systems, channel decoding errors result in occasional dipping of image quality, which is often acceptable to human eyes. For shorter IRA codes, although one can simply scale the code length down based on the same degree profile obtained for a longer code length, this does not typically produce good results. Thus, two new IRA codes (the source-optimized Code and the conventional Code ) are designed for transmitwith a ting SPIHT-coded QCIF images over a BSC transmission rate of 0.505 bpp. The code length for both codes bits. The D-R model in [33] is adopted with is . Similar to Code and Code , in and Code , we pick the same code rate designing Code and same for both codes, however, since the code length is short, we restrict the left degree to be 7. Hence, are different from that of the degree profiles of Code and and even though they are designed for the same channel and rate. In addition, Code is designed using the cost function (16). Density evolution gives the degree profile of Code as , and the degree proas file of Code . The block error rate is 6.27 for for Code . Code and 3.85 In simulations with transmitting the SPIHT-coded QCIF Mad image—the first frame of the QCIF Mother_Daughter at 0.505 bpp, we obtain sequence—over the BSC with an average PSNR of 30.69 dB with Code and 29.82 dB with Code . Note that, if there is no channel decoding error, the PSNR after SPIHT decoding is 31.46 dB. Therefore, using Code incurs 0.77-dB loss in PSNR due to decoding errors. This loss increases to 1.64 dB with code . Whenever channel decoding fails, we assume that the CRC bits can detect the
erroneous packets. The average PSNR during these error events is 25.37 dB with Code and 20.36 dB with Code . Code thus provides 5-dB gain on average over Code during error events. This performance difference during error events shows the real advantage of the source-optimized Code , because there is no performance difference between the two codes when the IRA decoder is error free. To give more detailed information, we plot in Fig. 3 the cumulative distribution functions of the locations of the first error bits for both Code and Code , assuming an all 0 input sequence without CRC bits. From Fig. 3 and the degree profiles of both codes, we see that 90% of first error bits are those of the lowest degree 2 for Code . However, only 55% of first error bits are those of degree 2 for Code . Thus, optimization with respect to the source code in designing Code effectively pushes the first error bit down along the bits source bitstream so that the distortion caused by bit errors are small. Fig. 4 shows the normalized frequency of bit errors versus bit position (from highest to lowest degrees) of Code and Code . Clearly, both codes provide UEP. However, with Code , most errors are concentrated in the last 1000 bits, and errors occur much less frequently on the preceding 4888 bits than with Code . This shows the difference made with our source-optimized code design methodology using the average distortion as the cost function. VI. CONCLUSION In this paper, we have presented a JSCC paradigm based on source-optimized IRA code design by taking into account the D-R performance of the scalable source code and minimizing a new cost function that is the average end-to-end distortion. We argued that scalability is an almost universality concept in source coding and made the connection between UEP, as necessitated by scalability, and irregularity in graph-based channel coding. Consequently, we motivated our new channel code design methodology as a means of optimizing the inherent UEP capability of IRA codes.
LAN et al.: SOURCE-OPTIMIZED IRREGULAR REPEAT ACCUMULATE CODES
Fig. 4.
1747
Normalized frequency of bit errors versus bit position for source-optimized (top) Code E and (bottom) conventional Code E .
Applying our source-optimized IRA codes to scalable transmission of SPIHT-coded images over BSCs, we have obtained the best published results in the literature. Our work represents the progress made in JSCC since the publication of [15] in 1997. Much of our limit-approaching performance is attributed to recent advance in near-capacity IRA code design. Because long code length is needed to approach the capacity, our simulations with long IRA codes show almost the same performance with or without source-based optimization. This agrees with Shannon’s separation principle. Using short codes for transmitting QCIF images, our simulations indicate that source-optimized IRA codes can significantly outperform conventional IRA codes of the same code rate during error events. This is because source-optimized IRA codes effectively push channel decoding errors towards the end of the source bitstream, where they contribute much less to the end-to-end distortion. Our proposed JSCC approach thus works better with relatively short code length when the separation principle no longer holds, although designing short IRA codes with good performance is still an active area of research in channel coding. Finally, we have only concerned ourselves with the SPIHT image coder in this paper, other scalable image coders (e.g., JPEG2000) and a new class of efficient three-dimensional scalable wavelet video coders (e.g., [8], [9]) can replace the SPIHT coder. APPENDIX A PROOF OF THEOREM 3.1 Proof: Because of the assumption that there are no cycles in the Tanner graphs, the estimate of a particular information bit at the th iteration is given by an estimator operating on a set of observations that form a tree of depth . For a particular informaof degree , define as the binary value carried by tion bit and as messages passed from its child (neighboring) nodes. Similarly, taking each one of these child nodes as a parent node, the message passed from its child nodes is given as (excluding ). Repeat the process until reaching leaf nodes. In this way, we construct a tree of depth (see Fig. 5) and denote it
Fig. 5. Support tree
as . Clearly, we have lection of variable nodes in from the channel carried by
for decoding B .
, where is the col. Denote as the observations , then we have (18)
where are the observations from the channel carried by leaf nodes. Under the assumption of independence, the message passing algorithms at the th iteration actually perform
1748
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 7, JULY 2006
Fig. 6. Support trees
maximum a posteriori (MAP) estimation of , i.e., observations at variable nodes of
, respectively, for decoding B and B
and
based on
.
in terms of conditional entropy, because and
, which is equivalent to
(21) From (19) and (21), it follows that (22) where is the message from the channel, the binary value , and the observations (LLRs) from the conveyed by channel associated with is partitioned to and where
and such that
. According to the MAP rule, , hence
Next, we have
,
denote the region of observations in and
, respectively. Note that, for
a MAP decoder, the probability of error is no greater than 1/2, since
for (23)
where is the joint pdf of and . Based on the argument above, we conclude that the probability of error is nonincreasing in the number of iterations.
(19) and, similarly, for
. Furthermore, we have
(20)
APPENDIX B PROOF OF COROLLARY 3.1 Proof: For any , consider the estimates of and associated with and of degree and , respectively, at the th iteration. The corresponding support trees and for decoding and , respectively, are is defined as the subtree of shown in Fig. 6, where with its first edges.
LAN et al.: SOURCE-OPTIMIZED IRREGULAR REPEAT ACCUMULATE CODES
Although and stem from and , respectively, they have the same number of edges; thus, the statistics and are the same of the estimates at bit nodes in for a given degree rofile. Therefore
Since
, from (19) to (23), we have
Combining the above two gives
We thus conclude that the variable nodes with more neighboring check nodes have probabilities of errors no greater than those with less neighboring check nodes. REFERENCES [1] C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, 1948. [2] J. L. Massey, Joint Source Channel Coding Communication Systems and Random Process Theory, J. K. Skwirzynski, Ed. Amsterdam, The Netherlands: Sijthoff and Nordhoff, 1978, pp. 279–293. [3] B. Hochwald and K. Zeger, “Tradeoff between source and channel coding,” IEEE Trans. Inf. Theory, vol. 43, no. 5, pp. 1412–1424, Sep. 1997. [4] J. Shapiro, “Embedded image coding using zero trees of wavelet coefficients,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3445–3463, Dec. 1993. [5] A. Said and W. Pearlman, “A new, fast, and efficient image codec based on set partitioning in hierarchical trees,” IEEE Trans. Circuits Syst. Video Technol., vol. 6, no. 3, pp. 243–250, Jun. 1996. [6] D. Taubman and M. Marcellin, JPEG2000: Image Compression Fundamentals, Standards, and Practice. Norwell, MA: Kluwer, 2001. [7] B. Kim, Z. Xiong, and W. Pearlman, “Very low bit-rate embedded video coding with 3-D set partitioning in hierarchical trees (3-D Spiht),” IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 8, pp. 1374–1387, Dec. 2000. [8] S.-T. Hsiang and J. Woods, “Embedded video coding using invertible motion compensated 3-D subband/wavelet filter bank,” Signal Process.: Image Commun., vol. 16, no. 8, pp. 705–724, May 2001. [9] A. Secker and D. Taubman, “Lifting-based invertible motion adaptive transform (LIMAT) framework for highly scalable video compression,” IEEE Trans. Image Process., vol. 12, no. 12, pp. 1530–1542, Dec. 2003. [10] C. Berrou and A. Glavieux, “Near optimum error correcting coding and decoding: Turbo-codes,” IEEE Trans. Commun., vol. 44, no. 5, pp. 1261–1271, Oct. 1996. [11] D. MacKay, “Good error-correcting codes based on very sparse matrices,” IEEE Trans. Inf. Theory, vol. 45, no. 2, pp. 399–431, Mar. 1999. [12] H. Jin, A. Khandekar, and R. McEliece, “Irregular repeat accumulate codes,” in Proc. 2nd Int. Symp. on Turbo codes, Brest, France, Sep. 2000, pp. 1–8. [13] T. Richardson and R. Urbanke, “The capacity of low-density paritycheck codes under message-passing decoding,” IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 599–618, Feb. 2001. [14] J. Hagenauer, “Rate-compatible punctured convolutional codes (RCPC codes) and their applications,” IEEE Trans. Commun., vol. 36, no. 2, pp. 389–400, Apr. 1988. [15] P. G. Sherwood and K. Zeger, “Progressive image coding for noisy channels,” IEEE Signal Process. Lett., vol. 4, no. 7, pp. 189–191, Jul. 1997. [16] “Special issue on error-resilient image and video transmission,” IEEE J. Sel. Areas Commun., vol. 18, no. 6, Jun. 2000. [17] “Special issue on wireless video,” IEEE Trans. Circuits Syst. Video Technol., vol. 12, no. 6, Jun. 2002. [18] “Special issue on streaming media,” IEEE Trans. Multimedia, vol. 6, no. 4, Apr. 2004.
1749
[19] Q. Chen and T. R. Fischer, “Image coding using robust quantization for noisy digital transmission,” IEEE Trans. Image Process., vol. 7, no. 4, pp. 496–505, Apr. 1998. [20] T.-T. Lam, G. P. Abousleman, and L. Karam, “Image coding with robust channel-optimized trellis-coded quantization,” IEEE J. Sel. Areas Commun., vol. 18, no. 6, pp. 940–951, Jun. 2000. [21] A. Kurtenbach and P. Wintz, “Quantizing for noisy channels,” IEEE Trans. Commun., vol. 17, no. 2, pp. 291–302, Apr. 1969. [22] N. Farvardin and V. Vaishampayan, “On the performance and complexity of channel-optimized vector quantizers,” IEEE Trans. Inf. Theory, vol. 37, no. 1, pp. 155–160, Jan. 1991. [23] A. Ayanoglu and R. M. Gray, “The design of joint source and channel trellis waveform coders,” IEEE Trans. Inf. Theory, vol. 33, no. 6, pp. 855–865, Nov. 1987. [24] M. Wang and T. R. Fischer, “Trellis coded quantization designed for noisy channels,” IEEE Trans. Inf. Theory, vol. 40, no. 6, pp. 1792–1802, Nov. 1994. [25] D. Rowitch and L. Milstein, “Rate compatible punctured turbo (RCPT) codes in a hybrid fec/arq system,” in Proc. Globecom, vol. 4, Phoenix, AZ, Nov. 1997, pp. 55–59. [26] O. Acikel and W. Ryan, “Punctured turbo-codes for BPSK/QPSK channels,” IEEE Trans. Commun., vol. 47, no. 9, pp. 1315–1323, Sep. 1999. [27] C. Lan, T. Chu, K. R. Narayanan, and Z. Xiong, “Scalable image and video transmission using irregular repeat accumulate (IRA) codes with fast algorithm for optimal unequal error protection,” IEEE Trans. Commun., vol. 52, no. 7, pp. 1092–1101, Jul. 2004. [28] J. Lu, A. Nosratinia, and B. Aazhang, “Source-channel rate allocation for progressive transmission of images,” IEEE Trans. Commun., vol. 51, no. 2, pp. 186–196, Feb. 2003. [29] V. Chande and N. Farvardin, “Progressive transmission of images over memoryless noisy channels,” IEEE J. Sel. Areas Commun., vol. 18, no. 6, pp. 850–860, Jun. 2000. [30] B. Banister, B. Belzer, and T. Fischer, “Robust image transmission using JPEG2000 and turbo-codes,” IEEE Signal Process. Lett., vol. 9, no. 4, pp. 117–119, Apr. 2002. [31] R. Hamzaoui, V. Stankovic, and Z. Xiong, “Fast algorithm for distortionbased error protection of embedded image codes,” IEEE Trans. Image Process., vol. 14, no. 10, pp. 1417–1421, Oct. 2005. [32] C. Lan, K. R. Narayanan, and Z. Xiong, “Source-optimized irregular repeat accumulate codes with inherent unequal error protection capabilities and their application to progressive image transmission,” presented at the 37th Asilomar Conf. Signals, Systems, Computers, Pacific Grove, CA, Nov. 2003. [33] S. Mallat and F. Falzon, “Analysis of low bit rate image transform coding,” IEEE Trans. Signal Process., vol. 46, no. 4, pp. 1027–1042, Apr. 1998. [34] A. Huebner, J. Freudenberger, R. Jordan, and M. Bossert, “Irregular turbo codes and unequal error protection,” in Proc. ISIT, Washington, DC, Jun. 2001, pp. 24–29. [35] G. Caire and E. Biglieri, “Parallel concatenated codes with unequal error protection,” IEEE Trans. Commun., vol. 46, no. 5, pp. 565–567, May 1998. [36] B. Vasic, A. Cvetkovic, Z. Wu, and M. Marcellin, “Robust image transmission with unequal error protection low-density parity-check codes,” presented at the Communication Theory Workshop, Mesa, AZ, Apr. 2003. [37] B. Vasic, A. Cvetkovic, S. Sankaranarayanan, and M. Marcellin, “Adaptive error protection low-density parity-check codes for joint source-channel coding,” in Proc. ISIT, Yokohama, Japan, Jun. 2003, pp. 267–267. [38] C. Poulliat, D. Declercq, and I. Fijalkow, “Optimization of LDPC codes for UEP channels,” presented at the ISIT, Chicago, IL, Jun. 2004. [39] T. Richardson, A. Shokrollahi, and R. Urbanke, “Design of capacity approaching irregular low-density parity-check codes,” IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 619–637, Feb. 2001. [40] S. Chung, T. Richardson, and R. Urbanke, “Analysis of sum-product decoding of low-density parity-check codes using a gaussian approximation,” IEEE Trans. Inf. Theory, vol. 47, no. 2, pp. 657–670, Feb. 2001. [41] A. Roumy, S. Guemghar, G. Caire, and S. Verdu, “Design methods for irregular repeat-accumulate codes,” IEEE Trans. Inf. Theory, vol. 50, no. 8, pp. 1711–1727, Aug. 2004. [42] H. Pfister, I. Sason, and R. Urbanke, “Bounds on the decoding complexity of punctured codes on graphs,” presented at the Allerton Conf., Monticello, IL, Sep. 2004. [43] S. t. Brink and G. Kramer, “Design of repeat-accumulate codes for iterative detection and decoding,” IEEE Trans. Signal Process., vol. 51, no. 11, pp. 2764–2772, Nov. 2003.
1750
[44] A. Shokrollahi, “New sequences of time erasure codes approaching channel capacity,” in Proc. 13th Int. Symp. Applied Algebra, Algebraic Algorithms Error-Correcting Codes, vol. 1719, Lectures Notes in Computer Science, 1999, pp. 65–76. [45] W. H. Equitz and T. M. Cover, “Successive refinement of information,” IEEE Trans. Inf. Theory, vol. 37, no. 2, pp. 269–275, Mar. 1991. [46] L. Lastras and T. Berger, “All sources are nearly successively refinable,” IEEE Trans. Inf. Theory, vol. 47, no. 3, pp. 918–926, Mar. 2001. [47] Y. Charfi, R. Hamzaoui, and D. Saupe, “Model-based real-time progressive transmission of images over noisy channels,” presented at the WCNC, New Orleans, LA, Mar. 2003. [48] S. Dolinar and D. Divsalar, “Weight distributions of turbo codes using random and non-random permutations,” JPL Progr. Rep., 1995.
Ching-Fu Lan was born in Taichung, Taiwan, R.O.C. He received the M.S. degree in telecommunication engineering from the National Chiao-Tung University, Taiwan, in June 1996, and the Ph.D. degree in electrical engineering from Texas A&M University, College Station, in 2004. He research interests include equalization, adaptive signal processing, resource allocation for multicarrier systems, error-correction codes, joint sourcechannel coding, and distributed source coding.
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 15, NO. 7, JULY 2006
Zixiang Xiong (S’91–M’96–SM’02) received the Ph.D. degree in electrical engineering from the University of Illinois at Urbana-Champaign, Urbana, in 1996. From 1997 to 1999, he was with the University of Hawaii, Honolulu. Since 1999, he has been with the Department of Electrical and Computer Engineering, Texas A&M University, College Station, where he is an Associate Professor. He spent the summers of 1998 and 1999 at Microsoft Research, Redmond, WA, and the summers of 2000 and 2001 at Microsoft Research, Beijing, China. His current research interests are network information theory, code designs and applications, genomic signal processing, and networked multimedia. Dr. Xiong received an NSF Career Award in 1999, an ARO Young Investigator Award in 2000, and an ONR Young Investigator Award in 2001. He also received faculty fellow awards in 2001, 2002, and 2003 from Texas A&M University. He served as Associate Editor for the IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY (from 1999 to 2005) and the IEEE TRANSACTIONS ON IMAGE PROCESSING (from 2002 to 2005). He is currently an Associate Editor for the IEEE TRANSACTIONS ON SIGNAL PROCESSING and the IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART B: CYBERNETICS.
Krishna R. Narayanan is an Associate Professor with the Department of Electrical and Computer Engineering, Texas A&M University, College Station. During the 2004–2005 academic year, he held visiting positions at the University of Illinois at Urbana Champaign, Urbana; the Institut Eurecom; and the Indian Institute of Science. His research interests are in channel coding, information theory, and signal processing, particularly codes on graphs, iterative decoding, and signal processing and joint source-channel coding. Mr. Narayanan is the recipient of the 2001 NSF Career award and the outstanding faculty award from the Department of Electrical and Computer Engineering, Texas A&M University, in 2003. He is currently an Editor for the IEEE TRANSACTIONS ON COMMUNICATIONS.
