Memory-Enhanced MMSE Decoding in Vector ...

IEICE TRANS. INF. & SYST., VOL.E86–D, NO.10 OCTOBER 2003

2218

PAPER

Memory-Enhanced MMSE Decoding in Vector Quantization Heng-Iang HSU† , Wen-Whei CHANG† , Xiaobei LIU†† , and Soo Ngee KOH†† , Nonmembers

SUMMARY An approach to minimum mean-squared error (MMSE) decoding for vector quantization over channels with memory is presented. The decoder is based on the Gilbert channel model that allows the exploitation of both intra- and interblock correlation of bit error sequences. We also develop a recursive algorithm for computing the a posteriori probability of a transmitted index sequence, and illustrate its performance in quantization of Gauss-Markov sources under noisy channel conditions. key words: MMSE decoding, vector quantization, Gilbert channel

1.

Introduction

Vector quantizers (VQs) are widely used in source coding applications. Transmitting VQ data over noisy channels changes the codevector indices and consequently leads to severe distortions in the reconstructed output. This has motivated investigations in exploiting the residual source redundancy for better decoding of the transmitted index sequence [1]–[3]. Conventional design approaches to joint source-channel decoding can be grouped into two categories: maximum a posteriori (MAP) and minimum mean-square error (MMSE). No matter which design scheme is used, special care must be taken to ensure that actual error characteristics are incorporated into the computation of channel transition probabilities. Real channels are characterized by error bursts due to the combined effects of intersymbol interference and multipath fading. A standard technique for VQ over a channel with memory is to use interleaving to render the channel memoryless and then design a decoding algorithm for the memoryless channel. This approach, however, often introduces large decoding delays and does not utilize the channel memory information. It is therefore believed that further improvement can be realized through a more precise characterization of the channel on which the decoder design is based [4]. In this study, we focused on the two-state Markov chain model proposed by Gilbert [5]. This choice is motivated Manuscript received July 15, 2002. Manuscript revised February 19, 2003. † The authors are with the Department of Communication Engineering, National Chiao-Tung University, Hsinchu, Taiwan, R.O.C. †† The authors are with the School of Electrical and Electronic Engineering, Nanyang Technological University, 639798 Singapore.

in part by its applicability to performance analysis of error-correcting codes on channels with memory [6], and partly because efficient methods [7] are available for estimating Gilbert model parameters of the given channel. 2.

Joint Source-Channel Coding Scheme

We will address the design of a d-dimensional, mbit/vector VQ whose output is to be transmitted over noisy channels with memory. Figure 1 gives the block diagram of the joint source-channel coding scheme based on VQ with MMSE decoding. We assume that the source to be encoded is a real-valued, discrete-time, stationary process {vt ; t = 0, 1, . . .}. In the coding process of VQ, source samples are grouped in blocks of d samples to form a sequence of input vectors vn ∈ Rd , where vn = (vnd , vnd+1 , . . . , vnd+d−1 ) represents the nth block of source samples. Given an input vector vn , the VQ encoder searches through the codebook for the codevector ci that best matches vn and then transmits the index xn = i to the receiver in binary format. Here, the index i is regarded as an integer representing the decimal equivalent of a binary codeword b(i) = (b0 (i), b1 (i), . . . , bm−1 (i)), where the bits bl (i) are determined by the natural binary code for the index i. The codebook consisting of M = 2m codevectors, C = {c0 , c1 , . . . , cM−1 }, is designed for a noiseless channel using the generalized Lloyd algorithm [8]. Assume that the channel’s input xn and output yn differ by an error en , so that the received bit combination is bl (yn ) = bl (xn )⊕bl (en ), l = 0, 1, . . . , m−1, where ⊕ denotes the bitwise modulo-2 addition. Instead of using a codebook-lookup decoder, we develop an instantaneous MMSE decoder which determines the optimal estimate of input vector at time n given all the observations prior to and including time n. The receiver consists of the a posteriori probability calculator followed by conditional mean estimation, and includes two subsystems. The first subsystem observes a long sequence of channel outputs, denoted by y1n = (y1 , y2 , . . . , yn ),

Fig. 1

Block diagram of a VQ system using MMSE decoding.

HSU et al.: MEMORY-ENHANCED MMSE DECODING IN VECTOR QUANTIZATION

2219

and uses them to compute the a posteriori probability P [xn |y1n ] for each of possibly transmitted indices xn ∈ {0, 1, . . . , M − 1}. The second subsystem uses the a posteriori probabilities to produce the MMSE estimate as follows: ˆn = v

M−1


ci P [xn = i|y1n ].

(1)

i=0

3.

Memory-Enhanced MMSE Decoding

Depending upon the choice of a posteriori probability calculator, a number of different decoder implementations can be realized. In this work, we exploit the residual source redundancy and channel memory information for better decoding of Gauss-Markov sources over noisy channels. When quantizing GaussMarkov sources, we model the residual redundancy by assuming that the transmitted index sequence forms a first-order Markov process with transition probabilities P [xn |xn−1 ]. This reduces the a posteriori probability to the following: P [xn |y1n ] =

M−1


P [xnn−1 |y1n ]

xn−1 =0

=

M−1


P [yn |xnn−1 , y1n−1 ] P [yn |y1n−1 ] xn−1 =0 · P [xn |xn−1 ]P [xn−1 |y1n−1 ].

(2)

Notice that specific knowledge about the channel is incorporated into P [xn |y1n ] through the channel transition probabilities Qy|x = P [yn |xnn−1 , y1n−1 ]. If we consider the channel to be stationary, the error process en is independent of the channel input xn . There are many models describing the correlation of bit error sequences. Two memory-enhanced MMSE decoders are considered: MMSE1 decoder incorporates only the intra-block memory of the channel as represented by Qy|x = P [en ], and MMSE2 decoder exploits the intra-block and inter-block memory of the channel as represented by Qy|x = P [en |en−1 ]. For notational convenience, bl (en ) and bl (en−1 ) will be denoted, respectively, as rm(n−1)+l and rm(n−2)+l , for l = 0, 1, . . . , m − 1. The channel transition probabilities can be expressed as Qy|x =

mn−1 

4.

Probability Recursions for Gilbert Channel

Parameterized probabilistic models which characterize some of the most relevant aspects of error statistics are often required in designing an error protection scheme. It is apparent from previous work on channel modeling [4] that we are confronted with contrasting requirements in selecting a good model. A model should be representative enough to describe real channel behavior and yet it should not be analytically complicated. To permit theoretical analysis, we assumed that codevector indices were transmitted over Gilbert channel model [5]. The Gilbert model consists of an error-free state G and a bad state B, in which errors occur with the probability 1 − h. The state transition probabilities are b and g for the G to B and B to G transitions, respectively. The model state-transition diagram is shown in Fig. 2. The effective bit error rate (BER) produced by the Gilbert channel is  = (1−h)b/(g+b). The memory of the Gilbert channel is due to the Markov structure of the state transitions, which leads to a dependence of the current channel state sk on sk−1 . The effectiveness of the MMSE decoding crucially depends on how well the error characteristics are incorporated into the calculation of channel transition probabilities. Although using channel memory information was previously proposed for MAP symbol decoding [9], their emphasis were placed upon channels with no inter-block memory. When only access to the intra-block memory is available, it was shown that the channel transition probabilities of the Gilbert channel have closed-form expressions that can be represented in terms of model parameters {h, b, g}. Under such conditions, we can proceed the MMSE1 decoding in a manner similar to the work of [9]. Extension of these results to channels with both intra- and inter-block memory has been found difficult. This motivated our research into trying to develop a general treatment of probability recursions for the Gilbert channel. The main result is a recursive implementation of MMSE2 decoder being closer to the optimal for channels with memory. The key for the calculation of channel transition ], interpreted probabilities Qy|x in (3) is P [rk = 1|rkk−1 0 as the bit error probability conditioned on past values. The following is devoted to a way of recursively com] from P [rk−1 = 1|rkk−2 ]. The puting of P [rk = 1|rkk−1 0 0 k−1 Gilbert channel has two features, P [sk |sk−1 , rk0 ] =

P [rk = 1|rkk−1 ]rk P [rk = 0|rkk−1 ]1−rk 0 0

k=m(n−1)

(3) where k0 = m(n − 1) for MMSE1 decoder and k0 = m(n − 2) for MMSE2 decoder. Fig. 2

Gilbert’s channel model.

IEICE TRANS. INF. & SYST., VOL.E86–D, NO.10 OCTOBER 2003

2220

P [sk |sk−1 ] and P [rk |sk , rkk−1 ] = P [rk |sk ], which facili0 tate the probability recursions. Its bit error probabilities can be formulated as P [rk = 1|rkk−1 ] 0 = P [rk = 1|sk = B, rkk−1 ]P [sk = B|rkk−1 ] 0 0 = (1 − h)P [sk = B|rkk−1 ]. 0

(4)

By successively applying conditional probability and the Markovian property of the channel, we have P [sk = B|rkk−1 ] 0 ]P [sk−1 = G|rkk−1 ] = P [sk = B|sk−1 = G, rkk−1 0 0 + P [sk = B|sk−1 = B, rkk−1 ]P [sk−1 = B|rkk−1 ] 0 0 = b + (1 − g − b) = b + (1 − g − b) ·

] P [sk−1 = B, rk−1 |rkk−2 0 P [rk−1 |rkk−2 ] 0 P [rk−1 |sk−1 = B] P [rk−1 |rkk−2 ] 0

] P [rk−1 = 1|rkk−2 0 . 1−h

(5)

Then substituting (5) into (4) to complete the recursive formula: P [rk = 1|rkk−1 ] 0  (1 − g)(1 − h), rk−1 = 1     P [rk−1 = 1|rkk−2 ] 0 = (1−h)b + (1−g −b)h , k−2  1−P [rk−1 = 1|rk0 ]    rk−1 = 0 (6) 5.

Experimental Results

Experiments were carried out to investigate the potential advantages of the combined use of residual source redundancy and channel memory information for MMSE decoding in channels with error bursts. The input signals are Gauss-Markov sources described by vt = ρvt−1 + wt , where wt is zero-mean, unit-variance white Gaussian noise, with correlation coefficients of ρ = 0.5 and ρ = 0.9. VQ systems with memoryenhanced MMSE1 and MMSE2 decoders were compared against the VQ-IL-MMSE, which consists of a VQ encoder, an interleaver/de-interleaver combination and an MMSE decoder designed for the memoryless BSC. Here, the interleaving was done by arranging the 100 m-bit binary codewords of their corresponding indices into 100 rows of a rectangular array and then transmitting them column by column. Table 1 presents the VQ results associated with various decoding algorithms for the case where the bits in the codevector indices are subjected to error sequences typical of the Gilbert channel with BER  =

0.01. The performances were measured in terms of SNR for VQ having the following codebook sizes and dimension values (M, d) = (16, 4), (64, 6), (16, 2), (64, 3). The results of these experiments clearly demonstrate the improved performance achievable using VQ-MMSE1 and VQ-MMSE2 in comparison to that of VQ-IL-MMSE. Furthermore, the improvement has a tendency to increase for higher rates and for more heavily correlated Gaussian sources. Compared with VQ-MMSE1, the better performance of VQ-MMSE2 can be attributed to its ability to compute the a posteriori probabilities taking inter-block and intra-block memory of the channel into consideration. To elaborate further, SNR performances of various decoding algorithms were examined for Gilbert channels with BER ranging from 10−3 to 10−1 . We provide results for experiments on rate 1 (M = 64, d = 6) and rate 2 (M = 64, d = 3) VQ systems in Figs. 3 and 4, respectively. It can be seen that the accuracy of the channel model used in developing the decoding algorithm is extremely important to the performance of the VQ. The investigation further showed that the improved performance achievable Table 1 SNR (dB) performance of different vector quantizers for Gauss-Markov sources over the Gilbert channel with BER
= 0.01. Rate

(M, d)

System

ρ = 0.5

ρ = 0.9

1

(16,4)

VQ-IL-MMSE VQ-MMSE1 VQ-MMSE2 VQ-IL-MMSE VQ-MMSE1 VQ-MMSE2 VQ-IL-MMSE VQ-MMSE1 VQ-MMSE2 VQ-IL-MMSE VQ-MMSE1 VQ-MMSE2

4.9781 5.4275 5.4404 4.8612 5.6128 5.6285 8.1156 9.0311 9.3010 7.6427 9.6007 9.7878

8.1839 9.2174 9.6475 7.8850 9.8638 10.2452 10.1267 11.2871 12.4343 9.8261 12.4904 13.3793

(64,6)

2

(16,2)

(64,3)

Fig. 3 SNR (dB) performance of different vector quantizers as a function of Gilbert channel BER. Codebook size = 64, vector dimension = 6, source = 1st-order Gauss-Markov (ρ = 0.9).

HSU et al.: MEMORY-ENHANCED MMSE DECODING IN VECTOR QUANTIZATION

2221

using memory-enhanced MMSE decoding is more noticeable for higher channel BER. The next step in the present investigation concerned the performance degradation that may result from using the VQ-MMSE2 under channel mismatch conditions. In Table 2, d refers to the BER value assumed in the design process, and a refers to the true BER. It can be seen that the VQMMSE2 is not very sensitive to a channel mismatch between the design and evaluation assumptions, particularly for low a values. We next investigated the situation where error sequences were generated using a narrowband mobile radio channel simulator. The channel condition was defined by the Jakes model [10] with the following parameters: carrier frequency = 900 MHz, vehicle speed = 100 km/h, number of low-frequency oscillators = 8, and average SNR = 10 dB. Using this model, we simulated a series of error sequences for optimum differential phase-shift keying (DPSK) modulation at a data rate of 20 kb/s. Table 3 presents the results obtained from transmission of VQ data over the Jakes

model. In using MMSE1 and MMSE2 decoders, the channel transition probabilities have to be combined with a priori knowledge of Gilbert model parameters which can be estimated once in advance using the gradient method [7]. Simulation results indicate that the proposed techniques used to refine the channel transition probabilities can match the decoder design to Jakes model and that improved performance requires better modeling of channel memory using the MMSE2 decoder. 6.

Conclusions

This study presented memory-enhanced extensions of MMSE decoding for vector quantization over noisy channels. We first emphasized the importance of matching the real channel behavior to the channel model on which the decoder design is based. This task was accomplished by using Gilbert’s two-state Markov chain model to characterize the statistical dependencies in error occurrences. Also proposed is an algorithm to associate Gilbert model parameters with recursive implementation of an instantaneous MMSE decoder. Simulation results indicate that the proposed algorithm leads to an MMSE decoder with increased robustness to channel errors. Acknowledgement This study was supported by the National Science Council, Republic of China, under contract NSC 902213-E-009-110, and Nanyang Technological University, Singapore, under the Tan Chin Tuan academic exchange fellowship. References

Fig. 4 SNR (dB) performance of different vector quantizers as a function of Gilbert channel BER. Codebook size = 64, vector dimension = 3, source = 1st-order Gauss-Markov (ρ = 0.9).

Table 2 SNR performance of the VQ-MMSE2 under channel mismatch conditions:
d = design BER,
a = actual BER; ρ = 0.9, M = 16, d = 4.
a = 0.001
a = 0.01
a = 0.1


d = 0.001


d = 0.01


d = 0.1

10.1142 9.6375 6.2735

10.1119 9.6581 6.4019

9.9756 9.5587 6.5220

Table 3 SNR (dB) performance of VQ-MMSE1 and VQ-MMSE2 for Gauss-Markov sources over the Jakes model. (M, d)

System

ρ=0.5

ρ=0.9

(64,6)

VQ-MMSE1 VQ-MMSE2

3.5479 3.5648

5.5290 5.8933

(64,3)

VQ-MMSE1 VQ-MMSE2

4.9224 5.0374

6.6928 7.1777

[1] N. Phamdo and N. Farvardin, “Optimal detection of discrete Markov sources over discrete memoryless channelsApplications to combined source-channel coding,” IEEE Trans. Inf. Theory, vol.40, no.1, pp.186–193, Jan. 1994. [2] T. Fingscheidt and P. Vary, “Softbit speech decoding: A new approach to error concealment,” IEEE Trans. Speech Audio Process., vol.9, no.3, pp.240–251, March 2001. [3] N. Phamdo, F. Alajaji, and N. Farvardin, “Quantization of memoryless and Gauss-Markov source over binary Markov channels,” IEEE Trans. Commun., vol.45, no.6, pp.668– 675, June 1997. [4] L.N. Kanal and A.R.K. Sastry, “Models for channels with memory and their applications to error control,” Proc. IEEE, vol.66, pp.724–744, July 1978. [5] E.N. Gilbert, “Capacity of a burst-noise channel,” Bell Syst. Tech. J., vol.39, pp.1253–1265, Sept. 1960. [6] A.I. Drukarev and K.P. Yiu, “Performance of errorcorrecting codes on channels with memory,” IEEE Trans. Commun., vol.COM-34, no.6, pp.513–521, June 1986. [7] J.Y. Chouinard, M. Lecours, and G.Y. Delisle, “Estimation of Gilbert’s and Fritchman’s models parameters using the gradient method for digital mobile radio channels,” IEEE Trans. Veh. Technol., vol.37, no.3, pp.158–166, Aug. 1988.

IEICE TRANS. INF. & SYST., VOL.E86–D, NO.10 OCTOBER 2003

2222

[8] Y. Linde, A. Buzo, and R.M. Gary, “An algorithm for vector quantizer design,” IEEE Trans. Commun., vol.COM-28, no.1, pp.84–95, Jan. 1980. [9] W. Turin, “MAP symbol decoding in channels with error bursts,” IEEE Trans. Inf. Theory, vol.47, no.5, pp.1832– 1838, July 2001. [10] W.C. Jakes, Microwave mobile communications, Wiley, New York, 1974.

Heng-Iang Hsu received the B.S. and M.S. degrees in communication engineering from National Chiao Tung University, Hsinchu, Taiwan, ROC, in 1996 and 1998, respectively. Currently, he is working toward the Ph.D. degree in communication engineering at National Chiao Tung University. His research interests include joint source-channel coding and wireless communication.

Wen-Whei Chang received the B.S. degree in communication engineering from National Chiao Tung University, Hsinchu, Taiwan, ROC, in 1980 and the M.Eng. and Ph.D. degrees in electrical engineering from Texas A&M University, College Station, TX, in 1985 and 1989, respectively. Since August 2001, he has been a professor with the Department of Communication Engineering, National Chiao Tung University, Hsinchu, Taiwan, ROC. His current research interests include speech processing, language indentification, and secure communication.

Xiaobei Liu received the B.S degree in electrical and communication engineering from Fudan University, Shanghai, China in 1998. From 1998 to 2000, she was an engineer with Datang Mobile Communications Equipment Co., LTD. Her work was focus on implementing turbo codes in 3G communication system. She is now a Ph.D student in Nanyang Technological University, Singapore.

Soo Ngee Koh received both the BEng degree from the University of Singapore and the BSc degree from the University of London in 1979. Upon graduation, he worked as an engineer at the Telecommunication Authority of Singapore. He obtained his MSc and PhD degrees from the Loughborough University of Technology, UK in 1981 and 1984 respectively. Prior to his return to Singapore, he worked as a consultant in wideband speech and audio coding at the British Telecom Research Laboratories in England. He joined Nanyang Technological University in 1985. He is currently a Professor and Head of the Communication Engineering Division in the School of Electrical and Electronic Engineering of NTU. He has over 100 publications in international journals and international conference proceedings. He holds two international patents on speech coder design. He was a co-recipient of the IREE (Australia) Norman Hayes best paper award in 1990. His research interests include speech processing, coding, enhancement and synthesis, joint source-channel coding, audio and video coding and communication signal processing.