A Method for Proving Multiterminal Source Coding

IEEE TRANSACTIONS

ON INFORMATION

THEORY,

VOL.

IT-27, NO.

5, SEPTEMBER

those of constructing good TDC and Abelian codes more efficiently and of decoding them.

1981

565

131 C. L. Chen, W. W. Peterson, and E. J. Weldon, Jr., “Some results on auasi-cvclic codes.” Inform. Contr.. vol. 15. DD.407-423. 1969.

and rational [41 G. E. ~Collins, “Cornmiter algebra of p&nomials functions.” Amer. Math. Mon.,-vol. 80, no. 7, pp. 725-755, 1973. [51 H. .I. Heleert and R. D. Stinaff. “Minimum-distance bounds for binary h&r codes,” IEEE Tran;. Inform. Theory, vol. IT-19, pp.

ACKNOWLEDGMENT

344-356, May 1973.

The author wishes to express his sincere thanks to Prof. M. Iri of the University of Tokyo for valuable advice and encouragement. Thanks are also due to Prof. T. Date of Hokkaido University for his help in programming the algorithm. During the course of this work we made use of the HITAC 8800 * 8700 computer at the University of Tokyo. REFERI~N~E~

S. D. Berman, “On the theory of group codes,” Cybernetics, vol. 3, no. 1, pp. 31-39, 1967. [2] S. D. Berman, “Semisimple cyclic and Abelian codes,” Cybernetics, vol. 3, no. 3, pp. 21-30, 1967. [I]

PI G. Hermarm, “Die Frage der endlich vielen Schritte in der Theorie

der polynomideale,” Mathematische Annalen, Bd. 95, pp. 736-788, 1926. [71 T. Ikai, H. Kosako, and Y. Kojima, “Basic theory of two-dimensional cyclic codes-Generator polynomials and the positions of check symbols,” IECE Trans., vol. 59-A, no. 4, pp. 31 l-318, 1976. R-U H. Imai, “A theory of two-dimensional cyclic codes,” Inform. Contr., vol. 34, pp. 1-21, 1977. ]91 F. .I. MacWilliams, “Binary codes which are ideals in the group algebra of an Abelian group,” Bell Syst. Tech. J., vol. 49, pp. 987-1011, 1970. [lOI S. Sakata, “General theory of doubly periodic arrays over an arbitrary finite field and its applications,” IEEE Trans. Inform. Theory, vol. IT-24, pp. 719-730, Nov. 1978. [ill J. M. Stein, V. K. Bhargava, and S. E. Tavares, “Weight distribution of some ‘best’ (3m, 2m) binary quasi-cyclic codes,” IEEE Trans. Inform. Theory, vol. IT-21, pp. 708-711, Nov. 1975. [ 121 B. L. van der Waerden, Modern Algebra, vol. II (F. Blum, English transl.). New York: Ungar, 1949.

A Method for Proving Multiterminal Source Coding Theorems JOHN C. KIEFFER

A bsfruct- The two-step method used by Wyner and Ziv to prove the Wyner-Ziv theorem is extended to prove sliding-block source coding theorems for coding a general finite-alphabet ergodic multiterminal source with respect to a single-letter fidelity criterion. The first step replaces every stochastic encoding in the network by a deterministic sliding-block encoding. The second step involves using the Slepian-Wolf theorem to adjust the sliding-block encodings so that they will have the desired rate, while introducing little additional distortion. The method is applied to give quick proofs of sliding-block versions of theorems of Berger, Kaspi, and Tung. Since the method obtains sliding-block coders directly without first obtaining block coders, the block coding results can be obtained as an easy corollary. The direct methods for proving results on block coding are more difficult and do not imply the corresponding results on sliding-block coding. Indeed, in the multiterminal case it is, in general, not known how to construct good sliding-block coders from good block coders.

Manuscript received March 10, 1980; revised January 12, 1981. This work was supported by NSF Grant MCS-7821335 and the Joint Services Electronics Program under Contract NOOOl4-79-C-0424. The author is with the Department of Mathematics, University of Missouri- Rolla, Rolla, MO 65401.

L

ET X, Y be jointly stationary and ergodic processes. Let the stationary process Z be a stochastic encoding of X. Suppose some block decoding of (Z, Y) yields an estimate of X to within the distortion level D. The WynerZiv theorem says that if R > I( X, Z 1 Y), then a block encoding Z of X at a rate less than R can be found so that (2, Y) yields an estimate of X within the level D by means of an appropriate block decoding. Wyner and Ziv prove their result [l l] in two steps. First, they find a block encoding U of X so that H(U] Y) = 1(X, UJ Y) 1, one attempts to convert a block encoding of each sub- If X is a discrete random variable, H(X) denotes its source Xi into a sliding-block encoding by inserting ran- entropy. If W is a stationary process H(W) denotes its dom “punctuation” in the Xi sequence and then using the entropy rate, defined by lim,, cf n - ‘H( W,, W,, . * . , W,). If Y is another process jointly stationary with W, H( W 1 Y) block encoder to encode the pieces of the Xi sequence lying denotes the conditional entropy rate of the process W between successive times where the punctuation has been inserted. However, in order for the decoder to operate, it given the process Y, which is equal to H( WY) - H(Y); also I( W, Y) denotes the mutual information rate of the would appear that the encoders of the separate subsources must insert their punctuation in the same positions, but processes W and Y, which is equal to H(W) - H(WI Y). If Z is a third process such that W, Y, Z are jointly since the encoders cannot cooperate, this is impossible. stationary, I( W, Y 1Z) denotes the conditional mutual inThis difficulty was overcome in noiseless multiterminal source coding [6] to obtain a sliding-block version of the formation rate of the processes W and Y given the process Z, which is equal to H(WI Z) - H(WI YZ). Slepian-Wolf theorem. We overcome this difficulty in this If w’; **,Wk are processes, the notation W’ + W2 paper for non-noiseless multiterminal source coding by --) . . . -+ Wk will denote that the processes form a Markov constructing the required sliding-block codes directly chain (in the indicated order). If {Xk}Tz’ is a sequence of without first finding block codes.

KIEFFER:

MULTITERMINAL

SOURCE

CODING

567

THEOREMS

processes with state space A and X is a process with state space A, the notation Xk 4 X denotes convergence in distribution, that is, Pr [Xk E E] + Pr [X E E], for every finite-dimensional cylinder subset E of A”. We call a process U with state space A aperiodic if Pr [U = U] = 0 for every u E Am. The following two lemmas allow US to accomplish the first step of our method. Their proofs are given in the Appendix. Lemma 1: Let

X, Y be jointly stationary processes, where X is aperiodic with state space A and Y has state space B. Then there exists a sequence of sliding-block

spectively;

tionary codings of Y such that YW” 4 YW and H(Y) W”) and for any sliding-

blockcodes +,, +*, Gg, hp4,+55,

a> H(htXYW”) I W”) + Ht+,(XYW) I W>, b)

~(cp,(XYW”),

+2(J3’w”)

q+,(xym c)

~(+oo

wn+*(XyWn)

~(4dXY)~

I W”+&XYW”))

--$

I wJ,wm I44xy)) +

(P*(xyw)

wJ,(xJw

l&w).

The following result, proven in the Appendix, is a version of the Slepian-Wolf theorem for stationary codes. It is used to accomplish the second step of our method. In the following, if X’, . . . , XN are processes and S is a nonempty subset of { 1,2,. . *,N}, X’ denotes the process Xjl . **Xjk where S = {j,, . . . , ji} and j, < **. < jk. SC denotes the complement of S(in { 1, **. , N}). Of course, if S is null (S = +), then by I(U, V) X’) we mean I( U, V) and by H( U 1Xs) we mean H(U). Theorem 1: Let X1; **,XN be jointly

stationary and ergodic processes defined on the probability space a. Let R,; . ->RN be positive numbers such that a)

H(Xs

[Xs’)


> --$H(Xl Y>.

that

4

I? < Xi andH(I?)

d)

Xi < 6”-4$‘I”(Di),

such that i = 1,e.o,N,

< Ri,

i = I,-*-,N.

Remarks: Because of the definition of the “< ” relation, the encoders and decoders in this theorem are stationary codes. If they are not sliding-block codes, by [3, theorem 3.11 they may be replaced by sliding-block codes. The block coding version of Theorem 2 was proved by Berger [l] and Tung [lo] for the case N = 2, X1X2 W’ W* an independent process. Proof: CaseI,AIlH(X’)=O: TakeJ$“‘=X’,i=l;*.,N. Case 2, All H(X’) > 0: For each i = 1; * - , N, X1 * * * . . XNW’ . . . wi-lwi+l . WN -+ Xi -+ W’. Hence by Lemmas 1 and 2, we can successively replace each W’ by a

process w”, where 1) 2)

J@I’< xi, i = 1;. .,N, conditions a), b) of Theorem 2 hold with the {Wi} replaced by the {pi}.

Condition b) for the {wi} reduces to 3)

H(@‘s

I @I”‘)
0: Set Tl = { 1 5 i 5 N: H( Xi) = O}, T, = { 1 I i I N: H(X’) > O}. Let S c T2. We have that 1(X’, WT1 I WT2-‘> = ~(xs, WTI 1xT~WTzps) = 0, since WTl + XT~WT~-s +

X’ and H(X’l)

= 0. Hence

z( xs, ws ( xT’w~-S) We will now prove the following theorem using our method. Theorem 2: Let X1; . a, XN, W’;

*a, WN be jointly stationary processes where X1, **a, XN are jointly ergodic and w’,. . . , WN are stochastic encodings of X1,. . . , XN, re-

I I( xs, wSwT1 1 w-) = I( xs, w=1 ) w-s) -=c zRi.

iES

+ I( xs, ws 1wq

568

IEEE TRANSACTIONS

Using Lemmas 1 and 2 much as in Case 2, we can successively replace each W’, i E T,, by a Wi < Xi so that H(l@’ I XT@IT2-s)


I(Y,WlX),

a)

R, ‘Z(X,Ul

R, >I(Y,VIUW) VW),

R, + R, + R, > I( XY, UVW)

b)

X
H(U(

Vti)

R, > H(VI

VI+).

= H(UV’I ?+‘)

By Theorem 1, pick I?, 920 that I%< U, f < V, H(c) < UV < UVW. It is easily verified that R,, H(V) < R,, c)- e) hold. Subcuse C, H(W I X) < R, < H(W): Find 6’ < W with H(c) = R, and W < XW. (First find by Theorem 1 aW< WwithH(W) H(UI VW) = H(UW(

VW@

R, > H(V(

UWti).

VW)

= H(VWI

Applying Theorem 1, there are 0, P such that i? < VW, t< VW, H(c)< R,, H(P) < R,, and UVW< I%%. Thus d) and e) hold. Note that fi < UW < XW < X?& Also, P < VW < Y, and so c) holds. APPENDIX

-Fix the finite sets A, B. Let the processes X, Y be the projections from Am X BM onto Am, B”, respectively. Let the process Y denote the identity map on Bm. By a channel [A, Y, B] we mean a collection Y = {vx: x E A”‘} of probability measures on Bm such that the map x --) v,(E) is measurable on A” for each measurable E C B”. If p is a measure on Am, let pv denote the measure on Am X B” such that

for each measurable E c A” and measurable F c Bm. In a subsequent proof, we will need to know some types of channels, which we now define. 1) [A, v, B] is stationary if vT.,(7”E) = v,(E) for all x, E.

KIEFFER:

MULTITERMINAL

SOURCE

CODING

569

THEOREMS

2) [A, v, B] is an n&order discrete memoryless channel (DMC) if there exists for each x E A” a measure $ on B’ such that for are indeeach x E Am, the variables _ 0. Proof of Lemma 1: Case 1, (X, Y) Ergodic: For each m, 6, E, by the source coding theorem for context dependent fidelity criteria [2, theorem 11, there is for n sufficiently large a block code +: A” - B” with n-‘H(+(X,;.. ,X,)) < Z( X, Y) + c, such that the frequency of each (a, b) E A” X B” in each (x,+(x)) is within 6 of Pr [(X,3.. . ,x,> = a, (Y,,. . . ,Y,) = b]. Randomizing over the starting time of the block code, we see that, redefining X and Y on a new probability space if necessary, there is a process Y jointly stationary with X such that a)

H(Y)

< Z(X, Y) + e, and

b)

(Pr [(X,;..,X,,,) -Pr[(X,;..

=a, ,X,)

(Y,;..,Yj)

= a, (Y,;..,Ym)

= B] = b] I< 28,

for every (a, b) E A” X B”. By [5, lemma 61 there is a slidingblock coding-of X, which jointly with X, yields a pair-process as close to (X, Y) in distribution as we desire. Since entropy rate is upper-semicontinuous with respect to convergence in distribution, we can assume in a) and b) that Y is some sliding-block coding of X. Hence there is a sequence {Y”} of sliding-block codings of X such that XY” $ XY and lim sup,, MZ( X, Y”) = h

SUP,+ mZZ(Y”)

tional

entropy

fim SUP,+ mzf(XI

stationary n-Markovian channel such that for each i E Z and x E Am, (E; . . ,z+n) has the same distribution under v,” and v,. Then (Al) holds, and each [A, v”, B] is a finite state indecomposable channel, and therefore ergodic. We can now appeal to Case 1. Case 3, X Nonergodic: Let the measure ~.t on A” be the distribution of X and let [A, v, B] be a stationary channel such that the distribution of Y conditioned on X is given by v. Given 6 > 0 and finite-dimensional cylinder subsets E,, . . . , Ek of Am X B”, there must exist a stationary code +: Am --f B” such that -IfL(~l+m) - H7ytxI y) I,

(A51

for every sliding-block code 9,. Consider the equation ff(cJ,(XYW”)

I W”) = H( XYW” 1 wn) - H(XYW”

1 W”l#l,( xuwn)).

If we take the lim sup of the left side and the lim inf of the right side, we can apply (A4), (A5) to conclude part a) of Lemma 2. Parts b) and c) follow because whether @= W or W”, ZL+ ,( XYF), (p,( XYw) ( w+,( XYW)) and I( +4( XY), W+,( XYW) I +5( XY)) can be written as linear combinations of terms of the following two types: ZZ(F( XY) ) G(XY)) and H( F( XYW) I W). Lemma 3: Let X, Y be jointly stationary and ergodic processes and let ZZ(XI Y) < 6. Then there exists a process U < X such that H(U) < 6 and X < UY. Proof: Pick positive e,, e2, . . . so that Zp3,,c, < 6 and H( X I Y) < e,. Applying repeatedly the sliding-block version of the Slepian- Wolf theorem [6, theorem 11, pick processes v’, v2, . . . < X so that H( V’) < er and

H( Xs I v’ . . . l”Y)

< ei+, 7

i= 1,2;...

Let V be the process with infinite state space such that v, = (v,‘,vy,

-.),

i E Z.

570

IEEE TRANSACTIONS

Then H(V), the Kolmogorov-Sinai entropy of V, is lim,,, H(V* . ~1V”) 5 Zy= IH( Vi) < 6. By Krieger’s generator theorem [8] there is a process U with finite state space isomorphic to V(i.e., ZJ < V and V-C U). Hence H(X, 1 VU) = H(X, 1 W) = lim “-CC H(X, 1 v’ -. . v”Y) = 0 and so X < UY. Also H(U) = H(V) -C 6. Since V-C X, we must have U < X. Proof of Theorem 1: Find a positive R: ( R,, i = 1; * . , N, so that condition a) of Theorem 1 still holds with the {Ri} in place of the {Ri}. By [6, theorem l] find for each i a process Vi < Xi so that H( Vi) < R: and H( Xi 1 V’ . . . VN) < Ri - R:. By Lemma 3, find U’ < Xi so that Xi < U’V’ . . . VN and H(U’) < Ri - R:. Set W’ = U’V’ for each i. Then conditions b) and c) hold.

ON INFORMATION

THEORY,

VOL.

IT-27, NO.

5, SEPTEMBER

1981

denendent fidelity criteria,” IEEE Trans. Inform. Theory, vol. IT-18, pp: 378-384,~1972. ,131 R. M. Grav. “Slidine-block source coding” IEEE Trans. Inform. a Theory, “01: IT-21, pp’. 357-368, 1975. -’ [41 A. Kaspi and T. Berger, “Rate-distortion for correlated sources with partially separated encoders,” to appear in IEEE Trans. Inform. Theory. [51 J. Kieffer, “On the transmission of Bernoulli sources over stationary channels,” Ann. Prob., vol. 8, pp. 942-961, 1980. “Some universal multiterminal source coding theorems,” ZnWI -, form. Contr. to appear. [71 J. Kieffer and M. Rahe, “Selecting universal partitions in ergodic theory,” Ann. Prob., vol. 46, pp. 93- 107, 1980.

PI W. Krieger, “On entropy and generators of measure-preserving transformations,”

Trans. Amer. Math. Sot., vol. 149, pp. 453-464,

1970. [91 P. C. Shields and D. L. Neuhoff, “Block and sliding-block source coding,” IEEE Trans. Znform. Theory, vol. IT-23, pp. 211-215, 1977.

REFERENCES

WI S. Y. Tung, “Multiterminal source coding,” Ph.D. thesis, Cornell [l]

T. Berger, “Multiterminal source coding,” in The Znformafion Theory Approach to Communications, G. Longo, Ed., CISM Courses and Lectures #229. New York: Springer-Verlag, 1978. [2] T. Berger and W. C. Yu, “Rate-distortion theory for context-

University, Ithaca, NY, 1977.

1111 A. D. Wyner and J. Ziv, “The rate-distortion function for source

coding with side information at the decoder,” ZEEE Trans. Znform. Theory, vol. IT-22, pp. I- 10, 1976.

Optimal Coding in Two-User White Gaussian Channels with Feedback KENKO UCHIDA

Abstract-The optimal coding problem in two-user white Gaussian channels with feedback is discussed. The messages are taken as GaussMarkov processes. The optimal decoder pair and the optimal linear encoder pair are developed. A nonlinear class (additive feedback type) of encoder pairs in which the optimal linear encoder pair is optimal is presented.

I.

INTRODUCTION

T

HIS PAPER deals with the problem of optimal coding in two-user white Gaussian channels with feedback. Both channels are multidimensional continuous white Gaussian channels and have a common noiseless feedback link. Two messages to be transmitted separately through the two channels are jointly Gauss-Markov processes described by the multidimensional linear stochastic differential equations, and the transmission interval is taken as Tc

00.

Some special cases of this problem have been reported. The scalar message and scalar single-channel cases have Manuscript received June 18, 1980; revised September 22, 1980. The author is with the Department of Electrical Engineering, Waseda University, Okubo 3-4-1, Shinjiku, Tokyo 160, Japan.

been almost exhaustively studied in [l], [2]. Extensions to vector variable cases have been reported in [3], [9], [lo], [ll]. In particular, Baqar [ll] showed that for a GaussMarkov vector source the optimal encoder structure is nonlinear and dictates a time-sharing use of the channel. Recently, the scalar two-channel case has been discussed in [4] where one of the two channels, however, is used only as a fixed transmission line for side information. Our generalization consists of the coding problem with two encoders and two decoders in vector two-user channels with vector sources. In this paper, the decoding errors of messages and the power constraints of channels are estimated in the sense of the matrix inequality order; the problem is reduced to a matrix minimization under matrix inequality constraints. First, the optimal decoders are derived. Next the optimal linear encoders are developed and their implications are discussed. Last, we present a nonlinear class (additive feedback type) of encoders in which the optimal linear encoder pair is optimal. To clarify the fundamental idea of our method we treat only the two-user case, but the extension to the multiuser case is straightforward.

001%9448,‘8 l/0900-0570$00.75

0198 1 IEEE