On the Optimum Average Distortion Attainable by Fixed-Rate Coding ...

190

IEEE TRANSACTIONS ON INFORMATION

This implies that (C8) (RI - 4 R, - 4 Ro - &E~III for all 6 > 0. SinceWnI is closed,it follows that R E WIII. This completesthe proof of the conversepart. Comment: We have been unable to upperboundfurther the right sidesof (C6) by sums of individual mutual informations. Sucha resultwould be significantand would lead to the converse of Theorem 2. The inherent difficulties we encountered were two. First, Q,,* in (C4) need not be a product channel. Second, each of the mutual informations on the right side of (C6) involved only two X-variables, whereas three X-variables are needed in each expression in order to apply independence and convert the

mutual informationsinto sums. REFERENCES [l] R. Ahlswede, “Multi-way communication channels,” presented at the 2nd Int. Symp. Information Theory, Tsahkadsor, Armenian S.S.R., 1971. [2] ;, “The capacity region of a channel with two senders and two recervers,” Ann. Prob., vol. 2, pp. 805-814, Oct. 1974. [3] R. B. Ash, Information Theory. New York: Interscience, 1965. [4] P. P. Bergmans, “Random coding theorems for broadcast

THEORY, VOL. IT-21,

NO.

2,

MARCH

1975

channels with degraded components,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 197-207, Mar. 1973. 151T. M. Cover, “Broadcast channels,” IEEE Trans. Inform. Theory, vol. IT-l& pp. 2-14, Jan. 1972. [61R. G. Gallager, Information Theory and Reliable Communication. New York: Wiley, 1968. “Coding for degraded broadcast channels,” presented at the [71 -, 3rd Int. Symp. Information Theory, Tallinn, Estonian S.S.R., 1973. 181C. E. Shannon, “A note on a partial ordering for communication channels,” Inform. Contr., vol. 1, pp. 390-397, 1958. “Two-way communication channels,” in Prqc. 4th Berkeley [91 -, Symp. Math. Statist. and Prob., vol. 1, 1961, pp. 611-644. 1101D. Slenian and J. K. Wolf. “A coding theorem for multinle access channels with correlated ‘sources,” %eZl Syst. Tech. J.; vol. 52, pp. 1037-1076, 1973. 1111M. L. Ulrey, “A coding theorem for a channel with s senders and r receivers,” to appearin Inform. Contr. 1121E. C. van der Meulen. “The discrete memorvless channel with two senders and one receiver,” presented at-the 2nd Int. Symp. Information Theory, Tsahkadsor, Armenian S.S.R., 1971. “On a problem by Ahlswede regarding the capacity region D31 -, of certain multi-way channels,” Inform. Co&r., vol. 25, pp. 351356, 1974. 1141 J. Wolfowitz, Coding Theorems of Information Theory, 2nd ed. New York: Springer-Verlag, 1964. u51 A. D. Wyner, “A theorem on the entropy of certain binary sequences and applications: Part II,” IEEE Trans. Inform. Theory, vol. IT-19, pp. 772-777, Nov. 1973.

On the Optimum Average Distortion Attainable by Fixed-Rate Coding of a Nonergodic Source JOHN C. KIEFFER

Absfruct-A simple proof is given for the Gray-Davisson result on the ergodic decomposition of the optimum average distortion attainable by fixed-rate coding of a nonergodic source, for the special case of a bounded distortion measure. A second proof is given for a more general class of distortion measures using a result of Ziv.

x = (X1,Xz,’* - J,) E Al”, Y = (Y~,YZ,**-,Y,,) E 4”. If E c B,” is a finite set and p is a source, define p,(E,p) = jA inf,,, P,(~“,Y> 4. If R > 0, let WbCL) = inf P,UL~, where the infimum is taken over all E c B,” such that the rate of E is no greater than R. (The rate of E is n-l In IEI, SECTION I where [El is the cardinality of E.) Let 6(R,p) = inf, 6,(R,p). N THIS section we discussthe recent result of Gray and The quantity 8(R,p) (which may be infinite) may be Davisson [l] on source coding for nonergodic sources. interpreted as the optimum averagedistortion attainable by We let the topological space A, be the source alphabet for coding the source ,u at the fixed rate R > 0. If p is an ergodic stationary source, 6(R,p) can be our sources,and let the measurespaceB, be the reproducing calculated using a source coding theorem and the converse alphabet. Let A be the topological space consisting of the of Berger [2, theorems 7.2.4 and 7.2.51, which imply the sequences(x1,x2, - * a) of elementsfrom A, with the product following result. topology. Let X”: A + A,” be the projection map such that X”(X1,X~; * a) = (x1,xz; * * ,x,). A source p is just a Bore1 Theorem 1: If ,u is an ergodic stationary source and if probability measure on A. Let p1 : A, x B, -+ [O,“c)) be there exists a y* E B, such that f pr(X’,y*) dp < co, then the distortion measure. (Assume p1 is jointly measurable.) 6(R,p) = D(R,p), R > 0, where D(. ,p) is the distortionFor n = 1,2;**, let pn: A,” x B,” + [O,co) be the rate function for the source ,u, which can be calculated by function such that p,(~,y) = n-l Cy=1 p,(Xi,yi), where a certain information-theoretic m inimization.

I

Manuscript received March 18, 1974; revised September 6, 1974. The author is with the Department of Mathematics, University of Missouri at Rolla, Rolla, MO. 65401.

Gray and Davisson [1] solved the problem of how to calculate 6(R,p) for nonergodic sources. They showed that under certain restrictions on A,, B,, and pl, the following

KIEFFER: FIXED-RATE CODING OF NONERGODIC SOURCE

191

holds. G iven the stationary source p, there exists for each shift transformation such that T(xl,xz, * * *) = (x2,x3,* * *). x E A an ergodic stationary sourcepX such that A function g : A + (- co,co) is said to be invariant if g(T(x)) = g(x), x E A. We need the following ergodic decomposition theorem of Rohlin, which was also used by WW = 1 WUJ 444 = s WAJ 444 A A Gray and Davisson in their paper [l]. R > 0. (1) Theorem2: Let p be a stationary source. Then for each (See [1, theorem 4.1) In the next section, we give a simple x E A, there exists a stationary ergodic sourcepX such that : proof of the Gray-Davisson formula (1) for the special i) for each bounded Bore1measurablef: A -+ (- CXJ,CO) case of a bounded distortion measure. It is a strong the function g : A + (- co,co), such that g(x) = assumption to assume one has a bounded distortion j f dp,, x E A, is Bore1measurableand invariant; measure since many distortion measures of practical ii) with f and g as in i), s hf dp = J hg dp, for every importance are unbounded. The proof of the Graybounded invariant Bore1measurablefunction h: A + Davisson formula given in [l] is more general than the (- CvfJ). proof we are going to give, in that the proof of Gray and Davisson applies to unbounded as well as bounded The following is what we wish to prove. distortion measures. Theorem3: If ~1is a stationary source and R > 0, then Before proceeding to the next section we prove two 8(R,pJ is a Bore1 measurablefunction of x and 6(R,p) = useful lemmas. Throughout the rest of the paper let Y be !A Wbx) &(X). the collection of all stationary sources. Lemma I: a) If E, c BIni, i = 1,2, and n, + n2 = n, To prove this, we will apply the following result, which then follows from Rohlin’s theorem using methods developedby Jacobs [3, pp. 38-40]. (In the following if {p,} is a sequence of sourcesand p is a source,we say that pL,+ p, if and only if Jf dk + Jf dp, f or every bounded continuous function b) If there exists y* E B, such that 1 p,(X’,y*) +(x) = f: A + (- cqco).) p* < co, where p E Y, then 8(R,p) 5 p*, R > 0, and Theorem4: Let F: 9’ + (- co,co) be a bounded function lim,, g) 8,(R,p) exists and equals 8(R,p), R > 0. such that: Proof: Part a) is easy to show and is left to the reader. a> f’G M l + A2~2) = V’W + A2F(~2L LA2 > 0, If the sets E, and E2 of part a) have rates no greater than PlJJ2 E 9; R, then E, x E, has a rate no greater than R. Taking an b) F is upper semicontinuous; that is, K + p implies infimum of the equation in part a) over such pairs (E1,E2), that F(p) 2 lim, sup F(pJ. we see that n&(R,p) I CF=, n,&,.(R,p). Also since the sequence{6,(R,p)} is bounded by p;, we apply a lemma of G iven p E 9, let {pX: x E A} be the ergodic sourcesgiven Gallager [4, p. 1121to conclude that lim, 8,(R,p) exists and by Theorem 2. Then F(pJ is a Bore1 measurablefunction equals 8(R,p) and thus that 6(R,p) I p*. of x and F(P) = fA F(A) 444. Therefore, to prove Theorem 3 we need only show that Lemma 2: Suppose there exists y* E B, such that the hypotheses of Theorem 4 hold with F( *) = 6(R, *). 1 pl(X’,y*) &(x) = p* < co, where p E 9. Then 6(R,p) This is what we do in the following lemmas. is a convex and, therefore, continuous function of R, Lemma 3: 6(R,1+, + /Z,/J~)= Xi”= 1 liS(Rypi), RJ,,& > R > 0. o;rl, + 12 = l;p&zE9. Proof: Let RI > R, > 0. Let R = C&, ~iRi, where Proof: Let &,Iz, > 0, h, + A2 = 1. Let rzl = [an,], the greatest integer in nl, and n2 = n - nl. For i = 1,2, if Ei c BIni ~ = i: li~i. i=l has rate no greater than Riy then E, x E2 has a rate no greater than R. Taking an infimum of the equation in Now Lemma la) over such pairs (E,,E,), we seethat n&(R,p) I E c B1”. P~(E,PL)= iil APn(Ed CL, niSnl(Ri,p). Dividing by n and letting n --f 00, we see that B(R,p) I CF=1 li6(R,~). Thus 6( * ,p) is convex and, Therefore, &(R,p) 2 Cf=, ~,8,(R,CLi).Letting n + CO,we therefore, continuous for positive rates. obtain S(R,p) 2 CF=“=,iliG(R,~i). Conversely, fix R’ > R. For n sufficiently large, if E,,E, c B,” have rates no II greater than R, then E, u E2 has a rate no greater than R’. In this section we assumethat the topological space A, Therefore, for it sufficiently large, is a metric space, a Bore1 subset, and a subspace of a complete separable metric space. We suppose that the distortion measure p1 is bounded (0 I p1 < C, for some nonnegative real number C) and that pl(*,v) is a continuous function for each y E B,. Let T: A + A be the SECTION

192

Taking the infimum over E, and E, and then letting n --f co, we obtain 6(R’,p) I x.i”=, li6(R,pu,). If we let R’ + R, then 6(R’,p) + B(R,p) by Lemma 2. Thus we conclude that 6(R,p) < C$, ,li6(R,pi), and then 6(R,p) = Ci”=1 A,&RPi)*

IEEE TRANSACTIONS ON INFORMATION

THEORY, MARCH 1975

7’. Thus inf g(Q) d&(s) = inf Let) G!Jw tsDrns F,, t~Grns F,

where D, is a countable dense subset of G,,,. Thus Lemma 4: Let {p,,} be a sequencein Y and let p E 9. inft~om SF,g&t) d&,(s) is a measurable function of w. Then p, + p implies that B(R,p) 2 lim , sup 8(R,,u”), Now R > 0. g(U) dL(s) as n + 00. g(s,t) his) t ss s Fn Proof: We supposethat p,, + CL.Fix E > 0. Pick m large and then E c B,“’ with a rate no greater than R so that Therefore , 6(R,p) 2 p,(E,p) - E. Since inf,,, P&!?,Y): A + (- qco) is a bounded continuous function, g(s,t) dl,(s) as n + 03. inf g(O) &is) t inf fEGms ,y teGms F, inf p,Gm, Y> dcl,, --f inf P,G”, Y) 4~. (If (h,} is a nondecreasingsequenceof continuous functions s Y~E s Y~E on a compact set K, then lim , (inf, h,) = inf, (lim, h,).) That is, p,(E,p”) + p,(E,p). Then for n sufficiently large, Thus inf,,, Js g&t) d&(s) is a measurable function of w. m 2 o3, we see that inf,,, Is g&t) d&,,(s) is a 6(R,p) 2 p,(E,p,) - 2~ 2 6(R,p”) - 2~. The lemma follows Letting from this. measurablefunction of w. We observe now that for fixed R > 0, the function Lemma 7: Let {pu,: x E A} be the ergodic decomposition F: 9’ + (- co,co) such that F(p) = 6(R,,u)is bounded and of p E 9. Then 6(R,p,) is a Bore1measurablefunction of x. satisfies the other hypothesesof Theorem 4. Therefore, the Proof: For each fixed ~1,it sufficesto show that 6,(R,,uL,) Gray-Davisson formula of Theorem 3 follows. is Bore1 measurable. Apply Lemma 6 with S = Al”, T = A,” x A,” x * * * x A,” (the Cartesian product of SECTION III [enR] copies of A,“); g(s,t) = inf, &,tJ, where for each In this final section of the paper we present an alternate i = 1,2,. . .,[enR], ti is the ith coordinate oft E T; W = A; for w E A, A, = pwn, where pwn is the projection of pL, to Proof Of Tlmx-em 3 Using a result Of ZiV [5]. we now assumethat the topological spaceA, is a metric spacewith A,“. Then inf, ss g(s,t) d&,(s) = i$(R,p,,,). metric d such that every bounded closed subset is compact Lemma 8: Let {/.L~:x E A} be the ergodic decomposition and that A, = B,. Let f: [O,co) --f [O,co) be a continuous of p E 9’. Then &R,p) 2 J @R,pJ dp(x), R > 0. nondecreasingfunction such that lim ,, m [f(x - a)/f (x)] = 1, for every a > 0. Define the distortion measure pl: Proof: Let E c A,” have a rate no greater than R. AI x Al --) P,~) so that P~(x,Y) = f(d(x,y)), x,y E A,. Applying Lemma 5, we seethat (Observe that, in contrast to Section II of this paper, the p1 just defined may be unbounded.) P,(J%P) = P,EAJ 44) 2 WU-4 44x). s s Since A, is a complete separable metric space, the Theorem 2 due to Rohlin holds. Given p E Y, let Taking the infimum, we conclude that {CL,:x E A} be the ergodic sources given by Theorem 2. A generalized Fubini theorem [6, theorem 2.6.71 implies the WL4 2 WQx) &L(x). s following lemma. Lemma 5: Let g be a nonnegative Bore1 measurable To obtain the reverse inequality, we need the following function defined on A. Then fA g dp, is a Bore1measurable result due to ziv ~5, theorem 11. function of x defined on A (possibly infinite for some x). Theorem5: Given R > E > 0, we may tind for each n a Also, JA [SA 9 &I 444 = JA 9 &. Lemma 6: Let S,T be metric spaces such that every set -% c Ar”p of rate no greater than R, such that for every closed bounded subset is compact, Let W be a measure CLE 9, space such that for each w E W, 1, is a Bore1 probability 6(R - E, p) + E 2 lim sup p,(E,,p). ” measure on S. We suppose that Jsf d& is a measurable function of w for each nonnegative Bore1 measurable Theorem 6: If {pX: x E A} is the ergodic decomposition function f defined on S. Let g be a nonnegative function of p E 9, and if there exists y* E A such that defined on the product topological space S x T. Assume g is continuous. Then inf,,, Js g&t) d&(s) is a measurable PIG~,Y*) & < ~0 function of w. s Proof: Let {F”},{G,} be sequencesof compact subsets then of S,T, respectively, such that F” t S, G, t T. Fix m ,n. For fixed w E W, SF,g&t) d&,,(s) is a continuous function on

KIEFFER: FIXED-RATE

193

CODING OF NONERGODIC SOURCE

Proof: Fix R > 0. Pick E > 0 so that R > E. Pick R’ > R. Let yn E A,” be the pointy, = (y*,y*; * *, y*). Let En’= E, u {y,}, where {E,,} is the sequence of sets in Theorem 5. Now for every 1 E 9, Theorem 5 implies 6(R - E, A) + E 2 lim sup p,(E,J) 2 lim sup p,(E,‘,;i). ”

n

Replacing ,I by pX and integrating, we conclude that E + /a@

- E, A,> 44x) 2 /liF

Now sup,,p,(E,‘,& s iI.f P~(X~,Y*)

sup P,(&‘,&)

dp(x).

I 1 pl(X1,y*) dp,. Since by Lemma 5,

&I 444 = .f P,W’,Y*)

&

< ~0, we see

that sup,,p,(E,‘,p,) is integrable with respect to p. Thus by Fatou’s lemma [6, p. 481, lim

s n

SUP

PG’J~

supe6(R - E,~J < S p,(X’,y*) with respectto p. Therefore, by theorem [6, p. 491, Ilim 6(R - E, p.J dp(x) = e+O J

dp, and is thus integrable the dominated convergence

P lim 6(R - E, p,) d&x)

J &+0

=

WA) 44x). s We then have j 6(R,pJ dp(x) 2 6(R’,,u). Letting R’ J R, we see that S 6(R,pJ dp(x) 2 8(R,p). We conclude that s d(R,p.J dp(x) = 6(R,p). For almost every x with respect to P, J pl(X1,y*)

&,

< cg and thus WUJ

= W&J

by

Theorem 1. Thus

d&) REFERENCES

2 lim SUP P,(J%‘,I~ n s

d4x)

= lim sup p,(E,,‘,p) 2 8(R’,p). n

(The last inequality follows because E,,’ has a rate no greater than R’, for n sufficiently large.) Therefore, E + s 6(R - E&J dp(x) 2 6(R’,p). Since s p,(X’,y*) dp, is integrable with respect to p, J pl(X1,y*) dp, < 00, for almost every x with respect to p. For these x, 6(. ,p.J is continuous for positive rates by Lemma 2. Also,

[l] R. M. Gray and L. D. Davisson, “Source coding theorems without the ergodic assumption,” IEEE Trans. Inform. Theory, vol. IT-20, pp. 502-516, July 1974. [2] T. Berger, Rate Distortion Theory. Englewood Cliffs, N.J.: Prentice-Hall,.. 1971. [3] K. Jacobs, “Uber die struktur der mittleren entropic;,” Math. Z., vol. 78, pp. 33-43, 1962. [4] R. G. Gallage?, Znformation Theory and Reliable Communication. [5] pzipF: WlleY~ 1968. Codmg of sources with unknown statistics-Part II: l%storfion relative to a fidelity criterion,” IEEE Trans. Inform. Theory, vol. IT-18, pp. 389-394. May 1972. [6] FG7;. Ash, Real Analysis and Probability. New York: Academic,