Strong converses in source coding relative to a ... - Semantic Scholar

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 2, MARCH 1991

257

Strong Converses in Source Coding Relative to a Fidelity Criterion John C. Kieffer, Senior Member, IEEE

Abstracf-Two strong converses are obtained for an abstract alphabet stationary ergodic source coded relative to an appropriate fidelity criterion. It is shown that given a distortion rate point ( D , R ) , which lies below the rate distortion curve, 1) block codes that operate at rate level R must encode sample source blocks at a rate exceeding D with probability tending to one as block length tends to infinity, and 2) variable-rate codes that operate at distortion level D must encode sample source blocks at a rate exceeding R with probability tending to one as block length tends to infinity. The previously known weak converses guaranteeonly that the indicated probabilities remain bounded away from zero as block length tends to infinity. The proofs of the strong converses involve sample converses in source coding theory that are presented in a companion paper. Index Tenns -Block and variable-rate source coding, coding theorem, strong converse.

lim Pr min d,((~,,*..,~,),y) S D ] =o,

I. INTRODUCTION

n+m

T

HIS PAPER concerns strong converses that arise when one codes a stationary ergodic information source relativb to a fidelity criterion. In this introduction, we discuss the strong converses that shall be obtained in the subsequent sections. First, we present some historical background. The first strong converse for stationary ergodic sources arose in 1953, in the context of noiseless block source coding. It is due to Khinchin [2] and McMillan [6], and states the following: Let XI, X,, . . . be the sequence of random outputs generated by a stationary ergodic finite-alphabet information source p. Let A be the source alphabet and let B, be a subset of A" (n 2 1) such that

& n-'log,(B,I < H,

(1)

n+m

where !B,I is our notation for the number of elements in the set B, and H is the entropy rate of the source p , given by

Then, lim Pr [( X , ; . . ,X,)

n+m

E

B,]

McMillan [6] proved the AEP for a stationary ergodic finitealphabet source.) In view of the form of the strong converse for noiseless block source coding stated above, it is not hard to infer what form a strong converse would take for block source coding relative to a fidelity criterion. (In the ensuing discussion we assume the reader is familiar with the terminology of rate distortion theory; we formally introduce this terminology later in the introduction.) Let A be the source alphabet (not necessarily finjte), let { X , }be the random sequence of source outputs, let A be the reproduction alphabet, and let {d,} be the assumed fidelity criterion. A strong converse for block source coding allows one to cpnclude that

= 0.

(2)

(Properly speaking, the arguments in Khinchin [21 yield (2) under Assumption (1) in case p is any finite-alphabet source satisfying the asymptotic equipartition property (AEP); Manuscript received May 23, 1989; revised August 15, 1990. This work was supported by NSF Grant NCR-8702176. The author is with the Department of Electrical Engineering, University of Minnesota, 200 Union Street SE, Minneapolis, MN 55455. IEEE Log Number 9040746.

whenever

[

yet?,

(3)

-

lim n-'log,IB,I 0 and let

-

lim r * ( B , ) < R ( D ) ,

(7)

n+m

where B, is an nth-order block code (n 2 1). Then lim Pr [ d ( X"IB,) ID] = 0. n+m

(8)

Remdrk: It is not hard to show, using the definition of rate distortion function, that -

lim Pr [ d( X"IBn) ID] < 1

(9)

n-m

follows from (7) under restrictions 3) and 4). (The simple proof is given in Appendix A.) The result which states that (9) follows from (7) is the weak conuerse for block source coding. Clearly, the strong converse for block source coding implies the weak converse for block source coding. Unlike 3) If n, m arc positive integers and x1E A", y , E k,x2 E the weak converse, the strong converse does not seem to A", y , E A", then follow easily from the definition of rate distortion function. Indeed, the proof of Theorem 1 we give later in the paper dn+m((xl, ~ 2 () Y 1, ~ 2 ) ) involves the use of a new result in source coding theory 14, 1niax[d,(xl,~l),dm(~2,~2)].Theorem 21. We now discuss variable-rate source coding, culminating in 4) cor each D > 0, there exists a countable subset {y,} of the statement of the second of our strong converses, viz., the A and a countable measurable partition {E,}of A such strong converse for variable-rate source coding. An nth-order that dl(x, y,) I D, x E E, for each y,, and uariable-rate code is &a triple C = (c$,J,, SI, where S is a countable subset of A",+ is a measurable function from A" - CCLl(E,)log,PI(E,) < W . into S, and J, is a one-to-one function from S onto a set of I finite binary words satisfying the prefix condition. Given an Assuming that 3) and 4) hold, we define the rate distortion nth-order variable-rate code C = (4,J,, S), there are two function of the source p relative to the fidelity criterion {d,J important quantities that are used to evaluate its perforto be the function R(. 1 on (0, m) given by mance, viz., its distortion leuel d*(C) and its rate function r ( . IC). These quantities are defined by R( D) 2 infR,( D ) , D > 0, n

where R J D ) is the infimum of Z(U; V)/n over all A" X d. valued random pairs ( U , V ) in which U has the same distri-

d*(C) a SUP dn(x,c$(x)), XEA"

r(xlC)an-'[lengthof $ ( c $ ( x ) ) ] ,

XEA".

259

KIEFFER: STRONG CONVERSES IN SOURCE CODING RELATIVE TO A FIDELITY CRITERION

We are now ready to state Theorem 2, our strong converse for variable-rate source coding. It shall be proved in a subsequent section of the paper. Theorem 2: Let 1) and 2) hold. Let R > 0 and let

-

lim d*(C,) < D ( R ) ,

(10)

Let $, (n 2 1) be a one-to-one function from S, onto a set of finite binary words obeying the prefix condition, such that length of $,( Y ) I [log, IB,I1+ 1,

n+-

(15)

4 , ( ~ ) = 4 , * ( ~ ) , d(xIBn) I D I

R ] = 0.

n-rm

(11)

Remark: It is not hard to show, using the definition of distortion rate function, that

-

lim Pr [ r( X"IC,) I R ] < 1

= &,(

otherwise.

x),

For each n 2 1, let C, be the variable-rate code C,,= (+,, $, S,), From (13), (14) and the way in which each C, was defined, we have d*(C,) I D (n 2 1) and so, by Proposition 1, we have

(12)

Iim Pr[r(X"IC,) 2 R ( D ) - r ] = I ,

n+m

~ > 0 .(16)

n+m

follows from (10) under restrictions 1) and 2). (The simple proof is given in Appendix A.) The result that states that (12) follows from (10) is the weak converse for variable-rate source coding. Clearly, the strong converse for variable-rate source coding implies the weak converse for variable-rate source coding. The strong converse seems to be a much deeper result than the weak converse, since we were not able to prove the strong converse for variable-rate source coding via simple reasoning based on the definition of distortion rate function; instead, our strong converse given later relies on a new source coding result [4, Theorem 11. There are three remaining sections to the paper, Sections 11-IV. Section I1 is devoted to the proof of Theorem 1; Section I11 is devoted to a proof of Theorem 2; and in Section IV, we present two new source coding theorems that represent a strengthening of traditional source coding theorems via the strong converses Theorems 1 and 2.

Suppose (8) fails. Then, in view of (161, -

This section is devoted to a proof of Theorem 1, the strong converse for block source coding. The proof of Theorem 1 involves the following result ([4], Theorem 2), which tells us about the asymptotic behavior of the rate functions of a sequence of variable-rate codes along the sample sequence of source symbols X,, X2, * * . Proposition I : Let 3) and 4) hold, and let D > 0. Then lim r ( X n l C n )2 R ( D ) ,

as.,

-

whenever C, is an nth-order variable-rate code satisfying d*(C,) ID(n 2 1). Proposition 1 is a type of result termed "sample converse" in [4]. Hence, we shall be using a "sample converse" to prove a "strong converse.'' Proof of Theorem I : For each n 2 1 , let B, be an nth-order block code and suppose that (7) holds. Let +,*(n 2 1) be a measurable function from A" into B, such that XEA".

(13)

In view of restriceon 4), we may fix for eack n 2 1 ,a pair (&,,j,J in which S, is a countable subset of A" and 4, is a measurable function from A" onto S, satisfying x

A". (14) Let [log, IB,I] denote the smallest positive irfteger greater than or equal to log, IB,l, n 2 1. Define S, = S, U B,, n 2 1. E

€1 > 0,

E>O. If d(xlB,) I D, then +,(x)

E B,,

(17)

whence, from (15),

r( xlc,) I n-'[10g2 IB,I]

+ n-I.

This observation, combined with (171, yields -

lim [ n-'[log2 lBnl] + n-'] 2 R ( D ) ,

n +m

which contradicts (7). This contradiction tells us that (8) 0 must hold. The proof of Theorem 1 is now complete.

111. STRONG CONVERSE FOR VARIABLE-RATE

SOURCECODING This section is devoted to a proof of Theorem 2, the strong converse for variable-rate source coding. Theorem 2 is proved using the following result ([4], Theorem l), which is another "sample converse.'' Proposition 2: Let 1) and 2) hold, and let R > 0. Then a.s.,

lim d( X"lB,) 2 D(R ) , -

n-m

whenever B, is an nth-order block code satisfying r*(B,) IR (n 2 1).

n +-

dn(X,#,*(x)) =d(xI~n),

r ( xW,) 2 R( D )-

lim Pr [ d( X ~ I B , )5 D ,

n+m

11. STRONGCONVERSE FOR BLOCK SOURCE CODING

d , ( x , & , ( x ) ) ID ,

Bn*

Let 4, ( n 2 1) be the measurable function from A" into S, given by

where C, is an nth-order variable-rate code (n 2 1). Then Iim Pr [ r( x"Ic,)

YE

Proof of Theorem 2: For each n 2 1 let C, = (4,, $, S,) be a nth-order variable-rate code and suqpose that (10) is satisfied. Let B, ( n 2 1) be the subset of A" defined by

B,

{ 4,( x ) : x

E

IR} .

A", r ( .IC,)

Application of Kraft's inequality yields

r*(B,) IR ,

n 2 1.

Application of Proposition 2 then yields 2D , )( R ) - E ] lim Pr [ d( x ~ ~ B

= 1,

E

> 0.

n-rm

If (11) is not satisfied, then

-

lim Pr [ r( X " ~ C , )I R , d( x ~ I B , )

2

D( R ) - €1 > 0,

n+m

€ > 0 . (18)

260

IEEE TRANSACTIONSON INFORMATION THEORY, VOL. 37, NO. 2, MARCH 1991

If r(xlC,,)I R , then by definition of E,,, 4,(x)

E E,,

and

The optimum distortion level among all variable-rate codes that encode X" subject to (22) is the quantity D , ( R , E ) defined by D , ( R , E ) $inf{d*(C): C satisfies (22)}. (23)

d(xIBn) sdn(xT4n(x)) I d * ( C n ) *

This observation, coupled with (181, yields

-

lim d*( C,,) 2 D( R ) ,

n+m

which contradicts (10). This contradiction tells us that (11) must be valid, thereby completing the proof of Theorem 2. 0

IV. NEWSOURCE CODING THEOREMS In this final section of the paper, we prove two new source coding theorems that are obtained with the aid of the strong converses Theorems 1 and 2. In order to present the first of these two coding theorems, we need to make the following definition. Definition: Let E be a fixed number satisfying 0 < E < 1. Let D > 0. Suppose we wish to encode the source block X" using an nth-order block code E, so that the following constraint is satisfied:

Note that for each n 2 1, there exists a variable-rate code C,, for which r(.IC,) = n-'. The code C,, satisfies constraint (22) if n 2 R-'. Hence, the set on the right-hand side of (23) is nonempty and D,,(R,E ) is well defined, provided n 2 R-'. It thus makes sense to examine the asymptotic behavior of D,,(R,E)as n -+m; the following theorem, the second of our two new source coding theorems, accomplishes this. Theorem 4: Let 1) and 2) ,hold,*and, in addition, suppose there is a countable subset S of A such that inf d , ( x , y ) = O ,

R,( D , E ) inf [ r*( E) : E satisfies (19)). (20) Note that if restrictions 3) and 4) hold, then the set on the right hand side of (20) is nonempty, and hence R , ( D , E ) is well defined for n 2 1. We are now ready for the first of our two new source coding theorems, which tells us the asymptotic behavior of R , ( D , E )as n + W . Theorem 3: Let 3) and 4) hold. Then lim R , ( D , E ) = R ( D ) ,

D>O,

O < E < ~ . (21)

n -+m

(24)

Then lim D , , ( R , e ) = D ( R ) ,

R>0,

O < E < ~ . (25)

n-m

Proofi The converse half of statement (25) is the statement

Pr [ d( X n l B ) I D ] 2 1 - E .

(19) The optimum rate level among all block codes that encode X" subject to (19) is the quantity R J D , E ) defined by

XEA.

YES

lim D , ( R , E ) ~ D ( R ) , R > 0 , -

OR(D),

D>O,

O < E < ~ .

In order to present the second of our two source coding theorems, we need to make the following definition. Definition: Let E be a fixed number satisfying 0 < E < 1. Let R > 0. Suppose we wish to encode the source block X" using an nth-order variable-rate code C, so that the following constraint is satisfied: Pr [ r ( X n ( C )I R ] 2 1 - E . ( 22)

lim Pr [ r ( X"ICn) < R ] = 1.

n+m

(29)

Statements b) and c) yield -

lim d*( C,,) ID( R ) .

n-+m

One easily concludes from (29)-(30) that (26) holds.

( 30) 0

Remark: In view of our observation at the beginning of the proof of Theorem 4, we point out here that the weak

261

KIEFFER: STRONG CONVERSESIN SOURCE CODING RELATIVETO A FIDELITY CRITERION

converse for variable-rate source coding mentioned in the Introduction can be given the following equivalent formulation: R>O,

rD(R),

-log P(V,) and -log P ( Y l , .* *,Y,) coincide. Hence, for every n z l ,

0 R } and D is any number a) V ,= W,, whenever d(XnlB,) I D; satisfying whenever d(XnlB,) > D. b) V, = (Y1;..,Y,) (A.12) lim d * ( C , ) < D < D( R). n +m From (A.4) and the way in which each W, and V, was defined, we have Suppose (A.9) fails. Then limPr[F,] = 0 and so the last term in ( A . l l ) must tend to zeroas n --tm through a subsequence Pr [ d,( x",v,) I D ]= 1 of integers. This would give us and therefore, by definition of the rate distortion function, lim E[ d,( X " , V,)] I D. (A.13) Iim n - l ~ x"; ( v,)2 R( D). n+m (A.5) n +m Statement (A.13), coupled with (A.10), would then yield For n 2 1, let E, be the event {d(X"IB,) > D} and let F, be D( R') I D , the event {d(X"IB,) 5 D}. Then, where D ( R + ) is the right-hand limit of D ( - ) at R. Since n - ,I( X " ; V,) D(.)is convex on (0,m) [l], D(.) is continuous on (0,031 and so D ( R + ) = D(R) I D, which contradicts (A.12). This contradiction allows us to conclude that (A.9) must be valid, completing the proof. 0

[i

APPENDIX B (A4 where IF" and ZE, are the indicator functions of the events F,, E,,, respectively. On F,, the random variables -log P(V,) and -log P( W,) coincide. Hence, the first term on the right side of (A.6) is less than or equal to n-'H(W,) and therefore less than or equal to r*(B,). On E,, the random variables

Lemma BZ: Let 1) and 2) hold. Let R > O . Then there exists for each n an nth-order block code BA such that -

lim r*(B,) < R

(B.1)

n+m

and lim P r [ d ( X " I B , ) I D ( R ) + 6 , ] n-m

=1,

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 37, NO. 2, MARCH 1991

262

where (6,) is a sequence of positive numbers tending to zero.

constructed one can verify that for n 2 N,

Proot Let 6 > 0 be arbitrary. It suffices to find for each

i=l

n an’nth-order block code B,, such that (B.1) holds and

lim Pr[d(X”IB,,)s D ( R ) + 6 ] = l .

(B.2)

n-+m

i

Since D ( . ) is continuous on (O,m), we may pick R’ satisfying 0 < R’< R and D(R‘) < D ( R ) + 6. Pick A > 0 SO small that R’+ A < R and D(R‘)+ A < D ( R ) + 6. BY a Source coding theorem for fidelity criteria satisfying 1) and 2) [I, Chap. 111, there exists N and an Nth-order block code B such that

n

+(n-I

i=l

1

By the pointwise ergodic theorem, the first two quantities within braces in the preceding tend to zero almost surely as n + m, whereas the third quantity within braces tends to E[d(XNIB)].Hence

-

lim d( X”IB,,) I E [ d( X N I B ) ]< D( R ) + 6 ,

r*( B ) IR’+ A < R

\

c ~ ( ( x ~ , . . . , x ~ +. ~ - ~ ) I B )

as.,

n-+w

E [ d( X N I B ) ]ID( R’) + A < D ( R ) + 6 .

and (B.2) holds.

REFERENCES For each t (1 It s N), we construct a sequence of sets ( B f ) : n r l } , in which each B f ) is a block code of order n, according to the following recipe: ( y l , . . y,,) is placed in B f ) if and only if every subblock (yi,yi+l,. yi+N-l) in which i = tmod N lies in B, and every coordinate of (yl,.-.,yn) not lying in such a subblock is equal to U * . Define a ,

a ,

Then (B.l) holds. From the way in which the B,’s were

[l] R. M. Gray, Entropy and Information Theory. New York Springer-Verlag, A990. [2] A. I. Khinchin, The entropy concept in probability theory,” Usp. Mat. Nay!, vol. 8, pp. 3-20, 1953. [3] J. C. Kieffer, Block coding for an ergodic source relative to a zero-one valued fidelity criterion,” IEEE Trans. Inform. Theory, vol. IT-24, no. 4, pp. 432-438, July 1978. [4] -, “Sample converses in source coding theory,” IEEE Trans. Inform. Theory, vol. 31, no. 2, pp. 257-262, Mar. 1991. [5] J. Komer, “Coding of an informatiof source having ambiguous alphabet and the entropy of graphs, Transactwns Sixth Prague Conference on Information Theory. Prague, Czechoslovakia: Academia, 1973, pp. 411-425. [6] B. McMillan, “The basic theorems of information theory,” Ann. Math. Stat., vol. 24, pp. 196-219, 1953.