A Rate-distortion Theorem For Arbitrary Discrete ... - Semantic Scholar

1666

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 4, JULY 1998

A Rate-Distortion Theorem for Arbitrary Discrete Sources Po-Ning Chen and Fady Alajaji

Abstract— A rate-distortion theorem for arbitrary (not necessarily stationary or ergodic) discrete-time finite-alphabet sources is given. This result, which provides the expression of the minimum -achievable fixedlength coding rate subject to a fidelity criterion, extends a recent data compression theorem by Steinberg and Verdu. ´ Index Terms—Arbitrary discrete sources, data compression, rate-distortion theory, Shannon theory.

II. PRELIMINARIES Consider a random process X defined by a sequence of finitedimensional distributions [2]

fX n = (X (n) ; 1 1 1 ; Xn(n) )g1 n=1 : 1

( )

is defined as the limsup in probability of the sequence of normalized conditional entropy densities n1 iY jX (Y n j X n ). Analogously, the conditional inf-entropy rate H (Y j X ) of Y given X is defined as the liminf in probability of n1 iY jX (Y n j n X ). III. GENERAL DATA COMPRESSION THEOREM

Definition 3.1 (e.g., [1]): Given a finite source alphabet X and a finite reproduction alphabet Y , a block code for data compression of blocklength n and size M is a mapping fn (1) : X n ! Y n that results in kfn k = M codewords of length n, where each codeword is a sequence of n reproducing letters. Definition 3.2: A distortion measure n (1; 1) is a mapping

n : X n 2 Y n ! 0 and 1 > " > 0. R is an "-achievable data compression rate at distortion D for a source X if there exists a sequence of data compression codes fn (1) with

(bn j an ) : (bn )

lim sup

n!1

1

n

nk = R

log kf

and

Manuscript received April 15, 1996; revised January 20, 1998. The work of P.-N. Chen was supported in part by National Chiao-Tung University. The work of F. Alajaji was supported in part by Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant OGP0183645. The material in this correspondence was presented in part at the International Symposium on Information Theory and Its Applications, Victoria, BC, Canada, September 1996. P.-N. Chen is with the Department of Communication Engineering, National Chiao-Tung University, HsinChu, Taiwan, R.O.C. F. Alajaji is with the Department of Mathematics and Statistics, Queen’s University, Kingston, Ont. K7L 3N6, Canada. Publisher Item Identifier S 0018-9448(98)03468-3.

Xf (X ) ()  "]  D:

sup[ : 

(3.1)

1 If A is a sequence of random variables, then its liminf in probability is n the largest extended real number  such that for all  > ,

nlim !1 Pr [An   0 ] = 0:

0

Similarly, its limsup in probability is the smallest extended real number such that for all  > [2]

0018–9448/98$10.00  1998 IEEE

0 nlim !1 Pr[An  + ] = 0:



IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 4, JULY 1998

Fig. 1. X f (X ) (D + ) > "

)

sup [ :

X f (X ) ()



"]



D

+

Note that (3.1) is equivalent to stating that the limsup of the probability of excessive distortion (i.e., distortion larger than D) is smaller than 1 0 ". The infimum "-achievable data compression rate at distortion D for X is denoted by T" (D; X ).

1667

:

Step 5: Define a sequence of codes ffn g by

fn (xn ) =

where

R" (D)

fW :sup[:

inf

( )

"]Dg

 X; Y ) I(X

xn

!1

n

and

= lim inf n!1

n

n

n (x ; y

n

)

C

c lim sup EY~ [PX (A ( n ))] n!1

< 1 0 ":

The proof of this claim is provided in the Appendix. Therefore, there exists (a sequence of) Cn3 such that

C

c 3 lim sup PX (A ( n )) n!1

< 1 0 ":

2 X n : n1 n (xn ; f (xn ))  D +

> ":

Hence sup[ :

X f (X ) ()  "]  D +

where the last step is clearly depicted in Fig. 1. This proves the forward part. 2) Converse part: We show that for any sequence of encoders ffn(1)gn1=1 , if sup [ :

X f (X ) ()  "]  D

lim sup n!1

n

then

 D + :

Step 4: Claim:

xn

PX

3  lim inf PX (A(Cn )) n!1 c 3 = 1 0 lim sup PX (A (Cn )) n!1

sup[ : X f (X ) ()  "]  D + :

1

2 X n : n1 n (xn; fn(xn ))  D +  A(Cn3 )

X f (X ) (D + )

1 kfnk  R" (D) + 2 n

~ be the channel distribution achieving R" (D), and Step 1: Let W ~. let PY~ be the Y -marginal of PX W Step 2: Let R = R" (D) + 2 . Choose M = enR n-blocks independently according to PY~ , and denote the resulting random set by Cn . Step 3: For a given Cn , we denote by A(Cn ) the set of sequences xn 2 X n such that there exists y n 2 Cn with

Y n.

since (8xn 2 A(Cn3 )) there exists y n 2 Cn3 such that n n (1=n)n (x ; y )  D + , which by definition of fn implies that (1=n)n (xn ; fn (xn ))  D + . Step 6: Consequently,

where the infimum is taken over all conditional distributions W = fPY jX g1 n=1 for which the joint distribution PX Y = PX W satisfies the distortion constraint. Proof: 1) Forward part (achievability): Choose > 0. We will prove the existence of a sequence of data compression codes with

lim sup

n (xn ; y n ); if xn 2 A(Cn3 ) otherwise

where 0 is a fixed default n-tuple in Then

Theorem 3.1 (General Data Compression Theorem): Fix D > 0 and 1 > " > 0. Let X and fn (1; 1)gn1 be given. Then

T" (D; X ) = R" (D)

arg miny 2C 0;

1

log

kfnk  R" (D):

Let ^ n (y n j xn ) W

1; 0;

if y n = fn (xn ) otherwise:

Let Y^ n denote the output corresponding to input X n and ^ n . Then to evaluate the statistical properties of the channel W random sequence f(1=n)n (X n ; fn (X n )gn1=1 under distribution

1668

IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 44, NO. 4, JULY 1998

PX is equivalent to evaluating those of the random sequence ^n f(1=n)n (X n; Y^ n )g1 n=1 under distribution PX W . Therefore, R" (D)

fW :sup[:

inf

( )"]Dg

 X; Y ) I(X

=



 X ; Y^ )  I(X  Y^ ) 0 H(Y^ j X )  H(  Y^ )  H( 1  lim sup log kfn k; n!1 n

PX (x ) 1 0 n

x

x

2

2X

PX (xn )

2X y

10

2Y x

y

2Y

n

M

n

PY~ (y )K(x ; y )

2Y ~  1 0 e0n(I (X ;Y )+ )

PY~ jX (y

2X

n

n

y

jx

n

n

n

M

)K(x ; y )

PX (xn )PY~ jX (xn ; y n )K(xn ; y n )

+ expf0en(R0R (D)0 ) g:

where the second inequality follows from [4, Theorem 8, property (d)] and the third inequality follows from the fact that H(Y^ j X )  0.

Therefore, ") lim sup EY~ [PX (An (Cn3 ))]  1 0 lim inf Pr A(n;

Claim (cf. Proof of Theorem 3.1):

lim sup EY~ [PX (Ac (Cn3 ))] < 1 0 ": n!1

< 1 0 ":

ACKNOWLEDGMENT

Proof: Step 1: Let

The authors would like to thank Prof. S. Verd´u for his valuable advice and encouragements.

supf : X Y~ ()  "g:

D(")

n!1

n!1

APPENDIX

REFERENCES

Define ") A(n;

(xn ; y n ) : and

1 n (xn ; y n )  D(") + n

1  X ; Y~ ) + : iX Y (xn ; y n )  I(X n

Since

lim inf Pr n!1

1 n (X n ; Y~ n )  D(") + n

D

>"

[1] R. E. Blahut, Principles and Practice of Information Theory. Reading, MA: Addison Wesley, 1988. [2] T. S. Han and S. Verd´u, “Approximation theory of output statistics,” IEEE Trans. Inform. Theory, vol. 39, pp. 752–772, May 1993. [3] Y. Steinberg and S. Verd´u, “Simulation of random processes and ratedistortion theory,” IEEE Trans. Inform. Theory, vol. I42, pp. 63–86, Jan. 1996. [4] S. Verd´u and T. S. Han, “A general formula for channel capacity,” IEEE Trans. Inform. Theory, vol. 40, pp. 1147–1157, July 1994.

and

lim inf Pr n!1

1 i ~ (X n ; Y~ n ) n X Y

E

 X ; Y~ ) +  I(X

On One Useful Inequality in Testing of Hypotheses =1

we have ") lim inf Pr A(n; = lim inf Pr (D \ E ) n!1 n!1  lim inf Pr (D) + lim inf Pr (E ) 0 1 n!1

> " + 1 0 1 = ":

Step 3: By following a similar argument in [3, Eqs. (9)–(12)], we obtain

= =

C x

PY~ (Cn3 )

2X

x 62A(C ) n

PX (x )

C

:x

PX (xn )

62A(C

)

Index Terms—Error probabilities, testing of hypotheses.

I. MAIN INEQUALITY ") A(n;

") 1; if (xn ; y n ) 2 A(n; 0; otherwise:

EY~ [PX (Ac (Cn3 ))]

Abstract—A simple proof of one probabilistic inequality is presented.

n!1

Step 2: Let K(xn ; y n ) be the indicator function of

K(xn ; y n ) =

Marat V. Burnashev

PY~ (Cn3 )

Let P and Q be two given probability measures on a measurable space ( ; ). We consider testing of hypotheses P and Q using one observation. For an arbitrary decision rule, let  and denote the two kinds of error probabilities. If both error probabilities have equal costs (or we want to minimize the maximum of them) then it is natural to investigate the minimal possible sum inf  + for the best decision rule.

XA

f

g

Manuscript received July 10, 1997; revised November 24, 1997. This work was supported by the Russian Foundation for Fundamental Research under Grants N 95-01-00136a and INTAS-94-469. The author is with the Institute for Problems of Information Transmission, Russian Academy of Sciences, 101447 Moscow, Russia. Publisher Item Identifier S 0018-9448(98)03470-1.

0018–9448/98$10.00  1998 IEEE