Deterministic Computation of Complexity, Information and ... - CiteSeerX

ISIT 1998, Cambridge, MA, USA, August 16 - August 21

Deterministic Computation of Complexity, Information and Entropy Mark R. Titchener Dept of Computer Science, The University of Auckland, Auckland, New Zealand. Email: markQtcode.auckland. ac .nz

Abstract - A new measure of string complexity [3] for finite strings is presented based on a specific recursive hierarchical string production process (c.f. [ 2 ] ) .From the maximal bound we deduce a relationship between complexity and total information content.

( ( # A , 0 , 0 , ...)

for 1 = 1

Given an alphabet A and prefix-free code W C A + , we define the generalized T-augmentation of W by: k

w:p; = U P ' ( W \ { P ) )

U

{Pk+').

(1)

is0

p E W is referred to as the T-prefix, and k E NINfas the corresponding T-expansion parameter. Applying Eqn (1) recursively starting with A , and subject to the recursive con>...,k,-i straints pl E A , pi E (Pl..--,Pi-I i = 2,.. . ,n, yields:

The right-most non-zero element position in d(') is the length of the maximal-length strings. The complexity of the strings of this length is simply CT(Z))= mi and found to be very accurately described by li(loge#A.lxl) where ( k i ,...,kn) li(n)= d u / l o g ( u ) . Conversely a lower bound for C T ( X )is) obtained by observing that for a single repeating symbol, Eqn (2) reduces to n = 1 , kl = 1x1 - 1. Thus: logz(lxl) 5 C T ( ~5 ) li()z)loge#A). We view generalized T-augmentation as a production proIn computing CT(2)as a function of file length for printed cess for the maximal length strings in A;;:::::;;'',),each having texts, for example, one finds a function closely of the form ) C is a constant for the source. the form x = p $ p k ; ' . . .p:'a, a E A. Conversely, given any C T ( Z )N L i ( v 1 ~ 1where x E A+, it is straight forward t o derive the vectors ( p i , . . . ,pn) Recognising that C represents in the context of the present model a bound on the expected compression for the source, and (k1,. . . , kn) respectively. We define our string complexity, denoted C T ( X )as , the ef- to be achieved by mapping the source string onto a maximal complexity string of equivalent complexity, we write C T ( Z )= fective number of T-augmentation steps required to generate li(& (x)x 1x1) where E is the expected entropy for the string x from A . More formally: x in nats/symbol. Thus we conclude that the complexity of a arity(k) C,(Z) = logz(k1+ 1). (2) string (in taugs) is simply the logarithmic integral of the total information E(z) x 1x1 (in nats/symbol x symbols = nats). i=l Given a file, we may easily compute C!r(x)from Eqn (2) An upper bound for CT(Z)as a function of string length and from this E ( x ) = lZC1(C~(~))/l~1. This was done for a is deduced from understanding the growth in the length number of English texts (alphabet sizes of 75 - 90 symbols) of the maximal-length strings with minimal T-augmentation. yielding entropy values ranging from 1.6-1.9 bits/char. This Let 4 E JVm, i.e. d = ( d l ,d z , . . . ,d,,. . .), denote a distribu- compares well with [l]in which an 'upper bound' of 1.75 bits tion vector of unbounded arity whose elements di E N rep- per character for full text is arrived at by constructing a word resent the number of code strings of length i in A;;:;::::;'';. trigram model from 583 million words of training text and More particularly let d(') denote the distribution resulting then estimating the cross-entropy. This correspondence is infrom exhaustive simple T-augmentation, that is, where all terpreted as corroborating evidence of the new deterministic code strings of length less than 1 are consumed in turn as theory. REFERENCES T-prefixes pi with corresponding ki = 1. We assume a unit vector g = (1,0,0, . . .) and define a shift operator a such that [l] P. F. Brown, V. J . Della Pietra, S. A. Della Pietra, J. C. Lai, and R. L. Mercer, "An Estimate of an Upper Bound for the d' = &)d- is given by: Entropy of English", Computational Linguistics, vol. 18, no. 1, pp. 3 2 4 0 , 1992. 0 for i < j d' = A. Lenipel and J. Ziv, "On the Complexity of Finite Sequences", [2] di-j for a > j

;,

son

c

(TI

-

' {

IEEE Trans. Inform. Theory, vol. 22, no. 1, pp. 75-81, January

1976. We recursively determined('), 1 = 1,2,. . . in terms of d("-') and mt-1, the value of the left-most non zero element of d('-'), [3] M. R. Titchener, "A Deterministic Theory of Complexity, Information and Entropy", in Recent Results, IEEE Information which is the number of smallest T-prefixes available of length Theory Workshop ITW-98, San Diego, February 1998. 1-1:

0-7803-5000-6/98/$10.000 1998 IEEE.

326