Using Information Theory Approach to Randomness Testing

Using Information Theory Approach to Randomness Testing ∗ B. Ya. Ryabko and V.A. Monarev

Abstract We address the problem of detecting deviations of binary sequence from randomness,which is very important for random number (RNG) and pseudorandom number generators (PRNG). Namely, we consider a null hypothesis H0 that a given bit sequence is generated by Bernoulli source with equal probabilities of 0 and 1 and the alternative hypothesis H1 that the sequence is generated by a stationary and ergodic source which differs from the source under H0 . We show that data compression methods can be used as a basis for such testing and describe two new tests for randomness, which are based on ideas of universal coding. Known statistical tests and suggested ones are applied for testing PRNGs, which are practically used. Those experiments show that the power of the new tests is greater than of many known algorithms.

Keywords: Hypothesis testing, Randomness testing, Random number testing, Universal code, Information Theory, Random number generator, Shannon entropy.

∗

The authors were supported by INTAS grant no. 00-738 and Russian Foundation for Basic Research under Grant no. 03-01-00495.

1

1

Introduction

The randomness testing of random number and pseudorandom number generators is used for many purposes including cryptographic, modeling and simulation applications; see, for example, Knuth, 1981; Maurer,1992; Rukhin and others, 2001. For such applications a required bit sequence should be true random, i.e., by definition, such a sequence could be interpreted as the result of the flips of a ”fair” coin with sides that are labeled ”0” and ”1” (for short, it is called a random sequence; see Rukhin and others, 2001). More formally, we will consider the main hypothesis H0 that a bit sequence is generated by the Bernoulli source with equal probabilities of 0’s and 1’s. Associated with this null hypothesis is the alternative hypothesis H1 that the sequence is generated by a stationary and ergodic source which generates letters from {0, 1} and differs from the source under H0 . In this paper we will consider some tests which are based on results and ideas of Information Theory and, in particular, the source coding theory. First, we show that a universal code can be used for randomness testing. (Let us recall that, by definition, the universal code can compress a sequence asymptotically till the Shannon entropy per letter when the sequence is generated by a stationary and ergodic source). If we take into account that the Shannon per-bit entropy is maximal (1 bit) if H0 is true and is less than 1 if H1 is true (Billingsley, 1965; Gallager, 1968), we see that it is natural to use this property and universal codes for randomness testing because, in principle, such a test can distinguish each deviation from randomness, which can be described in a framework of the stationary and ergodic source model. Loosely speaking, the test rejects H0 if a binary sequence can be compressed by a considered universal code (or a data compression method.) It should be noted that the idea to use the compressibility as a measure of randomness has a long history in mathematics. The point is that, on the one hand, the problem of randomness testing is quite important for practice, but, on the other hand, this problem is closely connected with such deep theoretical issues as the definition of randomness, the logical basis of probability theory, randomness and complexity, etc; see Kolmogorov, 1965; Li and Vitanyi, 1997; Knuth, 1981; Maurer,1992. Thus, Kolmogorov suggested to define the randomness of a sequence, informally, as the length of the shortest program, which can create the sequence (if one of the universal Turing machines is used as a computer). So, loosely speaking, the randomness (or Kolmogorov complexity) of the finite sequence is equal to its shortest de2

scription. It is known that the Kolmogorov complexity is not computable and, therefore, cannot be used for randomness testing. On the other hand, each lossless data compression code can be considered as a method for upper bounding the Kolmogorov complexity. Indeed, if x is a binary word, φ is a data compression code and φ(x) is the codeword of x, then the length of the codeword |φ(x)| is the upper bound for the Kolmogorov complexity of the word x. So, again we see that the codeword length of the lossless data compression method can be used for randomness testing. In this paper we suggest tests for randomness, which are based on results and ideas of the source coding theory. Firstly, we show how to build a test basing on any data compression method and give some examples of application of such test to PRNG’s testing. It should be noted that data compression methods were considered as a basis for randomness testing in literature. For example, Maurer’s Universal Statistical Test, Lempel-Ziv Compression Test and Approximate Entropy Test are connected with universal codes and are quite popular in practice, see, for example, Rukhin and others, 2001. In contrast to known methods, the suggested approach gives a possibility to make a test for randomness, basing on any lossless data compression method even if a distribution law of the codeword lengths is not known. Secondly, we describe two new tests, conceptually connected with universal codes. When both tests are applied, a tested sequence x1 x2 ...xn is divided into subwords x1 x2 ...xs , xs+1 xs+2 ...x2s , . . . , s ≥ 1, and the hypothesis H0∗ that the subwords obey the uniform distribution (i.e. each subword is generated with the probability 2−s ) is tested against H1∗ = ¬H0∗ . The key idea of the new tests is as follows. All subwords from the set {0, 1}s are ordered and this order changes after processing each subword xjs+1 xjs+2 ...x(j+1)s , j = 0, 1, . . . in such a way that, loosely speaking, the more frequent subwords have small ordinals. When the new tests are applied, the frequency of different ordinals are estimated (instead of frequencies of the subwords as for, say, chi- square test). The natural question is how to choose the block length s in such schemes. We show that, informally speaking, the block length s should be taken quite large due to the existence of so called two-faced processes. More precisely, it is shown that for each integer s∗ there exists such a process ξ that for each binary word u the process ξ creates u with the probability 2−|u| if the length of the u (|u|) is less than or equal to s∗ , but, on the other hand, the probability distribution ξ(v) is very far from uniform if the length of the 3

words v is greater than s∗ . (So, if we use a test with the block length s ≤ s∗ , the sequences generated by ξ will look like random, in spite of ξ is far from being random.) The outline of the paper is as follows. In Section 2 the general method for construction randomness testing algorithms basing on lossless data compressors is described. Two new tests for randomness, which are based on constructions of universal coding, as well as the two-faced processes, are described in the Section 3. In Section 4 the new tests are experimentally compared with methods from the ” A statistical test suite for random and pseudorandom number generators for cryptographic applications”, which was recently suggested by Rukhin and others, 2001. It turns out that the new tests are more powerful than known ones.

2

Data compression methods as a basis for randomness testing

2.1. Randomness testing based on data compression Let A be a finite alphabet and An be the set of all words of the length n S n ∞ over A, where n is an integer. By definition, A∗ = ∞ is the n=1 A and A set of all infinite words x1 x2 . . . over the alphabet A. A data compression method (or code) ϕ is defined as a set of mappings ϕn such that ϕn : An → {0, 1}∗ , n = 1, 2, . . . and for each pair of different words x, y ∈ An ϕn (x) 6= ϕn (y). Informally, it means that the code ϕ can be applied for compression of each message of any length n, n > 0 over alphabet A and the message can be decoded if its code is known. Now we can describe a statistical test which can be constructed basing ˆ 0 be a hypothesis that the words on any code ϕ. Let n be an integer and H n from the set A obey the uniform distribution, i.e., p(u) = |A|−n for each u ∈ {0, 1}n . (Here and below |x| is the length if x is a word, and the number of elements if x is a set.) Let a required level of significance (or a Type I error) be α, α ∈ (0, 1). The following main idea of a suggested test is quite natural: The well compressed words should be considered as non- random ˆ 0 should be rejected. More exactly, we define a critical value of the and H suggested test by tα = n log |A| − log(1/α) − 1 . (1) (Here and below log x = log2 x.) 4

ˆ 0 is accepted if Let u be a word from An . By definition, the hypothesis H (n) |ϕn (u)| > tα and rejected, if |ϕn (u)| ≤ tα . We denote this test by Γα,ϕ . Theorem 1. For each integer n and a code ϕ, the Type I error of the described test Γ(n) α,ϕ is not larger than α. Proof is given in Appendix. Comment 1. The described test can be modified in such a way that the Type I error will be equal to α. For this purpose we define the set Aγ by Aγ = {x : x ∈ An & |ϕn (x)| = γ} and an integer s for which the two following inequalities are valid: s X

n

|Aj | ≤ α|A|
(s + 1) then If for x ∈ An |ϕn (x)| ≤ s then H ˆ ˆ 0 is accepted with H0 is accepted and if |ϕn (x)| = (s + 1) the hypothesis H the probability s+1 X

(

|Aj | − α|A|n )/|As+1 |

j=1

and rejected with the probability s+1 X

1 −(

|Aj | − α|A|n )/|As+1 | .

j=1

(Here we used a randomized criterion, see for definition, for example, Kendall and Stuart, 1961, part 22.11.) We denote this test by Υ(n) α,ϕ . Claim 1. For each integer n and a code ϕ, the Type I error of the described test Υ(n) α,ϕ is equal to α. Proof is given in Appendix. We can see that this criterion has the level of significance (or Type I error) exactly α, whereas the first criterion, which is based on critical value (1), has the level of significance that could be less than α. In spite of this drawback, the first criterion may be more useful due to its simplicity. Moreover, such an approach gives a possibility to use a data compression method ψ for testing even in case where the distribution of the length |ψn (x)|, x ∈ An is not known. Comment 2. We have considered codes, for which different words of the same length have different codewords (In Information Theory sometimes 5

such codes are called non- singular.) Quite often a stronger restriction is required in Information Theory. Namely, it is required that each sequence ϕn (x1 )ϕn (x2 )...ϕ(xr ), r ≥ 1, of encoded words from the set An , n ≥ 1, can be uniquely decoded into x1 x2 ...xr . Such codes are called uniquely decodable. For example, let A = {a, b}, the code ψ1 (a) = 0, ψ1 (b) = 00, obviously, is non- singular, but is not uniquely decodable. (Indeed, the word 000 can be decoded in both ab and ba.) It is well known in Information Theory that a code ϕ can be uniquely decoded if the following Kraft inequality is valid: Σu∈An 2−|ϕn (u)| ≤ 1 ,

(3)

see, for ex., Gallager, 1968. If it is known that the code is uniquely decodable, the suggested critical value (1) can be changed. Let us define tˆα = n log |A| − log(1/α) .

(4)

ˆ 0 is Let, as before, u be a word from An . By definition, the hypothesis H accepted if |ϕn (u)| > tˆα and rejected, if |ϕ(u)| ≤ tˆα . We denote this test by ˆ (n) Γ α,ϕ . Claim 2. For each integer n and a uniquely decodable code ϕ, the Type ˆ (n) is not larger than α. I error of the described test Γ α,ϕ Proof is given in Appendix. So, we can see from (1) and (4) that the critical value is larger, if the code is uniquely decodable. On the other hand, the difference is quite small and (1) can be used without a large loose of the test power even in a case of the uniquely decodable codes. It should not be a surprise that the level of significance (or a Type I error) does not depend on the alternative hypothesis H1 , but, of course, the power of a test (and the Type II error) will be determined by H1 . The examples of testing by real data compression methods will be given in Section 4. 2.2. Randomness testing based on universal codes. We will consider the main hypothesis H0 that the letters of a given sequence x1 x2 ...xt , xi ∈ A, are independent and identically distributed (i.i.d.) with equal probabilities of all a ∈ A and the alternative hypothesis H1 that the sequence is generated by a stationary and ergodic source, which generates letters from A and differs from the source under H0 . (If A = {0, 1}, 6

i.i.d. coincides with Bernoulli source.) The definition of the stationary and ergodic source can be found in Billingsley, 1965, and Gallager, 1968. We will consider statistical tests, which are based on universal coding and universal prediction. First we define a universal code. By definition, ϕ is an universal code if for each stationary and ergodic source (or a process) π the following equality is valid with probability 1 (according to the measure π ) lim (|ϕn (x1 ...xn )|)/n = h(π) ,

n→∞

(5)

where h(π) is the Shannon entropy. (Its definition can be found, for example, in Billingsley, 1965; Gallager, 1968.) It is well known in Information Theory that h(π) = log |A| if H0 is true, and h(π) < log |A| if H1 is true, see, for ex., Billingsley, 1965; Gallager, 1968. From this property and (5) we can easily yield the following theorem. Theorem 2. Let ϕ be a universal code, α ∈ (0, 1) be a level of significance and a sequence x1 x2 ...xn be generated by a stationary ergodic source π. If the described above test Γ(n) α,ϕ is applied for testing H0 (against H1 ), then the Type I error is not larger than α, and the Type II error goes to 0, when n → ∞. So, we can see that each good universal code can be used as a basis for randomness testing. But converse proposition is not true. Let, for example, there be a code, whose codeword length is asymptotically equal to (0.5 + h(π)/2) for each source π (with probability 1, where, as before, h(π) is the Shannon entropy). This code is not good, because its codeword length does not trend to the entropy, but, obviously, such code could be used as a basis for a test of randomness. So, informally speaking, the set of tests is larger than the set of universal codes.

3

Two new tests for randomness and twofaced processes

Firstly, we suggest two tests which are based on ideas of universal coding, but they are described in such a way that can be understood without any knowledge of Information Theory. 3.1. The ”book stack” test

7

Let, as before, there be given an alphabet A = {a1 , ..., aS }, a source, which generates letters from A, and two following hypotheses: the source is i.i.d. and p(a1 ) = .... = p(aS ) = 1/S (H0 ) and H1 = ¬H0 . We should test the hypotheses basing on a sample x1 x2 ... xn , n ≥ 1 , generated by the source. When the ”book stack” test is applied, all letters from A are ordered from 1 to S and this order is changed after observing each letter xt according to the formula ν t+1 (a) =

   1,

if xt = a ; ν t (a) + 1, if ν t (a) < ν t (xt );   t ν (a), if ν t (a) > ν t (xt ) ,

(6)

where ν t is the order after observing x1 x2 ... xt , t = 1 , , ... , n , ν 1 is defined arbitrarily. (For ex., we can define ν 1 = {a1 , ..., aS }.) Let us explain informally (6). Suppose that the letters of A make a stack, like a stack of books and ν 1 (a) is a position of a in the stack. Let the first letter x1 of the word x1 x2 ... xn be a. If it takes i1 −th position in the stack (ν 1 (a) = i1 ), then take a out of the stack and put it on the top. (It means that the order is changed according to (6).) Repeat the procedure with the second letter x2 and the stack obtained, etc. It can help to understand the main idea of the suggested method if we take into account that, if H1 is true, then frequent letters from A (as frequently used books) will have relatively small numbers (will spend more time next to the top of the stack). On the other hand, if H0 is true, the probability to find each letter xi at each position j is equal to 1/S. Let us proceed with the description of the test. The set of all indexes {1, . . . , S} is divided into r, r ≥ 2, subsets A1 = {1, 2, . . . , k1 }, A2 = {k1 + 1, . . . , k2 }, . . . , Ar = {kr−1 + 1, . . . , kr }. Then, using x1 x2 ... xn , we calculate how many ν t (xt ), t = 1, ..., n, belong to a subset Ak , k = 1, ..., r. We define this number as nk (or, more formally, nk = |{t : ν t (xt ) ∈ Ak , t = 1, . . . , n}|, k = 1, ..., r.) Obviously, if H0 is true, the probability of the event ν t (xt ) ∈ Ak is equal to |Aj |/S. Then, using a ”common” chi- square test we ˆ 0 = P {ν t (xt ) ∈ Ak } = |Aj |/S basing on the empirical test the hypothesis H ˆ 1 = ¬H ˆ 0 . Let us recall that the value frequencies n1 , . . . , nr , against H x2 =

r X (ni − n(|Ai |/S))2 i=1

n(|Ai |/S)

,

(7)

is calculated, when chi- square test is applied, see, for ex., Kendall and Stuart, 1961. It is known that x2 asymptotically follows the χ-square distribution 8

ˆ 0 is true. If the level of significance with (k −1) degrees of freedom (χ2k−1 ) if H ˆ 0 is accepted (or a Type I error) of the χ2 test is α, α ∈ (0, 1), the hypothesis H 2 2 when x from (7) is less than the (1 − α) -value of the χk−1 distribution.; see, for ex., Kendall, Stuart, 1961. We do not describe the exact rule how to construct the subsets {A1 , A2 , . . . , Ar }, but we recommend to implement some experiments for finding the parameters, which make the sample size minimal (or, at least, acceptable). The point is that there are many cryptographic and other applications where it is possible to implement some experiments for optimizing the parameter values and, then, to test hypothesis basing on independent data. For example, in case of testing a PRNG it is possible to seek suitable parameters using a part of generated sequence and then to test the PRNG using a new part of the sequence. Let us consider a small example. Let A = {a1 , . . . , a6 }, r = 2, A1 = {a1 , a2 , a3 }, A2 = {a4 , a5 , a6 }, x1 . . . x8 = a3 a6 a3 a3 a6 a1 a6 a1 . If ν1 = 1, 2, 3, 4, 5, 6, then ν2 = 3, 1, 2, 4, 5, 6, ν3 = 6, 3, 1, 2, 4, 5, etc., and n1 = 7, n2 = 1. We can see that the letters a3 and a6 are quite frequent and the ”book stack” indicates this nonuniformity quite well. (Indeed, the average values of n1 and n2 equal 4, whereas the real values are 7 and 1, correspondingly.) Examples of practical applications of this test will be given in Section 4, but here we make two notes. Firstly, we pay attention to the complexity of this algorithm. The ”naive” method of transformation according to (6) could take the number of operations proportional to S, but there exist algorithms, which can perform all operations in (6) using O(log S) operations. Such algorithms can be based on AVL- trees, see, for ex., Aho,Hopcroft and Ulman, 1976. The last comment concerns with the name of the method. The ”book stack” structure is quite popular in Information Theory and Computer Science. In Information Theory this structure was firstly suggested as a basis of an universal code by Ryabko, 1980, and was rediscovered by Bently, Sleator, Tarjan, Wei in 1986, and Elias in 1987 (see also a comment of Ryabko (1987) about a history of this code). In English language literature this code is frequently called as ”Move-to-Front” (MTF) scheme as it was suggested by Bently, Sleator, Tarjan and Wei. Now this data structure is used in a so called caching and many other algorithms in Computer Science under the name ”Move-to-Front”. It is also worth noting that the book stack was firstly considered by a soviet mathematician M.L. Cetlin as an example of a

9

self- adaptive system in 1960’s, see Rozanov, 1971. 3.2. The order test This test is also based on changing the order ν t (a) of alphabet letters but the rule of the order change differs from (6). To describe the rule we first define λt+1 (a) as a count of occurrences of a in the word x1 . . . xt−1 xt . At each moment t the alphabet letters are ordered according to ν t in such a way that, by definition, for each pair of letters a and b ν t (a) ≺ ν t (b) if λt (a) ≤ λt (b). For example, if A = {a1 , a2 , a3 } and x1 x2 x3 = a3 a2 a3 , the possible orders can be as follows: ν 1 = (1, 2, 3), ν 2 = (3, 1, 2), ν 3 = (3, 2, 1), ν 4 = (3, 2, 1). In all other respects this method coincides with the book stack. (The set of all indexes {1, . . . , S} is divided into r subsets, etc.) Obviously, after observing each letter xt the value λt (xt ) should be increased and the order ν t should be changed. It is worth noting that there exist a data structure and algorithm, which allow maintaining the alphabet letters ordered in such a way that the number of operations spent is constant, independently of the size of the alphabet. This data structure was described by Moffat, 1999 and Ryabko, Rissanen, 2003. 3.3. Two- faced processes and the choice of the block length for a process testing There are quite many methods for testing H0 and H1 , where the bit stream is divided into words (blocks) of the length s, s ≥ 1, and the sequence of the blocks x1 x2 . . . xs , xs+1 . . . x2s , . . . is considered as letters, where each letter belongs to the alphabet Bs = {0, 1}s and has the probability 2−s , if H0 is true. For instance, both above described tests, methods from Ryabko, Stognienko and Shokin (2003) and quite many other algorithms belong to this kind. That is why the questions of choosing the block length s will be considered here. As it was mentioned in the introduction there exist two-faced processes, which, on the one hand, are far from being truly random, but, on the other hand, they can be distinguished from truly random only in the case when the block length s is large. From the information theoretical point of view the two- faced processes can be simply described as follows. For a two- faced process, which generates letters from {0, 1}, the limit Shannon entropy is (much) less than 1 and, on the other hand, the s− order entropy (hs ) is maximal (hs = 1 bit per letter) for relatively large s. We describe two families of two- faced processes T (k, π) and T¯(k, π), where k = 1, 2, . . . , and π ∈ (0, 1) are parameters. The processes T (k, π) and T¯(k, π) are Markov chains of the connectivity (memory) k, which generate 10

letters from {0, 1}. It is convenient to define them inductively. The process T (1, π) is defined by conditional probabilities PT (1,π) (0/0) = π, PT (1,π) (0/1) = 1 − π (obviously, PT (1,π) (1/0) = 1 − π, PT (1,π) (1/1) = π). The process T¯(1, π) is defined by PT¯(1,π) (0/0) = 1 − π, PT¯(1,π) (0/1) = π. Assume that T (k, π) and T¯(k, π) are defined and describe T (k + 1, π) and T¯(k + 1, π) as follows PT (k+1,π) (0/0u) = PT (k,π) (0/u), PT (k+1,π) (1/0u) = PT (k,π) (1/u), PT (k+1,π) (0/1u) = PT¯(k,π) (0/u), PT (k+1,π) (1/1u) = PT¯(k,π) (1/u), and, vice versa, PT¯(k+1,π) (0/0u) = PT¯(k,π) (0/u), PT¯(k+1,π) (1/0u) = PT¯(k,π) (1/u), PT¯(k+1,π) (0/1u) = PT (k,π) (0/u), PT¯(k+1,π) (1/1u) = PT (k,π) (1/u) for each u ∈ Bk (here vu is a concatenation of the words v and u). For example, PT (2,π) (0/00) = π, PT (2,π) (0/01) = 1−π, PT (2,π) (0/10) = 1−π, PT (2,π) (0/11) = π. The following Claim shows that the two-faced processes exist. Theorem 3. For each π ∈ (0, 1) the s-order Shannon entropy (hs ) of the processes T (k, π) and T¯(k, π) equals 1 bit per letter for s = 0, 1, . . . , k whereas the limit Shannon entropy (h∞ ) equals −(π log2 π + (1 − π) log2 (1 − π)). The proofs of the theorem is given in Appendix, but here we consider examples of ”typical” sequences of the processes T (1, π) and T¯(1, π) for π, say, 1/5. Such sequences could be as follows: 010101101010100101... and 000011111000111111000..... We can see that each sequence contains approximately one half of 1’s and one half of 0’s. (That is why the first order Shannon entropy is 1 per a letter.) On the other hand, both sequences do not look like truly random, because they, obviously, have too long subwords like either 101010.. or 000..11111... (In other words, the second order Shannon entropy is much less than 1 per letter.) Hence, if a randomness test is based on estimation of frequencies of 0’s and 1’s only, then such a test will not be able to find deviations from randomness. So, if we revert to the question about the block length of tests and take into account the existence of two- faced processes, it seems that the block length could be taken as large as possible. But it is not so. The following informal consideration could be useful for choosing the block length. The point is that statistical tests can be applied if words from the sequence 11

x1 x2 . . . xs , xs+1 . . . x2s , . . . , x(m−1)s+1 x(m−1)s+2 . . . xms

(8)

are repeated (at least a few times) with high probability (here ms is the sample length). Otherwise, if all words in (8) are unique (with high probability) when H0 is true, it should be so for close hypotheses H1 . That is why a sensible test cannot be constructed basing on a division into s−letter words. So, the word length s should be chosen in such a way that some words from the sequence (8) are repeated with high probability, when H0 is true. So, now our problem can be formulated as follows. There is a binary sequence x1 x2 . . . xn generated by the Bernoulli source with P (xi = 0) = P (xi = 1) = 1/2 and we want to find such a block length s that the sequence (8) with m = bn/sc, contains some repetitions (with high probability). This problem is well known in the probability theory and sometimes called as the birthday problem. Namely, the standard statement of the problem is as follows. There are S = 2s cells and m (= n/s) pellets. Each pellet is put in one of the cells with √ the probability 1/S. It is known in Probability Theory that, if m = c S, c > 0 then the average number of cells with at least two pellets equals c2 (1/2 + ◦(1) ), where S goes to ∞ ; see Kolchin, Sevast’yanov and Chistyakov, 1976. In our case the number of cells with at least two pellets is equal to the number of the words from the sequence (8) which are met two s (or more) √ times. Having into account that S = 2 , m = n/s, we obtain from m = c S, c > 0 an informal rule for choosing the length of words in (8): n  s2s/2

(9)

where n is the length of a sample x1 x2 ...xn , s is the block length. If s is much larger, the sequence (8) does not have repeated words (in case H0 ) and it is difficult to build a a sensible test. On the other hand, if s is much smaller, large classes of the alternative hypotheses cannot be tested (due to existence of the two-faced processes).

4

The experiments

In this part we describe some experiments carried out to compare new tests with known ones. We will compare order test, book stack test, tests which are based on standard data compression methods, and tests from Rukhin and others, 2001. The point is that the tests from Rukhin and others are 12

selected basing on comprehensive theoretical and experimental analysis and can be considered as the state-of-the-art in randomness testing. Besides, we will also test the method recently published by Ryabko, Stognienko, Shokin, (2003), because it was published later than the book of Rukhin and others. We used data generated by the PRNG ”RANDU” (described in Dudewicz and Ralley, 1981) and random bits from ”The Marsaglia Random Number CDROM”, see: http://stat.fsu.edu/diehard/cdrom/ ). RANDU is a linear congruent generators (LCG), which is defined by the following equality Xn+1 = (A Xn + C) mod M , where Xn is n-th generated number. RANDU is defined by parameters A = 216 + 3, C = 0, M = 231 , X0 = 1. Those kinds of sources of random data were chosen because random bits from ”The Marsaglia Random Number CDROM” are considered as good random numbers, whereas it is known that RANDU is not a good PRNG. It is known that the lowest digits of Xn are ”less random” than the leading digits (Knuth, 1981), that is why in our experiments with RANDU we extract an eight-bit word from each generated ˆ i = bXi /223 c. Xi by formula X The behavior of the tests was investigated for files of different lengths (see the tables below). We generated 100 different files of each length and applied each mentioned above test to each file with level of significance 0.01 (or less, see below). So, if a test is applied to a truly random bit sequence, on average 1 file from 100 should be rejected. All results are given in the tables, where integers in boxes are the number of rejected files (from 100). If a number of the rejections is not given for a certain length and test, it means that the test cannot be applied for files of such a length. The table 1 contains information about testing of sequences of different lengths generated by RANDU, whereas the table 2 contains results of application of all tests to 5 000 000- bit sequences either generated by RANDU or taken from ”The Marsaglia Random Number CDROM”. Let us first give some comments about the tests, which are based on popular data compression methods RAR and ARJ. In those cases we applied each of both methods to each file and estimated the length of compressed data. Then we use the test Γ(n) α,ϕ with the critical value (1) as follows. The 8 alphabet size |A| = 2 = 256, n log |A| is simply the length of file (in bits) before compression, (whereas n is the length in bytes). So, taking α = 0.01, from (1), we see that the hypothesis about randomness (H0 ) should be 13

rejected, if the length of compressed file is less than or equal to n log |A| − 8 bits. (Strictly speaking, in this case α ≤ 2−7 = 1/128.) So, taking into account that the length of computer files is measured in bytes, this rule is very simple: if the n−byte file is really compressed (i.e. the length of the encoded file is n − 1 bytes or less), this file is not random. So, the following tables contain numbers of cases, where files were really compressed. Let us now give some comments about parameters of the considered methods. As it was mentioned, we investigated all methods from the book of Rukhin and others (2001), the test of Ryabko, Stognienko and Shokin, 2003 (RSS test for short), the described above two tests based on data compression algorithms, the order tests and the book stack test. For some tests there are parameters, which should be specified. In such cases the values of parameters are given in the table in the row, which follows the test results. There are some tests from the book of Rukhin and others, where parameters can be chosen from a certain interval. In such cases we repeated all calculations three times, taking the minimal possible value of the parameter, the maximal one and the average one. Then the data for the case when the number of rejections of the hypothesis H0 was maximal, is given in the table. The choice of parameters for RSS, the book stack test and the order test was made on the basis of special experiments, which were carried out for independent data. (Those algorithms are implemented as a Java program and can be found on the internet, see http : //web.ict.nsc.ru/˜rng/.) In all cases such experiments have shown that for all three algorithms the optimal blocklength is close to the one defined by informal equality (9). We can see from the tables that the new tests can detect non-randomness more efficiently than the known ones. Seemingly, the main reason is that RSS, book stack tests and order test deal with such large blocklength as it is possible, whereas many other tests are focused on other goals. The second reason could be an ability for adaptation. The point is that the new tests can find subwords, which are more frequent than others, and use them for testing, whereas many other tests are looking for particular deviations from randomness. In conclusion, we can say that the obtained results show that the new tests, as well as the ideas of Information Theory in general, can be useful tools for randomness testing.

14

5

Appendix.

Proof of Theorem 1. First we estimate the number of words ϕ(u) whose length is less than or equal to an integer τ . Obviously, at most one word can be encoded by the empty codeword, at most two words by the words of the length 1, ..., at most 2i can be encoded by the words of length i, etc. Having taken into account that the codewords ϕn (u) 6= ϕn (v) for different u and v, we obtain the inequality |{u : |ϕn (u)| ≤ τ }| ≤

τ X

2i = 2τ +1 − 1.

i=0

From this inequality and (1) we can see that the number of words from the set {An }, whose codelength is less than or equal to tα = n log |A| − log(1/α) − 1, is not greater than 2n log |A|−log(1/α) . So, we obtained that |{u : |ϕn (u)| ≤ tα }| ≤ α|A|n . Taking into account that all words from An have equal probabilities if H0 is true, we obtain from the last inequality, (1) and the description of the test Γ(n) α,ϕ that P r{|ϕn (u)| ≤ tα |} ≤ (α|A|n /|A|n ) = α if H0 is true. The theorem is proved. Proof of Claim 1. The proof is based on a direct calculation of the probability of rejection for a case where H0 is true. From the description of the test Υ(n) α,ϕ and definition of s (see (1)) we obtain the following chain of equalities. P r{H0 is rejected } = P r{ |ϕn (u)| = s} s+1 X

+ P r{|ϕn (u)| = s + 1} ( 1 − (

|Aj | − α|A|n )/|As+1 | ) )

j=1

=

s s+1 X 1 X ( |A | + |A | ( 1 − ( |Aj | − α|A|n )/|As+1 | ) ) = α. j s+1 n A j=0 j=1

The claim is proved. Proof of Claim 2. We can think that tˆα in (4) is an integer. (Otherwise, we obtain the same test taking btˆα c as a new critical value of the test.) From 15

the Kraft inequality (3) we obtain that 1≥

ˆ 2−|ϕn (u)| ≥ |{u : | ϕn (u)| ≤ tˆα }| 2−tα .

X u∈An

This inequality and (4) yield: |{u : | ϕn (u)| ≤ tˆα }| ≤ α|A|n . If H0 is true then the probability of each u ∈ An equals |A|−n and from the last inequality we obtain that P r{|ϕ(u)| ≤ tˆα } = |A|−n |{u : | ϕn (u)| ≤ tˆα }| ≤ α, if H0 is true. The claim is proved. Proof of Theorem 3. First we show that p∗ (x1 ...xk ) = 2−k ,

(10)

(x1 ...xk ) ∈ {0, 1}k , is a stationary distribution for the processes T (k, π) and T¯(k, π) for all k = 1, 2, . . . and π ∈ (0, 1). We will use an induction on k. For the case k = 1 the equality (10) can be obtained by the direct calculations. For other values of k, (4) will be proved if we show that the system of equations p(x1 ...xk ) = p(0x1 ...xk−1 ) p(xk /0x1 ...xk−1 ) + p(1x1 ...xk−1 ) p(xk /1x1 ...xk−1 ) has the solution p(x1 ...xk ) = 2−k , (x1 ...xk ) ∈ {0, 1}k , k = 1, 2, . . .. It can be easily seen if we take into account that, by definition of T (k, π) and T¯(k, π), the equality p(xk /0x1 ...xk−1 ) + p(xk /1x1 ...xk−1 ) = 1 is valid for all (x1 ...xk ) ∈ {0, 1}k . From the definition T (k, π) and T¯(k, π) we can see that either p(0/x1 ...xk ) = π, p(1/x1 ...xk ) = 1 − π or p(0/x1 ...xk ) = 1 − π, p(1/x1 ...xk ) = π. That is why h(xk+1 /x1 ...xk ) = −(π log2 π + (1 − π) log2 (1 − π)) and, hence, h∞ = −(π log2 π + (1 − π) log2 (1 − π)). The claim is proved.

16

References [1] A.V.Aho,J.E. Hopcroft, J.D.Ulman. The desighn and analysis of computer algorithms , Reading, MA: Addison- Wesley, 1976. [2] Bently J.L., Sleator D.D., Tarjan R.E., Wei V.K. A Locally Adaptive Data Compression Scheme. Comm. ACM, v.29, 1986, pp.320-330. [3] P. Billingsley, Ergodic theory and information, John Wiley & Sons (1965). [4] E.J. Dudewicz and T.G. Ralley. The Handbook of Random Number Generation and Testing With TESTRAND Computer Code, v. 4 of American Series in Mathematical and Management Sciences. American Sciences Press, Inc., Columbus, Ohio, 1981. [5] Elias P. Interval and Recency Rank Source Coding: Two On-Line Adaptive Variable-Length Schemes, IEEE Trans. Inform. Theory, v.33, N 1,1987, pp.3-10. [6] Gallager R.G. Information Theory and Reliable Communication. Wiley, New York,1968. [7] Kendall M.G., Stuart A. The advanced theory of statistics; Vol.2: Inference and relationship . London, 1961. [8] Knuth D.E. The art of computer programming. Vol.2. Addison Wesley, 1981. [9] Kolmogorov A.N. Three approaches to the quantitative definition of information. Problems of Inform. Transmission, v.1, 1965, pp.3-11. [10] Krichevsky R. Universal Compression and Retrival. Kluver Academic Publishers, 1993. [11] Li M., Vitanyi P. An Introduction to Kolmogorov Complexity and Its Applications , Springer-Verlag, New York, 2nd Edition, 1997. [12] G. Marsaglia. The structure of linear congruential sequences. In: S. K. Zaremba, editor, Applications of Number Theory to Numerical Analysis, pages 248-285. Academic Press, New York, 1972.

17

[13] G. Marsaglia and A. Zaman. Monkey tests for random number generators. Computers Math. Applic., 26:1-10, 1993. [14] Maurer U. A universal statistical test for random bit generators., Journal of Cryptology, v.5, n.2, 1992, pp.89-105. [15] O. Moeschlin, E. Grycko, C. Pohl, and F. Steinert. Stochastics. Springer-Verlag, Berlin Heidelberg, 1998.

Experimental

[16] A. Moffat, An improved data structure for cumulative probability tables, 1999, Software – Practice and Experience, v.29, no. 7, pp.647-659. [17] Rozanov Yu.A. The Random Processes , Moscow, ”Nauka” (”Science”), 1971. [18] Rukhin A. and others. A statistical test suite for random and pseudorandom number generators for cryptographic applications. NIST Special Publication 800-22 (with revision dated May,15,2001). http://csrc.nist.gov/rng/SP800-22b.pdf [19] Ryabko B.Ya. Information Compression by a Book Stack, Problems of Information Transmission, v.16, N 4, 1980, pp.16-21. [20] Ryabko B.Ya. A locally adaptive data compression scheme (Letter), Comm. ACM, v.30, N 9, 1987, p.792. [21] B. Ryabko, J. Rissanen. Fast Adaptive Arithmetic Code for Large Alphabet Sources with Asymmetrical Distributions. IEEE Communications Letters,v. 7, no. 1, 2003,pp.33- 35. [22] B. Ya. Ryabko, V. S. Stognienko, Yu. I. Shokin. A new test for randomness and its application to some cryptographic problems. Journal of Statistical Planning and Inference, 2003, (accepted; available online, see: JSPI, doi:10.1016/S0378-3758(03)00149-6 )

18

Table 1 105 5 105 106 100 100 100 100 100 100 √ s s=20, |A1 | = 5 2 4 75 100 100 s=16 s=17 s=20 0 0 100 100 0 0 99 100 2 1 1 2 1 2 1 1 5 M=1000 M=2000 M = 10 M=20000 2 1 2 1 0 2 1 1 0 1 0 0 0 1 1 0 0 0 0 1 – – – 2 m=10 – – – 2 m=10 – – 1 1 L=6 L=7 Q=640 Q=1280 Approximate Entropy 1 2 2 7 parameters m=5 m=11 m=13 m=14 Random Excursions – – – 2 Random Excursions Variant – – – 2 Serial 0 1 2 2 parameters m=6 m=14 m=16 m=8 Lempel-Ziv Complexity – – – 1 Linear Complexity – – – 3 parameters M=2500 19 Name/Length (in bits) Order test Book stack parameters for both tests RRS parameters RAR ARJ Frequency Block Frequency parameters Cumulative Sums Runs Longest Run of Ones Rank Discrete Fourier Transform NonOverlapping Templates parameters Overlapping Templates parameters Universal Statistical parameters

5 104 56 42

Table 2 5 106 (RAN DU ) 5 106 (random bits) 100 3 100 0 √ s s=24, |A1 | = 5 2 100 1 s=24 s=24 100 0 100 0 2 1 2 1 6 M = 10 M = 105 3 2 2 2 2 0 1 1 89 9 5 5 m=10 m=10 4 1 m=10 m=10 1 2 L=9 L=9 Q=5120 Q=5120 Approximate Entropy 100 89 parameters m=17 m=17 Random Excursions 4 3 Random Excursions Variant 3 3 Serial 100 2 parameters m=19 m=19 Lempel-Ziv Complexity 0 0 Linear Complexity 4 3 parameters M=5000 M=2500 20 Name/Length (in bits) Order test Book stack parameters for both tests RRS parameters RAR ARJ Frequency Block Frequency parameters Cumulative Sums Runs Longest Run of Ones Rank Discrete Fourier Transform NonOverlapping Templates parameters Overlapping Templates parameters Universal Statistical parameters