Enumeration of Context-Free Languages and ... - cs.UManitoba.ca

Enumeration of Context-Free Languages and Related Structures

Alexander Okhotin Department of Mathematics, University of Turku FIN–20014 Turku, Finland

Michael Domaratzki Jodrey School of Computer Science, Acadia University Wolfville, NS B4P 2R6 Canada

Jeffrey Shallit School of Computer Science University of Waterloo Waterloo, Ontario N2L 3G1 Canada

Abstract In this paper, we consider the enumeration of context-free languages. In particular, for any reasonable descriptional complexity measure for context-free grammars, we demonstrate that the exact quantity of context-free languages of size is uncomputable. Nevertheless, we are able to give upper and lower bounds on the number of such languages. We also generalize our results to enumerate predicates that are recursively enumerable or co-recursively enumerable.

1

Introduction

Enumeration is the study of the number of distinct objects from an infinite set that are of a fixed, finite size. Enumeration has a long history of interaction with formal language theory, dating to the 1950s, where researchers were primarily interested in the enumeration of deterministic finite automata (DFAs). We refer the reader to Domaratzki et al. [2] for a bibliography of research on enumeration of finite automata. Recently, enumeration of other objects that characterize regular languages—particularly non-deterministic finite automata (NFAs) [2] and regular expressions [12]—has also been examined. These problems are often complicated by the fact that it is difficult to enumerate the number of NFAs, DFAs and regular expressions that generate distinct languages. Experimental results can also be obtained by exhaustively computing the number of objects generating distinct regular languages. However, we note that in each of these cases, the equivalence of two objects (DFAs, NFAs or regular expressions) is computable, a basic necessity in algorithms for explicitly computing these quantities. In this paper, we examine the problem of enumerating context-free languages (CFLs). It is well known that given two context-free grammars (CFGs), it is undecidable whether they generate

Research supported in part by NSERC. Supported by the Academy of Finland under grant 206039.

1

the same language. This eliminates the natural method for computing experimental results for the number of CFLs of a given size. However, it does not immediately preclude that enumerating the number of CFLs of size is computable given . To our knowledge, despite the simplicity of this problem, it has not been studied before. We note, however, that several authors have considered the unrelated problem of, for a fixed CFG , enumerating (exactly or approximately) the number of, or listing the, words of length generated by . For example, see D¨om¨osi [4], Bertoni et al. [1], Gore et al. [6] and Martinez [13]. We show that the enumeration of CFLs is uncomputable in general. Thus, it is impossible to compute the exact values algorithmically. However, we also give asymptotic bounds on the number of CFLs of size . We examine this problem with respect to different notions of the size of a CFG, including number of symbols and number of variables in CNF.

2

Preliminary Definitions

! " #

Let be a finite set of symbols, called letters. Then is the set of all finite sequences of letters from , which are called words. The empty word is the empty sequence of letters. The length , where , is , and is denoted . For any of a word and , we denote by the number of occurrences of in . A language is any subset of . For additional background in formal languages and automata theory, please see Yu [14] or Hopcroft and Ullman [11]. In particular, we assume that the reader is familiar with the concepts of DFAs and NFAs. where is an alphabet, is a A context-free grammar (CFG) is a 4-tuple finite set of variables (or non-terminals), is the unique start variable and is the set of productions. Each production in has the form where and . Let . We denote by the fact that there exists in with and for some . Let denote the reflexive, transitive closure of . The language generated by a CFG is denoted by

$

% '&(*))+,)-,. ( + + 13254 136( 4"7&(089. 4:) OW R n&.NAMTK>"V

n

We note that measure (a) is not well-behaved. However, there are also some compelling reasons for examining measure (a) as a descriptional complexity measure for CFGs. The most obvious is the link to the state complexity of regular languages. In the traditional conversion of a DFA to a (right-linear) CFG, a DFA with state complexity is converted to a grammar with variables. With this in mind, we can salvage the measure of “number of variables” as a descriptional complexity measure by insisting that our grammars be in Chomsky normal form (CNF). (A grammar is in CNF if all productions are of the form or , where are variables and is a terminal.) If is a CFL, we define

162 at a 172 1G) ) $ R 0 >&(*))+,)-,. is a CFG in CNF accepting $MTWV vcnf&$N., >j] :OK (i

This measure has been employed by Domaratzki et al.[3] for measuring the descriptional complexity tradeoffs between regular and context-free languages. We adopt this descriptional complexity measure in Section 5.2 for enumerating unary CFLs.

3

4

Uncomputability of Enumeration of CFLs

We begin with a proof that it is impossible to exactly compute the number of CFLs of a given size. We note that the undecidability result does not depend on the descriptional complexity measure, as long as it is itself computable and well-defined.

f&M.

Proof. Let us denote our well-behaved, computable descriptional complexity measure by n . Let })} be CFGs. Without loss of generality, we can assume that n&}. n&} . , since we can always add useless non-terminals or productions to pad the smaller grammar. Suppose that f&M. can be effectively computed for any given , and consider the following algorithm: Input: ) , such that n&. >n&} . A Let be the set of all CFGs of measure . Data structure: , a partition of . 1. Let ` /HOWKTh . 2. Compute >f&M. . 3. For all Q 4. For each group . 5. Determine whether S$M&. for all I W s 6. If )OW/ `$M&.T 8. If ¢ A 9. Break from for loop 10. Accept if } and } are in the same class in , reject otherwise. The general strategy is to create all equivalence classes of context-free languages, based on being able to compute f&M. . Then, once these classes have been created, we test if two grammars are equivalent by testing if they are in the same class. Since this is undecidable, computing f&M. must also be undecidable. To see that the algorithm works, consider that the last line (10) of the algorithm is eventually reached: since there are B Cf&M. distinct CFLs of size , by testing each grammar on words of successive lengths, eventually all equivalence classes will be separated from each other. Listing is trivial: we can simply list all possibilities for a CFG with size exhaustively. Theorem 1. Let denote the number of distinct CFLs of size with respect to a given, computable well-behaved descriptional complexity measure. Then the function is uncomputable.

Line (5) can be implemented by any general CFL parsing strategy: CYK or Earley’s Algorithm, for example. Line (10) can be tested since we only need to test for grammar equality, not equivalence, in order to conclude that and are or are not in the same class.

We can also consider the undecidability of other enumerative problems relating to context-free grammars.

f&M.

Theorem 2. Let denote the number of CFGs of size computable descriptional complexity measure) that generate

4

f&M.

(with respect to a well-behaved, . Then is not computable.

Proof. Supposing the computability of , let us construct an algorithm for solving the universality problem for context-free grammars (“determine, whether a given over generates ”), which is known to be undecidable, or, to be precise, -complete.

£¤

n&., L /OWm F)VVV¥¦ZT with ng& ., A . § 1. Let ` > 2. Compute >f&M. . 3. For all Q 4. For all ! $M&Z. 5. If ¨S 6. Remove from 7. if ¢ A 8. Accept if / , reject otherwise. The basic idea is the same as in Theorem 1, and it is more obvious in this case. There are /f&M. context-free grammars of size that generate , while each of the rest of the grammars cannot generate some string ©= . All these strings will eventually be found, and all grammars that generate languages other than will be removed from . Then lines (7–8) report the result. We note that, in contrast, the exact number of distinct CFGs of size that generate ª or a Input: , such that Let be all CFGs Data structure: , a subset of .

finite language is computable.

5

Enumerating CFLs

We now turn to the growth of some enumerative functions related to CFLs. Due to the results in the previous section, these functions are uncomputable. However, we are able to give estimates of their growth.

5.1

Enumerating CFLs by Number of Symbols

«

¬§&M. « Theorem 3. For all M)«/® , the following inequality holds: X Y¯&¬§§&M..>&±°®W.²X Yf&MW~"«~"®W.V Proof. For any word ³ F´ ´ ´ µ ´ over the alphabet ¶8·(38·O ´ T , CFG-like form 4I ¸2 4H 2 .. .. .. . . . 4H 2 µ

We first consider estimating the number of CFLs with symbols (i.e., the total sum of production denote the number of CFLs generated by CFGs lengths is ) over a letter alphabet. Let with symbols over a -letter alphabet. We begin with an upper bound:

5

defines a

4¯¹ 4¯6(>89

H4 L7 ( S&¡ (º~·«~·®W. »f L°c® MW (MW

where and . Note that if , then this does not define a valid CFG. However, this will suffice for our upper bound. Therefore, we see that for all words , we get at most one valid grammar with symbols. If is of length , there can be at most productions (at most every other symbol of can be ). Thus, we get and our upper bound is .

´

&MW~"«¼~"®W. »f

We now give a lower bound. We require the following result due to Domaratzki et al. [2]:

states over a M N½ »f ¾¿ Theorem 5. For all Q/® and «> , we have X Ym&¬g§&M.., À «¼«Z°"~"® LX YMÁ~"X YNi° À «¼«°¶~"® LX Ym& À «~"W.Â°"X Ym& À «~"W.Â~ À « ~" Proof. We consider the impact of converting a DFA with states over a « -letter alphabet to a CFG. Let Ã ³&Ä))ÅÆ)Çº[)È. . Define a CFG ³&ÄZ))+,)Ç[h. . For each Å¢&Çº)¥ ¥. ³ÇºÉ we define the production ÇºI2¥ |ÇÊÉ . Thus, for each of the M« transitions in Ã , we have a production À of length . Furthermore, for each final state ÇºËÈ , we have a production ÇºË2B , of length . À À Thus, the total number of symbols in is M«¤~AK ÈÌ>& «¼~AW. . Using Theorem 4, we obtain «]>

Theorem 4. The number of distinct regular languages generated by a DFA with letter alphabet is at least .

our result.

5.2

Enumerating CFLs by Number of variables

We now discuss enumeration of CFLs by the number of variables in a CNF CFG generating the language. We require the following theorem due to Domaratzki et al. [3]:

§»f T . Then there exists a CFG in QÀ*>Í ® Îand let $ be any subset of OP§)§)VVV¥) Ï Ð ~¶ variables generating $ . Let È¯§&M. give the number of distinct CFLs generated by a CNF CFG with at most variables over a « -letter alphabet. In the following theorem, « is considered a constant. gÒº¾ Theorem 7. ÈH§&M.ShÑf½ . Ë ¾ Proof. By Theorem 6, for some f&M.NAÓ&M. , each finite language $A\OP§)§)VVV¥) ½ T can be generated by a CNF CFG with variables. This establishes the lower bound. Ï For the upper bound, consider that there are ~"«¢ possible productions in a grammar with Ï variables in CNF: of the form 1ÔN2Õ1N |1Ô for any choice of ®]Ï )« . Each Òºsubset Ö . of the ~`«¢ possible productions defines a grammar, which yields an upper bound of Theorem 6. Let CNF with at most

We can make the discussion more precise for finite unary context-free languages.

×Z&M.

X Y¯&×Z&M.. Ø À Ï °¶®¼>X Y¯&×&M..M Ù Ï °9 ~ À Ù "° ÚZ&LX YMM.

Theorem 8. Let measure the number of distinct unary finite languages generated by a CFG in CNF with variables. Then satisfies

for

sufficiently large. 6

1G F)1Nh)VVV¡)1Ô 1¤ 1N2Ý1N ¥1N /« 1Ô2 1Ô |1Ô ¢)«Þ 12 Ò N Ö â gÏ Ò °9 ~>ß Ï possible productions, » § á Ò à possible sets of productions. Noting that the and mà Ò names of the non-start symbols are irrelevant, we can divide this quantity by &`°6®W.ã . Thus, by Ò Stirling’s Approximation, X Ym&×Z&M.., Ï °9 ~ ß Ï °"Ú&LX YMM. . For the lower bound, we make the lower bound of the proof of Theorem 7 more precise. In §Ò Ë ¾¹»f T can be generated by a CNF particular, if f&M.ä å °^® , then all subsets of OP)g)VVV¢) ½ À*Í Ò Ðg ß ÎÏ ~"A variables. This gives the result. CFG with & å °¶®W. ß

Proof. Let us first give the upper bound: Consider an arbitrary grammar in CNF with variables. with the start symbol. There are Without loss of generality, let the variables be different productions of the form with —the order of variables on the left-hand side is irrelevant, since unary languages commute, and the productions may be assumed to be of the form with , since the language generated is . Therefore, simplifying there are finite. Finally, there are productions of the form

Û Üf & Á° .&±° °¶®W.

5.3

Number of Universal Grammars

Motivated by Theorem 2, we can also give estimates on the number of CFGs generating . Note that we are not concerned with enumerating CFLs, as we were in the previous sections, but grammars. We first consider the measure of the number of variables in CNF. Note that CNF (this quantity grammars cannot generate , so the theorem discusses grammars that generate is also uncomputable, as is easily seen).

Ö

« . Denote by æç&M. the number of CNF CFGs with variables Ö A ® and ¢ >« . The following inequalities hold: &Á°"®W. Ï ~"&Á°"®W. ~"&Á°"®W.«]>X Ym&æçg&M..MA Ï ~9M« Proof. The upper bound is the trivial bound from Theorem 7. For the lower bound, consider a grammar in CNF with variables OW1G F)VVV¥)1NHT with 1¤ the start symbol. We set the productions of 1G to include the following productions: 1 2 1 1 1G Õ2 ZSV Ö Note that we now guarantee that the grammar generates . Now, every grammar including Ö from choosing the remaining these productions also generates . Thus, the lower bound )« . The following inequalities hold: &±° Ù «Z°"ëW.²X Ym&«Z~"®W.,>X Ym&êg&M..,/&Á°"®W.²X Ym&MW~«~"®W.V Ù «~®W . For the lower bound, we additionally require QÞ

Theorem 10. Let generating where

7

symbols

1 1 2 1ä1 1 2 QV 1 2 Ù Ø Ù Ø Then the total length of these productions is «¤~ . Consider the remaining length of !° «¤° Ù symbols. Any word over Ì8ZOW1GT of length ° «M°Áë can be extended to a production 172ì . For any such , adjoining it to the «~" productions above, we get a distinct grammar (provided us from duplicating the above productions). This gives a lower bound Á° Ù «°ëÞ/ , to prevent § § » Ï § » í . on êg&M. of &«Z~"®W. Proof. The upper bound is from Theorem 3. The lower bound is similar to the proof of Theorem 9. Consider again the grammar with fixed start symbol and productions

6

General results on Uncomputability of Enumerative Functions

Theorems 1 and 2 share a single method of argument: it is shown that knowing the values of an enumerative function allows one to solve a certain related problem, and the undecidability of that problem implies the uncomputability of the enumerative function. In this section we prove general theorems that can be used to establish results of this kind.

6.1

The number of objects satisfying a predicate

î

Consider the following abstract generalization of Theorem 1: instead of the set of context-free with any computable and well-behaved descriptional grammars, let us take any recursive set complexity measure defined on it; instead of the universality condition, let us take an arbitrary predicate on . We denote by the following quantity

n

{ &M. { &M. OPï`î R n&ïI.M AM) and +&ïI. holds. TK V That is, { &M. denotes the number of words of measure in î that satisfy the predicate + . Theorem 11. Let î be a recursive language, let ð be an oracle. Let + be a predicate on î that is recursively enumerable in ð or co-recursively enumerable in ð . Then, if { &M. is a computable function, then + is recursive in ð . Proof. Assume + is recursively enumerable (in ð ), let { be computable. Input: ï9î , such that ng&., A Let îLÌ >OPïI F)VVVºï¯¦ZT be all elements of î of measure Data structure: , a subset of îL . 1. Let ` ^ª 2. Compute >f&M. . 3. Start simulating the r.e. procedure for + simultaneously on all ñîL 4. For every ñ on which one of the procedures halts 5. Add ñ to 6. If ¢ A 7. End the simulation 8. Accept if ï9 , reject otherwise. + î

8

f&M.

All yes-instances of the predicate will eventually be found, as the corresponding r.e. procedures terminate. At this point the lines (6–7) will be invoked, and the acceptance condition in line (8) determines the result: is accepted if and only if is true. Since the constructed algorithm always halts, we have proved that is recursive in . The case of co-recursively enumerable (as in the special case treated in Theorem 1) follows is recursively enumerable in , and is computable. by observing that Therefore, by the first part of the theorem, is recursive in , and hence so is .

ï

ò,+

+ + ð ,ò +

ð

+&ï*.

hó { &M., î °Á { &M. ð +

£N ð

If the oracle is not used, the theorem infers recursiveness out of (co-)recursive enumerability of the predicate, provided that the enumerative function is computable. If we let be the oracle for the TM halting problem, then Theorem 11 states that if is in or in the arithmetical hierarchy and is computable, then . More generally, we have the following corollary:

+

{

+³ôA£N Corollary 1. Let î be a recursive set, let + be a predicate on î that is in ,8S£Ô for some «õ7® . If { is computable, then + is in ,äôÁ£N . In the following, ö denotes the symmetric difference of two sets: -: ö -²¼ >&-ç z÷:-²h.8t&-²ç÷ -ç F. . Corollary 2. For any predicate + that is in , ö £Ô , the corresponding function { is not computable. Corollary 3. For any predicate + that is complete for , or for £Ô , the corresponding function { is not computable. This allows us to obtain results of the following kind:

(a) The number of Turing machines of size sion problem is -complete).

that halt on is uncomputable (because the deci-

£¤

(b) The number of unambiguous context-free grammars of size problem is -complete).

is uncomputable (the decision

,

(c) The number of context-free grammars of size that generate languages with a context-free complement is not computable (the decision problem is -complete).

î

6.2

The number of equivalence classes

Let us similarly generalize Theorem 1, which states the uncomputability of the number of distinct CFLs generated by CFGs of a given size. In the general context, we consider a recursive set with a computable and well-behaved descriptional complexity measure , and an equivalence relation on , and we denote the number of equivalence classes on the elements of measure in by .

ø î î ùWúÔ&M.

î

n

ð

ø

ð

î

ùWúû&M. Proof. Let us start from the case of a recursively enumerable ø . Using the computability of ùú , construct the following decision procedure for ø :

Theorem 12. Let be a recursive language, let be an equivalence relation on that is recursively enumerable using an oracle or co-recursively enumerable using . Then, if is a computable function, then is recursive in .

ø

ð

9

ïîLÌ >)ng&ñm., î I of measure . îL 1. Let ` /HOWü TW)VVV¥)OWü ¦ Th . 2. Compute AùWúN&M. . 3. Start simulating the r.e. procedure for + simultaneously on all pairs &ï )