finiteness conditions on subgroups and formal language theory

FINITENESS CONDITIONS ON SUBGROUPS AND FORMAL LANGUAGE THEORY CHRISTIANE FROUGNY, JACQUES SAKAROVITCH, and PAUL SCHUPP [Received 31 December 1985—Revised 19 October 1987]

Dedicated to the memory of W. W. Boone ABSTRACT

We show in this article that the most usual finiteness conditions on a subgroup of a finitely generated group all have equivalent formulations in terms of formal language theory. This correspondence gives simple proofs of various theorems concerning intersections of subgroups and the preservation of finiteness conditions in a uniform manner. We then establish easily the theorems of Greibach and of Griffiths by considering free reductions of languages that describe the computations of pushdown automata in one case and of Turing machines in the other, thus making clear that they are essentially the same.

1. Introduction The profound connection between the theory of infinite discrete groups (combinatorial group theory) and decision problems has long been recognized. The cornerstone of this interrelationship is the unsolvability of the word problem, established independently by W. W. Boone [4] and P. S. Novikov [14], while its crowning achievement is the Higman embedding theorem: Let F be a finitely generated free group and let N be a normal subgroup of F. Then G = F/N is embeddable in afinitelypresented group if and only if N is recursively enumerable. Thus a certain finiteness condition is equivalent to the logical or 'languagetheoretic' condition that N is recursively enumerable. At a less profound level, we show in this article that the most usual finiteness conditions on a subgroup—having finite index, being finitely generated, or being the normal closure of finitely many elements—all have equivalent formulations in terms of formal language theory. The Chomsky hierarchy divides formal languages (in order of increasing complexity) into the classes of regular, context-free, context-sensitive and recursively enumerable languages. If 5 is a set of elements of a finitely generated group G = (X;R), then a set L of representatives of S is a set of words on the alphabet A = XUX~* such that for every element s of S there is at least one word in L which is equal to s in G and all words of L do represent elements of 5. A subset S of G has a regular, context-free, or context-sensitive enumeration if it has a set L of representatives which is a regular, context-free, or context-sensitive language in the usual sense of formal language theory. We shall also need the concept that a subset S of G is recognizable if the set L of all representatives of S is a regular language. This work was supported by the Programme de Recherches Coordonne"es Math6matiques et Informatique of the Ministere de la Recherche et de la Technologic The third author's work was supported by NSF grant DMS 83-02550. A.M.S. (1980) subject classification: 20F10. Proc. London Math. Soc. (3) 58 (1989) 74-88.

FINITENESS CONDITIONS ON SUBGROUPS

75

The results characterizing finiteness conditions on subgroups of a finitely generated group G = (X ;R) are the following. A subgroup H of G has finite index if and only if H is recognizable. A subgroup H of G is finitely generated if and only if H has a regular enumeration (which is a result of Anisimov and Seifert [2]). A normal subgroup N of G is the normal closure of finitely many elements if and only if N has a context-free enumeration. At this point the language hierarchy breaks down for groups—the subsets of G having a context-sensitive enumeration are exactly the recursively enumerable subsets. One may note that Theorems I and II immediately imply the old grouptheoretic result that a subgroup of finite index of a finitely generated group is finitely generated. As discussed in more detail under Corollary 3 in § 2 below, a finitely generated free group F has the property that a non-trivial normal subgroup N of F has finite index if and only if N is finitely generated. We may thus summarize finiteness conditions in the case of free groups in the following particularly succinct fashion. Scholium. Let F be a finitely generated free group, let N be a non-trivial normal subgroup of F and let G = F/N. Then G is finite if and only if N has a regular enumeration, G is finitely presentable if and only if N has a context-free enumeration, and G is embeddable in a finitely presented group if and only if N has a recursive enumeration. A question of considerable interest in group theory (and also in formal language theory) is the preservation of finiteness conditions under intersection. For example, the intersection of two subgroups of finite index is always of finite index while the intersection of two finitely generated subgroups need not be finitely generated. The situation in formal language theory is the following. The classes of regular, context-sensitive, and recursively enumerable languages are closed under intersection while the class of context-free language is definitely not closed under intersection. It is a basic fact about context-free languages that the intersection of a context-free language and a regular language is again contextfree. Using the interplay between finiteness conditions and formal languages we obtain simple proofs of various theorems concerning intersections of subgroups and the preservation of finiteness conditions in a uniform manner. Let G = (X ; R) be a finitely generated group and let H be a subgroup of finite index. The intersection of H and a finitely generated subgroup is again finitely generated. The intersection of H and a normal subgroup N which is the normal closure of finitely many elements contains a normal subgroup K which is the normal closure of finitely many elements and which has finite index in H(IN. We also obtain a simple proof of the theorem of Howson [11] that if F is a free group then the intersection of two finitely generated subgroups is again finitely generated. The third section is devoted to a series of examples in the free group F showing the failure of certain analogies between subgroups possessing context-free enumerations and the situation for context-free languages. In particular, a normal subgroup of F with a context-free enumeration need not be recursive. (This is simply the unsolvability of the word problem.) More interestingly, the intersection of a subgroup with a regular enumeration and a subgroup with a context-free enumeration need not have a context-free enumeration.

76

CHRISTIANE FROUGNY, JACQUES SAKAROVITCH, AND PAUL SCHUPP

In the last section we somewhat reverse the situation, applying the more or less group-theoretic point of view of considering free reductions to formal language theory. The principle is that it is easy to enumerate machine reductions up to free reductions. In particular we obtain simple proofs of two interesting theorems and show that they are essentially the same. The first is the theorem of Greibach [6] that the set of possible total states which can arise during the computation of a pushdown automaton M is a regular language. The second is the theorem of Griffiths [7] and of Stanat [15] that one can obtain every recursively enumerable language by taking the reduction of a context-free language and the intersection with a regular language. While the idea of considering the relationships between group theory and the lower levels of the Chomsky hierarchy occurred more or less simultaneously to several people the first article on the subject is that of Anisimov [1]. The subject has since expanded considerably. Since our present purpose is to continue this merger of two subjects, we carefully cite all the results needed from both areas. Our basic framework and notation follows Magnus, Karrass, and Solitar [13] for group theory and Hopcroft and Ullmann [10] for formal language theory. It is with marked sadness that we dedicate this article to the memory of our colleague W. W. Boone, who is no longer with us but whose influence on the subject of the relations between group theory and logic will always be present. 2. Finiteness condition and intersections In order to fix our notation and terminology we give a brief description of the group G defined by the presentation {X\R). The set X~x of inverses of generators is disjoint from X and there is a one-to-one correspondence x>-*x~l between X and X~\ We define (x'1)'1 to be x. We write A = X U X'1 and call A the alphabet and the elements of A letters. A word is a finite sequence of letters and we denote the set of all words over the alphabet A by A*. We denote the empty word by 1. The length \w\ of a word w is the number of letters in w. If w = ax... an then w"1 = a~l... of1. The set R of defining relators is an arbitrary subset of A*. Since we want G to be a group we need to consider the trivial relators, aa~x, where a eA. The trivial relators are understood and are not written as part of the presentation. Two words u and v on A* are equivalent (modulo R) if it is possible to transform u into v by a finite sequence of insertions and deletions of defining relators and trivial relators. The set of equivalence classes forms the group G where multiplication is concatenation of representatives, [u][v] = [uv], the identity element is the class of the empty word, and [w]"1 = [w"1]. As usual when working with equivalence relations, one uses the words of A* to name the elements of G and when doing group theory the distinction between the name of an element and the element itself is implicitly clear by context. The set A* is the free monoid generated by A where multiplication is simply concatenation of words and 1 is the identity element. The natural homomorphism a: A*—>G is defined by a(u) = [u]. Since we need to preserve the distinction between names and elements but we wish to avoid equivalence class notation we shall write either a(u) or 'the element w' to denote the element [u] of G. A language L over the alphabet A is simply a subset of A*.

FINITENESS CONDITIONS ON SUBGROUPS

77

As stated in the introduction, we follow Hopcroft and Ullman [10] for formal language theory. An important feature of formal language theory is that the classes of languages studied can be defined in either of two ways—acceptance by a machine of the appropriate type of generation by a grammar of the appropriate type. The Chomsky hierarchy arranges the four principal classes of languages according to the increasing generality of their grammars and corresponding machines as follows. Regular languages are defined by finite automata or left-linear grammars. Context-free languages are defined by pushdown automata or context-free grammars. Context-sensitive languages are defined by linear bounded Turing machines or context-sensitive grammars. Finally, recursively enumerable languages are defined by Turing machines or unrestricted grammars. Let G = (X ;R) be a finitely generated group, let A = XUX~1, and let a: A*^> G be the natural homomorphism. We shall say that a subset S of elements of G has a regular (respectively context-free, context-sensitive, recursive) enumeration if there exists a regular (respectively context-free, context-sensitive, recursively enumerable) language L^A* such that a(L) = S. In other words, S has a set of representatives which is a language of the appropriate type. Furthermore, we shall say that a subset S of G is recognizable if the set a~1(S) of all representatives of elements of S is a regular language. DEFINITION.

THEOREM I. A subgroup of afinitelygenerated group has finite index if and only if it is recognizable.

Proof. Let H be a subgroup of G = (X;R). By definition, the index of H is the index of the equivalence relation ~H defined by u ~H v if and only if v e Hu. On the other hand, the Nerode equivalence of a language L of A* denoted by =L is defined by u=L v if and only if for all z, uz eL if and only if vz e L. If L = oc~l{H), then the index of =L is equal to the index of ~H. A basic fact about regular languages is Nerode's theorem (cf. [10]) that L is regular if and only if =L has finite index. The next result is due to Anisimov and Seifert [2] but we give here a simpler proof. THEOREM II. A subgroup of afinitelygenerated group isfinitelygenerated if and only if it has a regular enumeration.

Proof. Let H be a subgroup of G = (X; R). First suppose that H is generated by the set {hlf..., hn}. For each hh there exists a representative M, in A* such that hi = a(Ui). Let L be the submonoid of A* generated by the set {uu . . . , « „ , Ui1, ...,u~1}, which we denote by {ux,...,un, u^1, ..., u " 1 } * . Clearly H = a(L). Kleene's theorem says that finite sets are regular and that the submonoid generated by a regular language is regular. Hence H has a regular enumeration. Conversely, let H = a(L) where L is a regular language. The pumping lemma for regular languages (see [10, p. 56]) states that there is a constant C > 0 such that if w is any word in L with \w\^C then there exists a factorization w = uvz such that \uv\ ^C,\v\^l, and for all i ss 0, uv'z e L.

78

CHRISTIANE FROUGNY, JACQUES SAKAROV1TCH, AND PAUL SCHUPP

Let U = {uvu^l u, v e A*, \uv\ ^ C, a(uvu~') e H}. The set U is clearly finite since its elements have bounded length and A is finite. We claim that H is equal to the subgroup, Gp(a(U)), generated by ce(U). For if the claim is false, pick an element of H\Gp(a(U)) having a representative w e L of length as small as possible. Now \w\^C since U contains all words of L of length less than C. Thus the pumping lemma gives a factorization w = uvz where \uv\ w where / and / are variables and w eA*. A basic theorem [10, Theorem 9.1] states that a language is regular if and only if it can be generated by a left-linear grammar. We shall obtain the set IT of possible total states of pushdown automaton M by taking the reductions of a regular language L over an alphabet A 3 Q U Z and then intersecting with QZ*. By the previous result, the reduction of a regular language remains regular and thus II is regular. Given a pushdown automaton M= (Q, 2, Z, 6, q0, z0, T), let Q~x and Z~x be sets in one-to-one correspondence with Q and Z respectively. We call A = QUZUQ~XU Z" 1 the description alphabet. Let A = {z~1z\ zeZ) be the set of 'right' reductions allowing a pair z~lz to be cancelled where 2 occurs to the right of z~\ THEOREM VII. The set of possible total states of a pushdown automaton is a regular language.

Proof. Given a pushdown automaton M = (Q, 2, Z, 6, q0, T) we keep the notation of the paragraph above. We turn to the definition of the appropriate left-linear grammar C. The set V of variables consists of a special start symbol 5 and variables Iq, q eQ. There is a special starting production S^>Iqozo. For each pair qeQ, z e Z such that there exists a symbol Ae(2U{e}) and a pair {q', w) e d(q, A, 2), there is a production Iq—>Iq-wz~*\ Finally, there is a production Iq—>q for each qeQ. The grammar C given above works in the following way. The only production applicable to the start symbol S is S-+IqoZo and qozo is the initial total state of M. One may think of applying a production Iq^*Iq'WZ~x as guessing that the symbol on top of the stack is 2, then inserting a z~x to cancel the 2, placing w on the stack, and then changing to the variable lq> which records that the state is now q'. If the guess is wrong then the letter z~x can never be cancelled. An easy induction shows that if there is a derivation S-*>Iq£, where £,eA*, then the unique reduced word which is congruent to qt, modulo A either contains a letter z~xeZ~x or is a possible total state of M. Furthermore, given a possible total state of M there exists a derivation S-**Iq£ such that the reduced form of qt, is the given total state. Finally, applying a production Iq^*q terminates a derivation by placing q at the left end of the terminal string. Letting L denote the language generated by C we have II = Red(L) D QZ* as desired. We now consider the computations of a possibly non-deterministic Turing machine M = (Q, 2, Z, 6, q0, b, T) where 2 is the input alphabet, Z 3 2 is the tape alphabet and b e Z is the symbol for blank. The transition function 6 is a function 6: QxZ^>P(QxZx{l,r})

86

CHRISTIANE FROUGNY, JACQUES SAKAROVITCH, AND PAUL SCHUPP

and (q1, z', d)ed(q, z) means that when M is in the state q reading the symbol z on the tape, then M can replace z by z', change state to q', and move to direction d. We shall assume that M always works on an input tape delimited by the special left and right end-markers # and / respectively. Thus M cannot move to the left of # and if M needs more working space, M replaces the symbol / by a blank and then writes / after the new blank. We assume that M starts in the initial state q0 with a tape #w>/ where w e 2* and with the reading-head scanning the first symbol to the right of # . An instantaneous description of M is a word £i