the dynamics of semigroups of rational functions i - Semantic Scholar

THE DYNAMICS OF SEMIGROUPS OF RATIONAL FUNCTIONS I A. HINKKANEN and G. J. MARTIN [Received 4 October 1993—Revised 10 July 1995]

ABSTRACT

This paper is concerned with a generalisation of the classical theory of the dynamics associated to the iteration of a rational mapping of the Riemann sphere, to the more general setting of the dynamics associated to an arbitrary semigroup of rational mappings. We are partly motivated by results of Gehring and Martin which show that certain parameter spaces for KJeinian groups are essentially the stable basins of infinity for certain polynomial semigroups. Here we discuss the structure of the Fatou and Julia sets and their basic properties. We investigate to what extent Sullivan's 'no wandering domains' theorem remains valid. We obtain a complete generalisation of the classical results concerning classification of basins and their associated dynamics under an algebraic hypothesis analogous to the group-theoretical notion of 'virtually abelian'. We show that, in general, polynomial semigroups can have wandering domains. We put forward some conjectures regarding what we believe might be true. We also prove a theorem about the existence of filled in Julia sets for certain polynomial semigroups with specific applications to the theory of Kleinian groups in mind.

1. Introduction This is the first of a series of papers where we extend the classical theory of the dynamics associated to the iteration of a rational function of a complex variable to the more general setting of an arbitrary semigroup of rational functions. These semigroups of rational functions may be finitely or infinitely generated and usually will satisfy some finiteness condition. For instance, we may assume that there are only a finite number of elements of a given degree in the semigroup. This is the case for finitely generated semigroups. A principal aim is to see how far the classical theory of Fatou and Julia [6,11] applies in this more general setting. Particularly, we consider the classification of periodic components of the Fatou set and the associated dynamics, and we are interested in seeing to what extent such things as Sullivan's no-wanderingdomains theorem [15] remains valid and to what extent the classification theorem for the dynamics on the Fatou set holds. Also we are interested in finding out what new phenomena can occur. In this paper we pose a number of questions, often phrased as conjectures, regarding the phenomena which seem to us to be particularly interesting. The situation in the general setting of rational semigroups is largely complicated by two facts. First, the Fatou set (where the dynamics are regarded as stable) is only forward invariant and the Julia set (where the dynamics are regarded as unstable) is only backward invariant. Classically both these sets are The research of the first author has been partially supported by the Alfred P. Sloan Foundation and by the U.S. National Science Foundation grant DMS 91-07336. The second author was partially supported by the Foundation of Research, Science and Technology, New Zealand. This research was completed while the first author was visiting the University of Auckland, Auckland, New Zealand. He wishes to thank the Department of Mathematics for its hospitality. 1991 Mathematics Subject Classification: 30D05, 58F23. Proc. London Math. Soc. (3) 73 (1996) 358-384.

SEMIGROUPS OF RATIONAL FUNCTIONS

359

completely invariant. Secondly, a rational semigroup need not be, and most often will not be, abelian. Classically the semigroup is actually cyclic. The simple algebraic structure in the cyclic case implies strong consequences concerning the possible dynamics. A case of particular concern is that of a polynomial semigroup. Here we would like to describe the structure of the filled-in Julia set. This is because of applications in the theory of discrete groups (which is where the initial motivation for this study came from). It turns out that various one-complex-dimensional moduli spaces for discrete groups can be described as the complements of the filled-in Julia sets for the dynamics of certain polynomial semigroups. This is explained in [7] and the polynomials are exhibited in [8]. A well-known instance is the moduli space of two-generator discrete free groups generated by parabolics. We shall describe the associated polynomial semigroup in a little detail later (see §7), but we point out that all hyperbolic two:bridge knot complements appear as points which have an essentially finite orbit under the semigroup. It is important from many points of view to get an accurate description of these spaces. From Teichmiiller theory, we know that the moduli space (and hence the filled-in Julia sets) are simply connected. But an important question is whether this region is locally connected and whether the closure is topologically a closed disk. We hope to approach these questions in subsequent papers using a dynamical systems approach. We offer here a first result concerning the attracting basin at infinity of a polynomial semigroup satisfying a certain finiteness condition; see §7. Another interesting problem is relating the algebraic structure of the semigroup to the dynamics. As we mentioned above in the case of iteration of a rational function, the semigroup generated is cyclic and therefore abelian. We expect that in the generic case a semigroup will be free on its generators and therefore have a large Julia set, often with non-empty interior. We shall try to develop criteria to make such statements rigorous. We shall see that even simple polynomial semigroups generated by two quadratic mappings are sufficiently complicated to exhibit new and interesting phenomena. We shall use A. Beardon's book [4] as a general reference for basic facts concerning the iteration of a rational function. We also recommend this book as a basic introduction to this theory. We thank the referees for their helpful comments. 2. Definitions and basic facts A rational semigroup G is a semigroup generated by a family of non-constant meromorphic functions, {/i,/2, ...,//i> •••}> defined on the Riemann sphere C with the semigroup operation being functional composition. We denote this situation by G

=

(fl,fl,

•••yfn, •••)•

Thus each g e G is a rational function and G is closed under composition. The rational semigroup generated by a single analytic function g is denoted (g). A rational semigroup G is called a polynomial semigroup if each g e G is a polynomial. We denote the nth iterate of/by/". We shall henceforth assume that a rational semigroup G contains at least one rational function of degree at least 2.

360

A. HINKKANEN AND G. J. MARTIN

All notions of convergence will be with respect to the spherical metric on the Riemann sphere C. A family of meromorphic functions & is normal in a region Q if every sequence of elements of 2F contains a subsequence which converges locally uniformly in Q to a meromorphic function (possibly the constant \a\}.

First observe that both rational functions z—>z2 and z-*z2/a map the regions {z: |z||fl|} into themselves, and therefore the semigroup they generate contains these regions in the associated set of normality N(G). Next observe that J(z2) = {z: \z\ = 1} and J(z2/a) = {z: \z\ = \a\}. As J(G) is backward invariant, we see that {z: \z\ = \a\i}^J(G) and then, by an elementary induction, that J(G) contains all circles of radius \a\' for any t = kl~" with 1 «£ k ^ 2". Since

SEMIGROUPS OF RATIONAL FUNCTIONS

361

J(G) is closed, the result claimed then follows. Alternatively, one may note that G = {z2"laq: 0^q l } into itself, so that {z: | z | > l } g N ( G ) , while J(2z2- 1) = [-1,1] and the inverse images of this interval under the mappings z2" are dense in the closed unit disk. (In fact Gn = (z2, Tn(z)) has this property for all n 5* 2.) EXAMPLE 2

There are some general theorems in the theory of discrete groups which show how to analyse the combination of two or more discrete groups. These are the Klein-Maskit combination theorems; see [12]. One can use the ideas used there to analyse the structure of the combination of two or more dynamical systems associated to the iteration of different rational maps in some special cases. These often provide good examples on which to test conjectures. Thus we present the simplest possible version here. We first establish an elementary lemma on which the result we seek is based. LEMMA 2.2. Let G be a rational semigroup and U a non-empty open set such that g(U)C\U = 0 for all but finitely many geG. Then U c N(G).

Proof. Apart from finitely many exceptional elements, G omits U on U. As U is open, it contains more than three points. Thus G is normal on U by Montel's criterion. Let G be a rational semigroup and U a (non-empty) open backward fundamental set for G. That is, g~\U)n U = 0 for all g EG\ {Identity} (thus U is also a

362

A. HINKKANEN AND G. J. MARTIN

forward fundamental set). Similarly, let V be a backward fundamental set for the rational semigroup H. By the lemma above, this implies that U UD

Theorem 2.3 is proved in the same way as Lemma 2.2. Actually, with a little care one can build up a picture of J(F) from J(G) and J{H) using the proof. To provide examples one might, for instance, take U and V to be annular regions about attracting fixed points of two rational maps / and g. Then the semigroup (/, g) is free on the two generators. There are other possibilities for combination theorems where one is able to deduce a little of the structure of the Julia sets. As a basic observation note that if G and H are rational semigroups and there is a set V for which f(V) (G). Also the identity mapping, being equal to [/,/] for any /, lies in 3>(G). As the rational functions that we are considering here are locally invertible outside a finite subset of the Riemann sphere, the commutators can locally be expressed using inverses. Thus many of the usual commutator identities for groups can be verified. We note the following examples: 1. [/,/] = Identity;

4.

5[f°g>gof)ogof=fog°[g>fl These identities can be verified by straightforward calculations. For instance, [f>g°fn] = [f>g] Allows because / ° g °/" = [/,g] ° g ° / " + 1 and a l s o / ° g o / « = LEMMA 4.2. Let G be a nearly abelian semigroup. Then for all f e G and for all 17 G 4>(G) there is a Mobius transformation y such that

(9)

f°l = y°f-

Moreover there are , if/ e $(G) such that y = if/ ° (j>. Proof. Given rj e $ ( G ) there are g,h e G such that goh = t]°h°g. (10)

/ o g ° h=f °

T; °

Then

h°g

and furthermore, there are ,il/ e $ ( G ) such that (11)

f ° g ° h = if/ ° g ° f ° h = if/ ° (f) °f ° h ° g.

As f,g,h are all rational functions, the desired equality / ° TJ = y ° / follows with y = ifj o (f).

We note that, if r/ e $(G) is given, it is not always possible to find any element y of the (semi)group generated by the functions in $(G) such that 17 °f =f ° y. To see this, set G = (f, g), where f(z) = z2 - i g = (f>°f, and $(z) = - z . Then / has a parabolic cycle at z = - J. Note that / ° / = / and f = f°y, no matter how y e \ ° 2° «Ai ° ^2°f it ~ y\ ° Ii\

O

fi2°fu°fi3

Jii °//4 Jh>

from which it follows that /', % °fh °fu = 72 °A °/«4 0 A % = 73 % % % % • The following lemma may be of interest in applications as it explains to what extent it suffices to consider the commutators of generators. The proof is straightforward, in view of the definitions and Lemma 4.2, and is therefore left to the reader. LEMMA 4.3. Suppose that G = (/i,/ 2 , ...,^,, •••) is a rational semigroup and that 3> is a family of Mb'bius transformations such that [fi, fk] e 4> for all j,k. Then for all f,g G G, we may write [/, g] = i ° ... ° n for some elements \,...,nofQ?.ln other words,

Notice that in Theorem 4.3, the group H is a subgroup of the conformal automorphism group of J(G). Finally in this section we wish to point out the following result, largely contained in A. Beardon's work [2,3]. THEOREM 4.4. Let X be a compact set in the plane which is not a round circle. Let G = {g: g is a polynomial and J(g) = X). Suppose that G contains a polynomial of degree at least 2. Then G is a nearly abelian polynomial semigroup. The set ^(G) is a finite collection of elliptic transformations and the group generated by $(G) is finite and cyclic.

Proof. The only thing not clear is that the commutator semigroup generates a finite cyclic group. To see this, recall that any commutator e O(G) is a linear polynomial (since itfixesinfinity) and that the group of linear transformations of a compact set, which is not a round circle, is finite cyclic. (To prove this last

370

A. HINKKANEN AND G. J. MARTIN

claim, notice that no / e Aut(A') can be loxodromic or parabolic, since one of its fixed points must be at (G) and U e °U. It is easy to verify from the precompactness of the family 3>(G), that si consists of a finite number of components of N(G). (For instance, we may normalise so that °° e N(G) and then observe that there are only finitely many components whose area is larger than any given constant. Since O(G) is precompact, there is a uniform bound on the amount any element of O(G) can decrease the area of any U e °ti.) We now observe that if g e G and U G % then g(U) e si. To see this, simply observe that for every integer m, we have g(U) = g(fm(U)) = Mm(g(U))),

(14)

and if m is taken sufficiently large, then fm(g(U)) e °ti. Next let V be any component of N(G) and suppose that V is wandering. Choose an infinite sequence g, e G such that the sets g,(V) = Vt are disjoint. Choose an integer n such that fn(V) = U e %. As / " has finite degree, the collection {fn(Vt)}?=1 must contain an infinite number of components of N(G). However, for each / we see that (15)

fW)

=f"(gi(V)) = 4>t ° gt °

for some $, e $(G). However, it is again easy to see from the precompactness of the set 3>(G) that in fact the set {(si): e 0>(G)} is a finite collection of components of N(G). We next construct an example of an infinitely generated polynomial semigroup with a wandering domain. It may be of some interest to note that our example nevertheless satisfies the condition of being 'of finite type', to be given later, at the beginning of §7.

Suppose that 0 < a < § and set (16)

p(z) = az(l+z).

Then 0 is an attracting fixed point of the mapping p. Let vv0 = \ and (17)

wn=p"(w0).

Then wn are real, positive and wn —» 0 as n —> °°. Let r0 = Aw0 and s0 = \r0, where A is a positive constant to be determined later. Define (18)

rn + l = arn(l + rn +2wn),

(19)

sn+l=asn(l+sn+2wn).

We first claim that the disks (20)

An = {z: | z - w j «£/•„}

are all disjoint. Since the wn are real and decrease to 0, it is easily seen that it suffices to show that (21)

rn + r n + 1 < \wn - wn+1\.

372

A. HINKKANEN AND G. J. MARTIN

Now \wn - wn+i\ = wn(l - a - awn) > \wn and rn + rn+1 = rn{\ +a+arn+

2awn) < 2rn.

Thus the disks will be disjoint if we can show for all n that Arn < wn. This is where we must choose A suitably. We first observe that wn and rn eventually tend to zero geometrically. In fact rn + l/rn^>a and wn+1/wn —>a as n —>°°. Therefore the following infinite product converges and the sequence of partial products increases:

(22)

f [ (1 + rn + wn) = a > 1.

We choose A with A *£ 1/(4a). Note that ro/wo = A < \. We compute that

From this we deduce by induction that

wn+i

- 5vvn ( 1 + r+w)n2, and if, for some m 2*0, {hv ° gm){V) is simply connected and sufficiently small (which can be arranged by taking V small enough), then also (h" ° gm){V) is simply connected for all n 2s v and this same m. Let us choose here n > max{v, n0} and then choose m depending on this n as before (so that (gm ° h")(V) is already known to be multiply connected). The map (hv ° gm)(£ + zg) - hv(zg) behaves like £k for £ in a neighbourhood of the origin, for some k 2= 2. Thus zg has a topological disk neighbourhood V for which (hy ° gm)(V) is simply connected, as required. This completes the proof of (33). Now fix m for which (33) holds. Set b = g(d)(zg)/d\ ¥> 0 and note that gU)(zg) = 0 for l^j^d-1. Suppose that (gm)u)(zh) = 0 for 1 =£/ ^ K -1 and that m (K) where K S* 1. For k,n s* 1, write ipk,n = gm+k ° h" and {g ) {zh)lK\=a*0,

Xk,n = h"°gm+k, implies that

so that iftk,n = 0. This shows that h'(hj(z8)) * 0 for all /ssO. We have h'(z)—* A as z-+Zh- Since h'{zg)-*Zh as ;—>°c and since 0 < |A| < 1, we obtain a contradiction by taking /: and n to be sufficiently large. Case l(ii). Suppose that U is parabolic for h. Then the parabolic fixed point of h lies on the boundary of U. Then for all m, we have (hn °gm)(zg) = h"(zg)—> zh e dU. However, (gm ° hn){zg) can be made to be arbitrarily close to zg by a judicious choice of m, for a given n. As zg is an interior point of U and z>, is a boundary point, we again reach a contradiction to the precompactness of the family {4>nJ. Case l(iii). Suppose that U is a Siegel disk for h. The orbit of zg under /i is

precompact in U and its closure is topologically a circle unless h{zg) = zg. Suppose that h(zg)^zg. Choose a small neighbourhood V of zg- Since the preimages of zg under (the iterates of) g are isolated in U, we can find an n such that h"(zg) is not

378

A. H1NKKANEN AND G. J. MARTIN

a preimage of zg under g. Then, if V is small enough and m is sufficiently large, the domain (hn ° gm)(V) is simply connected, while (gm ° h")(V) is not. Thus we conclude that h(zg) = zg. Next, after conjugating by a suitable Mbbius transformation, we may assume that zg - zh = 0 and that the power series expansion of g at the origin looks like zd + higher order terms. As h has a Siegel disk centred at 0, its power series expansion looks like cz + higher order terms with c = e2m9 and 6 irrational. It follows that [g, h] is an elliptic Mobius transformation of infinite order. Therefore U is a domain mapped onto itself by an elliptic Mobius transformation of infinite order with an interior fixed point. Thus U is a round disk. By the reflection principle applied to h in dU, we see that U cannot be the Siegel disk of a map of degree at least 2. This completes the proof of Case 1. Case 2. Suppose that U is attracting for g e G and let zg e V denote the attracting fixed point of g. We claim that U is attracting for each h G G of degree at least 2. Case 2(i). Suppose that U is superattracting for h. This case is covered in Case l(i). Case 2(ii). Suppose that U is parabolic for h. Then the proof given in Case l(ii) easily generalises. Case 2(iii). Suppose that U is a Siegel disk for h. As U is a component of N(G) = N(g), we find that cg e U for some critical point cg of g. The orbit of cg under h is again precompact and its closure is topologically a circle, unless h{cg) = cg. If h(cg) T* cg, then h" ° g has degree at least 2 near cg, and g ° h" does not, for a suitable choice of n. This leads to a contradiction. Thus h(cg) = cg. Similarly, there is c' e U with c' ¥^cg and g(c') = cg. Now g2 is branched over c', so that h would have to fix both cg and c'. This is impossible. Case 3. Suppose that U is parabolic for g e G and let zg denote the parabolic fixed point of g. As above, it is clear that zg e dU. We claim that U is parabolic for each h £ G of degree at least 2. Case 3(i). Suppose that U is attracting or superattracting for h. This case is covered in Cases l(ii) and 2(ii). Case 3(ii). Suppose that U is a Siegel disk for h. As above, a critical point cg of g lies in U. Also there is c' G U\{C8} with g(c') = cg. The argument concerning

the branching leads, as before, to a contradiction since it implies that cg and c' are both fixed by h. Suppose that U is of Siegel or Herman type, so that U can be mapped by a conformal mapping