Some Experimental Results with Tree Adjunct Grammar Guided Genetic Programming 1
2
N.X.Hoai , R.I. McKay , and D. Essam
1
2
School of Computer Science, University of New South Wales, ADFA campus, Canberra, ACT 2600, Australia,
[email protected] 2 rim,
[email protected]
Abstract. Tree-adjunct grammar guided genetic programming (TAG3P) [5] is a grammar guided genetic programming system that uses context-free grammars along with tree-adjunct grammars as means to set language bias for the genetic programming system. In this paper, we show the experimental results of TAG3P on two problems: symbolic regression and trigonometric identity discovery. The results show that TAG3P works well on those problems.
1 Introduction Genetic programming (GP), first introduced in [8], can be seen as a machine learning method, which induces a population of computer programs by evolutionary means [1] to solve some particular problems. Genetic programming has been used successfully in generating computer programs for solving a number of problems in a wide range of areas. In [5], we proposed a framework for a grammar-guided genetic programming system called tree-adjunct grammar guided genetic programming (TAG3P), which uses tree-adjunct grammars along with context-free grammars to guide the evolutionary process in genetic programming. In this paper, we show some experimental results of TAG3P on the symbolic regression and trigonometric identities problems, preliminary results have been presented in [3, 4]. The organization of the remainder of the paper is as follows. In section 2, we give a brief overview of related work on grammar guided genetic programming. The definition of tree adjunct grammars and the components of TAG3P will be given in section 3. Section 4 describes the problem spaces and experimental setups for TAG3P. The results will be given and discussed in section 5. The paper concludes with section 6, which contains conclusion and future work.
2 Grammar Guided Genetic Programming Grammar guided genetic programming systems are genetic programming systems that use grammars to set syntactical constraints on programs. The use of grammars
also helps these genetic programming systems to overcome the closure requirement in canonical genetic programming, which cannot always be fulfilled [17]. Using grammars to set syntactical constraints was first introduced by Whigham [17]: where context-free grammars were used. Whigham [18] also incorporated domain knowledge to modify grammars in order to bias the search process; he demonstrated positive results on the 6-multiplexer problem. Gruau [2] proved that using syntactical constraints can reduce the size of the search space. He also used context-free grammars to describe target languages but did not limit the depth of derivation trees, which led to severe growth in tree size [14]. Wong and Leung [19] used logic grammars to combine inductive logic programming and genetic programming. They have succeeded in incorporating domain knowledge into logic grammars to guide the evolutionary process of logic programs. However, because they did not maintain explicit parse trees, their system suffers from ambiguity when it tries to generate programs from parse trees [16]. Ryan et al. [16] proposed a system called grammatical evolution (GE), which can evolve programs in any language, provided that this language can be described by a context-free grammar. Their system differs from Whigham’s system in that they do not evolve derivation trees directly. Instead, genomes in GE are binary strings representing eight-bit numbers; each number is used to make the choice of the production rule for the non-terminal symbol being processed. GE has been shown to outperform canonical GP on a number of problems [9, 10, 12]. One of the great advantages of GE is the linear genome structure, which allows GE to employ well-studied operators of genetic algorithms, and to reduce the bias in treebased systems whereby the nodes closer to the leaves are often selected for genetic operators [1]. Moreover, the representation above provides a many-to-one genotypeto-phenotype map, which can provide neutral evolution in the genome space. On the other hand, GE cannot guarantee to generate legal genomes. This creates some problems such as introns and unnatural multiple uses of genes [5]. To cope with these problems, they used pruning to cut the unexpressed parts of genomes, and allowed the system optionally use wrapping [16].
3 Tree Adjunct Grammar Guided Genetic Programming 3.1 Tree Adjunct Grammars Tree-adjunct grammars are tree-rewriting systems, defined in [6] as follows: Definition 1: a tree-adjunct grammar comprises of 5-tuple (T, V, I, A, S), where T is a finite set of terminal symbols; V is a finite set of non-terminal symbols (T ∩ V = ∅); S ∈ V is a distinguished symbol called the start symbol. I is a set of trees called initial trees. An initial tree is defined as follows: the root node is S; all interior nodes are labelled by non-terminal symbols; each node on the frontier is labelled by a terminal symbol. A is a finite set of trees called auxiliary trees, which can be defined as follows: internal nodes are labelled by non-terminal symbols; a node on the frontier is labelled by a terminal or non-terminal symbol; there is a special non-terminal node on the frontier called the foot node. The requirement for the foot node is it must be
labelled by the same (non-terminal) symbol as the root node of the tree. We will follow the convention in [7] to mark the foot node with an asterisk (*). The trees in E= I ∪ A are called elementary trees. Initial trees and auxiliary trees are denoted α and β respectively; and a node labelled by a non-terminal (resp. terminal) symbol is sometime called a non-terminal (resp. terminal) node. An elementary tree is called X-type if its root is labelled by the non-terminal symbol X. The key operation used with tree-adjunct grammars is the adjunction of trees. Adjunction can build a new (derived) tree γ from an auxiliary tree β and a tree α (initial, auxiliary or derived). If a tree α has a non-terminal node labelled A, and β is an A-type tree then the adjunction of β into α to produce γ is as follows. Firstly, the sub-tree α1 rooted at A is temporarily disconnected from α. Next, β is attached to α to replace this sub-tree. Finally, α1 is attached back to the foot node of β. γ is the final derived tree achieved from this process. Adjunction is illustrated in Figure 1. A A
A A* A
Figure 1. Adjunction.
The tree set of a TAG can be defined as follows [6]: TG = {all tree t / t is completed and t is derived from some initial trees} Where a tree t is completed, if t is an initial tree or all of the leaf nodes of t are nonterminal nodes; and a tree t is said to be derived from a TAG G if and only if t results from an adjunction sequence (the derivation sequence) of the form: α β1(a1) β2(a2)... βn(an) , where n is an arbitrary integer, α , βi (i=1,2..n) are initial and auxiliary trees of G and ai (i=1,2..n) are node address where adjunctions take place. An adjunction sequence may be denoted as (*). The language LG generated by a TAG is then defined as the set of yields of all trees in TG. LG = {w ∈ T* / w is the yield of some tree t ∈ TG} The set of languages generated by TAGs (called TAL) is a superset of context-free languages; and is properly included in indexed languages [6]. More properties of TAL can be found in [7]. One special class of tree-adjunct grammars (TAGs) is lexicalized tree-adjunct grammars (LTAG) where each elementary tree of a LTAG must have at least one terminal node. It has been proved that for any context-free grammar G, there exists a LTAG Glex that generates the same language and tree set with G (Glex is then said to strongly lexicalize G) [7]. 3.2 Tree Adjunct Grammar Guided Genetic Programming In [5], we proposed a grammar guided genetic programming system called TAG3P, which uses a pairs consisting of a context-free grammar G and its corresponding
LTAG Glex to guide the evolutionary process. The main idea of TAG3P is to evolve the derivation sequence in Glex (genotype) rather than evolve the derivation tree in G as in [17]. Therefore, it creates a genotype-to-phenotype map. As in canonical genetic programming [8], TAG3P comprises of the following five main components: Program representation: a modified version of the linear derivation sequence (*), but the adjoining address of the tree βi is in the tree βi-1. Thus, the genome structure in TAG3P is linear and length-variant. Although the language and the tree set generated by LTAGs with the modified derivation sequence is yet to be determined, we have found pairs of G and Glex conforming to that derivation form for a number of standard problems in genetic programming [3, 4]. Initialization procedure: a procedure for initializing a population is given in [5]. To initialize an individual, TAG3P starts with selecting a length at random; next, it first picks up randomly an α tree of Glex then a random sequence of β trees and adjoining addresses. It has been proved that this procedure can always generate legal genomes of arbitrary and finite lengths [5]. Fitness Evaluation: the same as in canonical genetic programming [8]. Genetic operators: in [5], we proposed two types of crossover operators, namely one-point and two-point crossover, and three mutation operators, which are replacement, insertion and deletion. The crossover operators in TAG3P are similar to those in genetic algorithms; however, the crossover point(s) is chosen carefully so that only legal genomes are produced. In replacement, a gene is picked up at random and the adjoining address of that gene is replaced by another adjoining address (adjoining address replacement); or, the gene itself is replaced by a compatible gene (gene replacement) so that the resultant genome is still valid. In insertion and deletion, a gene is inserted into or deleted from the genome respectively. With these carefully designed operators, TAG3P is guaranteed to produce only legal genomes. Selection in TAG3P is similar to canonical genetic programming and other grammar-guided genetic programming systems. Currently, reproduction is not employed by TAG3P. Parameters: are minimum length of genomes - MIN_LENGTH, maximum length of genomes MAX_LENGTH, size of population - POP_SIZE, maximum number of generations – MAX_GEN and probabilities for genetic operators. Some analysis of the advantages of TAG3P can be found in [3, 4, 5].
4 The Problem Space 4.1 The Symbolic Regression Problem The symbolic regression problem can be stated as finding a function in symbolic form that fits a given finite sample of data [8]. In [8], the problem is restricted to finding a function of one independent variable. In his book, Koza uses the binary operators +, -, /, and the unary operators sin, cos, log, exp, which makes the program space can be described by a finitely ambiguous context-free grammar G and the corresponding lexicalized tree-adjunct grammar Glex as follows [3]: The context-free grammar for the symbolic regression problem: G = (N={EXP, PRE, OP, VAR,},T= {X, sin, cos, log, ep, +,-,*,/, (, )},P,{EXP}} where ep is the exponential function, and the rule set P={EXP→EXP OP EXP, EXP→PRE ( EXP),
EXP→ VAR, OP→ +, OP→ - , OP→ *, OP→ /, PRE→ sin, PRE→ cos, PRE→ log, PRE→ ep, VAR→ X}. The tree adjunct grammar for the symbolic regression problem: Glex= {N={EXP, PRE, OP,VAR},T={X, sin, cos, log, ep,+, -, *, /, (, )}, I, A) where I∪ A is as in Figure 2. α: EXP β1: VAR
β2:
EXP
β3:
EXP
EXP OP EXP* EXP OP EXP* EXP OP
X
VAR + β5:
X EXP
EXP* OP EXP +
VAR -
VAR
X β6:
X β7:
EXP
- VAR
*
β9: EXP
X β10: EXP
EXP EXP OP EXP* VAR /
β 8:
EXP
EXP* OP
X
EXP*
*
EXP* OP EXP
VAR
β 4:
EXP
X EXP
EXP
EXP* OP EXP
VAR
/
X β11: EXP
VAR X
β12: EXP
PRE ( EXP* )
PRE ( EXP* )
PRE ( EXP* )
PRE ( EXP* )
sin
cos
ep
log
Figure 2 Elementary trees for Glex.
Table 1 below summarizes the experimental setup for the problem. Objective
Terminal Operands Terminal Operators Fitness Cases Raw fitness Standardized Fitness Hits Wrapper Genetic Operators Parameters
Success predicate
Find a function of one independent variable and one dependant variable that fits a given sample of 20 (xi, yi) data points, where the target function is the quadratic 4 3 2 polynomial X + X + X + X. X (the independent variable). The binary operators +, -, *, / and the unary operators sin, cos, exp and log. The given sample of 20 points in the interval [-1..+1]. The sum, taken over 20 fitness cases, of the errors. Same as raw fitness. The number of fitness cases for which the error < 0.01. The adjunctions of elementary trees of Glex to generate the derivation trees of G. Tournament selection, one-point crossover and gene replacement. MIN_LENGTH=2, MAX_LENGTH=20, POP_SIZE= 500, MAX_GEN=30, probability for crossover=0.9 and probability for replacement= 0.01. An individual scores 20 hits.
Table1. Experimental setup for the symbolic regression problem.
4.2 The Trigonometric Identities Problem The problem of finding trigonometric identities can be stated as follows [8]. Given a trigonometric function in symbolic form, a system must try to discover the alternative representations of the function. For example, the function cos(2x) has one 2 (of many) well-known alternative representation as 1-2sin (x). Koza [8] has shown that genetic programming can solve this problem. The function he examined was cos(2x) (as in [10]). It is noted that the cos function should not be present in the function set otherwise the problem will be much like the symbolic regression; and with this simple function the problem will become trivial. Therefore, we use the same function set and terminal set as in [8, 10]. The context-free grammar and the LTAG for the problem are as follows [4]: The context-free grammar for the problem of finding trigonometric identities: G = (N, T, P, {EXP}). Where N={EXP, PRE, OP, VAR, NUM} is the set of non-terminal symbols; T= {X, sin, +, -, *, /, (, ), 1.0}, P, {EXP}) is the set of terminal symbols; EXP is the start symbol; and the rule set P={EXP→EXP OP EXP, EXP→ PRE(EXP), EXP→VAR, EXP→ NUM, OP→ +, OP→ - , OP→ *, OP→ /, PRE→ sin , VAR→ X, NUM→ 1.0}. The tree-adjunct grammar for the problem of finding trigonometric identities Glex= (N, T, I, A). Where N={EXP, PRE, OP, VAR, NUM} is the set of non-terminal symbols; T={ X, sin, +, -, *, /, (, ), 1.0} is the set of terminal symbols; and I and A are the sets of initial and auxiliary trees respectively. The set of the elementary trees I ∪ A is as in Figure 3. α1 :
E X P α2 : E X P VAR
NUM
X β1 0 :
EXP NUM
sin
β1 1 :
OP EXP* EXP +
EXP PRE ( EXP* )
1 .0
EXP
NUM
1 .0 β1 4 :
β9 :
β1 2 :
EXP
OP EXP* EXP -
NUM
1 .0 β1 5 :
EXP
NUM 1 .0
OP
β1 3 :
EXP
EXP* EX P
*
-
1 .0
EXP*
/
1 .0 β1 7 :
EXP
EXP EXP* OP NUM
OP
NUM
1 .0 β1 6 :
EXP
EXP* OP EXP EX P* OP +
EXP
*
EXP
EXP
EXP* O P
NUM 1 .0
/
EXP NUM 1 .0
Figure3. Elementary trees for Glex the β1-β8 are the same with the β1-β8 in Figure 2.
The experimental setup for the problem is summarized in table 2.
5 Results For the symbolic regression problem, TAG3P consistently discovered the target function. Our experiment consisted of 50 runs, 48 of those 50 runs discovered the
target function within the maximum number of generations, which makes the probability of success is 96%. The cumulative frequency is depicted in Figure 4 (left). The best and average fitness in each generation of one typical successful run is shown in Figure 4 (right). In comparison, on the same problem and parameter settings the probability of success was nearly 30% by GP ([8] - page 203), 2% by GE in generational mode, and 90% by GE in steady-state mode [11]. One of the reasons for the success of TAG3P on the above problem lies with the superiour capability of TAG3P in preserving and combining building blocks [4, 5]. Building blocks [15] are sub-trees which are not completed. Functionally, they can be viewed as some potential modules. For the target function above, the atomic building blocks could be X+t and X*t, where t is a parameter representing the incomplete portion of the tree, which are exactly β1 and β3 in Figure 2. During the evolutionary process, these building blocks are preserved and combined to make even better building blocks. For example, if β1, β3 and β3 are brought together in a chromosome 2 then the corresponding blocks in the phenotype space will be X+X + t after adjunction takes place. For the trigonometric identities problem, TAG3P did successfully evolve some alternative representations of the function cos(2x). We conducted 50 runs and 18 out of these runs (which is 36%) were successful. In comparison, the probability of success in GE on this problem was 4% and 87% for generational mode and steadystate mode respectively [10]. Objective
Terminal Operands Terminal Operators Fitness Cases Raw fitness Standardized Fitness Hits Wrapper Genetic Operators Parameters
Success predicate
Find a function of one independent variable and one constant that fits a given sample of 20 (xi, yi) data points, where the target function is cos(2x). X (the independent variable), 1.0 (the constant). The binary operators +, - , *, / and the unary operator sin. The given sample of 20 points in the interval [0..2π]. The sum, taken over 20 fitness cases, of the errors. Same as raw fitness. The number of fitness cases for which the error < 0.01. The adjunctions of elementary trees of the Glexto generate the derivation trees of the G. Tournament selection, one-point crossover and gene replacement. MIN_LENGTH=2,MAX_LENGTH=20,POP_SIZE=500 ,MAX_GEN=200, probability for crossover=0.9 and probability for replacement= 0.05. An individual scores 20 hits.
Table 2. The experimental setup for the trigonometric identities problem.
TAG3P converged to the three well-known alternative representations of function 2 cos(2x), namely, 1-2sin (x), sin(2x+π/2) and sin(π/2-2x); whereas in [8, 10] the 2 functions 1-2sin (x) and sin(π/2-2x) were discovered but not sin(π/2+2x)). The constants π/2 (≈1.570796) was evolved in a number of ways. Some of them were: 1+sin(sin(sin(sin(sin(sin(1/sin(sin(sin(sin(sin(sin(sin(sin(1))))))))))))))≈1.570727, 1+ sin(sin(sin(sin(sin(sin(sin(sin(sin(sin(1+1))))+1)+1)))))≈1.570812, 1+1/(1+1/(1+1/(1+
(1+1)))) =11/7≈1.571492. The cumulative frequency is shown in Figure 5 (left). The best and average fitness in each generation of one successful run which stopped at 2 generation 72 finding the function 1-2sin (x) is shown in Figure 5 (right). As investigated in [4], the efficiency of TAG3P in replicating building blocks and the nature of the search space made it have a tendency to converge towards the functions sin(π/2-2x) and sin(π/2+2x) (16/18 successful runs). The building blocks in this problem are: sin(2x+1 ± t), sin(2x+sin(1/sin(1) ± t), sin(2x+1+1/sin(1) ± t), sin(12x ± t), sin(sin(1/sin(1))-2x ± t), and sin(1/sin(1)+1-2x ± t), which frequently appeared during almost all these 50 runs [4]. Figure 6 (left) shows how these building blocks (using t=0) approximate the target function. The best and average fitness in each generation is also given in Figure 6 (right).
Figure 4. The results for symbolic regression problem; on the left is the cumulative frequency; on the right are the best and average fitness in each generation of one successful run, which stopped at generation 14 finding the target function (the dotted line is the average fitness in each generation scaled down by the Log10 function).
Figure 5. The results for trigonometric identities problem; on the left is the cumulative frequency; on the right are the best and average fitness in each generation of one successful run, 2 which stopped at generation 72 finding the function 1-2sin (x) (the dotted line is the average fitness in each generation scaled down by the Log10 function).
Figure 6. The results for trigonometric identities problem; on the left are some basic building blocks; on the right is the best and average fitness in each generation of one successful run, which stopped at generation 54 finding the function sin(π/2-2x) (the dotted line is the average fitness in each scaled down by the Log10 function).
6 Conclusion The experimental results show that TAG3P works well on the symbolic regression and trigonometric identities problems. The efficiency of replicating building blocks helped TAG3P converge towards the solutions. In future, we will experiment with TAG3P on these problems using different genetic operators (two-point crossover, insertion, deletion) and compare the result to TAG3P with one-point crossover. The performance of TAG3P on a number of other benchmark problems for genetic programming will also be investigated. The effectiveness of TAG3P in preserving and combining building blocks will be theoretically analyzed. We are also developing a tree adjunct grammar guided genetic programming system with the most general form of derivation sequence attempt to make TAG3P universal (i.e. to solve all the problem of which the problem space can be described by an arbitrary context-free grammar).
References 1. Banzhaf, W., Nordin, P., Keller, R.E., Francone, F.D.: Genetic Programming: An Introduction. Morgan Kaufmann Pub, USA (1998). 2. Gruau, F.: On Using Syntactic Constraints with Genetic Programming. In: Angeline, P. J., Kinnear, K.E. Jr. (eds): Advances in Genetic Programming. The MIT Press, USA (1996) 377-394. 3. Hoai, N.X.: Solving The Symbolic Regression Problem with Tree-Adjunct Grammar Guided Genetic Programming: The Preliminary Results. (To appear in the Proceedings of th The 5 Autraliasia-Japan Co-Joint Workshop on Evolutionary Computation, Nov 2001, Ontago New Zealand).
4. Hoai, N.X.: Solving Trigonometric Identities with Tree Adjunct Grammar Guided Genetic Programming. (To appear in the Proceedings of The First International Workshop on Hybrid Intelligent Systems (HIS’01), Dec 2001, Adelaide, Australia). 5. Hoai N.X., McKay, R.I.: A Framework for Tree Adjunct Grammar Guided Genetic Programming. In: Abbass, H.A., Barlow. M. (eds): Proceedings of the Post-graduate ADFA Conference on Computer Science (PACCS’01) (2001) 93-99. 6. Joshi, A.K., Levy, L.S., Takahashi, M.: Tree Adjunct Grammars. Journal of Computer and System Sciences, 10(1) (1975) 136-163. 7. Joshi, A.K., Schabes, Y.: Tree Adjoining Grammars. In: Rozenberg G, Saloma A (eds) Handbook of Formal Languages. Springer-Verlag, NY, USA, vol 3 (1997) 69-123. 8. Koza, J.: Genetic Programming. The MIT Press, USA (1992). 9. O’Neill, M.: Automatic Programming with Grammatical Evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference Workshop Program, July 13-17, Orlando, Florida USA (1999). 10. O’Neil, M., Collin, J.J.: Grammar Evolution: Solving Trigonometric Identities. In: Proceedings of Mendel ’98, (1998) 111-119. 11. O’Neill, M., Ryan, C.: Grammatical Evolution: A Steady State Approach. In: Proceedings of the Second International Workshop on Frontiers in Evolutionary Algorithms 1998, (1998) 419-423. 12. O’Neil, M., Ryan, C.: Under the Hood of Grammatical Evolution. In: Banzhaf, W.,.Daida, J., Eiben, A.E., Garzon, M.H., Hovana, V., Jakiela, M., Smith, R.E. (eds): GECCO-99, Morgan Kaufmann Pub, USA (1999). 13. O’Neill, M., Ryan, C.: Genetic Code Degeneracy: Implication for Grammatical Evolution and Beyond. In: Proceedings of the European Conference on Artificial Life (1999). 14. Ratle, A., Sebag, M.: Genetic Programming and Domain Knowledge: Beyond the Limitations of Grammar-Guided Machine Discovery. Ecole Polytechnique, France (2000). (Available at: www.researchindex.com. Accessed: 30, May, 2001). 15. Rosca, J.P. and Ballard, D.H.: Genetic Programming with Adaptive Representations. Technical Report 489, The University of Rochester, Feb 1994. 16. Ryan, C., Collin, J.J., O’Neill, M.: Grammatical Evolution: Evolving Programs for an Arbitrary Language. In: Lecture Note in Computer Science 1391, Proceedings of the First European Workshop on Genetic Programming, Springer-Verlag (1998) 83-95. 17. Whigham, P.: Grammatically-based Genetic Programming. In: Proceedings of the Workshop on Genetic Programming: From Theory to Real-World Applications, Morgan Kaufmann Pub (1995) 33-41. 18. Whigham, P.: Search Bias, Language Bias and Genetic Programming. In Genetic Programming 1996, The MIT Press, USA, (1996) 230-237. 19. Wong, M.L., Leung, K.S.: Evolving Recursive Functions for Even-Parity Problem Using Genetic Programming. In: Angeline, P.J., Kinnear, K.E. Jr. (eds): Advances in Genetic Programming, The MIT Press, USA (1996) 221-240.
