Intelligent Information Systems VII Proceedings of the Workshop held in Malbork, Poland, June 15-19 , 1998
Emergent Thought Automated Agent Design Mariusz Nowostawski ?
[email protected] Aaron Sloman
[email protected] School of Computer Science The University of Birmingham, Edgbaston, Birmingham B15 2TT, UK February 1998
Abstract. The following report presents the preliminary results and theoretical
backgrounds of a multi-layer evolutionary mechanism. The project aims at developing a mechanism for automated agent design within a pre-designed multi-layer architecture. The core mechanism used in the experiments is a Genetic Programming algorithm working in a specially designed condition-action rules system. Described multi-layer evolution paradigm is inspired by notion of the niche & design spaces. There are presented plans for experiments with the multi-layer coevolutionary mechanism for parallel distributed search within restrained search space. Attention is paid to evolutionary mechanisms in general, with no commitments to any particular problem. In theory the multi-layer evolutionary process can be applied to a number of systems and problems, e.g. to Genetic Programming.
Keywords: agents, genetic algorithms, evolutionary mechanisms, architectures ?
The work was written by the rst author under supervision and with co-operation with the second author.
2
M.Nowostawski, A.Sloman
1 Introduction The main research within this project Emergent Thought, ET, is concerned with algorithms for automated programming with the use of evolutionary technique, called GP, Genetic Programming. The problem is studied within a special case of this technique, namely a language for condition-action rules. The main task is to control the evolutionary process which works with a huge search space by setting so called design. The evolutionary algorithm can search only a subspace, which ts that pre-set design. This approach is used for automated programming of agents who act in a rich, complex, and dynamic environment. The experiments within this project are done in a kind of Virtual Reality. VRis no longer a believable copy of the reality around us, it is straightforward step into a reality which could be evolved via some other trajectory of possible Universe trajectories (in cosmological terms). It is a step into possible trajectories of realities which could happen, realities with di erent physical laws, with di erent geometries, and so on. Following that idea, and by extending it into terms like intelligence for example, we have a domain in which we can ground general theories about life, about intelligence, etc. It is possible to perform experiments within the domain which were not possible before. It is very promising for elds like psychology, philosophy and biology. AI can bene t from it as well. The arti cial evolutionary mechanisms are addressed in ET project, with stress of the underlying mechanisms of the evolution.
2 Agent technology The main requirement of developing an agent is to satisfy dynamic changes of the environment (if we assume that). This goal can be achieved only by autonomous entities, which can adapt to the changing environment, which can learn from experience, and which can create its own goals, make plans, and switch autonomously between di erent motives. Within that area there is a lot of problems, with representation, scheduling, decision making, deliberative mechanism, memory, etc. It is essential for an autonomous agent, to adapt itself to a changing or completely new environment. Although implementations of agent, which act in static environment, are quite successful, dynamic environments are still a big challenge (Brooks 1991), (Maes 1994). Despite technology & implementation details, we can treat an agent as a system with prede ned architecture. The simplest architecture model is based only on reactive layer, which can be seen as a mapping from possible environmental inputs (sensory stimuli from environment) to the proper reaction patterns. This layer is responsible for whole agent behaviour, see gure 1. There has been growing evidence for the role of emotion in human cognition. Some of the relevant work from the research on emotional systems in animals and humans stress the role of emotions in believable and autonomous behaviour. We can expand the architectural model by one more layer which is responsible for prede ned and gained emotions, and motives. According to traditional condition-action rules system, we can de ne also separate layer for controlling and planning sequences of
3
ET-Automated Agent Design thirst hunger pain/pleasure
Reactive layer Efectors
Sensors
Environment
Fig. 1. Architecture based on reactive-layer actions. This can be called deliberative layer. In ET project there is also implemented layer responsible for storing and managing beliefs. It is a kind of long term memory, about environment, but it does not represent a complete model of the environment. The architecture used within ET project is presented on gure 2. The arrows represent possible connections and dependencies between layers. When we take a closer look into
Meta-management
thirst hunger pain/pleasure
Emotions
Beliefs
Deliberative layer
Efectors
Reactive layer
Environment
Fig. 2. ET. Multi-layer agent architecture work done in di erent architectural models (Sloman 1993), especially within emotions and a ect area (Sloman and Logan 1997), we can suspect that all this mechanism, together with agents' beliefs (Kellog and Gmytrasiewicz n.d.) can help very much the designer in implementing complex behavioural rules. Sometimes (usually) the mapping between
4
M.Nowostawski, A.Sloman
environment stimuli and agents actions is so complex, that simple function based only on reactive layer is impossible to implement it, and even after nding the function close to our requirements it is hardly possible to adapt it to changing environment. If one can support the process of modelling that function by mechanisms mentioned before, the task become easier. One can decompose easier this one complex function into submechanisms, which can be develop separately. There is the chance of using some complex dependencies and subfunctions prede ned by designer to help to develop an agent design. As far as an agent is concerned, in both, subsymbolic and symbol processing, it seems that the main research is undertaken in reinforcement learning (Kaelbling and Littman 1996). With autonomous agents it is clear that unsupervised learning is a key issue while an agent should learn from interaction with the environment. The class of agents based on condition-action rules systems represents sequential decision problem. The agent interacts with dynamic environment within discrete-time. At the beginning of each time-slice, the agent observes ongoing information and its current state and selects one of the nite set of actions, based on the agent's decision rules. As a result, the agent and the environment change their states. In reinforcement learning, agent after each time-slice, or after set of performed actions is given a payo . The cycle repeats and the learning task is to nd a set of rules that maximises the expected total payo . Even if one deals with concurrent rule systems, we can easily adapt the sequential decision problem to this case and the only di erence is in introducing additional mechanism for concurrency support. ET project represents classic sequential decision problem with concurrent rule system, but it was impossible to apply a reinforcement learning technique, because of the evaluation function. There is no policy during simulation given by the designer to agents, and the only feedback to an agent is given after the whole cycle. The agent is given a single payo after its whole life, depending on states of other agents. One of the promising methods of learning in such stated problems, currently very popular and widely used is Genetic Programming. ET project is based on special case of Genetic Programming model, namely on condition-action rule based language. There is a research done here about possibilities of multi layer genetic programming model.
3 Multi-layer Genetic Programming Genetic Programming GP is a kind of evolutionary algorithm. It is a special case of genetic algorithms (Goldberg 1989). In GP we are dealing with computer programs as individuals. The problem is not coded directly into a genotype-phenotype structure. The program itself does not represent a solution, but the solution is gained by running those programs and collecting results. The results of those programs are essential for the solution of a given problem (Koza 1992). The tness of any individual is determined by evaluating the individual program (in ET set of condition-action rules) using domain dependent performance/cost function. In some application, in the condition-action rules languages there is alternative approach, in which the single individual represents only one rule, and the whole population represents the "program", i.e. set of rules applied to one particular agent.
ET-Automated Agent Design
5
From its nature, GP works on a very low level of the structure of a language itself. There exists attempts to improve GP, and to make it more ecient. Some of them are based on a hierarchy within the language. By organising functions and code primitives with a kind of hierarchical structure we can make the discovery of subprograms much more ecient and e ective. Those problems are described with details in Rosca and Ballard (1994) and Rosca (1995). Other researches are directed into discovering higher processes, e.g. like communication. The "Blind and Lazy" example described in Poli (1996) and some others can be set as a strong argument that the features which are discovered from the low-level processing, which emerge as a higher level structures from the lower-level processing are very important, or the most important features of evolutionary processing. (See also work of Miglino, Nol and Parisi (1997)). The aim of this research is to test a possibility of pushing GP search into pre-designed higher level structures. The main idea is to combine within single GP low level language and instructions set, together with some higher level mechanisms, to bound the search space. The next section presents and describes so called Multi-layer Genetic Programming approach. There exists a technique which makes GP search more convergent in " at" space of given language. The extension of the GP paradigm to cope with the automatic decomposition of a solution function is known as automatic de nition of function, ADF, introduced and discussed with details in work of Koza (1994). The schema theory on the genotype level (Holland 1975) in cases with complex genotype-phenotype dependency, and complex structures gives little help for understanding the evolutionary mechanism which the designer wants to control. It seems that underlying levels, usually abstract (called sometimes virtual) layers have very strong in uence on the eciency of the evolution (e.g. GP search). The schema theory is usually extended through higher order levels into building blocks paradigm. Building blocks in terms of GP are usually subtrees of programs, with xed, limited, or unlimited depth, which can be evaluated in terms of frequency and tness.
3.1 Search space and abstract layers Following notion introduced by Sloman (1994) we can propose for particular problem a pre-designed mapping between the niche space and architectures, which can be capable to support given needs and conditions. What we are interested in is not based on the low level, physical representation, but in the behavioural layer. In terms of computer programs we can have a large number of word processing packages, and all of them in terms of "behaviour" will be indistinguishable. What make them di erent is a design. We can see the program from many di erent points of view, seeing it on single, selected layer: e.g. as a nite automaton, which states are binary vectors (memory map in discrete timeslices), as a program in assembler mnemonics, as a program in higher order language, as an object-oriented system in a kind of objects-driven speci cation, etc. The most intriguing thing is that all those layers have their own laws, structure and rules, any of them can be seen as the most important one. All of them in uence each other, and
6
M.Nowostawski, A.Sloman
from the behavioural points of view are in 100% identical. But for GP the layer which it explores play the most important role. Nobody is going to write even simplest word processing system using assembler mnemonic. The reason why layers are so important lies in the size of the search space. It looks promising to use higher order layers for GP search, because we can restrain the possible search space by setting on the higher level a kind of design which is the only possible subspace of the whole search space (on the lowest level) to be explored by GP.
3.2 Taxonomy and models This section contains the simple mathematical model of multi-layer GP paradigm, MLGP. It is a preliminary theory developed for use within ET project. The generalisation of this model is expected to be de ned and can probably be applied to number of tasks, as a part of GA or GP algorithms. The natural method of applying abstract layers to an existing language is to extend the language by introducing structures in a kind of hierarchy. The higher order levels are implemented in the lower order levels. It is very similar to the ADF paradigm, but in this case we are not limited only to language structures, and so called abstract layer can include much richer structures than functions de ned by instructions in the lower order level. The essential idea is to implement relations or mappings from one layer into another. The multi-layer model can be seen generally as set of layers represented by a directed graph of dependencies. Lets assume that we have the lowest level of possible implementation called . Without taking implementation into account we can assume, that consists of nite set of states, namely:
l; m; m ; m ; n 2 N 0
00
= f!1 ; !2 ; :::; !ng Let be the higher level layer based on (implemented in) . Like the` previous layer, consists of a nite set of states. Each state is implemented in , it means that each of is a subset of . So is a set of subsets of . = f1 ; 2 ; :::; m g 8i = 1::m i and 2
Up to this point we are compatible with ADF paradigm. We de ned set of functions (mapping of states), which are implemented in the lowest language level. GP can use them together with lower level instructions (states) to de ne expected combination (sequence or set) of states. Let and in analogy to create another two layers implemented in . We have for = f1 ; 2 ; :::; m g 0
00
0
0
0
0
0
0
7
ET-Automated Agent Design
i and 2
The same equation can be de ned for . There are relations between , , and . Lets call them R , R and R respectively. There is no relation between , , and . Lets refer to such layers as layers without dependency. Lets introduce another layer, . Let be implemented in , and , , . As before, consists of nite set of states . Each consists of subset of , , , and . = f 1 ; 2 ; :::; l g
8i = 1::m
0
0
0
00
0
0
0
00
00
00
0
00
0
00
8i = 1::l
and
i ( [ [
2( [[ [ 0
00
0
[ ) 00
)
There exist only three possible and quite natural way of applying genetic operators (in particular Genetic Programming paradigm) to such prede ned model of layers: Type U (up) - this approach is based on the relations R , R , R and R . We let the GP use only these relations and it will look through the search space by exploring only possible subspace de ned (constrained) by R , R , R and R . Other relations, i.e. in the lower layers have to be xed, prede ned by the designer. In this model we are not allowing GP to control the evolution on the level of layers , , , so as said before, relations R , R and R are xed. The evaluation function is based on the overall performance on the highest level. This model represents simplest, and most close to traditional GP approach, in which the only di erence lies in prede ned architecture with xed lower level relations (mechanisms). Genetic operators work in the highest level, namely on the relations R , R , R and R . Those relations are tuned during the evolution to t the expected functional mapping. We should notice that GP search consists of four independent domains, each represented separately by proper relation. Some of genetic operators: mutation, crossover and recombination work separately for each of those relations, and the rest: selection and reproduction work for all of the layers within the whole population of individuals at the same time. Type D (down) - in this case GP search is based on relations R , R and R . Here the relations R , R , R and R are xed, prede ned by designer. The search is done by exploring subspace of the whole search space through relations R , R and R . There exists one evaluation functions which is based, as in the previous type U, on the overall performance on the highest level. Genetic operators work in the s level, namely on the relations R , R and R . Those relations are tuned during the evolution to t the expected functional mapping. Type M (middle) - this is more sophisticated case, in which GP use all possible relations in searching solution space. The designer have to set up evaluation functions for every layer. It can be done by setting a xed evaluation function, or by evaluating lower level layers by higher level evaluation functions. The search is done by exploring subspace of the whole search space through relations R , R and R , and R , R , R and R . Genetic operators work in all the previously mentioned before relations, which 0
00
0
00
0
00
0
00
0
00
0
0
0
00
00
00
0
00
0
00
00
0
8
M.Nowostawski, A.Sloman
means that there exists seven independent domains. From evaluation point of view there exist 4 abstract layers (which ML-GP works on) where three of them are dependency independent. This means that the designer has to set up at least one evaluation function for the layer , which can be used in lower order layers as a evaluation function as well. The designer can specify one, two or three other evaluation functions for layers , and respectively. In that case we can treat the entire system as a co-evolution of "independent" (on GP level) components. All relations are tuned during the evolution to t the expected functional mapping. Types U, D and M represents possible techniques in applying genetic operators (not only GP paradigm) to model built with layers. It can be easily adopted to any kind of arti cial evolutionary systems. The presented mathematical model with simple taxonomy can be extended to any kind of layers dependency. The important properties and possible models of applying genetic operators will stay the same. 0
00
3.3 Automated Agent Design Lets refer to Actions as A, to Reactive layer as R, Deliberative layer as D, Emotions as E and Beliefs as B. Internal variables are represented by V and external data by S (sensory data). This is just the example architecture, and of course there is possible to implement other architectures as well. In ET we use that one, to make experiments with in uence of Emotions and Meta-management layers in terms of eciency of GP search and behavioural performance. There are also possible tests with and without R layer. It is expected with use of R to develop simple states (within R layer) for danger situations (reaction without deliberating on stimuli - alarm system). As seen from dependency tracks, B is used just as long term memory about sensory inputs, and as such play passive role. For simplicity there is no generalisation mechanism implemented. We have fV, Sg as an input in discrete time-slices, fB, E, D, M, Rg as a set of possible states ( nite set of states) with memory ( nite automaton with memory), and fAg as an output. Layers can store particular states, which in uence the next transition, and this property is referred as memory. Implementation domain consists of possible mappings between relevant layers, e.g. within layer B there is mapping fV, Sg into B (fV; S g 7! B ), and following dependency track, there is mapping from B into D (B 7! D). The task in such de ned model is to nd the best mappings from fV, Sg into fAg (fV; S g 7! fAg). It can be done directly, by the simplest method based on only one layer, R which transforms fV, Sg into nite set of states, from which there exists one-toone or one-to-many relation with actions. The problem is that this " nite set of states" can be very big, and GP as relatively blind mechanism can have diculties in nding appropriate one. In perfect case we would have in nite set of states, and behaviour gained within this system would be as it is expected to be. But in limited set of states, it is very dicult to model them. We can assume, that the function which we are trying to model is extremely dicult, non continuous, with big number of local extrema. The uniform GP model has diculties and usually is not capable of solving it. ML-GP introduced in previous section can work in parallel on di erent layers, and thereby decompose that function into several sub-functions. By doing so, from one side we have better (more
ET-Automated Agent Design
9
complex and sophisticated) representation mechanisms, and from the other side we are restricting the search space by setting up only a subspace for exploring. It does not solve the problem, but it helps to include a knowledge about the problem itself into language structures. If there is one-to-many relation between system states and actions, we can face con ict resolution. It means that we can have contradicting actions selected, by some states of abstract layers. It is commonly noticed problem with multi-layer agent architectures, and as such is a subject of separate research. AAD implementation. We can implement ML mechanism using di erent techniques. In ET the layers are implemented in condition-action rule-based system, as a separate set of condition-action rules. Some layers can overlap each other. Within layers there is always only one selected rule. Layers work in parallel. GP operators like mutation crossover and recombination work within single layer. There is implemented evaluation function based on the overall behavioural performance, and some "penalty" functions de ned for each layer separately (usually on very low level). It includes for example that we prefer shorter rule set (a smaller number of rules within a given set) if the performance is comparable, or that within B layer, the number of states stored in database will keep within given limits (to remember enough but not too much). Within ET, the agents' environment consists of a 2D/3D, rectangular, and continuous terrain. The package can work in both 2D online mode and 3D o -line mode. The terrain consists of water, food, and obstacles. The environment is dynamic, that means the water, food, and obstacles can change their location in time autonomously or via user interaction. There are other agents moving around. The whole simulation environment is implemented in pop111 with use of sim agent package. Sim agent (Sloman and Poli 1996) is a programming library written in pop11 for rapidly implementing and testing out di erent agent architectures, it includes agents composed of several di erent sorts of concurrent interacting sub-systems, in an environment where there are other agents and objects.
4 Future work. There are several possible extension to the model discussed in this paper. One of them is based on research on the evolutionary process which itself searches for the best possible architecture, i.e. the best possible decomposition of the representation and design model. The design is not given by the real designer, but is discovered while evolutionary search explores the Design space (in analogy to the design and niche spaces notion (Sloman 1994)). It seems that this kind of embedded evolutionary processes is closely related to real evolution and co-evolution mechanisms - maybe. Mathematical models of such mechanism would be much more complicated, but it could guide the research on arti cial evolutionary mechanisms into areas which can bring better understanding of evolutionary processes and uniform evolutionary theory. 1
The Edinburgh AI language developed in the 60s and 70s and extended at Sussex University in the 80s and 90s, widely used by AI researchers across UK
10
M.Nowostawski, A.Sloman
Extending the multi-layer paradigm presented here with embedded (recurrent) evolutionary mechanisms seems to be one of several possible, and interesting areas of future research in this eld. In general studying arti cial evolutionary mechanism can bring very useful and powerful algorithms for hard problems. It can also bring answers to many questions put forward by biologists, philosophers and arti cial intelligence researchers.
Acknowledgements I would like to thank my supervisor Aaron Sloman for his professional support, help and theoretical and philosophical feedback, and my polish supervisor Halina Kwasnicka for her stimulating in uence on me.
References 0. Brooks, R. A. (1991). Intelligence wihtout reason, Technical report, Massechusetts Institute of Technology Arti cial Intelligence Labolatory. Goldberg, D. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning, AddisonWesley. Holland, J. (1975). Adaption in Natural and Arti cial Systems, MIT Press. Kaelbling, L. P. and Littman, M. L. (1996). Reinforcement learning: A survey, Journal of Arti cial Intelligence Research 4: 237{285. Kellog, T. and Gmytrasiewicz, P. J. (n.d.). Bayesian belif update in multi-agent systems, University of Texas at Arlington. Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of Natural Selection, MIT Press, Cambridge Massachusetts. Koza, J. R. (1994). Genetic Programming II: Automatic Discovery of Reusable Programs, MIT Press, Cambridge Massachusetts. Maes, P. (1994). Modeling adaptive autonomous agents, Arti cial Life Journal 1(1,2): 135{162. edited by C. Langton. Miglino, O., Nol , S. and Parisi, D. (1997). Discontinuity in evolution: how di erent levels of organization imply pre-adaptation, Departament of Psychology, University of Palermo, Italy. Poli, R. (1996). Parallel distributed genetic programming, Technical report, The University of Birmingham, School of Computer Science. Rosca, J. P. (1995). An analysis of hierarchical genetic programming, The University of Rochester, New York. Rosca, J. P. and Ballard, D. H. (1994). Genetic programming with adaptive representations, Technical Report 489, The University of Rochester, Computer Science Dept., New York. Sloman, A. (1993). The mind as a control system, in C. Hookway and D.Peterson (eds), Philosophy and the Cognitive Sciences, Cambridge University Press, pp. 69{110. Sloman, A. (1994). Explorations in design space, Proceedings 11th European Conference on AI, Amsterdam. Sloman, A. and Logan, B. (1997). Cognition and a ect: Architectures and tools, School of Computer Science, The University of Birmingham. Sloman, A. and Poli, R. (1996). Sim agent: A toolkit for exploring agent designs, in M. Wooldridge, J. Mueller and M. Tambe (eds), Intelligent Agents Vol II (ATAL-95), Springer-Verlag, pp. 392{ 407.
