Mining Data and Providing Explanation to Improve Learning in Geosimulation Eurico Vasconcelos, Vládia Pinheiro, Vasco Furtado University of Fortaleza, Mestrado em Informática Aplicada - MIA Av. Washington Soares 1321, Edson Queiroz, Fortaleza, CE, Brazil
[email protected] [email protected] [email protected]
ABSTRACT This article describes the ExpertCOP tutorial system, a Multi-agent geosimulator of the criminality in a region. In the ExpertCOP, students are police officers who allocate their available police force according to the selected region and then, they can follow the simulation process. The interaction between domain agents representing social entities as criminals and police patrols drive the simulation. The ExpertCOP intends to induce students to reflect about their actions related to resources allocation. To do this, a pedagogical agent aims to define interaction strategies between a student and a geosimulator in order to make simulated phenomena better understood. This agent uses a concept formation algorithm in order to identify patterns on simulation data and to formulate questions to the student about these patterns. Moreover, it explores the reasoning process of domain agents for providing explanations, which help the student to understand simulation events.
1. INTRODUCTION
Simulation aims to represent a phenomenon via another one and it is useful to measure, demonstrate, test, evaluate, foresee, and decrease risks and costs. In educational terms, simulation is important because it allows learning through the possibility of actually doing something (Piaget, 1976). The use of Multi-Agent Systems - MAS to simulate social environments has become broadly used (Khuwaja, 1996) (Gilbert, 1995). Social or urban environments are dynamic, non linear, and made of a great number of variables, characterizing the geosimulation (Benenson and Torrens, 2004). With the computational development of geographical information systems- GIS, multi-agent simulation systems benefited from them in terms of geographical representation of areas to be simulated. Geoprocessing brought precision and realism to the simulation (Wu, 2002). Gibbons (Gibbons 1994, 2001) suggests that there are few adequate tools for developing educational computer systems where intelligent agents support the interaction between the simulation model and the user. Despite recent proposals on new models and implementation of instructional layers in simulators (Mann and Batten, 2002) (Gibbons at al, 2001), few tools have been created specifically for geosimulation. These applications involve particular features as the geographic ones, which must be exploited by intelligent tutorial systems in order to enrich learning. This article describes the educational software ExpertCOP that considers the main characteristics, which we claim to be essential to a general architecture of an educational geosimulation. The ExpertCOP aims to enable police officers to better allocate their ostensive police force in an urban area. This software produces, based on a police resource allocation plan, simulations of how the criminality behaves in a certain period of time based on the defined allocation. The goal is to allow a critical analysis by police officers and system users/students, making them understand the causeand-effect relation of their decisions. Since geossimulation generates a great amount of data, it is necessary to make spacial, geographical, and statistical associations among these data to understand the cause and effect of the simulated events. Thus, we propose the use of an intelligent tutor agent as the data analysis supporting tool, the Pedagogical Agent. This agent uses a machine learning concept formation algorithm to identify patterns on simulation data, to create concepts representing these patterns, and to elaborate questions to the student about the learned concepts. Moreover, it explores the reasoning process of the domain agents for providing explanations, which help the student to understand simulation events.
This article is structured as follows: Initially, we present the state of the art, in which we describe the adopted
technologies and its correlations. In the next topic, we present a generic architecture of the pedagogical social geosimulation in order to describe the ExpertCOP system. Correlated works lead to comparisons and conclusions to finish this article.
2. BACKGROUND KNOWLEDGE
2.1 Simulation in education
Simulation is a scientific attempt that consists of performing an artificial reproduction, called model, of a real phenomenon that we want to study and observe its behavior (Drogoul, 1993).
Simulation has also been proving to be an excellent teaching tool, especially for complex situations, with high cost and risk. Practical applications can be seen in various areas, such as, in the aeronautical industry, nuclear industry, spatial exploration, and petrochemical industry. According to Gibbons (Gibbons, 1997), educational simulations have specific characteristics, like:
Simulations focus on and model real world elements, conceptual systems, and/or on our environment.
Involve users in interactions.
Simulations answer the user, and they are flexible and dynamic.
Simulations are developed to guide the learning of specific subjects.
2.2 Simulation and Multi-Agents Systems Multi-Agents Systems (MAS) study the behavior of an autonomous and organized group of agents aiming to solve problems or situations that are beyond the individual abilities of each agent. These systems are being successfully adopted in the simulation area because of the characteristics inherent to MAS, such as, agent autonomy and modularity that facilitate the representation of models to be simulated. The simulation based on MAS is a live simulation that differs from other types of computational simulations because simulated entities are individually modeled with the use of agents. According to Gilbert (Gilbert, 1995), the multi-agent approach (bottom-up) is appropriate to the study of social and urban systems. Social or urban environments are dynamic, non linear, and made of a great number of variables. In such environments, the individual behavior is relevant in the simulation general context.
A particular kind of simulation, called geosimulation, treats an urban phenomenon simulation model with a multiagent approach to simulate discrete, dynamic, and event-oriented systems (Benenson and Torrens, 2004). In geosimulated models, simulated urban phenomena are considered a result of the collective dynamic interaction among animate and inanimate entities that compose the environment. Geosimulation has been applied in the study of a variety of urban phenomena (Torrens, 2002). The Geographic Information System (GIS) is responsible to provide the “data ware” in geosimulations.
2.3 Knowledge Discovery in Databases - KDD Fayyad (Fayyad, 1996) defines Knowledge Discovery in Databases – KDD as a non-trivial process for identifying valuable and useful patterns on data. The KDD process involves some phases, which are: data selection, preprocessing, mining, and interpretation. These phases are iterative and some of them are often realized with the humaninteraction. Typically, the mining phase is an automatic one, where machine learning algorithms are used to discover clusterings, classifications, and associations on data.
Figure 1: Knowledge Discovery on Database phases.
Concept formation algorithms, as the COBWEB, (Fisher, 1989) generate a probabilistic concept tree, such as the one shown in Figure 2. Probabilistic concepts have attributes and values with an associated conditional probability of an entity having an attribute a with a value v, given the fact that this entity is covered by the concept C, P(a=v|C).
Figure 2: Example of a probabilistic concept tree.
3. A PEDAGOGICAL GEOSIMULATION ARCHITECTURE
Figure 3 depicts the main components of a generic architecture for educational geosimulation systems. On the left side of the figure, we describe the interface, which allows the student to interact with the multi-agent platform and the geoprocessed map. The GIS is responsible for characterizing, generating and managing the map, where the simulation occurs. It offers mechanisms to manipulate the system’s map elements. The user/student model, the parameters, and the log of the simulation are stored in databases. The right side of the figure represents the MAS platform, responsible for containing, controlling, and configuring agents in the system. This platform is composed of four types of agents: constructors, controllers, pedagogical, and domain agents. Here, we concentrate on describing the latter because they are more relevant for tutoring systems. For more details on the whole architecture, see (Furtado and Vasconcelos, 2004).
Figure 3: Pedagogical geosimulation architecture.
3.1 The pedagogical agent
The pedagogical agent (PA) is responsible for helping the user/student to understand the implicit and explicit information generated during the simulation process. It also is a PA’s mission to induce the student to reflect about the causes of simulation events. Therefore, the PA tries to identify patterns on simulation data by using concept formation algorithms in order to interrogate the students about the causes of the identified patterns and how this can help explain the simulation events.
The PA does the KDD process automatically, as it is illustrated in Figure 4. First, it gets data on event simulation and pre-processes them in order to produce new data from the geographic information (distance between relevant points
and the events, for instance). After the pre-processing, the PA submits data to a probabilistic concept formation algorithm. The generated concepts are characterized according to their attribute/value conditional probabilities. That is, a conceptual description is made of attribute/values relations with high probability. Finally, the pedagogical agent, by means of questions, models the characterized concepts.
Figure 4: KDD done by the Pedagogical Agent
3.2 The domain agent
The Domain agents represent entities to be simulated. Each group of agents acts in the system according to its specific behavior. Their internal structure is depicted in Figure 5. In this figure, we describe two cyclical parallel behaviors, an internal state, and a set of possible actions. One behavior is responsible for “listening” and interpreting messages from the environment, and updating the internal state according to the contents in the messages. The second behavior is responsible for the agent reasoning process and, eventually, the decision-making process. In this behavior, beliefs, desires, and intentions are modeled using the Unified Problem-Solving Method Description Language (UPML) (Fensel, 2003). This architecture for developing intelligent behavior provides elements to model the Task, which defines the problem that should be solved, the Problem-Solving Method (PSM) (Fensel, 1998), which defines the reasoning process, the Domain Model, which describes the domain knowledge, and the Ontologies, which provide the terminology used in the other elements. All these elements are described independently in order to enable reusability.
This intelligent behavior leads to the agent’s actions, which are responsible for the interaction with the environment.
The reason to use the UPML is that the development of the agent behavior using this framework makes the reasoning process explicit and part of the application. This fact enhances the quality of the explanation to be provided to the student because such explanations are based on the PSM control structure (Pinheiro, 2003).
Figure 5: ExpertCOP agent’s internal structure.
4. THE EXPERTCOP SYSTEM
4.1 Motivation
The police force allocation with its equipments in an urban area to perform a preventive policing is a tactical management activity that usually is decentralized by sub sectors in police departments spread in this area. What it is intended from those tactical managers is that they analyze the disposition of crime in their region and that they perform the allocation of the police force based on this analysis. We agree on the principle that by knowing where the crime is happening, they can perform an optimized allocation and, consequently, decrease the crime rate.
The volume of information that police departments have to analyze is one of the main factors to provide the society with efficient answers. Tactical managers that perform police allocations, for instance, have a lack of ability related to information analysis and decision making based on this analysis. In reality, understanding criminal mapping activities, even using GIS, is a non-trivial task. In addition to that, experiments in this domain cannot be performed without high risks because they result on loss of human lives. In this context, simulation systems for teaching and decision support are a primordial tool.
The ExpertCOP system aims to support education through the induction of reflection on simulated phenomena of crime rates in an urban area. The system receives as input a police resource allocation plan and it makes simulations of how the crime rate would behave in a certain period of time. The goal is to lead the student to understand the consequences of his/her allocation as well as understanding the cause-and-effect relations.
In the ExpertCOP system, the simulations occur in a learning environment with graphical visualizations that help
the student’s learning process. The system allows the student to manipulate parameters dynamically and analyze the results.
4.2 ExpertCOP architecture
The system architecture follows the same idea of the pedagogical geosimulation architecture, briefly described in this work. It is composed of cognitive and reactive heterogeneous agents, with closed mobility. Figure 6 shows the proposed architecture of the ExpertCOP system.
Figure 6: ExpertCOP System Architecture.
The ExpertCOP system user interface is composed of:
Digital map with axis of the streets of the area to be simulated. This map is generated, manipulated, and updated by the GIS system. The student can follow the dynamics of the domain agents (police patrol, criminals, and so on) acting through the map. The map contains several layers, such as street names, demographic density, traffic signs, crime description, crime type, etc. We use the Intergraph Geomedia® toolkit as the GIS of the system.
Configuration, controlling, and exhibition interface, which interacts with the multi-agent platform and the GIS. The interface is responsible for offering the system configuration, control, and answering tools to the student. Through this interface, the student controls the simulation and observes its results. The user interface was developed based in
the Java Server Pages – JSP technology, using the standard Model View Controler - MVC and the Apache-Tomcat as JSP container.
Graphics that allow the dynamic and static visualization of simulation relevant data. Visualization examples are bar graphics with the quantity of crimes that happened by week, weekday, or hour in a day. We use the JFreeChart library in Java applets to generate the graphics and to include them in the system interface. These applets are updated by the system in accordance with the generation of new events.
We use JADE (Java Agent DEveloper Framework) as the framework for creating the MAS platform. It is composed of control agents and domain agents. The control agents are:
GIS agent that is responsible for answering requests from the graphical interface, contacting the necessary groups of agents. It is responsible for updating the map with the generated simulation data.
Log agent that is responsible for recording all interactions among system agents in order to contain all data about the simulation and the configuration.
Graphical agent that compiles pertinent data to the domain, exchanging information with the log agent, and dynamically generating statistical graphics to the graphical interface.
The domain agents are:
Manager agents, which are responsible for the representation, administration, and interaction with agents of a certain type or class. This kind of agent is a middleware among agents of its own group and the rest of the system. They control pre-programmed activities of its agents, such as activation and deactivation.
Domain simulation agents, which represent in the system the police officers, criminals, citizens, the emergency central (911), and notable places1 in the urban region. Each group of agents possesses its specific modus operandi. From the interaction of these agents, we obtain different crime rates according to a good or bad resource allocation made by the student. It is important to point out that an intelligent behavior of the criminal agent was necessary. It typically realizes an assessment task when it evaluates a certain situation to decide if it commits or does not commit a crime.
1 Notable places are points of a region that are relevant to the objective of our simulation, like shopping centers, banks, parks and drugstores.
Therefore, we have modeled its behavior by means of a JAVA implementation of the abstract-and-match PSM2.
The domain ontology describes the communication structures that are familiar to the domain agents. The communication among system agents uses the Agent Communication Language - ACL protocol, adopted by the JADE multi-agent platform as the agent communication pattern. Since the JADE platform is distributed, groups of agents can still become physically separated in different hosts, making this separation transparent to the user.
4.3 The pedagogical agent in ExpertCOP
The pedagogical Agent in the ExpertCOP has the same mission exposed in the generic architecture mentioned previously. It tries to induce the student to reflect about the causes of simulation events. Some of its activities are:
Evaluation of the graphics on crimes per period of time, trying to identify periods with higher criminal occurrence levels and to relate it with some other simulation event, such as, days with less police service or days of social events, such as holidays.
Evaluation of crimes grouped per area, trying to relate these physical groupings to characteristics of the area, such as, geographic density, commercial activity, existence of slums, lack of police officers, means of dispersion (streets/avenues).
Evaluation of flows of events or agents within areas along the simulation process, trying to identify the reasons for these flows, like the migration of a certain kind of crime among areas in the map.
Besides data evaluation, the PA collects simulation data from the interaction among domain agents in order to apply the FORMVIEW (Reboucas, 2004) concept formation algorithm. Some examples of these data are crimes (date, hour, motive, type), patrols (start time, final time, stretch), escape routes, notable place coordinates, distance between events, agents and notable places, and social and economical data associated to geographic areas.
Moreover, the PA explores the reasoning process of the domain agents for providing explanations. Explanations in the ExpertCOP are adaptive to the student characteristics represented in the user/student model. For limitations of space, we cannot detail our description of how this is done. It suffices here to say that the students are classified according to their level of experience. Specific domain ontologies are constructed for each one of these levels in order to allow explanations, which use adequate concepts and terms to the student. For more details on this, see (Pinheiro,
2 For more details of this modeling see (Pinheiro, 2004).
2004).
4.4 System’s configuration and functioning
To perform the simulation, the user/student must initially configure the simulation parameters. This configuration consists of selecting the geographic area, in which the simulation will occur, and determining, in a date period offered by the system, the number of days for the simulation. Moreover, the user/student should determine the number of police teams that can be allocated. The selection and visualization of the area for the simulation are made through a geoprocessed map contained in the system interface. In Figure 7, we can see the system’s interface, which contains a control menu, dynamic graphics, and the geoprocessed city map. Based on the statistical data about crime rate and on their knowledge on police patrol, the student determines the areas for patrol and allocates the police teams available on the geoprocessed map. He/She should also define the police team characteristics, such as number of vehicles used, service shift, weapons to be used, etc.
Agents representing the police teams monitor an area, by following a program card (a card in which the police officer receives the mission to be performed with information on the areas, type, and patrol route). For instance, if the student defined a program card for the team using motorcycles, this team certainly will have a higher speed and will visit a larger area than a team that works on foot. The patrol function is to inhibit possible crimes that could happen in the neighborhood. We presume that the police presence is able to inhibit crimes in a certain area scope.
Figure 7 Interface of ExpertCOP.
After choosing the simulation period and area, the system acquires, from a database, all crimes that happened in the
selected area for the chosen period. The goal of the student is to provide an optimal allocation that avoids crimes that have happened in the city in the past, making it possible to evaluate the results over real data. In order to perform such crimes, criminal agents start to act. These agents move with intentions based on the abstract-and-match PSM, which implements a decision-making process that evaluates the criteria to perform a crime. These criteria are time, place, the presence or lack of police officers, and environmental factors, such as, demographic density, escape routes, and reaction risks.
The simulation process follow-up can be done by observing the activities on the map and by the statistical graphs generated by the system. At any moment, the simulation can be paused for the analysis of the current situation; in this analysis, the pedagogical agent can help the student. At each new allocation performed, the system will comparatively evaluate the simulated moments, showing to the student whether the modification brought a better effect to the crime rate or not. The PA also makes comparisons among results obtained in each simulation tour for evaluating learning improvements done by the student.
4.4.1 Example of PA formulating questions from data mining
An example of a probabilistic concept tree produced by the PA can be viewed in Figure 7, where nine concepts were formed as a result of the FORMVIEW processing. Police officers who are experts in the safety public domain observed that these concepts reflected interesting, but of dificult perception, facts about the simulation process. The PA evaluates the concepts and selects those attributes that have values with at least 80% of probability. This allows us to conclude, for instance, that the concept C_1.1.3 represents events where there is car theft on saturday nights in residential street corners at the aldeota neighborhood. Thus, this concept is displayed to the user/student as a question. For example, the PA shows the question above produced from the concept C_1.1.3:
“Did you realize that crime: theft, object: vehicle, week day: saturday, period: nigth, local: residential street, neighborhood: aldeota frequently ocurr together ?”.
Having this kind of information, the user/student can make changes in the alocation, aiming to avoid this situation.
Figure 8: Example of ExpertCOP concept tree formed by PA using FORMVIEW.
4.4.2 Explanation Capabilities
As we have already mentioned, the pedagogical agent can explain the task done by domain agents. The interaction example depicted in Figure 6a shows the explanation presented to a colonel (typically an expert on the task of police allocation) when he/she selects, on the map, the places where crimes occurred (points in red) to ask: Why did this crime happen? For a lieutenant (a debutant student), an explanation would be more detailed, as the one shown in Figure 9. After the explanation is provided to the student, he/she can continue to question about related objects in the explanation. For example, after the answer in Figure 9 is shown, the student can ask, “What is close?” Then, the PA explains how the PSM abstraction rule determines the value close to the criterion “notable point distance”. In this example, the notable point is slum and a distance less than two blocks was considered close.
-
(a) Why did this crime happen? The reason was: close to slum.
(b) Why did this crime happen? This crime happened because the place was close to slum, which is an escape route.
-
What is? Distance less than two blocks Figure 9: Explanation in the ExpertCOP Interface.
5. RELATED WORKS
The MAS simulation in education (Khuwaja, 1996) (Gibbons, 1997), the social simulation to support decisionmaking (Gimblett, 2002), and ITS works (Johnson, Rickel and Lester, 2000) (Ryder, Scolaro and Stokes, 2001) strongly influenced this research work. We try to explore the intersection between them. The use of simulation with educational goal was proposed in (Querec, 2003) to train firemen in fire situations in a city. In this work, however, the pedagogical aspect is not enough explored. We saw a lot of works of tutoring systems, virtual environments for training (Munro at al, 1997) (Rickel and Johnson, 1997), and distance education (Pereira 2000). Although these works aggregate pedagogical theories to computer systems based on agents, none of them contemplates Agent-based simulation with GIS and machine learning for a pedagogical purpose (Benenson, 2004) (Wu, 2002). The architecture proposed by Atolagbe (Atolagbe and Hlupic, 1996) and Draman (Draman, 1991) for educational simulation has also similar points with this work, although they do not emphasize on the power of simulation in GIS with the use of KDD to improve the student learning process.
6. CONCLUSION AND FUTURE WORKS
This work described two main components of the architecture for the development of pedagogical geosimulators. This architecture is based on the existence of a MAS to perform geosimulations and of a pedagogical agent that follows the simulation process and can define learning strategies, as well as use a conceptual clustering algorithm to search relations in the facts generated in the simulation. Based on this proposal, we developed the ExpertCOP system, a system focused on police officers education, related to resources allocation. The ExpertCOP was developed using JAVA, on the multi-agent JADE platform.
Initial trainings with users (police officers) interacting with the system were performed aiming to evaluate the learning process by using this tool. As a complement to the use of the system, a crime mapping course is being developed for police officers in the city of Fortaleza, where the ExpertCOP will be used as a tool for analysis and reflection of practical situations. A methodology to evaluate improvements on the student learning process after the ExpertCOP system use is in preparation.
We intend to continue this research on the ExpertCOP system, enhancing its functionalities, and increasing the training support, aiming to make it not only an educational tool, but a decision-making support tool as well. We must verify the applicability of the generic architecture described here in other domains, besides the possibility of developing
a tool for the development of geosimulation systems based on the architecture proposed.
REFERENCES Anderson, J. R., Corbett, A. T., Koedinger, K. R. and Pelletier, R. Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences, 1995. Atolagbe, T., and Hlupic, V. A Generic Architecture for Intelligent Instruction for Simulation Modelling, Proceedings of the 1996 Winter Simulation Conference WSC'96, San Diego, USA, December 1996. Benenson, I. and Torrens, P.M. “Geosimulation: object-based modeling of urban phenomena”. Computers, Environment and Urban Systems. Forthcoming, 2004. Draman, Murat. A generic architecture for intelligent simulation training systems, Proceedings of the 24th annual symposium on Simulation. New Orleans, Loisiana, United States, 1991. Drogoul, A. “De La Simulation Multi-Agents à La Résolution Collective de Problémes”. Paris: Institut Blaise Pascal, Thése de L’Universidté Paris VI, 1993. Fayyad, U. M.; Piatetsky, G.; Smyth, P.; Uthurusamy, R. From Data Mining to Knowledge Discovery: An Overview In: Advances in Knowledge Discovery and Data Mining. 1 ed. California: AAAI Press/The MIT Press, 1996. Fensel,D. and Benjamins,V.R., Key Issues for Automated Problem-Solving Methods Reuse.13th European Conference on Artificial Intelligence, ECAI98, Wiley & Sons Pub, 1998. Fensel,D. et al., The Unified Problem-Solving Method Development Language UPML. Knowledge and Information Systems, An International Journal, 5, 83-127, 2003. Fisher, D. Knowledge Acquisition via Incremental Conceptual Clustering. Machine Learning, v.2,n.2, 1987. Furtado, V. P. and Vasconcelos, J. E. Geosimulation in Education. In Proceedings of Agent-Based Simulation Workshop 2004, Lisbon, PT, May, 2004, (to appear). Gibbons, A. S., Lawless, K. A., Anderson, T. A., & Duffin, J.. The web and model-centered instruction. In B. H. Khan (Ed.), Webbased training (pp. 137-146) Englewood Cliffs, NJ: Educational Technology Publications, 2001. Gibbons, A. and Anderson, T. Model-centered instruction In ID2 Research 6th Annual Summer Institute, Utah: Utah State University, 1994. Gibbons, A. S. and Fairweather, P. G.; Anderson, T. A. “Simulation and Computer-Based Instruction: A Future View”. In, Instructional Development Paradigms. Educational Technology Publications: New Jersey, 1997. Gilbert, N; Conte, R. Artificial Societies: the Computer Simulation of Social Life. UCL Press, London - 1995. Gimblett, H. Randy. Integrating Geographic Information Systems and Agent-Based Modeling Techniques for Simulating Social and Ecological Processes. University of Arizona & Santa Fe Institute: Oxford University Press, 2002. Johnson, W.L., Rickel, J.W. and Lester, J.C. Animated Pedagogical Agents: Face-to-Face Interaction in Interactive learning
Environments, in International Journal of AI in Education - 2000. Khuwaja,R. and Desmarais,M. and Cheng,R. Intelligent Guide: Combining User Knowledge Assessment with pedagogical Guidance. International Conference on Intelligent Tutoring Systems - ITS’96, Proceedings...Berlim: Springer Verlag, 1996. Mann, M.D. and S. Batten. How to accommodate different learningstyles in computer-based instruction. Slice of Life Abstracts. Toronto, 2002. Michalski, R., Carbonnel, J., Mitchell, T. : Machine Learning, An Intelligence Approach. v.I, Tioga Publishing, CA. 1983. Munro, A., Johnson, M. C., Pizzini, Q. A., Surmon, D. S., Towne, D. M. and Wogulis, J. L. (1997). Authoring Simulation-Centred Tutors with RIDES. International Journal of Artificial Intelligence in Education, 1997. O'Shea, T. and Self, J. A. Learning and Teaching with Computers. Brighton, Harvester Press, 1983. Pereira, A. S. and Geyer, C. F. “Pedagogical Agent for Educational Environments in the Internet.” In: Proceedings of the International Conference on Eng. and Computer Education. São Paulo. 2000. Piaget, J. Le comportement, moteur de l’évolution. Gallimard, Paris - 1976. Pinheiro, V. Furtado, V. An Architecture for Interactive Knowledge-Based Systems. ACM International Conference Proceeding Series, Proceedings of the Latin American conference on Human-computer interaction, Rio de Janeiro, Brazil, 2003. Pinheiro, V. Furtado, V. Developing Interaction Capabilities in Knowledge-based Systems via Design Patterns. SBIA´2004 – Brazilian Symposium of Artificial Intelligence 2004, São Luis, Brazil, 2004. (Under review) Querrec R., Buche C. and Maffre E. et Chevaillier P.. "SecuReVi : Virtual environments for fire fighting training" VRIC 2003 (Colloque Laval Virtual 2003 Cinquième Conférence Internationale sur la Réalité Virtuelle), pages 169-175, Laval, France 14 16 Mai 2003. Reboucas R. Furtado, V.: Formation of Concepts using Discrete and continous Intelligence Research Symposium, FLAIRS, Miami, May, 2004, (to appear). Ryder, J. M., Scolaro, J. A., and Stokes, J. M. An instructional agent for UAV controller training. In Proceedings of UAVs Sixteenth International Conference. Bristol, UK: University of Bristol - 2001. Russel, S. Novig P.. Artificial Intelligence: A Modern Approach. 2.ed. New Jersey :Prentice Hall, 1995. Torrens, P.M. Cellular automata and multi-agent systems as planning support tools. In Planning Support Systems in practice, edited by S. Geertman and J. Stillwell. London: Springer-Verlag, 2002. Wu, F. Complexity and urban simulation: towards a computational laboratory. Geography Research Forum Vol. 22, pp. 22-40, 2002.
