IEEE Transactions on Computers C-29(12):1104-1113. 1980. [6] Durfee, Edmund, Lesser, Victor, and Corkill, Daniel, Trends in Cooperative Distributed Problem ...
An Agent-Based Approach to Distributed Data and Information Fusion Gregor Pavlin Marinus Maris Jan Nunnink Informatics Institute, University of Amsterdam E-mail: {gpavlin, nunnink, mmaris}@science.uva.nl Abstract This paper introduces a multi agent-based approach to fusion of heterogeneous data and information, which supports arbitrary distribution of fusion processes throughout a system of networked devices. In particular, the presented approach provides mechanisms for dynamic generation of well-structured models, which can cope with complex fusion problems in dynamic environments.
1. Introduction Intelligent process control and decision making in complex systems require adequate situation assessment which, however, requires processing of large amounts of heterogeneous sensory data and information from spatially dispersed sources. In traditional approaches sensory data and other information are processed centrally. However, central processing is not trivial in modern systems, where we deal with large quantities of information. In general, central approaches suffer from inadequate communication and processing capacity, vulnerability to single-point failures, etc. In addition, central approaches to fusion might not be well suited for dynamic systems (e.g. sensor constellations can change during the operation). Beside the centralized fusion process, this requires centralized reasoning about the states of the fusion system itself, which can result in massive flow of information and additional processing. In order to avoid problems of centralized fusion systems, we introduce Distributed Perception Networks (DPN), a multi agent-based approach to robust and efficient fusion of heterogeneous data and information. Our approach features rather simple building blocks implemented through agents of different types, which can dynamically be organized into fusion systems that can cope with partially unpredictable aspects of the real world. In this paper we focus primarily on the problem of self organization of multi agent fusion systems. We show that decomposition of complex fusion problems often provides
a basis for well structured approaches to task allocation and self organization of multi agent systems, which in turn allows robust and adaptive fusion.
2. Complex Data and Information Fusion A particular DPN provides a specific observation service in a certain spatial entity. Such services are normally based on the fusion of very heterogeneous data and information, which requires complex world models and reasoning mechanisms. However, in this section we show that for a large class of fusion problems, we can find clear problem decompositions, which in turn allow well structured solutions based on task sharing. In other words, in many cases we can accomplish very complex fusion tasks by combining results from simple predefined tasks.
2.1. Problem Decomposition In general, the main purpose of data and information fusion is estimation of events, which cannot be observed directly, but we can observe their symptoms. In other words, fusion is a mapping from different observations to potential causes, i.e. backward reasoning from the symptoms to the causes. Moreover, in a large class of observed systems we can identify chains of hidden events resulting in observable events, which can easily be described through causal models (see Figure 1.). Consequently, it makes sense to approach complex fusion problems with hierarchical fusion systems, which gradually map raw sensory data and simple types of information to higher-level information types used for the decision making. Typically, at the lowest levels we have to interpret the raw sensory data or simple information types. The resulting information is further fused in order to extract higher level information, etc. In other words, causal models facilitate a clear Problem Decomposition. Consequently, complex fusion tasks can be accomplished through hierarchies of simpler tasks, each corresponding to a partial estimation problem focusing on a particular
hidden event from a causal chain; i.e. by executing simple fusion tasks in a certain order, we can extract increasingly abstract information.
• •
Figure 1. Graph of a causal model.
2.2. Distributed Fusion Since a large class of fusion problems can easily be decomposed into well defined sub-problems, we can apply Cooperative Distributed Problem Solving (CDPS) principles [6]. However, the fusion systems supporting CDPS require adequate control at different levels of processing and components have to negotiate about sharing common sensing and computing resources, which requires significant reasoning capabilities. In this context it turns out, that multi agent based solutions [2] are very suitable. Namely, such paradigms support encapsulation of sub-tasks, reflecting the Problem Decomposition and, on the other hand, allow well structured approaches to reasoning about resource allocation and process control. In order to be able to cope with the problem diversity we introduce different types of agents, each specialized for a partial fusion task, a mapping from a certain subset of the available data/information to some higher-level-information type.
3. DPN Framework A Distributed Perception Network (DPN) consists of a hardware layer, a network of sensor suites and computers, which provide the interface to the system’s environment and a software layer, which implements the DPN functionality, such as (i) self configuration of a DPN, (ii) resource allocation and sharing, (iii) runtime control, and (iv) fusion of data and information from a system of interconnected information sources. The software layer consists of a system of communicating agents, which populate the network of computers. Each agent features: • • •
Reasoning engine, which implements the fusion as well as the negotiation services. Communication engine implements mapping between messages and internal states of the agent. Acquaintances model of some agent ai encodes (i) a list of Supplier agents providing ai with the required
information (ii) a list of Consumers, agents using the information supplied by agent ai. World model encodes causal relations between phenomena from the observed environments. Shared Ontologies [2] encode types and formats of relevant queries and replies; i.e. concepts, that a particular agent can understand.
In general, each DPN agent can be viewed as a data/information compression node, where several inputs are reduced to a single piece of information. We classify DPN agents into Sensor Agents and Fusion Agents. Each sensor agent runs on a certain Sensor Suite, has a direct access to the sensor data and implements a particular data interpretation. In addition, often sensor agents support sensor management, such as setting sensing parameters, etc. Fusion agents, on the other hand, support fusion of very heterogeneous information. In addition, the configuration as well as runtime properties of a DPN are based on Fusion Communicative Acts (FCA), Control Communicative Acts (CCA) and Sensor Management Communicative Acts (SMCA). Fusion Communicative Acts (FCA) establish the DPN fusion structure, which supports a particular type of fusion. In addition, FCAs are used for delivery of fused information to higher abstraction levels. In general, each agent of type ai can parse and generate a limited set of queries and replies, whose semantics is grounded in specific ontologies that q p q define the sets of concepts C i and C i . Set C i specifies the concepts that can be used in Configuration Queries (e.g. Calls For Proposals) and correspond to the p information types that agent ai can fuse. C i , on the other hand, encodes the concepts, which correspond to the information types that agent ai can provide to higher fusion levels. In addition, agent ai can provide the information only to the Initiators, whose Configuration p p Queries contain a concept from C i . Note that C i and q C i specify the types of agents that can be registered in the Acquaintances models of the agent ai ..
4. DPN Fusion Process A DPN fusion process is established through a system of cooperating agents, each accomplishing a small task corresponding to a partial fusion problem. Sensor agents interpret raw sensory data and pass the extracted low level information to the Fusion agents, which further extract higher level information. In other words, the fusion process is based on Task and Result Sharing, where each agent type provides very limited and well defined fusion services whose results are integrated by the higher level agents (Solution Synthesis).
4.1. World Models DPNs can fuse very heterogeneous data and information, which requires adequate world models that encode causal relations in observed systems. Moreover, each observation task requires a specific world model, which captures the current constellation of the used informationn sources and maps this constellation to a hypothesis about the state of the (hidden) event of interest. However, we deal with dynamic environments, where the available information sources are not knownprior to the operation. Therefore, a DPN world model must be assembled and updated at runtime. In order to facilitate such adaptive generation of world models, each DPN agent type implements a basic world modeling building block. Such basic units are integrated into complex world models via communication channels between the agents. In other words, the distribution of a DPN world model is defined through simple rules, which are implemented through the communication interface and restrict the possible communication combinations to small subsets of DPN agent types (see next subsection); i.e. through the FCAs partial world models are integrated into a global world model, which encodes a mapping between primitive data/information types and high-level information used for the decision making. While the types of building blocks and possible interactions are limited, the actual combinations of elements depend on the current state of the physical world, requested services and the state of the reasoning system. Consequently, a DPN world model can be highly dynamic. In general, the DPN world models consist of two types of building blocks. At the lowest level, different types of Sensor agents use world models for the sensory data interpretation, which are encoded in different procedural routines. However, Fusion agents support information fusion at higher levels, which is based on the Bayesian Networks [7].
4.2. DPN Fusion Structure Each DPN fusion process depends on the constellation of cooperating agents, which corresponds to a particular Problem/Task decomposition. In order to facilitate further analysis we introduce the concept of a DPN fusion structure, which specifies (i) a set of agents that can participate in a certain observation service and (ii) the communication channels between these agents used for flow of fusion information. In other words, the DPN fusion structure plays a central role in a DPN system, since it influences the properties of the fusion process. For each monitoring service a specific DPN fusion structure determines which types of information can be processed
and in what order this can be done. Each DPN fusion structure can be viewed as an artifact which describes possible topologies of a particular multi agent DPN system. Therefore, a DPN fusion structure is defined as a directed graph G DPN = ( ADPN , L DPN ) , where ADPN = {a1 , , a N } denotes the set of agents that can participate in a certain observation service while LDPN represents a set of communication channels (directed links) between these agents ADPN . The communication channels establish connections which allow information flow between the partial world models belonging to different DPN agents. the set of possible links in a DPN observing a certain place, is defined as:
LDPN = {lij = (ai , a j ) | ai ∈ A ∧ a j ∈ A ∧ (Ciq ∩ C jp ≠ ∅) ∧
(1)
( si (t ) ∩ sDPN ≠ ∅) ∧ ( s j (t ) ∩ sDPN ≠ ∅ )} where ai denotes an agent instance from the set A of all q agents populating the DPN hardware layer, C i denotes the set of concepts that can be used for the formulation of Configuration Query FCAs by ai, and C jp denotes the set of concepts that can be be mapped from FCAs to some actions by agent aj. si(t) represents the spatio-temporal subset of the real world, the so called focal environment, for which the agent ai is responsible, while sDPN denotes the spatial context (e.g. area) of a particular DPN. In addition, the set of agent instances that can participate in a DPN service in a particular spatial context is given by: ADPN = {ai | ai ∈ A ∧ ∃(l ∈ L DPN )[ai ∈ l ]} (2) Moreover, must avoid fusion cycles, which is guaranteed if the following constraint is satisfied for each agent ai:
∀(a j ∈ σ i )[C qj ∩ C ip = ∅] ,
(3)
where σ i denotes the set of all ai’s descendants. From the definition of the DPN fusion structure (see (1), (2), (3)), it is obvious that the general rules for combining different types of agents are encoded in ontologies, which define for each agent type specific parsing and query concepts Ciq and C ip , used for the formulation of FCAs. In other words, through ontologies used for Configuration FCAs, we specify beforehand WHAT types of agents can exchange information. However, we never know in advance WHICH agent instances are actually going to provide the required information. Obviously, a particular DPN fusion structure is a function of the required services, concepts used for the inter agent communication and available agent instances. Thus, a DPN fusion structure is a blueprint of the agent organization corresponding to a particular Problem Decomposition and available resources.
4.3. Self Configuration of a DPN One of the main DPN features is a “Concept driven” self configuration process, which allows a great flexibility through adaptive combination of standard Sensor and Fusion agents into complex fusion systems. In the previous subsection we showed that the fusion structure depends primarily on the ontologies encoding concepts used in FCAs. In this subsection we show how such local ontologies, which are dispersed throughout different agents of different types, define automated generation of meaningful DPNs, corresponding to particular Problem Decompositions. A DPN is initiated by specifying a high-level concept, corresponding to some information type needed for decision making, and the DPN agents gradually organize into a network that maps data and low-level information to the required high-level information. The self configuration process is started by some agent ai, which issues a Configuration Query FCA, specifying some spatiotemporal context sDPN and the type of required q information, a concept x from Ci encoded by ai’s local ontology; i.e. agent ai starts looking for other agents that might join the DPN and supply ai with the required information. If there is any agent aj within the spatial context sDPN that understands the request, then aj can respond with a Bid. Note that aj can understand the query p only if its ontology specifies C j that contains the concept x. If aj is chosen to be ai’s Supplier then it becomes part of the fusion system. In this way the partial world models of ai and aj are linked to a larger Bayesian Network. Note that a DPN configuration process, based on sending Configuration Query FCAs to other agents, can be viewed as a sequential search for eventual q p intersections between C i and C j , the concepts of the Consumer agent ai and potential Supplier agent aj, q respectively. Namely, Ci corresponds to the “needs” of p agent ai, while C j corresponds to the “skills” of agent aj, i.e. abilities to provide information of a type specified by p some concept in C j . In this context, an intersection q p Ci ∩ C j ≠ ∅ means that agents ai and aj can cooperate in solving a bigger fusion problem. In other words, possible intersections between the local ontologies used for Configuration FCAs correspond to meaningful information sharing between agents. In addition, agent ai can send Configuration Query FCAs, only after ai has become a member of a particular DPN. In this way, we achieve that the configuration process proceeds in a top-down manner, where higherlevel fusion agents search for eventual providers of lowerlevel information types. In other words, for a set of agent instances we can identify a partial order of link creations between different types of agents, which is encoded
q
through the local ontologies providing concepts Ci and Cip for each agent type. It should be emphasized that, the presented approach to self configuration does not require any central control or central knowledge of possible combinations of agents. Also note that the presented self configuration process corresponds to the Contract Net Protocol [5]. For each required piece of information the agent acts as an Initiator and sends out a Call For Proposals (CFP). Each potential Participant responds to a CFP with a bid, if it can understand the concept in the CFP and the request can be satisfied. Fusion agents often act both as an Initiator and a Participant.
5. Example We illustrate the configuration and operation of DPNs through a simple example. We assume a DPN for the detection of the burning fluid X in a certain area. The DPN uses different information sources, such as sensors for toxic gas X, smoke and flame detectors, instruments measuring pressure and flow in pipelines transporting fluid X. We further assume that there are several Sensor and Fusion agents populating a hardware infrastructure within the chemical factory. A DPN for detection of a burning fluid X is first initialized by some external agent that sends a Configuration Query to the agent community: •
“Request: fluid X is burning?; Location=area1”
While many different agents will receive that query simultaneously, only the Fusion agents specialized for the detection of the burning fluid X will understand the query. Moreover, only the agents dedicated to the detection of the burning fluid X in area 1 will register the external agent as a Consumer in their acquaintances model and notify that agent about their service. Moreover, these DPN agents can provide the information whether the fluid X is burning only through fusion of heterogeneous information provided by other agents. Therefore, such an agent will issue Configuration Queries, each corresponding to an information type that it can process: • • •
“Request: Is there Fire?; Location=area1” “Request: Is there gas X?; Location=area1”. “Request: Is there a Leak in fluid X pipeline?; Location=area1”.
If there are agents that can provide the information mentioned in the above queries, they will respond with a bid and if they are accepted as Producers they are provided with a contract. In this case direct communication is established between the agents. In addition, the Producers register the agent detecting the burning fluid X as a Consumer in their private Acquaintances models. If these Producers do not have access to the sensory data, they start acting as Consumers and issue further queries. The configuration process stops as soon as either there is no agent left that needs to issue a
Configuration Query (e.g. sensor agents) or no suitable producer can be found within a certain time limit. The communication channels established during the initialization phase are used for the generation of a distributed Bayesian Network (see Figure 1.), which is used for the fusion process, i.e. estimation of the existence of the burning fluid X event. Control Room
Burning Liquid X detection (area 1)
Burning Liquid Y detection (area 1)
Burning Liquid Z detection (area 2)
Burning Liquid X detection (area 2)
Fire detection (area 2)
6.1. Related Work Several authors have recognized the potential of distributed and cooperative paradigms in the fusion context [4, 6, 8]. However, in contrast to our work, they deal with fusion of rather homogeneous information types and spatial distribution of processes. While [3] introduces an elegant approach to agent-based inference with Bayesian Networks it does not address the problem of adaptive generation of fusion structures. In addition, there is work, which is complementary to our research, such as [1], which addresses communication and load optimization in distributed fusion systems.
Gas X detection (area 2) Gas Y detection (area 1)
Fire detection (area 1)
Flame detection (area 2)
Gas X detection (area 1) Flame detection (area 1)
Gas X smell (area 2)
Leak detection in Liquid X pipeline (area 1) Gas Y sesnsor (area 1)
Gas X sensor signal (area 2)
Bad smell (area 1) Gas X smell (area 1) Fume detection (area 1)
Fluid X pipeline pressure (area 1)
Gas X sensor signal (area 1)
6.2. Future Work Gas Y smell (area 1)
Fluid X pipeline flow ( area 1)
Bad smell (area 2)
Fume detection (area 2)
Figure 2. A DPN for specific monitoring is established during an initialization process in which different DPN agents try to find other agents that can provide appropriate information. Arrows indicate the information flow during the fusion process.
6. Discussion In this paper we show that the DPN problem decomposition approach allows self organization of multi agent fusion systems, which are suitable for a large class of complex fusion problems. The presented self configuration mechanism allows DPNs to adapt to different aspects of dynamic environments, which cannot be foreseen prior to the operation. For example, certain sensors might go off-line while new sensors might be introduced to the network. Each new sensor constellation must be captured in a particular world model used for the fusion. The agent based approach to task decomposition as well as data and process encapsulation facilitates arbitrary migration of Fusion Agents within a network of devices while the DPN fusion structure is preserved. This allows optimization of performance and makes the DPNs robust. In addition, the agent-based paradigm facilitates maintenance and modification of DPNs, which can be very complex. Typically, the fusion capabilities can be changed by simply adding a new or replacing an existing agent. This does not require recompilation of the entire system. We merely have to make sure that certain agent types implement intersecting ontologies and the DPN structure does not have any cycles (see (1), (2), (3)).
A recently developed DPN prototype, which features dynamic generation of simple DPN tree structures, will be enhanced through intelligent sensor allocation and fusion control. In addition, the future research will focus on the generation of DPNs with more complex topologies as well as improved fusion accuracy and robustness.
7. Acknowledgements We would like to thank Arnoud Visser from the UvA for invaluable comments and suggestions. This work has been funded by Delft Cooperation on Intelligent Systems.
8. References [1] J. Shen, V. Lesser, N. Carver, Minimizing Communication Cost in a Distributed Bayesian Network Using a Decentralized MDP, Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems, 2003. [2] Michael Wooldridge, Multi Agent Systems, Wiley, 2002. [3] Yang Xiang, Probabilistic Reasoning in Multiagent Systems: A Graphical Models Approach, Cambridge Univ. Press, 2002. [4] B. Horling, R. Mailler, M. Sims and V. Lesser, Using and Maintaining Organization in a Large-Scale Sensor Network, Proc. of Workshop on Autonomy, Delegation, and Control,2003 [5] R. Smith, The Contract Net Protocol: High-level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers C-29(12):1104-1113. 1980. [6] Durfee, Edmund, Lesser, Victor, and Corkill, Daniel, Trends in Cooperative Distributed Problem Solving. In IEEE Transactions on Knowledge and Data Engineering, Volume KDE-1, Number 1, pp. 63-83. March, 1989. [7] J. Pearl, Causality: Models, Reasoning and Inference, Cambridge University Press, 1999. [8] L. K. Soh and C. Tsatsoulis, Reflective Negotiating Agents for Real-Time Multisensor Target Tracking, Int. J. Conf. On Artificial Intelligence (IJCAI-01), Seattle, WA, 2001.
