AMI: An Adaptive Multi-Agent Framework for Augmented Cognition

AMI: An Adaptive Multi-Agent Framework for Augmented Cognition Matthew Johnson, Shri Kulkarni, Anil Raj, Roger Carff, and Jeffrey M. Bradshaw Institute for Human and Machine Cognition 40 South Alcaniz, Pensacola, FL 32502 {mjohnson, skulkarni, araj, rcarff, jbradshaw} @ihmc.us

Abstract In this paper, we report on our efforts to develop an Adaptive Multi-agent Integration framework, called AMI, and its application in the development of Augmented Cognition (AugCog) systems. Key components of AMI include: 1) Pluggable sensor architecture with dynamically substitutable measures for flexible, optimal exploitation of all available multi-sensory channels; and 2) An industrial-strength integration approach and providing high faulttolerance and dynamic workload distribution across virtually any implementation platform. AMI leverages IHMC’s Knowledgeable Agent-oriented Systems (KAoS) services to provide a flexible and generic approach for linking components. For AugCog systems, these components can be for sensing, detecting cognitive state, and modulating the application itself. AMI has served as the integration platform for a large multi-institutional team led by Honeywell Laboratories under funding from DARPA’s Improving Warfighter Information Intake Under Stress program. Multiple sensors and cognitive state measurement gauges have been independently developed by members of the team and straightforwardly integrated for rapid prototype development and evaluation. The flexibility of the AMI framework has been successfully demonstrated by accommodating the myriad of changes to hardware and algorithms required by the research advances achieved throughout the project.

1

Introduction

Research in the field of Augmented Cognition (AugCog) seeks to develop a closed loop humancomputer interaction system where the state of the human is measured, analyzed and adapted to automatically to improve performance (Schmorrow & Kruse, 2002). There are many cognitive measures under investigation such as memory, attention, stress, arousal, and workload Schmorrow & Kruse, 2002). From our initial research with the Honeywell team, it is clear that no one indicator will provide a universal solution. Each cognitive measure provides insight into a particular aspect of the human- computer interaction and its usefulness and accuracy are both highly dependant on both the environment and the domain of application. There is continuous advancement in sensor technology, cognitive measurement techniques, and adaptation strategies. Any integration framework has to be flexible enough to cope with the research advances inherent in a new and blossoming research field. It is clear from these requirements that system integration will be particularly challenging and critical to the viability of an AugCog system. There are many integration challenges involved in developing an AugCog system. The highly specialized sensors being employed are often limited to a specific platform by hardware restrictions or driver limitations. Advanced algorithms and large data sets can be computationally expensive and demanding on limited computational resources. Both of these considerations indicate the requirement for multiple computational platforms for a complete AugCog system and pose development of such a system as a heterogeneous distributed computing problem. There are many solutions to these types of problems, such as CORBA (Vinoski, S. 1997), Jini (Venners, 1999), Cougaar (Helsinger, A. & Wright, T. 2005) and the CoABS Grid(Kahn, M., & Cicalese, C. 2001). The AMI framework does not try to replace these techniques, but instead leverage their capabilities, as discussed in more detail in section 4. There

are two main tasks in distributed heterogeneous computing. The first is component integration, which involves getting everything to talk the same language. We address this issue in section 3. Once everything can talk the same language, the components must find each other. The discovery process is covered in section 4. After addressing the distributed computing issues, there still remain several functional and practical challenges of how to configure and manage a complete AugCog System in a way that allows flexible testing and evaluation. One of the goals of the AMI framework was not to provide merely an integration framework, but make use of the framework to provide a set of tools to empower researchers. In section 5 we describe some of the extra features of the AMI framework designed to help a researcher run, test and evaluate a system. Section 6 discusses future work that both addresses areas of weakness and looks toward future advances. Before we address the challenges directly, we will first provide an overview of the history and current state of the AugCog system developed under the DAPRA Improving Warfighter Information Intake Under Stress program.

2

AugCog System Overview

We have developed the AMI framework to support the requirements of the DARPA AugCog project as a member of the Honeywell Team. A basic diagram of the overall architecture is shown in Figure 1. The main components are the Cognitive State Assessor (CSA), the Augmentation Manager, and the application with which the user is interacting. The CSA contains all of the sensors and algorithms for capturing the state of the user. The Augmentation Manager determines how to adjust the application to improve performance based on the inputs of the CSA.

Figure 1 AugCog architecture

The AugCog system we developed was a complex integration challenge, since many components were not ready when we began developing the AMI framework. The cognitive state detection techniques were independently researched and developed by various team members and needed to be easily and seamlessly integrated into the system as they became available. Throughout the project we had several hardware and software additions, as well as frequent software updates to various components. Our current CWA configuration is shown in Figure 2. Most of the hardware provided multiple channels of data. Several of the sensors provide similar data but at different levels of quality. As algorithms were added to interpret cognitive state, they

would require certain inputs. The inputs might be from a single device, like Engagement requiring only EEG from Active Two. They may need to be obtained from several devices, like the Gaze algorithm which uses both head and eye movement input. Sometimes different devices provided similar information but had a different quality of data. For example, Arousal needed Inter-beat Interval (IBI) which is available from either Cardiax or Procomp, with Cardiax being the preferred source due to better quality data. AMI also had to handle multiple algorithms requiring the same data. For example, Active Two provided EEG data to Executive Load, Novelty, and Engagement algorithms.

Figure 2 Cognitive Workload Assessor

The AMI framework was not limited to the CWA, but also integrated the Augmentation Manager and the application as well. It has been used to flexibly integrate with both a simulated satellite management application (HCC) provided by NASA (Figure 3A) and a simulated dismounted soldier application developed for the Army Objective Force Warrior (OFW) using the Quake gaming engine (Figure 3B). IHMC extended the normal interfaces of these applications to include haptic devices such as a tongue display shown in Figure 3C.

Figure 3 AugCog Interfaces: A) HCC satellite management B) Quaked based OFW C) haptic tongue display

3

Component Integration Challenge

A core task of the AugCog system is the measurement of physiological data and the processing of that data to detect the cognitive state of the human. The primary goal of AMI is to provide a simple method for data from multiple distributed heterogeneous sensors, algorithms, and interface elements to be collected asynchronously, processed, and used to assist a human. For the Honeywell team, many of the cognitive state detection algorithms were developed independently by researchers whose primary focus is not engineering, but cognitive science. An important design consideration was to minimize the interface burden imposed on those researchers for integrating their component, and free them from the complex issues associated with distributed computing. AMI met this objective by providing a simple and flexible interface that can handle the diverse requirements of different sensors and algorithms. AMI uses agents to handle the complex distributed computing issues, but the framework does not require those wishing to integrate components to have knowledge of agents in order to interface with the framework. Instead developers implement a simple interface, described in section 3.1, which gives the agents the necessary access to the component. The agents themselves can run on any platform that supports Java. Once an interface is created, the component can be plugged into the architecture. Another nice feature of the interface is that it provides a natural method for decoupling the sensor from the algorithm, enabling the reuse of both components separately and minimizing the impact of changes to either component. The AMI framework provides an architecture, shown in Figure 4, for a Basic Agent to handle, and hide, the basic communication and configuration needs of the components being integrated into the AugCog system. The architecture includes four interfaces that allow for separation of different aspects of a component during integration: 1) 2) 3) 4)

Data Provider (required) provides basic data input and output for sensors or algorithms Manipulator (optional) provides data manipulation to incoming or outgoing data Calibrator (optional) provides mechanism to start and stop calibration Graphical User Interface (optional) provides a graphical window for testing, debugging, and other functions

Figure 4 AMI Agent Architecture

3.1

Data provider

Developers are only required to implement a few basic Java methods defined by the AMI Data Provider interface. Since many sensor drivers are written in C/C++, the AMI source code also provides a JNI (Java Native Interface) example that can be used to wrap C/C++ code. Developers can protect their source code by providing only the Java class files or a dynamically linked library for C code. They will also need to provide a description of the input requirements and outputs for their component including: initialization parameters (if required), required input including information about format and frequency (if required), expected output in the correct format and a description of what the data means and how to use it in the context of cognitive state assessment. The data format is based on a simple two dimensional array. Each row represents a data sample. Each column is assigned to a specific data channel. Each data sample is also required to have a timestamp generated by the frameworks time synchronization utility discussed in section 5.2, and an assessment of the quality of data as assessed by the developer of the component based on available information. An example of a set of three samples across three channels is depicted in Figure 5.

Sample 1 Sample 2 Sample 3

Timestamp 12345670.0 12345671.0 12345672.0

Certainty 1.0 1.0 1.0

Channel 1 33443.5 343545.232 352356.3

Channel 2 665.22323 2323.3323 1200.0

Figure 5 IHMC AugCog Standard Data Format

Channel 3 0.3234 -0.2323 0.2334

3.2

Manipulators

To keep the Data Providers simple and easy to build, we created the Manipulator interface. This interface is used to manipulate the data coming into or going out of a given Data Provider. During development we determined several functions that were frequently used to process data either as it was received or before it was sent. For example, we sometimes needed to provide a buffered average of the data to smooth the results. We incorporated many of these generic functions into the AMI framework as Manipulators. A Data Provider could be nested inside multiple Manipulators. For example, data could be scaled and buffered before being sent out. Manipulators can also be applied to single or multiple channels of the data to allow for the flexible application of manipulators to the data. Some Manipulators currently implemented in the AugCog code are: • • • • • • • • 3.3

Simple Calibrator - scales the specified channels by computing the mean and standard deviation for each source channel during the calibration period and then scaling (modified ZScore using standard deviation) the output of those channels. Output Buffer - buffers the output allowing the component providing the data to run faster than the component accessing the data. Averager - outputs the average of the last specified number samples of data. Input Synchronizer - used to over-sample data or synchronize asynchronous data. It outputs the last value at a specified frequency. Input Buffer – buffers incoming data to allow the component sending data to run faster than the component receiving data. Input Channel Averager - modifies the specified channels of the specified agent by replacing the channel data with a rolling average of the last specified number of channel values. Linear Channel Scaler - scales the specified channel between {-1, +1} Median Channel Filter - buffers the specified number of values of the specified channels and outputs median value. Calibration

Many of the physiological measures used for AugCog systems require calibration. The AMI framework provides some simple calibration techniques, such as linear scaling, but also allows the developer to extend or replace this with their own techniques through a simple interface. The interface adds hooks into their calibration process, and these hooks can then be generically handled by the AMI framework to allow calibration consistent with the domain of application. The calibration can be started or stopped by the graphical interface of the agent controlling the component or by a remote call from a separate component. 3.4

GUI

The GUI interface provides a way of separating the visual aspect of the Data Provider from the pure non-graphical data retrieval, data manipulation, and data broadcast. The AMI framework provides a basic GUI that provides some testing and debugging features. The basic GUI displays the input and output connections and provides the ability to view the raw inputs and outputs, record the data, and display it on a simple gauge as shown in Figure 6. This GUI can easily be

replaced by a more sophisticated one as required, or be disabled to reduce the computational load on the system.

Figure 6 Basic Agent GUI

4

Discovery Challenge

AMI leverages IHMC’s Knowledgeable Agent-oriented Systems (KAoS) services (Uszok, Bradshaw, Jeffers, 2004) to provide a flexible and generic approach for linking components. KAoS uses Java Agent Service (Arnold et al., 2002) to abstract the underlying implementation and provides a simplified and consistent interface to several distributed process communication mechanisms including TCP, CORBA (http://www.omg.org), Cougaar (http://www.cougaarsoftware.com), and the CoABS Grid (http://coabs.globalinfotek.com). KAoS also provides an advanced directory service that allows for the publication of not just the agent, but its capabilities as well. This enables component discovery based on capability vice stipulating a specific component name or location. Components with similar capabilities can be seamlessly and dynamically interchanged. The agents that wrap the components hide all the details of registration and querying for needed capabilities. They also handle establishing the required communication. Agents allow data to flow not only between different pieces of code, but also between different platforms. All that is required is a network connection. Agents automate the discovery process required to find necessary data providers. They flexibly handle establishing data exchanges between components and provide robust fault tolerance by handling service interruptions gracefully and attempting to re-establish the connection autonomously. This is critical in AugCog systems where new and untested components are added to the system. These components should not be able to break the entire system, even if an individual component fails completely. Hardware restrictions not withstanding (i.e., a sensor agent is locked to the platform to which the sensor hardware is physically attached), each agent could run on any platform. This allows the agents to harness any available computing power. This flexibility is needed to balance the work load when dealing with algorithms or sensors that were platform restricted or computationally expensive.

5

Tools for running, testing and development

The AMI framework includes a variety of tools for running, testing and evaluating AugCog systems.

5.1

System Management

Although our AugCog system had to handle distributed processes across diverse platforms, the complete system needed to be configured and run by users who may not have an advanced background in distributed computing. It also had to be able to handle changes in overall system configuration, since component availability and requirements frequently changed. Agent Launcher is a utility provided by the AMI framework to allow for easy system start-up. Running Agent Launcher automatically creates the necessary infrastructure for the AMI framework based on the desired configuration provided by a simple text file. It allows for both local and remote launching of agents and displays the system process ID numbers for each agent to help in debugging. Agent Launcher provides a simple push-button approach to running our AugCog System.

Figure 7 Agent Launcher GUI

5.2

Time Synchronization

Time synchronization is a difficult problem in distributed systems. The AMI framework does not claim to be the panacea for the distributed time synchronization issue, but it does provide a solution that met the accuracy requirements for our particular implementation. By incorporating the time stamping service into the framework, AMI frees the developer from having to worry about time synchronization issues. This also allows the system to be updated to an improved time synchronization technique without affecting the existing code. To enable data fusion across platforms each data sample must include an associated timestamp synchronized to a global reference. The AMI framework uses a Time Server to provide this reference for distributed agents. A single Time Client runs on each platform and can

service many agents. The clients periodically correct their platform time to correlate with the server. The corrections also take into account network delay with an assumption of symmetric network latency. Although this assumption may not be very accurate, it is mitigated by frequent corrections. The corrections, as well as the network latency can be analyzed through tools provided by AMI to verify that the system is working within acceptable margins of error. The resolution of the time returned by AMI framework depends upon the underlying operating system. For example, Microsoft Windows operating systems provide time-of-day accuracy of only 10 milliseconds. However, timestamp intervals can have a much greater resolution if the machine supports a high performance counter. The AMI framework provides this functionality for both Java and C/C++ implementations of the Data Provider. The AMI framework provides a simple utility for obtaining a timestamp for a data sample. However it is up to the developer to ensure the timestamp is obtained as close as possible to the actual sample time. 5.3

Data Logging

The AMI framework includes tools to assist in data collection. A critical component of the AugCog system is to be able to collect data for evaluation of a sensor or algorithm. A generic data logging interface is available to save data in a binary or text format, merge the data into one file and also a means of converting the data to a MATLAB .mat format file. The logging can be controlled locally or managed by the AMI frameworks Data Logging Agent. This agent provides an interface with a selectable list of the currently running agents and allows the user to tell all selected agents to start/stop logging. The file name and type can be specified and each session is distinguished by appending a timestamp. Another benefit of allowing the data to be collected by the framework is that it can be used to replace data output of a component when it is not available. This is a very useful testing tool in AugCog systems where specialized sensors are not available, expensive to operate, or require a lot of set up time. We can also use this feature to verify the performance of a component provided by the team members using the sample data they provided. We have also used the component simulation to reconstruct data after an experiment when there had been a failure.

6

Future Work

As we look toward future improvements to the system, we hope to address some of the weaknesses as well as expand the functionality in new ways. Two weaknesses we are addressing are directory service persistence and distribution. The directory service does not currently save it state in a manner that allows for recovery in the event of failure. We are working on a persistence mechanism to allow the directory service to recover its state after failure. Currently components need access to the directory service during initialization. Once connected, the system can run without the directory service. With wireless networks, occasionally agents can be cut off form the directory service. We are currently working on a distributed directory service that will increase the overall robustness of the system, particularly in wireless environments. Some of the new functionality we plan to add is to allow the agents to assist in selecting the preferred data source based on the quality of data provided. Agents could use predefined ontologies to prioritize their interactions in the architecture. We would also like to make the system self configuring by the agents to proactively starting their own data sources. For example, if the stress gauge needs EDR, pupil RMS, and HFQRS, it would take the initiative to

launch the required agents. We would also like agents to relocate autonomously to another platform, hardware restrictions not withstanding, to optimize utilization of available computational resources automatically. Currently we can only manually distribute the agents. Looking further out in the future, if AugCog systems become prevalent, there will be issues of data access privileges. These concerns could be operational driven, for example who needs the information to complete a mission, or they could be privacy driven, for example who is authorized to view the data. By using KAoS as our backbone for AMI, we are afforded access to rich domain and policy services. KAoS Domain Services provide the capability for groups of software components, people, resources, and other entities to be semantically described and structured into organizations of domains and subdomains to facilitate collaboration and external policy administration. KAoS Policy Services allow for the specification, management, conflict resolution, and enforcement of policies within domains. Although not as critical for the current small scale, single-user applications currently being developed, these services could prove to be invaluable in multi-user, cross-organizational applications.

7

Conclusion

AMI has served as the integration platform for a large multi-institutional team led by Honeywell Laboratories under funding from DARPA’s Improving Warfighter Information Intake Under Stress program. Multiple sensors and cognitive state measurement gauges have been independently developed by members of the team and straightforwardly integrated for rapid prototype development and evaluation. The flexibility of the AMI framework has been successfully demonstrated by accommodating the myriad of changes to hardware and algorithm required by the research advances achieved throughout the project. The use of agents as a basis for a framework in developing AugCog systems has proven to be an effective means of providing the dynamically adaptable and fault tolerant environment required by the advancing research field of Augmented Cognition. The AMI framework also provides several tools that have proven to be useful in the management, testing and evaluation of AugCog systems. References Arnold, G., Bradshaw, J, de hOra, B., Greenwood, D., Griss, M., Levine, D., McCabe, F., Spydell, A., Suguri, H., Ushijima, S. (2002) Java Agent Services Specification. http://www.java-agent.org/ Helsinger, A. & Wright, T. (2005) Cougaar: A Robust Configurable Multi Agent Platform, IEEE Aerospace Conference Kahn, M., & Cicalese, C. (2001). CoABS Grid Scalability Experiments. O. F. Rana (Ed.), Second International Workshop on Infrastructure for Scalable Multi-Agent Systems at the Fifth International Conference on Autonomous Agents. ACM Press Schmorrow, D. & Kruse, A. (2002) Tomorrow’s human computer interaction from vision to reality: building cognitively aware computational systems. Proceedings of the 38th International Applied Military Psychology Symposium, Amsterdam, Netherlands, 20-24 May. Uszok, A., Bradshaw, J. M., & Jeffers, R. (2004). KAoS: A policy and domain services framework for grid computing and grid computing and semantic web services. In C. Jensen, S. Poslad, & T. Dimitrakos (Eds.),Trust Management: Second International Conference (iTrust 2004) Proceedings,Oxford, UK, March/April, Lecture Notes in Computer Science 2995, Berlin: Springer, pp. 16-26. Venners, B. (1999) Objects, the Network, and Jini, http://www.artima.com/jini/jiniology/intro2.html, accessed on 15 FEB 2005 Vinoski, S. (1997) CORBA: Integrating Diverse Applications Within Distributed Heterogeneous Environments. IEEE Communications magazine, Vol. 14, No. 2.