Keywords: Multi-Agent System, Ambient Intelligence,. Ubiquitous Computing. Abstract. This paper presents a novel control system architecture which allows fast ...
Multiagent Control System with Mobile Ubiquitous Platform for Ambient Intelligence Kevin I-Kai Wang, Waleed H. Abdulla, Zoran Salcic, Neil DeSouza, Sashank Ramkumar Department of Electrical and Computer Engineering University of Auckland, Private Bag 92019, Auckland, New Zealand {kevin.wang, w.abdulla, z.salcic}@auckland.ac.nz Keywords: Multi-Agent System, Ambient Intelligence, Ubiquitous Computing
Abstract This paper presents a novel control system architecture which allows fast integration of different user interfaces and physical networks. A set of XML messages has been designed as standardized content language between user interfaces and the control system, regardless of the communication protocol used by a particular interface. Universal Plug and Play is integrated in the control system to provide interoperability between hybrid device networks. To evaluate the practicality and interoperability of the control system with different physical networks and user interfaces, a mobile ubiquitous platform has been proposed, which enables quick construction of ubiquitous testbeds in any physical environments. GUI and 3D virtual interfaces have been integrated in the system to control devices on RS-485 network, Bluetooth and IP network.
1 Introduction With advanced embedded technology, computational devices ubiquitously merge with individuals’ daily activities. The idea of Ambient Intelligence (AmI) [1] has become realistic. Demands for devices, interfaces, and processes to become more intelligent have increased dramatically due to insufficient availability of human supervisions and abundance of computational resources. Researchers from different disciplines have put on enormous efforts to advance different aspects of AmI applications. The MIT AI Lab started Intelligent Environment (IE) researches around mid 90s [2]. At that time, their research was focused on introducing intelligence via smart sensors and camera networks and can be considered as Human-Computer Interaction (HCI) and sensor network researches. In 1999, a Multi-Agent System (MAS) called Metaglue was developed to control the IE [3]. Over the past few years, intelligent knowledge-based resource management [4] and reactive behavior systems [5] had been integrated into the MAS to more intelligent control. The Adaptive Building Intelligence (ABI) project collaborated by several Swiss universities uses MAS
approach to provide general building services rather than personal space services [6]. The University of Essex focuses on online learning of personalized behavior using Fuzzy Inference System (FIS) [7, 8]. In University of Colorado, Artificial Neural Networks (ANNs) are used to control the lighting and heating services of a normal residential environment involving different types of living spaces such dining room, living room and bathroom [9]. It is obvious to see that most contemporary researches aim to develop more advanced algorithms, protocols or interfaces, which are most likely to be incompatible with each other. The incompatibility between individual advancements has made the research outcomes less practical in many real-life cases. In this project, a multiagent based control system architecture is proposed to provide: 1) interoperation between various physical device networks and 2) fast integration of different user interfaces. In order to achieve interoperation between physical devices, a well known device discovery protocol, Universal Plug and Play (UPnP), has been used as a middleware between the control system and various low level physical networks. A set of XML messages has been designed as high level content language between the communication of control system and user interfaces. The standardized XML messages allow developers to design user interfaces independently, regardless of any changes in the control system or physical networks Further, to speed up the construction of a ubiquitous platform for AmI researches, a mobile ubiquitous setup box has been proposed and implemented. The portable setup box can be connected with up to four different sensors and can control up to 3 different devices. The mobility of such setup boxes enables researchers to quickly set up an ubiquitous platform and perform on-site experiments in any physical locations other than their laboratories. The rest of the paper is presented as follows. Section 2 outlines the control system architecture and elaborates on the implementation of the control system. Section 3 demonstrates the construction and usage of the mobile ubiquitous platform. Section 4 evaluates the practicality of the system and section 5 concludes the paper.
Fig. 1 Multiagent control system overview.
2 Control System Architecture
2.1 Multiagent infrastructure
The goal of the proposed control system architecture is to provide flexibility and interoperability among different physical networks and high level user interfaces. An overview of the control system architecture is provided in Fig. 1. The control system architecture can be discussed in three components, namely the multiagent infrastructure, the XML communication API and UPnP middleware layer, in the following sections.
In this project, a well known agent platform, JADE (Java Agent DEvelopment Framework), is used to construct the underlying multiagent infrastructure [10]. Refer to Fig. 1, agents in the control system can be split into three groups: interface agents, learning agents and control/utility agents. Each user interface has a corresponding interface agent registered in the multiagent control system. The interface agents are responsible for transmitting system update events and receiving user control actions. The learning agents involve different types of adaptive learning algorithms such as fuzzy inference, neural network and decision tree, encapsulated as agent entities. Learning can
be performed based on the personalized user behavior data stored in the system database and hence generating automatic control actions. More detailed implementations and experiments on learning agents can be found in [1113]. The control point agents are essentially control points in the UPnP protocol, encapsulated as an agent entity. The control point agents are responsible for passing control messages to the corresponding UPnP software devices and listening for events from each registered software devices. As UPnP control points are implemented as agent entities, they can act as a middleware, providing unique communication channel over IP network to the high level multiagent control system. The benefits of using a multiagent infrastructure and employing an existing agent platform are manifolds. For all the control and interface modules encapsulated as agents, standardized communication and service discovery routines are provided, which speed up the development process of the control system. More importantly, through the support of the agent platform, agent entities can be dynamically started and terminated, which is a crucial feature for system maintenance and long period experiments.
required to communicate with the physical device using a specific communication protocol. 2.2 XML based control message As mentioned in previous sections, different user interfaces could be developed using different programming languages and use different communication protocols. In order to achieve fast integration between the control system and various user interfaces, a set of XML control messages are designed for the high level communication. In this way, the content of the communication are standardized regardless of different programming languages or communication protocols. All the XML messages are designed according to a common XML schema, which provides an easy way to design new and compatible XML messages in future interfaces. At this stage, a set of seven XML messages have been designed and incorporated into the control system. The first pair of XML messages, shown in Fig. 3 and 4, is used by the interface to register and deregister itself with the corresponding interface agent. The registration message provides essential information for the interface agent to establish communication with the interface. The pair of XML messages shown is used in one of our user interface case study for remote GUI interface over socket connection. Different protocols such as Bluetooth can be easily handled with a different pair of registration and deregistration messages. Interface name IP address 9000
Fig.3 Application registration message. Interface name
Fig.4 Application deregistration message.
Fig. 2 System communication stacks. The communication stacks within the multiagent control system is shown in Fig. 2. XML messages are used in communications between interface agents and user interfaces, regardless of different types of protocols and networks. Within the agent platform, each agent should register with the Directory Facilitator (DF) agent to publish services provided by itself. Agents can search for other agents based on the required services through DF agent and communicate with each other using the default Agent Communication Language (ACL). The searching and communication routines are provided by the JADE agent platform. The UPnP software devices are not part of the agent system and the communications between software devices and control point agents are done through UPnP protocol. Each UPnP software device is a software replica of an actual physical device and is
Once the communication link is established successfully, the user interface will perform a “query” action to retrieve a set of active devices that can currently be controlled. The query message is shown in Fig. 5.
query query
Fig.5 Device query message.
Respond to the query message, interface agent will send two types of XML messages according to the information provided by the underneath control agents. The first XML message shown in Fig. 6 contains a list of active devices that is currently available in the system. The second XML message shown in Fig. 7 contains status information for a particular device. It should be noticed that device status acknowledgement message is also used as an event updating message, to update different user interfaces when device status gets modified by other control routines such as automatic control routine or manual switch control.
Device 1 name Device 2 name
Fig.6 Device list acknowledgement message.
2.3 UPnP middleware To integrate hybrid device networks, Universal Plug and Play (UPnP) is used as a middleware [14]. In general, UPnP is an IP-based device discovery protocol which enables automatic device discovery, configuration and management [11, 15]. Hence, each individual device network is mapped onto the IP network and controlled as one integrated network. To enable UPnP communication, two UPnP components, UPnP software devices and UPnP control points, need to be implemented. The UPnP software device is responsible for communicating with the physical devices. It is the software replica of the underlying hardware. UPnP control point communicates with the software devices and passes necessary information to the high level learning and interface agents. Both UPnP control points and software devices are implemented in Java. The UPnP control points are also encapsulated as an agent entities, served as control agents within the multiagent control system. The UPnP components and protocols are implemented using CyberGarage UPnP Java API.
Device name Device status
Fig. 7 Device status acknowledgement message. After the user interface completes the initializing query process, users can perform control actions based on the list of active devices through the user interface. Fig. 8 and 9 shows the control message and the corresponding acknowledgement message.
3 Mobile Ubiquitous Platform Physical ubiquitous environment is essential for realistic and accurate experiments. However, construction of a ubiquitous platform which fits the corresponding control system is usually very time consuming. The fixed location of a ubiquitous platform also bears many constraints towards carrying out experiments in different physical locations or different environmental settings. In this project, we proposed a mobile ubiquitous setup box which allows fast construction of a ubiquitous environment in any physical locations and environments.
Device name perform value
Fig. 8 Control action message.
Error Value
Fig. 9 Control action acknowledgement message.
Fig. 10 Conceptual diagram of mobile ubiquitous setup box.
3.1 Mobile ubiquitous platform Generally speaking, a ubiquitous computing paradigm can be realized through a group of devices loosely connected over one or more networks, which are able to communicate with one another. Context-awareness, cooperative learning, and hence ambient intelligence can be achieved through the information and intelligence hidden in the communication. Due to the complexity of network connectivity and wiring of physical devices, ubiquitous computing testbeds are
Fig.11 Mobile physical testbed. commonly developed in a fixed location, such as a specialized laboratory. The fixed location ubiquitous testbed can only examine one type of physical environment settings, for example, office, studio apartment or normal household environment. The fixed location testbed also limits the number of experiments that can be carried out at the same time. In order to overcome these constraints, a mobile ubiquitous setup box is proposed. The conceptual diagram of the mobile ubiquitous setup box is shown in Fig. 10. Each setup box is consisted of two components, a digital switch and a 3-to-3 triac module. Each digital switch contains an internal light and temperature sensor and allows two additional sensors to be wired as inputs of the switch. A digital switch has three push buttons which allow manual controls for up to three physical devices. The outputs of push buttons are connected to a 3-to-3 triac module to provide three power plug interfaces for any arbitrary appliances. A setup box can connect to a host computer via a RS-232/USB interface module. Multiple setup boxes can be connected in a daisy chain connection to handle different groups and different amount of physical devices. 3.2 Physical Infrastructure Based on the mobile setup box, a mobile ubiquitous testbed can be easily built by connecting devices and sensors to a particular setup box. Fig.11 shows the physical infrastructure used in this project, constructed using two setup boxes.
Refer to Fig. 11, two setup boxes numbered 2 and 3 are used to control four lighting devices, one fan and one heater. Each setup box has light and temperature sensors to detect the local light intensity and temperature. Two additional pressure sensors have been connected to setup box 2 to detect the current location of the user. Both setup boxes are connected by one bus line (i.e. the daisy chain connection) and linked with host computer number 1. In order to link the ubiquitous setup box with proposed control system architecture, RS-485 communication API module and corresponding UPnP software devices are implemented on host computer 1, as depicted in Fig. 1. With different types of environments, different amount of setup boxes can be used to control a different set of physical appliances to quickly form a ubiquitous testbed. In this way, experiments can be performed concurrently by different group of people and in any physical locations and environments. Fig. 11 also demonstrates possible scenarios to extend the physical infrastructure to include hybrid physical networks and multiple user interfaces over Bluetooth, WIFI/Ethernet and Internet connections. Physical devices on other physical networks, such as Bluetooth, Zigbee or LonWorks, can be seamlessly integrated as part of the physical infrastructure as long as the physical communication medium exists. To incorporate different physical devices into the control system, corresponding communication API module and UPnP software devices need to be implemented. In case the physical devices communicate over IP network as UPnP protocol, the communication API module is not required. Due to the distributive nature of multiagent architecture and UPnP protocol, UPnP software devices
can be operated on a different host computer (e.g. computer 6) without modifying any code. User interfaces designed over different communication protocols such as Ethernet/WIFI, Bluetooth, or internet, can be integrated into the system by implementing the corresponding interface agents. Extra XML messages that maybe needed by different protocols can also be designed according to the XML schema provided by our research group.
15 shows the interface delay on different platforms and different communication protocols. The last two entries in Fig. 15 indicate there is an obvious delay over WIFI connection compared to wired Ethernet connection. It was observed during the experiments that the time delay is mainly dependent on the distance and obstacles such as concrete walls between the device and access point (AP). By comparing the first two entries, an extra delay can be found in PocketPC platform over the same communication link. Possible causes could be that J2SE is too heavy for such a mobile platform or the XML parsing process takes too long. Nevertheless, taking into consideration the advancement in mobile processing power, the proposed XML content language are expected to be more practical. Platform PocketPC Laptop Laptop
Communication protocol Delay (sec.) WIFI (802.11g) ~8 WIFI (802.11g) ~3 Ethernet N/A
Fig. 15 Interface delay comparison.
Fig. 12 GUI control interface.
Fig. 13 3D virtual interface [16].
4 System evaluation and future works In order to examine the practicality of using XML as a content language between high level user interfaces and the control system, two different user interfaces, the conventional GUI interface and 3D virtual interface shown in Fig. 12 and 13, have been implemented and integrated with the control system.
The 3D virtual interface was implemented using game technologies in C++, detailed implementation can be found in [16]. This particular interface was tested using the laptop platform over WIFI and Ethernet links. There is still a noticeable delay when communicating over WIFI link. However, the observed delay is slightly shorter compared with the corresponding delay in GUI interface (i.e. the second entry in Fig. 15). The slight difference in delay may be caused by the different efficiency of programming languages in parsing, transmitting and receiving XML messages using socket connection. The time required to integrate these two user interfaces into the control system by implementing corresponding interface agents is typically 3 to 5 days, for a properly trained engineering student. Different programming languages and platforms come with different resources and tools which is the main reason for the difference in implementation time. Nevertheless, using XML messages as content language makes the design and implementation of user interfaces independent from the control system and hence greatly reduces the time consumption to integrate different interfaces with the proposed control system.
Fig. 14 Evaluation interface platform.
Unlike integrating an interface, the time required to implement the communication API module for various physical device networks can vary in a wider time period, depends on the complexity, flexibility and programming resources of a particular physical network. In this project, the proprietary RS-485 API module takes more than 10 days to develop. In comparison, Bluetooth API module was finished in two days due to more programming resources are available for Bluetooth protocol.
The GUI interface was implemented in Java and tested on both platforms in Fig. 14, using socket connection. The PocketPC uses Mysaifu Java Virtual Machine (JVM), which is a free JVM supporting J2SE on Windows Mobile series operating systems. Therefore, the GUI interface can operate on both platforms without modifying the code. Fig.
Based on the current research progress, more user interfaces such as voice interface and mobile phone interface should be integrated with the proposed control system to further examine the practicality of XML content language. Interface agents for different communication protocols such as HTTP and Bluetooth should also be
Platform Processing Power Operating System PocketPC Intel PXA270 62MHz, 64MB Ram Windows Mobile 5.0 Laptop Intel Pentium M 750 (Dothan) 1.86GHz, 1GB Ram Windows XP
implemented to achieve a more comprehensive set of XML messages. Furthermore, communication API modules for common physical device networks such as LonWorks and Zigbee should be implemented to provide a better support for the control system to integrate different types of physical devices.
[6]
5 Conclusions In this paper, a multiagent control system architecture has been proposed to provide fast integration of different user interfaces and physical device networks. A set of XML messages has been designed and implemented as content language for communications between high level user interfaces and control system. The well known device discovery protocol, UPnP, has been incorporated into the control system architecture as a middleware to provide interoperability between devices over hybrid physical networks. To speed up the implementation of ubiquitous platforms, a mobile ubiquitous setup box has been proposed and constructed. The mobile setup box has demonstrated enormous time saving on constructing ubiquitous platform in any physical locations and environments. The control system with mobile ubiquitous testbed constructed using mobile setup box was evaluated in terms of its practicality on fast integration of different user interfaces and devices interoperation between different physical networks. GUI interface and 3D virtual interface have been successfully integrated in the system to control devices on RS-485 network, Bluetooth and IP network.
[7]
[8]
[9]
[10]
[11]
[12] Acknowledgement The author would like to thank Mr. Kevin Quin from Kristil Technology, New Zealand for his kind help on constructing the mobile ubiquitous setup box.
[13]
References [1]
[2]
[3]
[4]
[5]
Riva G., Loreti, P., Lunghi, M., and Davide, F., 2003, [14] “Presence 2010: The Emergence of Ambient Intelligence”, Ios Press, Amsterdam, The Netherlands. Brooks, R. A.: The Intelligent Room Project. In: Second International Conference on Cognitive Technology, pp. 271-278, 1997 Philips, B. A., 1999, “Metaglue: A Programming [15] Language for Multi-Agent Systems”, M.Eng. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA. Gajos, K., 2000, “A Knowledge-Based Resource Management System For The Intelligent Room”, [16] M.Eng. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA. Kulkarni, A. A., 2002, “A Reactive Behavioral System for the Intelligent Room”, M.Eng. thesis,
Massachusetts Institute of Technology, Cambridge, MA, USA. Rutishauser, U., and Schaefer, A., 2002, “Adaptive Building Intelligence: A multi-Agent approach”, diploma thesis, University of Applied Science Rapperswil, Switzerland and Institute of Neuroinformatics, Swiss Federal Institute of Technology and University of Zurich , Switzerland. Hagras, H., Colley, M., Callaghan, V., Clarke, G., Duman, H., and Holmes, A., “A Fuzzy Incremental Synchronous Learning Technique for EmbeddedAgents Learning and Control in Intelligent Inhabited Environments,” Proceedings of the 2002 IEEE International Conference on Fuzzy Systems, pp.139144, 2002. Doctor, F., Hagras, H., and Callaghan V., “Fuzzy Embedded Agent-Based Approach for Realizing Ambient Intelligence in Intelligent Inhabited Environment,” IEEE Transaction on. System, Man and Cybernetics, 35, no. 1, pp. 55 – 65, 2005. Mozer, M., Proceedings of American Association of Artificial Intelligence Spring Symposium on Intelligent Environment, pp. 110-114, 1998. K. I., Wang, W. H., Abdulla, Z., Salcic, A MultiAgent System for Intelligent Environments using JADE. In Proceedings of the 1st IEE International Workshop on Intelligent Environment, pp86-91, 2005. K. I., Wang, W. H., Abdulla, Z., Salcic, Distributed Embedded Intelligence Room with Multi-agent Cooperative Learning. Ubiquitous Intelligence and Computing. Lecture Notes in Computer Science, Vol. 4159, pp. 147-156, Springer-Verlag, Berlin Heidelberg, 2006. K. I-K. Wang, W. H. Abdulla, and Z. Salcic, “Multiagent Software Control System with Hybrid Intelligence”, Indulska et al. (Eds.): UIC 2007, LNCS 4611, Springer, pp. 1046-1055, 2007. K. I-K. Wang, W. H. Abdulla, and Z. Salcic, “Multiagent System with Hybrid Intelligence Using Neural Network and Fuzzy Inference Techniques”, H. G. Okuno and M. Ali et al. (Eds.): IEA/AIE 2007, LNAI 4570, Springer, pp. 0473-0482., 2007 A. A. N., Shirehjini, A Generic UPnP Architecture for Ambient Intelligence Meeting Rooms and a Control Point allowing for integrated 2D and 3D Interaction, in Proceedings of the 2005 joint conference on Smart objects and ambient intelligence: innovative context-aware services: usages and technologies, pp. 207-212, Grenoble, France, 2005. K. I., Wang, W. H., Abdulla, Z., Salcic, Multi-agent fuzzy inference control system for intelligent environments using JADE. In Proceedings of 2nd IET International Conference on Intelligent Environments, pp.285-294, 2006. K. I.-K. Wang, I. Y.-H. Chen, W. H. Abdulla, Z. Salcic, and B. Wuensche, “3D Virtual Interface for Ubiquitous Intelligent Environments”, In Proceedings of 3rd IET International Conference on Intelligent Environments, Ulm, Germany, pp. 268275, 2007.
