BLOGALPHA: HOME AUTOMATION ROBOT USING ONTOLOGY IN HOME ENVIRONMENT Kenta Cho TOSHIBA Corporation 1 Komukai-Toshiba-cho, Saiwai-ku, Kawasaki-shi, 212-8582, Japan email:
[email protected]
in a home. A home automation robot has sensors such as a camera and a microphone to sense situations and remote controllers that control legacy appliances. A robot moves autonomously to check the state of an environment and users, and to adjust the behavior of appliances so that they are adopted to a user’s needs. A robot can also be controlled via the network, and a user can use a robot as a remote control system of a home environment. Since a home robot works as an intelligent glue that connects and automates legacy appliances, users can introduce an intelligent home environment in their home at much lower cost. In this paper, we propose a home automation robot called BlogAlpha that has a blog interface that receives requests from users as a comment and writes the blog entry showing the results for these requests. Users can control a home environment remotely via BlogAlpha by using a blog interface on their PC or cellular phone. BlogAlpha uses an ontology about a home environment to interpret users’ requests and create concrete commands to act. We describe the benefits of using a home robot, an ontology and a blog interface to realize a home automation service in Section 2. The implementation of a BlogAlpha system is described in Section 3. We discuss advantages of using an ontology and a blog interface with a robot through typical examples in Section 4. The related works are described in Section 5. Section 6 concludes the paper.
ABSTRACT BlogAlpha is a home automation system that uses a home automation robot ApriAlpha to integrate the legacy appliances in a home. It provides a blog interface to receive the user’s request remotely in a natural language and show the state of the home. A robot works as intelligent glue that connects and automates legacy appliances, allowing users to introduce an intelligent home environment in their home at much lower cost. BlogAlpha uses the ontologies about the commodities in a home, the locations where these are placed and the tasks the robot can achieve. By using these ontologies, the robot can achieve the proper action to respond to a wide variety of user requests. KEY WORDS home automation, robot, ontology, blog
1 Introduction As computers become ubiquitous, it will become possible to support a human in a home environment with computers. There are intelligent home environments that provide an autoconfiguration service and a remote control service. Intelligent appliances are configurated automatically according to the user’s context and the user can control these appliances remotely while out of the house with a PC or a cellular phone. An intelligent home environment is filled with many smart devices that sense a user’s context and provide proper information or service to a user. These devices are networked to control and check a home environment remotely, and can provide many services such as a home security service. The problem concerning an intelligent home environment is cost. For example, in order to realize a home security service, it is necessary to introduce many cameras and sensors and these have to be connected by wiring or wirelessly. A home automation service also requires smart white goods such as an intelligent refrigerator and networked audio-video equipment. In many cases, there should be a home server that integrates information from devices and provides the remote control service via the Internet. Since these appliances are expensive, it is costly to build such an environment. Our approach to this problem involves using a home automation robot capable of integrating legacy appliances
549-074
Takahiro KAWAMURA TOSHIBA Corporation
2
Home automation robot
Recently a home automation system has been widely recognized as an application of the pervasive computing [1] and ambient intelligence [2] [3]. Smart devices embedded in a home environment are integrated to realize services in a home environment such as remote control, autoconfiguration and security. There are some requirements for a home automation system. [3] shows requirements that are clarified by a user’s experimental study. The foremost priority is that the user must always remain in control of the system and the system must be secure and safe. The second priority is that the system should provide the appropriate information to the right people. The third priority is that the system should integrate and combine functionality of appliances and should be cost saving. In view of these results, we focus on the two requirements described below.
197
• Providing the appropriate information and service flexibly according to the user’s request The system should provide the appropriate information and service to the user. To be appropriate and flexible, there should be wide varieties of services and the system should select the proper service from among them. A mechanism to avoid an inappropriate service is also important in order to preclude performance of an unsecured service for a user. • Low Cost If cost for the system is kept low, a home automation system can widely applied in homes. To achieve tasks corresponding to the user’s request, a system has to have knowledge about appliances in the home environment. Such knowledge includes the terms of these appliances, the category to which they belong, their functions and their installation locations. Such knowledge can be expressed with by ontologies and ontologies are used to model intelligent home environments [6]. A home automation system mainly consists of information appliances and sensors, and a system using a lot of such equipment is costly. There is another approach in which a home automation robot is used as an interface for users and appliances. [4] uses a robot to control appliances in a home via Bluetooth communication. A home robot can integrate legacy appliances that don’t have a capacity to connect a network and an autoconfiguration function. A robot moves around a home to sense the environment and control the appliances. A robot can receive a request from a user remotely and achieve corresponding tasks. If a robot can provide wide varieties of services that had been realized with many intelligent appliances so far, a user can obtain an intelligent home environment only by acquiring a robot and without replacing many legacy appliances in a home. So we take the approach of using a home automation robot with a home ontology to satisfy the two requirements previously mentioned. A home ontology is knowledge for a robot to derive the appropriate information and service. A home automation robot can also reduce the cost of a home automation system, because a home robot integrates legacy appliances and adds useful functions, such as autoconfiguration and remote control, a moreover, a user doesn’t have to replace many legacy appliances. For users to make requests to a home automation system, a web interface is often used [7]. Using a web interface, user can access a system remotely from any location where a browser is available. Most system of this kind have an exclusive UI [8], but if we want to realize UI for wide varieties of services, a general UI that has a natural language interface is proper for accepting any requests from user. [9] shows a system to communicate with robotic assistants by spoken language and change internal robot states such as attention to the person and the object.
Figure 1. ApriAlpha
Our system uses a blog system as an interface to a robot. A user can make a request to a robot as a comment in the form of a blog entry in a natural language. A robot processes the comment and recognizes what is the request of the user and how the robot can solve that request. The robot creates and processes a list of commands and posts an entry that represents the result of executing these commands. Recently a blog system is widely used, allowing many users to interact with a robot naturally by means of comments and entries in a blog. Since these interactions are performed in natural language, many varieties of requests can be handled in a system. We use an ontology to process input natural language and create commands to solve a user’s request in a comment. We use a robot called ApriAlpha [5] to realize a home automation system. We called our home automation system, realized with a combination of ApriAlpha and a blog system, BlogAlpha. In the next section, we explain the implementation of the BlogAlpha system.
3
Implementation of BlogAlpha system
BlogAlpha system consists of a home automation robot ApriAlpha (Fig.1) and a blog system. ApriAlpha receives a user’s request as a comment from a blog system, processes commands to fulfill that request and posts an entry to report the result. Fig.2 shows the architecture of the BlogAlpha system. The blog system can be included in ApriAlpha, but in this section we explain a system in which these two components are separated and connected via the network. BlogAlpha works as follows. 1. User inputs a request for BlogAlpha as a comment of
198
Commodity Ontology
Result
Request
Location and Components Ontology
Request Parser
Entry
Commands
Comment
Task Ontology Result
Blog System
Command Executor Camera IR Remote Controller
ApriAlpha
Figure 2. BlogAlpha Architecture
the blog system. These commands are ’switch on the TV’, ’go and check on the cat’s food dish’, ’bring me a glass of wine’, etc. 2. The comment is sent to ApriAlpha and ApriAlpha parses the user’s request with ontologies. 3. The request is broken down to a sequence of commands and ApriAlpha executes these commands. These commands consist of primitive actions of ApriAlpha such as moving to a specific position, controlling an appliance and taking a picture. 4. ApriAlpha writes an entry to the blog system that represents the result of the commands ApriAlpha executed. The entry can include a picture taken with ApriAlpha’s camera.
Figure 3. Commodity Ontology
Using BlogAlpha, a user can control home appliances and observe a home environment remotely via a blog interface. 3.1
• Commodity Ontology The commodity ontology represents the terms of commodities and the ’is-a’ relationships between these terms. These terms are classified into a stratified form. Fig.3 shows a part of the commodity ontology tree.
ApriAlpha
ApriAlpha is a robot used in a home and it can move around a home, control appliances and sense an environment. It is equipped with a wireless LAN, and can be connected with other PCs. It has a NLP (Natural Language Processing) module that receives a sentence representing a user’s request.
• Location and Components Ontology The location and components ontology represents where the commodities are placed in a home environment and which commodity includes another commodity such as a beer is put in a refrigerator and a suit is placed in a closet. This ontology includes the terms of locations and the data of physical locations in a home. Fig.4 shows a part of the location and components ontology.
3.2 Ontologies A home robot should know the knowledge about home appliances, commodities and the environment in order to provide the appropriate information and service. We use ontologies to represent this knowledge. The ontologies consist of the three types described below. These ontologies work as common sense that a robot should know when it works in a home environment.
• Task Ontology The task ontology represents a sequence of commands to resolve a user’s request. The task consists of a term index and a sequence of commands. A term index maps a user’s comment to a corresponding task. A
199
list, and if terms match a word or the node in the ontology, the corresponding sequence of commands is selected to resolve the user’s request. When the request parser compares the term with a node in the ontology, all nodes that have an ’is-a’ relationship can be matched with the term. So if there is a node for the task that ’bring me a drink’, this node can be matched with the user’s comment that requires bringing ’liqueur’, ’wine’ or ’Beaujolais’. 3. If the sequence of commands requires the information about the location that ApriAlpha should go to in order to achieve the task, the request parser refers to the location and components ontology, and retrieves the information as to where the commodity mentioned in a comment is located or stored. Figure 4. Location and Components Ontology
4. The selected sequence of commands and the information about the commodity and the location are sent to the command executor. 3.4
Command Executor
The command executor executes the sequence of commands sent from the request parser. Each command consists of the command name and parameters. ApriAlpha can execute the command described below. • MoveTo Using the information about the location, ApriAlpha moves to the location. The command executor has the information about the path to guide ApriAlpha to the target location.
Figure 5. Task Ontology
• TakePicture
sequence of commands is used to control ApriAlpha to achieve the task. With the task ontology, ApriAlpha can understand the user’s request in a comment and perform the task to resolve that request. Fig.5 shows a part of the task ontology. 3.3
ApriAlpha takes a picture with the camera, and sends it as the result of this command. • ControlAppliance By using the information about the commodity, ApriAlpha controls the appliance with the legacy IR remote controller.
Request Parser
By using these ontologies, ApriAlpha parses the user’s comment and performs the corresponding task. ApriAlpha has a request parser that creates a sequence of commands from the user’s request input as a comment. The request parser works as follows.
• Talk Speak the natural language sentence described in a parameter. At present, ApriAlpha can execute only these commands, but in the future, a home robot may have functions such as taking an object and controlling an appliance physically. So we assume the commands described below will become available in the future version of ApriAlpha.
1. A comment is divided into terms with Japanese language morphological analysis. 2. These terms are checked against the task ontology. A term index in the task ontology consists of a list of words, the node in the commodity ontology and the node in the location and components ontology. The divided terms in a comment are compared with that
• Take / Release By using a robot arm, ApriAlpha takes or releases a lightweight object.
200
• ControlAppliancePhysically
5. The BlogAlpha system resolves the location of the TV by the location and components ontology, and ApriAlpha moves to that location and uses an IR remote controller to switch on the TV.
An arm can also be used to control appliances and fittings such as doors and windows. The command executor can output the result of each command in a natural language form, and that result is sent to the blog system. 3.5
By using the blog interface, the user and his family can see the state of the home environment remotely via a web interface. All requests and their results are recorded as comments and entries, which the user can easily have another look at. In this example, the system starts with the request from users, but BlogAlpha can write the entry without an order from the user. For example, if an appliance has a trouble and it can send a fault signal to ApriAlpha, ApriAlpha writes the entry about the problem with the appliance and reports it to the family. In the future, if BlogAlpha can achieve the other commands described in Section 3, the system should understand more terms to solve the user’s request. The example described below is one of the typical scenarios of the future version of the BlogAlpha system. We assume ApriAlpha uses the voice recognition feature that was already implemented in the current system.
Blog System
The blog system is used as the user interface of the BlogAlpha system. We used the existing blog system and add the bridge to ApriAlpha. The bridge detects the input of a user’s comment and sends it to ApriAlpha, and ApriAlpha receives the result of commands and writes the result as the entry. If a user requires the status in a home to be checked, the entry can include a picture taken by the camera.
4
Discussion
In this section, we discuss whether the BlogAlpha system can have the flexibility to provide the appropriate service to solve the user’s request. We show some typical examples in BlogAlpha in order to investigate the flexibility of the system and examine whether the ontologies in the BlogAlpha system play an important role in dealing with the user’s request. 4.1
1. The user says to ApriAlpha, ’bring me a beer’. 2. The request parser uses the task ontology to create the sequence of commands, ’move to the place where beer exists’, ’get the beer’, and ’move to the user’. The task ontology includes the information about what kind of commodities can be carried with the arm. The request such as ’bring me a refrigerator’ isn’t achieved because the refrigerator node in the commodity ontology isn’t included in the corresponding task ontology.
Typical Example
By using the commands currently available, BlogAlpha can monitor and control a home environment remotely via the blog interface. This function is useful for providing a home security service. We describe typical examples of monitoring and controlling a home below.
3. The location where a beer exists is resolved with the location and components ontology. BlogAlpha figures out that the beer is put in the refrigerator and the refrigerator is located in the kitchen. 4. ApriAlpha goes to the kitchen, gets the beer, and brings it back to the user.
1. From an office, a user accesses to the blog of BlogAlpha, and inputs a comment, ’check the front door is locked’.
By using the commodity ontology, the location and components ontology and the task ontology, the BlogAlpha system can understand the many variations of user’s requests, figure out whether the system can solve that request, and create commands to solve the request. In more detail, the command ’get the beer’ should be decomposed into commands ’open the door’, ’get the beer’ and ’close the door’. We should examine introducing a planning feature that has rules for decomposing the command according to the type of command and the target object.
2. The request parser receives that comment and resolves ’the front door’ is a component of the entrance and the physical location of the entrance by using the location and components ontology. Using this information and the task ontology, the request parser creates the sequence of commands, ’move to the entrance’ and ’take a picture’. 3. ApriAlpha executes these commands, and writes an entry that includes a picture taken in response to the user’s comment.
5 4. The user checks the door is locked, but, for the purpose of security, the user wants to switch on the TV to pretend that there is someone at home. So he enters the comment ’switch on the TV’.
Related Works
There was an attempt to define the ontology in a home environment. DomoML [10] is a mark-up language for interoperability of human home interactions. DomoML consists of
201
References
the home environment resources ontology, DomoML-env, the functional composition ontology, DomoML-fun, and the ontology for the interaction of resources, DomoMLcom. DomoML-env [11] is a vocabulary to define physical resources of a domestic environment and it has a structure similar to that of the commodity ontology of BlogAlpha. The task ontology of BlogAlpha is a specified form to control a robot, but we have possibilities that by modeling the task with the combination of the function and the communication of commodities such as DomoML-fun and DomoML-com, we can create a general expression of tasks in a home environment.
[1] E.M. Petriu, N.D. Georganas, D.C. Petriu, D. Makrakis, and V.Z. Groza, Sensor-based information appliances, IEEE Instrum. Meas. Mag., vol. 3, pp. 3135, Dec. 2000. [2] Ceccaroni L., and Verdaguer X., Agent-oriented, multimedia, interactive services in home automation, In proceedings of the Second European Workshop on Multi-Agent Systems (EUMAS 2004), Barcelona, Spain, 2004.
Ontologies have also been used in an ambient intelligence environment to realize a resource awareness and a semantic service discovery [12], and for querying realworld services with a natural language [13]. For NLP services, the ontologies are used for resolving ambiguities in natural language sentences [14] [15]. NLP performed by the request parser in BlogAlpha is very simple for now, but since the commodity ontology and the location ontology works well to resolve ambiguities of user’s requests, the BlogAlpha system can find the proper task corresponding to a request.
6
[3] C. Rocker, M. Janse, N. Portolan, and N. A. Streitz, User Requirements for Intelligent Home Environments: A Scenario-Driven Approach and Empirical Cross-Cultural Study, In proceedings of the Smart Objects Ambient Intelligence Conference, (SOCEUSAI), Grenoble, France, pp. 111-116, October 1214, 2005. [4] Ho Seko Ahn, Jin Young Choi, Home Automation System using Intelligent Mobile Robot in Ubiquitous, URAmI (International Conference on Ubiquitous Robots and Ambient Intelligence), 2005. [5] Yoshimi T., Matsuhira N., Suzuki K., Yamamoto D., Ozaki F., Hirokawa J., Ogawa H., Development of a concept model of a robotic information home appliance, ApriAlpha, In proceedings of Intelligent Robots and Systems, 2004 (IROS 2004), Sept 28-Oct 2, 2004.
Conclusion and Future Work
We propose a home automation system called BlogAlpha that uses a home automation robot ApriAlpha to integrate the legacy appliances in a home and provides the blog interface to receive the user’s request remotely in a natural language and show the state of the home to the family. BlogAlpha uses the ontologies about the commodities in a home, the locations where these are placed and the tasks the robot can achieve. By using these ontologies, BlogAlpha can create a sequence of commands to respond to a wide variety of user’s requests.
[6] Tao Gu, Xiao Hang Wang, Hung Keng Pung, Da Qing Zhang, An Ontology-based Context Model in Intelligent Environments, In Proceedings of Communication Networks and Distributed Systems Modeling and Simulation Conference, San Diego, California, USA, January 2004. [7] F. Mohd-Yasin, S. Noori, R. Roy, C. Wilson, S. Ahmadi, C. E. Korman, M. E. Zaghloul, Distributed Wired and Wireless Sensors for the Home of the 21st Century Project, International Conference on Industrial Electronics, Technology and Automation (IETA) at Cairo, Egypt, December 19-21, 2001.
In the future version of BlogAlpha, to achieve the more complicated tasks, the request parser should have a planning feature to decompose the task according to the task type and the target object. The task ontology should be extended to describe rules for decomposing tasks. The system should also recognize the specific terms for each home such as the pet’s name. When the system finds an unknown term in the interaction between a robot and a user in a blog interface, by posting an entry to ask the user what the meaning of the term is, the system can find the node in the commodity ontology corresponding to the term.
[8] R. Agis, R. Carrillo, V. Moran, A. Gonzalez, C. Morillas, F. Pelayo, J.L. Bernier, Monitoring a mobile robot using a web interface, In Proceedings of the 3rd International Conference on Multimedia and Information and Communication Technologies in Education (m-ICTE2006), November 29-December 2 2006. [9] I. Toptsis, S. Li, B. Wrede, G. A. Fink, A multimodal dialog system for a mobile robot, In Proceedings of Conf. on Spoken Language Processing, vol. 1, Jeju, Korea, pp. 273-276, 2004.
Acknowledgements This work was supported by Ministry of Internal Affairs and Communications as part of the Network Robot project.
[10] Francesco Furfari, Lorenzo Sommaruga, Claudia Soria, Roberto Fresco, DomoML: the definition of a
202
standard markup for interoperability of human home interactions, In Proceedings of the 2nd European Union symposium on Ambient intelligence (EUSAI ’04), 2004. [11] Lorenzo Sommaruga, Antonio Perri, Francesco Furfari, DomoML-env: an ontology for Human Home Interaction, In Proceedings of SWAP 2005, the 2nd Italian Semantic Web Workshop, Trento, Italy, December 14-16, 2005. [12] Davy Preuveneers, Jan Van den Bergh, Dennis Wagelaar, Andy Georges, Peter Rigole, Tim Clerckx, Yolande Berbers, Karin Coninx, Viviane Jonckers, Koenraad De Bosschere, Towards an Extensible Context Ontology for Ambient Intelligence, Ambient Intelligence: Second European Symposium, EUSAI 2004. [13] K. Hiramatsu, J. Akahani, T. Satoh, Querying Real World Services Through the Semantic Web, 3rd International Semantic Web Conference (ISWC2004), LNCS 3298, pp.741-751, 2004. [14] K. Mahesh, S. Nirenburg, A Situated Ontology for Practical NLP. In Workshop on Basic Ontological Issues in Knowledge Sharing, International Joint Conference on Artificial Intelligence (IJCAI-95), Montreal, Canada, 19-20 August, 1995. [15] Dominique Estival, Chris Nowak, Andrew Zschorn, Towards Ontology-based Natural Language Processing, RDF/RDFS and OWL in Language Technology: 4th Workshop on NLP and XML (NLPXML-2004), ACL 2004, Barcelona, Spain, 2004.
203
