Semantic Robot Memory Store using 5W1H for ...

The 5th International Conference on the Advanced Mechatronics(ICAM2010)

Semantic Robot Memory Store using 5W1H for Service Tasks1 Hak Soo Kim*1, Jin Hyun Son*1, Gi Hyun Lim*2, Il Hong Suh*2 *1 Department of Computer Science and Engineering, Hanyang University Sa-3 ong, Sangrok-gu, Ansan, 426-791, Korea *2 College of Information and Communications, Hanyang University 17 Haengdang-dong, Seongdong-gu, Seoul, 133-791, Korea the context inferencing for service tasks. That is, an autonomous symbiotic human-robot having to provide the intelligent services should recognize a current context or situation like that there is “having a dinner” or “taking a lecture”. If a robot can more correctly recognize these situations, he or she will provide a predictable or intelligent service. In this regard, using the ontology concept is one of the techniques related with the context inferencing. Ontologies have been developed to provide a machine-processable semantics of information sources that can be communicated between different agents (software and humans) [3]. Many definitions of ontologies have been given in the last decade. Gruber [10] defined it as “An ontology is a formal, explicit specification of a shared conceptualization.” This concept facilitates the categorization and definition of things existing in the human world. Also, we can easily build the knowledge system to build an autonomous symbiotic human-robot. On the other hand, providing a machine-processable semantics of information means that a robot can understand the knowledge of humans and provide an intelligent service through inferring a current situation in the level of describing ontologies. As described above, we propose a semantic robot memory store, called Robot Brain Store (RBS), which supports that a context inferencing engine infers a context in a current situation. The basic idea of our proposed store is based on the human memory system in the area of mental mechanisms of Human [4]. The human memory can be categorized into Sensory Memory, Working Memory and Long-term Memory. By analyzing these memory systems, we designed Robot Brain Store that efficiently store and query ontologies in an autonomous symbiotic human-robot. In addition, we propose ontology-based 5W1H to support the context inferencing for service tasks. 5W1H that is known to consist of WHEN, WHERE, WHO, WHY, WHAT and HOW manages things that a robot performed. This episodic memory stores individual events and personal experience information at a specific time and place. The advantage of using 5W1H is able to more correctly predict a current situation from past facts. The rest of this paper is organized as follows: In Section 2, we first present the basis for classifying the robot knowledge into several robot brain stores. In Section 3, we propose the robot brain store for storing the ontology-based robot knowledge. Section 4 describes Temporal Episodic Structure Model managing

Abstract: Nowadays, many researches use the ontology concept to build the knowledge system supporting an autonomous symbiotic human-robot. These researches focus on the design of a database schema of storing ontologies, developing a reasoning system based on the rule. The most important goal of using the ontology concept is known to be the context inferencing. In this regard, we focus on designing the robot memory store, called Robot Brain Store with 5W1H, to support the context inferencing for service tasks. The lack of previous researches is as followings: i) not consider the methodical and semantic 5W1H, ii) poor connections between 5W1H and other concepts. To solve these problems, we designed the robot memory model that is the conceptual model for storing the ontology knowledge such as context, object, space, feature, human and 5W1H ontologies. Also, we designed EventsEpisodic RBS which supports the context inferencing to efficiently store and query 5W1H.

1. INTRODUCTION In the past decade, the robot technology has been developed to provide a better service for Human. The autonomous robot providing intelligent services means that a robot achieves a higher level request by itself [1]. More recently, its concept extends to a symbiotic human-robot system. A symbiotic relationship between Human and Robot is the interest concept in terms of being a partner of human person in the daily life [2]. In this regard, we argue that the most important technique to achieve this dream is the robot knowledge system and inferencing engine. Nowadays, many domains use the ontology concept to build the knowledge-based system supporting an autonomous symbiotic human-robot. These domains focus on designing a database schema for storing ontologies and developing a reasoning system based on the rule. The most important goal of using the ontology concept is known to 1

This work was performed for the Intelligent RoboticsDevelopment Program, one of the 21st Century Frontier R&D Programs funded by Korea Ministry of Commerce, Industry and Energy. This work was supported by the Korea Science and Engineering Foundation (KOSEF) grant funded by the Korea government (MOST) (No. R01-2007-000-20135-0).

579

Copyright © 2010 by the Japan Society of Mechanical Engineers

ontology-based 5W1H. Finally, we conclude our work in Section 5.

and SkillsProcedure Memory. Fig. 1 shows the relationship between three memories based on Tulving's theory [5]. If Working Memory recognizes "Bring Apple", then Working Memory System searches the related knowledge from three memories. For example, it searches the ontologies, such as Bring, Apple, Fruit and etc., from FactsSemantic Memory, and the skills information, such as Grasp, Goto, OpenRefrigerator and etc., from SkillsProcedure Memory, and the episodic information from EventsEpisodic Memory as shown in Fig. 1. The classification of the knowledge under a robot environment is very important because the explicit classification of the knowledge increases the system performance by reducing the insertion, deletion and update cost. In this regard, to remove the confuse for the classification, we made the comparison table as shown in Table. 1.

2. CLASSIFICATION OF ROBOT KNOWLEDGE We first present the relationship between the robot brain models to classify the robot knowledge into the knowledge models under a robot environment. In this paper, the robot brain models are based on the human memory system. In general, a human memory is classified into the declarative and non-declarative memories. The declarative memories are Facts/Semantic memory, the system that a human use to store common facts such as the knowledge of the world, and Events/Episodic memory, the system that a human use to store personal experience information at a specific time and place. Otherwise, the non-declarative memories are Skills/Habits, memory subsystem that enable a human to perform specific learned skills or habitual responses, Priming, Simple classical conditioning, and non-associative learning.

Table. 1 Comparison table between FactsSemantic and EventsEpisodic Memory

SkillsProcedure Memory FactsSemantic Memory Skills related with “Bring Apple”  Grasp, Goto, Open Skills, …

Bring, Apple, Fruit, …

Working Memory Bring Apple Reference the method information of What + How

Representation

Reference the semantic information of 5W1H

EventsEpisodic Memory  5W1H When

Where

Who

Why

What

How

Yesterday

Refrigerator

.

Eat

Apple

Bring

Fig. 1 Relationship between FactsSemantic Memory, EventsEpisodic Memory between SkillsProcedure Memory; processing "Bring Apple" in Working Memory means that the related knowledge from three memories is loaded into Working Memory. The important memory systems under a robot environment are known to Facts/Semantics memory, Events/Episodic memory and Skills/Procedure memory because these memories can well be represented into an ontology-based knowledge. According to the features of the human memory system, our robot brain models are classified into three memories: FactsSEmantic Memory, EventsEpisodic Memory, and SkillsProcedure Memory. In more detail, FactsSemantic Memory stores the concept knowledge such as Context, Object, Space, Feature and Human ontologies on RocOn [7]. EventsEpisodic Memory stores the experience information of a robot, using 5W1H that consists of WHEN, WHERE, WHO, WHY, WHAT and HOW. Finally, SkillsProcedure stores the skill information that is mapped to the rule information used by action or task planner. In practice, we designed the robot brain store in Section 3 from a motivation example describing the relationship between FactsSemantic Memory, EventsEpisodic Memory

FactsSemantic Memory A non-instance based representation - Concept-based - include concrete and abstract concepts (e.g., ‘house’, ‘cat’, ‘comedy’) - relationship between concepts (e.g., “a house can be made of wood”, “a cat says meow”)

Stores

General Knowledge (also rules and concept), and facts

Scope

N/A

Contents Share with other View

Factual information Possible Objectivity

EventsEpisodic Memory An instance-based representation - Each perceptual experience results in the encoding of a new instance of memory

Events and episodes In particular spatial and temporal context Private memories Impossible Subjectivity

3. ROBOT BRAIN STORE The inferencing ability of an autonomous symbiotic human-robot depends on the volume of the knowledge of a robot. That is, inefficient management of the robot's knowledge causes the bottle-neck of the inferencing system. In this regard, we first propose the store model for efficiently managing the knowledge as shown Fig. 2. In this paper, we use RocOn that is the ontology-based multi-layered robot knowledge framework. RocOn, called OMRKF, has four levels of knowledge; Perception, Model, Context and Activity. Each level of knowledge has three knowledge layers; high level, middle level and low level knowledge layer. In addition, each knowledge layer

580

has three ontology layers; meta ontology layer, ontology layer and ontology instance layer.

this regard, we propose EventEpisodic RBS with 5W1H structure based on ontologies.

Context Hierarchical Relationship

5W1H

Human Interaction

5W1H

[RocOn] Knowledge

Instances

Association

RocOn What TV watched at this hour yesterday?

Observation

Context Object

Yesterday, you watched “ESPN News”.

Space Feature

Providing the episodic information From 5W1H

[RocOn]

Human

Knowledge Association

Fig. 3 An example of using 5W1H in terms of interaction between Human and Robot

Action Task Sub-Task

4. TEMPORAL EPISODIC STRUCTURE MODEL

Primitive Behavior

In this section, we present Temporal Episodic Structure Model based on 5W1H as the knowledge model for EventsEpisodic RBS. Before describing TES model, we formally define the context. Previous researches has defined it as that the context is a situation for robot's actions, or under an environment. However, the definition of context has not only various concepts, but also the various representations such as the symbolic-based or the ontology-based context. In this regard, we formally define the context based on 5W1H to avoid the confusion.

 Rule-based Reasoning  Value-based Reasoning  Ontology-based Context Analogy Reasoning

Fig. 2 Robot brain store model for storing RocOn based on robot knowledge defined by ontologies. As shown in Fig. 2, the whole structure of our store model physically consists of FactsSemantic, EventsEpisodic, and SkillsProcedure RBS from the relationship between three memories of Fig. 1. All the RBS are based on ontology-based knowledge, especially using OWL language [5]. That is, as RBS uses RocOn for ontology-based knowledge framework, we only focus on EventsEpisodic RBS for managing 5W1H. Although we do not treat other RBSs, we argue that the physical classification of the robot knowledge is important factor. Finally, the proposed three RBS can used to support following reasonings/inferencings: Rule-based reasoning, value-based reasoning, ontology-based context analogy reasoning. EventsEpisodic RBS has two advantages as followings: i) Providing the episodic information for interacting between Human and Robot, ii) Supporting ontology-based context analogy reasoning. For first advantage, Fig. 3 shows the situation of an interaction between Human and Robot. Answers for some questions maybe depend on the episodic information under many interactions. If the episodic information is not well managed, the interaction such as shown in Fig. 3 cannot be achieved. For second advantage, many domains from now did not consider the episodic information for inferencing the context. If the episodic information are stored according to the time sequences and is correctly recognized, ontology-based context analogy reasoning can infer more correct context with reducing an error ratio. In

Definition 1 (Context): A context consists of WHAT and HOW based on ontologies. WHAT is ontologies becoming the purpose for Behavior. HOW is ontologies indicating the action of Object. Accordingly, a context is defined by followings: Context  ([ Object  WHAT ] , Behavior  HOW )

Fig. 4 shows how to describe the context such as "Take Lecture" and "Have Meal" from Definition 1. Notice that we use RocOn that is ontology-based multi-layered robot knowledge framework. That is, we add our concept for the context to RocOn. Context Pool (ObjectWhat, BehaviorHow,) Behavior Ontologies

Take

Have

Hungry Meal

Object Ontologies

Lecture

Seminar

Breakfast

Dinner Lunch

Fig. 4 An example of describing the context As shown in Definition 1 and Fig. 4, a context has a behavior ontology and the object ontologies. It has one more objects and a behavior because a number of objects maybe have one behavior. By using this concept for the context, we can clearly define it without the semantic confuse. As a result, this concept is connected to WHAT

581

When

Where

Who

Why

What

How

Laboratory

Professor

Present

Seminar

Take

Instance Knowledge

START TIME END TIME

2008-06-29 14:00:12 2008-06-29 18:00:12 2008-06-30 10:00:12

Chair LivingRoom

Mother

Hunger

Meal

Have

2008-07-01 15:00:00 2008-09-04 10:00:00

Desk

(color,brown) (height,100cm) (width,120cm)

Notebook Projector

LectureRoom

Student

Learn

Lecture

Take

2008-09-04 13:00:00

Desk Chair

(color,brown) (height,100cm) (width,60cm)

Fig. 5 An example of the episodic information. and HOW of 5W1H. 5W1H is defined as shown in Definition 2.

consider that a robot recognized the following context; "Students take a lecture to learn at a lecture room from 2008-09-04 10:00:00 to 2008-09-04 13:00:00". Then, the episodic information consists of WHEN (2008-09-04 10:00:00,2008-09-04 13:00:00), WHERE (LectureRoom), WHO (Student), WHY (Learn), WHAT (Lecture), HOW (Take), and the instance knowledge such as Desk and Chair as shown in Fig. 5. In many robot environments, the episodic information as described above maybe is very important to achieve the high-level context inferencing. However, applying the concept for the episode has two problems: i) difficulty classifying each element of 5W1H, ii) difficulty managing voluminous episodic information. For the first problem, the extraction of 5W1H information needs to interact between external modules such as Task/Action planner and Context reasoner. However, the problem for extracting WHY information still was not solved. For the second problem, the episodic information generating while a robot is working is voluminous and is difficult to manage. To solve this problem, many domains have used the information management based on Relational Database System. We only focus on the second problem because the first problem is beside the point in terms of designing and managing TES model. Efficient management of the episodic information determines the performance of an inferencing engine for supporting the service task. In this regard, we designed the database schema of EventsEpisodic RBS based Relational Database System. Fig. 6 shows the database schema of EventsEpisodic RBS implemented from TES model. Notice that all the RBS described above share Resource and owl_Triples Table because OWL ontologies are based on RDF Triple [9] with (subject, predicate, object). As RDF Triple is the smallest unit for describing OWL ontologies, we can represent all ontologies, e.g. Object, Context and Space ontologies, as a set of RDF triple. In this regard, the physical data structure to store OWL ontologies in Relational Database System has basically two tables like owl_Triples and Resource. Episode table for storing an episode consists of ID, startTime, endTime, where, who, why, what and how. As each attribute of its table should

Definition 2 (Temporal Episodic Structure (TES) Model): TES Model is the sequence of 5W1H to store the episodic information of a robot. Additionally, this has the time sequences of 5W1H with the performed period, from start time to end time, of specific context. That is, this has the following definition: TES  SEQUENCE  WHEN , WHERE ,WHAT ,WHY , WHO, HOW , IK , where WHEN is Time ontologies, WHERE is Space ontologies of RockOn, WHAT is Object ontologies of RockOn, WHY is Behavior ontologies, WHO is Human ontologies, HOW is Behavior ontologies, and IK is the recognized Instance Knowledge

As shown in Definition 2, Temporal Episodic Structure for storing 5W1H records the episodic information (episodes or past facts) under the temporal dimension. Separating between episodes is the context corresponding to specific events or situations. All elements are referred to Time, Space, Object, Behavior, Human, and Instances, respectively. Especially, Time ontologies are based on "Time Ontology in OWL [8]" proposed by W3C, representing Date Time Description(year, month, week, day, hour, minute, second), Duration Description(years, months, weeks, days, hours, minutes, seconds), Periodic Description(morning, noon, evening, summer, winter and etc.), Relative Duration Description(yesterday, today, tomorrow, last, next), and Temporal Entity(now, before, after, hasBegin, hasEnd). Also, for Human ontologies, we modeled the generic ontologies with several features; Uniqueness (an unique existence such as Face and Voice Model), Character Type( the conversation type), Social Network(the relationship between Human), and Personal Information(Name, Age, Phone, Gender, Address, and etc.). We are omitted since the description for these ontologies is beside the point of our paper. Fig. 5 shows several episodic information according to the start and end time of each past episode. For example,

582

refer to other ontologies belonged to oneself, there have one foreign key of Resource table as shown in Fig. 6.

is valid, then Query Generator generates an SQL, the query language on Relational Database System, for the valid 5W1H and SQL Executor runs the generated SQL to store a set of 5W1H. Episode Query Processor processes an input 5W1H query that has various patterns to search 5W1H from EventsEpisodic RBS. That is, the query processor delivers the parsed condition set for an input 5W1H over to 5W1H Query Validator. 5W1H Query Validator checks the uses of the ontologies mentioned in an input 5W1H. The validator must signal an error if the mentioned ontologies are not existed. After validating the correct uses of ontolgoies, the validated query condition set is delivered over to Query Condition Generator. Finally, Query Condition Generator generates an SQL with the Boolean expressions, located in WHERE-clause of an SQL, from the validated query condition set. In this paper, the 5W1H query has the following regular expression:

Fig. 6 Database schema of EventsEpisodic RBS Finally, the instance knowledge is stored in InstanceInEpisode table. Notice that its table is separated from Episode table because an episode can take many instances. The design of our database schema increases the performance of the update, insertion and deletion of the instance knowledge because it minimizes the knowledge duplication. Until now, we presented the database schema of EventsEpisodic RBS as the physical view of TES model on Relational Database System. We design the architecture and query API of Events Episodic RBS to use TES model. Episode Reasoner for Service Task

5W1H Result Set

5W1H Query

Query := Condition (‘&’ | ‘|’) Query Condition := '(' WHEN | WHERE | WHAT | WHO | WHY | HOW ')' WHEN := ‘WHEN = ‘ TimeDuration | StringType WHERE := ‘WHERE =‘ StringType WHAT := ‘WHAT =‘ StringType WHO := ‘WHO =‘ StringType WHY := ‘WHY =‘ StringType HOW := ‘HOW =‘ StringType

Episode Data Processor

Episode Query Processor 5W1H Query Condition Set

5W1H Query Validator

For example, consider that we want to query an episode: WHO is Student, WHEN is Yesterday, and WHERE is LectureRoom. An input 5W1H query for an example above is "(WHO='Student' & WHEN='Yesterday' ) | (WHERE='LectureRoom'). This query is finally converted into an SQL:

5W1H

Episode Validator 5W1H Checker

Validated Query Condition Set Valid 5W1H

Query Condition Generator Boolean Expression Of Query Condition

SELECT startTime, endTime, Where, What, Who, Why, How FROM Episode WHERE who = (SELECT id FROM Resource WHERE localName='Student') and where = (SELECT id FROM Resource WHERE localName='LectureRoom') and startTime like '2010-04-01 00:00' and endTime like '2010-04-01 23:59'

Query Generator

SQL Executor

EventsEpisodic RBS

Fig. 7 Architecture to store and query 5W1H Fig. 7 depicts our architecture to store and query 5W1H on EventsEpisodic RBS. Our architecture is divided into two parts: Episode Data Processor and Episode Query Processor. Episode Data Processor processes TES model described by OWL ontologies. After parsing TES model, it delivers a set of 5W1H over to Episode Validator for checking the existence of all elements of 5W1H. As all the described ontologies in 5W1H refer to Object, Space and Context ontologies, 5W1H Checker validates the existance of the mentioned ontologies from the previously stored ontologies. If 5W1H

As shown in above, in case of describing WHEN, Query Condition Generator translates Time ontologies into the date and time according to the current time because WHEN is represented as Time ontologies. Accordingly, the startTime and endTime for the interpreted time information is "2010-04-1 00:00" and "2010-04-01 23:59" if the current time is "2010-04-02 13:00".

583

Finally, Episode Reasoner for Service Task uses Episode Query Processor to infer the context. It can be the part module of the context inference engine.

Symbiotic Human-robot System”, International Journal of Automation and Control 2007 - Vol. 1, No.1, pp. 64 - 83, 2007. [3] Dieter Fensel, “Ontologies: A Silver Bullet for Knowledge Management and Electronic Commerce Second Edition”, Springer, 2004. [4] William Bechtel, “Mental Mechanisms: Philosophical Perspectives on Cognitive Neuroscience”, Lawrence Erlbaum, 2008. [5] Linda Laurila, “Neuropsychology of Semantic Memory: Theories, Models, and Tests”, School of Humanities and Informatics in University of Skovde, Sweden, 2007. [6] Deborah L. McGuinness, Frank van Harmelen: OWL Web Ontology Language Overview W3C Recommendation. Feb 2004. See http://www.w3.org/TR/owlfeatures/. [7] Il Hong Suh, Gi Hyun Lim, Wonil Hwang, and Hyowon Suh, "Ontology-based Multi-layered Robot Knowledge Framework (OMRKF) for Robot Intelligence", Proceedings of the 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 429 - 436, 2007. [8] Jerry R. Hobbs and Feng Pan, "Time Ontology in OWL", W3C Working Draft 27 September 2006. See http://www.w3.org/TR/owl-time/. [9] Dave beckett and et al.: RDF/XML Syntax Specification (Revised), W3C Recommendation 10 February 2004. See http://www.w3.org/TR/rdf-syntax-grammar/. [10] Thomas R. Gruber, "Toward Principles for the Design of Ontologies Used for Knowledge Sharing", International Journal of Human-Computer Studies, 43(5/6), pp. 907-928, 1995.

5. CONCLUSIONS AND FUTURE WORKS In this paper, we have presented EventsEpisodic RBS for TES model. This approach uses the ontology-based 5W1H that represents the episodic information into consisting of six attributes: WHEN, WHERE, WHAT, WHO, WHY and HOW. The advantage of using 5W1H provides more intelligent task services by using the episodic information for the interaction between Human and Robot. Also, it can support an ontology-based context analogy reasoning through the analysis of the past context stored in EventsEpisodic RBS. The additional advantage can efficiently manage voluminous episodic information while a robot is working. Finally, we designed the architecture to store and query 5W1H on EventsEpisodic RBS. We plan to extend the concept of ontology-based 5W1H to hierarchical ontology-based 5W1H. The current 5W1H has a flat structure according to the time sequence without the hierarchy between episodes. As future works, we plan to support the more correctable context inferencing for service tasks, applying the hierarchical concept. REFERENCES [1] Haruki Ueno, “A Knowledge-based Information Modeling for Autonomous Humanoid Service Robot”, IEICE Transaction on Information & System, Vol. E85-D, No. 4, pp. 657 - 665, 2007. [2] Tao Zhang and Haruki Ueno, “Knowledge model-based Adaptive Intelligent Control of Robots for a

584