Semantic Connection between Everyday Objects and a Sensor Network

Semantic Connection between Everyday Objects and a Sensor Network Michita Imai1 , Yutaka Hirota, Satoru Satake, and Hideyuki Kawashima Keio University, 3-14-1 Hiyoshi Kohoku Yokhama 223-8522, Japan, [email protected], http://www.ayu.ics.keio.ac.jp/

Abstract. This paper proposes middleware for achieving a semantic sensor network. It generates logical descriptions of the state of environments based on sensors attached to everyday objects. What a sensor network must do to describe an environment is to connect sensory readings with the everyday objects logically. We must prepare information about the logical connection even though they are physically connected. On the other hand, it is hard for us to prepare the information one by one. The semantic sensor network employs the class definitions of everyday objects for managing sensory data related to the everyday objects as the instances of the classes. In particular, what we must do in preparing the semantic sensor network is to just connect the sensory devices to the everyday objects because it refers to the RFID tags on the objects in generating the instances of the classes. Moreover, the generated instances become the basis for inferring the states of the environments. We have developed two physically grounded applications on the semantic sensor network; one is a GUI based system and the other is a robotic system.

1

Introduction

Sensor networks have been developed for physically grounded applications such as intelligent robots [1] [2], and intelligent rooms [3] [4] to recognize the activities of humans or the states of an environment. The recognized information can cover a wider area than the one that sensors and cameras mounted on the robots or located in the rooms obtain. Since each sensor node of a sensor network monitors the environment directly, the physically grounded applications can obtains more precise information. Those are reasons why many researchers start to employ the sensor networks for their applications. However, levels of representations of the obtained information are different depending on each application. Some applications may require raw sensory data, others may do information about events or objects. This paper proposes a sensor network which generates logical descriptions of an environment for many types of applications to share. The difficulty in generating the environmental descriptions is to extract meaningful information from sensory readings. However, the sensory readings have little meaning by themselves because they are just raw data. We must prepare a mechanism to extract meaningful information from the raw data. For

II

example, if a sensor network finds the location of humans, it must have knowledge of what features indicate the existence of humans. Moreover, it requires knowledge of what sensors can find the features and what situation makes them functional. In other words, the difficulty comes from connecting the sensory data to the features of an environment. There are several studies which deal with sensor networks with respect to the connection between sensory data and the features of environments. Michahelles et. al [5] has proposed middleware by using three layered model which interprets sensory data based on a attaching relation between a sensor node and an object. The relation with the object gives the sensory data an interpretation. For example, data from an acceleration sensor attached to a cup denotes the tilt or the movement of the cup. If the attaching relation is not given to the sensor network, the acceleration data cannot suggest any physical state. Although a lot of studies [6] [7] [2] have attached sensor nodes to everyday objects such as cups, sources, books, furniture, home electric appliances, they are different from Michahelles’ study. They do not prepare a common model of interpreting sensory data. In addition, studies on a description language are important in terms of preparing logical description of environments for anonymous physically grounded applications. The study [8] proposes a description language based on ontology for ubiquitous and pervasive computing. They try to describe the state of humans’ activities and interactions. The study [9] attempts to describe the state of a VR environment using logical expressions. Although the three layered model [5] interprets sensory data based on objects, it does not care about a cost of introducing new objects to the sensor network. Since the three layered model employs a primitive expression for defining the features of the objects, a designer must write down each feature separately. There is no generic expression of what features an object has. Moreover, he/she must give each sensor node the information of the attaching relationship with an object. The separate definitions of the features make the model of each object incomprehensible to the designer and prevent him/her from developing the sensor network efficiently. The method [8] to produce logical expressions based on ontology is also inadequate. The produced expressions are far from general use in terms of anonymous applications because information of activities and interactions of humans heavily depends on the requirement of each application. We must consider what information must be dealt with in terms of the general use. The study on VR [9] can describe information like the relationship between objects in an environment with a general expression. However, it cannot cope with the dynamics of the real world. This paper proposes middleware for achieving a semantic sensor network which we call SS. SS connects physically grounded applications to the information on the environment of everyday life via a sensor network. SS infers and describes the state of an environment using logical expressions, maintains the expressions along with dynamic changes in the environment, and answers queries about the environment. SS deals with an environment where wireless sensor nodes are attached to everyday objects. SS excludes humans activities or in-

III

teractions between humans from the target to make contents common to many types of applications as much as possible. The logical expressions are also employed to intend to make the contents shared easily. SS conducts semantics on the sensor network by preparing an interpretation model for sensory data and queries about the environment. The remarkable achievement of SS is to employ explicitly the concept of a class and an instance of everyday objects so as to construct the interpretation model efficiently. The class corresponds to the generic definition of an object. A designer can comprehensively prepare the features of the object and the possible states of the object in the class definition. The instance corresponds to a logical description associated with each object. The instance of an object consists of metadata, inference rules, and sensory data. SS generates the instance of an object by copying the contents of the associated class definition when identifying a physical connection between a sensor node and the object by a RFID tag. In other word, the interpretation model is not only automatically but also dynamically constructed depending on the physical connections. What a designer must do in constructing the interpretation model is to prepare only the class definitions. Moreover, SS can also infer the states of the objects and their relationship by referring to the generated instances. The reminders of the paper are organized as follows. Section 2 describes the requirement of physically grounded applications for sensor network and explains fundamental configurations which the paper employs as a sensor network. Section 3 proposes a semantic sensor network and explains what information it prepares to connect the physically grounded applications to the information of the real world. Section 4 describes a prototype system of SS and demonstrates the behaviors of SS by showing two examples. Section 5 is a conclusion of the paper with our future plans.

2 2.1

Applications on Sensor Networks Physically grounded applications and common use of a sensor network

Physically grounded applications such as ubiquitous computers, intelligent rooms, multi-modal interfaces, mobile human interfaces, and robotic systems possess their own interpretation model for interpreting sensory data to obtain information about the real world from sensor networks. Although sensory data, as we already mentioned in Introduction, have little meaning, the sensory data do not become a meaningful information before the model interprets them. Also each model is provided depending on what information each application requires. There are pieces of information which many applications can share on the sensor networks even though they gather the information to achieve different aims. For instance, an intelligent room gathers the information about the locations of objects to monitor the states of the objects and a robot does to

IV

manipulate the objects. Since the information about the location and the state of objects is general in terms of the difference between the applications, it is redundant that each applications possess its own model for obtaining the information. This paper proposes a middleware to prepare the information for many types of applications. However, it is difficult for the applications to share information about the activities of humans because what information must be extracted from the motions of humans considerably depends on the context of an interaction between a human and an application. For example, some results of the interpretation exit when a human raises his/her arm and moves his/her hand from side to side. Someone may intend to say goodbye or someone may call somebody with the motion. Since human’s activities must be interpreted depending on the context of the interaction, we leave the extraction of the information to each application. Moreover, we assume the other use of the middleware. That is, humans use the middleware directly throughout a GUI based system. This interface is also a physically grounded application. They can look for objects with the interface. Since the interface will fascinate many users, we also include the interface in the applications on the middleware. 2.2

Requirements in describing environments

The middleware we propose describes the states or the locations of objects, and the relationships between them to equip physically grounded applications with the basis of information about the real world. Since information used for the descriptions must be extracted from sensory data and be shared with many types of applications, there are several requirements in describing them. The middleware must deal with the description by connecting sensory data to an object in the real world [5]. The type or the feature of an object converts sensory data into meaningful data as the example of the cup and the acceleration sensor in Introduction. Although the value of the acceleration sensor does not express any situations by itself, the type and the feature of a cup gives an interpretation as the movement of the cup. In other words, the facts of the connections correspond to an interpretation model based on the cup. However, since there are vast numbers of objects in environments, preparing the information about the connection sensor by sensor or object by object increases the complexity of the models. For instance, we can prepare the relationship between a sensor si and an object which has features from fj to fk as follows. si → (fj , ..., fk ) (1) For example, the formula corresponds to the connection between a cup and an an acceleration sensor. Also, we can express a condition between objects. The following formula expresses that result occurs if the states < fi : sj > of several objects satisfy condition. if (< fi : sj >, ..., , < fk : sl >, condition) → result

(2)

V

Fig. 1. Connections between objects and sensor nodes

Here, < fi : sj > denotes the set of a feature fi and a value sj . Since the value sj comes from sensory data, < fi : sj > corresponds to the interpretation of the sensory data. For example, the formula (2) corresponds to the movement of the cup if condition inspects a change in the value of acceleration sensor. We must prepare those formulas object by object when we employ the defining method. In contrast to defining the formula (1) object by object, the middleware should have a framework for generically defining an object to reduce the complexity of designing the model. Moreover, since the formula (2) also depends on the type of an object, the definition of the object should include the formula (2). Also, what language the middleware employs is significant for the designer to develop an interpretation model or applications. Since the middleware describes the states of the environment, logical expressions are more appropriate than procedural languages. In particular, the logic expressions are also easily transformed into natural languages. We also employ the logical expression for the applications to give a query to the middleware. 2.3

Acquiring information of everyday objects

The paper employs everyday objects as the target of the description. Figure 1 shows the example of the sensors attached to a plastic bottle and a book. We use ultra sonic sensors to identify 3D location [10] of an object and MICA-MOTEs [11] to recognize the states of the object. MICA-MOTE is a wireless sensor node which has an acceleration sensor and thermometer. Also, we can add extra sensors to it. For example, a bend sensor can detect whether the book is open or closed. We propose a framework for generically defining the connections and the rules between the objects and the sensors. The middleware describes the states and the locations of the everyday objects and the relationship between them based on the definition of the objects. 2.4

Inference on sensor networks

We also employ inference on the descriptions of the everyday objects to obtain further descriptions of the environment. The middlware infers the description based on the rules which object definitions introduce.

VI

Since the sensory data indicate changes in the locations or the states of objects, an inference engine must be capable of dealing with symbols which appear and disappear dynamically. The engine must reflect on the changes during the inference to describe the new states of the real world. Also, we prepare three types of inference for queries given by physically grounded applications. The first is top-down inference triggered by given queries. The queries divides into more basic expressions according to inference rules until the expressions match descriptions the middleware already has. In other words, the middleware prepares semantics for the queries by introducing the inference on the interpretation model. The second is to monitor whether a desired condition occurs. We can also trigger the monitor using queries. The queries also divide into more basic descriptions but the middleware starts monitoring the appearance of the descriptions when it cannot divide the descriptions anymore. The third is bottom-up inference which proceeds from the basic descriptions such as that of the everyday objects to more abstract descriptions or relationships between them. Nowadays, computers have plenty of computational power. Since almost of them remain idle, the middleware can produce additional descriptions by using the remaining power. If the middleware infers the extra descriptions, the cost of top-down inference decreases because the times dividing the queries decreases. However, the three inferences must be achieved while descriptions appear and disappear.

3

Semantic Sensor network

The paper proposes a middleware named SS which becomes a basis of a semantic sensor network. It connects physically grounded applications to information about everyday objects. SS has several features as follows. – SS describes the states and the locations of everyday objects, or their relationships by using a logical expressions. – SS employs the set of sensory data and metadata as the data structure of the descriptions. The metadata corresponds to the logical expressions. – SS has the class definition of an everyday object which provides the feature of the objects and the basic inference rules of their possible state in the form of metadata. – SS creates the data structure of each object in the form of an instance of the class definition. The creation arises when a sensor node is attached to a everyday object. – SS also has relation inference rules for obtaining the relationship between objects. This rules are defined directly on SS regardless of the class definitions. Applications can add their own new relation inference rule to SS. – SS infers the locations of objects, their states, and the relationships between them using the inference rules, metadata, and sensory data. SS also expresses the results of the inference in the form of metadata.

VII

Fig. 2. The overview of Semantic Sensor Network

– SS interprets given queries based on the descriptions. The class definition corresponds to the generic definition of an object and resolves the complexity when preparing an interpretation model. Creating the instances constructs dynamically the interpretation model on a sensor network. Also, the model prepares semantics in interpreting given queries. 3.1

Metadata, sensory data, and inference rules

The data structure of metadata and sensory data is a basic element of the interpretation model on SS. Since the constituent of metadata corresponds to the feature of a everyday object, it puts meaningful interpretation on sensory data. Also, the relationship on the data structure corresponds to the formula (1). The metadata consist of logical expressions and basic inference rules. The logical expressions take the form of an atom ni , a feature fi , the pair of a feature and a value fi (v), and a binomial relation ri (nj , nk ) between objects. For example, the atom is the name of an object, the feature is the name of a material which the object is made of, the pair of the feature and the value is the color of the object, and the binomial relation is the relationship between the locations of two objects. Also, the logical expressions divide into static metadata and dynamic metadata. The static metadata is the set of default metadata which expresses the type of an object. The dynamic metadata is given as the result of the inference. The form of an inference rule is xi , xj , xk → xl . Each x denotes the logical expression. The number of the expression in the left side of the rule is from one to three. The rule expresses that the logical expression xl exists in the environment

VIII

if all expressions in the left side are satisfied. SS adds the new logical expression xl as dynamic metadata at that time. There are two types of rules: a basic inference rule and a relation inference rule. The basic inference rule is related to only each objects and used to infer the state of the object. Because of the restricted effect of the basic inference rule, it is encapsulated in the metadata of each object. Since SS uses the relation inference rule to identify the relationship between objects, SS manages the rule regardless of the instances of objects. Sensory data is expressed as the pair of senor ID and the value (ID, v). Since the description of an everyday object becomes the set of the static metadata, the sensory data, the dynamic metadata, and the basic inference rule, SS expresses it as follows. object[nj , (ID, v), ..., fk (v), ..., on(nj , nj k), ..., if (fi (sj ), ..., , fk (sl )) → xl , ...] For example, a red book on a desk is expressed as follows. Here, the example omits the basic inference rules. object[bookA, (#13, 40), (#25, 0.00), clolor(red), on(BookA, Desk)] 3.2

Class and instance of objects

The remarkable property of SS is to have class definitions of objects. The definition make it possible for a designer to introduce the elements of semantics in terms of the object definition. Since the object is a basis of meaningful components, humans easily design the semantic model on a sensor network. Figure 2 shows the overview of SS. The rectangle in the upper part of the figure denotes a class tree. Each class definition consists of static metadata and basic inference rules. Moreover, the class definitions use inheritance along the tree structure. An instance is generated when a sensor node is attached to an object (see also the rectangle in the center of Fig. 2). SS prepares basic inference rules and static metadata by copying the contents of class definition when generating the instance. SS also inserts sensor ID and some values in the static metadata. After preparing the instance, it starts reading sensory data and generating dynamic metadata. If an instance already exists when a sensor is attached to the object, SS adds sensor ID to the static metadata related to the object and starts giving sensory data to the instance. 3.3

Queries on SS

SS interprets given queries based on the relation inference rules. There are three types of queries: asking the environment, monitoring them, and inserting a new relation inference rule. SS employs a logical expression like first ordered logic.

IX

Fig. 3. Hardware structure of prototype system

For example, if you want to find an object whose color is red and which does not move, IsA(X, red), stay(X) is a query. To find X, SS carries out topdown inference and refers to object description based on the rule IsA(X, red) : −object[X, ......, color(red), ....]. If there is object[BookA, ..., stay(BookA), color(red), ....], X becomes BookA. Since stay(X) also uses a similar rule stay(X) : −object[X, ......, stay(X), ....], the other X also becomes BookA. Then, SS generates answer X = BookA. Here, since stay(BookA) is the result of inference referring to the location sensory data, it must be inferred. However, since SS always carries out bottomup inference, SS does not need carries out inference at the timing of answering the query. In addition, SS manages the appearance and disappearance of logical expressions by generating dependency trees between expressions. If an expression disappears, SS erases the expressions related to the reference from the tree.

4 4.1

Prototype System of SS and Physically Grounded Applications Prototype system

Figure 3 shows the hardware structure of a prototype system. The system consists of small sensor nodes, RFID tags, physically grounded applications, and a sever. The server is designed depending on the concept of SS. It has environment information DB and sensor information DB. The sensor information DB manages the class definitions of everyday objects. The environment information DB manages the instances of the objects and the relation inference rules.

X

Fig. 4. Example of generating instance

The prototype system employs RFID tags (µ tip developed by Hitachi co.) to identify a connection between a sensor node and an everyday object. SS generates the instance of an object when a user inputs ID of the object by a ID reader. However, since the ID reader is divided from each sensor in the current implementation, he/she must use the reader after attaching the sensor to the object. Figure 4 shows the example of generating instances. A user has picked a 3D location sensor from the book and attached it to the plastic bottle in the pictures. The graph under the pictures indicates the time sequence of the change in a sensor value and the change in the dynamic metadata which is relations between the bottle and the desk in the example. The above two lines correspond to the metadata and the lower two lines do the sensor’s levels of height from the floor. In the example, since the generated instance belongs to the class of a book at first, the metadata related to the book is generated. Although SS does not generate any metadata while removing the sensor, it generates the metadata related to the bottle after attaching it to the bottle.

4.2

Real world search with GUI

Figure 5 shows a GUI based system with a user can look for an everyday object or monitor the location and the state of the object. He/she can use keywords or logical expressions like a first ordered logic in the GUI. The GUI displays the metadata of a target object. In the figure, the user set a query to monitor the

XI

[A]

[B]

[C]

Fig. 5. Monitoring the states of the plastic bottle by GUI

plastic bottle. At first, it was on the desk. Then, he brought it by his hand. At last, he put it on the book. In response to the sequence, GUI displayed the result of monitoring it in the center: “onDesk,” “Bringing,” and “onBook.” Also, since the class definition of the bottle possess its picture as metadata, GUI shows it. In the example, GUI generates a query monitoring where(bottle, X) to ask SS to monitor the bottle. 4.3

The use of SS by Robot

Figure 6 shows that the robot uses SS to interact with a human. In the example, the robot also tries to monitor the location of the plastic bottle. He took the bottle and then put it on the book. In the example, the robot must obtain not only information about the logical position of the bottle, but also the exact location of the bottle to point to it. The query about the exact location must be added to the query as follows. monitoring where(bottle, X), monitoring location(bottle, Y ) The robot generates utterances “it is on the desk” (the left of Fig. 6), “you are bringing it” (the center of Fig. 6), and “it is on the book” (the right of Fig. 6). These utterance expressions are generated depending on the answers of the queries. The answers are On(bottle, desk), Bring(someone, bottle), and On(bottle, book). Since the logical expressions have somewhat similarity to the utterance expressions, SS also have an advantage when the applications generate the response to humans.

XII

[A]

[B]

[C]

Fig. 6. Monitoring the states of the plastic bottle by a robot

5

Conclusion

The paper proposes a middleware named SS as a basis of a semantic sensor network which connects physically grounded applications to information about the real world. SS gives sensory data metadata depending on the connection between a sensor node and an object. In particular, SS has the class definitions of objects which are the basis to prepare the metadata. Since SS generates the description of the environment as instances of the class definition, it can construct the interpretation model of sensory data dynamically. The paper developed a GUI based system and an intelligent robot as an applications on SS. They can obtain the descriptions of everyday objects throughout SS. SS employs a centralized server to describe the environment. However, we must take account of distributed servers for SS to extend over much larger area. Moreover, SS must deal with heterogeneous environments where some objects have sensors and the other not. In the future, we must deal with the environment to make SS a robust system.

References 1. Morioka, K., Lee, J., Hashimoto, H.: Human-following mobile robot in a distributed intelligent sensor network. IEEE Transactions on Industrial Electronics 51(1) (2004) 229–237 2. Chong, N.Y., Tanie, K.: Object directive manipulation through rfid. In: Proc. Int. Conf. on Control, Automation, and Systems. (2003) 22–25 3. Estrin, D., Culler, D., Pister, K., Sukhatme, G.: Connecting the physical world with pervasive networks. In: IEEE pervasive computing, 1(1). (2002) 59–69 4. Harris, K.J., Meyers, S., Brumitt, B., Hale, B., Shafer, M.: Multi-camera multiperson tracking for easyliving. In: Proc. of Third IEEE International Workshop on Visual Surveillance. (2000) 3–10 5. Michahelles, F., Antifakos, S., Schmidt, A., Schiele, B., Beigl, M.: Towards distributed awareness - an artifact based approach. In: Proc. of Sixth IEEE Workshop on Mobile Computing Systems and Applications (WMCSA2004). (2004) 82–93

XIII 6. Holmquist, L.E., Mattern, F., Schiele, B., Alahuhta, P., Beigl, M., Gellersen, H.: Smart-its friends: A technique for users to easily establish connections between smart artefacts. In: Proc. Ubicomp 2001, LNCS No. 2201. (2001) 116–122 7. Tapia, E.M., Intille, S.S., Larson, K.: Activity recognition in the home setting using simple and ubiquitous sensors. In: Proc. of PERVASIVE 2004, vol. LNCS 3001. (2004) 158–175 8. Chen, H., Perich, F., Finin, T., Joshi, A.: Soupa: Standard ontology for ubiquitous and pervasive computing. In: Proc. of the Sixth Annual International Conference on Mobile Computing and Networking. (2004) 22–26 9. Yashima, E., Saito, S., Okumura, M., Nakajima, M.: Expression of relative location in virtual world. In: Technical Report of IPSJ, 2001-CG-104. (2001) 9–12 10. Nishida, Y., Aizawa, H., Hori, T., Hoffman, N., Kanade, T., Kakikura, M.: 3d ultrasonic tagging system for observing human activity. In: Proc.of IROS-2003. (2003) 785–791 11. Madden, S.R., Franklin, M.J., Hellerstein, J.M., Hong, W.: The design of an acquisitional query processor for sensor networks. In: Proc. of ACM SIGMOD. (2003) 491–502