An Inferring Semantic System Based on Relational ... - IEEE Xplore

2015 IEEE International Conference on Autonomous Robot Systems and Competitions

An inferring semantic system based on relational models for mobile robotics Jonathan Crespo Herrero

Ram´on Barber Casta˜no

´ Oscar Mart´ınez Mozos

System Engineering and Automation Department University Carlos III of Madrid Email: [email protected]

System Engineering and Automation Department University Carlos III of Madrid Email: [email protected]

School of Computer Science University of Lincoln [email protected]

Abstract—Nowadays, there are many robots with the ability to move along its environment and they need a navigation system. In the semantic navigation paradigm, the ability to reason and to infer new knowledge is required. In this work a relational database is presented and proposed. This database allows to manage and to raise queries of semantic information from any environment in which a mobile robot would perform a semantic navigation, providing advantages such as conceptual simplicity and a fast method for switching between semantic environments, being unnecessary redefine new rules. In this paper, a target topological level is obtained from a semantic level destination.

I.

based on the objects links. This reasoning module is the base on which a semantic navigation can be created. The proposed design adapts to the ontology defined in the environment in which the robot will be able to navigate. II.

R ELATED

WORKS

In recent years, there is a growing interest for semantic knowledge in mobile robots navigation. It represents a breakthrough with respect to navigators that only use metric maps [1] , topological maps [2], or hybrid ones [3].In contrast, semantic mapping [4] is considered a better solution for interaction with humans. In [5], these concepts are applied in order to understand natural language interactions when a person requests a robot to find a new object and the robot must look for that object in the environment.

I NTRODUCTION

Currently, in the robotics world there is a general tendency of implementing behavior mechanisms based on human psychology. A high level of abstraction and the ability to handle a great amount of information on objects and places, eases inference systems to acquire even more information. Robots that are provided with cognitive architectures that will model the environment semantics, will have a large autonomy, will be robust and more efficient. These concepts are also applied to navigation in mobile robots.

All these concepts are used to classify the place where objects are located. That information is used to reach a given location to meet a specific target. Therefore, the robot can find places or rooms semantically related with the target, even if it is in a little-explored or unknown environment. This navigation level allows to complete the necessary information from a partial knowledge of the environment.

The first issue to deal with is how to represent knowledge. Ontologies can be seen as one of the best ways of obtaining this representation and there are groups who are developing ontologies for this purpose [6]. Hierarchies can be established where the highest levels of abstraction define its semantic interpretation. This knowledge is considered for the creation of semantic maps. According to [7] they are considered unique because they take into account all the information sources that have published. They combine information about the existence of objects in a room, land marks or signs, the topological structure, room appearance and size, the space geometry and the data provided by an interaction with human users. The environment ontology is manually codified been based on [8] and [9] they are supported by a wide literature on localization, mapping and places classification as it is explained in [10], [11], [12] [13], [14] or [15]. One difference between the work of [8] and the work proposed in this paper is that the rules defining the ontology hand-coded, must be created by a user with knowledge of logic. In our system, in the case of a manual insertion of knowledge, the user only need add room types that is required to differentiate and objects which may contain, in list form. An interesting idea that is also considered is the connectivity between rooms by probability. The kitchen is often connected to a corridor, but not to the bathroom, for example. This idea is also studied in [16] where probabilities on unknown space are used.

This article shows how a reasoning module works. This module uses a relational database that is used both to store objects and to perceive information to infer new knowledge

In [9], supported in the idea that humans partially adopt a hierarchical representation of spatial organization and a categorization of the spatial structures. They design a spatial

Semantic navigation allows the robot to relate what it perceives to the places in which it is located. This way, an environment model is managed being based on the objects and on the concepts that they represent. The more information that can be managed, the bigger the robot intelligence and autonomy will be. An object is defined by the knowledge of its usefulness and it is linked to other objects that its usefulness is connected in the some way. For example, a pen can be linked to the writing concept and related to sheet of paper. But, writing can also linked to work. And if it is also be linked to the drawing utility, it might be at the same time related to entertainment as well. Entertainment could be connected to relaxing activity.

978-1-4673-6991-6/15 $31.00 © 2015 IEEE DOI 10.1109/ICARSC.2015.22

83

representation divided in different abstraction levels. Therefore, the division of several types of knowledge is common as it is proposed in [9] where the following types of knowledge can be distinguished, been based on how to get them: Acquired knowledge (with its sensors), Confirmed knowledge (by an user), Innate conceptual knowledge (pre-set in the T-Box) and knowledge inferred. There is a constant search for a mechanism that catalogues and identifies a region of space. In [17], the authors try to differentiate areas based on the usefulness of objects that are found. A Naive Bayes Classifier is employed as in [18] to infer the place identification, in order to better group and create a set that represents an area. With this classification method, effectiveness is increased, as it is described in [19].

Fig. 1: Database design for semantic navigation

rooms, without creating new rules. The idea in [24] of keeping on one hand the abstract entities and on the other hand the physical objects perceived, both types of information related through links, remain in this architecture.

In [5] the authors also rely on probabilistic methods based on Naive Bayes for places classification and to obtain the probabilities of objects location. Relations between objectobject and objet-scenario can be used to predict the location of a variety of objects and scenarios.

In the case that the robot identifies objects in a same room with a common utility, it can then deduced that it is located in a room where the robot could find more objects with related utilities. Therefore, new categories for rooms are autonomously created.

Object recognition is another base on which a semantic navigation system may rely on. Although it is not the main subject of this work, it is necessary to consider the real possibilities that exist at present. There are methods used in other works of semantic navigation such as SIFT [20] combined with other methods used in [21], [22] or [23]. The development of a system of active visual search [16] must also be mentioned, it combines semantic hints to guide the process of object searching. III.

The objects and room identification also allows to detect possible inconsistencies. In [25] some methods of how to solve the situation are detailed. In our paper, one method for solve this is to find an object that catalogues the room. If the robot finds a stove in a room identified as living room, then it is also a kitchen. The incoherence is solved by cataloging that room as living room-kitchen, understanding that it is both things. If the object that defines the room cannot be found, that inconsistency must be stored and later the robot must decide how it should solve it. It is strange to find a refrigerator in a living room, but if it happens, the robot must keep the consistency by considering it as an exception.

I NFERENCE MECHANISM

A. Knowledge representation of the environment Robotic navigation systems require a way of modeling and interpreting the environment. In the case of semantic navigation, a large amount of information relating to the environment and to the objects contained needs to be extracted and represented. This representation requires an ontology to establish the concepts and the links of the objects in the environment.

B. Database design In the figure 1, where the database design is shown, a relational database diagram scheme is presented. It is a database diagram on which to build a semantic navigation system, without going into detail on the fields that contain each table, this scheme is used to understand the link among each other. There are different tables that are detailed in the following sections according to the way they are referred to.

Unlike what has is described in [24], where the system of representation of knowledge LOOM is used, or previously in [8], where they employ NeoClassic, depending on a reasoning engine where rules and facts are included, an equally efficient alternative that brings some advantages, is proposed . A relational database can be designed. A considerable advantage of using the proposed system, is the capacity for changing the environment without alterations of database. Using a reasoning engine, rules must be changed by programmers or an advanced IA when the environment (office, house...) is modified. But using the system described in this paper, is not necessary because there are no rules. In [9] user information may be added by a tour. However, the rules must be predefined, as in [8]. In our system the semantics of an environment can be changed from home to office for example, without knowledge representation of logical rules. This makes it easier the use by a final user and avoids the time spent to define rules. The relations between entities are implicit in the database design and only introducing new data is necessary. In addition, this architecture also allows the robot to deduce new types of

1) Basic information tables: This type of tables are those that arise when considering the four elements the robot must store : physical rooms (real locations sensorially perceived by the robot ), conceptual rooms (the type of room the robot can recognize), physical objects (objects perceived by sensors) and conceptual objects (each object’s semantic information that the robot must recognize). Therefore, being based on this description, the following tables exist: ConceptualRoom:This table contains the information of the rooms understood as concepts. The table records are those corresponding to the concepts kitchen, living room, dining room, etc. ConceptualObject:This table stores data relating to objects abstractly defined and understood as concepts. For example, Refrigerator, Sofa, Computer, Washing machine, etc.

84

With these tables the desired use level of the semantics of objects in the environment is achieved, in order to be able to search for them. For example, in the case of the utilities concepts, it can be mentioned that the computer is used to work. If the user requests to go to a place to work, the robot goes to the place where the computer is located.

PhysicalRoom: It Corresponds to physical rooms (understood as locations, places in space, regions delimited by walls, having real coordinate, etc) by which the robot can move. PhysicalObject:This table stores each real object that the robot sensors identifies. This table records can be Chair-1, Chair-2, Refrigerator-1, etc.

With this relational database model, the semantic knowledge is obtained using simple queries without the need of define rules as it has been described in previous works [24] [8].

Relating to these basic tables that represent the model, the following links that generate new tables exist because of the links of the type many-to-many: ConceptualObjectRoom (the relationship between ConceptualRoom and ConceptualObject gives as a result this table) and PhysicalConceptualRoom (the link between ConceptualRoom and Physicalroom produced this one).

IV.

In this section it is shown how the system using a relational database with the implicit knowledge in its design, has been implemented. The system has a topological navigator and the planner translates the semantic information in topological commands.

2) Links between primary tables: A part from the links which have been mentioned, that generate new tables, the following links between the primary tables exist: Between the tables PhysicalRoom and PhysicalObject there is another link that corresponds to the fact that a physical object is located in a single physical room. And a physical room may have an indeterminate amount of physical objects. In table PhysicalObject, in each record the physical room to which it corresponds, is also stored.

A. Semantic target obtention In this architecture it exists a class that defines the concept of semantic position as a contextual information unit related to the robot’s position. Creating an algorithm to locate the target, the input of that algorithm is an object of the class SemanticPosition. The class in this architecture contains attributes that correspond to a room code (the target room in this case) and the object code (the target object in this case), it is therefore understood, that a position is obtained in relation to a semantic reference point, which can be either objects or rooms. A variable of the class SemanticPosition represents the target the robot must go. Several different options depending on the information available in the database on the requested destination and the type of destination (object or room) can be given:

Between the tables ConceptualObject and PhysicalObject another obvious link is found, due to the fact that each physical object corresponds to a single conceptual object. This information is also added to the records of table PhysicalObject 3) Tables that allow searchs by semantic proximity: Figure 1 shows the designed database. In this database there are several tables that manage important information in order for the robot to find objects somehow related with other objects. The idea is, that if the robot does not locate a specific object but it can deduce with what other objects is linked to, then it begins to locate those other objects. To get this information, the database contains the following tables:

Unknown room. If the destination is an unknown room, the field of the room id of the object of the class SemanticPosition that was the function input, will still have void value and the program will launch the corresponding semantic exploration routine to try to locate a kitchen.

Interaction: Objects interact with other objects. This table can handle any type of interaction, although tests have been performed with a limited number of them. The interactions taken into account are: BE A, when an object is a subtype of another object; IS USED WITH, when an object is used with an object and IS INSIDE OF, when usually an object is inside another object. Utility: All objects have one or more utilities. And tendence is to group objects that have the same utility something related) in the same places. And also concepts created regarding kinds of room depending on what they used for.

S EMANTIC INFORMATION MANAGEMENT

Known room. If the target is a room, and if it has been explored, the variable target will have a value in the field of the room id corresponding to the physical room which is already known.

the (or are are

Known object in a known room. Another possibility is that the original target was an object, such as the refrigerator. In that case, after completing the function, the variable Semantictarget of the class SemanticPosition will have the id of the specific object (refrigerator-1) and of the room where it is located (room-3), considering that the object has been explored and located.

Meaning: The actions or utilities that an object has, often are associated with a specific meaning. The goal of the navigation may also be oriented to emotions or places that give a private feeling, such as calm or fun. For example, read is an action quiet.

Unknown object in a known room. If the robot does not have the previous information of examining a refrigerator in the kitchen, the value of the object identifier would be void and the robot would try to explore in room-3 (kitchen) looking for the refrigerator.

Characteristic: Objects may have features to best define the concept or even differentiate them from others. For example, on one hand, water may be cold, warm or hot. And that implies differences in location. Cold water may be found in the refrigerator or in a fountain, but hot water would come out from the fosset.

Unknown object in an unknown room. If the kitchen has neither been located, the robot will first explore looking for a kitchen. The conclusion that the robot must explore looking for

85

Fig. 3: Turtlebot

B. Semantic identification of the room By what has been mention so far, semantic information would be needed to identify the room from a vector of detected objects. There is a function in the system that deletes from the vector the objects already known, to only analyze those observed for the first time. And for all new objects, gets the room codes that the object returns with a query that returns a list of the types of room where the object can be found. If an object is found that its presence defines the room where it is located , then the list of room types will only have one element. For example, the stove is an object that can only be in the kitchen, so that room is a kitchen. If there is a stove in a room containing a sofa and a table and it has been identified as living room and dining room. In this system, the same room can match an infinite number of types of room. If the result of the previous query does not return a single record, it is because the refered object is not discriminatory enough to be able to conclude any identification.

Fig. 2: Diagram that shows the possible actions of the function TargetLocation in an example of finding a chair

V.

E XPERIMENTATION

To carry out tests on this system, focusing on the translation of a topological destination from a request to a semantic target, it has started from a hypothetical situation of a partially explored environment, with information contained in the table Ia (up), with semantics defined by the table Ib and deducted knowledge stored in table Ia (down). On these tables, that represent the knowledge the robot has, the system effectiveness has been tested. Semantic ultimate destination is a geometric target that is used in these tests for a turtlebot (figure 3).

a kitchen occurs both if the kitchen itself has been a requested target as if it has requested an object that may be found in the kitchen, such as the refrigerator. To help to understand the process of performance of this function, an example is shown in figure 2 where the object to look for is a chair. The function TargetLocation takes as input the the state of variable target and its output is the modification of that variable. In the example of locating an object (a Chair in figure 2), 5 possible outcomes A, B, C, D and E can be obtained. In A result, the chair was identified and therefore the room where the object was found was also identified. When the planner encounters a situation of this type, it directly sends the event to the low level navigator(the topological) to go to that room and afterwards to that object. However, for results B, C and D, the variable target does not have information about any physical object, but it has information on the search for a conceptual object CHAIR and it also adds information of the physical room where the robot should go to find a chair.

Name Chair-1 Chair-2 Chair-3 Refrigerator-1 Sofa-1 Desk-1

PhysicalObject ConceptualName CHAIR CHAIR CHAIR REFRIGERATOR SOFA DESK

RoomName Room-1 Room-2 Room-3 Room-1 Room-2 Room-3

ConceptualPhysicalRoom ConceptualName PhysicalName Room-1 KITCHEN Room-2 LIVING ROOM Room-3 OFFICE

In this function, four consultations that are implicit in the database knowledge have been mentioned. One query corresponds to the need of asking for all of a specific object instances that has already been stored. Other query is raised from the need to know the physical room where a specific object is found. Other query is created to meet the need of knowing in which room a specific type of object can be found. Last query is created to meet the need of knowing in which room a specific type of object can be found.

(a) Content of the table PhysicalObject and ConceptualPhysicalRoom

ConceptualObjectRoom RoomName ObjectName KITCHEN STOVE KITCHEN REFRIGERATOR KITCHEN SINK KITCHEN WASHING MACHINE KITCHEN HEATER LIVING ROOM SOFA LIVING ROOM COFFEE TABLE LIVING ROOM CHAIR LIVING ROOM TV SET LIVING ROOM HEATER LIVING ROOM BOOKCASE DINING ROOM CHAIR DINING ROOM DINING TABLE DINING ROOM HEATER OFFICE HEATER OFFICE CHAIR OFFICE BOOKCASE OFFICE DESK OFFICE COMPUTER BATHROOM WASHBASIN BATHROOM TOILET BATHROOM HEATER

(b) Content of the table ConceptualObjectRoom

TABLE I: Content of the experiments tables

86

Fig. 4: Test sequence with the CHAIR object

A. Physical rooms This test is intended to analyze the behavior of the planner when a conceptual room is requested to be found, corresponding with a known physical room. The test consisted of asking for the KITCHEN in the case where the robot knows that there is a room that is a kitchen. In this case, the system issues a command to go to room-1, which corresponds to the kitchen. Checking the table PhysicalConceptualRoom, it can be seen that if a query is raised asking for KITCHEN, the result is room-1. If the user asks again for the kitchen, the system search for another one, not having more kitchens in the database, explore the rest of the rooms.

Fig. 5: Test sequence with the CHAIR object when there is no match with no observed physical object

B. Physical objects The test described in this section, evaluates the system performance when attempting to find a conceptual object where real objects have been observed. In the initial environment, there are three real objects of the CHAIR type. When asked for a Chair, the system identifies Chair-1, Chair-2 and Chair3. It gets the information of the room in which the first chair is found and it issues a command to move to that room, which corresponds to room-1. When asking again for a chair, Chair2 is obtained, and it stays at room-2. With another query on Chair, Chair-3 is obtained and it issues the command to move to room-3, which corresponds with the last known chair. This behavior is visually described in figure 4.

Fig. 6: Test sequence with the WASHBASIN object

If it is the case of unknown objects, when asked again for CHAIR after having discarded the chairs from the previous test, the operation sequence can be seen in figure 5. it had to find a soft drink, coffee or fresh water. With this, the problem is reduced to the previous cases of searching for an object (known or unknown). Later, looking for a hot drink have been requested and the system worked out that it must get coffee, being a case of search by object (coffee) already solved. With the knowledge shown in figure 7 and telling the system that something relaxing was searched for, the response was trying to find a chair.

One more query on chair that can be seen in figure 4 and it issues the command to explore for CHAIRS, searching in rooms where chairs could be present (classrooms, offices, dining rooms). C. Unknown objects in unknown rooms This test is carried out to ensure the correct operation in cases in which it has been introduced as semantic target a conceptual object where no real object has not observed a real object and that object is found in unidentified rooms. The object asked for is the WASHBASIN. In figure 6 the system tries to find bathrooms observed in the data base, asked by an user.

E. Objects search by semantic proximity. Finally, with the knowledge shown in figure 7 and telling the system that a printer was needed, the system deduced that it must go to the office (in a home environment) to search for the computer because a printer may be found. For that, the system used the information of the relations between objects (computer-printer).

D. Object search by characteristics. Based on the knowledge shown in Figure 7, the system is asked for a something to drink and the system deduced that

87

[2] H. Choset and K. Nagatani, “Topological simultaneous localization and mapping (slam): Toward exact localization without explicit localization,” IEEE Transactions on Robotics and Automation, vol. 17, pp. 125– 137, 2001. [3] N. Tomatis, I. Nourbakhsh, and R. Siegwart, “Hybrid simultaneous localization and map building: A natural integration of topological and metric, robotics and autonomous systems,” 2003. [4] I. Kostavelis and A. Gasteratos, “Semantic mapping for mobile robotics tasks: A survey,” Robotics and Autonomous Systems, vol. 66, pp. 86– 103, Apr. 2015. [5] T. Kollar and N. Roy, “Utilizing object-object and object-scene context when planning to find things,” in IEEE International Conference on Robotics and Automation (ICRA), 2009. [6] C. Schlenoff, E. Prestes, R. Madhavan, P. Goncalves, H. Li, S. Balakirsky, T. Kramer, and E. Miguelanez, “An IEEE standard ontology for robotics and automation,” in Intelligent Robots and Systems (IROS), pp. 1337–1342, IEEE, 2012. [7] A. Pronobis and P. Jensfelt., “Large-scale semantic mapping and reasoning with heterogeneous modalities,” in Proceedings of the 2012 IEEE International Conference on Robotics and Automation (ICRA’12), 2012. [8] C. Galindo, A. Saffiotti, S. Coradeschi, P. Buschka, J. A. FernndezMadrigal, and J.Gonzlez, “Multi-hierarchical semantic maps for mobile robotics,” in Proc. of IROS, 2005. [9] H. Zender, O. M. Mozos, G.-J. M. K. P. Jensfelt, and W. Burgard, “Conceptual spatial representations for indoor mobile robots,” Robotics and Autonomous Systems (RAS), vol. 56, pp. 493–502, 2008. [10] M. Cummins and P. M. Newman, “Highly scalable apearance-only slam- fab-map 2.0,” in Proc of RSS, 2009. [11] F. Dayoub, G. Cielniak, and T. Duckett, “A sparse hybrid map for vision-guided mobile robots,” in Proc. of ECMR’11, 2011. [12] A. Torralba, K. Murphy, W. Freeman, and M. Rubin, “Context-based vision system for place and object recognition,” in Proc. of ICCV’03, 2003. [13] J. Wu, H. I. Christensen, and J. M. Rehg, “Visual place categorization: problem, dataset, and algorithm,” in Proc. of IROS’09, 2009. [14] A. Ranganathan, “Pliss: Detecting and labeling places using online change-point detection,” in Proc. of RSS’10, 2010. [15] A. Pronobis, O. M. Mozos, B. Caputo, and P. Jensfelt, “Multi-modal semantic place classification,” IJRR, vol. 29, 2010. [16] A. Aydemir, M. Gbelbecker, A. Pronobis, K. Sj, and P. Jensfelt., “Planbased object search and exploration using semantic spatial knowledge in the real world,” in Proceedings of the 5th European Conference on Mobile Robots (ECMR’11), 2011. [17] S.Vasudevan and R. Siegwart, “Bayesian space conceptualization and place classification for semantic maps in mobile robotics,” Robotics and Autonomous Systems (RAS), vol. 56, pp. 522–537, 2008. [18] R. O. Duda, P. E. Hart, and D. G. Stork, “Pattern classification,” Wiley Interscience, 2000. [19] S. Vasudevan and R. Siegwart, “A bayesian conceptualization of space for mobile robots,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2007. [20] D. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, 2004. [21] S. Ekvall, D. Kragic, and P. Jensfelt, “Object detection and mapping for service robot task, robotica,” International Journal of information, Education and Research in Robotics and Artificial Intelligence, vol. 25, pp. 175–187, 2007. [22] S. Ekvall and D.Kragic, “Receptive field coocurrence histograms for object detection,” in IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2005. [23] D. Lpez, “Combining object recognition and metric mapping for spatial modeling with mobile robots,” Master’s thesis, Royal Institute of Technology, 2007. [24] C. Galindo, J.-A. Fernndez-Madrigal, J. Gonzlez, and A. Saffiotti, “Robot task planning using semantic maps,” in Robotics and Autonomous Sistems, 2008. [25] C. Galindo and A. Saffiotti, “Inferring robot goals from violations of semantic knowledge,” in Robotics and Autonomous Systems, 2013.

Fig. 7: Knowledge of the of the robot represented on tables.

F. Unknown objects in known rooms In this tests block the result of asking for conceptual objects that have not been observed but its location must be known, as it is inferred in the reasoning module, has been examined. It is checked that such inference is carried out and that the topological destination is right. It has been tried to ask for HEATER. The system issues the command to go to all rooms with a heater. None was identified and it issued explore. G. Unknown rooms The system has been asked for a conceptual room that has no correspondence with a physical room because it has not been identified as a room. The room asked for was the BEDROOM. The system explores to search for the bedroom. As no room was identified as a bedroom in the database, the reasoning module returns the result that launches the explorer module looking for a bedroom. VI.

C ONCLUSION

In Robotics there is a trend to adopt models with biological inspiration in many ways and each time with greater intention to imitate human abilities. Among these abilities, the power to think symbolically, to extract semantic information from the environment and to reason with it to plan their actions is a goal to reach in Robotics with the use of cognitive architectures that require a semantic modeling. Systems perform a great autonomy and robustness unthinkable without a way of treating the contextual semantic information of the environment. It has also been described that an optimal and efficient option is a planner that uses a relational database for semantic navigation. The process is fast and easy to implement. The proposed design enables the combination of different types of rooms (living room, kitchen, etc) being a semantic knowledge more flexible and adaptable than what exists in other works where changes are required in semantics and in the rules, when adding new objects in the conceptual hierarchy or when identifying a hybrid room. Furthermore, an efficient search of objects is achieved. VII.

ACKNOWLEDGEMENTS

The authors would like to thank the RoboCity2030-II project (S2009/DPI-1559), funded by Programas de Actividades I+D en la Comunidad de Madrid and cofunded by Structural Funds of the EU. R EFERENCES [1]

K. Arras, Featur-based robot navigation in know and unknow environments. PhD thesis, Swiss Federal Institute of Technology Lausanne (EPFL), 2003.

88