A Knowledge-based Approach to Behavior Decision in Intelligent ...

A Knowledge-based Approach to Behavior Decision in Intelligent Vehicles Andreas D. Lattner, Jan D. Gehrke, Ingo J. Timm, and Otthein Herzog Center for Computing Technologies - TZI Universitaet Bremen, PO Box 330 440, D-28334 Bremen, Germany {adl|jgehrke|i.timm|herzog}@tzi.de

Abstract— Recent advances in the field of intelligent vehicles have shown that it is possible nowadays to provide the driver with useful assistance systems, or even letting a car drive autonomously over long distances on highways. Usually these approaches are on a rather quantitative level. A knowledgebased approach as presented here has the advantage of a better comprehensibility and allows for formulating and using common sense knowledge and traffic rules while reasoning. In our approach a knowledge base is the central component for higherlevel functionality. A qualitative mapping module abstracts from the quantitative data and stores symbolic facts in the knowledge base. The knowledge-based approach allows for easily integrating and adjusting background knowledge. Higher-level modules can query the knowledge base in order to evaluate the situation and decide what actions to perform. For the evaluation of the approach a prototype was developed in order to simulate traffic scenarios. In experiments behavior decision was applied for controlling the vehicle and its gaze.

I. I NTRODUCTION Recent advances in the field of intelligent vehicles have shown that it is possible nowadays to provide the driver with useful assistance systems like, e.g., lane departure warning, lane change assistants, adaptive cruise control, or even letting a car drive autonomously over long distances on highways [3], [4], [7], [18], [22]. Usually these approaches are on a rather quantitative level. A knowledge-based approach as presented in this paper has the advantage of a better comprehensibility and allows for formulating and using common sense knowledge and traffic rules while reasoning. We claim that a knowledge-based representation is not only needed in order to increase traceability and user acceptance w.r.t. intelligent vehicles but even mandatory to handle complex situations where background knowledge is needed to cope with traffic situations like those in cities. Some approaches addressing higher-level functionality like situation assessment, mission planning, and plan recognition use symbolic representations. Pellkofer et al. [17] perform a situation assessment in order to decide the behavior of the intelligent vehicle based on explicitly described capabilities. In their approach the situation assessment computes symbolic meanings by “situation aspects” [17, p. 497]. Among other things the name, values (fuzzy, degree of fulfillment), and the period of validity is described here.

Mock-Hecker [16] developed a methodology for recognizing critical traffic situations. He defined a symbolic representation for plans of traffic participants in order to describe their behavior based on STRIPS [9]. On this basis a conflict detection is performed. Possible conflicts are identified on a qualitative level and – if necessary – evaluated on a quantitative level where time is also taken into account. Dagli et al. [5], [6] address the prediction of actions of traffic participants with dynamic Bayesian networks. Their approach is based on motivations of actors, e.g., for changing the lane in order to keep a desired speed. With this approach lane change maneuvers could be recognized about 1.5 s before the actual maneuver. Schlenoff et al. [21] predict future positions of moving objects by assigning probabilities to discrete actions of vehicles. The probabilities of actions can be biased by weather, intentions, vehicle type etc. In our approach a knowledge base (KB) is the central component for higher-level functionality. Similarly to the work presented in [14] we perform a mapping from quantitative to qualitative data. A qualitative mapping module abstracts from the quantitative data and stores symbolic facts in the KB. The temporal dimension is represented by time intervals where certain facts hold. The knowledge-based approach allows for easily integrating and adjusting background knowledge. The higher-level modules can query the KB in order to evaluate the situation and decide what actions to perform. The paper is organized as follows: Section II gives an overview of the overall architecture. The knowledge-based situation assessment and behavior control is presented in Section III. The experimental evaluation is described in Section IV. The paper closes with conclusions in Section V. II. A RCHITECTURE The architecture presented here is a refinement of Dickmanns’ architecture [8] (see Fig. 1). Dickmanns’ architecture consists of different components in order to realize various cognitive functions in an intelligent vehicle. Situation assessment, behavior decision, and mission planning have different characteristics w.r.t. time criticality. While mission planning can be done in rather long cycles, situation assessment must be able to identify risks quickly.

Fig. 1.

Architecture adapted from Dickmanns

The situation assessment analyzes the situation perceived by the sensory information and recognizes situation patterns. Some decisions have to be made already at this level based on the current situation if they need a quick intervention to the maneuver control. The central component here is the qualitative representation of the information perceived by the sensors, which is mapped into a qualitative abstraction. The current situation can be evaluated by considering matching situation and behavior patterns. If a dangerous situation is identified a direct interaction with the vehicle control might be necessary in order to avoid a collision with some obstacle. Having modeled the current situation and possible actions of traffic participants future world states can be computed (“possible worlds”). This allows for identifying risks in the near future. The main task of the behavior decision is to create a shortterm plan about the next actions of the vehicle. Here, the next mission goal (from the mission planning component) and the current situation must be taken into account. This also includes the plans of other traffic participants which might interfere with the own plan. Mission planning is on the most abstract and least critical (w.r.t. time) level. Here, decisions are made on a very high level, e.g., for basic strategies (e.g., economic vs. time-saving driving) or route planning. The selected goals are passed to the behavior decision component where planning is performed on a more detailed level. III. K NOWLEDGE - BASED S ITUATION A SSESSMENT AND B EHAVIOR D ECISION This section introduces our knowledge-based approach to behavior decision. The following subsections describe the qualitative scene representation, the qualitative mapping, the definition of situation and risk patterns, and their application for behavior decision. A. Qualitative Scene Representation and Mapping We propose a symbolic representation of the world within the higher layers of the architecture. The quantitative data originating from the sensors is mapped onto a qualitative

abstraction. This builds the basis for higher level tasks like situation assessment, planning, and behavior control. In order to create such a representation, time series of different measures are divided into intervals by monotonicity-based or threshold-based segmentation algorithms [14]. This leads to time intervals for different properties of objects or object pairs, e.g., intervals where the distance between two objects decreases monotonically, or intervals where the velocity of an object can be described by a qualitative class (e.g., “high speed”). Using such an abstraction spatiotemporal patterns can be described in a human comprehensible way, e.g., “A approaches B” if the distance between those two objects decreases monotonically. The temporal dimension is represented by time intervals. This representation can be easily mapped to a qualitative representation like Allen’s interval logic [1]. In order to create such a qualitative scene representation qualitative mapping modules have to observe different quantitative time series or have to obtain information from other sources (e.g., object classification modules). The vocabulary for the scene description must be defined and the background knowledge must be modeled in advance. The qualitative mapping is done cyclically. During each cycle an update of the KB is performed, i.e., new facts are inserted into the KB or existing intervals are extended (if a relation is still valid). In order to find out if an existing interval must be extended the KB is queried if there exists such relation directly before the current mapping interval. The performance of the mapping cycles is crucial as it must be fast enough to provide information to the higher level components in order to allow the intelligent vehicle to act in time. Our knowledge base allows for storing relevant information for describing traffic situations, e.g.: • Object classes: This includes classes (e.g., truck), class properties (like capabilities or special traffic rules for a class), class hierarchy, and the assignment of instances to classes. • Topological information: In order to represent the position of dynamic objects in relation to the ground regions RCC-5 is used in this approach [2]. With these relations it can be represented whether an object region is disconnected, overlapping, inside, part of, or (spatially) equal to a ground region. • Spatial relations: It is useful to use other spatial representation in order to represent relations between objects, e.g., before/behind relations from an egocentric point of view. • Speed information: Information about speed and acceleration of single objects (e.g., high speed, decelerating). In this domain lateral speed, acceleration and position (w.r.t. to a lane) are also important. • Distance information: Distance classes and the development of distances between object pairs (e.g., close, approaching; cf. [15]). • Road network: Information about roads, lanes, junctions, lane regions and their connectivity. This is needed in order to know which region transfers are possible and

allowed. Traffic situation: Different information about signals, signs and traffic rules (possibly assigned to lanes and thus to vehicles on these lanes). • Background knowledge: Here, different rules can be set up in order to allow for deriving information from atomic facts, e.g., the definition of a one-way street (if all neighboring lanes just allow for driving in the same direction). The qualitative mapping leads to symbolic descriptions of dynamic scenes. The following example illustrates the representation: •

holds(rcc_part(iv, ivLane), 5, 30). holds(fast_speed(iv), 10, 30). meetsJunction(ivLane, junction1). holds(maxDist(close, iv, junction1), 25, 30).

The first relation specifies that the object region of the intelligent vehicle (“iv”) is part of the region of a lane (“ivLane”). The speed of the vehicle is fast and it is close to junction “junction1”. The lane “ivLane” meets the junction “junction1”. The last two parameters represent the temporal validity of the relation. Static information (as road connectivity) is independent of time, thus no start and end time points are defined. B. Situation and Risk Patterns Patterns are abstract descriptions of situations where certain conditions hold (cf. [11]). The patterns are based on the qualitative representation in the knowledge base. Complex patterns can be composed of the different basic predicates. Allen’s temporal relations between these predicates can be used for a more concise definition of patterns. Risk patterns extend the pattern description by the definition of risk variables and corresponding risk values [12]. An example for a pattern which finds dynamic objects in medium distance ahead of the intelligent vehicle is: maxDistHolds(medium_distance, Actor, iv, S, E), relationHolds(ahead, Actor, iv, S, E), isMemberOfClass(Actor, dynamic_object)

Here, “Actor” denotes a variable for a dynamic object. The variable “S” and “E” represent the start and end time points. C. Situation Assessment and Behavior Decision Different pattern matching modules observe the KB and notify their initiators if certain patterns are detected. In each matching cycle all patterns are evaluated and all valid assignments are derived by an inference engine. An assignment maps constants (i.e., objects) to variables. The inference engine just returns valid assignments w.r.t. the defined pattern and its temporal interrelations. For situation assessment different situation patterns must be defined. These patterns are checked at regular intervals. If a situation pattern holds, the derived information can be stored in the KB in order to allow the behavior decision module to select an appropriate behavior. Behavior decision can be realized similarly to situation assessment. Based on the initial and derived facts through

Fig. 2.

Architecture of the ASKOF prototype

situation assessment, behavior decision patterns determine which actions can be performed. In order to select the behavior the expected outcome of certain actions has to be evaluated. Here an evaluation function defines how to rate these possible future states. The action (or plan) which leads to the most promising situation is selected. IV. E XPERIMENTAL E VALUATION For the evaluation of our approach a prototype was developed. The architecture is shown in Fig. 2. The ASKOF demonstrator allows for setting up and simulating traffic situations with different regions (like roads, lanes, crossings) and dynamic objects (traffic participants). The movements in the simulation are mapped onto the qualitative representation which is shown in Section III. For the qualitative mapping different mapping modules were realized, e.g.: • SpeedMapper: Creates intervals with different speed classes (e.g., high speed) and the development of the speed (e.g., acceleration). • DistanceMapper: Creates intervals with different distance classes (e.g., far distance) and the development of the distance (e.g., approaching). • ClassMapper: Assigns the recognized class(es) to objects in the dynamic scene. • TopologyMapper: Creates the topological information between objects and ground regions. • RelativeDirectionMapper: Assigns egocentric direction information relative to the direction of an object (e.g., before/behind). • LateralPositionMapper: Creates intervals with different lateral positions within a lane (e.g., left, center, right) and the lateral movement (e.g., moving left, moving right). The different mapping modules create and update facts representing the belief about the world and assert them to the KB. For the ASKOF prototype we decided to use F-Logic as the representation language because of its representational power [10]. As implementation we used Flora-2 [13], [23] which is based on XSB [19], [20]. In order to test the feasibility two modules which use the qualitative representation have been developed: A behavior

Fig. 3.

Stopping - Oncoming traffic

decision for gaze control based on risk assessment and a simple behavior decision for locomotion control. For an evaluation the approach was tested in six different scenarios. In the basic setting of the scenarios there are four dynamic objects in the scene: the own vehicle, a bus, and two children. In the simulation the own vehicle has a camera attached to the car body which can be directed to focus different objects. In all scenarios the own vehicle and the bus approach the same crossing. It has to be identified if the autonomous vehicle must stop in order to grant the bus the right of way, or if the own way can be kept without stopping. For behavior decision for gaze control different sample risk patterns have been defined. In the risk assessment cycles the KB is queried with the risk patterns, all results are collected, and corresponding risk values are assigned to the risky objects. The gaze control is directed to the different objects identified as dangerous by applying a scheduling algorithm. (cf. [12]) For locomotion control situation patterns were defined which evaluate if a crossing is blocked or if another vehicle which approaches the crossing has the right of way. These patterns were used by a simple behavior decision customized to the scenarios. Here, the car is stopped before the crossing if the crossing is blocked or if another vehicle is approaching the crossing. The pattern which identifies if another vehicle has the right of way is: relationHolds(rcc_part, iv, IvLane, S, E), meetsJunction(IvLane, NextJunction), maxDistHolds(medium_distance, iv, Veh, S, E), relationHolds(decreasing_distance, iv, Veh, S, E), (relationHolds(rcc_part, Veh, VhLane, S, E); relationHolds(rcc_partial_overlap, Veh, VhLane, S, E) ), leadsToJunction(VhLane, NextJunction), not( (relationHolds(rcc_partial_overlap, Veh, OtherReg, S, E), OtherReg \= NextJunction, not leadsToJunction(OtherReg, NextJunction) ) )

It matches if the intelligent vehicle (IV) is in a lane that meets a junction and if another vehicle (“Veh”) is on a different lane that also leads to the junction. Additionally, the distance between the IV and the other vehicle should be less or equal to “medium distance” and decreasing. In order to check if the other vehicle has not entered the lay-by further conditions state that the other vehicle has not entered another region which does not lead to the junction. The experiments show that risky objects are identified correctly and that the gaze control is directed towards these objects. For locomotion control the autonomous car is stopped correctly if it is not allowed to enter the crossing due to oncoming traffic or a blocked crossing. Figure 3 shows a situation where the vehicle was stopped because of oncoming traffic. In the situation shown in Figure 4 the vehicle can proceed driving because the bus entered the lay-by. In order to evaluate the approach different measures were extracted from log files created by the prototype. Each of the six scenarios was repeated three times. The average duration for an assert command to the KB is 0.66 ms. A query takes 1.55 ms on the average1 . Table I shows the average duration and number of commands for the different mapping modules in the basic settings. The average duration for the pattern matching cycles with the same settings are shown in Table II. The time complexity was evaluated by increasing the number of objects. Fig. 5 shows the duration of the mapping and matching cycles with a growing number of objects. As it can be seen the matching cycles stay quite constant but the mapping cycles show approximately quadratic growth. Fig. 6 shows exemplarily the development of the mapping cycle duration for five and for ten objects for one run. In the 1 Experiments were run on a system with SuSE Linux 9.0 / 64 Bit with four 1.4 GHz AMD Opteron processors and 8 GB RAM. Due to the sequential processing of queries and commands within Flora-2/XSB the parallel computation power could not be used.

Fig. 4.

Turning left - No oncoming traffic

case of five objects after some initial expense the durations stay quite constant. If ten objects are processed it can be seen that duration grows rapidly after some cycles. The reason for this growth is that in the current implementation all values of time series are processed even if the mapping cycle takes too much time (due to the large number of relations between objects to map). If mapping takes too much time in the next step more values have to be processed by the qualitative abstraction. This problem can be addressed by reducing the number of objects or skipping values during qualitative mapping. V. C ONCLUSION In this paper we presented a knowledge-based approach to behavior decision. This approach presumes a qualitative scene representation which has to be created by a qualitative mapping. Patterns are defined by combining different predicates from the KB and possibly setting them into temporal interrelations. Relevant information about the scenes are stored in a KB. The KB is queried in order to assess the situation and to identify risky objects. For non-standard or complex traffic situations (e.g., in cities) it is usually hard to formulate all possible aspects relevant for situation assessment. A knowledge-based approach allows for realizing abstract rules and background knowledge and to use an inference engine in order to evaluate a situation. The experimental evaluation on simulated traffic scenes shows the feasibility of our approach. An analysis of the

Distance Speed Topology RelDirection Class

ACKNOWLEDGMENT The content of this paper is a partial result of the ASKOF project which was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant HE989/6-1. We would like to thank our student research assistants Jan Gerken and Hanna Bauerdick for many interesting discussions and ideas and their effort in the implementation of the ASKOF prototype. We also want to thank Matthias Goebl and Sebastian Dr¨ossler from the Institute for Realtime Computer Systems (RCS) at TU M¨unchen for their cooperation as members of the ASKOF project, especially for their support in designing the evaluation scenarios. TABLE II AVERAGE DURATION OF THE MATCHING CYCLES

TABLE I AVERAGE DURATION

time complexity with a growing number of objects point out difficulties during the mapping if the number of objects is large, but usually it is not necessary to take all objects in the environment into account. Thus, if the qualitative mapping can be reduced to relevant objects it is possible to handle the complexity, and usually there are less than ten relevant objects. In future work we would like to address the learning of behavior patterns and their application in order to create adaptive intelligent agents. It has to be investigated to what degree the performance of agents can be increased by learning and applying such patterns.

AND NO . OF COMMANDS OF THE MAPPING CYCLES

Cycle duration in ms 18.80 ± 08.13 6.35 ± 09.20 4.74 ± 03.26 8.30 ± 03.14 0.17 ± 01.85

#Commands/Cycle 23.03 ± 4.37 8.07 ± 0.70 2.67 ± 1.37 12.00 ± 0.05 0.02 ± 0.30

Child close, low speed Dyn. object in medium distance Stop - oncoming traffic Unknown object Child close, medium speed Child close, high speed Stop - crossing blocked

Cycle duration in ms 1.62 ± 04.45 3.16 ± 09.72 4.82 ± 14.00 1.20 ± 03.51 1.51 ± 01.29 1.67 ± 01.81 2.17 ± 05.41

Development of cycle duration

600

1000

Growing number of objects

800 600

Cycle duration in ms

400

500 400 300

0

0

100

200

200

Duration in ms

05 objects 10 objects

Mapping cycles Matching cycles

5

6

7

8

9

10

0

Number of objects

Fig. 5.

Duration of cycles at growing number of objects

R EFERENCES [1] J. F. Allen. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843, 1983. [2] B. Bennett. Spatial reasoning with propositional logics. In J. Doyle, E. Sandewall, and P. Torasso, editors, Principles of Knowledge Representation and Reasoning: Proceedings of the 4th International Conference (KR94), San Francisco, CA., 1994. Morgan Kaufmann. [3] M. Bertozzi, A. Broggi, and A. Fascioli. Vision-based intelligent vehicles: State of the art and perspectives. Journal of Robotics and Autonomous Systems, 32(1):1–16, June 2000. [4] I. Dagli, G. Breuel, H. Schittenhelm, and A. Schanz. Cutting-in vehicle recognition for ACC systems - towards feasible situation analysis methodologies. In Proceedings of the IEEE Intelligent Vehicles Symposium, pages 925–930, Parma, Italy, June 14-17 2004. [5] I. Dagli, M. Brost, and G. Breuel. Action recognition and prediction for driver assistance systems using dynamic belief networks. In Proceedings of the Conference on Agent Technologies, Infrastructures, Tools, and Applications for E-Services, pages 179–194, 2002. [6] I. Dagli and D. Reichardt. Motivation-based approach to behaviour prediction. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV’2002), pages 227–233, Versailles, 2002. [7] E. D. Dickmanns. The development of machine vision for road vehicles in the last decade. In Proceedings of the IEEE Intelligent Vehicles Symposium 2002 (IV’2002), pages 268–281, Versailles, France, June 17-21 2002. [8] E. D. Dickmanns. An advanced vision system for ground vehicles. In Proceedings of 1st International Workshop on In-Vehicle Cognitive Computer Vision Systems (IVCCVS), pages 1–12, Graz, Austria, April 3 2003. [9] R. E. Fikes and N. J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3– 4):189–208, 1971. [10] M. Kifer, G. Lausen, and J. Wu. Logical foundations of object-oriented and frame-based languages. Journal of the ACM, 42(4):741–843, July 1995. [11] A. D. Lattner and O. Herzog. Unsupervised learning of sequential patterns. In ICDM 2004 Workshop on Temporal Data Mining: Algorithms, Theory and Applications (TDM’04), Brighton, UK, November 1st 2004. [12] A. D. Lattner, I. J. Timm, M. Lorenz, and O. Herzog. Knowledgebased risk assessment for intelligent vehicles. In Proceedings of the International Conference on Integration of Knowledge Intensive Multi-Agent Systems KIMAS ’05: Modeling, Evolution and Engineering, Waltham, Massachusetts, USA, April 18 - 21 2005. To appear.

5

10

15

20

25

30

Mapping cycle

Fig. 6.

Development of mapping cycle duration over time

[13] B. Lud¨ascher, G. Yang, and M. Kifer. Flora: The secret of object-oriented logic programming. Technical report, SUNY at Stony Brook, July 13 2001. [14] A. Miene, A. D. Lattner, U. Visser, and O. Herzog. Dynamic-preserving qualitative motion description for intelligent vehicles. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV ’04), pages 642–646, Parma, Italy, June 14-17 2004. [15] A. Miene, U. Visser, and O. Herzog. Recognition and prediction of motion situations based on a qualitative motion description. In D. Polani, B. Browning, A. Bonarini, and K. Yoshida, editors, RoboCup 2003: Robot Soccer World Cup VII, volume 3020 of Lecture Notes in Computer Science, pages 77–88. Springer, 2004. [16] R. Mock-Hecker. Wissensbasierte Erkennung kritischer Verkehrssituationen - Erkennung von Plankonflikten. VDI-Verlag, D¨usseldorf, 1994. [17] M. Pellkofer and E. D. Dickmanns. Behavior decision in autonomous vehicles. In Proceedings of the IEEE Intelligent Vehicles Symposium, pages 495–500, Versaille, France, June 17-21 2002. [18] M. R¨uder, W. Enkelmann, and R. Garnitz. Highway lane change assistant. In Proceedings of the IEEE Intelligent Vehicles Symposium, pages 240–244, Versaille, France, June 17-21 2002. [19] K. Sagonas, T. Swift, and D. S. Warren. XSB as an efficient deductive database engine. In Proceedings of the ACM SIGMOD International Conference on the Management of Data, pages 442–453, Minneapolis, Minnesota, May 1994. [20] K. Sagonas, T. Swift, D. S. Warren, J. Freire, P. Rao, B. Cui, E. Johnson, L. de Castro, S. Dawson, and M. Kifer. The XSB system version 2.5 - volume 1: Programmer’s manual, volume 2: Libraries, interfaces and packages. http://xsb.sourceforge.net/manual1/manual1.pdf, http://xsb.sourceforge.net/manual2/manual2.pdf, June 27 2003. [21] C. Schlenoff, R. Madhavan, and S. Balakirsky. An approach to predicting the location of moving objects during on-road navigation. In Proceedings of the Workshop on Issues in Designing Physical Agents for Dynamic Real-Time Environments: World modeling, planning, learning, an communicating at the International Joint Conference on Artificial Intelligence (IJCAI-03), pages 71–79, 2003. [22] K. Weiss, N. Kaempchen, and A. Kirchner. Multiple-model tracking for the detection of lane change maneuvers. In Proceedings of the IEEE Intelligent Vehicles Symposium, pages 937–942, Parma, Italy, June 14-17 2004. [23] G. Yang, M. Kifer, and C. Zhao. Flora-2: User’s Manual - Version 0.92. Department of Computer Science, Stony Brook University, June 28 2003.