A Perception Strategy For A Surveillance System Claude Barrouil, Charles Castel, Patrick Fabiani, Roger Mampey, Patrick Secchi, Catherine Tessier 1 f*
[email protected]
Abstract. This paper presents the principle of a perception system meant for delivering alarms when dreaded activities are suspected. It includes a Symbolic Layer (SL) that deals with activities recognition and alarm decision, and a Resource Management System (RML) that, upon request, provides SL with information. The domain to be watched over includes several areas that cannot be checked at the same time because of the limited capabilities of the resources, and the crucial problem is to optimize the use of the resources. It is shown how sensors are dynamically allocated to limited parts of the scene and how it is decided between immediate alarm or additional information request without explicit situation assessment. The principle of an implementation within a real time multi-task architecture is also presented. keywords : perception, belief revision, robotics.
NL
The perception 2research program aims to design a surveil-
lance system that must warn when dreaded activities (DA) are suspected to occur in a dynamic environment. The applications are sensible areas surveillance as well as mission execution monitoring for autonomous robots. Zones where cars may appear pedestrian
Scene where dreaded activities have to be detected
The scene on gure 1 illustrates typical applications. The system uses sensors to collect data which are processed in order to detect that possible dreaded activities (DA) are going on, and then to send an alarm to human operators. 1 ONERA/DCSD/CD: Oce National d'E tudes et de Recherches Aerospatiales, Departement de Commande des Systemes et de Dynamique du vol, unite de recherches "Conduite et Decision", 2 avenue E.Belin, BP4025, F-31055 Toulouse cedex 4, France http://www.cert.fr/fr/dcsd/CD/CDPUB 2 http://www.cert.fr/fr/dcsd/PUB/PERCEPTION
c 1998 C.Barrouil & al. ECAI 98. 13th European Conference on Arti cial Intelligence Edited by Henri Prade Published in 1998 by John Wiley & Sons, Ltd.
sensors + signal processing
real world
DA
ME RML Figure 2.
1 Overview
Figure 1.
Activities (such as DA) are described by a temporal organization of conditions (CD). CDs are properties that are de ned on the world at a given time or within a given time interval. An example of activity in a car park context is \vehicle departure". An example of CD is \the pedestrian is motionless and close to a car". An example of DA is \vehicle robbery".
SL
alert !
Surveillance system organization.
The system is organized as follows: a symbolic layer (SL) matches DA description prototypes [5] with CD assessments delivered by a Resource Management Layer (RML). RML generates this information by processing data from the Numerical Layer (NL), which includes sensors and associated signal processing algorithms such as classi cation and tracking. An important hypothesis is that NL has limited capabilities with respect to the possible number of objects of interest and their dynamics. Therefore, it is necessary to optimize the scene observation strategy: which zone to check ? which object to track ? which condition to assess? The observation strategy is twofold: SL optimizes CD assessment requests addressed to RML, RML optimizes observation requests addressed to NL. This paper aims to present how RML and SL cooperate in order to achieve an overall ecient surveillance.
2 Resource Management
The resources to manage are sensors, such as color CCD cameras, infrared cameras and radars, with associated signal processing algorithms. A request to NL should specify i) the device to operate, ii) the observation direction and iii) the window in the sensor image where signal processing should be performed. In the perception project, NL returns phenomena (PH) that indicate changes within groups of neighboring pixels. Changes are detected by comparing 3 consecutive images. A PH has several attributes as: position in the speci ed window, degree of conformity with prede ned shapes ("car",
"lorry", "pedestrian"). NL does not detect motionless objects: when there is no image change, NL returns a particular report. RML controls NL in order to insure: detection of new objects: the places where new objects might appear are well known parts of the scene called scene entrances, tracking of objects of interest: localization updating and motionlessness or disappearance detection, CD assessment in order to reply to SL requests such as: Is it a motionless car in the direction where the pedestrian is walking to? The rst two items are merged, as tracking an object consists in nding a PH in a window centered on this object, in the same way as for the scene entrances. We call a surveillance task either a zone to be watched over or an object to be tracked. We call the tasks target either this zone or this object. Because of the inaccuracy on numerical PH attributes and of uncertainty on the symbolic ones, PH data cannot be used as they are (see [3] for related work on uncertainty in the project PERCEPTION).
2.1 Allocating sensors to surveillance tasks
NL-provided PHs are meant for discovering new moving objects and for updating beliefs about the known ones. [9, 10] present an approach for modeling belief erosion i.e. the propensity for properties beliefs to decrease when no new information allows updating to be performed. We consider that the dynamics of belief erosion is domain dependent: it depends only on the type of the target and property. In the example at hand, the basic properties are the location, the velocity and the motion direction for moving objects and the absence of PHs for zones. The belief about the target type (they can be \zone", \pedestrian", \car" or \lorry") does not change when time passes without updating. For every surveillance task, we de ne three parameters: the critical time Tc , the critical velocity Vc and the importance kc . Tc and Vc only depend on the type of the target: assume t0 denotes the last time a sensor was allocated to the task, and that, at time t0 the estimated target velocity was v0 (always 0 for a zone), then at time t (t > t0 ) the task criticity is given by the empirical formula: c = kc t ?T t0 (1 + Vv0 ) (1) c c When the beliefs upon the properties of a target are updated, the corresponding task criticity is reset to 0. When a target has been left without checking for a duration Tc , the criticity is equal to kc if the target was assessed to be motionless at time t0 . Otherwise, the criticity raises linearly with time and velocity. By default kc = 1. This value increases when SL requests information concerning the target and, concerning zones, when PHs are detected and not matched with existing objects. Equation 2 gives Tc and Vc for the possible object types: zone pedestrian car lorry Tc (s) 10 2 20 20 Vc (m=s) 1(*) 1 7 7
Robotics, Vision, and Signal Understanding
(2)
628
(*) this value has not importance as zone velocity is 0. When a perception resource is idle, RML evaluates the criticity for all the current surveillance tasks, and decides to check the most critical one. An internal database, called Environment Model delivers the window in the scene where the object is most likely be found at the current time, so that a request can be formulated to NL.
2.2 The Environment Model
The Environment Model (ME) is a database that manages beliefs about object motions. Any object is associated with a type and with a track that represents the succession of PHs that correspond to it. ME insures the following functions to be performed: the explanation of new PHs by matching with current object tracks. This matching, based on classic bayesian updating [1], uses models of the object dynamics that allow the matching likelihood to be assessed. When a matching is found, the track is updated. Like the critical parameters, these models depends on the object types. the creation of new tracks when no current track can explain a new PH with a sucient likelihood. In principle such PHs can be collected only at the scene entrances. when several PHs with the same type are matched to a new track, ME creates a new object for SL information and further tracking. the answers to RML requests about the object motions at a given time. If this time is greater than the most recent track updating date concerning this object, the dynamics model is used for predicting the motion attributes.
2.3 Condition assessment
RML answers SL requests such as: At time t is there a motionless car in the direction where the pedestrian P4 is likely to walk to? Neither NL nor ME can answer this question. RML has knowledge about how to organize requests to ME in order to evaluate this sort of condition. We chose to express this knowledge in logic programming (and RML is connected with a PROLOG interpeter for using it). SL formulates requests in PROLOG syntax. Example: time(t) ^ car(c) ^ motionless(c) ^ direction(P4,d) ^ relative-position(P4,c,d) ^ answer(c) ; The predicate time declares the reference date, direction asserts that d denotes P4's motion direction, car says that c is one of the known objects of type \car", motionless is selfexplicative, relative-position expresses that the relative position of P4 and c is compatible with direction d, and answer indicates that SL wants a reply that indicates the name of the car if the evaluation succeeds. For evaluating direction and motionless RML requests ME. Predicate car scans the list of objects of type \car" provided by ME when evaluating the predicate all-cars: car(c) ! all-cars(l) ^ element(c,l) ; When requesting ME, RML uses the time reference declared by time.
C.Barrouil & al.
Note that RML never requests NL during a condition evaluation. The multi-object surveillance allocation strategy is meant for providing ME with data independently of the current CD evaluation, and RML answers SL requests by using the current ME records as there are when the request is received. RML returns true or fail to SL, with possibly variable bindings in the rst case. As we use logic programming, true means that the CD is evaluated to true, but fail only means that RML failed to prove CD to be true. A more ecient way to assess CDs is to compute con dence degrees for true, which is under study.
3 A Strategy For Requesting Condition Assessments When some DA is partially recognized, SL may either i) sends an alarm, ii) give up with this DA, or iii) request RML for additional information. \Alarm" and \GiveUp" are called terminal decisions. We do not consider the problem of the most pertinent condition to assess; we focus on the problem of deciding to request RML before deciding on a particular terminal decision, or not. If SL asks RML for more information, the current DA likelihood will be increased, but time will have passed and some surveillance task(s) importance kc will also have increased. Then, does requesting RML bring a real enhancement as the nal decision is delayed and the sensors allocation is disturbed? It is common to consider problems of scene recognition, diagnosis and, more generally, belief updating [6] independantly of the end-user's needs. One reason is that it is tempting to apply the obvious rule (R):
8 >< if the pedestrian looks like a thief (R) then send an alarm >: else do nothing
cost GiveUp 70
minmax solution: freq(Alarm)=0.7 freq(GiveUp)=0.3
Alarm 30 21
V not−a−thief Figure 3.
The problem with (R) is that it is never certain whether the pedestrian looks like a thief, or not. We prefer to consider that DA recognition is meant for delivering pertinent alarms, and that the SL strategy for requesting RML should optimize a compromise between an unadequate immediate terminal decision and a sensor allocation disturbance. The Game Theory [11], especially the rectangular zero-sum case, is a nice formalism for such decision problems. It provides the so-called "minimax solution" to make a decision even when there is no knowledge at all about the state of the world, instead of using the dubious maximum entropy principle often used in classic decision theory [14, 16]. Classically, probabilities are used for representing uncertainty, but other formalisms, as possibilities [8, 7] or the evidence theory [15], allow partial ignorance to be represented and may also be used in decision making. Assume that a pedestrian was detected and that SL suspects he is trying to steal a car (DA). Without RML request possibility, SL should decide between "Alarm" and "GiveUp". Assume that, depending on the ac-
Robotics, Vision, and Signal Understanding
tual pedestrian's intention, the costs are: NOT a thief ACTUAL thief "Alarm" 30 0 (3) "GiveUp" 0 70 If a knowledge exists and is expressed by the probability V that the pedestrian is a thief, the cost expectation is 70V with "GiveUp" and is 30(1 ? V ) with "Alarm". So "Alarm" and "GiveUp"are associated with lines in diagram 3. Unfortunately, such knowledge as V seldom exists. Choosing randomly between "Alarm" and "GiveUp" with relative frequencies (p; 1 ? p) leads to a cost of 30p when the pedestrian is not-a-thief, and of 70(1 ? p) when he is a thief. The expectation is equal to 21 with the minimax value p = 0:7 in both cases, whatever V ; this can be considered as the optimal behavior in case there is no prior information about the pedestrian's intention. We note that this strategy cannot be found if we want to bet for a particular hypothesis about the pedestrian's intention before making the decision. This shows that, at least in some particular cases, given an uncertain knowledge V of the state of the world, there might be no hypothesis hj such that the optimal decision corresponding to hj is the optimal decision given V . Therefore, it might be an awkward approach to elect a certain most likely hypothesis h^ j from V and behave as if h^ j held.
629
0.3
thief
Uncertainty and Cost aggregation diagram
The initial set of terminal decisions is enlarged with new decisions, called strategies, when preliminary information collection is possible. An example of such strategies is S1 :
8 >> request RML < RML says the pedestrian is motionless S1 > if then Alarm, >: else GiveUp
Note that this strategy di ers from the simple rule (R) since it refers to something which can be known (the answers returned by RML) and not to the real state of the world that will never be known. This di erence makes that it is easier to answer the question is it necessary to request RML?
3.1 Quoting strategies
Quoting strategies requires the relationship between the state of the world and the possible information data to be described.
C.Barrouil & al.
For instance, with the example at hand, we assume that when the pedestrian is close to a car the relative frequency that RML considers for the property "the pedestrian is motionless" to be true depends on the fact that the pedestrian is a thief or not: NOT a thief ACTUAL thief (4) freq(motionless) 0:3 0:8 Note that these values depend not only on the thief's and honest pedestrian's behaviors, but also on the sensors and signal processing performance. Then, in the case of a not-a-thief, strategy S1 above leads to an alarm with the relative frequency 0:3, and to tracking renunciation with the complementary relative frequency 0:7. As table 3 gives the situation costs, S1 may be quoted with the expectation 0:3 30 + 0:7 0 = 9 when not-a-thief. Similarly, when a thief, S1 may be quoted with the expectation 0:8 0 + 0:2 70 = 14.
cost GiveUp 70
Alarm 30
14 12
S1
9 0.14
V
0.6
thief
not−a−thief Figure 4.
Mixing S1 with initial decisions
However, when the number of requests to RML is not limited, it can be shown (Wald tests [17, 12, 2]) that there exist two limits v and v" such that the optimal strategy S^ is recursive and uses belief updating: 0
8 >> while v < V < v" >> do : request RML, >< update beliefs V S^ > apply strategy S^. if V v >> "GiveUp" >> then else ( v " V ): "Alarm" :
3.2 Should the pedestrian be observed?
0
Assume just one observation is allowed, and that S1 is the only strategy considered; the problem is to choose in the set fAlarm, GiveUp, S1 g. The corresponding game matrix is: NOT a thief ACTUAL thief "Alarm" 30 0 (5) "GiveUp" 0 70 S1 9 14 S1 corresponds to a new line on gure 4 and the decision rule is: When a prior knowledge V exists, then:
8 >>if V 0:14 >< then "GiveUp" :6 >> else if Vthen 0"Alarm" >: else apply strategy S1
0
It is necessary to assume that a xed cost, say , is attached to an RML request, otherwise v = 0 and v = 1 so that the system inde nitely requests RML and never decides. (v ; v") only depends on the costs in table 3, on the information model in equation 4 and on . The existence of (v ; v") would de nitively separate scene recognition from decision making. However, the use of a recursive Wald strategy implies that there is no limit to the number of possible requests to RML and, therefore, there should be no time limit. That is not the case in the problem at hand. 0
00
0
0
When there is no prior knowledge (i.e. the knowledge is simply: 0 V 1), then the game minmax strategy is to choose randomly in the set fS1 ,Alarm,GiveUpg with the relative frequencies: f 76 ; 17 ; 0g. This minmax strategy returns a cost expectation of 12. It is better, of course, than the cost of 21 without information request, but the most important is that a terminal decision is made without identifying whether the pedestrian is likely to be a thief or not. This general result meets [13] who argue: "cognition cannot be fully separated from action and acting". The question seems to be: is cognition necessary? The result returned by RML is used for deciding between "Alarm" and "GiveUp" in accordance with S1 de nition, but it is not necessary to update V (with Bayes's formula), even if several requests to RML are possible as with strategy S2 :
8 >>request RML < says the pedestrian is motionless S2 >if RML then apply S1 >: otherwise "GiveUp"
Robotics, Vision, and Signal Understanding
minmax solution: freq(S1)=6/7 freq(Alarm)=1/7 freq(GiveUp)=0
3.3 Introducing time constraints When dealing with time constraints, modeling RML request costs by a xed number is not adequate. There is a cost because the nal decision is delayed and, thus, is possibly less ecient. Moreover, as we assume limited processing capabilities for RML, requesting to track one object implies discarding other object tracking, which induces other costs. With time constraints the request cost: i) does not increase linearly with time, ii) generally depends on the particular hypotheses about the state of the world, and iii) depends on the waiting surveillance tasks. Decisions (terminal decisions or strategies) cannot be quoted without time references, and the costs that appear in table 3 depend on time { or any equivalent parameter. For instance, the criticity of not sending an alarm as a pedestrian walks to a car is equal to zero if the pedestrian is an honest walker, and increases as he comes closer to the car if he is a thief. This dependence of the costs makes that (v ; v") are not constants. Thus, a recursive strategy as S^ cannot be simply 0
630
C.Barrouil & al.
de ned and, unfortunately, it is not legitimate to manage requests to RML for maintaining beliefs separately from the action choice process. In practice, because of the time constraints, the number of information requests prior to deciding on a particular terminal decision is not very large and introducing a limit N to the number of requests does not actually change the initial problem.
3.4 SL decision strategy
SL reasoning starts when some PH allows ME to update some object attributes. Then RML informs SL that this object is doing something. SL identi es DAs that are both not impossible and partly matched, and then identi es CDs whose evaluation would allow larger DA patterns to be matched [4]. All strategies with at most N RML requests are generated with a recursive algorithm based on the following principle: let R be the set of possible RML answers, let Sn be the set of strategies with at most n requests S0 is the set of terminal decisions Sn+1 is the set of applications R 7! Sn The complexity is exponential, but as N is limited (4 or 5) the problem is still tractable. These strategies allow the game matrix to be calculated: the lines are the strategies and the columns correspond to the current DAs. The minmax strategy is obtained with the simplex algorithm. If decision \Alarm" is made, an alarm is sent and SL reasoning stops. When the decision is \GiveUp", SL stops at once. Note that RML answers are not simply true or fail as, when returning true, RML may indicate complementary information like an object name. In the case RML would return a set of con dence values among R, the maximum element would be considered, if unique; if several elements would correspond to similar maximum con dence degrees, it would be considered that RML returns the answer fail.
4 Global Information Processing Figure 5 represents data ow in the system. ME 2
3
1
DA RML prolog
NL 1
Figure 5.
3
SL
4 2
Information processing in perception
ow 1 corresponds to RML requests to NL for providing
ME with data in order to update beliefs about the objects in the scene. ow 2 represents messages from RML to SL concerning objects events: they trigger SL reasoning. ow 3 is the CD evaluation activity ow 4 is the updating of the importance of the surveillance tasks that adapts the sensor use to SL needs.
Robotics, Vision, and Signal Understanding
631
5 Conclusion
The perception system generates pertinent warnings by the cooperation of two sub-systems with weakly linked strategies. The resource management layer activates sensors in order to avoid pertinent object attributes obsolescence. The symbolic layer requests condition evaluations in order to assess to what extend dreaded activities are likely to go on. Pertinent objects are those that appear in the symbolic layer information requests. It was shown that terminal decisions are decided by using the game theory approach without making explicit situation recognition and, even, that it could be awkward to try to elect a most likely situation and behave as if it held.
ACKNOWLEDGEMENTS
This work was carried out in the framework of the project
PERCEPTION which was supported by the DGA/DRET.
REFERENCES
[1] Yaakov Bar-Shalom and Thomas E. Fortmann, Tracking and Data Association, volume 179 of Mathematics in Science and Engineering, Academic Press, Inc., 1988. [2] M. Basseville, `Detecting changes in signals and systems - a survey', Automatica, 24(3), (1988). [3] Charles Castel, Corine Cossart, and Catherine Tessier, `Dealing with uncertainty in situation assessment: towards a symbolic approach', in UAI'98, 14th Conference on Uncertainty in Arti cial Intelligence, Madison, USA, (july 1998). [4] L. Chaudron, C. Cossart, N. Maille, and Tessier C., `An algebraic framework for a exible symbolic fusion', in IJCAI'97, (1997). Poster sessions. [5] Laurent Chaudron, Corine Cossart, Nicolas Maille, and Catherine Tessier, `A purely symbolic model for dynamic scene interpretation', International Journal on Arti cial Intelligence Tools, 6(4), 635{664, (July 1998). [6] Alvaro del Val, Pedrito Maynard-Reid, and Yoav Shoham, `Qualitative reasoning about perception and belief', in 15th IJCAI, pp. 508{513, (1997). [7] D Drainkov and J Lang, `Possibilistic decreasing persistance', in 9th UAI, pp. 469{476, (1993). [8] D. Dubois and H. Prade, Possibility Theory, an approach to the computerized processing of uncertainty, Plenum Press, New-York, 1988. [9] Patrick Fabiani, Representation dynamique de l'incertain et strategie de prise d'information pour un systeme autonome en environnement evolutif, Ph.D. dissertation, these SupAero, Toulouse, decembre 1996. [10] P.J. Fabiani, `Dynamics of beliefs and strategy of perception', in 12th ECAI, pp. 8{12. Wiley & Sons, (1996). [11] D. Fudenberg and J. Tirole, Game theory, Wiley interscience, MIT PRESS, 1995. [12] J.M. Mendel and K.S. Fu, Adaptive, Learning and Pattern Recognition Systems, volume 66 of Mathematics in Science and Engineering, Academic Press, 1970. [13] J. Penders and P. Braspenning, `Situated actions and cognition', in 15th IJCAI, pp. 1372{1377, (1993). [14] H. Rai a, Decision Analysis, Addision Wesley, 1968. [15] G. Shafer, A mathematical theory of evidence., Princeton University Press, 1976. [16] M. Tribus, Rational Descriptions, Decisions and Designs, Pergamon Press Inc., 1969. [17] A. Wald, Sequential Analysis, Wiley and Sons, 1947.
C.Barrouil & al.
