Method for computing all occurrences of a compound event from ...

US006941290B2

(12)

United States Patent

(10) Patent N0.: US 6,941,290 B2 (45) Date of Patent: Sep. 6, 2005

Siskind

(54)

METHOD FOR COMPUTING ALL

OTHER PUBLICATIONS

OCCURRENCES OFA COMPOUND EVENT

,

FROM OCCURRENCES 0F PRIMITIVE EVENTS

.

.

.

. .

.

Kerridge et al; Synchronization Primitives for Highly Par allel Discrete Event Simulations; Proceedings of the 32nd Annual Hawaii International Conference on System Sci

(75) Inventor: Je?'rey Mark Siskind, Lawrenceville,

69685; V01- Tracl? 8; Jan- 5_8’ _1999; PP _1_10-* _

(73) Assignee; NEC Laboratories America, Inc”

Allen; Maintaining Knowledge About Temporal Intervals;

NJ (Us)

_

Siskind; Grounding Language in Percept1on;Art1?c1al Intel

l1gence Review; vol. 8; Dec. 1994; pp 371—391.* Princeton, NJ (Us)

Communications of the ACM; vol. 26, Iss. 1; Nov. 1983; pp 832—843.*

(*)

Notice:

Subject to any disclaimer, the term of this

ChOW; A GeneraliZed Assertion Language; Proceedings of

patent is extended or adjusted under 35

the 2nd International Conference on Software Engineering;

U.S.C. 154(b) by 635 days.

OCI- 1976* Thiele et al; On FuZZy Temporal Logic; Second IEEE International Conference on FuZZy Systems; vol. 2; Mar. 28—Apr. 1, 1993; pp 1027—1032.* Abe. N. et al., “A Plot Understanding System on Reference

(21) APPL NO: 09/916,249 (22)

Filedi

Jul- 30, 2001

(65)

Pnor Pubhcatlon Data US 2002/0138458 A1 Sep. 26, 2002

.

.

.

to Both Image and Lanaguge,” Proceedings of the Seventh

International Joint Conference on Arti?cial Intelligence, Vancouver, Canada, pp. 77—84, Aug. 1981.

Related US. Application Data (60) 2P(r)(())‘6isional application No. 60/247,474, ?led on Nov. 10,

(Continued) Primary Examiner_AnthOny Knight Assistant Examiner—Meltin Bell

(51)

Int. c1.7 ......................... .. G06F 17/00; G06N 7/00;

G06N 7/08

(57)

ABSTRACT

A method for computing all occurrences of a compound event from occurrences of primitive events where the com

U-Sn Cl-

(58)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..

pound events. event The method is a de?ned includes Combination the steps of of: the (a)

Field of Search ............................ .. 706/25, 45, 58,

706/49, 53, 56, 57; 704/251 (56)

primitive event type is true; and (d) computing the com pound event occurrences, such occurrences being speci?ed

U.S. PATENT DOCUMENTS 3,647,978 A

*

* 10/1992

3/1972

5,301,320 A

*

5,966,523

A

*

6,021,403 A

*

Hill .......................... .. 704/251 Goodridge .... ..

.. 704/219

4/1994 McAtee et al. . 10/1999

Uchino

. ... ... .

2/2000 HoWitZ et al.

primitive event types; (b) de?ning combinations of the

primitive event types as a compound event type; (c) input ting the primitive event occurrences, such occurrences being speci?ed as the set of temporal intervals over which a given

References Cited

5,153,922 A

de?ning

705/9 . . . ..

703/2

706/45

as the set of temporal intervals over which the compound event type is true, where the sets of temporal intervals in steps (c) and (d) are speci?ed as smaller sets of spanning intervals, each spanning interval representing a set of inter vals.

6,424,370 B1 * 7/2002 Courtney ...... .. 348/143 6,785,663 B2 * 8/2004 Wang et al. ..... .. 706/45 6,813,312 B2 * 11/2004 Tullberg et al. ..... .. 375/240.01

5 Claims, 19 Drawing Sheets

De?ne pnmitive event type!

De?ne mrnbinniionr of iii: primitive event types as a

compound ever-ii type

Input the primitive event uccurrenccl. such occurrences

106

being speci?ed in the Sci pr

\empoml mbtrVI-l! werwhich a g'ven primitive event type is true, wherein the set: of

temporal intervals are speci?ed B5 smaller sels ufspanning

intervals, mn spanning interval representing a set ofinlervlls

Compute the compound :venr occurrences, such occurrences being rpeci?ed as me set of lempnru]

iniewnir iwer which the compound event type is true, wherein iii: sets

cftempm'd-l inmvnis m speci?ed as n-niiier rm ufspanning intervals,

mir spinning interval representing a set of intervals

103

US 6,941,290 B2 Page 2

OTHER PUBLICATIONS

Abe, N. et al., “A Learning of Object Structures by Verbal ism,” COLING 82, pp. 1—8, 1982. Adler, M.R., “Computer Interpretation of Peanuts Car toons,” 5th International Joint Conference on Arti?cial Intel

ligence, Cambridge, MA, pp. 608, Aug. 1977. Allen, J .R., “Maintaining Knowledge About Temporal Inter vals,” Communications of the ACM, vol. 26, No. 11, pp. 832—843, Nov. 1983. Blum, M. et al., “A Stability Test for Con?gurations of Blocks,” Arti?cial Intelligence Memo No. 168, Massachu setts Institute of Technology, Feb. 1970.

Bobick, A.F. et al., “Action Recognition using Probabilistic Parsing,” Proceedings of the IEEE Computer Society Con ference on Computer Vision and Pattern Recognition, pp. 196—202, Jun. 1998.

Borchardt, G.C., “A Computer Model for the Representation and Identi?cation of Physical Events,” Masters Thesis, University of Kansas, May 1984. Borchardt, G.C., “Events Calculus,” Proceedings of the Ninth International Joint Conference on Arti?cial Intelli

gence, pp. 524—527, Aug. 1985. Brand, M. et al., “Sensible Scenes: Visual Understanding of

Siskind, J.M., “Unsupervised Learning of Visually—Ob served Events,” AAAI Fall Symposium Series on Learning

Complex Behaviors in Adaptive Intelligence Systems, pp. 82—83, 1996.

Siskind, J .M., “Visual Event Perception”, Proceedings of the 9th NEC Research Symposium, Princeton, NJ, Mar. 1999. Siskind, J .M., “Visual Event Classi?cation via Force Dynamics,” Proceedings of the Seventeenth National Con ference on Arti?cial Intelligence, Aug. 2000. Siskind, J .M. et al., “A Maximum—Likelihood Approach to Visual Event Classi?cation,” Proceedings of the 4th Euro pean Conference on Computer Vision, Cambridge, UK, pp.

347—360, Apr. 1996. Starner, T.E., “Visual Recognition of American Sign Lan guage Using Hidden Markov Models,” Masters Thesis, Massachusetts Institute of Technology, Feb. 1995.

Talmy, L., “Force Dynamics in Language and Cognition,” Cognitive Science, vol. 12, pp. 49—100, 1988. Thibadeau, R., “Arti?cial Perception of Actions,” Cognitive Science, vol. 10, No. 2, pp. 117—149, 1986.

of the Eleventh National Conference on Arti?cial Intelli

Tsuji, S. et al., “Understanding a Simple Cartoon Film by a Computer Vision System,” Proceedings of the 5th Interna tional Joint Conference on Arti?cial Intelligence, Cambridge

gence, pp. 588—593, 1993.

MA, pp. 609—610, Aug. 1977.

Fahlman, SE, “A Planning System for Robot Construction Tasks,” Arti?cial Intelligence, vol. 5, No. 1, pp. 1—49, 1974.

Tsuji, S. et al., “Three Dimensional Movement Analysis of Dynamic Line Images,” Proceedings of the Sixth Interna

Complex Structures Through Causal Analysis,” Proceedings

Krifka, M., “Thematic Relations as Links BetWeen Nominal

Reference and Temporal Constitution,” Lexical Matters, Sag, I.A. (eds.), pp. 29—53, 1992. Mann, R. et al., “ToWards the Computational Perception on

Action,” Proceedings of the IEEE Computer Society Con ference on Computer Vision and Pattern Recognition, Santa

Barbara, CA, pp. 794—799, 1998. Mann, R. et al., “The Computational Perception of Scene

tional Joint Conference on Arti?cial Intelligence, Tokyo,

Japan, pp. 896—901, Aug. 1979. Tsuji, S. et al., “Tracking and Segmentation of Moving Objects in Dynamic Line Images,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 2, No. 6, pp. 516—522, 1980. WaltZ, D.L., “ToWard a Detailed Model of Processing for

Dynamics,” Computer Vision and Image Understanding,

Language Describing the Physical World,” Proceedings of

vol. 65, No. 2, pp. 113—128, Feb. 1997.

the Seventh International Joint Conference on Arti?cial

McCarthy, J., “Circumscription—A Form of Non—Mono tonic Reasoning,” Arti?cial Intelligence, vol. 13, pp. 27—39,

Intelligence, Vancouver, Canada, pp. 1—6, Aug. 1981. WaltZ, D.L., “Visual Analog Representations for Natural Language Understanding,” Proceedings of the Sixth Inter

1980.

Okada, N., “SUPP: Understanding Moving Picture Patterns Based on Linguistic Knowledge,” Proceedings of the Sixth International Joint Conference on Arti?cial Intelligence,

Tokyo, Japan, pp. 690—692, Aug. 1979. Regier, T.P., “The Acquisition of Lexical Seminatics for Spatial Terms: A Connectionist Model of Perceptual Cat egorization,” Ph.D. Thesis, University of California, Berke ley, 1992. Shoham, Y., “Temporal Logics in A1: Semantical and Onto logical Considerations,” Arti?cial Intelligence, vol. 33, pp. 89—104, 1987. Siskind, J.M., “Naive Physics, Event Perception, Lexical Semanics, and Language Acquisition,” Ph.D. Thesis, Mas sachusetts Institute of Technology, 1992. Siskind, J.M., “Axiomatic Support for Event Perception,” Proceedings of the AAAI—94 Workshop on the Integration of Natural Language and Vision Processing. Seattle, WA, pp. 153—160, Aug. 1994.

national Joint Conference on Arti?cial Intelligence, Tokyo,

Japan, pp. 926—934, Aug. 1979. Yamato, J. et al., “Recognizing Human Action in Time—Se

quential Images using Hidden Markov Model,” Proceedings of the IEEE Conference on Computer Vision and Pattern

Recognition, pp. 379—385, 1992. ChoW; A Generalized Assertion Language; Proceedings of the 2nd International Conference on SoftWare Engineering; Oct. 1976; pp 392—399.*

Thiele et al; On FuZZy Temporal Logic; Second IEEE International Conference on FuZZy Systems’, vol. 2., Mar.

28—Apr 1, 1993., pp 1027—1032.*

Kerridge et al; SynchroniZation Primitives for Highly Par allel Discrete Event Simulations’, Proceedings of the 32nd Annual HaWaii International Conference on System Sci

ences, vol. Track 8, Jan. 5—8, 1999, pp 1—10.*

Siskind, J .M., “Grounding Language in Perception,” Arti? cial Intelligence RevieW, vol. 8, pp. 371—391, Dec. 1994.

* cited by examiner

U.S. Patent

Sep. 6,2005

Sheet 1 0f 19

US 6,941,290 B2

102

De?ne primitive event types

De?ne combinations of the primitive event types as a

104

compound event type

Input the primitive event occurrences, such occurrences

106

being speci?ed as the set of

temporal intervals over which

/

a given primitive event type is true, wherein the sets of

temporal intervals are speci?ed as smaller sets of spanning

intervals, each spanning interval representing a set of intervals

Compute the compound event

10

occurrences, such occurrences being / speci?ed as the set of temporal intervals over which the compound event type is true, wherein the sets

of temporal intervals are speci?ed as smaller sets of spanning intervals,

each spanning interval representing a set of intervals

FIG. 1

8

U.S. Patent

Sep. 6, 2005

Sheet 2 0f 19

US 6,941,290 B2

Start

l Label primitive subexpressions of (D with spanning intervals that represent the sets of intervals over which the

corresponding primitive event types hold

Are there any

subexpressions that have not been labeled

with sets of spanning intervals?

Stop: output the set of spanning intervals that label the whole

(root) expression CD

Let (1)’ be some subexpression of (I) such that CD’is not labeled with sets of

spanning intervals 'but for vyhich all Y subexpressions (DI , MCI)" of II)’ are labeled with sets of spanning intervals

FIG. 2

Apply the appropriate formula for e (M, ') using the subroutines (i), i1 (1 i2,-1i, sPAN(i1, i2),€D(r, i), and $J(i,r, j) to compute a set of

spanning intervals to label (1)’

U.S. Patent

Sep. 6, 2005

US 6,941,290 B2

Sheet 5 0f 19

UENMUNtQHE 25ENNQBEEE32

uNNamkm

.65) (ATTAGKED'! RED cmlln?lmnln

FIG. 6A

FIG. 6B

U.S. Patent

Sep. 6,2005

Sheet 7 0f 19

US 6,941,290 B2

. M

U.S. Patent

US 6,941,290 B2

BEEmm3“295@N069a

U.S. Patent

Sep. 6,2005

Sheet 9 0f 19

US 6,941,290 B2

m

Spa wa

Ow.DE 25¢3

,9, “w W .. ,

U.S. Patent

Sep. 6,2005

Sheet 10 0f 19

US 6,941,290 B2

0Kb5 »

0258m

.05Dw

U.S. Patent

Sep. 6,2005

Sheet 11 0f 19

US 6,941,290 B2

GEmi.

U.S. Patent

Sep. 6,2005

Sheet 12 0f 19

(arr-nova nuvm m nwsn?lmuLluwll) (STACK WWII; RED ILUE GEDOO‘IHOJil-[ZIJOIH

FIG. 8A

FIG. 8B

FIG. 8C

FIG 8D

FIG. 8B

US 6,941,290 B2

U.S. Patent

Sep. 6,2005

Sheet 14 0f 19

US 6,941,290 B2

U.S. Patent

Sep. 6,2005

#N$35E

Sheet 15 0f 19

US 6,941,290 B2

U.S. Patent

Sep. 6,2005

Sheet 16 0f 19

US 6,941,290 B2

Q6 6

E

U.S. Patent

Sep. 6,2005

Sheet 17 0f 19

US 6,941,290 B2

FIG. 10A

FIG. 10B

FIG. 10C

FIG. 10D

U.S. Patent

Sep. 6,2005

Sheet 18 0f 19

US 6,941,290 B2

UEMH

25¢2

22:8.“

95EM:

.05m: N~Deni