Online Nonparametric Bayesian Activity Mining From Video

“... we may have knowledge of the past but cannot control it; we may control the future but have no knowledge of it.” Claude Shannon (1916-2001)

Online Nonparametric Bayesian Activity Mining From Video Vahid Bastani, Lucio Marcenaro, Carlo Regazzoni, Online Nonparametric Bayesian Activity Mining and Analysis From Surveillance Video, IEEE Transactions on Image Processing, vol.25 n.5, p.p.2089-2102, May 2016

Vahid Bastani, Lucio Marcenaro, Carlo Regazzoni, Incremental Nonlinear System Identification and Adaptive Particle Filtering Using Gaussian Process, arXiv:1608.08362, Aug 2016, (under review for IEEE Signal Processing Letters) Vahid Bastani, Lucio Marcenaro, Carlo Regazzoni, A Particle Filter Based Sequential Trajectory Classifier For Behaviour Analysis in Video Surveillance, Image Processing (ICIP), 2015 IEEE International Conference on, Quebec City, Canada, September 2015 9/17/2016

1

Motivation • Surveillance video is the biggest Big Data • Manual annotation is impossible and Supervised methods usage is limited • Storage is impossible • Online exploratory video analytic tools are necessary 9/17/2016

2

Objectives Problems I (Online Activity Mining): Only by observing moving objects over time • Determine how many different trajectory patterns exist (Clustering) • Learn the shape of trajectory patterns (Learning) Problems II (Online Classification and Abnormality Detection): Given set of learned trajectory patterns, for every moving object at each time instance: • Track the object’s position (state estimation) (Tracking) • Classify the object’s trajectory to one of the learned models or determine if it is a new unseen trajectory (Classification)

9/17/2016

3

Probabilistic Model of Trajectories 𝒙𝑛𝑡+1 = 𝒙𝑛𝑡 + 𝒇 𝒙𝑛𝑡 , 𝑡; 𝜽𝒓𝒏

Time-space dependent flow function

𝛼

𝜆

𝜽𝑚

𝒙𝑛𝑡

𝒙𝑛𝑡+1

𝒚𝑛𝑡

𝒚𝑛𝑡+1

∞ Parameter set determines the flow function shape

9/17/2016

𝑟𝑛

𝝅

𝑁

4

Learning-Recognition Loop “The term closed loop-learning process refers to the idea that one learns by determining what s desired and comparing what is actually taking place as measured at the process and feedback for comparison.” Harold Chestnut (1984)

Video Stream

Tracking Classification RB Particle Filter Memory

History Learning

Clustering

Incremental DPM over GPs 9/17/2016

5

Learning Flow Function Trajectory class indicator From RBPF tracking

Estimated flow at trajectory points + variance of estimation Estimated trajectory points ML Gaussian process kernel parameter

Incremental model update: Bayesian Committee Machine

Incremental model update: Stochastic Variational Sparse Gaussian Process • Model the dataset as variational distribution over some inducing points: {𝑥𝑖 , 𝑥𝑖 }𝑁 𝑖=1

𝑞 𝒖 = 𝒩(𝒎, 𝑺)

• Update variational parameters with every new sample using Stochastic Gradient Descent (SGD) Vahid Bastani, Lucio Marcenaro, Carlo Regazzoni, Incremental Nonlinear System Identification and Adaptive Particle Filtering Using Gaussian Process, arXiv:1608.08362, Aug 2016, (under review for IEEE Signal Processing Letters) 9/17/2016

8

BCM vs SVSGP BCM

SVSGP

Unbounded memory

Bounded memory

Complexity grows over time

Fixed computational complexity

Trajectory level updating

Sample level updating

9/17/2016

9

Incremental Dirichlet Allocation How do we calculate trajectory class indicator Number of trajectory assigned to Class r

Likelihood that the trajectory is from class r

Likelihood that the trajectory is from any new class How much surprise do you expect? (DP concentration parameter) Chinese Restaurant Process

Rao-Blackwellized Particle Filter We want to estimate joint posterior of state trajectory and trajectory class

Marginalize trajectory class and estimate only state trajectory posterior

Particle Filter

r

x(t-1)

x(t)

x(t+1)

y(t-)

y(t)

y(t+1)

Estimate trajectory class posterior given state and observation trajectory Conditional Likelihood

Tracking and Classification (2) Compute posterior trajectory class probabilities Randomly chose a trajectory class based on calculated class The Filter adaptively selects probabilities best dynamic model for tracking among K+1 model Sample updated state based on sampled trajectory class Compute particle weights Compute class conditional Likelihood

Demo – online learning and classification

9/17/2016

13

Demo – prediction by particle propagation

Cluster 1

9/17/2016

Cluster 2

14

Results

9/17/2016

15

Work in progress: Detecting environment saliency map Obstacles

Attractors

9/17/2016

16