Neuromorphic AER-DVS camera

Neuromorphic AER-DVS camera

Marek Otáhal Cyber, FEL CVUT [email protected] 25.3.2015

●

here

WordCloud

What can it do? ●

Fast (really!)

●

High dynamic range (bad lightning conditions)

●

Neuro-inspired → sparse output (HTM/CLA)

●

Filters (eye tracking, gestures, edge detector)

●

Sense of 3D

●

Low power

Motivation why? ●

●

Q: why not to use existing well known pre/postprocessing image algorithms? (edge detectors, face tracking, sparse encodings, …) A: they don't cut it at sufficient efficiency –

SPEED: latency down to 15us!!

–

Low power: 20mW (embedded)

–

No need for extra lightning (120dB dyn. Range)

–

Low cost (compared to high-speed setups, ~5000CHF)

Features (cont) - Speed ●

DVS speed: –

Latency 5us

–

Scale: ● ●

Fast: dot on grinding disc el. drill(high temp. resolution) Slow: sleep disorders (skipping useless periods of inactivity)

–

Temporal/spatial sparsity: sleep disorders

–

Specs: (example of one product) ●

(? low pixel resolution)

Principles ●

And the name explained

●

Neuromorphic –

“inspired by neuro-biological architectures” ●

DVS ~ retina

–

Neuristor, memristor, chips,Human Brain Project

–

Sparse representations ● ●

Spatial (only active pixels/neurons) Temporal (only when something happens = AER, eg “sleep”)

Principles (2) - AER AER = address-event representation ● “communications protocol for transferring spikes between bio-inspired chips” AER example ●

●

Principles (2) - AER AER ● “communications protocol for transferring spikes between bio-inspired chips” ● Translation to/from addresses/spikes AER example ●

Principles (2) - AER AER ● “communications protocol for transferring spikes between bio-inspired chips” ● Translation to/from addresses/spikes AER example ●

Principles (3) - DVS ● ● ●

DVS = dynamic vision system Retina inspired tech Conventional vs. DVS vision system advantages – Low power – Low cost – Fast – Sparse – Continuous – No spec. lighting needed

Principles (3) – DVS 2 ● ● ● ●

DVS = dynamic vision system Pixels (=cells) react asynchronously to relative light changes “ON/OFF” events eg. on increased/dec. light intensity on the pixel Explained in video –

Experiments ●

Connect your camera

●

Use recorded RAW data –

Basic stimuli: Moving hand, Orientation stimulus

–

Self locomotion (walking)

–

High speed: Juggling, Fast spinning dot

–

Surveillance: Street scene with cars and people walking, 3.5 days of mouse activity, Walking flies

–

Tracking: Cars passing under bridge over the 210 freeway in Pasadena, Eye tracking, Eye movements - pupil view

–

Driving: Driving in Pasadena to the post office, Driving on the Pasadena freeway

Programming ●

OSS repo jAER/cAER

●

Java, C++, (Python, Matlab scripts)

●

Drivers & GUI viewer for all OS (Win,Linux,OSX)

●

Filters: “programs” running on the DVS hw locally –

From filters like: Gaussian mixture, Space cells, Correlation,

–

To advanced: eye,face,hand-tracking, movement detection, car detection

–

Up to specific: driving XYZ robot,...

–

Complete list of filters

–

Interfacing (binding) to your programs - direct MM access, read from file, send TCP/UDP messages (personal: filters a bit messy; GUI crashes sometimes on Linux)

Use-cases & examples ●

Moving hand (obfuscated w glasses)

●

Robotic goalie

●

Tracking fast moving cars

●

Particle tracking in fluid dynamics

●

High spinning dot

●

others

Ideas for interfacing with neural networks ●

●

DVS: –

Uses sparse coding (suitable for CLA)

–

Many high-level “filters” → DVS useful as a preprocessor/encoder for NN

Programming –

For speed, use memory-access

–

otherwise TCP/UDP packets are nice ●

We could wrap jAER as a ROS node (eg as in NengoROS )

Future development Our areas of interest: ●

Anomaly detection in vision (jAER+HTM/CLA)

●

Vision Neuro-implants (jAER+visual prosthesis)

●

Spatiao-temporal orientation (jAER+Autonomous Agents) Next steps:

●

updating Nupic for vision domain (see nupic.vision) –

●

Mostly tasks on improving spatial clustering (SP) in vision

Pass data from DVS to CLA (encoder,region,sensor,...?)

Resources & Further reading ●

http://siliconretina.ini.uzh.ch/wiki/index.php

●

http://www.inilabs.com/products

●

http://sourceforge.net/p/jaer/wiki/Home/

●

https://github.com/numenta/nupic

●

https://github.com/nupic-community/nupic.vision

●

https://ojs.cvut.cz/ojs/index.php/ap/article/view/2272

●

https://indico.cern.ch/event/120853/session/2/contributio n/158/material/slides/1.pdf

Thank you ●

For your attention –

●

For questions and ideas, please reach us at nupic.vision or my email.

This work has been supported from research grant SGS14/144/OHK3/2T/13.