the roles of ontologies in machine intelligence

From machines that learn to machines that know: the roles of ontologies in machine intelligence* Alessandro Oltramari Bosch Research and Technology Center

*Not a state of the art! :-)

Semantic Deep Learning A New Role in the new season of the AI drama (and it’s getting noticed!)

3

We are in “Deep” trouble • Deep Learning is a Black Box •

You literally put Big Data in it, then you wait, and you get results

• What’s inside the Box? •

Algorithms that attempt to model data representation through many layers of non-linear transformations • Deep Convolutional Neural Network (CNN) → good at modeling data with strong topological structures and local correlations (images and speech) • Deep Conditional Restricted Boltzmann Machine (RBM) → good at modeling time-series (e.g., human motion, sensor data streams, etc.) • SVM, etc.

• Can we understand how the algorithms generate the

results? •

“It’d be like explaining Shakespeare to a dog” (Hod Lipson, Nature, Vol 538, 6 October 2016)

4

But let’s not go into “deep” panic • We are not dogs • The algorithmic processes are opaque, not the I/O • The Black Box problem can reduced to a more “tractable”

issue: how can we clarify the connection between data as input and data as output? •

Quantitative methods: measure the correlation •

E.g., Family of metrics that capture the degree of influence of inputs on outputs of the system*

• Semantic transparency: ontologies of input data (across layered neural networks) can be ultimately interlinked to ontologies of output data •

E.g. From visual signals, to pixels, bounded boxes, objects, persons, movements, complex activities, etc.**

*Datta, Anupam, Shayak Sen, and Yair Zick. "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems." Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 2016. **Oltramari, Vinokurov, Lebiere, Oh, Stentz. “Ontology-based cognitive system for contextual reasoning in robot architectures” 2014 AAAI Spring Symposium Series

03/08

5

Ontolog

Cognitive Engine

03/08

Ontolog

6

03/08

7

Ontolog

Ontology-driven Activity Recognition • Visual signals are processed by ML classifiers, output is mapped to ontology patterns of action • Classification of actions is based on semantic

structure (frames sequences

and

roles)

and

temporal

• New patterns can be learned • Natural Language Generation of scene descriptions

03/08

Ontolog

8

Semantic Aspects of Visual Pattern Recognition Conceptual packaging1 • Events are “packaged” (e.g., motion verbs come with path/direction/source/destination) • Reversible vs. Non-reversible events = Symmetric vs. Asymmetric (OPEN vs. CUT)

Compositional similarity2 • Similar events have high degree of overlapping stages

Conceptual primitives3 • Substantiality: objects generally cannot pass through one another. • Continuity: objects that appear in two locations at different time in the must have moved along the

connecting path. • Gravity: unsupported objects fall. __________________________________________________________________________________________________________________________________________________ 1

Majid, A., Boster, J.S., Bowerman, M. (2008). The cross-linguistic categorization of everyday events: A study of cutting and breaking. Cognition, 109, 235-250. 2 Biederman, I. (1987). Recognition-by-Components: A Theory of Human Image Understanding. Psychological Review, 94, 115-147. 3 Siskind, J.M. (1995). Grounding Language in Perception. Artificial Intelligence Review, 8, 371-391.

Data Integration A more traditional role...but in the coolest of the new industrial sectors: IoT

03/08

Ontolog

10

IoT, the core driver of Machine Intelligence • 2015: 9 Billion devices • 2020: 30-40 Billion devices ➢ 44 ZB of content

More than just ginormous numbers…

03/08

Ontolog

11

Sensors + KRR + Actuators = Ambient Machine Intelligence

03/08

Ontolog

12

Internet Of (Intelligent) Things “To realize Ambient Machine Intelligence, these things must understand the user’s context, including location, activities, cognitive/affective states, and social interactions, as well as the environment’s state”. Roggen, Daniel, et al. "Opportunistic human activity and context recognition." Computer-IEEE Computer Society- 46.EPFL-ARTICLE-182084 (2013): 36-45.

03/08

Ontolog

13

The Role of Users IoT • A user-centric knowledge base system is as important

as a sensor/data-centric one. It should represent: - demographics - location - behaviors - biometrics - emotions - beliefs - user preferences • privacy • security - social media - ...

03/08

Ontolog

14

Machine Learning Applications Source: McKinsey Report 2016

03/08

Ontolog

15

Machine Learning Applications Source: McKinsey Report 2016

03/08

Automotive, Retail

Ontolog

16

Source: McKinsey Report 2016

03/08

Energy, Healthcare

Ontolog

17

Source: McKinsey Report 2016

03/08

18

Ontolog

Ambient Machine Intelligence CONTEXT

KNOWLEDGE

INTELLIGENCE

Sensor Network Cloud Services

External Knowledge Representation Integration

Social and Environment

Machine Reasoning

Internal Knowledge Representation

Adaptive IoT Systems - Decision support - Predictive analytics - Autonomous agent

Cognition and Emotion Actuator Network

03/08

19

Ontolog


KNOWLEDGE

INTELLIGENCE

Sensor Network


Internal Ontology Cognition and Emotion

Machine Reasoning

Integration

External Ontology

Cloud Services


03/08

20

Ontolog


KNOWLEDGE

INTELLIGENCE


External Ontology Integration


Machine Learning

IoT Systems

Internal Ontology

- Decision support - Predictive analytics - Autonomous agent


03/08

21

Ontolog


KNOWLEDGE

INTELLIGENCE


External Ontology Integration


Machine Learning

Internal Ontology



22

Take-home messages • There’s a whole world beyond “ontology learning”, and my

presentation didn’t even scratch the surface of it! • Ontologies play different and yet crucial roles for enabling • •

semantic transparency in Deep Learning Networks intelligent services in IoT infrastructures through • • • •

sensor data fusion high level reasoning decision making predictive analytics

• The recipe for human intelligence is complex, but

perceptions and knowledge are two necessary ingredients. Machine Learning is good at clustering perceptions, but without ontologies that represent the knowledge aspects of those clusters, there can’t be any machine intelligence, or at least any form of machine intelligence capable of replicating cognitive capabilities.