Semantic Aspects of Visual Pattern Recognition. Conceptual packaging1. ⢠Events are âpackagedâ (e.g., motion verbs
From machines that learn to machines that know: the roles of ontologies in machine intelligence* Alessandro Oltramari Bosch Research and Technology Center
*Not a state of the art! :-)
Semantic Deep Learning A New Role in the new season of the AI drama (and it’s getting noticed!)
3
We are in “Deep” trouble • Deep Learning is a Black Box •
You literally put Big Data in it, then you wait, and you get results
• What’s inside the Box? •
Algorithms that attempt to model data representation through many layers of non-linear transformations • Deep Convolutional Neural Network (CNN) → good at modeling data with strong topological structures and local correlations (images and speech) • Deep Conditional Restricted Boltzmann Machine (RBM) → good at modeling time-series (e.g., human motion, sensor data streams, etc.) • SVM, etc.
• Can we understand how the algorithms generate the
results? •
“It’d be like explaining Shakespeare to a dog” (Hod Lipson, Nature, Vol 538, 6 October 2016)
4
But let’s not go into “deep” panic • We are not dogs • The algorithmic processes are opaque, not the I/O • The Black Box problem can reduced to a more “tractable”
issue: how can we clarify the connection between data as input and data as output? •
Quantitative methods: measure the correlation •
E.g., Family of metrics that capture the degree of influence of inputs on outputs of the system*
• Semantic transparency: ontologies of input data (across layered neural networks) can be ultimately interlinked to ontologies of output data •
E.g. From visual signals, to pixels, bounded boxes, objects, persons, movements, complex activities, etc.**
*Datta, Anupam, Shayak Sen, and Yair Zick. "Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems." Security and Privacy (SP), 2016 IEEE Symposium on. IEEE, 2016. **Oltramari, Vinokurov, Lebiere, Oh, Stentz. “Ontology-based cognitive system for contextual reasoning in robot architectures” 2014 AAAI Spring Symposium Series
03/08
5
Ontolog
Cognitive Engine
03/08
Ontolog
6
03/08
7
Ontolog
Ontology-driven Activity Recognition • Visual signals are processed by ML classifiers, output is mapped to ontology patterns of action • Classification of actions is based on semantic
structure (frames sequences
and
roles)
and
temporal
• New patterns can be learned • Natural Language Generation of scene descriptions
03/08
Ontolog
8
Semantic Aspects of Visual Pattern Recognition Conceptual packaging1 • Events are “packaged” (e.g., motion verbs come with path/direction/source/destination) • Reversible vs. Non-reversible events = Symmetric vs. Asymmetric (OPEN vs. CUT)
Compositional similarity2 • Similar events have high degree of overlapping stages
Conceptual primitives3 • Substantiality: objects generally cannot pass through one another. • Continuity: objects that appear in two locations at different time in the must have moved along the
connecting path. • Gravity: unsupported objects fall. __________________________________________________________________________________________________________________________________________________ 1
Majid, A., Boster, J.S., Bowerman, M. (2008). The cross-linguistic categorization of everyday events: A study of cutting and breaking. Cognition, 109, 235-250. 2 Biederman, I. (1987). Recognition-by-Components: A Theory of Human Image Understanding. Psychological Review, 94, 115-147. 3 Siskind, J.M. (1995). Grounding Language in Perception. Artificial Intelligence Review, 8, 371-391.
Data Integration A more traditional role...but in the coolest of the new industrial sectors: IoT
03/08
Ontolog
10
IoT, the core driver of Machine Intelligence • 2015: 9 Billion devices • 2020: 30-40 Billion devices ➢ 44 ZB of content
More than just ginormous numbers…
03/08
Ontolog
11
Sensors + KRR + Actuators = Ambient Machine Intelligence
03/08
Ontolog
12
Internet Of (Intelligent) Things “To realize Ambient Machine Intelligence, these things must understand the user’s context, including location, activities, cognitive/affective states, and social interactions, as well as the environment’s state”. Roggen, Daniel, et al. "Opportunistic human activity and context recognition." Computer-IEEE Computer Society- 46.EPFL-ARTICLE-182084 (2013): 36-45.
03/08
Ontolog
13
The Role of Users IoT • A user-centric knowledge base system is as important
as a sensor/data-centric one. It should represent: - demographics - location - behaviors - biometrics - emotions - beliefs - user preferences • privacy • security - social media - ...
03/08
Ontolog
14
Machine Learning Applications Source: McKinsey Report 2016
03/08
Ontolog
15
Machine Learning Applications Source: McKinsey Report 2016
03/08
Automotive, Retail
Ontolog
16
Source: McKinsey Report 2016
03/08
Energy, Healthcare
Ontolog
17
Source: McKinsey Report 2016
03/08
18
Ontolog
Ambient Machine Intelligence CONTEXT
KNOWLEDGE
INTELLIGENCE
Sensor Network Cloud Services
External Knowledge Representation Integration
Social and Environment
Machine Reasoning
Internal Knowledge Representation
Adaptive IoT Systems - Decision support - Predictive analytics - Autonomous agent
Cognition and Emotion Actuator Network
03/08
19
Ontolog
Ambient Machine Intelligence CONTEXT
KNOWLEDGE
INTELLIGENCE
Sensor Network
Social and Environment
Internal Ontology Cognition and Emotion
Machine Reasoning
Integration
External Ontology
Cloud Services
Adaptive IoT Systems - Decision support - Predictive analytics - Autonomous agent
03/08
20
Ontolog
Ambient Machine Intelligence CONTEXT
KNOWLEDGE
INTELLIGENCE
Sensor Network Cloud Services
External Ontology Integration
Social and Environment
Machine Learning
IoT Systems
Internal Ontology
- Decision support - Predictive analytics - Autonomous agent
Cognition and Emotion Actuator Network
03/08
21
Ontolog
Ambient Machine Intelligence CONTEXT
KNOWLEDGE
INTELLIGENCE
Sensor Network Cloud Services
External Ontology Integration
Social and Environment
Machine Learning
Internal Ontology
Adaptive IoT Systems - Decision support - Predictive analytics - Autonomous agent
Cognition and Emotion Actuator Network
22
Take-home messages • There’s a whole world beyond “ontology learning”, and my
presentation didn’t even scratch the surface of it! • Ontologies play different and yet crucial roles for enabling • •
semantic transparency in Deep Learning Networks intelligent services in IoT infrastructures through • • • •
sensor data fusion high level reasoning decision making predictive analytics
• The recipe for human intelligence is complex, but
perceptions and knowledge are two necessary ingredients. Machine Learning is good at clustering perceptions, but without ontologies that represent the knowledge aspects of those clusters, there can’t be any machine intelligence, or at least any form of machine intelligence capable of replicating cognitive capabilities.