Deep Reinforcement Learning in Serious Games

Deep Reinforcement Learning in Serious Games Analysis and Design of Deep Neural Network Architectures

Aline Dobrovsky, Uwe M. Borghoff, Marko Hofmann Institute for Software Technology Computer Science Department Universität der Bundeswehr München

Eurocast 2017

Outline 1. Introduction:  Serious Games Artificial Intelligence (AI) Challenges and Research 2. Approach:  Deep Reinforcement Learning  Deep Reinforcement Learning Architectures  Interactive Deep Reinforcement Learning (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

2

Serious Games Education/Training Interactive Computer Application: •

Challenging Goal

•

Fun to Play

•

Concept of Scoring

•

Knowledge: Can be Applied in the Real World

Entertainment

Screenshot from Battlefield3, EA DICE (2011)

Bergeron (2006)

Goal

Means

Screenshot from America‘s Army – Proving Grounds, U.S. Army (2015)

(Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

3

Serious Game Example: SanTrain TCCC: Tactical Combat Casualty Care

First Aid on the Battlefield

Education: • Cooperation of Various Experts • Realism : Precise Pathophysiologic Model of the Human Body (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

4

Artificial Intelligence in Serious Games NPC

Flow NPC Channel


Increasing Challenges

Increasing Challenges

Challenges

Goals HIGHand Constraints • ImmersionAnxiety and Flow • Constraints: Time, Money Increasing Skills • Prevent “Gaming the Game”

Increasing Skills

LOW LOW

Domains • NPC Control Boredom • Adaption

Skills / Time

HIGH

SanTrain: Enemy Positioning Example Concept of Flow by Mihaly Csikszentmihalyi (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

Serious Game AI Challenges and Research • Various, Specific Methods

3

Automated  Interactive Deep Reinforcement Adaption Learning? •

Interactive Learning

NPC

NPC


1

AI Generation

• Various, Specific Methods

 (Interactive) Deep Reinforcement Learning ?

2

 Interactive Deep Reinforcement Learning?

Experts' Influence

• Data • Feedback • Domain Knowledge Behaviour Bias Scenario Control

…



7

Deep Reinforcement Learning

Q-Learning Max Q(st,at)

*

Q values: Prediction of Reward a1

a2

…

an

s1

0

0

…

-1

s2

10

1

…

1

…

…

…

…

…

sn

-1

0

…

5

*R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 1998. (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

8

Deep Reinforcement Learning

Input Layer Hidden Layer(s) Output Layer

…

a1

a2

…

s t a t e

Artificial Neural Network→Connectionist Emergent Cognitive System


9


10

Basic Neural Network Architectures Feedforward

• Universal Approximator

Recurrent

• Turing-complete • Memory


11

Feedforward: Convolutional Neural Networks (CNN)

• Understanding Convolutional Neural Networks for NLP http://selmandesign.com/qa-on-machine-learning/ http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/ • Feature extraction using convolution from Stanford Deep Learning http://deeplearning.stanford.edu/wiki/index.php/Feature_extraction_using_convolution


12

CNN Example: Supervised Learning GoogLeNet Inception Architecture • • •

CNNs for Visual Recognition GoogLeNet: Inception Architecture Won the Image Classification Challenge 2014 (5,6% top 5 error rate)

* * Szegedy, Christian, et al. "Going deeper with convolutions." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.


* 13

CNN Example: Unsupervised Learning Unsupervised Learning of Visual Representations using Videos • • •

Unsupervised Tracking in Videos Objects: Similar Visual Representation in Deep Feature Space Performance Comparable to Previous Succesful Image Recognition Results

Wang, Xiaolong, and Abhinav Gupta. "Unsupervised learning of visual representations using videos." Proceedings of the IEEE International Conference on Computer Vision. 2015.


14

CNN Example: Deep Reinforcement Learning in Computer Games Mnih, Volodymyr, et al:

"Human-level control through deep reinforcement learning." Nature 518.7540 (2015)

+ quick-moving games - long-horizon games ? realistic behaviour Video Pinball, Revenge, Atari, Inc.Parker (1979)Bros (1984) Montezuma's


15

CNN Deep Learning Architectures Analyses and Experiments Theses: Deep Reinforcement Learning in Computer Games (Atari) • CNN Architectures • • • •

•

•

Blackboxes Configuration: Architectures, Parameters, Algorithms Learning Time, Hardware Image Recognition: Precise Classification with big Architectures Games: Small Architectures, Algorithm Improvements

CNN Visualization (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

16

CNN Deep Learning Architectures Analyses and Experiments Theses: Deep Reinforcement Learning in Computer Games (Atari) • CNN Architectures • CNN Visualization • Nice Gimmick -> Developer Tool • Comprehensive Toolboxes Available • Good Option for Error Detection • Use and Performance of Different Layers • Interpretation is Necessary (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

17

CNN Architectures: Conclusions • •

CNNs are Leading in Image Recognition CNNs can be Used in Games • • • • • •

•

•

NPC/Avatar Control Works on Raw Screen Input Replaces Symbol System AI? Meaningful Reward System Necessary Games: Quick Reaction, Short Horizon Long-Horizon, Planning Endless Configuration Possibilities, no General Rules Experiments: Long Training Times, Local Optima, Successful in “Simple Scenarios” with Complete Information Improvements through Changes of Learning Algorithms

Further Research Needed •

Methods with Memory and Planning • RNN Master Thesis • CNN and RNN Combinations (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

18


19


3

Automated  Interactive Deep Reinforcement Adaption ? Learning? •


NPC

NPC


1

AI Generation



2


Experts' Influence

?


…


Interactive Deep Reinforcement Learning: Communication & Interaction Possibilities a) Iterative Workflow:

Human Teacher

 Pure Learning + Feedback

b) Interactive Shaping c) Learning from Human Demonstration d) Active Learning

…

Agent (Deep Reinforcement Learning)


21


3

Automated  Interactive Deep Reinforcement Adaption Learning? •


NPC

NPC


1

AI Generation



2


Experts' Influence


…


Interactive Deep Reinforcement Learning: Player Communication & Interaction Adaption Behaviour Experts

Agents (Deep Reinforcement Learning)

Game Interface

Visualization

Interaction Interface


Controller

…

Settings

Developer (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

23

Conclusion • • • •

Challenges in Serious Game (AI) Development Deep Reinforcement Learning (DRL) Deep Reinforcement Learning Architectures Interactive Deep Reinforcement Learning (iDRL) Current Research and Approaches

• •

CNN in Games, RNN, Interactive Learning, Serious Game NPC AI and Adaption

• Eurocast Implications :  iDRL as broadly Appliable AI Concept  Interaction of Different Cognitive Systems and Beings (Interactive) Deep Reirning in Serious Games – Uwe M. Borghoff

24

Future Work and Open Issues •

Deep Reinforcement Learning •

Architecture Decisions or Indications?

•

Human-Computer Interaction in iDRL

•

DRL Framework for Serious Games

•

Serious Game Adaption

• Eurocast Implications :  iDRL as broadly Appliable AI Concept  Interaction of Different Cognitive Systems and Beings

•

Application Areas (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

25

Deep Reinforcement Learning Playe in Serious Games r Analysis and Design of Deep Neural Network Architectures

Adaption

Behaviour Experts

Agents (Deep Reinforcement Learning)

Game Interface Controller

Interaction Interface Interactive Learning

Visualization

…

Settings

Contact: Developer Aline Dobrovsky, Uwe M. Borghoff, Marko Hofmann

[email protected] (Interactive) Deep Reinforcement Learning in Serious Games – Uwe M. Borghoff

26