S. La Bua - Design and Implementation of Modules for ...

UNIVERSITY OF PALERMO POLYTECHNIC SCHOOL

Department of Industrial and Digital Innovation (DIID) Computer Science Engineering for Intelligent Systems

Design and Implementation of Modules for the Extraction of Biometric Parameters in an Augmented BCI Framework Master Degree Thesis of:

Salvatore La Bua

March, 2017

WWW.SLBLABS.COM

Introduction ▪What ◦ Investigate the effects of the interaction with a robotic agent on the mental status of the human player through brain signal analysis ◦ Acceptance of a robotic agent by the user ◦ Performance improvements over a classical BCI system

▪How ◦ Rock-Paper-Scissors game integration ◦ UniPA BCI Framework based on the P300 paradigm ◦ Augmented by ◦ Eye gaze coordinate acquisition ◦ Biometric feature extraction

S. La Bua

DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK

2

Introduction

Human-Robot Interaction (HRI) ▪HRI as a multidisciplinary research topic ◦ Artificial Intelligence ◦ Human-Computer Interaction ◦ Natural Language Processing ◦ Social Sciences ◦ Design

▪Model of the user’s expectation towards a robotic agent in a human-robot interaction

S. La Bua


3

Introduction

Brain-Computer Interfaces (BCI) ▪Direct communication between brain and external devices ◦ Non-Invasive ◦ Partially-Invasive ◦ Invasive

▪Brain Lobes ◦ Frontal: ◦ Temporal: ◦ Parietal: ◦ Occipital:

emotions, social behaviour

speech, hearing recognition sensory recognition

visual processing

▪Extraction of biometric features from brain signals S. La Bua


4

Introduction Visual Focus

▪Importance of eye gaze for direct interaction in a social environment ▪Interfaces dedicated to people affected by degenerative pathologies

▪Entertainment applications, such as games ▪Better advertisement placement

S. La Bua


5

Methodology

Background Information ▪Problem ◦ Effects of the behaviour of a robotic agent on the brain signals ◦ Trust context in Human-Robot Interaction

▪Feature Extraction ◦ Entropy: ◦ Energy: ◦ Mental Workload:

as a stress indicator as a concentration indicator as an index of engagement in the task

▪Brain waves types ◦ δ Delta: ◦ θ Theta: ◦ α Alpha: ◦ β Beta: ◦ γ Gamma: S. La Bua

Hz 0.5÷3 Hz 3÷8 Hz 8÷12 Hz 12÷38 Hz 38÷42

related to instinct, deep sleep related to emotions related to consciousness related to concentration, stress related to information processing


6

Methodology

The math behind Entropy:

𝐸 𝑠𝑖 = 𝑠𝑖2 log (𝑠𝑖2 ); 𝐸 𝑠 = − σ𝑖 𝐸 𝑠𝑖 Energy:

∞

𝐸𝑠 = ෍ 𝑥(𝑛)

2

𝑛=−∞

Mental Workload:

S. La Bua

𝛽𝑝𝑠𝑑 𝛼𝑝𝑠𝑑 + 𝜃𝑝𝑠𝑑 DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK

7

The Proposed Solution Architecture Structure

▪Action Selection ◦ ◦ ◦ ◦

Direct interface with the user Acquisition of bio-signals Acquisition of eye gaze coordinates Selection of the Base action

▪Feature Extraction and Analysis ◦ ◦ ◦ ◦

Bio-signals analysis Features extraction Features analysis Computation of Intention, Attention, Stress indices

▪Response Modulation ◦ Threshold of the Base action by means of the Intention index ◦ Modulation of the resulting action by means of Attention and Stress indices S. La Bua


8

The Proposed Solution Class Diagram

S. La Bua


9

The Proposed Solution Functional Blocks

Action Selection ◦ Eye-Tracking module ◦ Screen coordinates acquisition

◦ Weighing module ◦ Weighing of the BCI classifier response precision and the Eye-Tracking module response precision, by means of the user’s skill level

◦ ID Selection module ◦ Action selection by means of the weighted BCI classifier and Eye-Tracking module precisions S. La Bua


10


Feature Extraction and Analysis ◦ It makes use of external calls to the MATLAB engine ◦ Features extracted and analysed ◦ Correlation Factor: ◦ Energy: ◦ Entropy:

S. La Bua

related to the Intention index related to the Attention index related to the Stress index


11


Response Modulation ◦ Threshold module ◦ ID Selection validation by means of Intention index thresholding

◦ Modulation module ◦ In the case the selected ID has passed the validation step, the resulting action is modulated by means of the Attention and Stress indices

S. La Bua


12

The Proposed Solution Robotic Controller

S. La Bua


13

The Proposed Solution Utilisation Modes

Basic Mode ◦ Simplest mode

◦ Minimal number of modules involved ◦ Classical BCI approach ◦ P300 paradigm classification ◦ Direct Behaviour

S. La Bua


14


Hybrid Mode ◦ Advanced mode

◦ Eye-Tracking module ◦ Combination of brain signals and eye gaze ◦ User skill level as weighting parameter ◦ Composite Behaviour

S. La Bua


15


Bio-Hybrid Mode ◦ Complete mode

◦ Feature Extraction and Analysis functional block ◦ Response Modulation functional block ◦ Intention, Attention and Stress indices computation ◦ Modulated Behaviour

S. La Bua


16

Architecture

Eye-Tracking module P300 6x6 spelling matrix

S. La Bua

3x3 spelling window areas


17

Architecture

Eye-Tracking module Preliminary tests results SUBCATEGORIES FOR SINGLE ELEMENT

3-BY-3, 700X700PX 3-BY-3, 300X300PX 6-BY-6, 700X700PX 6-BY-6, 300X300PX SUBCATEGORIES FOR ROW SPAN SELECTION

3-BY-3, 700X700PX 3-BY-3, 300X300PX 6-BY-6, 700X700PX 6-BY-6, 300X300PX

AVERAGE BY PARAMETER 700X700PX 300X300PX GAIN WITH LARGER WINDOW AVERAGE BY PARAMETER 3-BY-3 6-BY-6 GAIN WITH LESS DENSE MATRIX

S. La Bua

FOCUS %

CENTRAL FOCUS %

LATERAL FOCUS %

EXTERNAL FOCUS %

100 98.4562 100 99.5997

99.9000 93.2697 84.7408 75.9943

0.1000 6.7303 2.7592 24.0057

0 1.5438 0 0.4003

FOCUS %

CENTRAL FOCUS %

LATERAL FOCUS %

EXTERNAL FOCUS %

74.2632 77.1340 69.5037 75.0674

93.9192 89.9075 96.3287 71.7202

6.0808 10.0925 3.6713 28.2798

25.7368 22.8660 30.4963 24.9326

FOCUS % 85.9417 87.5643 -1.8530% FOCUS % 87.4634 86.0427 +1.6512% DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK

CENTRAL FOCUS % 93.7222 82.7229 +13.2966% CENTRAL FOCUS % 94.2491 82.1960 +14.6639%

18

Architecture

Data Structures

Generic signal data structure fields CH 1

CH 2

···

CH N

A

B

C

N fields dedicated to the brain signals acquisition ◦ Ch 1 – Ch 16

3 auxiliary fields to carry peculiar information ◦ A, B, C

S. La Bua


19

Architecture

Data Structures Baseline Calibration signal

BASELINE CALIBRATION

S. La Bua

A (RED)

B (CYAN)

C (MAGENTA)

-2

EYES STATUS

0


20

Architecture

Data Structures Game Session signal

GAME SESSION

S. La Bua

A (RED)

B (CYAN)

C (MAGENTA)

TRIAL STATUS

TRIAL SUB-PHASE

GAZE TRACKING


21

Architecture

Data Structures

P300 Calibration signal P300 Calibration

A

B

C

Calibration target

Flashing tag

0

A

B

C

-1

Flashing tag

0

P300 Spelling signal P300 Spelling

S. La Bua


22

The Framework Main Interface

1. Basic settings ◦ P300-related settings ◦ Preset modes

3

1

2. Main functionalities

4

◦ Signal quality check ◦ P300 Calibration and Recognition ◦ Game session control

3. Interface modality ◦ Alphabetic or Symbolic

5

2

4. Devices ◦ Eye-Tracker settings

5. Plots and Indicators

6

◦ Signals and Indices visualisation

6. Output panel ◦ Feedback for the operator S. La Bua


24

The Framework

Baseline Acquisition Interface Control dialog window

S. La Bua

User dialog window


25

The Framework

Game Session Interface 1. Game modality 1

◦ Fair ◦ Cheat-to-Win/Lose

2. Trials number per session ◦ Initial Fair sub-session ◦ Middle Cheating sub-session ◦ Terminal Fair sub-session

2

3

4

3. Devices ◦ BCI signal acquisition ◦ Kinect gesture recognition ◦ Play against a robotic agent

4. Session panel ◦ Moves selection ◦ Trial temporal progress S. La Bua


27

Experiments Introduction

▪Purpose ◦ Investigate the effects of the interaction with a cheating robotic agent on the mental status of the human player ◦ Rock-Paper-Scissors game session

▪Scenarios ◦ The robot behaves according to the game’s rules ◦ The robot exhibits a cheat-to-win behaviour ◦ The robot exhibits a cheat-to-lose behaviour

▪Game Session Initial Fair sub-session S. La Bua

Cheating sub-session

Terminal Fair sub-session


28

Experiments Set-up

Subjects ◦ 16 Subjects ◦ Aged 18-51

Hardware ◦ g.tec g.USBamp ◦ g.tec g.GAMMAbox

◦ g.tec g.GAMMAcap2 ◦ Secondary standard PC screen ◦ Tobii EyeX eye tracker

◦ Kinect for Xbox One ◦ Telenoid

◦ Camera(s) S. La Bua


29

Experiments

EEG Electrodes configuration Channels-Electrodes correspondence L

R

Ch 01

F7

Ch 09

F8

Ch 02

F3

Ch 10

F4

Ch 03

FZ

Ch 11

T4

Ch 04

T3

Ch 12

C4

Ch 05

C3

Ch 13

T6

Ch 06

T5

Ch 14

P4

Ch 07

P3

Ch 15

PZ

Ch 08

O1

Ch 16

O2

S. La Bua


30

Experiments Protocol

S. La Bua


31

Experiments Protocol

S. La Bua


32

Experiments Subcategories

▪Sub-Session Analysis ◦ Analysis of the Baseline signal, Fair and Cheating sub-sessions

▪Trials Analysis ◦ Single trial analysis for each subject

▪Intra-Class Comparison ◦ Comparison between Cheat-to-Win and Cheat-to-Lose classes

▪Average Analysis ◦ Average over all subjects, by class and by sub-sessions S. La Bua


33

Experiments

Sub-Session Analysis Entropy

Energy

S. La Bua


34

Experiments

Sub-Session Analysis Mental Workload

S. La Bua

Visual Focus %


35

Experiments Trials Analysis

Summary: Entropy Energy Mental Workload

Visual Focus % S. La Bua


36


Entropy: Cheat-to-Win

S. La Bua

Cheat-to-Lose


37


Energy:

S. La Bua

Cheat-to-Win

Cheat-to-Lose


38


Workload: Cheat-to-Win

S. La Bua

Cheat-to-Lose


39


Focus %: Cheat-to-Win

S. La Bua

Cheat-to-Lose


40

Experiments

Intra-Class Comparison Entropy: Cheat-to-Win

Cheat-to-Lose

S. La Bua


41

Experiments

Intra-Class Comparison Energy: Cheat-to-Win

Cheat-to-Lose

S. La Bua


42

Experiments

Intra-Class Comparison Mental Workload: Cheat-to-Win

Cheat-to-Lose

S. La Bua


43

Experiments

Intra-Class Comparison Visual Focus percentage: Cheat-to-Win

Cheat-to-Lose

S. La Bua


44

Experiments

Average Analysis Entropy

ENTROPY

CHEAT WIN CHEAT LOSE

FAIR 1 MEAN 3.8584 3.7420

CHEAT STD DEV 0.2191 0.0850

MEAN 3.8998 3.7632

FAIR 2 STD DEV 0.2540 0.1177

MEAN 3.8742 3.7304

STD DEV 0.1891 0.1074

The entropy values do not show any particular evidence of stress S. La Bua


45

Experiments

Average Analysis Energy

ENERGY


FAIR 1 MEAN 0.2572 0.1498


MEAN 0.3032 0.1720

FAIR 2 STD DEV 0.2267 0.0948

MEAN 0.2254 0.1143

STD DEV 0.1951 0.0447

The energy values show higher concentration level for the Cheat-to-Win class S. La Bua


46

Experiments

Average Analysis Mental Workload

MENTAL WL


FAIR 1 MEAN 1.3798 1.0923


MEAN 0.8988 1.0382

FAIR 2 STD DEV 0.4215 0.3229

MEAN 0.9437 1.0777

STD DEV 0.4570 0.3936

The mental workload values show a slightly lower engagement level for the Cheat-to-Win class S. La Bua


47

Experiments

Average Analysis Visual Focus

FOCUS %


FAIR 1 MEAN 7.89100 4.59710

CHEAT STD DEV 8.93670 9.91690

MEAN 9.13020 3.24540

FAIR 2 STD DEV 11.3344 7.09430

MEAN 12.1404 2.20110

STD DEV 20.1567 4.79480

The visual focus values show higher visual attention level for the Cheat-to-Win class S. La Bua


48

Experiments Demo

S. La Bua


49

Conclusions and Future Works ▪A robotic agent that cheats to win is perceived as more agentic and human-like than a robot that cheats to lose

▪Some of the Questionnaire results Strongly Disagree

Strongly Agree

Unusual Behaviour

Fair Play

Intelligence

▪Trust related improvement ◦ Biometric features to mitigate or amplify the effects of the robotic agent behaviour on the subject’s emotional response S. La Bua


50

Future Works

Framework Extension

Sensor Aggregation functional block ◦ Galvanic Skin Response (GSR) sensor ◦ Heart Rate (HR) sensor ◦ Other physiological sensors

S. La Bua


51

Future Works

Extended Framework

S. La Bua


52

Thank you for your attention Salvatore La Bua [email protected] m

WWW.SLBLABS.COM

S. La Bua - Design and Implementation of Modules for ...

S. La Bua - Design and Implementation of Modules for ...

Suggest Documents

Design and Implementation of Fishery Modules in ... - Description

Design and Implementation of Fishery Modules in ... - Description

A. Forms BUA-1, BUA-2, BUA-3, and BUA-4.

Salvatore La Bua CV - SLB Labs

BUA

s. Baer and s. Rickart Modules

Bua: III. - Global Journal of

Design and Analysis of Mirror Modules for IXO and Beyond

Design and Implementation of Visualization Techniques for ...

Distributed S-Net Design and Implementation - Homepages of UvA ...

Design and Implementation of an Environment for

Design and implementation of assertions for the

Design and Implementation of an Environment for

Design and Implementation of Mobile Portal for

Design of Analysis Modules

Design of Analysis Modules

Robust Design Modules

Bua Komanisi! - SACP

DESIGN AND IMPLEMENTATION OF WINDOWED

design and implementation of performance

Design and Implementation of Standalone

DESIGN AND IMPLEMENTATION OF SMART

Analysis, Design, and Implementation

Database Design and Implementation