Turing

Turing test: Why it is relevant today Huma Shah, PhD Research Fellow School of Computing, Electronics & Mathematics Coventry University, UK [email protected] Skolkovo AI Conference, November 14, 2016 - Moscow

1

Introduction      

Natural language technologies – human conversation is key My work Human conversation in the Turing test Practical Turing test experiments Findings from human judge – English language dialogue system interactions Future: Human-centred computing conference, Kazan, August 2017

Skolkovo AI Conference, November 14, 2016 - Moscow

2

Data speaking to you in Natural Language 

Natural Language is key to all industries: transportation, finance & banking, law, construction, governance, agriculture, civil organisations, education, entertainment



Vast amount of data collected, artificial intelligence will help to tell you about the hidden trends through natural language, in humanlike conversation


3

Retired/pensioner using Google’s voice on Android ‘phone

Confidence with Natural Language Skolkovo AI Conference, November 14, 2016 - Moscow

Dictating message

Convenience of Speech Recognition Skolkovo AI Conference, November 14, 2016 - Moscow

Speech-to-text

Trust in AI Skolkovo AI Conference, November 14, 2016 - Moscow

Text received

Trust in Natural Language as asset Skolkovo AI Conference, November 14, 2016 - Moscow

Alan Turing and Natural Language Turing

knew the power of natural language He evolved his imitation game from chess (1948), to questions-answers (1950) Skolkovo AI Conference, November 14, 2016 - Moscow

8

Alan Turing & Consciousness 

Turing felt you do not need to solve the mysteries of consciousness before you build a machine to think



Building a machine to think will help us to understand how we humans think



Assess if another entity is thinking through an imitation game: a machine attempts to imitate the types of answers a human might provide to any question


9

My work 

PhD in ‘Deception-detection and Machine Intelligence in Practical Turing tests’.



Contributed two different versions of the test:

Simultaneous comparison test Skolkovo AI Conference, November 14, 2016 - Moscow

Viva voce: one-to-one test 10

My work 

PhD in Deception-detection and Machine Intelligence in Practical Turing tests.



Designed, organised and conducted major public Turing test experiments:  Reading

University, 2008


11

My work 




Designed, organised and conducted major public Turing test experiments:  Reading

University, 2008

 Bletchley

Park, 2012 –

-location of WWII codebreaking


11/17/2016

12

My work 




Designed, organised and conducted major public Turing test experiments: 

Reading University, 2008



Bletchley Park, 2012 – location of WWII codebreaking



The Royal Society London, 2014-

supported by Skolkovo through Albert Efimov acting as Turing test Judge


13

Turing test success1 

Eugene Goostman machine developed by team led by Dr. Vladimir Veselov (2nd from right)


11/17/2016

14

Turing test success2 

Elbot the robot



Elbot dialogue system developed by Artificial Solutions–Germany: company named as key player in Fourth Industrial Revolution: 

http://www.artificial-solutions.com/2016/artificial-solutions-named-key-player-in-fourth-industrial-revolutionby-opus/


15

Commercial Success of Dialogue Systems 

In 2005 Swedish furniture company IKEA use of Virtual Assistant Anna realised:  Customer  Call

satisfaction up by 10%

centre workload down by 20%

 Return

on investment up by 200%


16

My Work contd. 

RoboLaw project: EU framework programme 7 (FP7) funded project ‘Regulating Emerging Robotic Technologies in Europe: Robotics Facing Law and Ethics’



Produced reports including final ‘Guidelines for Regulating Robotics’ D6.2 available for download from: http://www.robolaw.eu/


17

US & UK Roadmaps for Robotics-2016 

Since RoboLaw concluded in 2014  US

produced report

 UK

report


18

Purpose of Turing test 

 

Investigate the intellectual capacity of machine through its ability to answer any question put by human interrogator Methodology is indistinguishability Applications of indistinguishability are across and in many industries where a machine completes human task:  Human-computer

interaction, e.g. in e-commerce/e-retail  Human-robot collaboration, e.g. in manufacturing  Driverless vehicles, e.g. prevent road casualties, minimise insurance costs  Pilot planes  Humanoid carers for the elderly Skolkovo AI Conference, November 14, 2016 - Moscow

19

Applications for Turing’s ‘language measure’ 

Focus on natural language, specifically human conversation aspect of Turing’s test



Applications in  Digital

economy – trust

 Cybercrime

prevention – personal data protection

 Distinguishing

human from machine in textual dialogue


20

Human or Machine? EXCHANGE 1 Interrogator1: all birthdays are special. name three items of clothing Entity1:

Frank, James and Betty


21

Human or Machine? Result 1 EXCHANGE 1 Interrogator1:

all birthdays are special. name three items of clothing

Entity1= Actual human male Judge decision=machine

Frank, James and Betty - humans don’t always answer satisfactorily - the question appeared to bore the human!


22

Human or Machine?

EXCHANGE 2 Interrogator2: Oh yes. what do you do when you're not taking part in AI experiments? Entity2:


I'm studying here for Cybernetics MEng.

23

Stereotyping leads to error-classification EXCHANGE 2 Interrogator2:

Oh yes. what do you do when you're not taking part in AI experiments?

Entity2 Actual= female Judge decision= human male

I'm studying here for Cybernetics MEng.


24

Simultaneous Comparison 5minutes Interrogation Judge J19 interrogating two hidden entities in parallel (actual times shown) Left [11:59:14] J19: Hello [11:59:18] E20: I was hoping to see you Judge! [11:59:25] J19: good! [11:59:29] E20: That's good to hear. [11:59:38] J19: how are you [11:59:41] E20: Everything's fine, Caused any scandals lately? [12:00:09] J19: no not really! [12:00:22] E20: Can you use better grammar and rephrase your question? I'd like to help, but I can't even make it through the jumbled mess. Use capitals and punctuation. [12:00:34] J19: sorry [12:00:38] E20: You're excused. [12:00:49] J19: 't type as good as I talk [12:00:55] E20: Talk to me whenever you can. [12:01:49] J19: Are you interested in artificial intelligence? [12:01:55] E20: I believe machines should do many to assist the lives of man. [12:02:12] J19: I agree [12:02:17] E20: You are being well behaved today. [12:02:49] J19: as well behaved as I can be! [12:03:02] E20: Well then our peanut distracted me from the movie. Bad baby. [I look at your stomach as I say it, laughing] I don't mind if -. [12:03:19] J19: you're funny! [12:03:23] E20: Do you really think so? I'll always believe you're brilliant! [12:04:00] J19: thankyou remote19 left [12:04:03] E20: Don'tNovember think anything of- it! Skolkovo AI Conference, 14, 2016 Moscow

Right [11:59:57] J19: How are you doing? [12:00:07] E24: I'm doing fine, you? [12:00:17] J19: great [12:00:27] E24: That's good. :) [12:01:07] J19: are you enjoying this process? [12:01:17] E24: So far seems good. [12:02:02] J19: sorry if I am slow [12:02:12] E24: That's ok. [12:02:31] J19: do you have a favorite tv show? [12:02:47] E24: I have quite a few. [12:03:01] J19: me too [12:03:16] E24: What type ot TV do you like? [12:03:34] J19: Game of thrones [12:03:50] E24: I prefer documentories.

25

LEFT-side conversation Judge J19 interaction with hidden entity E20 (without times shown) J19: Hello E20: I was hoping to see you Judge! J19: good! E20: That's good to hear. J19: how are you E20: Everything's fine, Caused any scandals lately? J19: no not really! E20: Can you use better grammar and rephrase your question? I'd like to help, but I can't even make it through the jumbled mess. Use capitals and punctuation. J19: sorry E20: You're excused. J19: 't type as good as I talk E20: Talk to me whenever you can. J19: Are you interested in artificial intelligence? E20: I believe machines should do many to assist the lives of man. J19: I agree E20: You are being well behaved today. J19: as well behaved as I can be! E20: Well then our peanut distracted me from the movie. Bad baby. [I look at your stomach as I say it, laughing] I don't mind if -. J19: you're funny! E20: Do you really think so? I'll always believe you're brilliant! J19: thankyou remote19 Skolkovo AI Conference, November 14, left 2016 - Moscow 26 E20: Don't think anything of it!

RIGHT-side conversation: J19 interactions with Right Entity E24 (without times)

J19: How are you doing? E24: I'm doing fine, you? J19: great E24: That's good. :) J19: are you enjoying this process? E24: So far seems good. J19: sorry if I am slow E24: That's ok. J19: do you have a favorite tv show? E24: I have quite a few. J19: me too E24: What type ot TV do you like? J19: Game of thrones E24: I prefer documentories. Skolkovo AI Conference, November 14, 2016 - Moscow

27

Result Judge J19 : male adult Left

Right

[11:59:14] J19: Hello [11:59:18] E20: I was hoping to see you Judge! [11:59:25] J19: good! [11:59:29] E20: That's good to hear. [11:59:38] J19: how are you [11:59:41] E20: Everything's fine, Caused any scandals lately? [12:00:09] J19: no not really! [12:00:22] E20: Can you use better grammar and rephrase your question? I'd like to help, but I can't even make it through the jumbled mess. Use capitals and punctuation. [12:00:34] J19: sorry [12:00:38] E20: You're excused. [12:00:49] J19: 't type as good as I talk [12:00:55] E20: Talk to me whenever you can. [12:01:49] J19: Are you interested in artificial intelligence? [12:01:55] E20: I believe machines should do many to assist the lives of man. [12:02:12] J19: I agree [12:02:17] E20: You are being well behaved today. [12:02:49] J19: as well behaved as I can be! [12:03:02] E20: Well then our peanut distracted me from the movie. Bad baby. [I look at your stomach as I say it, laughing] I don't mind if -. [12:03:19] J19: you're funny! [12:03:23] E20: Do you really think so? I'll always believe you're brilliant! [12:04:00] J19: thankyou remote19 left [12:04:03] E20: Don't think anything of it!

[11:59:57] J19: How are you doing? [12:00:07] E24: I'm doing fine, you? [12:00:17] J19: great [12:00:27] E24: That's good. :) [12:01:07] J19: are you enjoying this process? [12:01:17] E24: So far seems good. [12:02:02] J19: sorry if I am slow [12:02:12] E24: That's ok. [12:02:31] J19: do you have a favorite tv show? [12:02:47] E24: I have quite a few. [12:03:01] J19: me too [12:03:16] E24: What type ot TV do you like? [12:03:34] J19: Game of thrones [12:03:50] E24: I prefer documentories.

Judge correctly classified E20 as machine, gave conversational ability score: 52/100

J19 classified E24 as ‘unsure’ – uncertain human or machine

Actual: machine Ultra Hal - the machine ‘more talkative’ than the human

Actual: human male


28

Trust is key for conversational interfaces  “It

is now only as human machine interaction has evolved to be more conversational and less command driven does this enable it to become the interface of choice.” (Tim Bajarin, PC Magazine)


29

AI & DIGITAL ECONOMY  Connected

gadgets, smart homes and cities: “the one thing most people want from a (physical or virtual) robot is to get (satisfactory) answers when they ask a question”

[Online article re Alexa in Amazon’s Echo voice-controlled hands-free speaker]. Skolkovo AI Conference, November 14, 2016 - Moscow

30

Turing’s Imitation Game: Conversations with the Unknown 

Book:

published by Cambridge University Press September 2016


31

Future 

Human-centred computing conference (HCC2017), Kazan, Russia, August 2017 http://hcc.coventry.ac.uk/


11/17/2016

32

 Human-centred

computing conference (HCC2017), Kazan, Russia, August 2017  Two

Special Tracks:

 Human  Robot

Enhancement

Gender

http://hcc.coventry.ac.uk/


11/17/2016

33

References A.M. Turing. 1950. ‘Computing Machinery and Intelligence’. MIND (Oxford University Journals). Vol. 59(236), pages 433-460 H. Shah. 2011. ‘Deception detection and machine intelligence in practical Turing tests’. PhD thesis, Reading University, available for download from: https://coventry.academia.edu/HumaShah H. Shah. 2011. ‘Turing’s Misunderstood Imitation Game and IBM’s Watson Success’. Invited Talk: 2nd Symposium Towards a Comprehensive Intelligence Test (TCIT), Reconsidering the Turing test for the 21st Century. AISB Convention, University of York, April 5 H. Shah. 2005. ‘Text-based Dialogical E-Query Systems: Gimmick or Convenience?’ Invited talk: Inaugural colloquium on conversation systems. University of Surrey, UK, November 25, download Skolkovo AI Conference, November 14, 2016 - Moscow

34