Miltimodal Interactions Toy Robot

Galley Proof

8/11/2010; 13:17

File: nre613.tex; BOKCTP/wyn p. 1

1

NeuroRehabilitation 27 (2010) 1–7 DOI 10.3233/NRE-2010-0613 IOS Press

Multimodal interactions in free game play of children with autism and a mobile toy robot Irini Giannopulu∗ and Gilbert Pradel Laboratory of Computer Science, Integrative Biology and Complex Systems, University of Evry-Genopole, France

Abstract. Autism is a complex neuropsychological disorder characterized by qualitative alterations in social interaction and interpersonal communication. The aim of this study is to estimate the interaction between autistic children and a mobile toy robot during free spontaneous game play. The duration of different criteria including eye contact, touch, manipulation, and posture have been considered. The variety of interactions of children with autism and the mobile toy robot show that the children take an interest in playing with the robot. This study suggests that the mobile toy robot in an ecological situation such as free, spontaneous game play could be used as a mediator of social stimuli in order to reduce the impairment of autistic children skills related to social information understanding and interaction. Keywords: Autism, neural functioning, mobile toy robot, free spontaneous game play

1. Introduction Infantile autism is a severe neuropsychological disorder, characterized by three symptoms: i) repetitive and stereotypical behavior generally apparent by 3 years of age; ii) impairment in communication; iii) quantitative impairment in social interaction [13]. The current diagnosis of autism is based on family history, behavioural observation and assessments [1,7,23,35]. It’s genetic and neurocognitive aetiology is unknown though different hypotheses have and continue to be discussed. Autism is considered a complex multifactor disorder involving many genes [19,36,37]. These findings have given rise to new insights into neuronal circuits relevant to autism disorders. Neuroimaging studies have reported abnormal activity in the exterior [28,40] and the interior [9] temporal lobe which could explain impairment in the processing of non-verbal sounds [6], human voices [4], facial information [8] comprehension of other people’s intentions and [2] social function∗ Address for correspondence: Irini Giannopulu, Laboratory of Computer Science, Integrative Biology and Complex Systems, IBISC FRE CNRS 3190, University of Evry-Genopole, 40 Rue du Pelvoux, 91020 Evry Cedex, France. E-mail: [email protected].

ing [3,5,24]. This data could explain disorders in interpersonal interaction and social communication, which characterize autism. From a developmental viewpoint, the most widely accepted hypothesis in autism is the theory-of-mind deficit [18,33]. Even if this theory cannot account for the whole spectrum of autistic disorders, it raises many issues which not only involve mental representation of others but also social skills such as posture [22], eye contact [32], touching [17], and manipulating [15] that express social interaction [38]. Researchers have understood from early on the importance of the challenge that is, to facilitate the development of social interaction [31] in children with autism. Game play is a very important feature of early childhood and is of particular importance for children with autism. It could potentially increase quality of life, learning skills and social inclusion. However, play in children with autism is more like “learned routine” rather than “spontaneous” [39]. Autistic children display difficulty in their play activities, which could be associated with their retardation in communication and social development [16]. Free game play characterized by spontaneity could allow children with autism the possibility to express themselves and engage in satisfying social activity, which in turn could lead to devel-

ISSN 1053-8135/10/$27.50 ! 2010 – IOS Press and the authors. All rights reserved

Galley Proof

2

8/11/2010; 13:17


I. Giannopulu and G. Pradel / Multimodal interactions in free game play of children with autism and a mobile toy robot Table 1 General characteristics of population subjects child 1 child 2 child 3 child 4

a Childhood

Chronological age 7.11 8.6 9.5 8.2

sex M M F M

C.A.R.Sa 46.5 35.5 31.5 43.5

10b

ICD 20 to 34 35 to 49 20 to 34 20 to 34

Autism Rating Scale; classification of diseases.

b International

opment of their social skills. Many options are being explored to improve capacity for social interaction, one of which is to simplify the elements that constitute the interaction. Robots, virtual environments as well as other computer-based technologies are being increasingly utilized in game play and the education of autistic children. These types of environments seem to be more effective than real environments [10]. A variety of studies have been conducted. Though some of these are exploratory in nature, their results are encouraging. The Aurora project study investigates the use of robots (Labo-1, Kaspar, Robota doll, for example) in game play. The aim is to create a tool based on an autonomous robot that convinces autistic children to engage in a process of interaction [11,12,34]. The interactions are tested through the analysis of visual contact, joint attention, avoidance or fleeing, visual pursuit, and whether the child imitates the robot [14]. Using a variety of modalities for interaction such as music, color and visual contact, a sensitive robot named Tito was employed in social interaction with autistic children [25,26]. A very small fixed robot named Keepon captured and maintained visual contact with the child, drawing his attention and initiating some element of conversation [20,21]. Roboto used the form of an animated face (mouth, eyebrows, eyes) that, can cause behavior imitation from the part of the autistic child [27]. Taken together, these studies have shown that animate robots, humanoid or not, using different stimulations, encourage interaction in autistic children. Nonetheless, to date, only narrative descriptions of child-robot interaction have been reported. With the exception of Labo-1 in the Aurora project, so far, only fixed robots have been utilized reducing the child’s spontaneity and self – expression in game play. Our study, which is part of the Robautistic project developed in France, is focused on ways in which a mobile toy robot can engage autistic children in interactive activities and free, spontaneous game play. The interaction between the child and a mobile toy robot named “GIPY-1” which incites the child to engage in free spontaneous game play is analyzed. We hypothesized that the autistic child will

be in quasi-constant interaction with “GIPY-1” using a variety of ways to play with the robot. 2. Methods 2.1. Subjects Four children (3 boys and 1 girl) participated in this study. Their chronological ages ranged from 7 to 9 years old (mean 8.3 years). Their developmental age ranged from 2 to 4 years old. The children were diagnosed according to the Diagnostic and Statistical Manual of Disorders IV criteria of autism [13]. The Childhood Autism Rating Scale [35] had been administrated at the age of 6 years by an experienced clinical psychologist. The international classification of diseases was used to estimate intellectual disability (Table 1). At the time of the experiment all of the children were attending special education classes for autism. The study was approved by the local Ethical Committee and conformed to the Helsinki convention. All the parents were formally informed and agreed to the participation of their children in this study. Anonymity was guaranteed. The study was conducted in a day hospital. The experiment took place in a familiar room. A digital camera recorded the whole experiment. 2.2. Material 2.2.1. Room The room was 4.56 m by 3.34 m. A chair, a small wardrobe and a table on which the equipment needed for the framework of the study was placed (laptop and joystick), were used. In order to reduce the presence of elements that might disturb and avoid autistic bend, the room was left bare [30]. 2.2.2. Robot The IBISC home made mobile robot, called “GIPY1”, is cylindrical-shaped with a diameter of 20 cm and a height of 30 cm. A representation of a neutral facial expression constitutes the cladding of the robot: the round eyes and nose triangle were dyed olive green and the elliptical mouth was dyed red (Fig. 1). Everything was covered with a transparent plastic sheet. The simplicity of the robot was driven by the preference of autistic children for simple and predictable toy design [29]. An operator manipulated the robot via a wireless remote control using a joystick connected to a laptop. The robot could move forward, backward and turn on itself at low speed. These movements were did not vary from child to child.

Galley Proof

8/11/2010; 13:17


I. Giannopulu and G. Pradel / Multimodal interactions in free game play of children with autism and a mobile toy robot

3

Fig. 1. GIPY 1.

2.2.3. Protocol The duration of the session was 5 minutes. An observer was always present in the room. The robot was put on the ground in the center of the room beforehand, its stylized face toward the entrance. The game play session began in exactly the same way for each child. When the child and tutor entered in the room, the tele-operated robot carried out three movements (move forward, move back, 360◦ swivel). As in real social interaction, the child and the robot altered their responses. If the child approached, the robot moved back and conversely. If the child moved away from the robot, i.e., ignored the robot, the robot followed the child in order to attract his/her attention. If the child remained motionless, the robot approached or turned around in order to focus the attention of the child. All the movements were standardized across the children. 2.2.4. Analysis Two independent raters unfamiliar with the aim of the study completed the observations of the game play skills. Both performed the analyses of video sequences with Elan software. Prior to assessing game play improvement, inter-rater reliability was assessed to ensure that both raters who analyzed videotapes were consistent in their analyses. Inter-rater reliability was assessed using intra-class coefficients to make the comparison between them. The dependant variable was the time of child-robot interaction. This time was defined as the duration between the onset time and the offset time of each child’s behavior toward the robot. Four criteria were defined. These criteria were: 1) eye contact (looking at the

robot), 2) touch (touching the robot without manipulating it), 3) manipulation (operating the robot), and 4) posture (changing corporal position toward the robot). The duration of each criterion was calculated in seconds and was considered independently of the others. Concerning, for example, the characteristic “s/he looks at the immobile robot” (“eye contact”) the onset time corresponded to the time when the child looked at the robot and the offset time to the moment when the child looked away from the robot. Accordingly, we calculated the duration of all the characteristics of each criterion. We summed up the duration corresponding to each criterion for each child. Only the total duration is presented in the results section. 3. Results Table 2 gives the characteristics of each criterion identified in game play of children with autism and the robot. The total duration of robot-child interaction is given in Fig. 2. The mean time of child-robot interaction is 238,7 sec. In other words, the children spent more than 79% of their time (156 seconds for the first, 289 seconds for the second, 269 seconds for the third and 241 seconds for the forth child) playing with the robot. The duration of each robot-child interaction is presented in Fig. 3. The duration of “eye contact” is similar for all the children. However the analysis of the duration of “touching”, “manipulating” and “posture” clearly highlights inter individual differences. This analysis also showed how autistic children’s behavioral interaction with the robot changes over a period of time.

Galley Proof

4

8/11/2010; 13:17


I. Giannopulu and G. Pradel / Multimodal interactions in free game play of children with autism and a mobile toy robot Table 2 Characterization for each criterion Touch s/he puts the left hands on the robot; s/he puts the right hand on the robot; s/he touches the robot with both hands

Eye contact s/he looks at the immobile robot; s/he watches the robot turning; s/he watches the robot going away; s/he watches the robot approaching

Manipulation s/he seizes and blocks the robot with the two hands; s/he lifts of the robot; s/he stops the robot with both hands; s/he catches the robot; s/he returns the robot; s/he tilts the robot around itself and looks at its wheel; s/he puts back the robot upright

Posture s/he sits downs in front of robot; s/he bends towards the robot; s/he bends over the robot; s/he squats and bends over the robot; s/he steps over the robot

Fig. 2. Total duration of child-robot interaction.

4. Discussion The aim of this study was to analyze the interaction between autistic children and a mobile toy robot in free, spontaneous game play. The genetic and neurocognitive aetiology of autism and the importance of play in autistic children have already been highlighted. Consistent with our hypothesis, the children are quasiconstantly in interaction with the mobile robot using a variety of ways. As autism is a spectrum disorder where a large variation in abilities and interests among autistic children is apparent, the interaction of children and robots was evaluated on the level of each individual child. Consistent with various studies, the present study shows that the use of robots engages autistic children in interaction [10,12,14,20,21,25–27,34]. But contrary to previous studies where only a narrative description of robotchild interaction has been presented, we have calculat-

ed the duration of robot-child interaction during free, spontaneous game play. More precisely, the behavior of autistic children visa-vis the robot based on four criteria (“eye contact”, “touch”, “manipulation” and “posture”) has been analyzed and a temporal quantification of child-robot interaction with respect to the duration performed. The analysis revealed that the duration of eye contact behavior was similar for each child. Inter-individual differences were identified for the duration of “touching”, “manipulating” and “posture” behavior. This data demonstrated that the autistic children not only visually explored the robot but also engaged in different kinds of play with the robot. In other words, the autistic children clearly took an interest in playing with the mobile robot. In all the studies we have mentioned above, only fixed robots were used, with the exception of Labo1 [11]. In our study as with Labo-1 [11], the autistic children were invited to interact with the robot during free, spontaneous game play. Taken together, both stud-

Galley Proof

8/11/2010; 13:17



5

Fig. 3. Duration of child-robot interaction following each criterion.

ies have shown that autistic-children used a variety of behavior when playing with the robot in free game play. It seems that free game play could be a relevant ecological situation where an autistic child spontaneously interacts with the robot. These findings tell us that an ecological situation, i.e., free game play, encourages an autistic child to interact with the robot in a spontaneous manner. They also reveal that free, spontaneous game play with robots is possible with autistic children and could better facilitate the transfer of social and learnt abilities to real life. However, future robot development and longitudinal studies using a more statistically significant number of autistic children will tell us if the facilitation of real social interaction is attainable. What is important to demonstrate is whether and how autistic children can generalize learnt abilities during play with the robot to therapists and parents, i.e., proving that the robot could be used as a social mediator tool for the enrichment of child-human interaction. The robot-autistic child interaction is a growing and challenging area, which requires interdisciplinary collaboration between robotics scientists, computer scientists, engineers, and neuropsychologists. To develop this area, a multidisciplinary scientific approach is required. New experimental paradigms and scenarios should be investigated as we need to build new theories, computational models and methodologies in order to be able to replicate and confirm these scientific findings.

As autism is a neuropsychological disorder, brain abnormalities and brain-behavior connections at earlier stages of the disorder could be analyzed and should be considered when taking into account inter individual behaviors. To create rehabilitation tools such as a social mediator tool using robots for autistic children, we need to determine physiological, electrophysiological, psychophysiological and behavioral factors that can predict performance in social interactions while exploring irregularities of brain activity and brain connectivity. A robot could also be the appropriate tool to develop realistic behavior and emotional reactions in numerous social contexts in autism as well as in other related pathologies. Certainly, a robot could be used as a social stimuli mediator, with the ability to activate the same brain areas sensitive to humans in order to reduce the impairment of skills related to social and emotional information processing. To conclude, our results contribute to the development of a new researcher ecological design paradigm for a better understanding of autistic child and robot interaction, which could help autistic people improve their social interaction. Acknowledgments The authors thank ANR for its support. It is possible to find different information concerning the members of RobAutiSTIC as well as the project on http://robautistic.ibisc.univ-evry.fr.

Galley Proof

6

8/11/2010; 13:17


References [1] [2]

[3] [4]

[5]

[6]

[7]

[8]

[9]

[10] [11]

[12]

[13] [14]

[15]

[16]

[17]

[18]

[19]


J.L, Adrien, BECS: Batterie d’évaluation cognitive et socioe´ motionnelle, De Boeck Université, 2008. T. Allison, A, Puce and G, McCarthy, Social perception from visual cues: the role of STS region, Trends of Cognitive Sciences 4 (2000), 267–278. S. Baron-Cohen, A.M. Leslie and U. Frith, Does the autistic child have a theory of mind? Cognition 21 (1985), 37–46. P. Belin, R.J. Zatorre, A. Lafaille, P. Ahad and B. Pike, Voice selective areas in auditory human cortex, Nature 403 (2000), 309–312. M.K. Belmonte, G. Allen, A. Beckel-Mitchener, L.M. Boulanger, R.A. Carper, J. Sara and S.J. Webb, Autism and Abnormal Development of Brain Connectivity, The Journal of Neurosciences 42 (2004), 9228–9231. N. Boddaert, P. Berm, J.B. Poline, N. Chabanneg, M.C. Mouren-Shneoni, C. Barthelemy, Y. Samson and M. Zilboviciw, Temporal lobe dysfunction in autism: a PET auditory activation study, Neurolmage 13 (2001), S0128. Y. Bourreau, M. Gonnot, S. Roux and C. Barthélemy, Construction d’une e´ chelle d’évaluation des comportements répétitifs et restreints dans l’autisme (EC2R): Etude préliminaire, Bulletin Scientifique ARAPI 17 (2006), 62–64. F. Castelli, Understanding emotions from standardized facial expressions in autism and normal development, Autism 4 (2005), 428–449. M.K. Chung, K.M. Dalton, A.L Alexander and R.J. Davidson, Less white matter concentration in autism: 2D voxel-based morphometry, Neuroimage 23 (2004), 242–351. K. Dautenhahn and I. Werry, Towards interactive robots in autism therapy, Pragmatics and Cognition 121 (2004), 1–35. K. Dautenhahn, Socially intelligent robots: dimensions of human-robot interaction, Philosophical Transaction of the Royal Society B 362 (2007), 679–704. M. Davis, B. Robins, K. Dautenhahn, C.L. Nehaniv and S. Powell, eds, A comparison of interactive and robotic systems in therapy and education for children with autism. European Conference for the Advancement of Assistive Technology in Europe (AAATE’05) September 6-9 2005 IOS Press Lilli, pp. 353–357. DSM-IV-TR Manuel diagnostique et statistique des troubles mentaux, Paris, Editions Masson, 2003. A. Duquette, F. Michaud and H. Mercier, Exploring the use of a mobile robot as an imitation agent with children with low-functioning autism, Autonomous Robots – Special Issue on Socially Assistive Robotics 2 (2004), 147–157. A. Escalona, T. Field and J. Nadel, B. Lundy, Brief report: Imitation effects on children with autism, Journal of Autism and Developmental Disorders 32 (2002), 141–144. C. Ferrara and S.D. Hills, The responsiveness of autistic children to the predictability of social and non-social toys, Journal of Autism Development Disorders 10 (1980), 51–57. U. Frith and C.D. Frith, Development and neurophysiology of mentalizing, Philosophical Transaction of the Royal Society B Biological Science 358 (2003), 459–473. U. Frith, J. Morton and A.M. Leslie, The cognitive basis of a biological disorder: autism, Trends in Neurosciences 14 (1991), 433–438. L. Jorde, S. Hassted, E. Ritvo, A. Mason-Brothers, B. Freeman, C. Pingree, W.M. MacMahon, B. Petersen, W.R. Jenson and A. Mo, Complex segregation analysis of autism, American Journal of Human Genetics 49 (1991), 932–938.

[20]

[21]

[22] [23]

[24]

[25] [26]

[27]

[28]

[29]

[30]

[31]

[32]

[33] [34]

[35]

[36] [37]

H. Kozima, C. Nakagawa and Y. Yasuda, Children-robot interaction: a pilot study in autism therapy, Progress in Brain Research 164 (2007), 385–400. H. Kozima and H. Yano, Designing a robot for contingencydetection game. Working Notes Workshop Robotic and Virtual Interactive Systems in Autism Therapy; 2001 27–28 September; U.K. Hatfield. University of Hertfordshire, 2001. Technical Report No 364. A.M. Leslie, Pretending and believing: issues in the theory of ToMM, Cognition 50 (1994), 211–238. C. Lord, B.L. Leventhal, H. Edwin and J. Cook, Quantifying the phenotype in autism spectrum disorders, American Journal of Medical Genetics Part B: Neuropsychological Genetics 105 (2001), 36–38. J. Martineau, S. Cochin, R. Magne and C. Barthelemy, Impaired cortical activation in autistic children: Is the mirror neuron system involved? International Journal of Psychophysiology 68 (2008), 35–40. F. Michaud and S. Caron, Roball, the rolling robot, Autonomous Robots 12 (2002), 211–222. F. Michaud, T. Salter, A. Duquette and J.F. Laplante, Perspectives on mobile robots used as tools for pediatric rehabilitation, Assistive Technology-Special Issue on Intelligent System in Pediatric Rehabilitation 19 (2007), 14–19. J. Nadel, A. Revel, P. Andry and P. Gaussier, Toward communication: first imitations in infants, low-functioning children with autism and robots, Interaction Studies: Social Behaviour and Communication in Biological and Artificial System 5 (2004), 45–54. T. Ohnishi, H. Matsuda, T. Hashimoto, T. Kunihiro, M. Nishikawa, T. Uema and M. Sasaki, Abnormal regional blood flow in childhood autism, Brain 123 (2000), 1838–1844. J. Pedersen and J. Schelde, Behavioral aspects of infantile autism: an ethological description, European Child and Adolescent Psychiatry 6 (1997), 96–100. P. Planche, E. Lemonnier, K. Moalic, C. Labous and A. Lazartigues, Les modalités de traitement de l’information chez les enfants autistes, Annales Médico-Psychologiques 160 (2002), 559–564. L.M. Quirmbach, A.J. Lincoln, M.J. Feinberg-Gizzo, B.R. Ingersoll and S.M. Andrews, Social stories: mechanisms of effectiveness in increasing game play skills in children diagnosed with autism spectrum disorder using a pre-test post test repeated measures randomized control group design, Journal of Autism and Development Disorders 39 (2009), 299–321. T. Reed and C.A. Peterson, comparative study of autistic subjects’ performance at two levels of visual and cognitive perspective taking, Journal of Autism Developmental Disorders 20 (1990), 555–567. J. Russell, (Ed) Autism as a executive disorder. 1997 Oxford:Oxford University Press. T. Salter, I.P. Werry and F. Michaud, Going into the wild in child-robot interaction studies – Issues in social robotic development, International Journal of Robotics – Special Issue on Multidisciplinary Collaboration in Socially Assistive Robotics 1 (2007), 93–98. E. Schopler, R.J. Reichler, R.F. De Vellis and K. Daly, Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS), Journal of Autistic Development Disorders 10 (1980), 91–103. N.H. Sykes and J.A. Lamb Autism: The quest for the genes, Experimental Revue of Molecular Medecine 9 (2007), 1–15. P. Szatmari, A.D. Paterson, L. Zwaigenbaum, W. Roberts et al., Mapping autism risk loci using genetic linkage and chro-

Galley Proof

8/11/2010; 13:17


I. Giannopulu and G. Pradel / Multimodal interactions in free game play of children with autism and a mobile toy robot mosomal rearrangements, Nature Genetics 39 (2007), 319– 328. [38] B.A. Taylor, H. Hoch, B. Potter, A. Rodriguez, D. Spinnato and M. Kalaigian, Manipulating establishing operations to promote initiations toward peers in children with autism, Research in Development Disabilities 26 (2005), 385–392. [39] E. Williams, V. Reddy and A. Costal, Taking a closer look at

7

functional play in children with autism, Journal of Autism and Developmental Disorders 31 (2001), 67–77. [40] M. Zilbovicius, N. Boddaert, P. Belin, J.B. Poline, P. Remy, J.F. Mangin, L. Thivard, C. Barthelemy and Y. Samson, Temporal lobe dysfunction in childhood autism:a PET study, American Journal of Psychiatry 157 (2000), 1988–1993.