A Virtual Environment to Improve the Detection of ... - Semantic Scholar

sensors Article

A Virtual Environment to Improve the Detection of Oral-Facial Malfunction in Children with Cerebral Palsy María-Luisa Martín-Ruiz 1, *, Nuria Máximo-Bocanegra 2 and Laura Luna-Oliva 2 1 2

*

Department of Telematic and Electronic Engineering, Technical University of Madrid, Madrid 28031, Spain Department of Physiotherapy, Occupational Therapy, Rehabilitation and Physical Medicine, Faculty of Health Sciences Rey Juan Carlos University, Alcorcón, Madrid 28922, Spain; [email protected] (N.M.-B.); [email protected] (L.L.-O.) Correspondence: [email protected]; Tel.: +34-91-336-3770; Fax: +34-91-336-7817

Academic Editors: Fabrizio Lamberti, Andrea Sanna and Jon Rokne Received: 17 December 2015; Accepted: 21 March 2016; Published: 26 March 2016

Abstract: The importance of an early rehabilitation process in children with cerebral palsy (CP) is widely recognized. On the one hand, new and useful treatment tools such as rehabilitation systems based on interactive technologies have appeared for rehabilitation of gross motor movements. On the other hand, from the therapeutic point of view, performing rehabilitation exercises with the facial muscles can improve the swallowing process, the facial expression through the management of muscles in the face, and even the speech of children with cerebral palsy. However, it is difficult to find interactive games to improve the detection and evaluation of oral-facial musculature dysfunctions in children with CP. This paper describes a framework based on strategies developed for interactive serious games that is created both for typically developed children and children with disabilities. Four interactive games are the core of a Virtual Environment called SONRIE. This paper demonstrates the benefits of SONRIE to monitor children’s oral-facial difficulties. The next steps will focus on the validation of SONRIE to carry out the rehabilitation process of oral-facial musculature in children with cerebral palsy. Keywords: serious games; inclusive education; detection; virtual therapeutic environment; virtual reality

1. Early Detection and Intervention in Children with Cerebral Palsy Cerebral palsy (CP) is a continuous but non-progressive motion/posture, motor function disorder resulting from an injury in the developing brain primarily damaging the areas responsible for the postural control [1–3]. Motor disorders in people with CP are often accompanied by disturbances of sense, perception, cognition, communication, as well as behavioral issues, epilepsy and secondary musculoskeletal problems [4]. CP is the most common cause of severe disability in childhood [5], and can also involve physical, cognitive, emotional and social difficulties [6,7]. The different impairments that affect children with CP make it very hard, or even impossible, to perform many daily activities (getting dressed, personal hygiene, eating, going to school, and even walking and talking). Although the physical and motor difficulties and changes are often the most obvious, cognitive, emotional and behavior impairments are present in 50% to 75% of cases [8]. The effect of this whole set of problems goes beyond the individual; it can also affect the family and, furthermore society. A very important aspect of this disorder, which must be considered in addition to the abovementioned considerations, is that it affects a person as they are maturing, before developmental

Sensors 2016, 16, 444; doi:10.3390/s16040444

www.mdpi.com/journal/sensors

Sensors 2016, 16, 444

2 of 18

milestones have been reached. The acquisition of these milestones enables an individual’s participation in society, which, for the World Health Organization (WHO), would mean reaching a state of health. The injuries that cause CP may occur during pregnancy, childbirth, or in the first few years of life, and may be derived from several different causes, some of which are still unknown. Supportive treatments, medications, and surgery can help many individuals improve their motor skills and ability to communicate with the world [9]. According to the Surveillance of Cerebral Palsy in Europe (SCPE), a network made up of 20 European countries, the incidence of CP is currently 2 to 3 per 1000 births [3]. However, this number is rising, due to an increase in new cases and the rising life expectancy of those affected [5,10]. This rate goes up to 40–100 per 1000 live births among babies born very prematurely or with a very low birth weight [11]. Rates appear to be similar in both the developing and developed world [12]. The rate is higher in males than in females; in Europe it is 1.3 times more common in males [13]. It is, therefore, a primary healthcare problem, and its care must be encompassed in an integral care model, based on a set of interventions focused on the childhood population and its environment [14]. These interventions must have the objective of responding to the needs of children with developmental disorders, or at risk of acquiring them, from both the healthcare and the psychological and social perspectives to strive for the greatest degrees of personal autonomy and family, school and social integration. The importance of therapeutic intervention in CP is well accepted, hence new tools have been developed to create contexts and situations capable of facilitating the interaction between the individual and the environment [1,15]. In fact, the search for solutions that provide benefits for patients treated with Virtual Reality (VR)-based systems has included devices that were initially designed for other uses (Xbox 360, Kinect, Wii), but that have proven useful when applied in the medical world [16,17]. In addition, the low cost of these technologies, compared to specialized hardware, makes them even more attractive. The term VR was introduced by Jaron Lanier in 1986. However, due to the progress of technology, VR has undergone several transformations since then [18]. VR is classified according to the degree of immersion the user experiences. According to Burdea and Coiffet [19], the term immersion is the voluntary act of ignoring the different stimuli that make users perceive the experience as not real, allowing the system to hold users’ full attention and concentration. VR has been widely used in the field of medicine, since it can present challenging situations within a safe environment, while maintaining experimental control of the mediation and presentation of stimuli [20]. The first applications of this kind gave way to a series of psychosocial interventions for the treatment of phobias [21]. Later, its use was expanded into other areas, such as surgical training and neurological rehabilitation, among others [18]. In neuro-rehabilitation, VR is used as a tool for rehabilitation and evaluation in programs for training balance, posture and walking, activation of functions in the upper limbs, exercise and pain tolerance therapies, evaluation of daily activities, and evaluation of visual impairment after a stroke [22,23]. Reviewing the literature, we found the work of Luna et al., presenting a study that had the objective of evaluating the utility of a video game system based on VR technology: the Xbox 360 Kinect [24]. The system developed has the objective of aiding the conventional rehabilitation treatment that is traditionally prescribed to children with CP. The study included 11 children whose motor skills were measured, specifically, balance, speed when walking, running and jumping, and fine motor skills. The participants underwent the treatment with the Xbox 360 Kinect for eight weeks. The statistical study applied the Wilcoxon test [25], which showed significant differences before and after the treatment for all the values measured. In addition, the evolution of these children was monitored for a year, obtaining stable results carried out from the previous evaluation, which proves that this therapy has persistent long-term positive effects. The study performed by Sharan et al. is important because it was the first research to use VR-based therapy (Wii Fit) for post-operatory rehabilitation in children with CP [26]. The main conclusion of this study was that balance improved significantly in children who used this technology during

Sensors 2016, 16, 444

3 of 18

their rehabilitation process. Their participation level, motivation, cooperation and satisfaction during therapy also improved. Other studies of children with neural development problems are focused on autism. The main feature of this pathology is that these children have severe and generalized deficiencies in their social interactions and communication. The work of Blum-Dimaya et al. was focused on showing four children with autism how to use the video game Guitar Hero II [27]. In addition to the game, the children were given an activity chronogram that taught them how to work with the game. As a result, all four participants managed to learn how to play Guitar Hero II. These results were very beneficial, since the musical sounds also functioned as stimuli to enable verbal and non-verbal communication. This is possible as music acts at a neural level. Finally, we must include the results obtained during a systematic review of the literature. The work of Wang and Reid in 2011 contains a systematic review of the existing literature exploring VR systems used as treatment tools that are meant to deal with the main deficiencies of children with attention deficit disorder with and without hyperactivity (ADHD), autism and CP [28]. In this paper, they worked with 20 articles, for which they considered two aspects; the first was the analysis of general benefits of VR in the context of pediatric rehabilitation. The second was the testing of the results obtained in all the studies. An important result of this review was that simple VR systems used at home can help increase adherence to treatment programs in children, because these systems increase the degree of participation in activities and reduce the need to go to rehabilitation centers where conventional therapy is received. Furthermore, they found that the studies performed on children with CP used VR in combination with treadmill training, and the results were that it enabled the rehabilitation of the lower limbs. This fact allowed the children to achieve full immersion in the video game, bringing them into the virtual environment. The meta-analysis performed by Chen et al. intended to systematically evaluate the effect of VR in achieving mobility in the upper limbs, associating its effects to the importance of adhesion to treatment [29]. The conclusion of this review was that VR is a viable tool for improving motor function in the upper limbs of children with CP. Besides, it stated that it is necessary to continue this line of investigation to provide a conclusive recommendation. In this paper, we present a virtual environment to improve the detection and evaluation of oral-facial musculature dysfunctions in children with CP. This article is organized as follows. After this introduction, in Sections 2–4 we describe the Sistema de terapia basadO en KiNect paRa nIños con parálisis cErebral (Virtual Environment using Kinect for children with CP, known by its Spanish acronym SONRIE—“smile” in English), as an intervention and detection system. Sections 5 and 6 present a verification process performed with both typically developing children and children with CP. Finally, we discuss the technical aspects of the verification including the difficulties in the facial recognition of such a varied group of children. Moreover, we include a comparison to current therapies and detail the ethical issues taken into account. 2. SONRIE: Virtual Environment for Oral-Facial Intervention At present, new technologies provide opportunities for the rehabilitation of children with different neurological disorders [23,24,26–30]. Studies about Virtual Reality systems have good results for clinical use in pediatric rehabilitation, especially in CP [1]. In fact, researches focused on using VR system in CP interventions aimed to prove effectiveness of VR on the body structures and functions, activity and participation according to components in the International Classification of Functioning, Disability and Health [1]. Moreover, conventional rehabilitation programs are shorter and less intensive to guarantee optimal therapeutic results. They cannot satisfactorily increase the motivation of the patient or support activity participation [30]. Many studies have shown that the motivation of patients play a critical role in treatment results [24,26,30]. Hence, virtual environments can also provide more intriguing and competitive conditions, by increasing the motivation of the patient and ensuring active participation, so that less time is used to regain motor skills [1,31].

Sensors 2016, 16, 444

4 of 18

SONRIE detects oral-facial difficulties for children with CP between the ages of 4 and 12 by using a virtual environment. SONRIE uses the 360 Kinect sensor and is based on games so it is a low-cost solution. During the developmental years, a child spends a large part of his or her time playing. It is the means and the end through which the children receive the challenges that will enable them to reach the different developmental milestones, thanks to what they can learn through play. The importance of the need to help children with CP to play, whether it is by adapting and modifying games to enable their participation, creating games especially for them, or re-training the abilities that will help them participate in each game. In this case, SONRIE adapts accurately to this theoretical rationale, since it invites children to play within a game-oriented environment, which will induce responses from them that could contribute to the re-training of the oral-facial musculature. In addition, it provides an opportunity for play for children who, due to their pathology, face limitations in the type and amount of games in which they can participate in. It is necessary to consider the developmental age of the child with CP, to adjust the demands of the game to his or her characteristics, and ensure that the game provides the right challenge to enable learning, inclusion, and the acquisition of different types of abilities. Taking into account the information provided by the survey on disability and personal autonomy presented by the Spanish National Statistics Institute (2008) [32]: over 97% of children aged between 6 and 15 had some sort of disability between 2007/2008. Therefore, it is especially relevant that SONRIE System are going to be applied in schools, thus enabling a greater number of affected children to have access to it, and allowing it to be used in a controlled manner by educators that work with these children thus aiding motivation in these groups of children. Children with CP have difficulty controlling movement, muscle tone and posture, which evidently includes the muscles of the face, necessary for non-verbal expression and even the movements necessary for eating and swallowing. In fact, in the first 12 months of life, more than 90% had clinically significant oral motor dysfunction. In 60% of the children, severe feeding problems preceded the diagnosis of CP [33]. After the first year of life, these children present difficulties when they want to expressing emotions, as well as communicating, which may involve, that they have having social interaction problems. The mobility of these muscles can be impaired in different ways, one of which is that they produce movements known as “block” or not dissociated. These muscles will act in a non-dissociated way, because the child does not possess the capability of making one muscle independent from the other, causing his or her face to be expressionless [34]. Another difficulty appears when they cannot move a specific muscle (for example, the super ciliary muscle, for an angry expression). To achieve this muscle mobility, their body has learned to compensate by using other facial muscles to perform the function of that muscle. The alteration of muscle tone that these children present is usually mixed, which means that some muscles are spastic (the muscles remain permanently contracted), while others are hypotonic (with diminished muscle tone), normal, or fluctuate from one state to another. When the child is asked to perform a muscle movement in imitation, he or she may sometimes perform it correctly and other times be incapable of doing so. Hence, SONRIE detects if the child makes a compensation with another facial muscle for performing a particular movement, which is something difficult to detect for the therapist only by observing the movement. Besides, it may be useful to aid the neuro-rehabilitation of the oral-facial musculature in children. SONRIE stores information about other muscles used to compensate the movement and the time spent in the exercise, as well as the ability or inability to perform the exercise correctly. Consequently, it is considered that the movement is right if it is made within the time limit, if the facial muscles responsible for movement are activated and if there is no compensation with other muscles. The facial musculature helps us to communicate, it also participates in eating and basic oral-facial functions, such as suction, chewing, swallowing, etc. Therefore, the muscular impairment will also

Sensors 2016, 16, 444

5 of 18

affect the performance of these functions. The different games of the SONRIE System allow the user to first explore, and then work on the muscles involved in each of the proposed gestures. Future studies will verify if SONRIE is a serious game able to achieve motor learning. As noted by Sanchez-Cabeza et al. repetition of movements enable the development of new learning and new movement schemes in the brain. Furthermore, it allows the child to be an active part of the process, which contributes to an improved self-esteem and personal satisfaction, as well as better motor learning and greater with treatment [35]. Sensors 2016, compliance 16, 444 5 of 18 3. 3.SONRIE SONRIEFramework Framework Development Development InIn order necessary to tohave havethe theexperience experienceofofdifferent different types ordertotocreate createany anyinformation informationsystem, system, it it is is necessary types of of actors: final users, experts, analysts, developers; moreover, it is essential to design a methodology actors: final users, experts, analysts, developers; moreover, it is essential to design a methodology that covers the informationsystem—within system—withinthe theestimated estimated that covers thestages stagesrequired requiredto toprovide provide the the solution—an solution—an information timeframe, and Figure 11presents presentsthe thegeneral generalmethodology, methodology, which timeframe, andcovering coveringthe therequirements requirementsdefined. defined. Figure which covers Furthermore,it itisisimportant importanttoto coversthe theconstruction constructionphases phasesfor forSONRIE SONRIE System System development. development. Furthermore, indicatethat thatSONRIE SONRIESystem Systemisisfully fully developed, developed, now indicate now we we are are working workingininthe thevalidation validationstage stagewith with children a realenvironment environmentover over99months. months. children inin a real

Figure1.1.Work Workmethodology methodology used used for for the Figure the construction constructionof ofSONRIE. SONRIE.

In the Background Study stage, we analyzed the difficulties faced by these children when In the Background Study stage, we analyzed the difficulties faced by these children when performing movements that involve the oral-facial musculature. Furthermore, we worked on the performing movements that involvesolutions the oral-facial worked on the study study of possible technological with musculature. the objective Furthermore, of achieving we a neuro-rehabilitation of process possiblefor technological solutions with the objective of achieving a neuro-rehabilitation these children. Their findings in that moment were that, to this moment, there process had beenfor these children. Their findings in that moment were that, to this moment, there had been no specific no specific work done on the oral-facial region, and given its implications on the occupational work done on of thethese oral-facial region, and given its implications on the thetarget occupational performance performance children, we decided on this musculature as area. After that, we ofdecided these children, we decided on this musculature as the target area. After that, we decided that: that: SONRIE uses Microsoft’s Kinect sensor connected to a PC, with an integrated database SONRIE uses Microsoft’s Kinect sensor connected to a PC, with an integrated database and video and video instructions to form the video game. Kinect is a webcam-style add-on for the Windows instructions to formwhich the video game. Kinectuser is ainterface webcam-style add-on for the operating operating system, provides a natural (NUI) allowing users toWindows interact intuitively system, which provides a naturaldevice, user interface users to interact intuitively and without and without any intermediary such as (NUI) a gameallowing controller. any intermediary device, such as a game The child and the Kinect sensor willcontroller. be situated in the following positions: The child and the Kinect sensor will be situated in the following positions: • Child: placed at a distance of 140 cm from the Kinect sensor, standing upright and looking at it. Kinect: the atelevation difference between upright a child’s ‚ • Child: placed a distanceabove of 140 ground cm from is thethe Kinect sensor, standing and height lookingand at it. 80 cm/31.5 inches. ‚ Kinect: the elevation above ground is the difference between a child’s height and 80 cm/31.5 inches. The closer the positions are to these values, the greater the accuracy of the SONRIE system is. Figure 2 shows an example: a child 130 cm/51.18 inches tall, who uses the SONRIE system, will be placed at a distance of 140 cm from the sensor, and the sensor should be placed at a height of 50 cm/19.68 inches (130 cm/51.18 inch − 80 cm/31.5 inch = 50 cm/19.68 inch).

Sensors 2016, 16, 444

6 of 18

The closer the positions are to these values, the greater the accuracy of the SONRIE system is. Figure 2 shows an example: a child 130 cm/51.18 inches tall, who uses the SONRIE system, will be placed at a distance of 140 cm from the sensor, and the sensor should be placed at a height of Sensors 2016, 16, 444 6 of 18 50 cm/19.68 inches (130 cm/51.18 inch ´ 80 cm/31.5 inch = 50 cm/19.68 inch).

Figure 2. SONRIEgame gameenvironment environment configuration. Figure 2. SONRIE configuration.

Additionally, to make the system more intuitive, a red square appears in the upper left corner

Additionally, to make the system more intuitive, a red square appears in the upper left corner of of the SONRIE game that indicates where the child’s head should be positioned. the SONRIEOver game whereour thefacial child’s head should positioned. 40 that facialindicates muscles control expressions, which be allow us to show our mood or our Over 40 facial muscles control expressions, which one allow us to show mood or our current emotions. Besides, they our allowfacial human beings to perform of our most vitalour functions: Due Besides, to the amount muscles involved, andtothe variety one of their functions, the human being eating. current eating. emotions. theyof allow human beings perform of our most vital functions: can learn to independently move each of them. However, children with CP are not capable of doing Due to the amount of muscles involved, and the variety of their functions, the human being can learn this. For example, when trying to furrow the brow, they can raise one of the corners of their mouth, to independently move each of them. However, children with CP are not capable of doing this. For performing this movement involuntarily, and without getting their facial muscles to move example, when trying to furrow the brow, they can raise one of the corners of their mouth, performing independently. The main difficulties derived from their brain injuries [36], in this area, are: this movement involuntarily, and without getting their facial muscles to move independently. suctioning, communication, imitating facial expressions, involuntarily moving the tongue, babbling The main difficulties derived from their brain injuries [36], in this area, are: suctioning, communication, and drooling. The identification emotions in the face implies gathering information associated with the imitating facial expressions, of involuntarily moving the tongue, babbling and drooling. shape of the eyes, nose and mouth, the location of the apexes, wrinkles, lines and bulges. Estimates, The identification of emotions in the face implies gathering information associated with the such as age, could also be inferred from the methods employed to detect emotions, if attention is shape of the eyes, nose and mouth, the location of the apexes, wrinkles, lines and bulges. Estimates, focused on lines or the skin’s luminosity. In the case of children with CP, it is important to get them such as age, could also be inferred from the methods employed to detect emotions, if attention is to execute the movement when they want to, rather than involuntarily. These children face the focusedproblem on linesoforbelieving the skin’s luminosity. In the case childrencorrectly, with CP,when it is important to get that certain movements are of performed in reality, they arethem to executeexecuting the movement when they wantofto, rather than involuntarily. These children face the problem them incorrectly. Because this, a facial expression recognition motor is needed, to ensure that thethat movements by the children are adequate and offer feedback to the child the of believing certainperformed movements are performed correctly, when in reality, theyusing are executing system at that moment. them incorrectly. Because of this, a facial expression recognition motor is needed, to ensure that the last task to perform during this stage consisted of determining four movements as part of movementsThe performed by the children are adequate and offer feedback to the child using the system the facial movements proposed by SONRIE for children with CP: raising both eyebrows, blowing, at that moment. kissing and smiling. The occupational therapist decided to create activities in which joining the The last task to perform during stage consisted of determining movements as part of the simplest facial movements. Thesethis games achieved a process of detection four and treatment of children facial movements by motor SONRIE for children with CP: raising both eyebrows, with CP thatproposed not only have problems but also visual, cognitive or sensory perceptionblowing, disorders,kissing hearing andoccupational learning difficulties. and smiling. The therapist decided to create activities in which joining the simplest facial the games of achieved SONRIE can be configured to have different levels ofof difficulty. therapist movements.All These games a process of detection and treatment childrenThe with CP that not can decide the number of repetitions and the time limit by game. Moreover, with the aim of only have motor problems but also visual, cognitive or sensory perception disorders,making hearing and the SONRIE games more attractive for children between 4 and 12 years of age, we chose a medieval learning difficulties. theme (Figure 3). All the games of SONRIE can be configured to have different levels of difficulty. The therapist can decide the number of repetitions and the time limit by game. Moreover, with the aim of making the SONRIE games more attractive for children between 4 and 12 years of age, we chose a medieval theme (Figure 3).

Sensors 2016, 16, 444

7 of 18

Sensors 16, 444 Sensors 2016,2016, 16, 444

7 of 718of 18

Figure 3. Game theme SONRIE.

Figure 3. Game theme SONRIE.

test

SONRIE. SONRIE’s common theme is aFigure game3.inGame whichtheme the child must overcome several tests. When the test is complete, he or she may advance to the next screen (Figure 4). overcome several tests. When SONRIE’s common theme is a game in which the child must SONRIE’s is a game which thethe child must overcome tests. When the Each of common these teststheme is an exercise thatin will prompt four movements thatseveral the team considered is test complete, he orheshe may advance to the next screen (Figure 4). is complete, or she basic for a child with CP.may advance to the next screen (Figure 4).

the

Each of these teststests is an exercise thefour four movements the team considered Each of these is an exercisethat thatwill will prompt prompt the movements that that the team considered basic for a child withwith CP. CP. basic for a child

Figure 4. Game scenarios SONRIE. Eyebrow raising game. Furthermore, the development team recorded videos in which three girls (functioning as avatars) indicated Figure to the player howscenarios he or she had to perform each of the exercises. 4. Game SONRIE. Eyebrow raising game. We used the Face 4. Game Eyebrow raising game. to be detected. It Tracking SoftwareFigure Development Kitscenarios (SDK) forSONRIE. recognizing each of the movements can be used as marker to track humanteam faces with the Kinect camera attachedthree to a PC. The(functioning face tracking as Furthermore, the development recorded videos in which girls engine computes of semantic feature points asofwell as girls a 3D (functioning head pose. the Face avatars) indicated to 3D the positions player how he or she facial had to perform each the exercises. We used Face Furthermore, the development team recorded videos in which three as avatars) Tracking SDK results are also expressed infor terms of weightseach of sixofAction Units and 11 Shape Units, It Tracking Software Development Kit (SDK) recognizing the movements to be detected. indicated to the player how he or she had to perform each of the exercises. We used the Face Tracking are subset of defined in the CANDIDE-3 [37]. CANDIDE is a The parameterized can which be used asamarker towhat trackishuman faces with the Kinectmodel camera attached to a PC. face tracking Software Development Kit (SDK) for for recognizing each ofofthe movements to be detected. It can be used face mask specifically developed model-based coding human faces. Its low number of polygons engine computes 3D positions of semantic facial feature points as well as a 3D head pose. Face (approximately 100) faces allows with fast reconstruction with moderate computing power. is engine as marker to track human the Kinect camera attached to a PC. The CANDIDE face tracking Tracking SDK results are also expressed in terms of weights of six Action Units and 11 Shape Units, controlled by global and local Action Units (AUs). The local Action Units control the mimics of the computes 3D positions of semantic facial feature points as well as a 3D head pose. Face Tracking which are a subset of what is defined in the CANDIDE-3 model [37]. CANDIDE is a parameterized SDK face so that different expressions can be obtained. For example, we decided to use the AU2—Lip resultsface are mask also expressed terms offor weights of six coding ActionofUnits and 11 Shape Units, which are a subset specifically in developed model-based human faces. Its low number of polygons Stretcher to detect the blowing movement, as the lip stretchers records the shape of the mouth using (approximately 100) allows fast reconstruction with moderate computing power. CANDIDE is of what isdots defined in the the mouth. CANDIDE-3 [37]. CANDIDE a parameterized face mask specifically around In AU2,model the possible movement, isisexpressed with a threshold values controlled by global and local Action Units (AUs). The local Action Units control the mimics of the developed for model-based codingwith of human Its 1low number oflips polygons (approximately between −1 and 1 (0 is associated a neutral faces. expression; indicates that the are stretched; and face−1soindicates that different can be obtained. For example, wethe decided to use thethreshold AU2—Lip that theexpressions lips arewith totally together and rounded). We choose value −0.7 at the 100) allows fast reconstruction moderate computing power. CANDIDE is controlled by global

Stretcher to detect the blowing movement, as the lip stretchers records the shape of the mouth using

and local Action Units (AUs). The local Action Units control the mimics of the face so that different dots around the mouth. In AU2, the possible movement, is expressed with a threshold values expressions can be obtained. For example, we expression; decided to1 use the AU2—Lip to and detect the between −1 and 1 (0 is associated with a neutral indicates that the lips Stretcher are stretched; blowing movement, records shape We of the mouth using−0.7 dots around the mouth. −1 indicates that as thethe lipslip arestretchers totally together andthe rounded). choose the value at the threshold In AU2, the possible movement, is expressed with a threshold values between ´1 and 1 (0 is associated with a neutral expression; 1 indicates that the lips are stretched; and ´1 indicates that the lips are totally together and rounded). We choose the value ´0.7 at the threshold to indicate that the child is blowing, since ´1 indicates that the lips are completely rounded, and the blow has not rounded. The physical therapist, who works with the child, can configure the game according to the needs and progress of each child, in order to carry out an effective therapy. To achieve this, Table 1 shows

Sensors 2016, 16, 444

8 of 18

to indicate that the child is blowing, since −1 indicates that the lips are completely rounded, and the blow has not rounded. physical SensorsThe 2016, 16, 444 therapist, who works with the child, can configure the game according to the needs 8 of 18 and progress of each child, in order to carry out an effective therapy. To achieve this, Table 1 shows that it is necessary to persistently store some child-game configurations. For instance, the therapist that it is necessary to persistently store child-game configurations. For instance, thethe therapist configures the number of repetitions forsome each game and the time limit needed to carry out game. configures the number of repetitions for each game and the time limit needed to carry out the game. Table 1. Persistence information about games by children. Table 1. Persistence information about games by children. Child-Game Configuration Configuration (PrimaryChild-Game key) Id_Game (FK Game) (Primary (Primarykey) key) (Primary key) (Primary key) (Primary key)

Id_Game Id_Child Id_Child Date Date Repetitions Repetitions Timelimit limit Time

(Primary key) (FK Game) (Primary key) (FK Child) (Primary key) (FK Child) (Primary key) (Primary key) (Primary key)

Child Performs Game Child Id_Game Performs Game

(FK Game) Id_Game (FK Game) Id_Child (FK Child) Id_Child (FK Child) Date Date Executed Executed Time necessary Time necessary Offset by other muscles Offset by other muscles

It is is useful useful to to know In Table Table 11 called called “child “child It know the the results results of of running running the the games games of of each each child. child. In performs game”, game”, SONRIE SONRIE persistently persistently stores stores information information about about the the child child game game execution, execution, with with data data performs about if the child has executed or not the movement, the time it took and if the child needed to use about if the child has executed or not the movement, the time it took and if the child needed to use other muscles muscles to to perform perform the the movement. movement. other Thus, when the gaming platform starts executing, executing, the the stored stored values values associated associated with with each each child, child, Thus, when the gaming platform starts offering an adequate dynamic game, and, when finished, the results are stored for later review by aa offering an adequate dynamic game, and, when finished, the results are stored for later review by therapist. This This work work culminated culminated in in the the completion completion of of the the system’s system’s first first prototype, prototype, which which included included the the therapist. four aforementioned games and a Web Based Framework which allows the therapist to work with four aforementioned games and a Web Based Framework which allows the therapist to work with the child child information information and and to to configure configure games games for for the the children children with with different different possibilities possibilities to to adjust adjust the the the therapy for for the the child child (Figure (Figure 55 shows therapy shows the the access access interface interface for for SONRIE SONRIE Framework). Framework).

Figure SONRIEWeb Web based Figure 5.5.SONRIE basedframework. framework.

The The next next section section explains explains the the general general function function of of the the SONRIE SONRIE System System and and presents presents the the structure structure that should appear. appear. that decides decides when when each each of of the the games games should SONRIESystem SystemFunction Function 4. SONRIE asas long asas it is Play motivates children children and andmakes makesthem themwant wanttotoparticipate participateininaaparticular particularactivity, activity, long it designed within the parameters that regulate games. Additionally, if it is interactive technological, is designed within the parameters that regulate games. Additionally, if it isand interactive and its capacity to motivate children increases, and adhesion toand treatment improves. The following games technological, its capacity to motivate children increases, adhesion to treatment improves. The cover eachgames of the cover movements a simple, guided usingguided a motivating interface for the child. following each ofinthe movements in way, a simple, way, using a motivating interface for the child. 4.1. Start and Eyebrow Raising Game When the game begins, the introduction screen appears (shown in Figure 3); next, after the player’s skeleton has been detected, the eyebrow raising game begins to run.

Sensors 2016, 16, 444

9 of 18

4.1. Start and Eyebrow Raising Game Sensors 2016, 16, 444

9 of 18

When the game begins, the introduction screen appears (shown in Figure 3); next, after the 9 of 18 the eyebrow raising game begins to run. This game will be running until movement is appears detected,(shown or the in time limit3);for execution When the game begins, the introduction screen Figure next, after theof the player’s skeleton been detected, eyebrow raising game to run. exercise has beenhas reached. Figure 3the shows the start and endbegins interfaces of the eyebrow raising game. This game will be running until movement is detected, or the time limit for execution of the This game will be running until movement is detected, or the time for execution thechild As we will show below, we designed starting and ending screens thatlimit are stimulating forofthe exercise has been reached. Figure 3 shows the start and end interfaces of the eyebrow raising game. exercise been reached. Figure 3 shows the start and end interfaces of the eyebrow raising game.As for each has of the games. weAs will show show below, we designed starting and and ending screens thatthat areare stimulating forfor the we we after designed starting screens stimulating thechild childforthe If will this screenbelow, remains the half of time ending limit, because the child has not yet performed each of the games. for each of the games.will be heard to give the child positive reinforcement (see Figure 6). movement, a sound If this screen remains the child childhas hasnot notyet yetperformed performed If this screen remainsafter afterthe thehalf halfof oftime time limit, limit, because because the thethe movement, a sound will bebeheard (seeFigure Figure6). 6). movement, a sound will heardtotogive givethe thechild childpositive positive reinforcement reinforcement (see 4.1.2016, Start16, and Raising Game Sensors 444Eyebrow player’s skeleton has been detected,

Figure 6. Activity diagram: Start game and eyebrows rise. Figure and eyebrows eyebrowsrise. rise. Figure6.6.Activity Activitydiagram: diagram: Start Start game and

Once the gesture is considered appropriate, the number of times the game has been played is OnceIfthe considered appropriate, the number of times been played is counted. isgesture the first or second appropriate, time, the eyebrow video will playthe again, buthas if the count is higher, Once the itgesture is isconsidered the number of times thegame game has been played is counted. If it is the first or second time, the eyebrow video will play again, but if the count is higher, the decision be made to continue to eyebrow the next game. In other each mustis be played counted. If it is will the first or second time, the video will play words, again, but if game the count higher, the decision will be made to continue to the next game. In other words, each game must be played timeswill as repetitions valued, with time limitIntoother perform each exercisethe(see thethe decision be made to continue to theanext game. words, eachcorresponding game must be played the times as repetitions valued, with a time limit to perform each corresponding exercise (see Table 1). times as repetitions valued, with a time limit to perform each corresponding exercise (see Table 1). Table 1). 4.2.4.2. Blowing andand Kissing Games Blowing Kissing Games 4.2. Blowing and Kissing Games TheThe blowing (Figure 7) and kissing (Figure 8) games follow the same procedures as the previous blowing (Figure and kissing (Figure 8) games follow same procedures asprevious the previous The blowing (Figure 7)7)and kissing (Figure 8) games follow the the same procedures as the one: waiting until thethe movement is recognized or the time limit reached, to continue to the final video one: waiting until movement is recognized or the time limit reached, to continue to the final one: waiting until the movement is recognized or the time limit reached, to continue to the final videovideo forfor each game. each game. game. for each

Figure 7. Activity diagram: blowing game.

Figure 7. Activity diagram: blowing game.

Figure 7. Activity diagram: blowing game.

After this, the counter is reviewed to see if the same game is repeated or the next one can begin.

Sensors 2016, 16, 444 Sensors 2016, 16, 444

10 of 18 10 of 18

Sensors 2016, this, 16, 444 After the

10 of 18 counter is reviewed to see if the same game is repeated or the next one can begin.

After this, the counter is reviewed to see if the same game is repeated or the next one can begin.

Sensors 2016, 16, 444

10 of 18

After this, the counter is reviewed to see if the same game is repeated or the next one can begin.

Figure 8. Activity diagram: kissing games. Figure 8. Activity diagram: kissing games. 4.3. Smiling Game and End of the Figure Game 8. Activity diagram: kissing games.

4.3. Smiling Game and End of the Game The smiling (Figure begins after the kissing game (Figure 8), but unlike the previous 4.3. Smiling Game game and End of the 9) Game games, it does not have an end signals that the exercise has 8), been The smiling game (Figure 9) video beginsthat after the kissing game (Figure butperformed unlike thecorrectly; previous The smiling game (Figure 9) begins after the kissing game (Figure 8), but unlike the previous instead, it enters loop an until it has been performed correctly (the gesture is recognized or the time games, it does not ahave end video that signals that the exercise has been performed correctly; Figure 8.that Activity diagram: kissing games. games, itbeen doesreached) not haverepetition an end video the exercise hasvideo beenthat performed correctly; limit has aftersignals which that it shows way the indicates instead, it enters a loop until it hastimes, been performed correctly (thetogesture is recognized or the theend time instead, it enters a loop until it has been performed correctly (the gesture is recognized or the time of the game. Smiling Game and End of thetimes, Game after which it shows way to the video that indicates the end limit has4.3. been reached) repetition limit has been reached) repetition times, after which it shows way to the video that indicates the end of the game. The smiling game (Figure 9) begins after the kissing game (Figure 8), but unlike the previous of the game. games, it does not have an end video that signals that the exercise has been performed correctly; instead, it enters a loop until it has been performed correctly (the gesture is recognized or the time limit has been reached) repetition times, after which it shows way to the video that indicates the end of the game.

Figure 9. Activity diagram: smiling game. Figure 9. Activity in diagram: smiling game. We summarize the SONRIE functionality Figuresmiling 10, which shows all the screenshots of the Figure 9. Activity diagram: game. SONRIE game. The SONRIE serious game and its framework (Figure 5) have been developed by a Figure 9. Activity in diagram: smiling game. shows all the screenshots of the We summarize theofSONRIE functionality Figure 10,has which multidisciplinary team professionals. The game’s design considered the gained by We summarize the SONRIE functionality in Figure 10, which shows all experience the screenshots SONRIE game. The SONRIE serious game and its framework (Figure 5) have been developed of bythe a occupational therapist while working with children with special needs. We summarize the SONRIE functionality in Figure 10, which shows 5) all have the screenshots of the SONRIE game. The SONRIE serious game and its framework (Figure been developed by a multidisciplinary team of professionals. The game’s design has considered the experience gained by SONRIE game. The SONRIE serious game and its framework (Figure 5) have been developed by a multidisciplinary team ofwhile professionals. Thechildren game’s design has considered the experience gained by occupational therapist working with with special needs. the experience multidisciplinary team of professionals. The game’s design has considered gained by occupational therapist while working with children with special needs. occupational therapist while working with children with special needs.

Figure 10. Interaction screens with different games to motivate children. Figure 10. Interaction screens with different games to motivate children.

Figure 10. Interaction screens with different games to motivate children. Figure 10. Interaction screens with different games to motivate children.

Sensors 2016, 16, 444

11 of 18

4.4. The General Facial Recognition Algorithm for Obtaining the Accurately for Each Game Sensors 2016, 16, by 444Game) (Numerical Value

11 of 18

During the execution of each game in SONRIE, the the child obtainsfor a numerical value (between ´1 4.4. The General Facial Recognition Algorithm for Obtaining Accurately Each Game (Numerical Valueper by game, Game) this number indicates the precision and the execution speed of the facial movement and 300) supportedDuring by SONRIE for theofchild or the child withobtains typicala development. the execution each with game CP, in SONRIE, the child numerical value (between −1 Some possible values fornumber the facial recognition algorithm and 300) per game, this indicates the precision andare: the execution speed of the facial ‚ ‚ ‚

movement supported by SONRIE for themovement child with CP, oristhe childproposed with typical ´10: the child performs the opposite that being bydevelopment. SONRIE. Some possible values for the facial recognition algorithm are: Values between 0 and 99: the child makes a correct facial movement, nevertheless he/she does • reach −10: the performs the opposite movement that is precision being proposed SONRIE. not thechild minimum threshold which measures the of theby facial movement execution. • Values between 0 and 99: the child makes a correct facial movement, nevertheless he/she does by Values between 100 and 300: the child carries out successfully facial movement supported not reach the minimum threshold which measures the precision of the facial movement SONRIE serious game. The return value increases depending on the seconds needed and the execution. facial movement precision. •

Values between 100 and 300: the child carries out successfully facial movement supported by SONRIE serious game. The return value increases depending on the seconds needed and the 5. Verification Stage: Children with Typical Development facial movement precision.

Before beginning the experimentation process with children with CP, we evaluated a control VerificationofStage: Children with Typical Development group5. consisting 7 children with normal development who were 4 to 10 years old. We configured a time limit of 15 s for executing the facial movement repeated per child. Children Before beginning the experimentation process with childrenthree withtimes CP, we evaluated a control with group consisting ofdid 7 children with normal development who were 4 to able 10 years old. We normal development not have any disorder, therefore they were to carry outconfigured all the games time limitand of 15 s for executing facialbetween movement times per were child. obtained Children with with aprecision execution speedthe values 66 repeated and 269.three These values applying normal development did not have any disorder, therefore they were able to carry out all the the algorithm summarized in the above subsection. Hence, we could verify that all thegames SONRIE’s with precision and execution speed values between 66 and 269. These values were obtained applying games run correctly with healthy children. This verification stage has allowed to calibrate each game the algorithm summarized in the above subsection. Hence, we could verify that all the SONRIE’s to ensure that each movement was detected correctly by SONRIE. Moreover, we evaluated that the games run correctly with healthy children. This verification stage has allowed to calibrate each game facialtorecognition algorithm works satisfactory. Table 2 shows the average values obtained for each ensure that each movement was detected correctly by SONRIE. Moreover, we evaluated that the gamefacial in tests with healthy children. recognition algorithm works satisfactory. Table 2 shows the average values obtained for each game in tests with healthy children. Table 2. Average values obtained for each game in tests with healthy children. Table 2. Average values obtained for each game in tests with healthy children.

Eyebrow Blow Kiss Description EyebrowGame Blow Game Kiss Game Description Game Game Game Average values obtained by healthy children 136 92 128 Averagebetween values obtained healthy children 4 and 7 ofbyage 136 92 128 between 4 and 7 of agechildren Average values obtained by healthy 210 97 120 between 9 and 10 of Average values obtained byage healthy children 210 97 120 between 9 and 10 of age Average values obtained by healthy children 173 95 124 Average values obtained by healthy children 173 95 124

Smile Smile Game Game 101 101 118 118 110 110

Table 2 summarizes that healthy children got good scores on average performance on all of the Table 2 summarizes that healthy children got good scores on average performance on all of the games (95 the average value for the blow game and 173 the average value for the eyebrow game). games (95 the average value for the blow game and 173 the average value for the eyebrow game). Figure 11 shows two images of the where the the Kinect sensor is connected to to thethe computer which Figure 11 shows two images of tests the tests where Kinect sensor is connected computerinin SONRIE serious game is running. which SONRIE serious game is running.

Figure 11. Verification Stage with typical development children.

Figure 11. Verification Stage with typical development children.

Sensors 2016, 16, 444

12 of 18

Figure 12444 shows that healthy children perform similarly in the execution of each SONRIE 12 game. Sensors 2016, 16, of 18 The game “Smile” is the most irregular in its execution. In fact, this game obtained values between 67 and 159 corresponding to the precision in which the healthy child performs this movement. In our Figure shows that healthy children perform similarly in of each SONRIE game. opinion, this12game requires greater precision in its execution tothe be execution correctly detected by the Kinect The game “Smile” is the most irregular in its execution. In fact, this game obtained values between sensor. The “Smile” is the most complex of all of the movements implemented, having many 67 and 159 corresponding to the precision in which the healthy child performs this movement. In our variations. opinion, this gamethe requires inshows its execution to be correctly by the Kinect Furthermore, green greater line (seeprecision Figure 12) the results obtained bydetected a 9 year-old boy who sensor. The “Smile” is the most complex of all of the movements implemented, having many variations. was very focused during the test. As a consequence, he obtained the best results in almost all of the Furthermore, the green line (see Figure 12) shows the results obtained by a 9 year-old boy who was games. very focused during the test. As a consequence, he obtained the best results in almost all of the games.

Figure Figure 12. 12. Results Results obtained obtained by by children children with with typical typical development. development.

Figure 12 shows the best score for each healthy child. His/her scores were obtained by taking Figure 12 shows the best score for each healthy child. His/her scores were obtained by taking into account the best score in the performance of each game. into account the best score in the performance of each game. 6. 6. Validation ValidationStage: Stage:Children Childrenwith withCP CP After the SONRIE SONRIEverification verificationstage stagecarried carried out with healthy children, experimental After the out with healthy children, twotwo experimental tests tests took took place, both in a real environment: an integration school for children with CP. Before place, both in a real environment: an integration school for children with CP. Before the beginningthe of beginning of the validation at Bellas Vistas School, we have into account ethical issues to consider in the validation at Bellas Vistas School, we have into account ethical issues to consider in an intervention an intervention withpopulation: a very fragile population: study was approved by the Ethics of with a very fragile this study wasthis approved by the Ethics Committee ofCommittee the Rey Juan the Rey Juan Carlos University and was carried out in accordance with the Declaration of Helsinki. Carlos University and was carried out in accordance with the Declaration of Helsinki. Furthermore, Furthermore, this this report report shows shows the the protocol protocol for for this this research research which which must must be be properly properly explained explained to the parents of the children who participating in the study. Parents must of the to the parents of the children who participating in the study. Parents must approve of approve the participation participation of children in the study, by signing an acceptation letter. of children in the study, by signing an acceptation letter. This stage took tookplace placeinintwo twodifferent differentsessions. sessions.We We considered it relevant to carry This validation validation stage considered it relevant to carry outout the the tests in an environment that was familiar to the children with CP, and that did not imply a tests in an environment that was familiar to the children with CP, and that did not imply a disruption disruption to of thethese routine of these whoinparticipated in the study. Theaccompanied children were to the routine children who children participated the study. The children were by accompanied by their physical therapist, who helped them understand the dynamics of thetogame. their physical therapist, who helped them understand the dynamics of the game. In order verifyIn if order to verify if the children performed well during the exercises, theyusing wereaevaluated usinggestures a scale the children performed well during the exercises, they were evaluated scale of facial of facial gestures evaluation of children Cerebral Palsy3), (Likert scale-Table 3), physical created ad hoc by evaluation of children with Cerebral Palsywith (Likert scale-Table created ad hoc by the therapist the physical therapist working with the team. This scale was used before and after the verification working with the team. This scale was used before and after the verification stage. In this moment stage. In this moment children executed movements to the school’s physical children executed all the movements to all thethe school’s physical therapist without thetherapist SONRIEwithout system. the SONRIE system. The paper provides in Table 3 a novel scale for measuring facial movement The paper provides in Table 3 a novel scale for measuring facial movement capacity. Future researches capacity. Future researcheswith should check correlations otherorofacial scales such the “Nordic orofacial should check correlations other scales such us thewith “Nordic test us screening (NOT-S)” and test screening (NOT-S)” and should not be based only on observations [38]. should not be based only on observations [38].

13 of 18

Sensors 2016, 16, 444 Table 3. Scale of facial movements evaluation of children with Cerebral Palsy.

13 of 18

1.

A2.fewAtimes few times

3. Several Several times times

2.

3.

1. 1. Never Never

Table 3. Scale of facial movements evaluation of children with Cerebral Palsy.

Always 4. 4.Always

Sensors 2016, 16, 444

The child is able to express surprise or admiration raising

1. The child is able to express surprise or admiration raising eyebrow eyebrow 2.

The2. child ableistoable blow Theischild to blow

3.

The3. child ableistoable give kissa kiss Theischild to agive

4.

The child is able to smile 4.

The child is able to smile

The mainly thethe facial muscle movement involved in eachinexercise. The clinical The NOT-S NOT-Stest testcollects collects mainly facial muscle movement involved each exercise. The application is to improve muscles required for feeding and non-verbal communication [39]. Below, we clinical application is to improve muscles required for feeding and non-verbal communication [39]. show the results obtained from the 10 children with CP, all are students at an integration school, who Below, we show the results obtained from the 10 children with CP, all are students at an integration participated in the SONRIE’s validation stage. school, who participated in the SONRIE’s validation stage. The sample chosen for the system’s validation was made made up up of of children children with with CP, CP, all The sample chosen for the system’s validation was all between between 4 4 and Figure 13 13 shows and 12 12 years years of of age. age. Figure shows one one of of the the tests tests with with the the Kinect Kinect connected connected to to the the computer computer on on which which the the SONRIE SONRIE system system is is running. running.

Figure 13. SONRIE validation stage. Figure 13. SONRIE validation stage.

The selection criteria for children were: both genders, within the described age range, with a The selection for children were: both genders, within the described age range, with a diagnosis of CP andcriteria impairment of the oral-facial musculature. Two verification groups were created diagnosis CP and impairment of the oral-facial musculature. Two verification groups were created randomly,ofeach made up of five children. randomly, each made up of five children. The main aim of this validation test was to carry out a test about the way in which children with The mainthe aim of this validation testtowas to carry outresults a test about the obtained way in which with CP accepted SONRIE games, and contrast their to those in thechildren verification CP accepted the SONRIE games, and to contrast their results to those obtained in the verification

Sensors 2016, 16, 444 Sensors Sensors2016, 2016,16, 16,444 444

14 of 18 1414ofof1818

phase. As in the verification phase, we configured a maximum time threshold or time limit (15 s) and phase. As in the verification phase, we configured a maximum time threshold or time limit (15 s) and the execution of verification each exercise threewe times during awhich the child to perform thelimit movement. In phase. As in the phase, configured maximum timehad threshold or time (15 s) and the execution of each exercise three times during which the child had to perform the movement. In all the situations, the results the exercises were which stored the in achild database. Inperform the future will allow the the execution of each exerciseofthree times during had to theitmovement. In all the situations, the results of the exercises were stored in a database. In the future it will allow the therapist to specifically configure gamewere for every child. all the situations, the results of theeach exercises stored in a database. In the future it will allow the therapist to specifically configure each game forthat every child. In addition, we obtained interesting results are shown in Figures 14 and 15. Figure 14 shows therapist to specifically configure each game for every child. In addition, we obtained interesting results are shown in Figures 15. 14results shows a comparison between the scores of children withthat CP aged 10 Figures and14 12and and theFigure average In addition, we obtained interesting results that arebetween shown in 14 and 15. Figure 14 a comparison between the scores of children with CP aged between 10 and 12 and the average results obtained by healthy children (line green in Figure 14) evaluated in the verification stage. As shown shows a comparison between the scores of children with CP aged between 10 and 12 and the average obtained by the healthy children (line with greenCP, in regarding Figure 14)the evaluated in the verification stage. shown in Figure 14, scores of children execution of the games, moreAs irregular results obtained by healthy children (line green in Figure 14) evaluated in the are verification stage. in Figure 14, the scores of children with CP, regarding the execution of the games, are more irregular between them and for each child. This explains why it is so difficult to find big and homogeneous As shown in Figure 14, the scores of children with CP, regarding the execution of the games, are between them and for each This explains why it isthe so difficult to find big conducted and homogeneous samples, in addition thechild. difficulty in child. generalizing results in and this more irregular betweentothem and for each This explains why itofis studies so difficult to find big samples, in addition to the difficulty in generalizing the results of studies conducted in this population [40]. homogeneous samples, in addition to the difficulty in generalizing the results of studies conducted in population [40]. this population [40].

Figure 14. 14. Comparative obtained by by children children with with CP CP (10–12 (10–12 years) years) and and healthy healthy children. children. Figure Comparative results results obtained Figure 14. Comparative results obtained by children with CP (10–12 years) and healthy children.

As Figure 14 shows, during the execution of the games, we detected that a 10-year old boy, As 14 shows, during the of we that old boy, As Figure Figureto 14the shows, during the execution execution of the the games, games, we detected detected that aa 10-year 10-year old boy, corresponding red line in Figure 14, had a mobility impairment in the musculature of the upper corresponding to the red line in Figure 14, had a mobility impairment in the musculature of the upper corresponding red line in Figure 14, had a mobility impairment in thehe musculature of the upper part of the face,toathe problem that had previously gone undetected. Hence, could not perform the part of the face, aaproblem that had previously gone undetected. Hence, he could not perform the part of the face, problem that had previously gone undetected. Hence, he could not perform the movements even once. Finally, Figure 15 shows a comparison between the scores of children with movements even once. Finally, Figure 15 shows a comparison between the scores of children with CP movements even 4once. Figure 15 shows comparison between the scores children with CP aged between and 7Finally, with the average resultsaobtained by healthy children (lineof green in Figure aged between 4 and 7 with the average results obtained by healthy children (line(line green in Figure 15) CP aged between 4 and 7 with the average results obtained by healthy children green in Figure 15) evaluated in the verification stage. evaluated in the stage. 15) evaluated in verification the verification stage.

Figure Comparative results results obtained Figure 15. 15. Comparative obtained by by children children with with CP CP (4–7 (4–7 years) years) and and healthy healthy children. children. Figure 15. Comparative results obtained by children with CP (4–7 years) and healthy children.

one hand, hand, Figure 15 shows shows that that the the average average scores of all all healthy healthy children children (green line in On the one On15) the one hand,than Figure shows with that the scoresanticipate, of all healthy (green line in CP.average As we we might might anticipate, ischildren logical that that children children Figure are higher for 15 children CP. As it is logical Figure 15) are higher than for children with CP. As we might anticipate, it is logical that children without aapathology pathologyobtain obtainbetter betterscores scores than children with These normal scores, adjusted by without than children with CP.CP. These normal scores, adjusted by age, without a pathology obtain better scores than children with CP. These normal scores, adjusted by age,establish will establish normal data forgame eachwhen gamewe when we enlarge the of sample ofchildren. healthy children. On will normal data for each enlarge the sample healthy On the other age, will establish normal data for each game when we enlarge the sample of healthy children. On the other hand, shows that the games’execution efficient execution is not dependent ondata age,suggests the data hand, Figure 15 Figure shows 15 that the games’ efficient is not dependent on age, the the other hand, Figure 15 shows that the of games’ execution isbenot dependent age, the data suggests that it to is the related to of theimpairment. degree impairment. This hypothesis should beinon re-evaluated in that it is related degree Thisefficient hypothesis should re-evaluated future studies suggests that it is related to the degree of impairment. This hypothesis should be re-evaluated in future studies with the SONRIE with the SONRIE serious game. serious game. future studies with the SONRIE serious game.

Sensors 2016, 16, 444

15 of 18

7. Discussion In this moment, we present a discussion that focuses on three aspects to consider in evaluating a technology as it is presented in this paper. First, technical aspects of the verification: we evaluate the SONRIE functionality testing seven healthy children aged between 4 and 10 years. In these tests, we determined the correct distance between the child and the sensor and we decided that adequate lighting should be provided to optimize the functionality of the Kinect sensor. Furthermore, we adjusted the minimum time length and the number of repetitions every exercise to propose to the school physical therapist as a starting point to begin the assessment in a real environment. As second question, we want to show a comparison to current therapies: CP may affect oral motor skills, leading to speech delay, drooling and difficulties with sucking, swallowing, and chewing [41,42]. Despite the importance of the role of feeding for children with CP and their families, clinical experience shows that oral-motor treatment is not taken care of in the overall rehabilitation of these children. Present and past scientific studies mainly focus on feeding problems. The treatments carried out since 1979 by Borkowska [43] studied the most common problems: sucking, swallowing, breathing, chewing and drooling. Further studies reach different and even contradictory conclusions [44–47]. Borkowska tested the effect of rehabilitation of the feeding function on the development of visual-motor coordination and speech. He concluded that the training had a significant effect on feeding, and positive influence on the eye-hand coordination if the position was correct. These positive effects are corroborated on the study by Gisel [44] and Sıgan ˘ [40]. Later, Gisel et al. asserted that oral-motor training did not have a significant effect on feeding and growth [45]. Nevertheless, they found that this treatment reduced the feeding problems caused by aspiration. As to reducing drooling, we have found contradictory results too. While the study of Domaracki and Sisson did not find any treatment benefit [46], positive results were confirmed by the research of Sıgan ˘ [40] and Yam [47]. Furthermore, Sıgan ˘ et al. discovered in 2013 significant improvements in the treatment group when it comes to chewing, swallowing, drooling and independent feeding. Therefore, these studies show that there is not a single treatment protocol that is valid and established for the population in this area. Besides, there is not consensus regarding detection systems, screening and/or assessment tools. The scales used to determine the oral-motor dysfunction are mainly focused on feeding problems and they do not evaluate the facial muscles on the whole. Hence, these scales do not take into account those elements related to the role of emotional expression [48]. The clinical tool used to identify the presence of possible neuromuscular problems is the oral mechanism examination [38]. The observation of the feeding function, together with perceptual judgments about strength and joint range of motion are usually the way to obtain information on the performance of feeding. In addition, this kind of information is full of subjectivity; the validity and reliability of these assessments are unknown nowadays. Neither there are normative data that could help distinguish normal from abnormal performance [38]. SONRIE is a proposed technology that provides reliable data on the muscular function of the face as a whole. This research highlights the importance of the detection of oral-facial musculature impairments and represents a starting point to achieve an improvement treatment to be offered to children with CP. 8. Conclusions Using new technologies as a method for detection, intervention and rehabilitation in children with CP is a relatively new field, where we need to continue working to improve these children’s quality of life. For the neuro-rehabilitation of children with CP, it is necessary to have a real environment, where this therapy can be satisfactorily applied. In this environment, the child must feel comfortable and safe. In addition, it is necessary to have the collaboration of a professional that can introduce the child to the game and then measure the child’s degree of improvement as the therapy progresses. Considering the above, the school is the ideal environment to apply the system presented in this

Sensors 2016, 16, 444

16 of 18

research. However, other possible scenarios, such as specialized therapeutic centers and, in the future, the home environment, should not be ruled out. The SONRIE system can help these children, so they may work on their facial muscles. An improvement in muscular dissociation and an absence of compensations will mean that a child has better facial expressions, and he/she is more independent when eating, which not only contributes to improving his or her quality of life, but also that of his or her family members. The SONRIE system is a highly ecological environment, adapted for children, since it uses interactive games as therapeutic tools. Furthermore, SONRIE serious game allows professionals to create personalized therapies, configure personalized games and cushion the effects produced by CP in children who are treated with this interactive therapy. We need more than specific knowledge about CP and the oral-facial musculature that intervenes in the performance of different facial movements to satisfactorily develop SONRIE. Furthermore, it is essential to research existing technology and choose, in function of that research, the most adequate device to solve the problem. Lastly, it is necessary to observe these children as they perform the proposed movements, adjusting each of the games developed (smiling, kissing, blowing and raising eyebrows) and adapting them to the specific needs of each child in a real environment. SONRIE will aid the therapy process that children with CP undergo in schools, and will allow professionals to configure personalized therapies. In addition, in the future, we will evaluate whether it is possible to alleviate some of the effects that CP may have on the children treated with this therapy. The verification and validation results presented in this article justify the need for the better detection and intervention in the oral-facial musculature using a system like SONRIE. This system has proven to be effective because it is easy for professionals to use, motivational for children, and has a low cost. The selection of the game as the element that facilitates therapy was justified theoretically in the section “SONRIE: Virtual Environment for oral-facial rehabilitation”, and it has been proven by the validation process carried out in the school; the games caught the attention of the children and kept the children interested. In future works, we will use SONRIE as a tool for rehabilitation of oral-facial muscles in children with diseases. Acknowledgments: The authors wish to thank María del Carmen Martínez Culebras, physical therapist at Bellas Vistas School, Alcorcon, Madrid, Spain. This article is part of research conducted under EDUCERE project (Ubiquitous Detection Ecosystem to Care and Early Stimulation for Children with Developmental Disorders; TIN2013-47803-C2-1-R), supported by the Ministry of Education and Science of Spain through the National Plan for R + D + I (research, development, and innovation). Author Contributions: María-Luisa Martín-Ruiz and Nuria Máximo-Bocanegra defined the theoretical background and conceived and designed the case Study. María-Luisa Martín-Ruiz defined the system challenges and designed the framework and the games; Nuria Máximo-Bocanegra defines the games characteristics about Cerebral Palsy children necessities. Laura Luna-Oliva performed the case study; all the authors have supported and supervised the research work; María-Luisa Martín-Ruiz wrote the initial draft and Nuria Máximo-Bocanegra has contributed in the writing of the manuscript. All the authors have contributed in the revision process of the manuscript. Conflicts of Interest: The authors declare no conflict of interest.

References 1. 2. 3.

4.

Svraka, E. Cerebral Palsy—Challenges for the Future; InTech: Rijeka, Croatia, 2014; pp. 273–301. Rosenbaum, P.; Paneth, N.; Leviton, A.; Goldstein, M.; Bax, M.; Damiano, D.; Dan, B.; Jacobsson, B. A report: The definition and classification of cerebral palsy April 2006. Dev. Med. Child Neurol. Suppl. 2007, 49, 8–14. Cans, C. Surveillance of Cerebral Palsy in Europe, Surveillance of Cerebral Palsy in Europe: A collaboration of cerebral palsy surveys and registers. Surveillance of Cerebral Palsy in Europe (SCPE). Dev. Med. Child Neurol. 2000, 42, 816–824. [CrossRef] Bax, M.; Goldstein, M.; Rosenbaum, P.; Leviton, A.; Paneth, N.; Dan, B.; Jacobsson, B.; Damiano, D. Proposed definition and classification of cerebral palsy, April 2005. Dev. Med. Child Neurol. 2005, 47, 571–576. [CrossRef] [PubMed]

Sensors 2016, 16, 444

5.

6. 7.

8. 9.

10. 11. 12. 13. 14.

15.

16. 17.

18.

19. 20. 21. 22. 23.

24.

25. 26.

17 of 18

Hurley, D.S.; Sukal-Moulton, T.; Msall, M.-E.; Gaebler-Spira, D.; Krosschell, K.-J.; Dewald, J.-P. The cerebral palsy research registry: Development and progress toward national collaboration in the United States. J. Child Neurol. 2011, 26, 1534–1541. [CrossRef] [PubMed] Pascual, J.M.; Koenigsberger, M.R. Parálisis cerebral: Factores de riesgo prenatales. Rev. Neurol. 2003, 34, 275–280. Cancho-Candela, R.; Fernández-Alonso, J.-E.; Lanza-Fernández, E.; Lozano-Domínguez, M.-A.; Andrés de Llano, J.M.; Folgado–Toranzo, I. Estimation of the prevalence of cerebral palsy in the Autonomous Castilla and León Community (Spain) using a disabilities register. An. Pediatr. 2006, 65, 97–100. Puyuelo-Sanclemente, M. Psicología, audición y lenguaje en diferentes cuadros infantiles. Aspectos comunicativos y neuropsicológicos. Rev. Neurol. 2001, 32, 975–980. [PubMed] National Institute of Neurological Disorders. Cerebral Palsy Information Page. 20 March 2015. Available online: http://www.ninds.nih.gov/disorders/cerebral_palsy/cerebral_palsy.htm (accessed on 12 November 2015). Camacho-Salas, A.; Pallás-Alonso, C.-R.; de la Cruz-Bértolo, J.; Simón-de Las Heras, R.; Mateos-Beato, F. Cerebral palsy: The concept and population-based registers. Rev. Neurol. 2007, 45, 503–508. [PubMed] Surveillance of Cerebral Palsy in Europe, Cerebral Palsy. Available online: http://www.scpenetwork.eu/ en/cerebral-palsy/ (accessed on 28 April 2015). Yarnell, J. Epidemiology and Disease Prevention: A Global Approach, 2nd ed.; Oxford University Press: Oxford, UK, 2013. Johnson, A. Prevalence and characteristics of children with cerebral palsy in Europe. Dev. Med. Child Neurol. 2002, 44, 633–640. [CrossRef] Navas Contino, M.; Urquijo-Sarmiento, C.-M.; Gutiérrez-Escarrás, Y.; Durán-Morera, N.; Álvarez-Guerra- González, E. Propuesta de sistema para la atención integral al niño discapacitado por parálisis cerebral y a su familia. Med. Electrón. 2013, 17, 56–64. Machado, F.R.C.; Antunes, P.P.; de Moura-Souza, J.; Levandowski, D.C.; de Oliveira, A.A., Jr. Virtual Reality Technology for Rehabilitation of Cerebral Palsy: A Literature Review. Trends Psychol. 2014, 22, 565–577. [CrossRef] Reinkensmeyer, D.J.; Boninger, M.L. Technologies and combination therapies for enhancing movement training for people with a disability. J. Neuroeng. Rehabil. 2012, 9. [CrossRef] [PubMed] Summa, S.; Basteris, A.; Betti, E.; Sanguineti, V. Adaptive training with full-body movements to reduce bradykinesia in persons with Parkinson’s disease: A pilot study. J. Neuroeng. Rehabil. 2015, 12. [CrossRef] [PubMed] Moreno, F.; Jordan, O.; Esmitt, R.J.; Christiam, M.; Omaira, R.; Jeanlight, R.; Silvio, Á. Un framework para la rehabilitación física en miembros superiores con realidad virtual. In Proceedings of the Primera Conferencia Nacional de Computación, Informática y Sistemas (CoNCISa), Vargas, Venezuela, 7–11 October 2013. Burdea, G.; Coiffet, P. Tecnologías de la Realidad Virtual; Paidós: Barcelona, España, 1996. Sveistrup, H. Motor rehabilitation using virtual reality. J. Neuroeng. Rehabil. 2004, 1. [CrossRef] [PubMed] Rothbaum, B.O.; Williford, J.S.; North, M. Effectiveness of computer-generated (virtual reality) graded exposure in the treatment of acrophobia. Am. J. Psychiatry 1995, 152, 626–628. [PubMed] Broeren, J.; Samuelsson, H.; Stibrant-Sunnerhagen, K.; Blomstrand, C.; Rydmark, M. Neglect assessment as an application of virtual reality. Acta Neurol. Scand. 2007, 116, 157–163. [CrossRef] [PubMed] Jaume-i-Capo, A.; Martinez-Bueso, P.; Moya-Alcover, B.; Varona, J. Interactive rehabilitation system for improvement of balance therapies in people with cerebral palsy. IEEE Trans. Neural Syst. Rehabil. Eng. 2014, 22, 419–427. [CrossRef] [PubMed] Luna-Oliva, L.; Ortiz-Gutiérrez, R.M.; Cano-de la Cuerda, R.; Piédrola, R.M.; Alguacil-Diego, I.M.; Sánchez-Camarero, C.; Martínez Culebras, M.D.C. Kinect Xbox 360 as a therapeutic modality for children with cerebral palsy in a school environment: A preliminary study. NeuroRehabilitation 2013, 33, 513–521. [PubMed] Berenson, M. Estadística Para Administración y Economía; MCGraw-Hill: México, 1993; pp. 410–496. Sharan, D.; Ajeesh, P.S.; Rameshkumar, R.; Mathankumar, M.; Paulina, R.J.; Manjula, M. Virtual reality based therapy for post-operative rehabilitation of children with cerebral palsy. Work 2012, 41, 3612–3615. [PubMed]

Sensors 2016, 16, 444

27.

28. 29. 30. 31.

32. 33. 34. 35.

36. 37. 38. 39. 40. 41. 42. 43.

44. 45.

46. 47. 48.

18 of 18

Blum-Dimaya, A.; Reeve, S.A.; Reeve, K.F.; Hoch, H. Teaching children with autism to play a video game using activity schedules and game-embedded simultaneous video modeling. Educ. Treat. Child. 2010, 33, 351–370. [CrossRef] Wang, M.; Reid, D. Virtual reality in pediatric neurorehabilitation: Attention deficit hyperactivity disorder, autism and cerebral palsy. Neuroepidemiology 2011, 36, 2–18. [CrossRef] [PubMed] Chen, Y.P.; Lee, S.Y.; Howard, A.M. Effect of virtual reality on upper extremity function in children with cerebral palsy: A meta-analysis. Pediatr. Phys. Ther. 2014, 26, 289–300. [CrossRef] [PubMed] Majnemer, A.; Shevell, M.; Law, M.; Poulin, C.; Rosenbaum, P. Level of motivation in mastering challenging tasks in children with cerebral palsy. Dev. Med. Child Neurol. 2010, 52, 1120–1126. [CrossRef] [PubMed] Brütsch, K.; Schuler, T.; Koenig, A.; Zimmerli, L.; Koeneke, S.M.; Lünenburger, L.; Riener, R.; Jäncke, L.; Meyer-Heim, A. Influence of virtual reality soccer game on walking performance in robotic assisted gait training for children. J. Neuroeng. Rehabil. 2010, 7. [CrossRef] [PubMed] INE. Encuesta de discapacidad. In Autonomía Personal y Situaciones de Dependencia; Boletín Informativo del Instituto Nacional de Estadística: Madrid, España, 2008. Reilly, S.; Skuse, D.; Poblete, X. Prevalence of feeding problems and oral motor dysfunction in children with cerebral palsy: A community survey. J. Pediatr. 1996, 129, 877–882. [CrossRef] Edvinsson, S.E.; Lundqvist, L.O. Prevalence of orofacial dysfunction in cerebral palsy and its association with gross motor function and manual ability. Dev. Med. Child Neurol. 2016, 58, 385–394. [CrossRef] [PubMed] Sánchez-Cabeza, A.; Arana-Echevarría, J.L. Aprendizaje motor: Teorías y técnicas, En: Cano de la Cuerda R y. In Neurorrehabilitación: Métodos Específicos de Valoración y Tratamiento; Collado, V.S., Ed.; Médica Panamericana: Madrid, Spain, 2012. Bax, M.; Tydeman, C.; Flodmark, O. Clinical and MRI Correlates of Cerebral Palsy: The European Cerebral Palsy Study. JAMA 2006, 296, 1602–1608. [CrossRef] [PubMed] Ahlber, J. CANDIDE—A Parameterized Face. Available online: http://www.icg.isy.liu.se/candide/ (accessed on 21 September 2015). Clark, H.M. Neuromuscular treatments for speech and swallowing: A tutorial. Am. J. Speech Lang Pathol. 2003, 12, 400–415. [CrossRef] Bakke, M.; Bergendal, B.; McAllister, A.; Sjogreen, L.; Asten, P. Development and evaluation of a comprehensive screening for orofacial dysfunction. Swed. Dent. J. 2007, 31, 75–84. [PubMed] Sigan, S.N.; Uzunhan, T.A.; Aydinli, N.; Eraslan, E.; Ekici, B.; Çaliskan, M. Effects of oral motor therapy in children with cerebral palsy. Ann. Indian Acad. Neurol. 2013, 16, 342–346. [PubMed] Reilly, S.M.; Skuse, D.H.; Wolke, D.; Stevenson, J. Oral-motor dysfunction in children who fail to thrive: Organic or non-organic? Dev. Med. Child Neurol. 1999, 41, 115–122. [CrossRef] [PubMed] Senner, J.E.; Logemann, J.; Zecker, S.; Gaebler-Spira, D. Drooling, saliva production, and swallowing in cerebral palsy. Dev. Med. Child Neurol. 2004, 46, 801–806. [CrossRef] [PubMed] Borkowska, M. Effect of rehabilitation of the feeding function on the development of visual-motor coordination and speech in children with infantile cerebral palsy. Probl. Med. Wieku Rozwoj. 1979, 9, 209–234. [PubMed] Gisel, E.G. Oral-motor skills following sensorimotor intervention in the moderately eating-impaired child with cerebral palsy. Dysphagia 1994, 9, 180–192. [CrossRef] [PubMed] Gisel, E.G.; Applegale-Ferrante, T.; Benson, J.E.; Bosnia, J.F. Effect of oral sensorimotor treatment on measures of growth, eating efficiency and aspiration in the dysphagic child with cerebral palsy. Dev. Med. Child Neurol. 1995, 37, 528–543. [CrossRef] [PubMed] Domaracki, L.S.; Sisson, L.A. Decreasing drooling with oral motor stimulation in children with multiple disabilities. Am. J. Occup. Ther. 1990, 44, 680–684. [CrossRef] [PubMed] Yam, W.K.L.; Yang, H.L.C.; Abdullah, V.; Chan, C.Y.L. Management of drooling for children with neurological problems in Hong Kong. Brain Dev. 2006, 28, 24–29. [CrossRef] [PubMed] Mariusz, S. On Facial Expressions and Emotions RGB-D Database. In Beyond Databases, Architectures, and Structures; Springer International Publishing: New York, NY, USA, 2014; pp. 384–394. © 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons by Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).