A review of gesture-based computing research ... - Semantic Scholar

Computers & Education 78 (2014) 268e277

Contents lists available at ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Taking a signal: A review of gesture-based computing research in education Feng-Ru Sheu, Nian-Shing Chen* Department of Information Management, National Sun Yat-Sen University 70, Lienhai Rd., Kaohsiung, 80424, Taiwan

a r t i c l e i n f o

a b s t r a c t

Article history: Received 31 October 2013 Received in revised form 12 June 2014 Accepted 16 June 2014 Available online 24 June 2014

This study used content analysis of journal articles from 2001 to 2013 to explore the characteristics and trends of empirical research on gesture-based computing in education. Among the 3018 articles retrieved from 5 academic databases by a comprehensive search, 59 articles were identified manually and then analyzed. The distribution and trends analyzed were research methods, study disciplines, learning content, technology used, and intended settings of the gesture-based learning systems. Furthermore, instructional interventions were also analyzed based on the learning context or the sub-education domain to which they belonged to ascertain if any instructional intervention was applied in these systems. It was found that experimental design research is the most commonly used method (72.9%) followed by design-based research (20.3%). The findings indicate that Nintendo Wii is the gesture-based device that is the most often used (40%), while the domain in which the technology is most frequently used is special education (42.4%). The same trend was also found in a further analysis which identified that the domain which uses Wii the most is special education (70%). Among all the identified learning topics, motor skills learning has the highest percentage (44%). When grouping these topics into three domains of knowledge (procedural, conceptual, and both), the result demonstrates that both procedural and conceptual type of knowledge are equally distributed in the gesture-based learning studies. Finally, a comparison of instructional intervention of gesture-based learning systems in different sub-education domains is reported. © 2014 Elsevier Ltd. All rights reserved.

Keywords: Humanecomputer interface Interactive learning environments Interdisciplinary projects Pedagogical issues Teaching/learning strategies

1. Introduction As revealed by the Horizon Report (Johnson, Adams, & Cummins, 2012; Johnson, Levine, Smith, & Stone, 2010; Johnson, Smith, Willis, Levine, & Haywood, 2011), gesture-based devices as an emerging technology have opened up new opportunities for learning. The features of gesture-based devices allow the user as a controller to interact with the computer more directly through the use of motions and movements as naturally as in daily life (Johnson et al., 2012), such as by using speech, gestures, body movements, finger flips and even facial expressions and eye movements (Johnson et al., 2012, 2011; Wojciechowski, 2012). For example, DephJS from MIT allows users to interact with the Google Chrome web browser through gestures. Other examples such as the 3Gear System, MudPad, LZI Technology, and ZeroTuch (Johnson et al., 2012) also allow users to interact with computers through gestures and hand movements. Although initially gesture-based computing received great attention in gaming and in mobile devices, the potential for learning purposes has recently generated enormous interest among educators. The applications and development of gesture-based computing in training and education are continually expanding. Experiments and innovative teaching with these devices have been growing in many areas such as special education, physics, mathematics, physical therapy, arts, music, science, literacy, and social development (see the 2012 Horizon Report for a review). Educational researchers are not only interested in investigating the effects of gesture-based devices as a means of natural input, but also in the impact and effect it may have on other aspects of learning, such as memory (Chao, Huang, Fang, & Chen, 2013) and physical rehabilitation (Chang, Chen, & Huang, 2011). A sampling of exciting projects for the application of gesture-based computing

* Corresponding author. Tel.: þ886 988660445. E-mail addresses: [email protected] (F.-R. Sheu), [email protected], [email protected] (N.-S. Chen). http://dx.doi.org/10.1016/j.compedu.2014.06.008 0360-1315/© 2014 Elsevier Ltd. All rights reserved.

F.-R. Sheu, N.-S. Chen / Computers & Education 78 (2014) 268e277

269

includes EyeDraw, EyeMusic (Johnson et al., 2012), Marracle, and KinectEDucation. It is both an emerging technology in practice and a research topic of great interest. Due to its novelty and wide ranging areas of applicability, gesture-based learning is highly intriguing to educators, learning technology specialists, and researchers alike. However, as indicated in the 2012 Horizon Report (Johnson et al., 2012), researchers and developers are gaining a realization that gesture-based computing within education requires “interdisciplinary collaborations and innovative thinking about the very nature of teaching, learning and communicating.” Consequently, an analysis of gesture-based computing in education from a research perspective is essential for acquiring a better understanding of how these goals can be achieved (or advanced) and how the technology can be utilized to support learning. In this present study, the authors have analyzed the research on gesture-based learning published in the period from 2001 to 2013. This project was undertaken to answer the following questions: (1). What is the status of the gesture-based learning articles published from 2001 to 2012? What was the growth trend of the journal publications on gesture-based learning in this period? (2). What research methods were used in these studies? (3). What gesture-based computing technology is utilized in education, specifically for teaching and learning? (4). What were the primary disciplines of these studies? In addition, what learning topics or content areas were targeted by these gesturebased learning studies? (5). How were these gesture-based computing devices applied pedagogically? Were there pedagogical differences in different learning domains (knowledge domains)?

2. Methodology The purpose of this study was to explore and gain an understanding of the characteristics and the trend of studies on gesture-based computing in education with a specific interest in gesture-based learning systems that incorporate involvement of the learners' gestures related to the learning content. Content analysis is a suitable method to achieve this purpose. Content analyses have been conducted in a variety of professional fields for a similar purpose, such as in library and information science (Chang, 2012), psychology (Howard, Cole, & Maxwell, 1987), distance education (Rourke & Szabo, 2002), educational technology (Shih, Feng, & Tsai, 2008), educational psychology, and science education (Tsai & Wen, 2005). These content analysis studies have provided valuable insights into the overall research trends in each area as well as information and characteristics of particular topics. Over time, content analysis has proven to be a highly effective research method. 2.1. Data collection The data analyzed in this study cover research articles published in academic journals from the years 2001 through 2013. The data were first retrieved individually from these five selected databases: the Education Resources Information Center Digital Library (ERIC), Educational Research Complete (ERC), Association for Computing Machinery Digital Library (ACM), the Institute of Electrical and Electronics Engineers (IEEE), and SpringerLink. The first two databases are educational. ERIC is considered the largest database in education while ERC collects most journals in education (around 1200 titles). ACM and IEEE contain the largest digital libraries in computing and engineering; a high percentage of the collection in these two databases, however, are conference proceedings. In order to include more journal articles in computer science, SpringerLink was added. One of the challenges in this study was to identify a list of search terms for a comprehensive search. As a new research trend in education, the term “gesture-based computing” is relatively new and is not yet indexed as “controlled vocabulary” in the bibliographical database. There exist many alternative terms, such as gesture-based technology, gesture-based device, motion-sensor technology, etc., as the topic is still evolving. Moreover, gesture-based computing has intricate connections with other relevant technologies, for instance speech and facial recognition, motion sensor technology, robotic technology, haptic (touch or motion-based) feedback, and natural user interfaces, to name a few. As a result, it is necessary to use multiple terms to search databases in order to retrieve as many relevant articles as possible. Table 1 lists the search terms used to obtain the bibliographic records of articles from the databases. The search terms on gesture-based computing and related technology were collected. Additional terms describing the topics on gesture-based learning were also collected. Terms are grouped into three categories: (1) gesture-based computing related terms, (2) the cognition aspect, and (3) specific technology devices. The database search began in December 2012. A final search was conducted on May 21, 2014 for possible new studies that met the selection criteria. Given the condition that a keyword search (lowest indexing level) was the default search setting for most databases, using a multiple-term search approach, it is expected to retrieve thousands of return hits with much irrelevant data. The retrieved bibliographic data (approximately 3018 items) were subsequently filtered manually by the researchers based on the relevance, publication type (academic journal articles), and published year (2001e2013). Titles and abstracts of the articles were first used for selection. In some cases, complete articles were reviewed to ensure their relevance to the topic of gesture-based learning. A total of 59 articles were identified as being relevant to the topic of gesture-based learning and were analyzed in this study. Articles were included based on the following two criteria: Table 1 Search terms. Group

Search terms/Phrases

Group 1

Gesture-based computing; gesture-based learning; gesture-based interaction; gestural-based interface; natural user interface; NUI; embodied interaction; bodily-kinesthetic intelligence; physical interaction; motion-sensor technology; haptic feedback Embodied learning; Embodiment & Cognition, body motion & learning Xbox Kinect; Nintendo Wii; iPad; iPod; iPhone; smartphone; tablet

Group 2 Group 3

270


(1). Empirical research paper. Other types of papers such as “news,” “reviews,” “editorial materials” and “commentary articles” were all excluded from the study. Literature reviews were not considered empirical research papers, and, therefore, were also excluded. (2). Containing a gesture-based learning component in which the learner's gesture involvement must relate to target learning content in the learning process. Therefore, use of gesture-based devices as a means to simply flip eBooks and receive and send information was excluded. Studies purely focusing on the technical efficiency of the device, gaming, and robot design were also excluded.

2.2. Analysis The coding categories for analyzing attributes are (a) type of research article; (b) discipline/domain of the study; (c) topic of target learning content; (d) technologydgesture-based related devices used in the learning system; (e) the setting in which the study intended to implement the technology; and (f) instructional interventions (the applications/implementation of gesture-based computing (GBC) technology used pedagogically in the studies). All methods used in the studies were identified and recorded. These methods include experimental research, design-based research, literature review, and other. Information related to the learning conditions/situations were also reviewed, such as the domain of learning, the target learning content to teach/learn, and the GBC technology used in the designed learning system to support the learning. Finally, the instructional approach/interventions applied or used for supporting and/or facilitating teaching and learning in these studies were also analyzed. 3. Results and discussion 3.1. Growth trend of literature on gesture-based teaching & learning Fig. 1 indicates that there were 59 studies related to gesture-based learning during the period of 2001e2013. As shown in Fig. 1, the total number of studies each year was small before 2010, ranging from zero to two, marking the beginning of research on gesture-based computing for learning. A considerable increase in output can be observed during the period of 2010e2013, with the number of studies reaching a peak in 2011 and 2013. Moreover, more than 80% of the literature appeared within the same last three years (2011e2013). The increasing output of studies related to gesture-based learning in the subsequent years aligns well with the predictions from the Horizon (2010) report. In that report, gesture-based devices were among the six emerging technologies that were predicted to be continuously adopted for learning in higher education. Both imply that the potential impact of gesture-based computing devices on teaching and learning is worth the attention of educators, instructional designers, and educational researchers. These reports indicate that the number of scientific publications related to gesture-based learning research is likely to continue growing in the near future. 3.2. Types of research paper In this section, the design of the research and the methods that were used in these studies are categorized and analyzed. These research articles were categorized into three major types based on the research design described in the studies: (1) experimental research, (2) designbased research, and (3) miscellaneous research. Research methods used for collecting data, analyzing data, and/or for specific research purposes in each study were also analyzed. Each type of research addresses different research purposes. Table 2 shows the distribution of the types of paper based on the 59 articles. Although two articles used more than one method (i.e., design-based research plus action research), the primary research framework was general design-based research. Therefore, these two articles were counted as design-based research. In all, 43 studies (72.9%) used an experimental research design to examine the hypothesis that gesture-based technology has an impact on learning, and all studies here show a positive impact on the measured learning outcomes. The other approximately one-third of the

Fig. 1. The growth trend of Gesture-Based Learning Studies by Year.


271

Table 2 Distribution of types of paper. Research type (# of study)

Total

(%)

Experiment research Common experimental design (27) ABAB design (14) Quasi-experimental design(2) Design-based research User studies, Laban movement analysis, multiple case studies, grounded theory, usability test, mixed method with other methods (i.e., action) Other research Action research (1) General qualitative research (2) Case study (1) Total

43

(73%)

12

(20%)

4

(7%)

59

(100%)

studies are design-based research that aimed to explore relevant issues, to gain understanding, and to generate design recommendations for learning contexts and learning conditions involving gesture-based learning. Of the 43 experimental design research studies, 14 used the A-B-A-B reversal design that measures behavior change during these four phases: (1) a baseline (the first A), (2) an intervention (the first B), (3) withdrawal of the intervention (the second A), and (4) reintroduced intervention (the second B), which aims to examine whether an intervention has the intended effect on the individual participants. The A-BA-B design uses individual participants as their own control while other research designs, such as experimental and quasi-experimental designs, are concerned with the effects within or between groups. The A-B-A-B design focuses on the behavioral change of one individual and compares the effects on and changes in behavior at different time points. It is a common research design in the field of special education for identifying and documenting solutions/interventions for people with disabilities. Even with the same disability diagnosis, individuals with disabilities often have unique needs (e.g., what method works for what tasks and under what conditions). As a result, the intervention or support will need to be adaptable in different environments and it is important information for the design of and decisions regarding assistive solutions. Multiple methods are utilized and employed in order to fulfill those purposes and respond to those research questions. Methods are hybrid and may include usability tests, user studies, and traditional qualitative techniques such as interviews and observations. Some researchers focus especially on user studies, and therefore adopt methods with a specific focus such as usability tests, card sorting, diaries, and observations of interaction with the devices. Some adopt a particular approach and practical aim, such as grounded theory and action research. About 72% of the studies used an experimental design, whereas 20% of studies relied on design-based research. This finding indicates that the major trend of the current research in the field is trying to prove the concept that the application of gesture-based computing can support teaching and learning by measuring the effect of gesture-based learning “systems” on learning performance. 20% of the studies focus on the development of the learning systems, systematically studying the design, development, and evaluation of the process of creation educational interventions, including the innovative learning system itself. As a result, design-based research methods were employed. 3.3. Disciplines/domains of the study As listed in Table 3, the majority of studies were in the domain of special education (25 studies; 42%), followed by science and math education (14 studies; 24%). This coincides with the situation in practice: a significant number of examples found in practice that gesturesensing techniques have profound implications for disabled individuals and those with special needs, including physical disability, autism, Down syndrome, blindness, and multiple impairments. The studies categorized into other fields are music, museums, manufacturing, architecture, social development, and medical training. Four articles did not mention the specific educational fields they intended to apply the technology to, particularly those studies that focused on the instructional tactics/tools that can be applied for general topic learning. Therefore, they were categorized into general education. The studies were categorized by considering two major areas: (a) the age of the target learners (or intended learners) that the system was designed for, and (b) the topic of the content that the system was intended to teach or to support. Most studies fall clearly into the sub-categories in the K-12 system, such as special education, science education, math education, and physical education. Topics that focus on social and emotional learning were categorized into general education in this paper. Five of the studies were beyond the K-12 system. Some of the categories were across ages (or not age-specific), such as special education, physical activity/physical education, and studies in general education. We analyzed them with both attributes in mind. However, for special education, the age of the learners does not often fall into the typical age classification found in most education systems. Instead, they are dependent on the disabilities or challenges the learners face and the resources available in particular schools. When studies including multiple topics were revealed, we first investigated the primary topic of the learning. If studies had equal attention among topics, we put them into a separate category, with the label of multiple cases/topics. For example, if a study involved six cases that covered chemistry, physics, and math, it was categorized into multiple topics under science and math education. It caught the researchers' attention to see a low number of studies in physical education since gesture-based devices are kinetic-centered and would naturally be considered for physical activity, especially those made commercially and ready-to-use gaming products, such as Nintendo Wii and MicroSoft Xbox Kinect. Exergame, an exercise and video game, has experienced rapid growth and is a new instructional approach to promoting physical activity. It is important to note that this area has been an emergent field in recent years. After further investigation, the researchers found that most studies in physical education were published in the domain of sports and exercise and collected in bibliographic databases such as SportsDiscs, the most comprehensive, multidisciplinary, full-text bibliographic database covering key facets of sports research. Such databases include physical fitness, exercise, sports medicine, sports science, physical education, kinesiology, coaching, training, sports administration, officiating, sports law and legislation, college and university sport, disabled persons,

272

F.-R. Sheu, N.-S. Chen / Computers & Education 78 (2014) 268e277 Table 3 Distribution of types of discipline/domain. Edu. domain/Disciplines

Target ages of learning

Total

(%)

Special education Science & math education Physics (8) Math (4) Chemistry (1) Multiple topic on science (1) Physical education General education Other disciplines Music(1) Architecture(1) Manufacture(1) Medical training(1) Car design(1) Athletic training (1)

Cross age (K-12 -Adult) K-Elementary

25 14

(42%) (24%)

Cross age (K-12-adult) Cross age (K-adult)

4 10 6

(7%) (17%) (10%)

59

(100%)

Kids Adult Adult Adult Adult Adult Total

(higher (higher (higher (higher (higher

ed) ed) ed) ed) ed)

facility design and management, intramural and school sport, doping, health, health education, biomechanics, movement science, injury prevention rehabilitation, physical therapy, nutrition, exercise physiology, sports and exercise psychology, recreation, leisure studies, tourism, allied health, occupational health and therapy, public health, and more. By searching with the keyword “exergame” alone in the database of SportsDiscs, the research team retrieved 107 articles. In the case of the studies retrieved for this study, they were published in Problems of Education in the 21st Century, Computer and Education, Journal of Technology Integration in the Classroom, where there is a special interest in educational technology. 3.4. Topics of learning content Table 4 lists all the topics based on the 59 GBC related empirical studies. The topics across these educational domains vary. Most of them focus on motor skills type of learning (18 studies; 31%), including motor skills, behavior skills training, etc. These topics are alike in the sense that the primary goal or target skills to learn were kinetic-centered. When grouping 22 topics into three domains, that is, conceptual knowledge, procedural knowledge, and miscellaneous, the results demonstrated that both of learning contents (procedural and conceptual) are about equally distributed. 3.5. Technology (GBC devices) The technology listed here includes the major devices used to create a gesture-based learning environment based on the reviewed articles. As shown in Fig. 2, the most-utilized gesture-based device is Nintendo Wii (26 studies; 44%), including Wii remote, Wii balance board, Wii Fit, Wii console and Wii Nunchuk. MicroSoft Xbox Kinect, another popular commercially made product, was found to be the primary device in nine studies (15%). There are eight cases (studies) using their own design, combined with multiple gesture-based devices. These commercially made devices were mostly directly used without much modification in terms of technology. They were integrated into the learning activities. In addition, the emphasis of these studies is on the implementation/utilization of the technology without huge Table 4 Distribution of topic. Knowledge domain

#

Topics

Total

(%)

Procedural

1. 2. 3. 4. 5. 6. 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Motor skills learning (Including rehab) (12) Behavior skills training (BST) (6) Physical Activity (5) Breast examination training (1) Assembly/disassembly (1) Architecture configurability (1) Physics experiment (8) Math (5) Chemistry (1) Science (1) Animal behavior (1) Music (sound, tempo etc.) (1) Oral hygiene (1) Reading/Literacy (1) Concept of Multiple Intelligence (2) Car design (1) General concept (1) Eng. Vocabulary (as 2nd language) (1) Emotion and social development (6) Play therapy/game play (approach) (2) Story telling (approach) (1) Total

26

(44%)

24

(41%)

9

(15%)

59

(100%)

Conceptual

Miscellaneous


273

exertion involved in developing it, which is also one of the major reasons why the technology is affordable for teaching and learning. This is especially true in the field of special education. Of 26 studies using Wii related devices, eighteen (approx. 65%) were in special education, seven (27%) in physics, three (12%) in physical education/physical activity, and one (5.88%) in storytelling, which was categorized in general education in this review. They are all sub-domain of education, and the intended settings are schools. According to these articles, one of the major reasons for using this device is its affordability, at the relatively low cost of $130 compared with other devices, such as interactive whiteboards (priced from US $700 to $2200). This does not include those custom-made alternative devices for persons with special needs, which can be very expensive. In addition, they can be applied without having a computer programming background. In many cases, very little to no modification is required for educators to use it in their curriculum. This is the socalled “Wii therapy” in special education. One can find a significant number of websites, blogs and projects on “Wii therapy” for special needs individuals. In a few studies that involved complex cognitive and procedural types of learning, the learning systems are customized or designed specifically to meet the needs and the objectives of the learning. Examples include the breast examination procedure in medical training, assembly/disassembly training in manufacturing, and exploring design arrangements in architecture. Another interesting group is comprised of the (digital) tangible objects for learning. The learning systems used and described in these studies were created to meet very specific needs and learning objectives. For example, a giant tooth and toothbrush for children to learn oral hygiene (Sylla, Brance, Coutinho, & Coquet, 2012) and various objects responding to children's actions for learning tunes and tempo in music learning (Bakker, Antle, & van den Hoven, 2012). Although it is a very different “shape” or “line” of instructional technology, they share the same underpinning notion of embodiment-based learning, that is, body movement shapes cognition or cognition develops through bodies interacting with the external world. Studies have revealed the potential benefits of such embodied interaction for learning and development (Marshall, 2007; Marshall, Price, & Rogers, 2003; Mazalek & van den Hoven, 2009; O'Maley & Stanton, 2004; Zuckerman, Arida, & Resnick, 2005). The fields/domains of the studies that designed and developed their own gesture-based learning systems varied. They include studies in the medical field for breast examination training, music (MoSo), special education for autism, the manufacturing industry for assembly and disassembly training, and architecture for configurability practice. The learning contents of these studies are complex and skill specific such that one single device or ready-to-use commercial gaming product is unable to support the learning without requiring heavy modifications. In the case of manufacturing, gesture-based devices include the data glove, motion tracker, haptic feedback and virtual reality (VR). In architecture, devices include sensors, tags, cameras, projectors, etc. Lastly, in medical training, primary devices include physical breast simulators and VR. 3.6. Intended settings Shown in Table 5 are the results of the setting in which the gesture-based learning is intended to be implemented. Three major settings are classroom/school learning (64%), rehabilitation and physical therapy (22%), and job training (14%). Given that the larger context of these studies is in education, it is no surprise that the intended setting of the learning system designed in most studies is the classroom or school. However, given the change in learning resulting from the advancement of other emergent educational technologies such as mobile learning and ubiquitous learning, virtual learning places, etc., physical categories for learning setting are becoming increasingly difficult to define. Designed learning can be anywhere and, as a matter of the fact, that is the trend and the way of learning for the future. In addition to the settings mentioned in the studies, the applications/implementations of gesture-based learning found in these studies can be applied to multiple settings as well. In other words, we can also predict that the intended physical setting for learning in the future would be fuzzy to define. Or, there would not be a physical constraint for the design of learning systems; it could literally be anywhere at any time, even with gesture-based learning. 3.7. Technology intervention for learning: How gesture-based computing devices are used for supporting teaching and learning within different sub-education domains 3.7.1. Special education (25 studies) GBC was used for performance support (job aid and assistive technology for daily use), training materials/systems (job training), and physical therapy (rehabilitation). Among these articles, Shih and colleagues conducted a series of studies (Shih, 2011; Shih & Chang, 2012;

Fig. 2. Distribution of GBC Technology.

274

F.-R. Sheu, N.-S. Chen / Computers & Education 78 (2014) 268e277 Table 5 Intended settings for implementation. Setting

Total

(%)

School/Classroom Rehab/therapy Job training Total

38 13 8 59

(64 %) (22 %) (14 %) (100%)

Shih, Chang & Mohua, 2012; Shih, Chen & Shih, 2012; Shih, Chung, Shih & Chen, 2011; Shih, Shih & Chiang, 2010; Shih, Shih & Chu, 2010; Shih, Shih & Shih, 2011; Shih, Wang & Chang, 2012) on using Wii and related products to support learners with both physical and cognitive difficulties to learn daily life skills (motor skills) and specific job tasks (job training), physical rehabilitation, and other behavior skills training. Shih's team also turned Wii into an affordable assistive technology, as a switch that can be operated by people with physical difficulty handling typical switches. Interestingly, Burke, Andersen, Bowen, Howard, and Allen (2010) combined iPhones and other add-on devices as a performance support system that facilitates the learning of skill sequences by simplifying complex tasks and providing cues (just-in-time support). Petersson and Brooks (2007) used a robotic device and camera to design an environment that allows children (4e6 years old) to play games as a type of play therapy. GBC, specifically Exergame, was also used in special education for promoting physical activities (Hourcade, Bullock-Rest, & Hansen, 2012; Porayska-Pomsta et al., 2012). These studies show that these applications of GBC have a very positive impact on the learning results of learners with physical difficulties, cognitive difficulties, or both. The primary intervention of the technology used is for performance support by prompting cues which provide just-in-time information that is needed at the time of performing the tasks. Another major intervention is using GBC as a means of adaptive and assistive technology for learning. These findings are especially encouraging given that the cost is relatively lower than that of customized assistive technology. 3.7.2. Science and math education (14 studies) Studies in this category include topics on physics, chemistry, and mathematics. These articles described such topics as the use of GBC to support scientific discovery learning by providing simulations and digital manipulations. Some of these studies used GBC as a tool as part of larger experiments. Researchers such as Ochoa, Roonev, and Somers (2011), and Wheeler (2011) used Wiimote as a versatile and easy-to-use measuring device and made it part of their designed experiments and demonstration projects for Newton's Third Law (by colliding two n (2011) used an Wiimote-attached carts) and for a simulation of weightlessness (during a free fall of a dropped Wiimote). Pendrill and Rohle iPhone 4 to collect data of acceleration and rotation in a pendulum ride. rrez, Huth, and Lee (2011) and Abrahamson, Gutie rrez, Charoenying, Negrete, and Bumbacher (2012) created Abrahamson, Trninic, Gutie their own GBC learning system, called MIT, which consisted of a combination of various technologies, such as screen, program and controller for discovery-based math instruction. In this project, lists of instructional tactics (tutorial tactics) were embedded into discovery learning. The tactics for general contexts include Para-content framing, clinical-interview techniques, and discursive practices. Other tactics for specific purposes include establishing a joint problem space, eliciting orientation of view, strategies, reasoning, vocabulary, and knowledge (see Abrahamson et al., 2012 for a full list of tactics.) There are several other studies in this area. For instance, Majgaard, Misfeldt, and Nielsen (2011) created digital manipulations (robotic education tool and number blocks, I-BLOCKS) for children to learn place value and the concept of numbers. Christodoulou, Garyfallidou, Ioannidis, and Papatheodorou (2008) used haptic technology to create simulations for various science concepts for students to explore as a “metaphor” of feeling associated with the concepts. In general, simulations, digital manipulations, and tools for experiments were the primary types of instructional interventions reported when using GBC in science education. 3.7.3. General education (10 studies) Application of GBC is also found in learning emotions, social behaviors, and in facilitating collaboration. Barakova and Lourens (2010) used a robot in a game for children to learn emotions and social behaviors. Similarly, Bekker, Sturm, and Barakova (2010), Bekker, Sturm, and Eggen (2010) used six designs of interactive play objects for children to interactively play with objects to simulate social interaction and physical play. Another creative application of GBC was the creation of tangible objects for learning oral hygiene (see Sylla et al., 2012). In this project, Sylla and her colleagues designed a giant tooth and toothbrush for preschool children to interact with. Through playing, children learn the abstract concepts of germs on teeth and how to clean them. 3.7.4. Physical education (7 studies) Di Tore, Aiello, et al. (2012), Di Tore, D'Elia, et al. (2012) used Kinect for a visual-motor game to improve fifth grade students' integration skills. In terms of older learners, Grieser, Gao, Ransdell, and Simonson (2012) used Wii Fit with college students for balance training. Vernadakis, Gioftsidou, Antoniou, Ioannidis, and Giannousi (2012) compared Wii Fit and traditional methods of balance training with college students and found that Wii Fit is equally as effective as traditional training approaches. While there are other traditional approaches that are equally effective for fitness training, video-game exercise (Exergame) brings additional motivation and fun into the activity. Additionally, when using these technologies, exercise and activity are logged automatically and can be compared. Such data can also be used for further instructional strategies, such as showing accomplishments, friendly competition in games among students, and visualization of each part of the exercise. There is a strong parallel between the phenomena of “Wii therapy” in special education and “Exergame” in physical education. These GBC enabled learning systems or instructional approaches are effective for similar reasons, that is their affordability, acceptability, and the low technical skills required to use and maintain them. Students are familiar with the devices and, with minor or no modifications, they can be turned into an instructional technology (or assistive technology). As such, no high level programming skills are required to run the system. Often, these devices bring additional motivation and learner engagement.


275

3.7.5. Other education-related fields (6 studies) Of the 59 studies, six were used outside of the K-12 education system. For instance, Kotranza, Lok, eladisma, Pugh, and Lind (2009) created a simulation with physical fake breast, sensor, virtual reality, and glasses for medical students to learn breast examination procedures. As another example, Aleotti and Caselli (2011) used a data glove, motion tracker, and haptic feedback device to operate and manipulate objects in VR for product assembly and disassembly training. Binder et al. (2004) created a physical interface (tangible objects) for architecture and interaction design students for inspirational learning. The use of GBC in these three studies was more complicated than the cases we have discussed in previous sections. The scale and complexity of the learning content in the reported learning context in these studies were also higher. The interventions were required to provide proper simulations when going through each stage of complex procedural tasks. By providing a safe and lower cost environment (compared to real materials and situations in these fields), learners were able to engage in learning fairly complex and higher level types of skills, such as imitation, manipulation, developing precision, and articulation, so that these skills became automatized (neutralization) (Dave, 1970). 3.7.6. Summary Given the differences of primary instructional goals in each sub-education domain, the emphasis of interventions used in these reviewed studies varies from one domain to another. Some interventions emphasize on the engagement by the stimulations enabled by the GBC devices; some stress more on the ability to learn with physical activity; and some focus on the control of environmental stimulation as assistive technology. Whether or not they are used for lower level learning (i.e. memorizing concept, and repeat motor skills) or higher level complex learning (i.e. problem-solving, discovering, decision-making etc.), they all share the same underpinning theory: embodied cognition, which claims that bodily action will have influences on our cognition and that the motor system is connected to the mind system (Anderson, 2003; Beilock and Goldin-Meadow, 2010; Clark, 1999). Gestures or motor actions can externalize thoughts or mental representations (Alibali & Goldin-Meadow, 1993; Goldin-Meadow & Wagner, 2005) as conceptual metaphor and image schema (Gallese & Lakoff, 2005). Several studies indicated the same argument that body and mind are connected, and therefore body movements would assist learners in acquiring knowledge (psychomotor, conceptual, and affective knowledge), such as mirror neuron system and observational learning (Rizzolatti & Craighero, 2004; Van Gog, Paas, Marcus, Ayres, & Sweller, 2009), animations for motor skill learning (Wong et al., 2009), stimulations for discover learning (Abrahamson et al., 2012), as well as the studies reviewed in this study. 4. Limitations The applications in other professional areas are also notable. Significant amount of examples can be found in rehabilitation medical care (Chang, Han, & Tsai, 2013; Lee et al., 2013; Lin, Hsieh, & Lee, 2013; Yeh et al., 2012). The application of GBC for training/learning in some fields are complicated and often designed for higher level learning (such as medical procedural training and pilot training, etc.), therefore requiring rather sophisticated and complicated technology. This might explain why it might not be as widely spread in education in terms of number of uses. However, given the high potential of using GBC technology for learning and training, the rapid growth of applications in other fields are predictable if it is not happening already. When authors went through database search and literature, we also found studies in other fields, such as physical rehabilitation, medical procedure training, and exercise. We estimate application of gesture-based computing for learning, and training is also common in the field of physical therapy, especially the implementation in the root level, such as rehab exercise, at home program etc. Physical therapy and rehabilitation are kinetic centered, which involves practice in motor skills learning. Similar to practice in special education, physical therapy or rehabilitation in medical field faces some challenges as the therapy is often repetitive and boring. Gesture-based applications for rehab purposes are expecting to grow rapidly. This study focused primarily on educational and computing databases. Since gesture-based computing is also used for learning in practice in other professional fields, it would be interesting to see future studies investigating into other professional fields as well, such as medical training, physical therapy, the military, astronautics, pilot training, business, sports and exercise science. Such research would provide another perspective on how gesture-based technology can be utilized for learning fields other than education. 5. Implications and conclusions In this paper, the researchers used content analysis to categorize and analyze 59 empirical studies published from 2001 to 2013 that are related to the application of gesture-based computing in education. This study revealed the distribution and trends in research methods, the disciplines of the studies, learning content, the gesture-based enabled technology used, and the intended setting of the gesture-based learning system. As a whole, the results of the analysis show that gesture-based computing research is a growing field in education. In fact, most of the studies were published between the years 2010 and 2013. The majority of these studies (43 studies; 73%) used an experimental design as their primary research method to test the feasibility of using gesture-based computing to support teaching and learning. There is also a clear trend that many studies on this topic used design-based research, aiming to gain better understanding of how and why this type of innovation supports teaching and learning. In terms of technology, the most commonly used gesture-based computing devices are Nintendo Wii and related products because they are affordable, acceptable, and require low technical maintenance. Other commercial ready-to-use products, such as MS Kinect and smart phones were also used for the same reasons in education. These are especially important factors in special education since other customized adaptive and assistive technology is often very expensive and requires high maintenance. It is predictable that as technology becomes more advanced and more affordable in the future, the potential of using it for learning will be even greater. Applications of this type of GBC-enabled instructional system are found most in special education. GBC-enabled system provides more natural users interfaces that allow users to interact with computing system more naturally by using their body movements, gestures, voice, or even brainwave. This is especially helpful for users with physical limitations. Studies have found that gesture-based video gaming devices have a significant impact on people with special needs. Several profound studies have shown the effectiveness of such technology in supporting learning and engaging learners, especially with autistic learners. With the important driving factors, such as affordances and low

276


technical maintenance, we will see more applications and implementations of gesture-based computing in supporting teaching and learning in this field. Regarding the topics and types of learning domains, procedural types of learning are well supported by the gesture-based computingenabled learning systems (or approaches). Although conceptual and procedural learning cannot be separated completely, more than half of the studies placed more emphasis on procedural types of learning and training. It is reasonable to take advantage of gesture-based technology in procedural learning domains, as one of its essential characteristics is kinetic-centered. Finally, in addition to employing tangible or embodied interaction for learning and development, the types of instructional intervention found in supporting procedural types of learning were mostly aimed to enhance learning effectiveness during the process by providing support and stimulation, such as just-in-time information, cuing systems, simulation of operations, etc. In special education, such support mechanisms were even more salient and, in addition, they were utilized in assistive and adaptive roles. In science education, the researchers found numerous GBC applications used as a tool or as part of an exploring learning process. In math education, digital tangible learning objects (e.g., digital manipulatives) are used to learn abstract math concepts. Because gesture-based computing for learning is likely to become one of the primary trends in education, as predicted by the Horizon Report three years in a row, the supporting educational theories and design guidelines are important for better system design, development, and implementation. Pedagogical issues and instructional tactics that respond to this particular type of learning context will also need to be studied. This study also aims to provide an opportunity for researchers across disciplines to think about future studies on gesture-based learning. The use of gesture-based enabled devices as part of emergent learning technology (gesture-based learning) is related to disciplines such as educational technology, instructional design, human computer interaction, machine learning, artificial intelligence, psychology, learning sciences, psychometrics, and various branches of computational engineering. Therefore, innovative cross-disciplinary research and related publications that document specific gesture-based learning systems and their associated designs are now vital. Once completed, educational researchers can better understand how gesture-based computing can be effectively utilized and implemented for teaching and learning across a wide array of educational settings. Acknowledgments This research was supported by the National Science Council, Taiwan under project numbers NSC 102-2911-I-110-501, NSC 101-2511-S110-003-MY3, NSC 100-2511-S-110-001-MY3, and NSC 99-2511-S-110-004-MY3. References rrez, J., Charoenying, T., Negrete, A., & Bumbacher, E. (2012). Fostering hooks and shifts: tutorial tactics for guided mathematical discovery. Technology. Abrahamson, D., Gutie Knowledge & Learning, 17(1e2), 61e86. rrez, J. F., Huth, J., & Lee, R. G. (2011). Hooks and shifts: a dialectical study of mediated discovery. Technology, Knowledge & Learning, 16(1), Abrahamson, D., Trninic, D., Gutie 55e85. http://dx.doi.org/10.1007/s10758-011-9177-y. Aleotti, J., & Caselli, S. (2011). Physics-based virtual reality for task learning and intelligent disassembly planning. Virtual Reality, 15(1), 41e54. Alibali, M., & Goldin-Meadow, S. (1993). Gesture-speech mismatch and mechanisms of learning: what the hands reveal about a child's state of mind. Cognitive Psychology, 25(4), 468e523. Anderson, M. L. (2003). Embodied cognition: a field guide. Artificial Intelligence, 149(1), 91e130. Bakker, S., Antle, A. N., & Van Den Hoven, E. (2012). Embodied metaphors in tangible interaction design. Personal and Ubiquitous Computing, 16(4), 433e449. Barakova, E. I., & Lourens, T. (2010). Expressing and interpreting emotional movements in social games with robots. Personal Ubiquitous Computing, 14(5), 457e467. http:// dx.doi.org/10.1007/s00779-009-0263-2. Beilock, S., & Goldin-Meadow, S. (2010). Gesture changes thought by grounding it in action. Psychological Science, 21, 1605e1610. Bekker, T., Sturm, J., & Barakova, E. (2010). Design for social interaction through physical play in diverse contexts of use. Personal Ubiquitous Computing, 14(5), 381e383. http:// dx.doi.org/10.1007/s00779-009-0269-9. Bekker, T., Sturm, J., & Eggen, B. (2010). Designing playful interactions for social interaction and physical play. Personal Ubiquitous Computing, 14(5), 385e396. Binder, T., De Michelis, G., Gervautz, M., Jacucci, G., Matkovic, K., Psik, T., et al. (2004). Supporting configurability in a mixed-media environment for design students. Personal and Ubiquitous Computing, 8(5), 310e325. http://dx.doi.org/10.1007/s00779-004-0294-7. Burke, R. V., Andersen, M. N., Bowen, S. L., Howard, M. R., & Allen, K. D. (2010). Evaluation of two instruction methods to increase employment options for young adults with autism spectrum disorders. Research in Developmental Disabilities: A Multidisciplinary Journal, 31(6), 1223e1233. Chang, C. Y., Chien, Y. T., Chiang, C. Y., Lin, M. C., & Lai, H. C. (2013). Embodying gesture-based multimedia to improve learning. British Journal of Educational Technology, 44(1), E5eE9. http://dx.doi.org/10.1111/j.1467-8535.2012.01311.x. Chang, Y. J., Chen, S. F., & Huang, J. D. (2011). A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. Research in Developmental Disabilities: A Multidisciplinary Journal, 32(6), 2566e2570. http://dx.doi.org/10.1016/j.ridd.2011.07.002. Chang, Y. W. (2012). Tracking Scientometric research in Taiwan using bibliometric and content analysis. Journal of Library and Information Studies, 10(2), 1e20. Chao, K. J., Huang, H. W., Fang, W. C., & Chen, N. C. (2013). Embodied play to learn: exploring Kinect-facilitated memory performance. British Journal of Educational Technology, 44(5), E151eE155. http://dx.doi.org/10.1111/bjet.12018. Christodoulou, S. P., Garyfallidou, D. M., Ioannidis, G. S., & Papatheodorou, T. S. (2008). Interactive education based on haptic technologies and educational testing of an innovative system. International Journal of Emerging Technologies in Learning, 3(2), 4e19. Clark, A. (1999). An embodied cognitive science? Trends in Cognitive Science, 3, 345e351. Dave, R. H. (1970). Psychomotor levels. In R. J. Armstrong (Ed.), Developing and writing educational objectives (pp. 33e34). Tucson, AZ: Educational Innovators Press. Di Tore, S., Aiello, P., Palumbo, C., Vastola, R., Raiola, G., D'Elia, F., et al. (2012). Sensory motor interaction in virtual environment to promote teaching-learning process. Problems of Education in the 21st Century, 42, 29e37. Di Tore, S., D'Elia, F., Aiello, P., Carlomagno, N., & Sibilio, M. (2012). Didactics, movement and technology: new frontiers of the humanemachine interaction. Journal of Human Sport and Exercise, 7(1), 178e183. Gallese, V., & Lakoff, G. (2005). The brain's concept: the role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, 22(3/4), 455e479. Goldin-Meadow, S., & Wagner, S. (2005). How our hands help us learn. Trends in Cognitive Sciences, 9, 234e240. Grieser, J. D., Gao, Y., Ransdell, L., & Simonson, S. (2012). Determining intensity levels of selected wii fit activities in college aged individuals. Measurement in Physical Education and Exercise Science, 16(2), 135e150. Hourcade, J. P., Bullock-Rest, N. E., & Hansen, T. E. (2012). Multi-touch tablet applications and activities to enhance the social skills of children with autism spectrum disorders. Personal Ubiquitous Computing, 16(2), 157e168. Howard, G. S., Cole, D. A., & Maxwell, S. E. (1987). Reserach roductivity in psychology based on publication in the journals of the American Psychological Association. American Psychologist, 42(11), 975e986. http://dx.doi.org/10.1037/0003-066X.42.11.975. Johnson, L., Smith, R., Willis, H., Levine, A., & Haywood, K. (2011). The 2011 horizon report. Austin, Texas: The New Media Consortium. Johnson, L., Adams, S., & Cummins, M. (2012). The 2012 horizon report. Austin, Texas: The New Media Consortium.


277

Johnson, L., Levine, A., Smith, R., & Stone, S. (2010). The 2010 horizon report. Austin, Texas: The New Media Consortium. Kotranza, A., Lok, B., Deladisma, A., Pugh, C. M., & Lind, D. S. (2009). Mixed reality humans: evaluating behavior, usability, and acceptability. Visualization and Computer Graphics, IEEE Transactions on, 15(3), 369e382. http://dx.doi.org/10.1109/TVCG.2008.195. Lee, C., Park, H., Kim, J., Kim, B., Kim, L., & Kwon, G. H. (2014). Motor rehabilitation based on brain machine interface and Microsoft Kinect. In: Proceedings of the 2014 IEEE International Conference on Consumer Electronics (ICCE), (pp. 236-239). Lin, T. Y., Hsieh, C. H., & Lee, J. D. (2013). A kinect-based system for physical rehabilitation: utilizing tai chi exercises to improve movement disorders in patients with balance ability. In: Proceedings of the 2013 7th Asia Modelling Symposium (AMS), (pp. 149-153). Majgaard, G., Misfeldt, M., & Nielsen, J. (2011). How design-based research and action research contribute to the development of a new design for learning. Designs for Learning, 4(2), 8e27. Marshall, P. (2007). Do tangible interface enhance learning?. In Proceedings of the first international conference on tangible and embeded interaction (TEI 2007), Baton Rouge, Louisiana, (USA) (pp. 163e170). Marshall, P., Price, S., & Rogers, Y. (2003). Conceptualising tangibles to support learning. In Proceedings of the second conference on interaction design and children (IDC 2003), London (pp. 101e109). Mazalek, A., & Van Den Hoven, E. (2009). Framing tangible interaction frameworks. Artificial Intelligence for Engineering Design, Analysis and Manufacturing, 23(3), 225. Ochoa, R., Rooney, F. G., & Somers, W. J. (2011). Using the Wiimote in introductory physics experiments. The Physics Teacher, 49(1), 16e18. O'Malley, C., & Fraser, D. S. (2004). Literature review in learning with tangible technologies. Discussion Paper. utureLab. n, J. (2011). Acceleration and rotation in a pendulum ride, measured using an iPhone 4. Physics Education, 46(6), 676e681. Pendrill, A., & Rohle Petersson, E., & Brooks, A. (2007). Non-formal therapy and learning potentials through human gesture synchronised to robotic gesture. Universal Access in the Information Society, 6(2), 167e177. Porayska-Pomsta, K., Frauenberger, C., Pain, H., Rajendran, G., Smith, T., Menzies, R., et al. (2012). Developing technology for autism: an interdisciplinary approach. Personal Ubiquitous Computing, 16(2), 117e127. http://dx.doi.org/10.1007/s00779-011-0384-2. Rizzolatti, G., & Craighero, L. (2004). The mirror-neuron system. Annual Review of Neuroscience, 27, 169e192. Rourke, L., & Szabo, M. (2002). A content analysis of the “journal of distance education” 1986e2001. Journal of Distance Education, 17(1), 63e74. Shih, C. H. (2011). A standing location detector enabling people with developmental disabilities to control environmental stimulation through simple physical activities with Nintendo Wii balance boards. Research in Developmental Disabilities: A Multidisciplinary Journal, 32(2), 699e704. Shih, C. H., & Chang, M. L. (2012). A wireless object location detector enabling people with developmental disabilities to control environmental stimulation through simple occupational activities with Nintendo Wii balance Boards. Research in Developmental Disabilities: A Multidisciplinary Journal, 33(4), 983e989. Shih, C. H., Chang, M. L., & Mohua, Z. (2012). A three-dimensional object orientation detector assisting people with developmental disabilities to control their environmental stimulation through simple occupational activities with a Nintendo Wii Remote controller. Research in Developmental Disabilities: A Multidisciplinary Journal, 33(2), 484e489. Shih, C. H., Chen, L. C., & Shih, C. T. (2012). Assisting people with disabilities to actively improve their collaborative physical activities with Nintendo Wii balance boards by controlling environmental stimulation. Research in Developmental Disabilities: A Multidisciplinary Journal, 33(1), 39e44. Shih, C. H., Chung, C. C., Shih, C. T., & Chen, L. C. (2011). Enabling people with developmental disabilities to actively follow simple instructions and perform designated physical activities according to simple instructions with Nintendo Wii balance boards by controlling environmental stimulation. Research in Developmental Disabilities: A Multidisciplinary Journal, 32(6), 2780e2784. Shih, C. H., Shih, C. J., & Shih, C. T. (2011). Assisting people with multiple disabilities by actively keeping the head in an upright position with a Nintendo Wii remote controller through the control of an environmental stimulation. Research in Developmental Disabilities: A Multidisciplinary Journal, 32(5), 2005e2010. Shih, C. H., Shih, C. T., & Chiang, M. S. (2010). A new standing posture detector to enable people with multiple disabilities to control environmental stimulation by changing their standing posture through a commercial Wii Balance board. Research in Developmental Disabilities: A Multidisciplinary Journal, 31(1), 281e286. Shih, C. H., Shih, C. T., & Chu, C. L. (2010). Assisting people with multiple disabilities actively correct abnormal standing posture with a Nintendo Wii balance board through controlling environmental stimulation. Research in Developmental Disabilities: A Multidisciplinary Journal, 31(4), 936e942. Shih, C. H., Wang, S. H., & Chang, M. L. (2012). Enabling people with developmental disabilities to actively perform designated occupational activities according to simple instructions with a Nintendo Wii remote controller by controlling environmental stimulation. Research in Developmental Disabilities, 33(4), 1194e1199. http://dx.doi.org/ 10.1016/j.ridd.2012.02.008. Shih, M. L., Feng, J., & Tsai, C. C. (2008). Research and trends in the field of e-learning from 2001 to 2005: a content analysis of cognitive studies in selected journals. Computers & Education, 51(2), 955e967. http://dx.doi.org/10.1016/j.compedu.2007.10.004. Sylla, C., Branco, P., Coutinho, C., & Coquet, E. (2012). TUIs vs. GUIs: comparing the learning potential with preschoolers. Personal Ubiquitous Computing, 16(4), 421e432. Tsai, C. C., & Wen, L. M. C. (2005). Research and trends in science education from 1998 to 2002: a content analysis of publication in selected journals. International Journal of Science Education, 27(1), 3e14. http://dx.doi.org/10.1080/0950069042000243727. Van Gog, T., Paas, F., Marcus, N., Ayres, P., & Sweller, J. (2009). The mirror neuron system and observational learning: implications for the effectiveness of dynamic visualizations. Educational Psychology Review, 21(1), 21e30. Vernadakis, N., Gioftsidou, A., Antoniou, P., Ioannidis, D., & Giannousi, M. (2012). The impact of Nintendo Wii to physical education students' balance compared to the traditional approaches. Computers & Education, 59(2), 196e205. Wheeler, M. D. (2011). Physics experiments with Nintendo Wii controllers. Physics Education, 46(1), 57e63. Wojciechowski, A. (2012). Hand's poses recognition as a mean of communication with natural user interfaces. Bulletin of the Polish Academy of Sciences-technical Science, 60(2), 331e336. http://dx.doi.org/10.2478/v10175-012-0044-3. Wong, A., Marcus, N., Ayres, P., Smith, L., Cooper, G. A., Paas, F., et al. (2009). Instructional animations can be superior to statics when learning human motor skills. Computers in Human Behavior, 25(2), 339e347. Yeh, S.C., Hwang, W.Y., Huang, T.C., Liu, W.K., Chen, Y.T., & Hung, Y.P. (2012). A study for the application of body sensing in assisted rehabilitation training. In: Proceedings of the IEEE 2012 International Symposium on Computer, Consumer and Control (IS3C), (pp. 922-925). Zuckerman, O., Arida, S., & Resnick, M. (2005). Extending tangible interfaces for education: digital montessori-inspired manipulatives. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI 2005), Portland, Oregon (pp. 859e868).