applied sciences Article
Interactive Rhythm Learning System by Combining Tablet Computers and Robots Chien-Hsing Chou * and Yung-Long Chu Department of Electrical Engineering, Tamkang University, New Taipei City 25137, Taiwan;
[email protected] * Correspondence:
[email protected]; Tel.: +886-2-2621-5656 (ext. 2763) Academic Editor: Hung Nguyen Received: 30 December 2016; Accepted: 4 March 2017; Published: 7 March 2017
Abstract: This study proposes a percussion learning device that combines tablet computers and robots. This device comprises two systems: a rhythm teaching system, in which users can compose and practice rhythms by using a tablet computer, and a robot performance system. First, teachers compose the rhythm training contents on the tablet computer. Then, the learners practice these percussion exercises by using the tablet computer and a small drum set. The teaching system provides a new and user-friendly score editing interface for composing a rhythm exercise. It also provides a rhythm rating function to facilitate percussion training for children and improve the stability of rhythmic beating. To encourage children to practice percussion exercises, a robotic performance system is used to interact with the children; this system can perform percussion exercises for students to listen to and then help them practice the exercise. This interaction enhances children’s interest and motivation to learn and practice rhythm exercises. The results of experimental course and field trials reveal that the proposed system not only increases students’ interest and efficiency in learning but also helps them in understanding musical rhythms through interaction and composing simple rhythms. Keywords: percussion learning; interactive learning; robot; jazz drum
1. Introduction Music has three components, namely rhythm, melody, and harmony, of which rhythm is the most crucial [1,2]. Music loses its dynamics and becomes a combination of monotonous tones in the absence of rhythm, regardless of the presence of a beautiful melody or rich harmony. Music education experts have indicated that the visual and auditory nerves of children undergo the most rapid development between the ages of 4 and 12 years. Therefore, this period is optimal for introducing children to music. Music education is generally categorized into two types based on the teaching methods [1]: 1.
2.
Teaching through physical movements: Teachers play music and guide children in moving their bodies to music, thus enabling the children to experience rhythms through various physical movements [2]. Teaching using musical instruments: This is the most common method. By playing with various musical instruments, children learn about aspects of music (e.g., scales, notes, musical scores, melody, and rhythm).
Playing a percussion instrument may be relatively difficult for young children. Teachers must select appropriate instruments for young children according to their age and degree of muscle development. Moreover, children of this age usually face difficulties in remaining focused for a period of time. Learning outcomes may also be undermined if the teaching materials are uninteresting. Therefore, an ideal teaching style should enable children to learn in a pleasurable environment and Appl. Sci. 2017, 7, 258; doi:10.3390/app7030258
www.mdpi.com/journal/applsci
Appl. Sci. 2017, 7, 258
2 of 14
apply appropriate teaching aids, in addition to increasing the learning motivation and outcomes of children. To improve children’s learning motivation, interactive systems that involve the music instruments with electronic techniques are developed for learning music in recent years [3–14]. Bae et al. developed “Drum On” to reduce boredom and other limitations [3]. Drum On projects animated images on the drum and directly provides rhythm cues to the drummer. Kanke et al. developed a learning system for enabling users to efficiently play percussion instruments [4]. This system uses haptic and visual indications to improve users’ rhythm internalization performance. Li and Fu developed an augmented reality system for practicing or performing a musical instrument [5]. In this system, users can play a musical instrument through a visual prompt. Baba et al. developed a scanning device called “Gosen” that can play handwritten musical notes [6]. Gosen can help a user read musical scores, even for users who are not adequately versed in reading such scores. Raymaekers et al. developed “Game of Tones”, which projects an animated image on a piano [7]. This image enables a user to play the piano without knowing how to read a musical score. Tsui et al. developed a smartphone-based platform for performing a one-man orchestra [8]. On this platform, a user uses traditional bimanual conducting gestures with leap motions to direct the orchestra. Bengler and Bryan-Jinns developed “Polymetros”, an interactive system for musical experiences [9]. This system supports multiple players, and the users need not have musical skills. Yazdi and Lee developed “E-Santur”, an interactive game-based application with music instructions [10]. This application helps children learn music through interesting and engaging games. Waranusast et al. developed “muSurface”, an interactive surface with ability to interact with tangible musical symbols [11]. Users place tangible music symbols on the virtual musical staffs displayed on the playing surface, and muSurface plays the resulting melody according to the symbols. Hoffman and Weinberg developed “Shimon”, an interactive music system [12]. Users use choreographic gestures to direct a jazz-playing robot. Rasamimanana et al. developed “Urban Musical Game”, which uses sports balls [13]. This system plays a variety of sounds with identifying physical dynamics according to how the ball is manipulated. Lee et al. developed the interactive “TuneTrain” app to help children learn music using interesting games [14]. Since the release of the iPhone in 2008, mobile devices have become prevalent and integral to the lives of numerous people. Various music applications are currently available on mobile devices [15–19]. Toothy Buddies developed “Carnival of Animals”, which combines animals and interactive learning with an audiobook [15]. This app can enhance children’s musicality in terms of duration, rhythm, melody, phrase, and music form. Apple Inc. developed “GarageBand” [16], an app comprising many editable music instruments such as a drum, guitar, and piano producing realistic instrument sounds. A user can play up to 32 tracks individually or with another person. This app also realistically simulates instruments such as guitar strings vibrating while playing. Marion Wood developed “Woodchuck Rhythm” [17], a rythm game mobile app featuring seven rhythm patterns, each of which has corresponding scores, cartoon graphics, and audio files. Lei Chi Tak developed “Rhythm Training” [18], a rhythm training mobile app featuring a metronome function. Learners can understand the note value of each piece of notation using the metronome function and practice their rhythm through virtual drumheads. Flippfly LLC developed “Monkey Drum” [19], a music-making mobile app featuring virtual musical instruments such as drums and string instruments. Children can tap the mobile device in whatever rhythm they desire and this app will identify and playback using the created rhythm. These applications can simulate musical instruments, enabling children to use touchscreen interfaces to play instruments and generate immediate sound feedback. For learners, these applications save the cost of purchasing actual musical instruments. In addition, playing instruments on mobile devices is much easier than playing the actual instruments. This motivated us to design a percussion learning app on a tablet computer for studying rhythm. Children’s inability to remain focused while studying music constitutes another challenge. Furthermore, uninteresting teaching materials may undermine learning outcomes. To solve these problems, this study proposes a percussion learning device that combines tablet computers and robots.
Appl. Sci. 2017, 7, 258 Appl. Sci. 2017, 7, 258
3 of 14 3 of 14
problems, this study proposes a percussion learning device that combines tablet computers and robots. This comprises device comprises two systems: a rhythm teaching system, in which can compose This device two systems: a rhythm teaching system, in which users users can compose and and practice rhythms bya using a tablet computer, and a robot rhythm performance system. practice rhythms by using tablet computer, and a robot rhythm performance system. The rhythm The rhythm teaching system provides a new and user-friendly score editingfor interface for composing teaching system provides a new and user-friendly score editing interface composing a rhythm a rhythm exercise, in addition to providing a rhythm rating function to facilitate percussion training exercise, in addition to providing a rhythm rating function to facilitate percussion training for children for children improveofthe stabilitybeating. of rhythmic beating. The performance robot rhythmsystem performance system and improve and the stability rhythmic The robot rhythm interacts with interacts with children to attract their attention during learning, performs percussion exercises children to attract their attention during learning, performs percussion exercises for children to listen for to, children tothem listeninto, and assists them in practicing exercises. Such interactions increase children’s and assists practicing exercises. Such interactions increase children’s interest and motivation interest and practice motivation to learn and practice rhythm exercises. to learn and rhythm exercises. 2. Rhythm Rhythm Teaching TeachingSystem System 2. Inbasic basicmusic music education, cultivating a sense of rhythm is essential. An means effective means of In education, cultivating a sense of rhythm is essential. An effective of achieving achieving this to beat on percussion instruments (e.g., drums, and castanets). this is to beat onispercussion instruments (e.g., drums, tambourines, andtambourines, castanets). Nevertheless, this Nevertheless, type the of presence learning of requires the presence of music and type of learningthis requires music teachers, and children faceteachers, difficulties in children practicingface on difficulties practicing on their own atproposes home. Therefore, this study proposes a rhythm teaching their own atin home. Therefore, this study a rhythm teaching system for assisting children to system for assisting children to practice rhythms a tablet computer. Thislearning system interest can increase practice rhythms through a tablet computer. This through system can increase children’s and children’s learningininterest and in addition to enabling to create simple learning outcomes, addition to learning enabling outcomes, them to create simple musical scoresthem through interactions. musical scores through interactions. The main rhythm teachingmusical systemscore contains twoand main functions: The rhythm teaching system contains two functions: editing playing, and musicalrating. score editing and playing, and rhythm rating. rhythm 2.1. 2.1. Musical Musical Score Score Editing Editing and and Playing Playing Functions Functions Understanding Understanding musical musical scores scores is is aa challenge challenge for for music music beginners. beginners. Figure Figure 11 shows shows aa drum drum score score consisting The first three barsbars primarily involve the hi-hat, snare snare drum,drum, and bass consistingofofsix sixbars bars[20]. [20]. The first three primarily involve the hi-hat, anddrum. bass The note positions are at the top, middle, and bottom parts of the score. Specifically, the hi-hat should drum. The note positions are at the top, middle, and bottom parts of the score. Specifically, be in quarter notes ininaquarter consistent rhythm, whereasrhythm, the snare and bass joinbass in various theplayed hi-hat should be played notes in a consistent whereas thedrums snare and drums beats. The bass drum should be played in a quaver of fast rhythm. In the final three bars, the join in various beats. The bass drum should be played in a quaver of fast rhythm. In the finaltenor three drum andtenor cymbal areand introduced, andintroduced, the rhythmand becomes complicated. a learner wants play bars, the drum cymbal are the rhythm becomesIfcomplicated. If a to learner awants pieceto ofplay music as shown in Figure 1, heinorFigure she must to understand score in advance. a piece of music as shown 1, hebe or able she must be able to the understand the score However, this is a difficult task for a beginning learner. in advance. However, this is a difficult task for a beginning learner.
Figure1. 1. A A drum drum score. score. Figure
To relieve relieve the the burden burdenon onchildren, children, aarelatively relatively intuitive intuitive and and user-friendly user-friendly score score interface interface was was To designedfor forthe the rhythm teaching system. Figure 2 shows of the edited rhythm by the designed rhythm teaching system. Figure 2 shows a piecea ofpiece the rhythm byedited the proposed proposed system. The score involves five instruments, and teachers can edit and adapt the melody system. The score involves five instruments, and teachers can edit and adapt the melody for various for various instruments. The score of each instrument comprises continuous vertical grids. instruments. The score of each instrument comprises continuous vertical grids. Four continuous grids Four continuous grids represent a bar, and each bar is played within two seconds. The light blue bar represent a bar, and each bar is played within two seconds. The light blue bar represents a beating represents a beating point. For each grid, four types of beat points can be edited by tapping the grid point. For each grid, four types of beat points can be edited by tapping the grid on the tablet computer, on the tablet computer, as shown in Figure 3. The four beat point types are a quarter note (Figure 3a), as shown in Figure 3. The four beat point types are a quarter note (Figure 3a), an eighth note (Figure 3b), an eighth note (Figure 3b), two eighth notes (Figure 3c), and no beating point (Figure 3d). two eighth notes (Figure 3c), and no beating point (Figure 3d).
Appl. Sci. 2017, 7, 258
4 of 14
Appl. Sci. 2017, 7, 258 Appl. Sci. 2017, 7, 258
4 of 14 4 of 14
Figure 2. Musical and playing playing functions. Figure 2. Musical score score editing editing and Figure 2. Musical score editing and playing functions. functions.
(a) (b) (c) (d) (a) (b) (c) (d) Figure 3. Four types of beat style: (a) a quarter note; (b) a quarter note (back beat); (c) two eighth notes; Figure 3. Four types of beat style: (a) a quarter note; (b) a quarter note (back beat); (c) two eighth notes; and (d)3.noFour beating Figure typespoint. of beat style: (a) a quarter note; (b) a quarter note (back beat); (c) two eighth notes; and (d) no beating point. and (d) no beating point.
When an edited score is played by the rhythm teaching system (Figure 4), a cue line appears on When an edited score is played by the rhythm teaching system (Figure 4), a cue line appears on the screen of tabletscore computer; the by score then scrolled from top to(Figure bottom over time. If appears the beat When anthe edited is played theis teaching system a cue line the screen of the tablet computer; the score isrhythm then scrolled from top to bottom4), over time. If the beat points reach the cut tablet line, learners must corresponding instruments in time. For example, on the screen of the computer; thebeat scorethe is then scrolled from top to bottom over time. If the points reach the cut line, learners must beat the corresponding instruments in time. For example, in Figure 4, a bass drum is beaten. In addition, teachers can insert background music, which can beat points reach the cut line, learners must beat the corresponding instruments in time. For example, in Figure 4, a bass drum is beaten. In addition, teachers can insert background music, which can improve sense of rhythm practicing rhythm exercises. After the completion in Figurethe 4, alearner’s bass drum is beaten. In when addition, teachers can insert background music, which of cana improve the learner’s sense of rhythm when practicing rhythm exercises. After the completion of a score edition, the teaching system enables teachers to save the edited score and load it in the next improve the learner’s sensesystem of rhythm when practicing rhythm exercises. the completion of score edition, the teaching enables teachers to save the edited scoreAfter and load it in the next exercise.the Compared with conventional drumto scores, the edited scores shown onload this interface are arhythm score edition, teaching system enables teachers save the and in the next rhythm exercise. Compared with conventional drum scores, the scores score shown on this it interface are easier for beginners to read. According to field trials, most children understand the rhythms in are the rhythm exercise. Compared with conventional drum scores, the scores shown on this interface easier for beginners to read. According to field trials, most children understand the rhythms in the score within a short to period. some practice and with certain level of skills,the most children easier for beginners read. After According to field trials, mosta understand rhythms in can the score within a short period. After some practice and with a children certain level of skills, most children can play a piece of music completely on their own. In addition, children can more easily create their own score within a short period. After some practice and with a certain level of skills, most children can play a piece of music completely on their own. In addition, children can more easily create their own druma scores formusic listening and practicing their play piece of completely on theiron own. Inown. addition, children can more easily create their own drum scores for listening and practicing on their own. drum scores for listening and practicing on their own.
Appl. Sci. 2017, 7, 258
5 of 14
Appl. Sci. 2017, 7, 258
5 of 14
Appl. Sci. 2017, 7, 258
5 of 14
Figure 4. An edited score is played by the rhythm teaching system. Figure 4. An edited score is played by the rhythm teaching system. Figure 4. An edited score is played by the rhythm teaching system.
By applying musical score editing function, teachers can immediately edit scores of various By applying musical score editing editing function, teachers can immediately immediately editdifficulty scores of ofaccording various difficulties (Figuremusical 5) corresponding to students’ learning progress and adjust edit the By applying score function, teachers can scores various difficulties (Figure 5) corresponding to students’ learning progress and adjust the difficulty according to the practice or 5) test outcome. Thetodrum scorelearning contentprogress includesand both basicthe and advanced rhythm difficulties (Figure corresponding students’ adjust difficulty according to the practice or test outcome. The drum score content includes both basic and advanced rhythm for aorsingle drum, two drums, multiple percussion instruments. Inadvanced addition, rhythm students topractices the practice test outcome. The drum or score content includes both basic and practices for single drum, drum, two drums, or multiple percussion instruments. In addition, addition, students could alsofor conveniently edit or create their or own scores using this musical score editing function.students practices aa single two drums, multiple percussion instruments. In couldalso alsoconveniently convenientlyedit edit create their own scores using musical score editing function. could oror create their own scores using thisthis musical score editing function.
(a)
(b)
(c)
(a) (b) (c) Figure 5. The edited scores of three different difficulties: (a) basic; (b) advanced; (c) difficult. Figure 5. The edited scores of three different difficulties: (a) basic; (b) advanced; (c) difficult. Figure 5. The edited scores of three different difficulties: (a) basic; (b) advanced; (c) difficult.
2.2. Rhythm Rating Function 2.2. Rhythm Rating Function In the proposed teaching system, a rhythm rating function was also designed to facilitate rhythm training children and improve stability of rhythmic beating. As shown in Figurerhythm 6, five In thefor proposed teaching system,the a rhythm rating function was also designed to facilitate instrument located at the bottom. When the teaching system a percussion training for buttons childrenare and improve the stability of rhythmic beating. Asplays shown in Figureexercise, 6, five the score scrolls from to bottom time.When If thethe beat points reach theplays cut line, learners must then instrument buttons aretop located at theover bottom. teaching system a percussion exercise, tap on the corresponding instrument buttons. As a learner presses the buttons, the rating function the score scrolls from top to bottom over time. If the beat points reach cut line, learners must then tap on the corresponding instrument buttons. As a learner presses the buttons, the rating function
Appl. Sci. 2017, 7, 258
6 of 14
2.2. Rhythm Rating Function In the proposed teaching system, a rhythm rating function was also designed to facilitate rhythm training for children and improve the stability of rhythmic beating. As shown in Figure 6, five instrument buttons are located at the bottom. When the teaching system plays a percussion exercise, the score scrolls from top to bottom over time. If the beat points reach the cut line, learners must then tap on the corresponding instrument buttons. As a learner presses the buttons, the rating Appl. Sci. 2017, 7, 258 14 function determines whether the tapping occurs on the correct beats and records the number 6ofofcorrect taps; concurrently, the corresponding instrument sounds are generated for simulating this tapping determines whether the tapping occurs on the correct beats and records the number of correct taps; action. In addition, the learners instrument want to adjust theare difficulty level a percussion exercise, they can concurrently, the if corresponding sounds generated for of simulating this tapping action. disable some instruments tapping thethe check buttons top of theexercise, teaching system. Because In addition, if the learnersby want to adjust difficulty levelatofthe a percussion they can disable playing musical instruments on mobile devices is relatively easy for children, we provide another some instruments by tapping the check buttons at the top of the teaching system. Because playing learning style for children to improve beatingeasy stability and effectively assess the learning accuracy of musical instruments on mobile devicestheir is relatively for children, we provide another style forrhythms. children to improve their beating stability and effectively assess the accuracy of practicing practicing rhythms.
(a)
(b)
Figure 6. Rhythm rating function: (a) full instruments; (b) disable three instruments.
Figure 6. Rhythm rating function: (a) full instruments; (b) disable three instruments.
3. Interactive Robot Performance System
3. Interactive Robot Performance System
This study developed a portable and low-cost percussion performance system, as shown in Figure 7. The design idea of this performance system percussion refers to a jazz drum set. Tosystem, attract children’s This study developed a portable and low-cost performance as shown in attention, performance system applies daily necessities a plastic a glass) as Figure 7. Thethe design idea of this performance system refers (e.g., to a jazz drumbucket set. Toand attract children’s beating instruments and robots as drummers. The drum set includes a plastic bucket, iron lid, sugar attention, the performance system applies daily necessities (e.g., a plastic bucket and a glass) as beating sachet, glass, bell on the robots These instruments can bebucket, dividediron into the instruments andand robots as which drummers. The beat. drum set includes a plastic lid, following sugar sachet, four categories according to their sound effects: (1) low frequency; (2) high frequency; (3) with some glass, and bell on which the robots beat. These instruments can be divided into the following four fermata; and (4) particularly loud. These instruments are respectively the bass drum, snare drum, hicategories according to their sound effects: (1) low frequency; (2) high frequency; (3) with some hat, and crash cymbal in a drum set (Table 1). When the robots perform a percussion exercise, these fermata; and (4) particularly loud. These instruments are respectively the bass drum, snare drum, daily necessities can produce sounds similar to those made by a jazz drum set. Simultaneously, the hi-hat, andcan crash cymbal a drum setproduced (Table 1).byWhen the performance robots perform a percussion learner listen to the in rhythm music the robot system and follow exercise, it to these daily necessities can produce sounds similar to those made by a jazz drum set. Simultaneously, practice the percussion exercise using a small drum set, as depicted in Figure 8.
the learner can listen to the rhythm music produced by the robot performance system and follow it to 1. Instrument Selections. practice the percussion exercise usingTable a small drum set, as depicted in Figure 8. Sound Type Low-Frequency Sound High-Frequency Sound Sound with Some Fermata Loud Sound
Object Selected in Our Drum Set Plastic Bucket Iron Lid Sugar Sachet Glass, Bell
Instrument Type Bass Drum Snare Drum Hi-Hat Crash Cymbal
Appl. Sci. 2017, 7, 258
7 of 14
Table 1. Instrument Selections. Sound Type
Object Selected in Our Drum Set
Instrument Type
Low-Frequency Sound High-Frequency Sound Sound with Some Fermata Loud Sound
Plastic Bucket Iron Lid Sugar Sachet Glass, Bell
Bass Drum Snare Drum Hi-Hat Crash Cymbal
Appl. Sci. 2017, 7, 258
7 of 14
Appl. Sci. 2017, 7, 258
7 of 14
Figure 7. Interactive robot performance system.
Figure 7. Interactive robot performance system. Figure 7. Interactive robot performance system.
Figure 8. Learners listen to rhythm music produced by the robot performance system and follow the rhythm8.music to practice the percussion Figure Learners listen to rhythm musicexercise. produced by the robot performance system and follow the
Figure 8. Learners listen to rhythm music produced by the robot performance system and follow the rhythm music to practice the percussion exercise. rhythm music to performance practice the percussion exercise. In the robot system, five robots, constructed with Lego blocks, act as drummers to
perform therobot percussion exercise. Eachfive robot contains a servowith motor to blocks, generate action In the performance system, robots, constructed Lego actaasbeating drummers to (Figure Anpercussion embedded system (i.e., Arduino UNOconstructed [21]) servesmotor as the control core of the This perform the exercise. Each robot contains a servo to generate a beating action In the 9). robot performance system, five robots, with Lego blocks, act robot. as drummers system score information sent from the tablet computer motor (Figure receives 9). embedded system (i.e., Arduino UNO [21]) thethen control core ofthe the robot. This to perform theAn percussion exercise. Each robot contains aserves servoasand motor tocontrols generate aservo beating action to beat on the instruments. Compared with a traditional jazz drum set, our robot performance system system receives score information sent from the tablet computer and then controls the servo motor (Figure 9). An embedded system (i.e., Arduino UNO [21]) serves as the control core of the robot. smaller andinstruments. portable. InCompared additions,with its construction cost drum is US$40. According to the fieldsystem trials, tosystem beat onreceives the jazz set, our robotthen performance Thisis score information senta traditional from the tablet computer and controls the servo children practicingInwith the robot performance cost system. This interactive style can is smallerenjoy and portable. additions, its construction is US$40. According learning to the field trials, motor to beat on the instruments. Compared with a traditional jazz drum set, our robot performance improve of rhythm practicing rhythm exercises. children their enjoysense practicing withwhen the robot performance system. This interactive learning style can system is smaller and portable. In additions, its construction cost is US$40. According to the field improve their sense of rhythm when practicing rhythm exercises.
Appl. Sci. 2017, 7, 258
8 of 14
trials, children enjoy practicing with the robot performance system. This interactive learning style can improve their sense of rhythm when practicing rhythm exercises. Appl. Sci. 2017, 7, 258 8 of 14
Figure 9. Appearance of the robot. Figure 9. Appearance of the robot.
4. Experimental Course and Field Trials 4. Experimental Course and Field Trials To demonstrate the effectiveness of the proposed system, an experimental course was designed. To demonstrate the effectiveness of the proposed system, an experimental course was designed. This course incorporated the rhythm teaching system into traditional rhythm education to improve This course incorporated the rhythm teaching system into traditional rhythm education to improve students’ sense of rhythm and musical performance skills. Ten students aged five to six years students’ sense of rhythm and musical performance skills. Ten students aged five to six years participated in four 50-min classes over four weeks. Appendix A shows the details of the teaching participated in four 50-min classes over four weeks. Appendix A shows the details of the teaching plan. plan. In the first two classes, the children studied the basic sense of musical rhythm with traditional In the first two classes, the children studied the basic sense of musical rhythm with traditional rhythm rhythm education. In the next two classes, the children used small snare drums to engage in education. In the next two classes, the children used small snare drums to engage in interactive rhythm interactive rhythm practice. First, robots played according to a score, and the children listened to practice. First, robots played according to a score, and the children listened to them and observed them and observed their movements. Subsequently, the children were asked to replicate the rhythms their movements. Subsequently, the children were asked to replicate the rhythms according to the according to the score by using small snare drums and repeat practicing what they have learned. To score by using small snare drums and repeat practicing what they have learned. To demonstrate the demonstrate the effectiveness of the proposed system, a test was administered to each student before effectiveness of the proposed system, a test was administered to each student before and after rhythm and after rhythm education. The test methods are described as follows: education. The test methods are described as follows: A. The proposed rhythm rating function was used for assessment. This function played a test A. The proposed rhythm functionwere was used for assessment. function played testmelody. melody, melody, during whichrating the students required to performThis rhythms along withathe during which the students were required to perform rhythms along with the melody. The system The system then recorded the number of beats that the students played accurately. the number of beats thattothe students playedwith accurately. B. then Whenrecorded the students used snare drums play songs along the melodies, their performance the students used snare drums to play songs along with the melodies, their performance B. When was observed and assessed. was observed and assessed. The test melody used in this experiment consisted of 25 beats that were to be tapped using one handThe (Figure 10). Table 2 lists the experiment detailed testconsisted results ofof1025students. Before course, theusing students test melody used in this beats that werethe to be tapped one scored an average beating accuracy of 59.16%; after the course, the accuracy improved to 79.51%. This hand (Figure 10). Table 2 lists the detailed test results of 10 students. Before the course, the students improvement verifies the benefit of our teachingafter method. Most students achieved superior beating scored an average beating accuracy of 59.16%; the course, the accuracy improved to 79.51%. accuracy after the course; only student A played poorly after the course. Next, we investigated the This improvement verifies the benefit of our teaching method. Most students achieved superior beating performance of each student. In the pretest, the beating accuracy of student F was 44%; accuracy after the course; only student A played poorly after the course. Next, we investigated the we observed of that hestudent. moved his body alongthe with the rhythm when music Fwas but he was performance each In the pretest, beating accuracy of student wasplayed, 44%; we observed unable to tap the rhythm on time (he was often too slow) when he performed. However, after the that he moved his body along with the rhythm when music was played, but he was unable to tap the course, we observed that his sense of rhythm and musical performance skills improved significantly; rhythm on time (he was often too slow) when he performed. However, after the course, we observed he always tapped the rhythm on time, and his beating fromhe 44% to 84%. that his sense of rhythm and musical performance skillsaccuracy improvedincreased significantly; always tapped the Student J showed a beating accuracy of 61.76% during the pretest. We observed that she was rhythm on time, and his beating accuracy increased from 44% to 84%. unfamiliar with musicalarhythms and had difficulty them. Consequently, she that oftenshe tapped Student J showed beating accuracy of 61.76%following during the pretest. We observed was rhythms too early. However, she showed great interest in the course content. Subsequently, when unfamiliar with musical rhythms and had difficulty following them. Consequently, she often tapped performing interactive musical performance with the she could maintain her focus and rhythms tooan early. However, she showed great interest inrobot, the course content. Subsequently, when imitate the rhythms played by theperformance robot. In addition, sherobot, indicated her strong desire practice performing an interactive musical with the she could maintain her to focus and playing again after training. Her beating accuracy increased from 61.76% to 96.15% after the course.
Appl. Sci. 2017, 7, 258
9 of 14
imitate the rhythms played by the robot. In addition, she indicated her strong desire to practice playing again after training. Her beating accuracy increased from 61.76% to 96.15% after the course. Appl. Sci. 2017, 7, 258 9 of 14
Figure 10. Test musical score in the experiment. Figure 10. Test musical score in the experiment. Table 2. Testing results of 10 students. Table 2. Testing results of 10 students. Testing before the Rhythm Education Student Testing before the RhythmAccuracy Education Beating of IDStudent Correct Beating Number Beatingof Beating Correct Accuracy ID A 38Number 18 47.37% Beating Beating B A 26 38 11 18 42.31% 47.37% C B 31 26 14 11 45.16% 42.31% D C 26 31 21 14 80.77% 45.16% E D 26 26 18 21 69.23% 80.77% E 69.23% F 27 26 12 18 44.44% 44.44% G F 26 27 18 12 69.23% 69.23% H G 61 26 24 18 39.34% H 39.34% I 25 61 23 24 92.00% I 25 23 92.00% J 34 21 61.76% J 34 21 61.76% Average Beating Accuracy 59.16% Average Beating Accuracy 59.16%
Testing after the Rhythm Education Testing after the Rhythm Beating Correct Education Accuracy of Number Beating Accuracy Beating Beating Correct of 52 36.54% Number Beating19 Beating 85.19% 5227 19 23 36.54% 25 19 76.00% 27 23 85.19% 92.31% 2526 19 24 76.00% 74.07% 2627 24 20 92.31% 2725 20 21 74.07% 84.00% 2525 21 22 84.00% 88.00% 2524 22 16 88.00% 66.67% 2426 16 25 66.67% 96.15% 26 25 96.15% 26 25 96.15% 26 25 96.15% 79.51% 79.51%
To further test our system, field trials were performed at Maker Faire Tokyo and Maker further test our11). system, field trialsawere performedto at Maker Faire and Maker Faire Taipei FaireToTaipei (Figure We designed questionnaire evaluate theTokyo user interaction satisfaction (Figure We designed a questionnaire to evaluate the user interaction the proposed of the 11). proposed system. The following conclusions were derived satisfaction from the of field tests and system. The following questionnaire results: conclusions were derived from the field tests and questionnaire results:
• • • •
Childrenface facedifficulty difficulty performing complex rhythms on own. their However, own. However, our Children performing complex rhythms on their throughthrough our system, system, children can learn to play complex rhythms using simple system operations. children can learn to play complex rhythms using simple system operations. Children who who have have aa tendency tendency to to hit hit items items with with aa stick stick show show aa willingness willingness to to spend spend more more time time Children editing their own rhythm-based sheet music after using our system. editing their own rhythm-based sheet music after using our system. Because the the robot robot plays plays self-created self-created music, music, after after listening listening to to aa particular particular piece piece of of music music several several Because times, most most children children learn learn about about how how to to edit edit the the sheet sheet music music to to produce produce the the songs songs they they want want times, to play. to play. The robot performance attracts children’s attention and increases their willingness to learn. The robot performance attracts children’s attention and increases their willingness to learn. In the field trials, the loud music and actions produced by the robot performance system drew attention from numerous adults and children. The audience gave positive feedback regarding the system’s novel and interesting design concept. They agreed that our system provides the users with a pleasant experience through its interactive features. Because the score could be easily edited, this also prompts them to adjust the input score in order to create more amusing rhythms.
Appl. Sci. 2017, 7, 258
•
10 of 14
In the field trials, the loud music and actions produced by the robot performance system drew attention from numerous adults and children. The audience gave positive feedback regarding the system’s novel and interesting design concept. They agreed that our system provides the users with a pleasant experience through its interactive features. Because the score could be easily edited, this also prompts them to adjust the input score in order to create more amusing rhythms.
Appl. Sci. 2017, 7, 258
10 of 14
Appl. Sci. 2017, 7, 258
10 of 14
Figure 11. Field trials of our system at Maker Faire Tokyo. Figure 11. Field trials of our system at Maker Faire Tokyo. Figure 11. Field trials of our system at Maker Faire Tokyo.
Finally, by applying musical score editing function, teachers can teach students how to edit Finally, musical score function, teachers can teach how to a better scoreby byapplying themselves. For example, the score in Figure 12a iscan edited a student. However, Finally, by applying musical score editing editing function, teachers teachbystudents students how to edit edit aathe better score by themselves. For example, the score in Figure 12a is edited by a student. However, student’s composes For inconsistent andinedits too12a many loud sounds, then the teacher better score score by themselves. example,rhythm the score Figure is edited by a student. However, the student’s score composes inconsistent rhythm and editsscore too many sounds, then teacher could immediately the student how to reformand a better as the loud example shown in the Figure 12b. the student’s score teach composes inconsistent rhythm edits too many loud sounds, then the teacher could immediately teach the student how to reform a better score as the example shown in Figure 12b. could immediately teach the student how to reform a better score as the example shown in Figure 12b.
(a)
(b)
(a) (b) Figure 12. (a) Student’s score; (b) reforming a better score by the teacher. Figure 12. (a) Student’s score; (b) reforming a better score by the teacher. Figure 12. (a) Student’s score; (b) reforming a better score by the teacher.
5. Comparison of Rhythm Apps 5. Comparison of Rhythm Apps In this section, four rhythm Apps were applied for comparison, including Carnival of Animals [15], Woodchuck Rhythm [16], Rhythm Training [17], and Monkey Drum [18]. Table 3 lists functions In this section, four rhythm Apps were applied for comparison, including Carnival of the Animals [15], of our systemRhythm and four rhythm Apps. According to the comparison in Table 3, we find Woodchuck [16], Rhythm Training [17], and Monkey Drumresults [18]. Table 3 lists thecould functions that system more interactivity functions thanresults other inapps. another of ourour system and contains four rhythm Apps. Accordingand to the comparison TableIn 3, we couldway, find an experimental trail ofmore applying the four and rhythm apps was processed. ThreeInstudents ages that our system field contains interactivity functions than other apps. anotherof way,
Appl. Sci. 2017, 7, 258
11 of 14
5. Comparison of Rhythm Apps In this section, four rhythm Apps were applied for comparison, including Carnival of Animals [15], Woodchuck Rhythm [16], Rhythm Training [17], and Monkey Drum [18]. Table 3 lists the functions of our system and four rhythm Apps. According to the comparison results in Table 3, we could find that our system contains more interactivity and functions than other apps. In another way, an experimental field trail of applying the four rhythm apps was processed. Three students of ages 6, 7, and 9 years participated the experimental field trail over 10 days. A test the same to Section 4 was administered to three students before and after this field trail. Table 4 lists the detailed test results of three students. Before this field trail, the students scored an average beating accuracy of 76%; after the field trail, the accuracy improved to 83.33%. This improvement is not significant. The following conclusions were derived from this field trail:
•
•
•
•
•
During the field-trail process, most participating parents did not like having their children spend excessive time on these apps. Some parents were concerned that long-term usage of mobile devices would impair their children’s vision. During the field-trail process, we found that children were spending a small amount of time on rhythm training and an excessive amount of time on in-app minigames unrelated to rhythm training. Although the Carnival of Animals and Rhythm Training apps had interesting and attractive interfaces, these apps had monotonous audio feedback and operation, and learners may become bored after one or two uses. In addition, these apps allowed practice with only minimal rhythm combinations and do not allow customization. For the Monkey Drum app, children could tap the mobile device in whatever rhythm they desire. However, this app had poor interactivity and monotonous playback and was unable to keep children’s long-term interest. Although the Woodchuck Rhythm app applied seven rhythm patterns, each of which had corresponding scores, cartoon graphics, and audio file. However, this app had only a small rhythm library and did not include rhythms with long beats (e.g., half notes) Table 3. Comparison of our system and four rhythm apps. Heading
Our System
Carnival of Animals [15]
X
X
Interactivity Notation Learning Note Value Learning Variety of Musical Instruments Create and Edit Drum Scores Difficulty Adjustment In-app Simulation of Percussion Instruments
X X X X
Woodchuck Rhythm [16]
Rhythm Training [17]
X X X
X X
Monkey Drum [18]
X
X
X X
X
X
X
X
Table 4. Testing results of three students. Testing before the Field Trail Student ID A B C
Testing after the Field Trail
Beating Number
Correct Beating
Accuracy of Beating
Beating Number
Correct Beating
Accuracy of Beating
25 25 25
18 17 22
72% 68% 88%
25 25 25
21 19 24
84% 76% 96%
Average Beating Accuracy
76%
85.33%
Appl. Sci. 2017, 7, 258
12 of 14
These observations revealed that although these apps provided tremendous entertainment value, they did not facilitate favorable learning outcomes. By contrast, our system not only provides excellent interactive and entertainment features, but also enables desirable learning outcomes. 6. Conclusions and Future Works This study proposes an interactive percussion learning device for helping children create and study musical rhythms. This system combines Lego robots and a tablet PC running the developed interactive system. Through this system, which is easy to manage for young music learners, children can edit musical scores and control the robots’ movement. During this process, they acquire the basic concepts of musical rhythms and show interest and strong motivation to use this system. The rhythm learning system can improve children’s sense of rhythm and motivate them to follow rhythms and play the correct beats. During actual teaching, children can first listen to rhythms played by the robots and understand the arrangements of each instrument in the music. The results of the experimental course and field trials indicate that the proposed system not only increases students’ interest and efficiency in learning but also helps them in understanding musical rhythms through interaction and composing simple rhythms. Although a drum machine can synthesize drum sounds, the sound it generates is not as authentic as that produced using actual percussion instruments. In performances with real instruments, various sound effects can be generated using different strike strengths, and users can clearly feel the changes of musical volume and level. Striking a drum at differently location can produce various sounds that cannot be replicated by a drum machine. In the experimental course, we discovered that the students naturally followed the robot’s striking actions and rhythm to generate their own rhythmic body movements. In the future, we expect to project hints regarding beat points onto the practice drumhead. Users will thus be signaled in advance regarding which drum to strike and where it should be struck. Additionally, vibratory sensors will be installed on each drum to identify the beating accuracy and note the score sections where the user is prone to making a mistake or is unable to keep up with the rhythm. For the robot performance system, we will provide other items as robotic percussion instruments. For example, cups made of various materials can generate different loud tones, whereas wood veneers can be used to produce high-frequency sounds. Learners can also cultivate their ability to distinguish sounds by listening to percussion using different items. In the rhythm teaching system, we will introduce basic music theory (e.g., notes, score, and rhythm) and the various elements of a drum set (e.g., bass drum, snare drum, and hi-hat). Acknowledgments: This work was supported by the Ministry of Science and Technology, Taiwan, R.O.C., under the Grant MOST 105-2221-E-032-055. Author Contributions: Chien-Hsing Chou and Yung-Long Chu proposed the original idea and designed the experiments; Yung-Long Chu designed the system; Chien-Hsing Chou and Yung-Long Chu analyzed the data and wrote the paper. Conflicts of Interest: The authors declare no conflict of interest.
Appendix A Table A1. Teaching plan for music rhythm education. No heading line Course Title
An experimental course on the use of interactive teaching aids in musical rhythm education
Topic
Training one’s sense of musical rhythm
Source of Teaching Materials
Self-compiled, together with reference books
Teaching Aids Used
Castanets, maracas, an iPad, and an interactive robot performance system
Target students
Preschool Levels 2 and 3 (5–6-year-olds)
Appl. Sci. 2017, 7, 258
13 of 14
Table A1. Cont. Number and Length of Classes
Four 50-min classes
Course Objectives
To incorporate teaching aids designed in this study into traditional rhythm education to improve students’ sense of rhythm and musical performance skills in addition to elevating their willingness to learn
Course Content
Task 1: Musical rhythms were created using the body to imitate produce a variety of rhythms and train students to feel rhythmic changes. Next, students were encouraged to improvise and create different rhythms. Subsequently, the nursery rhyme “Two Tigers” (“Frère Jacques” in French) was played, and students were asked to replicate the rhythms in accordance with the rhythm of the song. Task 2: Students were taught quarter notes and eighth notes, in addition to training them to recognize the speed differences between them. First, students were asked to perform music using maracas and castanets. Subsequently, the proposed rhythm rating function in iPad was applied to facilitate rhythm training and improve the accuracy of rhythmic beating. Task 3: Students were taught to play snare drums using their hands to create different rhythms. First, robots played according to a score, and the children listened to them and observed their movements. The rhythm was composed of quarter and eighth notes. Subsequently, the children were asked to replicate in accordance with the rhythm of “Two Tigers” by using small snare drums and repeat practicing what they have learned. Task 4: The Chinese nursery rhymes “Two Tigers” and “Speeding Train” were played using the interactive robot performance system. Students were instructed to replicate the rhythms using snare drums according to and along with the rhythm of the songs.
Assessment Method
A test was administered to each student before and after the rhythm training. The test methods are described as follows: 1. The proposed rhythm rating function was used for assessment. This function played a test melody, during which the students were required to perform rhythms along with the melody. The system then recorded the number of beats that the students played accurately. 2. When the students used snare drums to play songs along with the melodies, their performance was observed and assessed.
References 1. 2. 3. 4.
5. 6.
7.
8.
Eduard, H. Vom Musikalisch-Schönen: Ein Beitrag zur Revision der Ästhetik der Tonkunst; JA Barth: Leipzig, Germany, 1854. Rea, P. Experiences in Movement with Music, Activities, and Theory; Delmar Publishers: New York, NY, USA, 1995. Bae, J.; Lee, B.; Cho, S.; Heo, Y.; Bang, H. Drum On: Interactive personal instrument learning system. In Proceedings of the SIGGRAPH Emerging Technologies, Los Angeles, CA, USA, 5–9 August 2012. Kanke, H.; Terada, T.; Tsukamoto, M. A Percussion Learning System Using Rhythm Internalization with Haptic Indications. In Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology, Iskandar, Malaysia, 16–19 November 2015. Li, W.H.A.; Fu, H. Augmented reflection of reality. In Proceedings of the SIGGRAPH Emerging Technologies, Los Angeles, CA, USA, 5–9 August 2012. Baba, T.; Kikukawa, Y.; Yoshiike, T.; Suzuki, T.; Shoji, R.; Kushiyama, K. Gosen: A Handwritten Notational Interface for Musical Performance and Learning Music. In Proceedings of the SIGGRAPH Emerging Technologies, Los Angeles, CA, USA, 5–9 August 2012. Raymaekers, L.; Vermeulen, J.; Luyten, K.; Coninx, K. Game of tones: Learning to play songs on a piano using projected instructions and games. In Proceedings of the CHI 2014, Toronto, ON, Canada, 26 April–1 May 2014; pp. 411–414. Tsui, C.K.; Law, C.H.; Fu, H. One-Man Orchestra: Conducting Smartphone Orchestra. In Proceedings of the SIGGRAPH Emerging Technologies, Los Angeles, CA, USA, 5–9 August 2012.
Appl. Sci. 2017, 7, 258
9. 10.
11.
12. 13.
14.
15. 16. 17. 18. 19. 20. 21.
14 of 14
Bengler, B.; Bryan-Kinn, N. Designing collaborative musical experiences for broad audiences. In Proceedings of the 9th ACM Conference on Creativity & Cognition, Sydney, Australia, 17–20 June 2013; pp. 234–242. Yazdi, K.M.; Lee, S.P. E-Santur: A customizable tool for learning Santur music instrument using game-based learning approach. In Proceedings of the 2011 International Conference on Research and Innovation in Information Systems, Kuala Lumpur, Malaysia, 23–24 November 2011. Waranusast, R.; Bang-ngoen, A.; Thipakorn, J. Interactive Tangible User Interface for Music Learning. In Proceedings of the 28th International Conference on Image and Vision Computing New Zealand, Wellington, New Zealand, 27–29 November 2013. Hoffman, G.; Weinberg, G. Shimon: An Interactive Improvisational Robotic Marimba Player. In Proceedings of the CHI 2010, Atlanta, GA, USA, 10–15 April 2010; pp. 3097–3102. Rasamimanana, N.; Bevilacqua, F.; Bloit, J.; Schnell, N.; Flety, E.; Cera, A.; Frechin, J.L.; Petrevski, U. The Urban Musical Game: Using Sport Balls as Musical Interfaces. In Proceedings of the CHI 2012, Austin, TX, USA, 5–10 May 2012; pp. 1027–1030. Lee, J.; Gea, A.; Levin, N.; Yang, C.; Deciron, R.; Lee, M.; Lu, P.L.; Xie, M. TuneTrain: Incidental Learning Through Interactive Music Creation. In Proceedings of the 2013 IEEE International Games Innovation Conference (IGIC), Vancouver, UK, 23–25 September 2013; pp. 130–133. Toothy Buddies. Carnival of Animals: Music Education for Your Kids. Available online: http://www. toothybuddies.com/carnival-of-the-animals/tw/ (accessed on 4 June 2013). Apple Inc. GaragBand. Available online: http://www.apple.com/tw/ios/garageband/ (accessed on 10 March 2011). Wood, M. Woodchuck Rhythm. Available online: https://itunes.apple.com/tw/app/woodchuck-rhythm/ id633329909?mt=8 (accessed on 8 March 2014). Tak, L.C. Rhythm Training. Available online: https://itunes.apple.com/us/app/rhythm-training-sightreading/id517896059?mt=8 (accessed on 29 April 2012). Flippfly LLC. Monkey Drum. Available online: https://itunes.apple.com/us/app/monkey-drum/ id483225765?mt=8 (accessed on 31 May 2012). Drum Score. Available online: http://dp-notebook.blogspot.tw/2013/09/blog-post.html (accessed on 7 September 2013). Arduino UNO. Available online: https://www.arduino.cc/en/Main/ArduinoBoardUno (accessed on 1 December 2006). © 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
