The i-Cube: Design Considerations for Block-based Digital ... - NTU

The i-Cube: Design Considerations for Block-based Digital Manipulatives and Their Applications Wooi Boon Goh L.L.Chamara Kasun Fitriani Jacquelyn Tan School of Computer Engineering, Nanyang Technological University Nanyang Avenue, Singapore 639798 [email protected] ABSTRACT

Manipulatives are tangible objects designed to support learning through exploratory arrangement and manipulation. The i-Cube is a cube-shaped digital manipulative that provides unique 3-D spatial awareness of the facets and orientation of neighboring i-Cubes. This paper discusses the considerations adopted in its design and the advantages of the proposed design to that of other cubebased tangible user interfaces. The i-Cubes are then employed in the design of two applications. MusiCube Arranger is a tangible music composition and layering system and Spelling Cube is an interactive system for learning spelling. These applications are used to illustrate how the unique features of the i-Cube can be exploited to implement novel tangible interactions such as free-form 3D stacking, interactive control through block orientation change and context-aware feedback. Author Keywords

Tangible user interfaces, Digital manipulatives, Education, Child-Computer Interaction. ACM Classification Keywords

H.5.2 User Interfaces – Input devices and strategies; INTRODUCTION

There is convincing evidence that significant changes occur in the brain during the early years of a child and during these period, much of the cognitive, social and emotional development are nurtured through an environment of constructive play and exploration. For example, many aspects of the traditional educational blocks are known to facilitate functional and symbolic play in early childhood [6, 26]. The educational benefits of such devices have been highlighted by educators such as Frederick Froebel [4] and the early kindergarten reformers such as Patty Smith Hill [5] and Maria Montessori [15], with her famous ‘pink Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. DIS 2012, June 11-15, 2012, Newcastle, UK. Copyright 2012 ACM 978-1-4503-1210-3/12/06...$10.00.

Wei Shou

tower’ building blocks. In fact, studies have demonstrated positive correlation between pre-school block play performance and math achievements in US high school and 7th grade [27].

Figure 1. Traditional block-based manipulatives for learning letters of the English alphabet and spelling.

In providing embedded technology support for children concept development through play, it is important to realize that young children (below 8 years) are pre-operational and rely significantly on their visual and auditory perception in acquiring knowledge [1]. Moreover, they are active learners who make use of their physical and social experiences to form an understanding of the world around them [13]. Though the potential of traditional wooden blocks (see Figure 1) for symbolic representation is powerful, their pedagogical scope can be significantly enhanced if they can be endowed with intelligent sensing capabilities and context-aware audio-visual feedback. This research aims to develop effective and flexible play-based pedagogies for young children using novel tangible blocks that are embedded with interactive elements and responsive audio-visual feedback. These intelligent blocks-based manipulatives have the ability to sense the child’s physical activity in relation to it. In response to this interaction, it can provide context-aware feedback via visual cues (e.g. colored lights) and auditory cues (e.g. musical tones, associative sound effects for objects) that will engage the multiple sensory modalities of the child during the learning process. In the hands of creative educators and application designers, this novel educational tool can be used to design a rich variety of play-based pedagogies that are both engaging and able to deliver tailored learning objectives. In order to create a tangible interactive tool to investigate the educational potential of intelligent block-based manipulatives, the Interactive Inter-communicating Interface (I3 or i-Cube) was designed and developed. This paper describes the design considerations employed in developing the i-Cube and discusses the implications of

different design decisions in comparison with other blockbased tangible user interfaces (TUI). The design of two interactive i-Cube based applications is described. RELATED WORK

The idea of cube-based tangible user interfaces and its use in learning-oriented applications is not new. The Display Cube [10] has been used as a learning cube to support textbased quizzes [23] and as a novel input/output device to a social learning software to teach children skills to cope with bullying [11]. The educational potential of the Display Cube is somewhat limited to its use as a tangible means of interacting with the computer. In all its applications so far, the cube interface is always used as a singular device. More in line with our notion of block-based digital manipulatives (DM) is the work of Zuckerman et al. [30], where they defined DM as computationally enhanced version of physical objects that can be used to widen the range of concepts children can explore through direct manipulation. They coined the notions of “Froebel-inspired Manipulatives” (FiMs) and “Montessori-inspired Manipulatives” (MiMs). FiMs are viewed as construction manipulatives for children to design real-world physical structures like planes and buildings, for example. In this genre would be block-like construction manipulatives like ActiveCube [25], Block Jam [16] and roBlocks [20], just to name a few. On the other hand, MiMs, which are also a set of building blocks but their primary focus is to allow children to arrange these digital blocks to model concepts and abstract structures. Zuckerman himself proposed two digital MiMs, namely SystemBlocks and FlowBlocks [30] to explore system concepts such as rates, feedback and manipulate abstract structures of dynamic processes. Other MiMs would include GameBlocks [22] and Siftables [14]. Siftable in particular, with its high quality LCD display is highly reconfigurable and can be used to teach many different abstract concepts that have planar spatial relationships. It is this notion of relating concepts that have inherent spatial relationships that is the strength of blockbased digital manipulatives. Hunter et al. [8] for example, used Siftables to develop a language learning application called Make a Riddle that uses word-to-block mapping to allow sentence formation. Many concepts in language, mathematics and music are sequential in nature and can be conveyed quite readily using spatial arrangements of blockbased manipulative. Spelling Bee [2] for example used tangible letter blocks to teach children spelling. A red colored light is used as context-aware feedback to tell the user if the word is arranged incorrectly. However, the 1-D design of Spelling Bee’s inter-connect and its inability to allow orientation variations severely limits the variety and dimensions of spatial relational concepts it can model An area that has received substantial interest from TUIbased application designers is interactive learning. Marshall [12] suggested that one type of learning that is particularly amenable to the use of TUIs is discovery-based learning

(DL), in which learning takes place when a child attempts to solve a problem and draws on his own prior knowledge. This method of learning requires learners to interact with their environment by exploring and manipulating objects. Kirschner et al. [9] argued that such a learning process need to begin with some guidance. Once learners gain confidence and compentency, they can then learn more effectively through discovery. From this perspective, the spatial awareness capabilities of the i-Cubes allows the incorporation of context-aware feedback, which can be employed to scaffold the DL learning process of novice learners. These aids can be selective disabled as the child progresses, allowing the educator to tailor the appropriate level of challenge to each individual learner. According to Marshall [12], our intuition that physical material supports learning has not been backed up by strong and consistent evidence from emprical studies. It is therefore important to develop flexible tools and applications that can be used to gather such emprical support and understand how, when, where and why tangible manipulatives can enhance children’s learning. This has been a motivating factor for the development of the i-Cube digital manipulatives. The design goals of the iCube have been towards the development of a tangible interface system that is customizable in its learning objectives, rich in its spatial relational expression and easy to manipulate, especially for young children. i-CUBE DESIGN CONSIDERATIONS

There exist many block-based TUIs. Some are merely tangible input devices that give the user a physical means to make one-of-n menu selections [10]. Others like the iCube, are designed as functional building elements of a larger educational [20, 30] or entertainment system [14]. Whatever the case may be, the various choices made during the design of the block-based TUI will constrain its function, application and usability. There were two particular design goals for the i-Cube that set it apart from most existing cube-shaped TUIs. Firstly, the i-Cube is the only cube-shaped digital manipulative we are aware of that is able to support full 3-D block face and orientation awareness when stacked with other cubes. This unique feature allows the i-Cube to be employed in symbolic play that has a direct mapping to our 3-D physical world. It can also support symbolic arrangements that are orientation-sensitive, such as the detecting the alignment of particular faces of a stack of cubes (see Figure 4b). Secondly, we envisage that tangible interaction with cubeshaped TUIs is likely to find pedagogical advantage when employed in interactive play that is of the free-form exploratory nature. Blocks can be speedily placed on top of each other or side-by-side without precise alignment. As such, the i-Cube has been intentionally designed to be easy to manipulate and arrange through the use of contactless sensing and sensor configurations that are tolerant to minor

misalignment between stacked blocks. We discuss these two main design goals and other minor design options adopted in the i-Cube system. DESIGN GOAL #1 – FULL 3-D SPATIAL AWARENESS

To achieve full 3-D spatial awareness between cubes that are stacked together, two technical challenges need to be addressed. The first is the ability of the cube-shaped TUI to be aware of other block faces that are in contact with all its six facets. Secondly, it should also be aware of which of the four relative orientations the touching faces are currently in. We discuss each of these in turn. Dimensions of Block Face-Awareness

Cube-shaped manipulatives are common because they are amenable to 3-D stacking and compact packing. But not all block-based TUIs can detect the presence of another “contacting” block face with this full 3-D flexibility. Figure 2 shows the various degrees of block face awareness configurations possible. 1-D block face awareness is the simplest to implement and is usually sufficient for exploring linear sequential concepts such as word spelling or arithmetic expressions. TUIs in this category include Dekel et al.’s Spelling Bee [2] and product’s like Hasbro’s Boogle Flash (see Figure 2a). 1-D linear arrangement of blocks (e.g. Hasbro’s Boogle Flash)

(a)

2-D planar arrangement of blocks (e.g. Mattel’s Cube World)

(b)

Full 3-D arrangements of blocks (e.g. ActiveCube, roBlocks and i-Cube)

(c)

Figure 2. Various dimensions of block face-awareness. (a) 1-D, (b) 2-D and (c) 3-D.

2-D planar block arrangements can explore more complex relationships such as in Zuckerman et al.’s FlowBlocks Digital MiM [30], which explores flow across T-junction (Note: some block modules in FlowBlocks do take a top placing add-on, which make them a little more that 2-D). TUIs in this category typically have inter-communication capabilities on four sides of the cube, with one top visually informative face and a non-functional bottom side. Other TUIs in this category include, Block Jam [16] and Siftables [14] (available as Sifteo Cubes [21]) and toys like Cube World (see Figure 2b). The most flexible block-face awareness configuration is the full 3-D variety. It can also be used to express 1-D and 2-D related concepts. Watanabe et al.’s ActiveCube [25] fits this 3-D definition as all cube faces are homogenous and allows the formation of 3-D shapes such as that shown in Figure 2c. The roBlocks of Schweikardt and Gross [20] is close but the functional variation of different block types requires some faces to play other roles besides interconnecting. The design goal of the i-Cube is to support full

3-D block face awareness as this flexibility permits the design of manipulatives that can spatial map semantic concepts that are 1-D (e.g. spelling words), 2-D (e.g. crossword puzzles) or 3-D (e.g. 3-D shape construction). Block Orientation-Awareness

A design consideration that is often neglected by many block-based manipulatives is the ability to detect relative orientation between blocks. Consider the scenario in Figure 3, where a child wants the “boy” to talk to the “girl”. If the system is orientation-aware, the child merely needs to rotate block #2 by a 180 degrees (half turn) to indicate this intention. No additional symbolic block is required. #4

An additional cube #4 with a “reverse arrow” is required to maintain N-S pairing

#1

#2

#3

#1

#2

#3

After flipping cube #2

Flip cube #2

After

Before

Figure 3. Difficulty in realizing block orientation-awareness as flipping a block 180 degrees normally creates incompatible symmetric coupling between blocks (i.e. S-S or N-N).

However, many digital manipulatives like FlowBlocks [30] and Spelling Bee [2] use magnet-based (N-S) coupling and others like roBlocks [20] and ActiveCubes [25] use “matching protrusion-to-recess” type mechanical coupling. The need to maintain asymmetric face compatibility (i.e. NS) prevents the implementation of the “half turn” rearrangement. Instead, another symbolic block is needed to specify the new directional relationship (see Figure 3). Exploring musical chords

Color adjustment

Decrease red intensity

Increase green intensity

(a)

F

G

E

E

C

C

Rotate top cube to hear sound of valid chord

(b)

Figure 4. Full orientation-awareness can allow the (a) rotation of the top cube to adjust different attributes with each quarter turn or (b) select appropriate symbolic combinations by rotating stacked cubes relative to each other.

Even more challenging is the design to detect 90 degree rotations (quarter turn) between blocks. Such orientationawareness allows interesting relative rotation-based manipulation to be realized (see Figure 4). In order to maximize its manipulative potential, the i-Cube has been designed with the capability for full orientation-awareness. This is achieved by having four symmetrically separated receiver sensors per cube face (see Figure 5b and 7).

DESIGN GOAL #2 – EASY ARRANGEMENT OF CUBES

The value proposition of tangible manipulatives is their ability to be physically re-arranged to construct real world models (FiMs) or model conceptual and abstract structures (MiMs) [30]. The ease in which these block can be arranged by the child has significant implications on how they can be used and the type of applications they can support. From this perspective, there are two basic ways such manipulatives can be arranged, either through physical coupling or proximity coupling. Physical coupling describes TUIs like ActiveCubes [25], FlowBlocks [30], Spelling Bee [2], Block Jam [16] and roBlocks [20], where physical contact is required to attach one block to another before inter-communication between blocks can be established through direct electrical connections. Either mechanical or magnetic techniques are used to ensure contacting facets of the block stays attached for reliable electrical interconnect. The ease of manipulation afforded by physical coupling is restricted. Dekel et al. [2] in their user testing of Spelling Bee commented that users found the small tight fitting connectors of the blocks difficult to connect. In short, the physical coupling approach is not amenable to the easy block re-arrangement requirements often needed in exploratory play. The alternative is to employ proximity coupling through the use of non-contact sensors. Examples of this approach include Siftables [14] and Hasbro’s Boogle Flash, which use optical sensing or Resnick et al.’s Programmable Beads [18], which uses inductive coupling or Parkes et al.’s Glume [17], which uses capacitive communication. Without the need for electrical coupling to establish interblock communication, the physical spatial relationship between blocks can be easily arranged. In this sense, their ease of manipulation approaches that of the traditional wooden alphabet blocks in Figure 1. However, proximity sensing technology such as optical sensing can establish communication even when blocks are quite a distance apart. Informal experiments with Hasbro’s Boogle Flash showed that the blocks can inter-communicate even when separated by distances larger than 15mm. This extensive range is not always desirable because the application may perceive two nearby blocks to be “arranged” next to each other when they appear to the child as visibly and obviously apart. Block-based manipulative that are meant to be used in the manner depicted in Figure 2 should preferably inter-communicate when cube faces are touching or are very close to each other. In the design of the i-Cube, the distance d in Figure 6a is kept small by the use of short-range inductive sensors shown in Figure 5a. These sensor pairs are able to trigger each other when the cube faces are in non-coupled physical contact or in close proximity. This means the i-Cube does not have the constraints associated with physical coupling required in electrical interconnect techniques, whilst avoiding the long distance false triggers problems of optical

sensing. The inductive sensor can also sense across the physical enclosure of the i-Cubes (made of non-ferrous acrylic plastic). Our experiments suggest that the i-Cube trigger distance d is about 3mm, with a standard deviation (SD) of 1mm. This close proximity triggering design goal is particularly helpful when many blocks are used in a confined space, like in the situation shown in Figure 14a. Plastic enclosures

Transmitter circuit

Receiver circuit

i-Cube 2

i-Cube 1

RX

Volt

Volt

TX

RX

Inductor

d

Inductor

Typical sensing distance d is < 3mm

Short bust signal received at RX

IRX RX

TX

time

time

Short bust signal sent out by TX

ITX

4 receivers RX per cube face

1 emitter per cube face

RX

(b)

(a)

Figure 5. (a) The short-range inductive sensor system used in the i-Cube. (b) The transmitter on one cube face communicates to one of four possible receivers on an adjacent cube face. Which receiver is triggered determines the relative orientation of the two adjacent cube faces.

Unfortunately, since no mechanical coupling is available for alignment, this leads to the next issue. Should intercommunication occur when block alignment is precise or near-enough? In exploratory play, it is often frustrating for the child when precise alignment is needed for him to specify 3D spatial relationship between blocks. To facilitate ease of manipulation, blocks should maintain communication when they are arranged in a near-enough fashion. Unlike optical sensors, no line-of-sight alignment is required by the inductive sensor technology employed in the i-Cube as the transmitter radiates a spherical magnetic field. This makes the i-Cube tolerant to slight shift misalignments (see Figure 6b). Our experiments suggest that the i-Cube can tolerate mean shift misalignments, s of 4.4mm, with a SD of 3.8mm. Small twist misalignment

Distance s

2

1 d

Detection may occur before distance d is small enough

(a)

Small shift misalignment

(b)

Angle t

(c)

Figure 6. Inter-communication limits between two i-Cubes. (a) The distance d when cubes can begin to sense each other. The maximum allowable misalignment in (b) shift offset, s and (c) twist angle, t.

In addition, the design cleverly places four inductive receivers at the mid-point of each edge of the cube face (see Figure 7). This configuration symmetrically places four proximity sensors whilst minimizing sensor distances to the cube face centre. This arrangement is more robust to the twist misalignment of Figure 6c because the

transmitter-receiver pair separation is much smaller per degree of rotational offset than would be experienced if sensors were placed at each corner of the cube face (the other possible symmetrical sensor placement strategy). From empirical studies, the i-Cube can tolerate mean twist misalignments angle t of 5.6°, with a SD of 3.4°.

troublesome and restrictive, this design decision allows each of the i-Cube’s six faces to individually connote unique visual concept that can be selected during the interactive block arrangement. Giving all cube faces visual information also fits well with the i-Cube’s ability to detect 3-D block arrangement and this is discussed next.

Key

9

Emitter i-Cube

Receiver

distance dedge < dcorner

Topical decals

Relevant letter is lit

8

Rabbits in red

Alternative corner placement strategy

Dog in blue

dcorner dedge

Edge sensor placement more tolerant to twist misalignment

Figure 7. The mid-point sensor placement strategy contributes more to the robustness of the i-Cube to twist misalignment compared to the corner placement strategy. OTHER DESIGN CONSIDERATIONS

Besides the two main design goals, the following sections describes two other design considerations, namely the issue of block face display and the distribution of computational resources in the tangible interactive system. Type and Configuration of Block Face Display

The decision to mount LCD displays on the cube faces is a major design consideration. CubeBrowser [29], Z-agon [28] and Display Cube [10] have low-res LCD display on all six cube faces. Such TUIs are generally used alone and serve as novel interfaces to allow gesture-based inputs. A more common choice is to mount an LCD display on only one cube face. Notable examples include Merrill et al.’s Siftable (Color LCD) [14], Netwon-Dunn et al.’s Block Jam [16] (LED matrix) and commercial products like Radica Games-Mattel’s Cube World (Black-White LCD). These TUIs are used as building block components but their arrangements are restricted to a planar configuration to avoid occlusion of the display face if another block is placed on top. Whilst it is tempting to provide high quality reconfigurable visual information on each cube face, practical considerations such as keeping the unit cost of each digital building block affordable must be taken into account. The robustness of the manipulatives to physical abuse is also a serious consideration with young children. LCD displays maybe prone to cracking when dropped. The complexity of the TUI’s embedded processor also increases with the number and quality of the display supported. And so is the power drain on the device’s built-in battery. It was decided at the onset, that the i-Cube would be designed with no LCD display. Each translucent face can be lit with a programmable colored light allowing its associated motif to be highlighted (see Figure 8). Different motifs are adhered to the six cube faces depending on the application or learning objectives. Albeit a little

(a)

Alphabet block

(b)

Animal block

Figure 8. Different visual motifs from topical decal sets can be adhered to all six faces of the i-Cube to illustrate letters of the English alphabet or pictorial concepts, for example. (a) A specific face can be individual lit to indicate an active concept or (b) with different colors to indicate concept groupings. Centralized versus Distributed Computation

A fundamental consideration when designing intelligent digital manipulatives is the issue of “what computation is being done where”? In a survey paper, Schweikardt and Gross [19] coined the phrase distributed computational toy to describe the inclusion of computation in more than one part of the toy. However, in the design of a system consisting of intelligent manipulatives, the issue of centralized versus distributed computation may not necessarily be just one or the other. In its most centralized form, each manipulative contains no computational element. An example of this is Spelling Bee [2], where each regular block consists of just appropriately valued resistors and two LEDs. All computation is done at the master block. This approach keeps the cost per block low but at the expense of limited functionality and reduced support for exploratory play. At the other extreme are systems such as the SystemBlocks and FlowBlocks Digital MiMs [30], roBlocks [20], Boda Blocks [3] and non-block based construction components like Beads [18] and Glume [17]. In these systems, each component is embedded with some computational capability and the resulting global system behavior or programme is determined solely by the arrangement and interaction between the arranged components. Somewhere in between the two extremes is the hybrid distributed computation (HDC) setup, where local computational capability is distributed among each manipulative but these manipulatives are centrally linked to a more powerful host computer. The host handles the demanding computational needs of the application. The HDC setup can be found in systems like Siftables [14] or Sifteo Cubes [21] (wireless link to host) and ActiveCubes [25] (wired link to host). With a centralized host computer, applications addressed by the TUIs can be easily and quickly reconfigured. In addition, local computation within

each manipulative makes them more intelligent, responsive and reduces the communication bandwidth with the host. Unfortunately, as a standalone product or toy, requiring a host computer increases the cost of the system and reduces its portability. Such remarks have been observed in recent tech reviews of the Sifteo Cubes, for example. This consideration is less critical for educational manipulatives that are used in a typical classroom setting. All i-Cubes maintain wireless communication with master i-Cube

Host computer receives interactive inputs from i-Cubes

All i-Cubes maintain wireless communication with host computer

Wireless communication between the master controller and all i-Cube processors are constantly maintained to carry out this fire-latch process so that the system is aware of the most current block arrangement configuration. i-Cube

Master controller i-Cube

(a)

(b)

Figure 9. The i-Cube system can be used as (a) a fully distributed or (b) a hybrid distributed computational system.

The wireless-capable i-Cube can be used in a fully distributed computational mode or in a HDC mode as shown in Figure 9a and 9b respectively. The HDC setup is preferred during application prototyping as programme development on the host computer with its unrestrictive memory constraints increases productivity. In addition, limited memory and processing capability on the lowpowered embedded processor within the i-Cube means only less computationally demanding applications can be developed for used without a host computer.

RGB LED

x y

Proximity sensors & inter-communication

z Processor with WC & re-chargeable battery

Orientation & motion detection

Embedded Sensors (3D accelerometer & proximity)

i-Cubes in play area

i-Cube

i-Cube

Wireless Communications (WC) and Embedded Processor

RGB LED

RGB Mini LED speaker RGB LED

Output Modalities (audio & tri-color LED displays)

Figure 10. Various categories of i-Cube components, including the embedded processor, the sensors (orientation, motion and proximity) and outputs (colored lights and audio).

Tri-color LED – The RGB light emitting diodes (LED) allow each i-Cube face to be lit with a different color. Audio output – A tiny vibration speaker allows sound to be rendered locally. During setup, compressed audio data is wirelessly communicated to the i-Cube from the host computer and stored in its 2GB microSD memory card. Local playback of the sound is accomplished when the processor streams selected audio data blocks in the microSD card to the audio amplifier.

THE i-CUBE IMPLEMENTATION

At the heart of the i-Cube is a low power Texas Instrument CC2510 microcontroller [24] with a SimpliciTI protocolcapable wireless module (802.15.4). The CPU operates on a 26 MHz clock, has 32kB flash memory and 4kB SRAM. The electronics are housed in a cube-shaped acrylic plastic container (see Figure 11a). Once the processor is programmed, the enclosure is sealed with a lid and four M3 nylon screws. Appropriate decals can then be stuck to the centre of each cube face (see Figure 11b). The peripheral component parts within the i-Cube are shown in Figure 10. Orientation and motion – A 3-axis Freescale MMA7260 accelerometer sense the orientation and tilt of the i-Cube and thus allow the cube’s top face to be computed. This sensor is also senses 3-D motion when the i-Cube is used as a gestural input device. Proximity contact with other cubes – An inductive proximity-sensing system allows the i-Cube to detect faceto-face contact with another i-Cube (see Figure 5). This inter-communication between i-Cubes allows the spatial arrangements of the cubes to be inferred by triggering the transmitter of each cube face of each cube to fire in a round robin fashion. The receiver circuit latches a response that gives the host computer system clues regarding the block face and orientation relationship between the i-Cubes.

(a)

(b)

Figure 11. The i-Cube prototypes. (a) The i-Cube electronics are housed inside an acrylic container of size 5 5 × 5 5 × 55mm. (b) The i-Cubes with alphabet decals attached to each of the six faces and the i-Cube electronics.

We discuss the design of two applications that make use of the unique features of the proposed i-Cube system, namely its ability to detect 3-D block face and block orientation. These applications also require the users to freely arrange cubes in an inter-changeable manner and therefore require TUIs that are easy to combine. APPLICATION #1 - THE MUSICUBE ARRANGER

The first is an exploratory music application. The MusiCube Arranger (MCA) is a TUI-based interactive system for children to explore and enjoy the creation of short repetitive musical sequences. Percussive rhythms can be interactively created by simply arranging different colored i-Cubes together and listening to its immediate rendition. Melodic sequences from other instruments can be

layered over the repetitive rhythmic sequences earlier to produce progressively more complex compositions. We describe how the different features are exploited in the MCA to realize its interactive functionalities.

created musical i-Cube various

System Setup

The basic system setup of the MCA is shown in Figure 12. A total of seven i-Cube units were used in the prototype implementation. These i-Cubes are in constant wireless communication with a host PC, which assumes a background role and is not involved in the actual physical interactive play. The PC computes the current 3D block face and orientation relationship between i-Cubes, executes the sequencing algorithms, supports the mixing and rendition of multiple layers of stored musical sequences. Latin percussion selected

Host playback stored music layers in the background

Host PC and sound system

ME cube renders selected sound locally and in arranged order

PLAY mode selected Instrument selector (IS)

B

A

D

C

C

Music elements (ME)

A

B

The user begins by selecting an instrument group using the IS cube. One typically starts by creating the percussive rhythmic sequence. When the MS cube is in the PLAY mode, flipping any ME cube (sensing tilt change) will cause the sound associated with the new top face to be rendered once and the colored light mapped to this sound to blink momentarily on the top face. This allows the user to explore the mapping of different sounds to the different cube faces. In this mode, the sound is explored serendipitously by flipping ME cubes. For a more informed way of selecting musical elements, the color cue associated with the ME cube faces can be turned on. This is done by flipping the MS cube to COLOR and then back to PLAY. Each selection of the COLOR mode toggles the color cue feature. Interestingly, this serendipitous feature can be turned into an enjoyable memory game to test children’s ability to remember which pair of ME cube faces share the same sound or music element. Sequencing Musical Elements

Wireless communication between host and i-Cubes

Mode selector (MS) ME play sequence

Exploring and Selecting Musical Elements

C

D

C

Figure 12. The system set up of the MusiCube Arranger and the musical sequence created by the ME cube arrangement. Functional Roles of the Various i-Cubes

Of the seven cubes, one i-Cube is designated the Mode Selector (MS). The MS cube is used to select the current mode of operation of the MCA. These six modes, which include functions like PLAY, STORE-NEW, STOREAPPEND, TEMPO, COLOR and RESET will be described shortly. Another i-Cube is designated the Instrument Selector (IS). The IS cube provides the selection of six different categories of musical instrument types. In our prototype implementation, these include Percussive (Latin Percussion and Bongo Drums) and Melodic (Piano, Flute, Bell and Guitar) instruments. Both MS and IS cube faces are labeled with appropriate visual or text motifs to aid selection (see Figure 12), and this is done by flipping the desired choice to top facing. The built-in accelerometer detects this selection by sensing the i-Cube’s static tilt relative to earth’s gravitational acceleration. The other five unlabeled cubes are the Musical Element (ME) cubes and are used to arrange the desired musical sequences. Each ME cube, with its six faces, provides the user with six different choices of sound. For example, in the piano instrument, repetition of notes from A to G of a single octave are distributed over the thirty available cube faces of the five ME cubes.

Once the desired sound on a ME cube has been identified, it can be placed next to the MS cube in PLAY mode to begin arranging the musical sequence in various configurations. Packing the ME cubes side by side in the basic 1-D configuration shown in Figure 12 will order the notes sequence to start playing from the ME cube closest to the MS cube (PLAY). Stacking ME cubes vertically allows different notes to be rendered simultaneously. This is similar to the composition of musical chord shown in Figure 4b and is made possible by the 3D block faceawareness capability of the i-Cube. A combination of sideby-side and vertical arrangements is also allowed. MS cube (TEMPO)

MS cube (TEMPO)

90° anti-clockwise rotation inserts a unit-time rest

(a)

Two 90° anticlockwise rotations insert two unit-time

(b)

Figure 13. Inserting different duration of rest periods using the orientation change of proximity sensor detection feature of the i-Cube. Insertion of (a) quarter rest and (b) half rest. Changing Tempo and Inserting Rest Period

Musical rest or silent periods can be inserted into the composed sequence by placing the MS cube in TEMPO mode over the note (or chord) where the rest period is to be inserted. The rest is inserted in sequence after the selected note. The block orientation detection feature of the i-Cube is used to perform this operation. A 90-degree anticlockwise rotation inserts a unit-time rest after the selected

ME cube (see Figure 13a) and an additional 90-degree anticlockwise rotation will produce a total of two unit-time rests (see Figure 13b). Clockwise rotations can be used to decrease or remove the rest period. Layering Different Musical Sequences

Layering is the process of creating multiple soundtracks separately and then mixing them together to be rendered in unison. Once the first musical sequence has been arranged and found to be satisfactory, the MS cube is flipped to STORE-NEW to transfer the composed sequence to the host PC, where it be repetitively rendered as a separate background audio layer. The MS cube is then changed to the PLAY mode so that another musical sequence can be arranged using the ME cubes and with the aid of the repetitive background audio. The second and subsequent musical sequences can be stored using the STORE-NEW mode, which means it will be stored in a new layer and render in unison with the previous layer. Alternatively, it can be stored to the host PC using the STORE-APPEND mode on the MS cube. In this case, the new musical sequence will be played immediately after the last stored sequence (i.e. appended) in the same layer. This APPEND feature allows longer musical sequences to be created using just five ME cubes. The MCA system can create longer musical sequences per arrangement but this will require the availability of more ME cubes. The last remaining mode on the MS cube is the RESET mode, which is used to delete the current and all previously created musical layers so that a new session can be initiated. The MusiCube Arranger is designed as an exploratory creative activity to help children learn musical concepts such as rhythm, tempo, harmony and melody. It also allows children to explore how different percussive sounds can be combined to produce interesting and complex rhythms. There are other music-oriented applications that use cube-based manipulative. Block Jam [16] and Siftable [14] are notable examples. MCA’s main differences with these examples are the use of six generic cube faces for composition and the use of full 3-D spatial block arrangements. As a result, one can freely stack “sound” one on top of another to produce simultaneous rendition of multiple notes in musical chords. This can allow a child to explore harmony in chord construction while constructing musical sequences. APPLICATION #2 – THE SPELLING CUBE

The second application is less exploratory in nature but demonstrates how the i-Cube can be used to provide helpful context-aware scaffolding feedback in a more taskoriented learning scenario. Spelling Cube (SC) is a discovery-based learning application that uses several iCubes to engage preschool children in learning how to spell short 3-6 letter English words (see Figure 14a). Unlike MusiCube Arranger, which only involves interaction with the i-Cube, SC combines the TUIs with a notebook display

to provide richer audio-visual feedback and instructions to the children. It can be argued that this is not the preferred manner of using TUIs since there is now two focus of attention for the child (the screen and the blocks). We did consider the alternative of using just audio cues to provide the spelling instructions. However, we were advised by preschool educators to engage as many of the young children’s sensory dimensions as possible (especially the visual) if we wanted to explore the use of such tools for preschool children (i.e. 3 to 6 years old). As it is, we also took advantage of the available screen to provide interesting animated stories as motivation to go through the required spelling list during field trial sessions. SC supports more than 400 different 3 to 6 letter words, especially those from the Dolch word list. Arrangement of blocks is done within a sloping soft rubber-lined container. Besides providing better top active face visibility, it helps resolve left-to-right ambiguity (see Figure 14b). Tilt angle of 25°

Sloping container Tilt of accelerometer’s x-axis within i-Cube resolve left-to-right direction of block arrangement (a)

(b)

Figure 14. (a) A child spelling the word “t-o-y-s” with Spelling Cube. (b) The sloping enclosure (25°) improves visibility of the cube’s top face and helps resolve the left-to-right ambiguity. Accommodating Diverse Abilities

The spelling ability of children can vary significantly because of the differing development rates in linguistic competency (e.g. between boys and girls) [30]. SC caters for diverse abilities. For example, the spelling tasks is customized by the teacher by changing the content of Microsoft Powerpoint (PPT) slides. This popular presentation tool removes the need to develop sophisticated authoring software for SC. Multimedia contents such as animation, images and sound files can be easily incorporated to make each spelling task interesting. Verbal instructions are important for very young children, whose instruction medium is usually oral in nature. SC sequences through the PPT slides using a text-based configuration file, which details the slide number, word to be spelt and the next slide number after task completion. Partially or completely spelt words (visual cues) presented can allow children to spell by simply matching letters on the screen with that on the cube face. If required, visual help prompts highlighting relevant letter faces or context-aware feedback (see Figure 15) can be turned off for more capable children. Context-Aware Scaffolding Feedback

Since the i-Cube is spatially-aware of the cube identity, face and orientation of its neighboring cube, the current word arrangement by the child is known to the system. Context-aware feedback is provided to the child to

progressively guide her towards the desired learning goal, which is to spell a given word correctly. In SC, help cues include highlighting relevant letters to aid cube face search and reduce possible letter permutations so that the child is able to reach the goal with greater ease. In addition, context-aware feedback is provided if there are incorrect letters in the sequence, there are correct letters but placed in the wrong position or any letter is inverted. Both lights (red for incorrect letters) and verbal feedback (e.g. “two letters are in the wrong position”) are given after the word is formed by pushing the i-Cubes together (see Figure 15b).

(b)

(a)

Figure 15. Context-aware cues and prompts from Spelling Cube. (a) Visual prompts (blue) light up cube faces with letters needed to spell the current word. (b) Context-aware visual (red) and verbal feedback can be given to the child when letters are incorrectly placed or a letter is upside down. SPELLING CUBE EVALUATION

The Spelling Cube system was field trialed with 37 preschool children between the ages of 3 to 6 years. The objective of the study was to observe the way children used the SC by themselves and with another classmate of the same age and evaluate their preferences for different features of the i-Cube and SC design. Each child was asked seven questions after having used the Spelling Cube to spell about 5-7 words (the difficulty of the word list was varied depending on the age group). Given the need to rate their response over a 5 point Likert scale, the majority of the children questioned were between the older ages of 5 to 6 years and their opinion sought in a way that they can understand, namely by using exaggerated expressions, visual charts and gestures. Questionnaire Results for Spelling Cube (SC)

Questions on SC Q1 Do you like playing with SC?

5

Q2 Do you like looking for the letters

4

Q3 Do you find the blue lit letters helpful?

3

Q4 Do you like the red light that tells

2

Q5 Do you like the lights and sound reward

using the blocks?

you when you are wrong?

when you got the answer right?

1

Q6 Do you like the click sound the blocks Q1

Q2

Q3

Q4

Q5

Question number

Q6

make when you turn them around?

Q7 Q7 Do you like the animation on the computer screen?

Figure 16. The mean user ratings for the 7 questions posed to the children and their associated standard deviation bar. User ratings were captured on a 5 point Likert scale, where 1 is (strongly disagree) and 5 is (strongly agree).

The results of the evaluation and the details of each question asked are shown in Figure 16. Overall, the children have a positive impression of Spelling Cube and its technological features. In terms of preference, the “click” audio feedback when the i-Cube is turned around was rated the lowest at 4.1 and the brief light and sound jingle played when the child got the correct answer was rated the highest at 4.8. With spelling being a rather taskoriented activity, the audio-visual reward (jingle) relating to a successful outcome produced very positive responses in the children, suggesting that these feature of the i-Cube could be exploited as a potential motivational strategy in future i-Cube applications. Children also likely the contextaware cues and prompts (see Figure 15) very much as indicated by the high scores of about 4.6 in both questions Q3 and Q4. Surprisingly, most children did not dislike the tedious tasks of looking for the appropriate letter by having to physically turn the cube to perform a visual search. This was inferred from the mean score of 4.6 for question Q2. CONCLUSIONS

A new block-based TUI called the i-Cube has been presented. Like the traditional wooden alphabet blocks, it is designed for easy manipulation. But unlike these blocks, it is imbued with the ability to sense motion, tilt, 3-D spatial relationships and provides context-aware audio-visual feedback. These embedded technologies provide us the ability to extend the pedagogical potential of manipulative block play. Indeed, the i-Cube was designed with the purpose of providing educators with a generic and flexible tool to better understand how tangible manipulatives can be used to support learning through play. As such, the i-Cube design takes into account practical issues related to the usability of block-based manipulative during exploratory activities. Much effort has gone into making the i-Cube support complete 3-D block arrangement flexibility so that they can be used to represent and model complex concepts and structures that can only be expressed using the three physical spatial dimensions of our real world. Field trial evaluation with 37 pre-school children using the Spelling Cube suggests that they have very positive preferences for physical tangibility and the audio-visual feedback afforded by the i-Cube. They also find the rudimentary context-aware feedback helpful during their attempt to spell words that were new to them. Evidence that tangible manipulatives facilitate collaborative activities was also observed but the potential of TUIs to support collaborative learning needs further investigations. ACKNOWLEDGEMENT

This research is supported by the Singapore National Research Foundation and Ministry of Education (NRFMOE) IDM for Education grant (NRF2008-IDM001-017). We wish to thank staff and children at Learning Vision @ NTU Childcare Centre for their participation in the Spelling Cube trials.

REFERENCES

1.

Bee, H. The developing child. (9th Ed.). Boston: MA, Allyn and Bacon, 2000.

16. Newton-Dunn, H., Nakano, H. and Gibson, J. Block Jam: a tangible interface for interactive music. Journal of New Music Research, 32 (4), pp 383-393, 2003.

2.

Dekel, A., Yavne, G., Ben-Tov, E. and Roschak, J. The Spelling Bee: an augmented physical block system that knows how to spell. In Proc. of ACE’07, 212-215, 2007.

17. Parkes, A., Ishii, H. and LeClerc, V. Glume: exploring material in a soft augmented modular modeling system. In Proc. of CHI‘06, 1211-1216, 2006.

3.

Eisenberg, M., Buechley, L. and Elumeze, N. Computational and construction kits: towards the next generation of tangible building media for children. In Proc. of Cognition and Exploratory Learning in the Digital Age (CELDA’04), 2004.

18. Resnick, M., Martin, F., Berg, R., Borovoy, R., Colella, V., Kramer, K. and Silverman, B. Digital manipulatives: new toys to think with. In Proc. of CHI’98, 281-287, 1998. 19. Schweikardt, E. and Gross, M.D. A brief survey of distributed computational toys. In Proc. of DIGITEL’07, 2007.

4.

Froebel, F. The education of man. (W.N. Hailman, Trans.). New York: Appleton, 1887.

5.

Fowlkes, M.A. Gifts from chilood’s godmother: Patty Smith Hill. Children Education, 6 (1), p44-49, 1984.

20. Schweikardt, E. and Gross, M.D.. roBlocks: a robotic construction kit for mathematics and science education. In Proc. of ICMI 2006, 72-75, 2006.

6.

Gura, P. (Ed.). Exploring learning: Young children and block play. New York: Paul Chapman, 1992.

21. Sifteo Cubes. https://www.sifteo.com/ .

7.

Horne., J. Gender differences in computerised and connventional educational tests. Journal of Computer Assisted Learning, 23 (1), 47-55, 2007.

8.

Hunter, S., Kalanithi, J. and Merrill, D. Make a Riddle and TeleStory: designing children’s application for the Siftables platform, In Proc. of IDC’10, 206-209, 2010.

9.

Kirchner, P.A., Sweller, J. and Clark, R.E. Why minimal guidance during instruction does not work: an anlaysis of the failure of constructivist, discovery, problem-based, experimental and inquiry-based teaching. Educational Psychologist, 41 (2), 75-86, 2006.

10. Kranz, M., Schmidt, D., Holleis, P. and Schmidt, A. A display cube as a tangible user interface. In Adjunct Proc. of UbiComp 2005, 2005. 11. Leichtenstern, K., Kranz, M., Holleis, P., Lösch, E. and Andre, E.. A tangible user interface as interaction and presentation device to a social learning software. 4th Intl. Conf. on Networked Sensing Sys. (INSS'07), 2007. 12. Marshall, P. Do tangible interfaces enhance learning? In Proc. of TEI’07, 163-170, 2007. 13. Meadows, S. The child as thinker: the development and acquisition of cognition in childhood. London: UK, Routledge, 1993. 14. Merrill, D., Kalanithi, J. and Maes, P. Siftables: Towards sensor network user interfaces. In Proc. of TEI’07, 2007. 15. Montessori, M. The Montessori method. New York: Schocken, 1964.

22. Smith, A.C. Using magnets in physical blocks that behave as programming objects. In Proc. of TEI’07, 147-150, 2007. 23. Terrenghi, L., Kranz, M., Holleis, P. and Schmidt, A. (2005). A Cube to learn: a tangible user interface for the design of a learning appliance. The 3rd International Conference on Appliance Design, 2005. 24. Texas Instruments - CC2510/CC2511 Development Kit. www.ti.com/tool/cc2510-cc2511dk . 25. Watanabe, R., Itoh, Y., Asai, M., Kitamura, Y., Kishino, F. and Kikuchi, H. The soul of ActiveCube – implementing a flexible, multimodal, three dimensional spatial tangible interface. ACM Computers in Entertainment, 2(4), 2004. 26. K. Wellhousen and J. Kieff, A constructivist approach to block play in early childhood. Canada: Delmar, Thomson Learning, 2001. 27. C. H. Wolfgang, L. L. Stannard and I. Jones. Block play performance among preschoolers as a predictor of later school achievement in mathematics. J. of Res. in Childhood Education, vol. 15 (2), p173-180, 2001. 28. Z-agon: Mobile multi-display browser cube. T. Matsumoto, D, Horiguchi, S. Nakashima and N. Okude. In Proc. of CHI 2006, 351-356, 2006. 29. Zeller, L. and Scherffig, L. CubeBrowser – a cognitive adapter to explore media databases. In Proceedings of CHI 2009, 2619-2622, 2009. 30. Zuckerman, O., Arida, S. and Resnick, M. Extending tangible interfaces for education: digital Montessoriinspired manipulatives. In Proc. of CHI’05, 859-868, 2005.