Integrating learning supports into the design of visual programming ...

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Integrating learning supports into the design of visual programming systems Chris DiGiano Center for Technology in Learning SRI International Ken Kahn Animated Programs Allen Cypher David Canfield Smith Stagecast Software, Inc. Send correspondence to: Chris DiGiano Center for Technology in Learning SRI International 333 Ravenswood Avenue Menlo Park, CA 94025 USA [email protected]

1

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Abstract The success of a programming system depends as much on the learnability of its language concepts as the usability of its interface. We argue that learnability can be significantly improved by integrating into the programming system learning supports that allow individuals to educate themselves about the syntax, semantics, and applications of a language. Reflecting on our experience with developing novice programming systems, we identify infrastructural characteristics of such systems that can make the integration of learning supports practical. We focus on five core facilities: annotatability, scriptability, monitorability, supplementability, and constrainability. Our hope is that our examination of these technical facilities and their tradeoffs can inform the design of future programming systems that better address the educational needs of their users.

Keywords Programming systems, learning, instructional design, end-user programming, visual programming, programming by example

Introduction Consider the following scenario: after years of intense development, a crack team of programming language designers at a prestigious research institute develop a powerful yet simple new programming paradigm. Their visual programming environment makes it easy for users to quickly solve real and interesting problems. Working with students at a local elementary school, the team members are excited to see that even children are able to use their language. With great fanfare, they release their language to the world over the Internet. But just as quickly as word of the language spreads, the frustrated queries accumulate in the team’s email inbox: “How do I change this behavior?” “What commands can I use to do such and such?” “I’m overwhelmed—where do I start?” What went wrong? In our experience, the scenario describes a common situation for visual programming system designers who fail to devote enough attention to learnability issues. Since programming languages have potentially infinite applications, it is difficult to anticipate the important concepts that users need or want to learn without a principled learner-centered design process. With visual programming languages, it is particularly easy for designers to overlook learning needs because of the common misconception that such graphical environments are inherently more comprehensible [1]. The team in our scenario could still try to make amends. They could add to the programming environment an exploratory mode that allowed users to experiment safely with a subset of the language. They could distribute a tutorial that oriented users to the language. They could provide interactive demonstrations and example programs on their Web site. They could even create a special area on their site to support a community of on-line learners who—even when the team members themselves are preoccupied—can help each other succeed with the language. Unfortunately, the architecture of their programming environment never anticipated the need to provide these kinds of support mechanisms. So, part of the team spend months adapting the programming environment, annotating screen snapshots for the tutorial, recording demonstrations, and writing instructions for retrieving example programs from the Internet. More time is spent trying to compress all the materials into reasonably sized packages for download by the large but confused audience. Meanwhile, the rest of the team have taken the opportunity to improve the programming environment interface. Unfortunately, the changes render the tutorial and demonstrations obsolete, even before they are released. By now, the onceexcited user community has moved on to other things. In the end, the biggest mistake was not that the team in this scenario failed to foresee critical learning obstacles, but rather that their architecture failed to easily accommodate the integration 2

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. of learning opportunities. Without such infrastructure, the team had to resort to ad hoc, brittle, and poorly integrated educational materials. We believe that poor infrastructure for learning supports is a common pitfall in programming system design. Indeed, the story reflects some of our own experiences as part of ambitious commercial design efforts, including LEGO® MINDSTORMS™ RCX Code, Animated Programs ToonTalk™, and Stagecast Creator™ (formerly Apple’s Cocoa). In this paper, we reflect on our own experiences and identify several design strategies that, if adopted early in the development process, can provide critical insurance against too-little, toolate learning supports in programming systems. We begin by reviewing from our own experience and the programming language design literature techniques that can introduce programming concepts to various types of learners. Next, we identify infrastructural characteristics of programming systems that can make the integration of learning supports practical. We focus on five core facilities: annotatability, scriptability, monitorability, supplementability, and constrainability. Our hope is that our examination of these technical facilities and their tradeoffs can inform the design of future programming systems that better address the educational needs of their users.

Background The success of a programming system depends as much on the learnability of its language concepts as the usability of its interface for selecting from and composing language structures. When creating a programming system, designers should consider integrating into their system learning supports that individuals can use to teach themselves the syntax, semantics, and applications of a programming system. This article was motivated by our own experiences in developing learning supports for several visual programming systems, which we describe briefly below. All are designed for novice programmers (sometimes referred to as “end users” or “casual programmers”) who want to control their computer in more detail but who have never taken formal programming classes. All but one of these systems was developed with children in mind. We use the systems as illustrations throughout the remainder of the paper. LEGO Mindstorms is a new line of programmable construction kits featuring a cellular-phonesized microcomputer that can form the base of autonomous robots and other “smart” things. The first product, the Robotics Invention System (RIS), includes a CD-ROM with a visual programming system, RCX Code, for specifying the behavior of these inventions. RCX Code consists of colorful language puzzle pieces that fit together to form stacks of commands or event handlers. Recently released is the Robotics Discovery Set, in which users create their programs directly on the face of a redesigned microcomputer. DiGiano helped design the RCX Code “Training Center” and “Programmopedia” for the Invention System, as well as the paper manual for the Discovery Set. Animated Programs ToonTalk [2, 3] is an animated world in which children build and run programs by performing actions on concrete objects. Children build real computer programs by giving messages to birds, training robots to work on boxes, loading up trucks, and using animated tools to copy, remove, and stretch items. ToonTalk includes several narrated demonstrations, a puzzle sequence consisting of about 70 puzzles, many sample creations, and an interactive coach/guide character. Stagecast Creator [4, 5] is a customizable platform enabling users to easily create interactive stories, puzzles, games, and simulation models—collectively known as “worlds.” Users create the characters in the simulation and program their behavior by defining the rules by which they interact. Creator comes with a built-in interactive tutorial, illustrated manuals, on-line help, sample worlds, a dynamic Web site, and a Teacher’s Guide that offers a wealth of ideas for incorporating the product into elementary or middle school curricula.

3

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Chart ’n’ Art [6, 7] is a programmable information graphics tool that combines standard directmanipulation drawing tools with a textual programming language to help users place and size elements with precision and to automate repetitive tasks. Unlike the above programming systems, Chart ’n’ Art is not a commercial product; rather, it is a prototype system developed to explore how users can learn a programming language incidentally through exposure to continuous system feedback called “disclosures.”

Learning Supports for Programming Systems Ideally, users of programmable systems would learn to create interesting and useful artifacts from a human tutor. Indeed, this is how our first user testers became familiar with RCX Code, Creator, ToonTalk, and Chart ’n’ Art. Obviously, personalized tutoring is not a practical way to educate users about commercially distributed software. Even in the school setting, where Creator is targeted, we cannot assume that users will have the individual attention of knowledgeable teachers. Instead, we believe it is up to designers of programming systems to integrate learning supports—defined as resources that individuals can use to teach themselves the syntax, semantics, and applications of a programming system. A single programming system may require a range of learning supports, because users of different ages, experiences, and learning styles approach the same software in very different ways. An important observation we have made from our experience with children using programming environments is that they differ in the degree to which they are motivated and effective at (1) exploring (on their own or with an animated guide), (2) following instructions, (3) watching a prerecorded demonstration, or (4) solving a puzzle sequence. Even the same child will prefer different styles of interaction at different times, depending on her prior experience with the software. Furthermore, even with programming systems oriented toward children, designers can expect a significant population of adult learners. For example, LEGO was surprised to learn that roughly half of all Mindstorms RIS kits were being purchased by adults who wanted to get involved in robotics. Stagecast also has a variety of adult users, including publishers who are creating supplementary material for their textbooks. Designers must be mindful of the broad range of potential users and integrate a corresponding variety of learning supports into their systems. Below, we identify what we consider the basic types of learning supports found in many programming systems. We discuss the utility and benefits of each of these with examples from the systems described above. Animated Demonstrations. An effective teaching method is to show or demonstrate program features or programming techniques to students. By recording keyboard and mouse events and replaying them, such demonstrations can be automated. Viewing demonstrations, ideally with synchronized narration or subtitles, can be a good way to learn about the components, allowable actions, and composition techniques of any complex software. Very brief demonstrations can show a very specific skill or technique. Hundreds of such brief demonstrations can be organized as an animated “how to” guide. Demonstrations can be interactive, allowing users to interrupt playback at any time and take control. Constrained Exploration. Some learners enjoy discovering things by exploring a programming system on their own. A key learning support for these kinds of users is an exploratory mode in which they are mostly free to experiment, but constraints gently keep them on a pedagogically productive path. In general, one can construct a system in which much of the functionality is hidden or inaccessible and is incrementally revealed as users master the system. Even popular video games such as Super Mario Brothers take this approach. These games provide a rich world with many kinds of objects and actions, but they start off in a simple section, and players must

4

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. explore to discover new kinds of things. Players have the illusion of freedom, but the designers of such games have carefully planned out what parts are revealed and when. Guides and Coaches. A great way to learn a new programming language or environment is to be guided or coached by a human colleague or teacher. Although this process is very difficult to automate, a quasi-intelligent interactive character can be very helpful. Ideally, the coach should maintain a user model to decide what to explain or suggest. The software needs to be aware of what the user is doing in order to provide advice. A well-known coaching system for textual programming languages is Cognitive Adaptive Computer Help [8], which provides relevant feedback as users enter programs. An example of coaching in the context of a visual programming system is ToonTalk’s Marty the Martian, shown in Figure 1.

Figure 1. Marty the Martian offering help in ToonTalk. The hand in the foreground is controlled by the user with a joystick or mouse. Internet Resources. Internet-based resources, such as reference guides, tips and tricks, frequently asked questions, and sample programs, are useful supplements to the standard installation of a programming system. Allowing resources to be extended on the Web gives the system’s learning supports room to grow. LEGO’s Programmopedia, for example, was created on CD-ROM and on-line. The CD-ROM has a deliberately simple design to introduce the help reference, so that the Web version does not seem foreign. The Web version allows for constant and frequent updates in response to common problems. Paper Resources. Paper-based materials have the unique advantage of being highly portable and technologically robust, that is, they are “compatible” with even the most anemic end-user computer. The Mindstorms Robotics Discovery Set includes an extensive manual to introduce children step by step to programming their microcomputer from the device’s front panel. ToonTalk has also experimented with pictorial “instruction books” that illustrate how to use the language to create sample programs. It is worth noting that although Stagecast developed an extensive set of paper resources for Creator, it rejected a paper-based introductory tutorial on the grounds that it would be too awkward to flip through the pages in the manual while trying out the interface for the first time. Self-Disclosure. In many programming systems, users can accomplish the same task in several ways. For example, dual-view systems such as Chart ’n’ Art (see Figure 2) allow users to manipulate shapes directly with the mouse or by typing in expressions that perform the manipulations with high precision and over large numbers of objects. In some programming-bydemonstration systems, users can specify actions by rehearsing changes to a scene with the mouse, or they can explicitly compose the underlying commands and conditionals. Self-

5

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. disclosure is a learning support that exposes users to these alternative programming techniques in the context of authentic problem solving. For example, as a user drags out a new shape in Chart ’n’ Art’s drawing window, the system discloses the equivalent textual command for shape creation. Similarly, when a user changes the appearance of a Creator character while recording a rule, the system later discloses the command “Set the appearance of character 4 to ‘Facing Left’.” Static Samples. Sample programs, included in most programming systems, can provide important opportunities for learning by example. For the purpose of this paper, we narrow this category of learning supports to static samples to distinguish it from animated demonstrations. Arguably, the more useful samples are those that are well documented and situated within meaningful problem contexts. Mindstorms RIS, ToonTalk, and Creator each come with static samples oriented around engaging problems for children. Many of these include comments or annotations that help children make sense of the sample.

Figure 2. The dual views of Chart ’n’ Art. Above is a drawing window for directly manipulating shapes. Below is the language window for typing expressions to operate on the drawing window. Other Learning Supports Although the above learning supports are obviously not exhaustive, they do offer a vocabulary for characterizing a wide range of resources. For example, the Mindstorms RCX Code Training Center is a tutorial that involves animated demonstrations and constrained exploration. Puzzles in ToonTalk combine a coach and constrained exploration. ToonTalk also includes a game-like sequence of puzzles that acts like a tutorial. The critical aspect of a puzzle is that a player needs to figure out how to do a difficult task in a simpler constrained world where system functions have been temporarily disabled.

6

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Table 1. Facilities in the architecture of programming systems and their application to common learning supports.

•

•

•

•

SelfDisclosures

•

Static Samples

•

• • •

Paper Resources

•

Internet Resources

Guides and Coaches

• • • •

Constrained Exploration

Facility Annotatability Scriptability Monitorability Supplementability Constrainability

Animated Demonstrati ons

Learning Support

• • •

•

Learning supports vary along both an active/passive dimension and a directed/creative dimension. Demonstrations, for example, are both passive and directed. A puzzle is active and somewhat creative, while a sequence of puzzles may be active and directed. Pictorial instruction books are directed and can also be thought of as active (since learners are involved in translating the images into on-screen actions). Exploration is typically both active and creative. Further discussion of the dimensions of learning supports can be found in [9].

Facilitating Learning Supports In our experience, there are three main requirements for a learning support to be integrated in a programming system: (1) sufficient human capacity for actually implementing the support, (2) sufficient computer capacity on the target machine to run the learning support (this obviously does not apply to paper materials), and (3) technical infrastructure in the programming system itself to enable the learning support. In developing our systems, we have had to reject some learning supports at various times for lack of one or more of these requirements. For example, QuickTime video demonstrations were rejected for Creator because users might not have sufficient screen real estate or processor power. Although a constrained exploration mode would have been desirable for Mindstorms’ RCX Code, it was impractical given the infrastructure provided by the environment’s underlying code base, Europress’s Click & Create. Lack of technical infrastructure is perhaps the most preventable reason for rejecting, compromising, or delaying learning supports. Reflecting on our own experiences, we realize that particular learning supports would have been much more practical had certain facilities been integrated into the architecture of our programming systems. In this section, we describe five core facilities: annotatability, scriptability, monitorability, supplementability, and constrainability. Table 1 summarizes how these techniques can facilitate each of the learning supports described in the preceding section. For each facility we examine implementation issues in providing the facility for one’s programming system. Obviously, the underlying implementation language for a programming system will affect how difficult this is. For example, whereas monitorability can be a challenge for C++-based systems, Lisp essentially provides this facility for free through its “advise” function. Our discussion generally assumes that developers are using a compiled language, such as Java or C++. To help developers decide whether a facility is worth the effort, we also identify potential consequences of leaving the facility out of a programming system.

7

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. Annotatability Annotations are a useful technique for displaying informational text and images (and even playing sounds) in context, located directly where the information is relevant. We use the term annotatability to describe the degree to which this is possible. Probably the most familiar form of annotation is “tool tips” or “hover text” that appears when one holds the cursor over user interface controls in productivity packages such as Microsoft Word. User interface designers continue to find new applications for annotations, such as for personalizing the Web browsing experience [10]. Annotations are distinguished from other kinds of feedback in the way they can position themselves near the object they are describing. If the annotated object moves to a new location, the annotation often moves with it. In the context of programming systems, annotation layers allow the learning engineer to present ancillary text or highlighting on top of existing learning supports or to integrate learning opportunities into normal programming activities: • Annotatability can be useful to mark up animated demonstrations to highlight and label the most important information in a demonstration playback, as well as to provide subtitles. • Annotation layers allow learning engineers to present students with an interface for navigating from task to task during constrained exploration. For example, Creator’s built-in tutorial employs a text box at the bottom of the Creator window for the display of a couple of sentences of instruction and a “Next” button to advance to the next task. • Annotation layers can also provide a mechanism for the display of guides and coaches, such as ToonTalk’s Marty the Martian (see Figure 1). Marty also makes use of text-to-speech technology to literally comment on what is happening on-screen. • Annotatability can provide creators of paper resources with a useful tool for marking up screen snapshots to be printed. This can be an alternative to using a drawing program to go over snapshots later. • As with demonstrations, annotatability can also add ancillary information and highlighting to program samples. This was done in the Mindstorms RCX Code environment to display Challenge instructions. • Finally, annotatability can be central to effective self-disclosure, as described in the following section. Implementation Issues Annotatability can be a challenging capability to incorporate into one’s programming system, because it typically requires defining a special class of objects that can float over icons and other graphics. In adding annotatability to Chart ’n’ Art, for example, DiGiano encountered three implementation hurdles: (1) drawing outside of windows, (2) effective positioning of annotations, and (3) animating annotations. The first problem is illustrated in Figure 3, a screen snapshot showing Chart ’n’ Art’s selection-labeling self-disclosure mechanism. Selection labels appear in Chart ’n’ Art after the user selects one or more objects in the drawing window. They are designed to expose learners to programming expressions for referring to one or more objects. When adding support for these annotations, it was soon obvious that labels would often be too long to fit within the drawing window. The workaround was to implement the labels as independent windows—without title bars or other controls—that could float on top of the drawing window. This solution led to the second issue, that of effectively positioning Chart ’n’ Art annotations after the user makes a selection. Centering the label beneath the objects being annotated worked only when there was sufficient screen space at the borders of the drawing window. A layout algorithm had to be added to shift the label to the right in the event it would be clipped by the left edge of the screen.

8

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Figure 3. A selection label in Chart ’n’ Art. To allow the label to extend beyond the boundaries of the drawing window, the label had to be implemented as its own independent window. At first, the annotation facilities built for selection labeling were also considered for animated language annotations called mouse disclosures. Mouse disclosures are a single line of text that appears just below and to the right of the cursor as one creates and manipulates shapes by dragging the mouse. They are designed to introduce users to creational commands, object properties, and the drawing window coordinate system. The initial implementation used the floating-window approach used in selection labeling. Unfortunately, the label window rendered far too slowly for the annotations to appear while the user was dragging the mouse—for example, while repositioning a shape. Indeed while the user was dragging out the outline for a new shape, Chart ’n’ Art’s underlying GUI framework, SK8 (an experimental multimedia authoring system from Apple Computer, Inc.), did not allow a window to be rerendered at all. Ultimately, a trap was added to SK8’s low-level mouse tracking and outline tracking (or “rubber banding”) routines that enabled text to be drawn in XOR mode while the user dragged the mouse. This technique produced an acceptable refresh rate for animated annotations. Risks in Ignoring Annotatability Although there are clear implementation challenges for annotatability, it is important to consider the risks in not providing such a capability in a programming system. There are three fundamental problems: (1) missing opportunities (such as Chart ’n’ Art disclosures) to provide integrated pedagogical feedback directly where users are currently focusing their attention; (2) missing opportunities to present information in a timely way, when users are most ready for it; and (3) not being able to automatically constrain the position of graphical or textual feedback relative to the position of programming objects on the screen. This latter problem was an issue in creating Mindstorms RCX Code Training Center. The design of the Training Center tutorial relied on the ability to point to code blocks within the programming environment. Ideally, this could have been implemented with a script that requested that the programming system display a pointer adjacent to a certain set of language puzzle pieces, regardless of where they happened to be on the screen. However, the RCX Code environment was not originally designed with “hooks” for specifying such annotations. For this and other reasons, it was decided that it would be easier to completely simulate the RCX Code environment within a Macromedia Director™ movie rather than adding annotation facilities to the programming system itself. To the untrained eye, the result, shown in Figure 4, looks and behaves (within the limited context of the Training Center’s “missions”) almost exactly like the real programming system. The catch is that any future changes to the programming environment must be replicated in the Director movie.

9

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Figure 4. A “training mission” in the Mindstorms RCX Code Training Center. Here the user is being asked to insert a new code block. The arrow to the right of the puzzle pieces and the text on the far right are annotations designed to guide the user through the task. Scriptability Scripting enables users of a system to compose a series of commands (sometimes referred to as “macros”) that perform idiosyncratic functions. Many advanced tools incorporate some form of scripting facility, the most powerful incarnations being languages that can be used to control and coordinate multiple programs. Examples include the scripting support in the graphical editor framework Topaz [11] and Apple Computer’s AppleScript™. We use the term “scriptability” to describe the degree to which a system allows itself to be controlled from a stored or computed script in place of genuine user events such as mouse clicks and keystrokes. A good analogy for a scriptable system is an old-fashioned player piano, which can make music by reading from a roll in place of waiting for a musician to touch the keys. Of all learning support facilities, scriptability is perhaps the single most important piece of infrastructure that can be built into a programming system. Scriptability allows the learning engineer to play back instructional sequences or take over the programming system when the learner needs help: • Scripts can specify how an animated demonstration unfolds from an initial saved state. • Guides and coaches can use scripts to perform some actions on behalf of a learner who is stuck and needs help completing part of a programming problem. • Self-disclosures can be scripted to occur at idle time to expose learners to new commands or variations on commands they have been using. An example of using scripting in this way is Chart ’n’ Art’s “pedagogical screen saver” [12], which during idle time makes random changes to the current drawing and discloses the commands that were used. The programming system reverts to the previous state as soon as the user touches the mouse or keyboard. Implementation Issues The choices for implementing scriptability are numerous and highly dependent on the desired mechanisms for authoring scripts (described in the next section, on monitorability). Research by Lieberman [13] and others continues to explore the tradeoffs in enabling one’s system to be scripted. In our work, we have used at least three techniques: (1) low-level event simulation, (2) hard-coded call sequences, and (3) application factoring. We use the term low-level event simulation to describe how a programming system can be “fooled” into accepting simulated 10

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. events such as mouse clicks and key presses instead of genuine ones from a real user. This is the approach used to implement ToonTalk’s animated demonstrations, as shown in Figure 5. Demonstrations are specified by a script file consisting of a “click stream” of interaction events. Integrated into the script are regular samples of the mouse locations, so that the cursor can be shown traveling from one interaction event to the next. In addition the script can contain voiceovers and subtitles.

Figure 5. A scene from one of ToonTalk’s animated demonstrations. A prerecorded script file drives the demonstration and provides subtitles for the bottom of the screen. A significant issue in ToonTalk’s approach is backward compatibility with legacy scripts. This turned out to be the most difficult aspect of maintaining ToonTalk demonstrations. An enhancement (or even a bug fix) might cause old scripts to no longer play correctly. To address this issue, ToonTalk records a version number in logs and behaves in the old way when playing an old script. This is difficult to do systematically and can lead to a slower and larger executable. An issue encountered during recent user tests of scripted ToonTalk demonstrations was that learners wanted to be able to speed up or slow down playback. They also wanted to be able to “rewind a section” and replay some part of a demonstration. Support for such playback controls is currently in beta testing. Instead of recording several minutes of input events, ToonTalk can now record short segments (typically 5 to 10 seconds long). Each segment includes a saved state of the entire ToonTalk world and a recording of input events that occurred after the state was saved. The interface now provides a way to jump forward or backward one or more segments. Kahn is planning to remake the demonstrations as a series of such sections. A fast-forward mode would be feasible only on machines that are faster than the speed of the recording machine. The unpredictable nature of playback speed also made it difficult to synchronize ToonTalk narration and subtitles with a demonstration. If narration and subtitles ran as a separate QuickTime movie, for example, it would be easy for them to end up ahead of or behind what was happening in the ToonTalk world being demonstrated. The solution was to break narration and subtitles into small pieces and trigger the playing of each piece from an event in the log. Note that this solution interferes with any attempt to provide the narration in a manner that streams off the Internet or a CD-ROM. Rather than playing back click streams, Creator uses hard-coded call sequences to drive the single animated demonstration found in its tutorial. The tutorial itself makes use of special tutorial commands added to Creator to display instructional text at the bottom of the screen, as 11

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. shown in Figure 6. At a particular step in the tutorial, an internal Creator function is triggered that invokes a sequence of movements by a special character who shows a rule being made. It is interesting to note that this capability was added only later as learning engineers realized the need to show rule creation in an animated fashion rather than step by step.

Figure 6. A typical screen from the Stagecast Creator tutorial. The instructional text appears in a small box at the bottom of the screen. Note how most of the buttons on the control panel are grayed out to constrain the user. The stop button on the control panel—which the user is instructed to click—has a flashing yellow border to attract the user’s attention. Probably the most systematic and robust technique for implementing scriptability is application factoring. First developed for SmallTalk’s model-view-controller paradigm [14], factoring involves implementing the interactive graphical application as two independent modules: (1) a user interface that parses primitive user actions into abstract high-level events, and (2) an application engine that actually responds to these high-level events. As a separate module, the high-level event-handling code can respond just as well to scripted events as it can to real-time user events. Unlike click streams, high-level events can refer to objects on the screen, not just locations, and can describe actions as being relative instead of absolute. An example of a highlevel event might be “move Rectangle 2 by {10,50}”, which could indicate that a particular rectangle should be moved a relative number of pixels. Application factoring is the approach used by Chart ’n’ Art. This foundation made it almost trivial to implement its pedagogical screen saver. One or more objects in the drawing window are selected at random, and a high-level event is generated that performs some relative action on them. Risks in Ignoring Scriptability An obvious risk in ignoring scriptability is that learning engineers for a programming system waste time hand-coding instructional sequences. In such a scenario, instructional developers must learn the complexities of the underlying implementation just like the language developers, making for an inefficient division of labor. A more subtle, but perhaps more troubling consequence of ignoring scriptability could be that, without it, third parties find it too difficult to

12

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. generate additional learning material. Such third parties could include, for example, a teacher who wants to augment the programming system with an interactive lesson on how to make biology simulations. Such specialized instruction cannot be expected from the developers of the system, but scripting can make it practical for teachers, advanced users, or domain experts to create the content for themselves. Without scripting facilities, third parties might have to request special access to the programming system implementation and then learn how to make use of it for their lessons. Monitorability Monitorability is the ability of a programming system to log the stream of events that comprises a user’s interaction with the system’s icons, menus, and other interface widgets. Learning supports can take advantage of monitorability in a variety of ways: • In scripting animated demonstrations, a learning engineer can simply record an exemplary session with the programming system, rather than painstakingly specify the playback script by hand. • Constrained explorations can monitor a learner’s path through the programming system and alter the exploration environment as needed. This technique is used by Creator’s tutorial, for example, to present learners with new instructions after they have successfully completed a stage. • By being able to monitor a learner’s progress in solving a puzzle, guides and coaches can better know when to reward the successful learner or interrupt with assistance. • In dual-view programming systems, such as Chart ’n’ Art, the ability to record actions in one view can enable the system to echo equivalent or related expressions for the other view. This “echoing” technique represents a type of self-disclosure that can orient learners to new programming expressions [6]. Implementation Issues The central challenge in implementing monitorability is recording meaningful event streams. As mentioned in the scriptability section, such streams can be at different levels of abstraction, from low-level events such as “mouse-down at {241, 372}, drag to {251, 422}” to higher-level events such as “move Rectangle 2 by {10,50}.” We use the terms low-level recording to describe the monitoring of low-level events and high-level recording to describe the monitoring of high-level events. ToonTalk uses a low-level recording technique to implement monitorability. In essence, the system records every input event and the time it occurred. This simple scheme became very complex in practice. One issue is time: since ToonTalk’s behavior is influenced by the real time, it was necessary to time stamp every input event and on replay use the elapsed time recorded in the log. If the system replaying the log is faster than the one used to record the log, then the replay will play too fast. This was easily fixed by having the system sleep an appropriate amount of time to slow the fast machine down. If the replay machine is slower than the original, then the replay is often too slow to be effective. The best solution to this problem is to record the demonstrations on a slow or average machine (or a fast machine that sleeps to emulate a slower machine). The disadvantage of this approach is that on a fast machine the frame rate is significantly slower than it would otherwise be, so the animation does not look as smooth and professional. Further complicating monitorability is accurately capturing the potentially unique circumstances of a particular recorded session: screen resolution, window size, state of the clipboard, state of user-persistent objects, state of files read or written, and sequence of random numbers generated since the beginning the session. Dealing with some of these factors correctly can cause the size of the log to increase dramatically as it includes more of the operating system and file system state.

13

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. In contrast to ToonTalk, Chart ’n’ Art employs a high-level recording approach to monitorability. As a factored application, it first translates all user interaction into abstract highlevel events called Canonical Language Structures (CLSs), which are a superset of AppleEvents. CLSs are used to drive real-time disclosures of how to perform operations using Chart ’n’ Art’s programming language. CLSs are actually designed to be language independent, which allows Chart ’n’ Art to self-disclose in multiple programming languages (although doing this in any truly general way is an unsolved problem in artificial intelligence). Currently, the system supports both SK8Script and Lisp. Risks in Ignoring Monitorability Without a consistent means of monitoring user actions within a programming system, it can be considerably harder for learning engineers to create interactive tutorials and puzzles. Doing so might require a special tutorial mode within the programming system or simulating the real programming environment within another application altogether. The latter is what happened in making the Mindstorms RCX Code Training Center. Subsets of the RCX Code environment were simulated in Macromedia Director, where we could better track how well users were following tutorial instructions. The former is what happened in making the Creator tutorial. Initially, Stagecast hoped that errors by tutorial users could be prevented with sufficiently detailed instructions. However, testing soon revealed that, even with good instructions, user actions (such as the creation of a new rule) needed to be checked for correctness. This meant writing more ad hoc, special-purpose code for the tutorial. Now, whenever a user clicks the Done button upon completing a rule, custom tutorial-mode code checks that rule against the correct one to make sure it matches. If it does not match, the system reverts the rule and backs the user up to the start of the step-by-step instructions for creating that rule. Supplementability We use the term supplementability to describe the ability of a programming system to be enriched with new and updated learning resources after the initial installation. A highly supplementable system makes it easy for learners to “pull” materials as needed from on-line libraries or for the distributor to “push” updated resources to users. In the future, perhaps such delivery concerns will seem quaint—when bandwidth becomes so great that huge initial downloads are acceptable or, better still, when users no longer install software but instead run it from the giant centralized servers of Application Service Providers. But in the near term, we expect that learning engineers will continue to struggle to (1) produce resources in time for a release date and (2) fit all useful resources into the installation package, whether CD-ROM or downloadable installer file. Supplementable programming systems can help mitigate these and other learning support issues: • Lengthy animated demonstrations, especially videos or direct screen recordings, can be made available to learners as optional materials that can be downloaded from the Internet. • Additional Internet resources, such as an extended reference guide, tips and tricks, and Frequently asked questions, can be added after the initial release. These items can be ones that were not quite ready for the release or were created in response to technical support queries. • A highly supplementable programming system can ease the process of acquiring and viewing static samples provided by the system’s designers or by a support network of other users eager to share what they have learned. Implementation Issues A well-designed supplementable infrastructure anticipates the need to plug in instructional modules at a later date, and as much as possible removes the onus from the user for downloading such modules from the Internet. In designing for supplements, one must first decide what types

14

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. of resources will be extensible: demonstrations, on-line help, puzzles, sample programs, etc. Then, one must work out the mechanics of how supplements will be added. Probably the simplest approach one can implement is to register all resources as file types openable from the programming system. Users can view supplements by double-clicking a file or initiating an Open command within the programming system and navigating to the item. Indeed, this is an option for users who wish to see new Creator tutorials and sample programs, as well as ToonTalk demonstrations and puzzles. However, our user tests have given rise to concerns about children’s ability to successfully download such items from the Web, save them on their hard disk, and find them again to open them later. There are at least three ways to simplify the way users download supplements. Probably the most common route is to register the programmable system as a “helper application” in the user’s browser. This is the approach taken by the Mindstorms RIS. When users click on a sample program with the MIME type “application/x-legomindstorm,” the browser can automatically launch the RCX Code environment with this sample program immediately available for viewing. A similar technique is used to support the exchange of not whole programs but individual “behaviors” in AgentSheets [15], a visual programming system that features rule-driven “agents” comparable to Creator’s characters. Unfortunately, ensuring that the helper application is properly registered can be a technical support nightmare. As a Java-based application, Creator takes a different approach. Sample programs are stored on Web pages, and a tiny Java applet links the downloaded world with the main Creator application already installed on the user’s hard disk. ToonTalk employs a third technique, which makes simple use of the system clipboard and the fact that all ToonTalk objects can be serialized into a textual encoding. To try out an exemplary object, users select the serialized encoding in a Web page, copy the text to the clipboard, and then paste the encoding into ToonTalk. ToonTalk recognizes the text as a serialization and automatically makes it appear as a new ToonTalk object. The above implementations facilitate supplemental learning supports that run locally on the user’s machine. Obviously, there are many other kinds of supplements that the user can view on the Web through a browser. Indeed, Mindstorms, Creator, and ToonTalk all have Web sites with extensive collections of learning resources and access to communities of fellow learners. Allowing the learner to take advantage of these resources effectively is yet another implementation challenge. The first problem is enabling both the programming system and a Web site to be viewed simultaneously. Unless implementers are so ambitious as to build a Web browser into their environment, this means the programming system ought to be able to run in a movable window that can share the desktop with a browser program. The second problem has more to do with interaction design: to be as supportive as possible, the Web site ought to have a familiar look and feel that complements the programming system. Our philosophy is that users have enough to learn about programming that they should not have to worry about navigating a foreign Web space. Concerns about consistency between the programming system and Web supports played a key role in the design of the Mindstorms Programmopedia. The Programmopedia is a set of How To’s for programming the RCX microcomputer. We knew when we designed How To entries for the CD-ROM that we were likely to want to supplement these later through the Web. This possibility led us to deliberately choose a simple Web-like interface for the CD-ROM version to avoid disorienting and perhaps disappointing users when they visited the Web version later. Figure 7 shows both the final CD-ROM version and the Web version (which contains numerous tips, tricks, and lessons not available on the CD).

15

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. Risks in Ignoring Supplementability Without facilities for supplementing learning supports, producers of programming systems may be forced to have all learning resources ready in advance of a release date. This is obviously an undesirable situation and may well lead to compromises in the breadth, depth, and overall quality of the resources generated. Furthermore, requiring that learning resources fit on the original release media may eliminate resource delivery options, such as the inclusion of digital video tutorials. In reality, of course, all programming systems can be supplemented as long as a learner is willing and able to download new files from the Internet. However, without facilities for hiding the complexities of the download process, whole populations of novice users may find this too difficult to accomplish. The result could be that those who have the most to gain from supplemental resources are the least likely to be able to take advantage of them.

(a)

(b) Figure 7. LEGO Mindstorms’ CD-ROM (a) and Web-based (b) “Programmopedia.” The Web version provides a familiar place for learners to find supplemental information. Constrainability A constrained user interface can prevent errors and allow users to gain experience and confidence by limiting them to a subset of the system’s functions. We use the term constrainability to describe the degree to which a system can adjust its feature set for particular periods of use. Modern search engines, such as Google™, demonstrate constrainability in the

16

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. way they initially limit users to a simple keyword search. The systems eliminate these constraints when users specifically request an “advanced” search. Constrainability can play an important role in the learnability of a system. For example, Catrambone and Carroll found that word processing users learned basic editing functions more quickly when their actions were constrained through a “training wheels” interface [16]. Interestingly, even after the training wheels were eliminated, users also seemed to perform advanced tasks with more confidence. Constrainability in programming systems can be leveraged by learning supports to keep novice programmers on a pedagogically productive path. We have found in user testing that it is all too easy for learners to misinterpret instructions and stray from a programming language tutorial or other sequenced resource. It is also easy for beginning users to inadvertently start using advanced, complex expressions while exploring on their own. (Ironically, this may be a particular problem for visual programming systems that make it too easy to quickly compose programs through drag and drop.) In our experience, constrainability is important to limit the surprises and distractions encountered by users so that they can focus on learning: • Constrainability makes exploration a practical kind of learning support, since users can start a programming session in a particular state and then safely be left to experiment. The session could be open-ended or be goal-directed, with a particular problem or puzzle to be solved. In the latter case, constrainability helps make the evaluation of a user’s solution more tractable. • Guides and coaches can manipulate constraints according to a user model, so as to scaffold the learning process. As a learner becomes more sophisticated, a guide can raise the bar by enabling new features. Implementation Issues Constraining a user’s actions is a challenging implementation problem and is tightly connected with other facilities, such as annotatability, scriptability, and monitorability. For example, a learning engineer may want to allow only particular actions until the user has reached a certain state, at which point more features are enabled. Monitorability can enable the system to detect that state, and scriptability can be used to put the user in the new state. The three primary challenges unique to constrainability are (1) disabling inappropriate features, (2) ensuring that the learner reaches a particular stage in a educational problem or puzzle, and (3) allowing the learning engineer to specify when the first and second should happen. For ToonTalk, these constrainability challenges are greatly eased by the fact that all tools are themselves programmable objects that can be customized, duplicated, or deleted. Such metaprogramming allows the learning engineer to construct and save arbitrarily constrained “worlds” for particular learning activities. For example, in some ToonTalk puzzles, certain objects or characters are simply not around, and in other puzzles, only crippled versions of some objects are available, such as a ToonTalk magic wand with only a limited amount of “magic” for copying things. Further constraints are facilitated through global switches that enable the learning engineer to turn off primitive operations such as number typing, text typing, function keys, and the ability to flip over objects to reprogram them. For example, number typing is disabled in ToonTalk’s second puzzle, which asks learners to create the number 4 from 2 and 2. Without number typing, the learner cannot directly create the 4 by using the keyboard, but instead must experiment to discover how addition is expressed in ToonTalk. Since Creator does not support the kind of metaprogramming offered by ToonTalk, custom constrainability facilities had to be added incrementally as user tests revealed what was important to keep learners on track. For example, initial tests of Creator’s tutorial showed that users (mainly 10-year-olds) would sometimes experiment by clicking on all sorts of buttons, not just following the instructions. So, Stagecast added the ability to gray out and disable all buttons that were not part of the current activity. To ensure that users would click on the correct button, Stagecast was compelled to add the capability of flashing buttons in the interface. 17

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE.

Risks in Ignoring Constrainability Perhaps the most significant problem in ignoring constrainability is that the less a programming system can be constrained, the more likely its interactive learning supports are to require explicit instructions and/or error handling. For example, special instructions may be needed in a tutorial asking users to ignore irrelevant or distracting parts of the interface. Should users attempt an inappropriate action in spite of the instructions, the support will probably need to trap this condition and possibly display an error message. Other Facilities The above facilities represent what we believe to be the core capabilities that a programming system can provide to facilitate the addition of learning supports. Below, we outline several other facilities that can also be important. Some may not directly facilitate learning supports but can make it easier for learners to make sense of their programming system. Dual-Purpose Documentability. Writing reference manuals for a language can be tedious, as can properly documenting the code that implements that language. If the structure of the code nicely parallels the structure of the target programming language, it is possible to combine these efforts somewhat. Dual-purpose documentability refers to the ability to use code analyzers, such as Sun Microsystems’ JavaDoc, to extract help text automatically from the implementation so that it can be used for end-user documentation. Referencability. Particular challenges arise in creating learning supports that are intended to be read aloud—either by actors doing voice-overs (as was done for the Mindstorms RCX Code Training Center) or by learners themselves (e.g., in reading a printed manual). One issue is referencing specific programming language structures so that a learning support can define them or describe how they can be combined in interesting ways. For visual languages, such references can become awkward if one employs abstract shapes or colors to differentiate programming elements. Imagine a voice-over saying, “Now double-click the bluish command shaped like an ear”! By considering referencability at the outset of programming system design, these problems can be avoided. Indeed, referencability was a factor in the decision to abandon a purely iconic display for the Scout microcomputer of the Mindstorms Robotics Discovery Kit. Short names for programming elements, such as “Seek” and “Loop,” were chosen instead, which allowed them to be integrated more easily into the text of the printed manual and in the spoken vocabulary of the learner. Even if icons remain an integral part of one’s language design, an obvious way to increase referencability is always to label the icons on the screen. Governability. Programs often run too fast for learners to appreciate what is going on and to debug incorrect programs. A governable programming system can run at different speeds, allowing learners to slow down a running program and then speed it up again. ToonTalk’s governability allows it to enter a slow-motion mode while users work directly with robots, but when they “walk away” in the ToonTalk virtual world, execution accelerates. ToonTalk also has a remote control for the system speed, which can be set by the programmer or by user programs. Similarly, Creator provides a ubiquitous speed controller at the bottom of the screen where learners can choose from slow, medium, fast, and “as fast as possible,” as shown in Figure 8.

Figure 8. Part of Creator’s control panel. The buttons on the right let the user choose from four different execution speeds.

18

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. Immediacy. Immediacy is the ability of a programming system to support the on-the-fly editing of a running program. For textual programming languages, immediacy often takes the form of a “message box” or “listener” window where commands can be given at any time. For visual programming systems, including Creator and ToonTalk, immediacy can mean that users can arbitrarily pause a program, change the instructions or rules for certain agents in the system, and then continue running the program under the altered configuration. Obviously, immediacy can be useful for capricious programmers who change their mind frequently, but, more importantly from a pedagogical perspective, immediacy enables users to employ a playful “guess-and-check” strategy when learning to program. Undoability. The ability to “run time backward” is important to give novices a sense of safety. They can experiment more freely if they can back out of problems. Undoability is a facility that received much attention in the design of Creator. It is not only possible to undo a recent user action in Creator, but also to back up a paused simulation to review the steps taken by the system.

Conclusions It is probably uncontroversial that the quality of integrating learning supports can determine the success of programming systems. Perhaps less obvious is the degree to which the infrastructure of a programming system can determine the practicality of creating and maintaining these learning supports. We believe that the facilities examined in this paper can make the difference between brittle, post hoc educational aids and rich, extensible learning opportunities. It is important to point out that the ideas in this paper are not limited to visual programming systems, but could also have application to textual programming environments and any other kind of nontrivial, open-ended software. Indeed, nearly every learning support we describe has been used, if not pioneered, in video game designs. For example, arcade games have employed animated demonstrations almost since their invention in 1972. Constrained exploration is common in many games, especially puzzle and adventure games. Guides and coaches are very common in adventure games. Sophisticated puzzle tutorials are common in classic games such as Lemmings and The Incredible Machine. Finally, it should be said that implementing the learning supports we identify does not guarantee a highly learnable programming system. User testing is important to evaluate and tune the effectiveness of the learning resources. Such testing has been central to the development of supports for Mindstorms, Creator, ToonTalk, and Chart ’n’ Art. Nonetheless, we hope the lessons we have learned can help future developers better appreciate the learnability consequences of programming system designs, allowing them to avoid the most common pitfalls and concentrate more fully on language innovations.

Acknowledgments We would like to thank Mike Dooley and Cathy Fett at LEGO for their high standards for the Mindstorms learning supports, Lauriann Serra and others at Interactive Factory for the chance to learn about CD-ROM production realities, and Lucy Omo, Ron Fried, and Chris Hartman for design and editing assistance. Mike Eisenberg deserves credit for many of the ideas behind Chart ’n’ Art. We also want to thank Jeffrey Tribble and Nik Heger for implementing the Creator tutorial. Our appreciation to Henry Lieberman for his comments on a draft of this article.

References 1. T. R. G. Green, M. Petre, & R. K. E. Bellamy (1991) Comprehensibility of visual and textual programs: A test of superlativisms against the ‘match-mismatch’ conjecture. In: Proc. Empirical Studies of Programmers: Fourth Workshop (J. Koenemann-Belliveau, T. G. Moher, & S. P. Robertson, eds.) Ablex Publishing, Stamford, CT, pp. 121-146. 19

TO A PPEAR WITH REVISIONS IN J. OF V ISUAL LANGUAGES AND C OMPUTING. PLEASE DO NOT CITE. 2. K. Kahn (1996) ToonTalk—An animated programming environment for children. Journal of Visual Languages and Computing, 7, 197-217. 3. K. Kahn (1999) ToonTalk Home Page Animated Programs. 4. D. C. Smith & A. Cypher (1998) Making programming easier for children. In: The Design of Children’s Technology (A. Druin, ed.) Morgan Kaufmann, San Francisco, pp. 201-222. 5. A. Cypher & D. C. Smith (1995) KidSim: End user programming of simulations. In: Proc. CHI 95, ACM Press, New York, pp. 27-34. 6. C. J. DiGiano (1996) Self-disclosing design tools: An incremental approach toward end-user programming. University of Colorado at Boulder, Technical Report CU-CS-822-96. 7. C. DiGiano & M. Eisenberg (1995) Self-disclosing design tools: a gentle introduction to enduser programming. In: Proc. DIS '95 Symposium on Designing Interaction Systems (G. Olson & S. Schuon, eds.) ACM Press, New York, pp. 189-197. 8. T. Selker (1994) COACH: a teaching agent that learns. Communications of the ACM, 37, 9299. 9. K. Kahn (1998) Helping children to learn hard things: Computer programming with familiar objects and actions. In: The Design of Children’s Technology (A. Druin, ed.) Morgan Kaufmann, San Francisco, pp. 223-241. 10. R. Barrett, P. P. Maglio, & D. C. Kellem (1997) How to personalize the Web. In: Proc. CHI 97, ACM Press, New York, pp. 75 - 82. 11. B. A. Myers (1998) Scripting graphical applications by demonstration. In: Proc. CHI 98, ACM Press, New York, pp. 534-541. 12. C. DiGiano & M. Eisenberg (1996) Designing pedagogical screen savers. In: Proc. CHI '96 Conference Companion (J. D. Mackinlay, ed.) ACM Press, New York, pp. 185-186. 13. H. Lieberman (1998) Integrating user interface agents with conventional applications. In: Proc. International Conference on Intelligent User Interfaces 98, ACM Press, New York, pp. 3946. 14. F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad, & M. Stal (1999) Pattern-Oriented Software Architecture: A system of patterns. John Wiley & Sons, Chichester. 15. A. Repenning, M. Rausch, J. Phillips, & A. Ioannidou (1998) Using agents as a currency of exchange between end-users. In: Proc. WebNET 98 World Conference of the WW, Internet, and Intranet, Association for the Advancement of Computing in Education, Charlottesville, VA, pp. 762-767. 16. R. Catrambone & J. M. Carroll (1987) Learning a word processing system with training wheels and guided exploration. In: Proc. CHI-GI 87, ACM Press, New York, pp. 169-173.

20