software sketchifying – bringing innovation into software development

SOFTWARE SKETCHIFYING – BRINGING INNOVATION INTO SOFTWARE DEVELOPMENT Željko Obrenović, Backbase, Amsterdam Email: [email protected], URL: http://obren.info/ Abstract: Inspired by general ideas about how the automotive industry brings innovation into lean manufacturing, we propose introducing into software product development an activity called software sketchifying. With this activity we want to stimulate and support software stakeholders to spend more time generating and considering alternative ideas before making a decision to proceed with engineering. To support and explore our view, we developed Sketchlet, a flexible toolset that empowers non-engineers to work with emerging software and hardware technologies, explore their possibilities, and create working examples – called sketchlets – that incorporate these emerging technologies. The main output of software sketchifying is a set of realistic proposals for new software solutions aimed at giving the engineering process a clear focus and goal. We also present some lessons learned in applying our ideas in several projects.

1 Introduction Software productivity is one of the main themes in software development. New initiatives, such as lean software development (see the Special Issues of Lean Software Development) provide a new view on software productivity by introducing principles such as eliminating waste, i.e. avoiding doing anything that does not directly support the customer [Poppendieck03]. Such initiatives are tightly coupled with more investment in understanding of our customers’ needs. After all, eliminating waste, when our requirements may radically change, may be a wasted effort itself. In this aspect, initiatives such as lean software development seem to be a nice fit with iterative and incremental software development methods and agile software development frameworks, as such frameworks promote continuous customer involvement [Coplie10]. However, iterative and incremental approaches have their limitations when we are working on truly innovative projects and with emerging technologies. In such projects there are many unknowns, stakeholders often do not have clear expectations about new technologies and understanding of user needs is more challenging. Inspired by general ideas about how the automotive industry brings innovation into manufacturing we produce introducing into software product development an activity called software sketchifying. With this activity we want to stimulate and support software stakeholders to spend more time generating and considering alternative ideas before making a decision to proceed with engineering. Our view on software sketchifying combines general ideas of sketching [Buxton07] and creativity support tools [Shneiderman07] with several existing software engineering approaches. To support and explore our view, we developed Sketchlet ( http://sketchlet.sourceforge.net /), a flexible toolset that empowers engineers and non-engineers to work with emerging software and hardware technologies, explore their possibilities, and create working examples – called sketchlets – that incorporate these emerging technologies. The main innovation of our approach is in promoting a new form of communication between software engineers and non-engineers in early stages of software development: Sketchlet facilitates early involvement of developers in an occasional ad-hoc basis, to extend the functionality of Sketchlet and bring into the exploration space of non-engineers new pieces of emerging technologies relevant for the ongoing exploration activity. H

H

In next section, we present our motivation, explaining that, similar to lean software development, our approach is inspired by some of the lessons from the automotive industry. After that we present the main idea of our approach, followed by the description of implementation of this idea in Sketchlet. We also discuss some of our experiences in using Sketchlet in several projects. Sidebar 1 provides an example scenario, based on a realistic example from our projects, which illustrates our vision. Online appendices A and B provide more technical details about our tools, and how they can be used to implement the example scenario.

2 Motivation: Some Lessons from the Automotive Industry Many see Henry Ford as one of the pioneers of lean manufacturing principles. The assembly-line production model of Ford’s Model T was inspiration for changes in the automotive industry, and there were numerous attempts to apply similar ideas in software development (e.g. see the chapter “Will the Real Henry Ford of Software Please Stand Up” in [Glass06]). While the assembly-line philosophy has been influential, less known story about Henry Ford is his approach to innovation, and the process that preceded the production of Model T. Between 1892 and the formation of his company in 1903, most of which time he worked for the Edison Illuminating Company, Ford built about 25 cars. The Ford Motor Company then built and sold eight models in the five years preceding the Model T—Models A, B, C, F, K, N, R, and S—and tested prototypes labeled with the 11 missing letters. Ford summed up this [Grudin08]: "I do not believe in starting to make until I have discovered the best possible thing. This, of course, does not mean that a product should never be changed, but I think that it will be found more economical in the end not even to try to produce an article until you have fully satisfied yourself that utility, design, and material are the best. If your researches do not give you that confidence, then keep right on searching until you find confidence.... I spent twelve years before I had a Model T that suited me. We did not attempt to go into real production until we had a real product.” While today’s automotive industry has changed significantly from Ford’s initial success, some of the Ford’s philosophy behind innovation still remains. For example, the “Nemawashi” principle of the Toyota Way states that decisions should be implemented rapidly, but that making decisions should be slow, by consensus, and after considering all options [Liker04]. Buxton, who also studied innovation in the automotive industry, noted that the production of a new car has explicit design phase that proceeds an engineering phase. A design phase starts with a broad exploration which culminates in the construction of a full-size clay model, which cost over a quarter of a million dollars [Buxton07, page 73]. Only after bringing the new concept to the high level of fidelity in terms of its form, business plan, and engineering plan will a project enter the “green-light” stage. Then there is still typically a year of engineering before the project can go into production.

3 The Problem Issues from the previous section are not new to software industry. However, we have richer history of well engineering wrong products: badly defined system requirements and poor communication with users remain among top factors that influence failure of software projects [Charette05]. Fred Brooks also noted that the hardest single part of building a software system is deciding precisely what to build [Brooks87]. Brooks proposed rapid prototyping of systems and iterative specification of requirements as a way to solve this problem. Many of existing software engineering methodologies, including the Rational Unified Process, Extreme Programming and other agile software development frameworks, follow such iterative and incremental approach. However, when it comes to true innovation, rapid prototyping and iterative and incremental development have their limitations. While prototyping can enable us to cheaply represent and test our ideas, and iterative and incremental development can help us to further develop our ideas based on the frequent user feedback, these approaches do not directly support us to come up with new ideas for a software product, nor do they encourage us to consider alternatives. Buxton went further in his critique of innovation capacity of iterative and incremental software development, stating that there is nothing that compares software product design to the development of new automobile [Buxton07, page 73]. Buxton argued that, at least in innovative software projects, there needs to be a distinct design phase followed by clear green-light process before proceeding to software product engineering. He argued that design and engineering are different activities, employ different processes, and people suited for one are typically not suited for the other.

4 Software Sketchifying Building on the Buxton’s suggestion, we propose introducing into software product development an activity called software sketchifying, as a complement to prototyping and engineering. Our intention is to stimulate software stakeholders to spend more time generating alternative ideas before they make a decision to proceed with software engineering. Sidebar 1 provides an illustration of our vision about software sketchifying.

Sidebar 1: Example Scenario To illustrate our vision about software sketchifying, we introduce an example scenario with Mirko as our main protagonist. Mirko is an interaction designer at a company X that builds software for new generations of cars with advanced sensing and display technologies. Mirko has recently joined the company to explore ideas for software applications that exploit novel opportunities, such as using data from a car radar, GPS sensors, and links to web services. Mirko’s first task is to explore the following two applications: (a) a system for warning about the proximity of other cars, and (b) a system for presentation of news in idle situations, such as when waiting for a traffic light or on a parking. Mirko is not a programmer, and not familiar with all technical possibilities of modern cars, but he uses a design environment through which he can access and explore software services and components related to his task without serious programming. To understand what is possible, Mirko first talks with several engineers in his company. They advise him to start with exploration of a car simulator, which provides realistic but safe environment to learn about new possibilities of cars. One of the engineers also writes a small adapter that connects the car simulator logger to Mirko’s design tool. After this adaptation, Mirko’s design environment can be easily connected to a company car simulator, enabling him to, through a simple spreadsheet-like interface, immediately use the data from the simulator, including the data about the speed of the car, the distance from the car in front of the driver, or status of indicators. To explore possibilities and try out some ideas, Mirko starts a design environment on his laptop and connects it to the simulator. After getting familiar with the possibilities of the simulator, Marko turns to his laptop to create a few “sketchlets”, a simple interactive pieces of software. In order to explore the options for implementation of a car proximity warning system, Mirko first considers three groups of options: graphical presentation, audio presentation, and haptic (vibration) presentation. For graphical presentation, he uses an editor in his design environment, and creates several simple drawings. He then opens the properties panel, and connects the variables from the car simulator to the graphical properties of drawn regions. For example, he creates a “sketch” where the transparency of an image dynamically changes as a function of a distance from the car in front of the driver. He then experiments with other graphical properties, such as image size, position or orientation of the image. He now goes back to the simulator, and tries each alternative. He also invites few of his colleagues to try out and comment on his ideas. After exploring graphical options, he proceeds to create audio “sketchlets”. He first tries to use a MIDI generator service and connects data coming from the sensor to MIDI note parameters, such as pitch, or tone duration. He also experiments with a text-to-speech service, generating speech based on the conditions derived from car data. Finally, he explores using an MP3 player, with prerecorded sounds. He goes back to the simulator, and tries these alternatives. Mirko also wants to try out usage of vibration modality to present navigation information, which the simulator currently does not support. Mirko decides to use a simple trick. He starts an application on his mobile phone, which enables his design environment to use and control the phone’s resources, including a vibrator. Using gaffer tape he then fixes the mobile phone to the steering wheel, and creates several “sketchlets”, where distance from the car in front is mapped to vibration patterns. Marko is aware that this is not a very elegant solution, but it enables him to explore basic opportunities that this modality offers, with available resources and with little work. Mirko also plays with some other options related to his second application, presenting the news. He starts a Google news service in his design environment, and creates a simple page, where an HTML output of the Google news service is presented. He then creates a condition for visibility of a page so that the news are visible, overlaid on a part of the windshield, only when the speed of the car is zero and auto is not in gear. He also experiments with speech services, enabling a user to set a news search query by speech. After finishing his work in the “lab”, Mirko decides to collect some real-world experiences, and to try some of his more promising “sketchlets”, in a real car. With help of engineers who are working on testing cars, Mirko gets an extension of his design environment that connects it to an on-board diagnostic (OBD) system on a car using a Bluetooth connection. With this addition, Mirko creates a simple setting using his smartphone as a presentation device, positioned under a windshield. He connects the smartphone to his laptop, which uses a simple remote desktop client to capture a part of a screen from his laptop. On the laptop, Mirko is running the “sketchlets” that he had created in the laboratory, but now connected to the car’s on-board diagnostic system. He asks a colleague to drive a car, while Mirko observes a situation, and video records a whole session for later analyses. During the process, Mirko constantly interacts with other stakeholders and he regularly presents his findings and lets clients and colleagues try out some of his “sketchlets”. In this way, Mirko is helping development of new products by providing realistic and tested ideas before and outside the main development activity.

Our work includes changing the approach how we develop software, and development of tools that can support our approach.

4.1 Software Sketchifying Approach One of the key characteristics of our approach is postponing the main development activity for the benefit of free exploration, following one of the main principles of creativity: to generate a good idea, you need to generate a number of ideas and dispose the bad ones [Glass06; Buxton07]. Another key characteristic of our approach is stimulating involvement of non-engineers in early stages of development. Such users often have relevant expertise that is important for understanding our customers and identifying their needs. More specifically, our example scenario (Sidebar 1) illustrates several points that we wanted to address with our approach:  

 

 



The main activity of our protagonist is exploration, learning about a problem and potential solutions, and answering a question about what to build. Such explorative activity is heuristic, creative, and based on trials and errors, rather than incremental and iterative. Our protagonist generated a number of ideas, most of which will be rejected. However, through this process, he learns important lessons, and generates novel ideas. These lessons and ideas are the main outcome of the activity. Exploration activity is not accidental, but disciplined and systematic. Our protagonist consciously decides to consider many ideas, rather than to polish some of them. The exploration is holistic, enabling designers to reflect on relations between user issues, software and hardware possibilities, and overall dynamics of human-computer interaction. Mirko’s ideas are influenced not only by software, but also by human factors, and problems related to car mechanics and equipment. The exploration enables early user involvement, as our protagonist creates simple but functional pieces of software that users could immediately try out. Working with adaptation of realistic pieces of software, such as car simulator, car, Web services, enables a designer to learn about possibilities and limitations of software technologies, and create conceptual proposals that are more realistic. While a designer cannot program and extends its environment, a designer is a part of a broader community of people working on a same project/company. These people can help a designer to learn and extend its exploration space, provided that such help would not require significant time. Such interaction empowers non-engineers to learn about possibilities and limitation of emerging technologies, and to explore and test their ideas and proposals without additional help of developers.

4.2 Software Sketchifying Tools To support and explore our approach, we developed Sketchlet (http://sketchlet.sourceforge.net), a Java-based software tool that combines elements from traditional sketching, software hacking, opportunistic software development, and end-user development. Sketchlet builds on the results of the Sketchify project (http://sketchify.sourceforge.net), which explored possibilities to improve early design stages and education of interaction designers. Sketchlet has two main roles:  It enables designers to create a number of simple pieces of software – sketchlets – as a way to quickly and cheaply explore software and hardware technologies and their potential applications, and  It supports involvement of software engineers in short ad-hoc sessions to bring to designers realistic pieces of technologies that may be useful for design exploration. Sketchlet enables designers to, through direct interaction, quick explore a number of software and hardware technologies, using simple and intuitive user interface. For examples, with a spreadsheet-like interface, a designer may connect a Google search web service to data coming from a car simulator and local speech-recognizer service. In this way, similar to Buxton’s proposal, we facilitate involvement of non-engineers in early stages of software product development.

One of the main innovations of our approach is enabling quick extensions of Sketchlet exploration space with links to relevant software and hardware technologies (see Appendix A for examples). To simplify such integration Sketchlet combines ideas from opportunistic software development with techniques used by hacking and mashup communities [Hartmann08, Obrenovic08]. Sketchlet extensions, as illustrated in our example scenario, even though conduced in ad-hoc (and often quick and dirty) fashion, are crucial to enable researchers and interaction designers to create realistic proposals for new software products. Involvement of engineers can empower nonengineers to learn about possibilities and limitation of emerging technologies, and to explore and test their ideas and proposals. Full description of Sketchlet is out of scope this paper. Appendix A and B contain more details about how Sketchlet can be used to implement our examples scenario. Interested readers may also find some additional information behind our motivation in [Obrenovic11]. We also encourage reader to download and try the tool themselves. In section 6.3 we discuss our tool in relation to other tools.

5 First Results We have developed and applied the ideas about software sketchifying in three projects that featured collaboration between software engineers, interaction designers, and researchers. In these projects interaction designers and researchers were primarily responsible for creating and evaluating novel conceptual proposal and ideas. In the project Connect & Drive1, for example, Sketchlet was used by several researchers to explore options for development of software systems for cooperative adaptive cruise control systems in cars, based on Wi-Fi communication between vehicles and road infrastructure. Within the Persuasive Technology, Allocation of Control, and Social Values2 project, Sketchlet played a similar role helping researchers to investigate software products for development of persuasive technologies that encourage people to hand over control to intelligent automation of cars. Within the REPAR (REsolving the PARadox in user centered design) 3 project – Sketchlet was used as one of the tools for flexible prototyping in user centered design processes, allowing designers to create and evaluate (ill-defined) product concepts in an early stage of product development. Although our work is still in early stages to draw definitive conclusions, we were able to observe several positive effects of our approach and tools on the design process. Firstly, we made possible broader and more constructive involvement of non-engineers, such as interaction designers, psychologist, and students. In this way, companies could benefit from their expertise and knowledge early in design process. Our tools empowered non-engineers to easily explore emerging technologies relevant to companies and to independently create and test their ideas. We were also able to promote different collaboration between engineers and designers (nonengineers) in early stages of development. Prior to our involvement, most companies followed the approach where designers were responsible for creating a conceptual proposal which they would give developers for implementation with little interaction except to clarify their design. With Sketchlet, the interaction between designers and engineers could work in two ways, where, in the early stages of design, engineers provide designers with simplified software components and services that they might use later in implementation (Figure 1). Software services that engineers connect to Sketchlet, although simplified, resemble real components, and sketchlets expressed in terms of these services are closer to the implementation platform that the engineers use. With this change we addressed one of the problems that companies experienced when designers and engineers needed to work together, namely that the engineers perceived the ideas of designers as being unrealistic, too distanced from technology, and not precise enough to be useful. Through the exploration of these services, designers can develop more realistic expectations about the

1 2 3

http://www.htas.nl/index.php?pid=106 http://hti.ieis.tue.nl/node/3344 http://www.repar-project.com/

possibilities and limitations of technologies, and incorporate this understanding into design proposals.

Figure. 1 With Sketchlet, the interaction between designers and engineers could work in two ways, where, in the early stages of design, engineers could provide designers with simplified software components and services that they might use later in implementation.

Lastly, we were also able to influence the mindset of companies toward more and broader explorations in early stages of software design. Sketchlet helped us to illustrate the potential of such exploration, and to inspire the companies to think how other tools could be used in a similar explorative way.

6 Discussion Here we discuss as well as the relation of software sketchifying, prototyping, engineering and existing software tools.

6.1 Main Benefits of Software Sketchifying Software sketchifying may help us to better define requirements for our products, so that the engineering process that proceeds afterwards can have a clear focus and goal. Our approach promotes direct exploration of emerging technologies, and creating working examples of simple pieces of software with these technologies as a way to identify potential problems and “provoke” reactions from users as early as possible. With our tools effects of design decisions could be experienced and tested before actual development starts. By focusing on exploration of possibilities and limitations of technologies in earliest stages of design, we may identify some of the problems of emerging technologies and related user issues before we have invested significant development effort. Discovery of such problems later in the design process may require a number of changes and additional efforts. This is particularly important in project with emerging technologies, where there are many unknowns, and where it is hard to predict how users may accept these technologies. Second, we promote constructive involvement of non-engineers in design process. In this way we may get help from experts from other fields relevant for our product (such as human psychology) and reduce the burden of developers. Moreover, as noted by Glass, users who understand the application problem to be solved are often more likely to produce innovation than computer technologists, who understand only the computing problem to be solved [Glass06, page 335]. While our approach requires occasional involvement of developers, such involvement is aimed to be conducted in short ad-hoc sessions, and is intended to empower non-engineers to explore further without help of developers.

6.2 Relation to Prototyping and Engineering Our view on software sketchifying complements existing prototyping and engineering approaches by its focus on free exploration and a trial-and-error approach, versus a more iterative and incremental approach of prototyping and engineering (Figure 2). We view software sketchifying as an activity that supports us in constructing a novel idea. It should be conducted as an explorative, trial-and-error activity, in which non-engineers could actively contribute. In this way we want to make closer to software design the practice of other engineering disciplines, where prior to main engineering activity there is a design phase where designers (usually non-engineers) are encouraged to freely explore ideas before consolidating a few of them for further development. Early in the design process, for instance, it would not be unusual for an industrial designer to generate 30 or more sketches a day, each possibly exploring a different concept [Buxton07, page 106]. Software sketchifying precedes prototyping, which tests, compares and further develops aspects of selected ideas. With a prototype in place, we can take an evolutionary approach to development. Prototyping should assess whether selected ideas are feasible, and should help us to make decision should we proceed with its engineering. In contrast to sketchifying, prototyping aims at making idea more detailed and concrete, rather than coming up with radically new ideas. Engineering turns the winning idea into a robust and usable product.

Figure. 2. (a) Idealized representation of classical relationship between sketchifying, prototyping and engineering.

6.3 Relation to Other Software Tools Our vision about software sketchifying, in principle, could be achieved with other tools. However, many of the current tools cannot fully support our vision as they are not optimized for free exploration and involvement of non-engineers. For example, to implement our example scenario we could use standard programming languages, such as Java, C#, C++, or interaction design oriented programming tools such as Flash and Processing. However, programming requires significant expertise, time and effort, and such an investment is simply too high for the intended purpose, which is generating new ideas and exploring possibilities. Existing low-fidelity prototyping environments provide ways to quickly create prototypes where inputs can be taken from external services or sensors [Rettig94, Lim08]. These environments may be an excellent choice for exploration of interactions in various domains. The problems that we were addressing, somehow cross these domains, and require a variety of sensory inputs and links to diverse software services, as well as addition of components specific to the companies we were

working with. In addition, such tools often require too much precise specification, partly because they are primarily developed for advanced prototyping rather than for free and broad exploration. Electronic sketching systems are another promising direction for design tools, enabling designers to create interactive systems with ease, using intuitive and natural pen gestures [Landay01]. The drawback of sketch-based systems, from the viewpoint of our example scenario, is that they are usually very specialized and domain specific, and have been successfully used only in inherently graphical domains that have a stable and well-known set of primitives, such as 2D and 3D graphics, or web sites. Another alternative is to use simple freehand drawings, and techniques such as screen prototyping. Such techniques can facilitate exploration of graphical elements of solutions. However, with these techniques overall interaction, where sensing devices and the dynamics of user responses are also taken into account, can be described only in very abstract terms. Moreover, paper sketching does not allow users to explore the possibilities and limitations of emerging technologies. Through direct exploring technologies, we can get more concrete ideas about how they may be employed best. Sketchlet borrows ideas from existing solutions, trying to overcome some of their limitations. We also see our tool as a complement to existing tools, rather as their replacement. In several occasions Sketchlet has been used in combination with other tools. For example, some of our user employed Max MSP for signal processing and audio effects, using Sketchlet to connect to sensor devices and for visualization. Interested readers can find a more detailed description of some of the existing solutions in our previous work [Obrenovic11].

7 Conclusions Our initial experiences with applying software sketchifying are encouraging. One of the most important limitations of our approach is that requires significant changes of current development culture, in its emphasis in postponing start of development to benefit free exploration, more active involvement of non-engineers and end-users, and new forms of interaction between engineers and non-engineers prior to the main development activity. Such changes, in our experiences, are not easy to produce, and without them our tools are less effective and tend to be used in a limited way. In future work we plan to work on a more general approach towards building software services and components, where each service could have two sets of application programming interfaces (APIs), one engineering API, with full functionality, and one sketchifying API, representing a simplified and limited sample of the full functionality. We also plan to address collaboration, as in current implementation our tools are primarily aimed for individual usage, and are of limited value in collaborative design sessions.

8 References 1. 2.

Brooks F. (1987), No Silver Bullet - Essence and Accidents of Software Engineering, IEEE Computer 20(4), pp. 10-19. Buxton B. (2007), Sketching User Experiences: Getting the Design Right and the Right Design, Morgan Kaufmann (March 30, 2007). 3. Charette R.N. (2005), Why software fails, IEEE Spectrum 42(9), 42-49. 4. Coplie J., Bjørnvig G., (2010), Lean Architecture: for Agile Software Development, Wiley. 5. Glass R.L. (2006), Software Creativity 2.0, developer.* Books. 6. Grudin J., (2008), Travel back in time: design methods of two billionaire industrialists, ACM interactions 15, 3 (May 2008), 30-33. 7. Hartmann B., Doorley S., Klemmer S. R., Hacking, Mashing, Gluing (2009): Understanding Opportunistic Design, IEEE Pervasive Computing, vol. 7, no. 3, pp. 46-54 8. Landay J.A and Myers B.A., (2001), Sketching Interfaces: Toward More Human Interface Design. Computer 34, 3, 5664. 9. Liker, J. (2004). The Toyota Way: 14 Management Principles from the World's Greatest Manufacturer. McGraw-Hill. 10. Lim Y.K, Stolterman E., and Tenenberg J. (2008), The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Trans. Comput.-Hum. Interact. 15, 2, Article 7.

11. Obrenovic Z., Gaševic D., Eliëns A. (2008), Stimulating Creativity through Opportunistic Software Development, IEEE Software 25(6), 64-70. 12. Obrenovic Z., Martens J.B. (2011), Sketching Interactive Systems with Sketchify, ACM Trans. on Computer Human Interaction, 18 (1), Article 4. 13. Poppendieck M. and Poppendieck T. (2003) Lean Software Development, Addison Wesley. 14. Rettig M. (1994), Prototyping for tiny fingers. Commun. ACM 37, 4, 21-27. 15. Shneiderman, B. (2007), Creativity support tools: accelerating discovery and innovation. Commun. ACM 50, 12, 20-32.

Online Supplementary Material for

Software Sketchifying – Bringing Innovation into Lean Software Development ——————————  ——————————

Željko Obrenović

Supplement Appendix A Appendix B

Title Some of Simplified Software Services in Sketchlet Implementation of the Example Scenario in Sketchlet

2

Appendix A: Some of Simplified Software Services in Sketchlet ——————————  ——————————

A

INTRODUCTION

Sketchlet Designer is designed with the goal to be easily extended. In this aspect Sketchlet Designer differs from other software design tools, where extensions are usually achieved through standardized APIs and plugins. Sketchlet Designer offers a developer a range of diverse APIs with similar functionality, so that a developer can select an API most suited for his current knowledge and skills. With the requirements of software sketchifying in mind, a key challenge is how to connect software to Sketchlet Designer without spending a considerable amount of time. We followed two approaches:  Borrowing ideas from opportunistic software development, and 

Using simple software hacking techniques.

Opportunistic Software System Development (OSSD) promotes the idea of melding together software components and systems that were not designed to work together [Ncube08]. Using some of these ideas, we developed a toolkit for opportunistic software composition, based on simplification and unification of interfaces toward diverse software components and services. The toolkit is based on our previous work and ideas in the domain of service-oriented computing, providing alternative ways to connect services to our environment, including very simple low-level TCP and UDP protocols, as well as SOAP, XML-RPC, OSC, and HTTP Get/Post protocols (for details see [Obrenovic08]). Using these tools, in our projects, engineers were able to bring into Sketchlet Designer dozens of diverse software components and services. One of the main advantages of service-oriented interfaces is that they enable integration of software components and services written in different programming languages. To integrate pieces of software that do not have programmable API, and are not available as open-source components, Sketchlet Designer supports some of the techniques used by hacking and mashup communities [Hartmann08], such as screen scraping, screen capturing, and screen poking. Screen scraping is a technique based on parsing of rendered user interface to gather data. Sketchlet Designer can analyse and extract any part of a web page in HTML or XML (such as RSS) formats. Screen capturing dynamically catches a part of a screen, which enables bringing and manipulation of graphical outputs applications, including multimedia applications such as video players. In Sketchlet Designer, such captured parts of a screen can be manipulated and transformed as any other graphical objects. Screen poking is a technique that enables generating synthetic mouse and keyboard events computationally. In this appendix we list the several of the simplified software services that have been included within Sketchlet, which we will use in Appendix B to describe the implementation of the example scenario (introduced in the main text, in Section 2). Our software services are real but “trimmed down” input/output devices and services that have been extracted from diverse domains. Such services can bring components from various domains within reach of designers, and allow them to incorporate real functionality within a design already at an early stage, without requiring programming skills on the side of a designer. The obvious advantage of this is of course that the designer and potential end users can develop a better “feel” for the application being designed. Currently supported software services include camera-based face and motion detectors, text-tospeech engines and speech recognizers, VRPN devices (such as 3D trackers and buttons), MP3 and MIDI players, Web services (such as a Google spelling checker and search engine), a Wordnet definition service, and many others. For each software service, we describe the designer interface towards these services, which consists of one or more variables connected to the service. Source code examples for many of these services are available as part of our open-source package1 . F

CAR SIMULATOR For purposes of several of our projects, we have used with Sketchlet a Car Simulator available at the Department of Industrial Design of Eindhoven University of Technology2. The Simulator consists of realistic driving seat and driving controls, with multiple screens simulating the image visible through the car windows, and the windshield. The simu1 2

http://sketchify.sf.net/

http://w3.id.tue.nl/nl/department/labs_and_facilities/car_simulator/

3

lator uses GreenDino software3, a Windows based 3D rendering platform. While the simulator software is not open, it enables writing of custom log clients which receive simulator status data in real time through the IP network. We adapted one of the existing C++ log clients to, instead of writing data on the console, use this data to update Sketchlet variables using the Sketchlet UDP interface. *

      

Green Dino Car Simulator

Variables speed

Description Current speed

accel

Current acceleration

throttle_pedal

Status of the throttle pedal

brake_pedal

Status of the brake pedal

gear

Current gear

distance_front

Distance of the car in front (in seconds)

speed_limit

Current speed limit

WEB SERVICES Our platform allows connecting various Web services, including the Google SOAP search service and spelling checker 4, Google News search HTTP RSS Web Service5, Yahoo Babel fish HTTP HTML translation service6, the Flickr image uploader7, and the Alexa statistical Web Service8. Our platform allows for easy adaptation to other Web services. * Google News Search Web Service

Babelfish translation Web service

         

Variables google-news-query

Description User query

google-news-title-

Title of the news item (n = 1..5)

google-news-date-

Date of the news item (n = 1..5)

google-news-link-

Link to the news item (n = 1..5)

google-news-status

Status of search (‘ready’, ‘working’, ‘finished’)

babelfish-input-text

Text to be translated

babelfish-input-language

Language of user text (‘en’, ‘nl’, …)

babelfish-output-language

Language of translation (‘en’, ‘nl’, …)

babelfish-translation

Translated text

babelfish-status

Status of translation service

MUSIC SOFTWARE SERVICES We currently support two music output tools: an MP3 music player, based on the jlGUI open-source Java MP3 player9, and a MIDI player, implemented using standard Java audio libraries. * MP3 Player

MIDI Player

      

Variables mp3-song mp3-command

‘start’, ‘stop’, ‘pause’, ‘next’, ‘previous’, ‘eject’

mp3-volume

Sound intensity

mp3-equalizer

Main equalizer level

mp3-equalizer-

Equalizer level per channel

midi-note

A note to be played in format “ ”

midi-instrument

Music instrument being played

http://www.greendino.nl/producten.html http://code.google.com/apis/soapsearch/ 5 http://news.google.com/ 6 http://babelfish.yahoo.com/ 7 http://www.flickr.com/services/api/ 8 http://aws.amazon.com/ 9 http://www.javazoom.net/jlgui/jlgui.html 3 4

Description URL or path of audio file to be played

4

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, MANUSCRIPT ID

SPEECH SERVICES In order to enable sketching of speech-enabled interactive systems, Sketchlet has been connected to several open-source textto-speech (TTS) engines and speech recognizers, including an English speech recognizer based on Sphinx-410 the FreeTTS English TTS engine11, the NEXTENS Dutch TTS engine12, and the Mary TTS engine that currently supports English, German and Tibetan13. The table below illustrates the variables being used to connect to FreeTTS and to the Sphinix-4 systems. *

FreeTTS text-tospeech engine

 

Sphinix-4 recognizer



speech



Variables tts-input tts-status

Description Text to be pronounced. Status of the engine: ‘loading’, ‘ready’, ‘talking’

speechcommand sphinix4status

Text produced by the recognizer Status of the engine: ‘loading’, ‘ready’, ‘talking’

WII REMOTE Wii Remote14, which connects to a PC using a Bluetooth link, is a complex sensing platform. It can track infra-red sources, and contains three acceleration sensors, various buttons, a vibrator, a simple speaker and some status LED diodes. It can also be used to connect more devices, such as Wii Nunchuk, which contains a joystick and more buttons. Other related devices, such as Wii Fit, can also be used. Our Wii software service is based on the C# demo programs that come with WiimoteLib 15. 5B

Wii Remote

Wii Fit

*           

Variables wii-accel-x wii-accel-y wii-accel-z wii-ir--x wii-ir--y wii--state wii-led--status wii-vibrate-ms wii-fit--status wii-guitar--status wii-fit-balance-

http://cmusphinx.sourceforge.net/sphinx4/ http://freetts.sourceforge.net/ 12 http://nextens.uvt.nl/ 13 http://mary.dfki.de/ *  = read by the service (parameters of a service); 14 http://en.wikipedia.org/wiki/Wii_Remote 15 http://www.codeplex.com/WiimoteLib

Description X-axis acceleration (in Gs) Y-axis acceleration (in Gs) Z-axis acceleration (in Gs) X position of a tracked infrared object (0..1.0) Y position of a tracked infrared object (0..1.0) State of each Wii Remote button (‘True’ or ‘False”) Status of led diodes on Wii (‘on’ or ‘off’) Causes the Wii Remote to vibrate for a given time (in milliseconds) The status of LED diodes on Wii Fit The status of LED diodes on Wii Guitar Device Data from one of four balance board sensors

10 11

 = updated by the service (results of a service)

5

JAVAME MOBILE PHONE SOFTWARE SERVICES Many mobile phones support the Java Platform Micro Edition (Java ME16 ), which provides programmers with access to phones resources. We have developed several mobile phone software services using standard JavaME libraries, including a GPS sensor service, a photo capturer, an SMS messenger, and a controller for the mobile phone vibrator. 8B

H

JavaME Enabled Mobile Phone

*     

Variables gps-status gps-latitude gps-longitude gps-last-latitude gps-last-longitude

Description Status of GPS module Detected latitude Detected longitude Last detected latitude Last detected longitude

    

mobile-vibrate-command mobile-vibrate-status sms-number sms-message sms-status

Command (millis) to cause vibration Status of the vibrator SMS number Text of SMS message Status of SMS module

FACE EXPRESSIONS We also support a simple face expression animation module, based on The Expression Toolkit 17 – an open-source procedural facial animation system. In our adaptation, the face animation runs in a separate window, and through variables a designer can set basic and complex facial expressions, as well as define the “mood” of the character. Face Expressions

Direction   

Variables face-expression face-composite-expression face-mood

6B

16 17

http://java.sun.com/javame/technology/ http://expression.sourceforge.net/

Description ID of the face expression to be animated (1..41) ID of one of 12 complex face expressions to be animated Face mood during animation (‘happy’, ‘sad’, ‘angry’, ‘scared’, ‘tired’, ‘skeptical’)

6

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, MANUSCRIPT ID

Appendix B: Implementation of the Example Scenario in Sketchlet ——————————  ——————————

B.1 INTRODUCTION In order to facilitate rapid sketch-like manipulation of software ideas, we need tools that are efficient, yet simple and easy to use, and accessible to non-engineers. We have explored two sets of such techniques: an extension of a freehand drawing editor, and end-user programming tools. Sketchlet Designer includes a fully featured graphical editor, which enables designers to create and manipulate images as in any other graphical editor. Additionally, we support several extensions related to software sketchifying. The first extension is support for page transitions, where it is possible to define interaction by enabling a designer to go from one page to another, based on user mouse or keyboard events, as well as on updates of variables from simplified software services. Sketchlet Designer automatically generates a state transitions diagram, enabling a designer to quickly get an overview of interaction paths. The second extension is introduction of an active regions layer. An active region is a discrete part of a sketchlet which can itself contain images and capture user events. These transformations can be connected to software samples, enabling dynamic visualization and manipulation of data. Our website http://sketchlet.sourceforge.net/tutorials/ provides examples and videos that illustrate graphical possibilities of Sketchlet Designer. Sketchlet Designer uses a spreadsheet-based model of cells and formulas. Page properties, active region properties, and global properties (called variables), are cells that can be referenced from other cells through formulas. For example, we may define the transparency of the page as having the value “=[region1.position x] / 500”. This means that the transparency of the page will change as a user horizontally moves the region named “region1”. In addition to the build-in support for formulas and cells, in Sketchlet Designer external spreadsheets and scripting languages can also be used to quickly outline behavior of sketchlets. We currently support OpenOffice.org CALC spreadsheets, and several higher-level scripting languages including Javascript, Python, BeanShell, Groovy, Ruby, TCL, Sleep, Haskell, and Prolog. In the main text we introduced the example scenario to illustrate our vision about software Sketchifying. In this appendix, we describe a possible implementation of this scenario in Sketchlet. The implementation is described from the viewpoint of the Sketchlet user, in this case Mirko, an iteration designer working on design of interactive software systems for cars. Appendix A gives a description of simplified services that most of which we need in order to implement the example scenario. From a designer point of view, Sketchlet looks as a drawing editor with several extensions (Figure B.1). These extension enable a designer to start and stop simplified software services, define properties, and connect events and properties, as well as to work with other applications, using “hacking” techniques. In the rest of the appendix, we will illustrate the Sketchlet abilities on several concrete examples.

xxxx-xxxx/0x/$xx.00 © 200x IEEE

7

(a)

(b)

Fig. B.1. Sketchlet editor. (a) the editor where a user can draw and manipulate graphical objects, start software services, and define connections among events using formulas or other forms of end-user programming. (b) a window where a user can select Sketchlet services; Sketchlet hides the complexity of software services, providing a user a simple interface to start the service and work with it through variables.

Consistent with the scenario, we describe the following elements:  Defining the graphical presentation of a car distance,  Defining the audio presentation of a car distance,  Defining the haptic (vibration) presentation of a car distance,  Defining the news presentation screen based on state of the car.

B.2 DEFINING GRAPHICAL PRESENTATION OF CAR DISTANCE In the first part of the example scenario, Mirko creates several “sketches” to graphically present the distance of the car in front of the driver, based on the data received from the car radar, and test this “sketches” with the car simulator. In the scenario, Marko does the following:  Start the car simulator service and connect it to the Sketchlet in order to discover what data the simulator will send him.  Create several graphical objects to graphically represent  Create a few additional variables to represent parameters that Mirko wants to explore.  Connect variables to properties of graphical objects using simple formulas. In this way, graphical objects will be transformed based on the updates of variables, such as change of distance of the car updated by the simulator service.

B.2.1 Starting the Car Simulator To connect the simulator to his environment, Mirko simply select the service from the list of Sketchlet services (Figure B.1b). Sketchlet will starts the service in the background, offer a Mirko simple interface to start and stop this services, as well as several variables through which Mirko can receives data from the simulator (for details see Appendix B, Section B.1). By using the simulator, Mirko automatically updates the variables in the variables table. B.2.2 Creating Images and Regions To create graphical elements, Mirko can use drawing tools and active regions. With drawing tools, which replicate the functionality of the standard image editor, Mirko can create the background image and import other images, such as the photograph of a car. With active regions, Mirko to create graphical objects that contain and graphically transform images, as well as capture events, such as user mouse events, or overlap with other regions. Figure B.2 shows one possible graphical representation of the car distance created by drawing tools and with several active regions. The back-

8

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, MANUSCRIPT ID

ground image is used to represent the presentation frame and some decorative elements, while five active region dynamically present relevant data. Region 1, through color and size, represents the difference between current car position and target car distance. Region 2, through its position on the screen, represents the current position of the car, relative to its distance from the car in front of the driver. Regions 3, 4, and 5 textually represent the distances in time between the driver car and the car in front.

Fig. B.2. A Sketchlet page with a graphical representation of the car distance, consisting of a background image and five active region.

B.2.3 Creating additional variables and using formulas In order to create an interactive presentation, Mirko now has to connect the car simulator variables to the properties of the region. Before that, Mirko decides to create a few auxiliary variables that will simplify his work later (Table B.1). For some of these variables, Mirko uses the formulas, and in these formulas Mirko can reference other variables, and use simple mathematical and logic operations. TABLE B.1. AUXILIARY VARIABLES. distance

=simulator_distance

target_distance t_far t_normal t_close current_x target_x

1.5 1.8 1.3 0.8 =200 + 600 * distance / (t_far + 0.5) =200 + 600 * target_distance / (t_far + 0.5)

In seconds, taken from the car simulator. Represents the time needed to reach the nearest car with the current speed. The target distance time. The time when the distance is considered far enough The time when the distance is considered normal The time when the distance is considered far enough Graphical position of current distance Graphical position of target distance

Connecting variables to region properties To create a desired visualization effects, Mirko now opens the properties of the active regions, and, by using formulas, defines the connection between the properties and the variables (Table B.2). For example, in Region 1, Mirko defines the color property as the function of the comparison between the current distance of the car and target distance: if the distance is smaller than the target, the color is red, otherwise it is green. To graphically represent the distance, Mirko also changes the properties x1 and x2 of this region. In Region 2, which represent the current position of the car, Mirko only changes the “position x” properties. And in regions 3, 4 and 5, only text property is changes using simple text templates. Mirko can also define connections between region properties by dragging a variable from the variable table, dropping it on the region, and selecting the property of the region that he would like to control through the variable.

9

TABLE B.2 CONNECTING THE PROPERTIES OF THE REGIONS W ITH VARIABLES THROUGH FORMULAS. Region 1 properties fill color x1 x2 Region 2 position x Region 3 Text Region 4 Text Region 5 Text

=if(distance > target_distance, "green", "red") =min(current_x, target_x) =max(current_x, target_x) = current_x sec sec sec

As Mirko interact with the Simulator, the variables are updated, and the visualization will change. Figure B.3 shows how the visualization looks for two different values of the variables.

distance=2.0 seconds, target_distance=1.6 distance=1.2seconds, target_distance=1.6 Fig. B.3. The image visible to the user for two states defined by different values of variables.

Mirko can also play with some other visualization options available in the Sketchlet services. For example, he could use a Sketchlet facial animation service (see Appendix B for details, Section B.7) and define the mood of the character based on the distance (Figure B.4). For example, Mirko could define that the character will be in a “happy” mood when the distance is around the target one, “scared”, when the distance is too close, or “tired” when the distance is much bigger than the target one.

face-mood = if(distance < target_distance – 0.5, "scared", if(distance > target_distance + 0.5, "tired", "happy")) Fig. B.4. Using the Sketchlet facial animation service and defining the mood of the character based on the distance variables.

Integration with the Simulator The visualization that Mirko has created is visible within Sketchlet environment on Mirko’s laptop. Mirko now wants to integrate his graphical representation more tightly with the user interface of the simulator. One simple approach that Mirko could follow is to keep the presentation on his laptop, but use a projector to overlay the image over the desired part of the simulator screen screen. Using remote desktop solutions, such as VNC [ref.], Mirko can also get a part of the screen visible and interactive on other computers, or mobile devices such as iPad, or iPhone. Sketchlet, however, offers few other options. Mirko could run Sketchlet on the simulator computer (in our case a Windows PC). Sketchlet enables designers to select which part of the page will be shown in the playback mode, and to position the window on any part of the screen. Additionally, Mirko can also shape, and define the transparency of the playback windows, as well as set that the windows will always be on the top of other windows (Figure B.5).

10

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, MANUSCRIPT ID

Fig. B.5. Integration of Sketchlet graphical objects with other graphical applications. In Sketchlet, a user can select which part of the page will be shown, and the user can define the size, position, shape and transparency of the window in which the part of the page will be shown. It is also possible to set other parameters, such as forcing the window to be always on the top of other windows.

Audio Presentation of Car Distance To create audio presentation of a car distance, Mirko could use any of the Sketchlet audio services: a MIDI player, an MP3 player, or a text-to-speech service (see Appendix B, Section B.3 and Section B.4 for details). Mirko can simply select a service from the list, and he will get the variables that he can manipulate in order to produce sound. Table B.3 shows some examples how these variables could be set to produce the audio effects relative based on the distance variables. TABLE B.3. A POSSIBLE SETTING OF VARIABLES OF SPECHIFY AUDIO SERVICES. MIDI player variables =if(distance < target_distance, 50 + (distance / t_far) * 50, “”) midi-key = if(distance < target_distance, 100 + (distance / t_far) * 900) midi-duration MP3 Player variables if(distance < target_distance, "keep_distance.mp3", “”) mp3-song Text-to-speech variables if(distance < target_distance, "Keep the distance. You are too close.", “”) tts-input

If Mirko were a more advanced use, her could also try some more advanced options of the Sketchlet, such as macros, or scripts. For example, Figure B.6 shows an implementation of scenario with a MIDI player where, if the car distance is smaller than the target distance, the sound will be given as beats with constant duration, but variable tone.

11

(a)

while (true) { if (distance < target_distance) { Sketchlet.update("midi-key", 50 + (target_distance – distance) * 50); Sketchlet.pause(0.4); } }

(b)

Fig. B.6. An example of using Sketchlet simplified programming options: (a) a Sketchlet macro, (b) a script written in Sketchlet Javascript editor.

Haptic Presentation of Car Distance To create a haptic (vibration) presentation of a car distance, Mirko could use two Sketchlet services: a JavaME enabled mobile phone, or game devices such as Wii Remote (see Appendix B, Section B.5 for details). Again, Mirko can simply select a service from the list, and he will get the variables that he can manipulate in order to produce vibration on these devices. Table B.4 shows some examples about how these variables could be set to produce the vibration effects relative based on the distance variables. In both cases, Mirko defines a formula with the duration of the vibration pulse. TABLE B.4 A POSSIBLE SETTING OF VARIABLES OF SKETCHLET HAPTIC SERVICES. Mobile Phone Variables mobile-vibrate =if(distance < target_distance, “1000”, “”) Wii Remote Variables wii-vibrate =if(distance < target_distance, “1000”, “”)

NEWS PRESENTATION BASED ON STATE OF THE CAR To explore the possibilities of presentation of news when a user is within car, but not driving, Mirko pursues two paths:  Using a Sketchlet adaptation of the Google news service,  Using screen capturing and screen poking, to integrate existing news search applications. In both of these cases, Mirko defines the interaction based on the data received from the car simulator, combine with the Sketchlet speech recognizer service used to enable a user to enter a keyword that will be used to search the news.

Implementation using Sketchlet Google News Web Service A Sketchlet adaptation of the Google news Services (see Appendix B, Section B.2 for details), enables a user to interact with the service through several variables. By updating the variable “google-news-query”, a users will trigger the service, and as a results several variables will be updated. The Sketchlet adapter currently presents only the top five news items, and for each news item it creates separate variables for a news item title, a date, a link and a short description. Figure B.7 and Table B.5 show the user interface and regions properties settings used to create a simple new presentation interface with the Sketchlet Google news service. By pronouncing the keyword, the user will call the service, which will update the text in the active regions. The visibility of the whole page is set so that the page is not visible when auto is moving, or in gear, but not when the user is driving.

12

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, MANUSCRIPT ID

Fig. B.7. A Sketchlet page with four active regions presenting first four titles of the Google News service. TABLE B.5 A POSSIBLE SETTING OF VARIABLES OF SKETCHLET HAPTIC SERVICES. google-news-query Page transparency layer 1 Region 1 text Region 2 text Region 3 text Region 4 text

=speech-command =if(speed > 0 || gear > 0, 1, 0) =google-news-title-1 =google-news-title-2 =google-news-title-3 =google-news-title-4

Implementation using Screen Capturing and Screen Poking Alternatively, by using hacking techniques supported by Sketchlet, Mirko could also use existing Web search applications, such as BBC News page18. Figure B.8 shows a Sketchlet page that a Mirko could create in this way.

Fig. B.8. Using screen capturing and screen poking to integrate existing new search Web application into the Sketchlet page.

In this example, Mirko opens an existing application in a separate window, possibly on a separate screen, and presents the part of the Window in the active region using the Sketchlet screen capturing support. To call the news service, Mirko can create a screen poking script, which will generate syntactic mouse and keyboard events on the screen. In this example, the script has the following steps: 18

http://news.bbc.co.uk

13

Mouse: Left Button Click at 1621,154 Keyboard: Type CTRL+A Clipboard: Paste Text “=speech-command” Keyboard: Type ENTER As a result, the Sketchlet will move the mouse at the given coordinates, synthesize a left mouse button click event, then synthesize keyboard events for “CTRL A” key combination, copy to the clipboard the content of the variable ‘speechcommand’, paste it, and synthesize the keyboard ENTER event to start the search in the news service. As the search results appear on the screen, they will also be visible in the Sketchlet region, which continuously captures the screen. In order to define screen capturing parameters and screen poking scripts, Mirko does not have to manually code them. Sketchlet offers Mirko the interface where he can specify this parameters “by example”, that is, by clicking and typing over a captured image of the whole screen, and by visually selecting the part of the screen that should be captured.

Wizard of Oz Setting In the end, Mirko also decides to take some of the created “sketches” within the real car, in a “Wizard-of-Oz setting, where Mirko will manually control the distance based on his observation, while one of his colleague will drive. For that purpose, Mirko has to adapt the sketches so that they can be manually controlled. For example, to change the visualization introduced in Figure B.9, he adds a scrollbar and changes the formula for the variable distance, originally based on the data from the simulator, so that it the distance is calculated based on the variable “scrollbar” updated by the scrollbar widget (Figure B.9).

distance=scrollbar_position * t_(far + 0.5) Fig. B.9. Extending and the sketch with a scrollbar, and changing the variable formulas in order to enable a manual control of the sketch.

14

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, MANUSCRIPT ID

APPENDICES REFERENCES 1. 2. 3.

Hartmann B., Doorley S., Klemmer S. R., Hacking, Mashing, Gluing: Understanding Opportunistic Design, IEEE Pervasive Computing, vol. 7, no. 3, July-September 2009, pp. 46-54. Ncube C., Oberndorf P., Kark A.W. (2008), Opportunistic Software Systems Development: Making Systems from What's Available, IEEE Software 25(6), pp. 38-41. Obrenovic Z., Gaševic D., Eliëns A. (2008), Stimulating Creativity through Opportunistic Software Development, IEEE Software 25(6), 6470.