Future of End-User Software Engineering: Beyond the Silos Margaret M. Burnett
Brad A. Myers
School of Electrical Engineering and Computer Science Oregon State University Corvallis, OR 97331 USA
Human Computer Interaction Institute Carnegie Mellon University Pittsburgh, PA 15213 USA
[email protected]
[email protected] Bureau of Labor, more than 12 million people in the U.S. say they do programming at work, and more than 50 million use spreadsheets and databases—compared to only about 3 million professional programmers in the United States [54]. Thus, depending on how conservatively one differentiates “use” from “program in”, the number of end-user programmers in the United States falls somewhere between 12 million and 50 million people—several times the number of professional programmers. Clearly then, end-user programming empowers—it has already empowered millions of end users to create their own software.
ABSTRACT End-user software engineering (EUSE) is a research area that aims to invent new kinds of technologies that collaborate with end users to improve the quality of their software. The practice that EUSE research aims to support is end users using new tools and methods to improve the quality of the software that they and other end users have created. There is a need for this outcome because research shows both that the number of end users creating their own software greatly exceeds the number of professional software developers, and that the software they create is riddled with errors. In this paper, we describe the present state of EUSE, and challenges in moving forward toward a bright future. We show how the future of EUSE may become over-siloed, restricting future researchers’ vision of what can be achieved. We then show that focusing on the in-the-moment intents of end-user developers can be used to derive a number of promising directions forward for EUSE researchers, and how theories can help us further desilo future EUSE research. Finally, we discuss how overcoming challenges for the future of end-user software engineering may also bring direct benefits to the future of “classic” software engineering.
Nardi defined end-user programmers as being distinct from professional developers in that end users’ programs are not the end in itself, but rather a means to accomplish their own tasks [39]. There are a variety of research systems and tools aimed at this audience; examples include an accountant creating spreadsheet formulas (which are computational instructions) to keep track of a budget in Excel or a web-literate end user building a quick mashup to help sort real estate based on custom criteria. Note that, although Nardi’s definition does not define all end-user programmers as lacking formal computer science training, many subpopulations of end-user programmers do lack such training. In this paper, we consider the present and future of an area of software engineering that addresses populations like these. The future of this area that we envision breaks down the silos that have begun to emerge. We focus on where these silos are, and the benefits that could ensue to end-user software engineering research in the absence of these silos.
Categories and Subject Descriptors D.2 [Software Engineering]. H.1.2 [Models and Principles]: User/Machine Systems.
General Terms Reliability, Human Factors, Theory
1.1 End-User Programming versus End-User Software Engineering
Keywords End-user software engineering (EUSE), end-user development
Some examples of the “programming environments” used by end users are shown in Figure 1. For example, CoScripter [38] (Figure 1(a)) allows web users to automate web-based tasks by demonstrating the desired sequence of actions on the web pages. AgentSheets [46] (Figure 1(b)) is aimed mainly at children and teachers. These AgentSheets users demonstrate “rules of behavior” for different characters in a game or simulation by placing the characters onto the game board and demonstrating both the circumstance that requires the new behavior and the behavior itself. LabVIEW [25] (Figure 1(c)) aims at a different audience: scientists and engineers doing sensor-based computation. LabVIEW users wire together nodes representing various sensors and measurement instruments to construct dataflow diagrams defining the desired computations. Finally, TouchDevelop [36] (Figure 1(d)) enables end users to create mobile apps of their own by choosing from applicable actions from a continually updating “smart” keyboard.
1. INTRODUCTION Software engineering is no longer solely applicable to professional software engineers. Anyone who develops software, whether as a professional developer or not, may well need to employ beyond-coding processes to help ensure that the software they develop is of sufficiently high quality for the purpose for which they intended it. Who, besides professional developers, actually develops software? According to data from the U.S. Bureau of Census and Permission to make digital or hard copies of all or part of this work for personal or Permission togranted make digital copies all or of this for classroom use is without or feehard provided that of copies arepart not made or work distributed personal classroom use is granted without feethis provided that are for profit or or commercial advantage and that copies bear notice and thecopies full citation onnot the made first page. components this work owned by others and than ACM or Copyrights distributedforfor profit orofcommercial advantage that must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, this notice and the full requires citationprior on the firstpermission page. Toand/or copya tocopies post onbear servers or to redistribute to lists, specific otherwise, or republish, to post on servers or to redistribute to lists, fee. Request permissions from
[email protected].
requires prior specific permission and/or a fee. FOSE’14, May 31 – June 7, 2014, Hyderabad, India FOSE'14,2014 MayACM 31 – June 7, 2014, Hyderabad, India. Copyright 978-1-4503-2865-4/14/05...$15.00 Copyright 2014 ACM 978-1-4503-2865-4/14/05…$15.00. http://dx.doi.org/10.1145/2593882.2593896
Using systems like these, end users create a variety of programs to use on their desktops, phones, and the web. (Following Ko et al.,
201
(a) CoScripter [38], used for web automations. (Left): script. (Right): Web context for the script. (Original screen shot.)
(b) AgentSheets [46], used for agent-based games and simulations. (Top): Program state that includes some truck objects. (Bottom): A visual rule that controls truck behavior. (Excerpted from http://sgd.cs.colorado.edu/wiki/Frogger_Tutorial under Creative Commons License http://creativecommons.org/licenses/by/3.0/)
(c) A dataflow diagram in LabVIEW [25], which is used primarily for lab applications. (Excerpted from http://upload.wikimedia.org/wikipedia/commons/b/b6/WikipediaFPandB D.png, under Wikipedia Commons License http://commons.wikimedia.org/wiki/Main_Page.)
(d) The mobile version of TouchDevelop [36] for developing mobile apps.
Figure 1: Several end-user programming environments. we define a program as “a collection of specifications that may take variable inputs, and that can be executed (or interpreted) by a device with computational capabilities” [29]. The variability of input values requires that the program be able to execute on future inputs, which is one way it is different from simply doing a computation once manually.) Some ways in which end users create these programs include writing and editing mathematical formulas, dragging and dropping objects onto a logical workspace, typing and editing textual code, sketching user interface states and indicating “hot spots” and transitions, connecting objects in a diagram, or demonstrating intended logic to the system.
Unfortunately however, the software that end users are creating with these techniques is riddled with errors. Evidence abounds of the pervasiveness of errors in the software that end users create (e.g., [19]). These errors can have significant impact. For example, after Hurricane Katrina, dozens of end users created web sites to allow updates about survivors’ last known locations [52]. Confusion reigned with these sites. Data on one site were often missing or inconsistent with data on the next site. Leveraging data across sites was not possible, due to differing data formats and uncertain quality [52]. Turning to a 2011 example from the European Spreadsheet Risks Interest Group’s huge collection of spreadsheet errors, a formula error caused the village of West
202
Baraboo, Wisconsin to underestimate the cost of a borrowing decision by about $400,000 [19, case FH1218). Even when errors in end-user-created software are non-catastrophic, however, their effects can matter. For example, web applications created by small-business owners to promote their businesses do just the opposite if they contain broken links or pages that display incorrectly, resulting in client frustration, loss of revenue, and/or loss of credibility [1].
considers software development tasks only one lifecycle phase at a time; and suffers from siloing individual tools and findings rather than building connective tissue between them. Such silos raise risks that an area can become overly narrow, building upon research in one silo or another, but losing the potential for big steps forward that can enable a research area to transform. Thus, this paper sets out a number of de-siloing challenges that we hope will influence the future of end-user software engineering. We consider them from the perspectives of who, when, why and how.
Given that so much of end-user-created software suffers from quality too low for the purposes for which it was created, there is a clear need for researchers and tool makers to address this. From much prior research in classic software engineering, we know that to improve quality, the software lifecycle beyond just the “create” stage must be supported. That is, end-user developers, like professional developers, must be supported beyond the coding stage of their software development.
2. END-USER SOFTWARE ENGINEERING’S “WHO” A key difference between end-user software engineering and traditional software engineering is the kind of developers served. A premise of classical software engineering research is that professional developers have some training and/or experience in software engineering methods, and also at least a little interest in following good software engineering practices. In contrast, enduser software engineering research cannot assume the presence of any of these.
To help end-user developers do just that, a new research area emerged about a decade ago to address this problem. The area is known as end-user software engineering (EUSE). As Ko et al. summarize, end-user software engineering technologies are “technologies that collaborate with end users to improve software quality” [29]. Thus, the differences between end-user programming and end-user software engineering are that end-user programming focuses mainly on how to allow end users to create their own programs, whereas end-user software engineering considers how to support the entire software lifecycle and its attendant quality issues. Differences between end-user software engineering and traditional software engineering are several, but they all stem from a difference in population.
From this difference, two requirements become clear for end-user software engineering tools and techniques. First, EUSE tools and techniques must be designed for audiences that are usually unfamiliar with professional software engineering practices (even though they may have programmed before). Second, these tools and techniques must fit well with their audience’s non-softwareengineering-oriented interests, motivations, and practices.
2.1 Today’s End-User Developer Target Audiences
When taking this lifecycle view of end users’ software, the term end-user developer becomes more appropriate than end-user programmer, drawing upon Lieberman et al.’s definition of enduser development: “a set of methods, techniques, and tools that allow users of software systems, who are acting as nonprofessional software developers, at some point to create, modify or extend a software artifact” [37]. This terminology points out that end users involved in software development may or may not actually create new code as part of what they do in the software development process. For example, they might participate in activities like configuring their software, co-designing their software side-by-side with professional programmers, and tailoring their software [15, 17, 20, 26, 61]. Because the term “end-user developer” implies participation in any software development lifecycle stage—not restricted to just creating code—for the remainder of this paper we will use the term enduser developers .
Fulfilling these requirements requires considering more deeply who the target audiences are. The term “end-user programmers” was once used synonymously with “novice programmers” to mean anyone with a lack of experience programming regardless of their motivations, but that trend largely disappeared after Nardi’s clarifying book on end-user programming [39]. Still, in today’s end-user software engineering literature, assumptions about the population of end-user developers to be served are sometimes left implicit, and differ from paper to paper. For example, some researchers use the term to mean people who have neither experience nor interest in software development per se (e.g., [50]), others use it to mean anyone using a particular programming environment that targets end users (e.g., [6, 36]), and others use it to mean people who can program in one particular application area only (such as end-user webmasters [47]).
1.2 A Future of End-User Software Engineering: Challenges in De-siloing
Such assumptions are true of some end-user developer populations, but not all of them. For example, counterexamples to many of the above assumptions can be seen in Scaffidi et al.’s survey responses from 831 information workers (subscribers to Information Week) [51]. As Table 1 shows, these users—77% of whom were in non-technical jobs—did a surprising amount of programming, often using only ordinary office productivity tools.
EUSE research is at a crossroads. Following too closely in the footsteps of its classic software engineering predecessors raises a risk of over-siloing the area, to the detriment of the end-user developers it aims to serve. When we refer to siloes, we mean “a system, process, department, etc. that operates in isolation from others” [40] (italics added).
One reasonable way to establish what assumptions to make about a target end-user developer population is to empirically study the target audience as part of designing EUSE tools for that audience, and some EUSE research has moved in this direction (e.g., accountants using spreadsheets [59] and web designers [41]). An advantage of informing EUSE tool designs by empirically studying a particular population is that it enables EUSE researchers to fulfill the two requirements enumerated at the
In one sense, end-user software engineering de-silos already: it removes the silo that fenced off software engineering benefits from everyone except professional software developers. However, end-user software engineering has not gone much further in desiloing, and we believe the area could benefit from a great deal more. For example, it primarily considers end-user developers only as (atomic) people, rarely anything smaller or larger;
203
across a wide span of end-user developers1, from scientists and mathematicians who program often in traditional languages [56] to spreadsheet users [59] to teachers [63] to museum exhibit designers [7].
Table 1. Programming-like features, in a variety of common software applications, used by at least 33% of the information workers [52]. Application
Used by (% of users)
Slide Editors & Word Processors
96.1
Spreadsheets
93.1
Databases
79.3
Web Pages
68.6
Feature used by Feature (% of users) Create document templates 73.3 Make inter-document hyperlinks 53.7 Record desktop editor macros 33.3 Use functions (e.g., “sum”) 89.3 Create charts in spreadsheets 80.1 Inter-spreadsheet references 66.2 Create spreadsheet templates 50.4 Record spreadsheet macros 42.3 Create/edit spreadsheet macros 38.6 Reference records by key 74.0 Create tables 71.7 Create database views 68.1 Create stored procedures 49.5 Create hyperlinks 65.2 Create web forms 52.2 Create web page scripts 42.9 Create JavaScript functions 34.3
The decoupling of intent from personal identity helps to de-silo end-user software engineering because it removes the need to separate end-user developers as people into groups: instead, researchers can focus on intent, which slices across many different classes of end-user developers. Intent as a central concept ties end-user software engineering research directly to the end-user developer’s motivations and values in that particular software development task, which in turn can affect their software engineering processes.
2.2.1 Intent as a basis for addressing the challenge Moving to intent as the key definer of when someone is acting as an end-user developer suggests the possibility of raising intent to a first-class factor in defining future EUSE tools and techniques. One way focusing on intent can help advance EUSE research is by revealing flawed assumptions in EUSE work. One common assumption is “if we build it they will come.” That is, if a new tool or feature becomes available that can improve users’ correctness or efficiency, then surely conscientious users will use it. However, a focus on intent clarifies that this assumption may not hold for end-user developers. For example, what might a laboratory scientist’s intent be, when working on a laboratory application to collect and measure data from a variety of sensors? Some possibilities might be “build my lab application” or “fix my broken lab application” or “add new capabilities to last year’s lab application.” Note that none of these intents sounds much like “find a new tool that might give me some useful new ways to guard the quality of my application.”
beginning of Section 2: namely, not relying upon familiarity with software engineering, and fitting with the target audience’s motivations and practices. However, a disadvantage is that it can encourage siloing EUSE research by population. Further, each silo (population) must be pre-identified and researched on its own, without including other end-user developer populations.
Since intent resides in people’s heads, understanding its implications on EUSE is facilitated by turning to applicable theories of how people think. Blackwell’s Model of Attention Investment [5] is a model applicable to this situation; it helps to explain how end-user developers make choices about what problem-solving tools and features to use in computer systems.
2.2 Challenge: Targeting a More Nuanced “Who” A de-siloing alternative can be derived from Nardi’s original definition. Recall that, according to Nardi [39], the distinguishing factor of professional programmers vs. end-user developers is a difference in intent, not a difference in experience. Indeed, Nardi goes on to explain that end users are not necessarily “casual,” “novice,” or “naive” users. Ko et al. emphasized this notion by defining end-user programming (and end-user development) as a role (a close cousin to high-level intent) rather than a person’s semi-permanent identity [29].
Blackwell’s model rests on three constructs: the users’ perceptions of the costs, benefits, and risks they will face in completing their task (accomplishing their intent) one way versus some other way. According to Blackwell, users expect that exploring an unknown tool or feature will have a cost, some benefits, and some risks. For example, in the lab example above, suppose the scientist’s usual way of accomplishing such tasks is to use the environment’s editor. If something goes wrong, the scientist can fix it using the editor. This has no learning cost, and if the scientist believes s/he is succeeding, it seems to have the benefit of accomplishing the task with no risk. Only if the scientist realizes that there are problems too great to fix that way (i.e., there is high risk that the benefits will not be realized) is s/he likely to consider switching to a new unknown tool.
Defining end-user developers based on intent allows a more nuanced version of “who”, thus widening the population of those who could be served by end-user software engineering research. Roles can include not only traditional populations of end-user developers like teachers and spreadsheet users engaging in software development, but also populations who, although they program in various ways fairly regularly, still have little interest in software engineering in at least some of their software development tasks.
Now suppose the scientist decides that a more powerful problemsolving tool is needed and, in looking around the environment, sees a tool that might help. The scientist ventures to open it up, and is now faced with a first impression of the cost of learning the tool and of the cost of using it for the particular problem at hand.
As an extreme example, a professor of software engineering might be systematic and rigorous in validating software created for distribution to other researchers, but ad hoc about validating a grant budget spreadsheet. Indeed, an ad hoc view of beyondcoding software engineering tasks has been shown to arise often
1
204
From now on, we will use this phrase as an abbreviation meaning “developers operating in an end-user development role.”
This first impression is the perceived cost the scientist is imagining. S/he is also considering the perceived risk (probability) that going down this path will be a waste of time or, worse, will leave the application in a mess from which it would be difficult to recover. If users perceive an EUSE tool’s costs or risks to be greater than the perceived benefits, they are unlikely to use the tool [5, 50].
software development skills. However, many end users are reluctant to do so—resistance to various software development tasks has been reported for a variety of end-user developers (e.g., [14, 42, 56, 63]). As these reports show, to end-user developers like these the benefits seem low, and/or the costs and/or risks seem high. Fortunately, the keys to attracting even reluctant learners to learn software development skills become clear from the ideas of intent and the cost/benefit/risk factors of the Attention Investment model. That is, a system can introduce appropriate software development practices to end-user developers by serving their actual intents (if known) to provide these users with the benefits they seek or to reduce their risks or costs in a way that also shapes their software development behaviors.
Viewing intent in light of Blackwell’s Model of Attention Investment points out that EUSE tools and techniques can target an intent-nuanced “who” by (1) connecting to a target end-user developer’s intent whenever it may arise, and (2) using that moment to entice them to take appropriate software engineering steps when the need arises by making clear from the outset the upfront and ongoing costs, benefits, and risks that are likely. One approach to accomplishing these two steps is to explicitly use inferences of users’ intentions as starting points of EUSE support tools. Another approach is to use curiosity-based enticement to draw end users into engaging at an appropriate moment with software engineering features. A third is to base tools on past intents of the target audience as predictors of “common” intents.
For example, the Whyline research began by first researching common debugging intents empirically. End user developers’ most common debugging intents were harvested by investigating their most common questions. The investigation showed that their
Although possible intent-focused approaches to EUSE techniques are relatively unexplored, the next section describes a special case with promising beginnings in that direction.
2.2.2 A special-case “who”: EUSE & learning An interesting case study in de-siloing by moving from “who” as people to “who” as roles has occurred in the realm of learning. Originally, tools for helping people learn software development skills aimed at teaching programming (e.g., syntax, semantics, and basic programming skills) to people who wanted to learn programming. However, a relatively recent trend is to reach out to people who may not be interested in programming to try to convince them to want to program—i.e., to manipulate intent, or in the terminology of the Blackwell model, by manipulating users’ perceptions of benefits. A system can manipulate intent in (at least) three ways: (1) it can build new intent, (2) it can serve existing intent (if intent is known), and/or (3) it can attempt to infer intent, often as a precursor to (1) or (2).
(a) Pex4Fun’s gamification [60] can build intent to program in people who like to play games. (Original screen shot.)
The aim of a system building new intent is to reach out to users who are motivated by some benefit other than “a program of my own.” Such systems present programming as a means to that other benefit, with the hoped-for result that the users then decide to use the programming system in order to attain that benefit. One example is to “gamify” the process of learning programming (e.g., Pex4Fun [60] (Figure 2(a)) and Gidget [35]), leveraging the drawing power of a game to engage users in learning programming as a side effect of playing the game.
(b) Whyline (Alice version) [31] directly serves an existing intent that has been reported empirically. Here, the user asks “why didn’t Pac resize 0.5?” (Original screen shot)
A system can build intent to learn other software development skills too, not just coding. For example, a few researchers have worked to draw users toward higher success at design. Besides offering a non-software-development benefit (like telling stories, or designing their own web pages), these works feature scaffolding to reduce users’ perceptions of the costs of achieving the benefit they seek. For example, Kerr et al. provided story templates to help users with story design ideas in the setting of StoryTelling Alice’s successor [27], and Lee et al. provided manipulable examples to help end users explore web page design ideas [34]. Approaches like these can help improve end-user developers’ software development skills.
(c) Idea Garden [10] first infers intent and cost, and then offers help to reduce the cost of achieving that intent.
Approaches like the ones discussed so far have been shown to be effective for users who are at least open to expanding their
Figure 2: Three ways systems can build upon intent.
205
most common questions were “why” and “why not”, so the Whyline was developed to allow users to ask those questions directly [31], and to answer them through scaffolded views of the kinds of problem-solving approaches professional developers use, such as tracing through calling sequences with their parameter values. See Figure 2(b) for a Whyline view of a calling sequence with parameter values.
coverage yet) to blue (fully covered). At any time, if the user notices that a value (like the “Max of...” cell in Figure 3(a)) is correct, she can check it off, and the coverage is automatically recalculated so that corresponding cell border colors update immediately in the spreadsheet. There has also been research to support other stages of the end users’ software development lifecycle, such as design [48], assertions [8, 45], reverse-engineering [16], refactoring [3, 58], and explicitly guarding against or detecting errors and software “smells” [2, 18, 24, 53]. Figure 3(b) shows an example of the Topes approach [53], which guards against errors.
Serving existing intents can also be combined with building new intents. For example, Surprise-Explain-Reward [64] is a strategy to entice users toward desirable software engineering behaviors. The strategy attempts to arouse the end-user developer’s curiosity about some aspect of their software that seems to be erroneous (such as a faulty cell formula), by decorating the cell if the tool has something to say about it. If the software’s accuracy is important to that developer, they may become curious and follow up. The decoration (notification that the software has something to say) is the surprise, the information they get by following up (e.g., by hovering over the decoration) is a brief explanation, and the reward is (ideally) a found error and a sense that there is less risk of the software producing incorrect answers. Surprise-explainreward has been used successfully in several EUSE settings; one example was to entice end users into incrementally entering assertions to guard spreadsheet correctness [8].
These approaches are steps forward in bringing to end-user developers some of the benefits of software engineering methods—but they do not go far enough. Most of these approaches address each lifecycle stage in isolation (e.g., just refactoring, or just designing) so, while they break out of the “coding” silo, they then erect new silos for each lifecycle phase.
3.2 Challenge: Beyond “Beyond Coding” Such silos for each lifecycle phase do not match the opportunistic workstyle of many end-user developers [7, 56]. In developing their software, exactly when are end-user developers doing design, when are they testing, and so on?
Another intent-oriented strategy is to target the end-user developer’s intent by inferring it or asking about it in context [10, 12]. For example, the Idea Garden (Figure 2(c)) infers intent from context, and also infers the costs of behaviors the end-user developer may be exhibiting, and then provides help that will reduce the user’s difficulties (cost) of accomplishing that intent [10].
For example, consider a teacher who wants to create a grades spreadsheet, and suppose she begins by entering a few column headings across the top (say, “ID” and “name”), and then typing in all her students’ IDs and names into the corresponding columns. So far she has not entered a single “line of code” (formula)—so is she in the design stage? In one way she is, because she has begun to specify the structure of the spreadsheet,
The approaches discussed here to de-siloing the learning of EUSE practices share a set of common attributes: (1) targeting end-user developer intent, often as a “right now” moment, (2) drawing enduser developers into doing just a little of the desired behavior at a time when it can benefit them, and (3) giving them the supports they need, just in time and in their own environment, so that they can succeed at the software engineering behaviors they are engaging in. These approaches generalize beyond learning settings, as some of the above examples have already demonstrated. That is, building new intent by serving an existing intent does not have to be leveraged only in “learning systems” or “learning modes.” Approaches like these can be included in any system that end-user developers use, to encourage incremental, life-long improvement of their software development skills, in the contexts of doing their ordinary work.
(a) The WYSIWYT testing approach for end-user developers [49, 21]. (Original screenshot.)
3. WHEN EUSE CAN HAPPEN From a lifecycle perspective, end-user software engineering begins, by definition, by moving beyond the coding phase silo to connect it with other phases of the software lifecycle.
3.1 Today’s EUSE: Beyond Coding Researchers in end-user software engineering have devised a variety of techniques to support software engineering activities beyond coding. Perhaps the first such example of explicit support beyond coding was the “What You See Is What You Test” (WYSIWYT) research to help end-user developers systematically test their spreadsheets [49, 21]. WYSIWYT incrementally tracks test coverage of the spreadsheet formulas (via definition-use association coverage of the formula subexpressions), and colorizes the cell borders on a continuum from red (no test
(b) Topes allow end-user developers to define and error-check data inputs [53]. (Original screen shot ) Figure 3: Examples of beyond-coding approaches.
206
and indeed, according to design theorist Schön, she is doing design according to the reflection-in-action model [11, 55]. However, from a traditional software engineering standpoint, she is not, because she has not specified any relationships or logic, yet has already skipped ahead to “data entry.” Suppose she next “implements” a little of the program (enters several formulas), and simultaneously “tests” a couple of the formulas (glances at their values based on the inputs she put in), and decides things are going well.
For example, consider “Frieda“, an office manager in charge of her department’s budget tracking. (Frieda was a participant in a set of interviews with spreadsheet users that the first author conducted. Frieda is not her real name.) Every year, the company she works for produces an updated budget tracking spreadsheet with the newest reporting requirements embedded in its structure and formulas. But this spreadsheet is not a perfect fit to the kinds of projects and sub-budgets she manages, so every year Frieda needs to change it. She does this by working with four variants of the spreadsheet at once: the one the company sent out last year (we will call it Official-lastYear), the one she derived from that one to fit her department’s needs (Dept-lastYear), the one the company sent out this year (Official-thisYear), and the one she is trying to put together for this year (Dept-thisYear).
This opportunistic mix of several elements from the software lifecycle is the way many of today’s end-user developers with no prior software engineering training work [7, 56]. Without support for navigating seamlessly among lifecycle phase “moments” in their own environments, these end-user developers may never choose to engage in tools that could help them. (Recall our emphasis on developers’ intents in Section 2.) Yet, as the software they create grows in complexity or criticality (e.g., [19, 57]), sometimes they need these supports to attain the quality their software needs.
Using these four variants, Frieda exploratively mixes reverse engineering, reuse, programming, testing, and debugging, mostly by trial-and-error. She begins this process by reminding herself of ways she changed last year’s by reverse engineering a few of the differences between Official-lastYear and Dept-lastYear. She then looks at the same portions of Official-thisYear to see if those same changes can easily be made, given her department’s current needs.
At a minimum, then, EUSE tools need to be integrated into the work styles and environments these developers use. To do so requires tool builders to ensure that the tools they build for various lifecycle stages are interactive and incremental, so that end-user developers can easily do a little using one lifecycle stage’s tool, then switch to another to do a little, then switch back. This may require entirely different reasoning algorithms than have used in the lifecycle phase counterparts in classic software engineering. For example, some EUSE efforts have been to develop new EUSE static and dynamic reasoning algorithms that are highly incremental and responsive so as to support exactly this kind of incremental phase switching (e.g., [23, 32, 49]).
She can reuse some of these same changes this year, but copying them into Dept-thisYear is troublesome, with some of the formulas automatically adjusting themselves to refer to DeptlastYear. She patches these up (if she notices them), then tries out some new columns or sections of Dept-thisYear to reflect her new projects. She mixes in “testing” along the way by entering some of the budget values for this year and eyeballing the values that come out, then debugs if she notices something amiss. At some point, she moves on to another set of related columns, repeating the cycle for these. Frieda has learned over the years to save some of her spreadsheet variants along the way (using a different filename for each), because she might decide that the way she did some of her changes was a bad idea, and she wants to revert to try a different way she had started before.
Still, from a de-siloing perspective, this approach is somewhat incremental. It is supportive of the fact that end-user developers will need to rapidly and intermittently switch in and out of their tools, but still silos the EUSE tools and research efforts by lifecycle phase.
To support this style of development, the exploratory programming project is creating a variety of visualizations, search facilities, and interaction techniques to provide effective ways for users to find, understand, assess, reuse and create variants, and be able to ask “why” questions to understand differences between variations of a system. It will also include supports for helping end-user developers enhance their problem-solving strategies when they encounter difficulties.
An alternative approach would be a more dramatic departure from classic software engineering approaches. We believe it has more potential to transform but also brings more challenges. In essence, it is to promote end-user developers’ work styles and intents as a way to define our vision, giving EUSE researchers the potential to transcend one-lifecycle-phase-at-a-time research silos in favor of bigger visions. One way to go about such an approach would be to frame the EUSE problem more along the lines of the problem-solving activity rather than the software engineering lifecycle. To see how this can be done, we describe an emerging EUSE research project (in which we are involved) that characterizes its target as supporting “exploratory programming” by end-user developers (http://www.exploratoryprogramming.org/).
A lifecycle view of the project would not capture everything here: exploratory programming includes design, implementation, testing, debugging, and reuse—but the human activity of exploration is more holistic than these five different lifecycle phases. For example, the lifecycle view would not capture the user backing out of part of a solution, and would not capture a tool helping the user find a useful problem-solving strategy. Thus, the exploratory programming example suggests that first-class consideration of developer intents (such as “explore”) may help EUSE researchers devise future EUSE approaches that transcend today’s lifecycle-phase silos.
The project targets opportunistic end-user developers who are exploring possible ways to achieve their development goals, working to understand as much as needed of unknown code fragments, and exploring ways to fit some of the pieces together. Some code fragments may come from the user’s own files, and others discovered on the web. Some may have been written by this user, some by other end-user developers, and some by professional developers. As things go wrong, the end-user developer may need to revisit some variants tried before that were working.
4. CHALLENGE: WHY AND HOW Devising new EUSE ideas for tools and techniques that actually help end-user developers is difficult if we do not understand why end-user developers might use them or how the new ideas could work. Understanding these whys and hows has a de-siloing theme in its ability to connect one type of knowledge with another.
207
One way to connect our tools and techniques with an understanding of the developers they target is to ground our EUSE research efforts in formative empirical studies, so as to understand the target group of end-user developers. The research paradigm of grounding research ideas with formative empirical work is common in the EUSE area, and many of the research efforts we have summarized here have proceeded in this way.
and data about artifacts. Demonstrating its usefulness, the results of their empirical case study on the concept of focus captured nuanced advantages and disadvantages of focusing developer attention on a small number of modules that prior atheoretic measures have not been able to expose.
5. DISCUSSION: WHAT EUSE HAS TO SAY TO “CLASSIC” SOFTWARE ENGINEERING RESEARCH
However, this research paradigm still leaves an important gap: a missing or weak connection among similar EUSE research projects. To close this gap, we argue for the importance of more theoretical grounding in future EUSE work. Theory can help yield a more holistic understanding of how EUSE results fit together. Although a few EUSE projects have been human-theorybased, such projects are still relatively rare.
EUSE research began with the following overarching research question: Is it possible to bring the benefits of rigorous
software engineering methodologies to end users? To investigate this goal, EUSE researchers have built upon a blend of foundations from software engineering and humancomputer interaction. The results may have implications not only for future EUSE research, but also for “classic” software engineering research into how to support professional software developers.
As an example of the benefits such theories can bring to EUSE, recall from Section 2 how the Model of Attention Investment [5] provides a generalized way to gain insights into how and whether an end-user developer might go about deciding whether to even try a new EUSE tool or feature. Indeed, this model has already been used to shape and predict acceptance of a variety of EUSE approaches (e.g., [8, 10, 50, 59, 64]). Informing EUSE approaches from the outset using a common model (here, Attention Investment) connects seemingly disparate tools and approaches through their shared set of fundamentals.
The challenges of supporting end-user developers to do software engineering activities lend a magnifying lens to problems that may also affect professional software developers. A case in point is the Whyline debugging technique, which was developed originally for end-user developers in the Alice setting (Figure 2(b)) [31], and was later used as the basis for a Whyline for professional developers working in Java [32]. Empirical evaluations of the tool showed that novice Java developers with the Whyline were twice as fast as expert Java developers without it.
A second way theory can benefit EUSE research is to guide the pre-study design and post-study interpretation of empirical studies. For example, Ko and Myers developed a theoretical framework and methodology for investigating causes of software errors in programming systems [30]. This framework draws from a cognitive theory of human errors and situates it with results from many of the pioneering empirical studies of programming. The resulting framework and related empirical methodology has been used to iteratively evaluate and design the end-user programming system Alice [13], and to then inform the work on Whyline for Alice [31].
In addition to the Whyline case, the theory research on software errors, born in the EUSE setting as discussed in Section 4, has influenced several mainstream software engineering efforts (e.g., [28, 62, 65]). Similarly, research into how gender differences tie with software tool usage, originally investigated in EUSE [4], was later shown to be applicable to professional software development tools as well [9].
Theory can also connect tools after the fact—even tools that, on the surface, seem to have little to do with one another. For example, Information Foraging Theory, although a relative newcomer to software engineering, has already been used to show commonalities and design patterns among more than 20 different software engineering tools spanning three domains: debugging, refactoring, and reuse [22]. Information Foraging Theory [43], which is rooted in earlier foraging theories from biology, is a predator/prey theory that can both explain and predict how software developers (information predators) go about gathering the information they need (the prey) when performing information-intensive software tasks like debugging. With the exception of emerging work by Kuttal et al. [33], Information Foraging Theory has not yet been applied to the design of EUSE tools.
At its core, the EUSE vision is of a different kind of partnership between developers and tools than is sometimes the case in classic software engineering. In the EUSE partnership vision, a developer’s intents and work style take priority over what a tool might “want” the developer to do—because if an end-user developer does not appreciate the tool’s behavior, the developer is likely to simply abandon it. Because of this priority, and also because of its dual heritage from software engineering and human-computer interaction, EUSE research is arguably ahead of classic software engineering in research that focuses squarely on the developers and their intents. As the examples above demonstrate, bringing these focuses into classic software engineering research can bring benefits to professional developers, just as they do to end-user developers.
Another predator/prey model, which centers on notions of ecosystem diversity, shows a fourth way theories can benefit future EUSE work: using theory-derived measures to understand why and how empirical measures of developer behaviors connect with empirical measures of artifacts that might seem to be surrogates for developer measures [44]. Posnett et al. used this theory to derive complementary measures of “focus” for software development (by professional developers): developer attention focus and module activity focus. By using symmetric measures and concepts for both human diversity of attention among models and module diversity of exposure to humans, the theory provides a new way to unify and to contrast results from data about humans
Indeed, going beyond the examples above, the use of more advanced intent-oriented approaches such as building intent and inferring intent (Section 2.2) could be used to entice professional developers into more productive or effective software engineering behaviors, or to deliver to them the information they need at the very moment they need it. Implications for classic software engineering research then could be to draw upon ideas we have presented for end-user software engineering, so as to bring benefits of end-user software engineering research back to professional developers.
208
6. CONCLUSION
in this material are those of the author(s) and do not necessarily reflect those of the National Science Foundation.
In this paper, we have argued for the benefits of de-siloing to achieve the kinds of transformative changes that could come about in EUSE’s future. Specifically, we advocate de-siloing EUSE research from the perspectives of who, when, why and how.
8. REFERENCES [1] Anonymous. 2013. User-unfriendly PR: Nearly 9 of 10 of Americans have negative feelings about brands with poorly performing websites and mobile apps, new SOASTA study reveals, Bulldog Reporter Daily Dog (June 20, 2013), https://www.bulldogreporter.com/dailydog/article/userunfriendly-pr-nearly-9-of-10-of-americans-have-negativefeelings-about-brands- (Accessed Dec. 8, 2013).
EUSE’s who: EUSE research is beginning to move beyond assumptions of a fairly homogenous population of end-user developers. Instead, we argue for the more modern notion that end-user development is a role, or intent, at some particular time, not a personal identity. An immediate benefit is that de-siloing who EUSE targets in this way can open the door to new populations of end-user developers, working alone or together. More fundamentally, intent-focused approaches can promote intent to a first-class concept in EUSE, allowing a rich set of intent-oriented operations, such as attempting to manipulate it, infer it, target it, or reward it.
[2] Abraham, R. and Erwig, M. 2009. Mutation operators for spreadsheets, IEEE Transactions on Software Engineering, 35, 1, 94-108. [3] Badame, S. and Dig, D. 2012. Refactoring meets spreadsheet formulas, IEEE International Conference on Software Maintenance, 399-409. [4] Beckwith, L., Burnett, M., Wiedenbeck, S., Cook, C., Sorte, S., Hastings, M. 2005. Effectiveness of end-user debugging software features: Are there gender issues? ACM Conference on Human Factors in Computing Systems (Portland, Oregon, April 2005), 869-878.
EUSE’s when: EUSE already de-silos end-user programming by moving research beyond the coding phase silo, but it does not go far enough. We argue that it is not wise, once beyond the coding silo, to then build a new silo for each phase of the software lifecycle. Instead, we advocate moving toward research projects that are not lifecycle-phase-centric, instead focusing on supporting end-user developers as they actually work—opportunistically, incrementally, and rarely one lifecycle phase at a time. One possible approach is to start with the activity as the developer sees it (e.g., “explore” my options in getting the functionality I want), rather than starting from a lifecycle phase perspective (e.g., “reuse”).
[5] Blackwell, A. 2002. First steps in programming: A rationale for attention investment models, IEEE Human-Centric Computing Languages and Environments (Arlington, VA, Sept. 2002), 2-10. [6] Bogart, C., Burnett, M., Cypher, A., Scaffidi, C. 2008. Enduser programming in the wild: A field study of CoScripter scripts, IEEE Symposium on Visual Languages and HumanCentric Computing (Herrshing am Ammersee, Germany, Sept. 2008), 39-46.
EUSE’s why and how: We advocate a greater incorporation of theory into EUSE research. Theories enable us to connect multiple research efforts, even if on the surface they have little in common. For example, creating multiple new EUSE approaches from a common set of foundations connects all the approaches based on those particular foundations. Finding ways to pattern-match existing tools to a common theory can also connect those tools. Theory can even be used to connect disparate data sets, if those data sets revolve around a concept that can be intellectually unified and symmetrically measured. In short, theories can enable EUSE researchers to see through separate silos by generalizing across them.
[7] Brandt, J., Guo, P., Lewenstein, J., and Klemmer, S. 2008. Opportunistic programming: How rapid ideation and prototyping occur in practice. ICSE’08 Workshop on EndUser Software Engineering. (Leipzig, Germany, May 2008). [8] Burnett, M., Cook, C., Pendse, O., Rothermel, G., Summet, J. and Wallace, C. 2003. End-user software engineering with assertions in the spreadsheet paradigm, International Conference on Software Engineering, (Portland, OR, May 2003), 93-103.
Realizing these goals will be challenging, and it would not be realistic to expect these kinds of changes in the short term. Still, we argue that end-user software engineering, with its dual heritage in software engineering and human-computer interaction, is in a good position to eventually achieve them. By doing so, we can not only change the reliability that end users can expect from the software they develop for themselves, but at the same time contribute to their problem-solving skills, to their enjoyment of their software development activities, and to our own ability as researchers to have bigger visions, beyond just one silo at a time.
[9] Burnett, M., Fleming, S., Iqbal, S., Venolia, G., Rajaram, V., Farooq, U., Grigoreanu, V., Czerwinski, M. Gender differences and programming environments: Across programming populations, ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, (Bolzano-Bozen, Italy, Sept. 2010), Article 28, 10 pages. [10] Cao, J., Kwan, I., Bahmani, F., Burnett, M., Fleming, S., Jordahl, J., Horvath, A., Yang, S. 2013. End-user programmers in trouble: Can the Idea Garden help them to help themselves? IEEE Symposium on Visual Languages and Human-Centric Computing (San Jose, CA, Sept. 2013), pp. 151-158.
7. ACKNOWLEDGMENTS We’d like to acknowledge the many researchers whose pioneering works have helped establish the area of end-user software engineering. We also thank the students and collaborators who contributed to our own work in end-user software engineering. Our work in developing this paper was supported in part by the National Science Foundation under grants IIS-1116724, CNS1240957, IIS-1314356, IIS-1314384, and CCF-1302113. Any opinions, findings and conclusions or recommendations expressed
[11] Cao, J., Riche, Y., Wiedenbeck, S., Burnett, M., Grigoreanu, V. 2010. End-user mashup programming: Through the design lens, ACM Conference on Human Factors in Computing Systems, 1009-1018. [12] Cao, X. and Iverson, L. 2006. Intentional access management: Making access control usable for end-users,
209
Symposium on Usable Privacy and Security (SOUPS), (July 2006, Pittsburgh, PA), 20-31.
[24] Hermans, F., Sedee, B., Pinzger, M., Van Deursen, A. 2013. Data clone detection and visualization in spreadsheets, International Conference on Software Engineering, 292-301.
[13] Conway, M., Audia, S., Burnette, T., Cosgrove, D., Christiansen, K., Deline, R., Durbin, K., Gossweiler, R., Koga, S., Long, C., Mallory, B., Miale, S., Monkaitis, J., Patten, J., Pierce, J., Shochet, J., Staack, D., Stearns, B., Stoakley, R., Sturgill, S., Viega, J., White, J., Williams, G., Pausch, R. 2000. Alice: Lessons learned from building a 3D system for novices, ACM Conference on Human Factors in Computing Systems, 486–493.
[25] Johnson, G. 1997. Labview Graphical Programming: Practical Applications in Instrumentation and Control, 2nd Edition. (McGraw-Hill, New York). [26] Kahler, H. 2001. More than words - Collaborative tailoring of a word processor. Journal of Universal Computer Science, 7, 9, 826-847. [27] Kerr, J., Chou, M., Ellis, R., Kelleher, C. 2013. Setting the scene: Scaffolding stories to benefit middle school students learning to program, IEEE Symposium on Visual Languages and Human-Centric Computing, (San Jose, CA, Sept. 2013), 95-98.
[14] Costabile, M., Mussio, P., Provenza, L., Piccinno, A. 2008. End users as unwitting software developers. ICSE’08 Workshop on End-User Software Engineering. (Leipzig, Germany, May 2008). 6-10. [15] Costabile, M., Mussio P., Provenza L., and Piccinno A. 2009. Supporting end users to be co-designers of their tools. 2nd International Symposium on End-User Development (LNCS 5435), (Siegen, Germany, Springer-Verlag, Berlin Heidelberg, March 2009), 70-85.
[28] Kim, S., Zimmermann, T., Whitehead, J., and Zeller, A. 2007. Predicting faults from cached history, International Conference on Software Engineering, 489-498
[16] Cunha, J., Fernandes, J., Mendes, J., Pacheco, H., Saraiva, J. 2012. Bidirectional transformation of model-driven spreadsheets, 5th International Conference on Model Transformation (ICMT ‘12), 105-120.
[29] Ko, A., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., Rosson, M., Rothermel, G., Shaw, M., Wiedenbeck, S. 2011. The state of the art in end-user software engineering, ACM Computing Surveys, 43, 3 (April 2011), Article 21, 44 pages.
[17] Eagan, J. and Stasko, J. 2008. The Buzz: Supporting user tailorability in awareness applications. ACM Conference on Human Factors in Computing Systems, (Florence, Italy, April 2008), 1729-1738.
[30] Ko, A. and Myers, B. 2005. A framework and methodology for studying the causes of software errors in programming systems, Journal of Visual Languages and Computing 16, 1 (Feb. 2005), 41-84.
[18] Erwig M., Abraham R., Cooperstein I., Kollmansberger S. 2005. Automatic generation and maintenance of correct spreadsheets. International Conference on Software Engineering (St. Louis, Missouri, May 2005), 136-145.
[31] Ko, A. and Myers, B. 2004. Designing the Whyline: A debugging interface for asking questions about program behavior, ACM Conference on Human Factors in Computing Systems (Vienna, Austria, April 2004), 151-158.
[19] European Spreadsheet Risk Interest Group, EuSpRIG horror stories: Spreadsheet mistakes news stories, http://www.eusprig.org/horror-stories.htm (accessed Dec. 8, 2013).
[32] Ko A. and Myers, B. 2008. Debugging reinvented: Asking and answering why and why not questions about program behavior, International Conference on Software Engineering (Leipzig, Germany, May 2008), 301-310.
[20] Fischer, G. and Giaccardi, E. 2006. Meta-design: A framework for the future of end user development. In H. Lieberman, F. Paternò, & V. Wulf (Eds.), End User Development (Kluwer Academic Publishers, Dordrecht, The Netherlands), 427-457.
[33] Kuttal, S., Sarma, A., Rothermel, G. 2013. Predator behavior in the wild web world of bugs: An information foraging theory perspective, IEEE Symposium on Visual Languages and Human-Centric Computing (San Jose, CA, Sept. 2013), 59-66.
[21] Fisher, M., Rothermel, G., Creelan T. and Burnett, M. 2006. Scaling a dataflow testing methodology to the multiparadigm world of commercial spreadsheets, IEEE International Symposium on Software Reliability Engineering, (Raleigh, NC, November 2006), 13-22.
[34] Lee, B., Kumar, S., Brafman, R., Klemmer, S. 2010. Designing with interactive example galleries, ACM Conference on Human Factors in Computing Systems (Atlanta, GA, Apr. 2010), 2257-2268. [35] Lee, M. and Ko, A. 2012. Investigating the role of purposeful goals on novices’ engagement in a programming game. IEEE Symposium on Visual Languages and Human-Centric Computing, 163-166.
[22] Fleming, S., Scaffidi, C., Piorkowski, D., Burnett, M., Bellamy, R., Lawrance, J., Kwan, I. 2013. An information foraging theory perspective on tools for debugging, refactoring, and reuse tasks, ACM Transactions on Software Engineering and Methodology, 22, 2, (March 2013), Article 14, 41 pages.
[36] Li, S., Xie, T., Tillmann, N. 2013. A comprehensive field study of end-user programming on mobile devices, IEEE Symposium on Visual Languages and Human-Centric Computing (San Jose, CA, Sept. 2013), 43-50.
[23] Groce, A., Kulesza, T., Zhang, C., Shamasunder, S., Burnett, M., Wong, W., Stumpf, S., Das, S., Shinsel, A., Bice, F., McIntosh, K. (to appear). You are the only possible oracle: Effective test selection for end users of interactive machine learning systems, IEEE Transactions on Software Engineering (to appear).
[37] Lieberman H., Paterno F. and Wulf V. (Eds). 2006. EndUser Development. (Kluwer / Springer. 2006). [38] Lin, J., Wong, J., Nichols, J., Cypher, A., Lau, T. 2009. Enduser programming of mashups with Vegemite, ACM International Conference on Intelligent User Interfaces (Sanibel Island, Florida, Feb. 2009), 97-106.
210
[39] Nardi B. A Small Matter of Programming: Perspectives on End User Computing. 1993. (Cambridge, MA: The MIT Press).
[53] Scaffidi, C., Myers, B., and Shaw, M. 2008. Topes: Reusable abstractions for validating data. International Conference on Software Engineering, 1-10.
[40] Oxford Dictionary, http://www.oxforddictionaries.com/us/definition/american_e nglish/silo (accessed Dec. 9, 2013).
[54] Scaffidi, C., Shaw, M., and Myers, B. 2005. Estimating the numbers of end users and end user programmers. IEEE Symposium on Visual Languages and Human-Centric Computing, 207-214.
[41] Ozenc, F., Kim, M., Zimmerman, J., Oney, S., Myers B. 2010. How to support designers in getting hold of the immaterial material of software, ACM Conference on Human Factors in Computing Systems (Atlanta, GA., April 2010), 2513-2522.
[55] Schön, D. 1983. The Reflective Practitioner: How Professionals Think In Action. (Basic Books, New York, NY). [56] Segal, J. 2007. Some problems of professional end user developers, IEEE Symposium on Visual Languages and Human-Centric Computing, (Couer d’Alene, Idaho, Sept. 2007), 111-118.
[42] Petre M. and Blackwell, A. 2007. Children as unwitting enduser programmers, IEEE Symposium on Visual Languages and Human-Centric Computing (Couer d’Alene, Idaho, Sept. 2007), 239-242.
[57] Shaw, M. 2004. Avoiding costly errors in your spreadsheets. Contractor’s Management Report 11, 2-4.
[43] Pirolli, P. and Card, S. 1999. Information foraging. Psych. Rev. 106, 4, 643–675.
[58] Stolee, K. and Elbaum, S. 2013. Identification, impact, and refactoring of smells in pipe-like web mashups, IEEE Transactions on Software Engineering 39, 12 (Dec. 2013), 1654-1679.
[44] Posnett, D., D’Souza, R., Devanbu, P., Filkov, V. 2013. Dual ecological measures of focus in software development, International Conference on Software Engineering, 452-461. [45] Raz O., Koopman P., Shaw M. 2002. Semantic anomaly detection in online data sources. International Conference on Software Engineering (Orlando, Florida, May 2002), 302312.
[59] Subrahmaniyan, N., Burnett, M. and Bogart, C. 2008. Software visualization for end-user programmers: Trial period obstacles, ACM Symposium on Software Visualization (Herrsching am Ammersee, Germany, September 2008), 135-144.
[46] Repenning, A. 2011. Programming goes back to school, Communications of the ACM, 55, 5 (May 2012), 38-40.
[60] Tillmann, N., de Halleux, J., Xie, T., Gulwani, S., Bishop, J. 2013. Teaching and learning programming and software engineering via interactive gaming, International Conference on Software Engineering (San Francisco, CA, May 2013), 1117-1126.
[47] Rode, J., Rosson, M., Perez-Quinones, M. 2006. End user development of web applications, End User Development (Springer), 161-182.
[48] Rosson, M., Sinha H., Bhattacharya, M., Zhao, D. 2007. Design planning in end-user web development. IEEE Symposium on Visual Languages and Human-Centric Computing, 189-196.
[61] Trigg, R. and Bødker, S. 1994. From implementation to design: Tailoring and the emergence of systematization. ACM Conference on Computer Supported Cooperative Work, (Chapel Hill, North Carolina, October 1994), 45-54.
[49] Rothermel, G., Burnett, M., Li, L., Dupuis, C. and Sheretov, A. 2001. A methodology for testing spreadsheets, ACM Trans. Software Engineering and Methodology 10, 1 (Jan. 2001), 110-147.
[62] Walia G. and Carver, J. 2009. A systematic literature review to identify and classify software requirement errors, Information and Software Technology 51. 1087–1109. [63] Wiedenbeck, S. 2005. Facilitators and inhibitors of end-user development by teachers in a school environment. IEEE Symposium on Visual Languages and Human-Centric Computing, 215–222.
[50] Ruthruff, J., Prabhakararao, S., Reichwein, J., Cook, C., Creswick, E. and Burnett, M. 2005. Interactive, visual fault localization support for end-user programmers, Journal of Visual Languages and Computing, 16, 1-2 (February/April 2005), 3-40.
[64] Wilson, A., Burnett, M., Beckwith, L., Granatir, O., Casburn, L., Cook, C., Durham, M. and Rothermel, G. 2003. Harnessing curiosity to increase correctness in end-user programming, ACM Conference on Human Factors in Computing Systems, (Ft. Lauderdale, FL, April 2003), 305312.
[51] Scaffidi, C., Ko, A., Myers, B., and Shaw, M. 2006. Dimensions characterizing programming feature usage by information workers, IEEE Symposium on Visual Languages and Human-Centric Computing, 59-64. [52] Scaffidi, C., Myers, B., and Shaw, M. 2006. Challenges, motivations, and success factors in the creation of Hurricane Katrina “person locator” web sites, Psych. Programming Interest Group Workshop (Sept. 2006), 9 pages.
[65] Xie, J., Chu, V., Lipford, H. and Melton, J. 2011. ASIDE: IDE support for web application security, ACM Annual Computer Security Applications Conference (Dec. 2011). 267-276.
211
