Common Ground: For a Sociology of Code

Common Ground: For a Sociology of Code Hamid Ekbia and Les Gasser (Submitted to Information Technology and People, special issue on Social Theory)

Abstract Software development is a highly technical, but also exceedingly collective, process. A thorough understanding of this process might, therefore, benefit from a sociological analysis of actors, artifacts, and activities. This was the premise of a set of studies that we conducted on development processes of Free/Open Source Software (F/OSS). In this narrative essay, we provide a reflective account of those studies, focusing on the various ways in which social theory contributed to the inquiry. In particular, we illustrate how a range of ideas, theories, and insights from sociologies of art, science, and economics helped us, respectively, make sense of our vast amount of data, develop novel conceptualizations of software artifacts, and understand the character of collective activities involved in F/OSS. Based on these reflections, we draw lessons on how these ideas can be expanded to other computational artifacts and activities.

Prelude “They don’t listen,” bemoaned a computer science colleague, who felt that his ideas went unnoticed, unappreciated, and simply ignored by the geographers who attended the two-day NSF workshop. Ironically, he is a computer scientist who works on social policy, and the geographers are Geographical Information Systems (GIS) experts with a knack for understanding technology and building computer models — perhaps why they all both invited to the same workshop. The workshop aimed to bridge knowledge across several quite different communities – GIS modelers and social scientists, geographers and computer scientists, academics and businesspeople, Europeans and North Americans, and so forth. With this context in mind, I expressed some sympathy with my colleague’s lament, I responded, “You’re right," but added, "and neither do we.” In a sense, this essay is a reflection on such exchanges. In it, we report on what can happen if and when computer scientists, IS researchers, and IT professionals — we — converse deeply with social scientists and take social theory seriously. We write about important gaps we found in techno-centric accounts of hard technology problems, specifically large software systems: how they come about, and how they are managed over long periods of time. We expose our routes to discovering, adopting, integrating, and eventually extending these social theories as rich and, we think, essential underpinnings, for understanding these complex technical artifacts, complementary to purely technical theory and practice. Over decades, computer science has developed a particular disciplinary understanding of its core concepts, which in the realm of software system development (the subject matter of the present inquiry) include notions such as “software,” “requirement,” “design,” “test” “module,” “specification,” “bug,” “patch,” and so on. As constructs, concepts like these manifest a high degree of persistence, having been sustained and linked together over years of mainly unquestioned application in the ontological machinery of the computing world. As Smith (1996) has argued, however, these concepts and theories do not do justice to computation in the wild: “that eruptive body of practices, techniques, networks, machines, and behavior that has so palpably revolutionized late-twentieth-century life” (and after!) In this essay, we want to demonstrate how social theory has helped us develop concepts and understandings that may do greater justice to computational practice. By "do greater justice" here we mean that in the end we

expect to provide added structure to several observable, examined, data-founded paradoxes central to some existing conceptual machinery of the theory and practice of large-scale systems development. We will try to place a more solid social theory-derived armature inside some floating islands of abstraction circulating commonly in the computing world. By exploring the uncharted territory where computing and social theory interact, we hope to map out a wider area of common ground. An Engineering Question Sticks its Head Out Computer software development is a multi-billion-dollar industry that is central to both the basic functioning and the international economy of the modern world. Yet software development is a tedious, costly, and complex process. Surprisingly, as a whole, current development practice still results in expensive, untimely, and unreliable products. For nearly half a century, software engineers have been trying to improve these circumstances by developing tools, techniques, models, and procedures that streamline development processes, optimize resources, and reduce expenses. Compared to the magnitude of these improvement efforts, however, the relative gains have been minimal. According to several sources of statistics circa 2006, roughly 25% of software projects run over budget, another 25% do not finish on time, and almost 45% get cancelled altogether. This leaves us with a basic question: Why does software development is so inefficient, expensive, and unreliable? A Sociological Question Emerges Beyond the cognitive, skill-based, and tool-supported practices of individual designers and programmers, software development is principally a collective activity. Software systems both large and small are designed, built, used, and maintained in a distributed way by groups and communities involving small numbers to thousands of contributors and tens of millions of users. Nowhere has this become more visible than through the rapid rise of "free/open-source software" (F/OSS) development communities. F/OSS projects range from simple programs (single function, hundreds of lines of code, rarely changing) to the extremely complex suites whose components change constantly and asynchronously (full operating systems with suites of affiliated applications and system libraries; complete office-processing, web-management/browsing, and program-development tool suites; high-performance analysis tools for physics, astronomy, and biology; enterprise resource management (ERP) systems; and more). These efforts represent active distributed collectives of designers, developers, maintainers, and "users", both organizational and individual. As one indication of some of the social complexity involved, the New York Times recently reported that the Mozilla Firefox web browser project has 75 to 100 million users, while "Some 1,000 to 2,000 people have contributed code to Firefox, according to the Mozilla Foundation. An estimated 10,000 people act as testers for the program, and an estimated 80,000 help spread the word. Similarly, by 2007, some 1,682 people have reported bugs against the Fedora Core 6 Linux operating system project. These observations suggest that one big part of the answer to the engineering question posed above may indeed lie in the social organization of software development. This point is taken quite seriously by many F/OSS proponents. Based on credos and principles distinct from those of the commercial software, F/OSS proponents attribute its success to the social processes of software development that allow the production of artifacts which are purportedly faster, cheaper, and more reliable. In regards to reliability, for instance, the proponents of F/OSS claim that the active participation of multitudes of volunteers brings about a high degree of quality assurance that is not attainable in commercial software. This claim is best expressed in what has come to be known as Linus’s Theorem: "Given enough eyeballs, all bugs are shallow." Whether we agree

with claims like this or not, the F/OSS model has provided a viable alternative to the rival proprietary model by the sheer fact that it “works.” In so doing, F/OSS shifts the original engineering question towards a sociological one: How can large distributed collectives of experts and nonexperts produce artifacts that are comparable in quality to those created by conventionally-organized teams of experts? Recognizing the sociological nature of this question, several years ago we began a study of the social processes of problem management in software developed by an open source community centered around what has now become the Mozilla Foundation. This software, the original "Mozilla application suite," comprised the Mozilla web browser, an associated email/news reader, and a web page composer. Work on it was supported by several development tools, most notably for our studies the problem-reporting and tracking database tool called Bugzilla (see Figure 1).1 Since the 1980s large software projects have used a widening array of tools to ease the management of knowledge for software development and maintenance. Modern tools such as mailing lists, wikis, blogs, bug reporting/analysis databases, and workflow tools have been integrated into software processes in both F/OSS and commercial development practices for many years now. These systems serve as media for representing ideas, designing, discussing, planning, developing strategy, coordinating, fighting, storytelling, and humor. They embody fullfledged social and organizational environments. Our goal in studying Bugzilla was to understand the social processes involved in fixing bugs and, more generally, in the development of software. We were working with hypotheses that linked variation in social processes (such as seeking/providing information, negotiating, resolving conflicts, establishing priorities, selecting lines of attack on problems, and making evaluations) to variation in outcomes (such as time to repair problems, quality of problem solutions, long-term robustness of repairs, and participant reputations). As we worked toward a clear, empiricallygrounded model of social processes in software maintenance, our first task was naturally to examine the Bugzilla data to find instances of bugs and of social processes, map their associations, and establish correspondences, and find correlations. But how? The “data” involved here consists of hundreds of thousands of "bug reports" arranged in a searchable repository structure. Figure 1 presents a general view of the repository structure and its data. Like all formalized problem-tracking repositories that we have examined, Bugzilla contains at least three kinds of data. First is formalized, sometimes even quantitative data such as timelines for tracking activities, status indicators, controlled vocabularies for specifying issue attributes (e.g. release version number, relevant operating system), lists of people involved in an issue, votes for importance, module in which the problem appears, and so on (See Figure 3 for an example). The second type of data includes informal information such as descriptions of issues and discussion of workarounds and repair strategies. The third kind of data is an evolving set of links and markers that denotes how the various ongoing streams of comments, activity records, and formal data in each bug report relate to similar information in other reports: which problems are interdependent, which are duplicates, which contain relevant information, and so on. This information is captured via textual references in the comments, and in formal fields such as "Depends on" and "Blocks" (See Figure 1). As of October 2007, the Bugzilla repository contains over 400,000 numbered reports, each containing dozens of comments (about 10 comments each on average) along with uploaded code patches and other multimedia content. This amounts to thousands of pages of informal discourse. As our aim was to link social process structures to outcomes (see above), we first needed methods to identify social processes within some comparable units of analysis. Units of analysis we considered included 1) bug reports, 2) sets of linked reports (which we called "bug report 1

At this time, some five years after the initiation of our study, both the original software collection and the parent organization itself have been significantly augmented and reorganized.

networks", individual comments, and related comment collections (since some bug reports contain literally hundreds of comments on different, distinct subtopics.). We needed a method that, for each type of social process considered, would partition the entire set of analytical units into two subsets: those that contained instances of the social process in question, and those that did not. Briefly, to do this we considered and attempted five methods: 1) straight qualitative human coding using grounded theory methods; 2) straight human-based coding using discourse analysis methods; 3) wholly automated extraction using computational text analysis; 4) wholly automated extraction using computational extraction of speech acts; 5) "computational amplification" of human coded data using machine learning methods to extend sets of human-generated phrasal signifiers. Our goals were lofty: to create and apply tools that could recognize social processes in general (social process types of interest could be specified as input) and that could handle huge collections of informal text data. In short, this proved impossible. Again, briefly: the informal Bugzilla text data is very "noisy" and much of the interpretive knowledge essential for automated methods is actually not in the text itself, but in hard-to-capture context; and the quality, scope, and specialized technical nature of the discourses precluded comprehensive human analysis at a representative scale. In short, using the (currently limited) best practices of these methods with our analysis resources that were severely limited relative to the scale of the problem, we couldn't squeeze out enough theoretically powerful sauce. With this scale and difficulty, we needed a way to become more theoretically efficient - we needed to be able to focus our analysis resources on a specific set of social practices under a specific analytical model - but such a model was just what we were trying to build. This began to feel like a chicken-and-egg problem; where could we start? Having attempted these well-established techniques of computational and human-based content/discourse analysis with limited success, we made a fortuitous move. We happened upon it quite indirectly and without looking explicitly: by browsing the websites of some old sociological friends, mentors, and acquaintances. We found our first theoretical muse in an unlikely source of inspiration: the sociology of art. Art to the Rescue Our inspiration came from a short article by Howard Becker titled "New Directions in the Sociology of Art" [Becker, 2003]. Although Becker’s work was known to us from the past, most intimately through his book Art Worlds, there was something about this article that made it very appealing. Not only was it short, concise, and intuitively simple, it seemed generalizable. Becker’s core insight is that a sociological study of a work of art (and, for that matter, a genre, style, or an entire art world) involves research over two spans of time — "upstream" and "downstream"— linked by the artwork itself. An upstream investigation studies how the work comes into being, and a downstream investigation follows the work after it is done. Where you place the work itself in this timeline — when it is "done" — is roughly the point when the work "leaves the hands" of its “creator.” This is a rather arbitrary point because, by and large, the analyst decides when it is appropriate to consider the work “done.” The juncture constitutes a crucial analytic moment in understanding the history of the work, and different positionings can obviously lead to different accounts. But generally, (upstream) artworks undergo a plastic process of assembly or forming of the physical work itself, followed (downstream) by a relatively stabilized physical existence (See Figure 3). Generally, what is interesting analytically in both the upstream and the downstream processes is a set of crucial moments that Becker explicates as follows. Upstream, the most fruitful episodes for analysis are conflicts. These are the moments when expectations break down. To re-establish order various groups of participants (suppliers, artists, lay people, etc.) need to articulate their positions clearly, so they make arguments, justifications, rationales, economics, and sentiments explicit. Participants' disagreements, quarrels, fights bring their positions and assumptions into visible relief, and Becker points out that the researcher can

obtain insight by paying attention to the specific mechanisms of conflict and resolution involved in this process. The downstream process is also dynamic, due to changes in (for example) physical location, ownership, awareness, interpretation, and relationship to other artworks. These episodes of change are the crucial moments - those at which a work changes hands, a shift in audiences takes place, or the work is disposed of, stored, or promoted. Here, as in science, the fate of what artists make is in the hands of others. We found these ideas very useful in thinking about software development, and pursued them further by applying them to our data from Bugzilla. Our goal was to turn this into a model that would give greater detail, more concreteness, and stronger explanatory power with regard to the “social processes” of software development. It was thus that the hourglass model was born. A Model Is Born Standard accounts in software engineering describe software development as a multistage process that consists of distinct periods such as analysis, design, coding, testing, delivery, and maintenance. In the classical waterfall model, for instance, these stages are depicted in a linear succession of distinct, well-defined, and self-contained set of activities (see Figure 2). A design phase follows analysis of system requirements, and design is in turn followed by implementation (coding). Each activity involves its own set of skills, techniques, and professional background — for example, those skilled at analysis may not be the best at testing or coding. The mismatch between this ideal model and real practice has given rise to several refined variations, where the boundaries between successive stages are permeable, each stage gets feedback from the ones succeeding it, and various activities such as analysis, design, and coding are spread out as “workflows” throughout all stages. For example, as a design phase proceeds developers may find it necessary to revisit analysis as they recognize overlooked dimensions and underspecified details. Although greatly refined compared to the classical model, these variations remain committed to a particular conception of software – namely, that of an engineered artifact that starts as a set of ideas or "requirements" held by users, programmers, or market analysts. In a formal software process, the concrete, labor-intensive, and highly skilled work of analysts reifies these ideas into specifications — physical artifacts that reflect the original ideas and their refinements. In the official story, designers and programmers interpret these specifications to create computer codes in specific (collections of) programming and "scripting" languages. These programs, again concretized as physical objects (compilable program files) and again representing the functionality captured in specifications, are converted to runnable form by mechanical tools (compilers, linkers, etc.).2 In the end, modern programs are assembled from many parts, some written directly by programmers to meet specifications and some linked and assembled in from pre-existing code libraries, often cascaded in chains of interdependency. Further, in recent web-based applications, the links and libraries used by programs are only assembled when a specific part of the program is actually executed, in a "just-in-time" assembly process that allows many dynamic, use-driven and data-dependent contextual dependencies and adaptations. Finally, the conventional story goes, the software is packaged, delivered, adopted and put into use by users in specific contexts for taking specific, concrete, local actions that more or less conform to the actions envisioned in the specification process. In fact, an essential requirement for this model is that the successive representations made through the software life cycle — the oscillating transformations from user 2

Compare the reliability of the compilation process as a direct, meaning-preserving translation of programs from one form into another, to the assumption of veracity in photographs due to their creation by automated mechanical apparatuses whose function is presumed reproducible.

needs to reified requirements, from requirements through interpretations to designs and code, from written codes to compiled, linked, assembled delivered products, and from packaged products through products-in-use reflecting actual specific behaviors, are "meaning-preserving." The late-phase delivered products must accurately reflect the early-stage goals, assumptions, and specifications. Many of the cost, time, and quality liabilities of modern software development spring from the need to sustain the accuracy of these transformations. Many development tools and practices explicitly aim at increasing the reliability of this correspondence in the face of very complex, distributed development processes with many stakeholders and contingencies both internal and external. In short, software, in this conception, is a reified object with a lifecycle that goes through phases of birth, adoption, and death. The eventual destiny of the software is by and large in the hands of designers (upstream) and, to some extent, business people and the market (downstream), each of which passes the artifact through its own filter. The survival of the best and the fittest software is guaranteed in a kind of Darwinian process. For more productive development processes that result in higher quality software and better fits to users' needs, it is essential to understand the unfolding social processes of design, development, and maintenance shape that how designs and specifications get created, how systems are implemented, and how problems or "bugs" are managed and repaired, and how (and whether) the goal-specification-design-product-use correspondence is maintained. There are many instances and models for the social processes of software development: the family of F/OSS models and the family of commercial models (Extreme/agile programming, closed-source, etc.) are two. Many of these models suggest that it is possible to involve some categories of users in software specification and quality assurance processes, again to maintain a closer correspondence between goals and delivered products. This user involvement is said to speed up (compress) design, development and maintenance cycles, to increase user buy-in and commitment to projects, and to expand usage over time. The insights provided by Becker's upstream/downstream concept lead us to a rather different model, one that exposes three new aspects of software production. First, "upstream" we see an unfolding collection of interlinked dramas that influence each other in a dynamic and somewhat unpredictable way, instead of a linearized series of stages. Second, "downstream," after software has been released, we find the emergence of a large ensemble of new behaviors occasioned by the post-creation activities of ownership, application, and engagement. These include very large collections of specific micro-analyses of software behavior in specific contexts, coupled with huge numbers of tiny re-specifications of small, localized parts or attributes of software. Third, we find that unlike a physical artwork, the physical manifestation of the software artifact — its actual code — moves away from the center of attention and becomes a side effect of other activities that we label below as "justification," "qualification," and "binding." Taken together, the insights that come from viewing the F/OSS software development model through the lens of the Becker-esque hourglass change our view of the practices and on-the-ground aims of distributed software construction. First, the focus on the upstream cascades of debates conceptualizes software process as an accumulation of locally-situated knowledge that is clearly historical and clearly contingent. It is not meaning-preserving in any meaningful sense, because there is no objectively specifiable meaning to be preserved. The roles of upstream specifications, interpretations, and reformulations are that they capture and codify debates, fixing them as contexts for later debates and in the process giving a historical structure to software. Second, the downstream focus on detailed and locally-specific micro-analysis activities changes the conception of a software "life cycle" away from one in which a governing meaning is reasserted again and again through documentation, help files, training, and "maintenance" processes (e.g. "bug fixing"). Instead, we find an ongoing re-imagining of the roles and

expectations of the artifact as new audiences and experiences engage it and as the network of representations and social interactions about the software is grows and is codified through tools such as Bugzilla. Third, and perhaps most interestingly, we also see some phenomena that center on the solidity, durability, persistence, and representativeness of the software artifact itself. These lead us in the end to extend the Becker hourglass model, as we note in more detail below. But briefly, here, instead of an hourglass whose neck is punctuated by the release of an object to the world — artwork or software — in the "F/OSS and Bugzilla" world we find an ongoing cycle of continuous redesign and re-application without end. The differentiation between upstream, downstream, and artifact seem, at least for software, to be differences in degree, but not in kind. Crucially, the collective means used to carry out the debates, micro-analyses, and re-specification and their voluminous attached "articulation work" are representational: they are computer mediated database, tracing, analysis, and communication tools. This means that, along the complex software ensemble that is under design and review, the objects of debate themselves (software behaviors, bugs, causes, values, patches) are also subject to (upstream) debate and (downstream) re-interpretation. We believe this cascaded reflexivity exhibits patterns and reflects underlying principles. To give a sense of these things and how they play out on the stage of the hourglass model we next need to focus on a particular trajectory of bug analysis, debate, and repair as represented in Bugzilla, and to examine closely the activities that are involved. Some details emerge We knew, rather intuitively, that there is more to software than just code. Our previous studies had revealed some of the other aspects of software that play into its development — e.g., value conflicts, the behavior of the software artifact, the (perceived) causal relationships between code and behavior, and so on. We had also learned from sociology that a great deal of people’s activities is devoted to making connections, building bridges, and mobilizing allies. Based on these preliminary insights, we engaged in a careful study of a particular bug report, known by its number as “BR28586,” and performed a color-coding of it along the dimensions of values (red), behavior (turquoise), connections (yellow), and code (blue). BR28586 was an interesting case because it dealt with an apparently simple issue — namely, how the browser should respond to connection failures: with a dialogue box, an error page, or with some other mechanism. However, despite its elusive simplicity, the bug had remained unresolved for more than seven years. This was amazing, given that seemingly more complex bugs get resolved in much shorter time spans. This preliminary effort revealed an interesting pattern. Scrolling through the color-coded bug report, one could easily discern a change in colors: a high density of red during the first two years, blue in the third year, and turquoise afterwards, with yellow spreading evenly throughout. In other words, it had taken two years for people to arrive at some level of consensus as to whether BR28586 does indeed represent a “bug,” another year to decide how to tackle it, and a few years after that to decide whether the proposed “fix” and the ensuing behavior was in fact satisfactory. In Becker’s language, it had taken a good two years between the initiation of the bug report and the point when the bug was “created,” between the conception of the bug and its birth, so to speak. After that, the bug (still open at the time of this writing) has taken on a life of its own, changing hands among developers, users, Bugzilla’s maintainers, etc. All this was revelatory, and also rewarding because it suggested that Becker’s insight seems to bear out in the realm of software, and that the process of software development does, in fact, manifest an hourglass pattern. However, there are still other questions about bugs and software that the hourglass model fails to address. First, there is the question of the constituent activities of the

process: How does one move from conflicts and arguments to the actions that are coordinated in real practice? How does one account for the way people come to agreement, and finally converge on a “solution,” in particular circumstances? Second, what is the nature of bugs as key players in these activities? How does our reformulation of software development as a social process affect our understanding of bugs? What is a bug in this new conception? These are legitimate questions that concern the engineer, but should also interest the social theorist if the purpose is to attain a shared understanding of processes. Without adequately addressing them, our hourglass model would be rendered superfluous. These questions are closely related, as we will see, but let us tackle them one a time. The Story Comes to Life Thinking about the process, an analogy strongly suggests itself. There seems to be obvious parallels between human diseases and software bugs, as both refer to instances of malfunctioning systems. There might, therefore, be useful lessons to be learned from the analogy. What do doctors typically do when a patient refers to them? Roughly speaking, (i) they examine the patient in search of outward or inward symptoms (this might include tests); (ii) they then try to make sense of the symptoms based on their knowledge of human maladies; and (iii) they implement a treatment. In short, there are three key actions of examination, diagnosis, and treatment. The diagnostic step is the crucial step, where the physician’s knowledge, experience, and expertise are brought to bear. But the individual physician’s contribution is just the tip of the iceberg, because it builds on the historically compiled experience of the medical profession as captured in their mammoth classification systems. The physician can only make an educated diagnosis by drawing on the categories, labels, and infrastructures developed in medicine in the course of centuries. Faced with a patient, they have to shift from an individual dealing with the disease to kinds of individuals dealing with kinds of diseases. In other words, in order to identify a condition as a case of X the physician should essentially classify it as X. They classify in order to identify. How about bugs? What do software developers have at their disposal in their bug-fixing activities? Taking the analogy seriously, it would seem that similar activities should be involved here as well. But close scrutiny reveals important differences. To begin with, the two situations are very different in terms of the process of diagnosis: One is largely at the discretion of a single physician, while the other involves multiple individuals. Secondly, one involves a well-structured classification system at its disposal, and the other does not. Finally, rather than bugs being considered one at a time, thousands of them compete for attention, resources, and solutions. In short, we are facing a situation where thousands of bugs are competing for the attention of dozens of developers who seem to have no shared frame of reference to draw on. Given all these contrasts, the bug-disease analogy seems to break down rather easily. It seems indeed magical that such an arrangement works at all, to the extent that it does. In order to see how the “magic” works, we need to take a closer look at a specific bug. Seven Years in the Life of A Bug To illustrate our findings, we follow here the development of a particular bug, which was initiated in February of 2001 with bug report number 28586 (henceforth BR28586; see Figure 3). This bug has to do with how Mozilla responds to network errors such as, for instance, the loss of connection while loading a webpage. Different browsers respond differently to such a situation, the most common of which is to show either an http error page or a dialog box. At the time of the reporting of the bug, Mozilla did the latter — that is, it showed a dialog saying something like `Mozilla's connection was refused by the server foo.bar.' The person who initiated the bug report

believed that this kind of response could be potentially taxing for the user — for instance, in a scenario where multiple windows are open when connection is lost, and the user would have to click “OK” for all of those pages. Instead of this, the initiator suggests, an error page should appear, not only because it saves a lot of effort in a scenario such as the above, but also because the error page can be used as a “place-holder” for the real page — “so, for example, after browsing elsewhere for a while I can use the Back menu to come back to the error page, and then click Reload to have another go at connecting, without having to type the address (or find a link to click on) again. I can't do that with a dialog” (BR28586: Description). At first glance, this seems to be a straightforward technical matter that could be resolved easily. But seven years later, and more than three hundred “comments” in between, this bug is still active with a status of ASSIGNED at the time of this writing. What is going on? In order to make sense of this situation, we need to take a close look at the comments. Comments deal with various activities of the relevant bug. We followed Becker’s approach outlined above, focusing on conflicts at the beginning. Our study of this bug and many others demonstrates that comments deal with at least three such activities — namely, those that have to do with: 1. the behavior in question, how it was caused, how it should be evaluated, and how it could be “fixed;” 2. the relationship between the bug in question and the rest of the software system, how it relates to other similar or dissimilar bugs and behaviors, how other participants (developers, users, etc.) conceive and judge it, and what implications those relationships have on the life of the bug; 3. the evolution of the bug over time and how it binds to external clocks and schedules. We call these activities, respectively, justification, qualification, and binding of a particular bug. Let us expound these activities by looking at BR28586. First Activity: Justification As the reporter of this RFE, a small note to those on the CC/supporters list: If you want this bug to be fixed, then fix it, or find someone who can. Voting for it won't get it fixed. Spamming the bug report with useless comments about how useful it would be to your particular company/ISP/tiddlywinks club/whatever won't get it fixed. The only thing that will get it fixed is code. Thank you. — The initiator of BR28586 Three months after the initial posting, one of the core developers asks a programmer to verify that the reported behavior is indeed observed in the latest commercial build, and then decides that this bug is “not critical to beta2,” although it is put on “[nsbeta2-] radar (BR28586: Comment #4, 5). This leads to a flurry of discussions as to whether the report is in fact of a bug, a request for enhancement or simply a matter of personal preference (Comments #7, 9, 11, 12), whether users should be given a choice (“preference setting”) between getting a dialog box, an error page, or yet another option such as an error sidebar (Comment #7, 14), whether the modification should also apply to other cases (such as when a requested form is not in the cache) or these cases should be filed as separate bugs (Comment #8, 37, 38), whether the system should respond differentially depending on the context — e.g., with an error page in the case of a main page and with a dialog box otherwise (Comment #13), and so on. There is also some conflict over the right behavioral

“model” for Mozilla — for instance, whether or not the Internet Explorer™ is a good example of a browser (Comments #37, 38, 39, 41). In brief, at this stage of BR28586’s development, there is a great deal of disagreement in terms of the classification of the behavior, possible remedies, and different paths to be followed. This level of uncertainty and conflict is particularly common at the early stages of the lifetime of the bug. In reporting something as a bug, people are essentially putting a description, interpretation, and judgment on their own observation or experience of the software, which might or might not be similar to what others experience and to how they interpret and judge that experience. These differences in experience and interpretation generate a space of uncertainty that needs to be negotiated and hopefully diminished in order for things to move forward. A major step in reducing uncertainty would be to explain the cause of a behavior. “It appears that the IFRAME elements are the culprit,” declares one developer (Comment #25), while others try to link this behavior to other ones (Comment #31). Providing a “solution” in the form of code, pseudo-code, ideas, or even questions is another step in this direction. In a single comment, someone invites action (“OK, so instead of complaining about this bug, let’s design a solution to this”), provides an idea (“Let’s implement an error service”), and poses new clarifying questions (“What should an nslErrorService look like? What methods do we need?”) (Comment #32). As the above examples illustrate, there are disagreements at every step of the way, especially at the beginning. These might have to do with any number of issues: what is the right behavior, feature, or model, what causes the current behavior, what is a good or a bad solution, what needs to be included or excluded, etc. That these are issues of value, and not purely technical, judgment can be discerned by noticing the linguistic signifiers in the content of the comments: “I hate the damn…,” “I’d really like to …,” “It would be really cool if…” The activity of justification, therefore, delineates at least the four components of the software artifact: behavior, cause, code, and value (see Figure 2). Although there is no particular order in how the focus of activities shifts through these components, the value aspect clearly dominates at the early stages of the bug’s lifetime. The sense of frustration expressed in Comment#28 by the initiator of BR28586 (quoted above), almost fifteen months after the report was opened, is an interesting reflection of the ubiquity of conflicts, disagreements, and value judgments. This makes intuitive sense as uncertainty is initially high and people have varying understandings and expectations of the behavior. Gradually, this gives way to the behavior and causal aspects, with the code having to wait until later (see Figure 3: a bar graph of the number of comments dealing with the four components). Second Activity: Qualification I was wondering why this bug is marked as an enhancement and targeted for 1.1alpha. It is a _very_ annoying bug for people with dialup or independable [sic] connections. I got my girlfriend to using Mozilla, and she likes it a lot (no popups and virusses!), but she is ready to go back to IE because of it. — A Mozilla user Justification is important because it provides an effective mechanism of qualification. In their extensive study of the relation between agreement and discord, Boltanski and Thevenot have found qualification as a key operation that people carry out when they construct lasting agreements from initially discordant situations (2006: 25). They characterize qualification as “the operation by which (a) one identifies a particular being by putting it into an equivalency class; (b)

one designates a being as a conventional epithet…; (c)one is inscribing a being within a performative code…; (d) one is referring to a common good. (p. 359, Note 1). In the present case, qualification is manifested in a set of activities aimed at linking a bug report to other bug reports, other systems, other people, and other entities. For to qualify or disqualify a bug, connections with other already qualified and less controversial beings need to be made (Thevenot 2006). Some of these activities are explicitly built into Bugzilla through the “duplicate” and “dependency” mechanisms (see above). These mechanisms essentially pin down a bug report in relation to others. Xxx of the 320 comments in BR28586 simply established such links: “Bug number x has been marked as a duplicate of this bug.” Other means of qualification, however, are more subtle and implicit. Some of them make a link from the current bug to another one: “See also bug 20838, [RFE] Unreachable server should cause cached data to be displayed.” (BR28586: Comment #56); “It’s worth noting that fixing bug 96976 will alleviate one of the most annoying problems of this bug” (Comment #48). Others make a link in the opposite direction: “My guess is that if this bug is fixed, the bug #74987 ‘Specifying an invalid URL …’ is likely to go away” (Comment #27); “Do people think that fixing 111904 would be a good solution if we do not fix this bug?” (C#133). And still others are explicit or implicit pronouncements of making or breaking alliances: “Good idea…” (C#1); “I agree…” (C#2); “And for god sakes, have a network admin design the page” (C#23); “Same as a lot of folks here. This bug is the one that prevent my company to use mozilla. It will solve problems (Bug 33469) of a lot of people, please vote.” (C#24); “Votes now showing: 49. Mine makes 50” (C#101); “Does Netscape marketing not see an opportunity here?” (C#80); “You’d have to talk to marketing, which is not strongly present here…” (C#83); Third Activity: Binding Although every bug report is designated a number that remains constant and identifies the bug uniquely, the bug report itself changes and evolves over time on a continuous basis. This fluid character of bug reports is largely due the electronic environment in which they operate. What is particularly important is that there is no common temporal frame of reference to synchronize the evolution of different bug reports. Rather each bug report runs on its own “clock,” changing at different rates and with different bindings to external schedules and clocks. This kind of asynchronous development creates room for great variability in the resolution time across bug reports (ref?), but it also poses interesting issues and challenges for software developers, which they try to meet by engaging in a third type of activity that we call “binding.” The most common form of binding is to synchronize the development of a specific bug with other constructs such as builds, versions, and releases. Like other aspects of bug fixing, binding also involves negotiation about when the resolution for a bug should occur. In Mozilla, as in most other software projects, coding of new functionality and bug fixing occur continuously and simultaneously. Occasionally the management of the project wishes to release a stable version of the software. These stable versions, or releases, are usually numbered (e.g., 1.0, 1.1, 1.2, 2.0 etc.). In the Mozilla project, these occasions are referred to as ‘milestones.’ One of the fixed fields in Bugzilla is labeled ‘Target Milestone.’ This field in Bugzilla is currently rendered as a drop-down from which a single value from a constrained set of values may be selected. This represents the current estimate of when the fix for a given bug should be supplied. Specific target milestones are often referenced in the text of the bug reports. Sometimes the reference is simple and cursory. Comment #12 in BR # 8075 is simply “M15,” which refers to remarking the bug report as targeted for completion by milestone M15 instead of M14. In BR28586, for instance, many comments explicitly deal with binding. “Update target milestone to 'Future' since this missed the 'mozilla1.1beta' train (C#225);” “This can't make it to mozilla1.2. It

needs to be tested in the alpha release” (C#230). Other times, binding takes the shape of explicit intervention by someone with the right privileges — for instance, trying to “shut down” a bug because it “has gotten a bit too long and convoluted” (C#266). Despite such authoritative interventions, however, the bug has continued to exist for many more years. This comment was issued in September 2003, but BR28568 is still well and alive almost four years later. It has taken on a life of its own. An Analogy Suggests Itself In brief, in the absence of a classification system for bug fixing, the community arrives at a shared understanding, and a solution through the activities of qualification, justification, and binding. This suggests a rather different analogy from the medical diagnosis one. Think of a car race such as Indy500, where dozens of racecars and drivers show up for the tournament. In order to be part of the field on the race day, each one must first qualify. To do this, the car-driver combination should justify their qualifications. Similarly, when bug reports are submitted, the first thing that needs to be done is qualification. As Eugene Garfinkel noticed many years ago, a similar process is at work in grand jury deliberation, where jurors engage in what he describes as retrospective deliberation in order to arrive at a verdict. A bug report is typically disqualified for one of two reasons: it is nonreplicable or there are data issues. The next step is prioritization. Many factors influence this process but the key concerns are user impact, hours estimate, and developer time before next release. Even though many bugs may qualify to be fixed, not all of them will make it to a given release. As in the case of Indy500, when you qualify you are also given a starting position. Since this process happens over time, new bugs may be reported that jump to the top of the priority order – just like a faster car knocking another out of “pole position” late in the day. In any case, the bug should be temporally synchronized with another event such as a version release in order for the process to come to a closure. This activity is what we called binding. Interlude As mentioned earlier, we aimed to uncover relations between bugs and social processes. We thought, that is, we take "bug" as a given type of object, look for instances of those objects, and categorize them. We do the same for social processes, and we relate the two. However, we never actually find bugs — only bug reports taking the form of unfolding discursive assemblages (see Figure 2). Bugs themselves are somewhere else. Where are they? Outside, in the code, in the behavior, in the values, in (relations with/to) the discussion and experience and expectations and tasks and practices of people engaged in these discussions, using multiple versions of the software? This is a major disconnect for us: literature, folk practice, and community discussion on/of software speaks of bugs, but we never find them. We do find what seem to be representations and allusions, but these are active, unstable, and contentious assemblages in their own right. Where are the bugs? In fact, in order to answer this question, we need to answer yet another basic question: What is a bug? (You can only determine where something is if you know what it is that you are looking for!). This is a question for which the hourglass model does not provide much of an answer. The model tells us how the object gets created, it points us to the activities involved in the process, but it does not tell us what the object is. This is a big loophole in the model, and to fix it we need to look elsewhere at what Latour (2005) calls “associology”: a sociology of associations. The Ontological Wall Stands in the Way

But don’t we already know the answer to this question? Don’t we know that the object is a “bug”? Theoretically speaking, the answer to this question is, “No.” What we have is a pretheoretical understanding of bugs, based on many decades of practice by computer professionals and amateurs. To take this to be equivalent to a theory of bugs would amount to what Brian C. Smith (1996) calls an “inscription error: a tendency for a theorist or observer, first, to write or project or impose or inscribe a set of ontological assumptions onto a computational system (…), and then, second, to read those assumptions or their consequences back off the systems, as if that constituted an independent empirical discovery or theoretical result (p.50; original emphasis). Smith traces the origins of this error in the analytic tradition (in philosophy, but also in science), where it is common to analyze semantical systems, such as people or computers, in the following manner: 1. To parse the relevant theoretical situation in advance into a set of objects, properties, types, relations, equivalence classes, etc.; 2. With that ontological parse in hand, to go on and proclaim something as an empirically justified result. To use a dramatic analogy, this would be as if a detective sets up a murder scene in advance, before the commitment of the crime, with all the fingerprint, DNA evidence, and whatnot to prove the defendant guilty. Similarly, in setting up the allegedly innocuous pretheoretical stage, Smith argues, “one is liable, even if unwittingly, to project so many presuppositions, biases, blindnesses, and advance clues about the “answer,” and in general to so thoroughly prefigure the target situation, without either apparent or genuine justification” (p. 16). This, of course, would shut all inquiry down. Smith characterizes his own experience of trying to reflexively inquire computing as “running up against an ontological wall” — where computing is broadly understood as what happens in the daily activities of software developers and users alike — (p.17). Our attempt to understand bugs, bug reports, and bug fixing echoes a very similar experience of running into an ontological wall, where one finds the received formal tradition in computer science encumbered with presuppositions, biases, and advance clues, where bugs are conceived as discernible objects reified in code, and where bug fixing is thought of as a surgical procedure of removing the malignant code and replacing it with a well-behaved one. This is, indeed, an overwhelming analytic experience. How could we penetrate this ontological wall? Has social theory equipped us with the right drilling tools for this? We Find a Tool to Penetrate the Wall Social theory has traditionally not been interested in things, objects, and nonhumans in general. This makes intuitive sense because the social seems to have everything to do with humans and not much to do with nonhumans. Not much, that is, until one looks closely at how human societies are held together, at which point objects start to spring up everywhere. This is a point that actornetwork theorists have taken great pains to bring home for more than two decades now. “Once built, the wall of bricks does not utter a word” Latour muses. “Once they have been filled in, the printed questionnaires remain in the archives forever unconnected with human intentions until they are made alive again by some historian” (2005: 79). Objects, in other words, tend to be silent and to leave no traces. “This is why specific tricks have to be invented to make them talk (ibid; original emphasis). Let us see what types of tools actor-network theory puts at our disposal for doing this.

Latour enumerates a number of occasions where an object’s activity is made easily visible. Among these are: •

Innovative situations — in the artisan’s workshops, the engineer’s design department, and the scientist’s laboratory;

•

Unfamiliar situations — such as those created by distance in time (archaeology), in space (ethnology), or in skills (learning);

•

Problematic situations — accidents, breakdowns, and strikes.

What is common among these situations is that they turn objects from intermediaries into mediators, from entities that only transfer an effect or information to ones that “transform, translate, distort, and modify the meaning or the elements they are supposed to carry” (Latour 2005: 39). As such, these mediators “cannot be counted as just one; they might count for one, for nothing, for several, or for infinity. Their input is never a good predictor of their output; their specificity has to be taken into account every time” (ibid). In Latour’s light, therefore, the kinds of situations that we deal with should easily provide occasions for objects to become mediators, as these situations bring innovation, distance, and breakdowns together in an interesting matrix. Ontological walls, it turns out, are not brick walls. They are more robust and solid, but by the same token more resistant to our analytic provocation — they don’t talk easily. Nor are bugs printed questionnaires that would “remain in the archive forever” — they demand attention, sooner or later. How could we, then, use ANT’s conceptual tools to penetrate the wall and to pay bugs their due respect? The clue is in the hands of the bug reports. Bug Reports Turn Into Mediators At this point in our analysis, we have made interesting headways. As mentioned earlier, problem reports are texts that contain formal information, examples, and unstructured comments. Despite the fact that these texts are called “bugs” and are referred to using formal nametags of the form “Bug ”, it is very clear from the discussion in the reports themselves that these reports are representations of problems. But they are not representations in the sense traditionally understood by the term – namely, bounded and faithful reflections of the phenomenon. Rather, they are partial and context-sensitive representations that are subject to multiple reinterpretations. We say this for various reasons: a) many reports sometimes refer to the same underlying problem; b) a single report may contain information on many problems; c) there are linkages and interdependencies across both problems and reports; and d) there is often considerable uncertainty and contention over the relationships between reports and problem(s) to which they refer. In brief, bug reports are mediators, not actual causes of phenomena, nor mere representations. How about bugs themselves? If bur reports are mediators, what are bugs? Bugs Become Quasi-Objects Having this understanding of bug reports as mediators at our disposal, we are now better equipped to develop a theoretical understanding of bugs. The common conception of bugs as reified in code is too narrow for our purposes — it only applies to the rare instances in software development where a small group of people work on a product with very precise specifications, allowing them to readily identify discrepancies between behavior and specs and trace the cause

down in code. In all other situations, such as those in F/OSS, bugs take on a totally different character. They turn into immanent hybrid entities — partly virtual and partly physical, fairly abstract and fairly concrete, somewhat shared and somewhat contentious — unlike most of the objects that we deal with in the routine activities of our daily lives: chairs, rocks, and billboards. Although, admittedly, the latter could also occasionally acquire some of those properties, but that is not their “normal” way of being. For bugs, it is, as it is for those objects that populate scientific laboratories and that actor-network theorists call quasi-objects. So what is a quasi-object? Latour describes it as something that “is made to act by a large starshaped web of mediators flowing in and out of it… [that] is made to exist by its many ties” (Latour 2005: 217). But, equally important, it is something that is constitutive of practices, participating in activities just as humans do. We have seen this clearly in the case of bugs. The bug associated with BR28586, for instance, is made to exist by its many ties to other entities such as bugs and bug reports (through duplication and dependence relationships), to other software artifacts (versions or releases of Mozilla, Internet Explorer, etc.), to people (developers, users, girlfriends), to organizations (the networking team, competitors, business organizations), and so on. In short, in the absence of a full-fledged classification system of bugs that would allow the identification of BR28586 as a case of X, the bug is pinned down by these multiple ties. The Model Matures It seems, at last, that we have attained a model of software development as a collective process, which is rather distinct from the standard model in software engineering. The difference between the two models derives from both the processes that they invoke and the conceptual apparatus that they apply in describing their artifacts and processes (see Table 1). As such, the models are not mutually exclusive; rather, they afford various understandings. The standard model of software engineering, as we saw, focuses on transformations between stages of developments and on the preservation of meaning across those stages. This model, by and large, excludes human actors from the picture, bringing them in only at the margins. The analysis of bug fixing as a collective process, on the other hand, provides an integrated framework that accounts not only for humans but also for non-human actors such as bugs and bug reports. The picture that emerges from this analysis is, therefore, more comprehensive and hence more realistic.

The Concept Software Bug Bug report Bug fixing Software development

Technical Meaning Code Defect Data source Resolution Transformations (Meaning-preserving)

Socio-technical Meaning Quasi-object Quasi-object Mediator Qualification Qualification (based on justification)

Table 1. Two ways of thinking about software artifacts and processes Postlude What we have done here is, in a way, experimental in character. It is the initial reconceptualization of some of the basic elements of software system development by drawing on social theory. By thinking of bug fixing and software development as collective processes involving activities such as external analysis and internal debate, social theory helped us understand concepts such as bug and software in a different light. What is interesting is that the various components of this analysis come from different places in social theory, each one of which makes a distinct

contribution to the overall picture: roughly speaking, the hourglass model provides the method for making sense of data; sociology of science provides concepts to think about human and nonhuman actors in the language of intermediaries, mediators, and quasi-objects; and economic sociology provides an integrated framework that focuses on activities and mechanisms. Although it would be premature to make generalizations on the basis of what we have offered here, we would like here to extend our findings along three dimensions: i) the extension of the notion of quasi-object to other computational artifacts; ii) the extension of the operations of justification and qualification; iii) the extension of social theory to embrace non-human entities. Quasi-Objects Proliferate Bugs are just one example of computational artifacts that could be understood as quasi-objects. When you put your profile on a social networking site such as MySpace, in a way you create a representation of yourself, your history, tastes, hobbies, loved ones, friends, friends of friends, and so on. But on closer scrutiny this is not a representation, traditionally understood as a stand-in for the real you, as it can be decoupled from you and take on a life of its own. Like characters in a good novel who take on, we are told, a life of their own, dragging the author along with them, your profile becomes an autonomous entity, more or less independent of you. In other words, your profile is a quasi-object — a mutable, unstable, and transient entity the fate of which is in the hands of others who take interest in it and who build bridges between you and their profiles. Similarly, when someone creates an entry in Wikipedia, others who take interest make adjustments, modifications, or even corrections. The process might go on for days, months, or years, until at some point it converges onto a consensus, but there is always a chance for revision. The entry is up for grab as long as there are people who take interest in it. All of these are examples of quasi-objects, we suggest, and could therefore be understood in terms similar to what we have seen with bugs. Electronic environments seem to be increasingly generating quasi-objects of different sorts, populating contemporary life with these entities. Regimes of Justification As we saw earlier, some of what distinguishes quasi-objects from “ordinary” objects of daily life (chairs, rocks, and bottles) is their immanent, hybrid, and transient character. However, as the case of bugs also illustrates, the processes that generate quasi-objects provide another feature that distinguishes them from more stabilized entities. We identified three such generative activities in the case of bugs: justification, qualification, and binding. On close examination, any quasi-object (a bug, a profile, an entry) is discursively constructed through the collective activities of the relevant community through: 1. Justification: A quasi-object goes through trials of strength, during which its values, virtues, and merits are discussed, contested, and negotiated; 2. Qualification: Having passed the test, a quasi-object becomes stabilized through a process of closure; 3. Binding: Having arrived at closure, a quasi-object is bound to more stable and less contested entities. Let us elaborate on this. Collective activity of the kind we observe in software development becomes possible if the involved parties can reach agreement on a number of dimensions: what is the goal, what is the cost, what is at stake, what is relevant, what is the evidence, and so on. Given

that actors (users, developers, volunteers) enter the situation with varying understandings, expectations, and interpretations, on what grounds do they reach agreement? This question becomes the more imperative if there is no pre-existing set of agreed-upon standards, norms, or classification systems to draw on — which seems to be the case in situations such as F/OSS. Broadly speaking, the answer is that people need to make compromises, and that the operations of justification and qualification provide the mechanisms for this. Qualification, as we saw, links an entity of questionable value with others that are less controversial. But qualification, in turn, takes place within different regimes of justification, which could be industrial in nature (efficiency, speed, etc.), but they could also be domestic (a girlfriend using Mozilla), civic (an organization using Mozilla), and so on (Boltanski and Thévenot 2006). This is a major sociological insight that could be invoked in understanding collective processes of design and development. Returning the Favor This work, as we said at the outset, is an exploration of a common ground between social theory and technical work. So far we have elaborated on how the latter can benefit from the former. How about the opposite? Are there lessons for social theory to be drawn from studies such as this? We believe that there is, and would like to briefly discuss this here. Social theory, more often than not, deals with technology from a distance. A brief look at sociological literature reveals that sociologists do not show great interest in the technical (Orlikowski and Iacono 2001). And to the extent that they do, the interest is more in technical people rather than technical processes and artifacts. We believe that this lack of attention is a major deficit for social theory, for two reasons: First, as Latour (2005) and other science studies writers have taken pain to demonstrate, objects and artifacts are social agents in their own right (as our analysis of bugs illustrates). Second, and more importantly, technical work, especially on large scales such as those of F/OSS, provides interesting occasions for collective action. As such, the social study of technical work creates unique opportunities for understanding mechanisms involved in collective action — for example, negotiation, conflict resolution, consensus building, and so on. These two facets of technical work — the artifacts and the mechanisms — are closely related, and while their combination is the source of strength and efficacy of technical work, it could also be a useful source of insight for the social theorist. References Becker, H. (2003). New Directions in Sociology of Art. Boltanski, L. and Thévenot, L. (2006 [1991]). On Justification: Economies of Worth. Translated by C. Porter. Princeton University Press. Latour, B. (2005). Reassembling the Social: An Introduction to Actor-Network Theory. Oxford University Press. Orlikowski, W.J., & Iacono, C.S. (2001). Desperately seeking the "IT" in IT research — A call to theorizing the IT artifact. Information Systems Research, 12(2): 121-124. Smith, B. (1996). On the Origin of Objects. The MIT Press.

Figure 1. Bugzilla repository structure

Analysis Spec Design Implement/Test Release/Use

bug Figure 2. The standard waterfall lifecycle model

Figure 3. The header of a bug report

upstream

conflicts

Quasi-object

transitions

Figure 4. The hourglass model