from the software engineering literature. Common ... Traditionally, software engineering research has sought ... Here is an outline of the topics covered in the.
Empirical Methods in Software Engineering Research Walter F. Tichy and Frank Padberg University of Karlsruhe, Germany {tichy | padberg}@ ira.uka.de
Abstract Over the past decade, empirical methods have gained acceptance for validating tools and methods in software research. This tutorial aims at acquainting participants with the main methods used in empirical work in software research, enabling them to evaluate empirical results for validity as well as providing the basis for carrying out empirical studies. A wide range of empirical approaches are covered, including case studies, experiments, field studies, and surveys. These approaches are introduced by prominent examples from the software engineering literature. Common pitfalls are pointed out. Participants also critique empirical papers in small discussion groups, based on worksheets with prepared questions.
1. Motivation Traditionally, software engineering research has sought methods and tools that improve software development and maintenance. In the past decade, the field has been broadened to include the empirical paradigm as a means for assessing and comparing software tools and techniques, for understanding the processes that go on in software development, and for informing theories about software and software development. Research papers now often include results of trials on multiple open source projects, analyze various properties of large software repositories, or compare tools and techniques in field studies or experiments. This situation is a marked improvement over the situation documented by one of the presenters in 1995 [1]. With this broadening of software engineering research comes the need for a better understanding of the empirical paradigm by practitioners and researchers alike. How are empirical results interpreted? When can one trust the results of empirical work? How are empirical studies conducted? Which type of study should one perform in which situation? What are the
limits of empirical methods and what are the pitfalls to look out for? Which role does statistics play? These are the questions this half-day tutorial will address.
2. Tutorial Goal The goal of the tutorial is to enable the participants to understand and evaluate empirical research in software engineering, and to provide the basis for performing their own empirical studies. Participants will be able to define, compare and contrast the main classes of empirical approaches, namely case studies, experiments, field studies, correlation studies, surveys, ethnographic studies, and meta-studies. They will learn how to identify potential weaknesses and assess the validity of empirical results. They will see empirical methods “in action” in the context of concrete papers from the software engineering literature. The intended audience are practitioners and researchers in software engineering. A general reference for the tutorial is the book on Experimental Methodology by Christensen [2].
3. Presenter Background The tutorial builds on our own empirical research, our previous tutorials [3], and our graduate-level course on empirical software engineering. We have taught the university course for the past five years at the University of Karlsruhe and the University of Clausthal.
4. Tutorial Outline The tutorial is divided into three parts. Each part takes roughly 1 hour. In the first part, we present the basics of empirical methodology in the context of software engineering. The second part consists of break-out sessions in which the participants review empirical papers under our guidance. In the third part, we moderate a discussion among all participants about the
methods and results, strengths and weaknesses of these papers.
4.1. Presentation: Empirical methodology We use slides and illustrative sample studies to present key empirical concepts, such as the concept of “validity” of an empirical study. The focus is on concepts and methods needed when carrying out or assessing an empirical study in software engineering [4] [5]. Here is an outline of the topics covered in the first part of the tutorial:
Role of empirical studies in software engineering Surveys Case studies Pros and cons of various types of empirical study Validity of empirical results Typical threats to validity Experimental designs for achieving validity Controlled experiments Basic methods for data analysis Setting up and carrying out an experimental plan Quality criteria for empirical studies.
4.2. Break-out sessions: Selected papers In the second part of the tutorial, the participants review and discuss selected empirical research papers in small groups. Each group works on a single paper. For each paper, we hand out a worksheet containing a list of questions about the paper. The questions make it easier for the participants to get started and to assess the paper from an empirical viewpoint. We provide constant face-to-face support during the break-out session. We offer papers about different topics in software engineering; every participant should find a paper of interest. The papers include good ones and flawed ones. Our selection is based on a paper’s utility for illustrating important aspects of empirical studies in software engineering. Some papers show how to prepare, conduct, and evaluate the result of an empirical study in an exemplary manner; others illustrate what can go wrong and how to avoid the problems.
4.3. Discussion: Assessment of the papers In the third part of the tutorial, each group of participants presents the result of their break-out session to the other groups. All participants discuss
each paper from an empirical viewpoint. The focus is on discussing the experimental setup and results of the papers; identifying both strengths and weaknesses in the experimental design and data analysis; assessing the validity of the results and the conclusions drawn by the authors. We moderate the discussions. All papers studied in the break-out sessions will be discussed at roughly equal length. The goal of the discussion is to further increase the participants’ knowledge of empirical research methods and to strengthen their self-confidence when it comes to evaluating empirical work in software engineering.
5. References [1] W. Tichy, P. Lukowicz, L. Prechelt, and E. Heinz: "Experimental Evaluation in Computer Science: A Quantitative Study", Journal of Systems and Software 15 (1995) 1-18 [2] L. Christensen, Experimental Methodology, Allyn & Bacon, 2001 [3] F. Padberg, “Understanding Empirical Research Results About Extreme Programming”, Tutorial at PROFES 2005 [4] B. Kitchenham, S. Pfleeger, L. Pickard, P. Jones, D. Hoaglin, K. ElEmam, and J. Rosenberg, “Preliminary guidelines for empirical research in software engineering”, Transactions on Software Engineering 28 (2002) 721-734 [5] W. Tichy, “Hints for reviewing empirical work in software engineering”, Empirical Software Engineering 5 (2000) 309-312
