Experimental Software EngineeRing NETwork
Empirical Methods and Studies in Software Engineering Experiences from ESERNET Reidar Conradi and Alf Inge Wang (Eds.)
Ch.0: About this book Alf Inge Wang and Reidar Conradi (Eds.) Norwegian University of Science and Technology (NTNU), NO-7491 Trondheim, Norway
[email protected] /
[email protected]
The book “Empirical Methods and Studies in Software Engineering – Experiences from ESERNET” is a result of the ESERNET project. ESERNET is a thematic network project in 2001-03 in the European Union’s 5th Framework Programme under contract number IST-2000-28754, see www.esernet.org. It has the ambition, in cooperation with related activities, to gradually change the mentality of software engineers and their organizations towards systematic empirical studies, for the purpose of long-term learning. The overall goal is therefore to collect, systematize and disseminate relevant and valid insight. ESERNET is lead by IESE in Kaiserslautern (Germany). The project responsible is Dieter Rombach, with project manager Christian Bunse. ESERNET has five founding members (contractors): • Blekinge Institute of Technology (BTH), Sweden: Claes Wohlin and Kennet Henningsson. • European Software Insitute (ESI), Spain: Elixabete Ostolaza and Elisa Gallo. • Fraunhofer Institute for Experimental Software Engineering (IESE), Germany: Christian Bunse, Andreas Jedlitschka , Markus Nick, and Holger Westing. • Norwegian University of Science and Technology (NTNU), Norway: Reidar Conradi, Letizia Jaccheri, Tor Stålhane, and Alf Inge Wang. • The Technical Research Centre of Finland; Finnish National Institute of Technology (VTT), Finland: Toni Sandelin and Matias Vierimaa. Sodalia in Trento (Italy) was originally a partner, but decided to withdraw from the ESERNET consortium due to an internal reorganization activity. In addition to the contractors, ESERNET has the following 22 participating members, with contact person indicated: • DaimlerChrysler AG, Germany: Frank Houdek. • DELTA Danish Electronics, Light & Acoustics, Denmark: Jørgen Bøegh. • Engineering Ingegneria Informatica S.p.A., Italy: Stefano de Panfilis. • Fraunhofer Center for Experimental Software Engineering, Maryland (FC-MD), USA: Victor R. Basili. • FZI Forschungszentrum Informatik, Germany: Thomas Genssler. • Lund University, Sweden: Martin Höst. • MARKET MAKER Software AG, Germany: Martin Verlage. • methodpark Software AG, Germany: Christian Knuevener. • Politecnico di Torino, Italy: Maurizio Morisio • PSIPENTA Software Systems GmbH, Germany: Torsten Sander. • Robert Bosch GmbH, Germany: Eberhard Hübner. • Simula Research Laboratory/University of Oslo, Norway: Dag Sjøberg. • Softlab, Germany: Wolfgang Koch. • SOLID Information Technology, Finland: Janne Järvinen.
3
• • • • • • • •
Universidad Politécnica de Madrid, Spain: Natalia Juristo. Università degli Studi dell´Insubria, Italy: Sandro Morasca. Università degli Studi di Bari, Italy: Giuseppe Visaggio. Università degli Studi di Roma "Tor Vergata", Italy: Giovanni Cantone. University of Calgary, Canada: Yingxu Wang. University of Castilla La Mancha, Spain: Mario Piattini. University of Kaiserslautern, Germany: Dieter Rombach and Marcus Ciolkowski. University of Strathclyde, UK: Marc Roper and Murray Wood.
Of the 22 participating members, 8 are industrial and 2 come from outside Europe. Many of the partners and members already work together.
Table of Contents
Ch.1: Introduction Reidar Conradi and Alf Inge Wang (Eds.) Norwegian University of Science and Technology (NTNU), NO-7491 Trondheim, Norway
[email protected] /
[email protected]
1
Motivation
Our societies crucially depend on high-quality software for a large part of their functions and activities. That means that software researchers, managers, and practitioners have to be familiar with what software technologies to use for which purpose. This again means that we have to systematically study and collect evidence on what technologies (UML etc.) will work – or not work – with what effects in which contexts (application areas, project models, societal/technological systems etc.). The field of software engineering and software technology is developing very fast. Perhaps as a consequence, there is seldom enough interest or opportunity for systematic investigation of how the underlying technology will actually perform. That is, we constantly introduce new concepts, methods, techniques and tools – or change existing ones – without much scientific reflection. But how can we learn and improve without systematic studies? Systematic empirical studies in software engineering first of all assume well-proven research methods to plan and carry out investigations, and to collect relevant data. Thereafter comes data analysis – quantitative and/or qualitative – and discussion. Finally comes generalisation for later reuse and learning, and follow-up activities such as changed work practices and dissemination through scientific articles, technical reports and experience bases. In other words, we apply the scientific method [1]! Common study types are controlled experiments, case studies, postmortem analyses, and surveys. Such a study usually starts with some hypotheses to be tested (verified or falsified), based on some previous insight or observations, or some external need. The hypotheses (“goal”) will set the premises for what kind of activities and metrics that are relevant. For instance, if the goal is to improve reliability, we may do a case study as part of an existing project. A suitable development activity as part of the study may be to try out (improved) inspection techniques for e.g. UML design artifacts. A partial metrics for this is the number and type of defects in different phases or subsystems, with data coming from previous studies and the recent one. However, a persistent problem in our field is lack of previous data, i.e. no solid “baseline” to compare with. All this means that there is a need for improved and better qualified empirical methods. But most importantly, there is a need to learn about and to apply such methods in practice, and to spread the results. That is, we should promote an empirical approach to our field of software engineering. The present book aims to contribute to this goal. It has six chapters describing different kinds of empirical studies, and methods for performing such studies. For instance, chapter 2 describes the common study types, and chapter 3 describes the ESERNET “template” for planning,
6
performing, analysing, and reporting empirical studies. The book has seven more chapters where actual studies are described and discussed. These studies mainly deal with inspections, testing, object-oriented techniques, and component-based software engineering.
2 ESERNET, Experimental Software Engineering Research NETwork ESERNET has been a thematic network project in 2001-03 in the European Union’s 5th Framework Programme under contract number IST-2000-28754, see www.esernet.org. It has the ambition, in cooperation with related activities, to gradually change the mentality of software engineers and their organisations towards systematic empirical studies, for the purpose of long-term learning. The overall goal is therefore to collect, systematise and disseminate reliable insight. This type of knowledge will serve as a scientific base for assessing, understanding, changing, innovating, and using software technologies. This task assumes a joint effort between academia, technology providers, software developers, and possibly end-users. One step in this direction is the current ESERNET Method and Experience Book to convey the insights from the ESERNET project. The target audience of the book is software engineers and managers, and corresponding researchers, teachers and students. ESERNET is lead by IESE in Kaiserslautern (Germany). The project responsible is Prof. Dieter Rombach, with project manager Dr. Christian Bunse. ESERNET has p.t. four other founding members: VTT Electronics in Oulu (Finland), NTNU in Trondheim (Norway), Blekinge Institute of Technology or BTH in Ronneby (Sweden), and ESI in Bilbao (Spain). Sodalia in Trento (Italy) was a sixth founding member, but has later withdrawn. Per June 2003, ESERNET has 27 participating members (including the five founding members). Of these, 8 are industrial and 2 come from outside Europe. Many of the partners and members already worked together before the project. ESERNET has actively cooperated with and brought synergy to many other initiatives and projects in software engineering, as illustrated by Fig. 1 below:
SMEs
SW-Intensive Industries
Applied SE Research
SE Education, Training, and Conferences
ESERNET users
ESERNET virtual network organisation
ISERN PROFES, CORONET, VASIE, network PERFECT, CLARIFI ESPINODEs, REBOOT PROFIT ESSI/PIEs
SE initiatives and projects (company-wide, national, European, global)
...
Fig. 1. ESERNET and related initiatives and projects
In Fig. 1, SW means software and SE means Software Engineering. ISERN is the International Software Engineering Research Network, established in 1992, see www.iese.fhg.de/ISERN. PROFES, PERFECT, REBOOT, CORONET, and CLARIFI have been EU-supported software engineering projects in the 1990s (see www.cordis.lu). PROFIT is a Norwegian industrial project on software process improvement (SPI) in 1999-2003 (www.geomatikk.no/profit). VASIE stands for Value Added Software Information for Europe, and is an information and experience base for EU-projects maintained by ESI in Bilbao, see www.esi.es/VASIE/vasie.html. ESPINODEs and ESSI/PIEs encompass over 400 small EU-projects on SPI for Smalland Medium-sized Enterprises (SMEs), many registered by VASIE. Lastly, to promote exchange of ideas and results, ESERNET has supported a researcher exchange program among its members.
3
ESERNET goals, workpackages, results, and empirical studies
The actual project goals are: • Establish an infrastructure for a collaboration network. • Evaluate software engineering practices and package the gained experience for reuse. • Use Experimental Software Engineering (ESE) methods to facilitate the systematic introduction of new practices. • Accelerate community-wide learning in software engineering. • Foster collaboration and exchange within and between industry and research. • Create synergy from existing projects, networks, and organisations.
8
• Initially focus on inspections, testing, object-oriented technologies, and Component-Based Software Engineering (CBSE).
6.
ESERNET results and deliverables include internal reports, four open public workshops, a web portal (via the mentioned www.esernet.org) with relevant links and including a knowledge base of performed empirical studies, and of course this method and experience book. ESERNET has applied especially four kinds of empirical studies: post-mortems and partly surveys (often in industry), controlled experiments (e.g. using students), and industrial studies (e.g. real case studies in industry). As described in chapter 4, such studies are often done in the following sequence: First, post-mortems or surveys are used to pre-analyse practical experience by using a specific software technology or process. Then, controlled experiments are used to test out more concrete hypotheses about the chosen study object (e.g. novel inspection techniques), often using university students. Lastly, case studies are used to investigate refined and adapted hypotheses in an industrial setting. Overall methods to support such studies are discussed in chapters 2 and 3, and lessons-learned and recommendations from using two such methods stand in chapters 6 and 7. The term Experimental Software Engineering or ESE covers six phased activities, taken from the Quality Improvement Paradigm (QIP) [2]: 1-3. Characterise/define/plan: Make an explicit improvement goal and a plan to try out some new process or technique. Extra training may be given and measurements should be planned. 4. Operate (do) the study, observe the effects, and collect the specified data. 5. Analyse/interpret the results: analyse and generalise. Present/package: package and disseminate for reuse, often in a web/experience base.
The QIP principle on project vs. corporate learning, using a double improvement loop each with the above six activities, is illustrated by Fig. 2 below. As mentioned, the ESERNET project is supporting its own experience and knowledge repository to support even cross-corporate learning, see chapter 5.
Corporate Learning Package and store experience
Characterize and understand
Analyze results
Provide process with feedback
Project Learning
Set goals
Execute
Choose processes, methods, tools, and techniques
Analyze results
Fig. 2. Project vs. organisational learning in QIP
4
About the book content
As mentioned, the book reports on method and experimental work in the project. It is organised in four parts, numbered I-IV. Part I contains a short introduction by the book editors, as chapter one. Part II contains six method chapters, numbered 2-7. Chapters 2, 3 and 4 describe general and specific methods for running empirical studies. Chapter 5 deals with supporting an experience and knowledge base for such work. Chapters 6 and 7 describe experience by using two empirical methods, respectively controlled experiments and surveys, from chapter 2. The six method chapters are: Chapter 2 contains an introduction to “Empirical Research Methods in Software Engineering” by Claes Wohlin, Martin Höst, and Kennet Henningsson from Blekinge Institute of Technology and Lund University (Sweden). Chapter 3 is about “Challenges and Recommendations when Increasing the Realism of Controlled Software Engineering Experiments” by Dag I. K. Sjøberg, Bente Anda, Erik Arisholm, Tore Dybå, Magne Jørgensen, Amela Karahasanovi?, and Marek Voká? from Simula Research Laboratory (Norway). Chapter 4 explains “Empirical Studies in ESERNET” by Toni Sandelin and Matias Vierimaa from VTT Electronics (Finland). Chapter 5 discusses how to establish, use and maintain “Software Engineering Knowledge Repositories” by Andreas Jedlitschka and Markus Nick from Fraunhofer Institute for Experimental Software Engineering (IESE) in Kaiserslautern (Germany).
10
Chapter 6 explains how to “Using Empirical Studies during Software Courses” by Jeffrey Carver, Letizia Jaccheri, Sandro Morasca, and Forrest Shull from (in order of appearance) University of Maryland (USA), Norwegian University of Science and Technology (Norway), Università degli Studi dell'Insubria (Italy), and Fraunhofer USA Center for Experimental Software Engineering Maryland (USA). Chapter 7 sums up “Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering” by Marcus Ciolkowski, Oliver Laitenberger, Sira Vegas, and Stefan Biffl from (in order of appearance) University of Kaiserslautern (Germany), Droege & Comp. GmbH (Germany), Universidad Politécnica de Madrid (Spain), and TU Wien (Austria). Part III contains seven experience chapters, numbered 8-14. Chapter 8 deals with using two variants of post-mortem analyses in industrial case studies. Chapters 9, 10, 11 and 12 all report on university experiments with novel inspection techniques and partly testing, applied to either object-oriented designs (in UML) or programs (in Java). Chapters 13 and 14 report, respectively, on a university experiment and an industrial case study with CBSE. Thus, all the topics from the original ESERNET work plan are present. The seven experience chapters are: Chapter 8 reports on a study on “Post Mortem - An Assessment of Two Approaches” by Tor Stålhane, Torgeir Dingsøyr, Geir Kjetil Hanssen, and Nils Brede Moe from Norwegian University of Science (Norway) and Technology and SINTEF (Norway). Chapter 9 reports on “Evaluating Checklist-Based and Use-Case-Driven Reading Techniques as Applied to Software Analysis and Design UML Artifacts” by Giovanni Cantone, Luca Colasanti, Zeiad A. Abdulnabi, Anna Lomartire, and Giuseppe Calavaro from University of Rome at Tor Vergata and Rational Software (Italy). Chapter 10 reports on an experiment with “Effectiveness of Code Reading and Functional Testing with Event-Driven Object-Oriented Software” by Giovanni Cantone, Zeiad A. Abdulnabi, Anna Lomartire, and Giuseppe Calavaro from University of Rome at “Tor Vergata” and Rational Software (Italy). Chapter 11 reports on “Experimentation with Usage-Based Reading” by Thomas Thelin, Magnus Erlansson, Martin Höst, and Claes Wohlin from Lund University and Blekinge Institute of Technology (Sweden). Chapter 12 reports on “Functional Testing, Structural Testing and Code Reading: What Fault Type do they Each Detect?” by Natalia Juristo and Sira Vegas from Universidad Politécnica de Madrid (Spain). Chapter 13 reports on “COTS Products Characterization: Proposal and Empirical Assessment” by Alessandro Bianchi, Danilo Caivano, Reidar Conradi, Letizia Jaccheri, Marco Torchiano, and Giuseppe Visaggio from University of Bari (Italy), Norwegian University of Science and Technology (Norway), Politecnico of Turin (Italy). Chapter 14 reports on a study on “Reuse Based Software Factory” by Manuel Prego from European Software Institute Bilbao (Spain). Part IV with chapter 15 contains an Appendix with a glossary, assembled by the book editors. NTNU, Trondheim, 13 May 2003 On behalf of the ESERNET project,
Reidar Conradi and Alf Inge Wang, coordinating editors.
5
References
[1] Victor R. Basili, "The Experimental Paradigm in Software Engineering", in H. D. Rombach, V. R. Basili, and R. W. Selby (Eds.): Experimental Software Engineering Issues: Critical Assessment and Future Directives, Springer Verlag, Lecture Notes 706 in Computer Science, 1993, 261 p. From International Workshop, Dagstuhl Castle, Germany, Sept. 1992. [2] Victor R. Basili and Gianluigi Caldiera: “Improving Software Quality by Reusing Knowledge and Experience”, Sloan Management Review, 37(1):55-64, Fall 1995.
