Evaluating Design Approaches in Extreme Programming René Noël1, Hernán Astudillo1, Marcello Visconti1, Javier Pereira2 1
Departamento de Informática, Universidad Técnica Federico Santa María (UTFSM), Valparaíso, Chile 2
Departamento de Informática Universidad Diego Portales, Santiago, Chile
[email protected],
[email protected],
[email protected],
[email protected]
Abstract. Extreme Programming (XP) is an alternative to traditional software development methods intensive on documentation and planning; XP proposes an emergent design approach, which adds complexity to the system structure only when it’s needed, avoiding a big up front design of the system. Some cases have been reported of developers questioning XP’s design practices and (sometimes implicitly) arguing for the naturalness of, and need for, planned design. The present paper outlines an experimental design in order to compare the actual impact of emergent and planned design approaches on a product’s internal quality, process' productivity, and developers' satisfaction. To enable replication of the experimental design by other teams, an Experimental Package has been also elaborated.
1. Introduction Extreme Programming (XP) [Beck 1999] is an Agile [Beck 1998] software design methodology which arises as an alternative to traditional processes, documentation and planning intensive, but relatively inflexible when technological or requirement changes arrive during the development process. XP proposes embracing change through a set of software development best practices, anchored on key principles and values that drive development team’s actions. One of the most novel tenets of XP is avoiding a big up front design of the system and focusing on the iteration at hand, but there are documented cases of development teams that resists to this approach. On these cases developers question XP’s design practices and (sometimes implicitly) argue for the naturalness of, and need for, planned design. In order to evaluate the potential impacts of the design approach, an experimental study has been considered as needed, because it allows controlling other variables which can impact over development process, and that can’t be addressed by other studies such as Case of Study’s analysis or Surveys that are based on already finished software projects where methodology has not been implemented uniformly. Section 2 exposes the research context, which consists on the problem definition, and a research review. Section 3 presents the main hypotheses of the research. Section 4 outlines the experimental design, presenting the treatments and variables of the research, and classification and description of the experiment. Section 5 details the threats to validity of the experiment. Section 6 explains the communitarian focus of the research, and Section 7 describes the future work of the research.
2. Research Context 2.1 Problem Definition One of the more novel tenets of XP is its Design Strategy. XP proposes that the best design is the one that emerges from every time constructing the simplest solution to the requirements of a given iteration, then refactoring the software when necessary, thus. The goal of this strategy is to add structural complexity to the system only when it’s needed, thus simplifying the implementation of potential change of requirements during development, and avoiding wasting effort doing a big up front design of the system. Although this emergent design approach seems to be a good choice to embrace change, there are reports of cases where its implementation has not been satisfactory. In cases of study reported from universities [Müller 2001, Keefe 2004], and from software research and development organisations [Harrison 2002, Rasmusson 2003, Murru], it has been reported that the participants on XP software projects don’t feel comfortable with the design approach: they insist on adding complexity not required at each specific iteration, on making detailed design documentation, and on designing in advance for all the system features, following a traditional or planed design approach. According to these observations, the problem could be defined as the insistence of developers on implementing software design practices that are restricted by the methodology. If these perceived observations are correct, they could impact a product’s internal quality, process productivity, and developer’s satisfaction. 2.2 Research Review Due to the radical change proposed on the software design approach of XP, some authors and practitioners have taken a more nuanced position to this matter: [Fowler] comments that planned and emergent approaches must be equilibrated on a development, although the original XP approach could work if a subset of enabling practices is implemented successfully. Code Science [CrossTalk 2002] presents a modified XP version created and implemented on a development organisation, that describes practices to integrate architecture concepts and planned design, but no formal definition of the method has been presented. Agile Modelling [AgileModelling] proposes a set of practices to modelling design artefacts based on agile principles, but it doesn’t specify when the models should be created on a software process. Experimental efforts to approach to the impact of design approaches have not been reported, although experimentation on XP projects is a rising topic: [Salo 2004] presents a controlled case study approach for agile software development empirical evaluation, a framework to gather experimental quantitative and qualitative data. In [Saff 2004] a proposed practice and tool for improve testing practice of XP is empirically evaluated, through a controlled experiment. The perception of students about a subset of XP practices has been evaluated on [Melnik 2005]. The perception is favourable to XP; unfortunately the study does not address design practices explicitly.
3. Hypotheses A first high level statement that this research is focused to validate is that a design approach that is natural to developers is a better way to develop software. In order to specify the meaning of better and the aspects of software development that will be affected, the following questions and hypotheses are formulated:
a. There exist differences on the Product’s Internal Quality by designing with an emergent or a planned approach? Because the planned approach considers instances to the identification of critical design elements, pattern existence and maintainability risks, the following null hypothesis is formulated: H0: The use of planned design approach produces software with an equal design quality than emergent design approach b. There exist differences on process productivity when doing the design with emergent or planned approach? Planned design approach requires effort to plan and document the design, while emergent approach focuses on obtain functional software in the less time possible: H0’: The use of an emergent software design approach is as productive as a planned approach. c. - There exist differences on development team members’ satisfaction with the applied methodology? Developers have been formed based on traditional software processes; a planned design approach could be more comfortable to them: H0’’: The use of a planned approach design is as comfortable to developers as the use of an emergent approach.
4. Experimental Design 4.1 Variables The object of study is the design approach implemented. Development teams that join the experiment will develop using a planned design approach or using XP’s original emergent design approach. Dependent variables and its associated metrics are listed bellow. a. - Product’s Internal Quality: Product’s quality will be evaluated through its design quality, and it will be measured considering the design that is embedded on the system’s code, and not the one that design specifications show. ISO 9126 [ISO] standard defines a set of quality features, applicable to the development process’ products, and metrics to assess this features. The quality features and its related metrics are the following: Modularity - Coupling: Measure of the degree of data sharing between modules. Will be applied to the classes elaborated during the experiment. Understandability - Number of inputs and outputs per module: design’s complexity metric, if designed elements are properly encapsulated, number should be low. Maintainability - Data flow complexity or cyclomatic number: complexity metric, considers the number of decisions on the algorithms of the code. b. - Development Process Productivity: it will be measured in terms of produced software, without considering design specifications and other sub products. Software size metrics will be used, detailed on ISO9126. Those are number of lines of code and number of functions. Additionally, if an Object Oriented paradigm is used, the number of written classes can be measured. c. - Development Team’s Satisfaction: the degree of team’s satisfaction with the methodology and its support to the design and construction tasks will be evaluated by surveys and polls. The goal is to detect instances where developers felt uncomfortable or lack of orientation on how to do the design of the software.
4.2 Experiment Description The experimental design will be oriented to reject thee three null hypotheses. According to [Zelkowitz 1997], a Controlled Experimentation Method will be used, classified as a Synthetic Environment Experiment: the experiment will be performed on a non industrial setting (academic setting with undergraduate students), and simulating a single design and construction iteration, which implements a subset of methodology’s practices. These are Iteration Planning, Pair Programming, Collective Ownership, Testing, Continuous Integration, Simple Design and Refactoring, which according to [Fowler] allow the success of XP design approach. According to the taxonomy presented on [Basili 1996], the experiment is designed to present cause-effect results, to be performed by novices, and on in-vitro environment. Under both taxonomies, the experiment chosen is the recommended to analyze human factors on software development. The subjects of the study are development teams, conformed at least by a pair of programmers. The treatments that will be applied are the design approaches: Treatment 1 consists on the subset of practices of the original XP methodology. Treatment 2 is a proposed modification of the subset of practices that integrates a planned design approach to the methodology. Each subject will perform one design and construction iteration, of about 4 hours, and will implement the same two requirements specified by the client (the experiment driver). Each treatment will be applied to the same number of subjects. The development teams will be formed searching to block factors that could impact on the results of the experiment, described on subsection 5.3, according to one factor and two levels experimental design [Montgomery 1991]. The activity consists on give an overview of the whole functionality to implement to both kinds of subjects, which is separated on two requirements. Then, the client (the experiment driver) explains in detail a first requirement. Both kinds of subjects starts developing, but just the ones under treatment 2 are allowed to plan the design based on the overview given, while treatment 1 subjects are restricted by its methodology to just implement basic functionality to accomplish the first requirement. Second requirement is explained to both teams, and at the end of the time box all subjects submit its code. 4.3 Communitarian Research In order to allow the establish with a higher scientific rigor the impact of design approaches on XP projects, the experiment has been designed focusing on its replication, through an Experimental Package. Experimental Package has the following elements: Experimental Design, Preparation Surveys for Participants, Capacitating Documents and Slides, Measures Checklist and Result’s History. Those elements will support the experiment’s preparation activities, the experiment itself, and the data gathering.
5. Threats to Validity 5.1 Threats to Internal Validity For a valid experience, it is necessary to guaranty that upon the measured variables doesn’t exist another than the independent variable influence, in this case, the design
approach used. Next, the detected threats to the internal validity and the proposed ways to lessen its effects are listed. Maturation: Change on the people’s behaviors between two measures, because each team will participate once along the experience. History: Change on the subjects due to events between two measures is not considered like a threat, because there is a single participation from each development team. Instrumentation: The threat of some variation on the measure instruments, measurement scales or observers between the measures will be mitigated obtaining quantitative data from the code produced, and measures will be made according to the standard metrics described, detailing for each of them how to make the accounting. Selection: Badly constituted teams risk will be mitigated by identifying the most important characteristics of the participants (see 5.3). Also, subjects will be capacitated on one of the two development methodologies (XP and XP + Planned Design), by means of 1.5 hours sessions, after the creation of development teams. Mortality: This threat refers to the lost of measures due to the leaving of the participants during the experiment, it will be controlled by motivating subjects with an evaluation. Process Conformance: This is the bigger threat on this activity, because it must be assured that the development teams which works with XP do not fall on planned design practices. There is not a mechanism that assures to get under control this issue, only a constant custody over the designs made by the teams that works with XP. 5.2 Threats to the External Validity The possibility to generalize the obtained results to others projects with XP or XP + Planned Design is a threat for the next factors: Representative Individuals: the subjects are only representatives under an academic field, and that is the reason why the results cannot be generalized for similar situations with professional development teams, but the activity can seat precedent for future experiences with professionals. Representative Processes: Because that the experience considers just a part from the development process, the results cannot be extended to complete developments using the evaluated design approaches. Representatives Artifacts: Due to the focus on a part of the development process only, there is no knowledge about other artifacts developed in an entire development process. This means that high level design decisions could be made in previous stages of the process. As it can be seen, the threats to the experiment’s external validity are real and they cannot be mitigated without implementing experimental activities on a real software development project. 5.3 Subjects Characteristics Because the activity is designed to be accomplished on an academic environment, where the participants can have very heterogeneous characteristics, it is necessary to control the ones that could influence over the experiment results. Those include: Programming paradigm knowledge, programming experience, programming expertise, traditional development participation, XP methodology knowledge, design specification languages knowledge, software engineering concepts training, career, gender and age.
In order to control the impact of these characteristics, an evaluation survey must be performed, which allows to conform the teams blocking differences on these aspects.
6. Future Work The next activity considered is to execute a pilot experiment that helps on the improvement of the experimental design, with a Software Engineering course, with 80 participants approximately. Future work is oriented to create networks to replicate the experiment, and share results using the experimental package described.
References K. Beck, “Extreme Programming Explained: Embrace Change”, Addison-Wesley, 1999 K. Beck, J. Greening, M. Fowler, R. Martin, “The Agile Manifesto”, available on http://www.agilealliance.com, 1998 M. Müller, W. Tichy, “Case Study: Extreme Programming in a University Environment”, 23 International Conference on Software Engineering (ICSE'01), 2001. K. Keefe, M. Dick, “Using Extreme Programming in a Capstone Project”, proceedings of the Sixth Conference on Australian Computing Education, 2004. B. Harrison, “Study of Extreme Programming in a Large Company”, Avaya Labs, 2002. J. Rasmusson, “Introducing XP into Greenfield Project: Software Magazine, May-June of 2003.
Lessons Learned”, IEEE
O. Murru, R. Deias, G. Mugheddu, “Assessing XP at a European Software Company” M. Fowler, “Is Design Dead?”, http://www.martinfowler.com/articles/designDead.html
available
on
Software Technology Support, “Odysey and Other Code Science Success Stories”, CrossTalk Journal, October, 2002. Available on http://www.stsc.hill.af.mil/crosstalk/2002/10/manzo.html. The Official Agile Modeling (AM) Site, available on http://www.agilemodeling.com O. Salo, P. Abrahamson, “Empirical Evaluation of Agile Software Development: the Controlled Case Study Approach”, Proceedings of International Conference on Product Focused Software Process Improvement, 2004. D. Saff, M. Ernst, “An experimental Evaluation of Continuous Testing During Development“, ISSTA ’04, July, 2004. G. Melnik, F. Maurer, “A Cross-Program Investigation of Student’s Perceptions of Agile Methods”, ICSE ’05, May, 2005. M. Zelkowitz, D.Wallace, “Experimental Validation In Software Engineering”, Conference of Empirical Assessment & Evaluation in Software Engineering, 1997. V. Basili, “The Role of Experimentation in Software Engineering: Past, Current and Future”, Proceedings of ICSE, 1996. ISO, International Organization for Standardization, http://www.iso.org D. Montgomery, “Diseño y Análisis de Experimentos”, Editorial Iberoamericana, 1991.
