2016 IEEE/ACM 38th IEEE International Conference on Software Engineering Companion
Visualizing the Effects of Requirements Evolution Shinobu Saito, Yukako Iimura
Hirokazu Tashiro
Software Innovation Center NTT CORPORATION Tokyo, Japan
NTT DATA CORPORATION Tokyo, Japan
[email protected]
{saito.shinobu, iimura.yukako}@lab.ntt.co.jp Aaron K. Massey
Annie I. Antón
Department of Information Systems University of Maryland, Baltimore County Baltimore, MD, USA
School of Interactive Computing Georgia Institute of Technology Atlanta, GA, USA
[email protected]
[email protected]
ABSTRACT Changes to software requirements occur throughout the software life cycle. Requirements engineers who maintain software systems in regulated environments must identify the affected artifacts when requirements change. This identification is critical to: (a) ensure continued compliance with regulations, and (b) accurately estimate budget requests. Previously, we introduced Requirements Evolution Charts (REC) to provide a visual representation of requirements evolution history. An REC is generated from the issue tickets in which requirements engineers record changes to requirements artifacts. Herein, we examine whether a REC helps software engineers conduct an impact analysis. Thirty experienced NTT requirements engineers with no prior domain knowledge identified the impact of seven requirements changes in a large-scale system governed by Japanese laws and regulations. They were divided into two groups of fifteen engineers. The REC group employed the REC to aid their efforts; the Non-REC group conducted their impact analysis without the REC. Participants in both groups identified which of the 139 artifacts were affected based on a history of 108 issue tickets. Our study reveals that engineers in the REC group identified the affected artifacts more accurately and quickly than the Non-REC group, suggesting that the REC is a valuable tool for examining the impact of requirements evolution.
Keywords Requirements Evolution, Impact Analysis, Issue Ticket, Change Requirements
1.
INTRODUCTION
In software development, requirements rarely remain unchanged throughout the development lifecycle. Requirements Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from
[email protected].
ICSE ’16 Companion May 14–22, 2016, Austin, TX, USA c 2016 ACM. ISBN 978-1-4503-4205-6/16/05. . . $15.00
DOI: 10.1145/2889160.2889237
152
changes constitute a critical challenge for software engineering [9]. In regulated environments, when legal requirements change, the requirements engineers must identify the affected artifacts to (a) ensure continued compliance with regulations, and (b) accurately estimate budget requests. Understanding the history and rationale for requirements evolution is valuable when conducting an impact analysis on software governed by laws and regulations [15]. When a regulation is revised, the artifacts that initially complied with the regulation may require revision to remain in compliance. In this situation, requirements engineers must be able to detect artifacts derived from the requirement directly affected by the regulatory change for two reasons. First, a change to regulatory requirements may introduce unanticipated and potentially non-compliant requirements––a risk that can only be mitigated by conducting a thorough and systematic impact analysis of the changed requirement(s) and all related software artifacts. Second, regulatory changes may have important financial implications. In the case of many public or government software systems, including the system we discuss throughout this paper, an annual budget request must be submitted to the client once and only once per year. It is imperative to “get it right” when estimating the cost for the project on a yearly basis––a risk that can only be mitigated by ensuring that the impact analysis accurately captures the cost imposed on the overall system development effort by changing regulations, requirements, and software artifacts. Issue tracking is a common way to manage requirements changes. Many open source and commercial products are used for this purpose [10, 14, 16]. Each time a requirement changes, requirements engineers create an issue ticket. The requirements engineer must then determine the scope of that change with respect to the existing requirements artifacts, update the affected artifacts, and record the rationale for the changes using issue tickets. Previously, we introduced RECs (Requirements Evolution Charts) to support issue tracking and improve management of requirements evolution through a visualization of the relationships between requirements [15]. Over the course of two years, we actively engaged with a commercial development effort to develop a large-scale document management and approval system governed by Japanese laws and regulations. The project is managed and
operated by one of the NTT Group companies [13]. The system is proprietary in nature; thus, we refer to it as the DMAS (Document Management and Approval System) throughout the remainder of the paper. There are now over 70 business processes supported by the DMAS, each of which has at most 130 requirements artifacts, yielding a total of approximately 7,500 requirements artifacts consisting of 5300 use cases and 2200 decision tables. The DMAS project’s requirements engineers use issue tracking to manage requirements changes during the requirements definition phase. The project follows a waterfall development model because it is large and complex with approximately 2,000 software engineers (of these, 50 are requirements engineers) contributing to the effort and the large number of requirements artifacts must be carefully scrutinized due to their regulatory nature and the pressure to accurately estimate project costs on an annual basis. The overall development was planned as a 7-year long effort, and avoiding delays for this project is critical to ensure its success. Once the requirements specification phase is completed, most original engineers will move on to other projects. When the design phase starts, new engineers may be assigned to maintain the requirements artifacts on behalf of the original team members. When a new software engineering team joins, they typically only refer to the latest artifacts and issue ticket lists due to their inexperience with the domain or the system. When stakeholders request a change, new members use the information in the existing ticket lists to conduct an impact analysis. Although studying all existing issue tickets may enable new members to acquire valuable requirements evolution knowledge, the process is tedious and time consuming, particularly for a large-scale software system such as DMAS. Even experienced engineers may fail to track requirements evolution events over a series of issue tickets. In a previous paper [15], we proposed a Requirements Evolution Chart (REC) to visually represent a software project’s history of requirements evolution. The REC is generated from issue tickets where requirements engineers updated software artifacts to address changes in requirements. In our prior research [15], we examined the effectiveness of the REC for identifying requirements evolution events that were otherwise overlooked by original engineers. In this paper, we examine whether the existing REC aids requirements engineers new to a project domain (i.e., new team members that are otherwise experienced requirements engineers) identify the impact of the requirements changes in the context of system compliance. We conduct this study by seeking to answer the following research question by means of a practical case study.
Section 6 presents potential threats to the validity of this work. Finally, Section 7 summarizes the paper and presents our plans for future work.
2.
BACKGROUND
2.1
Related Work
2.1.1
Managing Requirements Change
Requirements evolution is inevitable in any software development effort. We are interested in requirements management approaches focusing on responses to stakeholder requests and in tracing requirements evolution to identify the subsequent impacts of these changes. Harker et al. [9] noted the importance of considering requirements change classifications to manage requirements evolution. Specifically, they distinguished between stable and changing requirements (mutable, emergent, consequential, adaptive, and migration) to assess the scope of their impact, which is a goal of our work. They also identified the need for better representations of evolving requirements to improve communication [9]. Herein, we develop the REC to address this need. Jones [11] introduced the term “requirements creep” to discuss the inevitable evolution of requirements. Jones proposed several emerging technologies (e.g., prototypes, formal requirements inspections, and change-control boards) to clarify early requirements and minimize the disruptive effects and costs of changing requirements later. We believe that using an automatically generated REC, as proposed herein, may be more easily adapted to software projects that already use an issue ticking system. Carter et al. [2] proposed a tool-supported method to manage requirements creep. Their evolutionary prototyping model, called EPRAM, supports engineers seeking to prompt stakeholders to submit and clarify requests. For existing systems or development teams unfamiliar with EPRAM, our approach may be easier to implement because it relies only on the history of and the rationale for requirements evolution as expressed in issue tickets, which are already available for many software projects. Easterbrook and Nuseibeh [5] proposed an approach for managing inconsistencies arising from multi-perspective evolving specifications. They use state transition diagrams to specify required behavior of a target device. The partial specifications, called ViewPoints, are described based on each actor’s perspective at a particular moment in the development process. The approach, and tool support, helps design engineers resolve conflicts in evolving specifications by recording consistency relationships between diagrams. In contrast, our approach helps requirements engineers identify the impact of requirements change by visualizing an evolving history of whole artifacts.
RQ: Can experienced requirements engineers without DMAS domain knowledge use an existing REC to identify artifacts affected by change requests more accurately and quickly than similar requirements engineers attempting the same task without using the REC?
2.1.2
The remainder of this paper is organized as follows: Section 2 describes related work with an emphasis on requirements evolution. We introduce the previous paper, which includes an overview of DMAS and the REC. Section 3 describes our case study methodology. We describe how to use the REC for impact analysis on requirements changes and our analysis procedures. Section 4 details the case study results. Section 5 discusses the implications of this work.
Modeling Requirements Evolution
Wnuk et al. [17] proposed Feature Transition Charts (FTC) to visualize a large number of requirements. In particular, FTC visualizes scope dynamics within and across multiple projects (e.g., product line projects). The technique provides a comprehensive overview of the timing and magnitude of feature transitions in multiple projects. In contrast, we focus on a single large software system and use issue tickets, which include both new feature requests and defect corrections, 153
rather than feature lists. Because both feature requests and defect corrected can cause requirements change, we must focus on issue tickets. In addition, our goal is to support identification of all requirements affected by a change request, whereas Wnuk et al. seek to identify the scope of a requirement across multiple projects. Zowghi and Offen [19] presented a logical framework for modeling and reasoning about the evolution of requirements. This framework aims to formally manage continuous changes in requirements by maintaining the consistency and completeness of the requirements model. In our study, we employ a visual approach, rather than a logical framework, for modeling the requirements evolution. These are complementary approaches, and it may be possible to combine them to improve overall effectiveness in change management.
2.1.3
tween requirements engineers and business processes, which complicates requirements evolution management. Once the stakeholders approve requirements artifacts, the design and implementation will be contracted to another team. Most original members of the requirements engineering team will be contracted out after the requirements definition phase. The DMAS is a six-year development effort with two years devoted to requirements definition, two years to architectural design, and three years to implementation and testing. Note that the last year of the requirements definition phase overlaps with the first year of the architectural design phase. As previously mentioned, an annual budget request must be submitted to the client only once a year. Thus, the need for judicious impact assessment when regulatory or requirements changes are introduced makes it imperative to “get it right” when estimating the cost for the project on a yearly basis––a risk that can only be mitigated by ensuring that the impact analysis accurately captures the cost imposed on the overall system development effort by changing regulations, requirements, and software artifacts.
Predicting Requirements Evolution
Ant´ on and Potts [1] provided an approach to analyzing the evolution of user-visible system services. Through an investigation of a service system over a 50-year period, they collected the service growth record (i.e., chronology of service evolution for the system). Although our work is similar in that we both examine historical elements of a system or systems, our goals are different. We seek to identify requirements affected by a change, whereas they seek to understand or predict feature evolution. In addition, we examine internal issue tickets rather than user-visible features for tracking the requirements evolution. Maxwell et al. [12] presented a framework for predicting regulatory evolution. In the regulated domain, some rules from the proposed regulations are likely to change when the final regulations are released. The framework attempts to predict how a proposed regulation will change prior to publication. The prediction allows engineers to focus on specifying compliance requirements from the more stable areas of the regulations. In this work, we focus on detection of affected artifacts after a change in regulatory requirements is certain. Although we believe that incorporating some predictive elements may improve planning, we reserve this for our future work.
2.2
2.3
Requirements Evolution Charts
A Requirements Evolution Chart (REC) provides a visual representation of requirements evolution events over time. A REC is based upon the mapping of issue ticket operations to requirements evolution events. Table 1 shows an Issue Ticket Template [15]. Each issue ticket includes a ticket ID, a change request, rationale, update action (e.g., updated artifacts, artifact types, operation types), issue date, and close date.1 The operation types are Add, Change, and Delete. The artifact types are UC and DT, which are abbreviations for “Use Case” and “Decision Table.” Table 1. Issue Ticket Template
DMAS Overview
The system for this case is a very large (tens of millions of SLOC) document management and approval system (DMAS) governed by Japanese laws and regulations. The DMAS supports document approvals similar to those needed for building or construction permits or drug approvals in the United States. The DMAS supports 13 high-level business process groups (BPGs), each of which is responsible for a different part of the approval process for different types of submissions. In total, there are over 70 business processes allocated to these 13 BPGs. Examples of business process activities include document filings, approvals, rejections, reviews, and appeals. On average, the DMAS supports over 1,000 daily users and nearly 500,000 documents are submitted each year, with each submission triggering a complex review and approval process. The DMAS requirements engineering team comprises approximately 50 requirements engineers, one project manager, and five middle managers. Each middle manager manages 10 requirements engineers on average. Each requirements engineer is in charge of two or three business processes on average. Thus, there is a many-to-many relationship be-
We employ Cleland-Huang’s seven events [3] to record the kinds of requirements evolution that occur. When stakeholders request new requirements, requirements engineers make structural changes to the corresponding artifacts. ClelandHuang et al. [3] defined seven evolutionary events of change: Create, Inactivate, Modify, Merge, Refine, Decompose, and Replace. Figure 1 shows the events resulting from requirements evolution. Requirements artifacts are represented by the capital letters A, B, and C. The arrows represent structural changes from the artifacts on the left hand side to those on the right hand side. 1
Please note that issue tickets are sorted based on their close time, which is when the changes they describe affect the system. 154
mation about change requirements requested by stakeholders and the rationale for the update actions to artifacts for addressing a change requirement.
3.
CASE STUDY METHODOLOGY
Accurate measurement of empirical data is critical to understanding the extent to which the REC helps software engineers respond to regulatory changes through an impact analysis. To this end, we conducted a case study in which we asked participants to identify the artifacts affected when requirements changed in the DMAS project. As shown in Figure 3, participants were partitioned into two groups: a REC group that used the REC to assist their identification of affected software artifacts and a Non-REC group that used current best practices to identify affected software artifacts. The goal of this study was to assess whether the REC assisted participants in three performance criteria: 1. Speed as measured in time spent completing the analysis.
Figure 1. Seven Evolutionary Events
2. Accuracy as measured in comparison with an expert’s assessment of the correct set of affected artifacts.
In our prior work [15], we introduced seven different mapping rules. These mapping rules yield the following seven kinds of evolutionary events: “Create” produces a new artifact. “Inactivate” means an artifact is marked as deleted. “Modify” changes the value of an attribute in a given artifact. “Merge” combines two or more artifacts into a new artifact. “Refine” adds a new additional artifact to, for example, finetune an existing original artifact. “Decompose” takes an existing artifact and separates it into two or more artifacts. “Replace” substitutes one artifact with another. Figure 2 shows a sample REC with corresponding issue tickets. Our development tool [15] generates the image of the REC on the left side of the figure. The tool, which depends on Graphviz [8], visually represents requirements evolution from the information of the issue ticket list on the right side of the figure. The REC includes five evolutionary events: Refine, Decompose, Merge, Replace, and Modify. The corresponding four issue tickets appear on the right side; the eight columns (Ticket ID, Change Request, Rationale, Updated artifact, Artifact type, Operation type, Issue date, and close date) are taken from the issue ticket template. The REC contains five artifacts (i.e., UC_1 to UC_3 and DT_1 to DT_2) in the initial condition. The dotted lines are labeled time-series links; each represents a transition from one issue ticket to another. The initial condition is the starting point and it leads directly to the first issue ticket (t1). From there, the second time-series link starts at t1 and ends at t2, which is the ticket ID of the second issue ticket. In this way, both ends of each link represent consecutive ticket IDs of two issue tickets. The solid lines are labeled evolutionary links; each represents a change transition of an artifact in the requirements evolution. In this way, these links and labels visually represent a time series of requirements evolution events. On the other hand, artifact DT_2 is not connected to any evolutionary links because no issue tickets refer to this artifact, as shown in the issue tickets list on the right side of the figure. The REC supports requirements engineers seeking to understand requirements evolution. Issue tickets are mapped to evolutionary events in the REC. They also contain infor-
3. Agreement as measured by Fleiss Kappa [7]. We conducted this case study over two separate sessions with 30 participants in each session. Each session consisted of two groups of 15 participants each, one REC group and one Non-REC group. Our case study comprised a tutorial and a survey, with the former conducted before the latter. In the tutorial, we first explained the purpose of the study to all participants, and then gave them a 20-minute overview of the DMAS, requirements artifacts, and issue tickets. Then, after a short question and answer period about the explanation, we moved the REC group to another room and spent another 20 minutes introducing this group to the REC, detailing how it can be used for impact analysis. In the remainder of this section, we describe the use of the REC for impact analysis in Section 3.1. In Section 3.2, we detail the case study materials. In Section 3.3, we explain the participant selection process. In Section 3.4, we describe the statistical analysis techniques employed.
3.1
REC for Impact Analysis
The REC allows requirements engineers to focus on the scope of changed requirements. Figure 4 depicts the use of the REC for impact analysis, which consists of three steps: a) Receive change requirement: The requirements engineer receives a change to a requirement from a stakeholder. At the upper right of Figure 4, the reason for the change requirement is identified as a revision of Policy 01. b) Identify related issue ticket: The requirements engineer identifies the issue ticket(s) related to the change requirement. In Figure 4, the requirements engineer identifies the issue ticket t1 because the ticket contains the change of the artifact (i.e., artifact A) resulting from system compliance of Policy 01. c) Focus on impact scope: Referring to the REC, the requirements engineer selects the artifacts that are 155
Figure 2. REC and Corresponding Issue Tickets
Figure 3. Participants and Materials
derived from the original artifacts. These selected artifacts could be affected by the changed requirement. In Figure 4, the REC shows that artifacts G and J are both derived from artifact A. The two derivative artifacts are likely to be affected by the revision of Policy 01. The requirements engineer makes a note to examine artifacts G and J.
3.2
Figure 4. Focusing on Scope Using REC
Study Materials
Table 2. Case Study Materials
Table 2 displays the materials we provided to the case study participants. A total of 139 (= 1 + 64 + 74) requirements artifacts were created for one business process of the DMAS project. Similarly, 108 issue tickets were recorded at the stakeholder review meetings. The remainder of this subsection describes each of the five materials in the order presented in Table 2.
3.2.1
Requirements Artifacts
The DMAS business process is represented by a collection of business flows shown in Figure 5. We now define three types of requirements artifacts: 1. Use case: describes a sequence of events performed by 156
Figure 5. Requirements Artifacts Figure 6. Work Steps for Requirements Artifacts
actors (e.g., input and output, pre- and post-conditions, normal and exceptional scenarios). 2. Decision table: represents conditions at the end of a given use case that must exist for determining a plausible next use case in the business flow sequence [3].
appropriate or relevant experience with the aspects of the system relevant to the agenda for participation in each meeting. During each meeting, the attendees actively reference and review all the requirements artifacts. Two requirements review meetings are held each week. Each meeting lasts about two hours. There are no breaks during the meeting because they are conducted using Fagan-style [6] software inspections, which are normally limited to two hours.
3. Business flow: comprises use cases and decision tables, as indicated by the blue dotted arrows. Figure 5 also shows an example business flow, including the relationship between the use cases and the decision table. The business flow in the figure comprises six use cases (From “Request Order” to “Receive Result”), and one decision table (“Acceptance Check”).
3.2.2
d) Update Artifacts: Requirements engineers update requirements artifacts after each review meeting based on discussions held and suggestions made during the meetings.
Issue Ticket List
The requirements definition phase continued for nine weeks: three weeks for initially creating the requirements artifacts and six weeks for reviewing them with stakeholders. During this phase, stakeholder review meetings, which took place 13 times, are conducted to ensure requirements coverage for the new system. A separate requirements analysis effort is conducted for each business process. Two requirements engineers developed the requirements artifacts and recorded the issue tickets. Both were senior-level engineers with over 10 years of experience. The stakeholders for each business process include members of the NTT team and approximately five members of the customers’ IT staff and end users familiar with the legacy system that the DMAS will replace. As shown in Figure 6, requirements engineers and stakeholders work together in developing requirements artifacts. This effort comprises six work steps:
e) Approve: Stakeholders check if the updated artifacts reflect the review comments from the previous meeting. If the stakeholders approve the artifacts, they are finalized. f) Record Issue Ticket: When finalizing the artifacts, requirements engineers record the changes in the update action section of a new issue ticket.
3.2.3
REC
We provided only the REC group with the REC shown in Figure 7, which was generated from the issue tickets recorded by two requirements engineers. Due to space limitations, the figure only shows part of the complete REC. In the figure, the name of each artifact is expressed as a symbol. In the case study, the actual name of each artifact was represented in the REC.
a) Initial Request: The initial request to the overall system development effort from stakeholders is a set of requirements extracted from the legacy system’s specification and managed using an Excel spreadsheet. This original specification is incomplete, so stakeholder review meetings are conducted to ensure requirements coverage for the new system.
3.2.4
Question List and Answer Form
We created eight questions: one tutorial question and seven survey questions. Table 3 shows all questions we created. They were based on requirements changed as a result of stakeholder requests after the completion of the requirements definition phase. We used two requirements change types that affected the requirements artifacts: a scenario change and a decision-rule change. We also used three primary reasons for the change: change requests from other business processes, changes in regulations, and misunderstandings of regulations. Collaborating with the original members of the DMAS project’s
b) Create Artifacts: After receiving the initial request, requirements engineers create requirements artifacts (e.g., business flows, use cases, and decision tables) reflecting the request from the stakeholders. c) Review: A few days prior to each meeting, a middle manager proposes an agenda. Using the agenda as a basis, a customer representative selects stakeholders with 157
Table 3. Case Study Questions
Figure 7. REC (Case Study Material)
to balance the amount of experience among participants. Each group included seven participants with less than five year’s experiences in requirements engineering, and eight expert level participants who have over five year’s experience.
requirements engineering team, we prepared the case study questions. As shown in the last survey question (ID=Q7) in the table, we prepared a question combining scenario change and decision-rule change. We requested the original members to create an oracle (i.e., correct answers to the study questions) for the case study. It took them about one hour to identify the impact scope of change requirements for each question.
3.3
3.4
Study Analysis
To answer the research question we posed in Section 1, we employed three measures: (1) F-Measure, (2) Survey time, and (3) Fleiss’ Kappa [7]. F-Measure is a weighted harmonic mean of precision and recall. We used it as a comprehensive indicator of combined precision and recall. To calculate it, we analyzed the collected data and counted the number of responses and correct answers for each question. All participants were requested to record the amount of time it took them to answer each question. By collecting the data for this, we calculated the survey time for each question. In addition to the F-Measure and survey time, we employed Fleiss’ Kappa [7] for assessing the group participants’ agreement on the answers. We used the R Project with the Inter-Rater Reliability (IRR) package for calculating Fleiss’ Kappa. We compared three measures of both groups in the case study.
Study Participants
For this study, we approached one of NTT’s software development companies, and requested that they identify 30 individuals with no prior domain knowledge about DMAS but with experience in software development, including requirements engineering. Candidates for participation in this study came from seven company projects. Each has experience in software development for a specialized field, such as mobile, security, and networking. We requested each senior project manager to choose project members who have experience in requirements engineering. We recruited about 4 to 6 engineers per project, for a total of 30 participants. As we mentioned before, we split them into two groups of fifteen: the “REC group” and the “Non-REC group”. Only the REC group participants were trained to use and allowed to refer to the REC. The Non-REC group used a basic textual search (provided by the issue ticketing system) over the complete issue ticket list to identify changes. We sought
4.
CASE STUDY RESULTS
Our study reveals that engineers in the REC group identified the affected artifacts more accurately and quickly than the Non-REC group, suggesting that the REC is a valuable 158
tool for examining the impact of requirements evolution.2 We chose to use the first question as a part of the tutorial and asked all participants to attempt it. When they completed the exercise, we explained the correct answer to them. In the study, we examined seven questions and the participants’ answers to them. Table 4 details the results for each question for both case study groups. Figure 8 shows the distribution of both Recall and Precision for each participant. Figure 9 shows the distribution of both survey time and F-measure for each participant.
that the REC group members answered the questions more quickly. To test for statistical significance, we employed the twosample t-test.3 As shown in Table 5, the result of the t-test for two groups does not indicate a statistically significant difference for mean time to complete the survey between the two groups. To test for effect size, we employed Cohen’s D using pooled variance [4] and found a moderate effect.
4.2
4.3
Fleiss’ Kappa
As shown in Table 4, for each question the Fleiss’ Kappa values for the REC Group were higher than those for the Non-REC group. Participants’ agreement on the answers was statistically significant for both groups. From the results obtained in a comparison analysis on the three measures, we can confidently say that use of the REC results in a quicker, more accurate identification of affected artifacts. Two individual instances indicated the Non-REC group performed slightly better (survey time for Q3 and F-measure for Q5), and we do not claim that using the REC will result in better performance in all measures and in every scenario. However, all of our aggregate statistics confirm that using the REC results in significantly better general performance. The answer to our research question from Section 1 is, simply put, “Yes!”
Figure 8. Recall and Precision
5. 5.1
DISCUSSION Effectiveness of Visualization
In the case study, we provided the issue tickets list to both groups. The issue ticket list included information on the history of requirements evolution. The Non-REC group was able to track requirements evolution by searching and reading through all issue tickets. However, their F-measure and survey time were worse than those of the REC group. Although the Non- REC group was able to complete the survey faster than the REC group, we were unable to establish this difference as statistically significant. We believe this to be a result of the small sample size used in the study, which limited the possible artifacts that could have been affected. The DMAS is extremely large in scale; we believe a larger,
Figure 9. Survey Time and F-Measure
4.1
F-measure
The average F-Measure value for the REC Group was 0.798; for the Non-REC group it was about 22% lower at 0.620. This indicates that overall the REC group answered the questions more accurately than the Non-REC group. The average recall value for the REC group was only slightly (about 12%) higher than that for the Non-REC group, but the average precision value was about 46% higher. This indicates that the REC group’s improved precision increased the group’s F-Measure value. As we did with the survey time, we employed the twosample t-test to measure statistical significance for F-measure between the two groups. As shown in Table 5, the F-measure results exhibit a statistically significant difference. We also employed Cohen’s D using pooled variance [4] to determine effect size and found a large effect.
Survey Time
The average of total survey time for the REC group was 66.3 minutes and that for the Non-REC group was 77.8 minutes. This means that on average the REC group took about 15% less time than the Non-REC group, indicating
3 We employ the t-test because we are working with a small sample size from a population that we confirmed to be normally distributed using the Anderson-Darling Normality Test.
2 Our raw data is available online here: http://userpages. umbc.edu/˜akmassey/documents/rec-data.xlsx
159
Table 4. Case Study Measures
Table 5. Statistical Significance Metrics
pants to more readily acquire information on requirements evolution than participants without the REC.
6.
more comprehensive analysis would result in a statistically significant result. Our REC provides a visual representation of requirements evolution. Even if requirements evolution events increase, as they did in this case study, the REC makes it easy to trace the history of requirements evolution. As described in Section 4, the REC group participants were able to detect the derivative artifacts more quickly and accurately than those of the Non-REC group.
5.2
Primacy Over Text-Search
Generally, requirements engineers use a text-search (i.e., keyword-search) function in performing impact analysis. All the Non-REC group participants also used the function to answer the case study questions. In many text-search situations an enormous number of matching results are obtained. In such situations requirements engineers have to narrow down the artifacts returned by the search to those that could be affected, which may consume a lot of time. Our REC solves this problem by enabling engineers to concentrate their efforts only on artifacts that could be affected. This is demonstrated by the results given in Section 4, which showed that REC group participants were able to answer questions more quickly than those of the Non-REC group.
5.3
THREATS TO VALIDITY
When designing any case study, care should be taken to mitigate threats to validity. We make no causal inferences as a result of our study, so internal validity is not a concern [18]. External validity is the ability of a case study’s findings to be generalized to broader populations [18]. A possible threat to external validity is the fact that we analyzed only one part of a large document management system regulated by laws and regulations. To mitigate this threat, we selected a project in which a requirements engineering team created three types of requirements artifacts (i.e., business flow, use case, and decision table). These types of artifacts are not domain specific; they may be created by other projects in other domains. Issue tickets are similarly generalizable artifacts. Construct validity addresses the degree to which a case study is in accordance with the theoretical concepts used [18]. Three ways to reinforce construct validity are: use multiple sources of reliable evidence, establish a chain of evidence, and have key informants review draft case study reports [18]. By collecting artifacts and issue tickets from 13 review meetings over a total of six weeks with two different types of stakeholders (Information Technology Department staff members and end users), we used multiple sources for our study. To establish a chain of evidence, we carefully followed the case study process described in Section 3. We also maintained all requirements artifacts and issue tickets collected, as well as the relation between the artifacts and tickets. Finally, several members of NTT reviewed our draft case study report. Reliability is the ability to repeat a study and observe similar results [18]. To reinforce our study’s reliability, we carefully documented our approach in Sections 3 and 4.
7.
Training Aid for New Project Members
SUMMARY AND FUTURE WORK
In this paper, we employ a Requirements Evolution Chart (REC), which is a graphical representation of requirements evolution generated from issue tickets. We defined an issue ticket template that details the possible operations affecting requirements artifacts when stakeholders request a change to the requirements. By applying the seven mapping rules we defined, we identify requirements evolution events based on combinations of operations in the issue tickets. The REC visualizes a series of these events identified by the mapping rules and the issue ticket list. We conducted a case study to determine whether the REC
In large-scale system development projects, it generally takes a lot of time for new project members to understand the overall structure of requirements artifacts and the related history of and rationale for the requirements. The REC can aid new project members seeking to understand the relationship between evolutionary events and issue tickets and the rationale for these changes. As described in Section 4, we provided a total of 40 minutes of lecture time to REC group participants without domain knowledge on the case system (i.e., DMAS). Referring to the REC enabled these partici160
helps requirements engineers conduct an impact analysis in a regulated environment. The system used for the case was a large-scale system governed by Japanese laws and regulations. In the case study, 30 requirements engineers with no domain knowledge for the business process selected for this study were divided into two groups: an REC group and a Non-REC group. All participants identified the impact of seven change requirements among 139 artifacts and 108 issue tickets that were actually developed in the case. Only the REC group used the REC generated from the issue tickets. The analysis results we obtained indicate that the REC group identified the affected artifacts more accurately (with 15% higher Fmeasure values) and quickly (22% less survey time) than the Non-REC group. In particular, the F-measure value for the REC group is higher than that of the Non-REC group with a statistically significant difference. We also found that as a whole, the participants’ agreement (as measured by Fleiss’ Kappa) on the answers was statistically significant for both groups. In the future, we plan to develop a cooperative tool that will integrate our REC generation tool with existing issue tracking tools. In addition, we plan to develop a supporting tool for requirements engineers that will enable them to record issue tickets in accordance with the guidance detailed herein. The tool will help to prevent incorrect data due to human error during the creation of issue tickets. While developing and applying these tools, we plan to conduct additional case studies and further evaluate the effectiveness and implications of our approach in other phases of the SDLC. For example, RECs highlight aspects of the system that change over time. Aspects of the system that regularly change may need more focused testing or perhaps even re-architecting.
8.
[4] J. Cohen. Statistical Power Analysis for Behavioral Sciences. New York: Academic Press, Revised Edition edition, 1977. [5] S. Easterbrook and B. Nuseibeh. Managing inconsistencies in an evolving specification. In Proceedings of the Second IEEE International Symposium on Requirements Engineering, pages 48–55, Mar. 1995. [6] M. E. Fagan. Advances in software inspections. IEEE Transactions on Software Engineering, SE-12(7):744–751, July 1986. [7] J. L. Fleiss and J. Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 1973. [8] Graphviz. http://www.graphviz.org/, 2016. [9] S. D. P. Harker, K. D. Eason, and J. E. Dobson. The change and evolution of requirements as a challenge to the practice of software engineering. In Proceedings of IEEE International Symposium on Requirements Engineering, pages 266–272, Jan. 1993. [10] JIRA. http://www.atlassian.com/en/software/jira, 2016. [11] C. Jones. Strategies for managing requirements creep. Computer, 29(6):92–94, June 1996. [12] J. C. Maxwell, A. I. Anton, and P. Swire. Managing changing compliance requirements by predicting regulatory evolution. In Proceedings of the 20th IEEE International Requirements Engineering Conference, pages 101–110, Sept. 2012. [13] NTT. http://www.ntt.co.jp/index e.html/, 2016. [14] Redmine. http://www.redmine.org/, 2016. [15] S. Saito, Y. Iimura, K. Takahashi, A. K. Massey, and A. I. Ant´ on. Tracking Requirements Evolution by Using Issue Tickets: A Case Study of a Document Management and Approval System. In Companion Proceedings of the 36th International Conference on Software Engineering, ICSE Companion 2014, pages 245–254, New York, NY, USA, 2014. ACM. [16] The Trac Project. http://trac.edgewall.org/, 2016. [17] K. Wnuk, B. Regnell, and L. Karlsson. Feature Transition Charts for Visualization of Cross-Project Scope Evolution in Large-Scale Requirements Engineering for Product Lines. In Fourth International Workshop on Requirements Engineering Visualization (REV), pages 11–20, Sept. 2009. [18] R. Yin. Case Study Research: Design and Methods. Sage Publications, Third Edition edition, 2003. [19] D. Zowghi and R. Offen. A logical framework for modeling and reasoning about the evolution of requirements. In Proceedings of the Third IEEE International Symposium on Requirements Engineering, pages 247–257, Jan. 1997.
ACKNOWLEDGMENT
The authors are grateful to Messrs. T. Maeyama, R. Matsumoto, S. Oyama, M. Fukunaga, T. Shinya, and T. Kusano of NTT DATA, G. Suzuki and S. Yamada of NTT, K. Kajihara and S. Hiraoka of NTT Software for their assistance in the case study.
9.
REFERENCES
[1] A. I. Anton and C. Potts. Functional paleontology: the evolution of user-visible system services. IEEE Transactions on Software Engineering, 29(2):151–166, Feb. 2003. [2] R. A. Carter, A. I. Anton, A. Dagnino, and L. Williams. Evolving beyond requirements creep: a risk-based evolutionary prototyping model. In Proceedings of the Fifth IEEE International Symposium on Requirements Engineering, pages 94–101, 2001. [3] J. Cleland-Huang, C. K. Chang, and M. Christensen. Event-based traceability for managing evolutionary change. IEEE Transactions on Software Engineering, 29(9):796–810, Sept. 2003.
161
