Test Case Reuse in Enterprise Software Implementation

Test Case Reuse in Enterprise Software Implementation - An Experience Report Sachin Patel

Ramesh Kumar Kollana

TCS Innovation Labs Tata Consultancy Services Pune, India [email protected]

Assurance Services Unit Tata Consultancy Services Bhubaneshwar, India [email protected]

Abstract—Organizations are making substantial investments in Enterprise Software Implementation (ESI). IT service providers (ITSP) execute a large number of ESI projects, and these projects tend to have severe cost and schedule constraints. Tata Consultancy Services (TCS) is an ITSP that has developed a reusable test case repository with an objective of reducing the cost of testing in ESI projects. We conducted a study of the reuse program to analyze its impact. A set of metrics were defined and relevant data was gathered from users of the repository. In this paper, we document the findings of the study. Users of this repository have reported cost savings of up to 14.5% in their test design efforts. We describe the repository structure, reuse metrics and experience gained through reuse. We motivate the need for further research on test notations/meta-models for transaction-oriented business applications and variability management within these models.

organization of Harmony, reuse metrics, lessons learnt and motivations for research in test assets reuse. II.

I. INTRODUCTION Enterprise software (ES) is the term used to describe information systems designed to integrate and optimize the business operations in an enterprise. These include operations such as enterprise resource planning, accounting, business intelligence, customer relationship management, human resource management, and so on. An IT spending forecast by Gartner [1] estimates the total spending on ES at $297 billion, which is 7.89% of the worldwide IT spend in 2013. Industrial experiences [2] indicate that testing an Enterprise Software Implementation (ESI) may account for 25% to 50% of the total effort, depending on the number of custom features. In order to reduce this significant effort in the testing process, TCS has developed a reusable test case repository, which has helped reduce up to 14.5% of the test design effort in ESI projects. ESI projects involve tasks such as business process reengineering, configuration, customization, integration, testing, data migration, training and so on. They require specialized domain, technology and management skills and it is prevalent to choose an IT service provider to execute them. Tata Consultancy Services (TCS) is an IT services, consulting and business solutions organization. TCS provides testing and endto-end implementation services for ES such as SAP, Oracle ERP, Peoplesoft, Seibel and so on. In order to promote reuse in ESI projects, TCS has built a repository of reusable test cases, referred to as Harmony, hence forth. We defined a set of metrics to measure the impact of reuse and conducted a series of interviews with developers and users of Harmony to gather relevant data. In the following sections, we describe the

Specialized Instances

Standard Instances Product 1 A

C

Product 2 Product 3

Business Process Y

Pass ?

...

D

N end

E

Product 1 & Industry 1 For example : Oracle EBS for Semi Conductor industry

.. .

Scenarios A

C

E

Keywords—enterprise software implementation; test case reuse; test repository; variability management

ORGANIZATION OF HARMONY

A

A

C

C

D

...

.. .

...

.. .

D

Test Cases

Figure 1 Organization of Harmony

Figure 1 shows the hierarchical organization of Harmony. The topmost layer consists of various products and modules for which testing services are provided. The second layer consists of diagrams for the business processes in all the modules. Each business process is sub-divided into scenarios belonging to that process. The last layer contains the test cases designed to test each scenario. These test cases are designed for the standard instance of the product. The standard test cases are also specialized into industry specific repositories so that, they may have a higher reuse potential in projects of that industry. When a new ESI project is initiated, the Harmony team provides a specialized instance of the particular industry and product, if available; else a standard instance is provided. III. REUSE METRICS A number of software reuse metrics have been suggested in literature. Various categories of metrics such as: cost-benefit analysis, maturity assessment models, amount of reuse, reusability, reuse library metrics have been defined [3][4]. While most of these can be adapted to measure test case reuse and are useful from different viewpoints, it would be expensive to implement all of them. We used the Goal Question Metrics (GQM) [6] approach to select a subset of the metrics that are most relevant to our objective.

A. GQM approach The primary objective of the reuse program is to reduce testing costs through reuse. Table 1 shows the questions and metrics that were listed in the process. TABLE I.

GQM MODEL TO IDENTIFY REUSE M ETRICS

Goal : Reuse test cases to reduce the cost of testing ES implementations Purpose (1) Reduce Issue (2) Cost Object (3.1) Testing Object (3.2) Reuse test cases Viewpoint (4.1) ES Implementation Viewpoint (4.2) Organization Questions? Metric New Development Cost (NDC) What is the effort Relative Cost of Reuse (RCR) (2) (3.1) spent in developing Relative cost of writing for reuse test cases? (RCWR) Reuse % (3.2) (4.1) How many test (4.2) cases are reused? Total Reuse Level Return on Investment (ROI) Is this investment (2) (4.2) profitable? Payoff threshold Understandability Is the reuse library (3.2) organized Search Effectiveness optimally? Changeability Reuse % How does the Reuse Cost Avoidance (RCA) (2) (4.1) implementation Development Cost Avoidance (DCA) project benefit? Service Cost Avoidance (SCA)

B. Metric Definitions The metrics that were selected for the measurement program are defined below. In these definitions, we treat cost as equivalent to effort. Metric definitions have been adapted from [3][4] and [5] and are divided in 3 categories 1) Cost Benefit Metrics 1. 2. 3. 4. 5. 6. 7. 8.

New Development Cost (NDC): The historical cost to develop new test cases (# of person hours per test case) Relative Cost of Reuse (RCR): The portion of effort needed to reuse a test case versus writing it new (ratio) Relative Cost of Writing for Reuse (RCWR): The portion of the effort needed to write a reusable test case versus writing it for one-time use only. (ratio) Development Cost Avoidance (DCA): The cost avoided by reuse in the development phase (person months) Service Cost Avoidance (SCA): The cost avoided by reuse in the maintenance phase (person months) Reuse Cost Avoidance (RCA): The total financial benefit to a project due to reuse. (DCA + SCA person months) Return on Investment (ROI): The total financial benefit to the organization due to reuse (ratio: savings / investment) Payoff threshold: Number of reuses needed to recover all test case development costs (# of reused test cases) 2) Amount of Reuse Metrics

9.

Reuse %: Number of test cases reused divided by total number of test cases delivered in the project (ratio) 10. Total Reuse Level: Total number of test cases reused divided by total number of test cases delivered in the organization (ratio)

3) Reusability Assessment Metrics [5] provides some suggestions on measuring test case reusability. However, we did not have observable data for calculating reusability metrics. We have analyzed these aspects qualitatively from feedback provided by users. 11. Understandability: How well the classification method helps users understand the reusable test cases. 12. Search Effectiveness: How well the classification methods help users locate the reusable test cases. 13. Changeability: A test case is changeable if its structure and style are such that any changes can be made easily, completely, and consistently. IV. MEASUREMENT AND ANALYSIS We conducted interviews with 6 ES implementation testing teams and the Harmony development team. A questionnaire was developed with the objective of gathering qualitative and quantitative data relevant to the metrics listed in III.B. Following are few of the questions that were used in the discussions. 1. 2. 3. 4. 5. 6. 7.

What is the size and organization of the repository? How much effort went into making the repository? How many test cases were re-used without change / reused with changes / newly added? How much effort is required for re-use without change / re-use with changes / newly added test cases? How do you get updates from Harmony? What search criteria do you use when finding test cases? What kinds of changes are needed after reuse?

In Table II., we have listed the data gathered and the metrics calculated. We have not listed customer names and product names for the reason of confidentiality. The products include popular ES products like SAP, Oracle EBS, Peoplesoft and Seibel. Harmony consists of 16285 test cases for 6 ES products. We have not listed specialized instances since they have not been reused yet. The Implementation project data shows that Reuse % depends on the Level of customization in the implementation. Project C had a standard implementation with very few customizations and could reuse 90.1% of the test cases, whereas Project D was heavily customized and could reuse only 10.4% of the test cases. Other projects where the level of customization was “average” showed Reuse % in the range of 17% to 30%. Another important finding is that RCR is more than 0.4 for all projects other than C. This is attributed to the high number of “test cases reused with some change”. 55% of the reused test cases had to be changed after reuse. Majority of the changes were in navigations, form fields and naming conventions. Project F had many process level changes and has the highest RCR amongst the projects studied. Testers have to analyze every process, scenario and test case to make the changes. This causes the RCR to be high and RCA to be low. The project teams mentioned that they had no benefit of reuse in the maintenance phase since for every change they had to make the necessary changes in their copy of test cases. That is the reason why SCA is zero for all projects.

TABLE II.

# 1 2 3 4 5 6

Harmony Repository Data Product name # test cases Product 1 5288 Product 2 2216 Product 3 3851 Product 4 2590 Product 5 2161 Product 6 179 Total 16285

a b c d e f g

Development Effort RCWR Total Investment Total Savings Return on Investment Payoff Threshold Total Reuse Level

254.14 1 254.14 21.71 0.09 28738 9.16

REUSE METRICS

Enterprise Software Implementation Project Data Observed Data A B C D h Level of customization Avg. Avg. Low High i # of Product modules implemented 12 16 2 4 j # of newly developed test cases 770 2200 43 1632 k # of test cases reused without change 0 37 392 189 l # of test cases reused with some change 330 691 0 0 m Total # of test cases reused (k + l) 330 728 392 189 n Total # of test cases delivered (j + m) 1100 2928 435 1821 Calculated Metrics o Reuse % 30 24.86 90.11 10.38 p New test case Dev. effort (NDC) 17.17 45.69 6.79 28.42 q Relative Cost of Reuse (RCR) 0.5 0.4 0.2 0.5 r Effort for reusing test cases 2.58 4.54 1.22 1.47 s Development Cost Avoidance (DCA) 2.58 6.82 4.89 1.47 t Service Cost Avoidance (SCA) 0 0 0 0 u Reuse Cost Avoidance (RCA) 2.58 6.82 4.89 1.47 v % Effort saved 13.04 13.57 61.08 4.93

The ROI from this program is 0.09. ROI of 1 indicates that investments have paid off and ROI more than 1 indicates that investment is profitable. The investments will be paid off when 28738 test cases have been reused. At this point, 2292 test cases have been reused. Respondents told us that one of the pre-requisites to understand test cases is to understand the domain and product functionality. After that, it is easy to understand the process diagrams, scenarios and test cases. The user compares the project specific process with the Harmony process and narrows down on the scenarios to be reused. The user then reads the test cases for each scenario to identify candidates for reuse. This structure makes the repository easy to understand. A simplistic measure of search effectiveness is the ratio: number of relevant items found divided by total number of items returned by the search. The commonly used search criteria were process name, module name, scenario name and form name. The repository classification structure has all of these criteria except form name, which is embedded in the text. This helps the user to narrow down the search to scenario level. Hence, the maximum number of irrelevant test cases in a search cannot be more that the number of test cases in that scenario. Each scenario may have between 10 to 50 test cases depending on its complexity. For a repository containing 16285 test cases this number is relatively small. We analyzed the types of changes made to test cases in order to understand the aspect of changeability. Changes were of following types: process, scenario, form field, form navigation, naming conventions and test data. We posit that changeability is high if all the parts that need change have been separately specified and if the change has to be specified only once. Except for process, scenario and test data all other change types do not meet this requirement of changeability. V. OPPORTUNITIES FOR IMPROVEMENT In this section, we list the opportunities identified for improving the reuse program.

E Avg. 3 1540 398 202 600 2140

F Avg. 2 249 0 53 53 302

Total

28.04 33.40 0.4 3.75 5.62 0 5.62 15.13

17.55 4.71 0.6 0.50 0.33 0 0.33 6.35

26.27 136.2 0.43 14.06 21.71 0 21.71 14.45

39 6434 1016 1276 2292 8726

A. Service Cost Avoidance (SCA) Users of Harmony make a copy of the reusable test cases and merge them with other test cases developed in the project. If Harmony evolves due to changes in the domain or product upgrades, there is no way to automatically propagate changes to the users. Hence when there are changes the user has to analyze every test case and make necessary updates. This is the reason there are no reuse benefits in maintenance (SCA = 0). In case of components, one can link components dynamically or inherit / extend them to modify behavior, however there is no such mechanism available for test cases. If a technique to dynamically link test cases and modifications is developed, SCA could be increased. B. Relative Cost of Writing for Reuse (RCWR) An analysis of the repository reveals that there are many common features across various products and industry specific instances. For example the “Order to Cash” process is similar across different ERP products. However, every product would implement the process with subtle variations. Similarly there would be variations of the process across different industries. In the current structure, there is no way to specify the common and variable parts of the test repository. Other sources of variation include navigation, form fields and so on. A separate repository instance is being created for every variant. This causes RWCR to be high. This increases the investment cost and reduces ROI. This issue motivates us to consider variability modeling in the test case repository. C. Changeability The test cases are written in a natural language and contains fields like test scenario name, test description / steps, test data and expected results. Information like navigation, form field names are embedded in the test description. This structure makes it difficult to make changes that are global in nature. For example, if a new field is added to one of the forms, the user has to find all the test cases where this form is referred and

make the modifications. Changeability also has a direct corelation with RCR. Since more than half of test cases have to be changed after reuse, better changeability will result in lower RCR. Ideally, we would like to have a test notation / metamodel such that information such as scenarios, oracles, test data, UI interfaces are specified only once. This will reduce redundancy of information and number of changes required, which will in turn improve changeability. VI.

MOTIVATIONS FOR RESEARCH

In this section, we discuss the need for a test model for transaction-oriented business applications. A. Test Notation/Meta-model for transaction-oriented applications The common practice in ESI projects in to specify test cases is in a tabular form with the following information: Test steps explain the navigation and form fields to be entered on the user interface. Test data is specified or described for the major form fields of each step. Different observations are made at each step and compared with an expected behavior. Information about the data setup required for a test case is sometimes written in a separate column and sometimes is part of the test steps. The standards that have been defined for test notations are UML Test profile (UTP) [8] and Testing and Test Control Notation (TTCN) [9]. UTP provides a complete meta-model covering all testing concepts. The test behavior specification in UTP can be one of the UML structures such as a statetransition diagram, activity diagram or sequence diagram. None of these are designed to specify the information that we would like to specify in the test case. For example, one cannot specify user activity in the state-transition diagram. Activity diagrams can be used to specify user activity but one cannot specify oracles and navigations in activity diagrams. Sequence diagrams can be used to specify navigations but it will be difficult to describe form fields, test data and corresponding oracles on the diagram. TTCN is another notation widely used in a variety of domains such as telecommunications, automotive and medical devices [9]. There have also been case studies on application of TTCN3 for testing of Web applications [10]. Our objective is to specify test cases in a non-redundant manner and not necessarily to automate their execution. For example, if the user interface (UI) model of the product could be specified separately from the test behavior, then changes to the user interface do not have to be made in every test case, but can be made in UI model and propagated to all test cases. We could not find an easy way to do such things with TTCN-3. We suggest that a notation or an extension of existing standards (UTP, TTCN-3) be developed for specifying test cases of transaction-oriented business applications. B. Variability management in test models Once a suitable test model/notation is developed another need arises, that is of variability modeling. Variability modeling has received considerable attention from the research community. A review of the literature in variability

management [11] indicates that Feature Models (FM), use case diagrams and UML models have been used to model variability. In a case study [7], a FM was used to express variability and features were mapped to test cases. It was found that majority of the test cases span across multiple features which results in a many to many mapping between features and test cases. This results in too many test cases being selected while doing feature-based selection, many of them are not necessarily relevant to the search. In our opinion, this issue may not arise if variability is modeled in the test artifact instead of an application model. We have not come across cases where variability was modeled within the test artifact. Authors of [7] also discuss the need for variability management in test artifacts. In section V.B we discussed the dimensions of variation that exist in the Harmony repository. The repository needs to be continuously updated as new product versions are released. Similarly, after a product update, it should be possible to distribute new test cases to the people who have re-used Harmony test cases. This can be facilitated by developing a meta-model based test repository where commonality and variability can be specified. ACKNOWLEDGMENT The authors would like to thank the developers and the users of Harmony for sharing their experiences and insights on the reuse program. REFERENCES [1] [2]

http://www.gartner.com/id=2388215 Gerrard, P., "Test Methods and Tools for ERP Implementations," Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION, 2007, vol., no., pp.40,46, 10-14 Sept. 2007 [3] William Frakes and Carol Terry. “Software reuse: metrics and models”, ACM Comput. Surveys.Vol. 28, 2 (June 1996) [4] Poulin, Jeffrey S. and Joseph M. Caruso, “A Reuse Measurement and Return on Investment Model,” Proceedings of the Second International Workshop on Software Reusability, Lucca, Italy, 24-26 March 1993, pp. 152-166. [5] Zhang Juan; Lizhi Cai; Weiqing Tong; Yuan Song; Li Ying, "Test Case Reusability Metrics Model", 2nd International Conference on Computer Technology and Development (ICCTD), vol., no., pp.294,298 [6] V. Basili, G. Caldiera, and H.D. Rombach, “Goal Question Metric Approach”, Encyclopedia of Software Engineering, pp. 528-532, John Wiley & Sons, Inc., 1994 [7] Sachin Patel, Priya Gupta, Vipul Shah, “Feature Interaction Testing of Variability Intensive Systems”, 4th International workshop on Product Line Approaches in Software Engg., ICSE 2013 [8] http://utp.omg.org/ [9] Ina Schieferdecker, “Test Automation with TTCN-3 - State of the Art and a Future Perspective”, Testing Software and Systems, Lecture Notes in Computer Science Volume 6435, 2010, pp 1-14 [10] Cosmin Rentea, Ina Schieferdecker, Valentin Cristea, “Ensuring Quality of Web Applications by Client-side Testing Using TTCN-3”, 9th International Conference on Web Engineering [11] Lianping Chen, Muhammad Ali Babar, Nour Ali, “Variability management in software product lines: a systematic review”, In Proceedings of the 13th International Software Product Line Conference (SPLC '09). Carnegie Mellon University, Pittsburgh, PA, USA, 81-90.