Towards the Selection of Testable Use Cases and a ...

Towards the Selection of Testable Use Cases and a Real Experience Andrea Rodrigues, Plácido Rogério Pinheiro, Maikol Magalhães Rodrigues, Adriano Bessa Albuquerque, Francisca Márcia Gonçalves University of Fortaleza (UNIFOR) – Graduate Program in Applied Computer Sciences Av. Washington Soares, 1321 - Bl J Sl 30 - 60.811-341 - Fortaleza – Ce – Brazil

Abstract. Nowadays, software organizations have needed to develop excellent and reliable products. This scenario has helped to increase the relevance of quality assurance activities, especially the testing discipline. However, sometimes the time and resources are limited and not all tests can be executed, demanding organizations to decide what use cases should be tested, to guarantee the predictability of the project’s time and budget. The multiple criteria methodologies support decisions, considering many factors, not only professional experience. This paper presents a multiple criteria model to support the decision of selecting the use cases that should be tested. Keywords: Tests, Decision and Analysis Resolution, Multiple Criteria Decision Analysis.

1 Introduction Software testing is one of the disciplines that have the capability of providing assistance to improve the quality of an organization’s products, because its goal is to evaluate how the product meets the clients’ specified requirements through a controlled execution of the software. In some cases, when there is not enough time and resources to guarantee complete test coverage, software organizations should reduce their scope. This work has as main objective the implementation of an approach based on a methodology that provides a structured support to multicriteria decisions. This methodology assists the process of deciding which use cases should be selected to be tested by removing subjective decisions in a structured way. The multicriteria methodology helps to generate knowledge about the decision context and thus increases confidence of those making decisions on the outcome results [15]. Other applications which have multicriteria involving software engineering were defined in [9], [10] and [13]. This research was carried out to define and execute an approach to support software organizations at selecting use cases that should be tested. The approach was based on the model defined in [7].

2

Software Test

Testing consists of verifying, dynamically, the software behavior to determine if it adheres to the specifications and executes correctly in the projected environment. As the main objective is to find software failures, this activity has a destructive nature [5]. Software testing focuses on the product’s quality and can not be considered elementary, because many factors can compromise the success of this activity’s execution: (i) time limitations; (ii) resource limitations; (iii) lack of skilled professionals; (iv) insufficient knowledge of test procedures and techniques and adequate test planning; (v) subjectivity of requirement and test specifications; and (vi) increase of the systems’ complexity. Moreover, difficulties related to the test activity are also due to the great variety of combinations of input and output data and the large quantity of paths that make it infeasible to execute all possible test cases [2]. Test case is the definition of a specific condition to be tested. Its structure is based on input values, restrictions to its execution and expected results or behaviors [3]. Test cases are designed based on use cases, which represent interactions that users, external systems and hardware have with the software aiming to achieve an objective. The amount of test cases executed is one of the main factors which may influence the cost of testing. Therefore it is fundamental to define a test scope considering acceptance criteria and business risks [12]. Test criteria uphold the selection and evaluation of test cases aiming to increase the possibilities to provoke failures and to establish a high level of reliability on the products’ errors correction [6]. The criteria may be classified as: (i) test coverage criterion; (ii) test cases adequacy criterion; and (iii) test cases generation criterion. 3 Multiple Criteria Decision Analysis (MCDA) Decision-making should be exploited when deciding to execute or not some activities or to perform them applying some methods [7]. The multiple criteria decision analysis proposes to reduce the subjectivity on the decision-making. Nevertheless, the subjectivity will be always present, because the mathematically analyzed items are always results of human beings’ opinions. These multiple criteria models allow the decision-maker to analyze possible consequences of each action to obtain the best understanding of the relationships between actions and their goals [6]. Objective criteria should be considered, as well as subjective criteria, even being generally disperse and diffuse in a decision context, but they are extremely important to assess actions. 3.1

MACBETH Approach

The MACBETH methodology contemplates the understanding and learning of the problem content and is divided into three phases: structure, evaluation and recommendation, as we can see in Figure 1 [1].

Fig. 1. Phases of the Multiple Criteria Decision Aid The structure phase focuses on constructing a formal model, capable of being accepted by actors as a structure to represent and organize an entire group of evaluation criteria. It consists of analyzing a specific system and making potential alternatives of decision explicit. The evaluation phase produces matrixes of judgments and provides scales of cardinal value for every criterion. The tasks are implemented with the MACBETH methodology. In the recommendation phase, the results generated by MACBETH are analyzed using scales of values generated by the matrixes of judgments, which are composed of various actions that must be examined according to the decision-maker evaluation [7]. Structuring activities include: (i) definition of a family of fundamental points of view (FPV), (ii) construction of descriptors; and (iii) estimation of the impacts profiles of each action [1]. The construction of descriptors comprehends three stages: (i) description of each descriptor for each fundamental point of view; (ii) access impacts for each fundamental point of view, and (iii) analysis of impacts according to the fundamental points of view [14]. The descriptors are desired to: (i) turn operational the analysis of impacts of the options in a FPV, (ii) describe the impacts with respect to FPVs, (iii) improve the structure of the evaluation model, and (iv) verify the ordinal independence of the corresponding FPVs. The FPV becomes operational if there’s a set of impact levels associated with it, defined by Nj, which should be sorted in descending order by the decision makers. Thus, they constitute a range of local preference, limited by the higher level N*j, that has more attractiveness and the lower level N*j, of less attractiveness, should meet the following pre-ordering condition:

N *j

 Nk

1, j

Nk, j

Nk

1, j

 N* j .

The main difference of MACBETH [11] to other multiple criteria methods is that it requires only qualitative judgments of the difference between the elements’ attractiveness aiming to assign values to the options for each criterion and to weigh up the criteria. MACBETH applies the concept of attractiveness to measure the potential actions’ values. Therefore, when the decision-maker is demanded to judge the value of a potential action on a specific situation, he should think about his attraction to that action [4]. HIVIEW [8] is a tool to evaluate models defined using multiple criteria methodologies with an aggregation function, like MACBETH. By using HIVIEW, the decision-maker defines, analyzes, evaluates and justifies his preferences, considering

existent alternatives. This software facilitates a lot the analysis of complex problems, supporting the elaboration of the problem’s structure, specifying the criteria used to choose alternatives and to assign weights to the criteria. The alternatives are evaluated by comparing these criteria and a preference value is assigned to each alternative’s criteria. Additionally, it’s possible to change judgments and to compare the obtained answers graphically, providing information to the decision-maker for reevaluation. If necessary the decision can be rectified. 4

Model to Select What Use Cases Should Be Tested and the Experience of Use

The proposed model is based on the model presented on [7] and is composed of generic steps, grouped by the phases of the Multiple Criteria Decision Aid (MCDA). These steps are described in Table 1. Table 1. Model’s steps Phase

Step

Structure 1. Identify criteria

2. Identify actors and their weights

3. Assign priorities to criteria

4. Execute a partial evaluation

Description Identify criteria to be used on the use cases evaluation aiming to define their level of priority. Identify roles that will expose their point of view, considering also their roles on the decision-making process. Each actor should assign a weight to all criteria, considering the full test process and not only a specific use case. After the execution of the steps listed above, it is necessary to standardize the three sets of values, putting them on the same base (base 1). The goal is to perform a partial evaluation correctly, without any bias. Then, for each actor the three variables should be multiplied, considering each criterion, thus obtaining a specific score.

5. Calculate the general scores of the Calculate, for each actor, a score to criteria each criteria of each use case. Evaluation

6. Apply scores on MACBETH

Construct the matrix of judgments and obtain the cardinal value scales for each defined criteria.

Recommendation

7. Define the level of priorities of the Prioritize use cases which will be use cases tested, given an analysis of the results obtained from the previous phases.

4.1

Experience of use

The organization where the experience of use was performed is a government institution with 270 professionals allocated on the Information Technology Area, working on projects that support the organization’s businesses. In 2004, a test team was organized and being external to the projects and responsible for executing systemic tests. Nowadays, the company has high demands of time and resources which make it difficult to satisfy the desired testing coverage in all projects. Therefore, many projects reduce their testing scope to assure the delivery schedule. Priorities have to be applied to use cases and accordingly selected to decrease the testing scope. This is quite relative and varies according to the actors involved and to the criteria they judge relevant. The pilot project selected to apply the proposed approach was a project with a schedule restriction, because the organization agreed on a date with the workers’ union, resulting on a fine if it got delayed. The project’s life cycle was iterative/incremental and the model was applied on the project’s first iteration. The following use cases were part of the test cycle on which the model was applied: UC01_Execute_Sign_In_and_Sign_Out;UC04_ Search_Problems; UC05_Demand_Benefit_Permission; UC07_Search_ Demands_ Benefit_Permission;UC08_Approve_Demands_Benefit_Permission; UC10_Manage_Parameters; UC12_Excuse_Sign_in_and_Sign_out. Bellow, we explain how the steps were performed. 4.1.1 Structure In step “Identify criteria”, we held a meeting with all actors involved at selecting the criteria, given the criteria listed on Table 2. The selected criteria are those highlighted. A criterion (c) is a tool to evaluate tests that are susceptible to automation in terms of a certain point of view (PV) or concern of the actors responsible for the analysis. The quantity of criteria (n) may vary for each project.

Table 2. List of criteria Criteria

Reason

Question

Functionality (ISO 9126)

Do the specific functions and The specific functions and properties of the properties of the product satisfy the product should satisfy the user. user?

Reliability (ISO 9126)

The use case requires maturity on fault tolerance Does the use case require maturity on aspects. fault tolerance aspects?

Usability (ISO 9126)

The use case requires a strong interaction with Does the use case require a strong the user and therefore must be usable. interaction with the user, being essential a high level of usability?

Portability (ISO 9126)

The use case will be executed in different Will the use case be executed in environments. different environments?

Security (ISO 9126)

The use case requires a specific control, Does the use case require a specific reducing the access to information. control to reduce the access to informations?

Availability of time/resource (Gonçalves et al., 2006)

If a use case requires specific resources (people Are there enough and qualified with technological expertise, software and resources to test the use case? hardware resources, available budget) so that the test can be perfomed.

Repeatability The use case is being implemented on the first Will the use case’s test be repeated (Gonçalves et iterations and so will be tested many times. many times? al., 2006) Complexity The use case has a large quantity of associated Does the use case have a lot of (Gonçalves et business rules or depends on complex associated business rules or depend al., 2006) calculations. of complex calculations to function adequately? User The use case needs to be implemented because Does the use case have to be requirements it will satisfy any contractual or legal demand implemented because it will satisfy (Dustin, 2002) or the organization may have financial loss. any contractual or legal demand or it will prevent the organization against large financial loss? Operational The use case is related to the functions which Is the use case related characteristics are frequently used. frequently used functions? (Dustin, 2002)

to

In step “Identify actors and their weights”, the following actors were selected: (i) Project manager; (ii) Tests coordinator; (iii) Project’s system analyst; and (iv) Project’s test analyst. The actors answered the questions related to each criterion and the questionnaire applied to obtain the actors’ weight, which was defined considering the role performed by the actor on the project e his knowledge of the business for which the software was developed. A questionnaire was elaborated to obtain the weight of each actor (weight of actor – WA), embracing actor’s experience in activities related to tests; roles performed; participation in projects; training and participation in test conferences. Each item had a value and with the measurement of all items, the actor’s weight was obtained.

In step “Assign prioritization to the criteria”, each actor (a) classified the criteria according to their relevance for the project’s test process. In step “Execute a partial evaluation”, we multiplied, for each use case, the values of the actors’ point of view (PV), the actors’ weights (WA) and the criterion’s level of priorities, as can be seen on the formula below: Ex (a, c) = [PVx (a, c)] * [WA (a, c)] * [Level of Prioritization (a, c)] It is important to emphasize that the obtained values of the actors’ weights and level of priorities were equalized (on the same base – base 1), after they were informed so that a correct evaluation was possible without benefiting a value to the detriment of another. Finally, in step “Calculate the general scores of the criteria”, we calculated the median to obtain the final score of the use cases for each criterion. The value of the median, calculated for each use case, will be used as a basis to prioritize. The following equation illustrates the median calculation. x

x ((j/2) + 1)

[E x, (j+1)/2], otherwise. where x: represents the user case, j: number of actors and c: criterion 4.1.2 Evaluation The evaluation phase was supported by the HIVIEW software. In step “Apply scores on MACBETH”, we elaborated the matrixes of judgments in MACBETH for each criterion, having to calculate their subtraction of attractiveness. For this, the values of the median of the criterion are subtracted between the use cases (x1 to xk), considering the module of the obtained value. Figure 2 illustrates the matrix of judgments for the “Operational characteristics” criterion.

Fig. 2. Matrix of judgements to the criterion “Operational Characteristics”

Figure 3 depicts the level of importance for each analyzed criterion. According to the figure, the criterion with the highest weight was “Operational characteristics” and the one with the lowest weight was “Security”.

Fig. 3. Criteria’s level of prioritization 4.1.3 Recommendation The results obtained during the evaluation phase generate reports with graphics, and the use cases analyzed are ranked. In this step, we identified the use cases with the highest level of priorities. To represent the general classification of all analyzed criteria we used the graphic showed in Figure 4.

Fig. 4. Selection of Use Cases As we can see, the use case with the highest level of priorities was UC01_Execute_Sign_In_and_Sign_Out and the lowest level was UC12_Excuse_Sign_in_and_Sign_out.

4.2 Validating the results of the experience of use The model was applied at the beginning of the test cycle and we decided to test all use cases, aiming to analyze its adequacy and efficacy and to compare the obtained results. At the end of the test cycle, we calculated the percentage of errors, considering the quantity of use cases tested and the quantity of detected errors for each use case as displayed in Figure 5.

Fig. 5. Percentage of detected errors to Use Cases We could observe, considering the level of priorities assigned to the use cases obtained by the model execution, that the model assigned the highest level of priority to those use cases with a higher concentration of errors. However, the level of priority of UC07 was higher than those priorities of UC08, UC10 and UC12, but even so in UC07 were detected fewer errors.

5 Conclusion and Further Works This work helped us to conclude that the restriction of time and resources is a real problem on many organizations and that an approach to support the decision to select use cases to be tested is very relevant. Besides, with the experience, we could see that the execution of the proposed model was satisfactory, allowing us to take the decision not using just our subjective experiences. The application of the MACBETH approach was adequate to model the problem, but, others Multi Criteria Decision Aiding (MCDA) methodologies can be used or, if necessary, we can define a combination of them to define the model. As further works, it will be important to apply the model in other software projects and to define customized questionnaires for other projects’ characteristics. The proposed approach can be used in other contexts, such as: the selection of processes improvements to be deployed, as it often is not possible to introduce all the improvements at once, given the difficulty of assessing and measuring the effectiveness of implemented improvements. The selection of systemic tests that can be automated is another scenario where we can apply the model. Criteria such as cost, availability of resources and repeatability of tests should be considered.

References 1. Bana & Costa, C. A.; Beinat, E. & Vickerman, R. (2001). Model-structuring and impact assessment: qualitative analysis of policy attractiveness. CEG-IST Working Paper n.25. 2. Bandeira, L. R. P.: Product Quality from Test Automation: a Case Study. Monograph to obtain the B.Sc in Computer Science Title, University of the Ceara State (2005) 3. Craig, R. D., Jaskiel, S. P.: Systematic Software Testing, Artech House Publishers, Boston (2002) 4. Cunha, J. A. O. Decisius: Um Processo de Apoio à Decisão e sua Aplicação na Definição de um Índice de Produtividade para Projetos de Software, 2008, Dissertation of M.Sc., CIN/UFPE, Recife, PE, Brazil. 5. Dias, A.: Introduction to Software Testing. Magazine of Software Engineering (on-line), available at http:// www.devmedia.com.br/esm (2007) 6. Flament, M.: Multiple Criteria Glossary. Red Iberoamericana de Evaluación y Decisión Multicritério, Espanha, available at http://www.unesco.org.uy/redm/glosariom.htm (1999). 7. Gonçalves, F. M. G. S.; Donegan, P. M.; Castro, A. K.; Pinheiro, P. R.; Carvalho, A.; Belchior, A. D. : Multi-criteria Model for Selection of Automated System Tests. In: The IFIP International Conference on Research and Practical Issues of Enterprise Information Systems, 2006, Viena. IFIP International Federation for Information Processing (2006). 8. Pinheiro, P.R.; Castro, A.; Pinheiro, M. A Multicriteria Model Applied in the Diagnosis of Alzheimer Disease: A Bayesian Network, 11th IEEE International Conference on Computational Science and Engineering, July 2008, Page(s):15–22, Digital Object Identifier: 10.1109/CSE.2008.44 9. Marilia M.; CARVALHO, A. L.; FURTADO, M. E. S.; PINHEIRO, P. R.: A coevolutionary interaction design of digital TVapplications based on verbal decision analysis of user experiences. International Journal of Digital Culture and Electronic Tourism (IJDCET), v. 1, p. 312-324, 2009. 10.Marilia M.; CARVALHO, A. L.; FURTADO, M. E. S.; PINHEIRO, P. R. . Co-evolutionary Interaction Design of Digital TV Applications Based on Verbal Decision Analysis of User Experiences. WSKS 2008. Communications in Computer and Information Science, v. 19, p. 702-711, 2008. 1 Digital Object Identifier: 0.1007/978-3-540-87783-7 11. Advances in Decision Analysis, N. Meskens, M. Roubens (eds.), 1999, Kluwer

Academic Publishers, Book Series: Mathematical Modelling: Theory and Applications, vol. 4, pp. 131-157 12. Myers, G. J.: The Art of Software Testing, John Wiley & Sons, Inc., New York (1979) 13. OLIVEIRA, J. M. M.; Oliveira, K. B.; CASTRO, A. K.; PINHEIRO, P. R.; BELCHIOR, A. D. . Application of Multi-Criteria to Perform an Organizational Measurement Process. In: Elleithy, Khaled (Ed.). (Org.). Advances and Innovations in Systems, Computing Sciences and Software Engineering. Berlin Heidelberg: Springer Verlag, 2007, p. 247-252. 14.PINHEIRO, P. R.; SOUZA, G. G. C.; CASTRO, A. K. A.: Structuring of the

Multicriteria Problem for the Production of Journal. Pesqui. Oper. 2008, vol.28, n.2, pp. 203-216. Digital Object Identifier 10.1590/S0101-74382008000200002. 15.Evangelou, C.; Karacapilidis, N.; Khaled, O. A.: Interweaving knowledge

management, argumentation and decision making in a collaborative setting: the KAD ontology model. Int. J. of Knowledge and Learning 2005 - Vol. 1, No.1/2, pp. 130 - 145 Softwares Hiview Software, Catalyze Ltd. www.catalyze.co.uk M-Macbeth, www.m-macbeth.com