SOFTWARE ENGINEERING PROJECT MANAGEMENT Submitted by

Using Defect Taxonomies for Testing Requirements

ELG5100 - SOFTWARE ENGINEERING PROJECT MANAGEMENT A project report on

DEFECT TAXONOMIES FOR TESTING REQUIREMENTS

Submitted by Aarthi Thirumavalavan (100958800), M.Eng ECE, Carleton University

Deepak AnandaReddy(100967820), M.Eng ECE, Carleton University

Submitted to Professor Shervin Shirmohammadi in partial fulfillment of the requirements for the course ELG 5100

1


Abstract Requirements of any projects defines what exactly are the client needs. To deliver the final product to the client or customer it has to be testified which can be done by testing. Testing should be done from the initial stage of the project so that any mistakes can be corrected at the beginning stages instead of building a non-functional product. Initial stage testing refers to testing the requirements which will be discussed in this paper. For testing requirements we make use RTDT (requirement based testing with defect taxonomies) where the defects can be classified and linked, based on the existing techniques so that the bugs can be identified and linked where exactly failure is from which helps in correcting them quickly due to linking of defects. Why RTDT and advantages of using this RTDT will be discussed in this paper.

2


Contents Abstract ........................................................................................................................................... 2 1) INTRODUCTION ...................................................................................................................... 5 2) REQUIREMENTS BASED TESTING WITH DEFECT TAXONOMIES............................... 6 2.1) Requirements Specification: ................................................................................................ 7 2.2) Compare Costs and Decide on Application ......................................................................... 7 2.3) Create Defect Taxonomy ..................................................................................................... 8 3) Case Study .................................................................................................................................. 9 3.1) RT or RTDT ?...................................................................................................................... 9 3.2) Linking Requirements and Defect Taxonomy................................................................... 10 3.3) Test Planning ..................................................................................................................... 11 3.4) Test Design ........................................................................................................................ 12 3.5) Test Execution ................................................................................................................... 13 4) Results ...................................................................................................................................... 13 4.1) RTDT Results .................................................................................................................... 13 5) CONCLUSION & FUTURE WORK....................................................................................... 14 6) REFRENCES............................................................................................................................ 15

3


Figure 1 : Requirement based testing.............................................................................................. 5 Figure 2 : Survey by James Martin on software project failures .................................................... 5 Figure 3 : Process of requirements–based testing with defect taxonomies (RTDT) ...................... 7 Figure 4 : Defect Taxonomy of project under study....................................................................8 Figure 5 : Agile Method Work-Flow[6] ......................................................................................... 9 Figure 6 : Test Plan Methodology ................................................................................................ 12

4


1) INTRODUCTION To begin with, requirement based testing can be defined as the experimenting approach in which the test cases, conditions and data are derived from requirements. It is derived based on few functional and non functional attributes. According to recent research works, most of the software projects do not meet the schedule and budget goals specified by the estimate. One of the main reasons behind these failures is the poor quality of software which leads to rework of requirements, designing and coding. The figure.1 shows the process of requirement based testing.

Figure 1 : Requirement based testing According to a survey conducted by James Martin on defects in software projects [12], the results indicated that about 56% of the defects were contributed by poorly written or faulty or unclear requirements and the rest 44% is because of the design, coding and other defects. The survey results can be seen in the figure 2.

Figure 2 : Survey by James Martin on software project failures According to the survey results from [13], faulty requirements accounts to 82% of the application rework, 44% of the project cancellations and only 54% of the project requirements are sustained throughout the course of a project. Another research by the Standish group in 2000 showed that $84 billion was spent by American companies on cancelled software projects and another $192 billion on software projects exceeding their schedule and budget limits. The top reasons of these cancellation being : i) incomplete requirement specifications; ii) frequent alterations in requirements and iii) faulty user inputs to requirements. 5


From all these survey results, it is quite apparent that requirement defect contributes to more than half of the project failures and should be handled by an alternative approach. That is where the role of defect taxonomy comes in. A defect, in this context, is a deviation in a software document from the nearest correct document that makes it locally incorrect. Taxonomy, in general, is a system of classifying things. Summing up, defect taxonomy is the system of categorizing defects. Defect taxonomy yields information about the distribution of faults and failures in a project. It is also worthy of learning what kinds of errors occur in a software project.  

Understanding process characteristics: Recording defects and classifying them can help understand the process with respect to all those activities that produce defects. Understanding product characteristics: -Identifying points needing rework: Clusters of defects point to documents that need rework.

-Suggesting rework approaches: The kinds of defects may point out the best approach for doing the rework (e.g. direct repair, review, redesign, retest, etc.) 

Providing project guidance: Defect characteristics of artifacts may help with risk assessment for process decisions such as task priorities, go/no-go decisions, reimplementation decisions etc.

2) REQUIREMENTS BASED TESTING WITH DEFECT TAXONOMIES Michael Federer and Armin Beer in their work [1] has introduced a system-testing approach based on defect taxonomies which is known as the requirement based testing with defect taxonomies. This approach is executed using the classic test processes. The phases of testing requirements - test planning, designing, execution and evaluation are carried out using defect taxonomies. The figure.3 displays the various steps involved in requirement based testing process and also marks the direction of process flow and data flow.

6


Figure 3 : Process of requirements–based testing with defect taxonomies (RTDT)

2.1) Requirements Specification: The analyst along with the domain experts analyzes and lists out the requirements. Each requirement specified in the document has a unique identifier, name, a description, artifacts and priority value. The artifacts are important for specifying test cases. They can be use cases or business rules or graphical user interface. The priority value determines the importance of the defect, ranging between low, medium or high.

2.2) Compare Costs and Decide on Application Based on the requirements specifications, the cost of the project is estimated on the basis of cumulated time effort during the testing phases. It is the role of the testing manager and the project manager to decide whether the project is carried out with RT or RTDT. They together compare the estimated cost of RT and RTDT and whichever is cost efficient is carried out throughout the project. In the case study carried out in this paper, it has been seen that the cumulative time effort for RTDT is lesser than RT. Hence, RTDT has been applied in this project. 7


2.3) Create Defect Taxonomy A generic defect taxonomy was first introduced by Beizer [3] and it is being widely used in software testing. The following are the top-level nine categories defined by Beizer : (1) requirements, (2) features and functionality, (3) structural defects, (4) data, (5) implementation and coding, (6) integration, (7) system, software architecture, (8) test definition and execution as well as (9) unclassified defects. The defect taxonomy for the case under study is based on Beizer's theory of defect taxonomy. The figure.4 shows the defect taxonomy of the project under study. It consists of three levels of abstractions starting with the Beizer's taxonomy on the column 1. The higher level categories are further mapped to product specific categories which are in turn marked to lower level categories. The lower level categories have an assigned identifier and severity level. The value of severity levels in this project study ranges between blocker, critical, major, normal, minor, or trivial.

Figure 4 : Defect Taxonomy of project under study

8


3) Case Study The above methods are applied to a project which was from a public health insurance institution where they make use of iterative and incremental type of process for the development of project which follows agile model[6], Agile methodology for development has its own advantages where this type of development requires the entire team to be integrated as there is lot of information to be communicated because the project comes into live in many versions and many builds which implies the project is delivered more often and they are frequent feedback from the clients regarding the fixes and changes which is very important concern in our case as our testing takes place around requirement stage. The working of agile model can be seen from below figure

Figure 5 : Agile Method Work-Flow[6] The project which we are using for RTDT is a web application which was developed to a institution employees who help handicapped people. This project took nine months with help of seven engineers and about in four iterations and it is a web application which is service oriented which supports various applications.

3.1) RT or RTDT ? The given project had 41 requirements which were in the standard format which had an identifier, name, description, artifacts and priority value. Other information like use case diagram are to be written from process description in accordance with the business rules from out of these test case design can be written from requirements from which test cases can be written from which Testing can be done[8]. Here we consider one of the important requirement to decide which one is better either RT or RTDT. From [1] REQ_0027 which covers the search functionality which is used search client and his information. For this particular from the use 9


case diagram eleven paths and nine decision path from which all the test cases can be written for search and its results function. As we have seen RT and RTDT has its own advantages and disadvantages for every the project the test manager or the test lead will decide which method to be used for the classification of the defects which will raised during the test execution phase. The classification is based on how much it cost for testing which we can get a estimate during early stages by the manager. The cost depends on the number of requirements from which n number of test cases will be written. In this project the break-even point i.e. the estimated cost for RTDT was lesser when compared to RT so we use RTDT in this project.

3.2) Linking Requirements and Defect Taxonomy The main objective of this paper is to link the defects which are seen from the requirements to the defect taxonomy. We have already discussed a classified Defect Taxonomy in the previous sections of this paper. Because of this linking the defect to the requirement one can easily identify which functionality is facing the problem so it can be rightly attended and assigned to right developer for rectifying the problem which has arise in less duration of time and also due to linking we can make sure the entire project is assigned for testing without missing any of the requirements because there is a provision for that in the taxonomy where all aspects of the project are covered because all parts of codes from regular statements to all loops, decision and branches are covered based on their functionality and requirements. The taxonomy also covers the non functional and performance testing of the project like load and stress can also be linked. Therefore making sure the entire project is covered. The quality analyst will link the requirements and the taxonomy later which will be approved by the test manager. Here the tester will give priority to the requirements based on their functionality and dependency for example without logging into the web application the staff cannot search the client which implies the login page has to be tested first later on to the search. As the execution phase moves on we can see the defects which will be fall onto the specified defect categories. The testing requirement in our case is the search functionality REQ_0027 was assigned to the taxonomy which was given F1 in the Defect Categories(DC) when referred to the table it reads 'erroneous identification of a client is critical during a search' but for this requirement we can also assign to one more defect categories in the taxonomy table i.e. F7 which reads 'incorrect display of data' because of this linking we can observe the following factors  

RTDT have control on requirement quality criteria Using Taxonomy additional features were detected 10




Due to priority and weighted DC the quality of requirements completeness, verifiability, traceability can be improved

Coming back to the search functionality REQ_0027 which was initially assigned to medium priority after linking to the defect taxonomy table it was changed to critical as it is mentioned in the figure given below which is part of the table and due to its priority and defect severity after consulting the tester it was changed to critical and also while linking the U1 was not linked to any requirement which states that the performance testing was not linked and U1 functionality was that which was taken care of later making sure everything in the project was covered.

3.3) Test Planning As the name suggest the strategy of the testing life cycle will be planned here. Planning were as in SDLC or STLC life cycle will have equal importance, usually planning will carried on by test manager or the test lead which covers important aspects like planning of schedule, dividing the modules among the testers, what sort of tools to be used in testing, resources required for the testing phase, scope as of how much has to be covered in testing, which methodology will be used by testers and finally documentation of the entire testing life cycle so one can proper idea regarding the project and also helps in best practices and lessons learnt in the current project which will help in future similar projects. In our current project we make use of a Technique which was proposed by Derk-Jan de Grood where the technique was listed in a table[9] which is shown in below figure in the form of table and the public health insurance made use of that technique. From the table we can technique category and strength which are of three types low medium and high based on the strength along with the priority and severity from the defect category one can decide the number of test cases to be written based on which the cost factor and also the quality of the product depends on. Also from the techniques we can see scope at which it can cover any given project. It makes sure all requirement at least have a technique in the table to cover it from CRUD to the database and all the statements.

11


Figure 6 : Test Plan Methodology For our example REQ_0027 which had a high priority and its severity was critical which we decided from the linking of defect taxonomy so the test manger for our particular requirement had to assign a strength of three which is high from which number of test cases were decided to be fourteen out of which eleven were due to use case diagram and three to search for the special symbols so minimum of fourteen test cases had to be written for the particular requirement.

3.4) Test Design Here from the requirements one will write the test scenarios which defines a part of main functionality and from these scenarios we can write the test cases which has both positive and negative scenarios for the given requirement which will done by the QA or the testers. This is scripting phase of testing which is important and challengeable so that one can cover entire project but keeping time factor has key attribute. Based on the linked defect category with requirements, techniques and numbers tester has to write test cases which covers all the functionality of the project. Designing of test cases starts with use case description which links directly to the requirement, also contains precondition, expected results and actual steps to conduct the test. For our case REQ_0027 along to the fourteen test cases, eight were added which gives negatives test cases to check the error messages therefore a total of twenty-two cases as per standard procedure were written

12


3.5) Test Execution The written test cases will be made ready so that they can be executed will be maintained in the test infrastructure or the tool. Execution can be carried out manually or by making use automation which can be done with the help of automation tools which has test harness or writing the automated test scripts. The project traceability can be done here which give the status of the testing in execution phase to determine the status of the project. In our example REQ_0027 testing was done manually where if we have a few test cases to be executed then the time required is also less but when they are more test cases with less resources or with less time then testers can prioritize test cases based on the functionality, defect category and proceed with the execution so it can be completed well within the time. The defects which arise are to reported and documented so that it can be fixed where the defect is considered as bug and it has bug defect life cycle[10] where a developer will be assigned the bug for fixing and later retesting until the defect as been resolved.

4) Results After the execution the results of the execution which gives the status of the project and the quality of the product. Therefore after a execution cycle results will be documented by the manager and from the number of test cases passed and total number of test cases executed from their ratio and also based on the severity from which the quality can be determined. Based on the defect taxonomy table the severity of defect a better quality of hot fixes will be released For the given requirement REQ_0027 after the execution five defects were reported out of which one was assigned major severity but from the table it has to be given a critical severity because a important functionality was failed where a handicapped client was wrongly identified

4.1) RTDT Results By using RTDT test cases were improved and also taught us lessons on the tool support, requirements quality and also the cost factor which can be seen in detail below, For the same project if we had used the RT then there would have been more test cases which might increase the chances of the more defects. For our project A, with DEFECT TAXONOMY Number of requirements (NOR) = 41 Number of test cases (NOT) = 148 Number of logged defects (NOF) = 169 For our project B(similar to A), without DEFECT TAXONOMY Number of requirements (NOR) = 28 13


Number of test cases (NOT) = 170 Number of logged defects (NOF) = 114 Therefore, Number of test cases per requirement for , A = 3.61 B = 6.07 Effectiveness =

==> For A = 1.14 and For B = 0.67

By comparing A and B values for number of test cases were less in A which made use of defect taxonomy is half of B and the effectiveness of A is twice the B which was due to the Defect Taxonomy. These were not only measured quantitatively but also qualitatively by asking the testers and mangers who worked on it where they concluded saying RTDT helped them in finding high severity defects along with which the failures were assigned with accurate severity, bug releases were planned and released and also the final release of the project were scheduled and error feature deliveries were controlled.

5) CONCLUSION & FUTURE WORK Manual Testing make use of professional standard tools for conducting the testing but when RTDT were to be used additional tool support is required which is a very low requirement as RTDT requires spreadsheet by using them defect taxonomy can be linked to requirements , design technique and to the failures which serves the purpose. Therefore to conclude we can say due to effectiveness which was shown in our project one can say that the factors of requirements like the completeness, verifiability and exact detailed in requirements were proven which shows the quality was maintained for the project and along with that due to less number of test cases the time as well as the cost factor was reduced due to the use of RTDT and also tool required is spreadsheet and training testers in RTDT is a easy task where in overall there were many benefits due to RTDT, the main two factors are  

requirements quality was better and also the test cases from which the overall quality of product was increased Not only the product but also the process of making the product was improved where the releases and hot fixes were scheduled on time

Usage of RTDT were to be spread into vast field of project management where not only testing phase but also to the other stages of the project. The main field of implementation to be 14


considered in the risk management were Risk-based-testing with the help of RTDT has to studied so that entire project will successful.

6) REFRENCES Felderer, M. (n.d.). "Using Defect Taxonomies for Testing Requirements Creating a Defect Taxonomy". [1]

[2] M. Felderer and A. Beer, "Estimating the return on investment of defect taxonomy supported system testing in industrial projects". (SEAA 12),2012,pp. 426-430 [3] B.Beizer, Software Testing Techniques, Int'l Thomson Computer Press, 1990. [4] Akimov, E., Mikheeva, S., & Sinkin, Y. (2009). Requirements Testing Methodology in industrial projects, 197–202.

Liu, G., Huang, S., & Piao, X. (2008). Study on Requirement Testing Method Based on Alpha-Beta Cut-Off Procedure. 2008 International Conference on Internet Computing in Science and Engineering, 396–402. http://doi.org/10.1109/ICICSE.2008.36 [5]

Nguyen-cong, D. (2013). A review of effort estimation studies in agile, iterative and incremental software development. The 2013 RIVF International Conference on Computing & Communication Technologies - Research, Innovation, and Vision for Future (RIVF), 27–30. http://doi.org/10.1109/RIVF.2013.6719861 [6]

[7] "Bugzilla, ITrackers

and other bug trackers", Nicolas Serrano, Ismael Ciordia(2005). open

source, 11–13. Xu, S., Guo, R., Bai, Y., Zhang, Y., & Li, Y. (2013). A methodology for test suite optimization based on testing requirement. 2013 IEEE 4th International Conference on Software Engineering and Service Science, 216–219. http://doi.org/10.1109/ICSESS.2013.6615291 [8]

[9] D–J de Grood, Test Goal : Results- Driven Testing, Springer, 2008. [10] Van

Moll, J. H., Jacobs, J. C., Freimut, B., & Trienekens, J. J. M. (2002). The importance of life cycle modeling to defect detection and prevention. Intermag Europe 2002 Digest of Technical Papers. 2002 IEEE International Magnetics Conference (Cat.No.02CH37323), 144– 155. http://doi.org/10.1109/STEP.2002.1267624 [11] Mirarab, S., Ganjali, A., Tahvildari, L., Li, S., Liu, W., & Morrissey, M. (2008). A requirement-based software testing framework: An industrial practice. 2008 IEEE International Conference on Software Maintenance, 452–455. http://doi.org/10.1109/ICSM.2008.4658102 [12] James Martin,.,(1984), "An information systems manifesto", Prentice-Hall International, 1984. [13] Borland, (2006), "Eliminate the Testing Bottleneck". http://www.borland.com/us/solutions/ lifecycle-qualitymanagement/requirements-based-testing.html[15.02.2009]

15