Exploratory Software Testing in Agile Project

Exploratory Software Testing in Agile Project Beni Suranto Department of Informatics, Faculty of Industrial Technology Universitas Islam Indonesia Yogyakarta, Indonesia [email protected] Abstract—Software testing is an important process in a software development life-cycle as validation and verification mechanisms to guarantee the quality of the intended software product. Exploratory testing is a software testing where the testers may interact with the system in whatever way they want and use the information the system provides to react and generally explore the system functionalities without restraint. Exploratory testing allows the full power of the human brain to be brought to bear on finding bugs and verifying functionality without preconceived restrictions. This paper discuss about enacting exploratory software testing in a software project which was developing Student Marking System (SMS) in University XYZ. Keywords-software testing; exploratory; agile

I.

EXPLORATORY SOFTWARE TESTING

Software testing is an important process in a software development life-cycle as a “validation and verification” mechanism that has following goals: •

•

•

Validating whether the software is the right product (i.e. meeting the applicable standards and customer needs, whether it works as expected) [1]. A software can be labeled as a right product when it provide the functionalities needed by the customer, behave as the customer’s expectation, and give high level usability to help the customer solve their problems. Without testing process it is impossible for developer and customer to judge whether a software is a right product or not. In the context of Student Mark System (SMS) project, the developer team must perform the testing process to validate the system whether it is be able to help the stakeholders (i.e. teachers, assessment coordinators, faculty staff) to perform all marking stuff in an effective and efficient way or not. Verifying whether the software is right (i.e. matching the requirement specification)[2]. Testing as a verification mechanism is used to judge whether the functionalities embedded in the software is developed as specified in the user requirement specification. In the SMS project, the stakeholders use every weekly meeting to verify whether the demonstrated features correctly represent the requirements specified in the previous meeting or not. To discover faults, defects, errors, bugs in the software where the behaviour of the software is

incorrect, undesirable or does not conform to its specification[3]. Testing also very important to detect all possible undesirable system behaviour, such as incorrect computations, system crashes, data corruption, etc. Since the SMS deals with many computations and processes of “very important” data (i.e. student’s marks) , the developer team must perform the testing process to ensure that the system doesn’t bring any risk of unacceptable errors or bugs. Exploratory software testing is a software testing process with the scripts are removed entirely (or, the scripts’s rigidness relaxed) [4]. In this method, testers may interact with the system in whatever way they want and use the information the system provides to react and generally explore the system’s functionality without restraint. Advocates argue that exploratory testing allows the full power of the human brain to be brought to bear on finding bugs and verifying functionality without preconceived restrictions [5]. As one of software testing methods, exploratory testing has following goals [5]: •

To gain an understanding of how an application works, what its interface looks like, and what functionality it implements: In the SMS project, this goal is achieved with perform two different testing process. First, the developer team perform exploratory testings in the group meetings. Second, the developer team with the stakeholders together perform exploratory testing in the weekly client meetings. These two are complementary processes. Some bugs that were not discovered in the first process may be discovered in the second process. It is very often since the developer team and the stakeholders sometimes use different ways to explore the functionalities of the system.

•

To force the software to exhibit its capabilities: The idea is to make the software work hard and to ask it hard questions that put it through its paces. This may or may not find bugs, but it will certainly provide evidence that the software performs the function for which it was designed and that it satisfies its requirements.The main purpose of the SMS project is to help the stakeholders perform all marking stuff related processes in an effective and efficient way. It has a lot of specific requirements since different faculty might have different “ways” to do the marking

process for their students. To deal with this issue, some staff from different schools (i.e. School of Education and School of Chemistry and Physics) were involved in the exploratory testing in the weekly client meeting sessions to help the developer team to verify whether the system is satisfy their requirements or not. •

To find bugs: Exploring the edges of the application and hitting potential soft spots is a specialty of exploratory testing. The goal is purposeful which is exploration to identify untested and historically buggy functionality. Exploratory testing is well suited to be used by developer team which is developing modern web system using agile methods[3]. In an agile project, development cycles are short, leaving little time for formal script writing and maintenance. Also, system features often evolve quickly, so minimizing dependent artifacts (i.e. pre-prepared test cases) is a desirable attribute. The SMS project is using agile method with one week iteration process and each week it is very often that the stakeholders ask for changing some features the way they want.

The drawback to exploratory testing is that testers risk wasting a great deal of time wandering around an application looking for things to test and trying to find bugs[4]. The lack of preparation, structure, and guidance can lead to many unproductive hours and retesting the same functionality over and over, particularly when multiple testers or test teams are involved. This is where guidance comes into play. The developer team of the SMS project uses a technique called Session-Based Team Management (SBTM) to perform exploratory testings during the system development life cycle. II.

about 60-90 minutes for a exploratory testing session for each charter. •

Reviewable Result: takes the form of a session sheet, a page of text that follows a formal structure. This structure includes charter, coverage areas (not code coverage; typically product areas, product elements, quality criteria, or test techniques); start time, tester name(s); time breakdown; data files; test notes; bugs (where a “bug” is a problem that the tester reasonably believe represents a threat to the value of the product); issues (where an “issue” is a problem that threatens the value of the testing process—missing information, tools that are unavailable, expertise that might be required, questions that the tester might develop through the course of the session)

•

Debriefing: a conversation between the tester who performed the session and someone else—ideally a test lead or a test manager, but perhaps simply another tester. In the debrief, the session sheet is checked to make sure that it’s readable and understandable; the testers discuss the bugs and issues that were found; the testers makes sure that the protocol is being followed; and collaboration happen. The developer team of the SMS project usually spent about 15-30 minutes to perform debriefing for every exploratory testing session.

SESSION-BASED TEST MANAGEMENT (SBTM)

SBTM which was invented by James and Jonathan Bach in 2000 is a well-known guidance that can be used in exploratory testing method[6]. SBTM has four elements: •

Charter: a one- to three-sentence mission for a testing session. The charter is designed to be open-ended and inclusive, encouraging the tester to explore the application and affording opportunities for variation. Charters are not aimed to be comprehensive descriptions of what should be done, but the total set of charters for the entire project should include everything that is reasonably testable. Figure 1 shows two examples of charters that were used in exploratory testing for the SMS project.

•

Time Box: some period of time between 45 minutes and 2 ¼ hours, where a short session is one hour (+/15 minutes), a long session is two, and a normal session is 90 minutes. The intention here is to make the session short enough for accurate reporting, changes in plans (such as a session being impossible due to a broken build, or a session changing its charter because of a new priority), but long enough to perform appropriate setup, to get some good testing in, and to make debriefing efficient. Software engineers working on the SMS project usually spent

Fig. 1. Examples of charters in the exploratory testing for the SMS project

Figure 2 shows an example of reviewable result of exploratory testing for the SMS project.

components/assessments/parts maximum mark)

names

and

their

•

Actual marks input by the user using manual input or using the “mark upload” feature.

•

Every value that user select from all GUI input elements in every page (e.g. dropdown list for grading scheme, dropdown list for manual grade, dropdown list for supplementary exam, dropdown list for histogram data set, radio button for histogram bininterval)

B. State In exploratory testing context, a state of software is defined as “..a coordinate in the state space that contains exactly one value for every internal data structure.”[5] The state space is the cross-product of every internal variable in the system. It contains large number of different internal states that represent whether the system provide the right or wrong response to a particular input. Software state is a result of interaction between the software itself with its history of receiving input and its environment. The software’s state changes when inputs are applied and stored internally. Exploratory testing aims to test these changes. Is the software state updated properly according to the inputs ? Does the state of the software cause undesirable behavior ? Is there any inputs lead the software getting into states it should never be in ? Following are some examples of software states for the SMS which were tested with exploratory testing method : Fig. 2. An example of reviewable result in exploratory testing for the SMS Project

III.

OBJECTS IN EXPLORATORY SOFTWARE TESTING

A. User Input A general definition of an input is “..a stimulus generated from an environment that causes the system under test to respond in some manner.”[3]. The key point of the definition above is that an input originates from outside the system (e.g. given by the user) and causes some code in the system to execute. Following are some example of user inputs that were be considered in exploratory testing for the SMS : •

Username and password input by the user in the login page when the user want to login into the system.

•

Every click done by the user in all clickable things in every page (e.g. login button, SMS main menus, SMS tab menus, course link in the course navigation page, column header in the class list page, add/edit/delete button in assessment configuration page)

•

All data that the user must input to configure the component marking for a particular class (i.e.

•

Different types of user (e.g. admin, teacher) should get different states of the main page.

•

When the teacher upload new marks for a particular assessment, the states of all student data in the corresponding class should be updated.

•

When the the states should be list shoud scheme)

•

When the admin append the enrolment data with a new feed, the states of all affected classes should be updated (i.e. some new students will be included in the classes and some existing students in the classes will be labelled as “unenrolled”)

admin change the grading scheme option, of all class in the corresponding term updated (i.e. the final grades in the class be reproduced based on the new grading

C. User Data Current software generally interact with a large data store using database management system and/or file management system. The most challenging task in the user data testing process is how the testers be able to create a database with specific data as similar as possible to the type of data we expect our users to have[7].

Some important characteristics of real user database that must be considered by the testers in exploratory testing are : •

Since data is added and modified the real user database will evolve and it can grow very large.

•

Very often that there are data relationships and structures in the real user database that the testers unable to infer.

•

The real user database often require a very big storage space that is not available in the testing process.

Following are user data that were tested with exploratory testing in the SMS project : •

•

Enrolment Data: This data is used to populate the database, especially for the tables related with the enrolment processes. The developer team of the SMS project use anonymous files as enrolment feeds (given by the stakeholders) to test following functionalities : o

Upload enrolment data

o

Append enrolment data

o

Data reconciliation

Marks Data: This data contains marks for a particular assessment(s) in a particular class. This data is uploaded using “upload mark” feature which is available in the class list page. The developer team of the SMS project use marks data to test following functionalities : o

Marks data preview

o

Identifier and assessment columns mapping

o

Class list data update

The developer team of the SMS project use commaseparated values (csv) files for both enrolment data and marks data in the testing processes. D. Environment The definition of an environment is depends on the type of the system being tested. There are some elements contruct an environment of a particular system: the operating system which the system is running, the various applications that might interact with the system, file and setting that may affect the system, the network that the system is connected, and so on and so forth. In exploratory testing, the testers must consider anything that can affect the behavior of the system as part of the environment[7]. Following are some elements that were considered as parts of the environment in exploratory testing for the SMS : •

Operating system (Windows and Linux)

•

Browser (Internet Explorer, Google Chrome, Mozilla Firefox, Safari, Opera)

•

Hardware (different machines, different displays)

•

Network (localhost and network server)

From the results of the previous exploratory testing processes, the developer team of the SMS project found that browser was the element which brought many bugs and issues (i.e. compability problems). IV.

CONCLUSION

Exploratory testing is one of software testing methods that use “explore and find” approach to verify the functionalities and detect bugs in the system. This method is well-suited to be adopted in agile projects since it doesn’t need a lot of time to create the test plans (compared to the script-based methods) and also it encourages collaboration among software engineers in the developer team and between the developer team and the stakeholders of the system. SBTM give well-designed guidance to be used in exploratory software testing in order to manage the testing process. Some important objects that must be considered in exploratory software testing are input, user data, software state, and environment. REFERENCES [1] [2] [3] [4] [5] [6] [7]

Sommerville, Ian 2007, Software Engineering 8th Edition, AddisonWesley, New York. Ammann, Oaul 2008, Introduction to Software Testing, Cambridge University Press, New York R.D. Craig and S.P. Jaskiel, Systematic Software Testing, Boston: Artech House Publishers, 2002.Reference 5 "Exploratory software testing." SciTech Book News Dec. 2009. Academic OneFile. Web. 25 Sept. 2012. Bach, J. (2003). Exploratory Testing Explained. (http://www.satisfice.com/articles/et-article.pdf) J. Bach, "Session-Based Test Management", STQE, vol. 2, no. 6, 2000. J. Bach, "Exploratory Testing", in The Testing Practitioner, Second ed., E. van Veenendaal Ed., Den Bosch: UTN Publishers, 2004, pp. 253-265.