Advances in test generation for testing software and systems

International Journal on Software Tools for Technology Transfer manuscript No. (will be inserted by the editor)

Advances in test generation for testing software and systems An introduction to selected papers from ICTSS 2013 H¨ usn¨ u Yenig¨ un · Cemal Yilmaz · Andreas Ulrich

Received: date / Accepted: date

Abstract Among the software quality assurance techniques, testing stands out as one of the most frequently used techniques in practice. Consequently, testing is also widely studied in research. An important aspect in testing that receives much attention is the problem of automated test generation. This introductory paper summarizes the papers selected from the 25th IFIP International Conference on Testing Software and Systems (ICTSS 2013), which all deal with this topic. The field is briefly introduced by pointing out to some of the existing survey articles and relating the selected papers in this issue to each other and to the known body of knowledge. Keywords Software testing · Test generation · Model-based testing

1 Introduction Edsger W. Dijkstra’s famous statement “Program testing can be used to show the presence of bugs, but never to show their absence!” [6, pg. 7] is often used as a negative remark about testing. However, the first half of the statement is actually the very positive aspect, that testing can show the presence of bugs. When it comes to practice, this aspect of testing is the most dominant one. The second half of the statement is often H. Yenig¨ un Sabanci University, Istanbul, Turkey E-mail: [email protected] C. Yılmaz Sabanci University, Istanbul, Turkey E-mail: [email protected] A. Ulrich Siemens AG, Corporate Technology, Munich, Germany E-mail: [email protected]

considered as the weak point of testing that limits its usefulness. The alternative technique of program verification that aims at showing the absence of bugs tries to overcome this limitation. However even for verification techniques, the ability to showcase bugs is eminently important. Robert P. Kurshan shares his experience on the technology transfer of verification techniques in [14], where he argues that the real value of the verification methods lies in their power of falsification, i.e. in their ability to show the existence of bugs. Combined with the ease of application, testing has been the most widely applied software quality assurance technique over many decades. Due to its successful practical applications, considerable research efforts have been put into improving the effectiveness of testing in order to scale the techniques for handling ever more complex software systems. Thereof, the automation of testing activities is the key factor for improving test effectiveness. Automation covers four main activities: – Generation of tests, – Execution of these tests against the system under test, – Evaluation of the results from test executions, and – Management of tests, their execution and recorded results. Test automation is therefore a topic that is frequently discussed at a number of international research conferences on testing. One of them is the IFIP International Conference on Testing Software and Systems (ICTSS), which is a series of conferences addressing the conceptual, theoretic, and practical problems of testing of software systems comprising communication protocols, services, distributed platforms, middleware, embedded and cyber-physical systems as well as security infrastructures.

2

This special section of the International Journal on Software Tools for Technology Transfer contains six articles selected from the best papers of the 25th ICTSS conference which took place in Istanbul, Turkey, on November 13–15, 2013. All of them deal with the particular problem of test generation and model-based testing in a varying number of application cases. It is the second opportunity that selected papers from ICTSS are published in this journal. The first one covered the topic of model-based testing already in 2012 [23]. The re-occurrence of this subject emphasizes its importance in research as well as in practical application. 2 A brief review of test generation approaches Test generation has been commonly identified as an important research challenge, and it is one of the topics widely studied by researchers. In 2014, Alessandro Orso and Gregg Rothermel presented the results of an informal survey that they conducted by asking researchers in testing to comment on the most notable achievements of the research efforts and on the open challenges of the field [21]. The most common keywords in the responses from the experts are visually summarized in the form of a word cloud [21, Fig. 1]). In this figure the word “generation” stands out together with few others, such as “tools” and “practice”, as one of the terms most frequently mentioned in the responses from the experts. This observation confirms the importance of the topic of test generation approaches that need to come along with good tool support to cover practical needs. The recent survey in [1] covers techniques for test case and test data generation, which comprises different sections on various techniques including symbolic execution, model-based testing, combinatorial interaction testing, adaptive random testing and search-based testing that are all written by recognized experts in their respective fields. Besides providing a short introduction to these techniques, some open problems and existing tools implementing the techniques are addressed. The same techniques are also surveyed in [21]. For model-based testing, which can be dated back to the 1950s [19], an extensive survey is given in [3] and a systematic review of available approaches can be found in [5]. Quite a number of approaches based on models of different natures and using different formalisms have emerged over the time. However, Finite State Machine (FSM) based and Labeled Transition System (LTS) based approaches dominate this field of research. Since the field has been originated from functional testing of sequential circuits, where an FSM (usually a Mealy machine) is a natural modeling formalism, FSM

H¨ usn¨ u Yenig¨ un et al.

based approaches predate LTS based approaches. An FSM specifies the complete behavior of a System Under Test (SUT), and under some conditions (see e.g. [4, 17]) it is possible to derive complete test suites, i.e. test suites with full fault coverage. One line of research in FSM based testing has been to optimize the length of test sequences [30, 12, 28], while preserving the completeness of test suites. On the other hand, test derivation techniques that are based on some coverage criteria, such as transition coverage, state coverage, etc. have also been studied. We see that despite the initial focus on test generation using complete and deterministic FSMs, there is a recent and increasing interest on non-deterministic FSMs [25, 24, 15]. Such a trend is natural, as it is desirable to extend the applicability of these methods to wider and more complex classes of specifications and implementations. Describing the required behavior of highly complex systems at an abstract level, necessarily brings the notion of non-determinism into the picture. The other dominating formalism, LTS, is a more general formalism than the FSM modeling approach in the following sense. The Mealy machine semantics of an FSM require that the action associated by a state transition consists of an input applied to the SUT and an output produced by the SUT as a reaction to this input. An LTS, or more precisely an Input Output Transition Systems (IOTS), requires an action to be either an input, or an output, or a special unobservable action. Due to non-determinism in an IOTS, a conformance relation is introduced to produce the verdict of an executed test. The well-known ioco conformance relation was initially introduced by Tretmans [29]. It has been extended by researchers to address other types of systems such as timed systems [20, 16], symbolic transition systems [9], distributed testing systems [11], and concurrent systems [18]. The control part of the behavior can be modeled by an FSM or by an LTS easily. However, in order to take into account the data part of the system, these modeling formalisms need to be extended appropriately. Extended FSMs and Symbolic Transition Systems augment FSM and LTS formalisms by explicit modeling and handling of the data part. These extensions definitely simplify the modeling of data dependent behavior of the system. When generating tests from models with an explicit data part, one obvious approach could be to unfold the data. However, this would easily cause state explosion and it would also be a missed opportunity to take advantage of the semantical information coming from the data part that could be used while generating tests. In order to generate tests from these extended formalisms, specialized methods have been suggested. For

Advances in test generation for testing software and systems

example, Frantzen et al. treat data symbolically in their test generation algorithm [9]. Another promising approach is to partition the input data space into finitely many equivalence classes where each equivalence class consists of values triggering an equivalent behavior to each other [13]. When this is possible, a representative input value from each equivalence class is considered as a test case, accounting for the testing of the behavior of the SUT under the entire equivalence class. For both FSM and LTS based approaches, the test generation process analyzes the given model to construct a test suite. This analysis is based on the given test purpose or fault model, and can range from very simplistic graph traversal algorithms, such as the construction of a tour of all the transitions in the model, to complex analysis techniques, such as the application of model checking algorithms. The use of model checking for test generation is based on the idea of formulating test purposes as temporal logic formulae, and using model checkers to produce a counter example for the negation of these formulae. These counter examples serve as a test suite exercising the given test purposes. Applying model checking, a technique initially developed for the purpose of verification, for the generation of tests has been an interesting and fruitful combination of the two worlds, and a survey of the field is given in [10]. The cost of using formal methods is easily affordable for safety critical systems and therefore the technology transfer resistance is lower in such areas. For example, an overview of applying model checking for the verification of PLC software that is widely used in industrial automation is given by Ovatman et al. [22]. The current state of technology transfer of the testing approaches briefly introduced above can be characterized by the amount of tool support for them and their application in industry. A recent survey and a systematic review of the available commercial tools can be found in [27]. Regarding the use of the techniques in industry, an interesting survey has been carried out by Binder et al. to identify the impact of model-based testing in practice [2]. One of the findings is that almost 50% of the survey respondents (who are model-based testing practitioners) have a rollout/general use of the techniques in their projects. Although this is a considerable ratio, there is still the remaining 50% that needs to be won over. This would be possible by further research and development of testing methods and tools. We believe that the analysis of the results from this survey can provide hints to researchers on needed improvements for a better penetration of the model-based testing approaches in the industrial practice, and on the acceleration of the necessary technology transfer.

3

3 Selected papers From the ICTSS 2013 proceedings, six papers were selected that fit into the scope of test generation and model-based testing. The authors were asked to submit extended versions that are published now in this special section. These articles represent the latest advances in the field and cover various aspects of test generation. The article “On adaptive experiments for nondeterministic finite state machines” [15] by Natalia Kushik, Khaled El-Fakih, Nina Yevtushenko, and Ana R. Cavalli visits the topic of state identification and state verification for finite state machines. Generating tests from finite state machine specifications has already a long history. The research in this topic has recently focused on non-deterministic and incomplete specifications. The authors consider the construction of adaptive homing and distinguishing experiments (i.e. tests) from nondeterministic and possibly non-observable finite state machines. The necessary and sufficient conditions for the existence of such tests are given, and algorithms to derive them are provided together with a discussion on the upper bounds for the size of these tests. The topic of state identification for deterministic finite state machines is carried over in the article by WenLing Huang and Jan Peleska on “Complete model-based equivalence class testing” [13]. The authors present a strategy for equivalence class partitioning of the input domain such that the test generation problem can be reduced to the construction of a test suite that is complete for a given fault model. The application of the approach using model-based testing tools is explained and demonstrated in a case study on a SysML model. The next two articles use Petri nets as the basis to generate tests. Petri nets are known to be a more appropriate modeling notation when dealing with concurrent behavior. Jo˜ao Pascoal Faria and Ana C. R. Paiva contributed to this special section with their article titled “A toolset for conformance testing against UML sequence diagrams based on event-driven colored Petri nets” [8]. UML sequence diagrams are frequently used in practice for constructing partial behavioral models to express important test scenarios. The authors improve the existing work by considering the conversion of sequence diagrams to extended Petri nets, which combines ideas from colored Petri nets and event-driven Petri nets. An implementation of the technique and a case study performed with the tool are also presented. The second article on test generation from Petri nets is written by Hern´an Ponce de Le´on, Stefan Haar, and Delphine Longuet and titled “Model-based testing for concurrent systems: unfolding-based test selection” [18]. It considers Petri net specifications for mod-

4

eling concurrency and combines them with a new conformance relation called co-ioco, an extension of the known ioco relation to describe conformance of a concurrent implementation to the given Petri net specification. The presented test generation algorithm makes use of Petri net unfoldings. The authors also discuss different test coverage criteria to reduce testing efforts. After dealing with the generation of tests, they need to be implemented in executable tests next. This task is not trivial when the system shall be tested under real-life conditions that limits the possibility of the automated tester to reset the system under test after each test run. This problem is discussed in detail in the next article on “Generating test case chains for reactive systems” [26] by Peter Schrammel, Tom Melham, and Daniel Kroening. The authors attempt to minimize the overall execution time of a test suite by reducing the number of system resets between subsequent tests as much as possible. An implementation of the approach is reported together with an experimental evaluation against some benchmark examples. The test generation approaches introduced above deal with testing of systems that react asynchronously to changes of their environment. Another computing paradigm for reactive systems is based on synchronous programming that performs computations in a sequence of logical ticks, similar to the synchronous abstraction in digital circuits. Synchronous languages that build on this paradigm are frequently applied in industrial automation systems that use programmable logic controllers (PLCs). Testing of PLC programs has gained more attention in research recently, not least because factory and process automation solutions become more and more complex and, hence, error-prone. The last article on “Automated test generation using model checking: an industrial evaluation” by Eduard P. Enoiu, Adˇ sevi´c, Thomas J. Ostrand, Elaine J. Weyuker, nan Cauˇ Daniel Sundmark, and Paul Pettersson [7] deals exactly with this class of systems. It presents an evaluation of the use of model-checking techniques for test generation from PLC programs. The approach is evaluated on a number of examples taken from a train control and management system. Overall, the selected articles show the progress in the field of test generation. They make clear that new approaches need to be based on a strong formal foundation in order to solve practically relevant problems. Equally important, robust tools must be provided to deal with the peculiarities in industrial practice. The articles give indications about the direction research and tool development are heading for.

H¨ usn¨ u Yenig¨ un et al. Acknowledgements We would like to thank the authors of

the selected articles for their motivation to provide extended versions of their work originally submitted to ICTSS 2013, the reviewers of the articles for their relentless efforts on providing feedback and holding up the quality of the submitted work, and all members of the technical program committee of ICTSS 2013.

References 1. Anand, S., Burke, E.K., Chen, T.Y., Clark, J.A., Cohen, M.B., Grieskamp, W., Harman, M., Harrold, M.J., McMinn, P.: An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86(8), 1978–2001 (2013) 2. Binder, R.V., Legeard, B., Kramer, A.: Model-based testing: Where does it stand? Communications of the ACM 58(2), 52–56 (2015) 3. Broy, M., Jonsson, B., Katoen, J., Leucker, M., Pretschner, A. (eds.): Model-Based Testing of Reactive Systems, Advanced Lectures, Lecture Notes in Computer Science, vol. 3472. Springer (2005) 4. Chow, T.S.: Testing software design modeled by finitestate machines. IEEE Trans. Software Eng. 4(3), 178–187 (1978) 5. Dias Neto, A.C., Subramanyan, R., Vieira, M., Travassos, G.H.: A survey on model-based testing approaches: A systematic review. In: Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies, WEASELTech 2007, pp. 31–36. ACM (2007) 6. Dijkstra, E.W.: Notes on Structured Programming (1970). URL http://www.cs.utexas.edu/users/EWD/ ewd02xx/EWD249.PDF. Circulated privately ˇ sevi´ 7. Enoiu, E.P., Cauˇ c, A., Ostrand, T.J., Weyuker, E.J., Sundmark, D., Pettersson, P.: Automated test generation using model checking: an industrial evaluation. Software Tools for Technology Transfer (in this issue) 8. Faria, J.P., Paiva, A.C.R.: A toolset for conformance testing against uml sequence diagrams based on event-driven colored petri nets. Software Tools for Technology Transfer (in this issue) 9. Frantzen, L., Tretmans, J., Willemse, T.A.C.: Test generation based on symbolic specifications. In: J. Grabowski, B. Nielsen (eds.) Formal Approaches to Software Testing, 4th International Workshop, FATES 2004, Linz, Austria, September 21, 2004, Revised Selected Papers, Lecture Notes in Computer Science, vol. 3395, pp. 1–15. Springer (2005) 10. Fraser, G., Wotawa, F., Ammann, P.: Testing with model checkers: a survey. Software Testing, Verifification and Reliability 19(3), 215–261 (2009) 11. Hierons, R.M., Merayo, M.G., N´ un ˜ez, M.: Implementation relations for the distributed test architecture. In: K. Suzuki, T. Higashino, A. Ulrich, T. Hasegawa (eds.) Testing of Software and Communicating Systems, 20th IFIP TC 6/WG 6.1 International Conference, TestCom 2008, 8th International Workshop, FATES 2008, Tokyo, Japan, June 10-13, 2008, Proceedings, Lecture Notes in Computer Science, vol. 5047, pp. 200–215. Springer (2008) 12. Hierons, R.M., Ural, H.: Optimizing the length of checking sequences. IEEE Trans. Computers 55(5), 618–629 (2006) 13. Huang, W.l., Peleska, J.: Complete model-based equivalence class testing. Software Tools for Technology Transfer (in this issue)

Advances in test generation for testing software and systems

5

14. Kurshan, R.P.: Verification technology transfer. In: 25 Years of Model Checking – History, Achievements, Perspectives, pp. 46–64 (2008) 15. Kushik, N., El-Fakih, K., Yevtushenko, N., Cavalli, A.R.: On adaptive experiments for nondeterministic finite state machines. Software Tools for Technology Transfer (in this issue) 16. Larsen, K.G., Mikucionis, M., Nielsen, B.: Online testing of real-time systems using UPPAAL: status and future work. In: E. Brinksma, W. Grieskamp, J. Tretmans (eds.) Perspectives of Model-Based Testing, 5.10. September 2004, Dagstuhl Seminar Proceedings, vol. 04371. IBFI, Schloss Dagstuhl, Germany (2005) 17. Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines-a survey. Proceedings of the IEEE 84(8), 1090–1123 (1996) 18. Ponce de Le´ on, H., Haar, S., Longuet, D.: Model-based testing for concurrent systems: unfolding-based test selection. Software Tools for Technology Transfer (in this issue) 19. Moore, E.F.: Gedanken-experiments on sequential machines. In: C. Shannon, J. McCarthy (eds.) Automata Studies, pp. 129–153. Princeton University Press, Princeton, NJ (1956) 20. Nielsen, B., Skou, A.: Automated test generation from timed automata. Software Tools for Technology Transfer 5(1), 59–77 (2003) 21. Orso, A., Rothermel, G.: Software testing: a research travelogue (2000-2014). In: 36th International Conference on on Software Engineering, ICSE, Future of Software Engineering Track, FOSE, Hyderabad, India, pp. 117–132 (2014) ¨ 22. Ovatman, T., Aral, A., Polat, D., Unver, A.O.: An overview of model checking practices on verification of plc software. Software & Systems Modeling pp. 1–24 (2014) 23. Petrenko, A., da Silva Sim˜ ao, A., Maldonado, J.C.: Model-based testing of software and systems: recent ad-

vances and challenges. Software Tools for Technology Transfer 14(4), 383–386 (2012) 24. Petrenko, A., Yevtushenko, N.: Adaptive testing of nondeterministic systems with FSM. In: 15th International IEEE Symposium on High-Assurance Systems Engineering, HASE 2014, Miami Beach, FL, USA, January 9-11, 2014, pp. 224–228. IEEE Computer Society (2014) 25. Petrenko, A., Yevtushenko, N., von Bochmann, G.: Testing deterministic implementations from nondeterministic fsm specifications. In: Selected Proceedings of the IFIP TC6 9th International Workshop on Testing of Communicating Systems, pp. 125–140. Chapman & Hall, Ltd., London, UK, UK (1996) 26. Schrammel, P., Melham, T., Kroening, D.: Generating test case chains for reactive systems. Software Tools for Technology Transfer (in this issue) 27. Shafique, M., Labiche, Y.: A systematic review of statebased test tools. Software Tools for Technology Transfer 17(1), 59–76 (2015) 28. da Silva Sim˜ ao, A., Petrenko, A.: Generating checking sequences for partial reduced finite state machines. In: K. Suzuki, T. Higashino, A. Ulrich, T. Hasegawa (eds.) Testing of Software and Communicating Systems, 20th IFIP TC 6/WG 6.1 International Conference, TestCom 2008, 8th International Workshop, FATES 2008, Tokyo, Japan, June 10-13, 2008, Proceedings, Lecture Notes in Computer Science, vol. 5047, pp. 153–168. Springer (2008) 29. Tretmans, J.: Model based testing with labelled transition systems. In: R.M. Hierons, J.P. Bowen, M. Harman (eds.) Formal Methods and Testing, Lecture Notes in Computer Science, vol. 4949, pp. 1–38. Springer Berlin Heidelberg (2008) 30. Ural, H., Wu, X., Zhang, F.: On minimizing the lengths of checking sequences. IEEE Trans. Computers 46(1), 93–99 (1997)