Teaching students Property-based Testing - Prowess

Teaching students Property-based Testing ˚ Fredlund and Julio Mari˜no Clara Benac Earle, Lars-Ake

Thomas Arts

Babel Group Universidad Polit´ecnica de Madrid, Spain Email: {cbenac,lfredlund,jmarino}@fi.upm.es

QuviQ, Gothenburg, Sweden Email: [email protected]

Abstract—Testing is a crucial aspect of the development of dependable embedded systems, and therefore a significant effort is put into researching and developing efficient testing techniques. However, testing is not normally taught in specific courses at many universities, but rather as a peripheral activity to programming. In this paper, we report on three separate experiences at teaching an advanced testing technique, property-based testing, and a supporting tool, QuviQ QuickCheck, to both undergraduate and master students. Keywords—Testing and Debugging; Computer science education;

I.

I NTRODUCTION

Given the increasing presence of software in embedded systems, testing becomes a central activity for ensuring dependable embedded systems, especially in safety-critical systems. Several testing techniques have been proposed, for instance, test-driven development [1], model-based testing [2], etc. A technique that is recently receiving a lot of attention is Property-based testing [3], [4], strongly related to modelbased testing, but where the focus is on generating individual tests automatically from specifications in the form of higherlevel properties. QuviQ QuickCheck is a commercial propertybased testing tool that has been used, for instance, to test Radio Base Stations and other telecommunication software from Ericsson [5] or AUTOSAR software components for Volvo Car Corporation [6]. In other words, a tool evaluated and used in industry to improve effectiveness and efficiency in testing. In our experiences many students consider testing an uninteresting undertaking, and so prefer to focus on other software engineering tasks (e.g., programming). We believe that a systematic use of property-based testing could help to improve the situation by removing much of the tedium associated with writing individual unit tests, and instead permit students to focus on the more interesting and rewarding task of developing higher-level behavioural properties of the program or system under test. For this reason a number of teachers at the Universidad Polit´ecnica de Madrid (UPM) has set out to try to introduce property-based testing, an innovative technique used in industry, as part of the computer science and software engineering curriculum. Unfortunately in traditional undergraduate courses, testing is often a peripheral activity compared to programming. The situation at UPM (and other Spanish universities) is no different. Moreover, the Spanish educational system is centralized, meaning that even when the local will exists to introduce more

testing in the computer science curricula, it is a very difficult undertaking. For this reason the initial attempts to introduce property-based testing, which are described here, have been focused on students at or near the master’s level. In this paper, we then describe our experiences as lecturers in teaching various groups of students at UPM the technique of property-based testing. Common to all these initiatives is the use of the Quviq QuickCheck property-based testing tool. In 2013 we have taught this topic in a course targeting undergraduate students about to finish their degrees. In this course, students with good programming skills and a high ambition level were explicitly targeted. We offered them, on an entirely voluntary basis, an introductory course on advanced testing techniques. The outcome of the course was very encouraging. Students learned the techniques very quickly, and were capable of using QuickCheck in an effective manner. Moreover, QuickCheck was used to illustrate modern testing technology in a course on component-based software taught at UPM as part of the two-year European Master in Software Engineering international master program (part of the Erasmus Mundus program). In addition, property-based testing using Quickcheck was also taught in the Rigorous Software Development course in the Master in Software and Systems at UPM. Here, the students were taught how to test Java code using QuickCheck. Finally, because teaching advanced testing techniques is not only useful for students but for engineers working in companies, we have been organizing regular seminars for both students, university staff, and people from companies interested in the subject. In Sect. II we first introduce the tools used in the course, the Erlang programming language and the QuviQ QuickCheck testing tool. Next, Sect. III reports on the first teaching experience, teaching property-based testing to undergraduate students, while Sect. IV and V reports on two experiences in teaching property-based testing to master students. Finally, Sect. VI draws conclusions from these teaching efforts, and discusses the future plans for teaching property-based testing. II.

E RLANG AND Q UICK C HECK

This section briefly describes the tools used by students during the courses on property-based testing, the Erlang programming language which is used to formulate the test models, and the QuviQ QuickCheck tool which provides higher-level libraries for aiding testing, and which automates the testing.

A. Erlang Erlang [7], [8] is a programming language originally developed in the 80s at Ericsson for telecoms use but which is now applied in banking, messaging, finance, cloud computing, NoSQL databases, VOIP, embedded systems and web applications by companies such as Ericsson, Volvo, Motorola, Yahoo, T-Mobile, and BT. The main characteristics of Erlang are the inclusion of lightweight concurrency language constructs based on the actor model, a powerful handling of fault-tolerance, the transparent distribution mechanisms and the fact that the language has functional programming roots leading to a small, clean, code base. The use of Erlang as a basis for testing has a number of advantages, due to its roots in functional programming. The language has a large number of libraries providing side effect free implementations of many abstract data types, which makes it easy to express complex correctness conditions for testing. Moreover, it is easy to treat programs as data or as declarative specifications of the behaviour of software under test. B. QuviQ Quicheck QuickCheck was first implemented for Haskell [3] and, afterwards, for the Erlang programming language. Quviq QuickCheck [9] (here referred to as QuickCheck) is the commercial version of QuickCheck which is written in Erlang. The basic functionality of QuickCheck is simple: when supplied with an Erlang data term that encodes a boolean property, which may contain universally quantified variables, QuickCheck generates a random instantiation of the variables, and checks that the resulting boolean property is true. This procedure is by default repeated 100 times. If for some instantiation the property returns false, or a runtime exception occurs, an error has been found and testing terminates. The tool then provides the user with a reduced counter-example. For example, the mathematical property P P ≡ ∀(X, Y ) ∈ Z × Z · (X × Y >= 0) can be specified in QuickCheck with the following Erlang function p() −> ? FORALL ({X,Y},{int (), int ()}, X∗Y>=0).

Assuming that the function p is defined in the module test, we can invoke QuickCheck in the Erlang shell to test the property: 1> eqc: quickcheck (test:p()). .......... Failed ! After 11 tests. {−3,1} Shrinking .(1 times) {−1,1} false

The previous code refers to a generator of random integers int() provided by QuickCheck. The generated numbers are bound to two variables X and Y used to check that X times Y is greater or equal to 0. QuickCheck rapidly finds a counterexample that falsifies the property and the shrinking mechanism returns a smaller counter-example.

Complex data generators can be built by referring to simpler ones such as nat() e.g. in list(nat()) —lists of natural numbers— or by using the QuickCheck constructs ?LET and ?SUCHTHAT. The following two generators generate, respectively, positive natural numbers and even numbers: pos () −> ?LET(N,nat (),N+1). even () −> ? SUCHTHAT (N,pos (),N rem 2 == 0).

State Machines for Testing QuickCheck provides a state machine based library for testing APIs with side effects. The goal of the library is to enable a tester to easily generate randomly sequences of sensible calls to the API, and to decide if the execution of such a call sequence was successful (i.e., the API returned the expected results) or not. Technically the tester, with help from the library, builds a state machine that serves as a model of the behaviour of the API, and is used to generate the sequences of commands used to test the API. Concretely, a user of the state machine library needs to develop an abstract model of the API state (e.g., for a database API, a possibly abstract representation of the database state), which is used as the state of the finite machine. QuickCheck proceeds to generate the sequence of API commands by, in every model state, calling a user supplied function (xxx args) to randomly generate a symbolic representation of the next API command to invoke. Next the generated command is invoked by QuickCheck. Once the API command has returned a result, another user supplied function (xxx post) is called to decide if, given the model state, and a symbolic representation of the API command, the result of the command was the expected one. If not, the testing of the API revealed an error. Finally, the next model state is computed by calling another user defined function (xxx next) using the model state, a symbolic representation of the API command, and the call result as parameters. III. T EACHING U NDERGRADUATE STUDENTS P ROPERTY- BASED T ESTING WITH Q UICK C HECK As was mentioned earlier, to disseminate the knowledge concerning property-based testing methods and tools, we targeted undergraduate students with good programming skills, and who were highly motivated. In fact, some of them had come to us asking for possible ways of collaborating in our research work. They were all in their third (out of four) study year. We divided the course in two parts. First, we provided the students with books about the Erlang programming language [7], [8] so that they could learn Erlang by themselves. We encouraged them to program all the examples found in the books, and come to us whenever explanation was needed. This part of the course lasted around 3 months, since they were learning Erlang in their spare time. Once they felt confident using Erlang for writing mid-sized program, the second part of the course began. In this second part, QuickCheck and Property-based testing were introduced during four sessions, one session every two weeks. In these sessions, we used some of the official training

material provided by QuviQ, as well as, our own training material. The sessions were very practical, consisting of a first short lecture of around 15 minutes, followed by a handson session, where the students would solve some proposed problems using QuickCheck in their own laptops, and with our assistance. The first example address testing of the seq function in the list library. This function takes two numbers as input and produces a list with the numbers from first to last given input. We ask the students to write tests for this function on paper and collect them then on the whiteboard. Normally, we collect about 3 tests per student, ending up with quite a lot of tests, but few duplicates. It shows that many unit tests are possible and that it depend on persons which ones are created. We analyse the tests to come up with a property that generalises these tests. Since students have a tendency to avoid the error cases in tests, we can often summarize it with the following property: prop seq() −> ? FORALL ({From , To}, {int (), int ()}, length (lists:eq(From ,To)) == To − From +1).

Checking this property with QuickCheck immediately fails and shows a reduced counterexample, for instance, {1,−1}. The interesting observation for the students is that the property normally subsumes all tests they have collectively written down and nevertheless is able to find an error case that they overlooked. After explaining the basic data generators, the students were tasked with expressing test properties by themselves. Individually, they had to implement the following properties: •

if an element is not in a list, then deleting that element leaves the list unchanged

•

if an element is in a list, then deleting that element shortens the length of the list by one

•

adding an element to a list and deleting it again returns the original list

Students were capable of writing these properties with very little help from the lecturer, which they found very encouraging. Thus, from the very first session, students could write simple properties and check them with QuickCheck. In the following sessions, students learned to implement their own data generators, and to express more complex properties. IV.

T EACHING M ASTER STUDENTS P ROPERTY- BASED TESTING

During a five-year period a course on component-based software was taught at UPM, both as part of an European master part of the the Erasmus Mundus programme, as well as part of a local (to UPM) masters programme. The European Master in Software Engineering (EMSE for short) was a two year international master program focusing on software engineering taught in English, with the collaboration of a number of universities, e.g., Blekinge Institute of Technology, Sweden, University of Kaiserslautern, Germany, Free University of

Bozen-Bolzano, Italy, and Universidad Politcnica de Madrid, Spain. Students enrolled in the EMSE program were expected to move between the participating institutions. As part of the curriculum offered by UPM in EMSE a course on component-based software was provided. The topics ranged from engineering aspects of component-based software to themes such as e.g. specification and verification of component-based software. To introduce the students to the world of embedded system the AUTOSAR automotive software architecture was introduced. The course was regularly attended by around 15 to 20 highly motivated students. The UPM lecturers of the course felt that the level of programming skills (e.g., in Java in general and in various Java based programming frameworks in particular) demonstrated by the students was quite high, but that their skills in specifying component-based systems, and verification and validation of such systems, needed reinforcement. Thus one of the focal points of the course was on the use of propertybased testing both to specify and validate component-based systems. Students were introduced to the QuickCheck tool, and shown how it could be effective both for specification and validation, by demonstrating examples of its application to component-based systems. As an example, the students saw the specification and effective testing of a Java data structure using the QuickCheck state machine library. As an obligatory part of the course each student had to select a topic related to component-based software (e.g., a component-based programming framework, part of AUTOSAR, etc.), and proceeded to study the topic in detail, including a thorough evaluation of it based on the students experiences1 , e.g., if a programming framework was chosen a mandatory part of the study was using the framework to solve some task. The conclusions of the study were summarised in a short presentation and presented in more detail in a written report of around 15-20 pages. A small number of students choose to study the use of QuickCheck for component-based systems since QuickCheck is used by Volvo to test AUTOSAR components, but it is fair to say that other topics were more popular. In our opinion, to many students testing is not seen as a very exciting task, and so prefer to focus on other undertakings. We believe that a systematic use of property-based testing could help to improve the situation by removing much of the tedium associated with writing unit tests, and instead focus on the task of developing higher-level behavioural properties of the program or system under test. V.

T EACHING M ASTER STUDENTS TESTING JAVA CODE WITH Q UICK C HECK

At UPM, the Master in Software and Systems [10] offers students an overview of cutting-edge research conducted worldwide in the fields of software and systems. One of the courses, Rigorous Software Development [11], introduces students to the use of formal methods applied to the construction and verification of software. Formal methods have traditionally been advocated for improving the reliability of safety-relevant systems albeit at a high cost. In this context, property-based testing can be 1 rather

than reading about the topic in Wikipedia

considered a light formal method where tests are generated from logical specifications. To give students an idea of some testing technique which is middle ground between pure formal method techniques (theorem proving and model checking, for instance) and ordinary testing, we introduce PBT and the QuicCheck tool to the master students in the Rigorous Software Development course in the autumn 2013. In this course, we gave a few lectures on testing Java code with QuickCheck. In the following, we describe the examples used to teach students to give an idea of the teaching material used.

for (int i = 0; i < size; i++) { // first , we skip smaller elements int pos = 0; for (int j = 0; j < logical size && result [j] < list[i]; j ++){ pos = j+1; } // we insert list[i] result [pos] = list[i]; // then , we shift the greater elements : for (int k = logical size; k > pos; k−−){ result [k] = result [k−1]; } logical size ++; }

In order to show the approach we use a Java implementation of the insertion sort algorithm. This is shown in Fig. 1. A buggy version is shown in Fig. 2.

Fig. 2.

The functional specification of list sorting has two parts. The first one states that the output must be sorted. The second one has the for-loop replaced by an erroneous for loop.

> {ok ,NID} = java:start node (). {ok ,101 }

public class insertionSort { public static int [] iSort(int [] list ){ // we make space for the new list // returned , i.e. we do not change // the input list int size = list. length ; int [] result = new int[size ]; // as result is originally filled with // zeroes , and the input list might // contain negative values , // or zeroes , we need to keep in mind // the " logical " // length of the array int logical size = 0; // we insert all list ’s elements // in order in result for (int i = 0; i < size; i++) { // first , we skip smaller elements int pos = 0; for (int j = 0; j < logical size && result [j] < list[i]; j++){ pos = j+1; } // then , we shift the // greater elements : for (int k = logical size; k > pos; k−−){ result [k] = result [k−1]; } logical size ++; // finally , we insert list[i] result [pos] = list[i]; } return result ; } } Fig. 1.

A Java implementation of insertion sort.

A. Calling Java from Erlang We use of the JavaErlang library [12] to invoke Java code from Erlang. A sample session follows. First, we invoke the Erlang shell, and then a Java server node is launched which will be in charge of executing code as requested from the Erlang side:

A buggy loop in the insertion sort

The node thus created is passed as parameter of the different commands that translate Erlang data into Java, > Input = java:list to array(NID ,[6,2,3,1,5,4],int ). {object ,0 ,101}

call Java methods, > Output = java:call static(NID , insertionSort , iSort , [Input]). {object ,1 ,101}

or translate Java data back to Erlang: > java:array to list( Output ). [1,2,3,4,5,6]

B. Testing Java code using QuickCheck The properties to be tested using QuickCheck are written in an Erlang file, in this case named sorting.erl. First, we have a purely functional specification of sorting. i.e. there is nothing to do with Java or QuickCheck here. A predicate to state that a list of integers is sorted follows: isSorted ([]) −> true; isSorted ([ ]) −> true; isSorted ([A,B | Zs]) −> (A =< B) and isSorted ([B | Zs]).

Defining a predicate to check whether two lists are permutations of each other is only slightly harder: occurs ( ,[]) −> 0; occurs (X,[H | Ts]) when X==H −> 1 + ( occurs (X,Ts )); occurs (X,[ | Ts]) −> occurs (X,Ts). permuts (As ,Bs) −> lists:all (fun(X) −> occurs (X,As )== occurs (X,Bs) end , As++Bs).

propSorted (Node) −> ? FORALL (Xs , nelist (int ()), isSorted (java:array to list( java:call static(Node , insertionSort , iSort , [java:list to array(Node ,Xs ,int)])))). propSorted insertbefore(Node) −> ? FORALL (Xs , nelist (int ()), isSorted (java:array to list( java:call static(Node , insertionSort ib , iSort , [java:list to array(Node ,Xs ,int)] )))). propPermuts (Node) −> ? FORALL (Xs , nelist (int ()), permuts (Xs ,java:array to list( java:call static(Node , insertionSort , iSort , [java:list to array(Node ,Xs ,int)])))). propPermuts insertbefore(Node) −> ? FORALL (Xs , nelist (int ()), permuts (Xs ,java:array to list( java:call static(Node , insertionSort ib , iSort , [java:list to array(Node ,Xs ,int)] )))). Fig. 3.

Erlang properties for testing insertion sort.

Now, we define QuickCheck properties on Erlang lists for testing the effect of the Java sorting implementations on their corresponding Java relatives. This is shown in Figure 3. Let us try it! The “correct” implementation passes all tests: >eqc: quickcheck ( sorting : propSorted (NID )). ....................................... ........................................ ..................... OK , passed 100 tests true > eqc: quickcheck ( sorting : propPermuts (NID )). ....................................... ....................................... ...................... OK , passed 100 tests true

The buggy version passes the “isSorted” test (!!) > eqc: quickcheck ( sorting : propSorted insertbefore(NID )). ........................................... ........................................... .............. OK , passed 100 tests true

but fails the “permuts” one: eqc: quickcheck ( sorting : propPermuts insertbefore(NID )). .......... Failed ! After 11 tests. [0,−1] false

a completely voluntary course. As teachers, we were very impressed by the skills of these students. They learned Erlang, and were capable of using QuickCheck to solve the exercises provided, very quickly. On one hand, we can infer that they had the necessary background to understand the material. On the other hand, not having learned other testing techniques in depth, and having good mathematical and logical skills, it seemed natural to them to write properties to test their code, instead of individual test cases. Student feedback regarding the course was very positive. They told us that the material introduced in the course was very intellectually stimulating, and that in their estimation having acquired knowledge of property-based testing would prove highly useful in their future professional careers. In the case of the Master students, we introduced propertybased testing in two different courses: “Component-based Software” which is taught in the European Master in Software Engineering, and “Rigorous Software Development” which is taught in the Master in Software and Systems. The results were, again, encouraging and we received positive feedback from the students. In the future, we plan to continue teaching property-based testing and QuickCheck to students. We need to develop our teaching material further in order to provide students with bigger, real-life examples where they can apply what they have learn in the course. Moreover, we need to dedicate more time to this topic to give students time to work on these larger examples. ACKNOWLEDGMENT

VI.

C ONCLUSION AND F UTURE WORK

In this paper we have reported on the experiences from introduced advanced testing techniques, e.g., property-based testing, to mature undergraduate and master level students at UPM. In particular, the courses have focused on the use of the QuviQ QuickCheck tool to automatize property-based testing. For introducing property-based testing to undergraduate students, we selected a groups of students that were both good programmers and highly motivated students, and offered them

This work has been partially funded by the ARTEMIS JU under grant agreement 295373 (nSafeCer), and from National funding, the European Comission FP7 project ICT2011-317820 PROWESS, and the Spanish MINECO project TIN2012-39391-C04-02 STRONGSOFT. R EFERENCES [1]

D. Astels, Test Driven Development: A Practical Guide. Prentice Hall Professional Technical Reference, 2003.

[2]

[3]

[4] [5]

[6]

[7] [8] [9]

[10] [11] [12]

M. Utting, A. Pretschner, and B. Legeard, “A taxonomy of model-based testing approaches,” Softw. Test. Verif. Reliab., vol. 22, no. 5, pp. 297– 312, Aug. 2012. [Online]. Available: http://dx.doi.org/10.1002/stvr.456 K. Claessen and J. Hughes, “Quickcheck: a lightweight tool for random testing of haskell programs,” in ICFP, M. Odersky and P. Wadler, Eds. ACM, 2000, pp. 268–279. ScalaCheck. [Online]. Available: http://www.scalacheck.org/ documentation.html T. Arts, J. Hughes, J. Johansson, and U. Wiger, “Testing Telecoms Software with Quviq QuickCheck,” in ERLANG ’06: Proc. of the 2006 ACM SIGPLAN workshop on Erlang, Portland, Oregon, USA, 2006. R. Svenningsson, R. Johansson, T. Arts, and U. Norell, “Formal methods based acceptance testing for AUTOSAR exchangeability,” SAE Int. Journal of Passenger Cars Electronic and Electrical Systems, vol. 5, no. 2, May 2012. J. Armstrong, Programming Erlang: Software for a Concurrent World). The Pragmatic Bookshelf, 2007. F. Cesarini and S. Thompson, Erlang Programming. O’Reilly Media, 2009. T. Arts, J. Hughes, J. Johansson, and U. T. Wiger, “Testing telecoms software with quviq quickcheck,” in Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang, Portland, Oregon, USA, 2006, pp. 2– 10. Master in Software and Systems. [Online]. Available: http://muss.fi. upm.es/en/index.html Rigurous Software Development. [Online]. Available: http://lml.ls.fi. upm.es/rsd/ JavaErlang. [Online]. Available: http://babel.ls.fi.upm.es/∼fred/ JavaErlang/