Insights in Real Test-Driven Development

Insights in Real Test-Driven Development Peter Zimmerer Siemens AG Otto-Hahn-Ring 6 D-81739 Munich, Germany +49 89 636 42155

[email protected] is also one core practice of eXtreme programming (XP). Among others it handles the fact that creating unit tests after coding is often quite difficult because of the low testability of the implementation.

ABSTRACT Today, many people talk about test-driven development (TDD) and there is some hype to perform test-driven development in software projects. In this situation when many people tell you to do TDD it cannot be bad to ask the question: What is really behind TDD?

Some people state that test-first programming is not a testing technique but rather a specific way of specification, design, and development. Perhaps that is more a philosophical question, but to me test-first programming is testing because testing is not only test execution at the end but includes activities like creating test requirements, test specifications, test design, test cases, etc.

In this paper I want to share my view of TDD’s advantages and disadvantages and how the TDD concept can be extended to all levels of testing. Based on experiences gained from real-world projects employing TDD, I want to explain how to use TDD practices that support preventive testing throughout development and result in new ways of cooperation between developers and testers. This helps us to see new aspects of test-driven development and to get a better understanding how it fits into the big picture of software testing and development overall.

After a short introduction of the basics of TDD I want to explain five lessons learned and limitations of TDD which I have experienced in my work. That’s important to take into account, because as with any testing approach there is no silver bullet and no single testing approach will serve in every situation. That means the usage of TDD requires adequate skill and judgment in the selection and application dependent on the context.

Keywords Test-driven development (TDD), test-first development, eXtreme Programming (XP), preventive testing.

Understanding these lessons learned and limitations of TDD will result in a more comprehensive view of TDD. In addition, it will provide some guidelines on the implications of TDD and the ability to apply TDD to all levels of testing not only unit and acceptance testing.

1. INTRODUCTION To find a definition of test-driven development (TDD) is not difficult today – there are even many books which contain TDD in their title. So, I will start with the definition of TDD as given by Kent Beck in [1]:

2. TYPICAL USAGE When looking into practice we can see the following characteristic usage of test-driven development (TDD).

o never write a single line of code unless you have a failing automated test

2.1 Unit testing by the programmer

o eliminate duplication

Typically unit testing is done by the programmer. This is supported by different testing frameworks and tools, particularly by the different xUnit tools with many extensions (see http://www.xprogramming.com/software.htm) like JUnit (see http://www.junit.org/) for Java. In the meantime these frameworks and tools are very popular and are very often used in industry in the meantime.

which practically results in performing the following three steps in development: o write a test o write necessary code to pass the test o refactor.

2.2 Acceptance testing by the customer

Other related terms for test-driven development (TDD) which are used are test-first development and test-first programming. TDD

Acceptance testing is supported by different testing frameworks and tools as well. For example in the acceptance testing framework FIT (FrameworkForIntegratedTest, see http://fit.c2.com/) by Ward Cunningham and others tests are specified as tables and fixtures act as the glue between the written tests and the application’s code. This is ideal for data-centric tests where each test does the same kind of thing to different kinds of data.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Conference of the Association for Software Testing (CAST), June 5-7, 2006, Indianapolis, Indiana, USA.

Furthermore there is FitNesse, a fully integrated standalone wiki and acceptance testing framework based on FIT by Robert C.

1

Martin and Micah D. Martin (see http://fitnesse.org/). And there are others as well which often have the focus on web testing over HTTP (see http://www.xprogramming.com/software.htm).

4. LESSONS LEARNED 4.1 Lessons learned 1 – My view of TDD These facts result in my first lessons learned of TDD which I just call “my view of TDD”. I see TDD composed of two parts:

From my experience these frameworks and tools are suitable for specific kinds of systems and domains (for example web and Java applications) but are not used in industry so much up to now at least compared to the xUnit tools.

o test-first design and o test-first implementation

3. SOME BASICS ON TESTING

It is very beneficial to use the TDD approach (test-first design) already to create, evolve, and improve the requirements, design, architecture, etc. This includes early creation of abstract test cases (i.e. describe the expected behavior as a test and specify how you would test a requirement or a high-level design) as well as executable test cases implemented in detail (test-first implementation). TDD is possible and highly recommended on every test level, not only for unit and acceptance testing and it is driven by the idea of preventive testing! This emphasizes the importance and benefits of early testing activities like building a test model, creating abstract test cases which then represent a set of executable specifications, and proactive design for testability.

So far, we can see that TDD has a focus on unit testing and acceptance testing as well as a focus on the detailed implementation of tests. Now, let’s have a look on some basics on testing which we as testers (should) know for many years: o Finding bugs during test execution is not the optimum. The goal must always be to find bugs early during test specification, not late during test execution!!! o We as testers experience every day that specification artifacts like requirements, use cases, models, architecture, and design must be improved, extended and completed for testing which results in changes. That means that creating a test specification and test design already finds and prevents bugs, i.e. building a test specification is testing, for example by building a test model or creating abstract test cases! Furthermore, these tests represent a set of executable specifications.

4.2 Lessons learned 2 – Test-first implementation If we look in more detail on the implementation aspects of TDD, i.e. on test-first implementation, in real world projects I have experienced that sometimes things are not so easy to do in practice as described in some nice little demo examples. From my experience it is not the point that it is not possible at all, but the critical point is that sometimes the usage is highly cost dependent and might be too time-consuming dependent on what huge test environment is needed or how many mock objects have to be set up etc. This occurs especially for example in the areas of testing GUIs, web applications, distributed objects on application servers, event-based reactive systems, embedded systems, etc.

o Design for testability (which can be shortly characterized as visibility/observability and control) must be built in from the start. And that’s not a private affair of the testers and will have benefits for the whole project team. Altogether, these basic facts on testing provoke the following question: Why should we restrict TDD only to unit and acceptance testing with executable test cases implemented in detail?

For GUI testing for example there are also different testing frameworks and tools available (for example JFCUnit at http://jfcunit.sourceforge.net/ and Jemmy at http://jemmy.netbeans.org/) which in principle support the TDD approach for a GUI as well. Another approach is to divide the code into appropriate components that can be built, tested, and deployed separately. Most of the functionality (i.e. the business logic) is built outside the context of the user interface code using TDD. And the user interface code is just a very thin layer on top of rigorously tested code i.e. as much functionality as possible is built outside the GUI:

3.1 Preventive testing When looking back for many years we can see that the idea of TDD is not new. One origin of the TDD idea has been already given by Bill Hetzel more than 15 years ago by using the term preventive testing (see [2] and [3]). He said: Preventive Testing is built upon the observation that one of the most effective ways of specifying something is to describe (in detail) how you would accept (test) it if someone gave it to you. In this manner, by using testing to discover, identify, create, specify, influence and control requirements, architecture, design, implementation, deployment, and maintenance artifacts testware development leads software development. That means that the idea of TDD is nothing actually new, nothing new brought to us by XP or agile methods and the hype around it but rather a quite old idea (see also reports and references given in [5]). What‘s new is that these things are more and more really used in projects today. For me that‘s the real big benefit of the TDD-hype brought to us by XP and agile methods..

This good basic architectural style of a clear separation between business logic and user interface is not new and is already well known for a long time, like for example a 3-tier architecture and the model-view-controller (MVC) design pattern.

2

4.3 Lessons learned 3 – Innovation and invention

and later people thought about needed testing strategies, testing methods, and testing tools for them.

Many of the software we build today is very new, i.e. very innovative and may include some kind of invention as well. These circumstances can also make TDD more difficult to realize. To explain that I will use the following engineering example from history:

Or let’s take architectural and design patterns which claim to contain innovative approved best practices: here again the patterns have been invented first (or some have been reinvented based on already known knowledge) and later people started to think about how to test a specific design pattern, i.e. also design patterns have not been developed in a core TDD manner.

In 1886 the first car was invented by the Germans Carl Benz and Gottlieb Daimler. This was a very big innovative step done by very creative inventors which has changed our lives completely until today. In the first half of the 20th century already many cars have been produced and used by many people. But only 70 years later, in the fifties, Mercedes-Benz invented the first crash tests which are systematically done since 1959.

And what’s about all the innovative testing tools we get from the commercial testing tool industry? Do you think these tools are usually developed in a TDD manner? I don’t think so … Altogether, we should not restrict ourselves in our own innovation and inventiveness by performing the TDD approach too rigorously and by expecting that we can always define the test first. To get innovation in software is not an easy task at all and starting first with a test for a really new thing is even more difficult because TDD adds some kind of indirectness and an additional level of abstraction. We need creativity in our testing and development work to make our software more innovative and to do not limit the next small or big steps in software evolution.

4.4 Lessons learned 4 – Non-functional requirements TDD should also be used for non-functional requirements. Vague, low-quality requirements like “Our architecture shall be very flexible” are very very hard to test (indeed they are not testable at all). Instead, using test cases to create requirements and describe how you would test it (i.e. doing preventive testing) will result in finding bugs, improving the quality of the requirements and increasing testability. These test cases can be abstract and nonexecutable or concrete and implemented in detail dependent on the level of abstraction of the requirements. In the same way it is very helpful to develop performance requirements in a TDD manner

Now let’s think about what TDD would mean for this example from the history in engineering. We can draw the following conclusion:

On the other side there are some characteristics of non-functional requirements which limit the usage of the TDD approach. For example regarding security most of the security bugs are not in the area of the intended and known behavior which is very well explained in [4]:

Using the TDD approach to full extent would mean that somebody first invents the crash test and then afterwards the car to pass this crash test. Oops! That procedure is really difficult to do in practice. Rather I think that this is almost impossible to be the usual way of developing such a high innovative product like a car! Now, let’s try to map this example from engineering to our software world. Innovations in software can appear on many different levels like for example “some innovative lines of code”, “an innovative framework (especially the xUnit frameworks!)” up to “a new software technology”. At the same time it is very difficult to quantify the degree of innovation and invention in a specific piece of software and to determine the influence of the development method on the innovation. On a more abstract level when looking back in the history of software engineering we can see that in most cases (or always?) testing came after development and not vice versa.

But, how can I create a test up front for an unintended, unknown behavior? This example shows some limitations of TDD in the area of non-functional requirements as well.

For example for all the new innovative software technologies like object-orientation, component-based software, web technologies, aspect-oriented programming, grid computing, etc. first there was the idea, vision and invention of these new software technologies

3

4.5 Lessons learned 5 – Cost efficiency and predictability

paper I recommend taking the following items into account when realizing and enhancing your own TDD approach:

We as testers often complain that we are involved to late in development projects. I completely agree with that. But, we also must admit that in a high innovative, iterative/incremental, agile project there is at least the possibility that because of continuously changing early requirements and architectural prototypes we have to do a lot of rework in test design and test implementation itself. So, our job is to find the right balance in our projects for that when using the TDD approach especially when doing a lot of test implementation work at the beginning.

o TDD = Test-first design + test-first implementation Do not miss the benefits of the more conceptual usage of the TDD approach during creation and improvement of requirements, design, architecture, etc. o TDD is possible and strictly recommended on every test level, not only for unit and acceptance testing (preventive testing): let testing drive your development and maintenance at all! o The right project specific balance is the key for cost efficiency.

Furthermore TDD is not sufficient in real life because test cases created using a test-first approach or generated from (always incomplete) requirements and design are never enough and must be always extended and completed. We cannot predict everything, so we have to use techniques like exploratory testing as well. In additional, decisions made during implementation won't be well tested by tests exclusively created up front.

o TDD needs changes in development: process, people, and tooling. o TDD results in a closer cooperation of testers and developers. o TDD is neither 100% possible nor sufficient in real-world projects.

Altogether, doing test-first only is not enough but we additionally need some “afterwards” testing which I call test-second.

o TDD does not completely replace conventional afterwards software testing: so, do test-first as well as test-second. Apart from this, if you have not already used and practiced TDD then I highly recommend to start with TDD tomorrow! You will see you will deliver better software with higher quality, you will have less pain in your projects, and in the mid-term you will also speed up your project! Furthermore be aware of some circumstances and constraints which require a flexible and adjustable testing approach. But, to select and adapt your own approach that’s for what you are paid in your job as a test engineer, test lead, test manager or test consultant. Good luck!

5. EXPERIENCES In my experience TDD strongly increases visibility and importance of testing in the whole project. And as case studies have shown it delivers benefits concerning quality improvement and productivity (see [5]). Although TDD is often used in the context of an iterative/incremental, agile development process (so do I) it is not restricted to it. That means TDD and the idea of preventive testing is useful and applicable even in an oldfashioned waterfall process. TDD needs changes in the development concerning process, people (including management!), and tooling. And TDD results in a closer cooperation of testers and developers which is a big benefit as well.

7. REFERENCES [1] Kent Beck: Test-Driven Development by Example, AddisonWesley Professional, 2002 [2] R. Craig, S.P. Jaskiel: Systematic Software Testing, Artech House, 2002

If you do TDD sometimes it is not easy to find out how much of the implemented test code is really developed in a TDD way or which is developed more conventionally “afterwards”. I myself was confronted with this issue in a project when the developers asked me: “How do I test private member functions?” At first, from a technical point of view there are different answers and possibilities dependent on the used programming language (for example friends in C++, reflection in Java). But when I thought about that in more detail I came to the following interesting conclusion (which might be a bit to strict in general, but …):

[3] D. Gelperin, B. Hetzel, The Growth of Software Testing, Communications of the ACM, Vol. 31, Issue 6, June 1988, pp. 687-695 [4] James A. Whittaker, Herbert H. Thompson, Herbert Thompson: How to Break Software Security, Addison Wesley, 2003 [5] D. Janzen and H. Saiedian, Test-Driven Development: Concepts, Taxonomy, and Future Direction, IEEE Computer, Vol. 38, No. 9, September 2005, pp. 43-50

In core TDD this question is not allowed, i.e. it does not make sense, because in TDD we first have the designed and implemented test and then do some implementation to pass this test. I.e. the details of the implementation do not matter so much.

[6] Links http://www.xprogramming.com/software.htm/

Here, I recommend using these kinds of indicators to see whether and to what extent the TDD approach is really used in your projects.

http://www.junit.org/

6. SUMMARY: TEST-FIRST ⊕ TEST-SECOND

http://jfcunit.sourceforge.net/

http://fit.c2.com/ http://fitnesse.org/ http://jemmy.netbeans.org/ http://www.testdriven.com/

To summarize my understanding of TDD and the different lessons learned and limitations of TDD which I have described in this

4