Testing - Department of Distributed and Dependable Systems

Software Testing Peter Libič, Vlastimil Babka, Pavel Parízek, Tomáš Kalibera Department of Distributed and Dependable Systems http://d3s.mff.cuni.cz CHARLES UNIVERSITY PRAGUE

Faculty of Mathematics and Physics

Software Testing

• Empirical comparison of a software system behavior against an oracle, i.e.      

Specification Standard A similar software system (back-to-back) Earlier version of the system Customer expectation ...

Software Testing

• Can only find negative answers  The system does not work as expected  Passing tests does not prove no bugs • Finds load testing Reliability -> stress testing Security Usability (is user interface easy to understand?) Maintainability, testability

Testing Methods (Based on Access to System)

• Black-box testing  The tester knows nothing about the implementation of the software • No access to code, knowledge of architecture

 Tests are based on specification • Comparing inputs with expected outputs

• White-box testing  The tester has full knowledge of the implementation • Can test e.g. both public and private API

 The tester can modify the system for easier testing • Choosing paths through code (i.e. failure paths)

• Grey-box testing  The tester knows the system, but cannot modify it

Fault Injection – Testing Fault Paths • Software should be robust against certain faults  Hardware faults • Depending on the criticality of software • i.e. one-bit errors in communication, errors in system memory, disk full, …

 Faults of other software • OS against anything an application does • OS against failures detected by most drivers

• Faults are created on purpose  Too rare in real situation  Allows to test that the application is really robust

Testing on Different Levels • Unit testing (Component testing)  Testing of a piece of software, written by single (multiple) developer(s), in isolation

integrated modules

unit

unit

• e.g. Java classes and methods, …

 Developers specify test cases and verify whether they successfully passed

• • • • •

Integration testing System testing System integration testing Acceptance testing Alpha/Beta testing

unit

system

thirdparty

customer

Testing on Different Levels • Unit testing (Component testing) • Integration testing

integrated modules

unit

unit

 Checking of interaction among components

• System testing

unit

 Testing of a whole system in a target environment

• System integration testing

 Tests including 3rd party components

• Acceptance testing

system

thirdparty

 Customers specify requirements and properties, and check test results • “system testing performed by customers”

• Alpha/Beta testing

 “internal/external user acceptance testing”

customer

Typical Testing Process • Requirements analysis and test planning  Deciding which properties to test

• • • • •

Test development (Automated) test execution Analysis of test results Fixing of bugs in implementation Regression testing  Re-run of previously passed tests after modification • Modification: bug fix, change of functionality

 Checks whether the modification hasn’t broken something that worked

Unit Testing

Unit Testing

• Basic idea  Testing of minimal units in isolation • Individual classes and their methods (incl. internal methods)

 Focus on low-level properties of the unit’s implementation • Early check whether the implementation works properly • Verifies class/method contract

 Automatic, easily repeatable tests with clear answer (pass or fail)

Unit testing

• Pros  Safer implementation changes (mistakes are quickly caught)  Simpler integration (individual parts already trusted)  Implicitly up-to-date API/contract documentation  Can be part of design (TDD)

Unit Testing

• Cons  Slows down coding • But typically saves debugging time

 Needs developer discipline • Can't live without that, anyway

 Tests are limited • in scope • can be wrong  “Testing tests”: fault (defect) injection / mutation testing

Unit Testing – Test Design

• Definition of the tests  Developers write code that • Specifies the properties and test inputs • Verifies whether the tests successfully passed by comparing expected and actual output (state) of tested classes or methods

• Automated execution  Needs to be very fast!

Unit Testing – What to Test?

• Class  method contracts

• Separate branches in the code • Special (border) cases  “Off by 1”

• Inputs triggering previously discovered bugs

Unit Testing - Frameworks • Help and automate unit test creation, execution and evaluation • xUnit: JUnit, NUnit, CPPUnit, PyUnit, …  Common architecture: • • • • •

Test suite Fixture Test case Test execution Assertions

• Many of them:  http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

• Even for HTML: HTMLUnit

Test Case, Test Suite • Test suite  Collection of related test cases • e.g. all test cases for a single Java class

 Set-up of program’s and environment’s state

• Test case  Definition of a single test • Subject of testing  e.g. implementation of a Java method

• Test input • Expected output

JUnit • Unit testing framework for Java  Originally written by K. Beck and E. Gamma  http://www.junit.org

• Key features  Test cases are regular Java methods  Test results are analyzed in an automated way

• Two major versions are widely used  JUnit 4 • Based on @Annotations

 JUnit 3 • Based on fixed method names (prefixes) • If possible, use JUnit 4

Simple Test Case – JUnit 3 import junit.framework.TestCase; public class TestArrayList extends TestCase { public void testAdd() { List al = new ArrayList(); int origSize = al.size(); al.add(“abc”); int newSize = al.size();

“test” prefix identifies test case methods

assertEquals(“Different size!”, origSize + 1, newSize); assertTrue(al.contains(“abc”)); } } comparison of expected results with actual results

Simple Test Case – JUnit 4 import org.junit.*; import static org.junit.Assert.*; public class TestArrayList { @Test public void add() { List al = new ArrayList();

annotation identifies test case methods

int origSize = al.size(); al.add(“abc”); int newSize = al.size(); assertEquals(“Different size!”, origSize + 1, newSize); assertTrue(al.contains(“abc”)); } }

Assert Statements – org.unit.Assert public static void assertX ([message], ...) • assertEquals (T expected, T actual) • assertArrayEquals (T[] expected, T[] actual) • assertSame (Object expected, Object actual) • assertTrue (boolean condition) • assertFalse (boolean condition) • assertNull (Object object) • assertNotNull (Object object) • assertThat (T actual, Matcher matcher) • fail ([String message]) • ....

Running JUnit Tests (JUnit 4) • From within Java, e.g. main()  org.junit.runner.JUnitCore.runClasses (TestArrayList.class, ...);

• From command line  java org.junit.runner.JUnitCore TestArrayList [...]

• via Ant

…

Output of JUnit • Example ----------------------------------------Testsuite: TestArrayList Tests run: 8, Failures: 1, Errors: 0, Time elapsed: 0,5 sec Testcase: testAdd(TestArrayList): FAILED Different size! expected: was: … -----------------------------------------

• Failure  Anticipated problem caught by the test case • Some “assertXXX” statement fails, throwing an AssertionError • Any other uncatched unexpected exception (Junit 4)

• Error (JUnit 3 only)  Unexpected problem in the test case • Not caused by failure of a “assertXXX” statement • Any other uncatched exceptions (e.g. NPE) • Can be caused by the tested implementation or test itself

Fixture

• Context for a test case

 Set of objects in a known state

• Sharing of fixtures among test cases is possible

 Useful for tests that use common set of objects • Remember: Tests are code like every other code  Remove duplications!  Fixture is reset before each test  tests are isolated

Fixtures in JUnit • Initialization  Create objects, store references in fields • JUnit 4: @org.junit.Before • JUnit 3: protected void setUp()

• Clean-up  Releasing external resources • JUnit 4: @org.junit.After • JUnit 3.8: protected void tearDown()

• JUnit framework automatically executes the methods at appropriate time

Simple Test Case with a Fixture (JUnit 4) public class TestArrayList { private List al; @Before public void setUp() { al = new ArrayList(); al.add(“abc”); }

object forming the fixture

@Test public void testAdd() { int origSize = al.size(); al.add(“def”); int newSize = al.size(); assertEquals(“Different size!”, origSize + 1, newSize); assertTrue(al.contains(“def”)); } }

Expected Exceptions • Test cases can test for method calls that should throw a given exception with given parameters (e.g. array out of bounds)  Throwing no exception is a failure  Throwing a different exception is a failure • JUnit 4 @Test (expected=MyEx.class) public void smth () { doSomeOperationThatThrowsException (); }

• JUnit 3 (works in 4 as well) public void testSmth () { try { doSomeOperationThatThrowsException (); fail (“Exception ‘MyEx’ not thrown”); } catch (MyEx e) { /* exception thrown, ignore it */ } }

Recommended Practice for Writing JUnit Tests • Place tests in the same package as sources (not direc.)  Directory layout • src/cz/cuni/mff/myapp/MyUtil.java • tests/cz/cuni/mff/myapp/TestMyUtil.java

 Allows easy testing of protected methods

• Put only one assertion in each test case, if possible  JUnit reports only the first failure for each test case @Test public void testSmth() { // set-up the test doIt(); assertTrue(cond1); assertTrue(cond2); }

@Before public void setUp() { // set-up the shared fixture }

vs.

@Test public void testCond1() { doIt(); assertTrue(cond1); } @Test public void testCond2() { doIt(); assertTrue(cond2); }

Unit Testing and Dependencies among Objects

Unit Testing and Dependencies among Objects • Issue  Units often have dependencies  it is hard to test them in isolation • Complex (and slow) fixtures and tests

 Example private MyPersistenceMngr pm; public void setUp() { java.sql.Connection dbconn = ... // complex code (DB population) pm = new MyPersistenceMngr(dbconn); }

• Solution  Test Double objects [Meszaros: xUnit Test Patterns] • Dummy, Fake, Stub and Mock

Dependencies: Kinds of Test Doubles • Dummy  Passed around, never used (fill parameter lists) • Fake  Working, but simpler implementations (e.g. in-memory DB) • Stub  “Empty” implementation with predefined responses to needed calls  Example public class ConnectionStub implements Connection { public Statement createStatement() { return new StatementStub(); } ... }

• Mock object  Stub that also checks if it is called correctly by the object under test • “Behavior verification”

Using Mock Objects

• Usage scenario    

Create mock objects Define expectations on mock objects Invoke methods on the tested unit Verify expectations

Mock objects

• Other benefits of mock objects  Faster testing (mock database, mock remote server)  Better logging of tested code  Easy fault insertion – simulate rare situations  Substitute unavailable hardware

• Cons  Need to know internals of objects, harder maintenance  Real objects (originals) not tested

JMock: Framework for Mock Objects

• Helps defining mock objects  Including expected interactions from the tested object  Automatic validation of the interactions

• Integration with JUnit 4 • Extendable ...

Example of jMock Usage public class TestPersistenceMngr extends TestCase { Mockery context = new Mockery ();

context for mock objects

public void testLoad () { Connection mockConn = context.mock (Connection.class); MyPersistenceMngr pm = MyPersistenceMngr (mockConn); context.checking (new Expectations () {{ oneOf (mockConn).createStatement (); }}); pm.load();

domain-specific language is used for definition of expectations

context.assertIsSatisfied (); // verify other assertions } }

verification of expectations

Definition of Expectations • Single method call context.checking (new Expectations () {{ oneOf (mockConn).createStatement (); }});

• Expected method parameter values context.checking(new Expectations () {{ oneOf (mockConn).setAutoCommit (true); }});

• Expected return values context.checking(new Expectations () {{ oneOf (mockConn).isClosed (); will (returnValue (false)); }});

Definition of More Complex Expectations • Sequence of method calls context.checking (new Expectations() {{ final Sequence seq = context.sequence (“myseq1”); oneOf (mConn).createStatement(); inSequence (seq); oneOf (mConn).commit(); inSequence (seq); }});

• Any number of calls of a method (incl. none) context.checking (new Expectations () {{ allowing (mockConn).prepareStatement (); }});

JUnit vs. jMock (~ Classical and Mockist Testing) • Classic JUnit tests  State verification • “assertXXX” statements examine the state of a program fragment under test  Use real objects when possible (~ mini-integration testing)

• JUnit with mock objects  Behavior verification • Correct interaction among objects is checked  Use mock objects always

Recommended practice  Combination of state and behavior verification • JUnit + jMock  Nice comparison: Martin Fowler - Mocks Aren't Stubs

Assessing and Testing the Tests

Test Coverage • What is the percentage of "code" covered by tests? • Coverage criteria       

Function coverage Statement coverage Decision (branch) coverage – control statements Condition (predicate) coverage – boolean expressions Path coverage Entry/exit coverage Data flow coverage

• Safety critical systems  100% required (for certain criteria)

Tools for Assessing Code Coverage (Java)

• Basic idea  Instrument the code to log information needed to calculate coverage  Run the unit tests on the instrumented code, analyze the logs

• Main instrumentation approaches  Source code instrumentation  Bytecode instrumentation  Modified JVM or runtime agents such as JVMDI/JVMPI

• Many tools exist  Cobertura, Coverlipse, Clover ...

Cobertura

• Uses bytecode instrumentation • Supports line and branch coverage and also McCabe cyclomatic code complexity  Class, package, project level

• HTML/XML output • Ant/Maven integration...

Cobertura – Ant Example ...

Cobertura – Sample output

• http://cobertura.sourceforge.net/sample/

Coverlipse, Clover

• Coverlipse  Eclipse plugin – integration, navigation  Block (statement) and “all-uses” coverage • Data flow analysis – considers pairs of variable definition – usage

• Clover    

Commercial, free license available Source code instrumentation Statement and branch coverage Test optimization • Runs only tests covering code that changed since last build

Mutation Test – Testing the Tests

• Automated injection of a fault  Modification of the source code • • • • •

Deleting a statement Replacing a boolean expression with true or false Replacing an arithmetic operator with another one Replacing a comparison operator with another one Replacing a variable with another in the same scope and of the same type

 Modification of binary code • Replacing an instruction…

• Verifying that tests detect a fault  If not, code may be dead or tests incomplete

Mutation Test – Testing the Tests

• Tools  Jester (http://jester.sf.net)  Nester  Pester  Jumble (http://jumble.sourceforge.net/)

Other Levels of Testing

Other Levels of Testing • Levels    

Integration testing System testing Acceptance testing …

Testing larger fragments of a software system

• Tools  JUnit + jMock, … • More complex fixtures  real objects are used instead of mocks • Mocks are used for components that are assumed to be correct  For example, a SQL database or a third-party library

 Fit + FitNesse • Acceptance tests where expectations are defined via Wiki or Excel tables  Customers write expectations (Wiki, Excel)  Developers write test cases and fixtures (Java code)

Test Driven Development (TDD) - Overview • Development process – repetition of    

Write tests first Verify that they fail Implement functionality only to make them pass Refactor, check they still pass

• Advantages  More reverting than debugging (which is sometimes more efficient)  Good test coverage  Usually leads to simpler and cleaner design  Only needed code implemented

• Disadvantages  Discourages experimenting  Same blind spots in both test and implementation

Test Driven Development (TDD)

• Three stages:  RED: Failing test(s)  GREEN: All tests pass  Refactor: Eliminate duplicities

Interesting Tools

• Pex  Automatically generates test suites  High coverage • Generates “interesting” inputs for tested functions

 Visual Studio plugin  http://research.microsoft.com/en-us/projects/pex/  http://www.pexforfun.com/ - Puzzles to solve

• CHESS  Multi-threaded testing  Generates and reproduces interleavings  http://research.microsoft.com/en-us/projects/chess/

Links • Software Testing  http://en.wikipedia.org/wiki/Software_testing

• JUnit  http://junit.org  http://junit.sourceforge.net/doc/faq/faq.htm

• jMock  http://www.jmock.org/  http://martinfowler.com/articles/mocksArentStubs.html

• HtmlUnit  http://htmlunit.sourceforge.net/

Links

• Cobertura  http://cobertura.sourceforge.net/

• Coverlipse  http://coverlipse.sourceforge.net/

• Clover  http://www.atlassian.com/software/clover/

• Fit + FitNesse  http://fit.c2.com/  http://fitnesse.org/

Links

• Pex  http://research.microsoft.com/en-us/projects/pex/

• CHESS  http://research.microsoft.com/en-us/projects/chess/

• Jumble  http://jumble.sourceforge.net/ • Jester  http://jester.sourceforge.net/

Books

• Kent Beck: Test-Driven Development by Example • Steve McConnell: Code Complete