Test Suite Evolution - ICST 2012

S.T.A.R. @ Faculty of Informatics University of Lugano Switzerland

Supporting

Test Suite Evolution through Test Case Adaptation

Mehdi Mirzaaghaei

Fabrizio Pastore

Mauro Pezzè

public final class DateTimeField{ .. public final class CopticChronology{ .. }

Test Evolution

} public class Report{ .. addRule(int line, String file) } ReportTest{ test(){ Report r = new Report(); r.addRule(5, file ); assertFalse(r.isEmpty()); }

JodaTime r1.62


Test Evolution

} public class Report{ .. addRule(int line, Context String file) ctx) } ReportTest{ test(){ Report r = new Report(); r.addRule(5, file ); assertFalse(r.isEmpty()); }

JodaTime r1.62


Test Evolution

} public class Report{ .. addRule(int line, Context String file) ctx) } ReportTest{ test(){ Report r = new Report(); r.addRule(5, file ); assertFalse(r.isEmpty()); ReportTest{ test(){ Report r = new Report(); r.addRule(5, new Context( file ) ); assertFalse(r.isEmpty());

}

}

JodaTime r1.62


Test Evolution

public final class EthiopicChronology{ ..

}

public class Report{ } .. addRule(int line, Context String file) ctx) } ReportTest{ test(){ Report r = new Report(); r.addRule(5, file ); assertFalse(r.isEmpty()); ReportTest{

EthipicChronologyTest{ .. }

test(){ Report r = new Report(); r.addRule(5, new Context( file ) ); assertFalse(r.isEmpty());

}

}

JodaTime r1.62

public final class DateTimeField{ .. public final class CopticChronology{ .. } }

Test Evolution

public final class EthiopicChronology{ .. public class Report{ } .. addRule(int line, Context String file) ctx) }

CopticChronologyTest{ ReportTest{ EthipicChronologyTest{ Chronology COPTIC_UTC test(){ = CopticChronology.getInstanceUTC(); .. Report r =1,new DateTime epoch = new DateTime(1, 1, Report(); 0, 0, 0, 0, COPTIC_UTC); r.addRule(5, file ); long millis = epoch.getMillis(); } assertFalse(r.isEmpty()); ReportTest{ long end = new DateTime(3000, 1, 1, 0, 0, 0, 0, ISO_UTC).getMillis(); test(){ DateTimeField monthOfYear = COPTIC_UTC.monthOfYear(); } Report r = new Report(); r.addRule(5, new Context( file ) ); while (millis < end) { assertFalse(r.isEmpty()); int monthValue = monthOfYear.get(millis); if (monthValue < 1 || monthValue > 13) fail("Bad month: }" + millis); }

JodaTime r1.62

public final class DateTimeField{ .. public final class CopticChronology{ .. } }

Test Evolution

Reuse Information

public final class EthiopicChronology{ ..

public class Report{ } .. addRule(int line, Context String file) ctx) }

available in

Existing Test Cases

CopticChronologyTest{ ReportTest{ EthipicChronologyTest{ Chronology COPTIC_UTC test(){ = CopticChronology.getInstanceUTC(); .. Report r =1,new DateTime epoch = new DateTime(1, 1, Report(); 0, 0, 0, 0, COPTIC_UTC); r.addRule(5, file ); long millis = epoch.getMillis(); } assertFalse(r.isEmpty()); ReportTest{ long end = new DateTime(3000, 1, 1, 0, 0, 0, 0, ISO_UTC).getMillis();

to

Evolve Test Suites

test(){ DateTimeField monthOfYear = COPTIC_UTC.monthOfYear(); } Report r = new Report(); r.addRule(5, new Context( file ) ); while (millis < end) { assertFalse(r.isEmpty()); int monthValue = monthOfYear.get(millis); if (monthValue < 1 || monthValue > 13) fail("Bad month: }" + millis); }

JodaTime r1.62

EthiopicChronologyTest{

Test Evolution

public final class DateTimeField{ Chronology ETHIOPIC_UTC = EthiopicChronology.getInstanceUTC(); .. DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, ETHIOPIC_UTC); public long final class =CopticChronology{ millis epoch.getMillis(); .. } long end = new DateTime(3000, 1, 1, 0, 0, 0, 0, ISO_UTC).getMillis(); DateTimeField monthOfYear = ETHIOPIC_UTC.monthOfYear(); public final class EthiopicChronology{ } .. while (millis < end) { int monthValue public class Report{= monthOfYear.get( new Date(millis) ); } if (monthValue < 1 || monthValue > 13) .. fail("Bad month: " + millis); addRule(int line, Context String file) ctx) } } CopticChronologyTest{ ReportTest{ EthipicChronologyTest{ Chronology COPTIC_UTC test(){ = CopticChronology.getInstanceUTC(); .. Report r =1,new DateTime epoch = new DateTime(1, 1, Report(); 0, 0, 0, 0, COPTIC_UTC); r.addRule(5, file ); long millis = epoch.getMillis(); } assertFalse(r.isEmpty()); ReportTest{ long end = new DateTime(3000, 1, 1, 0, 0, 0, 0, ISO_UTC).getMillis(); test(){ DateTimeField monthOfYear = COPTIC_UTC.monthOfYear(); } Report r = new Report(); r.addRule(5, new Context( file ) ); while (millis < end) { assertFalse(r.isEmpty()); int monthValue = monthOfYear.get(millis); if (monthValue < 1 || monthValue > 13) fail("Bad month: }" + millis); }

JodaTime r1.62

Test Care Assistant (TCA) Add Test Cases for new Modules

Repair Compilation Errors due to Signature Changes

new Class in Hierarchy

new Interface Implementation

Parameter Addition

Parameter Type Change

New Overloaded Method

New Overridden Method

Parameter Removal

Return Type Change

Test Care Assistant (TCA) Add Test Cases for new Modules




Parameter Addition




Parameter Removal

Return Type Change

Test Care Assistant Add Test Cases for new Modules

Repair Compilation Errors in Signature Changes



Parameter Addition




Parameter Removal

Return Type Change

Generate Test Cases for Classes added to a Hierarchy Identify & Copy Candidate Test Cases Adapt Test Cases Update References to Class Under Test Repair Compilation Errors Repair Runtime Errors

Remove Redundant Test Cases

Generate Test Cases for Classes added to a Hierarchy Identify & Copy Candidate Test Cases Adapt Test Cases Update References to Class Under Test Repair Compilation Errors Repair Runtime Errors


Identify & Copy Candidate Test Cases Chronology BasicChrono GJChrono Julian

Gregorian

BuddhistChrono

FixedMonthChrono Coptic...

Ethiopic...

JulianTest

GregorianTest

CopticTest

BuddhistTest

testCalendar()

testCalendar() testMax()

testCalendar() testEpoch()

testCalendar() testEpoch() JodaTime r1.62


Gregorian

BuddhistChrono


Ethiopic...

JulianTest

GregorianTest

CopticTest

BuddhistTest

testCalendar()





Gregorian

EthiopicTest

BuddhistChrono


Ethiopic...

testCalendar_1() testCalendar_2() testCalendar_3() testCalendar_4() testEpoch_1() testEpoch_2() testMax()

JulianTest

GregorianTest

CopticTest

BuddhistTest

testCalendar()




Identify & Copy Candidate Test Cases Chronology COPTIC_UTC =Chronology CopticChronology.getInstanceUTC(); EthiopicTest DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, COPTIC_UTC); assertEquals( CopticChronology.AM, epoch.getEra() ) testCalendar_1() BasicChrono BuddhistChrono testCalendar_2() ... GJChrono Julian

Gregorian


Ethiopic...

testCalendar_3() testCalendar_4() testEpoch_1() testEpoch_2() testMax()

JulianTest

GregorianTest

CopticTest

BuddhistTest

testCalendar()




Update References to Class Under Test Chronology COPTIC_UTC =Chronology CopticChronology.getInstanceUTC(); EthiopicTest DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, COPTIC_UTC); 31111111 assertEquals( CopticChronology.AM, epoch.getEra() ) testCalendar_1() BasicChrono BuddhistChrono testCalendar_2() ... GJChrono Julian

JulianTest

Gregorian

GregorianTest


Ethiopic...

CopticTest


BuddhistTest

Chronology COPTIC_UTC = CopticChronology.getInstanceUTC(); testCalendar() testCalendar() testCalendar() testCalendar() DateTime epochtestMax() = new DateTime(1, 1, 1, 0, 0, 0, testEpoch() 0, COPTIC_UTC); testEpoch() assertEquals( CopticChronology.AM, epoch.getEra() ) ... JodaTime r1.62


Gregorian


Ethiopic...


EthiopicChronology CopticTest BuddhistTest Chronology COPTIC_UTC = CopticChronology.getInstanceUTC(); testCalendar() testCalendar() testCalendar() testCalendar() DateTime epochtestMax() = new DateTime(1, 1, 1, 0, 0, 0, testEpoch() 0, COPTIC_UTC); testEpoch() assertEquals( CopticChronology.AM, epoch.getEra() ) EthiopicChronology ... JodaTime r1.62 JulianTest

GregorianTest


Gregorian


Ethiopic...


ETHIOPIC CopticTest GregorianTest BuddhistTest Chronology COPTIC_UTC = CopticChronology.getInstanceUTC(); EthiopicChronology.getInstanceUTC(); ETHIOPIC testCalendar() testCalendar() testCalendar() testCalendar() DateTime epochtestMax() = new DateTime(1, 1, 1, 0, 0, 0, testEpoch() 0, COPTIC_UTC); testEpoch() assertEquals( EthiopicChronology.AM, CopticChronology.AM, epoch.getEra() epoch.getEra()) ) ... JodaTime r1.62 JulianTest

Adapt Candidate Test Cases

Undefined Fields

Missing Constructors

Missing Methods

Adapting Undefined Fields Chronology COPTIC_UTC = CopticChronology.getInstanceUTC(); DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, COPTIC_UTC); assertEquals( CopticChronology.AM, epoch.getEra() ) ... FixedMonthChrono CopticChronology +AM

EthiopicChronology +EE

Chronology ETHIOPIC_UTC = EthiopicChronology.getInstanceUTC(); DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, ETHIOPIC_UTC); assertEquals( EthiopicChronology.AM, epoch.getEra() )

Adapting Undefined Fields Chronology COPTIC_UTC = CopticChronology.getInstanceUTC(); DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, COPTIC_UTC); assertEquals( CopticChronology.AM, epoch.getEra() ) ... FixedMonthChrono CopticChronology +AM

EthiopicChronology +EE


Adapting Undefined Fields

class CopticChronology {

class EthiopicChronology {

/** Serialization lock */ long serialVersionUID = -59L;


Chronology COPTIC_UTC = CopticChronology.getInstanceUTC(); /** /** DateTime epoch = new DateTime(1, 1, 1, 0, 0, 0, 0, COPTIC_UTC); * Constant value for ... * Constant value for ... assertEquals( CopticChronology.AM, epoch.getEra() ) */ */ ...public static final int AM = public static final int EE = DateTimeConstants.CE;

DateTimeConstants.CE;

/** A singleton era field. */ /** A singleton era field. */ FixedMonthChrono private DateTimeField private ateTimeField ERA_FIELD = new ERA_FIELD = new CopticChronology EthiopicChronology SingleEraDateTimeField("AM"); SingleEraDateTimeField("EE");

+AM

+EE










+AM

+EE










+AM

+EE










+AM

+EE


EE

Adapting Undefined Constructors Output GraphicOutput public void testDefaultFont() { g = new GraphicsMock(); output = new GraphicsOutput( g, DefaultEnv.DEFAULT_FONT, fgColour, bgColour); [..] assertEquals("graphicsOutput", output.toString()); }

SVGOutput public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( g, DefaultEnv.DEFAULT_FONT, fgColour, bgColour); [..] assertEquals("graphicsOutput", output.toString()); } Barbecue 1.5b1

Adapting Undefined Constructors Output GraphicOutput

SVGOutput

public void testDefaultFont() { public void testDefaultFont() { g = new GraphicsMock(); g = new GraphicsMock(); output = new SVGOutput( output = new GraphicsOutput( g, g, DefaultEnv.DEFAULT_FONT, DefaultEnv.DEFAULT_FONT, fgColour, fgColour, bgColour); bgColour); [..] [..] assertEquals("graphicsOutput", assertEquals("graphicsOutput", Compilation error: output.toString()); output.toString()); The constructor } } SVGOutput(Graphics2D, Font,

Color, Color) is undefined Barbecue 1.5b1

Adapting Undefined Constructors Output GraphicOutput public void testDefaultFont() { g = new GraphicsMock(); output = new GraphicsOutput( g, DefaultEnv.DEFAULT_FONT, GraphicOutput( fgColour, Graphics2D, bgColour); Font font, [..] Color fgColor, assertEquals("BLA", Color bgColor) output.toString()); }

SVGOutput SVGOutput(

public void testDefaultFont() { Writer writer, g = new GraphicsMock(); font, outputFont = new SVGOutput( g,Color fgColor, Color bgColor, DefaultEnv.DEFAULT_FONT, double scalar, fgColour, String units ) bgColour); [..] assertEquals("BLA", output.toString()); SVGOutput( } Writer writer,

Font fonts)

Adapting Undefined Constructors Output GraphicOutput public void testDefaultFont() { g = new GraphicsMock(); output = new GraphicsOutput( g, DefaultEnv.DEFAULT_FONT, GraphicOutput( fgColour, Graphics2D, bgColour); Font font, [..] Color fgColor, assertEquals("BLA", Color bgColor) output.toString()); }


public void testDefaultFont() { Writer writer, g = new GraphicsMock(); font, outputFont = new SVGOutput( g,Color fgColor, Color bgColor, DefaultEnv.DEFAULT_FONT, double scalar, fgColour, String units ) bgColour); [..] assertEquals("BLA", output.toString()); SVGOutput( } Writer writer,

Font fonts)

Adapting Undefined Constructors Output GraphicOutput


public void testDefaultFont() { public void testDefaultFont() { Writer writer, g = new GraphicsMock(); g = new GraphicsMock(); +2 font, outputFont = new SVGOutput( output = new GraphicsOutput( g,Color fgColor, +2 g, Score=6 Color bgColor,+2 DefaultEnv.DEFAULT_FONT, DefaultEnv.DEFAULT_FONT, GraphicOutput( double scalar, fgColour, fgColour, Graphics2D, String units ) bgColour); bgColour); Font font, [..] [..] Color fgColor, assertEquals("BLA", assertEquals("BLA", Color bgColor) Score=2 output.toString()); SVGOutput( output.toString()); } } Writer writer,

Font fonts) +2

Adapting UndefinedSVGOutput( Constructors Writer writer, Font font, Output Color fgColor, Color bgColor, double scalar, GraphicOutput SVGOutput String units ) public void testDefaultFont() { g = new GraphicsMock(); output = new GraphicsOutput( g, DefaultEnv.DEFAULT_FONT, fgColour, bgColour); [..] assertEquals("graphicsOutput", output.toString()); }

public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( ...., DefaultEnv.DEFAULT_FONT, fgColour, bgColour, ..., ...; [..] assertEquals("graphicsOutput", output.toString()); } Barbecue 1.5b1


public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( ...., new StringWriter( EasyMock(Writer) ... ) ), DefaultEnv.DEFAULT_FONT, fgColour, bgColour, ..., ...; [..] assertEquals("graphicsOutput", output.toString()); } Barbecue 1.5b1


public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( ...., new StringWriter( EasyMock(Writer) ... ) ), DefaultEnv.DEFAULT_FONT, fgColour, bgColour, 0..., , “”...; ); [..] assertEquals("graphicsOutput", output.toString()); } Barbecue 1.5b1

Adapting Runtime Failures Test Failures: apply ReAssert

Daniel et. al.[ASE09,ISSTA10]

public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( new StringWriter(EasyMock(Writer)), DefaultEnv.DEFAULT_FONT, fgColour, bgColour, 0, “”; [..] assertEquals("graphicsOutput", output.toString()); }



public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( junit.framework.AssertionFailedError: new StringWriter(EasyMock(Writer)), expected: but DefaultEnv.DEFAULT_FONT, was: fgColour, bgColour, 0, “”; [..] assertEquals("graphicsOutput", output.toString()); }



public void testDefaultFont() { g = new GraphicsMock(); output = new SVGOutput( new StringWriter(EasyMock(Writer)), DefaultEnv.DEFAULT_FONT, fgColour, bgColour, 0, “”; "SVGOutput" [..] assertEquals("graphicsOutput", output.toString()); }

Adapting Runtime Failures Uncaught Exceptions: involve users

Add Test Cases Identify & Copy Candidate Test Cases Adapt Test Cases Update References to Class Under Test Repair Compilation Errors Repair Runtime Errors






Parameter Addition




Parameter Removal

Return Type Change

Repair Signature Changes BankAccount deposit(int cents)

BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());

BankAccount deposit(Money cents)


BankAccount deposit(Money cents) Compilation error: Method deposit(Money) in the type BankAccount is not applicable for the arguments (int)

BankAccount account = new BankAccount(); int amount = 500; account.deposit(amount); assertEquals( 500, account.getBalance());


BankAccount account = new BankAccount(); int amount = 500; Money money = new Money(500); account.deposit(amount); account.deposit(money); assertEquals( 500, account.getBalance());

BankAccount deposit(Money cents)

Repair Signature Changes Analyze the Change

Determine Initialization Values

Repair the Error

Repair Signature Changes Analyze the Change Change Type


Repair the Error

Parameter Add Parameter Type Change Parameter Removal Return Type Change

Repair Signature Changes BankAccount

Analyze the Change

deposit(int cents)

Change Type

deposit(Money cents)


Repair the Error



Analyze the Change

deposit(int cents)

Change Type




pairs

Repair the Error


Analyze the Change

deposit(int cents)

Change Type




pairs

Repair the Error Money money = new Money(500); account.deposit(money);

EVALUATION

Q1: APPLICABILITY Q2: EFFECTIVENESS





Parameter Addition




Parameter Removal

Return Type Change

Repair Signature Changes Applicability Subject # versions Cassandra 19 Geronimo 4 POI 6 Santuario 4 Shindig 6 Velocity 15 Wicket 7 Xindice 9 XMLBeans 7

Repair Signature Changes Applicability 100

75

50

% applicability

90.1

69.3

54.0

Average= 45.9

50.6

46.9 41.7 35.2 25 18.7

0

6.9 Cassandra Geronimo

POI

Santuario

Shindig

Velocity

Wicket

Xindice

XMLBeans

Repair Signature Changes Effectiveness

Project PMD JodaTime JFreeChart

# Versions # Tests 13 2 6

99 19 21

21

138

Repair Signature Changes Effectiveness 138

# of tests

104

138

128 105

69

35

0

Fixed Errors Valid Test Cases Same as Developers





Parameter Addition




Parameter Removal

Return Type Change

Applicability 74

Software version 275

JFreeChart

0.96

JodaTime Barbecue Xstream PMD

1.62 1.5 1.31 4.2

KLOC 28

120

classes 315

217

471

64 9 25 65

99 55 218 483

Applicability 80

74 % applicability

60

Average= 60.9%

275 28

40

315

120

20

0

JFreeChart JodaTime Barbecue Xstream 275/471

74/99

28/55

120/218

PMD 315/483

% classes for which TCA can generate tests

Effectiveness TCA

Developer’s

Test Generation Tools

Randoop

• random test generation • run for 50 sec on each class

• •

industrial, coverage based run for each classes

• •

search-based testing run with assertion generation

Subjects

classes

Xstream 1.31 PMD 4.2 Barbecue 1.5b1 JodaTime 1.62 JfreeChart 1.013 Total

36 56 12 32 204 340

Coverage Results % Instructions Covered

100

75

68.9 50

61.7

60.5 51.3 36.3

25

0

TCA

Developers Randoop

CodePro

EvoSuite

TCA and Evosuite are Complementary EvoSuite 6,208 Common 12,055 TCA 6,370

Covered Statements

Test Evolution

Test Care Assistant

Test Reuse Algorithms

Evaluation

FUTURE WORK

Test Evolution

Test Reuse Algorithms

•Evaluate other Test Reuse Algorithms •User study on readability of generated tests Test Care Assistant

Evaluation