Continuous Integration

A Taming of the Tornado U i Using

Ants on a Mobius Strip by M.C. Escher

Outline • • • • • • • • •

Integration Hell Software Development Nirvana What is Continuous Integration? p g Continuous Integration g Implementing CI Toolbox Success Criteria Good Resources Questions, comments, stories,… Contact info 2

Integration Hell

3

Software Development NIRVANA

4

Evolution of Continuous Integration

5

What is Continuous Integration?

“The macro process of object-oriented d development l t is i one off "continuous " ti integration." i t ti " ... At regular intervals, the process of "continuous integration" yields executable releases that grow in functionality at every release. ... It is through these milestones that management can measure progress p g and quality, q y, and hence anticipate, p , identify, and then actively attach risks on an ongoing basis.” -- Grady Booch, Object Object-Oriented Oriented Analysis and Design with Applications, 2nd ed, 1993

6

What is Continuous Integration? “A common practice at Microsoft and some other shrink-wrap software companies is the ‘daily build and smoke test’ p process. Every y file is compiled, p , linked, and combined into an executable program every day, and the program is then put through a ‘smoke smoke test’ test , a relatively simple check to see whether the product ‘smokes’ when it runs.” -- Steve McConnell, Daily Build and Smoke Test, IEEE Software Software, July 1996

Microsoft called daily builds the “Sync Pulse” of a project -- Microsoft Secrets, 1995 7

What is Continuous Integration? “Continuous Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates i t t att least l t daily d il - leading l di to t multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly g y reduced integration g problems p and allows a team to develop cohesive software more rapidly.” -- Martin Fowler, Continuous Integration, 2000

8

What is Continuous Integration? Continuous Integration and the Möbius Strip Mobius Strip p ¨ never-ending, g, one-sided shape p ¨ continuous A su surface ace with t o only yo one es side de a and do only yo one e bou boundary da y component. ¨ mathematical property of being non-orientable ¨ a ruled surface ¨ discovered independently by the German mathematicians August Ferdinand Möbius and Johann Benedict Listing in 1858

9

What is Continuous Integration? • P Provides id a powerful f l method th d off fi finding di d defects f t early enough to fix them easily • Continuous integration automatically: – Integrating code frequently (2 hours recommended) – Monitors source code – Compiles after every change – Unit tests compiled code – Notifies developers of problems as encountered 10

Return on Investment Complexity

=

Cost

Each connection represents potential for defect

Less Complexity = smaller, manageable defects 11

What is Continuous Integration? • Many statistics show projects fail more often than they succeed • All studies agree: – Projects very often fail. fail – They are late. – They are over-budget. over budget – They fail to deliver the product they were designed to produce. produce – Some projects don't produce any product at all all. 12

Why is it Important? -- History Lessons Toyota - Recalled 160,000 Prius (new hybrid) for software bug Mars Climate Orbiter - Destroyed due to navigation error Th Therac-25 25 - Delivered D li d lethal l th l radiation di ti d doses – 5 patients ti t died di d National Cancer Institute - In Panama City, therapy planning software miscalculates radiation dosage - 28 overdoses, 8 died UK Air Traffic Control - Upgrade problems caused thousands of passenger delays and plane groundings across the country Denver International Airport - Baggage handling system problems Cost estimate - $1.1 problems. $1 1 million lost per day FAA - Advanced Automation System overran planned budget by ~ $3 billion IRS - $8 billion software modernization program cost U U.S. S taxpayers $50 billion per year in lost revenue California motor vehicle - Registration system cancelled after topping $44 million in a long series of overruns B-2 bomber - Wouldn’t fly on its maiden flight because of a software problem Ariane 5 rocket - Software error caused blow up p on maiden launch Seattle, Computer-controlled ferries problems caused dozens of dozen dock crashes with damages > $7M

13

Why is it Important? -- The Future True Global Civilization - complete with a unified language, culture and planet-wide technological prowess. A global community it capable bl off sustaining t i i and d controlling t lli itits planet. l t Control of Global Warming - Human-caused global warming is visible in the air, water and melting ice, and is destined to get much worse in the future future. Alternate Energies and Fuel - Managing the planet's resources effectively using advances in computing, engineering and other fields fields. Geothermal power showing promise promise. Cloak of Partial Invisibility - a cloaking device that reroutes certain wavelengths of light, forcing them around objects. the cloaked object would appear nearly invisible in microwave light light. Molecular Memory - ultradense integrated memory circuit from nanowires and molecules developed. Problems are still being worked out of device; expect these as common by 2020. Microrobot - group of international scientists developing microrobot able to swim through arteries and digestive system. Exploding p g Robots - fleet of exploding p gp probes could p prepare p the way for warding off hazardous asteroids. Bomb/Drug Sniffing Robots - able to track dilute scents based on new smell-seeking algorithm called infotaxis.

14

Reality: Projects Fail - especially IT projects Battle cry is loud and clear “Projects are failing more often than they are successful. successful Something must be done!” done! But what ??? One Good Answer: Continuous Integration

15

Not Just For Agile • Popularized in the XP World • Any Software Development Lifecycle benefits from Continuous Integration g – Find problems early – Correct problems while fresh and less complex – Promote communications/collaboration

Scrum

Spiral Project Life Cycle

Rational Unified Process (RUP) Build 1

Requirements Analysis

Preliminary Design

Preliminary Design

Preliminary Design

Detailed Design

Detailed

g or Iterative Build Design Serial Incremental Serial, Incremental, Project Life Code & Cycle

Preliminary Design

Build 1

Build n

Code & Unit Test

System Integration & Test

Detailed Design

• Update Plan • Update Procedures Formal Test • Review Progress • Baseline Product

Code &

4. Manage & Plan System Integration

Build 1 . . . n

Detailed Design

Code & Unit Test

Formal Test

System Integration & Test

Note that the builds can be serial or overlapping

• Identify Risks • Analyze Risks • Evaluate Risks • Plan Risk Mitigation

Risk Analysis Review Plan Commitment C it t Risk Walkthrough

Final System Unit Test Integration & Test

System Integration & Test Formal Test

Management Review System Integration & Test

• Develop/Update Estimations • Define Cycle /Update Cycle Definition

Unit Test

Requirements Analysis

Detailed Design

Code & Unit Test

Definition Review

Preliminary Design

Waterfall Project Life Cycle Build n

Requirements Analysis

2. Analyze & Avert Risk

Evolutionary Project Life Cycle

Requirements Analysis

. . .

Overlapping Waterfall Project Life Cycle

Requirements Analysis

1. Define Approach

. . .

• Monitor & Review • Develop & Verify Product

Development Plan Commitment

Product Review

Formal Test

16

3. Develop Product

&that Test Formal Test Note the figure only represents one cycle in the spiral.

Getting from Here to There • Are you in the tornado? • How can you start the path to Nirvana? • It is a matter of continuous incremental i improvements! ! • A CI platform becomes a place to “bolt on” repeatability 17

The CI Process

18

The CI Process (continued)

19

Getting There • Backlog: – Establish source repository and standards – Identify and configure the Continuous Integration Server (the “build machine”) – Write a continuous integration script – Automate the build process and install into CI tool – Automate the deployment process and install into CI tool – Automate the Unit Test Suite execution and install into CI tool – Establish software development p team standards to create (and increase coverage of) automated testing

20

Source Repository and Standards • We must be able to revert to previous versions of any artifact. This is critical in being able to quickly recover from a CI build failure. failure Rule

Whyy

Version everything needed for the build.

“Everything” goes into the repository. Everything includes: test scripts, properties files, database schema, install scripts, SQL scripts, third party libraries, documentation …

Don’t version things that get built.

Nothing that gets built through a tool should be stored in the repository, only those things that are needed to perform a build. build 21

The CI Server • This is the “brains” brains of CI • Install and configure a CI tool – Candidates: • CruiseControl from ThoughtWorks (www.thoughtworks.com) (www.urbancode.com) urbancode com) • Anthill Professional from Urbancode (www

Repository

Communications Portal (email, web interface,…)

CI Server

Deployment Servers 22

“Clean Machine” Deployment • “Clean Clean Machine” Machine Automation • This should include only those tools that are too difficult to script the installation. installation This would include the operating system, compilers, JVM, .Net run-time, database servers … • Deb Deb’s s picture here Repository

Communications Portal (email, web interface,…)

CI S Server

D l Deployment tS Servers 23

Script the CI Process • See Process Diagram from previous slide • Timely Feedback • T Typically, i ll lless th than 15 minutes. i t If more, consider id a ““staged t d build” approach Rule

Why

Maintain “clean machine” build scripts

Scripts are written and maintained in such a way that someone should be able to walk up to a “clean p , and machine”,, execute the scripts, fully build the system.

p ffrom a single g Launch the CI scripts command.

Simplicity. p y 24

Automated Builds • Automate the build from source to executable • Common tools such as make and ant can be used • Can identify other tools/best practices that should be included in automated build – Candidates: • Build documentation (e.g. JavaDocs) • Automated code standards (e.g. FxCop, CheckStyle, and Jalopy) • Automated test coverage checkers (Did the developer embed automated testing?)

Rule

Why

The build process must be reusable across Dev, QA, and Production

This creates repeatability as moving from one software development stage to the next next.

The output of the build should be placed in a location where anyone can retrieve it

Builds need to be accessible to anyone on the team 25

Automated Deployments • Automate the Deployment to the “Clean Machine” • Typically done with the OS and DB scripting languages

R l Rule

Wh Why

The deployment script should allow different deployment targets. targets

This creates repeatability as moving from one software development stage to the next.

After deployment, system should execute a “Smoke Test” to ensure proper configuration

This will allow the production support team, and others, to use the script for deployment and have some validation of proper configuration without running through the full Unit Test Suite 26

Automated Unit Tests • Automate the execution of a Test Suite of Unit Test Cases • Choose a testing tool for each language – Candidates: Java - Junit,, .Net - Nunit,, …

• The entire suite of tests should be able to be executed from a single command • Establish automated testing standard Rule

Why

Always add tests to code when a code module is created or modified f

This will maintain and/or increase test coverage g off code base.

Synchronize with repository before checking in.

Always synchronize your working copy with changes in the repository (mainline or branch) b h) before b f checking h ki code d in. i 27

Other Standards Rule

Why

Only check it in if it works.

The mainline and branch must be kept in working condition so that others can safely check out working copies. copies

Run the scripts before you check in.

Always run the automated build/deploy and test script p in yyour local workingg copy py and development environment before checking code in. This approximates the CI process as closely as possible and will avoid those pesky “failure” emails.

Everyone commits every day.

By doing this frequently, developers quickly find out iff there's a conflict f between two developers. The key to fixing problems quickly is finding them quickly.

If the h CI build b ild breaks, b k fix fi it i right i h away.

The whole Th h l point i off working ki with i h CI is i that h you're ' always developing on a known stable base.

28

Success Criteria • Be prepared to invest some time and effort in the beginning – Saves time later – Builds quality into product

• • • • • •

Ensure you are working with the “latest” source code Never commit “broken” code Always do a “clean build” Never use CI as a “shaming event” Di ib Distribute ffailure il ffeedback db k as quickly i kl as possible ibl to team Build/Compile/Test cycle timing should be reasonable – Break build up into logical chunks – Add more machines – Minimize testing if

• Make the CI process a team decision • Look at third-party tools rather than build your own tool

29

Continuous Integration Toolbox

• Tool Considerations – Functionality F ti lit – Compatibility – Reliability – Longevity – Usability

30

Continuous Integration Toolbox • Typical T i lC Continuous ti IIntegration t ti Toolbox T lb – Continuous Integration System – provides reports, t statistics, t ti ti notifications, tifi ti etc. t to t furnish f i h the framework of the CI build process – Build B ild T Tools l – scripting i ti or automating t ti code d compilation to convert software to run on a computer – Repository – where software components are stored – Automated Testing Tools – automating the unit or developer level testing 31

Continuous Integration Toolbox

Continuous Integration System CruiseControl Hudson LuntBuild Continuum Draco.NET CI Factory y Drumbeat CI Tinderbox BuildBot Anthill

Beetlejuice Gump Sin Parabuild Pulse TeamCity Bamboo

Build Tools Shell/command script Ant NAnt Maven Groovyy MSBuild Make Visual Studio

Repository

Automated Testing Tools

SubVersion CVS Visual Sourcecode PVCS MKS Dimensions Vault Surround Mercurial CMSynergy

NUnit (.NET) ( NET) JUnit (Java) SUnit (Smalltalk) PYUnit (Python) CPPUnit (C++) UnitTest g ((GUI)) Dogtail HttpUnit (Web) Parasoft

Many tools available are Open Source and Free

32

Good Resources • Continuous Integration: Improving Software Quality and Reducing Risk by Paul Duvall, Steve Matyas, and Andrew Glover • Martin Fowler's introduction to Continuous Integration: http://www.martinfowler.com/articles/continuousIntegration .html • Extreme Programming Website: htt // http://www.extremeprogramming.org/rules/integrateoften.h t i / l /i t t ft h tml • Daily Build and Smoke Test by Steve McConnell: http://www.stevemcconnell.com/bp04.htm 33

Questions, comments, stories,…

34

Contact Us

President/CEO Tod Pryor [email protected]

Business Development/ Process Improvement Deb Jacobs [email protected]

11717 M Circle Omaha,, NE 68137 Phone: 402.445.4700 Toll free: 866.PTI.CORP Fax: 402 402.445.4759 445 4759

www.prioritytech.com 35