Refactoring Databases Evolutionary Database Development.

Refactoring Databases Evolutionary Database Development.

Pramod Sadalage ThoughtWorks Inc. Copyright 2007. Copyright is held by author.

Overview

• • • • •

Introduction Evolutionary Database Development I Evolutionary Database Development II Technical Foundation Conclusion

Copyright 2007. Copyright is held by author.

Traditional Database • • • • • • •

Continuous changes not allowed Production changes are rare Afraid to fix, design issues that are broken Migration for new release is a project Non-existent change management Non-existent testing of database migration Deployed application is missing database changes

Copyright 2007. Copyright is held by author.

Evolutionary Database Development I

• • • • •

Reality is “Business is Changing” Change management for database assets Functionality added in increments Facilitates automated testing Knowledge by developers of the functionality • Acknowledged importance of team interaction • DBA = Role != Person Copyright 2007. Copyright is held by author.

Configuration Management

• All Database assets under configuration management • Provide greater control over changes • Connects database and application

Copyright 2007. Copyright is held by author.

Make the DBA and Developers pair.

• Helps DBA understand the application • DBA has a better understanding of other areas of the business application • Will help you to write database tests • Data migration code for production data is critical • Gain knowledge of SQL Tuning etc • Make the team aware of production data • Understand performance implications • Better to have “Generalizing Specialists” Copyright 2007. Copyright is held by author.

Everyone should have a database!

Copyright 2007. Copyright is held by author.

Why?

• Database instance = application instance • Developers work independent of other developers • Liberty of experimenting with the database • Use ANT/Maven/Make/Rake to automate tasks • Database could just be a Schema • Reduces contention Copyright 2007. Copyright is held by author.

Nirvana...

Copyright 2007. Copyright is held by author.

Demo.

• Shows how new person joining project can startup effortlessly • Shows reduction in waste (Lean) • Shows how to share DBA tasks • Shows how the DBA will get to do more • Shows productivity gains by automation • Shows ANT targets to Create, Clean, Initialize and Drop databases Copyright 2007. Copyright is held by author.

Typical Environment

Developer Developer

Check in Check out

Source Control

Check out

Continuous Integration Uses

Uses

DB Server Server DB Integrate more than once a day. http://martinfowler.com/articles/continuousIntegration.html Copyright 2007. Copyright is held by author.

Continuous Integration Check in Application Code and Database Delta Scripts

Local dev environment

Central integration environment

DB delta Scripts

DB delta Scripts

ANT Maven Make

QA / UAT / PROD Database

Continuous Integration Engine

Source Control

QA / UAT / PROD Environment

Artifacts

Dev DB Check out all and build

CRUISE Database

Copyright 2007. Copyright is held by author.

Every project should have

• Version controlled schema creation scripts • Setup data/seed data creation script • Delta scripts that are maintained by Build/Day/Iteration/Release/Sequential etc • Way to create test data • Every alteration to the schema and setup data must be chronologically logged and sourcecontrolled with the application code • A fully-functional database instance can be built at any time for any build in the development timeline Copyright 2007. Copyright is held by author.

CI Demo

• Shows DBA/Developer collaboration • Allows to experiment with different design options • Shows DBA could be involved in the design of the functionality being developed. • Shows developers changing the model in their local schema. Copyright 2007. Copyright is held by author.

Evolutionary Database Development II

• A database refactoring is a simple change to a database schema that improves its design while retaining both its behavioral and informational semantics. • A database schema includes both structural aspects such as table and view definitions as well as functional aspects such as stored procedures and triggers. • Important: Database refactorings are a subset of schema transformations, but they do not add functionality. • www.databaserefactoring.com Copyright 2007. Copyright is held by author.

Database Refactoring is Hard

Other Applications You Know About

Other Applications You Don't Know About

Your Application

Persistence Frameworks

Your Database

Other Databases Data Extracts

Data Imports Data File

Data File Test Code

Copyright 2007. Copyright is held by author.

The change script (delta files)

• A stored collection of all changes(Schema and Data) made to the database. • Will likely be written in SQL, if feasible be automatically generated. • Should be source controlled • Is the tool by which instances will be rolled forward (backward if needed) • Should be named by Build/Release/Day/Sequential etc • Make every change such that rolling them back is possible, especially in production situations. Copyright 2007. Copyright is held by author.

Replace Column Demo

• Shows isolated change • Shows use of dbdeploy • Shows developer involvement • Shows migration and new development • Shows how application build links to database.

Copyright 2007. Copyright is held by author.

Continuous refactoring, controlled deployment

Copyright 2007. Copyright is held by author.

First time Deployment.

• Should be handled almost exactly the same as deploying code • Database creation script is used to create the production instance of the database. • Branch if necessary • The production instance will be either a clean copy of the application database schema or full of converted data • Demo Copyright 2007. Copyright is held by author.

Managing production

Handling the application in production is surprisingly simple. Because: • The database is being developed, tracked and sourced with the same rigor as code • The Change Scripts are your migration scripts, for new releases or patches.

Copyright 2007. Copyright is held by author.

Copyright 2007. Copyright is held by author.

46

8 44

7 42

6 40

5 38

4 36

3 34

2

Additional Releases

Additional Releases part II

• All of the change scripts should be applied following the last production release. In chronological order. (Change Logs are stored by Build/Release/Day/Sequential) • Since data migration scripts are part of change logs, data is also migrated • It is wise to test the migration on a test database Copyright 2007. Copyright is held by author.

Thinking forward

• Production database instances can be migrated to a future state of deployment. • New releases can be acceptance tested against the most recent data currently in production. • Migration is a snap, so why not deploy weekly or frequently.

Copyright 2007. Copyright is held by author.

Additional Resources

groups.yahoo.com/group/agileDatabases databaserefactoring.com martinfowler.com ambysoft.com/agileDatabaseTechniques.html dbdeploy.com Keep in touch databaserefactoring.com/PramodSadalage.html

Copyright 2007. Copyright is held by author.