Highly available web sites with Tomcat and Clustered JDBC - People

Highly available web sites with Tomcat and Clustered JDBC Emmanuel Cecchet

Motivations Database tier should be scalable highly available without modifying the client application database vendor independent on commodity hardware JDBC JDBC

JDBC

Internet

http://c-jdbc.objectweb.org/ - [email protected]

2 - 07/11/2003

Scaling the database tier – Alternative 1 (master-slave)

Cons failover time on master failure scalability Web frontend

App. server

Master

Internet


3 - 07/11/2003

Scaling the database tier – Alternative 2 (atomic broadcast)

Cons atomic broadcast scalability no client side load balancing heavy modifications of the database engine

Internet

Atomic broadcast


4 - 07/11/2003

Scaling the database tier – Alternative 3 (SMP)

Cons Cost Scalability limit Web frontend

App. server Database

Internet

Well-known database vendor here Well-known hardware + database vendors here http://c-jdbc.objectweb.org/ - [email protected]

5 - 07/11/2003

Scaling the database tier – Alternative 4 (shared disks)

Cons still expensive hardware availability Web frontend

App. server

Database

Disks

Internet

Another well-known database vendor http://c-jdbc.objectweb.org/ - [email protected]

6 - 07/11/2003

Outline

RAIDb C-JDBC Tomcat and C-JDBC Scalability High availability


7 - 07/11/2003

RAIDb - Definition

Redundant Array of Inexpensive Databases better performance and fault tolerance than a single database, at a low cost, by combining multiple database instances into an array of databases RAIDb levels offers various tradeoff of performance and fault tolerance


8 - 07/11/2003

Key ideas

RAIDb controller gives the view of a single database to the client balance the load on the database backends

RAIDb levels RAIDb-0: full partitioning RAIDb-1: full mirroring RAIDb-2: partial replication composition possible


9 - 07/11/2003

RAIDb levels

RAIDb-0 partitioning no duplication and no fault tolerance at least 2 nodes SQL requests RAIDb controller

table 1

table 2 & 3


table ...

table n-1

table n 10 - 07/11/2003

RAIDb levels

RAIDb-1 mirroring performance bounded by write broadcast at least 2 nodes SQL requests RAIDb controller

Full DB http://c-jdbc.objectweb.org/ - [email protected]

Full DB

Full DB

Full DB

Full DB 11 - 07/11/2003

RAIDb levels

RAIDb-2 partial replication at least 2 copies of each table at least 3 nodes SQL requests RAIDb controller

Full DB

table x


table y

table x & y

table z 12 - 07/11/2003

RAIDb levels composition SQL requests

RAIDb-1-0

RAIDb-1 controller

no limit to the composition deepness

RAIDb-0 controller

table w

table x & y

RAIDb-0 controller

table z

table w

table y

table x & z

RAIDb-0 controller

table w http://c-jdbc.objectweb.org/ - [email protected]

table x

table y

table z 13 - 07/11/2003

Outline



14 - 07/11/2003

C-JDBC overview

Middleware implementing RAIDb 100% Java implementation open source (LGPL)

Two components generic JDBC driver (C-JDBC driver) C-JDBC Controller

Read-one, Write all approach Supports heterogeneous databases


15 - 07/11/2003

architectural overview

Application server C-JDBC MySQL JDBC driver JVM


C-JDBC controller MySQL database JVM

16 - 07/11/2003

Inside the C-JDBC Controller Java client program (Servlet, EJB, ...)

C-JDBC driver

RMI Sockets

RMI Sockets

C-JDBC Controller XML configuration file Administration console

JMX

Configuration & administration

Virtual database

Virtual database

Authentication Manager


Request Manager

Request Manager

XML engine

Scheduler Recovery Log

Request Cache


Request Cache

Load balancer


Load balancer

Database Backend

Database Backend

Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

Oracle JDBC driver

Oracle JDBC driver

MySQL

MySQL

MySQL

Oracle

Oracle 17 - 07/11/2003

Virtual Database

gives the view of a single

Java client program (Servlet, EJB, ...)

C-JDBC driver

Virtual database Authentication Manager Request Manager Scheduler Recovery Log

Request Cache Load balancer

Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL

database establishes the mapping between the database name used by the application and the backend specific settings backends can be added and removed dynamically configured using an XML configuration file


18 - 07/11/2003


Matches real login/password


C-JDBC driver



Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL

used by the application with backend specific login/ password Administrator login to manage the virtual database


19 - 07/11/2003

Scheduler

Manages concurrency control Specific implementations for


C-JDBC driver



Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL

Single DB, RAIDb 0, 1 and 2 Query-level Optimistic and pessimistic transaction level


uses the database schema that is automatically fetched from backends

20 - 07/11/2003

Request cache caches results from SQL requests improved SQL statement analysis to


C-JDBC driver

limit cache invalidations

Virtual database

table based invalidations column based invalidations single-row SELECT optimization

Authentication Manager Request Manager Scheduler Recovery Log


Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL

request parsing possible in the


C-JDBC driver offload the controller parsing caching in the driver

21 - 07/11/2003

Load balancer 1/2 RAIDb-0


query directed to the backend having the

C-JDBC driver

needed tables Virtual database Authentication Manager

RAIDb-1 read executed by current thread write executed in parallel by a dedicated

Request Manager Scheduler Recovery Log

thread per backend result returned if one, majority or all commit if one node fails but others succeed, failing node is disabled


Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

RAIDb-2 same as RAIDb-1 except that writes are sent only to nodes owning the written table

MySQL

MySQL

MySQL


22 - 07/11/2003

Load balancer 2/2

Static load balancing policies


C-JDBC driver

Round-Robin (RR) Weighted Round-Robin (WRR)

Virtual database Authentication Manager Request Manager

Least Pending Requests First



Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL


(LPRF) request sent to the node that has the shortest pending request queue efficient if backends are homogeneous in terms of performance

23 - 07/11/2003

Connection Manager

Connection pooling for a


C-JDBC driver

backend Simple : no pooling RandomWait : blocking pool FailFast : non-blocking pool VariablePool : dynamic pool



Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL

Connection pools defined on a


per login basis resource management per login dedicated connections for admin 24 - 07/11/2003

Recovery Log

Checkpoints are associated


C-JDBC driver



Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL

with database dumps Record all updates and transaction markers since a checkpoint Used to resynchronize a database from a checkpoint JDBCRecoveryLog


store information in a database can be re-injected in a C-JDBC cluster for fault tolerance 25 - 07/11/2003

Functional overview Client application (Servlet, EJB, ...)

Client application (Servlet, EJB, ...)

connect login, password C-JDBC driver connect myDB Controller execute SELECT * FROM t C-JDBC Virtual database 1

C-JDBC driver

Authentication Manager Request Manager Scheduler Recovery Query result cache Log Load balancer


Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL 26 - 07/11/2003

Functional overview Client application (Servlet, EJB, ...)


C-JDBC driver

C-JDBC driver

execute INSERT INTO t …

C-JDBC Controller Virtual database 1 Authentication Manager Request Manager Scheduler Recovery Query result cache Log Load balancer


Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL 27 - 07/11/2003

Failures execute INSERT INTO t …



C-JDBC driver

C-JDBC driver

C-JDBC Controller

No 2 phasecommit parallel transactions failed nodes are automatically disabled


Virtual database 1 Authentication Manager Request Manager Scheduler Recovery Query result cache Log Load balancer Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL 28 - 07/11/2003

Outline



29 - 07/11/2003

Highly available web site Apache clustering L4 switch, RR-DNS, One-IP techniques, LVS, …

Tomcat clustering mod-jk, session replication

Database clustering C-JDBC

RR-DNS

mod-jk

Parsing cache Result cache Metadata cache

Internet


30 - 07/11/2003

Result cache

Cache contains a list of SQL_ResultSet Policy defined by queryPattern_Policy 3 policies EagerCaching: variable granularities for invalidations RelaxedCaching: invalidations based on timeout NoCaching: never cached RUBiS bidding mix with 450 clients Throughput (rq/min)

No cache

Coherent cache

Relaxed cache

3892

4184

4215

Avg response time

801 ms

284 ms

134 ms

Database CPU load

100%

85%

20%

C-JDBC CPU load

-

15%

7%


31 - 07/11/2003

Configuring Tomcat 5

Same setting for all Tomcat nodes Copy c-jdbc-driver.jar to $CATALINA_HOME/common/lib

Without connection pooling 1. load driver Class.forName("org.objectweb.cjdbc.driver.Driver") .newInstance();

2. get a connection Connection conn = DriverManager. getConnection("jdbc:cjdbc://host/myDB","user","password"); http://c-jdbc.objectweb.org/ - [email protected]

32 - 07/11/2003

Configuring Tomcat 5

With connection pooling factory org.apache.commons.dbcp.BasicDataSourceFactory driverClassName org.objectweb.cjdbc.driver.Driver url jdbc:cjdbc://localhost/myDB http://c-jdbc.objectweb.org/ - [email protected]

33 - 07/11/2003

Configuring C-JDBC Virtual database configuration file


36 - 07/11/2003

C-JDBC vertical scalability

allows nested RAIDb levels allows tree architecture for scalable write broadcast necessary with large number of backends C-JDBC driver re-injected in C-JDBC controller


37 - 07/11/2003

C-JDBC vertical scalability

RAIDb-1-1

Client program

with C-JDBC no limit to composition deepness

C-JDBC driver JVM

Client program

Client program

C-JDBC driver JVM

C-JDBC driver JVM

C-JDBC controller RAIDb-1 C-JDBC driver

C-JDBC controller RAIDb-1



MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL


MySQL

MySQL

MySQL

MySQL

MySQL 38 - 07/11/2003

Vertical scalability Addresses JVM scalability issues Distributing large number of connections on many backends

Client program

Client program

C-JDBC driver JVM

C-JDBC driver JVM

Client program C-JDBC driver JVM

C-JDBC controller Partial replication C-JDBC driver

C-JDBC controller Partial replication



DB native JDBC driver



DB 1 http://c-jdbc.objectweb.org/ - [email protected]

DB 2

DB 3

DB 4

DB 5

DB 6

DB7 39 - 07/11/2003

TPC-W benchmark (Amazon.com)

Throughput in requests per minute

Nearly linear speedups with the shopping mix 1600 1400 1200 1000

Single Database

800

Full replication

600

Partial replication

400 200 0 0

1

2

3

4

5

6

Number of nodes http://c-jdbc.objectweb.org/ - [email protected]

40 - 07/11/2003

Outline



41 - 07/11/2003

Controller replication Prevent the controller from being a single point of failure Group communication for controller synchronization C-JDBC driver supports multiple controllers with automatic failover

jdbc:c-jdbc://node1:25322,node2:12345/myDB Java client program

C-JDBC driver JVM

C-JDBC driver JVM

C-JDBC Controller MySQL PostgreSQL JDBC driver JDBC driver

MySQL


Java client program

PostgreSQL

JGroups

Java client program

Java client program

C-JDBC driver

C-JDBC driver

JVM

JVM

C-JDBC Controller PostgreSQL MySQL JDBC driver JDBC driver

PostgreSQL

MySQL

42 - 07/11/2003

Controller replication Client application (Servlet, EJB, ...)



C-JDBC driver

C-JDBC driver

C-JDBC driver

jdbc:cjdbc://node1,node2/myDB Total order reliable multicast

C-JDBC Controller

C-JDBC Controller

Distributed Request Manager

Distributed Request Manager

Virtual database 1

Virtual database 2



Request Manager

Request Manager

Scheduler

Scheduler

Query result cache

Query result cache

Load balancer

Load balancer

Database Backend

Database Backend

Database Backend

Database Backend

Connection Manager

Connection Manager

Connection Manager

Connection Manager

MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

Oracle JDBC driver

MySQL

MySQL

MySQL

Oracle


43 - 07/11/2003

Client program

Mixing horizontal & vertical scalability

C-JDBC driver JVM

Client program

Client program

C-JDBC driver JVM

C-JDBC driver

C-JDBC controller Full replication

C-JDBC controller Full replication DB native JDBC driver

JVM

C-JDBC driver

C-JDBC driver

C-JDBC controller Full replication DB 1

DB 2

DB native C-JDBC JDBC driver driver

C-JDBC controller Full replication DB native JDBC driver

DB 3


DB 4

DB 5

DB 6

DB 7

44 - 07/11/2003

Building initial checkpoint Octopus is an ETL tool Use Octopus to store a dump of the initial database state Currently done by the user using the database specific dump tool Octopus dump for initial checkpoint

EJB Container JOnAS, WebLogic, JBoss, WebSphere, ...

C-JDBC driver JVM

disabled

C-JDBC Controller Recovery Log PostgreSQL JDBC driver

PostgreSQL disabled http://c-jdbc.objectweb.org/ - [email protected]

45 - 07/11/2003

Logging Backend is enabled All database updates are logged (SQL statement, user, transaction, …)


C-JDBC driver JVM

JDBC Recovery Log

Octopus dump for initial checkpoint

enabled


PostgreSQL enabled http://c-jdbc.objectweb.org/ - [email protected]

46 - 07/11/2003

Adding new backends 1/3 Add new backends

EJB Container

while system online Restore dump corresponding to JDBC initial checkpoint Recovery Log with Octopus Octopus

JOnAS, WebLogic, JBoss, WebSphere, ...

C-JDBC driver JVM

enabled


dump for initial checkpoint

PostgreSQL PostgreSQL disabled enabled http://c-jdbc.objectweb.org/ - [email protected]

PostgreSQL disabled 47 - 07/11/2003

Adding new backends 2/3 EJB Container

Replay updates from the log


C-JDBC driver JVM

JDBC Recovery Log

enabled

C-JDBC Controller Recovery Log

Octopus

PostgreSQL JDBC driver



PostgreSQL enabled

PostgreSQL disabled 48 - 07/11/2003

Adding new backends 3/3 EJB Container

Enable backends


when done

C-JDBC driver JVM

JDBC Recovery Log

enabled


Octopus




PostgreSQL enabled

PostgreSQL enabled 49 - 07/11/2003

Making new checkpoints Disable one backend to have a coherent snapshot Mark the new checkpoint entry in the log Use Octopus to store JDBC the dump Recovery Log


C-JDBC driver JVM

enabled



...

Octopus


dump for dump last for last checkpoint checkpoint


PostgreSQL enabled


Making new checkpoints Replay missing updates from log


C-JDBC driver JVM

JDBC Recovery Log

enabled



...

Octopus


dump for dump last forfor last checkpoint checkpoint


PostgreSQL enabled


Making new checkpoints Re-enable backend


when done

C-JDBC driver JVM

JDBC Recovery Log

enabled



...

Octopus


dump for dump last for for last checkpoint checkpoint


PostgreSQL enabled


Handling failures A node fails! Automatically


disabled but should be fixed or changed JDBC by administrator Recovery Log

C-JDBC driver JVM

enabled



...

Octopus




PostgreSQL enabled


Recovery 1/3 Restore latest dump with Octopus


C-JDBC driver JVM

JDBC Recovery Log

enabled



...

Octopus


dump for dump last for last checkpoint checkpoint


PostgreSQL enabled


Recovery 2/3 Replay missing updates from log


C-JDBC driver JVM

JDBC Recovery Log

enabled



...

Octopus


dump for dump last forfor last checkpoint checkpoint


PostgreSQL enabled


Recovery 3/3 Re-enable backend


when done

C-JDBC driver JVM

JDBC Recovery Log

enabled



...

Octopus




PostgreSQL enabled


Demo

Administration tools HypersonicSQL in-memory databases JMX


57 - 07/11/2003

Conclusion

RAIDb C-JDBC open source middleware for database replication RDBMS independent

Tomcat & C-JDBC no application modification required performance scalability high availability transparent failover


58 - 07/11/2003

Q&A _________ Thanks to all users and contributors ...

http://c-jdbc.objectweb.org http://c-jdbc.objectweb.org/ - [email protected]

59 - 07/11/2003

Bonus slides

HORIZONTAL SCALABILITY

Horizontal scalability

JGroups for controller synchronization Groups messages for writes only Client 1

Client...

Client n

DB2

DB3


Client...

Controller1

Controller1

DB1

Client 1

DB4

DB1

DB2

Client n

Controller2

DB3

DB4

62 - 07/11/2003


Centralized write approach issues co

co

Con

D

D

Con

D

Con

D

D

D

co

Con

D

D

co

Con

D

D

?

D

D

? ?

?

Issues with transactions assigned to connections T1(Write)

T1(Read) Controller1

Controller2

Controller1

Controller2

Controller1

Controller2

DB1

DB2

DB1

DB2

DB1

DB2


T1(R)

T1(W)

T1(R) T2(W) 63 - 07/11/2003


General case for a write query 3 multicast + 2n unicast DistributedVirtualDatabase@controller2

DistributedVirtualDatabase@controller1

DistributedRequestManager

execWriteRequest(AbstractWriteRequest) return result

Distributed scheduling handler


Scheduler

Write query handler

Scheduler

Load Balancer


Send request to all controllers Wait for schedulers replies Send execute request order to all controllers

Distributed load balancer handler

Wait for load balancer replies Notify scheduler of completion

Notify controllers of completion success or not Return result


Load Balancer

Notify scheduler of completion

DistributedVirtualDatabase@controller3 DistributedRequestManager Distributed scheduling handler Distributed load balancer handler

Scheduler

Load Balancer

Notify scheduler of completion


64 - 07/11/2003


Solution: No backend sharing 1 multicast + n unicast [+ 1 multicast]

Controller1

DB1

DB2

Controller2

DB3

DB4

Controller3

DB5

DB6

DistributedVirtualDatabase@controller2 DistributedVirtualDatabase@controller1


execWriteRequest(AbstractWriteRequest) return result DistributedRequestManager


Scheduler


Load Balancer

Write query handler

Scheduler

Load Balancer

Send request to all controllers


Wait if all backends failed

DistributedVirtualDatabase@controller3

Distributed load balancer handler Wait for load balancer replies


Notify controller(s) that have failed if other have succeeded Return result

DistributedRequestManager Distributed scheduling handler Distributed load balancer handler

Scheduler

Load Balancer



65 - 07/11/2003


Issues with JGroups resources needed by a channel instability of throughput with UDP performance scalability

TCP better than UDP but unable to disable reliability on top of TCP unable to disable garbage collection ordering implementation is sub-optimal

Need for a new group communication layer optimized for cluster http://c-jdbc.objectweb.org/ - [email protected]

66 - 07/11/2003

Horizontal scalability JGroups performance on UDP/FastEthernet


67 - 07/11/2003

USE CASES

Budget High Availability High availability infrastructure “on a budget” Typical eCommerce setup

http://www.budget-ha.com


69 - 07/11/2003

OpenUSS: University Support System

eLearning High availability Portability

Apache

Apache

Apache

Enhydra Director

Enhydra Director

Enhydra Director

Linux, HP-UX, Windows InterBase, Firebird, PostgreSQL, HypersonicSQL

OpenUSS

OpenUSS

OpenUSS

OpenUSS

Enhydra server

Enhydra server

Enhydra server

Enhydra server

JOnAS J2EE server

JOnAS J2EE server

JOnAS J2EE server

JOnAS J2EE server

C-JDBC driver

C-JDBC driver

C-JDBC driver

C-JDBC driver

JVM

JVM

JVM

JVM

http://openuss.sourceforge.net

C-JDBC Controller Full replication


Firebird JDBC driver

Firebird JDBC driver

Firebird http://c-jdbc.objectweb.org/ - [email protected]

Firebird 70 - 07/11/2003

Flood alert system Disaster recovery Independent nodes synchronized with C-JDBC VPN for security issues http://floodalert.org

Flood alert system

Flood alert system

Flood alert system

JBoss J2EE server

JBoss J2EE server

JBoss J2EE server

C-JDBC driver

C-JDBC driver

C-JDBC driver



MySQL JDBC driver

MySQL JDBC driver

MySQL JDBC driver

MySQL

MySQL

MySQL


> 1000 miles


71 - 07/11/2003

J2EE benchmarking

Internet

Large scale J2EE clusters

emulated users

http://jmob.objectweb.org


72 - 07/11/2003

PERFORMANCE

TPC-W Browsing mix performance


74 - 07/11/2003

TPC-W Shopping mix performance


75 - 07/11/2003

TPC-W Ordering mix performance


76 - 07/11/2003