Scaling in the Cloud - O'Reilly Media

scaling  mysql  in  the  cloud:   a  success  story   by  

Connecting the World Through Games

dan  rogart  /  zynga   mohan  krishnan  /  zynga  

who are we? •  mohan krishnan: director of dba team at zynga –  [email protected]

•  dan rogart: sr. dba at zynga –  [email protected]

what is zynga? •  zynga makes social games •  Many of the top facebook games are from zynga: –  cityville –  farmville –  poker –  mafia wars

zynga •  we get a bit of traffic (from appdata.com):

what is the cloud? •  “A cloud is a pool of highly scalable, abstracted infrastructure that hosts your application, and is billed by consumption.” – James Staten of Forrester Research •  we use amazon ec2

agenda: •  configuration •  management •  performance challenges with a write heavy workload •  high availability •  sharding •  advantages of the cloud

node configuration •  m2.xlarge or m1.xlarge •  a compute unit = a 1.0-1.2 Ghz 2007 Opteron / Xeon processor type:  

RAM  (GB):   CPU  compute   units  [cores]:  

m1.xlarge  

15  

8  [4]  

m2.xlarge  

17  

6.5  [2]  

comments:  

oCen  mulD-­‐guest  tenant.     may  affect  available  NIC   bandwidth  

i/o subsystem •  datadir: –  4 EBS volumes –  software RAID 0 –  write throughput ~10 MB/s

•  backup mount: –  1 EBS volume –  write throughput ~3 MB/s

software •  CentOS •  MySQL 5.0.67 •  Xtrabackup

automated configuration •  we use RightScripts •  install ssh authorized keys •  install required packages (including mysql) •  pull standard my.cnf •  customize my.cnf for the instance (server-id, etc)

replication topology •  each shard has 1 master with 2 slaves

rightscale •  RS provides a central portal for managing instances in the cloud •  provides a nice gui for point and click management (good for working with a single instance) •  autoscaling arrays for web tier •  www.rightscale.com

rightscale api •  allows for scripting: –  spinning up large numbers of instances –  terminating large numbers of instances (be careful) –  mass renaming of servers –  pulling information about each server –  etc

rightscripts •  rightscripts are scripts you can add to a template to handle post-boot configuration tasks –  set up and configure your EBS volumes –  set up and configure user accounts (via puppet) –  set up and configure mysql –  basically shell scripts, so they are very flexible

monitoring •  time series performance data is gathered by collectd and graphed via rightscale •  you can aggregate graphs for a given metric across a large number of servers •  or you can drill down and look at details of a given server

aggregate example:

detail graph example: •  mysql threads

detail graph example: •  datadir disk throughput

monitoring / alerting •  nagios –  Standard mysql alerts –  our awesome SRE team gets alerts and escalates as required

•  splunk –  useful for aggregating and indexing logs (used more for web / memcache tier)

performance challenges •  EBS storage performance •  EBS variability •  i/o wait •  failure scenarios

EBS performance •  RAID 10 server with BBWC: –  latency is < 1 ms

•  in our 4x software RAID 0 stripe: –  nominal latency per volume is ~4-6 ms (for a write heavy workload) –  max write throughput is ~10 MB/s for random writes –  iops vary depending on the size per request, but for our workload the max is 800-1200 –  things will start to fall over before you hit max utilization

EBS variability •  EBS is a black box, but you are probably sharing resources with other EBS users •  performance can vary across many volumes according to time (overall degradation during busy times, for example) •  performance can vary on a per volume basis as well (one volume might have higher latencies while another is fine)

i/o wait •  nodes will have a lot of i/o wait due to EBS latency •  not a representation of a cpu bound workload •  can help identify nodes with nascent EBS issues •  we run irqbalance to try and even out the requests across multiple cores (not always successful)

cpu i/o wait on a 2 core box:

cpu i/o wait on a 4 core box:

failure scenarios •  EBS failure •  EBS degradation •  host unreachable (network) •  host down (hardware/power)

EBS volume failure •  volume stops responding: •  mysql will start to pile up: queries wait forever, connections hang, replication starts lagging •  iostat output:

EBS volume failure •  cpu io wait during a failure:

EBS volume failure •  can manifest as innodb page corruption when an underlying error corrupts a page •  can be detected by filesystem (which then puts the filesystem into read-only mode for recovery)

EBS degradation •  EBS performance can vary widely from nominal performance without failing completely •  you need to quantify the effect degradation has on the application and replace if required •  latency might climb as high as 20ms – multiple seconds on a volume •  sometimes it can be ignored

host unreachable •  sometimes a network issue •  sometimes an issue with underlying hardware •  sometimes a power failure •  network/power issues will sometimes self correct •  a down master will be failed over right away, but you can often wait when a slave dies

high availability

•  scripted failover •  scripted node replacement •  xtrabackup

scripted failover

scripted failover •  Allows failover to a slave very quickly •  Allows SRE team to perform work without DBA intervention •  Not totally automated: a human must decide to initiate the process

scripted node replacement •  rightscale makes it really easy to provision replacement nodes •  scripts make populating new nodes with data and setting up replication very easy •  you can have a new slave up in less than an hour (often much faster)

new slaves from backup •  takes most recent backup from source •  restores it on the new slave •  sets up replication as required

new slave from direct file copy •  shut down mysql, rsync over data files •  bring up mysql •  set up replication as required •  you start with less replication lag, but during the process you may not have any active slaves

xtrabackup •  taken at staggered times on ‘b’ and ‘c’ slaves •  EBS can cause a mysql crash sometimes when a backup is taken on a busy slave •  so: stop sql_thread, let io_thread continue to run during backups •  writes backups to a separate EBS volume •  we take snapshots of the EBS volume as well

rolling restore •  in many cases we copy backups to a “rolling restore” server every day and restore them •  keep 12 days of restored data •  uses mysqld_multi •  can have up to 3 days active on each rr box

rolling restore •  verifies the integrity of the backup set •  allows customer service / admin tools to get a historical view of user state •  allows for selective recovery of user data without direct DBA intervention

scalability by sharding •  build new shards as slaves replicating from current masters •  expansion: –  push new config –  break replication from existing shards to new shards

sharding

sharding

data cleanup •  after sharding in this manner ½ of your data will be “dead” on each shard •  needs to be cleaned up •  it shares pages with live data, fragmenting the stuff you really want •  so you are doing more i/o on a given request

when to shard •  slaves will start to have trouble staying in sync •  i/o subsystem is fully loaded (getting close to ~10 MB/ sec) •  usually these happen at the same time •  slaves will be your bottleneck first

when to shard slave  at  ~40%  capacity:  

slave  approaching  8M/s  –  Dme  to  shard:  

advantages of the cloud •  provisioning •  availability zones •  flexible storage •  security zones

on demand provisioning •  allows DBAs to provision servers •  allows DBAs to terminate servers (very carefully!) •  greatly reduces the time to deliver new servers •  pay only for what you use

availability zones •  each zone is discrete (independent power, networking, etc) •  we put masters / slaves in different zones •  allows for some extra redundancy when, say, a backhoe knocks out power to a zone

flexible storage •  EBS allows for a lot of flexibility •  variable volume sizes, any number of volumes •  snapshots are very helpful •  can be dynamically created and attached as required

security zones •  security zones allow you to easily isolate pools of servers or applications from each other •  very useful and easy to manage

concluding remarks •  you can run a large mysql installation in ec2 •  we’re hiring! (email [email protected] or [email protected] •  questions?

connecting the world through games