Sam's Video. Splash page experiment. Variations: ... Sam's Video. Splash page
experiment. Variations: ... (HTML + CSS + jQuery). Python. Google App Engine.
Dan Siroker Co-‐Founder & CEO
What is A/B & MVT tes0ng?
Media
Button
Splash page experiment Variations: Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video 5. Springfield Video 6. Sam’s Video
Button: “Sign Up”
Button: “Learn More”
Button: “Join Us Now”
Button: “Sign Up Now”
Splash page experiment Variations: Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video 5. Springfield Video 6. Sam’s Video
Media: “Get Involved”
Media: “Family”
Media: “Change”
Media: “Barack’s Video”
Media: “Springfield Video”
Media: “Sam’s Video”
Splash page experiment Variations: Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video 5. Springfield Video 6. Sam’s Video
Splash page experiment Variations: Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video 5. Springfield Video 6. Sam’s Video
Splash page experiment Variations: Button:
Media:
1. Sign Up
1. Get Involved Image
2. Learn More
2. Family Image
3. Join Us Now
3. Change Image
4. Sign Up Now
4. Barack’s Video 5. Springfield Video 6. Sam’s Video
Splash page experiment results
Splash page experiment results
Splash page experiment results
Splash page experiment results Email Subscriptions
Volunteers
Amount Raised
Original:
7,120,000
712,000
$143,000,000
+40.6%
+2,880,000
+288,000
+$57,000,000
New:
10,000,000
1,000,000
$200,000,000
Splash page experiment results Email Subscriptions
Volunteers
Amount Raised
Original:
7,120,000
712,000
$143,000,000
+40.6%
+2,880,000
+288,000
+$57,000,000
New:
10,000,000
1,000,000
$200,000,000
Splash page experiment results Email Subscriptions
Volunteers
Amount Raised
Original:
7,120,000
712,000
$143,000,000
+40.6%
+2,880,000
+288,000
+$57,000,000
New:
10,000,000
1,000,000
$200,000,000
Demo!
Old Architecture Editor
Implementa0on
Tracking
Results
edit.op(mizely.com
cdn.op(mizely.com
op(mizely.appspot.com
www.op(mizely.com
Frontend (HTML + CSS + jQuery)
Edgecast CDN
Pixel Tracking
Frontend
Increment pre-‐ computed counters
Fetch pre-‐computed counters
Python
Amazon S3 Code Genera0on
Google App Engine (Datastore, Task Queues, Memcache)
New Architecture Editor
Implementa0on
Tracking
Results
edit.op(mizely.com
cdn.op(mizely.com
log.op(mizely.com
www.op(mizely.com
Frontend (HTML + CSS + jQuery)
Edgecast CDN
Pixel Tracking
Frontend
Python
Code Genera0on
Amazon S3
Google App Engine (Datastore, Task Queues, Memcache)
New Architecture Editor
Implementa0on
Tracking
Results
edit.op(mizely.com
cdn.op(mizely.com
log.op(mizely.com
www.op(mizely.com
Frontend (HTML + CSS + jQuery)
Edgecast CDN
Pixel Tracking
Frontend
Python
Code Genera0on
Amazon S3
Google App Engine (Datastore, Task Queues, Memcache)
Splunk & Amazon EC2 to the rescue!
New Architecture Tracking
Results
log.op(mizely.com
www.op(mizely.com
Pixel Tracking
Frontend
New Architecture Tracking
Results
log.op(mizely.com
www.op(mizely.com
Pixel Tracking
Frontend
Amazon ELB EC2 Log Instance 1.Nginx/uWSGI/Python app writes to one EBS volume (200 GB, XFS) 2.Splunk reads from volume and forwards to dedicated index server EC2 Log Instance
...
New Architecture Tracking
Results
log.op(mizely.com
www.op(mizely.com
Pixel Tracking
Frontend
Amazon ELB EC2 Log Instance 1.Nginx/uWSGI/Python app writes to one EBS volume (200 GB, XFS) 2.Splunk reads from volume and forwards to dedicated index server EC2 Log Instance
...
EC2 Index Instance 4 EBS volumes (each 200 GB for 800 GB total) in RAID 0 with XFS
EC2 Index Instance
...
New Architecture Tracking
Results
log.op(mizely.com
www.op(mizely.com
Pixel Tracking
Frontend
Amazon ELB EC2 Log Instance 1.Nginx/uWSGI/Python app writes to one EBS volume (200 GB, XFS) 2.Splunk reads from volume and forwards to dedicated index server EC2 Log Instance
...
EC2 Index Instance 4 EBS volumes (each 200 GB for 800 GB total) in RAID 0 with XFS
EC2 Index Instance
...
EC2 Search Instance Searches across all index servers and aggregates
...
New Architecture Tracking
Results
log.op(mizely.com
www.op(mizely.com
Pixel Tracking
Frontend
Amazon ELB EC2 Log Instance 1.Nginx/uWSGI/Python app writes to one EBS volume (200 GB, XFS) 2.Splunk reads from volume and forwards to dedicated index server EC2 Log Instance
...
EC2 Index Instance 4 EBS volumes (each 200 GB for 800 GB total) in RAID 0 with XFS
EC2 Index Instance
...
Query Genera0on
Results Caching
EC2 Search Instance Searches across all index servers and aggregates
...
Why Splunk? 1. Flexible query language for post-‐hoc analysis 2. Horizontally scalable 3. Everything is controllable via the API 4. Runs on our own instances on AWS 5. Great documenta(on & knowledge base
Best Prac0ces & Recommenda0ons 1. Use Cloudkick for monitoring 2. Use iTerm profiles to make logging into mulAple instances easier 3. Put anything user-‐facing behind an ElasAc Load Balancer 4. Create an image (EBS AMI) for each instance type to make cloning & scaling easy 5. Use a code repository (e.g. GitHub) to store and track changes in configuraAons
Email me:
Follow me:
[email protected]
@dsiroker
Appendex #1: Amazon EC2 Specs High-‐CPU Extra Large Instance 7 GB of memory 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each) 1690 GB of instance storage 64-‐bit placorm I/O Performance: High API name: c1.xlarge
