Home
Add Document
Sign In
Create An Account
Innovate with Experimentation
Recommend Documents
No documents
Innovate with Experimentation
Download PDF
3 downloads
325 Views
4MB Size
Report
Comment
John Provine. Product Manager. Optimizely. Formerly: Google. Matt Auerbach. Dev Evangelist. Optimizely. Formerly: AWS, F
Innovate with Experimentation Incorporate A/B Testing Into Your Product Development
John Provine Product Manager Optimizely
Matt Auerbach Dev Evangelist Optimizely
Formerly: Google
Formerly: AWS, Facebook
Agenda ● ● ● ● ●
Why experimentation matters How to build an experimentation platform Key challenges and learnings Demo Q&A
A B
A
B
9,800 Google search experiments in 2016
www.google.com/search/howsearchworks/
1,653 Google search launches in 2016
www.google.com/search/howsearchworks/
“I haven’t failed.
I’ve just found 10,000 ways that won’t work.” Thomas Edison
medium.com/netflix-techblog
uber.com/blog/colorado
Experimentation-Driven Product Development
Bad Product Development (high risk) Build
Launch
Better Product Development (minimize risk) Build
Rollout
Cleanup
Experiment-Driven Product Development (maximize impact and minimize risk) Build
Experiment
Analyze
Rollout
Cleanup
Fail as soon as possible
How to build an experimentation platform that is fast and scales
from planout.experiment import SimpleExperiment from planout.ops.random import *
class FirstExperiment(SimpleExperiment): def assign(self, params, userid): params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'], unit=userid) params.button_text = WeightedChoice( choices=['Join now!', 'Sign up.'], weights=[0.3, 0.7], unit=userid)
my_exp = FirstExperiment(userid=12) # parameters may be accessed via the . operator print my_exp.get('button_text'), my_exp.get('button_color')
medium.com/netflix-techblog
medium.com/netflix-techblog
Analytics
medium.com/netflix-techblog
medium.com/netflix-techblog
eng.uber.com/data-viz-intel/
3 Key Ingredients for Robust Experimentation
Traffic Splitting
Remote Configuration
Analytics
Traffic Sampling Audience Targeting Mutual Exclusion State Persistence Multivariate Treatment Global Holdout QA Tool Automation Whitelisting Logging
Permissions Feature Toggles Feature Rollouts Kill Switch Scheduling Staging Environments Audience Definitions Webhooks REST API Audit Trail
Event Dispatching Event Storage Sessionization Statistical Methods Metric Definitions User Aliasing Segmentation DW Integration Notifications Visualizations
3 Key Challenges with Experimentation 1. Trust “How can I trust the experiment was set up and tracked correctly?” 2.
Process “What are the best instrumentation practices to minimize technical debt?”
3.
Maintenance “The engineer who built this left the company… now what?”
Client-side A/B testing for websites
Deep experimentation anywhere in the stack
2010
2017
Demo
Q&A
×
Report "Innovate with Experimentation"
Your name
Email
Reason
-Select Reason-
Pornographic
Defamatory
Illegal/Unlawful
Spam
Other Terms Of Service Violation
File a copyright complaint
Description
×
Sign In
Email
Password
Remember me
Forgot password?
Sign In
Our partners will collect data and use cookies for ad personalization and measurement.
Learn how we and our ad partner Google, collect and use data
.
Agree & close