Building your Company's - Optimizely

Building your Company’s

Data DNA

How to identify and use the right metrics to delight your customers, engage your audience, and optimize your business.

Building Your Company’s Data DNA

2

Table of Contents: Section 1: Introducing Data DNA Section 2: Finding your Company Metrics Understanding and Measuring Goals Metrics by Business Type Section 3: How to Take Action on your Metric Quantitative Data (with contributions from KISSmetrics) Qualitative Data (with contributions from Qualaroo) Section 4: Building Data into the Optimization Flywheel Six Steps of Optimization

Craft your Experiment Hypothesis



Assembling your Data DNA

Section 1: Introduction

Building Your Company’s Data DNA

4

There’s no question: data is here to stay. The promise of data, analytics, and web-based tools make the task of building a business online accessible and affordable. Access has created a new challenge: volume. Leveraging the entirety of a business’ data on an ongoing basis is tremendously valuable, but requires equally tremendous resources, beyond the reach or scope of many businesses’ technology investments.

Access to Data Increases Expectations In a recent survey, 78% of all marketers report feeling pressure to become more data-driven.* The challenge is to choose

Given these challenges, how can you make intelligent

which measurements of online perfor-

decisions, informed by data, to grow faster?

mance are essential to a data-driven

You can incorporate data to inform decisions you

strategy. *Teradata Data-Driven Marketing Survey 2013

make about growing your presence online. At each juncture, you have choices to make—how should I describe my product to my customers? How can I collect more email addresses to grow my audience? How should I roll out a new feature and collect feedback?

“The best data-driven companies don’t just passively store and analyze data, they actively generate actionable data by running experiments. The secret to getting value from data is testing, and if you’re looking to grow your online business, implementing well-executed, consistent A/B testing is a necessity.” —Wyatt Jenkins, Shutterstock*

A clear understanding of metrics and the right type of data at every level of an organization are essential for focus. Data is also key to uncovering customer insights and running a more effective optimization strategy, but should be properly aligned from the top-down in order to achieve success. *”A/B Testing and the Benefits of an Experimentation Culture,” Harvard Business Review.

Building Your Company’s Data DNA

5

How do I know if this guide is for me? In this guide, we’ll outline an approach to transforming data into action at every level of your organization.

You’ll benefit from this guide if: • You’re already capturing and reporting on data regularly internally • You have a team or company culture that is open to experimentation • You’re looking for methods to turn your company data into action • You’re looking to get more meaningful ‘wins’ from your optimization strategy

Here’s what you’ll learn:

Being Data-Driven is Tied to Performance Companies that rate themselves substantially ahead of their peers in their use of data are three times more likely to rate themselves as substantially ahead in financial performance, according to findings from the Economist Intelligence

• Why becoming data-driven as an organization should start with your optimization programs • How to align your organization around a failproof guiding light metric

Unit.* *Tableau Software & Economist Intelligence Report: “Fostering a Data-Driven Culture.”

• How to prioritize your optimization strategy against your shared company goals • How to move past random, ad-hoc A/B tests to an optimization process that produces wins (and if not winning tests, valuable insights derived from a hypothesis)

“Low hanging fruit tastes great, but the rest is worth the work, too.” —Patrick McKenzie, Kalzemeus Software

Building Your Company’s Data DNA

In this guide, we’ll cover many different types of data. The important point to remember is that the ideal of being ‘Data-Driven’ is actually to be ‘Data-Informed.’ Data points are an important set of inputs, but they are not a replacement for

6

What is A/B Testing? A/B testing is a simple way to test changes to your page against the current design and determine which ones produce positive results. It is a method to

human intuition and judgement. This guide will

validate that any new design or change to

provide a list of inputs that are strong positive

an element on your web pages or mo-

indicators for how to improve your company goals and optimization program. What data points

bile apps is improving your conversion rate. It is the assumption that you have never reached the ‘best’ version of your

cannot provide, however, is input on what your

website, mobile app, or product, and that

business metrics should be, or what initiatives you

the best method of learning about your

want to prioritize.

customers is through iterative testing and experimentation.

What is Optimization? Optimization begins with broad, sweeping changes and tests that encompass all of your website traffic, but can become more granular, by running A/B tests to understand how changes affect different types of visitors, and then adapting site content to eventually deliver a more personalized experience for each individual.

Section 2:

Finding your Company Metrics

Building Your Company’s Data DNA

You’ve probably wondered at some point: “What’s the fastest way for me to create business impact?” We could frame this question with two others: 1. “What’s the fastest way to make a measurable business impact?”

8

Many analytics experts have discussed the value of identifying a single metric, which should be carefully chosen and then rigorously tested. We recommend Lean Analytics by Ben Yoskovitz and Alistair Croll.

2. “How do I define success?” To reach the goal of becoming a data-driven organization or a data-informed individual, we recommend striving for one metric that defines the goals of your business. When you choose your ‘guiding light’ metric, ask these questions to determine whether it will stand the test of time, and generate strong optimization results:

Use these questions to test and strengthen your metric statement.

• Is it quantifiable? Your metric should be measurable and understandable at-a-glance.

Before you take any actions to improve

• Is it visible and clearly communicated? Your metric should serve as a reference point in conversations.

stand scrutiny and testing over time.

• Is it understandable? It should require very little explanation. • Is it comparative? You should be able to compare the metric to benchmarks, like your competitors’ performance or other points in time. • Is it a rate or ratio? Showing a rate of change is more illustrative than an absolute number. • Does it change your behavior? If the metric you choose doesn’t enact change, you need a different one.* • Is it attributed to revenue? For many people in organizations, this measurement is what communicates above all else. If you have difficulty attributing your efforts to value for the business, you may struggle proving ROI.

*Questions 3-6 reference Ben Yoskovitz and Alistair Croll, Author of Lean Analytics

your metric, ensure that you have tested it internally to ensure that it will with-

Building Your Company’s Data DNA

“What gets measured, gets managed.” —Peter Drucker, Business Management Thought Leader

Having a singular goal is helpful not only for organizational focus, but it will be invaluable for more specific tasks, like your optimization efforts. In order to run more intentional, ROI-positive A/B tests and experiments, you need a guiding metric that you can brainstorm around and prioritize against. We’ll cover the data inputs that can inform these types of tests in Section 3.

9

Building Your Company’s Data DNA

10

Understanding and Measuring Your Goals Use this framework as a starting point for determining what your priority should be for your business, and by association, your online marketing and other customer touchpoints. What parts of your website directly serve the metric you are working to improve? Think of these areas as prime candidates for optimization.

do you do on a day-to- 1. What day basis to grow your business?

metrics do you currently 4. What report on on a regular basis?

does this activity help you 2. What accomplish?

5.

If you were to pick one activity and corresponding goal to work on for the next 1-3 months, what would it be?

6.

Which of the metrics from Question #4 help you to understand the progress you are making on the answer to Question #5?

7.

Which of the metrics from Question #4 could you live without?

is the priority for each of 3. What these activities? Essential Important Nice-to-have

Building Your Company’s Data DNA

11

Metrics by Business Type If you:

Sell products or services

(E-Commerce, Platforms)

Publish content

(Media, user-generated content like Reddit and Wikipedia)

Develop a web-based tool

(SaaS or Mobile app company)

You might use your website to:

And could be measured as:

Collect orders for your product

Number of orders over time Average order value Repeat customers Conversion Rate

Distribute and engage your content

Time on site Articles viewed Votes or comments generated Social traffic

Capture, engage, and retain users

Signed up users Paying customers Up-sold customers Churned customers

Good candidates for top metric statements:

Not as strong metric statements:

These statements reflect strong metric statements sup-

These metrics are not supported by a strong rationale, and

ported by a clear rationale. Strive to consolidate and

are more likely to be vanity metrics. They are not enough (on

strengthen your overarching metrics so that everyone can

their own) to justify prioritizing for optimization.

understand and tailor their priorities to match them.

“I measure conversion rates from free trials to customers because my most important metric is acquiring new customers.”

“I measure pageviews to product pages on my website.”

“I measure average donation value, because my most important metric is dollars raised during our fundraising campaigns.”

“I measure social shares.”

“I measure conversion rate on my homepage, because that is where the majority of my new leads are acquired.”

“I measure the amount of traffic to my pricing page.”

Section 3: How to take Action to Improve your Metric

Building Your Company’s Data DNA

13

You may be wondering: how does eliminating metrics help me use data? The answer lies in the touchpoints you have with your customers. Once you have identified your overarching goal, you will be in a position to identify touchpoints that can be improved in service of the corresponding metric. In order to take action that will improve the outcome of your metric, it is important to effectively identify high-value areas of your website, product, or other customer touchpoint. We will explore the best ways to gather information about these high-value areas later in this section. Each business metric can be broken down to apply to different levels and functions within your organization. A team should plan to adopt a secondary metric that supports your top-level company goal. These levels may resemble the diagram to the right: At the most granular level, your website and product goals can be supported by running experiments in an attempt to improve the conversions that increase performance, which in turn affect your highest-level goals.

Hypothesi ze Tests to Increase AOV

Building Your Company’s Data DNA

14

How do you measure the success of these facets of your online business? Your efforts to present your

What is a conversion? A desired action

business online and capture interest, engagement, or

taking place on your at a customer

revenue require conversions.

touchpoint—a click, or registration, or product action, for instance.

Take stock of the conversions that support your

Your conversion rate is the current pace

top-priority metric. These areas are prime candidates

at which website visitors take action at a

for optimization, where you’ll run experiments and A/B tests in an effort to improve conversions.

given touchpoint. Conversion rate optimization and A/B testing describe the practice of experi-

Getting started with optimization is not the most difficult step to success; the challenge will present itself in the struggle to keep continuously A/B testing to find improvements, uncovering winning tests consistently. In the following section, we will explore how to use a combined qualitative and quantitative approach to build your Data DNA into your website and app performance. Then, we’ll discuss how to funnel these data points into experiments that uncover improvements and build your data-driven approach to optimization in Section 4.

Using Quantitative Data to Determine which Areas of your Site to Test Your website analytics are an indispensable tool for determining the strengths and weaknesses of your website. Leverage your website data in order to effectively prioritize and brainstorm the best experiments to run on your website.

menting with elements to try to improve a conversion rate.

Mapping Your Company’s Data DNA

15

KISSmetrics Contributed Section:

Using Funnels to Find Big Wins and Accelerate Growth Before you can find ways to accelerate your acquisition and growth, you’ll need to start tracking your core funnel.

What’s your Core Funnel? It’s slightly different for each business but it measures how many people move through the major steps of your acquisition. Don’t worry about tracking every little click, page, or action that people take to become a customer. Focus on the core steps. Let’s look at a few examples. SaaS (subscription as a service) businesses drive traffic, convert that traffic into free trials, get those free trials to use their product, and convince people to purchase an ongoing subscription to the software. So a SaaS funnel is built from these steps: SaaS Funnel

1. Visited site 2. Signed up for free trial 3. Used product 4. Purchased subscription

Mapping Your Company’s Data DNA

16

An ecommerce funnel looks pretty similar: E-Commerce Funnel

1. Visited site 2. Visited product detail page 3. Added product to cart 4. Entered checkout 5. Finished checkout

For whatever business model you have, break your acquisition down into 4-5 core steps that everyone goes through.

Tracking your Funnel Ideally, you’ll have a customer analytics tool that can track people as they move through the different steps of your funnel. When you get to the point where you’re doing a lot of optimizations on your funnel, have plenty of traffic, and acquire customers consistently, start looking at customer analytics tools that will help you track the entire funnel. Even if you’re not there yet, you still want to get a general sense for how your funnel performs. Google Analytics and some internal tracking will get you pretty far.

Mapping Your Company’s Data DNA

17

Ecommerce companies have it easy by setting up ecommerce tracking in Google Analytics. This tells you how many people purchase and their conversion rates. You’ll also want to configure custom events to see how many times products get added to a cart or visitors enter your checkout. For SaaS, set up Google Analytics and trigger a goal when people start a trial. You’ll also want to build out some internal tracking that tells you how many new accounts you acquired, how many of them started using the product, and how many of them purchased. Add this data to your internal customer database and query it every week or so to see how you’re doing. Don’t worry about having perfect tracking when you get started. Your main goal is to have a general idea for how your funnel performs.

Optimizing your Funnel As soon as someone’s ready to start optimizing a funnel, they usually pick a random element on a random step. It might be a button color on a call to action, a snippet of copy, an image, or a layout that just feels “off”. Then they’ll launch an A/B test if they have enough traffic or maybe they’ll just launch the change and hope for the best. After a few rounds of this, you’ll find that your conversions are exactly the same as where they are now. They won’t budge an inch. Why? Because it’s incredibly difficult to find tests that really make a difference. Most online marketing “best practices” give 2%-3% wins. Not only is it incredibly hard to detect small wins like this (you need a ton of data to test on), small wins aren’t useful to a business. They only matter if you can find a lot of them in a short time period. Which, again, requires large amounts of data.

Mapping Your Company’s Data DNA

18

So how do we find bigger wins consistently? Start with qualitative data on the step of your funnel that you struggle the most with. Qualitative data is feedback from users or customers that isn’t a metric. Surveys, customer interviews, feedback forms, usability studies, and heatmaps are all qualitative data. It’s common to have one step that is a major roadblock to acquisition. Focus on collecting feedback from your customers at that step of the process. Ask them why they don’t want to purchase, why they didn’t use the trial, or why they didn’t finish the checkout. Asking for targeted feedback on an underperforming step of your funnel will give you plenty of ideas on what to test. There’s never a 100% guarantee of finding a big win but you’ll definitely find them more frequently than if you test random elements across your funnels.

Mapping Your Company’s Data DNA

By examining your website analytics, you will be able to pinpoint valuable areas of your site that are in need of improvement. If you are successful in running A/B tests and experiments to improve these key areas of your website, you’ll be one or more steps closer to moving that ‘guiding light’ metric, and shifting your company towards a more data-driven approach to solving problems. Website analytics have a tremendous number of benefits for structuring your optimization strategy: • Quantifiable, high-value traffic segments • Common goals that sync across your website analytics to your website optimization platform • Ability to scale and analyze website data across segments, cohorts, and more Quantifiable analysis of your website is only one part of the data equation. To complete the picture of how your visitors are converting and engaging (and more importantly, why they aren’t), you’ll need to supplement your quantitative data with qualitative insights to enhance your data DNA.

Using Qualitative Insights to Determine which Areas of your Site to Test Qualitative, anecdotal feedback is an effective way to incorporate data into your business at any stage. This data is essential because it comes directly from your visitors and users.

19

Mapping Your Company’s Data DNA

This information is essential to determining why certain events occur on your website. When quantitative data provides insight into what happens (or isn’t happening), these qualitative data points can illustrate why. Why did that customer buy? What brought them to your website in the first place? What do your visitors expect from your product or offering?

“A solid test hypothesis is an informed solution to a real problem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the better it will be.” —Michael Aargaard, Conversion Rate Optimization Expert

Qualitative insights come in many forms, including, but not limited to: • User testing: For a fee, a person unfamiliar with your website will complete a task on your website while providing their thoughts and feedback. There is a bias inherent in this qualitative data, since the person knows they are being tested. • On-page surveys: Use these surveys to collect ‘in the moment’ feedback from visitors to a website. The more questions you ask, however, the lower your completion rate is likely to be. • Heat mapping: These tools will show which areas of a given web page attract the most ‘heat,’ or attention, from clicks and scrolling on your site. • Long-form surveys: Often delivered via email, these surveys can be used to collect in-depth feedback on your company’s positioning, perception from customers, and some of their most valued offerings or products from your business.

20

Mapping Your Company’s Data DNA

• Feedback from cancellations, returns and complaints: Take your customers’ constructive feedback, and decide what visible changes to your website could help to alleviate pain points and frustrations they might be having. • Live chat feedback: Uncovering pain points and visitor intent will be key to generating winning experiment ideas. • Customer feedback: Anecdotal feedback from engagements with customers on calls, at events, and in other formats. You can leverage these tools (many of which have affordable and easy-to-use plans for businesses) to collect feedback directly from the visitors to your website or customers. You must hone in on motivating factors, however, in order to collect meaningful feedback.

21

Building Your Company’s Data DNA

22

AAA: Ask your Audience Anything

To ask questions of your website visitors and collect more data points about what you should optimize, practice asking the right questions. Here are two types of questions you should ask:

1. Intent - What did your website visitors come to your website looking to find, or do? 2. Frustration - Could you find what you were looking for? What was broken?

A few tips to get even better data from your qualitative research: Be polite: You’re human - so are your website users. Ask the questions in an approachable, visitor-friendly manner. Phrase questions with a “What do you

Consider the types of questions you could ask to understand key parts of your website funnel:

think of your website? Kindly tell us:”

On your homepage:

Ask for honesty: Your website visitors

• What did you come to our website today hoping to accomplish?

approach.

are also nice people. It’s sometimes difficult to provide constructive feedback. “We’d appreciate your brutally honest

Before a key conversion:

• Do you have any additional questions about this product or service? Determine what information your visitors are looking for before they purchase. • Is there anything stopping you from completing this order? Perhaps the visitor needs more information to make their decision. Maybe an unexpected bug with your checkout flow is preventing them from converting.

After a key conversion:

• What made you purchase/sign up/donate to us today?

On your content site:

• What topics would you like to see us cover more? You could make this multiple choice or leave as a free response • Would you like to see more recommended content on this topic? Determine the best way to design recommended content modules for your content.

feedback as we work on improving our website experience for you.” Keep questions open: Try to avoid asking questions like, “What was frustrating about your experience today?” or, “What do you think of our [product or service]?” These questions predicate a certain type of response, and will bias your visitors’ responses. Ask visitors to self-select: You can also learn more about your audience by asking them about their interests, their business type, or products and services that they like to use. This will help you to segment your web traffic and experiment with running more targeted tests.

Building Your Company’s Data DNA

23

Now, it’s your turn: What are the steps (or clicks) in your website funnel, and what questions would you ask at each step to reduce friction and improve your visitors’ experience?

?

?

?

?

Mapping Your Company’s Data DNA

24

Online Success and Optimization:

Why Quantitative Data isn’t Enough Interview with Startup Marketer and Qualaroo CEO Sean Ellis

Q: A:

What do you think is the most common misconception about Conversion Rate Optimization and A/B testing? The number one misconception is that A/B testing is simply about running a test here and there and hoping for improved results. Without an optimization process that focuses on continual improvement A/B testing often fails to live up to it’s promise. Without organizational rigor to make A/B testing a priority, companies give up failing to see early wins, which ultimately costs them the long-term gains that come from a systematic approach to optimization.

Q: A:

Do you think that most companies online are effectively communicating with their customers? Why or why not? Most companies lack a process for regularly collecting and then taking action on user feedback. Users provide feedback in numerous ways, from bouncing off of web pages, to taking surveys, leaving reviews, filling out customer support tickets and posting on social media. With all of this feedback coming in, you’d expect companies to be constantly processing it and using it to improve the visitor experience, and ultimately, their business. But more often, feedback is triaged to manage customer complaints rather than used for true learning and

Mapping Your Company’s Data DNA

25

business improvement. The most successful companies have a process for collecting, parsing and using customer feedback to improve their business.

Q: A:

What is the best way to get actionable, qualitative data from a website survey? The single best way to get actionable qualitative data from a website survey is to use them to try to understand specific user behavior. For example, using a website survey on pages that have high bounce rates, or pages within your conversion funnel have high drop-off rates, can give you insights right from the customer that help you understand why they’re leaving. When you understand why a behavior is happening, it’s much easier to take action on the feedback and try to change the behavior.

Q: A:

Do you have any tips on how to choose which feedback should be incorporated into your testing pipeline? Feedback from qualified people is the most important. The people that are potential customers who aren’t converting are the ones you want to focus on. Ignore the people who aren’t qualified or interested in what you’re offering from the beginning. For example asking people who converted a question such as “What almost stopped you from signing up?” or “What made you decide to sign up?” helps you understand the needs of qualified visitors. This qualified feedback will help you sort through the data from exit surveys that include both qualified and unqualified responses.

Q:

What would you tell someone who is looking for help creating strong hypotheses for their tests?

Mapping Your Company’s Data DNA

A:

26

There are two ways to create a hypothesis. In the first case you can look at the data and then spend hours or days with your team trying to interpret what the data means and what you should test next. You can create hypotheses from your interpretation and from your team’s opinions. Or you can ask visitors to that page what the actual problem that they’re encountering is. No need to interpret data, just ask visitors and get immediate feedback. I believe asking visitors what issues they’re encountering on your site, conducting user research, is the best way to formulate strong hypotheses that make for valuable tests.

Q: A:

How do you think companies should approach staying focused when it comes to optimization? How do you measure the progress of your CRO program? Having the organizational rigor to stay focused on conversion optimization is the hardest part of this process. A/B testing is not a one-off project—rather it is a continuous process of improvement that needs to be in motion at all times. Econsultancy reports that 87% of companies doing A/B testing run between 1 and 5 tests each month. The best companies run many times that number. Companies need to commit to A/B testing as a core part of their digital marketing program and invest in it accordingly. Stick to the process and eventually it will become an addictive habit, with the organization constantly trying to outdo its previous test. Systematic, ongoing A/B testing is going from a competitive advantage to a competitive necessity. It’s a key requirement for online success today.

Building Your Company’s Data DNA

How Clearlink Built Qualitative Insights into their Data DNA Content and conversion services provider, Clearlink, uses survey software* to foster its data-driven approach to engaging with its audience and collecting qualitative feedback from its online audience.

While many times quantitative metrics might be interesting, they cannot tell the entire story when it comes to the customer’s path. In short, we know the ‘what’, but not the ‘why.’ For these issues we have to turn to qualitative research and customer insights to make sense of the customer needs and dive in deeper. This is how we innovate; this is how we iterate; this is how we make sense of the larger picture. —Rachel Johnson, Consumer Insights Research Manager, Clearlink

At Clearlink, the team asked the question: “How can we see inside the minds of our customers and be sure we are offering what they need?” Despite following testing best practices, the team has frequently encountered unusual or contradictory A/B test results. Utilizing qualitative data gives them insights into these results and helps them better understand and explain when met with this challenge. They hypothesized that using a qualitative insights tool in conjunction with their optimization software would enable them to better serve their customer needs through well-researched and well-executed tests, resulting in the ability to integrate and better understand tests’ success and failure, and ultimately leading the to understand our market better.

*Clearlink implemented Qualaroo survey software for the purpose of these experiments.

27

Building Your Company’s Data DNA

28

By using on-page surveys to offer time-sensitive promotions, the team was better able to serve more relevant promotions to their website visitors in a more time-effective manner. This led to overall higher conversion rates for their sites: • Stand-Alone vs. Bundle Pricing: The team learned from survey results that stand-alone pricing was what Clearlink’s customer was looking for. After testing stand-alone pricing messaging against bundled pricing options, stand-alone pricing won with a conversion rate improvement of +5.4%, contributing to a testing win of $41,500 in annualized revenue impact. • Lead Capture: The team hypothesized that by using the survey tool, they would be able to send the customer directly to their conversion funnel (the sales floor) by engaging them more quickly on the initial landing page. In addition to the phone number, they added promotional messaging (similar to display ads) to the messages to further drive conversion. Using the survey tool bypassed the normal lead time required for a custom overlay to be designed, developed, and pushed to live site, significantly increasing Marketing’s agility in responding to changing customer tastes, preferences, and demands. Using quantitative and qualitative research meth-

Clearlink’s mantra is: “The more often

ods together led to several “wow” moments for the

you fail, the more often you’ll win; so

Clearlink team. By combining the two data streams, qualitative and quantitative, they created an enhanced testing cycle, making the most of testing wins, successes, and failures. Qualitative findings sped up the testing cycle, and the team gained impactful insights more often. Clearlink’s prevailing sentiment is that “The more often you fail, the more often you’ll win; so speed up the frequency at which you fail.”

speed up the frequency at which you fail.”

Building Your Company’s Data DNA

Integrating quantitative and qualitative data into the optimization cycle creates a more effective testing program for attaining website goals. Although taking the time to add consumer research and the customer’s point of view can seem like the longer path, for teams like Clearlink it ultimately improved the team’s feedback loops and led to insights that previously might have gone undiscovered. At this point, you have used data to inform your strategy from your top-level business goals down to your website and product-level goals. Now that you have a sense of what could stand to be improved, let’s turn to brainstorming and prioritizing your new experiment ideas to complete your Data DNA sequence.

29

Section 4: Build Data into the Optimization Flywheel

Building Your Company’s Data DNA

You’re well on your way to running high-impact, ROI-positive experiments to improve your customer touchpoints. The next step is to distill the data you’ve collected and begin to take action. In this process, you will funnel your quantitative and qualitative data points into concepts for A/B tests that can be run as experiments on your website to improve key conversions and support your business goals. We call this six-step framework the “Optimization Flywheel.”

In order to see success with this approach, you should: • B ● uild a rigorous practice of consistently collecting quantitative and qualitative data about your website funnels. • Maintain visibility of your organization’s overarching metric. If your focus as a company changes, your website goals and individual experiment objectives should change accordingly. • Maintain a sense of experimentation as broadly as possible across your company. Any visible changes to your website help to understand what works best for your customers and visitors. The pursuit of this understanding should support the goals of the company as well, as discussed in Section 2.

31

Building a Data-Driven Optimization Strategy

What is the Optimization Flywheel? Optimization is a continuous cycle. In order to ensure that you are responsive to the intent and frustrations experienced by your audience, you can leverage optimization to run experiments and discover improvements to your website.

There are six steps of the Optimization Flywheel:

32

Building Your Company’s Data DNA

Step 1: Define Goals As defined in Section 2, it is important that you begin your optimization process clearly aligns to your business goals. A clear understanding of the metrics you are optimizing for will help with prioritization and enables continual iteration and learning from experiments.

Step 2: Determine Optimization Points Identify a step in your funnel that is a prime candidate for optimization. It might be your homepage call to action (CTA), your campaign landing pages, your checkout flow, or your recommended content. Make sure to choose an area for optimization that has a direct correlation to your business goals. The quantitative funnel analysis in Section 3 will provide context for which areas of your website can be improved. Will optimizing this step of your website experience create a measurable change to the metric you identified in Section 2?

Step 3: Hypothesize Improvements A strong hypothesis about how your experiment will perform is core to running a winning A/B test. Your qualitative data collected in Section 3 will provide clear indicators of what should be tested to better match your visitors’ intent and solve their frustrations with your product or website. We recommend you take the time to collaborate on hypotheses with *Reference “Craft your own Experiment Hypothesis” for more detailed instructions on hypothesis creation.

33

Building Your Company’s Data DNA

your team, and properly document them along with the data that you used to inform them. A hypothesis is the following type of statement: [I make this change to my website] If ________________________ , [the following outcome] then _________________ will occur.*

Hypothesize what types of changes to your website would produce a positive change in conversions.

Step 4: Create Variations When setting up an A/B or multivariate test, you must develop a variation of your website feature that you would like to test against the current version. Use your hypothesis and your understanding of your analytics and qualitative data to isolate a variable that can be changed. The fewer the variables, the more straightforward your test will be. Changing multiple variables at once is possible with multivariate testing, but complicates the test and confounds the hypothesis, making it difficult to simply prove or disprove.

“It’s about coming up with your hypothesis first and then developing a test from there. Rather than saying, ‘I want a test button created,’ or ‘I want to try two different layouts,’ you create a hypothesis first, for example, ‘I think green buttons attract more attention and therefore will have a higher clickthrough rate.” —Michael Burk, Senior Online Product Manager, Electronic Arts

34

Building Your Company’s Data DNA

Once you have isolated your variable, create variations of your website based on the modification of that variable.

Step 5: Run Tests Prepare an experiment using your A/B testing software of choice. For more detailed instructions on how to set up and run an experiment using Optimizely, visit our Knowledge Base.

Step 6: Measure the Impact Allow your experiment to run until it has reached a statistically significant result. Did your variation win, lose, or draw even with the current variation? If the test is a win, congratulations! Your hypothesis was correct, and you can continue to build upon that test by applying the data-informed learning to other areas of your site, or testing another more advanced hypothesis. At this point, make sure to extrapolate the value of your improvement to the website across all of your traffic over an extended period of time. What will your lift translate to in sign-ups, orders, et cetera over the course of a year? If your test was a draw or your variation lost, investigate why that might be. It could be the case that your hypothesis needed additional research, or that you didn’t account for a behavior or event that skewed your test results. In the event of a losing test or a

35

Building Your Company’s Data DNA

draw, it is still possible to gain additional insights that you had not anticipated through further analysis and discussion with your testing team. At this point, discuss what the data couldn’t account for as you planned the experiment. What would you do differently in your next hypothesis and experiment?

Putting it all Together Here’s an example of Data DNA in action: • Take an e-commerce company that wants to improve the value of each customer they acquire. This is the top-level metric that they have identified is an excellent indicator of the success of their business. • To improve this metric, the team uses website analytics to identify the top traffic pages for a high-value customer segment: returning traffic. They identify conversion points for returning traffic that differ from other traffic types. • To collect qualitative data, they survey their customers to understand why they may not be converting at this point. • Based on this data, the team hypothesizes a change that could help to improve the conversion rate for these visitors. They run an experiment and to determine whether their hypothesis was correct.

36

Mapping Your Company’s Data DNA

37

Craft your own Experiment Hypothesis What does a winning test hypothesis look like? It’s a representation of your Data DNA on a hyper-focused level. Your qualitative and quantitative data collection will help you formulate a strong, testable prediction.

“IF _____________ , THEN ___________ DUE TO ____________.” [Variable]

The Variable: A website element that can be modified, added, or taken away to produce a desired outcome.

[Rationale]

[Result]

Result: The predicted outcome. (More email sign-ups, clicks on a call to action, or another type of behavior.)

Rationale: Demonstrate that you have informed your hypothesis with research: what do you know about your visitors from your qualitative and quantitative research that indicates your hypothesis is correct?

Hypotheses are statements, not open-ended questions. They address a question with a proposed solution. Crafting a hypothesis to address an open question or problem on your website enforces a well-rationalized, thoughtful proposal for how to address that problem. To take your hypothesis even further, consider what you would learn if your prediction was proven correct or incorrect in an experiment. What would you learn in each scenario?

Building Your Company’s Data DNA

Assembling your Data DNA Now, it’s time for you to take action: • Assess your top metrics with your team and company at large. • Take stock of your web analytics, and bring together your sources of qualitative data from your customers. • Most importantly, take action by applying an experimental mentality to your online presence with an intentional, measurable approach to continuously optimizing interactions with your audience. By incorporating data into your metrics and goals at every level of your business, you’ll move your organization towards a data-driven process for making decisions and instrumenting change. More importantly, your data will become actionable at the customer touchpoint level, where you’ll better engage and convert your audience into valuable customers, repeat visitors, and users.

38

Building Your Company’s Data DNA

39

To learn more about best practices for people, process, and technology for a winning optimization strategy, download a copy of our Roadmap to Building a Data-Driven Optimization Team. Optimization can be instrumental in improving the performance of your paid search funnels. Learn how to maximize your marketing ROI by reading Optimizing your Conversion Engine: Search Engine Marketing.

ABOUT THIS GUIDE Building your Company’s Data DNA Written By: Shana Rusonis Content Marketing Specialist, Optimizely @srusonis Designed By: Jon Saquing Communication Designer, Optimizely @JSaq

ABOUT OPTIMIZELY Optimizely is the world’s leading optimization platform, providing A/B testing, multivariate testing, and personalization for websites and iOS applications. The platform’s ease of use empowers organizations

Thank you to: Ural Cebeci, KISSmetrics, Sean Ellis and Morgan Brown.

SOURCES

• •

“Teradata Data-Driven Marketing Survey,” Teradata. “Fostering a Data-Driven Culture,” Tableau Software & Economist Intelligence Report.

• • •

to conceive of and run experiments that help them make better datadriven decisions. With targeting and segmentation using powerful real-time data, Optimizely meets the diverse needs of any business looking

“A/B Testing and the Benefits of an Experimentation

to deliver unique experiences to their

Culture,” Harvard Business Review.

visitors.

Lean Analytics by Alistair Croll and Benjamin Yoskovitz. For more detail from Hiten Shah, Co-Founder of KISSmetrics: “Increase your Testing Success by Combining Quantitative and Qualitative Data,” OptiCon 2014.

To learn more about Optimizely, schedule a live demo today at

OPTIMIZELY.COM/DEMO

San Francisco Office 631 Howard Street, Suite 100 San Francisco, CA 94105 Amsterdam Office Nes 76 1012 KE Amsterdam

The Netherlands