Based on your hypotheses, what data do you need to look at? What metrics will help you prove or disprove the possible causes? By isolating different pieces of data that may be related to or causing the problem, you’ll be able to more easily spot trends or anomalies (step 2).

For example, you might segment the number of signups by country, channel, and web session duration to test your hypotheses (solving the marketing problem mentioned earlier).

If you’ve been in your role for more than a couple months, you likely understand what ‘normal’ metrics look like for your team. Based on that knowledge and using common sense, what do you notice? Is there any aspect of the data that appears abnormal? Perhaps you see trial signups have a clear 20% drop off in Australia.

If you haven’t yet established a baseline for ‘normal,’ use historical data as a starting point. For example, you might compare the signups this month to the signups in the same month last year. Or perhaps look at the trend of signups over the last 12 months for context.

Assess the impact of an anomaly or trend (spotted in step 2). This is a sanity check to see if the trend/anomaly you’ve spotted is significant enough to explain the underlying problem. The technical term for this is ‘practical significance.’

Continuing with our example of trial signups, some basic math helps us see that Australia only makes up for 5% of all signups. Since we’re trying to understand why there’s been a 10% overall drop in signups, it becomes clear that this dip in Australian signups isn’t the main cause.

Think of these steps as a fluid process - eyeballing your data may lead you to additional insights (step 1) and sometimes the cause might be so obvious that a sanity check (step 3) isn’t necessary. Other times, assessing your data (step 1) might lead to a new hypothesis.

You may have heard the question, “Are the results statistically significant?”

Statistical significance (or statistical significance testing) is a technical term for determining if the anomaly you notice is due to a sampling error or is, in fact, a consistent finding across all the data. (If you want to learn how to test your hypothesis for statistical significance, read more here and here.)

The meaning of ‘significance’ here can be confusing. In statistics, significance means the result (or anomaly) is accurate and verifiable (instead of a chance occurrence). Unlike the commonly known definition of significance, this term doesn’t imply ‘importance’ in the scientific world. So it’s possible to have statistically significant findings (accurate findings) that make no real difference in solving your problem - like the 20% drop in signups in Australia.

We often worry about whether our sample size is large enough to provide reliable results when we find statistical significance, but we also need to take into consideration whether these differences are meaningful in a real way.

Solving everyday business problems means we’re looking for anomalies or trends in our data that not only are statistically significant, but also practically significant.

In other words, we need to figure out what is making (and will make) a real impact on our signups, ticket response time, cart abandonment rate, and activation rate. To better understand this, let’s walk through an analysis of our four opening example problems.

Our ability to do great things with data will make a real difference in every aspect of our lives.

I think of data science as more like a practice than a job. Think of the scientific method, where you have to have a problem statement, generate a hypothesis, collect data, analyze data and then communicate the results and take action…. If you use the scientific method as a way to approach data-intensive projects, you’re more apt to be successful with your outcome.

The head of customer support, Jamie, has noticed response times for support tickets have increased and are too long. She needs to know what’s causing it and begins by jotting down the problem and brainstorming several hypotheses. Core problem: Our customer support ticket response time is too long. How can we reduce response time? Hypotheses: Ticket response time has increased because of… an influx of service-related tickets which have longer response times compared to product-related tickets.

challenges specific to one call center.

insufficient customer support team members which results in a backlog of tickets.

a recently launched product feature. After going through the initial data analysis steps, Jamie ended up with a list of potential causes (the hypotheses listed above). Now it’s time to analyze them. First, she explores whether there has been a recent shift in the service tickets compared to product tickets. Did it happen at the same time? Do those tickets take longer? Does the magnitude of that impact match the core problem (i.e. could this realistically be the cause)? Here’s what she found in the data related to the type of support ticket. Product-related tickets have a response time of 5.5 hours

Service-related tickets have a response time of 5.8 hours

The ratio of product-related tickets to service-related tickets has remained the same over the last month at roughly 60% and 40% of tickets respectively. The longer response times for service-related tickets are statistically significant (accurate difference) compared to the response time for product-related tickets because of the extremely large sample size (5,000 total tickets). However, this doesn’t practically impact our problem because the absolute difference of response time is so small (0.3 hours). Jamie decides to explore a speculative possible cause. Perhaps the increased response time is related to a specific geographic region. The company has 4 call centers, each in different time zones (Pacific, Mountain, Central and Eastern). After the data was split into the four time zones, she found the average response times for each was: Pacific: 5.1 hours

Mountain: 4.6 hours

Central: 15.5 hours

Eastern: 4.2 hours. By eyeballing the data, Jamie notices the Central time zone has a response time around 3x longer than the other time zones. This seems like it might be the primary cause and chats with the manager at that call center. She discovers the team is struggling to roll out their new support software because of a technical snag in connecting their phones to the software. This caused a backlog of support tickets and significantly increased response time. With this new insight, Jamie can focus on resolving the underlying challenge and help her team get back on track.

Liam, the marketing manager for a SaaS company sees that signup numbers have dipped. The CEO and VP of product have also noticed and are keen to understand the cause. Liam starts his data analysis with the following overview. Core problem: The number of trial signups for our product decreased this week. What is causing this dip? Hypotheses: Signups have decreased because of… public holidays in some regions.

recent changes to the marketing site (or site outages).

a website outage Monday morning causing an error during the signup process.

a decrease in conversion rate has decreased signups.

organic search rankings (for our product page) that dropped to the second page of search results. First, Liam is eager to know - how much does the data vary? And is this definitely out of the ordinary? Eyeballing a chart of signups over time confirms it is unusual and needs to be addressed. He begins looking at the data by day or hour. Was this a dip that has since recovered or are daily signups still lower? If it were an outage that’s been resolved, he would expect to see a dip in signups at the same time which has since recovered. However, he rules this out as the dip started before the outage and hasn’t recovered. Some minor styling changes were made to the signup form, but again the timings don’t quite line up. He also double checks the conversion rate for this page and it’s actually gone up slightly. No dice. He does notice the number of people arriving on the sign up page has decreased by around 10% and the timing seems to correlate with the overall dip. Based on the data he’s reviewed so far, he assumes the problem is upstream. Next, Liam considers the holiday hypothesis. It was a public holiday in some regions last week. Breaking down the signups by country over time, he sees the dip isn’t for just one country - it’s across the board. So he rules out this hypothesis too. At this point, he’s a bit lost. He decides to investigate whether some pay-per-click (PPC) landing page changes could be to blame. Ah, one of the campaigns is down 50%. Could that be it? No, since it only accounts for 1% of signups. Finally, he looks at signups by channel. Organic search (which accounts for 70% of signups) is down 20%. This looks interesting. Could it be a change in page ranking kicking in from on-page changes made a few weeks ago? Time to check the SEO tool. Bingo. The main keyword has dropped ranking and is now on the second page. This looks very suspicious. A sanity check of the numbers shows this more than accounts for the drop off. In addition to having an explanation for the drop in signups, Liam can create a strategy for recovering signups to present to the CEO and VP of Product.

Suppose an ecommerce manager, Simone, needs to figure out why the average cart abandonment rate for her ecommerce business is increasing. She dives into the relevant data with several hypotheses. Core problem: More potential customers are abandoning their online shopping carts. How can we decrease the average abandonment rate? Hypotheses: The average cart abandonment rate has increased because of… an increase in the absolute number of people beginning a cart (placing items in the cart).

a recent change to part of the checkout process.

seasonality (i.e. holidays, school breaks, etc.).

an end to a promotion which results in more people abandoning their carts.

a specific product. First, Simone checks to see if the absolute numbers have changed. Has the rate gone up because more people are starting a cart? Or is the absolute number about the same? If there’s been a surge of people who begin adding items to the cart but the number of people completing a purchase has remained the same, maybe something about the additional shoppers makes them less likely to convert. Simone notices the number of people beginning a cart has gone up slightly. She then asks around to see what might have changed. Have there been any promotions? Any new products launched? Could there be a seasonal impact? Has anything changed to the checkout process? Have prices been adjusted? (Note: this could vary greatly depending on the business and the range of products.) Simone learns there’s been a small change in the checkout flow. Instead of just listing the items in the cart, they now show a picture of each item. She thinks this is unlikely to be the issue, but the change does coincide with the time the rate of cart abandonment increased. To better understand the impact of this change, she splits the checkout process into different steps. Looking at the checkout process, the drop off isn’t on the page that’s been affected. In fact, more people are making it to the next step so this seems unlikely to be the culprit. Next up, Simone looks for any seasonal impacts by comparing this week’s abandonment rate to the same week in previous years. She also takes a quick look at the calendar for any possible clues. Since other metrics like sessions and email open rates weren’t affected, she concludes the cause wasn’t seasonality. Then she remembers a promotion recently ended. Could that be the cause? Perhaps when people realize the promotion is over, they’re more likely to abandon their cart. Simone looks at the proportion of checkouts using the promotion code before the increase in cart abandonment. It was only 5%. The change in abandonment rate is triple that, so this might be a contributing factor, but it’s still unlikely to be the primary culprit. Could it be changes to the inventory? Simone breaks down abandonment by product, but performance is fairly consistent across all products. While thinking of other possible causes, she reviews the checkout process once more. The big drop off is on the page that first shows the shipping price. Simone recalls making some cosmetic changes to the product pages and realizes timeline aligns perfectly with the increased abandonment rate. Going back over the changes she notices the shipping details are much less prominent than they used to be. Her new hypothesis is potential customers are abandoning their carts because they’re frustrated by the price. Their expectations are set at a lower price based on the product pages. As soon as they see the full price (including shipping), they’re more likely to drop off. At this point, Simone has gone as far as she can with her data. It’s time to test a change and track the results. She could run an A/B test, revert the product change to the previous design, or perhaps try a version with the shipping included. Simone now feels confident in her analysis and can take action to solve her cart abandonment problem.

Allie, the product manager, is responsible for understanding why the activation rate for her mobile productivity app has decreased. Core problem: After the initial download, the rate of people tracking their tasks in the app has decreased. How can we increase the activation rate? Hypotheses: The activation rate has decreased because… a change in the product makes people less likely to activate.

a new (and different) group of people have begun trying the product. She notices the proportion of downloaders who activate (open and begin using the app) have been steadily decreasing for the past 3 months (compared to the total number of downloads). Allie decides to get more context by looking at the absolute numbers before generating lots of possible causes. She discovers the absolute number of downloads has gone up significantly while the number of people activating has only increased slightly. She’s a little bit relieved that both absolute numbers are increasing. (Note: this may or may not be a problem depending on your acquisition channels. Ultimately it depends on if money is being wasted on the additional signups.) Allie proceeds with two broad hypotheses. A change in the product makes people less likely to activate. A new (and different) group of people have begun trying the product. This might explain the additional individuals downloading who are less likely to activate. Allie quickly determines there weren’t any changes to the initial experience in the app that correlate with the drop in activation. Now she looks more closely at who is downloading the app and if the demographics have changed. She splits downloaders by region over time. There’s a small increase in downloads from lower activating regions, but some quick math proves this is nowhere near enough to explain difference in activation rate. Next, Allie splits the downloads by channel (e.g. app store search, social ads, referrals, etc.) and notices that downloads where referrals were the channel have significantly increased. It seems roughly the same number as increased downloads she noted earlier. Digging deeper, she splits the download-to-activation rate by channel. This channel has a lower conversion rate. Allie chats with the marketing team and discovers the app was featured in a well known, high traffic article. And it’s not costing anything! What untapped potential might be here? Is there an opportunity to boost the activation rate from this source? What else could we be doing? Allie and the marketing manager are now able to have a data-informed conversation about what action to take next. Thanks to some quick data analysis, what began as a problem has become an opportunity. It's your turn... As you analyze your own data, remember to consider both the truthfulness of the difference (statistical significance) as well as the practical meaning of that difference (practical significance).

With more data driving operations in a business than ever before, leaders need to cultivate a culture that is data-driven, instead of believing in their gut instincts.

When we’re in the middle of our analysis, the data can sometimes play tricks on us. The truth is, even the most experienced statisticians have to watch out for these tricks - called data fallacies. The following tips will help you avoid some of the most common data fallacies. A common fallacy is assuming a dataset is trustworthy - until it’s discovered later in analysis that it’s not. Flip that. Make sure your data is trustworthy before you begin analysis. Keep your data in context to avoid cherry picking. In other words, ignore your personal bias and motives when analyzing your data. Cherry picking is selecting only the bits of data that support your claim while discarding the parts that don’t. For example, we might notice the support tickets for a new feature have increased in response time. If we stop there, we might conclude that the problem is the new product feature. But if we look at all the support tickets over the last two months, we may see an overall increased response time because the number of tickets have increased. It is absolutely essential that one should be neutral and not fall in love with the hypothesis. Start with a hypothesis to avoid data dredging. When looking for the cause of a problem, it might be tempting to dig through the data until a pattern emerges. At first glance, the pattern might be statistically significant, but further testing (e.g. checking if the trend continues, looking at related metrics, etc.) could reveal the pattern as a false positive. Data dredging is the failure to acknowledge that the correlation was in fact the result of chance. The way to avoid this fallacy is to begin with a hypothesis before analyzing your data, check related metrics, and/or test to see if the trend continues. For example, we might see that the activation rate for our mobile app has decreased since adding new pictures and a more detailed description to the app store. Before assuming the two are correlated, we can consider other related metrics or factors - like winter holidays. It just so happened that we updated the text and images right before the holidays - a slower period overall for the business - creating the illusion of correlation. If you torture the data long enough, it will confess to anything. Distinguish correlation from causation to avoid false causality. It’s easy to assume that because two events happen at the same time (correlate), one must cause the other. Not so fast! It’s always best to gather more data and look at possible third party causes. Sometimes patterns that seem correlated may be correlated to a third independent factor and not to each other. For example, we see that potential customers who abandon their online shopping cart tend to have a low total cart value (total cost of items in their cart when abandoned). At this point, we don’t have enough data to know if that’s a consistent correlation, result of chance, or perhaps caused by another factor. Digging deeper we may realize that the cost of shipping is causing higher cart abandonment rates since free shipping is only available for orders that exceed a certain minimum cart value. Shallow men believe in luck or in circumstance. Strong men believe in cause and effect.