However you do it, the point is to use an in vitro test to get validation that an in vivo test is worth the additional resources. Tools like Unbounce, Optimizely, and paid advertising make these kinds of in vitro tests easy and fast. And since you aren’t dealing with existing users who were acquired through different channels, had varying CACs, and have different lifetimes, the control is far higher.

One of my favorite mentors, Anantha Katragadda, told me when I was starting out in growth that if I couldn’t think of a way to fake a product experience in order to get validating data, I simply wasn’t trying hard enough.

Curate Your Own Journal of Failed Projects

One of the major challenges in science is that there is no journal of failed projects because only successful projects are published.

Let’s say a highly influential paper comes out with an obvious next step. Over the course of the next 10 years, 50 different scientists may say “Aha! I can’t believe nobody has published that no-brainer follow-up study yet. I will devote my own time and resources to it.” Unfortunately there may have been a flaw in the initial paper or some other reason that the no-brainer follow-up just won’t work. Fifty labs have now wasted valuable resources - an unfortunate result for the scientific community.

The best research labs approach this problem by seeking out negative results. They spend as much time asking for and sharing failed results as they do successful ones.

I’ve been in the unfortunate position where my team ran a test that another team in the organization had already run unsuccessfully. Ever since, I’ve made sure that every major project is shared even if the results aren’t favorable.

Sharing and collecting negative results isn’t fun, so it’s only going to happen if you:

Bake it into your process Support open sharing of failed experiments within your company culture

If you don’t do both of these things, failed projects will be swept under the rug by whomever is working on them, only to resurface later as wasted time and resources when an unsuspecting team member runs a similar experiment.

Here’s how I keep track of failed projects with the help of my growth team at Feastly:

Build a project pipeline that requires analysis and sharing before any project can move to “complete.”

Whenever we meet with outside colleagues, we ask for one thing that’s worked recently, as well as one thing that hasn’t.

We build acceptance of and learning from failed experiments into our culture. Growth teams are optimized for learning, so finding negative results isn’t a bad thing because it facilitates learning. We celebrate the learnings that result from failed experiments constantly. For us, an automated celebration message is sent to Slack anytime a project finds positive results OR negative ones, and we’re diligent about measuring individual performance independent of the ratio between positive and negative results.

Control and Blind Your Experiments

Growth teams optimize for speed, which often makes running clean projects harder, while the culture of “move fast and break things” is often used to justify taking liberties on controlled design, proper stats, or both.

Given how projects become intertwined, a misinterpreted or poorly designed project can unravel months of work or set teams up to chase losing opportunities. Uncontrolled results can become building blocks for resource intensive follow-ups, with the flawed data only becoming apparent as the follow-ups miss their mark.

To publish a paper, every result must disprove the null hypothesis. This means that you prove, with statistical significance, that the result you are seeing is not due to random chance or biased observation. Both of these premises are easy to miss, so here’s are a few straightforward tips to help you design sound experiments.

Understand How to Properly Control Your Experiments

Placebo controls are not just for medical research.

Let’s say you launch an ad or email campaign retargeting users who purchased and later churned with copy around “missing out on what everyone is talking about.”

Your hypothesis is that FOMO is an effective strategy to increase resurrection rates.

Having a control group that doesn’t get the ads or emails is not enough to validate your hypothesis, even if the results are outstanding. You need a “placebo” group that controls for both independent variables so that your groups get the same frequency of ads or emails, but with different copy, to isolate the impact of the copy.

For every experiment, make sure you control for every independent variable in your hypothesis.

Understand How to Properly Blind Your Experiments

Any project that relies on qualitative observation should be blinded.

Let’s say you’re watching user sessions to compare how high LTV users interact with a feature compared to low LTV users. At the end of the session, you give the user a grade from 0-10, 0 indicating they didn’t understand how to use the feature and 10 indicating they easily understood it.

Your hypothesis is that low LTV users don’t understand how to use the feature.

If you view 10 sessions from high LTV users, and then view 10 from low LTV users, your results will be invalid. Since you’re coming in with a hypothesis, you’re far more likely to perceive things that support your ideas.

The correct way to conduct this test would be to have a colleague assign a number to all 20 videos and then send them to you. You record the results and then send them back. They then “unblind” the data by noting which videos were of high LTV users and which weren’t.

When you’re designing every experiment, verify that human bias won’t skew the observation of the data.

Be Diligent About Statistics

This should go without saying, but in growth at early stage companies I estimate that only 20% of results are passed through the appropriate statistical tests. I’ve found that people often think that stats are used by nerds who are too cautious and don’t want to move quickly, but the opposite is true.

Stats tell you how much risk you’re taking on by accepting the results as true, which is a powerful data point if you’re moving at hyperspeed and iterating rapidly.

If you want to move more quickly you can simply increase the minimum P value on your tests.

Medical journals require a P value of .05, which means that the results should accurately disprove the null hypothesis 95% of the time. If you’re more risk tolerant and want to move faster on a particular project shoot for 85 or 90%. You’ll be wrong more often but it will take far less data to reach a decision.

Conclusion

As a field, growth is still in its infancy. While it’s advancing the approaches we take to grow companies, we still have quite a bit of refinement to do on our growth processes. Because in the science world the scientific method it has been adapted, evolved and optimized to produce better outcomes, we need to leverage what the science community has already learned. No one wants to reinvent the wheel! The same principles that have worked for science can be applied to running rapid iterative growth to produce more successful experiments on a faster timeline.

Must-Know Updates from Tech’s Growth Leaders

Get our weekly 5-min digest