Modern tech companies have figured out that data is their product. Whether you sell a service, a product, or content, what you really do is create value for your customer base — and every interaction with your product is a measurable amount of value. But the best data-driven companies don’t just passively store and analyze data, they actively generate actionable data by running experiments. The secret to getting value from data is testing, and if you’re looking to grow your online business, implementing well-executed, consistent A/B testing is a necessity.

At Shutterstock, where I work, we test everything: copy and link colors, relevance algorithms that rank our search results, language-detection functions, usability in downloading, pricing, video-playback design, and anything else you can see on our site (plus a lot you can’t see).

Shutterstock is the world’s largest creative marketplace, serving photography, illustrations, and video to more than 750,000 customers. And those customers have heavy image needs; we serve over three downloads per second. That’s a ton of data.

This means that we know more about our customers, statistically, than anyone else in our market. It also means that we can run more experiments with statistical significance faster than businesses with less user data. It’s one of our most important competitive advantages.

A/B Testing in Action

Search results are among the highest-trafficked pages on our site. A few years back, we started experimenting with a mosaic-display search-results page in our Labs area — an experimentation platform we use to try things quickly and get user feedback. In qualitative testing, customers really liked the design of the mosaic search grid, so we A/B tested it within the core Shutterstock experience. For those unfamiliar with A/B testing, it involves, at its core, showing different user experiences to different users to measure the impact of those differences. You can read more about it here.

Here are some of the details of the experiment, and what we learned:

Image sizes: We tested different image sizes to get just the right number of pixels on the screen.

New customers: We watched to see if new customers to our site would increase conversion. New customers act differently than existing ones, so you need to account for that. Sometimes existing customers suffer from change aversion.

Viewport size: Tracking the viewport size (the size of the screen customers used) was an important measure of understanding.

Watermarks: Image watermark vs. no watermark: Was the watermark inclusion distracting?

Hover: We experimented with the behavior of the hover when a user pauses on a particular image.

Before the test, we were convinced that removing the Watermark on our images would increase conversion because there would be less visual clutter on the page, but in testing we learned that removing the watermark created the opposite effect, disproving our gut instinct.

We ran enough tests to find two different designs that increased conversion, so we iterated on those designs and re-tested them before deciding on one. And we continue to test this search grid and make improvements for our customers on a regular basis.

Experimentation Culture

The most obvious benefit of a test like this is the ability to improve our product and increase our revenue. But there are indirect benefits of testing too, ones that manifest themselves once testing becomes ingrained in company culture.

First off, in an experimentation culture you kill off the HiPPOs. (That’s an acronym for the Highest Paid Person’s Opinion.) A/B testing is a sure way to get to the bottom of a decision without relying on anyone’s gut instinct. At Shutterstock, if a senior executive has an idea in a meeting, the response is simply “Let’s test it.”

Secondly, more of your ideas will see daylight in the form of tests, instead of being killed off on whiteboards and in presentations. When it’s easy to try your ideas, your team can stop speaking in the abstract about things that haven’t happened yet, and instead speak about results and next steps. Lastly, co-workers will be highly motivated, because they get to see their ideas live in the real world.

It’s not all roses, of course. Experimentation culture has some downsides, too. One of the big ones is that experimentation teams sometimes miss the next big innovation because they’re constantly making incremental improvements that show quickly in test results. Remember, some test results will show a negative outcome in the short term, but be better in the long term due to user change aversion. Also, testing strategy is hard, and there’s still a place for strategic thinking that moves your organization in new directions.

Define your experimentation culture along a vector of Brand vs. Optimization — your current status vs. your aspiration. On the Brand side of that spectrum are tight visual brand guidelines, consistency between offline and online marketing, and similar styles throughout the experience. On the Optimization end of the spectrum, anything goes: test the logo, test the header, try any color or copy, never trust your gut, and allow your product to have some inconsistency if it means conversion is higher. Your culture will fall somewhere in between.

We ran a helpful exercise with our product and marketing teams where we each put our favorite company on this spectrum, then debated where we thought Shutterstock was, and where it wanted to be. We came out of that meeting very far to the Optimization end of the spectrum.

Putting Experimentation Into Practice

Here are some helpful tips that can set you on your way to creating growth through experimentation. (For a longer list, check out this blog post.)

1. Keep the team small. All you need to perform tests is a 3-4 person team made up of an engineer, a designer/front-end developer, and a business analyst (or product owner). Make sure your designer can think iteratively and conduct tests, and that your product person has the skills to analyze tests as they happen, to avoid going to another department for test results.

2. Metrics: choose one, view many. The metric you tried to move probably won’t tell the whole story. Plenty of very smart people have been puzzled by A/B test results. You’ll need to look at lots of different metrics to figure out what change really happened, and more often than not, you’ll be incorrect. When a test fails, don’t give up. Instead, learn what happened (figure out, for example, which metric did move), and use that to inform future iterations.

3. De-averaging. Often a test doesn’t perform better on average, but does for particular customer segments, such as new vs. existing customers. The test may also be performing better for a particular geo, language, or even user persona. You won’t find these insights without looking beyond averages by digging into different segments.

4. Test small changes. Don’t spend months building a test just to throw it away when it doesn’t work. If you have to spend a long time creating it, then you’re doing it wrong. Find the smallest amount of development you can do to create a test based on your hypothesis; one variable at a time is best.

5. Re-test ideas. Tests can often fail because of seasonality, or because you missed one tiny nuance. That doesn’t mean you should throw away the test; keep a backlog of previously run tests, and try re-running a few later. You might be surprised what you find.

6. Don’t forget performance. Performance testing should be considered part of your design and optimization. Even a small page-load increase can foul what would otherwise be a winning design. Page weight and load should be tested alongside other tests. I’ve seen winning tests lose on performance issues alone.

For more on getting started, take a look at these slides.