Hi! You signed up for my weekly newsletter this past week, probably after purchasing Cadence & Slang or Distance. I write about all sorts of things every week, and I don't always talk about design and tech stuff. If you want to get a sense of how things have gone in the past, you can go back and read some old hits.



If you don't want to read these anymore, no hard feelings; there's an unsubscribe link at the bottom of this email. Onward!

I'd like to chat about A/B testing today. Let's say you run a website, and you want to change something on there. You could just make the change and upload it – but you're scared that it might be read the wrong way, or people might not like it.



In an A/B test, you give your original page to half of your visitors, and a revised version to the other half. You measure something on it, and let your users duke it out over time. Then you go with whichever version wins.

Do you test new ideas on your site? Small revisions? With real people?

I see. Why not? You haven't done it before. Well, how did you know that your wireframes made for a good design?

You don't. All design is speculation until it's been put in front of real people. And we're doing a hell of a lot of speculating these days.

A/B Testing Is Broken

…for you. Probably. Meanwhile, A/B testing is doing just fine for the people who know the value of it, and who put in the effort to conduct A/B tests. Everyone else has a lot of learning to do. Heck, we all have a lot of learning to do.

Here are a bunch of reasons why you don't A/B test:

You don't know how. You have to learn the capability of A/B testing; it isn't quite as simple as straight HTML development or UX techniques. Many utilities, like Optimizely and Visual Website Optimizer, exist to make the process a little simpler; but there are also server-side utilities to help, like A/Bingo for Rails.

You have to learn the capability of A/B testing; it isn't quite as simple as straight HTML development or UX techniques. Many utilities, like Optimizely and Visual Website Optimizer, exist to make the process a little simpler; but there are also server-side utilities to help, like A/Bingo for Rails. You didn't scope it into the project. Traditional design process don't account for A/B testing, and that means a goodly amount of them are fundamentally broken. Meanwhile, clients are complicit because they haven't been sold on the process, either.

Traditional design process don't account for A/B testing, and that means a goodly amount of them are fundamentally broken. Meanwhile, clients are complicit because they haven't been sold on the process, either. You don't know what to change. Let's assume you had the best, easiest-to-use, most data-robust A/B testing framework installed on your site for free tomorrow. What would you do with it? Would you even know? Conversion optimization may be practiced by a lot of really smart people right now, but it isn't widespread enough.

Let's assume you had the best, easiest-to-use, most data-robust A/B testing framework installed on your site for free tomorrow. What would you do with it? Would you even know? Conversion optimization may be practiced by a lot of really smart people right now, but it isn't widespread enough. Some people parse A/B testing as a black hat UX technique. It's not what it is, it's how you use it. Anyone can use A/B testing to manipulate or deceive their customers. Don't do that. Use A/B testing to make it easier and more pleasurable for people to use your product. If you can't do that honestly and with integrity, you have no business with A/B testing, much less putting something into the world in the first place.

It's not what it is, it's how you use it. Anyone can use A/B testing to manipulate or deceive their customers. Don't do that. Use A/B testing to make it easier and more pleasurable for people to use your product. If you can't do that honestly and with integrity, you have no business with A/B testing, much less putting something into the world in the first place. You think your work will run into a local maximum. Small improvements will lead you to designs that are increasingly difficult to optimize, and larger redesigns may make sense at that point. Bugt that is no excuse to not try A/B testing. You have no idea what you're leaving on the table by not testing.

In short…

You're afraid. All of that stuff above? You've thought about it and taken it seriously, and it petrifies you. Where the heck do you begin? Is there a book for this stuff? (There isn't. You've checked.) If you put a test online and it fails miserably, yes, that's lost business. But those losses can be quickly erased by your gradual wins. How do you justify short-term losses to your boss? To yourself? How do you work without a net?

I've thrown a lot of effort into my A/B testing service, Draft Revise, lately. In the past few months, I've helped double the sales of several small businesses. I'm going to spend the next couple of weeks talking about how I use A/B tests to help a lot of good people. I hope you enjoy it!

Thanks for reading,

— Nick Disabato

nickd@nickd.org