I went to college twice for practically nothing that I'm paid to do now, in part, due to my incredible ability to speak about any subject at length regardless of whether I know about it or not. I was once asked if I knew anything about A/B testing and I pulled on a wealth knowledge from an undergraduate degree in Political Science to statistic-ify my shoddy answer. But they bought it.When I actually started doing tests, I perceived it as a clinical, empirical activity; one that's devoid of personal experience or philosophy. What I realized, however, was that opinions and approaches vary as much as anything else. A personal ethos exists with how you approach testing that very much influences how your testing program will perform.

People Love Testing

My experience is that people love the concept of testing. Real world experimentation smacks of all the curiosity of a middle school science fair project, but with all the promise to make any business owners' dream come true.

A/B testing makes space numbers and fairy dust feel like real science.

Imagine it: A process that has some semblance of actual science and is designed to improve practically any business objective. Intrinsically measurable. Often associated with a bevy of lines or bar charts. All incredibly sexy, but nothing as alluring as knowledge. Testing gives definition to conversations with marketing folks that are often bullshit.It feels real and for many people it is.

Early Mistakes In A/B Testing

The logic behind A/B testing is easy to understand. The Pepsi Challenge, where random participants in a blind taste test were asked to choose their favorite soft drink, was simple enough for anyone to get: Two things are tested and one of those things outperformed the other. If you do this enough times then the result is the same then, well, that's the Truth.The problem wasn't in the logic, it was with the philosophy. The biggest mistake I made when starting A/B test was not making enough mistakes.Fear played a lot into it. I was new and the pressure to improve, in this instance, conversions through testing was incredibly high. In retrospect, this was unfortunate since fear—the fear of making mistakes—is the antithesis of what testing is: sheer exploration, luck, and nerve.

Superfluous Changes