This post is part of our Guide to Bayesian Statistics and received a update as a chapter in Bayesian Statistics the Fun Way!

We've covered the basics of Parameter Estimation pretty well at this point. We've seen how to use the PDF, CDF and Quantile function to learn the likelihood of certain values, and we've seen how we can add a Bayesian prior to our estimate. Now we want to use our estimates to compare two unknown parameters.

Keeping with our email example we are going to set up an A/B Test. We want to send out a new email and see if adding an image to the email helps or hurts the conversion rate. Normally when the weekly email is sent out it includes some image, for our test we're going to send one Variant with the image like we always do and another without the image. The test is called an A/B Test because we are comparing Variant A (with image) and Variant B (without).

We'll assume at this point we have 600 subscribers. Because we want to exploit the knowledge gained during our experiment we're only going to be running our test on 300 of these subscribers, that way we can give the remaining 300 what we believe to be the best variant. The 300 people we're going to test will be split up into two groups, A and B. Group A will receive an email like we always send, with a big picture at the top, and group B's will not have the picture.

Next we need to figure out what prior probability we are going to use. We've run an email campaign every week so we have a reasonable expectation that the probability of the recipient clicking the link to the blog on any given email should be around 30%. To make things simple we'll use the same prior for A and B. We'll also choose a pretty weak version of our prior because we don't really know how well we expect B to do, and this is a new email campaign so maybe other factors would cause a better or worse conversion anyway. We'll settle on Beta(3,7):