Another way of doing hypothesis tests, is to use the marginal likelihood which we have not talked about much yet.

Let us re-iterate Bayesian inference:

$$ \overbrace{P(\theta| D, I)}^{\text{posterior}} = \frac{\overbrace{P(D | \theta, I)}^{\text{likelihood}}\overbrace{P(\theta|I)}^{\text{prior}}}{\underbrace{P(D|I)}_{\text{marginal likelihood}}} $$

$\theta$ corresponds to the parameters of the binomial model, $D$ are the coin flips at interest and $I$ corresponds to additional information (e.g., a model or a hypothesis).

The marginal likelihood corresponds to the probability of the data where the parameters have been marginalized out:

$$ P(D | I) = \int P(D | \theta, I)P(\theta | I)d\theta $$

The evidence is the weighted average over all possible values of the parameters $\theta$ where the weights come from the prior. So basically, the marginal likelihood is an average of the likelihood weighted by the prior.

Generally, we can say that if the prior is well aligned with the data, then the evidence is rising with the strength of the prior. The evidence is the largest if the prior and the likelihood are concentrated over the same parameter regions and it is the lowest if they concentrate on different regions.

For comparing the plausibility of two models, we can resort to Bayes factors. Bayes factors are representing a Bayesian method for model comparison that include a natural Occam's razor guarding against overfitting. We can define the Bayes factor---note that we apply unbiased comparison assuming that all models are equally likely a priori---as follows:

$$ B_{1,2} = \frac{P(D | M_1)}{P(D|M_2)} $$

The strength of a Bayes factor can be determined by consulting the interpretation tables of Jeffrey or Kass and Raftery.

Instead of comparing models, we can now also compare hypotheses encoded in the form of priors. For example, for the null hypothesis of a fair coin, we could use a Beta prior with $\alpha=100$ and $\beta=100$. The stronger be believe in this hypothesis, the higher we would set the overall concentration of the Beta prior, e.g., $\alpha=1000$ and $\beta=1000$. This is a similar idea to the ROPE discussed above, with rising symmetric concentration, we would at one point only believe in $p=0.5$ without any tolerance. As an alternative hypothesis, we could use an "anything is possible" prior with $\alpha=1$ and $\beta=1$.

For the binomial model with a conjugate Beta prior, the marginal likelihood is defined as follows,

$$ P(D) = {{n}\choose{k}}\frac{B(k+\alpha,n-k+\beta)}{B(\alpha,\beta)} $$

where $B()$ is the Beta function.

Then, for alternative hypotheses $H_{null}$ and $H_{alt}$, the Bayes factor is defined as:

$$ \frac{P(n,k|H_{alt})}{P(n,k|H_{null})} = \frac{{{n}\choose{k}}\frac{B(k+\alpha_{alt},n-k+\beta_{alt})}{B(\alpha_{alt},\beta_{alt})}}{{{n}\choose{k}}\frac{B(k+\alpha_{null},n-k+\beta_{null})}{B(\alpha_{null},\beta_{null})}} $$

The binomial coefficient cancelc out, and we can calculate the Bayes factor: