Bayesian Inference: Hidden Success Probability

Suppose we have \( n \) independent Bernoulli trials with unknown probability of success \( p \), and we observe:

\( s \) successes
\( f \) failures
of course, enforcing the two conditions \( n \equiv s + f\) and \( s, f \in \mathbb{Z}_{>0} \)

1. Likelihood

Each trial has probability \( p \) of success and probability of \( 1 - p \) failure. The probability \( s \) successes over \( n \) trials follow a binomial distribution, which gives us the likelihood function:

\[ \mathcal{L}(s ~|~ p,f) = \binom{n}{s} \cdot p^s (1 - p)^f \propto p^s (1 - p)^f. \]

2. Prior

Let us choose the Beta distribution because it is the conjugate prior to the binomial likelihood.

The Beta distribution is parameterised by the hyperparameters \( \alpha \) and \( \beta \). A nice property of the Beta distribution is that when the hyperparameters \( (\alpha, \beta) = (1, 1) \), the Beta distribution corresponds to a Uniform distribution over \( [0, 1] \). Thus, our prior is:

\[ \pi(p) = \frac{p^{\alpha - 1} (1 - p)^{\beta - 1}}{\text{B}(\alpha, \beta)} \propto p^{\alpha - 1} (1 - p)^{\beta - 1} \]

where \( \text{B}(\alpha, \beta) \) is the Beta function, defined as in terms of the Gamma (\( \Gamma \)) function as:

\[ \text{B}(\alpha, \beta) = \frac{\Gamma(\alpha) \Gamma(\beta)}{\Gamma(\alpha + \beta)}. \]

3. Posterior

Using Bayes’ Theorem, we can estimate the posterior distribution of \( p \) given the data:

\[ \mathcal{P}(p ~|~ s,f) = \mathcal{L}(s ~|~ p,f) \cdot \pi(p) \propto p^{s + \alpha - 1} (1 - p)^{f + \beta - 1}. \]

Since \( \mathcal{P} \propto p^s (1 - p)^f \) is a Beta distribution, we readily have the normalisation constant (the Beta function as discussed above). Thus, the posterior distribution is:

\[ \boxed{\mathcal{P}(p ~|~ s,f; \alpha, \beta) = \mathrm{Beta}(s + \alpha, f + \beta)}. \]

Now, suppose we choose \( (\alpha, \beta) = (1,1) \) for an uninformative Uniform prior. Now, we can readily use known properties of the Beta distribution to give us the expectation value and other statistics in terms of successes and failures, such as:

Expectation Value: \[ \mathbb{E}[p ~|~ s,f] = \frac{s + 1}{s + f + 2} \]
Median: \[ \text{Median} \approx \frac{s + \frac{2}{3}}{s + f + \frac{4}{3}} \]
Variance: \[ \mathrm{Var}(p ~|~ s) = \frac{(s + 1)(f + 1)}{(s + f + 2)^2 (s + f + 3)} \]

Of course, these results can be generalised to for any \( (\alpha, \beta) \) hyperparameters.