Bayesian Priors, Likelihoods, and Posteriors

Building, updating, checking, and interpreting a Bayesian model

Statistics
Inference
Bayesian
A practical foundation for Bayesian inference: priors, likelihoods, posterior distributions, prediction, computation, checking, and sensitivity.
Author

Jiahong Sun

Published

September 23, 2026

Modified

September 23, 2026

1 A Probability Model for Unknown Quantities

Bayesian inference represents uncertainty about an unknown parameter or latent quantity \(\theta\) with probability. A complete model specifies both how data could be generated and which parameter values were plausible before the current data were observed:

\[ \theta\sim p(\theta), \qquad y\mid\theta\sim p(y\mid\theta). \]

Here \(p(\theta)\) is the prior distribution and \(p(y\mid\theta)\) is the sampling distribution. After observing \(y\), Bayes’ theorem gives

\[ \boxed{ p(\theta\mid y) =\frac{p(y\mid\theta)p(\theta)}{p(y)} =\frac{p(y\mid\theta)p(\theta)} {\int p(y\mid t)p(t)\,dt} } \]

for a continuous parameter; a sum replaces the integral for a discrete parameter. The resulting \(p(\theta\mid y)\) is the posterior distribution. The denominator

\[ p(y)=\int p(y\mid\theta)p(\theta)\,d\theta \]

is the prior predictive density, also called the marginal likelihood or model evidence. It normalizes the posterior and is important in some model-comparison methods, even though it can be omitted when deriving a posterior kernel:

\[ p(\theta\mid y)\propto p(y\mid\theta)p(\theta). \]

This formula is an identity, not a guarantee that the chosen prior or sampling model is scientifically adequate.

2 The Four Objects Have Different Roles

Object What varies? Main role Common mistake
Prior \(p(\theta)\) \(\theta\) Encodes uncertainty before the current data Calling a vague prior assumption-free
Likelihood \(L(\theta;y)=p(y\mid\theta)\) \(\theta\), with \(y\) fixed Measures relative support from the observed data Treating it as a probability distribution over \(\theta\)
Posterior \(p(\theta\mid y)\) \(\theta\), with \(y\) observed Represents updated uncertainty conditional on the model Reporting only its mode and discarding its spread or shape
Posterior predictive \(p(\tilde y\mid y)\) Future or replicated data \(\tilde y\) Propagates parameter uncertainty into observables Plugging in one estimate and ignoring posterior uncertainty

The likelihood is the same mathematical object used in likelihood and log-likelihood. Bayesian and likelihood-based analyses differ in what else they add and how they interpret uncertainty.

3 Choosing and Understanding a Prior

A prior should be stated on a scientifically interpretable scale and examined through its implications. Useful sources include previous commensurate studies, physical constraints, expert knowledge, or regularization that rules out implausible extremes.

Broad labels do not determine whether a prior is appropriate:

  • An informative prior concentrates substantial mass based on credible external information.
  • A weakly informative prior excludes unreasonable values while leaving a broad scientifically plausible range.
  • A reference, default, or purportedly noninformative prior follows a formal construction, but can still affect inference and may change under reparameterization.
  • An improper prior does not integrate to one. It can sometimes yield a proper posterior, but prior prediction is then undefined and posterior propriety must be proved rather than assumed.
  • An empirical-Bayes prior estimates hyperparameters from data. Its uncertainty and any reuse of the same data need explicit treatment.

Uniformity is scale-dependent. A density uniform in a probability \(p\) is not uniform in the log-odds \(\log\{p/(1-p)\}\). Every density is defined relative to a parameterization and base measure.

3.1 Prior predictive checking

Before fitting the model, simulate

\[ \theta^{\mathrm{sim}}\sim p(\theta), \qquad y^{\mathrm{sim}}\sim p(y\mid\theta^{\mathrm{sim}}). \]

The distribution

\[ p(y)=\int p(y\mid\theta)p(\theta)\,d\theta \]

shows what observable data the entire prior-plus-likelihood model regards as plausible. Impossible counts, implausible effect sizes, or unrealistic between-sample variation can reveal a poorly calibrated prior more clearly than inspecting its numerical scale alone. A prior predictive check tests implications, not whether a prior is uniquely correct.

4 Worked Derivation: Beta–Binomial Learning

Suppose the experimental units are \(n\) independently prepared specimens and \(Y\) is the number positive for a prespecified assay endpoint:

\[ Y\mid p\sim\operatorname{Binomial}(n,p), \qquad p\sim\operatorname{Beta}(a,b). \]

Holding the observed count \(y\) fixed and retaining terms that depend on \(p\),

\[ \begin{aligned} p(p\mid y) &\propto p^y(1-p)^{n-y} p^{a-1}(1-p)^{b-1}\\ &=p^{a+y-1}(1-p)^{b+n-y-1}. \end{aligned} \]

Therefore

\[ \boxed{p\mid y\sim\operatorname{Beta}(a+y,b+n-y).} \]

The beta family is conjugate to the binomial likelihood: the posterior has the same distributional form as the prior. Its mean is

\[ E(p\mid y)=\frac{a+y}{a+b+n} =\frac{a+b}{a+b+n}\frac{a}{a+b} +\frac{n}{a+b+n}\frac{y}{n}. \]

This is a weighted average of the prior mean and observed proportion. Describing \(a+b\) as an “effective prior sample size” is an algebraic analogy; it is not evidence that those units were actually observed under the current protocol.

For one future exchangeable specimen, integrating over posterior uncertainty gives

\[ P(\tilde Y=1\mid y) =\int_0^1 p\,p(p\mid y)\,dp =\frac{a+y}{a+b+n}. \]

For multiple future trials, integration produces a beta-binomial distribution, whose variance includes uncertainty about \(p\).

5 Posterior Summaries and Decisions

The posterior distribution is the inferential result. Summaries should follow the scientific question rather than habit.

  • A posterior mean minimizes posterior expected squared-error loss.
  • A posterior median minimizes posterior expected absolute-error loss.
  • A posterior mode is the maximum a posteriori estimate, but can be unstable or parameterization-dependent.
  • A \(100(1-\alpha)\%\) credible interval is a set \(C(y)\) satisfying \(P(\theta\in C(y)\mid y)=1-\alpha\) under the fitted model.
  • A posterior probability such as \(P(\theta>\theta_0\mid y)\) directly addresses a prespecified threshold, although the threshold should have scientific meaning.

A Bayesian credible interval conditions on the observed data and prior-model specification. It is not a frequentist confidence interval, whose defining probability concerns coverage across repeated samples. Neither interval interpretation survives an incorrect experimental unit, unmodeled selection, or a badly misspecified likelihood.

Decisions require more than estimation. With action \(a\) and loss \(L(a,\theta)\), a Bayes action minimizes posterior expected loss:

\[ a^*(y)=\operatorname*{arg\,min}_a E\{L(a,\theta)\mid y\}. \]

Thus a treatment, validation, or follow-up decision depends on its consequences, not merely on whether a posterior probability exceeds an arbitrary convention.

6 Computation and Computational Diagnostics

Conjugate models permit exact algebra, but most realistic posteriors require numerical work.

Method What it approximates Main diagnostic concern
Quadrature or a parameter grid Integrals on a low-dimensional space Grid range and resolution
Markov chain Monte Carlo Correlated draws from the posterior Mixing, divergences, effective sample size, and Monte Carlo error
Importance sampling Weighted draws from a proposal Unstable or highly concentrated weights
Variational inference A tractable distribution near the posterior Approximation bias and underestimated tails or dependence
Laplace approximation Local Gaussian shape around a mode Skewness, boundaries, weak identification, and multiple modes

An optimizer or sampler can converge for a poor model, and a good model can be approximated badly. Computational diagnostics and model criticism are separate requirements. Posterior summaries should report Monte Carlo error when simulation error is not negligible relative to scientific uncertainty.

7 Posterior Prediction and Model Checking

Posterior prediction averages the observation model over parameter uncertainty:

\[ \boxed{ p(\tilde y\mid y)= \int p(\tilde y\mid\theta)p(\theta\mid y)\,d\theta. } \]

For a posterior predictive check, generate replicated data \(y^{\mathrm{rep}}\) using the fitted model and compare scientifically relevant features \(T(y^{\mathrm{rep}})\) with \(T(y)\). Useful discrepancies might include the number of zeros, an upper-tail quantile, between-sample variability, or a batch contrast. A check should target ways the model could fail; matching a statistic already forced by the fit is weak evidence of adequacy.

Posterior predictive checks detect mismatch but do not automatically identify its cause. A discrepancy can arise from the likelihood, dependence structure, prior, covariates, measurement process, or selection mechanism. Model revision should be followed by new checks and transparent reporting rather than treating the same data as untouched confirmation.

8 Bioinformatics Interpretation and Replication

Consider estimating a batch-specific contamination probability from sequencing libraries. The estimand might be the probability that a newly prepared library from the target laboratory and protocol exceeds a prespecified contamination threshold. The independent experimental units are libraries, not reads; reads are technical measurements nested within libraries.

A defensible model may therefore include:

  1. a library-level likelihood for the threshold indicator or contamination fraction;
  2. a prior calibrated using comparable laboratories and protocols;
  3. batch or run effects if the target population includes multiple batches;
  4. an explicit measurement model when read depth makes library classifications uncertain; and
  5. posterior prediction for a new library, not merely a posterior for a read-level rate.

Statistically, the posterior quantifies uncertainty under this hierarchy. Biologically, relevance depends on specimen type, preparation chemistry, laboratory conditions, contamination definition, and target population. Millions of reads can reduce measurement error within a library, but they cannot replace biological or process replication across libraries.

9 Sensitivity and Failure Modes

Issue Why it matters Check or response
Prior–likelihood conflict A concentrated prior and incompatible data can produce surprising compromise or model failure Plot prior, likelihood, and posterior; inspect prior and posterior predictions
Weak identification Different parameter combinations explain the data similarly Inspect joint posterior geometry and predictions; redesign or justify constraints
Improper posterior No valid probability distribution exists Establish posterior propriety or use a proper prior
Pseudoreplication Posterior precision reflects reads or cells rather than independent samples Encode clustering or aggregate at the experimental-unit level
Model misspecification A narrow posterior can be confidently wrong Use targeted posterior predictive checks and alternative likelihoods
Prior sensitivity Weak data leave conclusions dependent on prior choices Refit with scientifically defensible alternatives and report changes
Selection or data reuse Conditioning mechanism is omitted or information is counted twice Model selection/ascertainment and separate prior construction from validation

Sensitivity analysis is most informative when alternatives are plausible and prespecified. Replacing a defensible prior with an arbitrarily extreme one does not measure practical robustness.

10 Connections and Take-home Messages

Bayesian analysis is a workflow, not just multiplication of a prior and likelihood. It requires a scientifically meaningful estimand, a generative model, calibrated priors, accurate computation, posterior uncertainty, predictive checking, and sensitivity analysis. Its probabilities are conditional on all of those choices.

11 References

  • Bayes, T. and Price, R. “An Essay towards Solving a Problem in the Doctrine of Chances.” Philosophical Transactions of the Royal Society of London 53 (1763): 370–418. Article.
  • Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. Bayesian Data Analysis, 3rd ed. CRC Press, 2013. Book resources.
  • Gelman, A., Vehtari, A., Simpson, D., et al. “Bayesian Workflow.” arXiv:2011.01808, 2020. Preprint.
  • Gelman, A., Meng, X.-L., and Stern, H. “Posterior Predictive Assessment of Model Fitness via Realized Discrepancies.” Statistica Sinica 6 (1996): 733–760. Article.
  • Stan Development Team. “Posterior and Prior Predictive Checks.” Stan User’s Guide. Official documentation.
Back to top