2 Coefficients and Contrasts

With one continuous predictor,

A one-unit increase in multiplies the conditional odds by . For a contrast of units, the odds ratio is . The associated probability difference is

which depends on the starting value and all other predictors. A constant odds ratio therefore does not imply a constant risk difference.

For a binary predictor in

compares conditional odds at common modeled values of . This is an adjusted association. A causal interpretation additionally requires a defensible treatment definition, exchangeability, positivity, consistency, and appropriate handling of selection, missingness, and measurement.

Interactions make the log-odds contrast depend on another predictor. With , the conditional odds ratio for at is . As with linear regression, lower-order coefficients refer to the other variable’s reference value.

3 Likelihood and Estimation

Conditional independence across experimental units gives the likelihood

and log-likelihood

Let and . Differentiation gives

The maximum-likelihood estimate solves . There is generally no closed form, so software uses Newton–Raphson, Fisher scoring, or the equivalent iteratively reweighted least-squares calculation. Concavity makes a finite, identified solution unique when the design has full rank and the outcomes are not separated by the predictors.

At a regular finite solution, the model-based covariance estimate is

Wald intervals use this local quadratic approximation. Likelihood-ratio or profile- likelihood intervals often behave better for sparse data or asymmetric likelihoods. All are conditional on the model, sampling structure, and prespecified analysis.

4 A Small Calculation

Suppose a fitted model for a binary assay failure is

where is a centered quality score. At ,

At , the odds are multiplied by , but the probability is , a risk increase of about 0.112 rather than 123%. At , the probability is about 0.401. The example shows why the coefficient, odds ratio, and probability contrast answer related but different questions.

5 Assumptions, Diagnostics, and Sensitivity

Residuals are not normally distributed in a Bernoulli model. Deviance and Pearson residuals can reveal unusual patterns, but a binary observation contains little unit-level residual information. Binning residuals or probabilities can aid a display, yet conclusions can depend on arbitrary bins. Diagnostics cannot prove that confounding is absent or that a future population follows the same process.

6 Bioinformatics Design and Interpretation

Consider whether 80 independent patient biopsies show a molecular response after treatment. A logistic model includes treatment, prespecified baseline severity, tissue type, and sequencing batch. Patients are the experimental units. Cells, reads, technical libraries, and repeated aliquots nested within one biopsy are not 80 additional independent treatment replicates.

If the outcome is instead defined for every cell, cell responses remain clustered within donors. A cell-level logistic model must represent donor dependence and the level at which treatment was assigned; pseudoreplication can otherwise make uncertainty implausibly small. If every responder was sequenced in one batch and every nonresponder in another, outcome and batch effects are not separately identified.

Statistically, a treatment coefficient describes a conditional log-odds contrast under the fitted model. Biologically, the result also depends on how response was defined, assay error, effect magnitude on an interpretable probability scale, patient heterogeneity, and transport to the target population. A small p-value or large odds ratio alone does not establish clinical or mechanistic importance.

7 Prediction Is a Separate Goal

Likelihood fit on development data does not establish predictive usefulness. For risk prediction, evaluate probabilities on data not used for flexible model development. Report calibration, discrimination, and an overall proper score; none replaces the others. A classification threshold additionally encodes the relative consequences of false positives and false negatives and should not be chosen from accuracy alone.

For explanation or causal estimation, held-out prediction is informative about transport but does not diagnose confounding. Conversely, a causally interpretable coefficient need not produce an accurate individual risk model. State the goal before selecting predictors, regularization, metrics, and validation design.

8 Connections and Take-home Messages

• Odds ratios, probabilities, and calibration develops effect-scale interpretation and probability validation.
• Likelihood and log-likelihood gives the inferential foundation for the Bernoulli likelihood.
• Maximum-likelihood estimation explains score equations, information, and nonregular failures.
• Categorical predictors and contrasts covers coding and estimable comparisons.
• Interaction terms and effect modification explains conditional effects and scale dependence.
• Robust standard errors and robust regression separates covariance robustness from mean-model correctness.

1. Logistic regression models a conditional Bernoulli probability through a linear predictor on the log-odds scale.
2. Exponentiated coefficients are conditional odds ratios, not probability differences or automatically causal effects.
3. Maximum likelihood is numerical and can fail to have a finite solution under separation.
4. Valid uncertainty depends on independent experimental units, adequate outcome information, identification, and a suitable data-generating model.
5. Prediction requires external or resampling-based validation of probabilities; coefficient inference and predictive performance are different tasks.

9 References

• Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B, 20(2), 215–232. https://doi.org/10.1111/j.2517-6161.1958.tb00292.x
• McCullagh, P., and Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Chapman and Hall/CRC. https://doi.org/10.1201/9780203753736
• Firth, D. (1993). Bias reduction of maximum likelihood estimates. Biometrika, 80(1), 27–38. https://doi.org/10.1093/biomet/80.1.27
• Heinze, G., and Schemper, M. (2002). A solution to the problem of separation in logistic regression. Statistics in Medicine, 21(16), 2409–2419. https://doi.org/10.1002/sim.1047
• R Core Team. glm: fitting generalized linear models; family: family objects for models. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/glm.html

2 Odds Are Not Probabilities

For probability , the odds are . If is a baseline probability and is an odds ratio, the corresponding exposed probability is

An odds ratio of 2 does not mean that the probability doubles. For example:

Baseline risk	Risk with OR = 2	Risk difference	Risk ratio
0.01	0.0198	0.0098	1.98
0.20	0.3333	0.1333	1.67
0.60	0.7500	0.1500	1.25

The rare-outcome approximation is useful only when both risks are small enough for the intended precision. It does not turn a conditional association into a causal effect.

3 Conditional and Marginal Effects

Logistic odds ratios are non-collapsible. Even if a covariate is not a confounder, a conditional odds ratio adjusted for that covariate can differ from the marginal odds ratio because averaging probabilities and taking odds are nonlinear operations. A change after adjustment is therefore not, by itself, evidence of confounding.

To obtain a population-average risk under exposure level , standardize the fitted probabilities over a declared target covariate distribution:

Then and are standardized risk difference and risk ratio estimates for the population represented by weights . Equal weights over the study sample, survey weights, and external target- population weights define different estimands. Confidence intervals should carry coefficient-estimation uncertainty and, where relevant, uncertainty in the target distribution; delta-method, bootstrap, or simulation-based propagation can be used under their respective assumptions.

For a causal interpretation, standardization additionally needs conditional exchangeability, positivity, consistency, and correct treatment/outcome and covariate modeling. It is still valuable descriptively when causal identification is not claimed, provided the target and model are explicit.

4 Calibration: Do Probabilities Mean What They Say?

A model is calibrated in a target setting when predicted probabilities agree with event frequencies at the relevant level. If all units assigned probability 0.20 were repeatedly observed under the same process, approximately 20% should have the event. Calibration is a property of predictions in a population and time period, not a permanent label attached to an algorithm.

Useful levels include:

• Calibration-in-the-large: Is the mean predicted risk equal to the observed event proportion? Systematic overprediction or underprediction shifts this.

• Calibration slope: Regress the outcome on the original prediction logit, , in validation data:

  

  Ideal values are and . A slope below one often signals predictions that are too extreme, as after overfitting; a slope above one means they vary too little in that population.

• Flexible calibration: Compare observed risk with predicted risk using a smooth curve and uncertainty, revealing local departures hidden by two numbers.

An intercept-only recalibration can correct a changed event rate while preserving relative predictions. Intercept-and-slope recalibration can also shrink or expand the prediction logit. Such updating should use appropriate data and be evaluated again; it does not repair missing predictors, changed outcome definitions, selection bias, or unsupported covariate regions.

5 Calibration, Discrimination, and Overall Accuracy

These properties answer different questions:

For binary outcomes, the Brier score is

The mean log loss is

Both are proper scoring rules: in expectation, honest probabilities optimize the score under the scoring convention. Log loss penalizes confident wrong predictions especially strongly. Compare scores with a meaningful reference model and use uncertainty intervals rather than reporting an isolated decimal.

A high AUC can coexist with poor calibration because ranking can remain unchanged when all probabilities are systematically too high. Conversely, a useful recalibration may improve probability accuracy without changing the AUC.

6 Validation Without Optimism

Apparent performance on the development data is optimistic, especially after predictor selection, transformation searches, hyperparameter tuning, or threshold selection. Every data-dependent modeling step must occur inside the resampling loop. Bootstrap validation or nested cross-validation can estimate internal optimism; a genuinely external sample tests transport to a new place, time, or population.

Calibration assessment itself needs enough independent events and non-events. Small validation sets yield noisy curves, intercepts, and slopes. Grouping predictions into arbitrary deciles hides within-bin structure and can create misleading shapes; show a smooth curve with the prediction distribution and uncertainty when possible.

Prevalence altered by case-control sampling requires special care. Slopes and exposure odds ratios can be estimable under the design, while the raw sample event fraction and uncorrected intercept do not estimate population risk. Absolute probabilities require known sampling fractions or another valid route to the target prevalence.

7 Thresholds and Decisions

Turning into a class label at 0.5 discards probability information and quietly assumes a particular balance of consequences. The relevant threshold is determined by the decision: costs of false positives and false negatives, available interventions, resource constraints, and patient or scientific values.

Sensitivity and specificity describe classification at a chosen threshold; they do not assess probability calibration. Accuracy can favor predicting the majority class for rare outcomes. Select and evaluate a threshold using data independent of model fitting, or account for selection in the validation procedure.

8 Bioinformatics Example

Suppose a gene-expression signature predicts whether a patient-derived organoid responds to a drug. Donor lines are experimental units. Multiple wells, cells, reads, or library preparations within a donor can improve measurement but are technical or subsample replication; treating them as independent validation cases exaggerates precision and permits donor identity to leak across train and test sets.

Split or resample at the donor level and keep all preprocessing, feature selection, normalization choices, and tuning inside each training fold. Report the number of independent donors, events and non-events, missingness, prediction horizon, and outcome definition. Validate calibration in the intended target population, whose response prevalence and assay pipeline may differ from the development study.

Statistically, a calibrated 0.70 prediction means that comparable predictions correspond to an event frequency near 70% in the target setting. Biologically, it does not identify the causal mechanism of response, guarantee benefit from an intervention, or establish that the signature transports across tissues, platforms, laboratories, and patient populations.

9 Diagnostics and Sensitivity Checklist

10 Common Misinterpretations

• “An odds ratio of 2 doubles risk.” It doubles odds; the risk ratio depends on baseline risk.
• “Adjustment moved the odds ratio, proving confounding.” Non-collapsibility can move conditional and marginal odds ratios even without confounding.
• “AUC of 0.9 means probabilities are accurate.” AUC evaluates ranking, not agreement between predicted and observed risks.
• “The Hosmer–Lemeshow p-value proves fit.” A grouped test is sensitive to grouping and sample size and cannot summarize calibration, specification, and transportability.
• “Technical replicates enlarge the validation sample.” Independent biological units determine generalization and inferential precision.

11 Connections and Take-home Messages

• Logistic regression derives the Bernoulli likelihood, coefficients, estimation, and separation.
• Effect sizes compares differences, ratios, odds ratios, and standardized effects.
• Regression uncertainty and coefficient interpretation distinguishes coefficient, contrast, mean-response, and prediction uncertainty.
• Interaction terms and effect modification explains why effects depend on scale and covariate level.
• Parametric bootstrap and nonparametric bootstrap describe two different uncertainty and validation tools.

1. Odds ratios, risk ratios, risk differences, and predicted probabilities answer different questions and should be named explicitly.
2. Logistic coefficients are conditional; standardized probabilities can target a declared population-average contrast.
3. Calibration must be checked in the intended population and period, separately from discrimination.
4. Honest validation repeats the entire modeling pipeline and respects the true experimental unit.
5. Probability thresholds are decisions with consequences, not intrinsic properties of a fitted model.

12 References

• Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B, 20(2), 215–232. https://doi.org/10.1111/j.2517-6161.1958.tb00292.x
• Greenland, S., Robins, J. M., and Pearl, J. (1999). Confounding and collapsibility in causal inference. Statistical Science, 14(1), 29–46. https://doi.org/10.1214/ss/1009211805
• Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2
• Steyerberg, E. W., Vickers, A. J., Cook, N. R., et al. (2010). Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology, 21(1), 128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2
• Van Calster, B., Nieboer, D., Vergouwe, Y., et al. (2016). A calibration hierarchy for risk models was defined: from utopia to empirical data. Journal of Clinical Epidemiology, 74, 167–176. https://doi.org/10.1016/j.jclinepi.2015.12.005
• Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., and Steyerberg, E. W. (2019). Calibration: the Achilles heel of predictive analytics. BMC Medicine, 17, 230. https://doi.org/10.1186/s12916-019-1466-7

2 Adjusted Means Are Model-based Predictions

Let the fitted model have coefficient vector . An adjusted mean for group at covariate value is

An adjusted contrast is therefore

Its estimated variance is

“Adjusted mean” is incomplete unless the analyst states where or how covariates were averaged. Equal weighting across groups targets a different population from weighting by the observed sample frequencies. Predictions at the grand mean of a covariate may describe no actual unit when the distribution is multimodal or when covariates interact.

3 The Parallel-slopes Assumption

The model above assumes the same covariate slope in every group. A model allowing effect modification is

Now the adjusted group contrast at is

Thus is only the contrast at the centering value. If an interaction is present, deleting it to recover a single ANCOVA p-value answers the wrong question. Show conditional contrasts or predictions over the region with support, and define any average contrast with explicit target-population weights.

Linearity in is also an assumption, not a property of covariance adjustment. Transformations or splines can model curvature while retaining regression-based group contrasts. Categorizing a continuous covariate usually discards information, creates artificial thresholds, and leaves residual confounding.

4 Regression-based ANOVA and Nested Models

ANOVA, ANCOVA, and ordinary regression use the same least-squares machinery. Suppose a reduced model has design matrix and a full model adds estimable columns, producing residual sums of squares and . Under the classical Gaussian linear model,

The test asks whether the added coefficients are jointly zero conditional on all terms retained in both models. This formulation makes the hypothesis explicit:

• adding all group indicators tests an adjusted group effect;
• adding group-by-covariate terms tests whether slopes differ;
• adding a block or batch factor tests that term conditional on the specified biological design;
• adding nonlinear basis terms tests the chosen departure from linearity.

Models compared by this partial test must be fitted to the same observations. If missingness changes the analysis set, a difference in residual sum of squares mixes model change with data change. Robust or cluster-aware Wald tests may be preferable when the classical variance assumptions fail.

5 Baseline Adjustment in Randomized Studies

In a randomized two-arm study with baseline measurement and follow-up , a common model is

Because treatment is randomized, chance baseline imbalance does not confound the treatment effect. Adjusting for a prognostic baseline value generally improves precision and targets the follow-up mean difference conditional on the prespecified model. It is commonly more efficient than analyzing change , which implicitly fixes the baseline coefficient at one and can amplify measurement error.

Baseline must be measured before assignment or treatment. Adjusting for a post-treatment variable can remove part of the effect, induce collider bias, or change the estimand. In observational studies, pre-exposure covariates may address measured confounding only if the adjustment set, functional forms, positivity, measurement, and missingness assumptions are defensible. More covariates are not automatically safer.

6 Imbalance, Coding, and Sums of Squares

With balanced orthogonal designs, sequential and partial decompositions coincide. With imbalance, a reported “adjusted effect” depends on the other terms in the model and on the contrast being tested. A useful workflow is:

1. draw the design and identify experimental units, randomization, blocks, and measured covariates;
2. define the target contrast and target covariate distribution;
3. encode the full mean structure, including prespecified interactions and nonlinearities;
4. verify rank, overlap, and estimability;
5. express the hypothesis as a coefficient contrast or reduced-versus-full model;
6. report predicted means and contrasts with uncertainty.

Software “Type I,” “Type II,” or “Type III” labels cannot replace steps 2 and 5. In particular, a main-effect test in the presence of interaction may average over factor levels using weights unrelated to the target population. Empty or sparse factor-by-covariate regions can turn adjusted comparisons into extrapolation.

7 Assumptions and Diagnostics

ANCOVA inherits the linear-model assumptions and adds special risks around covariate choice and overlap.

Residual diagnostics cannot prove that the adjustment set controls confounding. Conversely, a chance correlation between treatment and baseline in a randomized sample is not evidence that randomization failed. Design knowledge supplies assumptions that the observed data alone cannot verify.

8 Worked Bioinformatics Example

Suppose 48 patient-derived organoid lines are randomized to control or drug, and log-expression of a pathway at 24 hours is the outcome. Prespecified covariates are baseline pathway expression, tissue of origin, and sequencing batch.

The organoid donor line is the experimental unit. Replicate wells, libraries, and reads nested within one line are technical replication. If wells receive treatment independently but share a donor, the model must still represent donor clustering and the precise assignment scheme.

An ANCOVA contrast estimates the adjusted drug-control difference at specified baseline and tissue values, or averaged over a declared tissue and baseline distribution. A drug-by-baseline interaction would make that difference vary with baseline expression. Batch adjustment can improve precision when treatment is represented within batches; if treatment and batch are perfectly confounded, no regression can identify their separate effects.

Statistically, an interval quantifies uncertainty about the adjusted contrast under the model and design. Biologically, a nonzero contrast supports a treatment- associated change in this pathway for the sampled donor population and assay; it does not alone prove pathway mediation, clinical relevance, or transferability to patients.

9 Reporting and Connections

Report the estimand, experimental unit, design, covariate timing and selection, model formula, coding and centering, overlap, adjusted means and contrasts with their weighting scheme, intervals, joint tests, diagnostics, missing-data handling, and sensitivity to interactions, nonlinearities, influential units, and alternative justified adjustment sets.

• One-way and factorial ANOVA introduces designed group comparisons, factorial interactions, and omnibus tests.
• Multiple linear regression supplies the conditional-mean and partial-regression foundation.
• Categorical predictors and contrasts explains coding, estimability, and marginal-mean weights.
• Interaction terms and effect modification develops conditional contrasts and scale dependence.
• Linear-model assumptions and diagnostics separates mean, variance, dependence, and causal assumptions.
• Regression uncertainty and coefficient interpretation derives uncertainty for linear combinations and predictions.

1. ANCOVA is multiple regression with group indicators and covariates; adjusted means are model-based predictions.
2. Define where or over whom a contrast is averaged before interpreting it.
3. A common adjusted group effect requires an adequate parallel-slopes model; interaction makes the contrast covariate-dependent.
4. Baseline adjustment in randomized studies can improve precision, but post-treatment adjustment can change or bias the target effect.
5. Regression cannot repair absent overlap, treatment-batch confounding, or technical pseudoreplication.

10 References

• Cochran, W. G. (1957). Analysis of covariance: Its nature and uses. Biometrics, 13(3), 261–281. https://doi.org/10.2307/2527916
• Frison, L., and Pocock, S. J. (1992). Repeated measures in clinical trials: Analysis using mean summary statistics and its implications for design. Statistics in Medicine, 11(13), 1685–1704. https://doi.org/10.1002/sim.4780111304
• Vickers, A. J., and Altman, D. G. (2001). Statistics notes: Analysing controlled trials with baseline and follow up measurements. BMJ, 323, 1123–1124. https://doi.org/10.1136/bmj.323.7321.1123
• Van Breukelen, G. J. P. (2006). ANCOVA versus change from baseline: More power in randomized studies, more bias in nonrandomized studies. Journal of Clinical Epidemiology, 59(9), 920–925. https://doi.org/10.1016/j.jclinepi.2006.02.007
• Senn, S. (2006). Change from baseline and analysis of covariance revisited. Statistics in Medicine, 25(24), 4334–4344. https://doi.org/10.1002/sim.2682
• R Core Team. lm: fitting linear models; anova.lm: ANOVA tables for fitted linear models. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.html

2 One-way ANOVA as a Linear Model

With groups, the cell-means parameterization is

An equivalent regression uses an intercept and indicator columns. The coefficient values depend on the chosen coding, but fitted group means and the same estimable contrasts do not. This is why scientific conclusions should be reported as group means and contrasts rather than as unexplained “main-effect” coefficients.

For a balanced design with and total , the total corrected sum of squares decomposes as

The equality follows by writing ; the cross-product vanishes because residuals sum to zero within each group. In matrix terms, it is a projection decomposition from least-squares geometry.

Under independent Gaussian errors with common variance , the omnibus hypothesis

is tested by

The numerator measures improvement over an intercept-only model; the denominator estimates within-group noise. Rejecting the omnibus null says that the vector of means is not constant. It does not identify which differences matter, measure their size, or establish biological importance.

3 Factorial Designs and Interactions

A factorial experiment studies two or more factors together. For factors and , a two-way fixed-effects model is

The interaction term allows the contrast between levels of to vary across levels of . For two-level factors, an interaction contrast is a difference in differences:

If interaction is scientifically plausible, marginal main effects can hide the pattern. Report cell means, conditional contrasts, and an interaction plot with uncertainty. A nonsignificant interaction is not evidence that effects are identical; the design may simply be imprecise.

Factorial designs are efficient because every unit informs more than one factor. They also distinguish a biological interaction from a nuisance block or batch effect. Blocking reduces residual variation when blocks are known before randomization. Treating a block as a biological factor changes the question, so the design and estimands must be declared first.

4 Contrasts, Omnibus Tests, and Multiplicity

A contrast is a linear function

Examples include one treatment versus control, a linear dose trend, or the average of two treatments versus another pair. Prespecified contrasts often answer the scientific question more directly and with more power than an omnibus test followed by every pairwise comparison.

For fitted coefficient vector and contrast vector ,

Report , an interval, the outcome scale, and the multiplicity strategy. Tukey procedures suit all pairwise comparisons under their conditions; Dunnett procedures suit several treatments versus one control. Planned contrasts need not be delayed until an omnibus test rejects, but the full family of claims still determines the appropriate error control.

5 Balanced and Unbalanced Data

In a balanced, complete factorial design, orthogonal model terms yield the same sums of squares regardless of fitting order. With unequal cell sizes, missing cells, or correlated predictors, “the sum of squares for factor ” is not unique.

• Sequential sums of squares compare each term with those entered before it.
• Partial sums of squares compare a term after adjusting for specified other terms.
• Different software labels such comparisons Type I, II, or III, but the label is less informative than the exact nested models or contrasts tested.

Type III tests depend on parameterization and may target averages over cell combinations that are poorly represented or absent. Empty cells can make some main effects or interactions non-estimable. Inspect the design matrix, cell counts, rank, and the weighting used for marginal means. State the hypothesis as a contrast or explicit reduced-versus-full model comparison.

6 Assumptions, Diagnostics, and Alternatives

The classical fixed-effects ANOVA reference law uses independent Gaussian errors with common variance and a correctly specified mean structure.

Issue	Why it matters	Check and response
Wrong experimental unit	Degrees of freedom and standard errors become artificially small	Reconstruct assignment and nesting; aggregate or model clusters
Unequal variances	Classical pooled standard errors may be wrong	Plot group spreads; use Welch’s one-way test, variance modeling, or robust covariance when justified
Dependence	Replicates contribute less information than assumed	Use blocked, repeated-measures, cluster-aware, or mixed models
Skewness or heavy tails	Small-sample calibration and means may be unstable	Check residuals and influence; consider transformation, robust methods, or design-respecting randomization inference
Sparse or empty cells	Contrasts may rely on extrapolation or be non-estimable	Show cell counts, rank, and contrast weights; simplify claims
Many post hoc comparisons	False-positive risk increases	Prespecify priorities and control the relevant error family

ANOVA is often reasonably robust to moderate nonnormality in balanced designs, but balance does not rescue pseudoreplication, severe outliers, or dependence. Rank-based tests change the estimand and generally do not provide a drop-in factorial analysis. Permutation tests require transformations that are valid under the design and null being tested.

7 Worked Bioinformatics Design

Suppose 36 mice are randomized equally to three drugs, with sex used as a second factor and RNA-seq pathway score as the outcome. The factorial model estimates six drug-by-sex cell means. Primary estimands might be each drug-versus-control contrast averaged equally across sexes and the drug-by-sex interactions.

The mouse is the experimental unit. Two library preparations and thousands of reads from one mouse are technical observations, not extra randomized units. Sequencing lane can be included as a design or processing factor if it is not confounded with drug. If every control library was sequenced in one lane and every treated library in another, drug and lane effects cannot be separated by ANOVA.

Statistically, a drug contrast estimates a difference in mean pathway score on the declared scale. Biologically, it supports a treatment-associated change in that pathway under this organism, tissue, dose, and time point; it does not by itself demonstrate a molecular mechanism or generalize to other settings.

8 Reporting and Connections

Report the design, experimental unit, allocation, cell counts, outcome scale, model formula, prespecified contrasts, estimated means and contrasts with intervals, multiplicity adjustment, residual and influence checks, missing-data handling, and sensitivity analyses. An ANOVA table alone is not a scientific result.

• Categorical predictors and contrasts develops coding-invariant group comparisons.
• Interaction terms and effect modification explains conditional effects and effect-scale dependence.
• Linear-model assumptions and diagnostics links each regression assumption to the claim it supports.
• Regression uncertainty and coefficient interpretation derives uncertainty for coefficients, contrasts, and predictions.
• Family-wise error rate distinguishes weak and strong control over families of tests.
• ANCOVA and regression-based ANOVA adds continuous covariates and explicit nested-model comparisons.

1. ANOVA is a linear model for means and contrasts, not merely a table of sums of squares.
2. The independent experimental unit determines the information and degrees of freedom.
3. In factorial designs, interactions make effects conditional on other factors.
4. With imbalance, define hypotheses through contrasts or nested models instead of relying on an unexplained sums-of-squares type.
5. Effect estimates, intervals, diagnostics, and biological context matter more than an omnibus p-value alone.

9 References

• Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd. https://psychclassics.yorku.ca/Fisher/Methods/
• Box, G. E. P., Hunter, J. S., and Hunter, W. G. (2005). Statistics for Experimenters (2nd ed.). Wiley.
• Searle, S. R. (1971). Linear Models. Wiley.
• Langsrud, O. (2003). ANOVA for unbalanced data: Use Type II instead of Type III sums of squares. Statistics and Computing, 13, 163–167. https://doi.org/10.1023/A:1023260610025
• R Core Team. aov: fit an analysis of variance model; anova.lm: ANOVA tables for fitted linear models. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/aov.html

2 Treatment Coding and Its Interpretation

With three levels—control, drug A, and drug B—and control as the reference, treatment coding gives

The coefficients are

Thus the intercept is the control mean and each slope is a difference from control. Changing the reference level changes all three coefficient meanings, but the fitted group means remain .

Reference coding is convenient when one level is a genuine scientific baseline. It is less useful when the questions concern a dose trend, an average of several controls, or A versus B. Those questions should be stated as contrasts rather than inferred indirectly from whichever coefficients software happens to print.

3 A Contrast Is the Estimand

A linear contrast of level means is

The zero-sum condition makes the comparison invariant to adding the same constant to every level. Examples for include

Question	Contrast weights	Estimand
A versus control
B versus control
A versus B
Average drug versus control

Let be the fitted coefficient vector with covariance . If the desired comparison is , then

An interval or test follows from this estimate and standard error using a reference distribution appropriate to the design and covariance estimator. The contrast should be chosen because it answers a prespecified question, not because it produces the smallest p-value.

4 Other Coding Systems

Different full-rank codings emphasize different coordinates:

Orthogonal contrast vectors satisfy for . In a balanced one-way design, their sums of squares partition the factor sum of squares. In unbalanced or covariate-adjusted data, algebraic orthogonality of the weights does not necessarily imply uncorrelated estimates.

A factor with levels contributes degrees of freedom. A single coefficient test answers one coordinate-specific question; an omnibus degree-of-freedom test asks whether all relevant level differences are zero. A nonsignificant omnibus test does not validate every equality, and a significant omnibus test does not identify which contrast matters.

5 Adjustment Changes What Is Being Compared

With additional covariates,

factor coefficients are conditional comparisons at the same . If there are no factor-by-covariate interactions, the adjusted difference is constant over . With interactions, there is no single group difference until values or a distribution of are specified.

Estimated marginal means compute model-based predictions for each level and average them over a declared reference distribution of covariates. Equal weighting across cells, weighting by the observed sample, and weighting by a target population answer different questions. In unbalanced observational data, calling one of them “the adjusted mean” without stating the averaging rule hides part of the estimand.

Adjusted comparisons also require overlap. If a treatment occurs only in one batch, the treatment and batch columns are aliased: no coding scheme can recover their separate effects. Sparse treatment-by-tissue cells may technically permit a fit while making the requested contrast largely extrapolative.

6 Worked Example: Three Library-preparation Methods

Suppose RNA integrity is measured for independent donor samples processed by a standard method or alternatives and . The fitted adjusted means are

The planned contrast comparing the average alternative with the standard method is

That estimate does not equal the A-versus-S coefficient, the B-versus-S coefficient, or an unweighted average of all three means. Its standard error must use the covariance among the three fitted means; treating their individual standard errors as independent is generally wrong.

If the same donor material was split across methods, donor is a blocking or repeated-measures unit and the model must retain that pairing. If different donors received different methods, donors are the independent experimental units. Technical lanes, aliquots, and reads do not increase the biological replication unless the target explicitly concerns technical repeatability.

7 Diagnostics, Sensitivity, and Reporting

Before interpreting a factor or contrast:

1. Verify levels and reference meaning. Check spelling, unused levels, ordering, and whether a numeric code was accidentally treated as continuous.
2. Inspect the design matrix. Confirm rank, columns, aliases, empty cells, and the coding actually used by the software.
3. Inspect cell counts and overlap. Report independent units per relevant factor combination, not only the total record count.
4. Check the outcome model. Residual, variance, dependence, and influence diagnostics still apply; factor coding does not repair misspecification.
5. Define the contrast family. Prespecify the comparisons and use simultaneous intervals or multiplicity adjustment when many claims form one family.
6. Test coding invariance. Refit with another full-rank coding and confirm that fitted values and explicitly reconstructed contrasts agree.
7. State the averaging population. For marginal means, document covariate values and weights.

Report the contrast weights or plain-language comparison, estimate, uncertainty interval, scale and units, covariance method, adjustment set, and multiplicity procedure. A coefficient table alone is not a complete account of a multi-level factor.

8 Connections and Take-home Messages

• Multiple linear regression defines the design-matrix model in which factor columns are fitted.
• Least squares and the geometry of regression explains rank, estimability, and covariance of linear functions.
• Regression uncertainty and coefficient interpretation develops conditional coefficients, contrasts, and prediction uncertainty.
• Interaction terms and effect modification shows why a factor effect can depend on another predictor.
• Bonferroni, Holm, and Hochberg procedures addresses multiplicity across planned or exploratory contrasts.

1. A coding scheme parameterizes group means; a contrast defines the scientific comparison.
2. Full-rank recoding changes coefficient labels, not fitted values or consistently computed estimable contrasts.
3. In unbalanced data, adjusted or marginal means depend on an explicit covariate distribution and weighting rule.
4. Empty cells, aliasing, and poor overlap are design problems that recoding cannot solve.
5. Biological inference follows the number and structure of independent samples, not the number of technical observations.

9 References

• Fisher, R. A. (1935). The Design of Experiments. Oliver and Boyd.
• Searle, S. R., Speed, F. M., and Milliken, G. A. (1980). Population marginal means in the linear model: an alternative to least squares means. The American Statistician, 34(4), 216–221. https://doi.org/10.1080/00031305.1980.10483031
• Schad, D. J., Vasishth, S., Hohenstein, S., and Kliegl, R. (2020). How to capitalize on a priori contrasts in linear (mixed) models: a tutorial. Journal of Memory and Language, 110, 104038. https://doi.org/10.1016/j.jml.2019.104038
• Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. https://doi.org/10.1017/9781139161879
• R Core Team. contrasts: get and set contrast matrices, and model.matrix: construct design matrices. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/contrasts.html
• Lenth, R. V. emmeans: estimated marginal means. R package documentation. https://rvlenth.github.io/emmeans/

2 The Product-term Model

For predictors and , the linear conditional-mean model

has conditional slope

Thus is the change in the slope for a one-unit increase in . The “main effect” is the slope specifically at , while is the slope at . They are not average effects unless the coding and design make them so.

Centering at a meaningful value makes the slope at . It changes the lower-order coefficients but not fitted values or the interaction coefficient when the same linear scale is used. Centering can improve interpretability; it cannot create overlap or remove genuine collinearity between the scientific quantities of interest.

3 Categorical Interactions Are Differences in Differences

Let and be binary indicators. The model

implies four conditional means:

		Conditional mean
0	0
1	0
0	1
1	1

The interaction is the difference in differences:

It asks whether the A difference at differs from the A difference at . Reversing the order gives the same additive interaction. With factors having more than two levels, an interaction is a set of contrasts and should usually be assessed through scientifically chosen comparisons or an omnibus multi-degree-of-freedom test rather than one arbitrarily selected coefficient.

4 Scale Determines the Interaction

In a generalized linear model,

describes interaction on the link scale.

• In a linear model with identity link, it is departure from additivity of means.
• In a log-link model, it is departure from multiplicativity of means or risks.
• In logistic regression, it is departure from multiplicativity of odds.

A zero product coefficient in logistic regression does not imply constant risk differences or risk ratios. Conversely, heterogeneous odds ratios can coexist with nearly constant risk differences. For binary outcomes, report predicted risks for joint exposure groups and contrasts on scientifically relevant scales; do not let the default link silently define “no interaction.”

For two binary exposures, additive interaction in risks can be summarized by

when risk ratios use the unexposed group as reference. means no departure from additivity of risk relative to baseline. This is a different null from a product-term coefficient of zero in a log or logit model.

5 Statistical Interaction and Causal Effect Modification

An observed conditional association is not automatically a causal effect. A causal effect modifier is defined relative to an intervention contrast, outcome scale, covariate set, and target population. For example, an additive conditional average treatment effect at is

where is the potential outcome under exposure . Identifying from observational data requires consistency, conditional exchangeability, positivity within modifier strata, and an adequate measurement and sampling process. A regression product term supplies a functional form; it does not supply those assumptions.

Effect modification is population dependent. Stratum-specific effects may be stable while a marginal effect changes because the distribution of baseline risk or other covariates changes. Noncollapsible measures such as the odds ratio can also change between conditional and marginal analyses without confounding. Specify whether the target is conditional or marginal and how standardization to the target population is performed.

6 Estimation and Uncertainty

Suppose the fitted covariance matrix of is . The estimated slope at is

with variance

Plotting or predicted outcomes with pointwise or simultaneous uncertainty is usually clearer than reporting only the p-value for . Inference must still respect heteroskedasticity, clustering, repeated measures, and the number of independent experimental units.

Interaction tests often have low power because the modifier splits the available information and the product term is correlated with lower-order terms. Absence of statistical significance is not evidence of homogeneous effects. Report effect estimates and intervals in prespecified strata, and design studies around the interaction contrast rather than the easier main-effect calculation.

7 The Hierarchy Principle and Model Specification

When is included, retain and unless a scientifically justified parameterization deliberately omits them. Dropping lower-order terms imposes constraints such as equality of groups at ; it does not merely simplify the model. Keeping hierarchy also makes the model invariant to shifts of the predictor origin.

Product terms encode a linear change in a linear slope. They may be inadequate when either predictor has a nonlinear effect. A flexible model can include spline-by-group interactions, but flexibility increases data requirements and the risk of discovering unstable subgroups. Choices of cutpoints, modifiers, transformations, and outcome scales should be prespecified or transparently treated as exploratory.

8 Worked Bioinformatics Example

Suppose normalized expression of a pathway score is measured in independent patient biopsies. Treatment is randomized, tissue type is measured at baseline, and the model includes treatment, tissue, their interaction, and prespecified batch covariates.

If fitted treatment differences are log-expression units in tissue 0 and in tissue 1, then the additive interaction estimate is

Statistically, the data suggest that the treatment contrast on this scale is smaller in tissue 1. Biologically, that pattern might motivate a tissue-specific mechanism, but the interaction alone does not prove one. Tissue-specific assay behavior, batch imbalance, compositional shifts, or sparse treatment-by-tissue cells may produce the same pattern.

The patient is the experimental unit. Multiple cells, spots, images, or technical runs from one biopsy refine its measurement but do not become independent replicates of the treatment-by-tissue interaction. A cell-level model must retain patient clustering or aggregate to justified sample-level summaries. If tissue was not randomized, causal claims about tissue and mechanistic synergy require additional assumptions beyond treatment randomization.

9 Diagnostics and Sensitivity

An interaction analysis should examine more than the product-term p-value:

Dichotomizing a continuous modifier discards information, creates an arbitrary threshold, and can make subgroup conclusions unstable. Prefer a continuous functional form and show predictions over regions with adequate data. Avoid claims that an effect exists in one subgroup but not another merely because one subgroup p-value crosses 0.05 and the other does not; the relevant test compares the subgroup effects directly.

10 Reporting the Scientific Result

Report:

1. the outcome scale and effect measure;
2. the modifier, coding, reference values, and target population;
3. joint cell counts or the range of continuous-modifier support;
4. predicted outcomes and conditional effects with intervals at meaningful values;
5. the interaction contrast with uncertainty, not two separate significance labels;
6. adjustment, clustering, multiplicity, and exploratory selection procedures;
7. sensitivity to functional form, influential units, and alternative effect scales.

The result should say “the estimated treatment difference changed by … on the … scale,” rather than vaguely claiming that two variables “interact.”

11 Connections and Take-home Messages

• Categorical predictors and contrasts shows how group comparisons become estimable linear functions.
• Multiple linear regression provides the conditional-mean and design-matrix foundation.
• Regression uncertainty and coefficient interpretation derives uncertainty for linear combinations of coefficients.
• Linear-model assumptions and diagnostics covers functional-form, leverage, dependence, and extrapolation checks.
• Statistical power and sample-size calculation explains why the intended interaction must drive design planning.

1. An interaction is a departure from additivity on a declared model or effect scale; it is not an intrinsic property of two variables.
2. Lower-order coefficients in a product-term model are conditional effects at the other predictor’s reference value.
3. Interpret and plot conditional effects or predicted outcomes, with uncertainty, rather than relying on a product-term p-value.
4. Causal effect modification requires design and identification assumptions that regression algebra cannot provide.
5. Sparse joint cells, data-driven subgroup searches, and technical pseudoreplication make interaction estimates especially unstable.

12 References

• Cox, D. R. (1984). Interaction. International Statistical Review, 52(1), 1–31. https://doi.org/10.2307/1403235
• VanderWeele, T. J. (2009). On the distinction between interaction and effect modification. Epidemiology, 20(6), 863–871. https://doi.org/10.1097/EDE.0b013e3181ba333c
• Knol, M. J., and VanderWeele, T. J. (2012). Recommendations for presenting analyses of effect modification and interaction. International Journal of Epidemiology, 41(2), 514–520. https://doi.org/10.1093/ije/dyr218
• VanderWeele, T. J., and Knol, M. J. (2014). A tutorial on interaction. Epidemiologic Methods, 3(1), 33–72. https://doi.org/10.1515/em-2013-0005
• Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. https://doi.org/10.1017/9781139161879
• R Core Team. formula: model formulae and model.matrix: construct design matrices. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/formula.html

2 The Assumptions, One by One

3 Residuals and Leverage

Let

The diagonal is the leverage of unit . It depends only on the design matrix and measures how unusual the predictor configuration is relative to the others. Because , average leverage is .

Raw residual variances are unequal even under the homoskedastic model:

An internally studentized residual is therefore

Externally studentized residuals replace with an estimate computed after deleting unit , making them more useful for judging an unusual response. A large residual means poor vertical fit; high leverage means an unusual predictor pattern. Influence requires the combination: an observation far out in predictor space can materially move the fit when its response disagrees with the trend.

Cook’s distance summarizes the change in all fitted values after deleting unit :

There is no universal cutoff that turns an observation into an error. Leverage, studentized residuals, Cook’s distance, DFBETAs, and deletion fits are prompts for record verification and sensitivity analysis, not automatic deletion rules.

4 A Diagnostic Sequence

Diagnostics work best as a structured comparison between observed data and the model’s implications.

1. Verify design and units. Confirm randomization, nesting, repeated measures, batches, missingness, coding, and which units are independent.
2. Inspect predictors before fitting. Check ranges, sparse categories, impossible combinations, overlap, rank, and potential extrapolation.
3. Check the mean structure. Plot residuals against fitted values, continuous predictors, time or order, and scientifically important groups. Add a smooth trend as a diagnostic, not as proof of a replacement model.
4. Check conditional spread. Look for funnels, group-specific variance, and variance related to expected measurement scale.
5. Check distributional shape when it matters. A normal quantile–quantile plot is relevant to exact small-sample inference and tail behavior, not to the existence of the OLS estimate.
6. Assess influence. Examine leverage, externally studentized residuals, Cook’s distance, and coefficient changes under justified deletion checks.
7. Refit scientifically plausible alternatives. Compare transformations, curvature, interactions, variance models, cluster-aware uncertainty, and robust procedures.
8. Validate the intended use. For prediction, assess held-out calibration and error. For explanation, check whether the coefficient still answers the prespecified estimand.

Formal tests of normality or heteroskedasticity can supplement this workflow, but they do not replace plots and design reasoning. With small , they have little power; with very large , they can flag negligible deviations. Repeatedly changing a model until every diagnostic test is nonsignificant also makes the reported inference conditional on an unreported selection process.

5 Reading Common Diagnostic Displays

The same visual pattern can have several causes. A funnel may reflect true heteroskedasticity, an inappropriate response scale, or a mean–variance relation requiring a generalized linear model. Diagnostics localize tension; they do not identify a unique repair.

6 Worked Bioinformatics Example

Suppose normalized gene expression is measured for 48 independent patient specimens, with treatment, age, and sequencing batch as predictors. Each specimen was sequenced in two lanes and lane counts were combined before analysis.

• The 48 patients—not the 96 lanes or millions of reads—are the experimental units for treatment inference.
• A residual shift by batch suggests incomplete technical adjustment or an interaction, while a residual trend with fitted expression suggests mean or variance misspecification.
• One high-leverage patient may combine an age and treatment pattern scarcely represented elsewhere. Its influence reveals weak support for that conditional comparison; deletion cannot create the missing overlap.
• If residual variance increases with mean expression, robust standard errors may protect coefficient inference asymptotically, but a count model or mean–variance model may better represent the measurement process.

Statistically, diagnostics ask whether the coefficient and its uncertainty are stable under credible alternatives. Biologically, they do not establish that a gene-expression association is causal, functionally important, or transferable to another tissue or population.

7 What Diagnostics Cannot Prove

Residual checks use observed variables and the fitted sample. They cannot prove:

• absence of unmeasured confounding or selection bias;
• independence when the sampling hierarchy was discarded;
• correct measurement of the response and predictors;
• validity outside the observed predictor range or target population;
• biological importance of a statistically precise coefficient.

A satisfactory residual plot means only that the chosen diagnostic did not expose that kind of discrepancy at its available resolution. It is not a certificate that the model is true.

8 Connections and Take-home Messages

• Multiple linear regression defines the conditional-mean model and coefficient comparisons being diagnosed.
• Least squares and the geometry of regression derives projection, residual orthogonality, leverage, and rank.
• Regression uncertainty and coefficient interpretation separates coefficient, contrast, mean-response, and prediction uncertainty.
• Robust standard errors and robust regression distinguishes covariance protection from resistance to influential outcomes.
• Sensitivity analysis and robustness organizes conclusions across plausible assumptions and analysis choices.

1. State the inferential goal before deciding which assumptions matter.
2. Residuals diagnose the fitted mean and error structure but cannot establish exogeneity, causality, or correct replication.
3. Heteroskedasticity, dependence, nonlinearity, and influence are different problems and require different responses.
4. Investigate influential observations transparently; never delete them solely because a generic threshold was crossed.
5. In bioinformatics, preserving the biological experimental unit is more important than the nominal number of technical measurements.

9 References

• Belsley, D. A., Kuh, E., and Welsch, R. E. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley. https://doi.org/10.1002/0471725153
• Cook, R. D. (1977). Detection of influential observations in linear regression. Technometrics, 19(1), 15–18. https://doi.org/10.1080/00401706.1977.10489493
• Weisberg, S. (2014). Applied Linear Regression (4th ed.). Wiley.
• Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. https://doi.org/10.1017/9781139161879
• R Core Team. lm.influence: regression diagnostics. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.influence.html

2 Heteroskedasticity-robust Covariance

Consider

with independent errors but unequal conditional variances . OLS remains

Its conditional covariance is

where . The heteroskedasticity-consistent estimator replaces unknown variances with functions of the fitted residuals:

The outer matrices are the bread and the residual cross-products are the meat, hence “sandwich covariance.” Under appropriate moment, independence, rank, and leverage conditions, it consistently estimates the sampling covariance without specifying a parametric variance function.

2.1 HC0 through HC3

The variants differ in finite-sample residual corrections:

Estimator	Residual multiplier	Interpretation
HC0		White’s basic large-sample estimator
HC1		Degrees-of-freedom inflation
HC2		Corrects residual shrinkage using leverage
HC3		Stronger leverage correction, related to leave-one-out ideas

HC0 can be too optimistic in small samples because OLS residuals are shrunken, especially at high-leverage points. HC2 or HC3 is often preferable when the number of independent units is not large. None supplies exact finite-sample inference in general; the reference distribution and degrees of freedom still need justification.

3 What Robust Standard Errors Assume

Heteroskedasticity-consistent inference is not assumption-free. It still needs:

• an estimand defined by an adequate mean model or linear projection;
• zero conditional mean for an unbiased conditional-mean interpretation;
• independent observations for the usual HC estimator;
• a nonsingular, sufficiently stable design with no asymptotically dominant unit;
• enough independent units for the sandwich approximation and tail calibration.

If errors are correlated within independent clusters, the cluster sandwich uses cluster score sums:

with a suitable small-sample correction. Precision is driven primarily by the number and balance of independent clusters, not the total number of nested records. With few clusters, ordinary cluster-robust tests may be unreliable; design-based methods, small-sample corrections, or a justified hierarchical model may be needed.

4 Robust Regression Changes the Estimate

Least squares minimizes a quadratic loss:

Because squared loss grows rapidly, one extreme response can dominate the fit. An M-estimator instead solves

where is a robust residual scale. Huber’s loss is quadratic near zero and linear in the tails:

Its derivative, , bounds the contribution of a large standardized residual. The estimating equation is

Iteratively reweighted least squares expresses this as repeated weighted fits with

using at . Small residuals receive nearly full weight; large residuals receive less. The tuning constant trades Gaussian efficiency against tail resistance and should be reported.

5 Resistance Is Not Universal

Huber M-estimation limits the effect of large vertical residuals, but a point with extreme predictors can have high leverage and pull the fitted line toward itself, leaving a deceptively small residual. Robust regression therefore also needs design-space diagnostics. High-breakdown regression procedures jointly protect against more severe contamination, but can sacrifice efficiency, require tuning, and behave poorly when the uncontaminated design has weak support.

Downweighting is not evidence that an observation is erroneous. A discordant unit may reveal a real subgroup, a missing interaction, a batch effect, or a different data-generating regime. Report which units receive low weight, verify their provenance, and compare scientifically plausible models.

6 A Small Example

Suppose five specimens have

The last response is far above the trend of the first four. OLS gives and , so every fitted value is pulled by that unit. HC3 standard errors would acknowledge unstable OLS uncertainty but would leave those two coefficient estimates unchanged. Huber regression would downweight the large residual and estimate a line closer to the majority trend.

Neither response decides whether the last specimen is a recording error or a real biological subtype. The defensible analysis checks the record, examines its predictor leverage and batch, fits prespecified alternatives, and reports how the scientific contrast changes.

7 Choosing a Response to Model Tension

8 Uncertainty for Robust Regression

Robust-regression coefficients require their own sampling covariance calculation. A sandwich estimator can be built from the M-estimating equations, but it is not the same calculation as attaching HC standard errors to OLS. A bootstrap can be useful when it resamples independent experimental units and repeats every fitting and tuning step. Resampling individual cells or reads when inference concerns patients recreates pseudoreplication rather than uncertainty.

With small samples, heavy contamination, high leverage, or data-driven tuning, asymptotic normal intervals can be inaccurate. Simulation under scientifically plausible contamination, variance, and clustering patterns is a useful sensitivity check.

9 Bioinformatics Design and Interpretation

Consider protein abundance measured in multiple technical runs for 40 patient specimens. Treatment is the predictor of interest, while age and batch are adjustment variables.

• If variability differs by batch but patients are independent, HC3 may improve uncertainty for the OLS treatment contrast.
• If several observations come from each patient, uncertainty must recognize patient clustering; treating runs as independent makes either classical or HC standard errors too small.
• If one patient has a large residual, robust regression measures whether the fitted treatment contrast is driven by that response. It does not justify silently removing the patient.
• If treatment and batch barely overlap, neither method restores the missing comparison. The design—not the residual loss—is the limiting factor.

The statistical conclusion concerns a treatment coefficient on the chosen abundance scale and under the specified design. Biological interpretation also depends on assay reliability, pathway context, effect magnitude, tissue and population transportability, and whether the association represents an intervention effect.

10 Reporting Checklist

Report enough detail to identify both the estimator and its uncertainty:

1. experimental unit, clustering level, and number of independent units or clusters;
2. mean model, coefficient estimand, and whether OLS or robust regression produced the point estimate;
3. HC variant or cluster correction, degrees of freedom, and reference distribution;
4. robust loss, scale estimator, tuning constant, and convergence information;
5. leverage, residual, and influence findings without automatic deletion;
6. sensitivity of effect estimates, intervals, and scientific conclusions to credible alternatives.

Calling a result merely “robust” hides the most important information.

11 Connections and Take-home Messages

• Linear-model assumptions and diagnostics distinguishes mean, variance, dependence, rank, and distributional conditions.
• Regression uncertainty and coefficient interpretation defines coefficient covariance, contrasts, and prediction uncertainty.
• Standard errors develops sandwich and cluster-aware uncertainty more generally.
• Sensitivity analysis and robustness separates robust procedures from robustness of an entire scientific conclusion.
• When asymptotic approximations fail explains why a large nominal record count may provide little independent information.

1. Robust standard errors change uncertainty, not OLS coefficients.
2. Robust regression changes the point estimate by limiting the influence of large residuals.
3. HC covariance requires independent units; clustered data require a dependence-aware covariance or model.
4. Neither method repairs a wrong mean, confounding, poor overlap, or a mistaken experimental unit.
5. Report the exact estimator, covariance variant, tuning choices, diagnostics, and sensitivity—not simply that the analysis was robust.

12 References

• White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817–838. https://doi.org/10.2307/1912934
• MacKinnon, J. G., and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 29(3), 305–325. https://doi.org/10.1016/0304-4076(85)90158-7
• Huber, P. J. (1964). Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1), 73–101. https://doi.org/10.1214/aoms/1177703732
• Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J., and Stahel, W. A. (1986). Robust Statistics: The Approach Based on Influence Functions. Wiley. https://doi.org/10.1002/9781118186435
• Zeileis, A. (2004). Econometric computing with HC and HAC covariance matrix estimators. Journal of Statistical Software, 11(10), 1–17. https://doi.org/10.18637/jss.v011.i10
• Zeileis, A., Köll, S., and Graham, N. (2020). Various versatile variances: An object-oriented implementation of clustered covariances in R. Journal of Statistical Software, 95(1), 1–36. https://doi.org/10.18637/jss.v095.i01
• R Core Team. lm.influence: regression diagnostics. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.influence.html
• Zeileis, A. vcovHC: heteroskedasticity-consistent covariance matrix estimation. sandwich documentation. https://sandwich.r-forge.r-project.org/reference/vcovHC.html

2 Normal equations and orthogonal projection

Write the residual vector as . Expanding the residual sum of squares and differentiating gives

At a minimum,

Thus the residual is orthogonal to every design column. If has full column rank,

The fitted vector is

The hat matrix is symmetric and idempotent: and . These are the defining algebraic properties of an orthogonal projector. The residual-maker is also symmetric and idempotent, with .

Because the fitted and residual vectors are orthogonal,

With an intercept, the same geometry applied after centering yields the familiar decomposition

This identity is exact in the fitted sample. It does not show that the mean function is correct, that the predictors are causal, or that the model will predict new observations well.

3 What full rank means

Full column rank means that no design column is an exact linear combination of the others. It gives a unique coefficient vector. If rank is deficient, fitted values can still be unique while individual coefficients are not: several vectors produce the same point in .

Common causes include an intercept plus indicators for every category, a batch indicator identical to treatment, or more fitted coefficients than independent units. Near-linear dependence does not destroy uniqueness, but it creates long, flat directions in the least-squares surface. Small changes in the data can then produce large coefficient changes.

The projection should not be computed by explicitly forming . Forming the cross-product squares the condition number. QR factorization is the standard stable route for a full-rank problem; singular-value decomposition exposes rank and ill-conditioned directions directly.

4 Residualization and the Frisch–Waugh–Lovell result

Partition the design as , and let

The coefficient of in the full regression is

To see the result, minimize over for any fixed . The remaining residual is ; differentiating its squared length with respect to gives the displayed expression.

Geometrically, first remove from both and the components explained linearly by . The slope between those two residual vectors is the full-model coefficient. A coefficient therefore uses only predictor variation orthogonal to the other design columns. If almost no such variation remains, the denominator is small and the estimate is unstable.

5 Leverage and influence in the projection

The diagonal element of is the leverage of unit :

High leverage means that a row of the design is unusual in predictor space. It does not by itself imply a large residual or an error. Influence combines leverage with outcome disagreement: a high-leverage unit whose response follows the fitted pattern may change the coefficients little, whereas a high-leverage unit with a large residual can dominate them.

Under homoskedastic errors,

so raw residuals have different variances even when the original errors do not. Studentized residuals account for this geometric fact. Cook’s distance and DFBETAs then quantify how deleting a unit changes the fitted vector or a particular coefficient. Deletion diagnostics are sensitivity analyses, not automatic rules for discarding inconvenient observations.

6 Weighted geometry

If the conditional error covariance is known up to scale as , generalized least squares minimizes

This is ordinary projection after whitening by a matrix satisfying . Weighted least squares is the diagonal special case. The choice of weights changes the metric and therefore the projection; weights should encode a justified variance or sampling model, not merely shrink observations that fit poorly.

7 A small worked geometry

Fit an intercept and a centered predictor to . Orthogonality of the intercept and centered predictor gives

Hence and . Direct calculation confirms and . The normal equations are visible as right angles.

8 Bioinformatics design example

Suppose 48 independent donor samples are assayed for a pathway score. The design contains an intercept, treatment, age, sex, and sequencing batch. Each column is a direction in donor-level observation space. The adjusted treatment coefficient uses only the treatment direction remaining after its projection on the other columns.

If all treated donors were processed in batch B and all controls in batch A, treatment and batch point in the same design direction. Their separate effects are not identified. More reads or repeated libraries cannot create the missing donor-level contrast. If treatment and batch are merely highly associated, the contrast exists but is short, so uncertainty is large and estimates are sensitive to a few donors.

The statistical statement is about an adjusted mean contrast on the pathway-score scale under the model. Biological interpretation additionally depends on donor selection, tissue composition, assay normalization, effect magnitude, and external replication.

9 Assumptions and sensitivity checks

10 Connections and take-home messages

• Simple linear regression introduces the one-predictor model.
• Multiple linear regression develops adjusted coefficients and omitted-variable reasoning.
• Regression uncertainty and coefficient interpretation connects this geometry to standard errors, contrasts, and prediction.
• Standard errors explains repeated-sampling uncertainty more generally.
• Sensitivity analysis and robustness provides a broader framework for assumption checks.

1. OLS projects the response onto the column space of the design matrix.
2. The normal equations say that the residual is orthogonal to every included design direction.
3. Full rank identifies coefficients; small singular values reveal unstable directions even when a unique solution exists.
4. Partial-regression geometry shows exactly which predictor variation estimates an adjusted coefficient.
5. The geometry is a sample identity. Scientific interpretation and valid uncertainty still require assumptions about design, errors, and experimental units.

11 References

• Frisch, R., and Waugh, F. V. (1933). Partial time regressions as compared with individual trends. Econometrica, 1(4), 387–401. https://doi.org/10.2307/1907330
• Lovell, M. C. (1963). Seasonal adjustment of economic time series and multiple regression analysis. Journal of the American Statistical Association, 58(304), 993–1010. https://doi.org/10.1080/01621459.1963.10480682
• Golub, G. H., and Van Loan, C. F. (2013). Matrix Computations (4th ed.). Johns Hopkins University Press.
• Weisberg, S. (2014). Applied Linear Regression (4th ed.). Wiley.
• R Core Team. lm: Fitting linear models. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.html

2 Sampling uncertainty under the classical model

For

the OLS estimator is

If, conditional on the observed design, , then

Estimate by

where is the rank of the design, including the intercept when present. The standard error for coefficient is the square root of diagonal element of .

With independent normal errors, the interval

has exact conditional coverage under the model. Without normality, similar intervals are generally asymptotic and need regularity conditions. A 95% confidence interval is a procedure with 95% long-run coverage under those conditions; it is not a 95% probability statement about a fixed frequentist coefficient after the interval is observed.

3 Contrasts are often the real estimands

Scientific questions commonly concern a linear contrast , not one raw coefficient. Its estimate and classical variance are

Examples include a difference between two factor levels, an average treatment contrast across prespecified covariate values, or the slope at a chosen modifier value in an interaction model. Estimability requires that the contrast be determined by the design; software cannot recover a treatment-versus-batch contrast when treatment and batch are perfectly confounded.

Simultaneously examining many coefficients or choosing a contrast after seeing the estimates changes the relevant error rate. Prespecified contrasts, simultaneous intervals, multiplicity adjustment, or honest exploratory labeling should match the analysis goal.

4 Coding, scaling, and interactions

Coefficient values depend on parameterization even when fitted values do not.

• Replacing age in years by age in decades multiplies its coefficient and standard error by 10 but leaves its statistic and fitted values unchanged.
• Changing the reference level of a factor changes individual coefficients, not the modeled group means or appropriate contrasts.
• Centering a predictor changes the intercept and, with interactions, changes which conditional slope is labeled a “main effect.”

For

the slope of at is

Thus is the slope only at . Interpreting it as an overall effect is wrong unless zero is the intended reference value and a constant slope is the question. Report fitted contrasts across scientifically meaningful values when an interaction is present.

5 Robust covariance: what it fixes and what it does not

Under independent but heteroskedastic errors with covariance , the conditional covariance is

Heteroskedasticity-consistent estimators replace the unknown error variances by functions of squared residuals. HC2 and HC3 adjust for leverage and often behave better than the original HC0 estimator in modest samples. These methods change the estimated covariance, not the OLS coefficients.

Robust standard errors do not repair a wrong mean function, omitted-variable bias, dependence between units, measurement error, selection bias, or poor overlap. Cluster-robust covariance needs enough independent clusters and a scientifically correct clustering level. When there are few clusters, ordinary large-sample reference distributions can be unreliable.

6 Uncertainty for a mean and for a new observation

At a predictor vector , the fitted conditional mean is . Under the classical model,

A confidence interval for the mean response uses this variance. A prediction interval for one future response must also include irreducible outcome variation:

The prediction interval is therefore wider. Both formulas are model-based and refer to predictor values supported by the design. Extrapolation can produce numerical intervals while requiring untestable assumptions far beyond the data. Prediction for a new donor also differs from predicting another technical measurement from an already observed donor because the sources of variation are different.

7 Association, prediction, and causation

The same fitted coefficient can serve different goals:

• Description: summarize conditional associations in the observed or target population.
• Prediction: combine predictors to forecast outcomes for new units.
• Causal estimation: estimate a contrast under hypothetical interventions.

Causal interpretation requires design and identification assumptions beyond the regression equation: consistency, exchangeability after justified adjustment, positivity, and a well-defined intervention are typical requirements. Adding every available covariate is not a substitute. Adjustment for a confounder can reduce bias; adjustment for a mediator changes the target; adjustment for a collider can create bias.

For prediction, unbiased causal interpretation may be irrelevant, but out-of-sample validation, calibration, and transport to the deployment population become central. For description, the target population and scale must still be specified.

8 Worked bioinformatics interpretation

Suppose 80 independent patient biopsies are assigned to treatment or control and processed in four balanced sequencing batches. For log-expression , fit

If with a 95% confidence interval , the statistical interpretation is an estimated adjusted mean difference of 0.40 on the chosen log-expression scale, with an interval produced by a procedure whose coverage depends on the model and design assumptions. On a natural-log scale, is a ratio of geometric means under the log-linear interpretation; it is not automatically a 49% change for every patient.

The experimental units are the 80 patients. Technical libraries or millions of reads improve measurement but do not increase the treatment replication to more than 80. Statistical evidence does not establish biological importance: pathway context, absolute expression, tissue heterogeneity, a prespecified meaningful effect, and replication in new patients remain relevant.

9 Diagnostics and sensitivity

Uncertainty statements should be paired with effect estimates on meaningful scales, the number of independent units, the fitting and covariance method, and the exact contrast. A p-value alone leaves the target and magnitude unclear.

10 Connections and take-home messages

• Least squares and the geometry of regression explains projection, partial regression, and leverage.
• Multiple linear regression introduces adjusted coefficients and omitted-variable bias.
• Confidence intervals and standard errors develop the general inferential concepts.
• Effect sizes separates magnitude from evidence strength.
• Statistical versus practical significance connects intervals to scientifically meaningful thresholds.

1. Define the target population and estimand before interpreting a coefficient.
2. A coefficient is a conditional comparison determined by coding, scale, other predictors, and any interactions.
3. Standard errors quantify estimator variability under assumptions; they do not measure confounding, external validity, or biological importance.
4. Confidence intervals for mean responses and prediction intervals for new observations answer different questions.
5. Robust covariance addresses some variance misspecification, not a flawed data-generating model or nonindependent experimental units.

11 References

• White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817–838. https://doi.org/10.2307/1912934
• MacKinnon, J. G., and White, H. (1985). Some heteroskedasticity-consistent covariance matrix estimators with improved finite sample properties. Journal of Econometrics, 29(3), 305–325. https://doi.org/10.1016/0304-4076(85)90158-7
• Freedman, D. A. (2006). On the so-called “Huber sandwich estimator” and “robust standard errors.” The American Statistician, 60(4), 299–302. https://doi.org/10.1198/000313006X152207
• Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. https://doi.org/10.1017/9781139161879
• Weisberg, S. (2014). Applied Linear Regression (4th ed.). Wiley.
• R Core Team. predict.lm: Predict method for linear model fits. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/predict.lm.html

2 What a Coefficient Means

Holding the other modeled predictors fixed,

when the mean is linear and is continuous. Thus is the difference in conditional mean per one-unit change in , at common values of the other predictors. For a binary indicator, it is the adjusted mean contrast between the two encoded groups.

“Holding fixed” is a mathematical comparison, not necessarily a feasible intervention. If two covariates cannot vary independently in the target population, the coefficient may depend heavily on extrapolation. If a predictor is a confounder, adjustment may reduce bias; if it is a mediator or collider, adjustment may instead change or bias the target. Predictor selection must follow the scientific question and data-generating process, not an automated p-value search.

Without interaction terms, the model asserts that the slope for is the same at every modeled value of the other predictors. With an interaction ,

so the main-effect coefficient is the slope when . Centering can make that reference value scientifically meaningful without changing fitted values.

3 Least Squares and Projection

OLS minimizes

Differentiation gives the normal equations

If has full column rank,

Geometrically, the fitted vector is the orthogonal projection of onto the column space of , where

The residual vector is orthogonal to every included design column: . These sample identities do not prove that omitted causes are unrelated to included predictors.

The Frisch–Waugh–Lovell result gives an especially useful interpretation. Regress on the other predictors and retain its residuals; regress on those same predictors and retain its residuals. The slope between the two sets of residuals equals . A coefficient therefore uses only the variation in that is not linearly explained by the other design columns.

4 Identification and Sampling Uncertainty

Full column rank is an identification condition. Exact linear dependence—for example, an intercept plus indicators for every category—means that multiple coefficient vectors produce the same fitted values. Near dependence makes coefficients estimable but unstable.

Conditional on , if

then

Estimate by

The standard error of coefficient is the square root of the corresponding diagonal element of . Conditional normal errors give exact small-sample and reference distributions. Heteroskedasticity-consistent standard errors relax constant variance but do not repair a wrong mean function, dependence, confounding, or poor overlap.

5 Omitted Variables and Changing Coefficients

Suppose the true mean includes ,

but is omitted. In a population linear projection, the short-regression slope is

Omitted-variable bias therefore needs both an association between and the outcome conditional on and an association between and . Adding a variable can legitimately change a coefficient because it changes the conditional comparison, not merely because the original model was “incorrect.”

In causal work, the right adjustment set is determined by the causal structure. In prediction, a variable may be useful without having a causal interpretation. In descriptive work, both adjusted and unadjusted associations may answer useful but different questions.

6 Worked Bioinformatics Design

Consider an RNA-expression summary for each of 60 independent patient specimens. A prespecified model is

• compares adjusted mean expression between treatment groups at the same age and batch under the model.
• is the adjusted mean difference per year of age.
• describes a batch contrast; it is not a biological effect.

The experimental units are the 60 patients. Multiple lanes, reads, or technical replicates improve measurement or help estimate technical variation but do not turn the study into a larger patient sample. If treatment is completely confounded with batch, the design matrix cannot distinguish from ; no regression algorithm can recover information absent from the design.

Statistically, an estimated treatment coefficient and interval summarize an adjusted contrast on the chosen expression scale. Biologically, interpretation also requires the gene’s function, effect magnitude, normalization, tissue composition, and whether the specimens represent the target population.

7 Assumptions, Diagnostics, and Sensitivity

A useful diagnostic sequence is: verify design and units; inspect missingness and overlap; visualize residuals; examine leverage and influence; compare conventional and robust uncertainty; test scientifically plausible nonlinearities and interactions; and report how conclusions change. Diagnostics should be tied to a failure mechanism, not used as a ritual pass/fail checklist.

Variable selection after looking at the same outcome changes the sampling behavior of coefficients and p-values. If selection is unavoidable, separate exploration from confirmation, validate out of sample, and acknowledge post-selection uncertainty.

8 Connections and Take-home Messages

• Simple linear regression develops the one-predictor case and basic prediction intervals.
• Likelihood and log-likelihood connects Gaussian regression to likelihood-based estimation.
• Sensitivity analysis and robustness provides a framework for checking dependence on assumptions.
• Selection bias and winner’s curse explains why data-dependent reporting can inflate coefficients.

1. A multiple-regression coefficient is a conditional comparison given the other modeled predictors.
2. Least squares projects the response onto the design space; full column rank is required for unique coefficients.
3. Adjustment changes the estimand and can reduce, create, or leave bias depending on the data-generating structure.
4. Robust standard errors address variance misspecification, not confounding, dependence, lack of overlap, or an incorrect mean model.
5. In bioinformatics, independent biological specimens determine inferential replication; technical observations do not substitute for them.

9 References

• Frisch, R., and Waugh, F. V. (1933). Partial time regressions as compared with individual trends. Econometrica, 1(4), 387–401. https://doi.org/10.2307/1907330
• Lovell, M. C. (1963). Seasonal adjustment of economic time series and multiple regression analysis. Journal of the American Statistical Association, 58(304), 993–1010. https://doi.org/10.1080/01621459.1963.10480682
• White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817–838. https://doi.org/10.2307/1912934
• Weisberg, S. (2014). Applied Linear Regression (4th ed.). Wiley.
• Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. https://doi.org/10.1017/9781139161879
• R Core Team. lm: Fitting linear models. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.html

2 Least Squares

Ordinary least squares (OLS) chooses the line that minimizes the residual sum of squares

Write and . Differentiating RSS with respect to and gives the normal equations. Their solution is

Thus the fitted line passes through . The slope is sample covariation divided by variation in ; it is undefined if every is the same. This connects regression directly to covariance and correlation.

The fitted value and residual for unit are

With an intercept, and . These are algebraic properties of the fitted sample, not proof that the population errors have mean zero or are unrelated to .

3 Sampling Behavior and Uncertainty

Condition on the observed predictor values. If errors are independent with and , then

where . More independent units, less noise, and a wider informative spread of predictor values make the slope more precise.

Estimate the residual variance with

If the errors are conditionally normal, the exact pivot

produces a confidence interval and a test of a prespecified slope. Without normality, OLS can still be unbiased under the mean model, while inference is generally approximate. Robust standard errors, bootstrap procedures, or a model for changing variance may be more appropriate.

Distinguish three uncertainty targets:

• a confidence interval for the unknown mean slope ;
• a confidence interval for the mean response at a chosen ;
• a prediction interval for a new response at , which is wider because it includes new unit-level variation.

At , their estimated standard-error components are

and

Both grow away from the center of the observed predictor range.

4 Worked Example

For five independent specimens, suppose standardized exposure is and normalized expression is . Here , , , and

Therefore and . The fitted mean expression increases by 1.7 normalized units per one-unit increase in exposure. This is an estimate of association among specimens. It is not automatically an effect of intervening on exposure, and its biological meaning depends on the normalization scale.

The residuals are , so RSS is and . The calculation makes clear that the sample size for inference is five specimens—not the number of reads or repeated technical measurements used to construct each specimen-level expression value.

5 Assumptions, Diagnostics, and Sensitivity

Assumption or goal	Why it matters	Diagnostic evidence	Response
Linear conditional mean	Otherwise one slope averages a changing relationship	Residual-versus-fitted plot; smooth trend; domain knowledge	Transform variables, add curvature, or restrict the range
Independent experimental units	Dependence makes ordinary SEs too small	Study design, clusters, time order	Model clusters/repeated measures or aggregate at the correct unit
Zero conditional mean	Confounding or systematic error biases the slope	Design knowledge; negative controls; sensitivity analysis	Randomize, adjust measured causes, redesign; diagnostics alone cannot prove it
Constant conditional variance	Classical SEs and efficiency rely on it	Residual spread across fitted values	Robust SEs, variance modeling, transformation
No dominant observations	One point can determine the line	Leverage, studentized residuals, Cook’s distance	Verify data; report leave-one-out sensitivity; do not delete automatically
Relevant predictor range	Extrapolation is unsupported by observed data	Compare requested with observed range	Avoid or justify extrapolation mechanistically

Normal residuals are needed for exact small-sample and inference, not for the definition of OLS. A histogram or quantile–quantile plot can reveal severe skewness and outliers, but small samples make normality hard to assess.

Measurement error in usually increases residual variation. Classical random measurement error in is more dangerous: it commonly attenuates the estimated slope toward zero. Replicating the assay technically can characterize measurement error, but it does not create additional biological replication.

6 Association, Prediction, and Causation

The same fitted line can serve different aims:

• Description: summarize the observed specimen-level association.
• Prediction: predict outcomes for future units drawn under the same data process; evaluate out-of-sample error.
• Causal estimation: interpret as an intervention effect only with a credible design and assumptions about confounding, selection, interference, measurement, and model form.

Randomization can justify a causal comparison, but a linear functional form may still be a poor summary. Conversely, an excellent predictive fit does not remove confounding.

7 Connections and Take-home Messages

• Populations, samples, parameters, and estimands clarifies the target before the slope is estimated.
• Standard errors and confidence intervals develop repeated-sampling uncertainty.
• Multiple linear regression replaces the single predictor with a vector and changes coefficient interpretation to a conditional comparison.
• Statistical versus practical significance explains why a small p-value does not establish biological importance.

1. Simple regression models a conditional mean; observations need not lie on a line.
2. The OLS slope is sample covariation divided by predictor variation.
3. Uncertainty depends on independent experimental units, residual variation, and the informative spread of .
4. Residual diagnostics can expose model tension but cannot prove absence of confounding or selection bias.
5. Biological, technical, predictive, and causal interpretations must be stated separately.

8 References

• Fisher, R. A. (1922). The goodness of fit of regression formulae, and the distribution of regression coefficients. Journal of the Royal Statistical Society, 85(4), 597–612. https://doi.org/10.2307/2341124
• Gauss, C. F. (1823). Theoria combinationis observationum erroribus minimis obnoxiae. Commentationes Societatis Regiae Scientiarum Gottingensis Recentiores, 5.
• Weisberg, S. (2014). Applied Linear Regression (4th ed.). Wiley.
• Gelman, A., Hill, J., and Vehtari, A. (2020). Regression and Other Stories. Cambridge University Press. https://doi.org/10.1017/9781139161879
• R Core Team. lm: Fitting linear models. R statistical software documentation. https://stat.ethz.ch/R-manual/R-devel/library/stats/html/lm.html

2 Effect-size Inflation

Let be a population effect and

Without selection, is unbiased in this model. If results are highlighted only when , the reported distribution is truncated. Estimates that cross the threshold tend to have unusually large magnitudes.

For a positive true effect, Gelman and Carlin’s Type M exaggeration ratio is

A Type S error is the conditional probability of estimating the wrong sign,

When the signal-to-noise ratio is small, power is low, Type M exaggeration can be large, and a selected result can even have the wrong sign. Statistical significance does not protect the magnitude or direction.

3 Why Low Power Inflates Selected Effects

Suppose , , and a two-sided test uses . Discovery requires . The true effect is smaller than this threshold, so most estimates near the truth are not selected. Published discoveries must usually contain a positive noise excursion; a smaller negative tail can also cross in the wrong direction.

This mechanism is sometimes called the significance filter, winner’s curse, or regression-to-the-mean setup. Its severity is driven by the signal-to-noise ratio and selection rule, not merely by whether the true effect is nonzero. Larger studies reduce , increasing power and reducing conditional exaggeration for a fixed effect.

Inflation can accumulate beyond a single test:

Noisy data
  → many outcomes, subgroups, transformations, or models
  → select favorable analyses
  → select favorable studies for publication
  → promote the most striking narrative

Ordinary confidence intervals condition on the chosen analysis as though it had been fixed in advance. They usually do not include uncertainty from searching across analyses, studies, or interpretations.

4 Replication Is an Estimation Problem

For independent original and replication estimates,

Exact equality is neither expected nor generally the scientific hypothesis. Even if , their difference has standard error

If the original estimate was selected, its conditional center is not . Comparing whether two p-values fall on the same side of discards magnitude and uncertainty: identical effects can yield different significance decisions, and different effects can both be significant.

A replication should specify before data collection:

• the population, intervention or exposure, outcome, time point, and effect scale;
• the experimental unit and randomization or sampling design;
• the primary estimand and analysis;
• a scientifically meaningful agreement, equivalence, or prediction criterion;
• sample size based on a defensible effect, not the selected original estimate;
• how protocol departures and heterogeneity will be handled.

Useful analyses report both effect estimates and intervals, the difference between them, and—when scientifically justified—an equivalence interval or prediction interval. A replication p-value alone does not estimate reproducibility.

5 Sampling Variation, Bias, and Heterogeneity

A smaller replication effect has several possible explanations.

Heterogeneity should not become an unfalsifiable explanation. Proposed moderators need biological justification and enough independent studies or sites to estimate. Post hoc subgroup stories are another selection process and need new validation.

6 Reproducible Workflows Reduce Hidden Selection

Computational practice cannot guarantee statistical validity, but it makes the analysis auditable.

1. Preserve raw data and create processed data with scripted, versioned steps.
2. Record inclusion, exclusion, filtering, normalization, and quality-control decisions.
3. Fix random seeds and report stochastic algorithms, simulation error, and software versions.
4. Use dependency-locked environments and machine-readable parameters.
5. Separate exploratory analyses from prespecified confirmatory analyses.
6. Nest preprocessing and variable selection inside resampling or cross-validation folds.
7. Publish code, metadata, and provenance when consent and privacy permit; otherwise provide controlled-access instructions and synthetic tests.
8. Test the pipeline on data with known answers and rerun it from a clean environment.

Preregistration and registered reports reduce undisclosed flexibility, but a poor prespecified design remains poor. Sharing code exposes what was run, not why the assumptions identify the target. Reproducible workflows complement design, diagnostics, sensitivity analysis, and replication.

7 Planning with Type M and Type S Errors

Traditional power asks how often a rule rejects the null under an assumed effect. A more informative design analysis also simulates or calculates:

• the distribution of estimates and interval widths;
• Type S probability among selected results;
• Type M exaggeration among selected results;
• probability of estimating an effect above a practical threshold;
• interval coverage after the complete selection pipeline;
• predictive performance in a genuinely external sample.

The assumed true-effect range should come from prior studies corrected for selection, a meta-analysis that addresses heterogeneity and reporting bias, mechanistic constraints, or the smallest effect worth detecting. Using the original selected estimate as truth understates required sample size.

Shrinkage and hierarchical models can improve estimation by combining noisy effects with an estimated effect distribution. Their gains depend on exchangeability and an adequate prior or population model. They do not transform technical replicates into biological information, and shrinkage toward a poorly estimated center can itself mislead.

8 Evidence Synthesis and Publication Selection

If study inclusion depends on significance, direction, or novelty, the published literature is selected. Small studies require larger observed effects to cross a fixed threshold, creating a small-study-effect pattern. Funnel-plot asymmetry and regression tests can detect some patterns but are low-powered with few studies and are not specific to publication bias; genuine heterogeneity can produce the same shape.

Meta-analysis of biased inputs can yield a precise biased summary. Sensible practice includes prospective study registration, comprehensive searches, inclusion of null results, risk-of-bias assessment, sensitivity to selection models, and prediction intervals when effects vary. No single trim-and-fill or bias-correction method should be treated as proof that publication selection has been removed.

Cumulative evidence is better evaluated by the distribution of compatible effects, heterogeneity, design quality, and prospective predictions than by counting successful replications. Meta-analytic pooling also requires commensurate estimands; combining differently defined outcomes does not manufacture a common target.

9 Bioinformatics Design and Interpretation

Consider differential expression measured in thousands of genes from six treated and six control donors. Donors are the experimental units. Reads, technical library replicates, and cells improve measurement or characterize within-donor variation but do not raise the condition-level biological sample size to thousands.

Effect-size inflation arises when only genes with small adjusted p-values are plotted and their same-data log fold changes are interpreted literally. Shrinkage can stabilize estimates, but discovery estimates should still be distinguished from effects in independent donors. Pathways selected because they summarize the same top genes require validation in a new cohort or a selection-aware procedure.

Reproducibility requires the count matrix or a traceable route from raw reads, sample metadata, reference genome and annotation versions, filtering and normalization rules, the design matrix, contrast, software versions, and all quality-control exclusions. Batch correction must be learned without leaking validation outcomes. In single-cell experiments, donor-aware pseudobulk or mixed models preserve replication; repeating cells from the same donor does not demonstrate cross-donor reproducibility.

Statistical replication asks whether a prespecified effect is compatible across independent samples. Biological replication asks whether the conclusion persists across relevant organisms, tissues, environments, protocols, and laboratories. A stable p-value is neither definition.

10 A Decision Framework

Before calling a result reproducible, ask:

1. Same target? Are the estimand, population, outcome, and scale aligned?
2. Independent information? Are the replication units independent of discovery and analyzed at the correct level?
3. Visible selection? Were hypotheses, analyses, exclusions, and reporting rules disclosed?
4. Adequate precision? Can the new study distinguish absence, meaningful similarity, and meaningful difference?
5. Magnitude and uncertainty? Are estimates and intervals reported rather than only threshold decisions?
6. Heterogeneity modeled? Are contextual differences planned rather than invented after results?
7. Workflow auditable? Can another analyst regenerate processing and analysis from documented inputs?
8. Claims calibrated? Does the conclusion stay within the observed populations and design assumptions?

11 Common Failure Modes

• Same code equals replication: it establishes computational reproducibility only.
• Both studies must be significant: dichotomized p-values are compared instead of effects and uncertainty.
• The original estimate powers the replication: selection makes the design effect too large.
• More technical measurements imply more power: the biological experimental unit is ignored.
• A correction is treated as ground truth: selection models and shrinkage assumptions are not checked.
• Unexpected differences are automatically heterogeneity: sampling error, selection, and bias are not quantified first.
• Only successful pipelines are disclosed: analyst-level selection remains invisible despite shared final code.
• A non-significant replication proves no effect: an imprecise study can be compatible with both important and negligible effects.

12 Connections and Take-home Messages

• Effect sizes defines magnitude on interpretable scales.
• Statistical versus practical significance supplies meaningful agreement and decision thresholds.
• Selection bias and winner’s curse derives the conditioning that inflates discoveries.
• Statistical power and sample-size calculation supports prospective replication design.
• Confidence intervals and equivalence tests replace significance matching with explicit compatibility questions.
• Sensitivity analysis and robustness examines dependence on unverifiable assumptions and workflow choices.

Reproducibility has computational, statistical, and biological dimensions. Effect-size inflation is expected when noisy, low-powered estimates are filtered for significance or extremeness. Replication should be designed around a prespecified estimand and defensible effect range, then interpreted through magnitude, uncertainty, selection, and heterogeneity—not matching p-value labels.

13 References

• Gelman, A. and Carlin, J. “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science 9 (2014): 641–651. Article.
• Ioannidis, J. P. A. “Why Most Discovered True Associations Are Inflated.” Epidemiology 19 (2008): 640–648. Article.
• Open Science Collaboration. “Estimating the Reproducibility of Psychological Science.” Science 349 (2015): aac4716. Article.
• Button, K. S. et al. “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience.” Nature Reviews Neuroscience 14 (2013): 365–376. Article.
• Munafò, M. R. et al. “A Manifesto for Reproducible Science.” Nature Human Behaviour 1 (2017): 0021. Article.
• Patil, P., Peng, R. D., and Leek, J. T. “A Statistical Definition for Reproducibility and Replicability.” Proceedings of the National Academy of Sciences 113 (2016): 14648–14653. Article.

2 Start with the Target and Selection Rule

Suppose the target-population estimand is an association or effect . An estimator may be unbiased before selection,

but the relevant property after a discovery rule is

These are different operating characteristics. A realized estimate answers the intended question only if the sampling, missingness, inclusion, and reporting mechanisms preserve identification of the estimand or are correctly handled.

Three selections should be distinguished:

1. Units: which people, specimens, cells, records, or measurements enter the analysis?
2. Variables and models: which outcomes, predictors, transformations, and specifications are retained?
3. Results: which estimates cross a threshold and are reported, interpreted, or sent to replication?

The selection indicator and the information used to construct it belong in the analysis description. Saying that data were “quality controlled” is insufficient if the quality rule depends on exposure, outcome, batch, or a downstream consequence of both.

3 Structural Selection Bias

A causal diagram makes one important mechanism visible. If exposure and outcome both influence selection,

X  →  S  ←  Y

then is a collider. Even if and are independent in the target population, conditioning on can associate them. More generally, selection opens an otherwise blocked path when it conditions on a collider or its descendant.

For example, suppose either a strong biomarker or severe disease makes referral to a specialty clinic more likely. Within referred patients, low biomarker values may occur disproportionately among severe cases admitted for other reasons. A clinic-only association can differ in magnitude or direction from the population association.

Not all conditioning is harmful and not all selection produces bias for every target. Case-control sampling can consistently estimate an exposure odds ratio under its design, although the sampled outcome prevalence is not a population prevalence. The effect depends on the estimand, graph, sampling fractions, and model—not on the word “selection” alone.

4 Deriving the Winner’s Curse

Consider an approximately normal estimate

with known standard error . A one-sided discovery is declared when , equivalently . For a truncated normal distribution,

where and are the standard-normal density and distribution functions. The second term is positive. Selection above a threshold inflates the signed estimate even though was unbiased without selection.

For a two-sided rule , effects reported in their observed direction are exaggerated in magnitude, especially when is small. Under the null, selected estimates are false positives rather than inflated estimates of a nonzero effect. When many non-null candidates compete, the top-ranked estimate combines the underlying effect with unusually favorable noise; heterogeneous true effects do not remove this conditioning problem.

5 Regression to the Mean Is Not Biological Reversal

An independent replication estimate does not contain the discovery study’s favorable noise. Consequently, it often lies closer to the underlying effect. This regression to the mean is expected after selecting extremes and does not by itself show misconduct, biological reversal, or a failed replication.

Nevertheless, several other mechanisms can produce discovery–replication differences:

• population, ancestry, tissue, protocol, or endpoint heterogeneity;
• biased discovery or replication sampling;
• measurement and batch differences;
• model misspecification or different estimands;
• false-positive discovery;
• sampling variation in both studies.

A replication probability computed from the naive discovery estimate is usually too optimistic. It treats a selected overestimate as though it were the design effect and can leave the replication underpowered.

6 Prevention and Post-selection Inference

Design is usually stronger than correction after the fact.

• Separate discovery, estimation, and validation samples. Use discovery data to select and independent data to estimate or confirm.
• Prespecify eligibility, quality control, outcomes, models, contrasts, and the hypothesis family.
• Increase the number of independent experimental units; higher information reduces the noise that selection can exploit.
• Report the full search process, including null and unfavorable results.
• Reserve external cohorts and laboratories for transportability and replication, not repeated tuning.

When the same data must be used, methods include conditional or truncated likelihoods, selective confidence intervals, shrinkage or hierarchical models, empirical Bayes estimation, data splitting, and bootstrap bias correction. They answer different conditional or marginal questions and depend on correctly specifying the selection rule and model. A correction can reduce mean squared error yet overcorrect strong signals; there is no universal multiplier.

Inverse-probability selection weighting can identify a target-population estimand when selection probabilities are estimable, all relevant causes of selection and outcome are measured, positivity holds, and the outcome model or weighting strategy is appropriate. Weight instability signals weak support. Neither weighting nor imputation repairs an entirely unmeasured determinant without additional assumptions.

7 Diagnostics and Sensitivity Analysis

Selection bias is often not empirically testable from selected data alone. Useful checks expose mechanisms rather than certify their absence.

Compare target-population and selected-sample covariates when external data exist. Vary plausible selection odds or bias parameters and find the tipping point at which the scientific conclusion changes. A funnel plot or discovery–replication contrast can reveal patterns compatible with selection, but neither uniquely identifies its cause.

8 Bioinformatics Design and Interpretation

In a genome-wide association study, people are the experimental units and genotypes are predictors measured on those people. Millions of variants are hypotheses, not replicates. Selecting variants with and quoting their discovery odds ratios without adjustment invites winner’s-curse inflation. Replication planning should use corrected, external, or conservatively specified effects and should preserve the ancestry and phenotype definitions relevant to the target population.

In single-cell analysis, donors or independently randomized biological samples—not cells, droplets, reads, or genes—are normally the experimental units for condition-level inference. Filtering cells using a variable affected by both biological condition and cell state can change cell composition and induce selection. Differential-expression screening followed by pathway estimation in the same data also makes pathway summaries conditional on selection.

Statistically, a selected estimate describes the selected-data mechanism unless assumptions transport it to a wider population. Biologically, replication also asks whether the mechanism generalizes across cohorts, tissues, protocols, and laboratories. Correcting sampling bias does not guarantee biological transportability.

9 Common Misinterpretations

• “A representative-looking sample cannot be selected.” Balance on observed variables does not rule out selection through unobserved variables or their joint effects.
• “Only the p-value is selected; the effect estimate remains unbiased.” The p-value and effect are functions of the same noisy data.
• “A smaller replication estimate disproves the original mechanism.” Regression to the mean is expected, although heterogeneity and bias must also be assessed.
• “More cells solve selection bias.” Technical observations cannot replace independent biological sampling.
• “Cross-validation removes winner’s curse.” Cross-validation can estimate predictive performance if all selection is nested within folds; it does not automatically debias a reported scientific coefficient.
• “One adjustment fixes all selection.” Weighting, imputation, shrinkage, and selective likelihood target different mechanisms and require different assumptions.

10 Connections and Take-home Messages

• Populations, samples, parameters, and estimands defines the population to which a selected sample is meant to refer.
• Point estimators: bias, variance, and mean squared error provides the unconditional and conditional bias language.
• Statistical power and sample-size calculation explains why low-information discoveries have severe exaggeration and why naive effects underpower replication.
• Multiple-testing procedures control error rates for selected hypotheses but do not automatically provide unbiased effect estimates.
• Sensitivity analysis and robustness supplies a framework for uncertain selection mechanisms.

Selection changes the distribution under analysis. The winner’s curse follows when noisy estimates are retained because they are extreme, making unbiased unselected estimators biased after discovery. State the estimand and every material selection rule, preserve the experimental unit, separate discovery from estimation when possible, and treat post-selection correction as assumption-dependent rather than automatic.

11 References

• Hernán, M. A., Hernández-Díaz, S., and Robins, J. M. “A Structural Approach to Selection Bias.” Epidemiology 15 (2004): 615–625. Article.
• Ioannidis, J. P. A. “Why Most Discovered True Associations Are Inflated.” Epidemiology 19 (2008): 640–648. Article.
• Xiao, R. and Boehnke, M. “Quantifying and Correcting for the Winner’s Curse in Genetic Association Studies.” Genetic Epidemiology 33 (2009): 453–462. Article.
• Ferguson, J. P., Cho, J. H., Yang, C., and Zhao, H. “Empirical Bayes Correction for the Winner’s Curse in Genetic Association Studies.” Genetic Epidemiology 37 (2013): 60–68. Article.
• Taylor, J. and Tibshirani, R. J. “Statistical Learning and Selective Inference.” Proceedings of the National Academy of Sciences 112 (2015): 7629–7634. Article.

2 A Bernoulli Example

Suppose a fitted model implies that an event has probability . Set . Then

If 2,438 of simulated datasets meet the event, and the estimated Monte Carlo standard error is about . A simulation-only 95% interval is approximately , or . This interval describes numerical uncertainty in estimating the model-implied probability. It does not account for whether the fitted model or its estimated parameters are correct.

For a rare probability, observing zero events does not prove the probability is zero and makes the plug-in standard error zero. Exact binomial limits, importance sampling, or a much larger targeted simulation are preferable.

3 Three Layers of Uncertainty

Monte Carlo appears inside bootstrap, permutation, posterior, power, and missing-data procedures. The layers must be kept distinct.

Increasing attacks only the third layer. Ten million simulations from the wrong model estimate the wrong target very precisely.

For nested procedures, conditional and unconditional targets also differ. A parametric-bootstrap Monte Carlo error conditional on the fitted parameters is not the sampling uncertainty of the original estimator. Repeating multiple imputation more times reduces finite-imputation noise but cannot make an implausible missingness model credible.

4 Dependent Draws and Effective Sample Size

Markov chain Monte Carlo draws are dependent. For a stationary chain with autocorrelations for , the asymptotic variance of the mean is

The integrated autocorrelation time motivates

Treating correlated draws as independent usually understates error. Batch-means and spectral-variance estimators estimate the long-run variance rather than the ordinary marginal variance. Effective sample size is function-specific: a chain can estimate one posterior mean well while poorly exploring a tail probability. Convergence diagnostics assess evidence of failure; they do not certify that all important modes have been visited.

5 Precision Planning and Stopping

Simulation effort should be chosen from the required numerical precision. For independent draws, a target Monte Carlo standard error suggests

Because is unknown, run a pilot, estimate it, calculate the additional effort, and re-estimate precision. A sequential rule should be specified before looking for a favorable answer. Examples include stopping when a valid confidence-sequence width is below a tolerance or when an estimated Monte Carlo standard error is small relative to the reported scientific standard error.

Threshold decisions require special care. If a Monte Carlo p-value is near , it is not enough to print more digits. Increase until simulation uncertainty is negligible for the scientific decision, or report that the finite simulation does not resolve the threshold. For randomly sampled null transformations, the valid correction

prevents zero p-values, where simulated statistics are at least as extreme as the observed statistic.

6 Variance Reduction

More computation is not the only route to precision.

Method	Central idea	Main requirement
Antithetic variables	Pair negatively associated draws	A construction that preserves the target distribution
Control variates	Subtract a correlated quantity with known expectation	Correct known expectation and stable coefficient estimation
Importance sampling	Draw from and weight by	wherever the target contribution is nonzero; manageable weights
Stratification	Allocate draws across important regions	Known stratum probabilities and appropriate allocation
Common random numbers	Reuse random streams when comparing procedures	Interest in a difference, with induced positive correlation
Quasi-Monte Carlo	Use low-discrepancy deterministic point sets	A suitable transformation and uncertainty assessment by randomization

For importance sampling,

The identity is exact, but a few enormous weights can produce an unstable finite estimate or even infinite variance. Weight distributions, repeated runs, and effective sample size are diagnostics—not substitutes for checking support and moment conditions.

7 Reproducible Implementation

set.seed(20261007)
M <- 100000L

# Estimate P(Z > 2) for Z ~ N(0, 1)
hit <- rnorm(M) > 2
p_hat <- mean(hit)
mcse <- sqrt(p_hat * (1 - p_hat) / M)
c(estimate = p_hat, mcse = mcse)

A reproducible analysis records the pseudo-random-number generator, seed or stream-management scheme, software versions, number of draws, discarded draws if any, estimator, variance estimator, and stopping rule. Parallel simulations need independent, non-overlapping streams; calling the same seed separately on every worker can duplicate output. Reproducibility means another analyst can regenerate the calculation, not that one lucky seed should be preserved without checking Monte Carlo error.

8 Bioinformatics Applications

Monte Carlo computation is common in gene-set tests, permutation calibration, sequencing-depth simulations, posterior summaries, power studies, and parametric bootstrap checks. In a differential-expression power simulation, the biological specimen remains the experimental unit. Simulating thousands of reads or cells within each of three specimens does not create thousands of independent biological replicates.

A useful study separates two axes:

1. vary the scientific data-generating conditions, such as effect size, dispersion, batch imbalance, dropout, and number of biological replicates;
2. within each condition, use enough Monte Carlo replicates to estimate power, false-positive rate, or interval coverage to a stated numerical precision.

Statistically, simulation can estimate operating characteristics under a specified mechanism. Biologically, it cannot show that the mechanism resembles a new organism, tissue, protocol, or laboratory. Calibrate inputs from defensible data, explore plausible alternatives, and interpret conclusions conditional on those scenarios.

9 Failure Modes and Checks

• No Monte Carlo error reported: another seed could materially change the last digits or even a threshold decision.
• Ordinary standard error for dependent draws: autocorrelation is ignored; use a long-run variance estimator.
• Too few tail events: naive Monte Carlo cannot resolve an extreme probability.
• Numerical precision confused with scientific precision: larger does not add experimental units or remove model bias.
• Adaptive stopping for significance: repeatedly simulate until a p-value crosses a threshold without valid sequential calibration.
• Invalid simulation mechanism: omitted dependence, selection, batch structure, or parameter uncertainty makes the computational target scientifically irrelevant.
• Single performance number: average power or error hides sensitivity to effect heterogeneity and nuisance parameters.

Independent repeated runs, trace and autocorrelation plots for chains, batch estimates, weight diagnostics, and comparison with analytically solvable special cases help reveal computational problems. Simulation-based calibration can check an inferential algorithm under draws from its own assumed model, but success is conditional on that model.

10 Connections and Take-home Messages

• Expectation, variance, and moments supplies the expectation and variance identities behind Monte Carlo averages.
• Law of large numbers explains consistency, while the central limit theorem supports standard errors.
• Standard errors distinguishes estimator variability from a realized error.
• Parametric bootstrap, nonparametric bootstrap, and permutation and randomization tests use simulation for different reference distributions.
• Statistical power and sample-size calculation uses Monte Carlo when analytic operating characteristics are unavailable.

Monte Carlo estimation converts an expectation into an average, and its error must be estimated like any other sampling error. Dependence, heavy tails, rare events, and unstable weights can invalidate the familiar calculation. Report simulation precision separately from scientific uncertainty, plan around a meaningful tolerance, and use sensitivity analysis to examine the assumptions that extra simulation cannot repair.

11 References

• Robert, C. P. and Casella, G. Monte Carlo Statistical Methods, 2nd ed. Springer, 2004. Publisher page.
• Owen, A. B. Monte Carlo Theory, Methods and Examples. 2013. Online book.
• Flegal, J. M. and Jones, G. L. “Batch Means and Spectral Variance Estimators in Markov Chain Monte Carlo.” The Annals of Statistics 38 (2010): 1034–1070. Article.
• Phipson, B. and Smyth, G. K. “Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn.” Statistical Applications in Genetics and Molecular Biology 9 (2010), Article 39. Article.
• R Core Team. “Random Number Generation.” R statistical software documentation. Documentation.

2 Robust Estimation Is Not Sensitivity Analysis

The terms overlap but answer different questions.

A robust standard error changes the estimated variance under heteroskedasticity; it does not make a biased causal estimand unbiased. A robust regression estimator can limit the influence of extreme observations; it does not repair confounding or selection. Conversely, a sensitivity analysis may deliberately use a conventional estimator while varying assumptions that the data cannot test.

3 Start with the Estimand and the Threat

Before changing code, state the scientific estimand. For example,

is a population average causal effect, whereas a sample mean difference is only an estimator under a design and identification assumptions. Sensitivity should target the assumption that links the observed data to , not arbitrary cosmetic choices.

A disciplined workflow is:

1. Name the estimand and decision. Specify the effect scale and the smallest scientifically important difference.
2. Draw the data-generating process. Identify sampling, treatment assignment, measurement, missingness, dependence, and selection.
3. List nonidentified assumptions. Prioritize assumptions that the observed data cannot verify.
4. Parameterize deviations. Use interpretable bias parameters or bounded alternative models.
5. Choose plausible ranges before seeing favorable results. Justify them with design knowledge, validation data, negative controls, or external studies.
6. Recompute estimates and uncertainty. Preserve dependence and repeat the complete analysis pipeline.
7. Find tipping points. Report how strong a departure must be to change a scientific conclusion.

This process distinguishes uncertainty analysis, which propagates a probability model for uncertain inputs, from sensitivity analysis, which maps outcomes over specified alternatives. Assigning a probability distribution to an unidentified bias parameter is an additional assumption, not an automatic fact.

4 Local and Global Sensitivity

Suppose an analysis result is , where is a nuisance or bias parameter. A local sensitivity measure is

It describes an infinitesimal change near the baseline and can be computationally efficient. It may miss nonlinearities, thresholds, interactions, or distant but plausible alternatives.

Global sensitivity evaluates over a region . One-at-a-time changes are easy to explain but cannot expose interacting assumptions. A grid, factorial design, or carefully sampled parameter region can show joint departures. The region must remain scientifically coherent: not every mathematical combination corresponds to a possible study.

Useful summaries include:

• the range of point estimates and interval endpoints;
• the fraction of prespecified specifications supporting each conclusion;
• the boundary at which effect direction, practical importance, or interval compatibility changes;
• worst-case bounds under explicit constraints;
• a graph retaining the full relationship rather than only the most favorable analysis.

5 Quantitative Bias Analysis

Standard intervals usually condition on measured data and a statistical model; they do not automatically include bias from unmeasured confounding, selection, or misclassification. Quantitative bias analysis introduces parameters for those mechanisms.

For a binary exposure and outcome, a simple confounding bias factor can be written

under a particular bounding setup, where bounds the confounder–outcome risk-ratio association and bounds the exposure–confounder association. A corrected observed risk ratio can be explored as . The formula is useful only when its definitions and structural conditions fit the problem.

For linear regression, omitted-variable sensitivity can be expressed through partial values describing the association of an unobserved confounder with treatment and outcome after measured covariates. Benchmarking these strengths against observed covariates makes a tipping point interpretable, but an observed covariate is not automatically an upper bound for an unobserved one.

The E-value is another bound for risk-ratio effects: for ,

It describes a minimum joint association strength that an unmeasured confounder would need, on the risk-ratio scale under stated conditions, to explain away an association. It does not address selection bias, measurement error, multiple biases acting together, or the plausibility of such a confounder. A single scalar should not replace a mechanism-specific analysis.

6 A Tipping-point Example

Suppose the primary estimate is a log fold change of , and domain experts regard as the smallest biologically important magnitude. Let denote additive bias on the log scale, so the bias-adjusted effect is

The sign changes at , while practical importance is lost at . Reporting both thresholds is more informative than saying that several alternative models remained “significant.” A defensible analysis then asks whether batch imbalance, selection, normalization, or measurement error could plausibly produce bias of those sizes.

Uncertainty should also be updated when the bias correction is estimated rather than fixed. Treating an uncertain correction as known can move the point estimate while leaving intervals implausibly narrow.

7 Model, Data, and Workflow Perturbations

Sensitivity domains should follow the actual failure mechanisms.

Perturbations should be defensible alternatives, not every possible model and not a search for the desired answer. If several pipelines answer different estimands, their disagreement is not mere instability; the targets must first be separated.

8 Specification Curves Without Vote Counting

A multiverse analysis runs many reasonable specifications. Its value is transparency about analyst choices, but naive summaries can mislead. Specifications are dependent, may encode contradictory estimands, and are rarely sampled uniformly from a well-defined population of analyses. Counting how many p-values fall below is therefore not a calibrated probability that a claim is true.

Organize specifications by the assumptions they represent, display effect estimates and intervals, and mark the prespecified primary analysis. If an analytical choice is influenced by the observed results, ordinary intervals and p-values do not account for that selection. Sensitivity displays reveal dependence on choices; they do not automatically solve selective inference.

9 Bioinformatics Design and Interpretation

Consider an RNA-seq comparison with six donors per group, two sequencing lanes, and thousands of genes. Donors are biological experimental units; lanes and reads are technical observations. Useful sensitivity analyses might vary:

• whether lane or extraction batch is included in the design;
• defensible filtering thresholds fixed without using treatment labels;
• normalization strategy when composition bias is plausible;
• dispersion or prior-shrinkage settings;
• donor-level quality exclusions using prespecified criteria;
• hidden-factor adjustment while guarding against removal of biological signal;
• effect-size thresholds and false-discovery procedures.

Refitting after dropping individual cells or reads is not a meaningful independence check. A leave-one-donor-out analysis can reveal influence, but with a small sample it may also change balance and the estimand. Batch correction performed before cross-validation can leak information, and an alternative correction may change both signal and uncertainty.

Statistical robustness means that a result is stable over justified analysis and bias scenarios. Biological robustness additionally asks whether the magnitude, pathway interpretation, and generalization persist across tissues, cohorts, protocols, and laboratories. Neither is established by repeatedly analyzing the same specimens until a preferred method agrees.

10 Reporting a Sensitivity Analysis

A useful report states:

• the primary estimand, analysis, and scientific decision criterion;
• each threatened assumption and why the observed data cannot settle it;
• the alternative models or bias parameters and the rationale for their ranges;
• which elements were prespecified;
• estimates and compatible intervals on a scientifically interpretable scale;
• tipping points and whether they are plausible;
• cases where the analysis changes the target population or estimand;
• code and data provenance sufficient to reproduce every scenario.

Do not hide sensitivity results in a supplement when they materially alter the main claim. A fragile result can still be valuable if its dependence on assumptions is stated clearly; a stable result can still be biased if the explored perturbation set missed the dominant threat.

11 Common Failure Modes

• Changing everything at once: disagreement cannot be traced to an assumption; use structured single and joint perturbations.
• Data-driven plausible ranges: ranges are chosen after seeing which values preserve the claim.
• Only alternative p-values reported: magnitude, interval, and practical threshold are omitted.
• Robust standard errors called robust inference: variance adjustment is confused with protection from bias.
• One-at-a-time comfort: interacting departures are ignored.
• Implausible worst cases: mathematically extreme scenarios are presented without domain calibration.
• Researcher degrees of freedom ignored: a multiverse is shown as if the final specification had been chosen independently of the data.
• Pseudoreplication preserved: many technical observations make every pipeline look precise while the number of biological units stays small.

12 Connections and Take-home Messages

• Populations, samples, parameters, and estimands anchors every perturbation to a target.
• Statistical versus practical significance supplies the scientific threshold for a tipping-point analysis.
• Effect sizes helps distinguish stability of magnitude from stability of a threshold decision.
• Monte Carlo estimation and uncertainty quantifies numerical error when a sensitivity grid is simulated.
• When asymptotic approximations fail motivates finite-sample and alternative-model checks.

Sensitivity analysis asks how conclusions change when uncertain assumptions change. It should begin with the estimand and a concrete failure mechanism, use scientifically defensible ranges, propagate uncertainty, and report tipping points rather than a binary robustness label. Robustness to a narrow menu is not proof of validity; it is evidence conditional on the perturbations actually examined.

13 References

• Huber, P. J. Robust Statistics. Wiley, 1981. Publisher page.
• Greenland, S. “Basic Methods for Sensitivity Analysis of Biases.” International Journal of Epidemiology 25 (1996): 1107–1116. Article.
• Rosenbaum, P. R. Observational Studies, 2nd ed. Springer, 2002. Publisher page.
• VanderWeele, T. J. and Ding, P. “Sensitivity Analysis in Observational Research: Introducing the E-value.” Annals of Internal Medicine 167 (2017): 268–274. Article.
• Cinelli, C. and Hazlett, C. “Making Sense of Sensitivity: Extending Omitted Variable Bias.” Journal of the Royal Statistical Society: Series B 82 (2020): 39–67. Article.
• Steegen, S., Tuerlinckx, F., Gelman, A., and Vanpaemel, W. “Increasing Transparency Through a Multiverse Analysis.” Perspectives on Psychological Science 11 (2016): 702–712. Article.

2 The Benjamini–Hochberg Procedure

Let be valid ordered p-values. For a target FDR level , BH finds

and rejects . If the set is empty, it rejects none. It is a step-up procedure: one crossing at rank makes every smaller p-value a discovery, even if an earlier point lies above its own line.

The ordered BH-adjusted p-values are

The backward cumulative minimum makes the values monotone in rank. Rejecting tests with reproduces the step-up rule.

If of the null hypotheses are true and their p-values are independent, classical BH satisfies

The guarantee also extends to specific positive dependence conditions. It does not hold for every possible dependence structure. The Benjamini–Yekutieli modification replaces by , where

to obtain FDR control under arbitrary dependence, usually at a substantial power cost.

3 Storey’s Estimate of the Null Proportion

Write for the unknown proportion of true null hypotheses. Under a continuous, correctly calibrated null, null p-values are uniform, while sufficiently large p-values are unlikely to come from strong alternatives. For a tuning value , Storey’s basic estimator is

Using can increase power because the method estimates that some tested hypotheses are non-null. In practice, relying on one arbitrary is unstable. The qvalue software can estimate a smooth relationship over several values or use a bootstrap choice, and the result should be checked for sensitivity.

For a p-value threshold , let . A plug-in estimate of the FDR at that threshold is

For ordered p-values, the corresponding monotone q-value estimate is

Conceptually, a Storey q-value is the smallest estimated positive false discovery rate attainable when calling that test significant through a p-value threshold. It is a property of the entire family and the estimation procedure, not a posterior probability that this particular null is true. The local false discovery rate is a different quantity.

4 Worked Example

Suppose eight pre-specified tests yield

At , the first four BH boundaries are , , , and . The largest crossing is rank 3 because but and no later p-value crosses. BH therefore rejects the first three hypotheses.

With , three p-values exceed , giving the illustrative estimate

The first four plug-in q-values are , , , and , so the adaptive calculation would include the fourth test at 0.05. This extra discovery depends on estimating well. Eight tests are far too few for a stable tail-based estimate; the calculation illustrates the mechanism, not a recommended analysis.

5 Reproducible Implementation

Base R computes BH- and BY-adjusted p-values. The Bioconductor qvalue package estimates , q-values, and local FDR values.

p <- c(0.001, 0.006, 0.012, 0.028, 0.300, 0.550, 0.700, 0.900)

p.adjust(p, method = "BH")
p.adjust(p, method = "BY")

# BiocManager::install("qvalue")
fit <- qvalue::qvalue(p)
fit$pi0
fit$qvalues

Keep the full, unrounded p-values for computation and record the package version, tuning method, and number of hypotheses. Software output does not determine which hypotheses belong in the family.

6 Assumptions and Diagnostics

Issue	Why it matters	Useful check or response
Invalid null p-values	Neither BH nor q-values can repair anti-conservative tests	Use diagnostics, permutation, simulation, or negative controls to assess null calibration
Dependence	Classical BH is not guaranteed under arbitrary dependence	Justify positive-dependence conditions, compare BY or suitable dependence-aware methods, and perform sensitivity analysis
Estimation of	Storey q-values can be unstable or anti-conservative when the p-value tail is poorly estimated	Plot the p-value histogram and ; compare tuning choices and BH results
Discrete p-values	A uniform null approximation and tail estimator may be conservative or irregular	Use exact/discrete-aware methods or simulation appropriate to the test
Data-dependent filtering	Filtering may change null p-value behavior and the effective family	Establish independence under the null or incorporate selection into the analysis
Hidden structure	Batch effects, confounding, or model misspecification can create an excess of small p-values	Correct the design/model first; inspect genomic inflation and control features

A flat p-value histogram is not universally expected: alternatives concentrate values near zero, discrete tests produce spikes, and correlated hypotheses create visible sampling variation. Diagnostics must be interpreted relative to the data-generating process.

7 Bioinformatics Study Design

In differential-expression or association studies, genes, peaks, taxa, or variants are the hypotheses; donors, organisms, or independently randomized samples are usually the experimental units. Cells, reads, technical wells, and sequencing lanes nested within a sample are not independent biological replicates. Pseudoreplication can make the input p-values anti-conservative, after which an FDR procedure merely adjusts invalid evidence.

Before testing, specify the contrast, eligible feature universe, independent-filtering rule, and analysis family. Filtering low-information features can improve power when the filter is independent of the null test statistic, but filtering on observed effects or p-values invalidates the usual interpretation. Report effect sizes and uncertainty alongside q-values: an FDR threshold ranks statistical evidence within a family and does not establish biological importance, mechanism, or reproducibility.

8 Interpretation and Common Pitfalls

• FDR is an expectation over repeated studies, not a guarantee that exactly 5% of one realized discovery list is false.
• A q-value is not . That individual posterior-type quantity is closer to a model-based local FDR.
• BH-adjusted p-values are often labeled q-values in software, but Storey’s q-values additionally estimate ; state the method rather than relying on the column name.
• The largest crossing determines BH rejections. Do not stop at the first ordered p-value that misses its own boundary.
• Choosing the family after seeing results defeats prospective error control. Document all tested features and contrasts, including those not reported.
• Discovery control is not selection-bias correction. Effect estimates among selected features can still be exaggerated by winner’s curse.

9 Connections and Take-home Messages

• Null distributions and p-values explains the calibration required of every input p-value.
• Family-wise error rate and Bonferroni, Holm, and Hochberg procedures address the stricter event of any false rejection.
• False discovery rate distinguishes FDP, FDR, positive FDR, and local false discovery rates.
• Effect sizes and statistical versus practical significance keep scientific magnitude separate from a discovery threshold.
• Statistical power and sample-size calculation shows why the number of tests and expected signal fraction belong in study planning.

BH provides a transparent step-up rule with well-defined guarantees under independence and suitable positive dependence. Storey’s method can gain power by estimating , but that gain makes null calibration, tuning, and sensitivity analysis more important. Neither method compensates for the wrong experimental unit or a misspecified model.

10 References

• Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300. doi:10.1111/j.2517-6161.1995.tb02031.x.
• Benjamini, Y., & Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(4), 1165–1188. doi:10.1214/aos/1013699998.
• Storey, J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B, 64(3), 479–498. doi:10.1111/1467-9868.00346.
• Storey, J. D., & Tibshirani, R. (2003). Statistical significance for genomewide studies. Proceedings of the National Academy of Sciences, 100(16), 9440–9445. doi:10.1073/pnas.1530509100.
• Storey, J. D., Taylor, J. E., & Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: A unified approach. Journal of the Royal Statistical Society: Series B, 66(1), 187–205. doi:10.1111/j.1467-9868.2004.00439.x.
• R Core Team. p.adjust: Adjust p-values for multiple comparisons. R documentation.
• Storey, J. D., Bass, A. J., Dabney, A., & Robinson, D. qvalue: Q-value estimation for false discovery rate control. Bioconductor documentation.

2 Bonferroni from the Union Bound

For every true null , validity means for . If indexes the true nulls, the union bound gives

Therefore Bonferroni rejects when

Equivalently, its adjusted p-value is . The proof makes no assumption about dependence among tests. It can nevertheless be conservative because it protects against a worst-case union of false-rejection events and uses , although only hypotheses are truly null.

3 Holm’s Step-down Procedure

Let

be the ordered p-values, with their corresponding hypotheses. Holm’s algorithm starts with the smallest p-value:

1. Compare with .
2. If it passes, compare with , then continue with .
3. At the first failure, stop and retain that hypothesis and every hypothesis with a larger p-value.

The ordered Holm-adjusted p-values are

The running maximum preserves monotonicity: a larger raw p-value cannot receive a smaller adjusted value. Holm can be derived as a closed-testing procedure with Bonferroni local tests, which explains its strong FWER control under every configuration of true and false nulls and under arbitrary dependence. Because its first cutoff equals Bonferroni’s and later cutoffs become less stringent, Holm never rejects fewer hypotheses than Bonferroni.

4 Hochberg’s Step-up Procedure

Hochberg uses the same critical values but searches from the largest p-value downward:

1. Find the largest index satisfying

   

2. Reject . If no such exists, reject none.

Its ordered adjusted p-values are

Hochberg may reject several hypotheses because one later ordered p-value crosses its boundary. This gain is not free: the classical guarantee relies on independent p-values or forms of non-negative dependence that make the underlying Simes test valid. Under arbitrary or poorly understood dependence, Holm is the safer default of these three methods.

5 Worked Example

Consider four pre-specified hypotheses with ordered p-values

and .

Rank			Bonferroni adjusted	Holm adjusted	Hochberg adjusted
1	0.010	0.0125	0.040	0.040	0.040
2	0.020	0.0167	0.080	0.060	0.048
3	0.024	0.0250	0.096	0.060	0.048
4	0.500	0.0500	1.000	0.500	0.500

Bonferroni rejects only . Holm also rejects only because its step-down sequence stops at rank 2. Hochberg finds that and rejects the first three hypotheses, provided its dependence assumptions are defensible.

An adjusted p-value here is the smallest family-wise level at which that procedure would reject the hypothesis. It is not the probability that the null is true and not an estimate of the fraction of reported findings that are false.

6 Choosing and Defining the Family

Multiplicity is attached to a set of claims. A confirmatory experiment might define one family as all primary contrasts needed to support a single conclusion. Separate scientific questions can sometimes form separate families, but splitting must follow the design and logical structure rather than the observed p-values.

These adjustments correct a valid collection of p-values; they do not repair biased sampling, model misspecification, selective reporting, or an invalid null distribution.

7 Bioinformatics Design and Interpretation

For a targeted validation panel, genes or contrasts may be the hypotheses, but biological samples remain the experimental units. Repeated sequencing lanes, technical wells, or multiple cells from the same donor do not create independent biological replication. The model must first account for pairing, blocking, repeated measures, or clustering; multiplicity adjustment comes after valid per-hypothesis inference.

Genome-wide assays also produce strongly dependent tests because genes share pathways, samples, and latent factors. Holm retains its guarantee without modeling that dependence, although it may have low power. Hochberg should not be selected merely because it yields more rejections. Report the family definition, number of tests, adjustment method, effect estimates, and uncertainty intervals so that statistical control is separated from biological importance.

8 Common Pitfalls

• Using after filtering on the observed p-values: data-dependent filtering changes the tested family unless the filter is demonstrably independent of the null p-values.
• Treating Holm as unrelated thresholds: it is a sequential rule; after the first failure, later hypotheses are not rejected.
• Applying Hochberg under arbitrary dependence without justification: its classical advantage relies on stronger assumptions than Holm’s.
• Calling an adjusted p-value a posterior error probability: it calibrates a repeated-sampling decision rule, not belief in an individual hypothesis.
• Reporting only rejections: multiplicity control says nothing about effect magnitude, direction, or practical importance.

9 Connections and Take-home Messages

• Hypothesis tests and test statistics explains what each underlying test must establish.
• Null distributions and p-values develops p-value validity and calibration.
• Family-wise error rate distinguishes weak and strong error control.
• False discovery rate targets the expected false fraction rather than the chance of any false rejection.
• Statistical power and sample-size calculation shows why multiplicity belongs in prospective design.

The central practical rule is simple: use Holm rather than unmodified Bonferroni when strong FWER control under arbitrary dependence is wanted; use Hochberg only when its dependence conditions are credible. In every case, define the family before inspecting results and interpret adjusted evidence alongside effects and uncertainty.

10 References

• Bonferroni, C. E. (1936). Teoria statistica delle classi e calcolo delle probabilità. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commerciali di Firenze, 8, 3–62.
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70. JSTOR.
• Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika, 75(4), 800–802. doi:10.1093/biomet/75.4.800.
• R Core Team. p.adjust: Adjust p-values for multiple comparisons. R documentation.

2 Randomization Tests and Permutation Tests

The terms are often used interchangeably, but their strongest justifications differ.

Setting	Source of the reference distribution	Typical null	Scope of interpretation
Randomized experiment	Known treatment-assignment mechanism	Sharp null of no unit-level treatment effect	Design-based causal inference for the randomized units
Independent samples	Exchangeability of pooled observations under	Equality of the group distributions	Population comparison under sampling assumptions
Paired or blocked design	Within-pair swaps or within-block permutations	No effect within the permitted exchange units	Contrast respecting pairing or blocking
Regression or complex design	Residual, score, or label transformations with additional conditions	Model-specific restriction	Conditional on the validity of the chosen transformation

In a randomized experiment, write the potential outcomes for unit as and . Fisher’s sharp null is

Under this null every missing potential outcome is known from the observed outcome, so can be recomputed for every allowed assignment. This yields finite-sample calibration from the design without an outcome-distribution model.

The weak null of zero average treatment effect,

does not make all missing potential outcomes known. A test exact for the sharp null is not automatically exact for this weaker null. Studentized statistics can provide asymptotically valid inference for average effects under suitable randomized designs, but the target and conditions must be stated.

For observational two-sample data, freely permuting labels is exact when the pooled observations are exchangeable under the null—for example, when both samples have the same distribution. Equality of means alone does not generally imply exchangeability when variances or shapes differ.

3 A Small Exact Randomization Test

Four experimental units have observed outcomes

and exactly two units were randomly assigned to treatment. The observed treatment group contains outcomes 8 and 6, so

There are permitted assignments. If is the sum of the two treated outcomes, then the difference in means equals . The six values are

Only one is at least 3, so the upper-tail p-value is . Two of the six have absolute value at least 3, so the two-sided absolute-statistic p-value is . The smallest attainable upper-tail p-value is : exact tests are discrete, and a small design may be incapable of attaining .

This calculation conditions on the outcomes under the sharp null and averages over the assignment mechanism. It does not claim that the four observed outcomes are a random sample from a larger biological population. Generalizing beyond these units requires additional sampling or modeling assumptions.

4 Exhaustive and Monte Carlo Calibration

Exhaustive enumeration is feasible only when is modest. With treated among units, a completely randomized design has assignments. For large sets, draw valid random assignments and include the observed arrangement. A conservative Monte Carlo p-value is

The added ones prevent a zero p-value and give a valid randomized Monte Carlo test when the observed and simulated assignments are handled symmetrically. Its resolution is , and its simulation uncertainty should be small relative to any decision threshold. Increasing reduces Monte Carlo error but cannot fix an invalid permutation scheme.

# A reproducible Monte Carlo randomization test for a completely randomized design
set.seed(20261006)
y <- c(8, 6, 5, 3, 7, 4, 6, 5)
z <- c(1, 1, 1, 1, 0, 0, 0, 0)
stat <- function(labels) mean(y[labels == 1]) - mean(y[labels == 0])
t_obs <- stat(z)
B <- 9999
t_star <- replicate(B, stat(sample(z, replace = FALSE)))
p_two_sided <- (1 + sum(abs(t_star) >= abs(t_obs))) / (B + 1)
p_two_sided

The code is appropriate only because every allocation with four treated units is permitted. A blocked, paired, clustered, or unequal-probability design needs a generator for its actual assignment mechanism.

5 Restrictions Must Follow the Design

Post-treatment missingness, noncompliance, interference between units, and outcome-adaptive stopping can break the simple reassignment argument. They require an estimand and procedure that explicitly address those features.

6 Choosing a Statistic and Tail

Exactness under a sharp null does not require the statistic itself to follow a parametric distribution. Power, however, depends strongly on the statistic. A mean difference is sensitive to average shifts; a rank statistic emphasizes ordering; a maximum may target a sparse extreme response; and a studentized contrast adjusts for estimated variability.

The alternative determines the tail. A two-sided p-value can be defined through , doubled one-sided probabilities, or an ordering based on probability mass; these definitions need not agree for asymmetric or discrete randomization distributions. Specify the statistic and extremeness rule before examining the outcome.

Inverting a family of randomization tests can produce a confidence set, but under a constant additive-effect model this targets a model parameter stronger than an average effect. Report an effect estimate and compatible interval alongside the p-value; randomization calibration does not make magnitude optional.

7 Bioinformatics Design and Interpretation

Suppose treatment was randomized to biological specimens and thousands of genes were measured. The specimen is the assignment and permutation unit. Reads, technical lanes, and cells nested within a specimen must travel with that specimen; permuting cell labels independently creates pseudoreplication. If randomization occurred within batches or matched donor pairs, permutations must remain within those sets.

For an observational case–control expression study, exchangeability is an assumption rather than a consequence of randomization. Batch, ancestry, age, or other confounders can make unrestricted case-label permutations invalid. Residual permutation methods also rely on model conditions and are not universally exact. A sensitivity analysis should compare plausible adjustment and transformation schemes rather than selecting the smallest p-value.

Statistically, a small randomization p-value says the chosen statistic is unusual among assignments permitted under the null and design. Biologically, it does not establish a large effect, a mechanism, absence of interference, or generalizability to another population. Multiplicity across genes remains a separate problem.

8 Assumptions and Failure Modes

• Wrong unit: permuting cells, reads, or repeated measurements instead of independently assigned specimens exaggerates information.
• Wrong null: a sharp-null test is presented as an exact test of a zero average effect.
• Wrong transformation group: unrestricted shuffles violate pairs, blocks, clusters, or unequal assignment probabilities.
• Exchangeability assumed from “nonparametric”: different variances, shapes, or confounder distributions can invalidate pooled-label permutation under a mean-only null.
• Adaptive statistic: trying several outcomes, transformations, or tails and reporting the most favorable test omits the selection step from calibration.
• Too few assignments or simulations: attainable p-values are coarse, and Monte Carlo error can change threshold decisions.

9 Connections and Take-home Messages

• Hypothesis tests and test statistics defines the decision rule and calibration target.
• Null distributions and p-values explains tail areas and valid p-values.
• Nonparametric bootstrap resamples units with replacement to estimate sampling uncertainty; permutation tests transform labels or assignments under a null.
• Independence and conditional independence clarifies exchangeability and dependence.
• Family-wise error rate and false discovery rate address multiplicity after individual tests are calibrated.

Randomization inference obtains its strongest finite-sample guarantee from the treatment-assignment mechanism. Permutation inference in observational samples instead needs null exchangeability or a justified model-based transformation. The experimental unit, restricted assignment set, statistic, tail, and null must all align. Enumeration or many simulations improve numerical accuracy only after that scientific and design logic is correct.

10 References

• Fisher, R. A. The Design of Experiments. Oliver and Boyd, 1935.
• Pitman, E. J. G. “Significance Tests Which May Be Applied to Samples from Any Populations.” Supplement to the Journal of the Royal Statistical Society 4 (1937): 119–130. Article.
• Lehmann, E. L. and Romano, J. P. Testing Statistical Hypotheses, 3rd ed. Springer, 2005. Publisher page.
• Ernst, M. D. “Permutation Methods: A Basis for Exact Inference.” Statistical Science 19 (2004): 676–685. Article.
• Phipson, B. and Smyth, G. K. “Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn.” Statistical Applications in Genetics and Molecular Biology 9 (2010), Article 39. Article.
• Chung, E. and Romano, J. P. “Exact and Asymptotically Robust Permutation Tests.” The Annals of Statistics 41 (2013): 484–507. Article.

2 The Wilcoxon–Mann–Whitney Statistic

Let and be independent observations from distributions and . A useful population effect is the probability of superiority

Its estimator is the normalized Mann–Whitney statistic

Thus is the fraction of cross-group pairs in which is larger, with ties split equally. Under identical continuous distributions, . If is the sum of the pooled ranks for the sample, then

This algebra explains why the Mann–Whitney test and the Wilcoxon rank-sum test are equivalent after a change of statistic.

The exact pooled-label permutation test targets equality of the full distributions,

Under a pure location-shift model with common shape and spread, the procedure can be interpreted as a test of , and a Hodges–Lehmann estimator summarizes the shift through pairwise differences. Without that model, it is not simply a test of equal medians: differences in spread or shape can change its distribution and interpretation.

3 Worked Two-sample Example

Consider independent samples

Every one of the cross-group comparisons has , so and . The rank sum is , giving the same result:

Under the equality null, all allocations of three pooled ranks to the group are equally likely. Only one allocation gives and one gives the equally extreme , so the usual two-sided exact p-value is . Complete sample separation therefore does not attain 0.05 with only three observations per group.

The effect estimate describes these observed cross-group orderings. It does not show that every future value will exceed every future value, and the p-value’s discreteness reflects the small number of independent units.

4 Paired Data: Sign and Signed-rank Tests

For paired measurements define differences . Pairing is part of the design; the measurements must not be analyzed as independent groups.

The sign test uses only whether each nonzero is positive. Under independent continuous differences with median zero, the number of positive differences is binomial with success probability . It makes relatively weak shape assumptions but discards magnitudes.

The Wilcoxon signed-rank test ranks and sums ranks carrying positive signs. Its exact sign-flip null distribution requires the differences to be independent and symmetric about zero, with additional care for zeros and ties. Symmetry makes positive and negative signs equally likely conditional on the absolute differences. If the distribution of differences is strongly asymmetric, the signed-rank test is not generally a distribution-free test of the median.

Under a symmetric location model, the signed-rank procedure targets a central shift and the Hodges–Lehmann one-sample estimator is the median of Walsh averages for . This is not generally identical to the sample median difference.

5 More Than Two Groups or Blocks

For independent groups with sizes , pooled rank sums , and , the Kruskal–Wallis statistic is

With no ties and under equal group distributions, its large-sample null distribution is approximately . Ties require a variance correction, and sparse groups can make the approximation inaccurate. A significant global test says that the rank distributions are not all alike; it does not identify the groups that differ. Planned contrasts or multiplicity-adjusted follow-up comparisons are needed.

For complete block designs, the Friedman test ranks treatments within each block and compares treatment rank sums. It preserves the block as the independent replication unit. Applying Kruskal–Wallis to repeated measurements ignores within-block dependence, while applying Friedman to unrelated groups invents pairing that is not present.

6 Ties, Zeros, Exactness, and Software

Ties occur naturally in rounded assays, ordinal scores, counts, and measurements below a detection limit. Midranks preserve a sensible ordering statistic, but the no-tie exact distribution no longer applies. Software may use tie-corrected asymptotic approximations, conditional exact methods, or Monte Carlo calibration. Zeros in paired differences also require an explicit convention.

Base R’s wilcox.test() computes an exact p-value by default for small samples only when there are no ties; otherwise it uses a normal approximation, optionally with continuity correction. The reported statistic and confidence interval depend on whether the call is one-sample, paired, or two-sample. Record the alternative, exact/asymptotic choice, tie and zero handling, continuity correction, and effect estimate rather than reporting only “a Wilcoxon test.”

x <- c(1, 2, 3)
y <- c(4, 5, 6)

# Independent samples: exact equality-of-distributions test when there are no ties
stats::wilcox.test(y, x, alternative = "two.sided", exact = TRUE,
                   conf.int = TRUE)

# Several independent groups: asymptotic Kruskal--Wallis test
values <- c(2.1, 2.8, 3.0, 3.2, 4.1, 4.4, 4.8, 5.0, 5.4)
group <- factor(rep(c("A", "B", "C"), each = 3))
stats::kruskal.test(values ~ group)

An asymptotic p-value can be inaccurate with few independent units or many ties. An exact conditional or Monte Carlo method can improve calibration only if its exchangeability and design conditions are correct.

7 Efficiency, Robustness, and What Ranks Cannot Fix

Under normal location shifts, the Wilcoxon rank-sum test has high asymptotic efficiency relative to the two-sample test, while often losing less power under heavy tails. Yet “use ranks when normality fails” is too crude. Welch’s test can remain appropriate for a mean estimand with unequal variances, and robust regression or permutation of a studentized statistic may address other targets.

Ranks cannot repair confounding, selection bias, dependence, informative missingness, batch effects, or pseudoreplication. They also cannot estimate a mean difference once magnitudes have been discarded. Choose a method from the estimand and design, not from a preliminary normality test performed on the same small dataset.

8 Bioinformatics Design and Interpretation

For a two-condition assay, donors or independently sampled specimens are normally the experimental units. Genes, reads, cells, and technical replicates are nested measurements, not extra independent ranks. A donor-level rank-sum test may be defensible with independent groups; ranking thousands of cells as though each were an independent donor can yield severely anticonservative inference.

With matched specimens, analyze donor-level within-pair contrasts using a sign or signed-rank procedure when its assumptions match the target. If treatments were randomized within blocks, a blocked randomization test can use ranks while preserving the design. Detection limits can create many ties; then report the tied fraction, use a calibrated method, and assess whether the scientific target is abundance, detection probability, or both.

Statistically, a rank effect such as describes relative ordering. Biologically, it does not directly state a fold change, pathway mechanism, or clinically meaningful difference. Report raw-scale or scientifically meaningful effect summaries and intervals alongside rank-based inference, and adjust for multiplicity when many features are tested.

9 Common Misinterpretations

• “Mann–Whitney tests medians.” Only with suitable common-shape or location-shift assumptions can it be reduced to that interpretation.
• “Nonparametric means no assumptions.” Validity still depends on sampling, independence, exchangeability, and tie handling.
• “Ranks solve outliers.” They limit magnitude influence but do not make an influential specimen scientifically irrelevant or correct data errors.
• “Signed-rank is the paired rank-sum test.” It analyzes ranked within-pair differences and needs symmetry; the independent-sample rank-sum test has a different design and null.
• “A significant Kruskal–Wallis test identifies the changed group.” It is a global test; follow-up comparisons require their own estimates and multiplicity control.

10 Connections and Take-home Messages

• Permutation and randomization tests explains how exchangeability or assignment generates exact reference distributions.
• Effect sizes distinguishes relative-ordering effects from raw and standardized differences.
• One-sided, two-sided, and equivalence tests connects the scientific claim to the alternative and tail.
• Bonferroni, Holm, and Hochberg procedures covers adjusted follow-up tests.
• Sampling distributions explains exact and asymptotic calibration.

Ranks retain ordering while discarding distance, providing monotone invariance and useful robustness. The Wilcoxon–Mann–Whitney statistic naturally estimates a probability of superiority, not automatically a median difference. Paired, independent, and blocked designs require different rank procedures. Ties, zeros, small samples, dependence, and the intended estimand determine whether exact, asymptotic, or alternative methods are appropriate.

11 References

• Wilcoxon, F. “Individual Comparisons by Ranking Methods.” Biometrics Bulletin 1 (1945): 80–83. Article.
• Mann, H. B. and Whitney, D. R. “On a Test of Whether One of Two Random Variables Is Stochastically Larger than the Other.” The Annals of Mathematical Statistics 18 (1947): 50–60. Article.
• Kruskal, W. H. and Wallis, W. A. “Use of Ranks in One-Criterion Variance Analysis.” Journal of the American Statistical Association 47 (1952): 583–621. Article.
• Friedman, M. “The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance.” Journal of the American Statistical Association 32 (1937): 675–701. Article.
• Hollander, M., Wolfe, D. A., and Chicken, E. Nonparametric Statistical Methods, 3rd ed. Wiley, 2014. Publisher page.
• R Core Team. “Wilcoxon Rank Sum and Signed Rank Tests.” R statistical software documentation. Documentation.