Subscribe to our "Population in Perspective" Substack Here!
Researchers in the Demographic Science Unit at Oxford’s Nuffield Department of Population Health have developed an open-source Python library that helps scientists examine whether their conclusions remain convincing across different, but equally defensible, ways of analysing data.
RobustiPy allows researchers to explore an analytical “multiverse”: the collection of reasonable decisions that can arise when defining outcomes, choosing variables, specifying models and evaluating results. Instead of relying on a single preferred analysis, users can examine the wider pattern across many plausible alternatives and see which conclusions remain stable.
The work, by Daniel Valdenegro, Jiani Yan, Duiyi Dai and Charles Rahal, is described in a new paper in Patterns. RobustiPy combines large-scale specification searches, bootstrap-based uncertainty estimation, model selection and averaging, out-of-sample evaluation, joint summaries and explainable AI in one reproducible workflow.
The complete source code is available through the RobustiPy GitHub repository. It is released under the GNU General Public License v3.0, allowing researchers and developers to inspect, use, modify and redistribute the software under the terms of the licence. The package is also documented, available through the Python Package Index and archived for reproducibility.
Although RobustiPy was developed in a demographic science setting, its use is not limited to demography. The software is relevant wherever researchers face multiple reasonable ways to formulate and test a quantitative model. This is common in regression-based work, where choices about outcomes, predictors, controls, transformations and evaluation criteria can quickly create a large analytical space.
Consider a hypothetical study asking whether housing insecurity affects family formation. Researchers might measure housing insecurity through rent burden, overcrowding, residential instability or difficulty meeting housing costs. They might define family formation using the timing of a first birth, entry into cohabitation or stated fertility intentions. They could also make different defensible decisions about whether to account for employment, education, income, partnership status or local housing conditions.
A conventional analysis may report one selected combination. RobustiPy would allow the team to examine the defensible alternatives and assess how consistently the conclusion holds.
The results might show that housing insecurity is associated with delayed family formation across nearly every reasonable specification. That would provide a stronger and more transparent basis for confidence than a single model alone. Alternatively, RobustiPy might reveal that the association appears only for particular measures or combinations of controls. That would show where the evidence is sensitive and help researchers refine theory, improve measurement or plan further data collection.
A second hypothetical example could involve a study of a new biomarker for future disease risk. One analysis might find that the biomarker is statistically significant, but the result could change depending on how the outcome is defined, which patient characteristics are included or how predictive performance is assessed.
RobustiPy could test whether the association remains stable across those choices, quantify uncertainty through bootstrapping and evaluate whether the biomarker improves predictions for observations not used to fit the model. Researchers might find that it is consistently associated with disease risk and improves out-of-sample prediction. Or they might find a stable association but little additional predictive value. Either conclusion offers a more complete account of its scientific and practical importance.
“Demographers are used to asking how conclusions change across people, places and periods. RobustiPy applies the same curiosity to the analysis itself,” said Associate Professor Charles Rahal. “It gives every defensible model a chance to make its case—without requiring the reader to arrive with a wheelbarrow for the appendices.”
The paper demonstrates RobustiPy through five simulations and ten empirical examples spanning economics, sociology, psychology and medicine. The examples include linear and logistic models, panel-data settings, multiple outcome measures and predictive evaluation, illustrating that analytical choice extends well beyond any single discipline.
RobustiPy is not intended to replace scientific judgement. Researchers must still decide which variables, measurements and models belong in a defensible analytical multiverse. Weak assumptions do not become reliable simply because they are evaluated many times. Instead, the software makes those decisions explicit and shows how much they matter for the final conclusion.
By bringing tools that researchers would otherwise have to assemble separately into one package, RobustiPy lowers the technical barrier to large-scale robustness analysis. It helps researchers move beyond a single headline estimate and communicate whether findings persist across multiple reasonable views of the same problem.
For the Demographic Science Unit, the project reflects a broader commitment to transparent and reproducible quantitative research. Its applications include population health and demographic questions, but its value is much wider: RobustiPy offers researchers across disciplines a practical way to make analytical uncertainty visible, measurable and easier to communicate.
