Simulation-Based Sample Size Tools for Prediction Models
Overview
Provides a flexible, simulation-based toolkit for exploring how much data are needed to develop reliable prediction models. It works by repeatedly generating data, fitting models, and evaluating performance to show how sample size affects predictive accuracy, calibration, and overfitting. The package supports continuous, binary, and time-to-event outcomes and can be used with both regression-based modelling approaches and machine-learning methods. It is designed to help researchers plan studies, assess feasibility, and build more robust and generalisable models. The methods are described in Olaniran et al. (2026) doi:10.1186/s12874-026-02935-9 and Shamsutdinova et al. (2026) doi:10.48550/arXiv.2602.23507.
Data-generating mechanisms
- Added four signal-complexity levels: linear, quadratic, quadratic with interactions, and the Friedman benchmark function.
- Added control over nonlinear signal strength, predictor correlation, predictor distribution, and binary predictor prevalence through
complexityanddata_control. - Updated binary, continuous, and survival tuning so that calibration uses the same data-generating configuration as the sample-size simulation.
Models and performance metrics
- Added ridge regression across binary, continuous, and survival outcomes and refined the random-forest defaults and XGBoost tuning.
- Added calibration-slope squared error (
"csse") for all supported outcome types, defined as-(1 - slope)^2so that larger is better and0is perfect calibration. Unlike the calibration slope itself, it penalises slopes above one as well as below it.
Show all changes in 1.0.0Hide the rest of the changes in 1.0.0
- Routed calibration-slope searches through CSSE for the machine-learning models. The search looks for the sample size at which performance *equals* the target, which requires the metric to move monotonically with
n. The calibration slope does so for classical regression models, which overfit and approach one from below, but not for penalised and tree-based learners, whose slope can approach one from either side depending on how much the fit is shrunk, and so can meet the target at more than one sample size. CSSE is a squared distance from perfect calibration, so it improves monotonically regardless of the direction of miscalibration. Requestingmetric = "calibration_slope"withlasso,ridge,rf, orxgboostnow searches on the CSSE scale internally and translates the answer back, so targets and results stay on the calibration slope scale.
Sample-size search and results
- Improved adaptive bounds and detection of performance targets that appear unreachable.
- Improved fallback behaviour when model fitting or metric calculation fails.
- Forwarded additional engine arguments consistently through
simulate_custom()and the wrapper functions. - Expanded and standardised the metadata returned by wrapper simulations, which now records the full data-generating configuration alongside the search settings.
- Added an up-front runtime estimate: the adaptive first stage is timed and extrapolated to the full search. Runs expected to take more than 30 minutes report the estimate; runs expected to take more than two hours also warn and ask for confirmation before continuing. The prompt is interactive-only and can be disabled with
options(pmsims.confirm_long_runs = FALSE).
Printed output
- Reorganised the console summary into sections:
Inputsis now divided intoData-generating scenario,Model and performance, andSimulation, andResultsleads with the minimum sample size, followed by the performance expected at that sample size. - Reported the data-generating configuration under
Data-generating scenario: signal complexity, nonlinear strength, predictor distribution, predictor correlation, and binary predictor prevalence. - Reported performance estimates beneath a
Performance at N = ...heading, with the achieved value for the chosen metric shown alongside its target. - Expressed the sample-size target as a criterion including its direction, for example
Calibration slope >= 0.900, using the direction in which the metric improves. - Replaced internal identifiers with human-readable model names (
Random forestrather thanrf) and standardised names for statistical quantities (AUC and the C-statistic are both reported asC-statisticfor a binary outcome). - Combined the signal and noise predictor counts onto one line, and stopped repeating the model and mode in
Results. - Moved the mean/assurance explanation and the CSSE footnote below a closing rule, and de-emphasised them.
- Added
print(x, verbose = TRUE)for implementation-level detail: internal metric identifiers, the engine settings used for the search, and quantities recorded on an internal search scale.summary()now prints this detailed display.
Interface changes
- Updated the wrapper interface to express predictor configuration through
complexityanddata_control. The top-levelpredictor_typeandbinary_predictor_prevalencearguments have been removed; both are nowdata_controlfields. See the current examples and function documentation for details. - Candidate predictors are now correlated by default (
data_control$correlation = 0.3), whereas0.5.0generated independent predictors. Setcorrelation = 0to reproduce the earlier behaviour.
From NEWS.md in the release
Install
Health
- OK13 OKNow
- OK5 OK2026-09-18
Documentation
The repository also keeps web-only articles.
- Help pages
- 28 (23 internal)includes a package overview page
- Examples that run
- 33%2 run in a plain check · 3 only when donttest examples are enabled (R CMD check --as-cran) · 1 hold no code
- Exported functions with examples
- 100%
- Documented parameters
- 100%
- Return-value docs
- 100%
- Help pages citing references
- 7%
Downloads
Repository
- PRs merged (all time)
- 95%
- Median days to close, 50 most recently updated closed issues
- 100
- Median days to close, 50 most recently updated closed PRs
- 0
11 development-tooling and community-health practices detected across 6 families in the upstream repository
Checks run against github.com/pmsims-package/pmsims on 2026-10-18.
The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel
GitHub Pages site last deployed 2026-09-22 · The repository has a site deploy workflow
pkgdown site last built 2026-09-22
Show all practicesHide 2 practices
Dependencies
Code & Tests
Datasets
People & History
- 1.0.0Latest2026-09-17 · current release
- RR 4.6.0 released · 2026-04-24
Package metadata
- First published
- 2026-09-17
- Total releases
- 1 / 1 yrs
- License
- GPL (>= 3) OSI
- Minimum R
- ≥ 4.1.0
- Bundled data
- 653 KB / 4 files
- Download size
- 1.2 MB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("pmsims")The package provides a citation file that builds its entry when R runs it. Run citation() to see it.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-21, which the citation names so these numbers can be found later. More on citing and the projects behind them.