Skip to content

pmsims

1.0.0Package record

Simulation-Based Sample Size Tools for Prediction Models

0packages depend
549downloads / year
70.7%test coverage
13/13flavors OK

Overview

About

Provides a flexible, simulation-based toolkit for exploring how much data are needed to develop reliable prediction models. It works by repeatedly generating data, fitting models, and evaluating performance to show how sample size affects predictive accuracy, calibration, and overfitting. The package supports continuous, binary, and time-to-event outcomes and can be used with both regression-based modelling approaches and machine-learning methods. It is designed to help researchers plan studies, assess feasibility, and build more robust and generalisable models. The methods are described in Olaniran et al. (2026) doi:10.1186/s12874-026-02935-9 and Shamsutdinova et al. (2026) doi:10.48550/arXiv.2602.23507.

Changes in 1.0.0

Data-generating mechanisms

  • Added four signal-complexity levels: linear, quadratic, quadratic with interactions, and the Friedman benchmark function.
  • Added control over nonlinear signal strength, predictor correlation, predictor distribution, and binary predictor prevalence through complexity and data_control.
  • Updated binary, continuous, and survival tuning so that calibration uses the same data-generating configuration as the sample-size simulation.

Models and performance metrics

  • Added ridge regression across binary, continuous, and survival outcomes and refined the random-forest defaults and XGBoost tuning.
  • Added calibration-slope squared error ("csse") for all supported outcome types, defined as -(1 - slope)^2 so that larger is better and 0 is perfect calibration. Unlike the calibration slope itself, it penalises slopes above one as well as below it.
Show all changes in 1.0.0Hide the rest of the changes in 1.0.0
  • Routed calibration-slope searches through CSSE for the machine-learning models. The search looks for the sample size at which performance *equals* the target, which requires the metric to move monotonically with n. The calibration slope does so for classical regression models, which overfit and approach one from below, but not for penalised and tree-based learners, whose slope can approach one from either side depending on how much the fit is shrunk, and so can meet the target at more than one sample size. CSSE is a squared distance from perfect calibration, so it improves monotonically regardless of the direction of miscalibration. Requesting metric = "calibration_slope" with lasso, ridge, rf, or xgboost now searches on the CSSE scale internally and translates the answer back, so targets and results stay on the calibration slope scale.

Sample-size search and results

  • Improved adaptive bounds and detection of performance targets that appear unreachable.
  • Improved fallback behaviour when model fitting or metric calculation fails.
  • Forwarded additional engine arguments consistently through simulate_custom() and the wrapper functions.
  • Expanded and standardised the metadata returned by wrapper simulations, which now records the full data-generating configuration alongside the search settings.
  • Added an up-front runtime estimate: the adaptive first stage is timed and extrapolated to the full search. Runs expected to take more than 30 minutes report the estimate; runs expected to take more than two hours also warn and ask for confirmation before continuing. The prompt is interactive-only and can be disabled with options(pmsims.confirm_long_runs = FALSE).

Printed output

  • Reorganised the console summary into sections: Inputs is now divided into Data-generating scenario, Model and performance, and Simulation, and Results leads with the minimum sample size, followed by the performance expected at that sample size.
  • Reported the data-generating configuration under Data-generating scenario: signal complexity, nonlinear strength, predictor distribution, predictor correlation, and binary predictor prevalence.
  • Reported performance estimates beneath a Performance at N = ... heading, with the achieved value for the chosen metric shown alongside its target.
  • Expressed the sample-size target as a criterion including its direction, for example Calibration slope >= 0.900, using the direction in which the metric improves.
  • Replaced internal identifiers with human-readable model names (Random forest rather than rf) and standardised names for statistical quantities (AUC and the C-statistic are both reported as C-statistic for a binary outcome).
  • Combined the signal and noise predictor counts onto one line, and stopped repeating the model and mode in Results.
  • Moved the mean/assurance explanation and the CSSE footnote below a closing rule, and de-emphasised them.
  • Added print(x, verbose = TRUE) for implementation-level detail: internal metric identifiers, the engine settings used for the search, and quantities recorded on an internal search scale. summary() now prints this detailed display.

Interface changes

  • Updated the wrapper interface to express predictor configuration through complexity and data_control. The top-level predictor_type and binary_predictor_prevalence arguments have been removed; both are now data_control fields. See the current examples and function documentation for details.
  • Candidate predictors are now correlated by default (data_control$correlation = 0.3), whereas 0.5.0 generated independent predictors. Set correlation = 0 to reproduce the earlier behaviour.

From NEWS.md in the release

Install

Health

CRAN checks
13OK
Slowest check: 3.5 min · r-oldrel-windows-x86_64
Status changes
  • OK13 OKNow
  • OK5 OK2026-09-18
Advisories

none known

Documentation

In release 1.0.0
READMEYes · 433 wordsVignettesYes · 1 · dynamicNEWSYes · 100% structured · mentions 1.0.0Citation fileYes · cites a publicationWebsiteYes · pkgdown

The repository also keeps web-only articles.

Help pages
28 (23 internal)includes a package overview page
Examples that run
33%2 run in a plain check · 3 only when donttest examples are enabled (R CMD check --as-cran) · 1 hold no code
Exported functions with examples
100%
Documented parameters
100%
Return-value docs
100%
Help pages citing references
7%

Downloads

549
CRAN downloads in the past year
Rank #23,401 · ~2/day · ~46/mo
Daily download trend is not available in this view yet.
48130 days
54990 days
5491 year
Compare downloads with other packages →
Also on7 r2u1 autocran

Repository

4Stars
2Forks
16Open issues
0Open PRs
2Releases
692Commits
9Commit identities
machine-learningprediction-modelsample-sizesimulationr
License file reads as GPL-3.0 · 692 commits · Last activity 2026-09-17 · Discussions: 8 posts
Responsiveness
How these are sampled
PRs merged (all time)
95%
Median days to close, 50 most recently updated closed issues
100
Median days to close, 50 most recently updated closed PRs
0
PracticesBeta

11 development-tooling and community-health practices detected across 6 families in the upstream repository

Checks run against github.com/pmsims-package/pmsims on 2026-10-18.

Continuous integration (2)
GitHub ActionsR CMD check workflow

The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel

Docs source (4)
README.Rmdpkgdown

GitHub Pages site last deployed 2026-09-22 · The repository has a site deploy workflow

pkgdown site last built 2026-09-22

Coverage (1)
Coverage reporting (Codecov)
Reproducibility and dev environment (1)
data-raw/
Show all practicesHide 2 practices
CRAN release process (1)
cran-comments.md
Lint, format, editor (2)
Lint workflowAir
How this is detected·Detection ruleset v3 (2026-09-26)

Dependencies

Declared dependencies
21 external dependencies (excludes base and recommended)

Installing pmsims adds 97 packages (50 with compiled code, 89 MB of source tarballs). Heaviest: stringi 11 MB, TH.data 8.2 MB, ggplot2 6.0 MB.

Depends (1)
R >= 4.1.0
Imports (9)
LinkingTo (0)
none
Enhances (0)
none
Reverse dependencies
0direct
0indirect

Nothing depends on this yet.

Datasets

People & History

People (7)
Author, Maintainer since 1.0.0 (2026)
Releases (1)
  • 1.0.0Latest
    2026-09-17 · current release
  • R
    R 4.6.0 released · 2026-04-24

Package metadata

First published
2026-09-17
Total releases
1 / 1 yrs
License
GPL (>= 3) OSI
Minimum R
≥ 4.1.0
Bundled data
653 KB / 4 files
Download size
1.2 MB

Cite

Cite this package

Run in R for the authors' preferred citation:

citation("pmsims")

The package provides a citation file that builds its entry when R runs it. Run citation() to see it.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

APA

Balamuta, J. J. (2026). R Observatory: Metrics for pmsims version 1.0.0 [Data set]. HJJB, LLC. Data release v2026-10-21. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-21, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →

Privacy choices

These apply to this browser and are stored on this device only. Nothing about your choice is sent to us.

Read the privacy policy