Automated Multi-Outcome Machine Learning Combination Models
Overview
Provides automated machine learning workflows for survival analysis, binary classification, continuous outcomes, and ordinal outcomes. The package trains and combines model variants across user-supplied multi-cohort data, evaluates survival models by leave-one-out cross-validation using Harrell's concordance index, binary models by leave-one-out cross-validation using receiver operating characteristic area under the curve, continuous models by out-of-fold root mean squared error and R-squared, and ordinal models by out-of-fold quadratic weighted kappa. It renders reproducible reports in Hypertext Markup Language (HTML) with figures and diagnostics. The survival workflow supports penalized and tree-based Cox proportional hazards models, stepwise Cox models, partial least squares regression for Cox models, supervised principal components, gradient boosting machine Cox models, survival support vector machines (survival-SVM), random survival forests, and optional 'CoxBoost'. The binary workflow supports penalized logistic regression, logistic baselines, gradient boosting machines, random forests, principal component analysis (PCA) logistic regression, and Gaussian naive Bayes variants. Continuous and ordinal workflows reuse an 18-variant regression registry with penalized, linear, boosted, forest, PCA, and baseline families. The optional 'CoxBoost' model is enabled when the suggested 'CoxBoost' package is installed; it is used conditionally and is not a strong dependency. Optional model backends are checked at run time so missing backend packages skip only the affected model variants rather than blocking installation of the whole package. Methods build on Friedman et al. (2010) doi:10.18637/jss.v033.i01, Bair and Tibshirani (2004) doi:10.1371/journal.pbio.0020108, Ishwaran et al. (2008) doi:10.1214/08-AOAS169, Blanche et al. (2013) doi:10.1002/sim.5958, and Binder and Schumacher (2008) doi:10.1186/1471-2105-9-14.
- CRAN pretest fix: replaced the
extreme_surv_screen()regression test with a deterministic survival-only fixture usingstepwise_cox, so the test no longer depends on optionalglmnetbehavior and remains stable on Windows and Debian R-devel incoming checks. - Single-cohort report fix: cohort-aware survival, binary, continuous, and ordinal report tables now use a unified panel rule. A single real cohort is labelled by its cohort name, a literal
Allcohort is labelledOverall, and only multi-cohort analyses addOverallplus individual cohorts. This removes duplicated single-cohort panels such asOverallplusTCGA_LUAD_PanCan. - Survival publication figures now adapt more layout settings to the actual rendered data: feature-importance and SHAP-style plots use data-driven left margins and point sizes, nomogram row spacing expands with the number of horizons, calibration and decision-curve legends choose less occupied plot corners, and the combination benchmark allocates relatively more space to the side performance bars.
Show all changes in 1.0.0Hide the rest of the changes in 1.0.0
- Binary publication figures now use the same data-aware plotting utilities for ROC, precision-recall, calibration, decision-curve, confusion-matrix, feature-importance, and benchmark plots, reducing label clipping and legend overlap in single- and multi-cohort outputs.
- Continuous and ordinal publication figures now compute ranking/importance margins from the longest model or feature label and reduce point size in observed-vs-predicted or observed-vs-score scatter plots as sample size increases. Ordinal confusion-matrix labels and cell counts also scale with the number and length of classes.
- DESCRIPTION wording was tightened for CRAN by removing unnecessary acronym parentheticals where the full method name is already written out.
- Routed model-evaluation and ensemble-fitting progress output through
log_message()soinitialize_auto_logging()captures messages such asEvaluating ...andFitting ...in the log file as well as the console. - Changed survival-SVM candidate evaluation to use k-fold resampling by default (
surv_svm_resampling = "kfold") to avoid known single-row prediction failures fromsurvivalsvmunder leave-one-out cross-validation. - Improved publication plotting defaults: heatmaps drop all-NA rows while marking partial NA cells, heatmap color scales adapt to the finite metric range, forest plots use data-aware x-axis limits, and continuous residual histograms use data-driven breaks and axis ranges.
- Added continuous and ordinal ensemble-member fitting progress messages.
- Exported
automlr_input_to_surv_xy(),automlr_input_to_binary_xy(),automlr_input_to_continuous_xy(), andautomlr_input_to_ordinal_xy()so users can call lower-level evaluation functions without relying on internal:::helpers. - Added optional automatic threshold recommendations for threshold-style ensemble selection:
auto_min_cindex,auto_min_auc,auto_min_r2, andauto_min_qwk, controlled byauto_quantile. - Added helper functions
recommend_surv_cindex_threshold(),recommend_binary_auc_threshold(),recommend_continuous_r2_threshold(), andrecommend_ordinal_qwk_threshold()for explicit threshold review. - Moved heavyweight model engines from strong imports to optional suggested dependencies so installation no longer requires all modelling backends.
- Added
check_automlr_dependencies()to report available and missing model backends, optional features, expected skip/degradation behavior, and install commands. - Made logging and parallel execution degrade gracefully when optional
log4r,future, orfuture.applypackages are unavailable. - Added continuous-outcome workflows:
prepare_continuous_cohort_input(),continuousmlr_parameters(),fit_continuous_ensemble(),export_continuous_results(), andrender_continuous_report(). - Added ordinal-outcome workflows:
prepare_ordinal_cohort_input(),ordinalmlr_parameters(),fit_ordinal_ensemble(),export_ordinal_results(), andrender_ordinal_report(). - Added 18 default continuous/ordinal model variants across penalized regression, linear / stepwise linear regression, GBM, random forest, PCA-linear, and mean-baseline families.
- Continuous model selection defaults to out-of-fold RMSE, with MAE, R-squared, Pearson, Spearman, cohort performance, observed-vs-predicted, residual, and feature-importance diagnostics.
- Ordinal model selection defaults to out-of-fold quadratic weighted kappa, with accuracy, balanced accuracy, class MAE, score RMSE, Spearman, confusion-matrix, observed-score, and feature-importance diagnostics.
- Hardened the binary-classification workflow: multi-class outcomes are now rejected by default unless users explicitly request
collapse_other = TRUE, andnegative_classcan be supplied for clear positive / negative mapping. - Added binary preprocessing audits for missingness filtering, median imputation, zero / low-variance filtering, and optional feature standardization.
- Added binary k-fold and repeated k-fold resampling options while preserving LOOCV as the default.
- Split binary exported predictions into apparent and out-of-fold probabilities/classes; default diagnostic tables and plots now use out-of-fold probabilities.
- Fixed binary
strategy = "threshold"report/export compatibility. - Added binary
model_performance_forest.csvandfig9_model_performance_forestwith OOF ROC AUC and approximate 95% CI. - Added a binary-classification workflow parallel to the survival workflow:
prepare_binary_cohort_input(),binarymlr_parameters(),fit_binary_ensemble(),export_binary_results(), and binary summary helpers. - Added 18 default binary model variants across 9 algorithm families: penalized logistic regression, standard / stepwise logistic regression, GBM, random forest, PCA-logistic, and Gaussian naive Bayes.
- Added binary single-model and single-/two-model probability-combination ranking by LOOCV ROC AUC, with PR-AUC, Brier score, threshold metrics, cohort AUC stability, calibration, DCA, confusion matrix, and feature importance exported as diagnostics.
- Added default explainability and clinical-utility outputs to regular survival exports: permutation feature importance, SHAP-style median-ablation summary and dependence plots, risk-score nomogram, calibration curve, decision curve analysis, and a model C-index forest plot.
- Refined the nomogram and SHAP-style figure set after FigureYa / regplot and SHAP documentation review: the nomogram now uses a points / total-points / event-risk ruler layout with risk-score distribution marks, while the SHAP figures follow mean-absolute-contribution bar, beeswarm summary, and dependence-with-density conventions.
- Added corresponding CSV audit tables:
feature_importance.csv,shap_approx_contributions.csv,risk_score_nomogram.csv,calibration_curve.csv,dca_curve.csv,model_cindex_forest.csv, andrisk_prediction_horizon.csv. - Embedded the final publication figure set directly in the default HTML report while keeping diagnostic figures in a separate diagnostic folder.
- Clarified the four regular-analysis interpretation checkpoints in the default report bundle: data preparation, base-model screening, ensemble selection, and explainability / clinical utility.
- Added
summarize_explainability_results()and bilingual interpretation text explaining that SHAP-style outputs are median-ablation approximations and that nomogram / calibration / DCA diagnostics are based on the final risk-score Cox calibration. - Stored the training feature matrix inside fitted survival ensembles so exported explainability diagnostics can be regenerated from the fitted object.
- Validated the regular workflow on a 100-sample TCGA-LUAD test dataset with all publication figures, CSV tables, HTML report, and bilingual summaries generated successfully.
From NEWS.md in the release
Install
Health
ERRORtestsr-oldrel-macos-arm64
Running ‘testthat.R’ [34s/35s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> library(testthat)
> library(AutoMLR)
AutoMLR loaded (1.0.0). Call initialize_auto_logging() to enable file logging, start_parallel() to enable parallel execution.
>
> test_check("AutoMLR")
2026-10-22 05:25:32.532 R[30891:201220] XType: Using static font registry.
Saving _problems/test-survival-workflow-275.R
[ FAIL 1 | WARN 3269 | SKIP 1 | PASS 372 ]
══ Skipped tests (1) ═══════════════════════════════════════════════════════════
• {plsRcox} cannot be loaded (1): 'test-survival-workflow.R:52:3'
══ Failed tests ════════════════════════════════════════════════════════════════
── Failure ('test-survival-workflow.R:274:5'): fit_surv_ensemble auto-uses automlr_input cohorts for stability ──
Expected `file.exists(file.path(export_dir, "figures", "publication", "fig4_multi_cohort_time_roc_panel.png"))` to be TRUE.
Differences:
`actual`: FALSE
`expected`: TRUE
[ FAIL 1 | WARN 3269 | SKIP 1 | PASS 372 ]
Error:
! Test failures.
Execution halted- ERROR12 OK, 1 ERRORNow
- ERROR7 OK, 1 ERROR2026-06-08
Documentation
- Help pages
- 96
- Examples that run
- no examples
- Exported functions with examples
- 0%
- Documented parameters
- 98%
- Return-value docs
- 100%
- Help pages citing references
- 0%
Downloads
Dependencies
Installing AutoMLR adds no packages beyond those that come with R.
Code & Tests
People & History
- 1.0.0Latest2026-06-07 · current release
- RR 4.6.0 released · 2026-04-24
Package metadata
- First published
- 2026-06-07
- Total releases
- 1 / 1 yrs
- License
- MIT + file LICENSE OSI
- Minimum R
- ≥ 4.1
- Download size
- 3.6 MB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("AutoMLR")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-21, which the citation names so these numbers can be found later. More on citing and the projects behind them.