R Observatory: Metrics for probcal version 0.2.0

Balamuta, James Joseph

doi:10.5281/zenodo.21843040

probcal

0.2.0Package record

Calibration of Binary and Multiclass Probabilities

0packages depend

699downloads / year

89.2%test coverage

13/13flavors OK

Overview

About

Maintained by Pedro Rafael Diniz MarinhoCRAN page ↗GitHub ↗Website ↗Issues ↗

Provides S3 calibrators, metrics, and diagnostics for binary and multiclass probability calibration. Binary methods include Platt scaling, temperature scaling, beta calibration, histogram binning, and isotonic regression. Multiclass methods include temperature scaling, vector scaling, Dirichlet calibration, and a one-vs-rest wrapper for the binary calibrators. A calibration-inference layer adds debiased calibration errors, bootstrap confidence intervals, and a kernel calibration hypothesis test for binary and multiclass predictions, including the strong (canonical) multiclass case. Methods follow Platt (1999, ISBN:9780262194488), Zadrozny and Elkan (2002) doi:10.1145/775047.775151, Guo et al. (2017) https://proceedings.mlr.press/v70/guo17a.html, Kull et al. (2017) doi:10.1214/17-EJS1338SI, Kull et al. (2019) doi:10.48550/arXiv.1910.12656, Widmann et al. (2019) doi:10.48550/arXiv.1910.11385, and Kumar et al. (2019) doi:10.48550/arXiv.1909.10155.

Changes in 0.2.0

Added a calibration-inference layer for binary and multiclass predictions.
skce() computes the squared kernel calibration error of Widmann et al. (2019) with three estimators: the unbiased U-statistic ("uq"), the unbiased linear-time estimator ("ul"), and the biased V-statistic ("biased"). It supports the binary/confidence reduction, the strong (canonical) multiclass form via a matrix-valued kernel, and the classwise one-vs-rest reduction. In the binary and confidence cases mmce() equals sqrt(skce(..., estimator = "biased")).
cal_test() performs a kernel calibration hypothesis test (H0: the model is calibrated). The default method = "bootstrap" uses the more powerful quadratic estimator with a wild bootstrap (Widmann et al. 2019, Theorem G.2); method = "asymptotic" uses the faster linear-estimator normal test (Lemma 3). It returns an object of class c("cal_test", "htest"). The test targets are binary, "confidence", and the strong "canonical" multiclass form; the classwise average is available only as a point estimate from skce().
cal_ci() returns a percentile bootstrap confidence interval for ece(), skce(), mmce(), mce(), or ace(), as a classed cal_ci object with a print() method.
skce() and cal_test() accept bandwidth = "median" for the median-heuristic kernel scale (Widmann et al. 2019), recommended for the canonical multiclass form; the fixed 0.2 remains the default.
ece() gains debiased (the debiased squared-ECE estimator of Kumar, Liang & Ma 2019, Definition 5.2), strategy (equal-width or equal-mass bins, Roelofs et al. 2022), and norm ("l1" or "l2"). The norm and debiased choices are independent; debiasing is defined only for norm = "l2". Defaults (norm = "l1", debiased = FALSE, strategy = "width") reproduce the previous numeric output exactly.
stratified_folds() (used by cal_cv()) now scopes the optional fold-assignment seed with withr::local_seed() instead of touching .Random.seed directly; withr was added to Imports. The fold assignments for a given seed are unchanged.
inst/CITATION now reads the version from the package metadata, and the package author is also declared as copyright holder in Authors@R.
The README documents the calibration-inference layer (skce(), cal_test(), cal_ci(), and the debiased, strategy, and norm arguments of ece()) with a worked example.

From NEWS.md in the release

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("probcal", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("prdm0/probcal")

Development version from prdm0/probcal. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-probcal

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled. We list this channel from recorded r2u download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-probcal

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-probcal

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed). We list this channel from recorded autoCRAN download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

Health

CRAN checks

13OK

CRAN check page ↗

Slowest check: 3.3 min · r-oldrel-windows-x86_64

Status changes

OK13 OK2026-08-05
NOTE12 OK, 1 NOTE2026-08-01
OK7 OK2026-07-10

Advisories

none known

How this is measured·How advisories are found

Documentation

In release 0.2.0

READMEYes · 975 wordsVignettesYes · 5 · dynamicNEWSYes · 100% structured · mentions 0.2.0Citation fileYes · software entry onlyWebsiteYes · pkgdown

Help pages: 19
Examples that run: 100%19 run in a plain check
Exported functions with examples: 100%
Documented parameters: 99%
Return-value docs: 100%
Help pages citing references: 84%

Vignettes (5)

How these are measured

Downloads

699

CRAN downloads in the past year

Rank #22,284 · ~2/day · ~58/mo

Releases R versions

Daily download trend is not available in this view yet.

20530 days

58790 days

6991 year

Compare downloads with other packages →

Also on19 r2u4 autocran

How downloads are counted

Repository

prdm0/probcal ↗Account: prdm0

0Stars

0Forks

0Open issues

0Open PRs

1Releases

8Commits

1Commit identities

machine-learningrlanrstatsrstats-packagestatistical-learningstatistics

8 commits · Last activity 2026-07-07

Responsiveness

not measured

How these are measured

PracticesBeta

8 development-tooling and community-health practices detected across 4 families in the upstream repository

Checks run against github.com/prdm0/probcal on 2026-10-11.

Continuous integration (2)

GitHub ActionsR CMD check workflow

The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel

Docs source (3)

README.Rmdpkgdown

GitHub Pages site last deployed 2026-07-07

pkgdown site last built 2026-07-02, before release 0.2.0 (2026-07-09)

CRAN release process (2)

cran-comments.mdCRAN-SUBMISSION

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v3 (2026-09-26)

AI coding toolsBeta

This repository shows signs of AI coding tools.

Claude Codefirst seen on or before 2026-06-23
Named in .Rbuildignore: .claude, CLAUDE.md
Named in .gitignore: CLAUDE.md
Commits: at least 2 crediting it in the message
OpenCodefirst seen on or before 2026-06-23
Named in .Rbuildignore: .opencode
Named in .gitignore: .opencode

Models named in its commits

Claude Opus 4.82 commits · Jun 2026

Counts are commits, not lines of code. Commits by a tool's accounts count once across all of its accounts. Commits crediting it are the largest count among its credit lines, a minimum.How this is detected·Detection ruleset v4 (updated 2026-09-27)

Dependencies

Declared dependencies

11 external dependencies (excludes base and recommended)

Installing probcal adds 17 packages (7 with compiled code, 14 MB of source tarballs). Heaviest: ggplot2 6.0 MB, isoband 1.5 MB, farver 1.2 MB.

Depends (1)

R >= 4.1.0

Imports (4)

cli ggplot2statswithr

LinkingTo (0)

none

Suggests (8)

betacal dplyr knitr pkgdown reticulate rmarkdown roxygen2 testthat

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

0direct

0indirect

Nothing depends on this yet.

How these are counted

Code & Tests

People & History

People (1)

Pedro Rafael Diniz Marinho

Releases (1)

0.2.0Latest
record
2026-07-09 · current release
R
R 4.6.0 released · 2026-04-24

Package metadata

First published: 2026-07-09
Total releases: 1 / 1 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 4.1.0
Download size: 1.6 MB

Cite

Cite this package

Run in R for the authors' entry:

citation("probcal")

The package provides a citation file, so citation() returns the authors' own entry.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

Format

APA

Balamuta, J. J. (2026). R Observatory: Metrics for probcal version 0.2.0 [Data set]. HJJB, LLC. Data release v2026-10-19. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-19, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →