Decision Accuracy and Consistency for Rater-Mediated Exams
Overview
Answers "would this candidate have passed with a different set of raters?" for rater-mediated exams such as oral examinations, objective structured clinical examinations and essay scoring. Builds on the many-facet extension of the rating scale model (Andrich, 1978, doi:10.1007/BF02293814) to compute counterfactual pass probabilities under the observed, an average-severity and a random rater panel, under an explicit decision rule (raw total, fair average or measure), and splits expected misclassification into measurement error and rater assignment, extending item response theory classification accuracy (Lee, 2010, doi:10.1111/j.1745-3984.2009.00096.x) to rater effects. Supports panels in which every rater scores every task and designs in which each case is scored by a different examiner, with rating categories or continuous scores such as 0-100 (through a linear many-facet model). Models can be fitted with 'TAM' or built-in estimators.
- Designs with one examiner per case (or any partially overlapping rating design) are now supported.
df_counterfactual()compares each candidate's observed examiners with random reassignments that draw, for each case, an examiner qualified for that case, without repeating an examiner for the same candidate.df_simulate()gainsdesign = "per_item"andraters_per_item. - Continuous scores (for example 0-100) are supported through a linear many-facet model,
score = ability - case difficulty - examiner severity + error.df_data()gainsscale = c("auto", "ordinal", "continuous");df_fit()uses the new"linear"engine for continuous scores;df_simulate()gainsscale = "continuous". - New decision rule
df_cut(value, "raw_mean"): pass if the mean observed rating reachesvalue(e.g. 70), convenient when candidates have different numbers of ratings.
Show all changes in 0.2.0Hide the rest of the changes in 0.2.0
df_fit()stops with a clear message when the rating network is disconnected, because rater severities would then not be comparable.- New synthetic data set
oral_exam: 300 candidates, 12 cases, one examiner per case, 0-100 scores. - Results for the existing crossed designs with ordinal scores are unchanged.
From NEWS.md in the release
Install
Health
- OK13 OKNow
- OK2 OK2026-10-07
Documentation
- Help pages
- 9
- Examples that run
- 100%9 run in a plain check
- Exported functions with examples
- 100%
- Documented parameters
- 77%
- Return-value docs
- 100%
- Help pages citing references
- 0%
Downloads
Repository
8 development-tooling and community-health practices detected across 5 families in the upstream repository
Checks run against github.com/edidatasolutions/decisionfacets on 2026-10-18.
The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel
GitHub Pages site last deployed 2026-10-10 · The repository has a site deploy workflow
pkgdown site last built 2026-10-10, before release 0.2.0 (2026-10-21)
Show all practicesHide 1 practice
This repository shows signs of AI coding tools.
- Claude Codefirst seen 2026-10-07Commits: at least 4 crediting it in the message
- Claude Opus 5.54 commits · Oct 2026
Dependencies
Installing decisionfacets adds no packages beyond those that come with R.
Code & Tests
Datasets
People & History
Pick two releases to compare their code metrics.
- 0.2.0Latest
- 0.1.02026-10-07 ·
3 days in CRAN's queue
Listed in CRAN's incoming queue from 2026-10-03 21:57 to 2026-10-07 06:45 UTC, in 16 readings. - RR 4.6.0 released · 2026-04-24
Package metadata
- First published
- 2026-10-07
- Total releases
- 2 / 1 yrs
- License
- MIT + file LICENSE OSI
- Minimum R
- ≥ 4.1
- Bundled data
- 6.9 KB / 1 file
- Download size
- 46 KB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("decisionfacets")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-21, which the citation names so these numbers can be found later. More on citing and the projects behind them.