Item Response Theory Calibration with a Mixed Subjects Design
Overview
Integrates large language model generated item responses into psychometric calibration studies through a mixed-subjects design for unidimensional two-parameter and one-parameter logistic item response theory models. Human pilot responses are augmented with model-generated responses using a prediction-powered inference estimator (Angelopoulos, Bates, Fannjiang, Jordan and Zrnic (2023) doi:10.1126/science.adi6000; Angelopoulos, Duchi and Zrnic (2023) doi:10.48550/arXiv.2311.01453) adapted to marginal maximum-likelihood estimation, following the mixed-subjects design of Broska, Howes and van Loon (2025) doi:10.1177/00491241251326865. The estimator is anchored to the human responses and is asymptotically unbiased for the human item parameters at any tuning weight; the weight on the synthetic responses is chosen to minimize propagated ability-score risk, down-weighting uninformative or biased generated responses. Louis-corrected sandwich standard errors, ability scoring, cross-fitted tuning, and scale linking are also provided.
Initial CRAN release.
- Mixed-subjects 2PL/1PL IRT calibration that augments human responses with LLM-generated responses through a PPI++ marginal-MML estimator (
fit_mixed_subjects_mml()and relatives). The estimator is anchored to the human data and is asymptotically unbiased for the human item parameters at any tuning weight. - Power tuning by **ability-score risk** (
tune_lambda_ability_risk()), which selects the tuning weight by direct 1-D optimization of propagated ability-recovery risk (passmethod = "grid"to scan a grid instead). Also included: a theoretical PPI++ score diagnostic (tune_lambda_ppi_score()), cross-fitted tuning (tune_lambda_ability_risk_crossfit(), the recommended workflow for reported analyses), and experimental per-item tuning (tune_lambda_ability_risk_item()). All non-experimental tuners use the marginal-MML estimator by default; the frozen expected-count estimator remains available viafit_fnbut is discouraged.
Show all changes in 1.0.0Hide the rest of the changes in 1.0.0
- Louis-corrected marginal sandwich covariance through the
vcov()S3 method (vcov_mixed_subjects_mml()), with ability scoring and item-parameter uncertainty propagation (score_theta(),ability_risk()). - Vignettes covering the recommended workflow, lambda tuning, the 1PL model, per-item tuning, scale linking, and a simulation-validation study; an
R-CMD-checkGitHub Actions workflow. - Currently
predictedandgenerateddata must be **binary 0/1 responses** in the high-level fitting and PPI-score functions; the low-level quadrature utilities accept fractional input.
From NEWS.md in the release
Install
Health
- OK13 OK2026-08-05
- NOTE12 OK, 1 NOTE2026-08-01
- OK7 OK2026-06-26
Documentation
- Help pages
- 32
- Examples that run
- 100%27 run in a plain check
- Exported functions with examples
- 84%
- Documented parameters
- 91%
- Return-value docs
- 100%
- Help pages citing references
- 0%
- Calibrating with a Weakly-Informative, Biased LLM
- Choosing Lambda in Mixed-Subjects IRT HTML
- IRT Linking and Gradient Asymmetry: Diagnostic Guide
- Mixed-Subjects 1PL Calibration HTML
- Mixed-Subjects IRT Calibration HTML
- Per-Item Lambda (Experimental) HTML
- Simulation Validation of the Mixed-Subjects MML Estimator
- Understanding Ability-Risk Tuning HTML
Downloads
Repository
Releases over time
Commits over time
8 development-tooling and community-health practices detected across 5 families in the upstream repository
Checks run against github.com/klintkanopka/mixedsubjectsirt on 2026-10-22.
The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel
GitHub Pages site last deployed 2026-09-14 · The repository has a site deploy workflow
pkgdown site last built 2026-08-14
Show all practicesHide 1 practice
Dependencies
Installing mixedsubjectsirt adds 65 packages (32 with compiled code, 27 MB of source tarballs). Heaviest: SimDesign 2.4 MB, Rcpp 1.8 MB, GPArotation 1.6 MB.
Code & Tests
People & History
- 1.0.0Latest2026-06-25 · current release
- RR 4.6.0 released · 2026-04-24
Package metadata
- First published
- 2026-06-25
- Total releases
- 1 / 1 yrs
- License
- MIT + file LICENSE OSI
- Download size
- 491 KB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("mixedsubjectsirt")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-25, which the citation names so these numbers can be found later. More on citing and the projects behind them.