Gtheory4LLM

Studies the reliability and generalizability of subjective judgments produced by large language models (LLMs), including annotation, rating, and structured LLM-as-a-judge tasks. Specifies evaluator, prompt, generation, and repeated-run facets through crossed or explicitly nested random sources with configurable item interactions. Fits univariate models, joint Gaussian models, and joint discrete models for binary, ordinal, and unordered categorical outcomes, with source-specific covariance. Gaussian models use exact balanced likelihood; discrete models use a dense first-order Laplace approximation with Gaussian latent random effects. Supported balanced decision studies compare evaluator, prompt, and replication allocations using observed Gaussian or explicitly requested latent binary and ordinal reliability, with random or fixed facets after Brennan (2001). Gaussian fits report asymptotic Wald standard errors for their variance components and delta-method intervals for the coefficients; discrete fits report point estimates only. Provides fixed-layout Gaussian batch projections, bounded cost-aware allocation comparisons, and Gaussian simulation-based pilot precision planning. Scalar nominal reliability and joint Gaussian-discrete fitting are not implemented. Includes three publicly archived LLM annotation datasets covering hate-speech, mental-health, and drug-review tasks. Discrete fitting is limited to small models; the preflight report describes supported designs and computational limits. Generalizability coefficients follow the variance-decomposition framework of Brennan (2001) <doi:10.1007/978-1-4757-3456-0>.

Releases

2

Needs compilation

No

Needs R

>= 4.5.0

• Text mining and NLP

• graphics
• Matrix
• methods
• OpenMx
• stats
• utils

Installing it adds

12 packages, 27.5 MB of source

Of those, need compilation

10

• knitr
• lme4
• ordinal
• rmarkdown

Lines by kind (16,983)

Lines by language

Languages are counted from file extensions.

Files

109

R functions

222

Object systems

S3: 25 methods

Median

6

Mean

11.58

Highest

234

Median lines per function

15

R functions

222

Calls

359

Unconnected

13

Groups

1

Lines counted

6,050

Lines the tests ran

5,155

Tests passed

Yes

Measured with

covr 3.6.5 on R 4.6.1

Has tests

Yes

Test frameworks

testthat, tinytest

Test cases

156

Lines of tests

5,781

Test lines per line of code

0.7

• How many evaluators, prompts, and runs?
• Plan annotation effort, costs, and pilot precision

README

Yes, 2,653 words

Release notes file

Yes

Help pages

26

Lines of documentation

2,405

Commits

179

Open pull requests

0

Tagged releases

7

Latest tagged release

v0.6.0, 21 Oct 2026

In development

Default branch at 1.0.0.9000, ahead of CRAN 0.5.0 (checked 2026-10-25)

Issues closed in

20 days (median)

Pull requests closed in

0 days (median)

Pull requests merged

95%

Created

13 Sep 2026

Licence file reads as

GPL-3.0

Method, repository practices

Across Veronica0206

<!-- release-identity:start -->

Source version: 0.5.0.

For versioned archives, manuals and publication status, see the

repository manifest

and GitHub releases.

These repository records are excluded from the package archive; this source

version does not assert that a corresponding release has been published.

<!-- release-identity:end -->

This version adds bounded study-planning tools. Historical release assets and

the numerical acceptance rules are unchanged.

• gt_batch_reliability() and gt_batch_dstudy() project item-score, named contrast and aggregate error under a fitted equal fixed Gaussian batching layout. Targets are conditional on that layout; no new-batch-size effect or universal batch-adjusted G is inferred.
• gt_plan() enumerates supplied allocation grids and ranks them using dated, user-supplied cost assumptions, constraints and point or available lower-bound targets. It retains infeasible candidates, Pareto options and baseline marginal benefits without changing the declared score universe.
• gt_simulate() generates balanced Gaussian studies and gt_pilot_plan() refits prospective pilot designs, retaining all attempts, numerical refusals, interval availability and Monte Carlo uncertainty. These are model-based design simulations, not calibrated bootstrap intervals or coverage guarantees.
• Batch D-study data frames retain allocation counts and essential interpretation fields in ordinary columns, so CSV exports remain interpretable without R attributes. Pilot replicate ledgers retain numerical acceptance-failure codes and the selected refit attempt without retaining whole fit objects.
• Batch projections preserve distinct numeric item identifiers, including large adjacent values that have the same default printed representation. Use the returned layout$item keys to name contrast and aggregate weights. This corrects a false refusal after an otherwise valid fit; covariance estimation and projection arithmetic are unchanged.