R Observatory: Metrics for pairwiseLLM version 1.3.1

Balamuta, James Joseph

doi:10.5281/zenodo.21843040

pairwiseLLM

1.3.1Package record

Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation

0packages depend

4.4Kdownloads / year

91.8%test coverage

13/13flavors OK

Overview

About

Maintained by Sterett H. MercerCRAN page ↗GitHub ↗Website ↗Issues ↗

Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. The package additionally supports adaptive pairing workflows that iteratively select comparisons based on model uncertainty to improve ranking efficiency. Results can be modeled using frequentist or Bayesian Bradley–Terry–Luce models (Bradley & Terry, 1952 doi:10.2307/2334029; see also Caron & Doucet, 2012 doi:10.1080/10618600.2012.638220) or Elo rating methods (see Clark et al., 2018 doi:10.1371/journal.pone.0190393) to derive writing quality scores. For information on pairwise comparisons and comparative judgement, see Thurstone (1927) doi:10.1037/h0070288 and Heldsinger & Humphry (2010) doi:10.1007/BF03216919.

Changes in 1.3.1

New Features

Adaptive linking

Added cross-set adaptive linking through adaptive_rank(), with one-spoke and multi-spoke hub-and-spoke run modes for placing separately ranked sets on a shared scale.
Phase A can run canonical within-set ranking, import saved Phase A artifacts, or combine imported and newly run sets. Phase B uses anchored-joint estimation with hard-locked hub parameters, globally shared judge parameters, and concurrent spoke processing.
Added D-optimal cross-set candidate selection, spoke-aware comparison budgets, and fixed per-refit held-out probes with explicit coverage, calibration, and stopping gates.

Show all changes in 1.3.1Hide the rest of the changes in 1.3.1

Added persistent Phase A manifests and linking sessions, resume support, link_stage_log, and explicit blocker and invalid-judgment reporting for auditable long-running workflows.

Model support & live API improvements

Added a separate Vertex AI Gemini live backend via backend = "vertex" with VERTEX_API_KEY.
Vertex remains live-only in this series; generic batch wrappers reject backend = "vertex" explicitly rather than routing it through Gemini batch helpers.
Clarified provider-specific service_tier behavior for Gemini Developer API and Vertex AI Gemini API in the package docs, README, and vignettes.
Added request-shape handling for the OpenAI GPT-5.x and GPT-5.6 model families.

Bayesian BTL usability

Added exported build_btl_results_data() to convert conventional ID1/ID2/better_id outcomes into the canonical input required by fit_bayes_btl_mcmc().
Standalone Bayesian BTL fits now report reliability and sampler diagnostics through the same result and log surfaces used by adaptive workflows.

Improvements

Reduced adaptive-linking refit, candidate-generation, logging, and checkpoint overhead for large runs, including bounded candidate construction and analytic anchored-joint derivatives.
Hardened adaptive-linking resume behavior, Phase A stopping scope, spoke routing, probe scheduling, and Phase B pair accounting.

Documentation

Updated the backend support matrix to distinguish Gemini Developer API from Vertex AI Gemini API and to document the live-only Vertex scope.
Replaced the undated supported-model list with a dated, machine-readable compatibility registry that separates backend implementation, live/batch testing, and current provider availability.
Corrected stale provider identifiers, API-key and CmdStan setup guidance, OpenAI Flex wording, cost-estimator summaries, and broken public examples.

Bug fixes

Fixed Anthropic extended-thinking live requests when users explicitly set the required temperature = 1.
OpenAI Responses live requests now forward and enforce an explicitly supplied max_output_tokens limit.
Changed omitted temperature and top_p controls to use model/provider defaults instead of injecting deterministic sampling values. Provider-required reasoning constraints are still enforced.
Provider smoke runs now exit unsuccessfully when a selected API key is absent, unless missing-key skips were explicitly allowed.

From NEWS.md in the release

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("pairwiseLLM", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("shmercer/pairwiseLLM")

Development version from shmercer/pairwiseLLM. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-pairwisellm

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled. We list this channel from recorded r2u download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-pairwiseLLM

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-pairwiseLLM

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed). We list this channel from recorded autoCRAN download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

Health

CRAN checks

13OK

CRAN check page ↗

Slowest check: 21.7 min · r-oldrel-windows-x86_64

Status changes

OK13 OK2026-08-05
NOTE12 OK, 1 NOTE2026-08-01
OK13 OK2026-06-08
ERROR12 OK, 1 ERROR2026-06-07
OK14 OK2026-03-30

Show 2 earlier changesHide 2 earlier changes

ERROR13 OK, 1 ERROR2026-03-28
OK14 OK2026-03-10

Advisories

none known

How this is measured·How advisories are found

Documentation

In release 1.3.1

READMEYes · 2,504 wordsVignettesYes · 11 · dynamicNEWSYes · 100% structured · mentions 1.3.1Citation fileYes · software entry onlyWebsiteYes · pkgdown

Help pages: 95 (5 internal)includes a package overview page
Examples that run: 59%53 run in a plain check · 37 never run
Exported functions with examples: 100%
Documented parameters: 99%
Return-value docs: 100%
Help pages citing references: 1%

Vignettes (11)

Show 1 moreHide 1 vignette

Standalone Bayesian BTL with CmdStan HTML

How these are measured

Downloads

4.4K

CRAN downloads in the past year

Rank #11,038 · ~12/day · ~368/mo

Releases R versions

Daily download trend is not available in this view yet.

34130 days

96390 days

4.4K1 year

Compare downloads with other packages →

Also on67 r2u3 autocran

How downloads are counted

Repository

shmercer/pairwisellm ↗Account: shmercer

3Stars

0Forks

0Open issues

0Open PRs

3Releases

388Commits

1Commit identities

assessment-toolelo-ratinglarge-language-modelsollamapairwise-comparisonrr-packagerstats

388 commits · Last activity 2026-10-19 · 0% stars, 30d · Default branch at 1.6.0, ahead of CRAN 1.3.1 (checked 2026-10-22)

Responsiveness

How these are sampled

PRs merged (all time): 97%
Median days to close, 50 most recently updated closed issues: 0
Median days to close, 50 most recently updated closed PRs: 0

Stars over time

2026-02-03 · 12026-07-07 · 3

How these are measured

PracticesBeta

14 development-tooling and community-health practices detected across 8 families in the upstream repository

Checks run against github.com/shmercer/pairwisellm on 2026-10-22.

Continuous integration (2)

GitHub ActionsR CMD check workflow

The R CMD check workflow runs on Linux, macOS and Windows

Docs source (4)

README.Rmdpkgdown

GitHub Pages site last deployed 2026-10-19 · The repository has a site deploy workflow

pkgdown site last built 2026-10-19

Coverage (1)

Coverage reporting

Reproducibility and dev environment (1)

data-raw/

Show all practicesHide 4 practices

CRAN release process (2)

cran-comments.mdCRAN-SUBMISSION

Research and citation (1)

codemeta.json

Lint, format, editor (2)

lintrRStudio project

Git structural (1)

.gitattributes

How this is detected·Detection ruleset v4 (2026-10-20)

AI coding toolsBeta

This repository shows signs of AI coding tools.

Codexfirst seen 2025-12-02
Named in .Rbuildignore: .codex
Named in .gitignore: .codex
Pull request it wrote: opened by a project member
Claude Codefirst seen 2026-09-09
Named in .Rbuildignore: .claude

How this is detected·Detection ruleset v4 (updated 2026-09-27)

Dependencies

Declared dependencies

25 external dependencies (excludes base and recommended)

Installing pairwiseLLM adds 21 packages (14 with compiled code, 8.3 MB of source tarballs). Heaviest: openssl 1.1 MB, jsonlite 1.0 MB, vctrs 1.0 MB.

Depends (1)

R >= 4.1

Imports (11)

cli curl dplyr httr2 jsonlite rlangstatstibble tidyselecttoolsutils

LinkingTo (0)

none

Suggests (17)

BradleyTerry2 cmdstanr EloChoice future future.apply knitr mockery pkgload purrr readr rmarkdown sirt stringr testthat tidyr vctrs withr

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

0direct

0indirect

Nothing depends on this yet.

How these are counted

Code & Tests

Datasets

People & History

People (1)

Sterett H. Mercer

Releases (2)

Pick two releases to compare their code metrics.

1.3.1Latest
record
2026-09-07 · current release · diff ↗ ·
added 23 exports
adaptive_get_logs, adaptive_item_log, adaptive_rank, adaptive_rank_resume, adaptive_rank_run_live, adaptive_rank_start, adaptive_results_history, adaptive_round_log, adaptive_step_log, build_btl_results_data, estimate_llm_pairs_cost, fit_bayes_btl_mcmc, llm_resume_multi_batches, llm_submit_pairs_multi_batch, load_adaptive_session, make_adaptive_judge_llm, save_adaptive_session, submit_vertex_pairs_live, summarize_adaptive, summarize_items, summarize_refits, validate_session_dir, vertex_compare_pair_live
R
R 4.6.0 released · 2026-04-24
1.1.0
record
2025-12-22
R
R 4.5.0 released · 2025-04-11

Package metadata

First published: 2025-12-22
Total releases: 2 / 1 yrs
License: MIT + file LICENSE OSI
Additional repositories: stan-dev.r-universe.dev
Minimum R: ≥ 4.1
Bundled data: 176 KB / 5 files
Download size: 1.5 MB

Cite

Cite this package

Run in R for the authors' entry:

citation("pairwiseLLM")

The package provides a citation file, so citation() returns the authors' own entry.

It points to 10.32614/cran.package.pairwisellm.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

Format

APA

Balamuta, J. J. (2026). R Observatory: Metrics for pairwiseLLM version 1.3.1 [Data set]. HJJB, LLC. Data release v2026-10-25. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-25, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →