Large Language Model Evaluation
Overview
A port of 'Inspect', a widely adopted 'Python' framework for large language model evaluation. Specifically aimed at 'ellmer' users who want to measure the effectiveness of their large language model-based products, the package supports prompt engineering, tool usage, multi-turn dialog, and model graded evaluations.
vitals 0.4.0 includes a number of new features and performance improvements, in addition to tightening integration with Inspect's log viewer.
Features
- New
claude_code()andcodex()solvers evaluate the Claude Code and Codex coding agents on a task's dataset, so the same dataset and R scorer can grade both your own agent harness and an off-the-shelf coding agent. The agents run in a Docker sandbox via Python Inspect's inspect_swe package (resolved automatically with reticulate); their transcripts are read back into ellmer Chat objects for scoring and logging. - New
vitals_log_read()reads an eval log file back into a tibble of samples, reconstructing solver (and, for model-graded scorers, scorer) chats as ellmer Chat objects. - Eval logs are substantially more faithful to their source chats. Reasoning content, standardized stop reasons, per-turn cached token counts, tool parameter schemas, and structured tool errors are now written to the log rather than dropped or approximated, and model strings follow Inspect's
provider/modelconvention.
Show all changes in 0.4.0Hide the rest of the changes in 0.4.0
Performance improvements
- The log viewer serves log files as-is rather than parsing and re-serializing them, making opening a log effectively instant (previously several seconds for logs tens of MBs in size).
vitals_view()now reads only the leading bytes of each log file when
generating the homepage; listing a directory of large logs is roughly 250x
faster.
- Eval log files are dramatically smaller (roughly 4x for multi-turn, tool-heavy evals). Repeated content in a sample's events and base64 images in its messages are now de-duplicated into the sample's
attachmentspool, mirroring Python Inspect's behavior.
Bug fixes and minor improvements
$log()now records its fallback temporary directory on the task and reports the path it wrote to when noVITALS_LOG_DIRis set, so the automatic$view()succeeds.Task$new()now assigns a valid task name when given an inline (unnamed)dataset, so$log()succeeds.vitals_bundle()now tolerates logs whose results contain no scores.vitals_view()will now show correct metadata for each log. Previously, the log viewer could display one log's metadata (task name, model, score) in place of another's, both in the log listing and when clicking into a log (#208).detect_pattern()now supportscase_sensitive = TRUEon R 4.5 and later.- The log viewer will now appropriately display tool calls called in parallel.
From NEWS.md in the release
Install
Health
- OK13 OK2026-08-04
- NOTE12 OK, 1 NOTE2026-08-01
- OK13 OK2026-06-09
- ERROR12 OK, 1 ERROR2026-06-07
- OK12 OK2026-04-25
Show 1 earlier changeHide 1 earlier change
- NOTE11 OK, 3 NOTE2026-03-10
Documentation
The repository also keeps web-only articles.
- Help pages
- 13 (1 internal)includes a package overview page
- Examples that run
- 100%12 run in a plain check (7 of them under a condition)
- Exported functions with examples
- 100%
- Documented parameters
- 100%
- Return-value docs
- 91%
- Help pages citing references
- 0%
Downloads
Repository
- PRs merged (all time)
- 92%
- Median days to close, 50 most recently updated closed issues
- 4
- Median days to close, 50 most recently updated closed PRs
- 0
Stars over time
Forks over time
Releases over time
Issues over time
PRs over time
Commits over time
13 development-tooling and community-health practices detected across 7 families in the upstream repository
Checks run against github.com/tidyverse/vitals on 2026-10-21.
The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel
GitHub Pages site last deployed 2026-09-02 · The repository has a site deploy workflow
pkgdown site last built 2026-09-02
Show all practicesHide 3 practices
This repository shows signs of AI coding tools.
- Claude Codefirst seen 2025-03-04Seen in the repository: .claude, .claude/agents, CLAUDE.mdNamed in .Rbuildignore: .claude, CLAUDE.mdPull request it wrote: opened by a project member
- GitHub Copilotfirst seen 2025-11-20Pull request: opened by its own accountCommits: 1 by its own account · at least 1 crediting it in the message
Dependencies
Installing vitals adds 35 packages (22 with compiled code, 31 MB of source tarballs). Heaviest: stringi 11 MB, promises 3.1 MB, ellmer 3.0 MB.
Code & Tests
Datasets
People & History
Pick two releases to compare their code metrics.
- 0.4.0Latest
- 0.3.0
- RR 4.6.0 released · 2026-04-24
- 0.2.02025-12-01 · diff ↗
- 0.1.02025-06-24
- RR 4.5.0 released · 2025-04-11
Package metadata
- First published
- 2025-06-24
- Total releases
- 4 / 1 yrs
- License
- MIT + file LICENSE OSI
- Minimum R
- ≥ 4.1
- Bundled data
- 14 KB / 1 file
- Download size
- 3.3 MB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("vitals")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-24, which the citation names so these numbers can be found later. More on citing and the projects behind them.