Skip to content

vitals

0.4.0Package record

Large Language Model Evaluation

0packages dependsuggested by 1
4.8Kdownloads / year
84.9%test coverage
13/13flavors OK

Overview

About

A port of 'Inspect', a widely adopted 'Python' framework for large language model evaluation. Specifically aimed at 'ellmer' users who want to measure the effectiveness of their large language model-based products, the package supports prompt engineering, tool usage, multi-turn dialog, and model graded evaluations.

Changes in 0.4.0

vitals 0.4.0 includes a number of new features and performance improvements, in addition to tightening integration with Inspect's log viewer.

Features

  • New claude_code() and codex() solvers evaluate the Claude Code and Codex coding agents on a task's dataset, so the same dataset and R scorer can grade both your own agent harness and an off-the-shelf coding agent. The agents run in a Docker sandbox via Python Inspect's inspect_swe package (resolved automatically with reticulate); their transcripts are read back into ellmer Chat objects for scoring and logging.
  • New vitals_log_read() reads an eval log file back into a tibble of samples, reconstructing solver (and, for model-graded scorers, scorer) chats as ellmer Chat objects.
  • Eval logs are substantially more faithful to their source chats. Reasoning content, standardized stop reasons, per-turn cached token counts, tool parameter schemas, and structured tool errors are now written to the log rather than dropped or approximated, and model strings follow Inspect's provider/model convention.
Show all changes in 0.4.0Hide the rest of the changes in 0.4.0

Performance improvements

  • The log viewer serves log files as-is rather than parsing and re-serializing them, making opening a log effectively instant (previously several seconds for logs tens of MBs in size).
  • vitals_view() now reads only the leading bytes of each log file when

generating the homepage; listing a directory of large logs is roughly 250x

faster.

  • Eval log files are dramatically smaller (roughly 4x for multi-turn, tool-heavy evals). Repeated content in a sample's events and base64 images in its messages are now de-duplicated into the sample's attachments pool, mirroring Python Inspect's behavior.

Bug fixes and minor improvements

  • $log() now records its fallback temporary directory on the task and reports the path it wrote to when no VITALS_LOG_DIR is set, so the automatic $view() succeeds.
  • Task$new() now assigns a valid task name when given an inline (unnamed) dataset, so $log() succeeds.
  • vitals_bundle() now tolerates logs whose results contain no scores.
  • vitals_view() will now show correct metadata for each log. Previously, the log viewer could display one log's metadata (task name, model, score) in place of another's, both in the log listing and when clicking into a log (#208).
  • detect_pattern() now supports case_sensitive = TRUE on R 4.5 and later.
  • The log viewer will now appropriately display tool calls called in parallel.

From NEWS.md in the release

Install

Health

CRAN checks
13OK
Slowest check: 2.3 min · r-oldrel-windows-x86_64
Status changes
  • OK13 OK2026-08-04
  • NOTE12 OK, 1 NOTE2026-08-01
  • OK13 OK2026-06-09
  • ERROR12 OK, 1 ERROR2026-06-07
  • OK12 OK2026-04-25
Show 1 earlier changeHide 1 earlier change
  • NOTE11 OK, 3 NOTE2026-03-10
Advisories

none known

Documentation

In release 0.4.0
READMEYes · 369 wordsVignettesNoNEWSYes · 100% structured · mentions 0.4.0Citation fileNoWebsiteYes · pkgdown

The repository also keeps web-only articles.

Help pages
13 (1 internal)includes a package overview page
Examples that run
100%12 run in a plain check (7 of them under a condition)
Exported functions with examples
100%
Documented parameters
100%
Return-value docs
91%
Help pages citing references
0%

Downloads

4.8K
CRAN downloads in the past year
Rank #5,149 · ~13/day · ~397/mo
Daily download trend is not available in this view yet.
52930 days
1.6K90 days
4.8K1 year
Compare downloads with other packages →
Also on649 r2u1 autocran

Repository

61Stars
13Forks
7Open issues
0Open PRs
4Releases
469Commits
12Commit identities
469 commits · Last activity 2026-09-02 · +1.7% stars, 30d · Default branch at 0.4.0.9000, ahead of CRAN 0.4.0 (checked 2026-10-21)
Responsiveness
How these are sampled
PRs merged (all time)
92%
Median days to close, 50 most recently updated closed issues
4
Median days to close, 50 most recently updated closed PRs
0

Stars over time

2026-07-14 · 572026-09-24 · 61
PracticesBeta

13 development-tooling and community-health practices detected across 7 families in the upstream repository

Checks run against github.com/tidyverse/vitals on 2026-10-21.

Continuous integration (2)
GitHub ActionsR CMD check workflow

The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel

Governance and community (1)
Code of conduct
Docs source (4)
README.Rmdpkgdown

GitHub Pages site last deployed 2026-09-02 · The repository has a site deploy workflow

pkgdown site last built 2026-09-02

Coverage (1)
Coverage reporting (Codecov)
Show all practicesHide 3 practices
Reproducibility and dev environment (1)
data-raw/
CRAN release process (1)
cran-comments.md
Lint, format, editor (3)
AirVS CodeRStudio project
How this is detected·Detection ruleset v4 (2026-10-20)
AI coding toolsBeta

This repository shows signs of AI coding tools.

  • Claude Codefirst seen 2025-03-04
    Seen in the repository: .claude, .claude/agents, CLAUDE.md
    Named in .Rbuildignore: .claude, CLAUDE.md
    Pull request it wrote: opened by a project member
  • GitHub Copilotfirst seen 2025-11-20
    Pull request: opened by its own account
    Commits: 1 by its own account · at least 1 crediting it in the message
Counts are commits, not lines of code. Commits by a tool's accounts count once across all of its accounts. Commits crediting it are the largest count among its credit lines, a minimum.How this is detected·Detection ruleset v4 (updated 2026-09-27)

Dependencies

Declared dependencies
23 external dependencies (excludes base and recommended)

Installing vitals adds 35 packages (22 with compiled code, 31 MB of source tarballs). Heaviest: stringi 11 MB, promises 3.1 MB, ellmer 3.0 MB.

Depends (1)
R >= 4.1
LinkingTo (0)
none
Enhances (0)
none
Reverse dependencies
0direct
0indirect
Suggested by 1
Suggests, most downloaded first

Datasets

People & History

People (5)
Author, Maintainer since 0.1.0 (2025)
Contributor
Contributor
Copyright holder, Funder
Releases (4)

Pick two releases to compare their code metrics.

  • 0.4.0Latest
    record
    2026-05-15 · current release · diff ↗ ·
    added 3 exports
    claude_code, codex, vitals_log_read
  • 0.3.0
    record
    2026-05-15 · diff ↗ ·
    added 1 export
    generate_structured
  • R
    R 4.6.0 released · 2026-04-24
  • 0.2.0
    record
    2025-12-01 · diff ↗
  • 0.1.0
    record
    2025-06-24
  • R
    R 4.5.0 released · 2025-04-11

Package metadata

First published
2025-06-24
Total releases
4 / 1 yrs
License
MIT + file LICENSE OSI
Minimum R
≥ 4.1
Bundled data
14 KB / 1 file
Download size
3.3 MB

Cite

Cite this package

Run in R for the authors' preferred citation:

citation("vitals")
Couch, S., Posit Software, PBC, Cetinkaya-Rundel, M., Kuhn, M., & Wickham, H. (2026). vitals: Large Language Model Evaluation (Version 0.4.0) [Computer software]. https://doi.org/10.32614/CRAN.package.vitals

This package has no citation file, so citation() returns this entry built from DESCRIPTION.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

APA

Balamuta, J. J. (2026). R Observatory: Metrics for vitals version 0.4.0 [Data set]. HJJB, LLC. Data release v2026-10-23. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-23, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →

Privacy choices

These apply to this browser and are stored on this device only. Nothing about your choice is sent to us.

Read the privacy policy