R Observatory: Metrics for wordpiece version 2.1.3

Balamuta, James Joseph

doi:10.5281/zenodo.21843040

wordpiece

2.1.3Package record

R Implementation of Wordpiece Tokenization

1packages dependsuggested by 1

9.6Kdownloads / year

100.0%test coverage

0/13flavors OK

Overview

About

Maintained by Jonathan BrattCRAN page ↗GitHub ↗Issues ↗

Apply 'Wordpiece' (arXiv:1609.08144) tokenization to input text, given an appropriate vocabulary. The 'BERT' (arXiv:1810.04805) tokenization conventions are used by default.

Changes in 2.1.3

Implemented various speed improvements, improving speed and memory usage by over 1000x. (#27, @jonathanbratt)
Removed purrr dependency. (#30, @jonthegeek)

From NEWS.md in the release

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("wordpiece", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("macmillancontentscience/wordpiece")

Development version from macmillancontentscience/wordpiece. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-wordpiece

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled. We list this channel from recorded r2u download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-wordpiece

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-wordpiece

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed). We list this channel from recorded autoCRAN download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

Health

CRAN checks

13NOTE

Check messages

NOTERd filesr-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc, r-devel-linux-x86_64-fedora-clang, r-devel-linux-x86_64-fedora-gcc, r-devel-windows-x86_64, r-oldrel-macos-arm64, r-oldrel-macos-x86_64, r-oldrel-windows-x86_64, r-patched-linux-x86_64, r-release-linux-x86_64, r-release-macos-arm64, r-release-macos-x86_64, r-release-windows-x86_64

checkRd: (-1) wordpiece_cache_dir.Rd:16-17: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:18-19: Lost braces in \itemize; meant \describe ?
checkRd: (-1) wordpiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ?

NOTECRAN incoming feasibilityr-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc

Maintainer: ‘Jonathan Bratt <jonathan.bratt@macmillan.com>’

The Description field contains
  Apply 'Wordpiece' (<arXiv:1609.08144>) tokenization to input text,
  given an appropriate vocabulary. The 'BERT' (<arXiv:1810.04805>)
Please refer to arXiv e-prints via their arXiv DOI <doi:10.48550/arXiv.YYMM.NNNNN>.

CRAN check page ↗

Slowest check: 1.3 min · r-devel-windows-x86_64

Status changes

NOTE13 NOTENow
NOTE14 NOTE2026-03-10

Advisories

none known

How this is measured·How advisories are found

Documentation

In release 2.1.3

READMEYes · 142 wordsVignettesYes · 1 · dynamicNEWSYes · 100% structured · mentions 2.1.3Citation fileNoWebsitenot declared in DESCRIPTION

Help pages: 15 (9 internal)
Examples that run: 100%3 run in a plain check
Exported functions with examples: 43%
Documented parameters: 100%
Return-value docs: 86%
Help pages citing references: 0%

Vignettes (1)

Using wordpiece HTML

How these are measured

Downloads

9.6K

CRAN downloads in the past year

Rank #13,323 · ~26/day · ~797/mo

Releases R versions

Daily download trend is not available in this view yet.

39430 days

85090 days

9.6K1 year

Compare downloads with other packages →

Also on83 r2u3 autocran

How downloads are counted

Repository

macmillancontentscience/wordpiece ↗Organization: macmillancontentscience

8Stars

1Forks

4Open issues

0Open PRs

3Releases

21Commits

2Commit identities

21 commits · Last activity 2022-03-03

Responsiveness

How these are sampled

PRs merged (all time): 100%
Median days to close, 50 most recently updated closed issues: 8
Median days to close, 50 most recently updated closed PRs: 0

How these are measured

PracticesBeta

5 development-tooling and community-health practices detected across 4 families in the upstream repository

Checks run against github.com/macmillancontentscience/wordpiece on 2026-10-11.

Governance and community (2)

Code of conductContributing guide

Docs source (1)

README.Rmd

CRAN release process (1)

cran-comments.md

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v3 (2026-09-26)

Dependencies

Declared dependencies

11 external dependencies (excludes base and recommended)

Installing wordpiece adds 18 packages (12 with compiled code, 16 MB of source tarballs). Heaviest: stringi 11 MB, fs 1.4 MB, vctrs 1.0 MB.

Depends (1)

R >= 3.3.0

Imports (7)

dlr fastmatch memoise piecemaker rlang stringi wordpiece.data

LinkingTo (0)

none

Suggests (4)

covr knitr rmarkdown testthat

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

1direct

0indirect

Imports 1

Suggested by 1

Depends, Imports or LinkingTo, most downloaded first

Rhobots

Suggests, most downloaded first

textrecipes

How these are counted

Code & Tests

Datasets

People & History

People (3)

Jonathan Bratt

Author, Maintainer since 1.0.2 (2021)

Jon Harmon

Author

Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning

Copyright holder

Releases (3)

Pick two releases to compare their code metrics.

R
R 4.6.0 released · 2026-04-24
R
R 4.5.0 released · 2025-04-11
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
R
R 4.2.0 released · 2022-04-22
2.1.3Latest
record
2022-03-03 · current release · diff ↗
2.0.1
record
2021-10-18 · diff ↗ ·
removed 1 export
get_cache_dir

added 4 exports
prepare_vocab, set_wordpiece_cache_dir, wordpiece_cache_dir, wordpiece_vocab
R
R 4.1.0 released · 2021-05-18
1.0.2
record
2021-02-11
R
R 4.0.0 released · 2020-04-24

Package metadata

First published: 2021-02-11
Total releases: 3 / 5 yrs
License: Apache License (>= 2)
Minimum R: ≥ 3.3.0
Download size: 19 KB

Cite

Cite this package

Run in R for the authors' preferred citation:

citation("wordpiece")

Format

Bratt, J., Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning, & Harmon, J. (2022). wordpiece: R Implementation of Wordpiece Tokenization (Version 2.1.3) [Computer software]. https://doi.org/10.32614/CRAN.package.wordpiece

This package has no citation file, so citation() returns this entry built from DESCRIPTION.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

Format

APA

Balamuta, J. J. (2026). R Observatory: Metrics for wordpiece version 2.1.3 [Data set]. HJJB, LLC. Data release v2026-10-18. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-18, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →