Tools for Preparing Text for Tokenizers
Overview
Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.
- Streamlined
validate_utf8()to work consistently across OS and R versions.
From NEWS.md in the release
Install
Health
- OK13 OK2026-08-05
- NOTE12 OK, 1 NOTE2026-08-01
- OK14 OK2026-03-10
Documentation
- Help pages
- 13 (3 internal)includes a package overview page
- Examples that run
- 100%10 run in a plain check
- Exported functions with examples
- 100%
- Documented parameters
- 100%
- Return-value docs
- 100%
- Help pages citing references
- 0%
Downloads
Repository
- PRs merged (all time)
- 100%
- Median days to close, 50 most recently updated closed issues
- 16
- Median days to close, 50 most recently updated closed PRs
- 1
Releases over time
Issues over time
PRs over time
13 development-tooling and community-health practices detected across 7 families in the upstream repository
Checks run against github.com/macmillancontentscience/piecemaker on 2026-10-18.
The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel
GitHub Pages site last deployed 2023-06-02 · The repository has a site deploy workflow
pkgdown site last built 2023-06-02
Show all practicesHide 3 practices
Dependencies
Installing piecemaker adds 8 packages (6 with compiled code, 14 MB of source tarballs). Heaviest: stringi 11 MB, vctrs 1.0 MB, rlang 766 KB.
Code & Tests
People & History
Pick two releases to compare their code metrics.
- RR 4.6.0 released · 2026-04-24
- RR 4.5.0 released · 2025-04-11
- RR 4.4.0 released · 2024-04-24
- 1.0.2Latest
- RR 4.3.0 released · 2023-04-21
- RR 4.2.0 released · 2022-04-22
- 1.0.12022-03-03 · diff ↗
- 1.0.02021-08-06
- RR 4.1.0 released · 2021-05-18
Package metadata
- First published
- 2021-08-06
- Total releases
- 3 / 5 yrs
- License
- Apache License (>= 2)
- Minimum R
- ≥ 2.10
- Download size
- 13 KB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("piecemaker")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-21, which the citation names so these numbers can be found later. More on citing and the projects behind them.