R Observatory: Metrics for tokenizers version 0.3.0

Balamuta, James Joseph

doi:10.5281/zenodo.21843040

tokenizers

0.3.0

Fast, Consistent Tokenization of Natural Language Text

17packages depend

503.7Kdownloads / year

98.2%test coverage

8/13checks pass

Overview

About

Maintained by Lincoln MullenFirst published 2016-04-029 releasesCRAN page ↗GitHub ↗

Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("tokenizers", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("ropensci/tokenizers")

Development version from ropensci/tokenizers. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-tokenizers

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-tokenizers

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-tokenizers

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

5NOTE8OK

Failing flavors

NOTE r-devel-linux-x86_64-debian-clang
NOTE r-devel-linux-x86_64-debian-gcc
NOTE r-devel-linux-x86_64-fedora-clang
NOTE r-devel-linux-x86_64-fedora-gcc
NOTE r-devel-windows-x86_64

Slowest check: 3.3 min · r-devel-windows-x86_64

Code health

Yes

Tests · ratio 1.64

98.2%

Coverage · measured lines

100%

Documentation · exports

3

Dependencies · direct

Check history

NOTE2026-04-22
12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-04-18
11 OK · 2 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
NOTE2026-03-10
12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Documentation

READMEYes · 437 wordsVignettesYes · dynamicpkgdown siteNoNEWSYes · 100% structuredCode of conductNoContributing guideNo

Examples that run: 86%
Documented parameters: 100%
Return-value docs: 86%
References docs: 11%

Downloads

503.7K

CRAN downloads in the past year

Rank #633 · ~1.4K/day · ~42K/mo

Releases R versions

Daily download trend is not available in this view yet.

28.2K30 days

93.3K90 days

503.7K1 year

Compare downloads with other packages →

Also on2.3K r2u19 autocran7.3K conda_forge317 c2d4u

Repository

GitHub ↗ ropensci/tokenizers

187Stars

24Forks

1Open issues

0Open PRs

7Releases

211Commits

14Contributors

text-miningtokenizerrstatsnlprr-packagepeer-reviewed

211 commits · Last activity 2024-03-27 · 0% stars, 30d

Stars over time

2025-03-07 · 1862026-07-14 · 187

Repository practices

Upstream repositoryBeta

10 development-tooling and community-health practices detected across 6 families in the upstream repository

Checks run against github.com/ropensci/tokenizers on 2026-08-16.

Continuous integration (2)

Travis CIAppVeyor

Reproducibility and dev environment (3)

data-raw/MakefileDockerfile

CRAN release process (2)

cran-comments.mdCRAN-SUBMISSION

Docs source (1)

README.Rmd

Show all practices

Research and citation (1)

JOSS paper

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v1 (2026-07-18)

Dependencies

Declared dependencies

9 external dependencies (excludes base and recommended)

Depends (1)

R >= 3.1.3

Imports (3)

stringi Rcpp SnowballC

LinkingTo (1)

Rcpp

Suggests (5)

covr knitr rmarkdown stopwords testthat

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

17direct

43indirect

WhatsR blocking covfefe deeplr easieRnmt pdfsearch pkgmatch proustr rslp rwig textrecipes tidytext wactor edgarWebR llmshieldr sumup torchdatasets

Code & Tests

Datasets

People & History

People (5)

Maintainer (1)

Lincoln Mullen

Author, Maintainer

Authors (1)

Lincoln Mullen

Author, Maintainer

Contributors (4)

Jeffrey Arnold

Contributor · added in 0.2.0

Kenneth Benoit

Contributor · added in 0.2.0

Os Keyes

Contributor · added in 0.2.0

Dmitriy Selivanov

Contributor

Package Timeline

9 releases. Pick two to compare their code metrics. R releases are shown for context.

R
R 4.6.0 released · 2026-04-24
R
R 4.5.0 released · 2025-04-11
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
0.3.0Latest
2022-12-22 · current release · diff ↗
0.2.3
2022-09-23 · diff ↗
R
R 4.2.0 released · 2022-04-22
R
R 4.1.0 released · 2021-05-18
R
R 4.0.0 released · 2020-04-24
R
R 3.6.0 released · 2019-04-26
R
R 3.5.0 released · 2018-04-23
0.2.1
2018-03-29 · diff ↗
0.2.0
2018-03-21 · diff ↗
R
R 3.4.0 released · 2017-04-21
0.1.4
2016-08-29 · diff ↗
0.1.3
2016-08-18 · diff ↗

Show 5 earlier events

R
R 3.3.0 released · 2016-05-03
0.1.2
2016-04-14 · diff ↗
0.1.1
2016-04-04 · diff ↗
0.1.0
2016-04-02
R
R 3.2.0 released · 2015-04-16

Package metadata

First published: 2016-04-02
Total releases: 9 / 10 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 3.1.3
Bundled data: 379 KB / 1 file
Download size: 448 KB
Installed size: not tracked yet
With dependencies: not tracked yet

Appears in task views

Natural Language Processing

Cite

Cite this package

Run in R for the authors' preferred citation:

citation("tokenizers")

The authors publish a JOSS paper, so they have said how they want this cited. Run citation() and use what it gives you.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

Format

APA

Balamuta, J. J. (2026). R Observatory: Metrics for tokenizers version 0.3.0 [Data set]. HJJB, LLC. Data release v2026-08-18. https://doi.org/10.5281/zenodo.21843040

From data release v2026-08-18, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →