tokenizers
Fast, Consistent Tokenization of Natural Language Text
Description
Convert natural language text into tokens. Includes tokenizers for shingled n-grams, skip n-grams, words, word stems, sentences, paragraphs, characters, shingled characters, lines, Penn Treebank, regular expressions, as well as functions for counting characters, words, and sentences, and a function for splitting longer texts into separate documents, each with the same number of words. The tokenizers have a consistent interface, and the package is built on the 'stringi' and 'Rcpp' packages for fast yet correct tokenization in 'UTF-8'.
Downloads
31.6K
Last 30 days
637th
116.7K
Last 90 days
511.1K
Last year
Trend: -22.1% (30d vs prior 30d)
117
Last 30 days
501
Last 90 days
2.3K
Last year
Trend: -29.9% (30d vs prior 30d)
9
Last 7 days
22
Last 30 days
0
All-time
autoCRAN-only: this name is served only by autoCRAN, so the count is exact.
365
Last 30 days
1.2K
Last 90 days
7.2K
Last year
Trend: -27.9% (30d vs prior 30d)
CRAN Check Status
Show all 13 flavors
| Flavor | Status |
|---|---|
| r-devel-linux-x86_64-debian-clang | NOTE |
| r-devel-linux-x86_64-debian-gcc | NOTE |
| r-devel-linux-x86_64-fedora-clang | OK |
| r-devel-linux-x86_64-fedora-gcc | OK |
| r-devel-windows-x86_64 | OK |
| r-oldrel-macos-arm64 | OK |
| r-oldrel-macos-x86_64 | OK |
| r-oldrel-windows-x86_64 | OK |
| r-patched-linux-x86_64 | OK |
| r-release-linux-x86_64 | OK |
| r-release-macos-arm64 | OK |
| r-release-macos-x86_64 | OK |
| r-release-windows-x86_64 | OK |
Check details (2 non-OK)
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
Check History
NOTE 12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Apr 22, 2026
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
ERROR 11 OK · 2 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE Apr 18, 2026
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
whether package can be installed
Installation failed. See 'd:/Rcompile/CRANpkg/local/4.6/tokenizers.Rcheck/00install.out' for details.
NOTE 12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 10, 2026
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
CRAN incoming feasibility
Maintainer: ‘Lincoln Mullen <lincoln@lincolnmullen.com>’ Package CITATION file contains call(s) to old-style personList() or as.personList(). Please use c() on person objects instead. Package CITATION file contains call(s) to old-style citEntry(). Please use bibentry() instead.
Code & tests
Open call graph →Code intelligence has not been computed for this package yet.
Code
Structure
Lines of code
3,448
Files
56
Compiled share
6.8%
Has compiled src
Yes
Language breakdown
API
Exported functions
15
Internal functions
16
Recent export changes
Testing & CI
Has tests
Yes
Test-to-code ratio
1.64
testthat edition
–
CI present
No
CI type
[]
PR gated
No
Docs
Return-value doc rate
85.7%
\dontrun example ratio
14.3%
Roxygen coverage
100%
Has pkgdown
No
NEWS present
Yes
Health & Security signals
Informational signals; not verdicts.
on.exit coverage
–
Unsafe pattern score
0
Dep constraint coverage
100%
Secret pattern count
0
Bundled 3rd-party code
2 items
Portability & License
Min R version
3.1.3
System requirements
–
C++ standard
–
License
MIT + file LICENSE
License flags
SPDX valid, OSI approved
History
Versions
9
First release
2016-04-02
Latest release
2022-12-22
Avg cadence
51 days
Cold removal rate
100%
Dep drift
0
LOC over versions
Per-file churn detail lives in the source pipeline: https://github.com/r-observatory/cran-code-metrics.