rtiktoken
A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language Models
v0.0.7
·
Apr 14, 2025
·
MIT + file LICENSE
Description
A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.
Downloads
235
Last 30 days
19319th
635
Last 90 days
2.6K
Last year
Trend: +18.1% (30d vs prior 30d)
CRAN Check Status
9
NOTE
5
OK
Show all 14 flavors
| Flavor | Status |
|---|---|
| r-devel-linux-x86_64-debian-clang | NOTE |
| r-devel-linux-x86_64-debian-gcc | NOTE |
| r-devel-linux-x86_64-fedora-clang | NOTE |
| r-devel-linux-x86_64-fedora-gcc | NOTE |
| r-devel-macos-arm64 | OK |
| r-devel-windows-x86_64 | NOTE |
| r-oldrel-macos-arm64 | NOTE |
| r-oldrel-macos-x86_64 | NOTE |
| r-oldrel-windows-x86_64 | NOTE |
| r-patched-linux-x86_64 | NOTE |
| r-release-linux-x86_64 | OK |
| r-release-macos-arm64 | OK |
| r-release-macos-x86_64 | OK |
| r-release-windows-x86_64 | OK |
Check details (9 non-OK)
NOTE
r-devel-linux-x86_64-debian-clang
compiled code
File ‘rtiktoken/libs/rtiktoken.so’:
Found non-API calls to R: ‘R_NamespaceRegistry’, ‘R_Srcref’,
‘R_UnboundValue’
Compiled code should not call non-API entry points in R.
See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual,
and section ‘Moving into C API compliance’ for issues with the use of
non-API entry points.
NOTE
r-devel-linux-x86_64-debian-gcc
compiled code
File ‘rtiktoken/libs/rtiktoken.so’: Found non-API calls to R: ‘R_NamespaceRegistry’, ‘R_UnboundValue’ Compiled code should not call non-API entry points in R. See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual, and section ‘Moving into C API compliance’ for issues with the use of non-API entry points.
NOTE
r-devel-linux-x86_64-fedora-clang
compiled code
File ‘rtiktoken/libs/rtiktoken.so’: Found non-API calls to R: ‘R_NamespaceRegistry’, ‘R_UnboundValue’ Compiled code should not call non-API entry points in R. See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual, and section ‘Moving into C API compliance’ for issues with the use of non-API entry points.
NOTE
r-devel-linux-x86_64-fedora-gcc
compiled code
File ‘rtiktoken/libs/rtiktoken.so’:
Found non-API calls to R: ‘R_NamespaceRegistry’, ‘R_Srcref’,
‘R_UnboundValue’
Compiled code should not call non-API entry points in R.
See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual,
and section ‘Moving into C API compliance’ for issues with the use of
non-API entry points.
NOTE
r-devel-windows-x86_64
compiled code
File 'rtiktoken/libs/x64/rtiktoken.dll':
Found non-API calls to R: 'R_NamespaceRegistry', 'R_Srcref',
'R_UnboundValue'
Compiled code should not call non-API entry points in R.
See 'Writing portable packages' in the 'Writing R Extensions' manual,
and section 'Moving into C API compliance' for issues with the use of
non-API entry points.
NOTE
r-oldrel-macos-arm64
installed package size
installed size is 12.7Mb
sub-directories of 1Mb or more:
libs 12.6Mb
NOTE
r-oldrel-macos-x86_64
installed package size
installed size is 12.5Mb
sub-directories of 1Mb or more:
libs 12.4Mb
NOTE
r-oldrel-windows-x86_64
installed package size
installed size is 9.8Mb
sub-directories of 1Mb or more:
libs 9.7Mb
NOTE
r-patched-linux-x86_64
compiled code
File ‘rtiktoken/libs/rtiktoken.so’:
Found non-API calls to R: ‘R_MissingArg’, ‘R_NamespaceRegistry’,
‘R_Srcref’, ‘R_UnboundValue’
Compiled code should not call non-API entry points in R.
See ‘Writing portable packages’ in the ‘Writing R Extensions’ manual,
and section ‘Moving into C API compliance’ for issues with the use of
non-API entry points.
Check History
NOTE 11 OK · 3 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 10, 2026
NOTE
r-oldrel-macos-arm64
installed package size
installed size is 12.7Mb
sub-directories of 1Mb or more:
libs 12.6Mb
NOTE
r-oldrel-macos-x86_64
installed package size
installed size is 12.5Mb
sub-directories of 1Mb or more:
libs 12.4Mb
NOTE
r-oldrel-windows-x86_64
installed package size
installed size is 9.8Mb
sub-directories of 1Mb or more:
libs 9.7Mb