sentencepiece

0.2.5

Text Tokenization using Byte Pair Encoding and Unigram Modelling

1packages depend

16Kdownloads / year

–test coverage

13/13checks pass

Overview

About

Maintained by Jan WijffelsFirst published 2020-06-048 releasesCRAN page ↗GitHub ↗

Unsupervised text tokenizer allowing to perform byte pair encoding and unigram modelling. Wraps the 'sentencepiece' library https://github.com/google/sentencepiece which provides a language independent tokenizer to split text in words and smaller subword units. The techniques are explained in the paper "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" by Taku Kudo and John Richardson (2018) doi:10.18653/v1/D18-2012. Provides as well straightforward access to pretrained byte pair encoding models and subword embeddings trained on Wikipedia using 'word2vec', as described in "BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages" by Benjamin Heinzerling and Michael Strube (2018) http://www.lrec-conf.org/proceedings/lrec2018/pdf/1049.pdf.

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("sentencepiece", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("bnosac/sentencepiece")

Development version from bnosac/sentencepiece. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-sentencepiece

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-sentencepiece

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-sentencepiece

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

13OK

Slowest check: 15.5 min · r-oldrel-windows-x86_64

Code health

None

Tests · ratio 0.00

not tracked

Coverage

100%

Documentation · exports

Dependencies · direct

Check history

OK2026-06-08
13 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-06-07
12 OK · 0 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
OK2026-04-25
12 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
NOTE2026-04-22
11 OK · 3 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-04-18
10 OK · 3 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE

Show 3 earlier snapshots

NOTE2026-04-10
11 OK · 3 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-04-09
10 OK · 3 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
NOTE2026-03-10
11 OK · 3 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Documentation

READMEYes · 254 wordsVignettesNopkgdown siteNoNEWSYes · 100% structuredCode of conductNoContributing guideNo

Examples that run: 100%
Documented parameters: 94%
Return-value docs: 100%
References docs: 0%

Downloads

16K

CRAN downloads in the past year

Rank #10,736 · ~44/day · ~1.3K/mo

Releases R versions

Daily download trend is not available in this view yet.

31730 days

1K90 days

16K1 year

Compare downloads with other packages →

Also on473 r2u33 autocran57 c2d4u

Repository

GitHub ↗ bnosac/sentencepiece

29Stars

6Forks

2Open issues

1Open PRs

9Releases

83Commits

2Contributors

sentencepiecebyteword-segmentationnatural-language-processing

License MPL-2.0 · 83 commits · Last activity 2026-02-09 · 0% stars, 30d

Stars over time

2025-07-01 · 262026-07-07 · 29

Repository practices

Upstream repositoryBeta

2 development-tooling and community-health practices detected across 2 families in the upstream repository

Checks run against github.com/bnosac/sentencepiece on 2026-07-19.

Continuous integration (1)

GitHub Actions

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v1 (2026-07-18)

Dependencies

Declared dependencies

4 external dependencies (excludes base and recommended)

Depends (1)

R >= 2.10

Imports (2)

Rcppstats

LinkingTo (1)

Rcpp

Suggests (2)

tokenizers.bpe word2vec

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

1direct

0indirect

textrecipes

Code & Tests

Code Composition

C 177,403 (83%)C++ 32,221 (15%)R 2,824 (1%)Rd 604 (0%)

Code characteristics

Object systems

S33

Cyclomatic complexity: 1.0 median / 10 max
System requirements: 1 external

Test coverage

Line coverage

–

Expression

–

Tests / Examples

–

Functions

2603 10 exported

Complexity

2.7 avg / 10 max

Call network

2603 nodes / 3709 edges

Call graph

Open call graph →

Loading call graph…

Lowest coverage

Per-function coverage is not measured for this package yet.

People & History

People (19)

Maintainer (1)

Jan Wijffels

Author, Maintainer, Copyright holder

Authors (1)

Jan Wijffels

Author, Maintainer, Copyright holder

Contributors (17)

Anton Carver (Google Inc.)

Contributor, Copyright holder

Benjamin Heinzerling

Contributor, Copyright holder

Chris Atenasio (Google Inc.)

Contributor, Copyright holder

Daisuke Okanohara

Contributor, Copyright holder

Google Inc.

Contributor, Copyright holder

Google Inc.

Contributor, Copyright holder

Jason Hsueh (Google Inc.)

Contributor, Copyright holder

Jeff Dean (Google Inc.)

Contributor, Copyright holder

Show 9 more contributors

Jim Meehan (Google Inc.)

Contributor, Copyright holder

Kenton Varda (Google Inc.)

Contributor, Copyright holder

Laszlo Csomor (Google Inc.)

Contributor, Copyright holder

Maxim Lifantsev (Google Inc.)

Contributor, Copyright holder

Sanjay Ghemawat (Google Inc.)

Contributor, Copyright holder

Susumu Yata

Contributor, Copyright holder

The Abseil Authors

Contributor, Copyright holder

Wink Saville (Google Inc.)

Contributor, Copyright holder

Yuta Mori

Contributor, Copyright holder

Jan Wijffels

Author, Maintainer, Copyright holder

Anton Carver (Google Inc.)

Contributor, Copyright holder

Benjamin Heinzerling

Contributor, Copyright holder

Chris Atenasio (Google Inc.)

Contributor, Copyright holder

Daisuke Okanohara

Contributor, Copyright holder

Google Inc.

Contributor, Copyright holder

Google Inc.

Contributor, Copyright holder

Jason Hsueh (Google Inc.)

Contributor, Copyright holder

Show 11 more copyright holders

Jeff Dean (Google Inc.)

Contributor, Copyright holder

Jim Meehan (Google Inc.)

Contributor, Copyright holder

Kenton Varda (Google Inc.)

Contributor, Copyright holder

Laszlo Csomor (Google Inc.)

Contributor, Copyright holder

Maxim Lifantsev (Google Inc.)

Contributor, Copyright holder

Sanjay Ghemawat (Google Inc.)

Contributor, Copyright holder

Susumu Yata

Contributor, Copyright holder

The Abseil Authors

Contributor, Copyright holder

Wink Saville (Google Inc.)

Contributor, Copyright holder

Yuta Mori

Contributor, Copyright holder

BNOSAC

Package Timeline

8 releases. Pick two to compare their code metrics. R releases are shown for context.

R
R 4.6.0 released · 2026-04-24
0.2.5Latest
2026-02-09 · current release · diff ↗
0.2.4
2025-11-27 · diff ↗
R
R 4.5.0 released · 2025-04-11
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
0.2.3
2022-11-13 · diff ↗
0.2.2
2022-11-09 · diff ↗
R
R 4.2.0 released · 2022-04-22
0.2.1
2021-12-21 · diff ↗
0.2
2021-12-15 · diff ↗
R
R 4.1.0 released · 2021-05-18
0.1.2
2020-06-08 · diff ↗
0.1.1
2020-06-04
R
R 4.0.0 released · 2020-04-24

Package metadata

First published: 2020-06-04
Total releases: 8 / 6 yrs
License: MPL-2.0 OSI
Minimum R: ≥ 2.10
Download size: 2.9 MB
Installed size: not tracked yet
With dependencies: not tracked yet

Appears in task views

Natural Language Processing

Report a problem with this page →