morphemepiece

1.2.3

Morpheme Tokenization

0packages depend

3.5Kdownloads / year

100.0%test coverage

0/13checks pass

Overview

About

Maintained by Jonathan BrattFirst published 2021-09-096 releasesCRAN page ↗GitHub ↗

Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("morphemepiece", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("macmillancontentscience/morphemepiece")

Development version from macmillancontentscience/morphemepiece. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-morphemepiece

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-morphemepiece

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-morphemepiece

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed).

Health

CRAN checks

13NOTE

Failing flavors

NOTE r-devel-linux-x86_64-debian-clang
NOTE r-devel-linux-x86_64-debian-gcc
NOTE r-devel-linux-x86_64-fedora-clang
NOTE r-devel-linux-x86_64-fedora-gcc
NOTE r-devel-windows-x86_64
NOTE r-oldrel-macos-arm64
NOTE r-oldrel-macos-x86_64
NOTE r-oldrel-windows-x86_64
NOTE r-patched-linux-x86_64
NOTE r-release-linux-x86_64
NOTE r-release-macos-arm64
NOTE r-release-macos-x86_64
NOTE r-release-windows-x86_64

Slowest check: 1.3 min · r-release-macos-x86_64

Code health

Yes

Tests · ratio 0.17

100.0%

Coverage · measured lines

100%

Documentation · exports

Dependencies · direct

Check history

NOTE2026-06-09
0 OK · 13 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE
ERROR2026-06-08
0 OK · 12 NOTE · 0 WARNING · 1 ERROR · 0 FAILURE
NOTE2026-03-10
0 OK · 14 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE

Documentation

READMEYes · 109 wordsVignettesYes · dynamicpkgdown siteNoNEWSYes · 100% structuredCode of conductNoContributing guideNo

Examples that run: 100%
Documented parameters: 100%
Return-value docs: 89%
References docs: 0%

Downloads

3.5K

CRAN downloads in the past year

Rank #13,779 · ~10/day · ~292/mo

Releases R versions

Daily download trend is not available in this view yet.

21930 days

79290 days

3.5K1 year

Compare downloads with other packages →

Also on94 r2u11 autocran

Repository

GitHub ↗ macmillancontentscience/morphemepiece

11Stars

0Forks

4Open issues

0Open PRs

2Releases

27Commits

2Contributors

27 commits · Last activity 2022-04-15

Stars over time

2024-10-03 · 112026-07-07 · 11

Repository practices

Upstream repositoryBeta

2 development-tooling and community-health practices detected across 2 families in the upstream repository

Checks run against github.com/macmillancontentscience/morphemepiece on 2026-07-19.

CRAN release process (1)

cran-comments.md

Docs source (1)

README.Rmd

How this is detected·Detection ruleset v1 (2026-07-18)

Dependencies

Declared dependencies

18 external dependencies (excludes base and recommended)

Depends (0)

none

Imports (10)

dlr fastmatch magrittr memoise morphemepiece.data piecemaker purrr readr rlang stringr

LinkingTo (0)

none

Suggests (9)

dplyr fs ggplot2 here knitr remotes rmarkdown testthatutils

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

0direct

0indirect

Nothing depends on this yet.

Code & Tests

People & History

People (3)

Maintainer (1)

Jonathan Bratt

Author, Maintainer

Authors (2)

Jonathan Bratt

Author, Maintainer

Jon Harmon

Author

Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning

Package Timeline

6 releases. Pick two to compare their code metrics. R releases are shown for context.

R
R 4.6.0 released · 2026-04-24
R
R 4.5.0 released · 2025-04-11
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
R
R 4.2.0 released · 2022-04-22
1.2.3Latest
2022-04-16 · current release · diff ↗
1.2.1
2022-04-08 · diff ↗
1.1.2
2021-12-06 · diff ↗
1.1.0
2021-10-27 · diff ↗
1.0.1
2021-09-17 · diff ↗
1.0.0
2021-09-09
R
R 4.1.0 released · 2021-05-18

Package metadata

First published: 2021-09-09
Total releases: 6 / 5 yrs
License: Apache License (>= 2)
Download size: 55 KB
Installed size: not tracked yet
With dependencies: not tracked yet

Report a problem with this page →