Morpheme Tokenization
Overview
Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.
- Update tests to be independent of morphemepiece.data version.
From NEWS.md in the release
Install
Health
NOTERd filesr-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc, r-devel-linux-x86_64-fedora-clang, r-devel-linux-x86_64-fedora-gcc, r-devel-windows-x86_64, r-oldrel-macos-arm64, r-oldrel-macos-x86_64, r-oldrel-windows-x86_64, r-patched-linux-x86_64, r-release-linux-x86_64, r-release-macos-arm64, r-release-macos-x86_64, r-release-windows-x86_64
checkRd: (-1) morphemepiece_cache_dir.Rd:17-19: Lost braces in \itemize; meant \describe ? checkRd: (-1) morphemepiece_cache_dir.Rd:20-21: Lost braces in \itemize; meant \describe ? checkRd: (-1) morphemepiece_cache_dir.Rd:22-24: Lost braces in \itemize; meant \describe ?
NOTER code for possible problemsr-devel-linux-x86_64-debian-clang, r-devel-linux-x86_64-debian-gcc, r-devel-linux-x86_64-fedora-clang, r-devel-linux-x86_64-fedora-gcc, r-devel-windows-x86_64
Found calls to structure() using deprecated special names: morphemepiece/tests/testthat/test-morphemepiece.R (.Names: 2) '.Names' should be changed to 'names'.
- NOTE13 NOTE2026-06-09
- ERROR12 NOTE, 1 ERROR2026-06-08
- NOTE14 NOTE2026-03-10
Documentation
- Help pages
- 18 (9 internal)includes a package overview page
- Examples that run
- 100%1 run in a plain check
- Exported functions with examples
- 11%
- Documented parameters
- 100%
- Return-value docs
- 89%
- Help pages citing references
- 0%
Downloads
Repository
- PRs merged (all time)
- 100%
- Median days to close, 50 most recently updated closed issues
- 169
- Median days to close, 50 most recently updated closed PRs
- 0
Stars over time
Releases over time
Issues over time
PRs over time
4 development-tooling and community-health practices detected across 3 families in the upstream repository
Checks run against github.com/macmillancontentscience/morphemepiece on 2026-10-21.
Dependencies
Installing morphemepiece adds 37 packages (21 with compiled code, 23 MB of source tarballs). Heaviest: stringi 11 MB, morphemepiece.data 3.4 MB, fs 1.4 MB.
Code & Tests
People & History
Pick two releases to compare their code metrics.
- RR 4.6.0 released · 2026-04-24
- RR 4.5.0 released · 2025-04-11
- RR 4.4.0 released · 2024-04-24
- RR 4.3.0 released · 2023-04-21
- RR 4.2.0 released · 2022-04-22
- 1.2.3Latest
- 1.2.12022-04-08 · diff ↗
- 1.1.22021-12-06 · diff ↗
- 1.1.02021-10-27 · diff ↗
- 1.0.12021-09-17 · diff ↗
- 1.0.02021-09-09
- RR 4.1.0 released · 2021-05-18
Package metadata
- First published
- 2021-09-09
- Total releases
- 6 / 5 yrs
- License
- Apache License (>= 2)
- Download size
- 55 KB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("morphemepiece")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-24, which the citation names so these numbers can be found later. More on citing and the projects behind them.