R Observatory: Metrics for tall version 1.0.1

Balamuta, James Joseph

doi:10.5281/zenodo.21843040

tall needs 2 packages that CRAN will archive unless their check problems are fixed, the earliest by Oct 22. See which.

tall

1.0.1

Text Analysis for All

0packages depend

7.4Kdownloads / year

60.6%test coverage

13/13flavors OK

Overview

About

Maintained by Massimo AriaCRAN page ↗GitHub ↗Website ↗Issues ↗

An R 'shiny' app designed for diverse text analysis tasks, offering a wide range of methodologies tailored to Natural Language Processing (NLP) needs. It is a versatile, general-purpose tool for analyzing textual data. 'tall' features a comprehensive workflow, including data cleaning, preprocessing, statistical analysis, and visualization, all integrated for effective text analysis.

Changes in 1.0.1

Bug fix (Settings > Tall AI): **a valid Gemini API key was reported as refused, and a new user could not run TALL AI at all.** Key validation sent a real generateContent request to a hard-coded gemini-2.5-flash (settings.R), and Google has stopped serving the 2.5 family to newly created API keys — so the probe returned 404 ... no longer available to new users and the app said "API key seems be not valid", whatever model was selected. The first-run default was 2.5-flash too (loadGeminiModel()), so even a key that got through had nothing it could call. Same defect as bibliometrix #637, found by carrying that fix across.
**Validation is now model-agnostic**: the new .geminiAvailableModels() asks for the model CATALOGUE (GET /v1beta/models) instead of invoking a model, so no future retirement can break it. geminiValidateKey() returns the key's verdict and the chosen model's availability as two separate answers.

Show all changes in 1.0.1Hide the rest of the changes in 1.0.1

**The selected model is checked**: if it is not in the catalogue for that key, Settings says so — instead of letting the user discover it as an HTTP 404 at the first analysis.
**Errors are told apart.** grepl("HTTP\\s*[1-5][0-9]{2}") collapsed every status from 100 to 599 into one "invalid key" message; 400/401 now mention the key, 403 a permission, 404 the model, and a transport failure says it is a connection problem. req_error() also lets Google's own message through: resp_body_string() was being handed a hand-built list, so the real reason was never shown.
**A connection failure no longer crashes the check.** The connection-level branch in gemini_ai() was commented out, leaving status_code = NA, so if (resp$status_code == 400) evaluated if (NA) and the user saw *"missing value where TRUE/FALSE needed"* instead of "there is no network".
**Defaults moved off the retired family** to gemini-3.5-flash-lite (transport default, first-run default and model selector). A stored 2.5-flash — which was the old default rather than anyone's choice — is carried forward; a deliberately chosen 2.5-flash-lite is kept. The 2.5 pair stays selectable under a **Legacy** group, since keys created before the cut-off still reach it.
**Preview ids removed from the selector** (3-flash-preview, 3.1-pro-preview): they carry their own expiry date. gemini-flash-latest added.
setGeminiAPI() removed: unreachable (no callers anywhere in the package) and a duplicate of the same faulty probe. abstractive_summary(), also unreachable and superseded by the inline block in documents.R, had its own stale 2.0-flash default corrected and is now marked as superseded.
Bug fix (Documents > Topic Modeling > Find optimal K): **the page tuned LDA with a different algorithm from the one that then estimates the model.** tmTuningAsync() fitted LDA(dtm, k, method = "VEM") while tmEstimate() fits LDA(dtm, K, method = "Gibbs", control = list(iter = 500, seed = seed)), so the number of topics a user was shown had been chosen for a model class they never fitted — two different inference algorithms, one picking K for the other. CTM was never affected (the same CTM() runs on both pages) and STM tunes through stm::searchK, which fits stm models. The tuning now uses method = "Gibbs" with tmEstimate()'s own control list, iterations included, so the model that is scored and the model that is estimated are the same fit. ⚠ **The recommended K will change on existing corpora** — that is the point of the fix, not a side effect. evaluate_single_k() is kept in step for the same reason, although nothing calls it.
Performance (Documents > Topic Modeling > Find optimal K): the grid is fitted **across cores** instead of one model after another, and each worker now also computes its own model's logLik and perplexity — which were half the wall clock when they were left to the calling process — returning only the numbers the metrics need (exp(beta) and len %*% gamma, a vector of length K) rather than the fitted model. Measured on a 22,053-unit corpus over the default grid K = 2..20: **36.7s to 5.8s, 6.4x**. The metrics come back bit-identical, which is what makes the change safe to describe as performance: every fit is independent and carries the same seed, so the result cannot depend on the order the workers finish in. If a cluster cannot be created the grid is fitted serially rather than the page failing, and on a grid small enough to finish instantly the cluster's own start-up dominates (0.2s to 0.4s on a 593-unit corpus over K = 2..6). tmTuningAsync() no longer returns the fitted models: nothing read them, and carrying them back through the socket was the single largest cost.
Bug fix (Documents > Topic Modeling > Find optimal K): on the "Multi-Metric Comparison" chart, two of the four curves were drawn upside down against the chart's own y axis, which reads "Normalized Score (0 = worst, 1 = best)". tmMultiMetricPlot() carried a per-metric flag named decreasing and its normalisation branch treated that flag as "higher is better", while the flags themselves had been assigned as though it meant "lower is better". Under LDA and CTM this put the best K of **CaoJuan2009** and of **Perplexity** — both minimised — at the BOTTOM of a chart that says the bottom is worst; under STM it did the same to **Semantic Coherence** and to the **Lower Bound**, both of which are maximised. A reader comparing curves was therefore reading two of them backwards. The flag is now higher_better, set from what each metric actually wants, and 1 is the good end for every curve on both branches (verified by execution: all eight metrics now plot their optimum at y = 1). The recommended K values do not change, and could not have: find_elbow() ignores its decreasing argument. Negating the metric reflects the whole configuration about the x axis, and the distance from a point to a line is invariant under reflection, so which.max(distances) returns the same index either way — confirmed on 2000 random metric shapes, identical 2000/2000, and on the patch itself, where every K came back unchanged. The argument is kept, because it documents which way a metric runs, but it is now labelled as having no effect so that nobody "fixes" the flags expecting a different K.
Documentation fix (Documents > Topic Modeling > Find optimal K): the "K Recommendation" tab described Deveaud 2014 as "Jensen-Shannon divergence between topic pairs. Lower = more separated topics." A divergence is larger when distributions are further apart, which is why Deveaud 2014 is maximised; the sentence said the opposite of both the statistic and the code beside it.
Bug fix (Documents > Topic Modeling > Model Estimation): the document labels of theta were wrong under every estimation method, and the two branches were wrong in two different ways. tmEstimate() (LDA and CTM) labelled each row with unique(x$doc_id)[as.numeric(row.names(tmResult$topics))], but those row names are topic_level_id values — the ANALYSIS UNIT, a sentence whenever group includes sentence_id — not positions in the list of documents, so indexing the document vector with a unit id returned NA for every unit past the number of documents and misattributed the few that did resolve. Measured on mobydick at sentence units (836 units, 10 documents): 826 of 836 labels were NA, and of the 10 that were not, only ONE was right. stmEstimate() took the first nrow(theta) entries of the unit list instead, which slides the labels as soon as the unit ids are not the leading ones — they are neither contiguous (LemmaSelection() drops units, so the ids skip: 3, 4, 5, 7, …) nor guaranteed to start at 1 (empty units are dropped before the fit). Measured on frankenstein at sentence units, 153 of 593 rows were attributed to the wrong document — silently, since no label came back NA. The visible symptom of both is "Topic by Docs Plot", whose y axis became a column of NA under LDA/CTM and a plausible but wrong set of documents under STM. It went unnoticed because the "Topics" selector defaults to Docs, where unit ids and document positions coincide. Each row is now matched back to its own unit.
Bug fix (Import > Wikipedia pages): nothing on this path was URL-encoded, and each of the three consequences was reachable from the ordinary use of the menu. A page whose title contains a non-ASCII character — Zürich, Café, Pokémon — made the API answer *400 Bad Request*, and because one failing page stops the extraction loop, a single such hit among the results left the import with nothing. A search phrase containing & had everything after it read as a new API parameter, so the query silently ran on the truncated phrase. And a phrase containing an en dash was turned into one containing a literal space by wikiSearch()'s own gsub("–", " ", …), which makes the URL illegal and aborts the import with a connection error rather than the "No results found!" message. Both the search phrase and the page title are now passed through URLencode(reserved = TRUE).
Bug fix (Documents > Summarization > Extractive): textrankDocument() joined the ranked sentences back to the corpus by the TEXT of the sentence (by = c("sentence")), not by its id. Any document that repeats a sentence therefore produced a many-to-many join: a text said k times contributed k^2 rows instead of k, and every copy inherited every paragraph that carries that text. Because nrow(s) is what abstractingDocument() takes its slider percentages of, the summary length moved with the duplication as well. On the usairlines collection a four-sentence tweet whose text repeats three times was rendered as ten sentences; mobydick_ita repeats 819 of its 12,356. The join now uses the sentence id, as the neighbouring highlightSentences() already did.
Bug fix (Documents > Summarization > Extractive): the view failed on a grouped corpus, on its own default setting. "Summarize: Documents" lists original document ids (ids() un-groups before listing), while the summary ran on values$dfTag, still keyed by group — so the selected id was absent from the data and the view stopped inside textrank_sentences() with nrow(data) > 1 is not TRUE. Grouping under FEATURES and pressing Run was enough to reach it. The frame is now un-grouped whenever the unit is not "Groups".
Bug fix (Settings > Tall AI): the chosen Gemini model and output size were never remembered. They were written to ~/.tall_gemini_model.txt and read back from ~/tall/.tall_gemini_model.txt — a different file — so every session started on 2.5-flash / medium whatever the user had picked. Two further defects in the same path: the model observer saved input$gemini_output_model, an input that does not exist (the control is gemini_output_size), so choosing a model also dropped the output size; and loadGeminiModel() tested length(model == 1) — the length of a *logical* vector, hence always at least 1 — so it appended a third element to an already complete pair. There is now a single geminiModelFile() used by both the writer and the reader.
Bug fix (Settings > Working Folder): "Clean Model Cache" removed the whole ~/tall folder, not the model cache. It took the Gemini API key (.tall_gemini_key.txt), the graph-export settings (.tall_graph_settings.txt), the working-folder pointer (tallWD.tall) and any .tall project the user had saved in that folder — none of which can be downloaded again — under a button that names only the models. It now removes ~/tall/language_models and says so, and the alert states what has been kept.
Bug fix (start-up): an empty or blank ~/tall/tallWD.tall aborted the session with argument is of length zero. readLines() of an empty file returns character(0), and file.exists(character(0)) returns logical(0), which if cannot evaluate — and wdFolder() is called from resetValues() before there is any UI to report the error in. A blank pointer file is now treated as no working folder, and removed.
Bug fix (Words > Word Embeddings > Similarity): the community-detection step built its lookup table with data.frame(as.list(membership(cluster))), which runs make.names() over the vertex names. Every term that is not a syntactic R name therefore lost the join that follows and was **silently dropped from the network** — no warning, no gap, just a smaller plot. This is not an exotic case: it covers hyphens and apostrophes (e-mail), a leading digit (3d, 30), everything TALL's own entity tagging produces (hashtags, mentions, emoji, URLs) and R's reserved words, so ordinary English terms such as next, break, in and for disappeared too. On the US-airlines corpus 6,789 of the 16,537 selected lemmas were affected, including 2 of the 100 words the view is asked to plot. The table is now built directly from the membership names and every node reaches the canvas.
Bug fix (Words > Word Embeddings > Similarity): when no pair of words reached the hard-coded 0.5 similarity cutoff the empty edge list reached graph_from_data_frame(), and the view died with Can't rename columns that don't exist. Column 'V1' doesn't exist. — a dplyr message with nothing to connect it to the cutoff. The network now renders a single label saying that no pair reaches the threshold.
Bug fix (Words > Word Embeddings > Similarity > UMAP): with a single word to place, the label de-overlap routine ran 1:(nrow(df) - 1), which counts **down** to 0, so it compared row 0 with row 1 and stopped on missing value where TRUE/FALSE needed. Nothing can overlap with itself, and the loop is now skipped in that case.
Bug fix (Words > Word Embeddings): word2vec keeps a </s> sentence-boundary sentinel in its model matrix. It is not a word of the corpus and its vector is essentially untrained, but every consumer read the matrix with as.matrix() and included it: on Frankenstein its norm is 4.47 against a mean absolute component of 0.81 for real terms, which made it the Min or the Max in 10 of the 20 dimensions of the Training tab, shifted Kurtosis by up to 183 and dropped the variance explained by PC1 from 0.591 to 0.501 — in the very chart that tab exists to show. It could also surface as a neighbour in the similarity network and as a row of the exported matrix. A new w2vMatrix() accessor drops it, and all consumers go through it.
Bug fix (Documents > Supervised Classification > Download Results): the export read test_data$target, but the model frame names that column .target_class, and "target" is not a prefix of it, so $ could not reach it either. Two outcomes, both reproduced on a real corpus: normally $target returned NULL and tibble() raised a recycling error **inside the download handler**, so the whole .xlsx failed and none of its five sheets was written; and on a corpus that happens to contain a term whose sanitised column name is exactly target — the US-airlines tweets do — $ resolved to that TF-IDF column instead, and the file was written with numbers in the Actual column and Correct false on every row. The lookup is now [[".target_class"]], which also cannot fall back to partial matching.
Bug fix (Documents > Supervised Classification): ranger's prediction.error was displayed as "OOB Prediction Error (%)" in the training summary, the success dialog and the exported workbook. For a probability = TRUE forest that field is the out-of-bag **Brier score**, not a misclassification rate — 0.21 on the reference run, which invited reading it as "21% of documents misclassified". It is now labelled and formatted as a Brier score.
Bug fix (Documents > Supervised Classification): the training-set accuracy was computed from **in-sample** predictions, so it is optimistically biased by co

The NEWS section is longer than shown here.

From NEWS.md in the release

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)

install.packages("tall", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("massimoaria/tall")

Development version from massimoaria/tall. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-tall

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled. We list this channel from recorded r2u download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-tall

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

Health

CRAN checks

13OK

CRAN check page ↗

Slowest check: 5.7 min · r-devel-windows-x86_64

Status changes

OK13 OK2026-08-04
NOTE12 OK, 1 NOTE2026-08-01
OK13 OK2026-06-09
ERROR12 OK, 1 ERROR2026-06-08
OK12 OK2026-04-25

Show 4 earlier changesHide 4 earlier changes

ERROR8 OK, 5 NOTE, 1 ERROR2026-04-17
NOTE11 OK, 3 NOTE2026-04-10
ERROR10 OK, 3 NOTE, 1 ERROR2026-04-09
NOTE11 OK, 3 NOTE2026-03-10

Advisories

none known

How this is measured·How advisories are found

Documentation

In release 1.0.1

READMEYes · 1,338 wordsVignettesNoNEWSYes · 100% structured · mentions 1.0.1Citation fileYes · cites a publicationWebsiteYes

Help pages: 16 (3 internal)
Examples that run: 33%3 run in a plain check · 3 only when donttest examples are enabled (R CMD check --as-cran) · 3 never run
Exported functions with examples: 67%
Documented parameters: 100%
Return-value docs: 100%
Help pages citing references: 6%

How these are measured

Downloads

7.4K

CRAN downloads in the past year

Rank #4,851 · ~20/day · ~615/mo

Releases R versions

Daily download trend is not available in this view yet.

70730 days

1.7K90 days

7.4K1 year

Compare downloads with other packages →

Also on461 r2u

How downloads are counted

Repository

massimoaria/tall ↗Account: massimoaria

40Stars

10Forks

9Open issues

0Open PRs

7Releases

685Commits

9Commit identities

r-shinytext-analysis-and-sentiment-analysistext-classificationtext-miningtextual-analysis

685 commits · Last activity 2026-09-18 · 0% stars, 30d · The default branch showed an older version (1.0.0.9000) than CRAN on 2026-10-11

Responsiveness

How these are sampled

PRs merged (all time): 99%
Median days to close, 50 most recently updated closed issues: 6
Median days to close, 50 most recently updated closed PRs: 0

Stars over time

2025-09-29 · 232026-07-30 · 40

How these are measured

PracticesBeta

3 development-tooling and community-health practices detected across 3 families in the upstream repository

Checks run against github.com/massimoaria/tall on 2026-10-11.

Docs source (1)

README.Rmd

CRAN release process (1)

cran-comments.md

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v3 (2026-09-26)

AI coding toolsBeta

This repository shows signs of AI coding tools.

Claude Codefirst seen on or before 2026-02-19
Named in .Rbuildignore: .claude
Named in .gitignore: .claude
Pull request it wrote: opened by a project member
Commits: at least 18 crediting it in the message

Models named in its commits

Claude Opus 516 commits · Jul to Aug 2026
Claude Opus 4.62 commits · Feb 2026

Counts are commits, not lines of code. Commits by a tool's accounts count once across all of its accounts. Commits crediting it are the largest count among its credit lines, a minimum.How this is detected·Detection ruleset v4 (updated 2026-09-27)

Dependencies

Declared dependencies

49 external dependencies (excludes base and recommended)

Installing tall adds 177 packages (97 with compiled code, 158 MB of source tarballs). Heaviest: BH 14 MB, stringi 11 MB, ggplot2 6.0 MB.

Depends (2)

R >= 3.5.0shiny

LinkingTo (1)

Rcpp

Suggests (1)

testthat

Enhances (0)

none

Dependencies facing archival

CRAN will archive these unless their check problems are fixed by the date shown. This package needs each one to install.

stmfix by Oct 23
quanteda through stmfix by Oct 22

Dependency license breakdown →

Reverse dependencies

0direct

0indirect

Nothing depends on this yet.

How these are counted

Code & Tests

Datasets

People & History

People (5)

Author

Author

Contributor

Michelangelo Misuraca

Contributor

Releases (10)

Pick two releases to compare their code metrics.

1.0.1Latest
record
2026-09-18 · current release · diff ↗
R
R 4.6.0 released · 2026-04-24
1.0.0
record
2026-04-15 · diff ↗ ·
added 3 exports
compute_syntactic_complexity, extract_noun_phrases, extract_svo_triplets
0.5.2
record
2026-02-12 · diff ↗
0.5.1
record
2025-12-12 · diff ↗
0.5.0
record
2025-12-06 · diff ↗ ·
added 4 exports
calculate_ngram_is, process_multiwords_fast, txt_recode_fast, txt_recode_ngram_fast
0.4.0
record
2025-10-23 · diff ↗
0.3.0
record
2025-06-03 · diff ↗
0.2.0
record
2025-04-16 · diff ↗
R
R 4.5.0 released · 2025-04-11
0.1.1
record
2025-03-13 · diff ↗
0.1.0
record
2025-02-06
R
R 4.4.0 released · 2024-04-24

Package metadata

First published: 2025-02-06
Total releases: 10 / 1 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 3.5.0
Bundled data: 700 KB / 1 file
Download size: 2.0 MB

Cite

Cite this package

Run in R for the authors' entry:

citation("tall")

The package provides a citation file, so citation() returns the authors' own entry.

It points to 10.1016/j.softx.2026.102590.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

Format

APA

Balamuta, J. J. (2026). R Observatory: Metrics for tall version 1.0.1 [Data set]. HJJB, LLC. Data release v2026-10-17. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-17, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →