R Observatory: Metrics for robotstxt version 0.7.15

Balamuta, James Joseph

doi:10.5281/zenodo.21843040

robotstxt

0.7.15Package record

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

1packages dependsuggested by 4

24.9Kdownloads / year

86.2%test coverage

13/13flavors OK

Overview

About

Maintained by Pedro BaltazarCRAN page ↗GitHub ↗Website ↗Issues ↗

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Changes in 0.7.15

CRAN compliance - Packages which use Internet resources should fail gracefully
CRAN compliance - fix R CMD check NOTES.

From NEWS.md in the release

Install

R (CRAN)pak (dev)Ubuntu (r2u)Fedora (COPR)openSUSE (OBS)

install.packages("robotstxt", repos = "https://cloud.r-project.org/")

Installs the current CRAN release. Prebuilt binaries on Windows and macOS.

if (!requireNamespace("pak", quietly = TRUE))
  install.packages("pak", repos = "https://cloud.r-project.org/")
pak::pak("ropensci/robotstxt")

Development version from ropensci/robotstxt. Installs pak first if it is not already available.

Release

# enable cranapt (r2u) if it is not already set up, then install
grep -rqs r2u.stat.illinois.edu /etc/apt/sources.list /etc/apt/sources.list.d/ \
  || curl -fsSL https://raw.githubusercontent.com/eddelbuettel/r2u/master/inst/scripts/add_cranapt_noble.sh | sudo bash
sudo apt install -y r-cran-robotstxt

Fast Ubuntu binaries via r2u (no compilation). The cranapt setup runs once and is skipped when it is already enabled. We list this channel from recorded r2u download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

bspm (used by r2u) also needs make: sudo apt install make

# enable the iucar/cran COPR if it is not already present, then install
ls /etc/yum.repos.d/*iucar*cran*.repo >/dev/null 2>&1 \
  || sudo dnf copr enable -y iucar/cran
sudo dnf install -y R-CRAN-robotstxt

RPMs from the iucar/cran COPR, rebuilt from CRAN.

For install.packages() integration in R, also: sudo dnf install R-CoprManager

# add the autoCRAN OBS repo (Tumbleweed) if it is missing, then install
zypper -q lr | grep -qi autoCRAN \
  || sudo zypper addrepo https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo
sudo zypper install -y R-robotstxt

RPMs auto-built from CRAN on openSUSE OBS (Tumbleweed). We list this channel from recorded autoCRAN download activity rather than from a live index of the repository, so treat it as last known rather than guaranteed.

Health

CRAN checks

13OK

CRAN check page ↗

Slowest check: 1.4 min · r-oldrel-macos-x86_64

Status changes

OK13 OK2026-08-05
NOTE12 OK, 1 NOTE2026-08-01
OK14 OK2026-03-10

Advisories

none known

How this is measured·How advisories are found

Documentation

In release 0.7.15

README· In the repository, left out of the releaseVignettesYes · 1 · staticNEWSYes · 67% structured · mentions 0.7.15Citation fileNoWebsiteYes · pkgdown

Help pages: 33 (9 internal)
Examples that run: 0%2 never run
Exported functions with examples: 10%
Documented parameters: 98%
Return-value docs: 44%
Help pages citing references: 0%

Vignettes (1)

using_robotstxt

How these are measured

Downloads

24.9K

CRAN downloads in the past year

Rank #3,168 · ~68/day · ~2.1K/mo

Releases R versions

Daily download trend is not available in this view yet.

1.7K30 days

3.9K90 days

24.9K1 year

Compare downloads with other packages →

Also on303 r2u7 autocran789 conda_forge132 c2d4u

How downloads are counted

Repository

ropensci/robotstxt ↗Organization: ropensci

69Stars

11Forks

1Open issues

0Open PRs

7Releases

315Commits

11Commit identities

robotstxtcrawlerwebscrapingspiderscraperrrstatsr-package

315 commits · Last activity 2024-11-15 · 0% stars, 30d · Default branch at 0.7.15.9000, ahead of CRAN 0.7.15 (checked 2026-10-22)

Responsiveness

How these are sampled

PRs merged (all time): 97%
Median days to close, 50 most recently updated closed issues: 7
Median days to close, 50 most recently updated closed PRs: 0

Stars over time

2025-03-27 · 672026-07-07 · 69

How these are measured

PracticesBeta

11 development-tooling and community-health practices detected across 7 families in the upstream repository

Checks run against github.com/ropensci/robotstxt on 2026-10-22.

Continuous integration (2)

GitHub ActionsR CMD check workflow

The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel

Governance and community (2)

Code of conduct (inherited from ropensci)Contributing guide

Docs source (2)

README.Rmdpkgdown

Coverage (1)

Coverage reporting

Show all practicesHide 3 practices

Reproducibility and dev environment (1)

data-raw/

CRAN release process (2)

cran-comments.mdCRAN-SUBMISSION

Lint, format, editor (1)

RStudio project

How this is detected·Detection ruleset v4 (2026-10-20)

Dependencies

Declared dependencies

11 external dependencies (excludes base and recommended)

Installing robotstxt adds 24 packages (16 with compiled code, 20 MB of source tarballs). Heaviest: stringi 11 MB, Rcpp 1.8 MB, openssl 1.1 MB.

Depends (1)

R >= 3.0.0

Imports (6)

stringr httr spiderbar future.apply magrittrutils

LinkingTo (0)

none

Suggests (6)

knitr rmarkdown dplyr testthat covr curl

Enhances (0)

none

Dependency license breakdown →

Reverse dependencies

1direct

0indirect

Imports 1

Suggested by 4

Depends, Imports or LinkingTo, most downloaded first

ralger

Suggests, most downloaded first

spiderbar webchem vosonSML newsanchor

How these are counted

Code & Tests

People & History

People (5)

Pedro Baltazar

Author, Maintainer since 0.7.15 (2024) · added in 0.7.15

Peter Meissner

Author

Kun Ren

Author, Copyright holder · added in 0.7.4

Author and copyright holder of list_merge.R.

Rich Fitz John

Contributor · added in 0.3.2

original release code review

Oliver Keys

Contributor

original release code review

Previously maintained by Peter Meissner (0.1.2 to 0.7.13).

Releases (12)

Pick two releases to compare their code metrics.

R
R 4.6.0 released · 2026-04-24
R
R 4.5.0 released · 2025-04-11
0.7.15Latest
record
2024-08-29 · current release · diff ↗
R
R 4.4.0 released · 2024-04-24
R
R 4.3.0 released · 2023-04-21
R
R 4.2.0 released · 2022-04-22
R
R 4.1.0 released · 2021-05-18
0.7.13
record
2020-09-03 · diff ↗
0.7.8
record
2020-07-25 · diff ↗
0.7.7
record
2020-06-27 · diff ↗
0.7.4
record
2020-05-31 · diff ↗ ·
added 12 exports
get_robotstxt_http_get, on_client_error_default, on_domain_change_default, on_file_type_mismatch_default, on_not_found_default, on_redirect_default, on_server_error_default, on_sub_domain_change_default, on_suspect_content_default, request_handler_handler, rt_last_http, rt_request_handler
R
R 4.0.0 released · 2020-04-24
R
R 3.6.0 released · 2019-04-26
0.6.2
record
2018-07-18 · diff ↗
R
R 3.5.0 released · 2018-04-23
0.6.0
record
2018-02-11 · diff ↗

Show 8 earlier eventsHide 8 earlier events

0.5.2
record
2017-11-12 · diff ↗ ·
added 1 export
get_robotstxts
0.4.1
record
2017-09-01 · diff ↗ ·
added 1 export
%>%
0.4.0
record
2017-07-16 · diff ↗ ·
added 1 export
is_valid_robotstxt
R
R 3.4.0 released · 2017-04-21
0.3.2
record
2016-12-05 · diff ↗
R
R 3.3.0 released · 2016-05-03
0.1.2
record
2016-02-08
R
R 3.2.0 released · 2015-04-16

Package metadata

First published: 2016-02-08
Total releases: 12 / 10 yrs
License: MIT + file LICENSE OSI
Minimum R: ≥ 3.0.0
Download size: 131 KB

Appears in task views

Web Technologies and Services

Cite

Cite this package

Run in R for the authors' preferred citation:

citation("robotstxt")

Format

Baltazar, P., Fitz John, R., Keys, O., Meissner, P., & Ren, K. (2024). robotstxt: A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker (Version 0.7.15) [Computer software]. https://doi.org/10.32614/CRAN.package.robotstxt

This package has no citation file, so citation() returns this entry built from DESCRIPTION.

Cite the R Observatory

For a number measured here: a download total, a coverage figure, an archival date.

Format

APA

Balamuta, J. J. (2026). R Observatory: Metrics for robotstxt version 0.7.15 [Data set]. HJJB, LLC. Data release v2026-10-25. https://doi.org/10.5281/zenodo.21843040

From data release v2026-10-25, which the citation names so these numbers can be found later. More on citing and the projects behind them.

Report a problem with this page →