Skip to content

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker

v0.7.15 · Aug 29, 2024 · MIT + file LICENSE
CRAN GitHub Mirror

Description

Provides functions to download and parse 'robots.txt' files. Ultimately the package makes it easy to check if bots (spiders, crawler, scrapers, ...) are allowed to access specific resources on a domain.

Downloads

1.7K

Last 30 days

3052nd

6.6K

Last 90 days

37.3K

Last year

Trend: -15.7% (30d vs prior 30d)

CRAN Check Status

14 OK
Show all 14 flavors
Flavor Status
r-devel-linux-x86_64-debian-clang OK
r-devel-linux-x86_64-debian-gcc OK
r-devel-linux-x86_64-fedora-clang OK
r-devel-linux-x86_64-fedora-gcc OK
r-devel-macos-arm64 OK
r-devel-windows-x86_64 OK
r-oldrel-macos-arm64 OK
r-oldrel-macos-x86_64 OK
r-oldrel-windows-x86_64 OK
r-patched-linux-x86_64 OK
r-release-linux-x86_64 OK
r-release-macos-arm64 OK
r-release-macos-x86_64 OK
r-release-windows-x86_64 OK

Check History

OK 14 OK · 0 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 10, 2026

Reverse Dependencies (7)

imports

BAwiR polite ralger

suggests

newsanchor spiderbar vosonSML webchem

Dependency Network

Dependencies Reverse dependencies stringr httr spiderbar future.apply magrittr BAwiR polite ralger newsanchor spiderbar vosonSML webchem robotstxt

Version History

new 0.7.15 Mar 10, 2026
updated 0.7.15 ← 0.7.13 diff Aug 28, 2024
updated 0.7.13 ← 0.7.8 diff Sep 2, 2020
updated 0.7.8 ← 0.7.7 diff Jul 24, 2020
updated 0.7.7 ← 0.7.4 diff Jun 26, 2020
updated 0.7.4 ← 0.6.2 diff May 30, 2020
updated 0.6.2 ← 0.6.0 diff Jul 17, 2018
updated 0.6.0 ← 0.5.2 diff Feb 10, 2018
updated 0.5.2 ← 0.4.1 diff Nov 11, 2017
updated 0.4.1 ← 0.4.0 diff Aug 31, 2017
updated 0.4.0 ← 0.3.2 diff Jul 15, 2017
updated 0.3.2 ← 0.1.2 diff Dec 4, 2016
new 0.1.2 Feb 7, 2016