spiderbar
Parse and Test Robots Exclusion Protocol Files and Rules
v0.2.5
·
Feb 11, 2023
·
MIT + file LICENSE
Description
The 'Robots Exclusion Protocol' <https://www.robotstxt.org/orig.html> documents a set of standards for allowing or excluding robot/spider crawling of different areas of site content. Tools are provided which wrap The 'rep-cpp' <https://github.com/seomoz/rep-cpp> C++ library for processing these 'robots.txt' files.
Downloads
1.5K
Last 30 days
3192nd
6K
Last 90 days
36.2K
Last year
Trend: -14% (30d vs prior 30d)
CRAN Check Status
2
NOTE
12
OK
Show all 14 flavors
| Flavor | Status |
|---|---|
| r-devel-linux-x86_64-debian-clang | NOTE |
| r-devel-linux-x86_64-debian-gcc | NOTE |
| r-devel-linux-x86_64-fedora-clang | OK |
| r-devel-linux-x86_64-fedora-gcc | OK |
| r-devel-macos-arm64 | OK |
| r-devel-windows-x86_64 | OK |
| r-oldrel-macos-arm64 | OK |
| r-oldrel-macos-x86_64 | OK |
| r-oldrel-windows-x86_64 | OK |
| r-patched-linux-x86_64 | OK |
| r-release-linux-x86_64 | OK |
| r-release-macos-arm64 | OK |
| r-release-macos-x86_64 | OK |
| r-release-windows-x86_64 | OK |
Check details (2 non-OK)
NOTE
r-devel-linux-x86_64-debian-clang
CRAN incoming feasibility
Maintainer: ‘Bob Rudis <bob@rud.is>’
No Authors@R field in DESCRIPTION.
Please add one, modifying
Authors@R: c(person(given = "Bob",
family = "Rudis",
role = c("aut", "cre"),
email = "bob@rud.is",
comment = "bob@rud.is"),
person(family = "SEOmoz"),
person(family = "Inc",
role = "aut"))
as necessary.
NOTE
r-devel-linux-x86_64-debian-gcc
CRAN incoming feasibility
Maintainer: ‘Bob Rudis <bob@rud.is>’
No Authors@R field in DESCRIPTION.
Please add one, modifying
Authors@R: c(person(given = "Bob",
family = "Rudis",
role = c("aut", "cre"),
email = "bob@rud.is",
comment = "bob@rud.is"),
person(family = "SEOmoz"),
person(family = "Inc",
role = "aut"))
as necessary.
Check History
NOTE 12 OK · 2 NOTE · 0 WARNING · 0 ERROR · 0 FAILURE Mar 10, 2026
NOTE
r-devel-linux-x86_64-debian-clang
CRAN incoming feasibility
Maintainer: ‘Bob Rudis <bob@rud.is>’
No Authors@R field in DESCRIPTION.
Please add one, modifying
Authors@R: c(person(given = "Bob",
family = "Rudis",
role = c("aut", "cre"),
email = "bob@rud.is",
comment = "bob@rud.is"),
person(family = "SEOmoz"),
person(family = "Inc",
role = "aut"))
as necessary.
NOTE
r-devel-linux-x86_64-debian-gcc
CRAN incoming feasibility
Maintainer: ‘Bob Rudis <bob@rud.is>’
No Authors@R field in DESCRIPTION.
Please add one, modifying
Authors@R: c(person(given = "Bob",
family = "Rudis",
role = c("aut", "cre"),
email = "bob@rud.is",
comment = "bob@rud.is"),
person(family = "SEOmoz"),
person(family = "Inc",
role = "aut"))
as necessary.