Parse, Clean, and Normalize URLs
Overview
A lightweight toolkit for extracting structured information from URLs. Includes functions for parsing, normalizing protocols, extracting domains, and constructing clean URLs. Domain and public-suffix extraction is delegated to the 'pslr' package, which implements the Public Suffix List from https://publicsuffix.org. Punycode and IDNA encoding is handled by the 'punycoder' package.
New features
- Exported names stay in US English, and their British spellings are now accepted as aliases:
serialise_url()is the same function asserialize_url(), andget_clean_url(),get_path(),safe_parse_url()andsafe_parse_urls()takepath_normalisationas an alias ofpath_normalization. The alias is the last formal, so no positional call shifts; supplying both spellings is an error, and the alias counts as explicitly supplied in theurl_standardconflict check.url_profile(),canonical_join()andresolve_url()accept it through.... An abbreviated argument such aspath_normal =no longer partially matches, because it now fits both spellings; spell the name in full (SEOR-qwomlgjd,SEOR-oytkybis). check_schemes()reports a sixth reasons token,no-authority, for a URL that has a scheme but carries no authority:mailto:someone@example.com,javascript:alert(1),foo:bar. It is computed from the parse, not fromget_host(), and never contradicts the parse record: underurl_standard = "whatwg"only a non-special scheme can get it, because the WHATWG URL Standard gives every special-scheme URL a host with or without//(http:example.comhas hostexample.com); under"rfc3986"an authority exists only after//(RFC 3986 §3), sohttp:example.comgets it there. A row with no scheme never does.?check_schemesand ruling RUL-021 had said such a token could not be computed becauseget_host()reads the@inmailto:someone@example.comas a userinfo delimiter. That was wrong: the parse record'shostisNAthere, andget_host()returns the recipient's domain as ADR 0012 D7 extraction metadata (evidence in RURL-hlbbbpdy). Ruling RUL-022 records the correction. No parse result changes (RURL-ktpjscne).get_url_diagnostics()reports a new host fact,domain-invalid-ace-label, for a host with a label that beginsxn--but is not a genuine A-label: its Punycode decode fails, or the decoded label fails the UTS #46 Validity Criteria (§4.1) under the WHATWG URL Standard's non-strict flags.xn--a.example,xn--.exampleandxn--ASCII-.exampleget it;xn--bcher-kva.exampledoes not. Each label is judged alone, so a label that fails the Bidi rule only because of another label does not fire it. The parse does not move: the WHATWG host parser runs the domain parser withbeStrict= false, which keeps an ASCII host lowercased when ToASCII fails, and WPT pinshttps://xn--/as a success. The token fires under both standards, like the rest of thedomain-*family. Unlike the rest of the vocabulary, it is complete: it fires on every host meeting its predicate, so a consumer may rely on its absence.check_hosts()lists it among thereasons. rurl checks UTS #46 criterion 4 (a decoded label must not begin withxn--) itself, because punycoder 1.2.1 does not. Ruling RUL-023 records the scope (RURL-vicyvlvh).
Bug fixes
host_encoding = "unicode"again renders an A-label whose ASCII part holds a code point other than a letter, digit or hyphen:get_host("http://xn--a_-wia.example/", host_encoding = "unicode")isa_ä.exampleunder everyurl_standard. punycoder 1.3.0 stopped decoding such labels, so the host kept itsxn--spelling, although RFC 3492 §6.2 accepts any basic code point and UTS #46 §4 (ToUnicode, withUseSTD3ASCIIRulesfalse, as the WHATWG URL Standard's domain to Unicode uses it) decodes them. rurl now decodes a label punycoder rejects with its own RFC 3492 decoder, and renders these hosts the same under punycoder 1.2.1 and 1.3.0. An A-label whose ASCII part holds#,/,:,?or@keeps its spelling, so a rendered host never gains a delimiter (RFC 3986 §3.2.2). Under punycoder 1.2.1 that moves one reachable case:url_standard = "rfc3986"with a percent-encoded colon, such asxn--;~%3A_-8qa3179i, which rendered a:inside the host. The frozenNULLprofile renders as it did under punycoder 1.2.1 (RURL-oizpyvdz).- Under
url_standard = "whatwg", a host whose UTS-46 mapping produces a forbidden domain code point now fails the parse, as the WHATWG URL Standard's host parser requires: it tests for one on the result of domain to ASCII (host parsing, the "domain-invalid-code-point" step). The gate tested only the source spelling, so the fullwidth#/?:%(U+FF03, U+FF0F, U+FF1F, U+FF1A, U+FF05) and the no-break and ideographic spaces (U+00A0, U+3000) passed layer 1, andserialize_url()wrote the mapped#/?:%or space into the host, where a re-parse reads a different host:http://127.0.0.1#.evil.com/serialized tohttp://127.0.0.1#.evil.com/. These now getlayer1_syntax_verdict = "fail",parse_status = "error"and anNAserialization.rfc3986and the frozenNULLprofile do not move (RURL-crsrkcoh). safe_parse_url(),safe_parse_urls()and the accessors built on them no longer fail on a long URL with "variable names are limited to 10000 bytes". The caches store each entry under its key as an R variable name, which R caps at 10,000 bytes, and non-ASCII characters are escaped in the key at six bytes each, so the error came from about 9,900 ASCII characters or about 1,700 non-ASCII ones. An internationalized host whose ASCII form passed 10,000 bytes failed the same way through the Punycode cache, even withrurl_cache_config(full_parse = FALSE). A key past the cap now skips the cache and is recomputed on every call; results are unchanged (RURL-tlmoybsl).
Documentation
BugReportspoints at <https://gitlab.com/bart-turczynski/rurl/-/issues>, the form R's CRAN incoming check requires for a gitlab.com tracker; a browser following that link is redirected to/-/work_items(RURL-nomzooml).- The human-facing tracker links in
SECURITY.md,codemeta.jsonand.bestpractices.jsonnow givehttps://gitlab.com/bart-turczynski/rurl/-/work_items, the address GitLab serves directly;DESCRIPTION'sBugReports:deliberately keeps the/-/issuesform, because only that form clears the CRAN incoming check (RURL-ahatycrd). - The help pages and
NEWS.mdare now spelled in US English throughout, matching the package's declaredLanguage: en-US, andinst/WORDLISTno longer admits British spellings (SEOR-kfiqpymb).
Show all changes in 3.1.0Hide the rest of the changes in 3.1.0
Internal
- The agent instructions point at the house
agent-workflowandfpskills for the git workflow,.Rbuildignoreno longer lists theFP_CLAUDE.mdfile this repository never had, anddesign/agent-workflow.mdrecords that an anonymous GitLab 404 needs a same-run control before it says anything about this project (SEOR-ipwcbcov, RURL-zrlkftor). scripts/bestpractices-url.pyis vendored from seor, with a pre-push hook that runs its offline self-test when the script changes..bestpractices.jsonnow names GitLab as the home of the project and GitHub only as its read-only mirror, and describes today's CI: GitLab pipelines onmainrun the consistency gates, while the test suite runs in the pre-push gate and in CI only on tags or by hand (SEOR-grrcptww)..gitlab-ci.ymlnow carries a top-levelworkflow:block that suppresses both the branch pipeline and the merge-request pipeline, leaving exactly one pipeline per merge, onmain. A feature-branch push and an API-triggered pipeline on a non-default branch now create no pipeline at all; a pipeline started by hand from Build > Pipelines > Run pipeline still works on any ref and runsgates,citation-versionandcheckthere, whilepagesis pinned tomainso a branch can be verified but never published. The local pre-push gate remains the thing that verifies a change before it merges (SEOR-bmgkzhvy).- The OSS Index dependency audit in
tests/testthat/test-security.Rnow scopes to hard dependencies (Depends+Imports) instead of theSuggeststree, which pulled in oysteR's own dependencies and failed the gate on acurlvulnerability that rurl does not ship (RURL-mafkcwnu). - The OSS Index audit in
tests/testthat/test-security.Rnow requires every reported advisory to carry an explicit disposition in an allow-list (tests/testthat/helper-security.R, empty today because nothing is reported): an unlisted advisory fails, a listed advisory that is no longer reported fails, and a row past its review date warns. UnderOSSINDEX_AUDIT_REQUIRED=truea missingoysteRor missing credentials fail instead of skipping. The pre-push gate'sLC_ALL=Ctest cell no longer runs this audit or the OSV audit intests/testthat/test-osv.R, so a new upstream advisory cannot block an unrelated push; run them withtestthat::test_local(filter = "^(security|osv)$")(SEOR-fftbjnpl). scripts/check-bugreports.py, ported from pagerankr, now guards theBugReportssplit:DESCRIPTIONmust keep the/-/issuesform CRAN's incoming check accepts,codemeta.json,SECURITY.mdand.bestpractices.jsonmust link/-/work_items, and no other human-facing file may reintroduce a/-/issueslink. It runs as its own pre-push hook next tocheck-citationand in thecitation-versionCI job (SEOR-ocbtrrnl)..gitlab-ci.ymlgainsosv-auditandsecurity-auditjobs that runtests/testthat/test-osv.Randtests/testthat/test-security.Ragainst an installed rurl, only in a pipeline schedule that setsSCHEDULE_KIND=dependency-audit;security-auditsetsOSSINDEX_AUDIT_REQUIRED=true, so missing OSS Index credentials fail the job instead of skipping. Nothing else runs these audits since they left the pre-push gate, and the schedule and theOSSINDEX_USER/OSSINDEX_TOKENvariables still have to be created on the project.tools/local-ci.sh --allleaves schedule-only jobs out (SEOR-fftbjnpl).tools/cran-comments-gate.Rno longer requires thecran-comments.mdspan pin to name the development version inDESCRIPTION. It checks the pin against the release that version names —3.0.1while at3.0.1.9000— so the release checklist's post-release version bump is executable, and--onlinecompares whichever end of the span CRAN actually serves in each phase (RURL-efbcrhjc).tools/curl-zero-gate.Rnow checks that rurl does not depend on curl, and no longer polices the word. The raw-text scan (C6), its path allowlist, the staleness check on that allowlist (C0) and the oracle-provenance exemption are gone, so a comment, NEWS entry, design note, fixture orinst/WORDLISTentry that names curl or libcurl passes. C5 now fails only on a\link[curl]{...}cross-reference or a curl call in an Rd file's\examplesor\usage, not on prose. The DESCRIPTION, NAMESPACE, parsed-code and clean-room checks are unchanged, and the self-test now covers a curl load in the clean room (RURL-dtzvekmf)..lintrno longer carries a#comment header:read.dcf()skips comment lines only from R 4.6, so on R 4.5 and olderlintr::lint_package()aborted withInvalid DCF formatand the package got no lint at all. The rationale for each deviation from the goodpractice linter set now lives indesign/verification.md, with the rule to keep.lintrcomment-free and ASCII-only (SEOR-qfxolldq).tools/local-ci.shnow runs each job script withbashwhen the job's image has it and with POSIXshotherwise, as the GitLab runner does. Thecitation-versionjob's image,python:3.13-alpine, ships onlysh, so the runner failed that job before any of its commands ran. A failing command in ash-only image still fails the job (RURL-dswufxky).tools/local-ci-plan.Rnow plans a job whosescript:holds a block-scalar (- |) entry, sotools/local-ci.sh --allno longer aborts at thepagesjob with "a script entry spans multiple lines". The block is emitted as one entry, verbatim, the way the GitLab runner writes it into the job shell: it runs as one unit, and a failing command inside it fails the job.--listshows it under a single$, and every job without a block entry plans byte-identically to before. A new--self-testmode pins the single-line and the block rendering, and runs both underbash(RURL-gysfdtcd).- The
checkandpagesCI jobs installhunspellfrom CRAN over Debian'sr-cran-hunspell. Debian's build drops the bundled dictionaries for the systemen_US.aff, whoseWORDCHARSkeeps digits inside a token, so in ther-basecontainerspelling::spell_check_package()flagged 61 words, such asPV1andsha256, that a CRAN build ofhunspellchecks asPVandshaand passes. The spelling step now reads the same dictionaries locally and in CI (RURL-czvqosis). - The gate now runs
Rscript tools/local-ci-plan.R --self-test.tools/verify-manifest.ymldeclares it as a self-test-onlylocal-ci-planjob, sotools/verify.Rselects it whenevertools/local-ci-plan.Rchanges, runs it on--releaseand whenever it cannot tell what changed, and fails the run when a case fails. Until now nothing ran it (RURL-fzeadcfc).
From NEWS.md in the release
Install
Health
- OK13 OK2026-08-04
- NOTE12 OK, 1 NOTE2026-08-01
- OK6 OK2026-06-20
Documentation
- Help pages
- 37 (2 internal)includes a package overview page
- Examples that run
- 100%37 run in a plain check
- Exported functions with examples
- 100%
- Documented parameters
- 98%
- Return-value docs
- 100%
- Help pages citing references
- 0%
Downloads
Repository
- PRs merged (all time)
- 99%
- Median days to close, 50 most recently updated closed issues
- not measured
- Median days to close, 50 most recently updated closed PRs
- not measured
Releases over time
- github.com/bart-turczynski/rurl · last recorded on 2026-09-06
Repository gitlab.com/bart-turczynski/rurl is linked, but repository-practices checks have not run for it yet. Checks are GitHub-only for now.
Dependencies
Installing rurl adds 6 packages (5 with compiled code, 14 MB of source tarballs). Heaviest: stringi 11 MB, Rcpp 1.8 MB, punycoder 517 KB.
Code & Tests
People & History
Pick two releases to compare their code metrics.
- 3.1.0Latest
- 3.0.12026-09-09 · diff ↗ ·
added 21 exports
check_hosts, check_schemes, format_url, get_host_type, get_mailto_recipients, get_parse_verdicts, get_scheme_class, get_url_diagnostics, get_url_key, is_valid_host, query_param_summary, resolve_url, serialize_url, url_anti_join, url_full_join, url_inner_join, url_key_policy, url_left_join, url_profile, url_right_join, url_semi_join - 1.2.02026-06-19
- RR 4.6.0 released · 2026-04-24
Package metadata
- First published
- 2026-06-19
- Total releases
- 3 / 1 yrs
- License
- MIT + file LICENSE OSI
- Minimum R
- ≥ 4.0.0
- Download size
- 1.0 MB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("rurl")The package provides a citation file that builds its entry when R runs it. Run citation() to see it.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-19, which the citation names so these numbers can be found later. More on citing and the projects behind them.