Seamless AWS Cloud Bursting for Parallel R Workloads
Overview
A 'future' backend that enables seamless execution of parallel R workloads on 'Amazon Web Services' ('AWS', https://aws.amazon.com), including 'EC2' and 'Fargate'. 'staRburst' handles environment synchronization, data transfer, quota management, and worker orchestration automatically, allowing users to scale from local execution to 100+ cloud workers with a single line of code change.
Cost & cleanup fixes (important)
- **The
max_hourly_costguard now works on the EC2 backend (the default).** Previouslyestimate_cost()returnedper_hourfor Fargate buttotal_per_hourfor EC2, and the guard readper_hour— which isNULLon EC2 — so the cost ceiling was silently never enforced on the default backend.estimate_cost()now returns a normalizedhourly_ratefor every backend, and the guard, cost display, andstarburst_map()/starburst_cluster()all use it. - **
cost_alert_thresholdis now enforced** — it warns when a plan's estimated hourly rate exceeds it (previously it was stored but never read). - **
starburst_config(auto_cleanup_s3 = …)now actually controls S3 cleanup.** The runtime read a different key (getOption("starburst.cleanup_s3")) and ignored the stored setting; cleanup now resolves option →auto_cleanup_s3config → default TRUE.
Show all changes in 0.3.9Hide the rest of the changes in 0.3.9
- **
starburst_map()/starburst_cluster()cost estimates match the backend used** — they forwardlaunch_type/instance_type/use_spot, so an EC2 job no longer reports a Fargate price. - **Cost estimates now use live AWS pricing** (Pricing API for On-Demand, EC2 spot-price history for Spot), cached per session with a built-in static-rate fallback when offline. Reported costs are **estimates** (measured runtime × current rate), not AWS billing figures; docs reworded accordingly.
New features
- **
starburst_setup()now provisions the default EC2 capacity provider** (forc7g.xlarge) so the default EC2 backend works out of the box — previously the firststarburst_map()/plan(starburst)run failed because no capacity provider existed. It is created atDesiredCapacity = 0(no billable instances launch during setup). Passsetup_ec2 = FALSEto skip (Fargate-only users). (#37) - **Custom EC2 instance types are now provisioned automatically on first use.** Selecting a non-default
instance_type(e.g.c8a.xlarge) previously failed with an opaque "AutoScalingGroup not found" because only the default was provisioned. Now the capacity provider/ASG is created on demand at first launch (a one-time ~1–2 min step; no instances launched by provisioning), or on failure you get the exactstarburst_setup_ec2(instance_types = "…")command to run.
Behavior changes
- **Detached
session$collect()now returns an entry for every terminal task**, including failures. A failed task comes back as a structured failure (list(error = TRUE, message = …, task_id = …)) alongside successful results, instead of being silently omitted — matching the documented contract. - Clarified
session$cleanup(): by default it stops workers and marks the session terminated but **preserves S3 objects**; passforce = TRUEto also delete them. (Behavior unchanged; the docs were wrong and are now correct.)
Breaking changes
- **Removed the
platformargument fromstarburst_map()andstarburst_cluster().** It was ignored (architecture is inferred frominstance_type: Graviton*g.*→ ARM64, else x86_64), and its"X86_64"default even contradicted the ARM64c7g.xlargedefault. Drop it from calls; the backend picks the architecture frominstance_type. (platformremains onstarburst_estimate(), where it is functional.) - **Renamed
starburst_config(max_cost_per_job=)tomax_hourly_cost.** The limit was always enforced as an hourly *rate* (USD/hour), not a total-job cap; the new name matches the behavior. Update anystarburst_config(max_cost_per_job = ...)calls tomax_hourly_cost = ....
Bug fixes
- **
starburst_map()andstarburst_cluster()now acceptlaunch_type,instance_type, anduse_spot** and forward them to the backend. Previously these functions had no backend arguments, so a call likestarburst_map(x, f, launch_type = "FARGATE")silently passedlaunch_typeto the mapped function.f(typically an "unused argument" error) instead of switching the backend — contradicting the 0.3.7 migration note. Backend selection now works uniformly acrossplan(starburst),starburst_map(),starburst_cluster(), andstarburst_session(). - Fixed broken/misleading examples: the README detached-session example used a nonexistent
detached = TRUEargument; S3-read examples used baseread.csv()/readRDS(url())which cannot reads3://; and the API rate-limit example aggregated to 10× its stated global limit across workers. - **Public base images are now built multi-arch (amd64 + arm64).** They were published amd64-only, so
use_public_base = TRUEfailed the multi-arch environment build with "no match for platform in manifest" for arm64. Thebuild-base-imagesworkflow now builds both architectures (matching the env build), so the public base path works. (#39)
Documentation
- Consistency sweep addressing an external documentation review: EC2 is now consistently presented as the default/recommended backend with Fargate as the optional alternative; the README and example console output match the current engine (ASCII status markers, one task per input element — no chunking step); the Getting Started tutorial leads with
starburst_map(); a grouped reference index and an architecture diagram were added; andstarburst_config()keys are now catalogued. - Trimmed the Getting Started tutorial: the inline Monte Carlo / bootstrap / genomics snippets (which showed unbatched
future_map(1:10000, …)and hand-written speedup claims) now link to the canonical, measured example articles and the Workload Shapes / Performance guides. The doc-consistency guard (tools/check-docs.R) now also flags naivefuture_map()/future_lapply()big-count calls, not juststarburst_map(). - Removed the historical
ROADMAP.md(a pre-1.0, Fargate-first implementation plan that also carried a stale "MIT" license note); the README Roadmap section is the single source of truth.
From NEWS.md in the release
Install
Health
- OK13 OK2026-10-17
- NOTE12 OK, 1 NOTE2026-10-01
- OK13 OK2026-08-21
- NOTE12 OK, 1 NOTE2026-08-17
- OK13 OK2026-08-04
Show 6 earlier changesHide 6 earlier changes
- NOTE12 OK, 1 NOTE2026-08-01
- OK13 OK2026-07-29
- ERROR12 OK, 1 ERROR2026-07-24
- OK13 OK2026-05-02
- ERROR11 OK, 1 ERROR2026-04-25
- OK5 OK2026-03-20
Documentation
- Help pages
- 163 (134 internal)includes a package overview page
- Examples that run
- 6%1 run in a plain check · 16 only when donttest examples are enabled (R CMD check --as-cran)
- Exported functions with examples
- 80%
- Documented parameters
- 98%
- Return-value docs
- 100%
- Help pages citing references
- 0%
- Detached Session Mode HTML
- Example: Bootstrap Confidence Intervals for A/B Testing HTML
- Example: Monte Carlo Portfolio Simulation HTML
- Example: Parallel Bulk API Calls HTML
- Example: Parallel Feature Engineering HTML
- Example: Parallel Geospatial Analysis HTML
- Example: Parallel Grid Search for Hyperparameter Tuning HTML
- Example: Parallel Portfolio Risk Modeling HTML
- Example: Parallel Report Generation HTML
- Getting Started with staRburst HTML
Show 4 moreHide 4 vignettes
Downloads
Repository
- PRs merged (all time)
- 100%
- Median days to close, 50 most recently updated closed issues
- 16
- Median days to close, 50 most recently updated closed PRs
- 0
Stars over time
Forks over time
Releases over time
Issues over time
PRs over time
Commits over time
Commit identities over time
13 development-tooling and community-health practices detected across 7 families in the upstream repository
Checks run against github.com/scttfrdmn/starburst on 2026-10-21.
The R CMD check workflow runs on Linux, macOS and Windows, and on R-devel
GitHub Pages site last deployed 2026-09-13 · The repository has a site deploy workflow
pkgdown site last built 2026-09-13
Show all practicesHide 3 practices
This repository shows signs of AI coding tools.
- Claude Codefirst seen 2026-02-12Seen in the repository: CLAUDE.mdNamed in .Rbuildignore: .claude, CLAUDE.md
Dependencies
Installing starburst adds 36 packages (22 with compiled code, 19 MB of source tarballs). Heaviest: Rcpp 1.8 MB, renv 1.5 MB, paws.compute 1.4 MB.
Code & Tests
People & History
Pick two releases to compare their code metrics.
- 0.3.9Latest
- RR 4.6.0 released · 2026-04-24
- 0.3.82026-03-19
- RR 4.5.0 released · 2025-04-11
Package metadata
- First published
- 2026-03-19
- Total releases
- 2 / 1 yrs
- License
- Apache License 2.0 OSI
- Minimum R
- ≥ 4.0.0
- Download size
- 1.6 MB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("starburst")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-24, which the citation names so these numbers can be found later. More on citing and the projects behind them.