New Experimental Design Based Subsampling Methods for Big Data
Overview
Subsampling methods for big data under different models and assumptions. Starting with linear regression and leading to Generalised Linear Models, softmax regression, and quantile regression. Specifically, the model-robust subsampling method proposed in Mahendran, A., Thompson, H., and McGree, J. M. (2023) doi:10.1007/s00362-023-01446-9, where multiple models can describe the big data, and the subsampling framework for potentially misspecified Generalised Linear Models in Mahendran, A., Thompson, H., and McGree, J. M. (2025) doi:10.48550/arXiv.2510.05902.
Install
Health
ERRORexamplesr-devel-linux-x86_64-debian-gcc
Running examples in ‘NeEDS4BigData-Ex.R’ failed
The error most likely occurred in:
> base::assign(".ptime", proc.time(), pos = "CheckExEnv")
> ### Name: modelMissLinSub
> ### Title: Subsampling under linear regression for a potentially
> ### misspecified model
> ### Aliases: modelMissLinSub
>
> ### ** Examples
>
> Beta <- c(-1, 0.75, 0.75, 1); Var_Epsilon <- 0.5;
> family <- "linear"; N <- 500
> X_1 <- replicate(2, stats::runif(n = N, min = -1, max = 1))
>
> Temp <- Rfast::rowprods(X_1)
> Misspecification <- (Temp - mean(Temp)) / sqrt(mean(Temp^2) - mean(Temp)^2)
> X_Data <- cbind(X0 = 1, X_1)
>
> Full_Data <- GenModelMissGLMdata(N, X_Data, Misspecification, Beta, Var_Epsilon, family)
> r0 <- 40; rf <- rep(10 * c(8, 12), 25)
> Original_Data <- Full_Data$Complete_Data[, -ncol(Full_Data$Complete_Data)]
>
> Results <- modelMissLinSub(r0 = r0, rf = rf,
+ Y = as.matrix(Original_Data[, 1]),
+ X = as.matrix(Original_Data[, -1]),
+ N = N, Alpha = 10, proportion = 0.5)
50% or >=50% of the big data is used to help find AMSE for the subsamples,
this could take some time.
Error in a$specials[[i]] : subscript out of bounds
Calls: modelMissLinSub ... predict.glm -> predict.lm -> delete.response -> off
Execution halted
Examples with CPU (user + system) or elapsed time > 5s
user system elapsed
ALoptimalGLMSub 3.878 0.551 6.044ERRORtestsr-devel-linux-x86_64-debian-gcc
Running ‘spelling.R’ [0s/0s]
Running ‘testthat.R’ [12s/16s]
Running the tests in ‘tests/testthat.R’ failed.
Complete output:
> # This file is part of the standard setup for testthat.
> # It is recommended that you do not modify it.
> #
> # Where should you do additional test configuration?
> # Learn more about the roles of various files in:
> # * https://r-pkgs.org/testing-design.html#sec-tests-files-overview
> # * https://testthat.r-lib.org/articles/special-files.html
>
> library(testthat)
> library(NeEDS4BigData)
>
> test_check("NeEDS4BigData")
Saving _problems/test-modelMissLinSub-15.R
Saving _problems/test-modelMissLogSub-14.R
Saving _problems/test-modelMissPoiSub-14.R
Saving _problems/test-plot_AMSE-15.R
[ FAIL 4 | WARN 0 | SKIP 0 | PASS 283 ]
══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test-modelMissLinSub.R:12:1'): (code run outside of `test_that()`) ──
<subscriptOutOfBoundsError/error
...[truncated]...
Gam(...)
8. └─stats::predict.glm(...)
9. └─stats::predict.lm(...)
10. └─stats::delete.response(tt)
11. └─stats (local) off(a$specials[[i]])
── Error ('test-plot_AMSE.R:12:1'): (code run outside of `test_that()`) ────────
<subscriptOutOfBoundsError/error/condition>
Error in `a$specials[[i]]`: subscript out of bounds
Backtrace:
▆
1. ├─base::suppressMessages(...) at test-plot_AMSE.R:12:1
2. │ └─base::withCallingHandlers(...)
3. ├─base::suppressWarnings(...)
4. │ └─base::withCallingHandlers(...)
5. └─NeEDS4BigData::modelMissLinSub(...)
6. └─gam::predict.Gam(fit_GAM, newdata = data.frame(X))
7. └─gam::newdata.predict.Gam(...)
8. └─stats::predict.glm(...)
9. └─stats::predict.lm(...)
10. └─stats::delete.response(tt)
11. └─stats (local) off(a$specials[[i]])
[ FAIL 4 | WARN 0 | SKIP 0 | PASS 283 ]
Error:
! Test failures.
Execution halted- ERROR12 OK, 1 ERROR2026-10-24
- OK13 OK2026-08-12
- ERROR12 OK, 1 ERROR2026-08-09
- OK13 OK2026-08-05
- NOTE12 OK, 1 NOTE2026-08-01
Show 10 earlier changesHide 10 earlier changes
- OK13 OK2026-06-20
- ERROR12 OK, 1 ERROR2026-06-18
- OK13 OK2026-05-24
- ERROR12 OK, 1 ERROR2026-05-22
- OK12 OK2026-04-25
- NOTE11 OK, 3 NOTE2026-04-15
- ERROR10 OK, 3 NOTE, 1 ERROR2026-04-11
- NOTE11 OK, 3 NOTE2026-03-25
- ERROR11 OK, 2 NOTE, 1 ERROR2026-03-22
- NOTE11 OK, 3 NOTE2026-03-10
Documentation
- Help pages
- 18
- Examples that run
- 100%16 run in a plain check
- Exported functions with examples
- 87%
- Documented parameters
- 100%
- Return-value docs
- 100%
- Help pages citing references
- 78% with a bibliography file
Downloads
Repository
Stars over time
10 development-tooling and community-health practices detected across 6 families in the upstream repository
Checks run against github.com/amalan-constat/needs4bigdata on 2026-10-21.
Show all practicesHide 2 practices
Dependencies
Code & Tests
Datasets
People & History
- RR 4.6.0 released · 2026-04-24
- 1.0.1Latest2025-10-22 · current release
- RR 4.5.0 released · 2025-04-11
Package metadata
- First published
- 2025-10-22
- Total releases
- 1 / 1 yrs
- License
- MIT + file LICENSE OSI
- Minimum R
- ≥ 4.1.0
- Bundled data
- 6.7 MB / 3 files
- Download size
- 7.5 MB
Cite
Cite this package
Run in R for the authors' preferred citation:
citation("NeEDS4BigData")This package has no citation file, so citation() returns this entry built from DESCRIPTION.
Cite the R Observatory
For a number measured here: a download total, a coverage figure, an archival date.
From data release v2026-10-24, which the citation names so these numbers can be found later. More on citing and the projects behind them.