Repository navigation
revdep bbotk fails to create a column by group after unserializing a data.table #7498
Description
Activity
This is due to #6789. A check for
ok < 1 || truelength(x) < ncol(x)+length(newnames)used to follow theok == 0check in the common case. Now it's been moved inside theif (missingby || bynull || (!byjoin && !length(byval)))branch, so it doesn't happen for group-byoperations.Too bad
dogroupshas to assign the result by itself, so there's no post-factum check on the R side afterjvalis known.A similar error for {circhelp}:
circhelp::Pascucci_et_al_2019_data[, prev_ori := shift(orientation), by = observer] # Error in `[.data.table`(circhelp::Pascucci_et_al_2019_data, , `:=`(prev_ori, : # This data.table has either been loaded from disk (e.g. using readRDS()/load()) or constructed manually (e.g. using structure()). Please run setDT() or setalloccol() on it first (to pre-allocate space for new columns) before assigning by reference to it.Maybe revert #6789 since it is obviously causing the most problems?
Another similar error in {DramaAnalysis}:
DramaAnalysis::utteranceStatistics(DramaAnalysis::rksp.0) # Error in `[.data.table`(text, , `:=`(dl, .N), .(corpus, drama)) : # Internal error in dogroups: Trying to add new column by reference but tl is full; setalloccol should have run first at R level before getting to this point. Please report to the data.table issues tracker.Again similar for {hicp}
library(data.table) download.file( "https://github.com/cran/hicp/raw/refs/heads/master/vignettes/data/hicp_itemweights.RData", tmp<-tempfile()) load(tmp) item.weights[, "t1" := hicp::tree(id=coicop, w=values, flag=TRUE, settings=list(w.tol=0.1)), by=c("geo","time")] # Error in `[.data.table`(item.weights, , `:=`("t1", hicp::tree(id = coicop, : # Internal error in dogroups: Trying to add new column by reference but tl is full; setalloccol should have run first at R level before getting to this point. Please report to the data.table issues tracker.
Looks like {webtrackR} is also having the same issue:
Running ‘testthat.R’ ERROR Running the tests in ‘tests/testthat.R’ failed. Last 13 lines of output: 1. └─webtrackR::add_duration(wt) at test-summarize.R:62:5 2. ├─...[] 3. └─data.table:::`[.data.table`(...) ── Error ('test-summarize.R:108:5'): sum_durations testdt_specific ───────────── Error in ``[.data.table`(wt, , `:=`(duration, as.numeric(data.table::shift(timestamp, n = 1, type = "lead", fill = NA) - timestamp)), by = "panelist_id")`: Internal error in dogroups: Trying to add new column by reference but tl is full; setalloccol should have run first at R level before getting to this point. Please report to the data.table issues tracker. Backtrace: ▆ 1. └─webtrackR::add_duration(wt) at test-summarize.R:108:5 2. ├─...[] 3. └─data.table:::`[.data.table`(...) [ FAIL 12 | WARN 22 | SKIP 7 | PASS 107 ] Error: ! Test failures. Execution haltedWe still need to detect column deletion before attempting to resize the table. Some of it could be detected statically (before evaluating
jsub), but not all (thinkfoo[, bar := if (runif(1) < .5) baz else NULL]). We're stuck between the Scilla of some revdeps attempting to delete columns on non-selfrefoktables (which used to work but now doesn't) and the Charybdis of this check only being fully possible afterjvalis known. #7502 is a kludge that passes ourR CMD checkand fixesbbotk, but that's not much.And {ume}, {SeaVal}, {metR}
And {blocking}
Ditto for {slim}:
slim::dialysis[, fully_observed := (length(month) == 5), by = id] # Error in `[.data.table`(slim::dialysis, , `:=`(fully_observed, (length(month) == : # Internal error in dogroups: Trying to add new column by reference but tl is full; setalloccol should have run first at R level before getting to this point. Please report to the data.table issues tracker.
A slightly different error message for {tidyrules}, but seems at first glance like the same issue:
tidyrules::tidy(partykit::ctree(bill_length_mm ~ ., data = palmerpenguins::penguins)) # Error in `[.data.table`(~.df, , `:=`(average = weighted.mean(response, : # Internal error in assign: input dt has not been allocated enough column slots. l=3, tl=3, adding 1. Please report to the data.table issues tracker.
Ditto {tidytable}:
tidytable::add_count(tidytable::tidytable(g = c(1, 2, 2, 2), val = c(1, 1, 2, 3)), g) # Error in `[.data.table`(~.df, , `:=`(n = .N), by = "g", keyby = FALSE) : # Internal error in dogroups: Trying to add new column by reference but tl is full; setalloccol should have run first at R level before getting to this point. Please report to the data.table issues tracker.
Fixed by #7502 and confirmed by a manual reverse dependency check.
From their tests:
The code is trying to add a new column by group to a freshly unserialized
data.table. This used to work (replacing the table with an over-allocated shallow copy in the caller's environment) but now doesn't (with a different error message):This is similar to #7488, but in a different code path.