Internal error: DT passed to assign has not been allocated enough column slots

It appears that set() does not always allocate enough slots. Example:

library(data.table)
DT = data.table(a = runif(10))

my.set = function(x, i = NULL, j, value) {
    if (truelength(DT) < length(DT)) {
       stop("bad input")
    }
    if (truelength(DT) == length(DT)) {
        cat("do we need to call alloc.col or setDT here?\n")
    }
    # note: switching from set() to DT[, c(j) := value] works here
    set(x, i, j, value)
    
    if (truelength(DT) < length(DT)) {
       stop("bad output")
    }
    invisible(x)
}

set.seed(6860)
while(ncol(DT) < 10000L){

    new.name = paste("V", ncol(DT) + 1L)
    new.value = sample(nrow(DT))
    
    my.set(DT, j = new.name, value = new.value)
}

As indicated in the comment above, switching from "set(x, i, j, value)" to "DT[, c(j) := value]" makes it work. The situation seems similar as to what was reported in #1830.

sessionInfo() is:

R version 3.6.0 (2019-04-26) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /home/btyner/R360/lib64/R/lib/libRblas.so
LAPACK: /home/btyner/R360/lib64/R/lib/libRlapack.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

loaded via a namespace (and not attached):
[1] compiler_3.6.0

This is still an issue in 1.14.8:

Here is a mini-example for a unit test:

# options('datatable.alloccol' = 1024L)
df <- data.table::data.table(x = 1)

for (i in 1:2000) data.table::set(df, j = as.character(i), value = 1)

# Error in data.table::set(df, j = as.character(i), value = 1) :   
# Internal error: DT passed to assign has not been allocated enough column slots. l=1025, tl=1025, adding 1

Adjusting datatable.alloccol resolves the issue.

setalloccol is executed at the end of data.table() with default argument from options('datatable.alloccol').

Would adding alloccol as an argument to data.table() be a viable option, as a way to override the default? This would also draw attention to the relevant documentation as requested in #1831.

Currently, one has two options: either adjust global options before making a DT, or execute setalloccol again on the created DT.

Will this be fixed at some point? If not, what would be the recommended way to deal with that?
Probably that in scenarios like for (i in 1:2000) data.table::set(df, j = as.character(i), value = 1) you should use := instead?
Like for (i in 1:2000) df[, :=(colname = 1), env = list(colname = as.character(i))]
It is a bit strange that in the documentation it says that set would be "appropriate for use inside a for loop" when in fact only := works in a for loop (at least in this scenario)

@Noskario

@Noskario

Probably that in scenarios like for (i in 1:2000) data.table::set(df, j = as.character(i), value = 1) you should use := instead?

If you find yourself creating a data.table with many columns, you should reconsider your whole approach. The package is much better optimized for many rows than for many columns. Consider carefully if you shouldn't switch to something that creates a more "long-format" data.table.

If you want to stay with your approach, check if length(DT) < truelength(DT) and increase the truelength with setalloccol if it isn't. Or, if you know the number of iterations, add the necessary number to the truelength using setalloccol before the loop.

@Noskario

@Noskario Sorry for the late reply. As mentioned by Roland, you can either call setalloccol upfront and add the nubmer of necessary slots to your data.table which is definetely the recommended approach.

If you do not really care about memory (a column slot takes up 8 KB IINM) you can also the default of options(datatable.alloccol) which is currently set 1024

(textproc/R-data.table) Updated 1.16.4 to 1.18.4, Fix build against 4…

Navigation Menu

Repository navigation

Internal error: DT passed to assign has not been allocated enough column slots #4100

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Internal error: DT passed to assign has not been allocated enough column slots #4100

Description

Activity

DavorJ commented on Jul 10, 2023

DavorJ commented on Jul 10, 2023

Noskario commented on Dec 10, 2025

ecoRoland2 commented on Dec 10, 2025

ben-schwen commented on Dec 28, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions