fread should not create data.tables with more than 2^31 rows

I just imported a huge file with more than 2^31 rows using fread on a machine with 512 GB RAM. It worked great. Unfortunately, I couldn't do anything with the resulting data.table. Autoprinting or subsetting returned a "long vectors not supported yet" error, which isn't surprising.

I suggest to stop fread before the data.table gets too large. Maybe it could return (with a warning) as many rows as R supports.

The fread documentation mentions field width limits but I don't see a mention that R has a limit regarding the number of rows.

My solution was to use nrows and skip to import as two data.tables but that was after I wasted time with reading the whole file.

Related: #570, #3957.

Rather than turn off this support, I'd rather add support -- could you be more specific about which operations are failing?

Happy to do so. Feel free to ask me to test other operations. However, I'm using the server too much right now to install package versions from github. That would have to wait until I don't have looming deadlines.

I'm using R 4.5.1 and data.table 1.17.8 via RStudio Server 2024.12.1+563 "Kousa Dogwood" Release (27771613951643d8987af2b2fb0c752081a3a853, 2025-02-02) for Ubuntu Jammy.

The data.table has about 3e9 rows and 4 numeric columns. Looks like the main issue is in dim.data.table but there might be others down the line.

DT <- fread("<file path>", nThread = 5)

DT
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

print(DT)
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

nrow(DT)
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

DT[1:10]
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

setnames(DT, 1, "test")
#works

DT[, .N, by = test] #result would have about 36e6 rows
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

DT[, lapply(.SD, class)]
#Error in dim.data.table(x) : 
#  long vectors not supported yet: ../../src/include/Rinlinedfuns.h:551

Fixing dim opens the can of worms that dim can return either integer or double depending on the size. Might even fully switch to double if we want long vector support.

There is a value in following base R convention on that, although it may not be very robust. If we go with always double then we could provide transition option, disable long vector, which keeps everything as it was.

If we could pull this off, this could be a great competitive advantage, because none of the other data.frames can handle long vectors:

> data.frame(raw(2^31))
Error in if (mirn && nrows[i] > 0L) { :
  missing value where TRUE/FALSE needed
In addition: Warning message:
In attributes(.Data) <- c(attributes(.Data), attrib) :
  NAs introduced by coercion to integer range
> S4Vectors::DataFrame(x = raw(2^31))
Error in S4Vectors::DataFrame(x = raw(2^31)) :
  cannot coerce class "raw" to a DataFrame
In addition: Warning message:
In new_DataFrame(setNames(list(from), "X"), nrows = length(from)) :
  NAs introduced by coercion to integer range
> tibble::tibble(raw(2^31))
Error in `new_tibble()`:
! `nrow` must be a nonnegative whole number smaller than 2^31.
Run `rlang::last_trace()` to see where the error occurred.

Introducing long vector support will require replacing the approximately grep -rPic '(?<!x|TRUE|SET)length *\(' src/ | awk '{sub(".*:","");a+=$0}END{print a}' = 634 uses of length() with xlength() and R_len_t/int used as an index type with R_xlen_t. Eventually we'll bump against R's own limits (a large portion of R itself is not ready for dim() to return anything but an INTSXP vector, especially for matrices), but it could be interesting to see how far can this be taken.

I just want to give thumbs up to do this from the genetics community.

We are starting to approach an era where our data easily reaches > 2^31 -1 rows,
so we have to spend a lot of time currently doing smart tweaks to avoid the data.table row limit.

And from our tests, no formats we have tested beats data.table + fst, for storage and processing of tabular data.

The human genome has ~ 3 billion positions over all chromosomes, so even 2^32 row limit would fix a lot of the issues we have, in that we currently need to chunk the data.tables and make index files to point to the correct positions etc.

my understanding is the code changes required may not be that drastic, however, it's a bit hard to do testing on personal machines.

if you are working with these huge data sets, you are very well suited to offering patches -- something to consider!

If you could share the minimal viable set of required operations and a couple of datasets, with slightly below 2^32 rows and slightly above, or scripts that would generate them, that would also help.

# Running on intel xeon, 0.5TB ram server
library(data.table)

# Factor from ints
.factor_from_codes <- function(codes, levels) {
  structure(codes, levels = levels, class = "factor")
}

# Make a toy genome coverage dataset, you can remove some columns to reduce memory footprint
make_ofst_dt <- function(n, start = 1L, chr = 1L, chr_levels = c("chr1", "chr2"), strand_levels = c("+", "-")) {
  data.table(
    seqnames = .factor_from_codes(rep.int(as.integer(chr), n) , chr_levels),
    start    = start + seq_len(n) - 1L,               # 1..n (fits since n ~ 1.43e9 < 2^31-1)
    strand   = .factor_from_codes(rep.int(1L, n), strand_levels),
    score    = rep.int(1L, n)
  )
}
# Make a list of length 2 (i.e. with 2 data.tables, that either peaks 2^31 total rows or not)
# sizes that sum to > 2^31
n_chunks <- 2
base_size <- 2
base <- floor(2^31 / base_size)                  # 1,431,655,765
sizes <- rep(base, n_chunks) + 2
start_index <- c(1, cumsum(sizes) / 2)
# 70GB object that fails, ~ 30 seconds to make
dt_list <- lapply(seq.int(n_chunks), function(i) {
  message("-i ", i)
  make_ofst_dt(sizes[i], start_index[i], i)
})


rbindlist(dt_list) # this will fail
rm(dt_list); gc()
# 70GB object that does not fail, ~ 30 seconds to make
# sizes that sum to  2^31 - 10*2 + 2*2, i.e. valid
dt_list <- lapply(seq.int(n_chunks), function(i) {
  message("-i ", i)
  make_ofst_dt(sizes[i] - 10, start_index[i], i)
})
rbindlist(dt_list) #this will not fail
rm(dt_list); gc()

I do feel morally obligated to point out that resource planners will not be a fan of enabling even more relative RAM usage in 2026 😂

OK, what about this compromise:

You set up an official 64bit row index branch (or I guess it is 2^63 -1 since it will be signed)
Then either you or me make a pull request to that new branch with regex replace for all length calls with xlength,and same for R_xlen_t.
then most importantly, I and others with servers do extensive testing on our end before we move on, that could not hurt right ?

Even if branch is not merged before 2027, that is fine, since most of us using large servers know how to use GitHub branches of R packages anyway.

Long vector support is smth I defintely will try/want to work on after finishing delete/insert with growable vectors.

Replacing length with xlength is not the problem. Minor functions like dim who will diverge from the default generics etc. will be

Navigation Menu

Repository navigation

fread should not create data.tables with more than 2^31 rows #7438

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fread should not create data.tables with more than 2^31 rows #7438

Description

Activity

MichaelChirico commented on Nov 25, 2025

ecoRoland2 commented on Nov 26, 2025

ben-schwen commented on Nov 26, 2025

jangorecki commented on Nov 26, 2025

aitap commented on Nov 26, 2025

Roleren commented on Jan 22, 2026

MichaelChirico commented on Jan 22, 2026

aitap commented on Jan 22, 2026

Roleren commented on Jan 22, 2026

MichaelChirico commented on Jan 22, 2026

Roleren commented on Jan 22, 2026

ben-schwen commented on Jan 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions