Repository navigation
Need long-vector support #3957
Description
Activity
Would anyone know why
data.framehas a limit in rows in the first place? I couldn't find an answer on R-help. Just wondering why they implemented it this way?@renkun-ken I know we have built long vector support in a few places, it might be helpful to make a prioritized list of where it's still not implemented?
@MichaelChirico Do we have a list of places where long vectors are supported or could be supported?
I don't think it make sense to start with anything else than
forder(which is neded for grouping and joining). AFAIK Matt has that on his radar. Once we will haveforderworking for long vectors, then the rest can follow. Just speculating now... but we may ended up making an abstraction layer to handle int32/int64 length, because supporting long vectors always may impose performance penalty even for int32 length vectors. It all needs benchmarking to make proper decisions. My suggestion is to hold on with this issue for @mattdowle.Reacted by Michael Chirico@robbyjo, I saw your patch. I appreciate your need for long data.tables. You're not alone; I would like to see them happen. Maybe Dan Luu is right and testing, not code review is the future, but for now we the data.table developers still have a duty to our users to actually fit into our heads and understand the code that they run every day. This complicates acceptance of external large patches (a commit history would help!), especially those generated by a tool that doesn't need human understanding to work. Use of zero-length
rownamesattribute for long tables is a bit suspect; proper tests for long vector support will require materialising the ALTREP and working with the large allocations.int64_tfor in-memory indices is an unfortunate historical choice caused by compiler warnings;R_xlen_texpresses the meaning better.rowIndex-related helpers that dispatch on type inside loops aren't necessarily great for performance. Partial ALTREP support is risky due to how we useTRUELENGTH.@Roleren, I haven't forgotten about your comment. The largest computers I currently have almost unlimited access to in the university have 32G of RAM (and also 32G of relatively fast swap on an NVMe SSD, but that's it). It would be really helpful if you could make a script that would take a toy dataset with
$2^{31}-1$ rows and perform all the merges, rolling by-group operations, mutations, et cetera, that you need to work, while fitting in the smallest amount of RAM possible. We could then extend the toy dataset to$2^{31}$ rows (and as much as feasible above that) and try to implement long vector support for all of those, until the script works.Meanwhile, R-core is working on native 64-bit integers in base R, so ideally there should be a forward-compatible way to implement that.
Reacted by Jan Gorecki- Oooh native 64 bit ints will be nice!…On Tue, Sep 15, 2026 at 7:45 AM aitap ***@***.***> wrote: *aitap* left a comment (Rdatatable/data.table#3957) <#3957 (comment)> @robbyjo <https://github.com/robbyjo>, I saw your patch <#570 (comment)>. I appreciate your need for long data.tables. You're not alone; I would like to see them happen. Maybe Dan Luu is right and testing, not code review is the future <https://danluu.com/ai-coding/>, but for now we the data.table developers still have a duty to our users to actually fit into our heads and understand the code that they run every day. This complicates acceptance of external large patches (a commit history would help!), especially those generated by a tool that doesn't need human understanding to work. Use of zero-length rownames attribute for long tables is a bit suspect; proper tests for long vector support will require materialising the ALTREP and working with the large allocations. int64_t for in-memory indices is an unfortunate historical choice caused by compiler warnings; R_xlen_t expresses the meaning better. rowIndex-related helpers that dispatch on type inside loops aren't necessarily great for performance. Partial ALTREP support is risky due to how we use TRUELENGTH. @Roleren <https://github.com/Roleren>, I haven't forgotten about your comment <#7438 (comment)>. The largest computers I currently have access to in the university have 32G of RAM (and also 32G of relatively fast swap on an NVMe SSD, but that's it). It would be really helpful if you could make a script that would take a toy dataset with $2^{31}-1$ rows and perform all the merges, rolling by-group operations, mutations, et cetera, that you need to work, while fitting in the smallest amount of RAM possible. We could then extend the toy dataset to $2^{31}$ rows (and as much as feasible above that) and try to implement long vector support for all of those. Meanwhile, R-core is working on native 64-bit integers <https://stat.ethz.ch/pipermail/r-devel/2026-July/084631.html> in base R, so ideally there should be a forward-compatible way to implement that. — Reply to this email directly, view it on GitHub <#3957?email_source=notifications&email_token=AAEN7VUDW622NODRD5CNDKL5PETWVA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNRXHE3DENBWGEZ2M4TFMFZW63VGNVQW45LBNSSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5679624613>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEN7VTWCIATDCQ3BTMA5JT5PETWVAVCNFSNUABDKJSXA33TNF2G64TZHMZDANJZG42TAOJ3JFZXG5LFHM2TANJRGM3DGNZTUF3AE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>Reacted by Jan Gorecki
Just wanted to mention I have a working 64 bit data.table fork here:
data.table link
I have an idea that those of us that need 64 bit badly, work together to produce a good product now (outside of the main data.table repo), with extensive testing that must be manually checked by people (no blind ai build).I started this earlier doing the build manually and now I am using Astra xhigh currently, supported features are:
- Implemented class data.table_long (64bit support for main functions)
- fread, rbindlist both support automatic conversion from 32 to 64 bit row index size
- DT[, sum(score), by = .(col1, col2, col3)] etc supports fast aggregation of tables up to 2^64 rows
- Added 350 new tests for better code coverage and existing tests all pass.
We could then compare our forks and use the best from all of them. When we believe it is ready (maybe in 2027 ish, we could start discussing a possible merge, lets see what the R guys does with native 64bit support etc). I am actively developing on this patch whenever I have some spare time.
Possible link topic would be: #570Also a note, I have a also made a related fork of fst / fstcore:
fst linkWhich now supports:
- Supports direct conversion from fst to data.table_long (name of the 64bit data.table class), instead of the default base R list output.
- direct appending of columns to disc file!
- I have started implementing appending of rows to disc file (a bit more tricky, since the last 4k chunk must be zstd recompressed per column)
Also if anyone has new formats with random access for very large tabular data, that is even close to the data.table + fst combo, let me know. I have played a bit with qs2, but it is not even close, and the multi-threading there is much less reliable compared to fst.
Currently,
data.tabledoes not support long vectors as columns. Long vector support was not requested much (related: #2751, #2340). However, I do need handle data with long vectors more frequently as I step more into the area of high frequency data.I'm not sure how difficult it is to support long vectors but it would be very nice and much appreciated to put this on the roadmap.