Repository navigation
Be less conservative about ALTREP #4697
Description
Discovered via SO Q&A:
https://stackoverflow.com/a/63770450/3576984
Currently j expands all ALTREPs, which can lead to significant performance degradation:
system.time(for (i in 1:40732170) {})
# user system elapsed
# 0.764 0.004 0.880
data.table()[ , system.time(for (i in 1:40732170) {})]
# user system elapsed
# 3.756 0.020 3.927
I don't know enough about ALTREP to say for sure which cases we can/can't avoid expansion, but hopefully we can do better than the above.
(and actually I'm not 100% sure ALTREP is to blame here, but seems the likeliest candidate)
Activity
ColeMiller1 commentedon Sep 11, 2020
Should we also be more aggressive about using
ALTREP? That is,.Iis almost free now.irowscould be refactored from beingNULLand might (?) end up in simpler code.R 3.5.0 is 2.5 years old so I think we're getting close to where incorporating it makes sense.
MichaelChirico commentedon Nov 22, 2020
Great talk from Gabe Becker here:
https://youtu.be/8i7ziLqsE2s
I think a low hanging fruit could be to set (first) keys to KNOWN_INCREASING, that could e.g. let R by very fast in doing
unique(DT$key).jangorecki commentedon Nov 22, 2020
Unless there are NAs which R places at the end and not at the front.
MichaelChirico commentedon Nov 22, 2020
Actually no! one of the sortedness enum fit our case if I'm not mistaken:
SORTED_INCR_NA_1STlink to slides here:
https://www.bioconductor.org/help/course-materials/
ben-schwen commentedon Jan 18, 2024
Well if we manage to support altrep vectors as columns then out-of-memory data.table should be easier targetable.
tdhock commentedon Jan 18, 2024
is there a more convincing real example?
it seems like in a real example if you are computing something over a large loop you will also be returning a large number of rows, and you should have memory for that, so optimizing for ALTREP is not necessary.
jangorecki commentedon Jan 18, 2024
out-of-memory data.table is very desired feature due to severity of consequences when you just don't have enough memory. Together with long vector support these were two main points on the roadmap for data.table new features that Matt was working on. As he said, long vector support is not that great deal as long as there is no out-of-memory data.table.