Repository navigation
[R-Forge #1458] Add fast insert() function #660
Description
Submitted by: Matt Dowle; Assigned to: Nobody; R-Forge link
insert(DT, list(...))
and
insert(DT, ...)
This would grow DT, and memcpy the value in.
Possibly, even respecting keys, and doing a memmove to make space.
Related, tables() could report length and truelength separately. There could be an allocate(DT,n) function which increases truelength but not the length (allowing many quick insert()s if user knows that is about to happen).
Activity
jangorecki commentedon Nov 15, 2015
+1
This would be useful to use data.table as reporting engine. Using Rserve any other language can outsource the job to data.table engine, insert allows to constant access without the need to "reload" data.table to put new rows.
ssssman commentedon Dec 16, 2016
Fast insertion will be a very nice feature to have. We can use data.table as an in-memory database and build a R server, just like KDB+ and q.
http://code.kx.com/wiki/Cookbook/PerformanceTips
q)load
:trade q)row: first trade q)\t do[1000000; insert[trade; row]]1968
ssssman commentedon Dec 16, 2016
Appending a list to the nested list is quite fast. If data.table is the list of list, fast insertion is possible?
runBenchmark <- function(n) {
microbenchmark(times = 5,
list_= {
a <- list(NA)
for(i in 1:n) {a <- list(a, list(i))}
}
}
jangorecki commentedon Dec 16, 2016
@ssssman data.table is column oriented, so when you append to list, you append columns, not rows.
ssssman commentedon Dec 17, 2016
@jangorecki understood. Do you think it is possible to add one-row data.table to another much bigger one quickly at microsecond level? rbind will take about 0.2 millisecond.
jangorecki commentedon Dec 17, 2016
@ssssman depends what you expect. You still need to pre-allocate "empty" (full of NAs probably) rows. Without pre-allocating memory I don't think it would be possible to insert in-place. Then you need to handle those "empty" rows in your application, and insert values there when needed. It is not perfect as you most likely will need to filter out those rows for most operations, and you still need to pre-allocate them. Also maintaining key is getting tricky (if inserted key columns are always bigger than existing one, then setting
sortedattribute manually should be enough).MichaelChirico commentedon May 2, 2019
@jangorecki This is for
inserting rows? Is there a difference to #635? Reprex would help as I don't quite followjangorecki commentedon May 2, 2019
It is different but it requires same changes in underlying structure to over-allocate rows. This need to be changed the very lowest level in our C code, and will need to changes in multiple places.
jangorecki commentedon Nov 17, 2022
in linked issue another api has been proposed. API that works inserting into data.frame
jangorecki commentedon Nov 17, 2022
We have to overallocate rows in a data.table (which means overallocate each column, set growable bit). Then inserting is possible in-place, up to the limit we have over allocated.
jangorecki commentedon Oct 30, 2023
Another useful feature of such fast insert could be preserving the key. That would require a merge in-place into overallocated space (some extra memcpy), rather than just insert after existing data into overallocated space.