Repository navigation
data.table new column := slower than base R (?) #921
Description
Activity
@szilard thanks! R3.1 doesn't update in-place. It shallow copies other columns and adds the new column..
As to why this happens in data.table, just ran a
debugonce()on[.data.table. The issue comes down to the same for-loop as shown in #727. This change seems to have happened during theby=.EACHIimplementation.Thanks @arunsrinivasan!
Yes, I meant base R >=3.1 is not copying the existing columns. So
df$y <- 2*df$xtakes now approx the same asy <- 2*df$x.I wonder if data.table can be made faster. First, I increase a bit the data size:
library(data.table) dt <- data.table(x = runif(500e6)) df <- as.data.frame(dt) system.time( df$y <- 2*df$x ) system.time( dt[,y := 2*x] )I get now ~ 2 sec for base R and ~ 5 sec for data.table.
Then I clean up (
dt[,y := NULL]which takes suprisingly ~ 3 sec), and finally I run this:Rprof() invisible(dt[,y := 2*x]) Rprof(NULL) summaryRprof()and get this:
$by.self self.time self.pct total.time total.pct ".Call" 2.10 40.70 2.10 40.70 "*" 2.02 39.15 2.02 39.15 "seq_len" 1.04 20.16 1.04 20.16 $by.total total.time total.pct self.time self.pct "[" 5.16 100.00 0.00 0.00 "[.data.table" 5.16 100.00 0.00 0.00 ".Call" 2.10 40.70 2.10 40.70 "copy" 2.10 40.70 0.00 0.00 "*" 2.02 39.15 2.02 39.15 "eval" 2.02 39.15 0.00 0.00 "seq_len" 1.04 20.16 1.04 20.16 $sample.interval [1] 0.02 $sampling.time [1] 5.16Any potential sources for speed-up?
I've already linked to the other issue, where I've listed what I think is the cause - there's an unwanted
copyin this case. Maybe something else in addition as well, from the timings you show. It's very much fixable. Thanks again.Awesome, thanks (I've seen the other issue, just was not obvious for me it's an easy fix). Also wondering about the speedup, but I guess ~ 2 sec in this case (see profile above).
What about the
seq_len? That's another ~ 1 sec, there must be aseq_len(500e6)call which takes precisely that much (~ 1 sec)? Is that avoidable?(With those 2 changes now it would be on par with base R ~ 2 sec total.)
Right. Don't really understand why there's a
seq_lenas well.I think the
seq_lencall that takes ~ 1 sec is thisSDenv$.I = seq_len(SDenv$.N)here https://github.com/Rdatatable/data.table/blob/master/R/data.table.R#L1075Thanks a lot. Will take a look.
We could really do with a timing regression framework! Like Python's vbench.I can also confirm a lot of performance impact by migrating to 1.9.4.
Is there a proposed timeframe for this bugfix? I suppose I could revert back to 1.9.2 for the time being.
@alexcpsec performance impact with respect to just
:=? If something else, it'd be great to have a new post with an example.It is a very convoluted function I have that makes heavy use of
:=where I had a significant hold-up.I could try to Rprofile the whole thing with both 1.9.2 and 1.9.4 versions so I can give you a better idea of how the versions compare to each other in a larger piece of code. Would this be useful for you?
@alexcpsec, yes, that would be incredibly useful, in the absence of a reproducible example. Thanks.
Now I get this:
# clean session library(data.table) dt = data.table(x = runif(100e6)) system.time(dt[,y := 2*x]) # user system elapsed # 0.384 0.563 0.956 df = data.frame(x = runif(100e6)) system.time( df$y <- 2*df$x ) # user system elapsed # 0.376 0.554 0.933
The timings are more or less the same, and we can't avoid a copy here. But there are cases where we can delay the copy like R v3.1.0+ does, by shallow copying. That'll be taken care of in #617.
@arunsrinivasan Awesome!! Great fix.
While data.table new column := used to be >100x faster than base R, base R (>=3.1) updates now data.frames in place and caught up. I wonder why data.table is slower for example in this case:
I get:
base R: 0.272 0.300 0.572 (user system elapsed)
data.table: 0.696 0.744 1.444
R 3.1.1 data.table 1.9.4.