Repository navigation
parallel optimization for common operators like +, sum and many others #2919
Description
Activity
gcc-8 and 20 cpus:
> system.time(a1<-sum(x)) user system elapsed 1.161 0.000 1.161 > system.time(a2<-fsum(x)) user system elapsed 2.973 0.000 0.151actually not just aggregations but also other common operators
20 cores> system.time(a1<-`+`(x, y)) user system elapsed 1.499 1.842 3.342 > system.time(a2<-fadd(x, y)) user system elapsed 5.073 5.014 0.520- changed the title
[-]parallel aggregations with openmp[/-][+]parallel optimization for common operators like `+`, `sum` and many others[/+]on Aug 17, 2018 I would say this is out of scope for data.table, if anything it should be a task for GForce and then handled by parallel group processing.
data.table being the host of a bunch of parallelization routines doesn't seem appropriate, better for a different package to be responsible for that (possibly even under Rdatatable org, but not in data.table package). I'll close for now.
This issue was about aggregates without grouping so GForce won't apply here
We can use openmp to parallelize some common aggregations like
sum.It has to play nice with gforce optimization, which makes it a little bit more tricky to do than implementation shown below.
It can give nice speed up, on
rnorm(1e9)and roundoff