lapply GForce opt could work also without .SD #5032

Activity

1. jangorecki

   added

   GForceissues relating to optimized grouping calculations (GForce)issues relating to optimized grouping calculations (GForce)

   on May 31, 2021
2. jangorecki commented on May 31, 2021

   

   jangorecki

   on May 31, 2021

   Last edited by jangorecki

   MemberAuthor

   More actions

   also when having manually expanded lapply it does work

   d[, list(sum(Petal.Length), sum(Sepal.Length)), by=Species, verbose=TRUE]
   #lapply optimization is on, j unchanged as 'list(sum(Petal.Length), sum(Sepal.Length))'
   #GForce optimized j to 'list(gsum(Petal.Length), gsum(Sepal.Length))'
   #Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.000

   but it doesn't work if we will want to combine two lists like this (below is just one list, but using c() already switches of GForce):

   d[, c(list(sum(Petal.Length), sum(Sepal.Length))), by=Species, verbose=TRUE]
   #lapply optimization is on, j unchanged as 'c(list(sum(Petal.Length), sum(Sepal.Length)))'
   #GForce is on, left j unchanged
   #Old mean optimization is on, left j unchanged.
   #Making each group and running j (GForce FALSE) ... 

   so the only thing that works is to manually combine both lists as well

   d[, list(sum(Petal.Length), sum(Sepal.Length), mean(Petal.Width), mean(Sepal.Width)), by=Species, verbose=TRUE]
   #lapply optimization is on, j unchanged as 'list(sum(Petal.Length), sum(Sepal.Length), mean(Petal.Width), mean(Sepal.Width))'
   #GForce optimized j to 'list(gsum(Petal.Length), gsum(Sepal.Length), gmean(Petal.Width), gmean(Sepal.Width))'
   #Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.000
3. myoung3 commented on Jul 1, 2021

   

   myoung3

   on Jul 1, 2021

   Contributor

   More actions

   weirdly c() doesn't always switch off GForce. See

   library(data.table)
   #> Warning: package 'data.table' was built under R version 4.0.5
   options(datatable.verbose=TRUE)
   d = as.data.table(iris)
   
   d[, c(lapply(.SD,sum), lapply(.SD,mean)), by=Species, verbose=TRUE]
   #> Finding groups using forderv ... forder.c received 150 rows and 1 columns
   #> 0.000s elapsed (0.000s cpu) 
   #> Finding group sizes from the positions (can be avoided to save RAM) ... 0.000s elapsed (0.000s cpu) 
   #> lapply optimization changed j from 'c(lapply(.SD, sum), lapply(.SD, mean))' to 'list(sum(Sepal.Length), sum(Sepal.Width), sum(Petal.Length), sum(Petal.Width), mean(Sepal.Length), mean(Sepal.Width), mean(Petal.Length), mean(Petal.Width))'
   #> GForce optimized j to 'list(gsum(Sepal.Length), gsum(Sepal.Width), gsum(Petal.Length), gsum(Petal.Width), gmean(Sepal.Length), gmean(Sepal.Width), gmean(Petal.Length), gmean(Petal.Width))'
   #> Making each group and running j (GForce TRUE) ... gforce initial population of grp took 0.001
   #> gforce assign high and low took 0.000
   #> This gsum took (narm=FALSE) ... gather took ... 0.001s
   #> 0.001s
   #> This gsum took (narm=FALSE) ... gather took ... 0.000s
   #> 0.000s
   #> This gsum took (narm=FALSE) ... gather took ... 0.000s
   #> 0.001s
   #> This gsum took (narm=FALSE) ... gather took ... 0.000s
   #> 0.000s
   #> This gsum took (narm=FALSE) ... gather took ... 0.000s
   #> 0.000s
   #> This gsum took (narm=FALSE) ... gather took ... 0.000s
   #> 0.000s
   #> This gsum took (narm=FALSE) ... gather took ... 0.001s
   #> 0.001s
   #> This gsum took (narm=FALSE) ... gather took ... 0.000s
   #> 0.000s
   #> gforce eval took 0.003
   #> 0.000s elapsed (0.000s cpu)
   #>       Species Sepal.Length Sepal.Width Petal.Length Petal.Width Sepal.Length
   #> 1:     setosa        250.3       171.4         73.1        12.3        5.006
   #> 2: versicolor        296.8       138.5        213.0        66.3        5.936
   #> 3:  virginica        329.4       148.7        277.6       101.3        6.588
   #>    Sepal.Width Petal.Length Petal.Width
   #> 1:       3.428        1.462       0.246
   #> 2:       2.770        4.260       1.326
   #> 3:       2.974        5.552       2.026
4. ben-schwen

   self-assigned this

   on Oct 28, 2025
5. ben-schwen

   closed this as completedin #7401on Jan 15, 2026