Rerun repeated uniqueN test #3438

Activity

1. mattdowle

   added this to the 1.12.4 milestone on Mar 1, 2019
2. jangorecki

   modified the milestones: 1.12.4, 1.13.0 on Sep 17, 2019
3. mattdowle

   modified the milestones: 1.12.7, 1.12.9 on Dec 8, 2019
4. jangorecki commented on Apr 7, 2020

   

   jangorecki

   on Apr 7, 2020

   Last edited by jangorecki

   Member

   More actions

   Up to date timings below. There is still a problem in uniqueN by group.
   We basically need what @mattdowle described in #3743 (comment) and #1120 would also be nice.

   busy machine (20 cores)

   #[1] 10
   #   user  system elapsed 
   #172.084   1.640  18.179 
   
   #[1] 1
   #   user  system elapsed 
   # 10.608   0.596  11.206 
   
   #[1] 20
   #still computing

   idle machine (32 cores)

   #[1] 16
   #   user  system elapsed 
   #258.708   0.876  16.638  
   
   #[1] 1
   #   user  system elapsed 
   #  8.405   0.132   8.539 
   
   #[1] 32
   #    user   system  elapsed 
   #1066.565    1.180   35.085 

   code

   library(data.table)
   N_X = 1e6
   n_day = 60
   n_clientid = 1e5
   n_Platform = 7
   X = data.table(
     day = sample(1:n_day, N_X, TRUE),
     clientid = as.character(sample(1:n_clientid, N_X, TRUE)),
     Platform = as.character(sample(1:n_Platform, N_X, TRUE))
   )
   
   setDTthreads(NULL) # default
   getDTthreads()
   system.time(
     X[, .(x = uniqueN(day) - 1L,
           first_active_day = min(day),
           last_active_day = max(day)),
       by = .(Platform, clientid)]
   )
   
   setDTthreads(1)
   getDTthreads()
   system.time(
     X[, .(x = uniqueN(day) - 1L,
           first_active_day = min(day),
           last_active_day = max(day)),
       by = .(Platform, clientid)]
   )
   
   setDTthreads(0)
   getDTthreads()
   system.time(
     X[, .(x = uniqueN(day) - 1L,
           first_active_day = min(day),
           last_active_day = max(day)),
       by = .(Platform, clientid)]
   )
5. jangorecki

   added

   performance

   on Apr 7, 2020
6. jangorecki

   linked a pull request that will close this issuethrottle threads for iterated small data tasks #4484on Jun 13, 2020
7. jangorecki

   modified the milestones: 1.12.11, 1.12.9 on Jun 13, 2020
8. mattdowle commented on Jun 18, 2020

   

   mattdowle

   on Jun 18, 2020

   MemberAuthor

   More actions

   Closed by #4484. See benchmark near the top of the top comment here: #4484 (comment)
9. mattdowle

   closed this as completedin #4484on Jun 18, 2020