Repository navigation
progress indicator/bar for "by" operations #3060
Description
I frequently run by operations that take minutes and sometimes hours to complete. To deal with the uncertainty of what's going on, I often resort to printing .BY in the i-expression:
dt = data.table(a = 1:5)
dt[, {print(.BY)
Sys.sleep(10)
5}
, by = a]
Would be great to have an automatic progress bar for these, similar to fread, if the expected run time is greater than e.g. 30s. Expected run time can be estimated from actual average time spent running the i-expression for each by item so far * number of remaining by items.
Activity
franknarf1 commentedon Sep 21, 2018
Dupe of closed #1409
I manually add progress-printers when I see an operation going slowly, but it'd be nice if it were automatic whenever time > 30 s as you suggest.
mattdowle commentedon Sep 22, 2018
Related: #3050
jangorecki commentedon Sep 22, 2018
@franknarf1 agree with Frank, but lets print it only for
interactive().eantonya commentedon Sep 25, 2018
I've been thinking about this, and ideally I'd like to see the following information:
MichaelChirico commentedon Sep 26, 2018
Kodiologist commentedon Oct 1, 2019
I too would like this as an opt-in feature of data.table. In the meantime, here's a code snippet to manually add a progress bar with a percentage, given a data table
d. ReplaceGROUP_EXPRESSIONandJ_EXPRESSIONto taste.jangorecki commentedon Jun 27, 2020
iis not run for each item inby, even if it would, then still such estimation would be quite inaccurate because it doesn't say anything aboutj, to make that automatically we would have to measure time of iterations over progressing groups evaluated inj, and then decide to print at some threshold. This unfortunatelly would impose an overhead, it may be better to enable this feature explicitly rather than automatically.MichaelChirico commentedon Jun 28, 2020
the same can be said of the progress bar for fread/fwrite right? at least w.r.t. the overhead concern
Kodiologist commentedon Jun 28, 2020
You could use
pbapplyas a dependency, since it already implements time estimation. In my example above, you'd replacetxtProgressBarwithstartpbandsetTxtProgressBarwithsetpb.jangorecki commentedon Jun 28, 2020
Time estimation, once we start to evaluate
j, is not something we should be worried about. Previous comment was about time estimation before we start evaluatingj. Recent PR by @MichaelChirico provides efficient way to know how many groups we have being insidej, so.GRP/.NGRPis not a problem. Problem is that to handle that automatically, when time is longer than a threshold, we have to sum timings always, for all iterations, and this will add overhead. Just R C API impose quite big overhead there already so it doesn't look fine for me to add this one, by default.Kodiologist commentedon Jun 28, 2020
Having to turn it on explicitly with an argument, like
progress = TRUE, makes sense to me.joshhwuu commentedon Jul 15, 2024
What would be the best way to opt-in to this feature? I was wondering what something like
showProgress = TRUEwould look like within the square bracket syntax.Kodiologist commentedon Jul 15, 2024
I imagine a typical call would look like
dt[, by = ..., progress = TRUE, ...], where the last...isj.