Repository navigation
frollapply could optionally use mirai+mori rather than base R parallel's fork #7891
Description
Activity
@MichaelChirico @aitap @ben-schwen @tdhock just wanted to check with you if we all agree that we are open for adding mirai and mori to suggested dependencies?
Reacted by Michael Chirico and aitapKeeping it Suggests SGTM. Is the proposal to fall back to {parallel} if {mirai} unavailable, or to disable parallelism in that case?
The former means continuing to maintain two code paths, while the latter means possibly less functionality if {mirai} is difficult to install in some environments (I'm not familiar enough to say if that's moot).
Has R declared similar functionality as {mirai} offers as out-of-scope / won't be supported in {parallel}?
Is the proposal to fall back to {parallel} if {mirai} unavailable, or to disable parallelism in that case?
I would say that on linux default should be still parallel. mirai has to spawn fresh R session for each worker so overhead of that will be still there.
Has R declared similar functionality as {mirai} offers as out-of-scope / won't be supported in {parallel}?
I think it is supported, but we are not using high level wrappers that could let us use mirai through parallel, we are using fork-related functions exported from parallel.
It should be fine to add both packages to Suggests.
moriis as minimal as it gets, I'm actually not sure why itSuggests: mirai; there seem to be no references to it inside the package. We could reimplement our own shared-memory ALTREP classes, but it's not fun, andmorialready does the job. (Anything non-ALTREP'pable is still serialized.morirequires R ≥ 4.3, although character / integer / logical / numeric ALTREP classes appeared in 3.5.0 and raw / complex appeared in 3.6.0. Support for lists does require 4.3.0 or above.)(Edit: shared memory is actually not zero-copy, but copy-once: to share a vector, we have to copy it into a new shared memory region. Not even Linux is crazy enough to let a program
mmap()someone else's address space through/proc/PID/memfor zero-copy sharing.)mirairequiresnanonext(libnng), which has problems with TLS and HTTP in some configurations, but should work fine for our purposes (as a replacement forparallel::mcparallelon Windows).Has R declared similar functionality as {mirai} offers as out-of-scope / won't be supported in {parallel}?
Support for
miraiclusters inparallelhas to do with snow-style (distributed) clusters whichdata.tabledoesn't use.Many people have wanted something like
parallel::mcparallelon Windows, but the exact semantics are very hard to replicate (you'd have to be Cygwin). Running something in a background R process, on the other hand, is relatively easy (start withtools::Rbut usesystem2(wait = FALSE)and some other means of discovering whether the worker process is done).mirai's added value is in doing everything much faster:libnnginstead of temporary files for IPC and a custom serialization format for common object kinds instead of baseserialize(). In theory, we could use justmoriand hand-launched child processes.Users whose R is not forkable (RStudio, Ark, ...) will probably prefer the non-
mcparallelapproach even on non-Windows.Suggests sounds ok to me (although I don’t use frollapply).
For parallel interfaces I tend to do something like thisfrollapply = function(…, LAPPLY=lapply){ … }
then the user can provide a parallel function similar to lapply (future_lapply etc), or just keep regular lapply (useful for debugging).
See for exampleproj_compute_all()in https://github.com/tdhock/mlr3resampling/blob/main/R/proj.R
frollapply now uses base R's parallel package fork for parallel computation of FUN. Adding mirai and mori packages together it could provide an alternative backend for parallel processing in frollapply.
Cost of it means 2 extra suggested dependency (3 pkgs in total, including nanonext, dep of mirai).
Benefits are that parallel processing will work on Windows, and other OSes where fork is not available.
Moreover, posit workbenchmark is not handling fork well, so that could be an alternative for posit workbench users.
data.table/man/frollapply.Rd
Line 6 in e26cf1b
note that frollapply is still experimental which gives more freedom to re-work