Repository navigation
Interruption problems in frollapply - #7428
Conversation
Instead of calling invokeRestart("abort") in the interrupt handler,
return from it. This continues the dispatch of the interrupt and lets an
outer handler catch it:
tryCatch(
frollapply(1:1e6, 1, \(.) { Sys.sleep(ret <- sum(.)); ret}),
interrupt = \(e) 'interrupted'
)
^C[1] "interrupted"
With invokeRestart("abort"), the interrupt cannot be handled further.
While handling an interrupt, ask mccollect() to wait for the child process to exit (with a warning) in order to avoid producing zombies. Otherwise a process that is too slow to react to SIGTERM will remain a zombie until the parent process exits.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7428 +/- ##
==========================================
- Coverage 99.06% 99.06% -0.01%
==========================================
Files 86 86
Lines 16613 16612 -1
==========================================
- Hits 16458 16457 -1
Misses 155 155 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Could the zombie processes also be the reason that the codecov github job gets stuck in #7288? |
Generated via commit 22de4e8 Download link for the artifact containing the test results: ↓ atime-results.zip
|
|
Apparently there are some other packages having the same problem |
|
Can't say no with 100% certainty, but I don't see how |
|
News should not be needed as this is change to code added in dev. It could eventually explain current interrupt behavior but I think better to have it in manual than news. |
|
Need to double check whether interrupting the first |
Since the PIDs of our worker processes could have been reused, first test them using waitid(NOWAIT) to make sure they are still ours.
|
It turns out this is possible: library(parallel)
jobs <- list(
mcparallel(TRUE, 'completes immediately'),
mcparallel(repeat Sys.sleep(1), 'hangs')
)
mccollect(jobs)This hangs, so interrupt R. Since the first job is complete, its process has terminated, the parent process has while (system(paste('test $$ -eq', jobs[[1]]$pid)) != 0) {} # the loop eventually completes!This shows that In theory, we could use Our salvation is the POSIX siginfo_t info;
bool can_terminate = waitid(P_PID, pid, &info, WCONTINUED | WEXITED | WNOHANG | WNOWAIT | WSTOPPED) == 0; // would be -1 if pid isn't oursNow we can see: options(datatable.verbose = TRUE)
trace(tools::pskill, quote(message('terminating: ', toString(pid))), print = FALSE)
setDTthreads(4)
writeLines('foo', 'foo')
frollapply(1:1e4, 1, \(.) { if (suppressWarnings(file.remove('foo'))) Sys.sleep(10); 42 } ) -> val
# frollapply running on 4 CPU threads
# ^C
# terminating: 11111
# (followed by harmless warnings)The PIDs previously belonging to the three processes that were fast to return the job result were not erroneously terminated. Sorry this took so long. |
And fix a silly typo.
5 participants
There were two problems with the way
frollapplyhandled interruptions:invokeRestart("abort")in the interrupt handler prevented any further error or calling handlers from running (but not exit handlers liketryCatch(finally=...)oron.exit(); those kept working). I think that it's better to return from the calling handler, letting R continue the dispatch to other possible handlers:frollapplysometimes left zombie processes. The only way to reap them was to unload theparallelparallel (or quit R). I think that the parent process was too quick to callmccollect()after sending theSIGINT, so the child process did not have enough time to process theSIGINT, quit, and be ready to be collected bywaitpid(). I think that the zombie processes can be avoided by forcingmccollect()to wait, but that opens opportunities for further problems.mccollect()ing a terminated process results in a warning: do we needsuppressWarnings()? If the child processes are really hung (e.g. due to the rolling function causing a deadlock), a second interrupt seems to interruptmccollectdespite thesuspendInterrupts()(???) and unblock the parent process: