Scheduler plugin to dump cluster state on close #5660

Activity

1. gjoseph92

   added

   diagnostics

   good second issueClearly described, educational, but less trivial than "good first issue".Clearly described, educational, but less trivial than "good first issue".

   on Jan 13, 2022
2. gjoseph92 commented on Jan 20, 2022

   

   gjoseph92

   on Jan 20, 2022

   CollaboratorAuthor

   More actions

   Actually, a SchedulerPlugin may not work for debugging #5675. The close callback on plugins doesn't run until after the workers have already been removed:

   distributed/distributed/diagnostics/plugin.py

   Lines 55 to 59 in 2a3ee56

   	async def close(self):
   	"""Run when the scheduler closes down
   	This runs at the beginning of the Scheduler shutdown process, but after
   	workers have been asked to shut down gracefully

   We could, however, use a preload, since teardown() gets called before any workers are removed:

   distributed/distributed/scheduler.py

   Lines 4155 to 4159 in 2a3ee56

   	for preload in self.preloads:
   	await preload.teardown()
   	if close_workers:
   	await self.broadcast(msg={"op": "close_gracefully"}, nanny=True)

   A preload will be a bit less ergonomic to install, though.
3. ian-r-rose commented on Jan 20, 2022

   

   ian-r-rose

   on Jan 20, 2022

   Collaborator

   More actions

   Actually, a SchedulerPlugin may not work for debugging #5675. The close callback on plugins doesn't run until after the workers have already been removed:

   We could also add a before_close() plugin hook. I've also run into this issue before in a different context, and adding some more granularity to plugin lifecycles is pretty cheap.
4. jrbourbeau commented on Jan 20, 2022

   

   jrbourbeau

   on Jan 20, 2022

   Member

   More actions

   Yeah, if adding a hook for before workers have been removed would be useful for other problems, then adding a new method seems sensible
5. gjoseph92 commented on Jan 20, 2022

   

   gjoseph92

   on Jan 20, 2022

   CollaboratorAuthor

   More actions

   Sounds good. Looking back through blames, it seems that scheduler plugin behavior has always been this way, but without much explanation of why. Having this option would probably be useful in many cases (especially if you're doing something dangerous and have worker plugins doing some stateful thing, and want to communicate to your worker plugins before shutdown).
6. gjoseph92 commented on Feb 23, 2022

   

   gjoseph92

   on Feb 23, 2022

   CollaboratorAuthor

   More actions

   Hm, I don't know what I was talking about here.

   This runs at the beginning of the Scheduler shutdown process, but after workers have been asked to shut down gracefully

   is only the case if close_workers=True is passed to close. This doesn't happen when invoked from the idle timeout (I don't know why—I feel like close_workers should be true if the cluster is shutting down due to idleness). It seems to only ever be set to True if Client.shutdown() is called and the client doesn't have a cluster object attached.

   So though I'm not sure if close_workers actually should be True for the idle timeout, for the purposes of debugging #5675, we don't a before_close hook or anything like that.
7. fjetter

   assigned

   sjperkins

   on Mar 22, 2022
8. mrocklin

   closed this as completedin #5983on Mar 30, 2022