Repository navigation
Dashboard restart during verification loses the review dispatch; re-request discards the finished result and re-runs all gates #4532
Description
Problem
A dashboard restart while a verification worker is running loses that verification's review dispatch. The worker survives the restart and writes a passed result, but the code that waits for the result and then dispatches the reviewer lived in the old dashboard process. When the deacon later re-requests review, the supervisor ignores the finished result and starts a fresh full verification. That costs another ~25-45 minutes of serialized gate time (the single CPU admission slot, PAN-4311) for a head that already passed.
Observed on PAN-4515, 2026-10-04 (times UTC):
- 03:52:21: verification starts; worker 1870433 (
~/.overdeck/verification-workers/pan-4515/worker-1791085939727-1693887.log). - 04:21: the dashboard restarts (
pan reload+pan restart approve). - 04:36:43: the worker writes
result-1791085939727-1693887.json={"outcome":"passed"}, and the journal recordsverification.passed. No reviewer is dispatched. - 04:41:52: deacon-lite re-requests:
Agent re-review request (1/25): a dashboard restart during verification left the review undispatched. - 04:41:54: a new worker (
worker-1791088912886-10483.log) starts a complete re-run.
PAN-4257, PAN-4498 and PAN-4508 had workers running across the same restart and are exposed in the same way.
Root cause
src/lib/cloister/verification-worker-supervisor.ts, runSupervisedVerificationInternal, line 174:
if (existing && existing.workspacePath === workspacePath && isProcessAlive(existing.pid) && !existsSync(existing.resultPath)) {A worker is joined only while it is alive and has no result yet. A worker that has already finished, and whose result nobody consumed because its waiter died with the old dashboard process, falls through to a fresh worker. Nothing on dashboard boot re-attaches waiters to workers that survived the restart (activeSupervisorRuns is in-memory only).
Fix
- Consume an unconsumed result. In
runSupervisedVerificationInternal, when the existing worker state has a result file, the result is unconsumed, and the worker's verified head equals the current workspace HEAD (or the HEAD differs only by the worker's own target-branch sync merge, as PAN-4527 defines it), return that result without starting a new worker. Record consumption in the worker state so one result is never consumed twice. - Re-attach on boot. On dashboard boot, scan
~/.overdeck/verification-workers/*/state.json. For each live worker, or each finished worker with an unconsumed result, resume the same post-verification continuation thatrequest-reviewruns (push the verified branch, dispatch review), as if the original request were still waiting. - Coordinate with PAN-4527 (head matching) and PAN-4531 (short-circuit on red CI). The head-equality rule must be the same in all three.
Acceptance criteria
- Given a worker state with a finished
passedresult for the current HEAD and no consumer, when review is re-requested, then no new worker starts and review dispatches (supervisor unit test). - Given the same, but the agent has pushed a new commit since, then a fresh worker starts (no regression with PAN-4527).
- Given a live worker at dashboard boot, when it later writes
passed, then the review dispatches without any re-request (boot-adoption test; use fake timers for polling). - A result is consumed at most once (unit test).
-
docs/PIPELINE-GATES.mdstates that verification survives a dashboard restart and that results are adopted, not re-run.
Routing
This is review/verification machinery (TENET-10): supervised pan handoff, not pan start. Bundle it with PAN-4527 and PAN-4531.
Activity