Repository navigation
Verification runs ~25 min of local gates on a head whose CI test job already failed, holding the single CPU admission slot #4531
Description
Problem
Verification keeps running every local gate (install, typecheck, frontend-typecheck, lint, build, server-bundle: about 25 minutes) on a head whose CI test job has already failed. Only after all of that does it fail test with "The CI test job already failed on this head". Heavy gates hold the machine-wide CPU admission slot one at a time (src/lib/cloister/quality-gate-admission.ts, PAN-4311). A verification run that is certain to fail therefore blocks every other issue's verification, and the agent's own isolation test runs, which enter the same queue.
Observed 2026-10-04 on PAN-4259 (times UTC):
- ~04:10: a verification worker starts on head
2665f45c. - 04:14:56: the CI test job fails on
2665f45c(test-shard (4/4),.overdeck/verification/2026-10-04T04-14-56-585Z-2665f45c.json). - 04:37: the worker still holds the admission slot running
lintfor that head (~/.overdeck/verification-workers/admission/owner.json:{"issueId":"PAN-4259","gateName":"lint"}). Seven waiters are queued behind it, including the PAN-4259 agent's ownvitest run tests/unit/lib/access-tokens.test.tsisolation run, which it needs in order to fix the CI failure.
Root cause
src/lib/cloister/verification-runner.ts ~lines 578-590:
const ciTestRed = testsMode === 'ci'
&& !localGateResults.some((r) => !r.passed && r.required !== false)
&& readLatestCiTestResult(workspacePath, headShort) === 'failed';The CI verdict for the head is read only after every local gate has run. Nothing checks it before a gate starts, or between gates.
Fix
- In
citests mode, readreadLatestCiTestResult(workspacePath, headShort)before the first local gate and again before each later gate requests CPU admission. If it isfailed, stop at once: record the same failedtestgate result and output as today, mark the remaining local gates as not run, and release the admission slot. - Keep today's message and attempt accounting: a re-request on the same head still counts against the attempt budget (PAN-3965 behavior).
- Use the head actually being verified. Coordinate with PAN-4527, which fixes the launch-time
headShortmismatch, and do not reintroduce it.
Acceptance criteria
- Given CI recorded
failedfor the verified head before verification starts, the runner runs no local gate, never requests CPU admission, and returnsfailedattestwith the existing message (runner unit test). - Given CI flips to
failedwhile a local gate is running, the next gate does not request admission, and the run endsfailedattest(unit test with fake timers for any polling). - Given CI is
passedor unknown, behavior is unchanged (existing tests stay green). -
docs/PIPELINE-GATES.md(verification gate,citests mode) states that a known-red CI head short-circuits local gates.
Routing
This is review/verification machinery (TENET-10). Route it through a supervised pan handoff, together with or after PAN-4527.
Activity
eltmon commentedon Oct 4, 2026
Second instance of the same mechanism, 2026-10-04 05:34 UTC: the runner checks whether the PR merged only before the gates and after all of them (
skipMergedVerificationatsrc/lib/cloister/verification-runner.tslines 304, 364, 420 and 592), never between gates. PAN-4498's duplicate verification worker (pid 3018247, started 05:06 by a deacon-lite re-request) kept queueing gates after PR #4521 merged at 05:30. PAN-4508's duplicate (pid 3032771) held the single CPU admission slot onlintwhile PR #4524 was already APPROVED.Extend the fix: the per-gate pre-admission check should stop the run when any decided condition holds: CI test job red for the verified head, PR merged (
skipMergedVerification), or the head is no longer the PR head. Add an acceptance criterion: given the PR merges while a gate is running, the next gate does not request admission and the run ends with the existing merged-skip outcome.