Skip to content

Verification runs ~25 min of local gates on a head whose CI test job already failed, holding the single CPU admission slot #4531

Open

Description

@eltmon

Problem

Verification keeps running every local gate (install, typecheck, frontend-typecheck, lint, build, server-bundle: about 25 minutes) on a head whose CI test job has already failed. Only after all of that does it fail test with "The CI test job already failed on this head". Heavy gates hold the machine-wide CPU admission slot one at a time (src/lib/cloister/quality-gate-admission.ts, PAN-4311). A verification run that is certain to fail therefore blocks every other issue's verification, and the agent's own isolation test runs, which enter the same queue.

Observed 2026-10-04 on PAN-4259 (times UTC):

  • ~04:10: a verification worker starts on head 2665f45c.
  • 04:14:56: the CI test job fails on 2665f45c (test-shard (4/4), .overdeck/verification/2026-10-04T04-14-56-585Z-2665f45c.json).
  • 04:37: the worker still holds the admission slot running lint for that head (~/.overdeck/verification-workers/admission/owner.json: {"issueId":"PAN-4259","gateName":"lint"}). Seven waiters are queued behind it, including the PAN-4259 agent's own vitest run tests/unit/lib/access-tokens.test.ts isolation run, which it needs in order to fix the CI failure.

Root cause

src/lib/cloister/verification-runner.ts ~lines 578-590:

const ciTestRed = testsMode === 'ci'
  && !localGateResults.some((r) => !r.passed && r.required !== false)
  && readLatestCiTestResult(workspacePath, headShort) === 'failed';

The CI verdict for the head is read only after every local gate has run. Nothing checks it before a gate starts, or between gates.

Fix

  1. In ci tests mode, read readLatestCiTestResult(workspacePath, headShort) before the first local gate and again before each later gate requests CPU admission. If it is failed, stop at once: record the same failed test gate result and output as today, mark the remaining local gates as not run, and release the admission slot.
  2. Keep today's message and attempt accounting: a re-request on the same head still counts against the attempt budget (PAN-3965 behavior).
  3. Use the head actually being verified. Coordinate with PAN-4527, which fixes the launch-time headShort mismatch, and do not reintroduce it.

Acceptance criteria

  • Given CI recorded failed for the verified head before verification starts, the runner runs no local gate, never requests CPU admission, and returns failed at test with the existing message (runner unit test).
  • Given CI flips to failed while a local gate is running, the next gate does not request admission, and the run ends failed at test (unit test with fake timers for any polling).
  • Given CI is passed or unknown, behavior is unchanged (existing tests stay green).
  • docs/PIPELINE-GATES.md (verification gate, ci tests mode) states that a known-red CI head short-circuits local gates.

Routing

This is review/verification machinery (TENET-10). Route it through a supervised pan handoff, together with or after PAN-4527.

Activity

  1. self-assigned this
    on Oct 4, 2026
  2. added
    bugSomething isn't working
    substrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline
    needs-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)
    on Oct 4, 2026
  3. eltmon commented on Oct 4, 2026

    @eltmon
    OwnerAuthor

    Second instance of the same mechanism, 2026-10-04 05:34 UTC: the runner checks whether the PR merged only before the gates and after all of them (skipMergedVerification at src/lib/cloister/verification-runner.ts lines 304, 364, 420 and 592), never between gates. PAN-4498's duplicate verification worker (pid 3018247, started 05:06 by a deacon-lite re-request) kept queueing gates after PR #4521 merged at 05:30. PAN-4508's duplicate (pid 3032771) held the single CPU admission slot on lint while PR #4524 was already APPROVED.

    Extend the fix: the per-gate pre-admission check should stop the run when any decided condition holds: CI test job red for the verified head, PR merged (skipMergedVerification), or the head is no longer the PR head. Add an acceptance criterion: given the PR merges while a gate is running, the next gate does not request admission and the run ends with the existing merged-skip outcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingneeds-handoffPipeline-runtime refactor: supervised pan handoff only, never autonomous pickup (TENET-10)performancesubstrate-improvementNon-blocker Panopticon infrastructure improvement — file, don't patch inline

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions