Label events cancel the in-flight PR run, and the surviving no-op leaves SKIPPED checks that satisfy branch protection #3560

Summary

A label applied to an open pull request cancels the real CI run for that pull request and replaces it with a run that does nothing. The required contexts end up SKIPPED rather than missing, and branch protection treats SKIPPED as satisfied, so the pull request looks mergeable while no build, no unit test and no e2e ever ran on that head.

Mechanism

.github/workflows/pull-requests.yaml subscribes to labeled alongside the real triggers (types: [opened, synchronize, reopened, labeled], line 11) and declares a per-pull-request concurrency group with cancel-in-progress: true (lines 14 to 16). Every label event therefore enters the same group as the run from the push and cancels it.

Observed on #3552, head 80a20904. The pull request opened at 10:26:48Z and the real run 30997452148 started at 10:26:53Z. Three labels were applied at 10:27:17Z, which is three separate labeled events, and they produced three runs at 10:27:19Z: 30997480009 and 30997480053 were cancelled by their successors, and 30997480441 survived as a no-op. The same three events also took down Automatic Backport (30997480404 and 30997480554 cancelled, 30997480755 skipped). A fourth label at 10:30:28Z produced no run at all, because it was applied with GITHUB_TOKEN and events made with that token do not start workflows. Labels applied by a third-party GitHub App do.

The guards inside the workflow do not help, because they run too late. plan carries if: github.event.action != 'labeled' || github.event.label.name == 'full-e2e' (line 31) and every other job hangs off plan, so a label run correctly does no work. Concurrency is evaluated before any job condition, so by the time the guard skips the jobs the real run is already cancelled. The guard prevents wasted work, not the cancellation.

The result is worse than a cancelled run. gh pr checks --json on #3552 reports Plan build, Unit & controller tests, Build, Build Talos, Finalize, E2E Tests, prepare and backport as SKIPPED, and branch protection counts a skipped required context as satisfied. The plain text output of gh pr checks renders the same state as CANCELLED, which reads as a failure, so the JSON output is the one to trust. Issue #3257 records the same consequence reached by a different route, fork pull requests, so that half of this is already known.

This has been worked around by hand at least once. Commit 007d0b1a2 on main is an empty commit, zero files changed and tree identical to its parent, whose only purpose was to re-trigger CI, and its message describes this exact failure including "cancel-in-progress killed the in-flight api-review-gate run that was carrying the real signal". There is no automation behind it: labeled is still in types, cancel-in-progress is still unconditional, and no workflow reacts to a skipped or cancelled pull request run. An affected pull request stays that way until someone pushes.

Possible directions

I have not picked one of these.

1. Drop labeled from types and make the manual full-e2e opt-in actionable another way.
2. Keep the trigger and make the cancellation conditional, for example by putting the event action into the concurrency group key, or by setting cancel-in-progress to false for label events.
3. Keep both and stop the no-op run from publishing SKIPPED contexts over a live run, so branch protection sees the real result or nothing.