Skip to content

fix(ci): escalate unclassified paths in TIA instead of skipping e2e - #3511

Merged
Aleksei Sviridkin (lexfrei) merged 10 commits into
mainfrom
ci/select-e2e-fail-safe
Aug 8, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 10 commits into
mainfrom
ci/select-e2e-fail-safe

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Closes #3392. Stacked on top of #3262 — that PR turns the E2E Tests merge gate into a commit status, which is what makes this fail-open path matter: a skipped required job used to pass branch protection either way, but a status posted success with no suite run is an explicit green verdict on untested code.

hack/select-e2e.sh silently selected nothing for a changed path it did not recognise outside packages/. Both lanes read an empty selection as "skip Chainsaw" and then report green. The worked example from the issue is hack/common-envs.mk, which supplies the tag, push and output flags of every image in the tree and matched nothing: full_suite_pattern covered hack/*.sh and hack/*.bats but not hack/*.mk. Walking the tree turned up five more in the same position — pkg/, tools/, go.mod/go.sum, hack/lib/ and hack/buildkitd.toml — plus .github/workflows/e2e-fork.yaml, which #3262 adds while pull-requests.yaml was already escalated.

As the issue argues, a blanket "empty selection means full suite" fallback would be the wrong fix, because it escalates exactly the inert paths the design deliberately skips. Instead every changed path now lands in one of three classes by an explicit rule:

  • full_suite_pattern — shared build inputs and the e2e harness: packages/library/, packages/core/, the Go trees (api/, cmd/, internal/, pkg/), the codegen under tools/, go.mod/go.sum, hack/*.sh|*.bats|*.mk, hack/lib/, hack/buildkitd.toml, hack/e2e-*.yaml, the Makefile, and the workflows that run the suite.
  • inert_config_pattern — paths that cannot affect what e2e exercises: examples/, .github/ (bar the e2e workflows), .claude/, .gemini/, img/, hack/testdata/, the codegen boilerplate header, and the top-level meta files (LICENSE, .gitignore, .pre-commit-config.yaml, .coderabbit.yaml).
  • the PackageSource graph for packages/(apps|system|extra)/, unchanged.

A path matching none of them escalates to the full suite and logs which path forced it, so the next path someone adds fails safe rather than fails open. The two lists are checked in that order, so a specific escalation beats a broad inert directory — .github/ is inert, .github/workflows/e2e-fork.yaml is not. *.md, docs/ and dashboards/ stay ahead of both, so a README.md under packages/core/ does not inherit that tree's escalation.

The second commit removes the || [ -z "$SELECTED_APPS" ] arm from the Chainsaw step in both lanes. Under a step gated on skip != 'true' it could never decide anything the FULL_E2E test had not, while the comment above it advertised "full suite when the selection is empty" in the file whose whole point is that a green status means tests ran.

Deliberate choices worth a second opinion

  • Which workflows escalate. The four that actually invoke the suite (pull-requests, e2e-fork, e2e-tag, nightly), enumerated rather than matched by prefix so an unrelated workflow does not burn a full run. That does mean an e2e-tag.yaml or nightly.yaml edit now costs a full PR suite even though the lane it changes is not the PR lane. Easy to narrow to the two PR lanes if that is the wrong trade.
  • go.mod/go.sum escalate. A dependency bump changes the shipped binaries, so this is the honest classification, but it does make Renovate's Go PRs pay for a full suite.

Verification

  • hack/select-e2e_test.bats: 25 tests green (15 existing, 10 new), covering both sides of the classification.
  • Mutation-checked rather than asserted: reverting the fall-through fails the unclassified-path test; dropping .github/ from the inert list fails the non-e2e-workflow test; swapping the order of the two list checks fails the existing pull-requests.yaml test.
  • The per-entry escalations are belt-and-braces by design: removing hack/*.mk from the pattern still escalates via the fall-through, so that test pins the outcome rather than the pattern entry. Called out so nobody reads it as stronger coverage than it is.
  • make bats-unit-tests: 417 tests, 0 failures. actionlint clean on both workflows.
  • Self-check: this branch's own diff selects the full suite (it touches both e2e workflows and a hack/*.sh).

Downstream repositories

Walked the trigger map in docs/agents/contributing.md against the diff file by file. The only hack/-related trigger is cozystack/ccp, which gates on hack/package.mk and hack/common-envs.mk existing as anchor files and on what make generate does — nothing here is moved or renamed, neither anchor file is touched, and no make target changes behaviour (BATS_UNIT_FILES already globbed hack/*.bats). The rest of the diff is e2e workflows and agent docs, which appear nowhere in the map.

Release note

fix(ci): Test-Impact Analysis now classifies every changed path explicitly and escalates an unrecognised one to the full E2E suite. Previously a path the selector did not recognise (`hack/*.mk`, `pkg/`, `go.mod`, `hack/lib/`, the fork E2E workflow) selected no suites, which both E2E lanes read as "nothing to test" and reported the required `E2E Tests` gate green with no suite run.

Summary by CodeRabbit

  • Bug Fixes

    • Improved end-to-end test selection so unrecognized or high-impact changes trigger the full suite instead of being skipped.
    • Inert-only changes now safely skip testing, while empty selections no longer trigger the full suite.
    • Full-suite testing is triggered explicitly by the full-e2e label or applicable change types.
  • Tests

    • Added coverage for build files, source code, workflows, scripts, documentation, metadata, and unclassified paths.
  • Documentation

    • Clarified end-to-end test selection, escalation rules, skipped file categories, and workflow handling.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The E2E selector now classifies changed paths as full-suite, inert, graph-scoped, or unmatched. Unmatched paths escalate to the full suite. Workflows run Chainsaw only for FULL_E2E=true or selected applications.

Changes

E2E selection and execution

Layer / File(s) Summary
Path classification and escalation
hack/select-e2e.sh, hack/select-e2e_test.bats, docs/agents/e2e-testing.md, .github/workflows/e2e-fork.yaml
The selector expands full-suite patterns, adds explicit inert-path handling, preserves final unterminated lines, escalates unmatched paths, and documents each outcome. Tests cover these classifications.
Workflow suite execution
.github/workflows/e2e-fork.yaml, .github/workflows/pull-requests.yaml
The workflows use FULL_E2E=true as the only full-suite condition. Inert selections skip Chainsaw, while unmatched paths no longer produce silent empty selections.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ChangedPaths
  participant SelectE2E
  participant E2EWorkflow
  participant Chainsaw
  ChangedPaths->>SelectE2E: classify changed paths
  SelectE2E->>E2EWorkflow: return SELECTED_APPS and FULL_E2E
  E2EWorkflow->>Chainsaw: run full suite when FULL_E2E=true
  E2EWorkflow->>Chainsaw: run selected suites when SELECTED_APPS is non-empty
  E2EWorkflow-->>Chainsaw: skip when selection is explicitly inert
Loading

Possibly related PRs

Suggested reviewers: lllamnyp

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: unclassified TIA paths now escalate to the full E2E suite.
Linked Issues check ✅ Passed The changes satisfy issue #3392 by preserving inert-path skips and escalating unclassified paths in both E2E workflows.
Out of Scope Changes check ✅ Passed The workflow, selector, documentation, and test changes directly support the linked issue and PR objectives.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/select-e2e-fail-safe

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/ci Issues or PRs related to CI workflows, GitHub Actions, automation kind/bug Categorizes issue or PR as related to a bug labels Aug 3, 2026
@dosubot dosubot Bot added the area/testing Issues or PRs related to testing (e2e, bats, unit tests) label Aug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — the classification is the right shape, the fall-through escalates in the safe direction, and the tests pin both sides of it. Nothing in the diff blocks. The stack does: this sits on #3262, which is at REQUEST_CHANGES, so it cannot land first.

What I checked, and how

Rather than read the two regexes, I ran the selector over the whole tree, hack/select-e2e.sh against git ls-files (4827 paths), and read the escalation log. Exactly 69 paths reach the new fall-through, and the classification of everything else matches what the PR body claims.

The four workflows enumerated in full_suite_pattern are exactly the output of grep -rl test-chainsaw .github/workflows/ (pull-requests, nightly, e2e-tag, e2e-fork), so the maintenance rule in the comment holds as written today.

The arm removed by the second commit really was unreachable, in both lanes. skip is derived from the same variable the arm tested, so a run that reaches the Chainsaw step with skip != 'true' either carries a non-empty selection or had select skipped outright by the full-e2e label, which the surviving FULL_E2E test covers. Deleting it is a strict simplification.

Ordering is safe: inert_config_pattern contains no packages/ prefix, so moving the inert check ahead of the graph lookup cannot shadow a real app path, and the new continue after the graph branch is what keeps a matched app path out of the fall-through. Test 25 pins the mixed inert-plus-real case. 25/25 green locally under the repo harness.

Non-blocking follow-ups

  1. 66 of the 69 fall-through paths already exist and are inert; each now buys a full suite. test/check-readiness/** (51 files) is the golden-file Go test for the readiness CLI, run by make test-check-readiness, a unit test whose subject lives in cmd/, which escalates on its own. packages/tests/cozy-lib-tests/** (12) is a helm-unittest chart. hack/e2e-apps/monitoring-oidc-{system,customconfig}.bats (2) are the last two unported files from the pre-Chainsaw suite; nothing in any Makefile or workflow runs them, and the surviving references are # Chainsaw port of hack/e2e-apps/<x>.bats comments. packages/{apps,system,extra}/Makefile (3) are arguably a fair escalation and hack/e2e-chainsaw/.gitignore (1) is noise. The PR body's walk looked for paths missing from the escalation list, not for paths missing from the inert one, so the two trees that are actively edited, the helm-unittest chart in particular, pay a full suite per PR. Three entries in inert_config_pattern closes it, which is what the script's own comment says to do.
  2. .github/ is the only blanket-inert directory that also holds live e2e inputs. Enumerating the four suite-running workflows is the right call on cost; the alternative escalates every codeql.yml tweak. What is not covered is the fifth one: a new workflow that runs the suite lands under an inert prefix and is silently not escalated, which is the same shape as #3392 with a smaller blast radius. Cheaper than changing the classification is pinning it: a unit test asserting that the workflows matching test-chainsaw are exactly the set in full_suite_pattern turns a future omission into a red test instead of a quiet green gate.
  3. #3330 is not closed here, and it is the mirror image of what is. This diff makes an empty selection mean "verified inert" instead of "TIA had no idea". The sibling case survives untouched: a non-empty selection that should have been the full suite. Reproduced on this head: packages/system/cozystack-basics/templates/ingress-hostname-policy.yaml alone selects 20 suites; the same path plus hack/e2e-chainsaw/postgres/chainsaw-test.yaml selects postgres and nothing else. A packages/system/ source with no *-application descendants escalates only by leaving final_apps empty (hack/select-e2e.sh:224-230), and :215 unions the per-suite edits into final before that emptiness test, so one suite edit anywhere in the diff swallows the escalation silently. Nothing here introduced or worsened it, the tail is byte-identical to the base, but it is the same defect class as the one this PR fixes, and this is the natural place to turn that escalation into a flag unioned with the selection rather than an inference from an empty list, pinned by the two-path case above.
  4. On the two trades the PR body flags, both read right as written. go.mod/go.sum escalating is the honest classification and a dependency bump is precisely the PR you want the full suite on. e2e-tag and nightly escalating on a PR is a handful of PRs a year, and editing a lane's suite invocation without running the suite once is how those lanes rot.

Checked and dropped

The new cases use trap 'rm -rf "$tmp"' EXIT, which the tree restricts for BATS tests. Dropped: the ban in the e2e testing doc is scoped to the cluster suite and explicitly permits a self-contained temp-dir trap, all 15 pre-existing cases in this same file already use the identical line, and the unit harness runs each case in a subshell, so the trap is contained per test rather than overwriting its predecessor.

Also dropped: the independent tooling pass returned no findings on this diff.

`select-e2e.sh` silently selected nothing for a changed path it did not
recognise outside `packages/`. Both e2e lanes read an empty selection as
"skip Chainsaw" and then post the required "E2E Tests" status green, so
an unrecognised path produced a green merge gate with no suite run. The
worked example is `hack/common-envs.mk`, which supplies the tag, push
and output flags of every image in the tree and matched nothing:
`full_suite_pattern` covered `hack/*.sh` and `hack/*.bats` but not
`hack/*.mk`. `pkg/`, `tools/`, `go.mod`, `hack/lib/` and
`hack/buildkitd.toml` were in the same position, as was
`.github/workflows/e2e-fork.yaml` while `pull-requests.yaml` was
already escalated.

A blanket "empty means full suite" fallback would be the wrong fix: it
escalates the inert paths the design deliberately skips. Instead every
path now lands in one of three classes by an explicit rule — the
full-suite list for shared build inputs and the e2e harness, an inert
list for paths that cannot affect what e2e exercises, and the
PackageSource graph for `packages/(apps|system|extra)/`. A path
matching none of them escalates and logs why, so the next path someone
adds fails safe instead of failing open.

The two lists are checked in that order, so a specific escalation beats
a broad inert directory: `.github/` is inert, the workflows that run
the suite are not. `*.md` and `docs/`/`dashboards/` stay ahead of both,
so a README under `packages/core/` does not inherit that tree's
escalation.

Ten tests cover both sides of the classification. Mutation-checked:
reverting the fall-through fails the unclassified-path test, dropping
`.github/` from the inert list fails the non-e2e-workflow test, and
swapping the two list checks fails the existing pull-requests.yaml
test. Note the per-entry escalations are belt-and-braces — removing
`hack/*.mk` from the pattern still escalates via the fall-through,
which is the point of having it.

Closes #3392

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
Both lanes ran `if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]`
under a step gated on `steps.select.outputs.skip != 'true'`. When
`select` runs, an empty result sets `skip=true` and the step never
starts, so the only way to arrive with no selection is the `full-e2e`
label — which sets `FULL_E2E=true`. The second test could never decide
anything the first had not, while the comment above it advertised
"full suite when the selection is empty" in the file whose whole point
is that a green status means tests ran.

Testing `FULL_E2E` alone keeps the real invariant visible. The comments
now say what an empty selection means after the preceding commit: every
changed path was explicitly classified inert, not that the selector had
no idea.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/select-e2e_test.bats`:
- Around line 181-188: Remove the test-level EXIT trap from each affected test
in hack/select-e2e_test.bats at lines 181-188, 190-197, 199-206, 208-215,
217-224, 226-233, 235-242, 244-256, 258-267, and 269-276. In every test, add rm
-rf "$tmp" after the final assertion so cleanup occurs only on success while
failed tests preserve their scratch directories and Bats retains its TAP
bookkeeping.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 94c02aa9-cc5f-4ba8-956a-a524636c452c

📥 Commits

Reviewing files that changed from the base of the PR and between 7dca642 and ee4a482.

📒 Files selected for processing (5)
  • .github/workflows/e2e-fork.yaml
  • .github/workflows/pull-requests.yaml
  • docs/agents/e2e-testing.md
  • hack/select-e2e.sh
  • hack/select-e2e_test.bats

Comment thread hack/select-e2e_test.bats
The privileged resolve step explains why the TIA inputs come from the
base repo, and closed with a caveat about the selector: that it escalates
only for an unrecognised path inside packages/ and silently selects
nothing for a path it does not recognise at all, naming hack/*.mk as the
example. That was true of the selector this branch replaces.

Measured against the selector as it now stands, hack/common-envs.mk
escalates to the full suite, and so does the fork workflow file itself.
An empty selection no longer means the path went unrecognised; it means
the path was classified as unable to affect what e2e exercises. The
difference is the whole point of the change, and the caveat now
contradicts it in the one comment a reader consults before touching the
privileged lane.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
The classifier this branch introduces is only total if every line
reaches the loop that applies it. POSIX read assigns the final line and
then returns non-zero when the input does not end in a newline, so a
plain `while read` drops it. The fall-through that escalates an
unrecognised path lives inside the loop body, so the dropped line is
never classified at all and the selection comes back empty. Both lanes
read empty as "skip Chainsaw" and post the required status green.

The shape that matters is a list whose only unclassified path is last:
`docs/x.md` followed by an unrecognised path with no terminator selects
nothing, where the same list with a terminator escalates to all 21
suites. The safety net this branch is built around is bypassed by
exactly the input it exists to catch.

No caller triggers it today, because `git diff --name-only` terminates
its output and the fork lane appends a newline explicitly. That makes it
caller discipline documented in one of two callers, which the next
caller will not read. Guard it where the assumption lives.

Both new cases are red without the guard and the twenty-five existing
ones stay green, so they pin this behaviour rather than the loop's
general shape. They clean up inline instead of through an EXIT trap:
that trap replaces the one the bats binary installs for its own
bookkeeping, and a test failing under it can print no TAP line at all,
which is the opposite of what a regression pin is for.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Four places on this branch describe an empty selection as "classified
inert", against a list of three outcomes that every path is said to land
in. There is a fourth: an edit under examples/backups/<app>/ where the
app has no Chainsaw suite selects nothing, decided inline where backup
examples are matched rather than through the inert list. The rule is
deliberate and explained where it lives; it just is not one of the three
the summary enumerates.

The property the two lanes actually depend on is weaker and true: an
empty selection means some rule decided the change cannot affect a
suite, never that nothing looked at the path. State that instead, in the
header that defines the classes and in the privileged lane's comment
that consumes the result.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Four places said an empty selection means the inert pattern matched, and
sent the reader to `inert_config_pattern` to see which paths those are.
Documentation, dashboards and `*.md` are dropped before that pattern is
ever consulted, and a backups example whose app has no suite is dropped
where backup examples are matched. A reader checking the pattern for any
of them finds nothing and has to reconstruct the order by reading the
script. Name the rules that produce an empty selection instead, and keep
the claim to the one the lanes rely on: empty is a decision some rule
reached, not a path nothing looked at.

The same bullet claimed every path is classified. One is not. The inert
pattern makes `.github/` inert as a whole directory and the workflows
that run the suite are exempted from it by name, so a workflow added
under `.github/workflows/` reads as inert and the fall-through never
sees it. Today's enumeration is correct and I checked it against `rg -l
test-chainsaw`, so this is not a live hole, but the sentence saying it
cannot exist is what makes the reminder next to that list look
decorative. Record it as the guard it is.

Separately, the fork lane's rationale for appending a newline described
a mechanism this branch removes: the reader now guards its own loop, so
the terminator is no longer what keeps the last path alive. Say that it
is now the text-file contract and a second line of defence across two
files, which is a reason to keep it and a different reason than the one
written there.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
The ten tests added with the classifier each installed `trap 'rm -rf
"$tmp"' EXIT`, taking this file from fifteen such traps to twenty-five.
`hack/cozyreport.bats` freezes that count per unconverted file, so
`make unit-tests` failed, and that runs in the required checks job.

The count is the visible half. The reason it is frozen is that an EXIT
trap replaces the one the bats binary installs for its own bookkeeping,
and a test failing under it prints no TAP line at all. Reverting the
step-6 escalation in a scratch copy made the flagship pin for this whole
change, the one asserting an unclassified path escalates, vanish from
the output: the run reported executing twenty-six of twenty-seven tests
and never named the missing one. After this change the same mutation
prints `not ok 21` with its name, file and line.

Clean up inline before the assertion instead, which is what the two
tests added alongside them already did. Behaviour is unchanged, all
twenty-seven still pass, and the frozen count returns to fifteen rather
than being re-frozen higher, since re-freezing would bank ten fresh
instances of the thing the freeze exists to stop.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
The previous pass corrected the same explanation in the selector header
and in the in-tree lane and missed this one, which sits in the fork
lane's selection step. It still said an empty selection means "verified
inert" and pointed only at the inert pattern, and it lacked the caveat
the other two copies gained: that a workflow added under
`.github/workflows/` reads as inert until it is named in the escalation
list.

Three copies of one explanation is the underlying problem; leaving one
of them a version behind is the way that problem shows up. A correction
is not finished while a copy of the old claim is still alive.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Two defects introduced by the previous two commits.

The in-tree lane's comment said `skip=true` cannot mean "a workflow
nobody classified", and then explained, after a colon, exactly how it
can: `.github/` is inert as a directory and the e2e workflows are
exempted from it by name, so a new one reads as inert. The clause meant
to warn a reader instead told them the one unsafe case is safe, in the
copy consulted when deciding whether to trust a green gate. The other
three copies of the caveat state it correctly.

The scratch-directory cleanup landed before the last assertion in every
test added here. That passes the EXIT-trap guard and inverts the reason
the guard exists: both runners set `-e`, so cleanup placed last is
unreachable on failure and the fixture survives for inspection, which
`docs/agents/e2e-testing.md` §3 states outright and the already-converted
`hack/select-install_test.bats` follows. Placed early, a failing test
deletes the evidence of its own failure, and one test had the call
between two assertions, so which of them failed decided whether the
directory survived.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
Three of the four workflows on the escalation list cannot be exercised
by the run they trigger. `e2e-fork` runs on `workflow_run`, which always
takes the default-branch copy; `e2e-tag` runs on `workflow_call` from a
tag push; `nightly` runs on a schedule. Only `pull-requests.yaml` is
executed from the pull request's own head, so only there does escalating
test the edit rather than the tree around it.

That argument existed in the reference doc and was removed by this
branch rather than answered, and the list it argued against is what
replaced it. Measured over six months of main, 26 commits touch one of
the three with no other escalating path; each now pays a full Chainsaw
run it used to skip, on a suite whose flake rate is the reason people
avoid running it for nothing.

Keeping them is defensible: the rule stays one idea rather than two, and
dropping them leaves those files inert, which reads as an oversight
rather than a decision. What was not defensible was paying it silently.
The trade now sits next to the pattern that makes it, with the number,
so the next person weighing it starts from measurement instead of
rediscovering the mechanism. The reference doc points there rather than
carrying its own copy.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Rebased onto main after #3262, the diff is down to five files, and the fail-safe now holds at the root: the reader-side read || [ -n "$file" ] guard means an unterminated last line cannot skip the fall-through escalation this PR exists for, with dedicated regression cases for exactly that input. Two full review passes at this head found no blocking defect.

The measured half-year replay is worth stating plainly: about 8% of commits start paying a full suite, and nearly all of that cost is closed fail-opens. Changes under pkg/ (84 commits) and hack/e2e-apps/ (119) previously selected nothing and could green the required gate with no suite run at all. The change is also monotone against the whole replay: nothing that escalated before escalates less now.

The escalation cost for the three workflows that do not execute from the PR head is recorded next to the pattern with its measured price and the conditions for revisiting, so the trade-off is explicit instead of silent.

Transition note: paths not yet classified (for example test/ and packages/tests/) now escalate to the full suite by design and print a classify-me line; the first commit touching such a path pays one full run and the fix is a one-line classification.

Reviewed with Opus assistance.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 68e1e41 into main Aug 8, 2026
42 of 43 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the ci/select-e2e-fail-safe branch August 8, 2026 03:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci Issues or PRs related to CI workflows, GitHub Actions, automation area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

e2e: an empty TIA selection concludes "E2E Tests" green with no suite run

2 participants