Repository navigation
fix(ci): escalate unclassified paths in TIA instead of skipping e2e - #3511
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe E2E selector now classifies changed paths as full-suite, inert, graph-scoped, or unmatched. Unmatched paths escalate to the full suite. Workflows run Chainsaw only for ChangesE2E selection and execution
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ChangedPaths
participant SelectE2E
participant E2EWorkflow
participant Chainsaw
ChangedPaths->>SelectE2E: classify changed paths
SelectE2E->>E2EWorkflow: return SELECTED_APPS and FULL_E2E
E2EWorkflow->>Chainsaw: run full suite when FULL_E2E=true
E2EWorkflow->>Chainsaw: run selected suites when SELECTED_APPS is non-empty
E2EWorkflow-->>Chainsaw: skip when selection is explicitly inert
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
LGTM — the classification is the right shape, the fall-through escalates in the safe direction, and the tests pin both sides of it. Nothing in the diff blocks. The stack does: this sits on #3262, which is at REQUEST_CHANGES, so it cannot land first.
What I checked, and how
Rather than read the two regexes, I ran the selector over the whole tree, hack/select-e2e.sh against git ls-files (4827 paths), and read the escalation log. Exactly 69 paths reach the new fall-through, and the classification of everything else matches what the PR body claims.
The four workflows enumerated in full_suite_pattern are exactly the output of grep -rl test-chainsaw .github/workflows/ (pull-requests, nightly, e2e-tag, e2e-fork), so the maintenance rule in the comment holds as written today.
The arm removed by the second commit really was unreachable, in both lanes. skip is derived from the same variable the arm tested, so a run that reaches the Chainsaw step with skip != 'true' either carries a non-empty selection or had select skipped outright by the full-e2e label, which the surviving FULL_E2E test covers. Deleting it is a strict simplification.
Ordering is safe: inert_config_pattern contains no packages/ prefix, so moving the inert check ahead of the graph lookup cannot shadow a real app path, and the new continue after the graph branch is what keeps a matched app path out of the fall-through. Test 25 pins the mixed inert-plus-real case. 25/25 green locally under the repo harness.
Non-blocking follow-ups
- 66 of the 69 fall-through paths already exist and are inert; each now buys a full suite.
test/check-readiness/**(51 files) is the golden-file Go test for the readiness CLI, run bymake test-check-readiness, a unit test whose subject lives incmd/, which escalates on its own.packages/tests/cozy-lib-tests/**(12) is a helm-unittest chart.hack/e2e-apps/monitoring-oidc-{system,customconfig}.bats(2) are the last two unported files from the pre-Chainsaw suite; nothing in any Makefile or workflow runs them, and the surviving references are# Chainsaw port of hack/e2e-apps/<x>.batscomments.packages/{apps,system,extra}/Makefile(3) are arguably a fair escalation andhack/e2e-chainsaw/.gitignore(1) is noise. The PR body's walk looked for paths missing from the escalation list, not for paths missing from the inert one, so the two trees that are actively edited, the helm-unittest chart in particular, pay a full suite per PR. Three entries ininert_config_patterncloses it, which is what the script's own comment says to do. .github/is the only blanket-inert directory that also holds live e2e inputs. Enumerating the four suite-running workflows is the right call on cost; the alternative escalates everycodeql.ymltweak. What is not covered is the fifth one: a new workflow that runs the suite lands under an inert prefix and is silently not escalated, which is the same shape as #3392 with a smaller blast radius. Cheaper than changing the classification is pinning it: a unit test asserting that the workflows matchingtest-chainsaware exactly the set infull_suite_patternturns a future omission into a red test instead of a quiet green gate.- #3330 is not closed here, and it is the mirror image of what is. This diff makes an empty selection mean "verified inert" instead of "TIA had no idea". The sibling case survives untouched: a non-empty selection that should have been the full suite. Reproduced on this head:
packages/system/cozystack-basics/templates/ingress-hostname-policy.yamlalone selects 20 suites; the same path plushack/e2e-chainsaw/postgres/chainsaw-test.yamlselectspostgresand nothing else. Apackages/system/source with no*-applicationdescendants escalates only by leavingfinal_appsempty (hack/select-e2e.sh:224-230), and:215unions the per-suite edits intofinalbefore that emptiness test, so one suite edit anywhere in the diff swallows the escalation silently. Nothing here introduced or worsened it, the tail is byte-identical to the base, but it is the same defect class as the one this PR fixes, and this is the natural place to turn that escalation into a flag unioned with the selection rather than an inference from an empty list, pinned by the two-path case above. - On the two trades the PR body flags, both read right as written.
go.mod/go.sumescalating is the honest classification and a dependency bump is precisely the PR you want the full suite on.e2e-tagandnightlyescalating on a PR is a handful of PRs a year, and editing a lane's suite invocation without running the suite once is how those lanes rot.
Checked and dropped
The new cases use trap 'rm -rf "$tmp"' EXIT, which the tree restricts for BATS tests. Dropped: the ban in the e2e testing doc is scoped to the cluster suite and explicitly permits a self-contained temp-dir trap, all 15 pre-existing cases in this same file already use the identical line, and the unit harness runs each case in a subshell, so the trap is contained per test rather than overwriting its predecessor.
Also dropped: the independent tooling pass returned no findings on this diff.
`select-e2e.sh` silently selected nothing for a changed path it did not recognise outside `packages/`. Both e2e lanes read an empty selection as "skip Chainsaw" and then post the required "E2E Tests" status green, so an unrecognised path produced a green merge gate with no suite run. The worked example is `hack/common-envs.mk`, which supplies the tag, push and output flags of every image in the tree and matched nothing: `full_suite_pattern` covered `hack/*.sh` and `hack/*.bats` but not `hack/*.mk`. `pkg/`, `tools/`, `go.mod`, `hack/lib/` and `hack/buildkitd.toml` were in the same position, as was `.github/workflows/e2e-fork.yaml` while `pull-requests.yaml` was already escalated. A blanket "empty means full suite" fallback would be the wrong fix: it escalates the inert paths the design deliberately skips. Instead every path now lands in one of three classes by an explicit rule — the full-suite list for shared build inputs and the e2e harness, an inert list for paths that cannot affect what e2e exercises, and the PackageSource graph for `packages/(apps|system|extra)/`. A path matching none of them escalates and logs why, so the next path someone adds fails safe instead of failing open. The two lists are checked in that order, so a specific escalation beats a broad inert directory: `.github/` is inert, the workflows that run the suite are not. `*.md` and `docs/`/`dashboards/` stay ahead of both, so a README under `packages/core/` does not inherit that tree's escalation. Ten tests cover both sides of the classification. Mutation-checked: reverting the fall-through fails the unclassified-path test, dropping `.github/` from the inert list fails the non-e2e-workflow test, and swapping the two list checks fails the existing pull-requests.yaml test. Note the per-entry escalations are belt-and-braces — removing `hack/*.mk` from the pattern still escalates via the fall-through, which is the point of having it. Closes #3392 Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
Both lanes ran `if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]` under a step gated on `steps.select.outputs.skip != 'true'`. When `select` runs, an empty result sets `skip=true` and the step never starts, so the only way to arrive with no selection is the `full-e2e` label — which sets `FULL_E2E=true`. The second test could never decide anything the first had not, while the comment above it advertised "full suite when the selection is empty" in the file whose whole point is that a green status means tests ran. Testing `FULL_E2E` alone keeps the real invariant visible. The comments now say what an empty selection means after the preceding commit: every changed path was explicitly classified inert, not that the selector had no idea. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
674355d to
ee4a482
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/select-e2e_test.bats`:
- Around line 181-188: Remove the test-level EXIT trap from each affected test
in hack/select-e2e_test.bats at lines 181-188, 190-197, 199-206, 208-215,
217-224, 226-233, 235-242, 244-256, 258-267, and 269-276. In every test, add rm
-rf "$tmp" after the final assertion so cleanup occurs only on success while
failed tests preserve their scratch directories and Bats retains its TAP
bookkeeping.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 94c02aa9-cc5f-4ba8-956a-a524636c452c
📒 Files selected for processing (5)
.github/workflows/e2e-fork.yaml.github/workflows/pull-requests.yamldocs/agents/e2e-testing.mdhack/select-e2e.shhack/select-e2e_test.bats
The privileged resolve step explains why the TIA inputs come from the base repo, and closed with a caveat about the selector: that it escalates only for an unrecognised path inside packages/ and silently selects nothing for a path it does not recognise at all, naming hack/*.mk as the example. That was true of the selector this branch replaces. Measured against the selector as it now stands, hack/common-envs.mk escalates to the full suite, and so does the fork workflow file itself. An empty selection no longer means the path went unrecognised; it means the path was classified as unable to affect what e2e exercises. The difference is the whole point of the change, and the caveat now contradicts it in the one comment a reader consults before touching the privileged lane. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The classifier this branch introduces is only total if every line reaches the loop that applies it. POSIX read assigns the final line and then returns non-zero when the input does not end in a newline, so a plain `while read` drops it. The fall-through that escalates an unrecognised path lives inside the loop body, so the dropped line is never classified at all and the selection comes back empty. Both lanes read empty as "skip Chainsaw" and post the required status green. The shape that matters is a list whose only unclassified path is last: `docs/x.md` followed by an unrecognised path with no terminator selects nothing, where the same list with a terminator escalates to all 21 suites. The safety net this branch is built around is bypassed by exactly the input it exists to catch. No caller triggers it today, because `git diff --name-only` terminates its output and the fork lane appends a newline explicitly. That makes it caller discipline documented in one of two callers, which the next caller will not read. Guard it where the assumption lives. Both new cases are red without the guard and the twenty-five existing ones stay green, so they pin this behaviour rather than the loop's general shape. They clean up inline instead of through an EXIT trap: that trap replaces the one the bats binary installs for its own bookkeeping, and a test failing under it can print no TAP line at all, which is the opposite of what a regression pin is for. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Four places on this branch describe an empty selection as "classified inert", against a list of three outcomes that every path is said to land in. There is a fourth: an edit under examples/backups/<app>/ where the app has no Chainsaw suite selects nothing, decided inline where backup examples are matched rather than through the inert list. The rule is deliberate and explained where it lives; it just is not one of the three the summary enumerates. The property the two lanes actually depend on is weaker and true: an empty selection means some rule decided the change cannot affect a suite, never that nothing looked at the path. State that instead, in the header that defines the classes and in the privileged lane's comment that consumes the result. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Four places said an empty selection means the inert pattern matched, and sent the reader to `inert_config_pattern` to see which paths those are. Documentation, dashboards and `*.md` are dropped before that pattern is ever consulted, and a backups example whose app has no suite is dropped where backup examples are matched. A reader checking the pattern for any of them finds nothing and has to reconstruct the order by reading the script. Name the rules that produce an empty selection instead, and keep the claim to the one the lanes rely on: empty is a decision some rule reached, not a path nothing looked at. The same bullet claimed every path is classified. One is not. The inert pattern makes `.github/` inert as a whole directory and the workflows that run the suite are exempted from it by name, so a workflow added under `.github/workflows/` reads as inert and the fall-through never sees it. Today's enumeration is correct and I checked it against `rg -l test-chainsaw`, so this is not a live hole, but the sentence saying it cannot exist is what makes the reminder next to that list look decorative. Record it as the guard it is. Separately, the fork lane's rationale for appending a newline described a mechanism this branch removes: the reader now guards its own loop, so the terminator is no longer what keeps the last path alive. Say that it is now the text-file contract and a second line of defence across two files, which is a reason to keep it and a different reason than the one written there. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The ten tests added with the classifier each installed `trap 'rm -rf "$tmp"' EXIT`, taking this file from fifteen such traps to twenty-five. `hack/cozyreport.bats` freezes that count per unconverted file, so `make unit-tests` failed, and that runs in the required checks job. The count is the visible half. The reason it is frozen is that an EXIT trap replaces the one the bats binary installs for its own bookkeeping, and a test failing under it prints no TAP line at all. Reverting the step-6 escalation in a scratch copy made the flagship pin for this whole change, the one asserting an unclassified path escalates, vanish from the output: the run reported executing twenty-six of twenty-seven tests and never named the missing one. After this change the same mutation prints `not ok 21` with its name, file and line. Clean up inline before the assertion instead, which is what the two tests added alongside them already did. Behaviour is unchanged, all twenty-seven still pass, and the frozen count returns to fifteen rather than being re-frozen higher, since re-freezing would bank ten fresh instances of the thing the freeze exists to stop. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
The previous pass corrected the same explanation in the selector header and in the in-tree lane and missed this one, which sits in the fork lane's selection step. It still said an empty selection means "verified inert" and pointed only at the inert pattern, and it lacked the caveat the other two copies gained: that a workflow added under `.github/workflows/` reads as inert until it is named in the escalation list. Three copies of one explanation is the underlying problem; leaving one of them a version behind is the way that problem shows up. A correction is not finished while a copy of the old claim is still alive. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Two defects introduced by the previous two commits. The in-tree lane's comment said `skip=true` cannot mean "a workflow nobody classified", and then explained, after a colon, exactly how it can: `.github/` is inert as a directory and the e2e workflows are exempted from it by name, so a new one reads as inert. The clause meant to warn a reader instead told them the one unsafe case is safe, in the copy consulted when deciding whether to trust a green gate. The other three copies of the caveat state it correctly. The scratch-directory cleanup landed before the last assertion in every test added here. That passes the EXIT-trap guard and inverts the reason the guard exists: both runners set `-e`, so cleanup placed last is unreachable on failure and the fixture survives for inspection, which `docs/agents/e2e-testing.md` §3 states outright and the already-converted `hack/select-install_test.bats` follows. Placed early, a failing test deletes the evidence of its own failure, and one test had the call between two assertions, so which of them failed decided whether the directory survived. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Three of the four workflows on the escalation list cannot be exercised by the run they trigger. `e2e-fork` runs on `workflow_run`, which always takes the default-branch copy; `e2e-tag` runs on `workflow_call` from a tag push; `nightly` runs on a schedule. Only `pull-requests.yaml` is executed from the pull request's own head, so only there does escalating test the edit rather than the tree around it. That argument existed in the reference doc and was removed by this branch rather than answered, and the list it argued against is what replaced it. Measured over six months of main, 26 commits touch one of the three with no other escalating path; each now pays a full Chainsaw run it used to skip, on a suite whose flake rate is the reason people avoid running it for nothing. Keeping them is defensible: the rule stays one idea rather than two, and dropping them leaves those files inert, which reads as an oversight rather than a decision. What was not defensible was paying it silently. The trade now sits next to the pattern that makes it, with the number, so the next person weighing it starts from measurement instead of rediscovering the mechanism. The reference doc points there rather than carrying its own copy. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
LGTM. Rebased onto main after #3262, the diff is down to five files, and the fail-safe now holds at the root: the reader-side read || [ -n "$file" ] guard means an unterminated last line cannot skip the fall-through escalation this PR exists for, with dedicated regression cases for exactly that input. Two full review passes at this head found no blocking defect.
The measured half-year replay is worth stating plainly: about 8% of commits start paying a full suite, and nearly all of that cost is closed fail-opens. Changes under pkg/ (84 commits) and hack/e2e-apps/ (119) previously selected nothing and could green the required gate with no suite run at all. The change is also monotone against the whole replay: nothing that escalated before escalates less now.
The escalation cost for the three workflows that do not execute from the PR head is recorded next to the pattern with its measured price and the conditions for revisiting, so the trade-off is explicit instead of silent.
Transition note: paths not yet classified (for example test/ and packages/tests/) now escalate to the full suite by design and print a classify-me line; the first commit touching such a path pays one full run and the fix is a one-line classification.
Reviewed with Opus assistance.
What this PR does
Closes #3392. Stacked on top of #3262 — that PR turns the
E2E Testsmerge gate into a commit status, which is what makes this fail-open path matter: a skipped required job used to pass branch protection either way, but a status postedsuccesswith no suite run is an explicit green verdict on untested code.hack/select-e2e.shsilently selected nothing for a changed path it did not recognise outsidepackages/. Both lanes read an empty selection as "skip Chainsaw" and then report green. The worked example from the issue ishack/common-envs.mk, which supplies the tag, push and output flags of every image in the tree and matched nothing:full_suite_patterncoveredhack/*.shandhack/*.batsbut nothack/*.mk. Walking the tree turned up five more in the same position —pkg/,tools/,go.mod/go.sum,hack/lib/andhack/buildkitd.toml— plus.github/workflows/e2e-fork.yaml, which #3262 adds whilepull-requests.yamlwas already escalated.As the issue argues, a blanket "empty selection means full suite" fallback would be the wrong fix, because it escalates exactly the inert paths the design deliberately skips. Instead every changed path now lands in one of three classes by an explicit rule:
full_suite_pattern— shared build inputs and the e2e harness:packages/library/,packages/core/, the Go trees (api/,cmd/,internal/,pkg/), the codegen undertools/,go.mod/go.sum,hack/*.sh|*.bats|*.mk,hack/lib/,hack/buildkitd.toml,hack/e2e-*.yaml, theMakefile, and the workflows that run the suite.inert_config_pattern— paths that cannot affect what e2e exercises:examples/,.github/(bar the e2e workflows),.claude/,.gemini/,img/,hack/testdata/, the codegen boilerplate header, and the top-level meta files (LICENSE,.gitignore,.pre-commit-config.yaml,.coderabbit.yaml).packages/(apps|system|extra)/, unchanged.A path matching none of them escalates to the full suite and logs which path forced it, so the next path someone adds fails safe rather than fails open. The two lists are checked in that order, so a specific escalation beats a broad inert directory —
.github/is inert,.github/workflows/e2e-fork.yamlis not.*.md,docs/anddashboards/stay ahead of both, so aREADME.mdunderpackages/core/does not inherit that tree's escalation.The second commit removes the
|| [ -z "$SELECTED_APPS" ]arm from the Chainsaw step in both lanes. Under a step gated onskip != 'true'it could never decide anything theFULL_E2Etest had not, while the comment above it advertised "full suite when the selection is empty" in the file whose whole point is that a green status means tests ran.Deliberate choices worth a second opinion
pull-requests,e2e-fork,e2e-tag,nightly), enumerated rather than matched by prefix so an unrelated workflow does not burn a full run. That does mean ane2e-tag.yamlornightly.yamledit now costs a full PR suite even though the lane it changes is not the PR lane. Easy to narrow to the two PR lanes if that is the wrong trade.go.mod/go.sumescalate. A dependency bump changes the shipped binaries, so this is the honest classification, but it does make Renovate's Go PRs pay for a full suite.Verification
hack/select-e2e_test.bats: 25 tests green (15 existing, 10 new), covering both sides of the classification..github/from the inert list fails the non-e2e-workflow test; swapping the order of the two list checks fails the existingpull-requests.yamltest.hack/*.mkfrom the pattern still escalates via the fall-through, so that test pins the outcome rather than the pattern entry. Called out so nobody reads it as stronger coverage than it is.make bats-unit-tests: 417 tests, 0 failures.actionlintclean on both workflows.hack/*.sh).Downstream repositories
Walked the trigger map in
docs/agents/contributing.mdagainst the diff file by file. The onlyhack/-related trigger iscozystack/ccp, which gates onhack/package.mkandhack/common-envs.mkexisting as anchor files and on whatmake generatedoes — nothing here is moved or renamed, neither anchor file is touched, and no make target changes behaviour (BATS_UNIT_FILESalready globbedhack/*.bats). The rest of the diff is e2e workflows and agent docs, which appear nowhere in the map.Release note
Summary by CodeRabbit
Bug Fixes
full-e2elabel or applicable change types.Tests
Documentation