test(e2e): the runner and the library disagree on where a collector writes, so a suite can be green on one and broken on the other #3711

hack/cozytest.sh:95 defaults COZY_REPORT_DIR to the repo-relative _out/cozyreport. The library defaults the same variable to the absolute /workspace/_out/cozyreport at three sites in hack/e2e-chainsaw/_lib/run-kubernetes.sh, lines 333, 764 and 910 (git show origin/main:hack/e2e-chainsaw/_lib/run-kubernetes.sh | grep -n 'COZY_REPORT_DIR:-/workspace/_out/cozyreport'). Nothing reconciles the two, and which one applies depends entirely on how the suite was started.

For a suite that drives a real collector and does not set the variable itself, that produces two distinct failures, and they do not overlap. Under the bare bats binary the library default applies, /workspace is not writable, and the collector dies on mkdir -p before issuing a read, so any assertion about what it read fails for a reason unrelated to the code under test. Under hack/cozytest.sh the collector succeeds and writes into the working tree at _out/cozyreport/snapshots/kubernetes/<collector>/.

That second path is the more consequential of the two, and it is the reason this is worth fixing rather than documenting. Both defaults derive the snapshot directory from COZY_SNAPSHOT_NAME:-kubernetes, which is the same name the real kubernetes-latest and kubernetes-previous suites use. A unit run and a real e2e run therefore write collector artifacts to the same path. Nothing distinguishes them once written, so a stale COLLECTION-FAILED.txt left by a unit test is indistinguishable from one produced by a failing e2e, in a directory whose entire purpose is to be read as evidence about a failing e2e.

Scope, counted on origin/main rather than asserted. Six hack/*.bats source the library, by the criterion of carrying a . or source line for it (for f in $(git ls-tree --name-only -r origin/main hack | grep '\.bats$'); do git show "origin/main:$f" | grep -qE '^[[:space:]]*(\.|source)[[:space:]]+hack/e2e-chainsaw/_lib/run-kubernetes\.sh' && echo "$f"; done; mentions of the path do not count, which is why a looser grep returns more). Of those six, three drive a collector at all, and two of the three set COZY_REPORT_DIR to a temp directory of their own: run-kubernetes-serial-console_test.bats and run-kubernetes-talos-diagnostics_test.bats. The third, run-kubernetes-node-join_test.bats, sets it nowhere.

It is nonetheless safe on main today, and the reason is worth stating because it is what makes this latent rather than broken. Only two collectors in the library write into COZY_REPORT_DIR, cozy_capture_tenant_serial_console at line 763 and cozy_capture_tenant_talos at line 908, and stub_collectors in that suite replaces both. Safety therefore rests on the current contents of one stub list, not on anything about the suite. Adding a collector that writes artifacts, or removing one of those two stubs, is enough to produce the first instance. That is exactly how this was found, in a branch that does both.

Two directions for a fix. Making the library and the runner agree on one default removes the divergence but leaves both writing under the shared snapshot name. Giving unit runs a distinct COZY_SNAPSHOT_NAME, or requiring every suite that drives a collector to point COZY_REPORT_DIR at its own temp directory, which two of the three already do, separates the artifacts, which is the part that matters.