e2e: a Chainsaw failure in finally is reported as a pass and gets no host snapshot #4408

When a Chainsaw test fails only in its teardown, the job log says FAIL, the JUnit report says pass, and the host snapshot is never taken. Both tenant Kubernetes suites do their drain and storage checks in finally, so their teardown failures come with the least evidence of any.

The example is run 35750017261, attempt 1. The job log ends with --- FAIL: chainsaw/kubernetes-previous (983.00s) and Failed tests 1, because cozy_cleanup returned non-zero in the finally block of hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml (the drain failure from #4374). The chainsaw-report artifact of the same attempt has <testsuite name="kubernetes-previous" tests="1" failures="0"> and no <failure> element at all.

The report side is in Chainsaw v0.2.15, pkg/runner/runner.go. runCatch runs the catch, finally and step cleanup operations, and it calls runAction with a nil step report (line 414). runAction records an operation only when that report is set (line 439). The JUnit writer in pkg/report/junit.go builds <failure> from recorded operations only, so an error in finally fails the Go test and never reaches the XML.

The missing snapshot comes from the same function. finally is deferred first (line 310) and catch second (line 328), so catch runs before finally and checks failed() while the try block still looks green. The crust-gather snapshot in the global error.catch of hack/e2e-chainsaw/.chainsaw.yaml is skipped. That is why the #4374 artifact has no YAML for the leftover importer pod, and nothing that shows whether it had a deletionTimestamp.

Anyone reading the JUnit file to see what failed, person or script, will not find these failures. The comment above the finally block also says it runs "after the crust-gather snapshots on a failed run", which is true only when the try block failed.

Two ways I see to fix it. Move the drain and LINSTOR baseline checks from finally into a last try step, so a failure there is recorded and triggers the catch, and keep only the unconditional cleanup in finally. Or let cozy_cleanup take its own snapshot before it returns non-zero. Passing the step report through runCatch would fix the report upstream, but that is a Chainsaw change.