test(e2e): migrate the E2E app suite from BATS to Kyverno Chainsaw #2979

Goal

Migrate the E2E app test suite from BATS to Kyverno Chainsaw — a declarative, Kubernetes-native E2E framework (CNCF, KUTTL successor) — replacing the per-app BATS loop in hack/e2e-apps/.

Context

This was piloted and grown to the full suite in #2826 (postgres + bucket → all 18 app suites). That PR is being postponed out of the immediate #2397 "move E2E off the PR pipeline" push and tracked here, so the Chainsaw exploration doesn't gate the keystone E2E work. The branch feat/e2e-chainsaw-example stands as the reference implementation.

Why

• Declarative assert collapses the timeout … until kubectl get … && kubectl wait pair (~100 occurrences) into a single existence-and-state poll, with a structured diff and automatic events / describe / podLogs capture on failure (today only harbor.bats does this, by hand).
• Removes a class of bash-level flakiness; the per-app 3× retry loop goes away (assertion polling replaces the fixed-timeout flakiness it papered over).
• Declarative tests are easier to select per-PR — complements the TIA work in feat(ci): test-impact analysis E2E (default-on) + release E2E workflow #2559 and epic [Epic] Diff-driven & scenario-based E2E testing for Cozystack #2847.

Scope (from the #2826 prototype)

• Port all 18 app suites under hack/e2e-chainsaw/: postgres, bucket, mariadb, mongodb, redis, qdrant, clickhouse, kafka, etcd, openbao, harbor, foundationdb, external-dns, kuberture, vminstance, gateway, kubernetes-latest, kubernetes-previous.
• Keep imperative logic in script steps (openbao init/unseal, kuberture external-dns probes, vminstance, gateway admission/impersonation, kubernetes-*); reuse hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
• CI: the e2e job runs chainsaw test hack/e2e-chainsaw/ via a new test-chainsaw target and uploads the JUnit chainsaw-report.xml; add the chainsaw binary to the e2e-sandbox image. install-cozystack and test-openapi stay BATS.

Open questions / risks to resolve before adopting

• Chainsaw v0.2.15 JMESPath bug: number-literal comparisons (ports[?port == + "N" + ]) mis-evaluate; the prototype works around it with the (ports[*].port) projection form. Confirm fixed/tracked upstream before relying on number-literal filters.
• TIA integration: how Chainsaw test directories map to select-e2e.sh per-app selection (today keyed on hack/e2e-apps/*.bats).
• BATS retirement cadence: dual-run (Chainsaw alongside BATS) vs hard cutover. test(e2e): migrate app suite from BATS to Chainsaw and wire into CI #2826 removes the .bats files outright — decide whether to soak in parallel first.
• Soak the three platform-dependent suites (gateway, kuberture, external-dns) and the two heavyweight kubernetes-* suites on full-platform CI before cutover.

References

• Prototype PR: test(e2e): migrate app suite from BATS to Chainsaw and wire into CI #2826 (feat/e2e-chainsaw-example)
• Related: Move E2E tests out of PR pipeline to release/nightly #2397 (move E2E off PR pipeline), [Epic] Diff-driven & scenario-based E2E testing for Cozystack #2847 (epic: diff-driven & scenario E2E), feat(ci): test-impact analysis E2E (default-on) + release E2E workflow #2559 (TIA)