fix(apps/kubernetes): pre-delete hook hangs deleting tenant addon HelmReleases when the tenant has no working nodes #3271

Summary

The kubernetes app's pre-delete hook (packages/apps/kubernetes/templates/delete.yaml, Job <release>-pre-cleanup) can hang indefinitely deleting a tenant's in-cluster addon HelmReleases when the tenant cluster has no working nodes — e.g. LINSTOR/ZFS resource exhaustion, worker VMs that never booted/joined, or a nodeGroup at minReplicas: 0. Because the hook's HelmRelease deletion is an unbounded kubectl delete … --wait=true, the tenant Kubernetes CR never finalizes, so deleting a broken tenant stalls (and, under GitOps, wedges the helm-controller working on the stuck uninstalls).

Root cause

delete.yaml Step 2:

kubectl -n <ns> delete helmreleases.helm.toolkit.fluxcd.io \
  -l "cozystack.io/target-cluster-name=<release>" \
  --ignore-not-found=true --wait=true          # <-- no --timeout

The child HelmReleases (in-tenant cilium, coredns, csi, ingress-nginx, …) can only uninstall cleanly if the tenant cluster is reachable and schedulable. When the tenant has no Ready nodes, the in-tenant cilium HelmRelease sits in InstallFailed (cilium-operator Deployment never leaves InProgress — nothing to schedule it on), and its uninstall/remediation drags for many minutes or does not converge. --wait=true with no timeout blocks the pre-delete Job on that, so:

1. <release>-pre-cleanup Job never reaches "completed"
2. Helm uninstall of the parent kubernetes-<release> HelmRelease fails: timeout waiting for: [Job/<ns>/<release>-pre-cleanup status: 'InProgress']
3. the Kubernetes CR finalizer never runs to completion → the CR (and its namespace) stays stuck deleting.

Contrast: Step 4 (TenantControlPlane delete) already does the right thing — --wait=true --timeout=120s || echo WARNING … — so it degrades gracefully. Step 2 (and, to a lesser extent, Steps 3/5/6) do not.

Impact

• Operators: deleting a tenant Kubernetes cluster that never became healthy (storage exhausted, nodes never joined, scaled to 0) can hang teardown and leave the namespace Terminating.
• CI: the render-side kubernetes-oidc-system / kubernetes-oidc-customconfig e2e suites (which create a control-plane-only tenant with 0 workers and delete it immediately) deterministically hit this. Under the Chainsaw suite (test(e2e): migrate app suite from BATS to Chainsaw and wire into CI #2826), which enforces cleanup completion, both fail at context deadline exceeded after the 5 min cleanup budget. The legacy BATS suite masked it (kubectl delete … --timeout=2m 2>/dev/null || true), so it was invisible on main.

Evidence

• PR test(e2e): migrate app suite from BATS to Chainsaw and wire into CI #2826, E2E run 29021040367 (attempt 2). cozyreport summary.txt:
  UninstallFailed helmrelease/kubernetes-oidc-byo — Helm uninstall failed … timeout waiting for: [Job/tenant-test/kubernetes-oidc-byo-pre-cleanup status: 'InProgress']
• Child HR stuck: kubernetes-oidc-byo-cilium — InstallFailed … timeout waiting for: [Deployment/cozy-cilium/cilium-operator status: 'InProgress'] (no nodes → operator never schedules).
• The TenantControlPlane itself did eventually delete (~10 min), confirming it's teardown-slowness on the child HRs, not a Kamaji DataStore deadlock.

Proposed fix (directions, for discussion)

Make the pre-delete hook resilient to an unreachable/nodeless tenant so teardown always completes:

1. Bound Step 2's kubectl delete helmreleases … --wait=true with a --timeout and || echo WARNING … (mirror Step 4).
2. After the bounded wait, for any child HRs still present (uninstall couldn't converge because the tenant is gone/nodeless), strip the Flux finalizers so they can be GC'd — safe here because the whole tenant (and its control plane) is being destroyed, so there is nothing left to orphan.
3. Apply the same bounded-wait treatment to Steps 3/5/6 for consistency.

This is a delicate hook with teardown-ordering history (#3062, #3078) — the finalizer-force in (2) needs care to not fire while the tenant is still legitimately alive (gate it on the TCP being confirmed gone, as Step 4b already does for the datastore secret).

Acceptance

• Deleting a Kubernetes CR whose tenant has no Ready nodes completes teardown within a bounded time (no indefinite hang, no Terminating namespace).
• The two render-side OIDC e2e suites tear down within the Chainsaw cleanup budget.