kamaji: deleted TenantControlPlane stays on its finalizer after a stalled reconcile #4489

Describe the bug

A deleted TenantControlPlane can stay forever on finalizer.kamaji.clastix.io, keeping its control-plane Deployment running. In one e2e run the TCP of a deleted Kubernetes app was still there 30 minutes after deletion, with both apiserver pods up.

Kamaji runs the TenantControlPlane controller with a single worker by default, and cozystack does not override --max-concurrent-tcp-reconciles or --controller-reconcile-timeout (default 30s). In this run one reconcile of the TCP ran for at least 13 minutes. After it, every reconcile, the deletion path included, reached the etcd DataStore call with an already-expired context, failed with DeadlineExceeded, and returned before removing the finalizer.

Environment

• Cozystack: PR chore(deps): roll up pending Renovate updates and move to Go 1.27.1 #4447 run, kamaji built from clastix/kamaji 26.3.6-edge (packages/system/kamaji/images/kamaji/Dockerfile at the PR head). Main has since moved to 26.9.4-edge (b31aa03); whether the stall still happens there is not known.
• Provider: in-tree e2e sandbox

To Reproduce

Not reproduced on demand. Seen in the kubernetes-latest chainsaw suite of run 36012456733 (attempt 1, PR #4447, 2026-09-24): the suite passed, deleted its app, and the TCP tenant-test/kubernetes-test-latest-version never went away.

Expected behaviour

After the Kubernetes app is deleted, the TCP loses its finalizer and is removed together with its Deployment within seconds, as happened for kubernetes-test-previous-version in the same run (deletion finished in under a second at 16:27:19 UTC).

Actual behaviour

Timeline from the kamaji manager log (UTC), filtered to lines that carry a reconcileID:

1. 16:29:29: last "has been reconciled" from the main TCP controller for this TCP.
2. 16:35:52: spec update, 16:36:59: deletionTimestamp set. The main controller logs nothing for the TCP between 16:29:30 and 16:50:06. The one line in that window, "soot manager completed its process" at 16:36:59, comes from the soot controller, which is a separate controller that logs under the same name.
3. 16:50:06: a reconcile ends with "unable to create the DataStore user". Its stack goes through the create/update branch, so it read the object before 16:36:59.
4. 16:55:13 and 17:01:56: deletion reconciles log "marked for deletion" and fail about 70µs later:

2026-09-24T16:55:13Z	INFO	marked for deletion, performing clean-up	{"controller": "tenantcontrolplane", "controllerGroup": "kamaji.clastix.io", "controllerKind": "TenantControlPlane", "TenantControlPlane": {"name":"kubernetes-test-latest-version","namespace":"tenant-test"}, "namespace": "tenant-test", "name": "kubernetes-test-latest-version", "reconcileID": "78163948-9c1c-43ef-9e72-fa7a7f216caf"}
{"level":"warn","ts":"2026-09-24T16:55:13.730999Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:68","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x13e4f627a3c0/etcd.tenant-root.svc:2379","peer":"Peer{Addr: <nil>, LocalAddr: <nil>, AuthInfo: <nil>}","method":"/etcdserverpb.Auth/RoleGet","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

At the snapshot (about 17:08) the TCP has generation: 5 and status.observedGeneration: 3, still carries finalizer.kamaji.clastix.io, and is also present in the end-of-run report. The kamaji pod holds its leader lease and idles at about 1.4m CPU, so it is not starved. The tenant-root etcd pods have no restarts; they went through leader elections between 16:31:39 and 16:33:04 and had a stable leader after that. The two leaked apiserver pods request 2986344448 bytes of memory each, and in e2e that memory stays taken for the following suites.

Additional context

What is missing is which call in the stalled reconcile ignores the reconcile context. That needs a goroutine dump from a reproduction. The etcd elections overlap the start of the stall, but I have no mechanism linking them.

Possible fixes: upstream, bound the call that ignores ctx. On our side, raise --max-concurrent-tcp-reconciles so one stuck TCP does not freeze reconciliation for every tenant, and enable pprof so the next occurrence can be dumped (--pprof-bind-address exists from 26.9.4-edge on, it is not in 26.3.6-edge). The e2e kubernetes suites should also wait for the TCP and its Deployment to be gone, so this shows up in the suite that caused it rather than as memory pressure later (see #4488 for the same effect from Kafka).