Repository navigation
e2e: kubernetes-oidc-system fails in cleanup when the tenant cluster outlives the 5m budget #3737
Description
kubernetes-oidc-system can fail after every one of its assertions has passed, because deleting the tenant cluster it created does not finish inside the cleanup budget.
Observed
Run 31337303277. The test body completes:
23:21:21 | kubernetes-oidc-system | keycloak-objects-in-cozy-realm | ASSERT | DONE
23:21:21 | kubernetes-oidc-system | keycloak-objects-in-cozy-realm | TRY | END
23:21:21 | kubernetes-oidc-system | create-kubernetes-oidc-system | CLEANUP| BEGIN
23:21:21 | kubernetes-oidc-system | create-kubernetes-oidc-system | DELETE | OK
23:26:21 | kubernetes-oidc-system | create-kubernetes-oidc-system | CLEANUP| ERROR
=== ERROR
context deadline exceeded
DELETE | OK means the delete call was accepted, not that the object went away. Chainsaw then waits for it to disappear and gives up at exactly five minutes, which is the cleanup: 5m budget in hack/e2e-chainsaw/.chainsaw.yaml:23. The suite is reported red on a step that tests nothing.
Why this one and not others
The comment above that budget already anticipates the problem: it says delete and cleanup are generous because finalizer-draining CRs can take minutes to reclaim their backing. Tearing a tenant Kubernetes CR down unwinds a Cluster API cluster, its machines, the KubeVirt VMIs behind them and their DataVolumes, and each layer waits for the one below.
Correction to two claims made earlier in this issue, both refuted by measurement (see the comment below for the numbers). The first said a tenant Kubernetes CR is the heaviest object in that category. It is not: the kubernetes-latest and kubernetes-previous suites reclaim a full cluster with workers in 91s and 106s, well under what these fixtures spend. The second said the fixtures render zero workers. They render replicas: 0, but the cluster-autoscaler shipped by the same chart scales md0 to 1 in response to unschedulable addon Pods, which packages/apps/kubernetes/values.yaml documents as intended behaviour. So each fixture does create a worker VM.
What actually costs the time is that the delete arrives roughly three seconds after the create, while the install is still in flight: helm-controller cannot begin the uninstall until the in-flight install action returns, and the pre-delete hook then spends two bounded 120s waits of its own.
Effect on triage
A red kubernetes-oidc-system currently means either "the OIDC wiring is wrong" or "the previous cluster took too long to disappear", and the two are only distinguishable by reading how far the log got before the error. The failure text, context deadline exceeded, names neither.
Scope
Pre-existing, and independent of the branch the run belongs to: the diff there does not touch OIDC, Cluster API or KubeVirt. Recorded from a single occurrence, so the frequency is unknown; the run list would answer it, but a five-minute ceiling on a teardown of this shape is worth stating regardless of how often it is currently crossed.
Activity
lexfrei commentedon Aug 10, 2026
Second occurrence, on the other OIDC fixture, and it reports the same timeout under a different error string.
Run
31373204907, suitekubernetes-oidc-customconfig:Same shape as the first report: every assertion passes,
TRY | ENDis reached, the delete is accepted, and the wait runs out after exactly 300 seconds. Both fixtures create a tenantKubernetesCR with zero worker replicas, so both spend the budget on the same teardown.The error text is worth noting because it does not look like a timeout. When the context is nearly spent, client-go refuses to queue the next request rather than issuing it and waiting, so the deadline surfaces through the rate limiter instead of as
context deadline exceeded. Anyone grepping for the earlier string will not find this run, and the wording points at API throttling, which is not what happened here. Both spellings mean the cleanup budget ran out.So the failing suite name is not a stable signal either: the same teardown produced
kubernetes-oidc-systemin one run andkubernetes-oidc-customconfigin another, purely on which fixture happened to be torn down when the cluster was slowest to release.dosubot commentedon Aug 10, 2026
This matches the issue tracked in #3271 [1], which was resolved by commit
02b87b63f— that fix bounded Step 2'skubectl delete helmreleases … --wait=truewith--timeout=120sand force-clears leftover Flux finalizers on child HRs still present after the timeout.That fix is present on
main, and has been further hardened by recent commits (b60c08f11on 2026-08-07 switched KamajiControlPlane/DataVolume/LB Service deletes to--wait=false;0b7a8f67dadded safe-state checks before finalizer removal) [2].The likely reason the timeout is still being hit: the pre-delete Job has no
activeDeadlineSeconds, and the individual step timeouts can aggregate past 5 minutes — Step 2 (child HR delete) is 120 s, Step 4 (TenantControlPlane delete) is another 120 s, plus Pod startup (image pull from Docker Hub with no mirror can be slow) and the remaining steps [3]. If any of those steps actually uses its full timeout, the 5-minute Chainsaw cleanup budget in.chainsaw.yaml[4] is exceeded.The two error strings are the same root cause — the context's deadline was reached — but surface differently depending on whether the last call was in-flight (
context deadline exceeded) or the rate limiter refused to even queue a new request (rate: Wait(n=1) would exceed context deadline). Worth noting in CI grep patterns.To reply, just mention @dosu.
Docs are dead. Just use Dosu.
lexfrei commentedon Aug 11, 2026
What remains inside the 5m budget after the install is taken out of it.
The dominant term in this failure was the remainder of the install: the CR is deleted about three seconds after it is applied, a release cannot be uninstalled while its install action is running, so
cleanupwas paying for the tail of the install before it paid for any teardown at all. That part is addressable in the fixtures and is what the pending change addresses.This is the part that is not, so it is written down before the change lands rather than after it is mistaken for a complete fix.
packages/apps/kubernetes/templates/delete.yamlruns as the release's pre-delete hook, and on a tenant whose worker never joined it spends most of the 5m budget on its own::160) deletes the child HelmReleases with--wait=true --timeout=120s. The step's own comment states the case it was written for: a tenant with no Ready nodes, includingminReplicas: 0, where those uninstalls cannot converge. On expiry it falls through to force-clearing their finalizers, so the timeout is a floor on the time spent, not a failure.:206) deletes the TenantControlPlane with--wait=true --timeout=120s, also falling through on expiry to Step 4b.docker.io/clastix/kubectl:v1.32(:63), which the template notes comes from Docker Hub with no mirror in front of it.Both fixtures in this issue are exactly the shape those bounds were written for: their tenants have no joined worker at teardown. So the two waits are floors rather than ceilings there, and ~240s of a 300s budget can be spent before the remaining steps begin.
The hook carries no
activeDeadlineSeconds, and that is a decision rather than an omission. Its own comment at:25-41derives it: a deadline has to sit above the sum of the waits below it or it cuts runs that were going to finish, and far enough under helm-controller's uninstall budget to be what helm reports rather than what it waits through, and those two bounds currently leave no gap. It also names where room would come from when they do (spec.uninstall.timeouton the generated HelmRelease, not squeezing the waits) and which existing hook to copy the shape from (packages/apps/bucket/templates/hooks/cleanup-acme.yaml, deadline and retry budget set together). So this is not a request to add a deadline; adding one against the current arithmetic would cut teardowns that were going to succeed.What it means for this issue: taking the install out of
cleanupremoves the term that was crossing the budget, and it does not bound the teardown. The failure mode becomes much less likely rather than impossible, and if either OIDC suite reds in cleanup again after that change, these two 120s waits plus the unmirrored pull are the first place to look rather than the install.One number worth having when that happens. On run
31373204907, reading the install's return from the hook Jobs' completion events and the cleanup boundaries from the job log: the System suite's uninstall took roughly 137 to 157 seconds once the install had returned, comfortably inside 300s. The CustomConfig suite's had consumed at least 141 seconds and had not finished when the 300s cut fell, so it has a lower bound and no observed upper one. Teardown-alone timing is therefore measured for one fixture and open for the other.lexfrei commentedon Aug 11, 2026
Correction to the comment above: the install remainder is the smaller of the two terms, not the dominant one.
That comment opens by calling the install remainder "the dominant term in this failure". It is the term that was crossing the budget, which is not the same thing and is not the larger half. Re-derived from run
31373204907, with every input quoted so this can be refuted rather than taken.The System suite's cleanup boundaries come from the job log's own timestamps:
CLEANUP | BEGINat11:01:53.5448,CLEANUP | ENDat11:06:07.2553, so 253.71s in total. The install action returns when the last blocking post-install hook Job completes, and that Job'sCompletedevent is read in thekubectl get eventscapture at11:30:50.5584with an age of27m. Ages there are truncated to whole minutes, so27mplaces the completion in the half-open interval (11:02:50.558,11:03:50.558].Splitting the 253.71s on that interval gives an install remainder of (57.0, 117.0]s and a teardown of [136.7, 196.7)s. The two do not overlap: the largest possible remainder is 117s and the smallest possible teardown is 137s, so the teardown is the larger half at every point in the interval.
On the earlier figure, which this supersedes. The comment above states "the System suite's uninstall took roughly 137 to 157 seconds once the install had returned". Its lower bound agrees with the derivation here; its upper bound of 157s cannot be reproduced from the timestamps above, and I cannot reconstruct what it was derived from. Treat 137–197s as the figure that follows from the stated inputs, and the earlier upper bound as unsupported rather than as independent confirmation.
The CustomConfig suite is the same shape and stays censored.
CLEANUP | BEGINat11:06:11.2212,CLEANUP | ERRORat11:11:11.2223— 300.001s, the budget exactly. Its Job'sCompletedevent reads22min the same capture, placing completion in (11:07:50.486,11:08:50.486], so the remainder is (99.3, 159.3]s and the teardown is at least 140.7s with no observable upper bound, because it had not finished when the cut fell.What this changes and what it does not. Removing the install remainder is still what brings the suite under 300s: 253.71s minus (57, 117]s leaves [137, 197)s against a 300s budget, and subtracting the same term from the CustomConfig case clears it too. The direction of the fix is unaffected. What changes is the reason it works, and that makes the rest of the comment above more load-bearing rather than less — the teardown is the bigger half, nothing in the fixtures bounds it, and the pre-delete hook's two 120s waits plus an unmirrored image pull are where to look if either suite reds in cleanup again.
For anyone re-deriving any of this: the truncation is the whole reason these are intervals rather than points. Ages in a
kubectl get eventscapture are truncated to whole minutes above 10m while the log's own marks are exact to sub-second, so an age of27mmeans "between 27 and 28 minutes ago", not "27 minutes ago". Reading those ages as rounded instead of truncated shifts every interval here by half a minute, so a derivation has to state which it assumes.myasnikovdaniil commentedon Sep 22, 2026
Closing, suite this is about is gone. #4020 folded both standalone oidc suites into the latest-version tenant cluster, so there is no
kubernetes-oidc-systemchainsaw test to time out in cleanup anymore. Wait for parent HR Ready before the assertions lives inrun_kubernetes_testitself since 401993a, and #3802 pins that ordering so it doesn't drift back.