Repository navigation
e2e: the OIDC fixtures each provision a worker VM they claim not to, with no drain barrier #3739
Description
Both OIDC chainsaw fixtures describe themselves as render-side and claim they create no workers. They create one each, along with its 20Gi DataVolume, and nothing drains it before the storage-heavy kubernetes-* suites run next.
The claim
hack/e2e-chainsaw/kubernetes-oidc-system/ and kubernetes-oidc-customconfig/ both head their fixture with a variant of:
Result: zero KubevirtMachines, zero worker DataVolumes, no DRBD churn overlapping the kubernetes-previous suite (see #3231).
The reasoning behind it is sound as far as it goes: nodeGroups is omitted, so the schema emits a default md0 with minReplicas: 0, and the MachineDeployment renders with spec.replicas: 0.
What the runs show
Runs 31337303277 and 31373204907, for both fixtures:
SuccessfulCreate machinedeployment/<release>-md0 Created MachineSet <release>-md0-66zms
SuccessfulScale machinedeployment/<release>-md0 Scaled MachineSet <release>-md0-66zms: 0 -> 1
SuccessfulCreate machineset/<release>-md0-66zms Created machine "<release>-md0-66zms-qtjzv"
pod/importer-<release>-md0-66zms-qtjzv-disk-system
pod/virt-launcher-<release>-md0-66zms-qtjzv-w2d4r
Rendering with zero replicas is not the same as staying at zero. The cluster-autoscaler that the same chart installs scales md0 up in response to the unschedulable Pods of the in-tenant addons, which packages/apps/kubernetes/values.yaml documents as the intended behaviour of an empty nodeGroups: the MachineDeployment "provisions no workers until an unschedulable Pod triggers the cluster-autoscaler".
So each fixture provisions a worker VM and a 20Gi DataVolume it never asked for and never waits for.
Why it matters beyond the wrong comment
The claim is load-bearing for the fixtures' design. They exist in this shape specifically so that no DRBD churn overlaps the kubernetes-* suites that follow, which is the failure #3231 was about. That protection does not hold: by the timing in these runs the virt-launcher Pod appears at the end of the cleanup window or after it, so the worker outlives the fixture and its storage is still being reclaimed while the next suite starts. The churn came back through a different actor.
Two things are worth separating here. The wrong comment is cheap to fix. The absent barrier is not, and deciding what to do about it needs a call: either stop the autoscaler from acting in these fixtures, or make them wait for the worker they caused to be drained, or accept the overlap and drop the claim that they avoid it.
Scope
Pre-existing, and independent of the cleanup-budget failure in #3737 that surfaced it. The comments were written when the fixtures were ported to Chainsaw; the autoscaler behaviour they miss is older than that.
Activity
lexfrei commentedon Aug 11, 2026
Two places outside the two fixtures carry the description this issue is about. I am recording them as addresses rather than fixing them, because they need different judgements.
packages/apps/kubernetes/tests/nodegroups_default_test.yaml:145, stale name rather than a false claim. The helm-unittest case labels itsminReplicas: 0scenario "the render-side OIDC bats path". Two things are off. The Kubernetes OIDC bats originals no longer exist, onlyhack/e2e-apps/monitoring-oidc-system.batsandhack/e2e-apps/monitoring-oidc-customconfig.batsremain, so the label points at a path that was ported away. And "render-side" is the description that turns out not to hold for those suites, for the reason this issue documents. Blame isb3aac80830, 2026-07-07, well before the branch that corrected the fixtures.The substance of the comment is still accurate, which is why it is worth separating: "no worker DataVolumes provisioned until scale-out" is exactly right, and the autoscaler scaling
md0from 0 to 1 is the scale-out. A reader can take the sentence literally and be correct, then take the parenthetical as evidence that no workers appear in practice and be wrong. So this is a naming fix, not a correctness fix, and it belongs to whoever changes that test next rather than to a dedicated change.hack/e2e-apps/monitoring-oidc-system.bats:3andmonitoring-oidc-customconfig.bats:4, same wording, NOT verified. Both call themselves "render-side only", andmonitoring-oidc-system.bats:82adds "We do not wait on HR Ready: that requires a full VictoriaMetrics + Postgres bringup which the render-side test deliberately avoids".I have not checked whether those claims hold. They are a different application (the Grafana instance under
monitoring), a different chart, and a different set of hook Jobs, and none of the evidence in this issue is about them. What I can say is only that the wording matches, and that the wording matched in the Kubernetes suites for a reason that had nothing to do with the phrase itself: the suite's assertions were render-side while the install it triggered was not, and the two were conflated.So the question worth asking of those two suites is not "is the comment wrong" but "does that chart's install action outlive its render-side assertions, and if so what pays for the difference". Whoever picks this up should treat it as an open question with a known shape, not as a reported defect, since there is no observation behind it yet.
lexfrei commentedon Aug 11, 2026
A fourth address, and this one is a plural claim that is true of one member and false of the other.
packages/apps/kubernetes/templates/oidc-rbac-job.yaml:207-208, inside themode=Nonecleanup branch:"these" is the pair deleted on the two lines below it,
<release>-oidc-kubeconfigand<release>-oidc-authn-config. The claim holds for one of them and not the other:oidc-authn-config.yamlgates onSystemat:30andCustomConfigat:75, so it does render for both. Correct.oidc-kubeconfig-secret.yaml:17is{{- if eq .Values.oidc.mode "System" }}and nothing else. System only.So
<release>-oidc-kubeconfignever exists atCustomConfig, and a sentence written for two objects is accurate for one of them.Nothing breaks. The delete carries
--ignore-not-found, so naming a Secret that was never rendered is a no-op, and the cleanup does what it says. The cost is entirely to the next reader: this is the sentence someone consults to learn which modes create that Secret, and it answers with one mode too many.Worth recording here rather than filing separately because the tree now holds two statements about that fact and they disagree. The
kubernetes-oidc-customconfigfixture reasons about the same question when it explains why its absence proof for<release>-oidc-kubeconfighas to run after the bootstrap Job, and it gets the mode gating right. Whichever of the two a reader finds first decides what they believe.The fix is one clause: name the two Secrets separately, or narrow "these" to the one the sentence is true of. It belongs to whoever next touches that template rather than to a change of its own.
lexfrei commentedon Aug 12, 2026
The tenant lives about 258s in these fixtures, not the ~3s the fixture shape suggests. A Helm release can't be uninstalled while its install action is still running, so the DELETE blocks until the install returns, and everything the release created, including the worker Machine the autoscaler scaled up and its disk import, exists for all of it.
Measured on run
31373204907, job93410526958, suitekubernetes-oidc-system. All timestamps below are the ISO prefix at the head of each log line, which carries microseconds. Chainsaw's own column inside the line is rounded to the second, and mixing the two costs up to a second per interval. Both sit in the same line and the rounded one is the more visible of them.The middle line is the
Completedevent on<release>-oidc-bootstrap, read as27min thekubectl get eventscapture at11:30:50.5584002Z. Event ages there are truncated to whole minutes above 10m, so it is a half-open interval and the only inexact figure here. That imprecision comes from the data, the rounding above would have been mine.So the real window is 66x the DELETE, or 86x if you take the nominal three seconds literally. Someone starting from 3s reads any change to this window as enormous, and compares against a number that was never true.
Two things this does not claim. Window length is not DRBD load: how long a tenant exists and how much churn its disk causes are different quantities, and nothing here measures the second one. And this is one run, not a distribution.
kubernetes-oidc-customconfigon the same job never finished its cleanup at all,CLEANUP | BEGINat11:06:11.2211540ZandCLEANUP | ERRORat11:11:11.2222879Z, which is 300.001s, the budget exactly. That suite has a lower bound and no observed upper one.lexfrei commentedon Aug 13, 2026
One more consequence of the worker that does get provisioned here, recorded on this issue rather than separately because it goes away if the worker does.
Neither OIDC fixture injects
spec.talos.imageFactoryURL. The script-driven suites get it fromtalos_spec_blockinhack/e2e-chainsaw/_lib/run-kubernetes.sh, which decides at runtime whether the in-sandbox Talos image cache (hack/e2e-talos-image-cache.yaml) is up and only then points the CR at it;hack/run-kubernetes-talos-spec_test.batspins both directions of that decision. A static Chainsaw manifest has no way to make it, so the field is simply absent.So while the autoscaler scales md0 to one worker, that worker's disk import resolves against the upstream image factory instead of the sandbox cache: one external pull per OIDC suite run, where a script-driven suite would have used the mirror. No assertion in either fixture depends on it.
lexfrei commentedon Sep 7, 2026
The cause named here is gone from main. Worker pools moved to a separate chart in the control-plane-only split (1e5ba15):
values.yamlhas nonodeGroupskey left, andhelm templaterenders zero MachineDeployment for aKubernetesCR. The autoscaler is still deployed, but it discovers groups through--cloud-provider=clusterapi, so with no CAPI objects there is nothing for it to scale from 0 to 1.control_plane_only_test.yamlkeeps it that way: four documents asserted, negative containsDocument for each worker kind.I didn't re-measure a live run. If a worker VM still shows up in these suites on a real cluster it comes from outside the parent chart and needs its own issue. The drain barrier half goes away with the worker.
Closing as resolved by the split unless someone sees it differently.