Repository navigation
e2e: two more failure-path reads can starve the tenant snapshot (image-cache reachability, VMI capture walk) #3666
Description
Activity
Two more sites on the same failure path, with a delete-specific nuance worth spelling out.
run-kubernetes.shlines 400, 402, 503 each runkubectl delete ... --wait=true --timeout=10swith no--request-timeoutand no outertimeout. The--timeout=10sbounds only the wait-for-deletion, not the DELETE request itself: that is--request-timeout, which defaults to 0 (unbounded). So against a hung apiserver the DELETE call has no ceiling at all, and the three deletes sit on the same failure path ahead of the tenant snapshot. Line 899'skubectl get vmisand the adjacentkubectl waitcarry--request-timeout/--timeoutbut no wall-clocktimeout -k, so they too can outrun the op budget. All pre-existing, same class as the sites above. The cleanest fix is the phase-budget approach the node-join diagnostics change adopts (a single deadline for the whole failure-diagnostic phase, past which later collectors refuse out loud rather than each carrying its own arithmetic), applied to this path too.Fuller enumeration of the guest-Talos capture path, which is the largest nominal member of the failure-path budget (~620s). These reads carry only
--request-timeout/--timeout, not a wall-clocktimeout -k, so an apiserver that hangs rather than answers can outrun the op budget here just as on the sites above:run-kubernetes.shlines 353, 400, 402, 409, 411, 413, 441, 503, 507, 508, 511, 899 (delete/get/wait/exec on the tenant path). Same class as the reachability helper and the VMI walk already in this issue. The cleanest fix is the phase-budget approach the node-join diagnostics change adopts: a single deadline for the whole failure-diagnostic phase, past which later collectors refuse to start rather than each carrying its own arithmetic, applied to this path too. Related happy-path decision defect is tracked separately in #3668.Worth recording so the next person does not chase a number that cannot exist: on this path the bringup ceilings alone already exceed the op timeout before any diagnostic read happens. The kubernetes suite op runs under
timeout: 50m, while the ceilings of the bringup steps plus the node-join deadline sum past that on their own. So the honest statement about the failure path is not "the worst case fits", it is "the worst case is not bounded by construction, and the phase budget bounds what the diagnostics may START, not what the whole path may cost".That is why the node-join diagnostics change derives its phase budget from an inequality rather than asserting a total: budget + the largest single collector's overshoot + the snapshot must fit in what remains after bringup, since a collector admitted one second before the deadline still runs its full cost. Anyone tightening this later should fix the unbounded members listed above first; until then a total for the path is not a number anyone can honestly write down.
Correcting the count in the description:
_talos_image_cache_reachable_from_tenantmakes SEVEN unbounded kubectl calls, not six. Lines 93, 100, 109, 123, 130, 137, 142 inhack/e2e-chainsaw/_lib/talos-image-cache.sh. The one the earlier count missed is thedelete pod --wait=falsethat follows the probe:--wait=falsedrops the wait for deletion, not the DELETE request itself, so it hangs on a wedged apiserver exactly like the other six. Same nuance as the--timeoutnote above, in the opposite flag.Two more failure branches in the same file belong to this issue, both ending in the same
exit 1with the snapshot EXIT trap already armed, so an unbounded read here costs the same artifact as the ones above.The backend-readiness branch (the block that opens with "backend readiness failed: the Pod was schedulable but did not become Available within 300s") runs
describe deployment,describe pods -landget events --sort-bywith no wall-clock bound, two of them through the tenant kubeconfig, which is the connection most likely to be the reason the branch was taken at all. The HelmRelease-history branch a little further down carries the same shape: aget hrread and twodescribe hrcalls with no bound.Neither is touched by the node-join diagnostics change, and there is no interaction with its phase budget, because these branches exit before the diagnostics phase opens and its deadline reads as zero there. Now that
cozy_diag_readexists in the same file, each of these is a one-line change; the reason to do them together with the sites already listed is that they all protect the same artifact.Two more members of the same failure path, and a note about how they are accounted for.
The budget inventory in both
chainsaw-test.yamlfiles lists the internal budgets of this path by name and ends by telling the reader to keep the outer op above "these budgets", which reads as an exhaustive list. It is not. Two things are missing from it.talos_image_cache_diagnose(hack/e2e-chainsaw/_lib/talos-image-cache.sh:192, called fromrun-kubernetes.sh:1411) opens by calling the reachability helper, which carries--pod-running-timeout=90son akubectl run --attachwrapped around a curl budget of--retry 8 --retry-all-errors --retry-delay 5 --connect-timeout 5 --max-time 45. The diagnose function then makes three further reads with no wrapper at all (:204,:207,:209): aget deploy,pod,svc,endpointslice, aget ciliumclusterwidenetworkpolicy, and alogs -l.The importer-pod loop at
run-kubernetes.sh:1258-1263has no bound of any kind. The pod list comes fromkubectl get pods -o nameunwrapped, and each matching pod costs two more unwrappedkubectl logscalls, current and--previous. It is the one member of this path whose cost scales with the number of pods while nothing but the op's owntimeout: 50mcontains it.The accounting matters as much as the two sites. A list that names six budgets and closes with "keep the op above these" is read as complete by the next person sizing the op, so an unlisted member is worse than an unbounded one that everybody knows about. Whatever fix bounds these reads should also make the inventory say how it stays in step with the code, or stop claiming to be exhaustive.
- added a commit that references this issue
on Aug 9, 2026 - added a commit that references this issue
on Aug 14, 2026 One more member of the same class, added with the block IO capture: the budget guard in
hack/run-kubernetes-node-join_test.batsprices the collectors gated ahead of the console by walking thecozy_capture_* || truecall form, so a bounded read placed ahead of the console in any other form is not in its sum. The LINSTOR resource state read on the node-join failure path is one: up to 25s of the same phase budget, ahead of the console, invisible to the guard. Any further read added in that form widens the gap the same way. Fix shape: teach the guard the second call form, or write such reads in the form it already walks.- added a commit that references this issue
on Sep 5, 2026
Summary
Two more reads on the kubernetes e2e failure path can spend far more than their share of the op budget, both in
hack/e2e-chainsaw/_lib/, both the same class as the node-join diagnostics block: a read whose cost is the cluster's to set, on the path that runs precisely when the cluster is unhealthy.1.
_talos_image_cache_reachable_from_tenantintalos-image-cache.sh. Sevenkubectlcalls with no outer bound (lines 93, 100, 109, 123, 130, 137, 142), including a blockingdelete pod, anexec, and adelete pod --wait=falsewhose flag drops the wait for deletion but not the DELETE request itself, plus a curl budget of--retry 8 --retry-all-errors --retry-delay 5 --connect-timeout 5 --max-time 45behind--pod-running-timeout=90s. Worst case is on the order of 12m30s. The retry budget was designed for the DECISION path, to ride out the seconds Cilium needs to program a freshly-applied allow policy; on the FAILURE path the same budget is paid again where the policy has already been in place for ~25 minutes, so the retries buy nothing and cost the tenant snapshot behind them.2.
cozy_capture_tenant_talosinrun-kubernetes.sh(around line 884).get vmis -o jsoncarries only--request-timeout=30s, no outer bound, and the walk over VMI rows has no cap. AtmaxReplicas10 that is roughly 500s where the budget arithmetic assumed ~100s.3. The guest-Talos capture path in
run-kubernetes.sh. Lines 353, 400, 402, 409, 411, 413, 441, 503, 507, 508, 511, 899 carry only--request-timeout/--timeout, never a wall-clocktimeout -k. This is the largest nominal member of the failure-path budget, roughly 620s. Note the delete-specific nuance:--timeouton a delete bounds the wait for deletion, not the DELETE request, which is--request-timeoutand defaults to 0.4. The bringup ceilings already exceed the op timeout. The kubernetes suite op runs under
timeout: 50m, while the bringup step ceilings plus the node-join deadline sum past that on their own. So the honest statement is not "the worst case fits" but "the worst case is not bounded by construction". Anyone tightening this should fix the unbounded members above first; until then a total for this path is not a number that can honestly be written down.5. Two failure branches further down the same file. The backend-readiness branch (the block opening with "backend readiness failed: the Pod was schedulable but did not become Available within 300s", around lines 1831-1833) runs
describe deployment,describe pods -landget events --sort-bywith no wall-clock bound, two of them through the tenant kubeconfig — the connection most likely to be the reason the branch was taken at all. The HelmRelease-history branch below it has the same shape: aget hrread and twodescribe hrcalls, unbounded. Both end in the sameexit 1with the snapshot EXIT trap already armed, so an unbounded read here costs the same artifact as every site above. Neither interacts with the diagnostics phase budget, because these branches exit before that phase opens.6. The rest of the node-join diagnostics block, for completeness. Two more collectors in the same block carry no bound at all, while their neighbours bound themselves: the DataVolume section at
run-kubernetes.sh:1256-1257(get datavolume,pvcanddescribe datavolume) together with the importer-pod loop noted in the comments below, and the CSR section at:1272-1274(get csrthrough the tenant kubeconfig, and the signer sidecar log). The tenant-kubeconfig reads matter most here for the same reason as everywhere on this path: that connection is the one most likely to be the reason the branch was taken.Why it matters
The node-join diagnostics bounding closes the arithmetic for its own block, but these sit on the same failure path ahead of or around the tenant crust-gather snapshot, so an unhealthy tenant apiserver can still burn the op budget here and lose the largest artifact. Item 1 is the single member that per-read bounding of the diagnostics block does not close, because touching the shared reachability helper risks the happy-path mirror selection.
Fix shape
For item 1, split the retry budget by path: the decision path keeps its Cilium-programming ride-out, the failure path uses a short bound since the policy is long settled. For item 2, give
get vmisan outer bound and cap the VMI walk the way the other capture walks are capped. Item 5 is a handful of one-line changes now that a bounded-read helper exists in the same file. Neither item 1 nor item 2 should be folded into the diagnostics-block PR, since item 1 touches a helper the happy path depends on.