Repository navigation
e2e: a transient apiserver error makes the whole suite fall back to the public Talos factory, cached suite-wide #3668
Description
Activity
Status note so the fix and the issue do not get confused with each other: the change in flight closes the FIRST door only, the
kubectl get deployat the top ofresolve_talos_image_factory_url. This issue is not fully closed by it, and should stay open afterwards.The reason is that the decision is cached once, at the end of the function, regardless of which inner check produced the negative answer. Inside the "deployment is present" branch every probe still conflates "the mirror does not work" with "the probe could not be run": the
kubectl rollout statuswait, and the three reads in_talos_image_cache_reachable_from_tenant(the pod lookup, theexec ... find, and the deploy image read). A transient apiserver error in any of them still reaches the cache write with an empty URL, which is exactly the suite-wide fallback this issue is named after. The fix shape is the same one the in-flight change uses, applied a layer down: either cache only a positive decision and a proven negative, or introduce one "the probe never ran" signal and return on it without writing the file. Note thatrollout statushas no indicator distinguishing "not ready" from "could not ask" other than its own timeout, so it needs its own treatment.Two related facts measured on the same file while confirming the above. The pod lookup uses
-o jsonpath='{.items[0].metadata.name}', which is a hard error on an empty list rather than empty output, so "no Running mirror pod" and "the apiserver refused" are again the same non-zero exit;{.items[*]...}with ahead -n1, or--ignore-not-found -o name, separates them. And none of the file's kubectl calls carried--request-timeoutbefore this change (0 of 17), while the siblingrun-kubernetes.shuses it on 18 calls and documents why: the flag bounds the HTTP request, an outertimeoutbounds the client's retries. The kubectl default is 0, meaning no timeout at all, so each of these probes is unbounded on a wedged apiserver.- added a commit that references this issue
on Aug 9, 2026
Summary
resolve_talos_image_factory_urlinhack/e2e-chainsaw/_lib/talos-image-cache.shdecides whether to use the in-cluster Talos image mirror withif kubectl -n kube-system get deploy talos-image-cache >/dev/null 2>&1(line 158). Any non-zero exit is read as "the mirror is not deployed", including a transient connection-refused, an Unauthorized, or an apiserver blip, not only a genuine absence. The result is written to_TALOS_IMAGE_FACTORY_DECISION_FILEand cached for the whole suite (lines 153-154, 182), so one bad check pins the decision for every worker that follows.Why it matters
This is the happy path, and the decision it makes is high-blast-radius: when it wrongly concludes "no mirror", every tenant worker in the suite falls back to the public Image Factory over live egress, which is exactly the dependency the mirror exists to remove. A transient apiserver hiccup during this one check therefore turns into suite-wide public-factory pulls, and a slow or throttled public factory is a plausible contributor to the node-join / image-pull timeouts the e2e suite flakes on. The failure is silent: the suite runs, just against the flaky dependency, so it reads as environment noise rather than a decision bug.
Same class as the image-cache gate defect just fixed on the failure path (a
kubectl getexit code does not distinguish "absent" from "could not ask"), but here on the happy path where the cost is a whole-suite fallback rather than one diagnostic line.Fix shape
Distinguish "deployment absent" from "could not determine":
kubectl get deploy ... --ignore-not-found -o namegives exit 0 plus empty output for a real absence and a non-zero exit for a refused/unauthorized/timed-out check. Treat only the empty-output case as "no mirror"; on an indeterminate result, either retry briefly or fail loud rather than silently pinning the public-factory decision for the suite. Do not cache an indeterminate result.Two more sites in the same function, not covered by the fix in flight
Checked against
fix/talos-factory-decision-indeterminate, both still present there._apply_talos_image_cache_egress_policy || truediscards its return value, so a failed policy apply changes neither the decision nor whether it is cached. The endpoint is committed and cached while the tenant default-deny drops traffic to it, which is the same shape as the unanswered lookup this issue is about: an outcome the function cannot observe is treated as a good one. The sibling helper on the ghcr side takes the opposite branch, refusing to cache when its own allow fails.The decision file is written with a bare
printf ... > "$file". The same file is read from another suite in the shared sandbox, so a concurrent reader can see a partially written decision. The ghcr helper writes atomically.Both belong here rather than in a separate issue: same function, same decision, same cache. Splitting them off would make it easy to fix half and close the ticket.