Repository navigation
computeplane: worker pools always pull Talos from the public factory and cannot use the platform image cache #4700
Description
Activity
- added a commit that references this issue
on Oct 6, 2026 Another computeplane red that fits this, on #4802 (run 38074878460, job 114283924737, attempt 1). That PR doesn't change the pool: the md0 node group renders byte-identical on its merge base. Timeline from the suite's event dump taken at 19:26:05 UTC:
- 19:19:28: the assert on
computeplane-cluster-ingress-nginxstarts. The release waits oncomputeplane-cluster-cilium(DependencyNotReady). - ~19:20:34 and ~19:20:54: the two md0 worker DataVolumes schedule their importers, both pulling from
https://factory.talos.dev/.... Both first report "Importer pod cannot be scheduled" because the scratch PVC is not there yet. - ~19:22:30 and ~19:23:03: Import Complete.
- 19:22:48: the assert gives up with
status: {}. - ~19:24:24: the tenant cilium release installs.
So the workers came up after the window, and their disks were imported from the public factory. What I couldn't establish: why the importers started only a minute into the assert, and how much of the ~2 minutes per import is the download itself.
- 19:19:28: the assert on
Correction to my previous comment: the computeplane red on #4802 was not caused by this issue. It is #4720. The
computeplane-cluster-ingress-nginxassert has a 35m timeout and failed after 3m20s. At 19:22:48.83 UTC two host kube-apiservers loggedetcdserver: leader changed, and chainsaw recordedASSERT ERRORabout 130 ms later. So the poll was aborted. The workers were not late.The import times from the same job don't show the public factory being slower either. Each one is measured from the first importer pod getting scheduled to the worker's virt-launcher pod getting scheduled after the import, using the events in the cozyreport snapshot:
Suite Source Worker 1 Worker 2 computeplane factory.talos.dev 2m30s ~2m36s kubernetes-latest in-sandbox cache 2m38s 2m34s kubernetes-previous in-sandbox cache 2m08s 1m56s Two imports per source is a small sample, and nothing in it favours the cache. On this sandbox, pointing computeplane at the cache would not have changed the timeline.
So this issue is a production gap, not a CI one. On a platform without egress to factory.talos.dev, a ComputePlane pool has no way to boot: the module drops
talosfrom the pool, and the platform has no Talos source of its own to pass down. Fixing that needs a platform-level setting that the module can forward, which is a values/API change. I'll agree on its shape before writing it.Dropping the CI motivation here: the red that pointed at this issue was #4720, and the import times above show no gain from the cache.
What stays is the air-gap gap. A ComputePlane pool cannot boot without egress to the public Image Factory, because the platform has no Talos image source the module could forward to its pools. Adding one means a new public key in the platform values, so it needs a design proposal in cozystack/community first. Keeping this open until that lands.
packages/extra/computeplane/templates/cluster.yamlpasses each nodeGroup tokubernetes-nodesthrough apickallowlist (around line 211). By design,talosandosImageare left out so a tenant cannot choose the worker OS image, and the comment says a ComputePlane pool takes the module's worker OS image. But the module itself sets no Talos source either, so every ComputePlane worker downloads its installer from the public Image Factory, even where the platform runs an image cache (the e2e sandbox does). That makes worker bring-up depend on public egress: slower, and exposed to the slow-egress node-join failures seen in the tenant Kubernetes suites (#3231). Fix shape: have the module, not the tenant, pass the platform-level Talos source (the image cache / factory the operator configured in the platform values) into each pool, so the restriction on tenant choice stays and the platform's own cache is used. Found while designing the computeplane e2e suite (#3369).