Repository navigation
Shipped charts reference public images by :latest, which pins nothing and forces a registry round-trip on every pod start #3578
Description
Eight image: references in first-party charts use a :latest tag. Seven pin nothing at all. The eighth carries a digest, and only the tag text is latest.
packages/system/kubevirt-csi-node/templates/deploy.yaml:204 quay.io/openshift/origin-csi-node-driver-registrar:latest
packages/system/kubevirt-csi-node/templates/deploy.yaml:228 quay.io/openshift/origin-csi-livenessprobe:latest
packages/apps/kubernetes/templates/csi/deploy.yaml:82 quay.io/openshift/origin-csi-external-provisioner:latest
packages/apps/kubernetes/templates/csi/deploy.yaml:112 quay.io/openshift/origin-csi-external-attacher:latest
packages/apps/kubernetes/templates/csi/deploy.yaml:141 quay.io/openshift/origin-csi-livenessprobe:latest
packages/apps/http-cache/templates/haproxy/deployment.yaml:34 haproxy:latest
packages/apps/tcp-balancer/templates/deployment.yaml:35 haproxy:latest
packages/system/flux-plunger/values.yaml:1 ghcr.io/cozystack/cozystack/flux-plunger:latest@sha256:6a6ec93...
Three consequences. The third is measured rather than argued.
What runs is not decided by this repository. A tenant installed today and one installed next month get whatever latest pointed at on those days, and nothing in git records the difference. Rolling a cozystack release back does not roll these back either.
Every pod start goes to the registry. A :latest tag makes kubelet default imagePullPolicy to Always, so having the layers already on the node does not save the round-trip. None of the seven sets a policy explicitly, so all seven inherit it.
The cost lands on the critical path of an install. In run 31025557528, suite kubernetes-previous, the tenant kubevirt-csi-node DaemonSet spent its release's entire 10m budget fetching: origin-csi-node-driver-registrar:latest took 2m16.833s on one worker and 4m1.224s on the other, and the second pod finished starting six seconds after the HelmRelease install timeout fired. That failure is #3576.
flux-plunger already shows the shape of the fix, a digest carried next to the tag, and the update tooling moves those the way it moves the rest of the tree. Whether the openshift CSI sidecars should additionally be mirrored into the project's own registry is a separate question, and the answer does not block pinning them.
Activity
lexfrei commentedon Aug 6, 2026
The body attributes the ten minutes of pulling to the
:latesttag. That is the wrong target, and it makes the tag look like a throughput problem when the measured cost belongs elsewhere.Those workers were freshly provisioned, so their image cache was empty. A cold cache downloads every layer whatever the tag says, and
origin-csi-node-driver-registrarwould have taken the same 2m16.833s and 4m1.224s pinned by digest. The measurement stands as a fact about #3576. It is not evidence about the tag.What the tag costs, in order of how much it matters.
A cluster cannot be reproduced. What runs on a tenant installed in March and on one installed today differs with nothing in git to say how, and rolling a release back does not roll these back.
Every pod start goes to the registry, because
:latestmakes kubelet defaultimagePullPolicytoAlwaysand none of the seven sets a policy explicitly.That default is what hurts an air-gapped or mirror-only cluster. Under
Alwaysa node that already holds the layers still fails to start the container when the registry is unreachable, since kubelet does not fall back to the local copy. So baking these images into a golden worker image buys nothing while the tag stays. Redirecting the pull is a separate lever that works regardless, as Talosmachine.registries.mirrorsaccepts a*catch-all; what the tag takes away is the ability to state which content was mirrored or baked.Two things widen the scope.
The class is in the test harness too.
hack/e2e-chainsaw/_lib/run-kubernetes.sh:758runsimage: busyboxwith no tag, which is:latestimplicitly, inside a 10m wait whose own comment at:770already blames it:host CDI prime PVC + tenant CSI mount + busybox pull worst-case bursts past 5m.These pulls are anonymous, from shared runner addresses, so a Docker Hub 429 is a hard e2e failure rather than a slow one. The tree has the precedent for the fix: the ouroboros kubectl sidecar moved off Docker Hub to
mirror.gcr.iofor this exact reason in #2807, backported in #2835 and recorded indocs/changelogs/v1.4.3.md, and PR base-image pulls were routed the same way in #3042.Last, on the pull-cost side rather than the pinning side,
hack/e2e-prepull-images.shalready exists because "a HelmRelease can time out while one node is still pulling", which is #3576 in one sentence. It runs against the host cluster, and the sidecars that timed out are in the tenant.fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases (#…