Repository navigation
ci: guard against pinned base-image digests disappearing upstream #3305
Description
Activity
Investigated this. Summary of findings, then a concrete proposal — the second half is the part that actually addresses the durable problem, and it is not what the issue originally proposed.
Findings
1. Quay retirement is instantaneous, not a delayed GC. The superseded manifest stops resolving the moment the tag moves. From Quay's tag-history API for
centos/centos:stream9:sha256:3714c89f… start 2026-07-21T02:04:01Z (active) sha256:a93e3316… start 2026-07-15T03:20:27Z end 2026-07-21T02:04:01Z → MANIFEST_UNKNOWN sha256:0c9db821… start 2026-07-07T00:49:47Z end 2026-07-15T03:20:27Z → MANIFEST_UNKNOWNMeasured gap between the tag move on July 15 and the first red build: 5h40m.
This is not CentOS-specific. The same probe against
quay.io/fedora/fedora:latest(4/4 historical digests gone) andquay.io/centos/centos:stream10(3/3 gone) shows the same behaviour, so it should be treated as a property of the registry, not a one-off.2. It has already recurred. #3303 (Jul 15) and #3386 (Jul 21) are byte-identical fixes six days apart.
stream9rolls every 6–8 days. The digest merged in #3386 is currently the live tag, so it is next in line to be retired.3. The blast radius is one reference. Of the digest-pinned bases in the repo, everything on
docker.io/libraryandgcr.io/distrolessprovably retains superseded manifests — verified by taking four superseded digests out of this repo's own git history (golang:1.26,golang:1.26-alpine,alpine:3.24,alpine:3.22) and confirming they all still resolve after their tags moved. The only pin on a non-retaining registry wascentos:stream9. The two remaining Quay pins (poseidon/matchbox:v0.11.0,cilium/cilium:v1.19.5) are immutable version tags that do not roll.4. A faster Renovate schedule cannot fix this. Three independent reasons: the exposure window opens at T+0 so no cron frequency avoids it;
stream9rolls Tue/Wed while the digest group is scheduledbefore 6am on monday, i.e. anti-phased; and withdo-not-merge/holdplusautomerge: false, recovery latency is human review, not cron frequency. Both repins in practice landed faster (9h and 6h43m) than any schedule change would have delivered.5.
release-1.4andrelease-1.5were never affected because they still carry the unpinnedFROM quay.io/centos/centos:stream9. Closing the backports #3309/#3310 was correct.Immediate fix
#3400 moves the runtime base to
docker.io/library/almalinux:9, keeping the digest pin. Verified equivalent by running both bases with the Dockerfile's exact install line: all eight mount binaries at identical paths,e2fsprogs/xfsprogsat identical NVRs, same four-architecture index, and the driver is builtCGO_ENABLED=0so the base libc is not linked.Explicitly not proposed: unpinning, or mirroring bases into a project-controlled registry. Unpinning would trade a supply-chain property for an availability one on a privileged CSI node image. Mirroring is real infrastructure — script, workflow, schedule, retention carve-out, ongoing ownership — to preserve access to a single base that has a one-line alternative.
What #3400 does not solve
It lowers the probability of a dead pin but does nothing about impact or recovery latency, which is the durable half of this issue:
Any dead digest still blocks the entire platform build. The root
buildtarget runs 29 packages serially, andpackages/apps/kubernetesis 4th. One unresolvable base aborts the remaining 25 packages and leaves the sharedmode=maxbuild cache unwritten, so subsequent PRs are slower as well. This is true for any package, not just this one.Recovery is gated by design.
do-not-merge/holdandautomerge: falseare deliberate policy and should stay, but they mean a repin cannot land quickly even once the problem is known.Suggested follow-ups, in order of leverage:
- Do not let one package abort the whole serial build on
main. Collect failures and continue, so a dead base costs one package rather than the platform. Release tag builds should keep failing hard — a partial platform must not ship. - A documented emergency override (an explicit build-arg, opt-in, never automatic) so a maintainer can unblock builds within minutes while the proper repin goes through normal review. An automatic digest-to-tag fallback should be rejected — it silently defeats the pin exactly when the pin matters.
- A scheduled resolve check scoped to pins on non-retaining registries — after fix(kubernetes): move kubevirt-csi-driver base off quay.io/centos to almalinux #3400 that is two references, so it is cheap. The value is lead time, not detection: it should open the repin PR before anyone needs to build, rather than reporting a breakage that is already loud.
- A registry allowlist check on changed
FROMlines, so a future base on a registry that retires manifests fails CI at review time instead of six days later.
Happy to split these into separate PRs if the direction looks right. Item 1 is the one that actually converts "the platform is blocked" into "one package is blocked", so I would start there.
- Do not let one package abort the whole serial build on
- added a commit that references this issue
on Jul 21, 2026 - changed the title
[-]ci: prevent retired base-image digests from blocking builds[/-][+]ci: guard against pinned base-image digests disappearing upstream[/+]on Aug 14, 2026 Rewrote this as a guard specification. The two merged fixes addressed one incident rather than the class: #3386 re-pinned a single digest and #3400 moved that one image to a registry that retains manifests. Neither added a check, so every other pin still fails the same way, a
MANIFEST_UNKNOWNinside an unrelated PR's build.The incident also recurred, which the original body did not capture: #3303 re-pinned on 2026-07-15, the newly pinned digest was deleted too, and #3386 re-pinned again on 2026-07-21 before #3400 moved off quay.io the same day. Two blocking incidents in six days, both fixed by hand.
Current exposure, verified with
crane manifeston 2026-08-13: 17 unique digest-pinned bases across 41 Dockerfiles, all resolving. Two sit on quay.io (poseidon/matchbox:v0.11.0,cilium/cilium:v1.19.5), both on immutable-style release tags rather than a rolling one, which is lower likelihood than thestream9case but the same failure mode. The retirement asymmetry is measured: the retiredcentos:stream9digest is stillMANIFEST_UNKNOWNtoday, while three of our owndocker.io/librarypins are already behind their tag head and still resolve.Scoped the proposal down from the original mirror-and-repin plan to a liveness guard in two existing lanes, a
base-digestsjob innightly.yamlfor daily detection and the same script from thechecksjob on Dockerfile-touching PRs, because two incidents in six days and zero in the 23 days since #3400 does not justify standing up a mirror registry and a sync job. Keeping this open: the guard does not exist yet, and one third-party ref (seaweedfs-cosi-driver.tag) is not digest-pinned at all, so it sits outside both Renovate'spinDigestsrule and any digest check.- added and removed
on Sep 23, 2026
Problem
Dockerfile base images are pinned to immutable digests for reproducibility and for OpenSSF Scorecard Pinned-Dependencies (
.github/renovate.jsonsetspinDigests: truefor thedockerfileandgithub-actionsmanagers). Some registries retire the superseded manifest as soon as a tag moves, so a pinned digest can stop resolving with no warning and no diff, and the failure lands inside whatever unrelated PR happens to rebuild that package next.This has already happened twice, on the same image, six days apart:
quay.io/centos/centos:stream9@sha256:0c9db821…returnedMANIFEST_UNKNOWNand blocked unrelated PR builds atpackages/apps/kubernetes(failing run). Fixed by hand in fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver #3303.docker.io/library/almalinux.Neither fix added a check, so the exposure is unchanged for every other pin in the tree.
Why a guard rather than a mirror
The original version of this issue proposed mirroring volatile bases into a project-controlled registry and pinning the mirrored digest. That is more machinery than the measured failure rate supports: two incidents in six days on one image, zero in the 23 days since #3400 removed the last rolling-tag pin on a retiring registry. A mirror means a registry to keep, a sync job to maintain, and a rewrite of every
FROM. What #3400 did instead, move to a registry that retains, cost one line and solved the real case.So the durable part of this issue is detection, not indirection. Prefer a retaining registry when picking a base, and have CI tell us when a pin is gone instead of letting a PR author discover it.
What is exposed today
38
Dockerfilefiles plus 1Containerfileunderpackages/*/*/images/pin 17 unique base refs. Verified withcrane manifeston 2026-08-13: all 17 resolve. Nothing is broken right now.By registry:
quay.ioquay.io/centos/centos:stream9@sha256:0c9db821…is stillMANIFEST_UNKNOWNtoday while the tag headssha256:04bad0b0…docker.ionon-librarygcr.iodistroless/static:nonroot@sha256:d29e660c…is behind headsha256:f7f8f729…and still resolvesdocker.io/librarygolang:1.26(pinned079e5980…, head705e964a…),debian:bookworm(30482e87…/813017f3…),node:22-alpine(16e22a55…/c610fcdf…)registry.k8s.ioThe two
quay.iopins:quay.io/poseidon/matchbox:v0.11.0@sha256:06bcdae8…, atpackages/core/talos/images/matchbox/Dockerfile:1quay.io/cilium/cilium:v1.19.5@sha256:20fbbc14…, atpackages/system/cilium/images/cilium/Dockerfile:1Both sit on immutable-style release tags rather than a rolling tag like
stream9, so upstream would have to re-push a released version for them to break. That is lower likelihood, not a different failure mode.Also in scope, from
packages/*/*/images/*.tag(13 of the 18 files hold first-party refs the build stamps, 5 hold third-party ones, carrying 3 unique refs):docker.io/alpine/k8s:1.33.4@sha256:b0523f0a…inkubernetes-nodes/images/kubectl.tag,kubernetes/images/kubectl.tag,monitoring/images/kubectl.tagdocker.io/library/busybox:1.37.0@sha256:1487d0af…inkubernetes/images/busybox.tagghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1inextra/seaweedfs/images/seaweedfs-cosi-driver.tag, not digest-pinned, so Renovate'spinDigestsrule cannot reach it (it covers thedockerfileandgithub-actionsmanagers only) and a digest liveness check has nothing to testUnpinned bases, which cannot 404 but can change content with no diff, listed for completeness rather than as work:
gcr.io/distroless/static:nonroot-${TARGETARCH}, theARG BASEIMAGEdefault atpackages/apps/kubernetes/images/cluster-autoscaler/Dockerfile:3.packages/apps/kubernetes/Makefilepasses no--build-arg BASEIMAGE, so this default is what builds.quay.io/metallb/controller:${VERSION}and:speaker, atpackages/system/metallb/images/metallb/Dockerfile:5,7, withVERSIONfrom the vendored chart'sappVersion.kubeovn/kube-ovn-base:$BASE_TAGandgolang:1.26-bookworm, atpackages/system/kubeovn/images/kubeovn/Dockerfile:5,19,39.Proposed guard
What it checks
A script,
hack/check-base-digests.sh, POSIX sh, that:FROM …@sha256:…andARG …=…@sha256:…lines inpackages/*/*/images/**/Dockerfileand**/Containerfile, plus refs inpackages/*/*/images/*.tagwhose repository is notghcr.io/cozystack/cozystack/*. Deduplicates by ref: today that is 19 digest-pinned refs to resolve (17 Dockerfile bases plusalpine/k8sandbusyboxfrom the.tagfiles), spread across many more occurrence sites, plus one tag-only ref to flag as unpinned.crane manifestorskopeo inspect --raw, whichever the lane already has, andnightly.yamlalready installs skopeo).file:line, and exits non-zero when any ref is definitively gone.Fail-open discipline, stated because getting it wrong here is worse than not having the guard. A
MANIFEST_UNKNOWNor a 404 is a failure. A timeout, a 5xx, a rate limit or an auth error is not a failure, but it must be reported as::warning::and the summary must say how many refs went unchecked. The guard must never print a clean result when a lookup did not complete, because a check that fails open silently reads exactly like a check that passed.Where it runs
Two places, both existing lanes. No new workflow file: this repo prefers extending a lane and asks for a measured failure frequency first, and two incidents in six days does not justify standing up a workflow of its own.
Scheduled detection, a
base-digestsjob in.github/workflows/nightly.yaml. That workflow already runscron: '0 2 * * *'and already installs skopeo in itsmirrorjob. The new job needs no secrets (every base is anonymously pullable), no build, and runs onubuntu-latestin seconds. It gates nothing, it fails so the nightly goes red and names the ref and itsfile:linein a::error::annotation. Detection window drops from "whenever something next rebuilds that package" to under 24 hours.PR-time signal, invoke the same script from the existing
checksjob in.github/workflows/pull-requests.yaml, conditioned on the PR having touched aDockerfile,Containerfileor*.tag. This catches a bad digest at review time, including one a Renovate digest PR is proposing. Becausechecksis a required status, this invocation fails only on a definitiveMANIFEST_UNKNOWN/404 and treats every transport error as a pass with a warning.An offline unit test,
hack/check-base-digests.batsundermake unit-tests, covers the enumerator against fixtures: that it findsFROMandARGforms, that it skips first-partyghcr.io/cozystack/cozystack/*refs, that it deduplicates, and that a tag-only ref is reported as unpinned rather than silently dropped. Follow the harness constraints documented inhack/image-pin-consistency.bats(hack/cozytest.shis not real bats: norun, no$status, noskip, repo-root-relative paths, andenv -u REGISTRYwhere the ambient CIREGISTRYwould skew a tree scan). The network resolve stays out ofunit-testsso no PR-gating unit test acquires a network dependency.What it does not do
Also worth deciding
packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tagholdsghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1with no digest, so it is outside Renovate'spinDigestsrule and outside anything a digest check can verify. Either pin it, or record it here as a deliberate exception. The original version of this issue asked for exactly that ("document any deliberate exceptions where a rolling tag must remain unpinned") and this is the only case in the tree.Definition of done
hack/check-base-digests.shexists, is covered offline byhack/check-base-digests.batsundermake unit-tests, and reports unchecked refs rather than swallowing them..github/workflows/nightly.yamland on Dockerfile-touching PRs from the existingchecksjob.