Repository navigation
fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver - #3303
Conversation
IvanHunters
added
kind/backport
github-actions
Bot
added
area/kubernetes
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesKubeVirt CSI driver runtime
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
the
size/XS
…i-driver quay.io/centos/centos:stream9 is a rolling tag; upstream deleted the pinned digest (sha256:0c9db821...) when it published a new stream9 build, so every build of image-kubevirt-csi-driver fails with "not found" — on open PRs and on main. Re-pin to the current live stream9 digest (sha256:a93e3316...). This is an emergency CI unblock, not a durable fix: stream9 is garbage-collected, so upstream will delete this digest too on its next rebuild and the same "not found" breakage recurs until the pin is bumped again — the same operation Renovate's weekly, non-automerged digest PR performs. A durable fix (mirror the base image into cozystack's own registry, or move to an immutable-digest base such as UBI9) is out of scope for this hotfix and left as follow-up. Signed-off-by: Ivan Okhotnikov <[email protected]>
37559e0 to
ce4b263
Compare
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical CI failure where the kubevirt-csi-driver image build was failing due to a missing base image digest. By re-pinning the Dockerfile to a currently available digest for the centos:stream9 image, the build process is restored. This is a temporary hotfix to unblock CI, with plans for a more robust solution to handle rolling tags in the future. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
dosubot
Bot
added
the
area/build
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the base image digest for quay.io/centos/centos:stream9 in the kubevirt-csi-driver Dockerfile. There are no review comments, and I have no feedback to provide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-3303-to-release-1.5
git worktree add --checkout .worktree/backport-3303-to-release-1.5 backport-3303-to-release-1.5
cd .worktree/backport-3303-to-release-1.5
git reset --hard HEAD^
git cherry-pick -x ce4b263ca6841f0da4084c38008c4f8b29a0ade3
git push --force-with-lease |
|
Created backport PR for
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-3303-to-release-1.4
git worktree add --checkout .worktree/backport-3303-to-release-1.4 backport-3303-to-release-1.4
cd .worktree/backport-3303-to-release-1.4
git reset --hard HEAD^
git cherry-pick -x ce4b263ca6841f0da4084c38008c4f8b29a0ade3
git push --force-with-lease |
|
previous versions does not this digest pin enforcement so backport not needed |
…almalinux (#3400) ## What this PR does `packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile` pinned its runtime stage to `quay.io/centos/centos:stream9` by digest. Quay serves **only currently-tagged manifests** — when a rolling tag moves, the superseded digest immediately returns `MANIFEST_UNKNOWN`, with no grace period. `stream9` rolls roughly every six days, so the pin has already died twice: | Date | Fix | Digest pinned | Status today | |---|---|---|---| | Jul 15 | #3303 | `0c9db821…` → `a93e3316…` | `0c9db821` gone | | Jul 21 | #3386 | `a93e3316…` → `3714c89f…` | `a93e3316` gone | Each death broke `make build` on `main` and on every PR triggering a full rebuild. Because the root `build` target is serial, a failure in `packages/apps/kubernetes` (4th of 29) aborts the remaining packages and leaves the shared build cache unwritten. Repinning cannot fix this: the freshly pinned digest is always the current live tag, so it is next in line to be retired. This is not specific to CentOS — the same probe against `quay.io/fedora/fedora:latest` and `quay.io/centos/centos:stream10` found every historical digest unfetchable. This PR keeps the digest pin and changes the *source* to a registry that provably retains superseded manifests, making the pin durable rather than decorative. Renovate policy is unchanged — `pinDigests` stays on, and `almalinux:9` is an ordinary rolling tag on a retaining registry exactly like the `golang`, `alpine` and `debian` bases already in the grouped digest PR. ### Verification Both bases were run locally with the Dockerfile's exact install line (`dnf install -y e2fsprogs xfsprogs nfs-utils`): | Binary | CentOS Stream 9 | AlmaLinux 9 | |---|---|---| | `mkfs.ext4` | `/usr/sbin/mkfs.ext4` | `/usr/sbin/mkfs.ext4` | | `mkfs.xfs` | `/usr/sbin/mkfs.xfs` | `/usr/sbin/mkfs.xfs` | | `blkid` | `/usr/sbin/blkid` | `/usr/sbin/blkid` | | `mount` | `/usr/bin/mount` | `/usr/bin/mount` | | `umount` | `/usr/bin/umount` | `/usr/bin/umount` | | `fsck.ext4` | `/usr/sbin/fsck.ext4` | `/usr/sbin/fsck.ext4` | | `fsck.xfs` | `/usr/sbin/fsck.xfs` | `/usr/sbin/fsck.xfs` | | `mount.nfs` | `/usr/sbin/mount.nfs` | `/usr/sbin/mount.nfs` | Every path `k8s.io/mount-utils` shells out to is identical. Package versions: | Package | CentOS Stream 9 | AlmaLinux 9 | |---|---|---| | `e2fsprogs` | 1.46.5-8.el9 | 1.46.5-8.el9 | | `xfsprogs` | 6.4.0-7.el9 | 6.4.0-7.el9 | | `nfs-utils` | 2.5.4-44.el9 | 2.5.4-42.el9_8 | | `util-linux` | 2.37.4-26.el9 | 2.37.4-25.el9 | | `glibc` | 2.34-274.el9 | 2.34-270.el9_8 | Same upstream versions throughout; the el9 release-number lag is expected since AlmaLinux is downstream of RHEL 9 while CentOS Stream 9 is upstream. `e2fsprogs` and `xfsprogs` — the two that govern `mkfs`/`fsck` behaviour — are identical NVRs. `glibc` is not linked at all: the driver is built `CGO_ENABLED=0`. The pinned digest is the OCI image index covering `amd64`, `arm64`, `ppc64le` and `s390x` — the same architecture set as the previous base. Retention was verified empirically rather than assumed: four superseded `docker.io/library` digests taken from this repo's own git history (`golang:1.26`, `golang:1.26-alpine`, `alpine:3.24`, `alpine:3.22`) all still resolve after their tags moved. ### Scope This addresses the immediate recurring breakage only. The broader concern in #3305 — that any retired digest blocks the entire platform build, with no advance warning and no fast path to repin — is not solved here and remains open. ### Release note ```release-note fix(kubernetes): move the kubevirt-csi-driver runtime base from quay.io/centos/centos:stream9 to docker.io/library/almalinux:9. Quay retires superseded manifests as soon as a rolling tag moves, which broke builds twice in six days; AlmaLinux 9 provides an equivalent RHEL 9 userspace on a registry that retains them, so the digest pin remains valid over time. No functional change to the driver. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the container runtime environment to use AlmaLinux 9 instead of CentOS Stream 9. * Kubernetes CSI driver behavior and startup configuration remain unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…almalinux quay.io serves only currently-tagged manifests: when a rolling tag moves, the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned `centos:stream9` digest has died twice in six days (cozystack#3303, cozystack#3386), each time breaking `make build` on main and on every PR that triggers a full rebuild. Repinning is not a fix -- the newly pinned digest is always the live tag, so it is next in line to be retired. Move the runtime stage to a registry that provably retains superseded manifests instead, which makes the digest pin durable rather than decorative. Pinning policy is unchanged. Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`, both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4, fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le and s390x -- the same architecture set as the previous base. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
3 participants
What this PR does
packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfilepins its runtime base image toquay.io/centos/centos:stream9by digest.stream9is a rolling tag, and upstream deleted the pinned digest (sha256:0c9db821…) when it published a fresh stream9 build. As a resultmake image-kubevirt-csi-drivernow fails withnot foundon every build — open PRs andmainalike (theBuild packages/apps/kubernetesjob goes red before E2E even starts).Re-pin to the current live
stream9digest (sha256:a93e3316…, verified pullable and multi-arch: amd64/arm64/ppc64le/s390x). No Dockerfile logic changes.Known limitation and follow-up
This is an emergency CI unblock, not a durable fix.
stream9is a rolling, garbage-collected tag: upstream deletes old digests on every rebuild, so the samenot foundbreakage will recur the next time CentOS rolls the tag faster than the pin is bumped. This is the same operation Renovate's weekly (non-automerged) digest PR performs.A durable fix is out of scope for this hotfix and left as follow-up:
registry.access.redhat.com/ubi9), orDownstream repositories
Release note
Summary by CodeRabbit