Skip to content

fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver - #3386

Merged
myasnikovdaniil merged 1 commit into
mainfrom
fix/kubevirt-csi-centos-digest
Jul 21, 2026
Merged

myasnikovdaniil merged 1 commit into
mainfrom
fix/kubevirt-csi-centos-digest

Conversation

@androndo

@androndo Andrey Kolkov (androndo) commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

quay.io/centos/centos:stream9 is a rolling tag — upstream deletes the previously-pinned digest whenever it publishes a new stream9 build. The digest re-pinned in ce4b263 (sha256:a93e3316…) has now itself been deleted upstream, so every kubevirt-csi-driver image build fails:

#3 ERROR: quay.io/centos/centos:stream9@sha256:a93e3316…: not found
ERROR: failed to solve: … failed to resolve source metadata … not found
make: *** [Makefile:43: image-kubevirt-csi-driver] Error 1

This breaks the Build packages/apps/kubernetes job on main and on every open PR (e.g. observed on #3383).

Re-pins to the current stream9 manifest-list digest sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926, resolved via docker buildx imagetools inspect quay.io/centos/centos:stream9.

Note: this is the second recurrence of the same class of breakage (ce4b263 fixed the first). Because stream9 is a rolling tag, a pinned digest will keep rotting; a follow-up could track it via renovate or pin a date-stamped/immutable tag instead. Out of scope here — this just unblocks CI.

NONE

Summary by CodeRabbit

  • Chores
    • Updated the container build to use a refreshed CentOS Stream 9 base image.
    • No end-user-facing functionality or behavior changed.

…i-driver

quay.io/centos/centos:stream9 is a rolling tag; upstream deletes the pinned
digest whenever it publishes a new stream9 build. The digest re-pinned in
ce4b263 (sha256:a93e3316...) has now itself been deleted, so every
kubevirt-csi-driver image build fails with:

  quay.io/centos/centos:stream9@sha256:a93e3316...: not found
  failed to resolve source metadata ...
  make: *** [image-kubevirt-csi-driver] Error 1

breaking the "Build packages/apps/kubernetes" job on main and every PR.

Re-pin to the current stream9 manifest-list digest
(sha256:3714c89f1903dac5a5e1eb6c02d84189ff6d962a0c18df01feba0a5406856926),
resolved via `docker buildx imagetools inspect quay.io/centos/centos:stream9`.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Signed-off-by: Andrey Kolkov <[email protected]>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a CI breakage where the kubevirt-csi-driver image build was failing due to a missing digest for the rolling centos:stream9 tag. By re-pinning to the current valid manifest-list digest, the build process is restored.

Highlights

  • Image Digest Update: Updated the quay.io/centos/centos:stream9 image digest in the Dockerfile to resolve build failures caused by the upstream removal of the previously pinned digest.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/XS This PR changes 0-9 lines, ignoring generated files labels Jul 21, 2026
@dosubot dosubot Bot added the area/build Issues or PRs related to image build infrastructure, multi-arch support label Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a02a1bd9-890c-4b34-947d-3dd1520800d9

📥 Commits

Reviewing files that changed from the base of the PR and between f5227ee and 9c3bae0.

📒 Files selected for processing (1)
  • packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile

📝 Walkthrough

Walkthrough

The KubeVirt CSI driver Dockerfile’s second-stage CentOS Stream 9 base image digest was updated. Build steps, installed packages, copied artifacts, and the entrypoint remain unchanged.

Changes

KubeVirt CSI driver image

Layer / File(s) Summary
Update base image digest
packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile
The second-stage FROM instruction now references a different CentOS Stream 9 image digest.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: lexfrei

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the digest re-pin for kubevirt-csi-driver and matches the main change in the Dockerfile.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/kubevirt-csi-centos-digest

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the base image digest for the quay.io/centos/centos:stream9 image in the kubevirt-csi-driver Dockerfile. There are no review comments, and I have no feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Unfortunately, it is known issue #3305.

@myasnikovdaniil
myasnikovdaniil merged commit 1b777d8 into main Jul 21, 2026
19 of 20 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the fix/kubevirt-csi-centos-digest branch July 21, 2026 08:47
myasnikovdaniil added a commit that referenced this pull request Jul 21, 2026
…almalinux (#3400)

## What this PR does

`packages/apps/kubernetes/images/kubevirt-csi-driver/Dockerfile` pinned
its runtime stage to `quay.io/centos/centos:stream9` by digest. Quay
serves **only currently-tagged manifests** — when a rolling tag moves,
the superseded digest immediately returns `MANIFEST_UNKNOWN`, with no
grace period.

`stream9` rolls roughly every six days, so the pin has already died
twice:

| Date | Fix | Digest pinned | Status today |
|---|---|---|---|
| Jul 15 | #3303 | `0c9db821…` → `a93e3316…` | `0c9db821` gone |
| Jul 21 | #3386 | `a93e3316…` → `3714c89f…` | `a93e3316` gone |

Each death broke `make build` on `main` and on every PR triggering a
full rebuild. Because the root `build` target is serial, a failure in
`packages/apps/kubernetes` (4th of 29) aborts the remaining packages and
leaves the shared build cache unwritten.

Repinning cannot fix this: the freshly pinned digest is always the
current live tag, so it is next in line to be retired. This is not
specific to CentOS — the same probe against
`quay.io/fedora/fedora:latest` and `quay.io/centos/centos:stream10`
found every historical digest unfetchable.

This PR keeps the digest pin and changes the *source* to a registry that
provably retains superseded manifests, making the pin durable rather
than decorative. Renovate policy is unchanged — `pinDigests` stays on,
and `almalinux:9` is an ordinary rolling tag on a retaining registry
exactly like the `golang`, `alpine` and `debian` bases already in the
grouped digest PR.

### Verification

Both bases were run locally with the Dockerfile's exact install line
(`dnf install -y e2fsprogs xfsprogs nfs-utils`):

| Binary | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `mkfs.ext4` | `/usr/sbin/mkfs.ext4` | `/usr/sbin/mkfs.ext4` |
| `mkfs.xfs` | `/usr/sbin/mkfs.xfs` | `/usr/sbin/mkfs.xfs` |
| `blkid` | `/usr/sbin/blkid` | `/usr/sbin/blkid` |
| `mount` | `/usr/bin/mount` | `/usr/bin/mount` |
| `umount` | `/usr/bin/umount` | `/usr/bin/umount` |
| `fsck.ext4` | `/usr/sbin/fsck.ext4` | `/usr/sbin/fsck.ext4` |
| `fsck.xfs` | `/usr/sbin/fsck.xfs` | `/usr/sbin/fsck.xfs` |
| `mount.nfs` | `/usr/sbin/mount.nfs` | `/usr/sbin/mount.nfs` |

Every path `k8s.io/mount-utils` shells out to is identical. Package
versions:

| Package | CentOS Stream 9 | AlmaLinux 9 |
|---|---|---|
| `e2fsprogs` | 1.46.5-8.el9 | 1.46.5-8.el9 |
| `xfsprogs` | 6.4.0-7.el9 | 6.4.0-7.el9 |
| `nfs-utils` | 2.5.4-44.el9 | 2.5.4-42.el9_8 |
| `util-linux` | 2.37.4-26.el9 | 2.37.4-25.el9 |
| `glibc` | 2.34-274.el9 | 2.34-270.el9_8 |

Same upstream versions throughout; the el9 release-number lag is
expected since AlmaLinux is downstream of RHEL 9 while CentOS Stream 9
is upstream. `e2fsprogs` and `xfsprogs` — the two that govern
`mkfs`/`fsck` behaviour — are identical NVRs. `glibc` is not linked at
all: the driver is built `CGO_ENABLED=0`.

The pinned digest is the OCI image index covering `amd64`, `arm64`,
`ppc64le` and `s390x` — the same architecture set as the previous base.

Retention was verified empirically rather than assumed: four superseded
`docker.io/library` digests taken from this repo's own git history
(`golang:1.26`, `golang:1.26-alpine`, `alpine:3.24`, `alpine:3.22`) all
still resolve after their tags moved.

### Scope

This addresses the immediate recurring breakage only. The broader
concern in #3305 — that any retired digest blocks the entire platform
build, with no advance warning and no fast path to repin — is not solved
here and remains open.

### Release note

```release-note
fix(kubernetes): move the kubevirt-csi-driver runtime base from quay.io/centos/centos:stream9 to docker.io/library/almalinux:9. Quay retires superseded manifests as soon as a rolling tag moves, which broke builds twice in six days; AlmaLinux 9 provides an equivalent RHEL 9 userspace on a registry that retains them, so the digest pin remains valid over time. No functional change to the driver.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated the container runtime environment to use AlmaLinux 9 instead
of CentOS Stream 9.
* Kubernetes CSI driver behavior and startup configuration remain
unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
pull Bot pushed a commit to medampudi/cozystack that referenced this pull request Jul 21, 2026
…almalinux

quay.io serves only currently-tagged manifests: when a rolling tag moves,
the superseded digest immediately returns MANIFEST_UNKNOWN. The pinned
`centos:stream9` digest has died twice in six days (cozystack#3303, cozystack#3386), each
time breaking `make build` on main and on every PR that triggers a full
rebuild.

Repinning is not a fix -- the newly pinned digest is always the live tag,
so it is next in line to be retired. Move the runtime stage to a registry
that provably retains superseded manifests instead, which makes the digest
pin durable rather than decorative. Pinning policy is unchanged.

Verified equivalent: after `dnf install -y e2fsprogs xfsprogs nfs-utils`,
both bases provide mkfs.ext4, mkfs.xfs, blkid, mount, umount, fsck.ext4,
fsck.xfs and mount.nfs at identical paths. e2fsprogs and xfsprogs are
identical NVRs. The driver is built CGO_ENABLED=0, so the base libc is not
linked. The pinned digest is the OCI index covering amd64, arm64, ppc64le
and s390x -- the same architecture set as the previous base.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
myasnikovdaniil added a commit that referenced this pull request Jul 22, 2026
## What this PR does

Adds `docs/changelogs/v1.6.0.md` (667 lines), generated locally with the
process in `docs/agents/changelog.md`. Normally `tags.yaml`'s
`generate-changelog` job produces this on the stable tag push, but that
job never ran for `v1.6.0` — see the note below.

**Coverage boundary.** The changelog runs through #3386, i.e. the
release period ending at the `v1.6.0-rc.4` cut. Roughly eight PRs merged
to `main` after that cut are deliberately absent, because `v1.6.0` is a
promotion rather than a rebuild: every shipped artifact is pinned to an
rc.4 digest (`cozystack-packages@sha256:bf682087…`,
`cozystack-operator@sha256:93b305f7…`), so that later code is in the
tagged tree but not in anything users install. The changelog describes
what shipped.

**Why this is a manual PR.** `generate-changelog` and
`update-website-docs` are jobs in `tags.yaml`, triggered by `push:
tags`. The `v1.6.0` tag was pushed by `pull-requests-release.yaml`'s
finalize job using the `cozystack-ci` app token, and no workflow run was
created for that ref — every rc tag, pushed by `cut-prerelease.yaml`
with the same app, did trigger it. `v1.6.0` is the first stable tag
created through the promote flow, so this path had never run before. The
cause is not yet established and needs a repro rather than a guess; the
website docs versioning is still outstanding for the same reason.

### Screenshots

N/A — documentation only.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, which is a single new file under `docs/changelogs/`. No trigger
matches: the map covers package layout, release assets, CRDs, workflow
behaviour and naming contracts, none of which a changelog file touches.
The separate repository list in `docs/agents/changelog.md` was already
consumed while generating this file — it feeds the notes, it is not a
code coupling.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
  * Added comprehensive release notes for version 1.6.0.
* Documented breaking changes, upgrade prerequisites, manual upgrade
steps, and post-upgrade expectations.
* Highlighted major features, deprecations, security updates, bug fixes,
operational notes, and component version updates.
* Included configuration, CI, development, and related-project changelog
updates.
  * Added contributor acknowledgements and a link to the full changelog.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Issues or PRs related to image build infrastructure, multi-arch support area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/XS This PR changes 0-9 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants