Repository navigation
flux-plunger, keycloak-operator, kilo and redis-operator have in-repo image builds no CI path ever runs #3416
Description
Summary
The root Makefile build: recipe is the single build list every pipeline iterates: hack/build-matrix.sh parses it for the PR/main build matrices, and release-prep (.github/workflows/tags.yaml) runs plain make build. Four packages ship a Makefile image: target that builds, pushes and stamps a ref into the tree, but are absent from that list — so no CI path (PR build, main build, release-prep, nightly) ever rebuilds them. Their pins are frozen until someone rebuilds by hand.
Evidence
Checked at 91c076137 (head of #3404; same on main). Each package below has an image: target in packages/system/<pkg>/Makefile; none appears in the root Makefile build: recipe, and git grep over the Makefile and .github/workflows/ finds no other invocation.
| Package | Committed pin | Last refreshed |
|---|---|---|
flux-plunger |
flux-plunger:latest@sha256:6a6ec9… |
2026-01-09, at introduction (2d1c8aa) — never since |
keycloak-operator |
keycloak-operator:latest@sha256:272792… |
2026-03-10, manual update (cfd57f8) |
kilo |
kilo:v1.5.0@sha256:fe837f… |
2026-06-23, manual rebuild (1b4c8b6) |
redis-operator |
redis-operator:v1.3.0-rc1@sha256:a4012e… |
2025-10-23 (a545ff3) — frozen at a v1.3.0 release candidate for 9 months |
Consequences
- A source fix merged to any of these images stays inert forever — this is strictly worse than objectstorage-controller: merged BucketClaim requeue fix (#3034) is not deployed — pinned digest predates it (harbor E2E flake + systemic gap) #3143, which covers digests going stale between releases for packages that are in the list; these are never repinned even by release-prep.
redis-operatorhas carried an rc tag across multiple stable releases, and two images are pinned vialatest@…— both violate the tag-accuracy contract indocs/agents/image-refs.md(fix(ci): cover every image-ref storage shape in promote, retag and mirror #3404): the digest keeps them reproducible, but the tag misrepresents what runs and no:vX.Y.Zconvention lookup can find them.
Non-goal
kubeovn is also absent from the build list, but deliberately: its image is built and pushed by cozystack/kubeovn-chart. This issue covers only packages whose build lives in-repo and is never invoked.
Options
- A — add all four to the
build:list. They become ordinary build units, repinned at every release-prep. Notekeycloak-operator's stamp writes only.image.tag, leaving the siblingregistry:key untouched — fine for release builds, but it interacts with the nightly split-host rewrite gap documented indocs/agents/image-refs.md. - B — move a build out of the repo and treat the package as externally-built like kubeovn, with a component-versioned pin.
- C — per-package decision, but at minimum get
redis-operatoroff the rc pin.
Related
- fix(ci): cover every image-ref storage shape in promote, retag and mirror #3404 — documents the invariant this violates (
docs/agents/image-refs.md). - objectstorage-controller: merged BucketClaim requeue fix (#3034) is not deployed — pinned digest predates it (harbor E2E flake + systemic gap) #3143 — the adjacent gap: packages that are in the list still go stale between releases.
Activity
docs(agents): correct issue ref, mirror claim and example in image-refs
lexfrei commentedon Aug 4, 2026
Another source fix just landed in this trap, and the history shows it is not the first.
packages/system/keycloak-operatornow carries a patch that makes the EDP operator stop deleting a user's realm roles whenspec.rolesis unset (upstream report: epam/edp-keycloak-operator#371). The Dockerfile applies everything inpatches/, so the fix is correct in the tree and unreachable in every cluster:hack/build-matrix.shreturns an empty matrix for a change to that directory, and the package is absent from the rootbuild:list, so no workflow rebuilds it.Two earlier commits went the same way without anyone noticing.
9c5fbb69amoved the builder to Go 1.26 and5b9b420ffbumped the base image digests, both editing the Dockerfile and neither updating the pin. The image running today is therefore built from an older Go on older base images than the tree describes, which makes this more than a stale-feature problem: security bumps to these four packages are silently not shipping.Not asking for a decision here, just adding weight. The choice between adding the packages to
build:and dropping the in-repo build is still open, and the split-host mirror gap that option A wakes up is real, so it wants doing deliberately rather than as a side effect of an unrelated patch.fix(keycloak): keep KeycloakRealmUser roles when spec.roles is unset (#…
lexfrei commentedon Sep 2, 2026
One more for the list:
packages/system/kubeovnstamps a first-party reference into its values (threeyq -isites in its Makefile) and has no entry in the rootbuild:target either, so it sits in the same class as flux-plunger, keycloak-operator and kilo. Found by computing the set mechanically (packages that stamp minus packages the build list iterates) rather than by reading the enumeration in docs/agents/image-refs.md, which names three; that sentence is what drifted, and a hand-maintained list will drift again, so the doc might better point at the computation. redis-operator leaves this list with #2729, which registers its image build.lexfrei commentedon Sep 27, 2026
This becomes a release blocker with #4505. That PR fails a pre-release when a pinned image is not an amd64+arm64 index, and it builds the arm64 half only for packages in the root
build:list. flux-plunger and keycloak-operator are still outside that list, and their pins are single amd64 manifests, so the first rc after #4505 would fail on them. kilo is outside the list too, but its hand-built pin is already a two-arch index, so it passes for now. redis-operator is in the list since then. I'm preparing a PR that adds the remaining three, and #4505 waits for it.