Skip to content

flux-plunger, keycloak-operator, kilo and redis-operator have in-repo image builds no CI path ever runs #3416

Description

@myasnikovdaniil

Summary

The root Makefile build: recipe is the single build list every pipeline iterates: hack/build-matrix.sh parses it for the PR/main build matrices, and release-prep (.github/workflows/tags.yaml) runs plain make build. Four packages ship a Makefile image: target that builds, pushes and stamps a ref into the tree, but are absent from that list — so no CI path (PR build, main build, release-prep, nightly) ever rebuilds them. Their pins are frozen until someone rebuilds by hand.

Evidence

Checked at 91c076137 (head of #3404; same on main). Each package below has an image: target in packages/system/<pkg>/Makefile; none appears in the root Makefile build: recipe, and git grep over the Makefile and .github/workflows/ finds no other invocation.

Package Committed pin Last refreshed
flux-plunger flux-plunger:latest@sha256:6a6ec9… 2026-01-09, at introduction (2d1c8aa) — never since
keycloak-operator keycloak-operator:latest@sha256:272792… 2026-03-10, manual update (cfd57f8)
kilo kilo:v1.5.0@sha256:fe837f… 2026-06-23, manual rebuild (1b4c8b6)
redis-operator redis-operator:v1.3.0-rc1@sha256:a4012e… 2025-10-23 (a545ff3) — frozen at a v1.3.0 release candidate for 9 months

Consequences

Non-goal

kubeovn is also absent from the build list, but deliberately: its image is built and pushed by cozystack/kubeovn-chart. This issue covers only packages whose build lives in-repo and is never invoked.

Options

  • A — add all four to the build: list. They become ordinary build units, repinned at every release-prep. Note keycloak-operator's stamp writes only .image.tag, leaving the sibling registry: key untouched — fine for release builds, but it interacts with the nightly split-host rewrite gap documented in docs/agents/image-refs.md.
  • B — move a build out of the repo and treat the package as externally-built like kubeovn, with a component-versioned pin.
  • C — per-package decision, but at minimum get redis-operator off the rc pin.

Related

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/buildIssues or PRs related to image build infrastructure, multi-arch support
    on Jul 22, 2026
  2. lexfrei commented on Aug 4, 2026

    @lexfrei
    Contributor

    Another source fix just landed in this trap, and the history shows it is not the first.

    packages/system/keycloak-operator now carries a patch that makes the EDP operator stop deleting a user's realm roles when spec.roles is unset (upstream report: epam/edp-keycloak-operator#371). The Dockerfile applies everything in patches/, so the fix is correct in the tree and unreachable in every cluster: hack/build-matrix.sh returns an empty matrix for a change to that directory, and the package is absent from the root build: list, so no workflow rebuilds it.

    Two earlier commits went the same way without anyone noticing. 9c5fbb69a moved the builder to Go 1.26 and 5b9b420ff bumped the base image digests, both editing the Dockerfile and neither updating the pin. The image running today is therefore built from an older Go on older base images than the tree describes, which makes this more than a stale-feature problem: security bumps to these four packages are silently not shipping.

    Not asking for a decision here, just adding weight. The choice between adding the packages to build: and dropping the in-repo build is still open, and the split-host mirror gap that option A wakes up is real, so it wants doing deliberately rather than as a side effect of an unrelated patch.

  3. lexfrei commented on Sep 2, 2026

    @lexfrei
    Contributor

    One more for the list: packages/system/kubeovn stamps a first-party reference into its values (three yq -i sites in its Makefile) and has no entry in the root build: target either, so it sits in the same class as flux-plunger, keycloak-operator and kilo. Found by computing the set mechanically (packages that stamp minus packages the build list iterates) rather than by reading the enumeration in docs/agents/image-refs.md, which names three; that sentence is what drifted, and a hand-maintained list will drift again, so the doc might better point at the computation. redis-operator leaves this list with #2729, which registers its image build.

  4. added
    area/ciIssues or PRs related to CI workflows, GitHub Actions, automation
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  5. lexfrei commented on Sep 27, 2026

    @lexfrei
    Contributor

    This becomes a release blocker with #4505. That PR fails a pre-release when a pinned image is not an amd64+arm64 index, and it builds the arm64 half only for packages in the root build: list. flux-plunger and keycloak-operator are still outside that list, and their pins are single amd64 manifests, so the first rc after #4505 would fail on them. kilo is outside the list too, but its hand-built pin is already a two-arch index, so it passes for now. redis-operator is in the list since then. I'm preparing a PR that adds the remaining three, and #4505 waits for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/buildIssues or PRs related to image build infrastructure, multi-arch supportarea/ciIssues or PRs related to CI workflows, GitHub Actions, automationkind/bugCategorizes issue or PR as related to a bugpriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions