Finish the CI image-build refactor: SBOM emission and pr-* tag retention

Status as of 2026-10-01

Checked against main at bc400dff1. One of the three "Still live" items is done (the bypassing Makefiles, #4485), multi-arch builds landed in another form than the matrix axis this epic deferred, and the runner notes changed. SBOM and OCIR pr-* retention are untouched. More below under each heading.

Status

This epic has been narrowed. The premise it was filed on has expired, and two of its three headline deliverables have shipped. What remains is listed under "Still live" below. Everything under "Landed" is closed out with proof and needs no further work.

The original background blamed a dedicated self-hosted runner (cozy-runner-1) whose single embedded buildkit contended under concurrent builds, plus a cold registry cache after #2711 repointed --cache-from at a tag that stopped being published. Both causes are gone. Every job now runs on the ephemeral Oracle VM pool, each build job creates its own docker-container buildkit, and a mode=max registry cache is warmed on every push to main. gh api repos/cozystack/cozystack/actions/runners returns total_count: 0. One workflow on main targets self-hosted on purpose, e2e-proxmox.yaml ([self-hosted, proxmox], gated behind the PROXMOX_NIGHTLY variable which is unset, not triggered by pull requests).

Landed

Phase 0, verifications:

OCIR accepts buildkit mode=max cache manifests. The cache-args macro in hack/common-envs.mk ships oci-mediatypes=true,image-manifest=true and is in production use.
A smaller shared-runner label exists and is used: oracle-vm-4cpu-16gb-x86-64 for checks, build, build-talos and finalize in .github/workflows/pull-requests.yaml.
buildx works on the shared pool: docker/setup-buildx-action with driver: docker-container in .github/workflows/pull-requests.yaml.

Phase 1, registry cache and warmer (#2938's design, landed via #2983):

--cache-to type=inline is gone. hack/common-envs.mk cache-args emits --cache-from type=registry,ref=$(CACHE_REGISTRY)/<img>:buildcache always, and --cache-to ...,mode=max only when WRITE_CACHE=1.
WRITE_CACHE defaults to 0, so PR builds are read-only against the cache and cannot race on the cache manifest.
.github/workflows/build-main.yaml warms the cache on every push to main with WRITE_CACHE: '1'. .github/workflows/build-release.yaml deliberately reads only (WRITE_CACHE: '0'), keeping build-main the single serialized writer. ci: warm the build cache on release-1.5 pushes #3469 warmed release-1.5 separately, on that branch only, via a branch-scoped CACHE_REGISTRY namespace.

Phase 2, matrix and digest merge (#2983):

plan emits the matrix from hack/build-matrix.sh, which parses the root Makefile build: target so CI cannot drift from what make build builds.
build is one job per package, fail-fast: false, each with its own docker-container buildkit. The shared-buildkit contention class is severed by construction.
Each job uploads a pr-patch-fragment-<pkg> artifact holding only its own package's digest diff, and finalize reassembles them with git apply --3way into the unified pr.patch that e2e consumes unchanged, then builds the installer last so flux push artifact --path=packages bundles an already-patched tree.
Talos is its own always-on leg (build-talos), and hack/build-matrix.sh excludes packages/core/talos and packages/core/installer from the matrix with the reasoning inline.
The parity gate against a legacy make build was run once at cutover and is not a standing test.

Phase 3, change scoping and runner cutover:

Change scoping ships, though by parsing the Makefile rather than with paths-filter. Shared-dependency fan-out is full_rebuild_pattern in hack/build-matrix.sh, plus a targeted seaweedfs to objectstorage-controller rule.
runs-on is the shared ephemeral label everywhere on main. The debug label no longer flips runs-on, it only opens the breakpoint on the runner the job already holds.
The dedicated runner is decommissioned (ci: move remaining workflows off the persistent self-hosted runner #3268). No repo-registered runner remains. The physical host and the breakpoint rendezvous server are not checked from the repo, BREAKPOINT_ENDPOINT is still read by pull-requests.yaml and e2e-tag.yaml.

Still live

Probe OCIR referrers/attestation support, then turn SBOM on. Still open on 2026-10-01, SBOM ?= 0 and no workflow sets it. hack/common-envs.mk already carries the seam: SBOM ?= 0, and --sbom=true is appended to BUILDX_ARGS when SBOM=1. It is off because an unsupported attestation push fails the build outright, and nobody has probed OCIR. This is the one blocker for every remaining SBOM item. Done means: the probe result recorded here, and either SBOM=1 set in the build workflows or a decision to deliver SBOMs as workflow artifacts instead.
~~Bring the three bypassing package Makefiles under BUILDX_ARGS.~~ Done in fix(build): pass the shared buildx flags from every image target #4485 (merged 2026-09-28). packages/system/dashboard/Makefile, packages/system/flux-plunger/Makefile and packages/system/kubeovn-plunger/Makefile append $(BUILDX_ARGS) now, so a flag added there reaches those images too.
Retention or GC for pr-* tags on OCIR. .github/workflows/retention.yaml prunes GHCR nightly container-package versions only (dated 0.0.0-nightly.* tags, KEEP: 14, via the GitHub Packages API). Nothing prunes OCIR, and every PR build pushes pr-<N>-<sha> per built image. Registry accumulation persists even now that runners are ephemeral. Separate from this, the GHCR job itself is broken: its package listing returns HTTP 400 and it has been red on every scheduled run since 2026-08-18 (tracked as A19 in [Tracking] Remaining CI / build / release / E2E automation work #3278).

Reviewed and deliberately left alone

origin/release-1.5 still carries 10 unconditional runs-on: [self-hosted] sites and origin/release-1.4 carries 9, all naming the decommissioned runner. This was reviewed and is deliberately not being cleaned up. release-1.4 is closed and takes no further patches, so its 9 sites are dead text outright. On release-1.5 the lanes that carry the label do not fire, and the ones that do already target shared runners, so a pointwise fix when it is actually needed beats a sweep now. The evidence backs that. Nothing is queued or waiting repo-wide. auto-release.yaml cannot fire, because schedule only triggers for the workflow file on the default branch and the file is absent from main. update-releasenotes.yaml is gated on push: branches: [main], which a release-branch push cannot match. backport.yaml and pull-requests-release.yaml do get triggered on release-line PRs, but every recent run is completed/skipped, and a skipped job never requests a runner. The only workflow that actually runs on release-1.5 is build-main.yaml, which #3469 already moved to oracle-vm-24cpu-96gb-x86-64.

The one residual worth knowing about was tags.yaml and release-e2e.yaml on the release lines, which fire on push: tags: v*. tags.yaml on release-1.5 runs on oracle-vm-24cpu-96gb-x86-64 now and the v1.5.4 tag build succeeded. release-e2e.yaml on release-1.5 is still [self-hosted] and the v1.5.4 run of it was cancelled, which fits a label nothing serves. I haven't looked at release-1.4, it is closed.

The stale comment in pull-requests.yaml ("only the debug label flips runs-on") is gone from main.

Deferred, not dropped

Parallelising the release build. Partly done: tags.yaml builds per architecture now (build-amd64 and build-arm64 jobs), but the amd64 make build is still one serial run. build-main.yaml and build-release.yaml stay serial on purpose, a single serialized writer is what keeps the shared :buildcache manifest race-free.
docker buildx bake consolidation of the per-package buildx invocations. Not started.
Multi-arch. Landed in another form than a platform matrix axis: DEFAULT_PLATFORMS is linux/amd64,linux/arm64 in hack/common-envs.mk, hack/stitch-multiarch.sh stitches the per-architecture images into one index, and there is a build-arm64-nightly.yaml workflow. PRs feat(build): build package images for amd64 and arm64 #4498, ci(release): build pre-release images for amd64 and arm64 and gate releases on it #4505, feat(vpn): build shadowbox for amd64 and arm64 #4521 and feat(build): build Talos, Harbor, the Velero KubeVirt plugin and the unlisted images for arm64 #4552.

Obsolete

The original "Background" section and its three numbered causes are historical. cozy-runner-1 is decommissioned, the buildkit-contention failure class is retired by one buildkit per job, and the inline-cache multistage problem is fixed by mode=max. The three "Open questions" are also resolved: #2855 was merged as interim relief (2026-06-09), a smaller shared-runner label does exist and is in use, and SBOM delivery is still open pending the referrers probe (item 1 above).

References

Landed here: fix(build): read --cache-from from ghcr :latest so PR builds warm-start #2855, ci: consolidated platform & e2e stabilization batch #2948, refactor(build): parallel matrix image builds on ephemeral runners #2983, feat(release): immutable tags & rc→stable promotion (#2677) #3017, fix(ci): restore e2e for fork PRs and close the merge-gate bypass #3262, ci: move remaining workflows off the persistent self-hosted runner #3268, ci: warm the build cache on release-1.5 pushes #3469, fix(build): pass the shared buildx flags from every image target #4485, feat(build): build package images for amd64 and arm64 #4498, ci(release): build pre-release images for amd64 and arm64 and gate releases on it #4505, feat(vpn): build shadowbox for amd64 and arm64 #4521, feat(build): build Talos, Harbor, the Velero KubeVirt plugin and the unlisted images for arm64 #4552
Closed without merging, superseded by refactor(build): parallel matrix image builds on ephemeral runners #2983: refactor(build): mode=max registry cache with a main-only warmer #2938, ci(build): isolate each PR build on its own ephemeral runner VM #2939
Closed without merging, unrelated to this epic: test(metallb): assert digest-pinned image form, not exact version literal #2873, fix(ci, kubernetes): build container disks concurrently to unblock Build #3457
Cache regression this epic started from: refactor(build): standardize image tagging to fix concurrent PR push conflicts #2711
Touch points: .github/workflows/pull-requests.yaml, .github/workflows/tags.yaml, .github/workflows/build-main.yaml, .github/workflows/build-release.yaml, .github/workflows/retention.yaml, hack/common-envs.mk, hack/build-matrix.sh, packages/**/Makefile

Plan amendment — live root-cause finding re-sequences the work

I caught the build-hang in the act and captured a daemon goroutine dump (SIGUSR1) while a build was wedged. The mechanism is now pinned. The end-state (Decisions 1–7) is unchanged, but the diagnosis gets sharper and the sequencing should move isolation ahead of the cache work.

Proven root cause

While a build was hung: ~290 BuildKit goroutines parked in solver/exporter.go + bboltcachestorage/storage.go + sync.Mutex.Lock, buildx on futex_wait_queue, zero registry connections, zero build CPU, disk at 4%. So cause #2 is specifically BuildKit's single-writer bbolt cache-metadata lock + the exporter mutex. make build is strictly serial within a job (no -j), so the pile-up is purely cross-job across the agents sharing one embedded buildkit. This confirms only per-job buildkit isolation (Decision 3) resolves it — cache mode, context size, prune, and daemon restart cannot. Notably, the captured wedge was a read-only PR build (WRITE_CACHE=0, no mode=max write), so mode=max is not implicated in the hang.

Proposed amendments

1. Background cause #2 — append:

Mechanism confirmed (live SIGUSR1 goroutine dump of a hung build): the contention is BuildKit's single-writer bbolt cache-metadata lock + the exporter mutex — ~290 goroutines parked in solver/exporter.go + bboltcachestorage/storage.go + sync.Mutex.Lock, buildx on futex_wait_queue, zero registry connections and zero build CPU (disk at 4%). make build is serial within a job (no -j), so the pile-up is purely cross-job. Only per-job buildkit isolation (Decision 3) resolves it; cache mode, context size, prune, and daemon restart cannot.

2. Decisions — framing note (does not change Decisions 1–2):

The mode=max registry cache + main warmer is the cache layer that isolated/ephemeral builders require for build-time — it is not a fix for the contention hang. It runs on the same single-writer store, so on the shared embedded buildkit it cannot reduce contention and only adds exporter/bbolt pressure (verified: a hung build wedged read-only, WRITE_CACHE=0). Sequence Phase 1 with, not before, a contention fix.

3. Decision 3 — name the driver + ephemerality gate:

Matrix, one image per job — provided each job runs on a truly ephemeral one-job-per-VM runner OR uses a docker-container-driver builder (never the shared embedded default docker driver). The "by construction" guarantee holds only if no two build jobs ever share a dockerd; verify pool ephemerality empirically before relying on it. docker-container is also required because the default driver does not fully support type=registry,mode=max cache export.

4. Decision 4 — cross-package coupling caveat:

"Patches only its package values.yaml" is too narrow here: (a) cross-package artifacts exist (packages/core/talos writes packages/extra/bootbox/images/matchbox.tag, consumed via Files.Get); (b) ~11 of 32 image targets do not yq-patch a values.yaml; (c) make build ends with make manifests, which renders _out/assets/* after core digests are patched and now rides in pr.patch. So the collect job must carry cross-package artifacts, run make manifests once after all core/platform/installer image jobs land their digests, and the parity gate must diff the full pr.patch.

5. New checklist section — insert before Phase 1, and reframe Phase 1:

Phase 0.5 — Interim contention relief (ship BEFORE Phase 1; throwaway at Phase 3)

Only per-job isolation fixes the proven contention; the cache work runs on the shared buildkit and cannot fix the hang. Bridge the gap:

Cap concurrent builds per daemon (reduce runner agents to 1–2, or a host flock/semaphore with SIGKILL-safe auto-release). Avoid a GHA concurrency group — it cancels excess jobs and can't key to the physical daemon.

OR pull Decision 7 forward for the Build job: flip runs-on: [self-hosted] → the oracle-vm pool the e2e job already uses (one line). Verify the pool is one-job-per-VM and has the build toolchain first.

Build-liveness observability: alert when a Build step exceeds ~~12 min + --progress=rawjson stall signature (single BUILDX_ARGS chokepoint). Shrinks MTTD 30m→~~12m.

And retitle Phase 1: "mode=max registry cache + warmer — the cache layer isolated builders need; sequence WITH Phase 0.5, not as the hang fix."

6. Open questions — add:

Is the oracle-vm pool genuinely one-job-per-VM (else contention returns under the matrix)? Does its image carry the full build toolchain + the privileged talos-nocloud imager? Cold-VM full-build time vs the 30-min wall?

Does mode=max cache-to actually function on the driver CI uses? The embedded docker driver does not fully support type=registry,mode=max — verify it isn't silently degrading to inline.

Does mode=max measurably increase exporter/bbolt hold time on the warmer? (Run the warmer isolated + SIGUSR1 under mode=max write.)

Net change

End-state unchanged (Decisions 1–7 stand); Decisions 3 and 4 get sharper. The only structural change is sequencing: interim isolation/cap before Phase 1, and Phase 1 reframed as caching rather than the hang fix.

ci(build): warm the main cache on an isolated docker-container builder

ci(build): route flux-plunger and kubeovn-plunger through $(BUILDER)

ci(build): install helm-unittest plugin on ephemeral build runners

ci(build): install flux CLI on ephemeral build runners

refactor(build): mode=max registry cache with a main-only warmer

ci(build): warm the main cache on an isolated docker-container builder

ci(build): route flux-plunger and kubeovn-plunger through $(BUILDER)

ci(build): isolate each PR build on its own ephemeral runner VM

ci(build): install helm-unittest plugin on ephemeral build runners

ci(build): install flux CLI on ephemeral build runners

Status update (2026-06-22)

Phase 1 and part of Phase 3 landed on main via #2948 (which carried the commits from #2938 d3958c5b and #2939 eba31194). Closing #2938/#2939 as superseded and ticking the boxes above.

Done

Phase 1 — mode=max registry :buildcache + main-only warmer (build-main.yaml); PR builds read-only; inline cache removed.
Phase 3 (partial) — build moved to the ephemeral oracle-vm-4cpu-16gb-x86-64 runner, with debug still routing to self-hosted.
Open Q1 resolved — fix(build): read --cache-from from ghcr :latest so PR builds warm-start #2855 was the interim ghcr :latest repoint, now superseded by the :buildcache cache.
Open Q2 resolved — smaller runner label exists (oracle-vm-4cpu-16gb-x86-64).

Remaining

Phase 1 — confirm cache HITs in PR build logs (the only Phase 1 box still open; fix(ci): create buildx builder in the build step's DOCKER_CONFIG #2962 stabilized the warmer's buildx builder today).
Phase 2 (untouched) — plan → matrix(1 image/job) → collect, per-package patch artifacts, parity gate vs make build, Talos nocloud as its own leg, SBOM.
Phase 3 (finish) — per-package dynamic paths-filter matrix (currently a coarse !docs/** filter) + shared-dependency fan-out; decommission the dedicated runner.
Open Q3 (SBOM delivery) — still open; SBOM not yet implemented.

ci(build): gate finalize on the unit/controller checks job

feat(release): immutable tags & rc→stable promotion (#2677) (#3017)

Status update (2026-07-08)

Phase 2 landed on main via #2983 (parallel per-package matrix) and #3148 (overlay current-main images for packages a PR didn't rebuild). Verified against main, not the checkboxes.

Done

Phase 2 — plan → build (matrix, 1 image/job) → build-talos → finalize. hack/build-matrix.sh computes the matrix from the PR diff; finalize reassembles per-package values.yaml patch fragments into pr.patch; e2e consumes it unchanged. First CI run validated end-to-end (plan → 25 parallel builds → talos → finalize → install). The one-time build-parity.yaml cutover gate served its purpose and was removed.
Phase 3 (change-scoping) — per-package dynamic matrix + shared-dependency fan-out are in build-matrix.sh; build runs on the ephemeral oracle-vm-4cpu-16gb-x86-64 pool (self-hosted only on the debug breakpoint).

Remaining

SBOM (Open-Q3) — plumbed but gated off (SBOM ?= 0, --provenance=false in hack/common-envs.mk). Needs the OCIR referrers/attestation probe, then --sbom=true + a delivery decision (attestation vs artifact vs release attachment).
Decommission the dedicated self-hosted runner — builds no longer depend on it; only the debug path remains.

build(flux-shard-operator): read the shared registry build cache

ci: move remaining workflows off the persistent self-hosted runner

ci: move remaining workflows off the persistent self-hosted runner (#…

ci: warm the build cache on release-1.5 pushes (#3469)

docs(ci): correct the stale shared-runner claims in the fork e2e lane

Narrowed this epic against origin/main at 09dc7f4. Two of the three headline deliverables shipped, so keeping the old body made the whole thing read as untouched work.

Landed: the per-image parallel matrix on ephemeral runners (#2983, strategy.matrix.pkg at .github/workflows/pull-requests.yaml:233-236, driven by hack/build-matrix.sh), the mode=max registry cache with a main-only warmer (hack/common-envs.mk cache-args plus .github/workflows/build-main.yaml), change-scoped builds with shared-dependency fan-out, and the runner cutover (#3268, gh api repos/cozystack/cozystack/actions/runners returns total_count: 0 and no workflow on main targets self-hosted).

Still live, and why the issue stays open. SBOM never shipped: the seam exists as SBOM ?= 0 in hack/common-envs.mk, and it is off because the Phase 0 OCIR referrers/attestation probe was never run and an unsupported attestation push fails the build. Three package Makefiles (dashboard, flux-plunger, kubeovn-plunger) hand-roll their buildx flags and never append $(BUILDX_ARGS), so four images would skip --sbom=true even once it is on. And nothing prunes pr-* tags on OCIR, since retention.yaml only prunes GHCR nightlies.

One thing checked and deliberately not tracked: release-1.4 and release-1.5 still carry 9 and 10 unconditional runs-on: [self-hosted] sites naming the decommissioned runner, but those lanes do not fire. auto-release.yaml cannot trigger from a non-default branch, update-releasenotes.yaml is gated on pushes to main, and backport.yaml and pull-requests-release.yaml skip before requesting a runner on every recent run. The body records it, with the one untested path called out: tags.yaml and release-e2e.yaml fire on tag pushes and last ran on 2026-07-07, two days before #3268 removed the runner.

Also recorded as deferred rather than dropped: .github/workflows/tags.yaml still runs a single serial make build, the one build path the matrix work never reached.

docs(gtr): finalize editorial markers and pin submission version (#4221)

Checked against main at bc400dff1, body updated.

The bypassing Makefiles (item 2) are fixed in #4485. Multi-arch landed, but as per-architecture builds stitched into one index (#4498, #4505, #4521, #4552) and not as the platform matrix axis this epic deferred, and tags.yaml builds per architecture now. SBOM (item 1) and OCIR pr-* retention (item 3) are untouched, SBOM ?= 0 and nothing prunes OCIR. The GHCR retention job is broken separately, that is A19 in #3278.

Runner notes changed. e2e-proxmox.yaml uses [self-hosted, proxmox] on purpose, it is gated behind a variable that is unset. tags.yaml on release-1.5 runs on the Oracle pool now and the v1.5.4 tag build passed, release-e2e.yaml there is still [self-hosted].

Navigation Menu

Repository navigation

Finish the CI image-build refactor: SBOM emission and pr-* tag retention #2937

Status as of 2026-10-01

Status

Landed

Still live

Reviewed and deliberately left alone

Deferred, not dropped

Obsolete

References

Plan amendment — live root-cause finding re-sequences the work

Proven root cause

Proposed amendments

Phase 0.5 — Interim contention relief (ship BEFORE Phase 1; throwaway at Phase 3)

Net change

Status update (2026-06-22)

9 remaining items

Status update (2026-07-08)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Finish the CI image-build refactor: SBOM emission and pr-* tag retention #2937

Description

Status as of 2026-10-01

Status

Landed

Still live

Reviewed and deliberately left alone

Deferred, not dropped

Obsolete

References

Activity

myasnikovdaniil commented on Jun 17, 2026

Plan amendment — live root-cause finding re-sequences the work

Proven root cause

Proposed amendments

Phase 0.5 — Interim contention relief (ship BEFORE Phase 1; throwaway at Phase 3)

Net change

myasnikovdaniil commented on Jun 22, 2026

Status update (2026-06-22)

9 remaining items

myasnikovdaniil commented on Jul 8, 2026

Status update (2026-07-08)

myasnikovdaniil commented on Aug 14, 2026

myasnikovdaniil commented on Oct 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions