Repository navigation
ci: warm the build cache on release-1.5 pushes - #3469
Merged
Merged
Conversation
Every release-1.5 PR Build job dies at the 30-minute timeout, and all of it is spent in one target. `image-ubuntu-container-disk` builds a disk per Kubernetes minor -- six of them -- and each spends ~4-5min inside a libguestfs appliance installing that minor's kubelet/kubeadm into its own copy of the cloud image. Serially that is ~28min of the 30min budget: v1.35 6:00, v1.34 4:21, v1.33 4:27, v1.32 4:39, v1.31 4:42, v1.30 cancelled None of it is cached, and no warmer will fix that on its own. Only build-main.yaml sets WRITE_CACHE=1, and main dropped this image from `make image` in 39cdc88 six days before release-1.5 was cut, so the :*-buildcache refs it reads have never been written -- the CI log shows all six importers failing with "not found". release-1.5 is now the only branch that builds this image at all. The six builds share no per-version work, so run them concurrently instead. A sub-make carries its own -j, keeping the root `make build` serial, and buildkit dedups the shared guestfish and cloud-image stages across the concurrent solves. --output-sync=target groups each version's log under its target name; six interleaved buildx progress streams have no per-solve identifier and are unreadable. Verified the per-version buildx/echo/rm command set is byte-identical to the loop it replaces across PUBLISH_VERSIONED, PUBLISH_FLOATING and WRITE_CACHE combinations. Moving off the 4cpu/16gb shape because six concurrent libguestfs appliances want ~9gb of RAM and a core apiece. The comment justifying the small shape assumed a warm cache, which no longer holds here. This also stops swallowing failures. The loop separated versions with `;`, so make saw only the last version's exit status: a failed build for any other version left its committed .tag file stale and the job green. Under the sub-make a failure in any version fails the target. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
Nothing has ever warmed this branch's build cache. This workflow is a copy of main's warmer that still triggers on `push: branches: [main]`, so on release-1.5 it has never fired once -- and it targets [self-hosted], the decommissioned cozy-runner-1 (#2937), so it could not have run even if it did fire. The consequence is a PR Build that runs cold end to end and exceeds its 30-minute timeout. Two separate reasons, both traceable to 39cdc88 ("bootstrap tenant workers with Talos via CABPT") landing on main six days before this branch was cut: * that commit dropped ubuntu-container-disk from `make image`, so main stopped writing its six :<minor>-buildcache refs. release-1.5 is the only branch that still builds the image, and the CI log shows all six importers failing with "not found" -- ~28min serial, cut to ~10min by parallelising the target in #3457. * every other ref carries main's sources, which have diverged from this branch, so most of the remaining 28 packages miss too -- ~38min across the full build, with no single hot spot left to fix. Point it at this branch, move it to the ephemeral shape main's warmer already uses, and write to a branch-scoped CACHE_REGISTRY namespace. The namespace matters: writing the shared refs from here would replace main's layers with release-1.5's, so main's next PR build would miss on everything and blow its own 30-min timeout until the next main push re-warmed it. pull-requests.yaml has to read the same namespace, which the following commit does. The filename stays build-main.yaml even though it no longer warms main. Renaming it would make any later backport touching main's copy arrive as a whole new file rather than a modification, which is the failure mode the backport bot already hits on this branch. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
Point the Build job at the namespace build-main.yaml now writes for this branch. Without this the warmer would run and nothing would read it: the cache-from refs would still resolve to the shared namespace main writes, whose layers carry main's sources and miss on most of this branch's 28 packages. Builds only read the cache -- WRITE_CACHE defaults to 0, so concurrent PR builds still never race on a cache manifest, and a miss degrades to a cold build rather than an error. Raise the timeout to 75 minutes at the same time. A warm build is ~15min, but the namespace starts empty, so this PR and the next one run cold at ~45min: six ubuntu-container-disk images plus 28 packages from scratch. Anything that later invalidates the cache broadly -- a Dockerfile edit, a go.mod bump -- costs the same again. 30 minutes only ever fit a warm build, and when it did not fit, the job reported a timeout rather than whatever actually went wrong. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
size/M
dosubot
Bot
added
area/build
myasnikovdaniil
changed the base branch from
ci/parallel-container-disk
to
release-1.5
July 28, 2026 09:30
IvanHunters
approved these changes
Jul 28, 2026
IvanHunters
left a comment
IvanHunters
left a comment
Collaborator
There was a problem hiding this comment.
LGTM. CI/build-infra only, no runtime chart/CRD/RBAC/migration surface. Cache wiring verified end-to-end: both workflows set CACHE_REGISTRY at workflow level to the same branch-scoped ref; the warmer writes (WRITE_CACHE=1) and PR builds read-only, so writer/reader target identical refs with no manifest race. Runner label oracle-vm-24cpu-96gb-x86-64 is a real in-use custom label; the actionlint "unknown runner" note is a false positive.
Non-blocking notes:
- Stacked on #3457 — merge after it lands (or confirm pulling its Makefile parallelization into release-1.5 here is intended), otherwise the merge-base diff drags #3457's commit in.
- The warmer only fires on push to release-1.5, i.e. post-merge, so the merging PR and near-term PRs still build cold (~45m) — consistent with the raised 75m timeout.
- Adjacent lanes (tags.yaml, release-e2e.yaml, pull-requests-release.yaml) still target the decommissioned [self-hosted] runner — follow-up, out of scope here.
myasnikovdaniil
added a commit
that referenced
this pull request
Jul 31, 2026
## What this PR does Adds the flux CLI to the release-1.5 cache warmer, which #3469 moved onto the ephemeral runner shape without carrying over the toolchain step that main's warmer and this branch's `pull-requests.yaml` both have. `make build` finishes at `packages/core/installer`, whose `image-packages` target shells out to `flux push artifact`. The self-hosted runner the warmer used to target had flux baked in; the ephemeral shape does not. Confirmed on the first warmer run after #3469 merged ([run 30430753152](https://github.com/cozystack/cozystack/actions/runs/30430753152)): it failed at 49 minutes with `/bin/sh: 1: flux: not found` on `Makefile:35`, the last of 28 packages. The cache itself came out warm, because every package exports its cache as it builds and installer is last: | | | | --- | --- | | cache export operations | 87 | | cache manifests written | 70 | | repos created under `CACHE_REGISTRY` | 34 | | `ubuntu-container-disk` refs (`v1.30`–`v1.35`, `mode=max`) | all 6 | | real auth or export failures | 0 | So the only loss was the `cozystack-packages:release-1.5` artifact push, and nothing downstream depends on it — PR builds build the installer themselves with flux present. This also settles the open question from #3469: the registry credentials can create repositories under the new `cozystack-cache-release-1.5/*` namespace, so no fallback to a cache-tag suffix is needed. Fixing it still matters. A permanently red warmer means a genuine breakage is indistinguishable from this known-broken tail, and nothing else reports on that workflow. The alternative would be teaching the warmer to run the 28 image builds without the artifact push, but no such target exists and it would diverge from main's warmer for no gain. Installing the CLI keeps the two warmers identical in shape, and matches main publishing `cozystack-packages:main` from its own warmer. ### Release note ```release-note NONE ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Nothing has ever warmed
release-1.5's build cache, so every PR build on this branch runs cold end to end and exceeds its 30-minute timeout.This branch's copy of
build-main.yamlis dead code. It triggers onpush: branches: [main], which never happens here, and it targets[self-hosted]— the decommissioned cozy-runner-1 (#2937) — so it could not have run even if it fired. This PR points it at this branch instead.Both halves of the cold build trace back to 39cdc88 (
feat(kubernetes): bootstrap tenant workers with Talos via CABPT) landing onmainon 2026-06-16, six days beforerelease-1.5was cut on 2026-06-22:ubuntu-container-diskfrommake image, somainstopped writing its six:<minor>-buildcacherefs.release-1.5is now the only branch that still builds the image, and the CI log shows all six importers failing withnot found. Serially that was ~28min of the 30min budget; fix(ci, kubernetes): build container disks concurrently to unblock Build #3457 parallelises the target down to ~10min.main's sources, which have diverged from this branch, so most of the remaining 28 packages miss too. Measured on the fix(ci, kubernetes): build container disks concurrently to unblock Build #3457 run: ~38min across the full build, and no single hot spot left to attack —apps/kubernetesat 15m07s andsystem/linstorat 5m39s, everything else 20-60s each.Changes
push: [main]push: [release-1.5][self-hosted](decommissioned)oracle-vm-24cpu-96gb-x86-64cozystack/cozystack-cache-release-1.5/IMAGE_TAGmainrelease-1.5pull-requests.yamlgets the sameCACHE_REGISTRYso PR builds read what the warmer writes, and its Build timeout goes to 75 minutes.The cache namespace is branch-scoped on purpose. Writing the shared refs from here would replace
main's layers with this branch's, somain's next PR build would miss on everything and blow its own 30-minute timeout until the nextmainpush re-warmed it.CACHE_REGISTRYalready exists as a knob inhack/common-envs.mk, so this is one env var per job rather than new machinery. Image tags are unaffected — they still resolve tocozystack/<image>, only cache refs move.The timeout raise is not optional. The new namespace starts empty, so this PR and the next one still build cold at ~45min, and anything that later invalidates the cache broadly — a Dockerfile edit, a
go.modbump — costs the same again. 30 minutes only ever fit a warm build, and when it did not fit, the job reported a timeout rather than whatever actually went wrong.Verification
--tag iad.ocir.io/…/cozystack/ubuntu-container-disk:v1.35-release-1.5alongside--cache-from/--cache-toon…/cozystack-cache-release-1.5/ubuntu-container-disk:v1.35-buildcache.WRITE_CACHEstays unset in PR builds, so they remain read-only against the cache and never race on a manifest.Notes for the reviewer
pull-requests.yaml, so keeping them independent guaranteed a conflict. GitHub retargets this torelease-1.5automatically once fix(ci, kubernetes): build container disks concurrently to unblock Build #3457 merges.build-main.yamleven though it no longer warmsmain. Renaming it would make any later backport touchingmain's copy arrive as a whole new file rather than a modification, which is the failure mode the backport bot already hits on this branch.cozystack-cache-release-1.5/*path. It is the same tenancy namespace that already auto-creates repos as packages get added, so it should hold — but if it does not,--cache-tofails the warmer, and the fallback is a cache-tag suffix inhack/common-envs.mkinstead of a separate namespace.release-1.5, so the warmer fires on merge and takes ~50-60min cold. PR builds after that should be ~15min. The warmer cannot be triggered manually —workflow_dispatchis only available for workflows present on the default branch, and this one exists only here.tags.yaml,release-e2e.yamlandpull-requests-release.yamlon this branch all still sayruns-on: [self-hosted], so the release and release-e2e lanes are broken the same way this warmer was.Release note