Repository navigation
fix(ci, kubernetes): build container disks concurrently to unblock Build - #3457
myasnikovdaniil wants to merge 1 commit into
Conversation
Every release-1.5 PR Build job dies at the 30-minute timeout, and all of it is spent in one target. `image-ubuntu-container-disk` builds a disk per Kubernetes minor -- six of them -- and each spends ~4-5min inside a libguestfs appliance installing that minor's kubelet/kubeadm into its own copy of the cloud image. Serially that is ~28min of the 30min budget: v1.35 6:00, v1.34 4:21, v1.33 4:27, v1.32 4:39, v1.31 4:42, v1.30 cancelled None of it is cached, and no warmer will fix that on its own. Only build-main.yaml sets WRITE_CACHE=1, and main dropped this image from `make image` in 39cdc88 six days before release-1.5 was cut, so the :*-buildcache refs it reads have never been written -- the CI log shows all six importers failing with "not found". release-1.5 is now the only branch that builds this image at all. The six builds share no per-version work, so run them concurrently instead. A sub-make carries its own -j, keeping the root `make build` serial, and buildkit dedups the shared guestfish and cloud-image stages across the concurrent solves. --output-sync=target groups each version's log under its target name; six interleaved buildx progress streams have no per-solve identifier and are unreadable. Verified the per-version buildx/echo/rm command set is byte-identical to the loop it replaces across PUBLISH_VERSIONED, PUBLISH_FLOATING and WRITE_CACHE combinations. Moving off the 4cpu/16gb shape because six concurrent libguestfs appliances want ~9gb of RAM and a core apiece. The comment justifying the small shape assumed a warm cache, which no longer holds here. This also stops swallowing failures. The loop separated versions with `;`, so make saw only the last version's exit status: a failed build for any other version left its committed .tag file stale and the job green. Under the sub-make a failure in any version fails the target. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Superseded by #3469, which carries this commit plus the release-1.5 cache warmer and the timeout raise it needs to go green. Collapsing the stack into one PR. |
## What this PR does Nothing has ever warmed `release-1.5`'s build cache, so every PR build on this branch runs cold end to end and exceeds its 30-minute timeout. This branch's copy of `build-main.yaml` is dead code. It triggers on `push: branches: [main]`, which never happens here, and it targets `[self-hosted]` — the decommissioned cozy-runner-1 (#2937) — so it could not have run even if it fired. This PR points it at this branch instead. Both halves of the cold build trace back to 39cdc88 (`feat(kubernetes): bootstrap tenant workers with Talos via CABPT`) landing on `main` on 2026-06-16, six days before `release-1.5` was cut on 2026-06-22: - That commit dropped `ubuntu-container-disk` from `make image`, so `main` stopped writing its six `:<minor>-buildcache` refs. `release-1.5` is now the only branch that still builds the image, and the CI log shows all six importers failing with `not found`. Serially that was ~28min of the 30min budget; #3457 parallelises the target down to ~10min. - Every other cache ref carries `main`'s sources, which have diverged from this branch, so most of the remaining 28 packages miss too. Measured on the #3457 run: ~38min across the full build, and no single hot spot left to attack — `apps/kubernetes` at 15m07s and `system/linstor` at 5m39s, everything else 20-60s each. ### Changes | | before | after | | --- | --- | --- | | trigger | `push: [main]` | `push: [release-1.5]` | | runner | `[self-hosted]` (decommissioned) | `oracle-vm-24cpu-96gb-x86-64` | | cache namespace | shared `cozystack/` | `cozystack-cache-release-1.5/` | | `IMAGE_TAG` | `main` | `release-1.5` | | warmer timeout | 90 | 120 | `pull-requests.yaml` gets the same `CACHE_REGISTRY` so PR builds read what the warmer writes, and its Build timeout goes to 75 minutes. The cache namespace is branch-scoped on purpose. Writing the shared refs from here would replace `main`'s layers with this branch's, so `main`'s next PR build would miss on everything and blow its own 30-minute timeout until the next `main` push re-warmed it. `CACHE_REGISTRY` already exists as a knob in `hack/common-envs.mk`, so this is one env var per job rather than new machinery. Image tags are unaffected — they still resolve to `cozystack/<image>`, only cache refs move. The timeout raise is not optional. The new namespace starts empty, so this PR and the next one still build cold at ~45min, and anything that later invalidates the cache broadly — a Dockerfile edit, a `go.mod` bump — costs the same again. 30 minutes only ever fit a warm build, and when it did not fit, the job reported a timeout rather than whatever actually went wrong. ### Verification - Both workflows are asserted to agree on the cache namespace, rather than eyeballed. - Confirmed the env split renders as intended: `--tag iad.ocir.io/…/cozystack/ubuntu-container-disk:v1.35-release-1.5` alongside `--cache-from`/`--cache-to` on `…/cozystack-cache-release-1.5/ubuntu-container-disk:v1.35-buildcache`. - `WRITE_CACHE` stays unset in PR builds, so they remain read-only against the cache and never race on a manifest. ### Notes for the reviewer - **Stacked on #3457.** Both PRs edit adjacent lines of `pull-requests.yaml`, so keeping them independent guaranteed a conflict. GitHub retargets this to `release-1.5` automatically once #3457 merges. - **The filename stays `build-main.yaml`** even though it no longer warms `main`. Renaming it would make any later backport touching `main`'s copy arrive as a whole new file rather than a modification, which is the failure mode the backport bot already hits on this branch. - **One assumption I could not test locally:** the registry credentials need repo-create permission for the new `cozystack-cache-release-1.5/*` path. It is the same tenancy namespace that already auto-creates repos as packages get added, so it should hold — but if it does not, `--cache-to` fails the warmer, and the fallback is a cache-tag suffix in `hack/common-envs.mk` instead of a separate namespace. - **Sequencing:** the merge commit is itself a push to `release-1.5`, so the warmer fires on merge and takes ~50-60min cold. PR builds after that should be ~15min. The warmer cannot be triggered manually — `workflow_dispatch` is only available for workflows present on the default branch, and this one exists only here. - **Out of scope, but adjacent:** `tags.yaml`, `release-e2e.yaml` and `pull-requests-release.yaml` on this branch all still say `runs-on: [self-hosted]`, so the release and release-e2e lanes are broken the same way this warmer was. ### Release note ```release-note NONE ```
What this PR does
Every
release-1.5PR Build job dies at the 30-minute timeout, and all of it is spent in one target.image-ubuntu-container-diskbuilds one disk per Kubernetes minor — six of them — and each spends ~4-5min inside a libguestfs appliance installing that minor's kubelet/kubeadm into its own copy of the cloud image. Serially that is ~28min of the 30min budget, measured on the Build job of #3326:None of it is cached, and a cache warmer would not fix it on its own. Only
build-main.yamlsetsWRITE_CACHE=1, andmaindropped this image frommake imagein 39cdc88 six days beforerelease-1.5was cut, so the:*-buildcacherefs the build reads have never been written — the CI log shows all six importers failing withnot found.release-1.5is now the only branch that builds this image at all, so its build has been cold since the branch was created.The six builds share no per-version work, so this runs them concurrently instead. A sub-make carries its own
-j, which keeps the rootmake buildserial, and buildkit dedups the shared guestfish and cloud-image stages across the concurrent solves.--output-sync=targetgroups each version's log under its target name; six interleaved buildx progress streams carry no per-solve identifier and are unreadable otherwise.Build also moves off the
4cpu/16gbshape, because six concurrent libguestfs appliances want roughly 9gb of RAM and a core apiece. The comment justifying the small shape assumed a warm mode=max cache, which does not hold for this image on this branch.timeout-minutesdeliberately stays at 30 — fitting inside it is the point of the change.This also stops swallowing failures. The loop separated versions with
;, so make saw only the last version's exit status: a failed build for any other version left its committed.tagfile stale and the job green. Under the sub-make, a failure in any version fails the target.Verification
buildx/echo/rmcommand set is byte-identical to the loop it replaces, checked across defaults,PUBLISH_VERSIONED=1, andIMAGE_TAG+PUBLISH_FLOATING=1+WRITE_CACHE=1together — so the versioned tag, the floating tag and--cache-toall still expand correctly.make -n imagestill emits exactly 9buildxinvocations (6 disks plus cloud-provider, csi-driver, cluster-autoscaler); the new pattern rule shadows none of them..tagand.jsonfilenames are unchanged, so theFiles.Getlookups intemplates/cluster.yamlstill resolve.;-separated loop returns exit 0 with a failing non-final version, the sub-make returns exit 2.Wall clock is the one thing only CI can confirm. The slowest single build carries the one-time guestfish and cloud-image work that the others reuse, so the target should land near that ~6min rather than 28.
Once this is green it unblocks the other stuck
release-1.5backports on retest: #3402, #3370, #3366, #3364, #3356, #3326, #3321, #3202.Release note