Skip to content

fix(ci, kubernetes): build container disks concurrently to unblock Build - #3457

Closed
myasnikovdaniil wants to merge 1 commit into
release-1.5from
ci/parallel-container-disk
Closed

myasnikovdaniil wants to merge 1 commit into
release-1.5from
ci/parallel-container-disk

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

What this PR does

Every release-1.5 PR Build job dies at the 30-minute timeout, and all of it is spent in one target. image-ubuntu-container-disk builds one disk per Kubernetes minor — six of them — and each spends ~4-5min inside a libguestfs appliance installing that minor's kubelet/kubeadm into its own copy of the cloud image. Serially that is ~28min of the 30min budget, measured on the Build job of #3326:

version wall clock
v1.35 6:00
v1.34 4:21
v1.33 4:27
v1.32 4:39
v1.31 4:42
v1.30 cancelled at the timeout

None of it is cached, and a cache warmer would not fix it on its own. Only build-main.yaml sets WRITE_CACHE=1, and main dropped this image from make image in 39cdc88 six days before release-1.5 was cut, so the :*-buildcache refs the build reads have never been written — the CI log shows all six importers failing with not found. release-1.5 is now the only branch that builds this image at all, so its build has been cold since the branch was created.

The six builds share no per-version work, so this runs them concurrently instead. A sub-make carries its own -j, which keeps the root make build serial, and buildkit dedups the shared guestfish and cloud-image stages across the concurrent solves. --output-sync=target groups each version's log under its target name; six interleaved buildx progress streams carry no per-solve identifier and are unreadable otherwise.

Build also moves off the 4cpu/16gb shape, because six concurrent libguestfs appliances want roughly 9gb of RAM and a core apiece. The comment justifying the small shape assumed a warm mode=max cache, which does not hold for this image on this branch. timeout-minutes deliberately stays at 30 — fitting inside it is the point of the change.

This also stops swallowing failures. The loop separated versions with ;, so make saw only the last version's exit status: a failed build for any other version left its committed .tag file stale and the job green. Under the sub-make, a failure in any version fails the target.

Verification

  • The per-version buildx / echo / rm command set is byte-identical to the loop it replaces, checked across defaults, PUBLISH_VERSIONED=1, and IMAGE_TAG + PUBLISH_FLOATING=1 + WRITE_CACHE=1 together — so the versioned tag, the floating tag and --cache-to all still expand correctly.
  • make -n image still emits exactly 9 buildx invocations (6 disks plus cloud-provider, csi-driver, cluster-autoscaler); the new pattern rule shadows none of them.
  • .tag and .json filenames are unchanged, so the Files.Get lookups in templates/cluster.yaml still resolve.
  • Failure propagation was reproduced on a reduced Makefile: the ;-separated loop returns exit 0 with a failing non-final version, the sub-make returns exit 2.

Wall clock is the one thing only CI can confirm. The slowest single build carries the one-time guestfish and cloud-image work that the others reuse, so the target should land near that ~6min rather than 28.

Once this is green it unblocks the other stuck release-1.5 backports on retest: #3402, #3370, #3366, #3364, #3356, #3326, #3321, #3202.

Release note

fix(ci): build the per-Kubernetes-minor `ubuntu-container-disk` images concurrently so the release-1.5 Build job no longer exceeds its 30-minute timeout, and fail the build when any single version fails instead of only the last one.

Every release-1.5 PR Build job dies at the 30-minute timeout, and all of
it is spent in one target. `image-ubuntu-container-disk` builds a disk
per Kubernetes minor -- six of them -- and each spends ~4-5min inside a
libguestfs appliance installing that minor's kubelet/kubeadm into its own
copy of the cloud image. Serially that is ~28min of the 30min budget:

  v1.35 6:00, v1.34 4:21, v1.33 4:27, v1.32 4:39, v1.31 4:42, v1.30 cancelled

None of it is cached, and no warmer will fix that on its own. Only
build-main.yaml sets WRITE_CACHE=1, and main dropped this image from
`make image` in 39cdc88 six days before release-1.5 was cut, so the
:*-buildcache refs it reads have never been written -- the CI log shows
all six importers failing with "not found". release-1.5 is now the only
branch that builds this image at all.

The six builds share no per-version work, so run them concurrently
instead. A sub-make carries its own -j, keeping the root `make build`
serial, and buildkit dedups the shared guestfish and cloud-image stages
across the concurrent solves. --output-sync=target groups each version's
log under its target name; six interleaved buildx progress streams have
no per-solve identifier and are unreadable. Verified the per-version
buildx/echo/rm command set is byte-identical to the loop it replaces
across PUBLISH_VERSIONED, PUBLISH_FLOATING and WRITE_CACHE combinations.

Moving off the 4cpu/16gb shape because six concurrent libguestfs
appliances want ~9gb of RAM and a core apiece. The comment justifying
the small shape assumed a warm cache, which no longer holds here.

This also stops swallowing failures. The loop separated versions with
`;`, so make saw only the last version's exit status: a failed build for
any other version left its committed .tag file stale and the job green.
Under the sub-make a failure in any version fails the target.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4bae3053-19a4-4271-b294-0b4df2acb6ae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/parallel-container-disk

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dosubot dosubot Bot added area/build Issues or PRs related to image build infrastructure, multi-arch support area/ci Issues or PRs related to CI workflows, GitHub Actions, automation kind/bug Categorizes issue or PR as related to a bug labels Jul 27, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app size/M This PR changes 30-99 lines, ignoring generated files labels Jul 27, 2026
@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

Superseded by #3469, which carries this commit plus the release-1.5 cache warmer and the timeout raise it needs to go green. Collapsing the stack into one PR.

myasnikovdaniil added a commit that referenced this pull request Jul 29, 2026
## What this PR does

Nothing has ever warmed `release-1.5`'s build cache, so every PR build
on this branch runs cold end to end and exceeds its 30-minute timeout.

This branch's copy of `build-main.yaml` is dead code. It triggers on
`push: branches: [main]`, which never happens here, and it targets
`[self-hosted]` — the decommissioned cozy-runner-1 (#2937) — so it could
not have run even if it fired. This PR points it at this branch instead.

Both halves of the cold build trace back to 39cdc88
(`feat(kubernetes): bootstrap tenant workers with Talos via CABPT`)
landing on `main` on 2026-06-16, six days before `release-1.5` was cut
on 2026-06-22:

- That commit dropped `ubuntu-container-disk` from `make image`, so
`main` stopped writing its six `:<minor>-buildcache` refs. `release-1.5`
is now the only branch that still builds the image, and the CI log shows
all six importers failing with `not found`. Serially that was ~28min of
the 30min budget; #3457 parallelises the target down to ~10min.
- Every other cache ref carries `main`'s sources, which have diverged
from this branch, so most of the remaining 28 packages miss too.
Measured on the #3457 run: ~38min across the full build, and no single
hot spot left to attack — `apps/kubernetes` at 15m07s and
`system/linstor` at 5m39s, everything else 20-60s each.

### Changes

| | before | after |
| --- | --- | --- |
| trigger | `push: [main]` | `push: [release-1.5]` |
| runner | `[self-hosted]` (decommissioned) |
`oracle-vm-24cpu-96gb-x86-64` |
| cache namespace | shared `cozystack/` | `cozystack-cache-release-1.5/`
|
| `IMAGE_TAG` | `main` | `release-1.5` |
| warmer timeout | 90 | 120 |

`pull-requests.yaml` gets the same `CACHE_REGISTRY` so PR builds read
what the warmer writes, and its Build timeout goes to 75 minutes.

The cache namespace is branch-scoped on purpose. Writing the shared refs
from here would replace `main`'s layers with this branch's, so `main`'s
next PR build would miss on everything and blow its own 30-minute
timeout until the next `main` push re-warmed it. `CACHE_REGISTRY`
already exists as a knob in `hack/common-envs.mk`, so this is one env
var per job rather than new machinery. Image tags are unaffected — they
still resolve to `cozystack/<image>`, only cache refs move.

The timeout raise is not optional. The new namespace starts empty, so
this PR and the next one still build cold at ~45min, and anything that
later invalidates the cache broadly — a Dockerfile edit, a `go.mod` bump
— costs the same again. 30 minutes only ever fit a warm build, and when
it did not fit, the job reported a timeout rather than whatever actually
went wrong.

### Verification

- Both workflows are asserted to agree on the cache namespace, rather
than eyeballed.
- Confirmed the env split renders as intended: `--tag
iad.ocir.io/…/cozystack/ubuntu-container-disk:v1.35-release-1.5`
alongside `--cache-from`/`--cache-to` on
`…/cozystack-cache-release-1.5/ubuntu-container-disk:v1.35-buildcache`.
- `WRITE_CACHE` stays unset in PR builds, so they remain read-only
against the cache and never race on a manifest.

### Notes for the reviewer

- **Stacked on #3457.** Both PRs edit adjacent lines of
`pull-requests.yaml`, so keeping them independent guaranteed a conflict.
GitHub retargets this to `release-1.5` automatically once #3457 merges.
- **The filename stays `build-main.yaml`** even though it no longer
warms `main`. Renaming it would make any later backport touching
`main`'s copy arrive as a whole new file rather than a modification,
which is the failure mode the backport bot already hits on this branch.
- **One assumption I could not test locally:** the registry credentials
need repo-create permission for the new `cozystack-cache-release-1.5/*`
path. It is the same tenancy namespace that already auto-creates repos
as packages get added, so it should hold — but if it does not,
`--cache-to` fails the warmer, and the fallback is a cache-tag suffix in
`hack/common-envs.mk` instead of a separate namespace.
- **Sequencing:** the merge commit is itself a push to `release-1.5`, so
the warmer fires on merge and takes ~50-60min cold. PR builds after that
should be ~15min. The warmer cannot be triggered manually —
`workflow_dispatch` is only available for workflows present on the
default branch, and this one exists only here.
- **Out of scope, but adjacent:** `tags.yaml`, `release-e2e.yaml` and
`pull-requests-release.yaml` on this branch all still say `runs-on:
[self-hosted]`, so the release and release-e2e lanes are broken the same
way this warmer was.

### Release note

```release-note
NONE
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Issues or PRs related to image build infrastructure, multi-arch support area/ci Issues or PRs related to CI workflows, GitHub Actions, automation area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant