Repository navigation
feat(talos): build the Talos installer and matchbox for arm64 - #4507
Closed
Aleksei Sviridkin (lexfrei) wants to merge 5 commits into
Closed
Aleksei Sviridkin (lexfrei) wants to merge 5 commits into
Aleksei Sviridkin (lexfrei) wants to merge 5 commits into
Conversation
Cozystack runs on arm64 nodes, but the Talos assets it builds are amd64 only, so arm64 clusters have to use an Image Factory schematic instead of the cozystack build. The arm64 profiles install drbd and zfs, which Cozystack needs, plus the firmware of the PCIe NICs whose drivers the arm64 Talos kernel carries. x86 CPU microcode is useless there, and the arm64 builds of amdgpu and i915 are empty. The generator checks every listed extension for a build of the target arch so an upstream change fails generation rather than the imager. The amd64 profiles keep their names and content. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The package pinned linux/amd64 because its assets were amd64 only. With arm64 profiles it follows PLATFORM like every other image. The installer is published as one index under every tag. skopeo cannot assemble an index from the imager's docker archives, so the archives go into one OCI layout and a small script writes the index there for skopeo to push. buildx could build it from OCI layout sources, but the release job builds on the classic docker driver, which has no such source. matchbox carries the kernel and initramfs of both arches whatever PLATFORM is, rather than those of TARGETARCH: the node matchbox runs on is not the arch of the machines it boots, so a per-arch image would hand an amd64 machine an arm64 kernel whenever the pod lands on an arm64 node. The amd64 assets stay at their old paths, and a group keyed on the arch iPXE reports sends arm64 machines to the arm64 ones. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-by: LLM
The talos package now builds for every arch in PLATFORM, and the jobs that build it without setting PLATFORM would pick up arm64 as well: the PR Talos build, the nightly disk and the release assets. None of them has a consumer for arm64 output, since e2e boots amd64 and the release publishes the amd64 assets only, so each one pins amd64 the way the image builds already do. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The build notes said the talos package pins linux/amd64, which no longer holds and would steer an agent into re-adding the pin. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
arm64 users can only take the Talos ISO, disk images, kernel and initramfs from the Image Factory today, because a release carries the amd64 ones only. The tag build now builds the assets for both arches and uploads the arm64 set next to the amd64 one under the same naming. It also pushes the talos installer image as a two-arch index, because a node installed from the arm64 assets pulls its installer from that image. The rest of the build stays amd64. Signed-off-by: Aleksei Sviridkin <[email protected]> Assisted-by: LLM
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: cozystack/cozystack/.coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (21)
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 5 remain after this review. 📝 WalkthroughWalkthroughTalos now builds installer and release assets for configured architectures, including ARM64. The build creates a combined installer index, and the Matchbox image and configuration include ARM64 boot assets. Build workflows and asset uploads specify architecture behavior. ChangesTalos multi-platform support
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~25 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant TalosMakefile
participant TalosImager
participant InstallerIndex
participant OCILayout
TalosMakefile->>TalosImager: Build installer for each selected architecture
TalosImager-->>TalosMakefile: Return installer image output
TalosMakefile->>OCILayout: Copy installer images
TalosMakefile->>InstallerIndex: Request index for selected architectures
InstallerIndex->>OCILayout: Write index blob and index reference
TalosMakefile->>OCILayout: Export or publish indexed image
Merge Risk: ⚪ Minimal · up to No confirmed issue remains that should block the Talos ARM64 build and boot changes from merging after normal checks. Security Architecture ReviewSecurity architecture risk: 🟡 Moderate · up to ARM64 machines gain a new boot path and installer image, while existing AMD64 builds must remain compatible. The build includes checks for missing architecture support, but the deployed boot-selection behavior has not been verified here. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
area/platform
This was referenced Sep 27, 2026
IvanHunters
previously approved these changes
Sep 28, 2026
IvanHunters
left a comment
IvanHunters
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed statically in a hermetic clone. Ran the native arm64 Talos imager to confirm cross-arch asset generation, verified every new arm64 systemExtension pin resolves to a multi-arch index, and confirmed matchbox routes arm64 machines ahead of the default group. LGTM. Minor non-blocking notes only.
Aleksei Sviridkin (lexfrei)
dismissed
IvanHunters’s stale review
September 28, 2026 16:43
The base branch was changed.
Contributor
Author
This was referenced Sep 28, 2026
Aleksei Sviridkin (lexfrei)
added a commit
that referenced
this pull request
Sep 28, 2026
…unlisted images for arm64 (#4552) ## What this PR does This brings together four approved PRs that give the remaining first-party images an arm64 build. Each of them lost its approval when #4498 merged and GitHub retargeted it to main, so I merged them into one branch to review and approve once. Each original PR has the full description and review history, and I'll close them in favour of this one. - #4507 builds the Talos installer and matchbox for arm64. Releases also publish the arm64 ISO, disk images, kernel and initramfs. - #4515 builds the KubeVirt Velero plugin in-tree for amd64 and arm64. - #4525 adds flux-plunger, keycloak-operator, kilo and migration-controller to the root `build:` list, so CI rebuilds them like every other image. A test keeps any new image target from being left out. - #4528 rebuilds the Harbor v2.15.1 images for amd64 and arm64, because upstream publishes v2.15 for amd64 only. Release builds still run with `PLATFORM=linux/amd64`, except the Talos installer, so these images ship arm64 only once #4505 adds the arm64 leg. This PR makes that leg buildable. A few commits are new on top of the originals. The velero test no longer pins the plugin's image string, because the release build stamps its own rebuild over it. It now checks the plugin init container by repository name. A promote-retag test comment now says what it guards, and the image-refs doc no longer calls its fixed gaps open. The merge with main keeps the Harbor portal in the list of stages whose output has no architecture. ### Screenshots Not a UI change. ### Downstream repositories I left every box empty and opened nothing in other repositories. Two repositories are touched, and neither follow-up is needed before this merges: - cozystack/website names the amd64 assets in `content/en/docs/next/install/talos/iso.md` and `install/providers/oracle-cloud.md`. The install pages could offer the arm64 downloads once a release carries them. No asset is renamed, so the current links keep working. - cozystack/talm points at `ghcr.io/cozystack/cozystack/talos:<version>` in `charts/cozystack/values.yaml`. Once a release publishes that tag as a two-arch index, the same reference works on arm64 nodes, so nothing there has to change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(talos): the Talos installer image is built for amd64 and arm64, with arm64 profiles that carry drbd and zfs, and the bootbox matchbox image network-boots arm64 machines as well as amd64 ones. Releases publish the arm64 Talos ISO, disk images, kernel and initramfs. feat(velero): the KubeVirt Velero plugin is now built in-tree for amd64 and arm64, so the velero server starts on arm64 clusters. build: flux-plunger, keycloak-operator, kilo and migration-controller are now rebuilt by CI like every other first-party image, so their pinned images pick up source patches and base-image updates and are published for amd64 and arm64. feat(harbor): the Harbor application now runs on arm64 nodes. Its component images are rebuilt from the pinned upstream v2.15.1 release for amd64 and arm64, because upstream publishes v2.15 images for amd64 only. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Talos images and boot assets now support both AMD64 and ARM64, including architecture-specific installation and cloud images. * Added build support for Harbor components and the KubeVirt Velero plugin. * Nightly builds now include ARM64 release assets. * **Improvements** * Harbor uses pinned chart and image versions, with built image references recorded automatically. * Added checks to catch image packages missing from the main build and to verify multi-architecture installer output. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Aleksei Sviridkin (lexfrei)
added a commit
that referenced
this pull request
Sep 29, 2026
…leases on it (#4505) ## What this PR does Pre-release builds now publish every image as an amd64 and arm64 index, and a release fails if anything it ships is single-arch. Main, release lines and pull requests keep building amd64 only. It builds on #4498, which made every image buildable for both architectures. Part of #1961. **This PR is on hold until everything under "Merge after" has landed.** Merged earlier, the new gate fails the next rc on the images those PRs fix. ### How a pre-release is built A pre-release tag gets a second job, for arm64, next to the existing release job. That covers `-rc.N`, `-beta.N` and `-alpha.N` tags. It runs on the CNCF arm64 pool (`oracle-vm-24cpu-96gb-arm64`), builds the same images with `PLATFORM=linux/arm64`, and pushes them as `<tag>-arm64`. `prepare-release` waits for it, and if the arm64 job fails the rc fails with it. There is no amd64-only fallback. Stable tags do not run it: promotion copies the rc digests with `skopeo copy --multi-arch all`, so a stable release inherits the indexes. After the amd64 build, the stitch joins each amd64 image with its arm64 twin into one index. It lives in `hack/stitch-multiarch.sh`. Every tag of the amd64 image moves to the index, the component-versioned ones included. The script then rewrites the digests in the tree and republishes the packages artifact and the installer chart pinned on them. It reads refs through `hack/lib/image-refs.sh`, and it fails if an old digest survives anywhere outside `charts/`. ### The gate A new check fails the rc when a pinned digest in the tree is not an index with both linux/amd64 and linux/arm64. It lives in `hack/verify-multiarch.sh`. It checks first-party and third-party refs alike. Images that are amd64 by nature go into `hack/multiarch-allowlist`, one repository per line, each with a mandatory reason. Today that is only the e2e sandbox. An entry that matches nothing is reported as stale. The static gate only sees digest-pinned refs in the tree. A tag-only ref, an image from a vendored chart default, or one an operator starts at runtime is invisible to it. So the rc e2e also audits every image its nodes pulled, with the same check and the same allowlist. Neither covers a package the e2e never installs. ### Nightly A new nightly workflow builds main for arm64. It keeps the arm64 build cache warm, since the rc job only reads it. It also prints the single-arch refs that would block the next rc. Nothing it builds is published. ### Smaller changes - `CACHE_TAG` in `hack/common-envs.mk` moves the default cache ref, so the arm64 build keeps a cache of its own. - `MATRIX_ARCH=arm64` makes `hack/build-matrix.sh` leave out the amd64-only e2e sandbox. - The kamaji provider image is pushed under the build's `IMAGE_TAG` like every other image. Its Makefile set `IMAGE_TAG` itself, so concurrent builds overwrote each other's component tag. Closes #4503. - `hack/nightly-mirror.sh` verifies a copy against the raw manifest digest. Plain `skopeo inspect` resolves an index to one platform's child. - matchbox is rebuilt for both platforms in the amd64 release job. The arm64 leg skips the talos package, so there is no arm64 half to stitch, and its Dockerfile only copies files, so no QEMU is needed. Releases then network-boot arm64 machines too. Closes #4524. The stitch also rewrites the kamaji ref inside `files/components.gz`, recompressed with `gzip -n` so the bytes are reproducible, so the kamaji control-plane provider ships multi-arch too. keda and kuberture now name the repository next to their pinned digest, so the gate can resolve them; their rendered manifests do not change. ### Merge after - #4552 builds the Talos installer, matchbox, Harbor and the Velero KubeVirt plugin for arm64, and adds flux-plunger, keycloak-operator, kilo and migration-controller to the root `build:` list. It replaces #4507, #4515, #4525 and #4528. - #4549 moves ingress-nginx, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy to their multi-arch releases, which are already published. The rest of the stack is merged: #4485, #4498, #4512, #4517, #4521 and #4522 here, and the multi-arch PRs in ingress-nginx-with-protobuf-exporter, cozy-proxy, cozystack-scheduler and keycloak-kms-proxy. ### Verification The bats suites pass: stitch, verify-multiarch, build-matrix, common-envs, nightly-mirror and the release contracts. The first three also pass under `hack/cozytest.sh` with dash. Each new test was red before its implementation, and the verify-multiarch and stitch tests each have a mutation that turns them red. I ran `verify-multiarch --report` against main. It lists the first-party refs the stitch will fix, plus the third-party and special cases above. The workflows have not run yet. The arm64 job's tools and duration, the stitch against real registries, and the audit are checked for the first time on the nightly and on the next rc. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Nothing under `hack/` is moved or renamed, and no make target changes its default behaviour: `CACHE_TAG` and `MATRIX_ARCH` are opt-in. The satellite repositories listed under "Merge after" are prerequisites, not follow-ups this change forces on them. ### Release note ```release-note ci(release): pre-release builds publish every image as an amd64 and arm64 multi-arch index, and a release fails if any image it ships or pulls in e2e lacks either architecture. Stable releases inherit the indexes through promotion. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Release candidates now build amd64 and arm64 images in parallel and combine eligible images into verified multi-architecture indexes before publication. * A nightly arm64 image build is available for testing and supplies a build cache for release-candidate builds. * **Bug Fixes** * End-to-end checks audit whether pulled images support both architectures. Audit failures block prerelease checks, while stable releases continue with a warning. * End-to-end test artifacts now include collected image references and multi-architecture audit results. * **Documentation** * Updated release and image guidance covers multi-architecture builds, verification checks, and troubleshooting, including arm64 build and stitching failures. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This makes the Talos installer image and matchbox work on arm64. It is stacked on #4498, which builds the other images for amd64 and arm64 but pins the talos package to amd64, because its assets were amd64 only. This PR removes that pin.
The profile generator now writes arm64 profiles next to the amd64 ones. The amd64 profiles keep their names and content, byte for byte. The arm64 set has drbd and zfs, plus the bnx2-bnx2x, intel-ice-firmware and qlogic-firmware packages, since the arm64 Talos kernel ships the drivers for those NICs. It leaves out amd-ucode and intel-ucode, which are x86 CPU microcode, and amdgpu and i915, whose arm64 builds are empty. The script explains this list and fails if a listed extension has no build for the target arch.
The installer is built for each arch in
PLATFORMand pushed as one index under every tag. skopeo cannot build an index from the imager's docker archives. So the archives go into one OCI layout, a small script (hack/installer-index.sh) writes the index there, and skopeo pushes it. I did not use buildx for this: it can take an OCI layout as a source, but not on the classic docker driver the tag build runs on. Fork PRs still exporttalos.oci.tar, which now holds an index.Every matchbox image carries the kernel and initramfs of both arches, whatever
PLATFORMis. Picking them byTARGETARCHwould be wrong, because the node matchbox runs on says nothing about the machines it boots: an arm64 pod would give amd64 machines an arm64 kernel. The amd64 assets stay at their old paths. A new arm64 group matches thearchparameter that matchbox'sboot.ipxesends, and serves the arm64 kernel withconsole=ttyAMA0. The bootbox ConfigMaps get the same profile and group.CI keeps building Talos for amd64 only. The PR Talos build, the nightly disk and the tag build's
make assetsnow setPLATFORM: linux/amd64, same as the image builds. The PR job still has to build the arm64 kernel and initramfs for matchbox. That took about one minute locally, probably 2-3 minutes on the 4-CPU runner.The last commit changes release engineering. With it, the tag build builds the release assets for both arches and pushes the talos installer again as a two-arch index after
make build, andhack/upload-assets.shuploads the arm64 ISO, disk images, kernel and initramfs. That adds six imager runs to the tag build, which I have not timed on the release runner. Promotion is not affected, becausepromote-rc.yamlruns the rc branch's ownupload-assets.sh.packages/core/testingstays amd64, since the e2e sandbox boots underqemu-system-x86_64.I tested this on an arm64 host. The imager built the arm64 installer, kernel and initramfs: the kernel is an Aarch64 EFI image and the installer config says arm64. The talos image pushed to a local registry resolves to one amd64+arm64 index under the build tag, the Talos version and
latest. matchbox built for both platforms carries both arches' assets,/ipxe?arch=arm64returns the arm64 profile, andarch=x86_64returns the default one. The changed bats files and the newhack/talos-installer-index_test.batspass under dash.Screenshots
No UI changes.
Downstream repositories
I left every box empty and opened nothing in other repositories. Two repositories are touched, and neither follow-up is needed before this merges:
cozystack/website names the amd64 assets in
content/en/docs/next/install/talos/iso.mdandinstall/providers/oracle-cloud.md. If the release commit goes in, the install pages could offer the arm64 downloads. No asset is renamed, so the current links keep working.cozystack/talm points at
ghcr.io/cozystack/cozystack/talos:<version>incharts/cozystack/values.yaml. Once a release publishes that tag as a two-arch index, the same reference works on arm64 nodes, so nothing there has to change.No downstream repository is affected by this change
cozystack/website - follow-up:
cozystack/terraform-provider-cozystack - follow-up:
cozystack/ansible-cozystack - follow-up:
cozystack/ccp - follow-up:
cozystack/talm - follow-up:
cozystack/cozyhr - follow-up:
cozystack/cozy-proxy - follow-up:
cozystack/cozystack-telemetry-server - follow-up:
cozystack/external-apps-example - follow-up:
cozystack/examples - follow-up:
cozystack/community - follow-up:
Release note
Summary by CodeRabbit
New Features
Bug Fixes
Documentation