Repository navigation
feat: ARM64 support for Generic Kubernetes (non-Talos) #1961
Description
Summary
Enable ARM64 architecture support for Cozystack on generic Kubernetes clusters (k3s, kubeadm, RKE2, etc.) using the isp-full-generic variant.
This is a subset of the broader ARM initiative (#903) focused specifically on non-Talos deployments.
Status: Proof of concept validated. Base platform deployed on a 3-node ARM64 k3s cluster — all 84 HelmReleases reconciled, 130 pods running. Remaining work focuses on CI integration and fixing edge cases in specific packages.
Current Status
| Category | Count | Status |
|---|---|---|
| Custom images (total in project) | 42 | Audited |
| ARM64-ready (built natively or switched to upstream) | 36 | ✅ |
| Deployment-tested on ARM64 | 84/84 HelmReleases | ✅ All reconciled |
| Remaining blockers | 3 | 🔴 See below |
Build infrastructure already supports multi-arch via PLATFORM variable (#519).
Completed Work
Images built natively for ARM64
All custom Cozystack images (28 total) were built on ARM64 hardware and verified. This includes own code (Go controllers, dashboard, proxies), patched upstreams (kamaji, kilo, redis-operator, linstor-csi, cluster-autoscaler, kubevirt-cloud-provider, clickhouse-backup, mariadb-backup, s3manager, COSI), and re-published images (kubevirt-csi-driver, nginx-cache).
Additionally, cozy-proxy and local-ccm from separate repositories were built (local-ccm required fixing hardcoded GOARCH=amd64).
Images switched to multi-arch upstream (8)
These were previously republished without patches. Upstream now provides multi-arch builds:
| Image | Upstream replacement | Notes |
|---|---|---|
| cilium | quay.io/cilium/cilium:v1.18.6 |
Multi-arch upstream |
| metallb-controller | quay.io/metallb/controller:v0.15.2 |
Multi-arch upstream |
| metallb-speaker | quay.io/metallb/speaker:v0.15.2 |
Multi-arch upstream |
| piraeus-server | quay.io/piraeusdatastore/piraeus-server:v1.32.3 |
Multi-arch (4 cozystack patches not applied — acceptable for testing) |
| kubeovn | docker.io/kubeovn/kube-ovn:v1.14.25 |
Multi-arch (custom leader-checker not included — acceptable for testing) |
| objectstorage-controller | registry.k8s.io/sig-storage/objectstorage-controller:v0.2.2 |
All COSI patches merged upstream |
| objectstorage-sidecar | registry.k8s.io/sig-storage/objectstorage-sidecar:v0.2.2 |
All COSI patches merged upstream |
| fdb-kubernetes-monitor | foundationdb/fdb-kubernetes-monitor:7.3.63 / 7.4.1 |
amd64-only 7.1 init container removed; 7.3+ are multi-arch |
Upstream patches merged (can be dropped on next update)
| Repo | Patch | PR |
|---|---|---|
clastix/kamaji |
Label preservation on mutate | #992 |
piraeusdatastore/linstor-csi |
RWX validation | #403 |
kubernetes-sigs/container-object-storage-interface |
Deletion handling | #89 |
kubernetes-sigs/container-object-storage-interface |
Bucket name format | #90 |
apple/foundationdb |
ARM64 fdb-kubernetes-monitor | #12140 |
Issues resolved during deployment
| Issue | Resolution |
|---|---|
fdb-kubernetes-monitor:7.1.67 amd64-only |
Removed 7.1 init container, kept 7.3 + 7.4 (multi-arch upstream) |
| k3s built-in metrics-server overwrites Cozystack RBAC | Disabled k3s addon (--disable=metrics-server), forced Flux reinstall |
| COSI custom images | All patches merged upstream — switched to registry.k8s.io/sig-storage/* |
Remaining Work
1. CSI sidecar images in kubevirt-csi (blocker for nested Kubernetes)
KubeVirt CSI templates use OpenShift CSI sidecar images inherited from kubevirt/csi-driver upstream manifests. These are amd64-only.
Replace with official Kubernetes SIG Storage images (already used by piraeus/LINSTOR in the same project):
| Current | Replacement |
|---|---|
quay.io/openshift/origin-csi-livenessprobe:latest |
registry.k8s.io/sig-storage/livenessprobe:v2.17.0 |
quay.io/openshift/origin-csi-node-driver-registrar:latest |
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.15.0 |
quay.io/openshift/origin-csi-external-provisioner:latest |
registry.k8s.io/sig-storage/csi-provisioner:v6.1.0 |
quay.io/openshift/origin-csi-external-attacher:latest |
registry.k8s.io/sig-storage/csi-attacher:v4.10.0 |
Why this replacement is safe
OpenShift images are forks of upstream Kubernetes CSI:
- kubernetes-csi/livenessprobe →
openshift/csi-livenessprobe - kubernetes-csi/node-driver-registrar →
openshift/csi-node-driver-registrar - kubernetes-csi/external-provisioner →
openshift/csi-external-provisioner - kubernetes-csi/external-attacher →
openshift/csi-external-attacher
Precedent: Piraeus/LINSTOR CSI in the same Cozystack deployment already uses registry.k8s.io/sig-storage/* images at these exact versions. The replacement aligns kubevirt-csi with the existing pattern.
Impact on existing users: None expected.
- Same CSI specification
- Same core functionality
- No configuration or API changes required
Benefits:
- ARM64 support (amd64, arm64, ppc64le, s390x)
- Pinned versions instead of
:latest - Direct upstream maintenance
- Consistent with piraeus CSI sidecar images already in the project
2. Kilo CNI stage (latent build bug)
Upstream kilo builds multi-arch via Makefile: separate docker build --build-arg GOARCH=arm64 per architecture. Cozystack builds via docker buildx build --platform instead. BuildKit auto-sets TARGETARCH but does NOT override custom ARG defaults.
In packages/system/kilo/images/kilo/Dockerfile:
- Builder stage — uses
TARGETARCH(correct) - CNI stage — uses
ARG GOARCH=amd64(hardcoded, NOT overridden by BuildKit)
When building with PLATFORM=linux/arm64 via buildx, Go binaries compile for arm64, but CNI plugins download for amd64. Currently latent — native ARM64 builds work (host arch matches), and CI only builds amd64. Will break when multi-arch CI is enabled.
Kilo is the only package in cozystack with this pattern — all others use TARGETARCH directly.
Note: Kilo upstream is low-maintenance (Go 1.18, last release Apr 2023). Active fork: castai/kilo.
3. ubuntu-container-disk (blocker for nested K8s VMs)
Requires qemu-img and guestfish (libguestfs) to build the VM disk image. Cannot be cross-built — needs a Linux build environment. This image is required for nested Kubernetes clusters running inside KubeVirt VMs.
4. local-ccm GOARCH hardcoded (separate repo)
cozystack/local-ccm Dockerfile has GOARCH=amd64 hardcoded. Fix: use TARGETARCH or ARG GOARCH=${TARGETARCH:-amd64}. Affects bare-metal deployments using the local cloud controller manager.
Checklist
- Build all custom images for ARM64
- Switch re-published images to upstream multi-arch where possible
- Deploy and validate on ARM64 k3s cluster (84/84 HelmReleases)
- Replace CSI sidecar images in
packages/apps/kubernetes/templates/csi/deploy.yaml - Replace CSI sidecar images in
packages/system/kubevirt-csi-node/templates/deploy.yaml - Fix kilo Dockerfile CNI stage:
ARG GOARCH=amd64→ARG GOARCH=${TARGETARCH:-amd64} - Fix local-ccm Dockerfile: remove hardcoded
GOARCH=amd64(cozystack/local-ccm) - Add Linux CI step for ubuntu-container-disk build
- Enable
PLATFORM=linux/arm64in CI (orlinux/amd64,linux/arm64for multi-arch) - Verify full app stack on ARM64 (nested Kubernetes, databases, storage)
Out of Scope
Talos-specific blockers (tracked in #903):
- Talos profiles and assets hardcoded to amd64
- GitHub Actions workflows for Talos
Related
- Epic: [Epic] Cozystack on ARM #903
- Images checklist: More ARM versions of docker images for components #984
- Build infrastructure: Cross arch build for images #519 (done)
Activity
kvaps commentedon May 1, 2026
I shared my broader view on ARM support in #903: #903 (comment)
A few specific notes on the blockers listed here:
squat/kiloimage ([kilo] Switch from fork to upstream squat/kilo #2259).ubuntu-container-disk-v1.30…v1.35), so each version is its own pinned image instead of a single rebuild target.TARGETARCHplumbing).From my side, the main remaining blocker is the build pipeline shape: ARM has to run as a separate pipeline, or builds need to happen on different runners. Build time and reliability must not regress vs. what we have today for amd64. That's the firm constraint.
One option for that would be splitting the repository in two: keep
cozystack,cozystack-operator,cozystack-controller,lineage-webhook, and the core packages here, and move the rest into a separate pluggable repository. That would split the test surface along the same boundary and let each side's pipeline scale independently — including ARM, which could land first on the lighter framework half without waiting for the full image set.Before going down that path we'd need a basic CEP in cozystack/community covering the split boundary, the contract between the two repositories, and the migration plan.
dosubot commentedon Jul 31, 2026
Hi, Aleksei Sviridkin (@lexfrei). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.
Issue Summary
isp-full-genericvariant, separate from the broader Talos-focused ARM effort.kiloandubuntu-container-diskwere noted as no longer blocking, andkubevirt-csiandlocal-ccmwere expected to be straightforward to adapt.Next Steps
Thanks for your understanding and contribution.
fix(dashboard): honour PLATFORM instead of hardcoding linux/amd64
feat(build): build package images for amd64 and arm64 (#4498)
ci(release): build pre-release images for amd64 and arm64 and gate re…