Repository navigation
Proxmox as a supported platform: where the work stands and what a decision needs #3481
Description
Activity
the provisioning layer is the thing to specify first, and the rest is
reviewable in pieces afterwardsThat is what this is. The provisioning layer works, and the rest is split the
way the issue asks for.The hard blocker was not a capmox incompatibility
#69 called automated VM provisioning "a hard blocker, not a remaining 10%", and
in eight months nobody produced a VM throughProxmoxMachine. It turns out that
was four independent packaging and configuration causes, each hidden behind the
previous one:PROXMOX_URLcarried/api2/json, which capmox appends itself — every call
went to/api2/json/api2/json/…and returned 501.- No IPAM provider was installed. capmox allocates from an
InClusterIPPooland
has no DHCP mode, soProxmoxClusternever reachedReady. templateSelector.matchTagsis set equality, not containment. A template with
four tags is not matched by naming one:found 0 VM templates with tags.- capmox waits for the qemu-guest-agent, which Talos does not ship —
error waiting for agent: timed out.
None of them is in the provider. That is why the earlier work could look
complete and still never boot a machine, and it is the part I would ask a
reviewer to weigh: the thing that stalled this for two years was not a missing
capability.What runs now
On Proxmox VE 9.2.11, Talos v1.13.6, Cozystack's own CAPI stack:
Cozystack → CAPI + Kamaji + capmox + IPAM → linked-clone VM on Proxmox (8 KiB per worker, not 10.2 GiB) → Talos via the nocloud ISO → Ready node in a tenant cluster → CCM: providerID + topology labels → CSI: PVC bound to a real disk on the hypervisor$ KUBECONFIG=<tenant> kubectl get node -o wide kubernetes-px-md0-jmzds-h9npx Ready v1.35.6 10.0.0.160 Talos (v1.13.6) $ KUBECONFIG=<tenant> kubectl get pvc csi-probe csi-probe Bound pvc-831a2db6-… 1Gi RWO proxmox $ qm config 106 scsi1: vm-disks1:vm-9999-pvc-831a2db6-…,size=1GThree PRs, in the order you named
- Provisioning — capmox v0.7.7 and in-cluster IPAM v1.0.3 as optional
packages, plus the bundle wiring. Reviewable on its own. - Substrate switch — one
substrate: kubevirt | proxmoxvalue in
apps/kubernetesandapps/kubernetes-nodes, not parallel charts. The
KubeVirt path renders byte-for-byte unchanged; the golden snapshots pin it. - Storage and cloud-controller —
proxmox-csiandproxmox-ccm. Stacked on
2, because it is the substrate value it gates on.
make unit-testspasses on each branch independently, and each is codegen-clean.On the question the issue actually asks
You are right that the components working is a different question from Proxmox
being supported, and that the second one is a commitment rather than a merge.
Answering it directly rather than leaving it implied: I am offering to carry it.What that means concretely:
- Hardware. A Proxmox VE 9 host (48 threads, 251 GiB RAM, ZFS) dedicated to
this. That is the thing upstream CI does not have — e2e here runs on
oracle-vm-*runners, and a Proxmox substrate cannot be exercised on them at
all, which is a large part of why this has never been testable. - e2e. A self-hosted runner on that host, so the substrate is covered by the
same bats/chainsaw suite as everything else rather than by my word. I would
write the job and keep it green. - Maintenance. capmox tracks the Cluster API contract and moves with it —
v0.8 already declares v1beta2, which is why v0.7.7 is pinned here. Following
those bumps, keeping the vendored components current, and triaging Proxmox
issues is the recurring work, and I am taking it.
What does not exist today, plainly: there is no e2e coverage for Proxmox in
CI, and everything above was verified by hand — one hypervisor, one tenant
cluster. That is the gap. It is also the first thing I would close, and closing
it is only worth the effort if the direction is acceptable to you — so I am
asking for the answer rather than assuming it.Two limitations worth knowing before reviewing:
proxmox-cloud-controller-managerchart 0.2.30 is incompatible with PVE 9's
HA-groups API (ha groups have been migrated to rules, every five minutes).
Node initialization is unaffected.TalosConfigTemplate.specis immutable by schema, so the reconcile Job cannot
rewrite an apiserver endpoint it has already published. Pre-existing, but it
becomes visible on this substrate.
If the answer is still no, that remains more useful than limbo — and the four
causes above are worth recording either way, so the next person does not spend
eight months on them.The three pull requests are open, in the order above:
- feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages #4087 —
feat(cluster-api): the Cluster API providers, capmox v0.7.7 and in-cluster IPAM v1.0.3, as optional packages plus the bundle wiring. This is the provisioning layer. - feat(kubernetes): add a proxmox substrate for managed clusters and worker pools #4088 —
feat(kubernetes): thesubstrateswitch inkubernetesandkubernetes-nodes. The KubeVirt path renders byte-for-byte unchanged; the golden snapshots pin it. - feat(kubernetes): storage and cloud-controller integration for Proxmox-backed tenants #4089 —
feat(kubernetes): the Proxmox CSI driver and cloud-controller-manager.
#4087 and #4088 share no files and can be reviewed in parallel. #4089 builds on #4088, and since GitHub cannot base a cross-fork PR on another fork's branch, its diff carries #4088's commits until that one merges — the two that belong to it are the last two.
DCO, codegen-drift, pre-commit and the package builds pass on all three.
- feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages #4087 —
On shared storage, Proxmox-backed tenants cannot be isolated from each other at the volume level. That matters for the decision here. I found it reviewing #4089, but no change to that PR fixes it.
The storage and cloud-controller components keep a Proxmox token inside the tenant cluster, where the tenant can read it. The CSI driver and the cloud-controller-manager both run there with a Proxmox API token. The token lands in plain text, in the chart's Secrets and in the Helm release Secrets. The tenant is cluster-admin in that cluster. The ACL the upstream charts document for the token sits on
/. With it a tenant can read every VM's config, detach another tenant's disks, delete volumes, and attach any volume on the storage to one of its own workers.Narrowing the token does not help on shared storage. Every volume the CSI driver creates is owned by the same placeholder VMID. Deleting a volume needs
Datastore.Allocateon the storage, and the same privilege lets the holder attach any volume on it. So there is no token scope that lets one tenant manage its own volumes without reaching everyone else's. Per-tenant tokens would narrow VM access, not this.Tearing a tenant down leaves its data behind with no owner. Teardown removes the CSI driver first, and nothing deletes the PVCs. Destroying a VM only frees volumes owned by that VM's ID. The CSI volumes stay behind with their data, and since they all carry the same owner there is no way to tell afterwards whose they were.
So Proxmox-backed tenants need either storage that is not shared between them, or something outside the tenant cluster that holds the token and brokers volume operations. The series as it stands does neither. The rest of the findings are in the reviews on the three pull requests.
Reacted by Marian Koreniuk- added and removed
on Sep 23, 2026 - added 3 commits that reference this issue
on Sep 25, 2026 The series this issue asked for is in
main. Where it stands, and what is still open:Merged
- feat(kubernetes): add a proxmox substrate for managed clusters and worker pools #4088 (
a285a5ef3) —substrate: kubevirt | proxmoxinapps/kubernetesandapps/kubernetes-nodes; the KubeVirt path renders unchanged. - fix(capi): ship the core provider manifests the repository actually holds #4483 (
e883f6f7f) — the core Cluster API provider manifests, regenerated from the repository's own files. - feat(cluster-api): add the Proxmox infrastructure provider and in-cluster IPAM as optional packages #4087 (
4a2dea2d0) — capmox v0.7.7 and in-cluster IPAM v1.0.3 as optional packages, with the bundle wiring and sha256-pinned components. - feat(kubernetes): storage and cloud-controller integration for Proxmox-backed tenants #4089 (
6580d2eeb) — the Proxmox CSI driver and cloud-controller-manager.
The isolation point raised above, and what #4089 does about it
The token no longer reaches the tenant. Both the CSI controller and the CCM run in the management cluster, in the tenant's namespace, and read the operator's Secret there; the tenant cluster gets the node plugin, the CSIDriver object and the storage classes, none of which carry a credential. Two runs on a live stand scanned every Secret in the tenant cluster for the token and found none.
Scope is one ACL per tenant on a Proxmox resource pool:
proxmox.poolon everyKubernetesNodesof the cluster, capmox adds each worker VM to it as it clones, replacements included, and the README's table lists the privileges each token needs on/pool/<pool>and on the tenant's storage — checked live with privilege-separated tokens holding exactly those rows, including a Machine replacement that the CSI attached to with no ACL touched. One thing learned the hard way is now in the README: Proxmox caps a privilege-separated token at its user's permissions, so the roles have to be granted to the user as well.Volumes are owned per cluster (
controllerVmIDderived from the tenant namespace and the release name), so leftovers on shared storage are attributable. Deleting a cluster frees its disks: the pre-delete hook releases the claims while the driver is still up, deleting the claims before the Pods that hold them, and waits for the PVs.What stays open by design, as stated in the README: a storage shared between tenants cannot be scoped —
Datastore.Allocateon it is what deletes volumes and also what lets any volume on it be attached — so tenant-level isolation of volumes needs a storage per tenant. The controllers act on what the tenant's own PV objects name, which is the same reach a cluster-admin of the tenant has anyway.Also in #4089
The worker reconcile Job now replaces the TalosConfigTemplate when CABPT's webhook refuses to patch it (the kubelet flag this series adds changes that spec on existing pools). This changes the KubeVirt path too: every existing worker pool gets a new reconcile Job on upgrade, and a pool whose live template has drifted from the render gets it recreated instead of failing. Machines already running keep the bootstrap data they joined with. If #3523 lands, its hashed template name has to be reflected in that Job's RBAC
resourceNamesand in thekubectl deletetarget, as discussed on the PR.Not done
- e2e for the Proxmox substrate: the suite and its workflow are in the tree (
hack/e2e-chainsaw/kubernetes-proxmox,.github/workflows/e2e-proxmox.yaml), parked — the workflow runs onworkflow_dispatchor a nightly gated onvars.PROXMOX_NIGHTLY, and needs a self-hosted runner with theproxmoxlabel that this repository does not have. Until then the substrate is covered by unit tests and by manual runs against a stand, not by CI. - The Proxmox CCM image v0.15.0 logs
ha groups have been migrated to rulesagainst PVE 9.x every reconcile; node initialisation is unaffected.
I will keep following capmox's Cluster API contract bumps and triaging Proxmox issues, as offered above. Thanks to Aleksei Sviridkin (@lexfrei) for five rounds of review that reshaped this into something that can be trusted with a hypervisor token.
Reacted by Aleksei Sviridkin- feat(kubernetes): add a proxmox substrate for managed clusters and worker pools #4088 (
Aleksei Sviridkin (@lexfrei) Since the Proxmox integration PRs have now been merged and I have been added as a reviewer for the related components, would it make sense to update this issue with the current status and clarify the maintenance model for the Proxmox integration?
My understanding is that the Proxmox integration can continue to be developed and maintained as part of the open-source Cozystack project.
I’m happy to continue contributing to and maintaining this part of the project, including the related components, testing and further integration work. It would be useful to document this in the issue so that the current status and ownership are clear. FYI Timur Tukaev (@tym83) Timofei Larkin (@lllamnyp) Andrei Kvapil (@kvaps) myasnikovdaniil Kingdon Barrett (@kingdonb) Matthieu ROBIN (@matthieu-robin) mattia-eleuteri IvanHunters
- added a commit that references this issue
on Oct 7, 2026 The networking piece is up for discussion, on top of the provisioning layer that #4087, #4483, #4088 and #4089 put in
main:- Design proposal: docs(design-proposals): add Proxmox-backed VPC subnets community#93
- Implementation, as a draft until the proposal is discussed: feat(vpc,kubernetes): back VPC subnets with Proxmox VLANs #4813
It puts Proxmox VMs, the workers of Proxmox-backed tenant clusters included, on a subnet of the tenant's own VPC, with the VLAN, bridge and IP pool allocated by the platform rather than taken from tenant values. It does not change how VMs are provisioned, and it does not settle the question this issue asks.
- added a commit that references this issue
on Oct 11, 2026
Moving the Proxmox work out of a stalled pull request and into a place where it can be discussed, since a 32k-line draft is not a format anyone can respond to. This issue records where the effort actually stands. It is not a commitment to ship Proxmox support, and it does not carry a decision either way.
Where it stands
#1927 carried a
paas-proxmoxbundle preset: proxmox-csi, proxmox-worker, proxmox-ve, the Cluster API provider wiring and the platform bundle entries, across 157 files. It stayed a draft from January, conflicts with main, and nothing in that direction exists in the tree today beyond two module-install helper scripts under the linstor and piraeus packages. I closed it, because the diff had drifted far enough that reviewing it line by line would have told us nothing about the actual question.The earlier tracking issue, #69, was closed by the stale bot in July rather than by anyone deciding the topic. Its last substantive comment is worth repeating here, because it comes from the person who wrote the PR:
So the storage and networking pieces got most of the attention, while the layer that creates the VMs in the first place stayed unspecified. That ordering is what left the work unable to land.
What a decision would need
The open question is not whether the individual components work. It is whether Proxmox becomes a supported way to run Cozystack, which means someone maintains it, tests it, and answers issues about it for as long as it ships. That is a larger commitment than any single PR represents, and it should be made deliberately rather than accumulated by merging parts.
If the answer turns out to be yes, the provisioning layer is the thing to specify first, and the rest is reviewable in pieces afterwards: the CSI driver, the cloud-controller integration and the bundle wiring are separable and each small enough to reason about. If the answer is no, saying so plainly is more useful to everyone who has asked about Proxmox over the past two years than leaving the topic in limbo, which is where it has been.
Either outcome is fine. Drifting is what is not.