Repository navigation
docs(design-proposals): add Proxmox-backed VPC subnets - #93
Marian Koreniuk (themoriarti) wants to merge 1 commit into
Conversation
Proxmox VMs, the workers of Proxmox-backed tenant clusters among them, need a place in a tenant's VPC instead of a raw bridge and an address range handed out by the admin. The proposal maps a VPC subnet onto a platform-allocated Proxmox VLAN, a Kube-OVN underlay subnet of the tenant's own VPC router and an in-cluster IPAM pool, and lists the open questions and the alternatives, Proxmox SDN among them. It is the networking piece on top of the provisioning layer that cozystack/cozystack#3481 asked to be specified first. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Aleksei Sviridkin (lexfrei)
left a comment
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
Marian Koreniuk (@themoriarti) The problem is real. On main a tenant picks the Proxmox bridge and vlan for its workers in the kubernetes-nodes values, and the address range in proxmox.ipv4Config, so nothing stops one tenant from putting its workers on another tenant's VLAN. That gap needs closing. I don't think this design is the way to close it now, for two reasons.
First, the size. The isolation gap only needs the platform to own the bridge, VLAN and pool of each tenant network: the VLAN allocator, the namespaced InClusterIPPool, and the admission policy that keeps machine NICs on the namespace's own networks. Everything else here is a feature on top of that. It puts VMs on L2 next to pods of a VPC subnet, adds a controller that writes HA_Chassis_Group rows into the OVN northbound database with Kube-OVN's own certificate, runs egress gateways with port security off, and needs a transit router outside the cluster. The NB writer is the part I would not take into the tree. The proposal itself says the right fix is in Kube-OVN, the gateway cells were run on v1.15.10 while main ships v1.15.26, and every Kube-OVN bump would have to be checked against it. So my answer to open question 2 is to wait for Kube-OVN, or to send the change there first.
Second, the boundary between tenants. The Goals say isolation holds at every layer, but traffic from one VPC to another VPC's CIDR is stopped only by the operator's transit router (Security, open question 1). Cozystack does not ship that router, does not test it, and cannot see when it is misconfigured. A boundary between tenants has to be something the platform owns, so the drop inside OVN belongs in the design and not in the open questions.
I'd split it. A smaller proposal for platform-allocated VLANs and pools with the admission policy closes the gap on main and stands on its own. The VPC integration can follow once Kube-OVN handles the gateway chassis and the drop between VPCs is designed in. Whether Proxmox becomes a supported platform is still the open question in cozystack/cozystack#3481, and a smaller first piece is easier to commit to while that is undecided.
2 participants
Design proposal for Proxmox-backed VPC subnets: a tenant puts Proxmox VMs, including the workers of a Proxmox-backed
Kubernetescluster, on a subnet of its ownVirtualPrivateCloud, and the platform allocates the VLAN, the bridge and the IP pool instead of taking them from tenant values.It comes from cozystack/cozystack#3481, which asks for a deliberate decision on Proxmox and for the provisioning layer to be specified first. That layer is in
mainnow (cozystack/cozystack#4087, cozystack/cozystack#4483, cozystack/cozystack#4088, cozystack/cozystack#4089); this proposal is the networking piece on top of it and does not change how VMs are provisioned. It meets #86 (per-cluster infrastructure backends) only at theproxmox.networkvalues of thekubernetesandkubernetes-nodesapplications, which would move with a Proxmox backend if #86 is accepted.What it asks reviewers to weigh:
proxmox.cozystack.io/v1alpha1, with an admin-ownedProxmoxNetworkZoneand a per-subnetProxmoxNetwork;HA_Chassis_Grouprows to the OVN northbound database, outside Kube-OVN, so that a VM reached through a localnet port can resolve its gateway, as a stopgap until Kube-OVN does it itself;VpcEgressGatewayonto a shared transit network, with the L3 boundary between VPCs held by an admin-run transit router until a platform-owned address space allows a drop inside OVN;The implementation is a draft pull request in cozystack/cozystack: cozystack/cozystack#4813. The proposal's numbers come from a run of its shell-based end-to-end suite on a two-host Proxmox VE 9 cluster; the Chainsaw suites that replace it have not run on a cluster yet.
Before review
decisions/directory, or says below why none is needed.Not a revision: this is a new proposal.
DCO
git commit --signoff).Assisted-by: LLM