Repository navigation
feat(vpc,kubernetes): back VPC subnets with Proxmox VLANs - #4813
Draft
Marian Koreniuk (themoriarti) wants to merge 8 commits into
Draft
Marian Koreniuk (themoriarti) wants to merge 8 commits into
Marian Koreniuk (themoriarti) wants to merge 8 commits into
Conversation
….15.0
The v0.15.0 release binary is built with gitVersion "0.15.0", without the
leading v. component-base v0.37 parses that version in the init() of its
legacy metrics registry and panics:
panic: version string "0.15.0" doesn't match expected regular
expression: "^v(\d+\.\d+\.\d+)"
so the Pod never starts. On a test cluster the CCM crash-looped for 30
hours (268 restarts); the one existing worker kept the providerID it
already had, but no new Node could be initialised.
v0.16 builds with goreleaser and sets gitVersion to v{{.Version}}. The capmox
provider-id mode this template configures (proxmox://<SMBIOS UUID>) is the
same in v0.16 - the refactor there only adds parsing of that format.
A unit test now refuses the v0.15.0 tag.
Assisted-by: LLM
Signed-off-by: Marian Koreniuk <[email protected]>
Kube-OVN parses Vpc.spec.vpcPeerings[].localConnectIP as a CIDR (util.CheckCidrs). A bare address fails that check, and the error aborts the whole Vpc reconcile, so a peered VPC also stops applying its static and policy routes. Both ends already take neighbouring addresses of one /30; writing the prefix makes them parse. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
A Proxmox VM can join a VPC subnet only if the subnet's logical switch reaches the hypervisors on a VLAN. Something has to pick that VLAN, keep two subnets from sharing it, and keep it until the last VM on the subnet has given its address back. Helm cannot do any of that: it has no allocator, and deleting a release would take the Vlan away from VMs that still use it. ProxmoxNetworkZone is the admin's side (bridge, Kube-OVN ProviderNetwork, VLAN range, transit for egress); ProxmoxNetwork is one request for one subnet. The controller names the Kube-OVN Vlan after the Subnet, so after a restart it finds every allocation in the cluster instead of allocating again, and it holds the Vlan, the Subnet and the IP pool with finalizers until the pool has no address in use. Kube-OVN drops ARP that arrives from a localnet port for the IP of a distributed router port, so a VM outside the cluster never resolves its gateway. Until Kube-OVN can do this itself, the controller puts the router port of each Proxmox-backed subnet into an HA_Chassis_Group of the zone's trunk nodes. The group is keyed by VPC, so every such port of one VPC is active on the same chassis and the path to the VPC's egress gateway never crosses Geneve, where Cilium's kube-proxy replacement rewrote LoadBalancer traffic. It writes to the OVN NB with Kube-OVN's own client certificate, read through the API, and refuses a group whose external_ids name another owner. The Kube-OVN and IPAM types are mirrored with only the fields read here, so this module does not take on their k8s.io dependencies. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
capmox takes any bridge and VLAN tag a ProxmoxMachine names, and Multus resolves an attachment in whatever namespace its name carries. Without admission a tenant who may create machines, VMs or pods could put a NIC on another tenant's VLAN or on the transit network every egress gateway shares. The package therefore ships ValidatingAdmissionPolicies next to the controller: a Proxmox NIC on a zone bridge must be one of its own namespace's networks, and a KubeVirt VM or a pod may attach only networks of its own namespace. The CRDs are a separate component installed first, so the zones in the package's values render on the first install rather than on a later re-render that Flux never performs. The package also carries the trunk ProviderNetwork, the zones and their transit network with an ACL that keeps the egress gateways of different VPCs apart, metrics, alerts, and a read-only role that lets a tenant see its own networks. It is opt-in in the iaas bundle for the same reason capmox is: nothing works until the hypervisors and the nodes carry the trunk. It brings the in-cluster IPAM provider along, because the VM addresses of every Proxmox-backed subnet live in an InClusterIPPool. The image is in the root build list, so the build stamps its reference like every other first-party image. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
Proxmox VMs, the workers of a Proxmox-backed Kubernetes cluster among them, have so far lived on a raw bridge and an address range handed out by the admin, outside any VPC. With subnets[].proxmox the tenant puts them on a subnet of its own VPC instead: the subnet becomes a VLAN-backed logical switch of the VPC's router, the router port is the VMs' gateway, and VMs and pods on the subnet are L2 neighbours. The VLAN, bridge and pool come from the platform and the tenant cannot set them. VM addresses come from one namespaced InClusterIPPool over vmRange, and the same range is excluded from Kube-OVN, so pods and VMs never get the same address. The pool skips the IPAM provider's delete webhook: that webhook refuses a pool with addresses in use, which left helm uninstall stuck half way, while the provider's finalizer already holds the pool until the last address is released. A VM outside the cluster has no node to masquerade through, so egress.enabled adds a VpcEgressGateway that SNATs traffic leaving the VPC onto the zone's transit network; a route narrower than 0.0.0.0/0, such as one to a peered VPC, keeps its next hop. A static route's next hop must now lie inside the VPC or on a peering link: one on the shared transit network would hand the VPC's traffic to another VPC's egress gateway. A Proxmox-backed subnet cannot be turned back into an overlay subnet in place, nor dropped while VMs hold addresses on it, because Kube-OVN would delete the logical switch under them. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
A Proxmox-backed cluster had to be given a raw bridge, a VLAN tag and an address range, values the tenant has to get from the admin and that admission cannot tell apart from someone else's. proxmox.network.vpc and .subnet name a Proxmox-backed subnet of the tenant's own VPC instead. The charts read the subnet's ProxmoxNetwork in the release namespace and give capmox its bridge, VLAN and InClusterIPPool, so every worker address comes from the subnet's pool. A pool can add more NICs the same way with proxmox.additionalNetworks. A raw bridge or vlan next to a subnet is refused rather than merged: it would either be ignored or override the platform's allocation, and overriding it is how a tenant would reach a VLAN that is not its own. capmox v0.7 refuses a ProxmoxCluster without ipv4Config and builds a pool from it. For a VPC-backed cluster that pool holds only the gateway, which the IPAM provider never hands out, so a machine without a pool reference waits instead of taking an address the subnet's pool may also hand out. The lookup lives in cozy-lib so both charts fail with the same message when the subnet has no ProxmoxNetwork or it is not Ready yet. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
Unit and chart tests cannot show how the controller, the vpc chart, Kube-OVN and admission behave together, and part of it exists only on a real hypervisor: the inner VLAN tag surviving the uplink, the VPC router port answering ARP from outside the cluster, a CAPI worker joining through the egress gateway, and isolation between tenants at the VM level. proxmox-network runs in the regular lanes. The e2e cluster has no Proxmox and no spare NIC, so its provider network selects no node: the zone stays not Ready and every network stops at NoTrunkNodes, which still shows that the controller reached the OVN NB and found the router port. The suite installs the vpc chart from the checkout with --wait=false, because the application's release would wait on a network that cannot become Ready there. It checks VLAN allocation, exhaustion and release, the Vlan, Subnet, pool and OVN localnet tag, a network held while an address on it is in use, and admission of pods and KubeVirt VMs by server-side dry run, each refusal next to an admitted control. The platform install enables the opt-in package; a run that does not select the suite disables it again through the selector's complement. proxmox-network-env needs a Cozystack cluster on Proxmox VE with trunk NICs, a zone bridge and a transit router, so it ships as chainsaw-test.yaml.disabled like kubernetes-proxmox and is run by hand. Its phases live in _lib/run-proxmox-network.sh and take every detail of the environment from COZY_PXNET_* variables. Every isolation check is paired with a positive control, so a check cannot pass only because the path under test is broken altogether, and the design's open questions are reported without failing the suite. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
The feature spans the vpc, kubernetes and kubernetes-nodes charts, a new controller, Kube-OVN, capmox, the in-cluster IPAM provider and the hypervisors, and its contract with the admin (trunk, zone, transit) lives in none of them. The document records the ownership of every object, the lifecycle and failure handling, the security model, operations, the questions a live run on Proxmox VE left open, and a phased plan for Proxmox SDN. It points to the design proposal in cozystack/community and to cozystack#3481, where Proxmox as a platform is discussed. Assisted-by: LLM Signed-off-by: Marian Koreniuk <[email protected]>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
area/kubernetes
This was referenced Oct 7, 2026
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| MinVersion: tls.VersionTLS12, | ||
| Certificates: []tls.Certificate{cert}, | ||
| // The chain is verified below, without a host name. | ||
| InsecureSkipVerify: true, // #nosec G402 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Part of #3481. Design proposal: cozystack/community#93
This is the networking piece for Proxmox-backed tenant clusters, on top of the provisioning layer that #4087, #4483, #4088 and #4089 put in
main. It is opened as a draft so that the proposal can be discussed first; the decision on Proxmox as a supported platform stays in #3481.A tenant can extend a subnet of its VPC onto Proxmox VE by adding
proxmox: {}to the subnet, and put Proxmox VMs on it, the workers of a Proxmox-backedKubernetescluster included. The subnet becomes a VLAN-backed logical switch of the tenant's own VPC router: VMs and pods on it are L2 neighbours, the router port is the VMs' gateway, and the VPC's isolation covers the VMs too. The platform picks the VLAN, the bridge and the IP pool, and the tenant cannot set them. Until now a Proxmox-backed cluster needed a raw bridge, VLAN tag and address range from the admin, outside any VPC, and admission had no way to tell a tenant's own VLAN from someone else's.The owner of every object, lifecycle, failure handling, the security model and operations are in
docs/proxmox-networking.md.proxmox.cozystack.io/v1alpha1:ProxmoxNetworkZone(cluster-scoped, owned by the admin: bridge, Kube-OVN ProviderNetwork, VLAN range, transit network for egress) andProxmoxNetwork(one VLAN request for one subnet, rendered by the vpc chart).proxmox-network-controller: allocates a Kube-OVNVlanfrom the zone's range and names it after the Subnet, so a restart recovers every allocation instead of allocating again; holds the Vlan, the Subnet and the IP pool with finalizers until no VM holds an address; puts each Proxmox-backed router port into an OVNHA_Chassis_Group(see below); annotates each namespace with its ready networks for admission; collects orphaned Vlans; exportscozy_proxmox_*metrics.packages/system/proxmox-network(andproxmox-network-crds, installed first): opt-in in the iaas bundle withbundles.enabledPackages: [cozystack.proxmox-network], which also enables the in-cluster IPAM provider. It ships the controller, its RBAC, the trunk ProviderNetwork, the zones, a transit network whose ACL keeps the egress gateways of different VPCs apart, ValidatingAdmissionPolicies, alerts and a read-only role that lets a tenant see its own networks.subnets[].proxmox(zone,vmRange) andegress(enabled,replicas,internalCidr): a VpcEgressGateway that SNATs traffic leaving the VPC onto the zone's transit network. VM addresses come from one namespacedInClusterIPPoolovervmRange, which is excluded from Kube-OVN, so a pod and a VM never get the same address.proxmox.network.{vpc,subnet}instead ofproxmox.ipv4Config. kubernetes-nodes:proxmox.network.{vpc,subnet}for net0 andproxmox.additionalNetworks[]for more NICs. The charts read the subnet'sProxmoxNetworkand give capmox the bridge, the VLAN and anipv4PoolRef, so every worker address comes from the subnet's pool. A rawbridge/vlannext to a subnet is refused.hack/e2e-chainsaw/proxmox-networkruns in the regular lanes and needs no hypervisor. The e2e nodes have one NIC, so its provider network selects no node and every network stops atNoTrunkNodes, which still shows that the controller reached the OVN NB and found the router port. It covers the package and CRDs, VLAN allocation, exhaustion and release, the Kube-OVN objects and the localnet tag, a network held while an address is in use, and admission of pods and KubeVirt VMs, each refusal next to an admitted control.hack/select-e2e.shmaps the package to the suite, and runs that do not select it disable the package again.hack/e2e-chainsaw/proxmox-network-envneeds Proxmox VE with trunk NICs, a zone bridge and a transit router. It ships aschainsaw-test.yaml.disabled, likekubernetes-proxmox, and is run by hand (see itsREADME.md); every environment detail comes fromCOZY_PXNET_*variables.The commits can be reviewed one by one. The first two are independent fixes, listed separately in the proposal; they can move to separate PRs if that is preferred:
fix(kubernetes): the Proxmox CCM moves from v0.15.0, which panics at start on its own version string, to v0.16.1.fix(vpc): each end of a VPC peering gets a/30. Kube-OVN parseslocalConnectIPas a CIDR and fails the whole Vpc reconcile, routes included, on a bare address.Changes that reach existing installations
nextHopIPmust lie inside one of the VPC's subnets, or in 169.254.0.0/16 when peers are declared. A VPC with a next hop elsewhere stops rendering after the upgrade withroute to <cidr>: next hop <ip> is outside every subnet of this VPC. A next hop on the shared transit network would otherwise hand a VPC's traffic to another VPC's egress gateway.localConnectIPnow carries/30(thefix(vpc)commit).cozy-*,kube-system, the package namespace, the egress gateway namespaces andadmission.exemptNamespaces. A pod or a KubeVirt VM may attach only NetworkAttachmentDefinitions and Kube-OVN subnets of its own namespace, and a*.kubernetes.io/logical_switchpin other than the namespace's default switch is refused on CREATE, so a copy of a running pod that keeps Kube-OVN's annotations (kubectl debug --copy-to, a restored backup) is refused as well.admission.vmNetworkNamespaceandadmission.podNetworkNamespaceswitch them off.The controller writes to the OVN northbound database
Kube-OVN drops ARP that arrives from a localnet port for the IP of a distributed router port, so a VM outside the cluster never resolves its gateway. Making the router port a gateway port with an
HA_Chassis_Groupof the zone's trunk nodes removes that drop. Kube-OVN has no field for it, so the controller manages one group per Proxmox-backed router port,px-<port>, withexternal_idsowner=proxmox-networkandnetwork=<namespace>/<name>. It refuses a group another owner has marked, removes the group before it frees the VLAN, and reportsGatewayReadyandcozy_proxmox_network_gateway_*. It connects with Kube-OVN's client certificate, read through the API with a Role limited byresourceNames, and writes through the Service that selects the RAFT leader. The groups are keyed by VPC, so every Proxmox-backed port of one VPC is active on the same chassis: with the ports of one VPC on different chassis, the path to the egress gateway crossed Geneve, where Cilium's kube-proxy replacement rewrote traffic to a LoadBalancer IP.--manage-gateway-chassis=falseturns it off. This is meant to go away once Kube-OVN sets the group for such subnets itself; that change has not been proposed to Kube-OVN yet.Known limits
docs/proxmox-networking.md§8, §10, §12 question 9; an open question of the proposal).gateway_mtu, so a full-size DF packet from a VM to an overlay pod is dropped silently.How this was tested
On this branch, rebased on
main:make unit-tests test-controllers: the Go targets (go-unit-tests,go-module-tests,test-controllers,test-check-readiness,test-backport-audit,rd-presets-check,migrations-target-check),helm-unit-testsandbats-posix-compat-testspass.bats-unit-testsshows the same failures asmainon the same host and none of its own (local environment: no docker, a different yq; listed on request).go test -racefor./internal/proxmoxnetworkcontroller/...,./api/proxmox/...,./cmd/proxmox-network-controller/...andapi/apps/v1alpha1, against a fake OVN NB for the gateway.go vet,gofmtandgo build ./...are clean.cozyvalues-genoutput are in sync. The one diff a localcozyvalues-genleaves,x-cozystack-namein the kubernetes schema, is the same onmain.chainsaw lint teston both suites;bash -nand shellcheck on the suite helper; the guard suiteschainsaw-leak-guard_test,chainsaw-op-budgets,bats-runner-coverage,platform-packages_test(with a new test for the opt-in package),run-kubernetes-install-wait_test,select-e2e_testandselect-install_test;hack/check-commit-trailers.shon the range.End to end, the shell form of the environment suite ran on a two-host Proxmox VE 9 cluster (a VLAN-aware bridge whose tenant VLANs ride inside one outer VLAN between the hosts, MTU 9000) next to a Talos management cluster running Kube-OVN v1.15.10 with Cilium's kube-proxy replacement, capmox v0.7.7 and in-cluster IPAM v1.0.3. Two tenants, each with a VPC of two Proxmox-backed subnets and one overlay subnet, and a Proxmox-backed Kubernetes cluster per tenant whose worker has net0 on one subnet (and, for tenant A, net1 on the other):
The open checks are the known limits above; the skipped ones are cells that topology could not exercise, such as the gateways of two VPCs on different chassis. The resilience phase restarts the controller and kube-ovn-cni and stops capmox during scaling with no change to any allocation; the lifecycle phase deletes a VPC under live VMs (the Vlan, the Subnet and the pool stay held) and then the cluster, after which VLANs, pools, subnets, addresses, OVN groups and the namespace annotation are all gone.
mainnow ships Kube-OVN v1.15.26, so the gateway checks have to be repeated on it.Screenshots
No UI change.
Downstream repositories
Walking the trigger map against this diff, three repositories are reached: the proposal, the website and the Terraform provider. The website follow-up is a draft PR until this one is decided; the provider pins released API types, so its follow-up is an issue until a tagged release carries these fields.
content/en/docs/next/guides/platform-stack/_index.md,operations/configuration/licenses.md), the iaas bundle's new opt-in package (operations/configuration/variants.md), and an operator guide for zones and the trunk.subnets[].proxmox,egress), kubernetes (proxmox.network) and kubernetes-nodes (proxmox.network,proxmox.additionalNetworks) schemas.Release note
Assisted-by: LLM