Repository navigation
Integration with Proxmox (paas-proxmox bundle) #69
Description
Activity
- changed the title
[-]Integration with Proxmox (paas-proxmox)[/-][+]Integration with Proxmox (paas-proxmox bundle)[/+]on Apr 9, 2024 I plan to implement LINSTORE directly into Proxmox itself, but integrating LINSTORE and proxmox disk management into the cozystack using the cluster API and possibly the create operator, this still needs to be discussed and most likely it will be a separate task.
I see two options how can you utilize Proxmox with cozystack.
Option one, where you create management cozystack cluster inside the proxmox VMs:
This is more safe, as it allows to isolate cozystack from the hardware nodes, but it still runs databases and tenant Kubernetes clusters for multiple users in the same virtual machines of management cluster.
There is an another option, where proxmox nodes become used the same way as Kubernetes nodes, and we just replace KubeVirt virtualization by Proxmox virtualization:
This setup looks more interesting. As it is more native to cozystack approach, but also allows you to simplify management of hypervisor by providing the oportunity for installing LINSTOR and extra things on it.
Personaly I like the second option more.
Another question is how to provide stable Kubernetes on Proxmox nodes. I was doing the following steps:
- create three small VMs on proxmox and install TalosLinux on them - they will be used as control-plane nodes for our hardware cluster
- Join proxmox nodes as workers to these VMs:
VIP="192.168.100.5" mkdir -p /etc/kubernetes/pki talosctl -n "$VIP" cat /etc/kubernetes/kubeconfig-kubelet > /etc/kubernetes/kubelet.conf talosctl -n "$VIP" cat /etc/kubernetes/bootstrap-kubeconfig > /etc/kubernetes/bootstrap-kubelet.conf talosctl -n "$VIP" cat /etc/kubernetes/pki/ca.crt > /etc/kubernetes/pki/ca.crt sed -i "/server:/ s|:.*|: https://${VIP}:6443|g" \ /etc/kubernetes/kubelet.conf \ /etc/kubernetes/bootstrap-kubelet.conf clusterDomain=$(talosctl -n "$VIP" get kubeletconfig -o jsonpath="{.spec.clusterDomain}") clusterDNS=$(talosctl -n "$VIP" get kubeletconfig -o jsonpath="{.spec.clusterDNS}") cat > /var/lib/kubelet/config.yaml <<EOT kind: KubeletConfiguration apiVersion: kubelet.config.k8s.io/v1beta1 authentication: anonymous: enabled: false webhook: enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook clusterDomain: "$clusterDomain" clusterDNS: $clusterDNS runtimeRequestTimeout: "0s" cgroupDriver: systemd EOT systemctl restart kubelet
I was using this setup for a long time it works quite well. Another option is using k3s or something like that.
Reacted by Nguyen Duc Quynh, spa-skyson and mr-addamsAndrei Kvapil (@kvaps) You presented a pretty good idea, but I would like to give customers the opportunity to determine what type of isolation will be for Database as a service and other services that are in a cluster mode or have two replica instances. That is, so that they can deploy them in LXC or KVM.
Tenant k8s workers will be in VM only.
For now the minimum installation will be on 2 servers, but ideally on three.
So you want to make Cozystack manage LXC containers and run DBs inside of them?
I guess this would be really challengeable, as you will need to replace operators with the custom logic.
Integrating process for proxmox
- Prepare ansible role install 3 proxmox server - done
-
Install LINSTOR as shared storage on proxmox- will be used current default cozystack solution - Prepare setup script cozystack in VMs - in progress 95% done
- Integrate proxmox servers to cozystack as a workers in managment k8s
- Integrate Proxmox CSI - in progress 99% done, writing test
- Integrate Proxmox CSI node - assessment of the complexity of integration - testing
- Integrate Proxmox CCM - testing
- Use internal network for proxmox and for LINSTOR based on VLAN - minimal requirments DRBD 9.2.9
-
Investigate Kubemox for manage LXC- not suitable for use - Integrate Cluster API - part implemented by Remi, in correction process.
- Intergate MetaLB or haproxy - simple method MetalLB
-
Сhanges in service packages for the ability to run on local disks- use LINSTORE
So you want to make Cozystack manage LXC containers and run DBs inside of them?
Andrei Kvapil (@kvaps) To give the user the opportunity to choose either LXC or VM, if the implementation will be difficult, then it will not be done until it is done, but it is worth including such an opportunity in the architecture.
If it can help:
A Proxmox CCM Project
https://github.com/sergelogvinov/proxmox-cloud-controller-managerA Proxmox CSI Project:
https://github.com/sergelogvinov/proxmox-csi-pluginReacted by Andrei KvapilReacted by Marian Koreniuk- linked a pull request that will close this issuefeat: Proxmox Integration Roadmap and Documentation (September 2025) #107
on Apr 25, 2024 If it can help:
A Proxmox CCM Project https://github.com/sergelogvinov/proxmox-cloud-controller-manager
A Proxmox CSI Project: https://github.com/sergelogvinov/proxmox-csi-plugin
Thx, we know about these projects, the integration for them is already ready.
Marian Koreniuk (@themoriarti) Can I help you with the integration of Proxmox into Cozystack? And if yes, how?
Reacted by Marian KoreniukMarian Koreniuk (@themoriarti) Can I help you with the integration of Proxmox into Cozystack? And if yes, how?
Sure, we are always open to cooperation, in this thread there is an high architectural design #69 (comment) , there is a check list for the integration process #69 (comment) and #69 (comment) , you can take any of the parts and start preparation, branch is tied to this submission. If some kind of discussion is needed, then there is either a Slack or Telegram channel, or we can create a separate channel for integration, for example, in Telegram. Open to any suggestions and help.
Regarding high architectural design, I would like to make the suggestions below:
- Kube-OVN: keep its use and use hookscripts in Proxmox to add the necessary information to the tap/veth interface of the VM/Container to make it compatible with OVN.
- LINSTOR: Give the possibility to choose between LINSTOR or CEPH. For our part, we prefer the use of Ceph. The proxmox-csi will do this without any problem.
- L2 Connectivity: we also think that using VLAN is very good.
- Cluster-API: it seems that a project already exists https://image-builder.sigs.k8s.io/capi/providers/proxmox. It's not enough to change the Infrastructure Provider to Proxmox? I'm sorry, I lack knowledge on the subject.
- Load-Balancers: Do you want to set up layer2/layer3 load-balancers? If so, MetalLB will be best. For layer4 load-balancers, we prefer HAProxy.
I can work on integrating OVN into Proxmox. In my opinion, we should be able to manage this with a hookscript that will make a call to the Kube-OVN API to retrieve the iface-id of the VM and apply it to the tap/veth interface.
This would keep the Tenants isolated using Kube-OVN. We could even create layer 3 Load-Balancers in Kube-OVN. MetalLB would therefore only be used to provide a pool of IPs.
I can also try working on Cozystack's CAPI.
Could you tell me the files containing the code that needs to be adapted?Reacted by Marian Koreniuk34 remaining items
Hi, Andrei Kvapil (@kvaps). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.
Issue Summary:
- You and Marian Koreniuk (@themoriarti) have made significant progress integrating management and tenant Kubernetes clusters with Proxmox VMs, including proxmox-csi, proxmox-ccm, and switching from kube-ovn to Cilium.
- The integration is about 90% complete and considered production-ready, covering VLAN setup, LINSTOR storage, Cluster API, and MetalLB load balancing.
- The main remaining blocker is automating VM provisioning.
- Future plans include adding LXC runtime support for databases and allowing user choice of runtime (LXC/VM/Pod), pending the proxmox-lxcri OCI runtime project.
- Documentation and a WIP branch are available, and collaboration is encouraged.
Next Steps:
- Please let me know if this issue is still relevant to the latest version of the cozystack repository by commenting here to keep the discussion open.
- Otherwise, I will automatically close this issue in 7 days.
Thank you for your understanding and contribution!
Reacted by Marian KoreniukAndrei Kvapil (@kvaps) It remains to make a final list of tests when installing, maintenance when updates and unit tests and e2e are made, will it be correct to translate everything into GO language?
Hi, Andrei Kvapil (@kvaps). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.
Issue Summary:
- Integration of CozyStack management with tenant Kubernetes clusters on Proxmox VMs is about 90% complete and production-ready.
- Stable ProxmoxCluster operation and MetalLB load balancing are in place.
- Main blocker is automation of VM provisioning.
- Future plans include LXC runtime support for databases and user choice of runtime (LXC/VM/Pod) pending proxmox-lxcri OCI runtime project.
- Comprehensive documentation and testing frameworks exist; collaboration continues on the WIP branch.
Next Steps:
- Please confirm if this issue is still relevant with the latest version of CozyStack and if you plan to continue progress.
- If no further updates are provided, this issue will be automatically closed in 7 days.
Thank you for your understanding and contribution!
Reacted by Marian KoreniukThe issue is still relevant, but the current status description is misleading.
The system is not production-ready due to the lack of automated VM provisioning. This is a hard blocker, not a “remaining 10%”.
Next step is to define and implement a deterministic provisioning layer for Proxmox VMs. Without this, the platform cannot be considered operational.
Proposed direction:
- Define provisioning interface (API / Cluster API / Terraform)
- Integrate it with cluster lifecycle (create/update/delete)
- Ensure idempotency and reproducibility
LXC runtime and multi-runtime support are separate concerns and should not block the base infrastructure delivery.
I will continue work in this direction.
Reacted by Ivan KolodrivskyiThis issue has had no activity for 60 days and was marked
lifecycle/stale.
It will be closed in 14 days unless commented or labelledlifecycle/frozen.Closed because no activity followed the
lifecycle/stalewarning.
Reopen if the issue is still relevant.- added a commit that references this issue
on Sep 25, 2026
phase1: adapting the management cluster to work on Proxmox VMs
we need to add the following components:
disable kube-ovn (leave only Cilium)phase1.5: how to achieve L2 connectivity with Proxmox?
phase2: adapting tenant clusters to work on Proxmox VMs: