Integration with Proxmox (paas-proxmox bundle)

phase1: adapting the management cluster to work on Proxmox VMs

we need to add the following components:

proxmox-csi
proxmox-ccm
Hybrid LINSTOR inside k8s + based on proxmox - in proggress
~~disable kube-ovn (leave only Cilium)~~

phase1.5: how to achieve L2 connectivity with Proxmox?

VLAN internal in one DC

phase2: adapting tenant clusters to work on Proxmox VMs:

Modify Cluster-API to order VMs in Proxmox (proxmox-infrastructure-provider) - in progress
Load balancers - what if we use MetalLB instead of kubevirt-ccm
Storage - proxmox-csi instead of kubevirt-csi

I plan to implement LINSTORE directly into Proxmox itself, but integrating LINSTORE and proxmox disk management into the cozystack using the cluster API and possibly the create operator, this still needs to be discussed and most likely it will be a separate task.

I see two options how can you utilize Proxmox with cozystack.

Option one, where you create management cozystack cluster inside the proxmox VMs:

This is more safe, as it allows to isolate cozystack from the hardware nodes, but it still runs databases and tenant Kubernetes clusters for multiple users in the same virtual machines of management cluster.

There is an another option, where proxmox nodes become used the same way as Kubernetes nodes, and we just replace KubeVirt virtualization by Proxmox virtualization:

This setup looks more interesting. As it is more native to cozystack approach, but also allows you to simplify management of hypervisor by providing the oportunity for installing LINSTOR and extra things on it.

Personaly I like the second option more.

Another question is how to provide stable Kubernetes on Proxmox nodes. I was doing the following steps:

create three small VMs on proxmox and install TalosLinux on them - they will be used as control-plane nodes for our hardware cluster
Join proxmox nodes as workers to these VMs:

VIP="192.168.100.5"

mkdir -p /etc/kubernetes/pki
talosctl -n "$VIP" cat /etc/kubernetes/kubeconfig-kubelet > /etc/kubernetes/kubelet.conf
talosctl -n "$VIP" cat /etc/kubernetes/bootstrap-kubeconfig > /etc/kubernetes/bootstrap-kubelet.conf
talosctl -n "$VIP" cat /etc/kubernetes/pki/ca.crt > /etc/kubernetes/pki/ca.crt

sed -i "/server:/ s|:.*|: https://${VIP}:6443|g" \
  /etc/kubernetes/kubelet.conf \
  /etc/kubernetes/bootstrap-kubelet.conf


clusterDomain=$(talosctl -n "$VIP" get kubeletconfig -o jsonpath="{.spec.clusterDomain}")
clusterDNS=$(talosctl -n "$VIP" get kubeletconfig -o jsonpath="{.spec.clusterDNS}")
cat > /var/lib/kubelet/config.yaml <<EOT
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
clusterDomain: "$clusterDomain"
clusterDNS: $clusterDNS
runtimeRequestTimeout: "0s"
cgroupDriver: systemd
EOT

systemctl restart kubelet

I was using this setup for a long time it works quite well. Another option is using k3s or something like that.

Andrei Kvapil (@kvaps)

Andrei Kvapil (@kvaps) You presented a pretty good idea, but I would like to give customers the opportunity to determine what type of isolation will be for Database as a service and other services that are in a cluster mode or have two replica instances. That is, so that they can deploy them in LXC or KVM.

Tenant k8s workers will be in VM only.

For now the minimum installation will be on 2 servers, but ideally on three.

So you want to make Cozystack manage LXC containers and run DBs inside of them?

I guess this would be really challengeable, as you will need to replace operators with the custom logic.

Integrating process for proxmox

Prepare ansible role install 3 proxmox server - done
~~Install LINSTOR as shared storage on proxmox~~ - will be used current default cozystack solution
Prepare setup script cozystack in VMs - in progress 95% done
Integrate proxmox servers to cozystack as a workers in managment k8s
Integrate Proxmox CSI - in progress 99% done, writing test
Integrate Proxmox CSI node - assessment of the complexity of integration - testing
Integrate Proxmox CCM - testing
Use internal network for proxmox and for LINSTOR based on VLAN - minimal requirments DRBD 9.2.9
~~Investigate Kubemox for manage LXC~~ - not suitable for use
Integrate Cluster API - part implemented by Remi, in correction process.
Intergate MetaLB or haproxy - simple method MetalLB
~~Сhanges in service packages for the ability to run on local disks~~ - use LINSTORE

Andrei Kvapil (@kvaps)

So you want to make Cozystack manage LXC containers and run DBs inside of them?

Andrei Kvapil (@kvaps) To give the user the opportunity to choose either LXC or VM, if the implementation will be difficult, then it will not be done until it is done, but it is worth including such an opportunity in the architecture.

If it can help:

A Proxmox CCM Project
https://github.com/sergelogvinov/proxmox-cloud-controller-manager

A Proxmox CSI Project:
https://github.com/sergelogvinov/proxmox-csi-plugin

If it can help:

A Proxmox CCM Project https://github.com/sergelogvinov/proxmox-cloud-controller-manager

A Proxmox CSI Project: https://github.com/sergelogvinov/proxmox-csi-plugin

Thx, we know about these projects, the integration for them is already ready.

Marian Koreniuk (@themoriarti)

Marian Koreniuk (@themoriarti) Can I help you with the integration of Proxmox into Cozystack? And if yes, how?

Marian Koreniuk (@themoriarti)

Marian Koreniuk (@themoriarti) Can I help you with the integration of Proxmox into Cozystack? And if yes, how?

Sure, we are always open to cooperation, in this thread there is an high architectural design #69 (comment) , there is a check list for the integration process #69 (comment) and #69 (comment) , you can take any of the parts and start preparation, branch is tied to this submission. If some kind of discussion is needed, then there is either a Slack or Telegram channel, or we can create a separate channel for integration, for example, in Telegram. Open to any suggestions and help.

Regarding high architectural design, I would like to make the suggestions below:

Kube-OVN: keep its use and use hookscripts in Proxmox to add the necessary information to the tap/veth interface of the VM/Container to make it compatible with OVN.
LINSTOR: Give the possibility to choose between LINSTOR or CEPH. For our part, we prefer the use of Ceph. The proxmox-csi will do this without any problem.
L2 Connectivity: we also think that using VLAN is very good.
Cluster-API: it seems that a project already exists https://image-builder.sigs.k8s.io/capi/providers/proxmox. It's not enough to change the Infrastructure Provider to Proxmox? I'm sorry, I lack knowledge on the subject.
Load-Balancers: Do you want to set up layer2/layer3 load-balancers? If so, MetalLB will be best. For layer4 load-balancers, we prefer HAProxy.

I can work on integrating OVN into Proxmox. In my opinion, we should be able to manage this with a hookscript that will make a call to the Kube-OVN API to retrieve the iface-id of the VM and apply it to the tap/veth interface.
This would keep the Tenants isolated using Kube-OVN. We could even create layer 3 Load-Balancers in Kube-OVN. MetalLB would therefore only be used to provide a pool of IPs.
I can also try working on Cozystack's CAPI.
Could you tell me the files containing the code that needs to be adapted?

Andrei Kvapil (@kvaps)

Hi, Andrei Kvapil (@kvaps). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

Issue Summary:

You and Marian Koreniuk (@themoriarti) have made significant progress integrating management and tenant Kubernetes clusters with Proxmox VMs, including proxmox-csi, proxmox-ccm, and switching from kube-ovn to Cilium.
The integration is about 90% complete and considered production-ready, covering VLAN setup, LINSTOR storage, Cluster API, and MetalLB load balancing.
The main remaining blocker is automating VM provisioning.
Future plans include adding LXC runtime support for databases and allowing user choice of runtime (LXC/VM/Pod), pending the proxmox-lxcri OCI runtime project.
Documentation and a WIP branch are available, and collaboration is encouraged.

Next Steps:

Please let me know if this issue is still relevant to the latest version of the cozystack repository by commenting here to keep the discussion open.
Otherwise, I will automatically close this issue in 7 days.

Thank you for your understanding and contribution!

Andrei Kvapil (@kvaps)

Andrei Kvapil (@kvaps) It remains to make a final list of tests when installing, maintenance when updates and unit tests and e2e are made, will it be correct to translate everything into GO language?

Andrei Kvapil (@kvaps)

Hi, Andrei Kvapil (@kvaps). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

Issue Summary:

Integration of CozyStack management with tenant Kubernetes clusters on Proxmox VMs is about 90% complete and production-ready.
Stable ProxmoxCluster operation and MetalLB load balancing are in place.
Main blocker is automation of VM provisioning.
Future plans include LXC runtime support for databases and user choice of runtime (LXC/VM/Pod) pending proxmox-lxcri OCI runtime project.
Comprehensive documentation and testing frameworks exist; collaboration continues on the WIP branch.

Next Steps:

Please confirm if this issue is still relevant with the latest version of CozyStack and if you plan to continue progress.
If no further updates are provided, this issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!

The issue is still relevant, but the current status description is misleading.

The system is not production-ready due to the lack of automated VM provisioning. This is a hard blocker, not a “remaining 10%”.

Next step is to define and implement a deterministic provisioning layer for Proxmox VMs. Without this, the platform cannot be considered operational.

Proposed direction:

Define provisioning interface (API / Cluster API / Terraform)
Integrate it with cluster lifecycle (create/update/delete)
Ensure idempotency and reproducibility

LXC runtime and multi-runtime support are separate concerns and should not block the base infrastructure delivery.

I will continue work in this direction.

This issue has had no activity for 60 days and was marked lifecycle/stale.
It will be closed in 14 days unless commented or labelled lifecycle/frozen.

Closed because no activity followed the lifecycle/stale warning.
Reopen if the issue is still relevant.

feat(cluster-api): add the Proxmox infrastructure provider and in-clu…

Navigation Menu

Repository navigation

Integration with Proxmox (paas-proxmox bundle) #69

34 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Integration with Proxmox (paas-proxmox bundle) #69

Description

Activity

themoriarti commented on Apr 12, 2024

kvaps commented on Apr 12, 2024

themoriarti commented on Apr 12, 2024

kvaps commented on Apr 13, 2024

themoriarti commented on Apr 15, 2024

themoriarti commented on Apr 15, 2024

remipcomaite commented on Apr 23, 2024

themoriarti commented on Apr 30, 2024

remipcomaite commented on Apr 30, 2024

themoriarti commented on Apr 30, 2024

remipcomaite commented on Apr 30, 2024

34 remaining items

dosubot commented on Jan 23, 2026

themoriarti commented on Jan 26, 2026

dosubot commented on Apr 28, 2026

themoriarti commented on Apr 28, 2026

github-actions commented on Jun 28, 2026

github-actions commented on Jul 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions