Kubernetes app: kubeadm join fails when cluster DNS domain is not cluster.local #2372

Activity

1. dosubot

   added

   kind/bugCategorizes issue or PR as related to a bugCategorizes issue or PR as related to a bug

   on Apr 11, 2026
2. dosubot commented on Apr 11, 2026

   

   dosubotbot

   on Apr 11, 2026 – with DosuBot

   More actions

   Hi Rajesh Medampudi (@medampudi)! I'm Dosu and I’m helping the cozystack team.

   This is a confirmed bug. The kubeadm join template in packages/apps/kubernetes/templates/cluster.yaml constructs the apiServerEndpoint using a short service DNS name without the cluster domain suffix [1]:

   apiServerEndpoint: {{ $.Release.Name }}.{{ $.Release.Namespace }}.svc:6443

   This relies on DNS search domains being present in the worker VM's /etc/resolv.conf, which doesn't hold when worker VMs get their DNS config via DHCP from the KubeVirt virt-launcher.

   The same short FQDN pattern also affects Cilium's k8sServiceHost configuration [2]:

   k8sServiceHost: {{ .Release.Name }}.{{ .Release.Namespace }}.svc

   This is actually the same class of bug that was already fixed for monitoring agents in v1.0.0-rc.1 — vmagent and fluent-bit were failing because service addresses used short DNS names without the cluster domain suffix [3]. That fix used the _cluster.cluster-domain value to construct full FQDNs [4]:

   {{- $clusterDomain := (index .Values._cluster "cluster-domain") | default "cluster.local" }}

   The fix here would follow the same pattern — append the cluster domain to both the kubeadm join endpoint and the Cilium k8sServiceHost:

   {{- $clusterDomain := (index .Values._cluster "cluster-domain") | default "cluster.local" }}
   
   # In KubeadmConfigTemplate:
   apiServerEndpoint: {{ $.Release.Name }}.{{ $.Release.Namespace }}.svc.{{ $clusterDomain }}:6443
   
   # In Cilium config:
   k8sServiceHost: {{ .Release.Name }}.{{ .Release.Namespace }}.svc.{{ $clusterDomain }}

   Worth noting: there's also a separate issue tracking DNS resolution problems with .local domains specifically, since RFC 6762 reserves .local for mDNS and systemd-resolved routes those queries to mDNS instead of CoreDNS. That could compound this issue for VMs using systemd-resolved.

   To reply, just mention @dosu.

   ---

   ^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}
      
3. medampudi commented on Apr 11, 2026

   

   Rajesh Medampudi (medampudi)

   on Apr 11, 2026

   Author

   More actions

   Root Cause Confirmed (2026-04-11)

   After a full cluster reinstall with clusterDomain: cluster.local, the managed K8s worker nodes still fail to join.

   The actual issue

   The kubeadm join config generated by Kamaji/CAPI uses a partial FQDN:

   apiServerEndpoint: kubernetes-botz-infra.tenant-simbotix.svc:6443

   This requires DNS search domain expansion to resolve. However, KubeVirt worker VMs don't have the cluster's search domains in their /etc/resolv.conf — they get DNS via DHCP which only provides the nameserver (10.96.0.10) but not the search domains.

   Proof

   From a pod in the same namespace (which HAS the search domains):

   $ nslookup kubernetes-botz-infra.tenant-simbotix.svc
   ** server can't find kubernetes-botz-infra.tenant-simbotix.svc: NXDOMAIN
   
   $ nslookup kubernetes-botz-infra.tenant-simbotix.svc.cluster.local
   Name: kubernetes-botz-infra.tenant-simbotix.svc.cluster.local
   Address: 10.96.80.135
   

   Even a pod with search domains gets NXDOMAIN on the partial name! The full FQDN works fine.

   Fix needed

   Kamaji should generate the full FQDN in the kubeadm join config:

   apiServerEndpoint: kubernetes-botz-infra.tenant-simbotix.svc.cluster.local:6443

   This is independent of the cluster domain setting (cozy.local vs cluster.local) — the issue is that the .svc suffix without the cluster domain never resolves.

   Environment

   • Cozystack v1.1.4
   • Talos v1.12.6 / K8s v1.34.3
   • clusterDomain: cluster.local
   • Kamaji control plane in tenant namespace
4. Rajesh Medampudi (medampudi)

   added a commit that references this issue on Apr 11, 2026

   fix(kubernetes): use full FQDN for kubeadm join apiServerEndpoint

   ...

   This issue will close once commit 179c51c is merged into the 'main' branch.

   179c51c
5. Aleksei Sviridkin (lexfrei)

   added

   area/kubernetesIssues or PRs related to the tenant Kubernetes appIssues or PRs related to the tenant Kubernetes app

   priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releaseMust be staffed and worked on either currently, or very soon, ideally in time for the next release

   triage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on

   on Apr 27, 2026
6. medampudi commented on May 3, 2026

   

   Rajesh Medampudi (medampudi)

   on May 3, 2026

   Author

   More actions

   Andrei Kvapil (@kvaps) any thoughts on this one? Root cause confirmed in my comment above (Kamaji-generated kubeadm join config uses partial .svc FQDN that requires search-domain expansion not present in KubeVirt VM /etc/resolv.conf). Happy to PR the fix in Kamaji's join-config template if you agree on the direction.
7. dosubot commented on Aug 2, 2026

   

   dosubotbot

   on Aug 2, 2026 – with DosuBot

   More actions

   Hi, Rajesh Medampudi (@medampudi). I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

   Issue Summary

   • You confirmed the root cause is that Kamaji/CAPI is generating a kubeadm join endpoint with a partial service name (...svc:6443) instead of a full FQDN.
   • Worker VMs that only receive DNS nameservers via DHCP cannot resolve the short name because they do not have the cluster search domains in /etc/resolv.conf.
   • You verified the short name returns NXDOMAIN, while the full ...svc.cluster.local name resolves correctly from within the cluster.
   • The requested fix is for Kamaji to emit the full service FQDN in the join config, regardless of whether the cluster domain is cluster.local or custom.

   Next Steps

   • Please confirm whether this issue is still relevant on the latest version of the cozystack repository.
   • If it is still relevant, you can keep the discussion open by commenting here; otherwise, this issue will be automatically closed in 7 days.

   Thanks for your understanding and contribution.
8. dosubot

   added

   stale

   on Aug 2, 2026
9. github-actions

   added

   lifecycle/staleDenotes an issue or PR has remained open with no activity and has become staleDenotes an issue or PR has remained open with no activity and has become stale

   and removed

   stale

   on Aug 3, 2026
10. Aleksei Sviridkin (lexfrei)

    added a commit that references this issue on Oct 6, 2026

    fix(vm-instance): supply cloud-init network-data to netplan guests (#…

    ...

    f532542