Tenant cluster service/pod CIDR collides with management cluster → DNS loop → all image pulls fail #4161

Summary

The kubernetes chart hardcodes the tenant cluster's pod and service CIDRs to the same values as the management cluster (10.243.0.0/16 pod, 10.95.0.0/16 service). When the management cluster uses those CIDRs, the tenant's CoreDNS service IP (10.95.0.10) shadows the management cluster's CoreDNS ClusterIP that the worker VMs are configured to use as their host resolver. The result is a DNS self-loop: every external name resolution from the tenant returns SERVFAIL, so every image that isn't preloaded on the node fails to pull, and the whole tenant stack (cert-manager, flux, metrics-server, velero, victoria-metrics, konnectivity, the second coredns replica, kubevirt-csi) sits in ImagePullBackOff.

This is not operator error: the tenant Kubernetes CR (apps.cozystack.io/v1alpha1) exposes no field to set the pod/service CIDR (its spec is storageClass, version, host, addons, controlPlane, images, talos, oidc). The CIDRs are hardcoded in the chart, so there is no supported way to avoid the collision.

Root cause

The tenant CIDRs are hardcoded, not derived:

• packages/apps/kubernetes/templates/cluster.yaml:56-57

  {{- $podCIDR := "10.243.0.0/16" }}
  {{- $serviceCIDR := "10.95.0.0/16" }}

• packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml:388-389 (same two literals)
• packages/apps/kubernetes/templates/helmreleases/coredns.yaml:4

  clusterIP: "10.95.0.10"

The management cluster runs --service-cluster-ip-range=10.95.0.0/16 and --cluster-cidr=10.243.0.0/16, so the tenant's hardcoded CIDRs are byte-for-byte identical to the management cluster's.

Mechanism (how the collision becomes a DNS loop)

1. packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml sets each worker VM's host-level nameserver to ${COREDNS_IP} — the management cluster's kube-dns ClusterIP, i.e. 10.95.0.10 (the inline comment states the intent: "Management CoreDNS as the worker's host resolver").

2. Because the tenant's service CIDR is also 10.95.0.0/16, the tenant's CoreDNS is also pinned to 10.95.0.10. Inside the worker VM, the tenant's Cilium intercepts 10.95.0.10 and DNATs it to the tenant's CoreDNS, not the management's.

3. The tenant CoreDNS Corefile is forward . /etc/resolv.conf, and the CoreDNS pod's dnsPolicy: Default means its /etc/resolv.conf is the node's — which is nameserver 10.95.0.10 again. So CoreDNS forwards external queries to itself → the loop plugin drops them → SERVFAIL.

4. In-cluster names (cluster.local) still resolve (served from etcd by the kubernetes plugin), which is why the cluster looks half-alive, but every external name (quay.io, ghcr.io, registry.k8s.io, docker.io) dies at the DNS stage.

Evidence

• Worker node /etc/resolv.conf (from the generated Talos machine config in kubernetes-clustor-worker-*-bootstrap-data):

  machine:
    network:
      nameservers: [10.95.0.10]
      searchDomains: [svc.cozy.local, cozy.local, svc.cluster.local, cluster.local]

• Tenant CoreDNS ConfigMap (kube-system/coredns): forward . /etc/resolv.conf.
• Pull failure (from pod events):

  failed to do request: Head "https://quay.io/v2/...": dial tcp: lookup quay.io on 10.95.0.10:53: server misbehaving
  

• Probe pods confirmed the split: getent hosts kubernetes.default → exit 0 (in-cluster OK); getent hosts quay.io → exit 2 / 40s timeout (external broken). After pointing the tenant CoreDNS forward at the management CoreDNS pod IPs, getent hosts quay.io → exit 0 and the cluster recovered.

Why this is a real bug (not a misconfiguration)

The management cluster's CIDRs live in the operator's own machine configuration (a standalone Talos cluster — see the related note in #3740), outside this repository. The tenant chart must not assume the management cluster uses any particular CIDR. Hardcoding 10.243.0.0/16 / 10.95.0.0/16 breaks the moment the management cluster uses anything else — and, more fundamentally, it means two tenants would collide with each other (both hardcode the same service CIDR).

Proposed fix

1. Add a per-tenant networking field to the Kubernetes CRD (apps.cozystack.io/v1alpha1) — e.g. spec.networking.serviceCIDR / spec.networking.podCIDR — or have the operator auto-allocate a unique range per tenant.

2. Replace the hardcoded literals in cluster.yaml, talos-reconcile-job.yaml, and coredns.yaml with the per-tenant values, deriving the CoreDNS ClusterIP from the service CIDR (e.g. <serviceCIDR base>.10) instead of the literal 10.95.0.10.

3. Add validation (and documentation) that the tenant's service CIDR and pod CIDR must not overlap the management cluster's service CIDR, pod CIDR, or joinCIDR — mirroring the documentation gap called out in networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740.

Workaround (for operators hitting this today)

Point the tenant CoreDNS at a reachable upstream that isn't shadowed by the collision. The management CoreDNS is reachable from the tenant VMs at its pod IPs (not its ClusterIP, which collides). Via the Kubernetes CR:

spec:
  addons:
    coredns:
      valuesOverride:
        coredns:
          servers:
            - zones:
                - zone: .
                  use_tcp: true
              port: 53
              plugins:
                - name: errors
                - name: health
                  configBlock: |-
                    lameduck 10s
                - name: ready
                - name: kubernetes
                  parameters: cluster.local in-addr.arpa ip6.arpa
                  configBlock: |-
                    pods insecure
                    fallthrough in-addr.arpa ip6.arpa
                    ttl 30
                - name: prometheus
                  parameters: 0.0.0.0:9153
                - name: forward
                  parameters: . <mgmt-coredns-pod-ip-1> <mgmt-coredns-pod-ip-2>
                - name: cache
                  parameters: 30
                - name: loop
                - name: reload
                - name: import
                  parameters: /etc/coredns/custom/*.override
                - name: loadbalance

Caveat: the management CoreDNS pod IPs are ephemeral (they change if those pods are rescheduled), so this is a stopgap, not a fix.

Related

• networking: cilium takes its per-node infrastructure IPs from the pod CIDR kube-ovn allocates pods from #3740 — same root theme (hardcoded/overlapping CIDRs, undocumented non-overlap constraint), different overlap (management-cluster kube-ovn vs cilium pod CIDR).