Skip to content

VMs using systemd-resolved cannot resolve *.cozy.local: DHCP does not provide DNS search domains #2085

Open
Open
VMs using systemd-resolved cannot resolve *.cozy.local: DHCP does not provide DNS search domains#2085
Labels
area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

Description

@mattia-eleuteri
mattia-eleuteri
opened on Feb 23, 2026

Problem

Virtual machines deployed via the vm-instance chart that use systemd-resolved (Debian 12+, Ubuntu 22.04+, Fedora 33+, etc.) cannot resolve Kubernetes service FQDNs ending in .cozy.local.

$ resolvectl query my-service.my-namespace.svc.cozy.local
my-service.my-namespace.svc.cozy.local: resolve call failed: DNSSEC validation failed: refused

All *.cozy.local lookups fail with REFUSED, making it impossible for VMs to reach cluster services by DNS name.

Root cause

The .local TLD is reserved for multicast DNS (mDNS) per RFC 6762. systemd-resolved enforces this: any query for *.local is routed to mDNS (which is typically disabled in server VMs), not to the configured unicast DNS server (CoreDNS).

KubeVirt's virt-launcher DHCP provides the CoreDNS server IP to the VM, but does not provide DNS search domains. Without search domains or a ~local routing domain hint, systemd-resolved has no reason to send .local queries to CoreDNS and falls back to mDNS → REFUSED.

Expected behavior

VMs should be able to resolve:

  • <service>.<namespace>.svc.cozy.local (FQDN)
  • <service> (short name, via search domain)

Evidence inside the VM

$ resolvectl status
Link 2 (enp1s0)
    Current DNS Server: 10.96.0.10          # ← CoreDNS IP (correct)
    DNS Servers: 10.96.0.10
    DNS Domain: ~.                           # ← no search domains, no ~local routing

The missing piece is DNS Domain: <namespace>.svc.cozy.local svc.cozy.local cozy.local ~local.

Affected code

packages/apps/vm-instance

templates/vm.yaml — The cloudInitNoCloud volume only has secretRef (userdata). There is no networkDataSecretRef:

      - name: cloudinitdisk
        cloudInitNoCloud:
          secretRef:
            name: {{ include "virtual-machine.fullname" . }}-cloud-init
          # networkDataSecretRef is missing

templates/secret.yaml — Only generates userdata (cloud-init #cloud-config). No network-data secret exists.

Reproduction

  1. Deploy a VM using the vm-instance chart with a Debian 12 or Ubuntu 24.04 golden image
  2. SSH into the VM
  3. Try to resolve any cluster service: 
    resolvectl query kubernetes.default.svc.cozy.local
# → REFUSED
  4. Check resolvectl status — no search domains are configured on the DHCP link

Proposed fix

Add a network-data secret to the vm-instance chart that injects DNS search domains via cloud-init network config v1.

1. New template: templates/secret-network-config.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ include "virtual-machine.fullname" . }}-network-config
stringData:
  networkdata: |
    network:
      version: 1
      config:
      - type: nameserver
        address: []
        search:
        - {{ .Release.Namespace }}.svc.cozy.local
        - svc.cozy.local
        - cozy.local
      - type: physical
        name: enp1s0
        subnets:
        - type: dhcp

Key points:

  • address: [] — DNS server IP is already provided by DHCP; we only need to add search domains
  • The nameserver block is global and coexists with per-interface DHCP config
  • The search domains tell systemd-resolved to route cozy.local (and subdomains) to the unicast DNS server

2. Update templates/vm.yaml

Add networkDataSecretRef alongside the existing secretRef:

      - name: cloudinitdisk
        cloudInitNoCloud:
          secretRef:
            name: {{ include "virtual-machine.fullname" . }}-cloud-init
          networkDataSecretRef:
            name: {{ include "virtual-machine.fullname" . }}-network-config

Compatibility

  • secretRef (userdata) and networkDataSecretRef (network config) are independent fields in KubeVirt's cloudInitNoCloud — they do not conflict
  • Users setting custom cloudInit (userdata) are unaffected since network-data is a separate secret
  • The nameserver block in cloud-init network config is global and does not interfere with DHCP

Open question

The cluster domain (cozy.local) is currently hardcoded. Ideally it should come from a platform-level value (e.g., networking.clusterDomain or a global Helm value). The exact mechanism to pass this to app-level charts is left to maintainers.

Workaround

Until fixed, users can manually configure DNS inside the VM using a systemd-networkd drop-in:

sudo mkdir -p /etc/systemd/network/10-netplan-enp1s0.network.d
sudo tee /etc/systemd/network/10-netplan-enp1s0.network.d/dns-fix.conf <<EOF
[Network]
Domains=<namespace>.svc.cozy.local svc.cozy.local cozy.local ~local ~.
EOF
sudo systemctl restart systemd-networkd systemd-resolved

Replace <namespace> with the actual tenant namespace. The ~local routing domain forces systemd-resolved to send .local queries to the unicast DNS server instead of mDNS.

Note: netplan does not support routing domains (~ prefix), so systemd-networkd drop-ins must be used directly.

Activity

  1. dosubot
    added
    kind/bugCategorizes issue or PR as related to a bug
    on Feb 23, 2026
  2. Aleksei Sviridkin (lexfrei)
    added
    area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    triage/acceptedIndicates an issue is ready to be actively worked on
    on Apr 27, 2026

  3. dosubot commented on Jul 27, 2026

    @dosubot
    dosubotbot
    on Jul 27, 2026 – with DosuBot

    Hi, mattia-eleuteri. I'm Dosu, and I'm helping the cozystack team manage their backlog and am marking this issue as stale.

    Issue Summary

    • VMs created with the vm-instance chart and systemd-resolved are unable to resolve Kubernetes service names under *.cozy.local.
    • The reported cause is that DHCP provides only the DNS server IP, but not the search domains or the ~local routing hint needed by systemd-resolved.
    • Without that routing hint, queries are routed to mDNS and fail with REFUSED.
    • The suggested fix is to provide network data via networkDataSecretRef so the VM receives the required DNS domains.

    Next Steps

    • Please confirm whether this is still relevant on the latest version of the cozystack repository; if it is, you can keep the discussion open by commenting here.
    • Otherwise, this issue will be automatically closed in 7 days.

    Thanks for your understanding and contribution.

  4. dosubot
    added
    stale
    on Jul 27, 2026

  5. mattia-eleuteri commented on Jul 27, 2026

    @mattia-eleuteri
    mattia-eleuteri
    on Jul 27, 2026 via email
    CollaboratorAuthor
  6. dosubot
    removed
    stale
    on Jul 27, 2026

  7. dosubot commented on Jul 27, 2026

    @dosubot
    dosubotbot
    on Jul 27, 2026 – with DosuBot

    Timofei Larkin (@lllamnyp) This issue is still relevant per the reporter, who confirmed the vm-instance/systemd-resolved DNS resolution problem under *.cozy.local remains unresolved. Could you please take a look and help with this?

  8. lexfrei commented on Sep 7, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Sep 7, 2026
    Contributor

    virt-launcher does send the search domains, so the root cause here needs a correction. At v1.9.0 StartDHCP reads the launcher pod's /etc/resolv.conf and passes the parsed search list down (kubevirt/kubevirt: pkg/network/driver/common.go), and the DHCP server puts it on the wire as option 119 alongside option 15 (kubevirt/kubevirt: pkg/network/dhcp/server/server.go). The launcher pod's resolv.conf carries <namespace>.svc.cozy.local svc.cozy.local cozy.local, so those domains are already being advertised to the guest.

    The guest drops them. UseDomains= defaults to no in systemd-networkd, per systemd.network(5): "When none of them are specified, defaults to no." A netplan-managed link ignores DHCP-supplied domains until something turns that on, which fits the resolvectl status output above: DNS Domain: ~. is a link that was offered domains and kept none of them.

    The .local part needs a correction too. resolved routes *.local to mDNS only after failing to match a search or routing domain. In dns_scope_good_domain() the search-domain loop runs first and returns a positive scope on a suffix match, and the .local exclusion sits below it, reached only when nothing matched. The comment on that branch, in systemd/systemd: src/resolve/resolved-dns-scope.c on current main:

    If networks use .local in their private setups, they are supposed to also add .local to their search domains, which we already checked above. Otherwise, we consider .local specific to mDNS and won't send such queries ordinary DNS servers.

    A plain cozy.local search domain is therefore enough to route those lookups to CoreDNS. ~local works, but it is not needed, and netplan's missing routing-domain support stops being a blocker for the fix proposed here.

    So the chain is: KubeVirt advertises the list, the guest discards it on UseDomains=no, resolved is left with nothing matching cozy.local, the .local exclusion fires, REFUSED. Anything that lands cozy.local on the link fixes it, which includes the network-config secret proposed above and equally dhcp4-overrides: use-domains: true.

    Separate caveat on the proposed shape. networkDataSecretRef replaces cloud-init's fallback network configuration instead of adding to it, so the secret has to configure the interface as well, and the snippet names enp1s0 literally. A guest whose first NIC comes up under another name gets a configuration for a device it does not have and loses DHCP altogether, which is a worse failure than the DNS problem and would land on every VM the chart renders. Worth settling how the interface is identified before that shape ships.

  9. mattia-eleuteri commented on Sep 17, 2026

    @mattia-eleuteri
    mattia-eleuteri
    on Sep 17, 2026
    CollaboratorAuthor

    Aleksei Sviridkin (@lexfrei) your correction holds on both counts, and it changed the fix. I went looking for why the search domains were being dropped rather than never sent, and ended up somewhere I did not expect.

    UseDomains=no is the whole of it, so dhcp4-overrides: use-domains: true is sufficient and the nameserver block I proposed is not needed at all. That also disposes of the open question at the bottom of the original report: the domains on the wire come from the launcher pod's /etc/resolv.conf, so they already carry whatever cluster domain the cluster actually runs, and nothing has to be hardcoded or plumbed down from a platform value. #2372 stops touching this.

    Your objection to the shape was right too, and for a wider reason than the one interface name. networkDataSecretRef replaces cloud-init's fallback configuration, and the fallback covers exactly one interface — which means the same missing piece produces a second, separate failure I had been treating as unrelated. A VM attached to a VPC subnet never configures that interface, on Debian and Ubuntu only. virt-launcher does serve it: newDHCPConfigurator in pkg/network/setup/launcher/configurator.go gates only on iface.Bridge != nil, so a DHCP server runs on every bridge-bound interface, secondary ones included, and BridgeConfigGenerator.Generate hands out the address the CNI put on the pod-side interface together with that link's MTU. On a cluster here the secondary pod interface carries its Kube-OVN address and enableDHCP is unset on every VPC subnet, so the offer is virt-launcher's. Nothing in the guest asks for it. AlmaLinux and Windows come up on the same subnets, which is the tell — with no server at all, none of the three would.

    The part worth reporting: this used to work. packages/apps/virtual-machine shipped templates/secret-network-config.yaml and a networkDataSecretRef in c63fb20 ([apps] virtual-machine: add vpc support, 2025-10-21), gated on .Values.systemDisk.image. 08a5f98 (feat(platform): remove virtual-machine application, 2026-02-12) deleted the application and took the template with it, and vm-instance never had one — git log --all -S networkDataSecretRef -- packages/apps/vm-instance/ comes back empty. The gate could not be ported as written, since systemDisk.image disappeared in the same split, which is probably why nobody did. #2166 came out of the same migration.

    #4306 restores it, as a draft. It matches en* and eth* instead of naming interfaces by index, which is what the old template did and what you flagged here; it describes every interface, since it replaces the fallback; and use-domains: true closes this issue without a hardcoded domain. The gate is an instanceProfile of ubuntu or debian, deliberately narrow because cloud-init only passes a v2 configuration through verbatim for the netplan renderer — handing a match stanza to a guest rendered by sysconfig would leave it with no DHCP at all. Whether that proxy is the right one is the open question on the PR; a MAC-matched v1 configuration would cover every guest but needs the chart to pin MACs, which is a larger change.

  10. Aleksei Sviridkin (lexfrei)
    added a commit that references this issue on Oct 6, 2026

    fix(vm-instance): supply cloud-init network-data to netplan guests (#…

    f532542
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/virtualizationIssues or PRs related to virtualization (kubevirt, cdi, vmi, vm-import)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions