Skip to content

Cilium 1.20 with kube-ovn chaining classifies node-local host traffic to a pod as world #4781

Open

Description

On Cilium v1.20.2 with kube-ovn chaining, traffic from a node's host network namespace to a pod on the same node is classified as world instead of host. On v1.19.5 the same path arrived as host. Traffic from the host to a pod on another node is still classified correctly. The datapath runs generic-veth chaining with kube-ovn as the chaining target.

Any ingress rule keyed on world or host misfires for this traffic. The kube-ovn webhook and ingress-nginx admission policies had a fromEntities: [world] deny on the webhook port, so the API server timed out calling the webhook replica on its own node (failed calling webhook ... context deadline exceeded). #3343 replaces that deny with the two /1 CIDR halves of each address family. This only works around it for those policies. An allow rule with fromEntities: [host] on a default-deny endpoint would drop the same traffic.

The drop happens on delivery to the pod. cilium-dbg monitor --type drop shows the SYN from the node IP dropped in tail_ipv4_to_endpoint (bpf_lxc.c:2477) with identity world. The host mark is still there at to-netdev on ovn0 and is gone by the time the packet reaches the pod's veth. Without the mark, the to-container path falls back to an ipcache lookup. That lookup ignores a HOST_ID result for a reserved source (bpf_lxc.c:2429-2444 at v1.20.2), so the identity stays world. Setting enable-identity-mark=true does not change it, and bpf-lb-sock-hostns-only=false does not either. I haven't found the 1.20 change that caused this.

The real fix is in the datapath. Next step is a bisect between v1.19.5 and v1.20.2 on a kube-ovn chained cluster. With the commit found, either carry a patch under packages/system/cilium/images or report it upstream.

Activity

  1. lexfrei commented on Oct 6, 2026

    @lexfrei
    ContributorAuthor

    Found the cause. It is upstream commit 990f79be17 (cilium/cilium#43955, "bpf: Generalize SNAT conflict detection"), first released in v1.20.0.

    In v1.19.5 nodeport_has_nat_conflict_ipv4() in bpf/lib/nodeport_egress.h only fired on the direct-routing device and only for the direct-routing source address. Since that commit it fires for any packet carrying MARK_MAGIC_HOST on every interface bpf_host is attached to. The conflict path then calls ctx_snat_done_set(), which replaces the host mark with MARK_MAGIC_SNAT_DONE. With kube-ovn chaining, host traffic to a pod on the same node leaves through ovn0, where bpf_host runs, so the mark is gone by the time the packet reaches the pod and inherit_identity_from_host() falls back to world. On our clusters the direct-routing device is the physical NIC, which is why 1.19.5 never took this path for ovn0.

    pwru on one SYN from the node IP to a local pod shows where it happens:

    ip_output            mark=0xc00 (MARK_MAGIC_HOST)
    __dev_queue_xmit     ovn0 0xc00
    dev_hard_start_xmit  ovn0 0x300 (MARK_MAGIC_SNAT_DONE)
    ovs_vport_send       0x300 -> pod veth, dropped as world
    

    This is also why enable-identity-mark made no difference. #3343 now carries a patch to the Cilium image that skips the conflict check when the destination is a local pod. With it, the webhook policies keep their world deny and local calls pass, and kubelet probes pass under a cluster-only policy. I'll keep this issue open until a Cilium release we ship has the fix upstream and the patch is removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions