Repository navigation
restrict-system-components blocks LINSTOR satellite cross-node traffic when Cilium is chained with kube-ovn #3337
Description
What happened
LINSTOR controller pod on node A cannot reach satellite on node B (port 3367 SSL). Only the satellite on the same node as the controller is Online. Cross-node pod-to-host TCP connections to port 3367 are silently dropped.
Root cause
restrict-system-components CCNP has:
ingressDeny:
- fromEntities:
- world
toPorts:
- ports:
- port: "3367" # ← blocks LINSTOR satelliteIn CNI chaining mode (generic-veth with kube-ovn), cross-node pod-to-host traffic arrives at the destination node with the pod overlay IP as source. Cilium ipcache does not have identity entries for pods managed by kube-ovn (they are not Cilium endpoints), so the traffic is classified as identity world→unknown and dropped.
Note: ingressDeny has priority over ingress in Cilium policy evaluation. Moving port 3367 to another CCNP (e.g. restrict-drbd-reactor) with ingress.fromEntities: [cluster] would NOT help — the ingressDeny.fromEntities: [world] rule is evaluated first, and since the traffic IS classified as world, it would still be denied.
Confirmed via cilium monitor:
xx drop (Policy denied by denylist) identity world->unknown:
10.244.0.204:37124 -> 10.88.148.82:3367 tcp SYN
Why this does not affect all clusters
In standard kube-ovn distributed-gateway mode with natOutgoing: true, pod→remote-node traffic gets SNAT to the node IP. Cilium then classifies it as host (allowed by ingress.fromEntities). On affected clusters, SNAT is not applied.
Short-term fix
Remove port 3367 (and 3366) from restrict-system-components ingressDeny.
Long-term fix
Fix Cilium identity classification in generic-veth chaining mode with kube-ovn. Cross-node pod traffic should be classified as cluster, not world.
Steps to reproduce
- Deploy cozystack with
networking: kubeovn-cilium, Cilium in chaining mode - Enable LINSTOR on a multi-node cluster
kubectl exec -n cozy-linstor deploy/linstor-controller -- linstor node list— satellites on remote nodes are OFFLINE- TCP connection from controller pod to remote node IP:3367 fails (
world→ denied) - HostNetwork pod on same node can reach remote satellite:3367 (
host→ allowed) cilium monitor --type dropon remote node confirms:Policy denied by denylist, identity world->unknown
Environment
- Cozystack version: current main
- CNI: Cilium + kube-ovn (generic-veth chaining)
- Affected component:
cilium-networkpolicy→restrict-system-componentsCCNP
Activity
dosubot commentedon Jul 17, 2026
Hi Nikita Aboltin (@2mind)! I'm Dosu and I'm helping the Cozystack team.
Your analysis looks accurate. Port 3367 is indeed listed in the
restrict-system-componentsCCNP'singressDenyblock [1], which deniesworldentity access. The key problem is that in Cilium chaining mode with kube-ovn, cross-node pod traffic gets classified asworld→unknownbecause Cilium ipcache doesn't have identity entries for kube-ovn-managed pods.The
restrict-drbd-reactorpolicy in the LINSTOR chart [2] already follows the pattern you described — it allowsfromEntities: [world, host, cluster]while only denyingworldon port 9942 (DRBD reactor). Moving ports 3366/3367 there (or at least adding afromEntities: [cluster]exception inrestrict-system-components) would let intra-cluster LINSTOR satellite traffic through while still blocking external access.One thing to note:
restrict-drbd-reactoris templated in the LINSTOR Helm chart atpackages/system/linstor/templates/networkpolicy.yaml[3], whilerestrict-system-componentslives inpackages/system/cilium-networkpolicy/templates/networkpolicy.yaml[4]. If the fix moves ports torestrict-drbd-reactor, the cilium-networkpolicy chart's test suite will also need updating to remove the port 3367 assertion.No prior PRs or issues were found tracking this, so this appears to be a new report.
To reply, just mention @dosu.
Docs are dead. Just use Dosu.
2mind commentedon Jul 17, 2026
# (@dosu) This comment addresses the original version of the issue. The body has since been updated with a correction:
ingressDenyhas priority overingressin Cilium, so moving port 3367 fromrestrict-system-componentstorestrict-drbd-reactordoes NOT help — that policy also hasingressDeny.fromEntities: [world], and since cross-node pod traffic IS classified asworld(due to missing identity in chaining mode), it would still be denied regardless ofingress.fromEntities: [cluster].Short-term fix: remove port 3367 from the
ingressDenylist entirely.Long-term fix: fix Cilium identity classification in
generic-vethchaining mode so pod traffic iscluster, notworld.dosubot commentedon Jul 17, 2026
Thanks for the correction, Nikita Aboltin (@2mind). You're right — since Cilium's
ingressDenytakes priority overingressallow rules, moving port 3367 intorestrict-drbd-reactorwouldn't help because that policy also deniesworldon specific ports, and cross-node pod traffic is misclassified asworldin chaining mode.Looking at the current
restrict-system-componentspolicy [1], port 3367 sits alongside etcd, kubelet, and metallb ports in theingressDenyblock. The policy also has a broadingressallow from[world, host, cluster][2], so removing 3367 from the deny list would still leave the general ingress allow in place — LINSTOR satellite traffic would flow while the other system ports remain protected.Your two-track approach makes sense:
ingressDenylist inrestrict-system-components. This unblocks LINSTOR satellite connectivity without affecting the deny rules for the other 13 ports.generic-vethchaining mode so kube-ovn-managed pod traffic gets a properclusteridentity instead ofworld.Are you planning to submit a PR for the short-term fix?
To reply, just mention @dosu.
Docs are dead. Just use Dosu.
lexfrei commentedon Aug 14, 2026
Confirmed against the tree, and the analysis here holds.
packages/system/cilium-networkpolicy/templates/networkpolicy.yamldenies 3367 throughingressDeny.fromEntities: [world], and the chart ships no values file at all, so the port list is hardcoded and cannot be overridden through Package values. That means there is no upgrade-surviving workaround today: an inline edit is reverted on the next reconcile, and a second CCNP cannot rescue it because deny wins over allow.Worth naming why it does not reproduce everywhere, since that is the confusing part: the trigger is the identity, not the port. Under chaining with kube-ovn the pods are not Cilium endpoints, so cross-node pod-to-host traffic arrives classified as
world. Where the controller happens to talk to a satellite on its own node, or where a cluster is not chained, the deny is never reached and everything looks fine.One candidate for a fix, and it wants testing on a real cluster before anyone commits to it: replace
fromEntities: [world]withfromCIDRSet: 0.0.0.0/0plusexceptfor the platform's own pod, service and join CIDRs. Cilium allowsexceptonfromCIDRSet, deny still takes precedence over allow, and CIDR selectors do not match node identities unlesspolicy-cidr-match-mode=nodesis set, so the host leg should stay as it is. The platform already knows those three CIDRs and hands them to kube-ovn, so they would have to be threaded into this chart as well. What needs verifying is that overlay traffic then lands outside the deny instead of finding its way back intoworldby another path.The cheaper interim is to parameterise the port list so an operator can drop 3367 without forking the chart. That unblocks LINSTOR but leaves the same classification problem standing for every other port in the list.