Repository navigation
ingress-nginx admission webhook unreachable on main since #3798: every Ingress creation fails admission #3983
Description
What is broken
Creating any Ingress on main fails admission:
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io":
Post "https://root-ingress-controller-admission.tenant-root.svc:443/networking/v1/ingresses?timeout=10s":
dial tcp 10.96.37.241:443: connect: no route to host
The API server cannot reach the ingress-nginx admission webhook, so every Ingress create/update is rejected. The full E2E suite is red on main because of it.
When it started
Occurrences of validate.nginx.ingress.kubernetes.io in the nightly E2E (full suite) log:
| nightly | started | occurrences |
|---|---|---|
| 2026-08-26 | 02:33 | 0 |
| 2026-08-27 | 03:49 | 0 |
| 2026-08-28 | 04:47 | many, job failed |
#3798 merged at 2026-08-27 16:59 — between the last clean night and the first broken one.
What in #3798 could do this
It added packages/system/ingress-nginx/templates/admission-webhook-networkpolicy.yaml: a CiliumNetworkPolicy that denies traffic to the webhook port. The failure is that exact port being unreachable.
The policy intends to allow the API server: it argues host-network callers are not Cilium endpoints, so fromEndpoints never matches them. True for that clause — but the policy also denies fromEntities: [world], which is a separate rule and has nothing to do with endpoints. Cilium here runs chained over kube-ovn (cni-chaining-mode=generic-veth, cni-chaining-target=kube-ovn), so it is worth checking what identity API-server traffic actually gets. If it lands in world, the deny hits the API server.
Not claiming that is the cause — it is the first thing to check. Other candidates: the deny uses dig controller.admissionWebhooks.port 8443, which may no longer match what the Service targets after the chart bump.
Why CI on the PR was green
Two separate reasons, and the first one is the important one.
The test that catches this never ran on the PR. The case is ingress-hostname-policy, one of 15 tests inside the gateway suite (hack/e2e-chainsaw/gateway/chainsaw-test.yaml). Chainsaw reports the TEST name in its output, but test selection works on the suite DIRECTORY name, and nothing maps ingress-nginx to gateway. packages/system/ingress-nginx belongs to three PackageSources, and src_to_suites() turns them into:
| source | suite name | directory exists? |
|---|---|---|
ingress-application |
ingress |
no, dropped |
ingress-nginx |
ingress-nginx |
no, dropped |
kubernetes-application |
kubernetes-latest / -previous / -oidc-* |
yes, selected |
That is exactly what the PR's run selected: Selected apps: kubernetes-latest kubernetes-oidc-customconfig kubernetes-oidc-system kubernetes-previous. The gateway suite maps only from gateway-application, so it runs when the gateway package changes — not when ingress-nginx does. It ran the next night in the full suite, and failed there.
So E2E (in-tree): success on #3798 is honest: the only test covering this was not part of the run.
One line in hack/select-e2e.sh closes it:
ingress-application|ingress-nginx) echo gateway ;;This is the same class the comment in that file records for #3665 — a suite unreachable from its own package — arrived at from the other side: the package cannot reach the suite that covers it. The general shape of that hole, which is not specific to ingress-nginx, is tracked in #3985: escalation to the full suite fires only when a package maps to NO suite, so a package that maps to some suites silently loses the others that cover it.
The unit tests could not have caught it either. The eight tests added with the policy all assert the rendered YAML: which selectors it carries, which namespaces it spans, that it can be disabled. None exercises whether the API server can still reach the webhook, which is the property that broke.
How to confirm in five minutes
Set admissionWebhookPolicy.enabled: false and create an Ingress.
- Works → the policy is the cause.
- Still fails → the policy is not the cause, look at the chart bump.
Activity
IvanHunters commentedon Aug 28, 2026
Ran this down on a cozystack v1.6.2 management cluster to pin the mechanism. The short version: the deny is real and enforced, but on my setup the API server still reaches the webhook, which narrows the cause to which identity Cilium gives the API-server → webhook traffic rather than to the chart bump or a version.
Setup — same CNI stack as the failing nightly: Cilium 1.19.5, kube-ovn
v1.15.10(same digestsha256:741299cb),cni-chaining-mode=generic-veth,cni-chaining-target=kube-ovn. Root ingress intenant-root, webhookroot-ingress-controller-admissiontargeting 8443.What I applied — the PR's policy rendered for the root namespace (
enableDefaultDeny.ingress: false, the splitingressDenyfromEndpoints, and the separatefromEntities: [world]deny on 8443), selecting the controller byapp.kubernetes.io/name=ingress-nginx+component=controller.Result — admission keeps working here. Ingress create/update succeeds under the policy, and
cilium-dbg monitor --type dropon the controller's node shows zero drops on 8443 during the create.But the policy is not a no-op — it is enforced on the controller endpoint:
So the deny is in the datapath, yet the API server is not caught by it. On this cluster the API-server → webhook traffic resolves to
host/remote-node(apiserver is host-network, controller pod is onnode1), and both are excluded from selector A by the implicit cluster-label filter and are notworld, so neither deny rule matches. That is why it survives here.Where this leaves the diagnosis. Cilium, kube-ovn, and the chaining config are byte-identical to the failing nightly, so the break is not a version difference. The variable is the identity the datapath assigns to the API-server → webhook-Service path, and that is not deterministic across setups — it depends on which node the serving apiserver instance is on relative to the controller pod, and on how kube-ovn SNATs the ClusterIP hop. Where that path lands in
world(or otherwise inside a deny clause), every Ingress admission fails exactly as reported; where it lands inhost/remote-node, it passes. This matches thefromEntities: [world]suspicion in the issue: the policy's correctness rests on an identity classification that is not guaranteed to hold, so the fix probably has to make the API-server carve-out identity-independent (drop the standaloneworlddeny, or add an explicit allow for the apiserver/host path) rather than rely onworldnot covering it.One thing this does rule out: the port is not the problem on my render — the webhook Service targets 8443 and the deny is on 8443, and it still passes. So
dig controller.admissionWebhooks.port 8443matching the Service is not the failure here; the identity path is.I could not reproduce the actual failure on 1.6.2, so I cannot bisect it further from this cluster — it needs the E2E environment where the path resolves to the denied identity. Happy to run a specific probe there if it helps (e.g.
cilium-dbg monitor --type dropon the controller node plus the identity of the dropped source) — that single line would confirm which identity the apiserver hop actually gets on the failing setup.kvaps commentedon Aug 28, 2026
The CI-gap analysis is the valuable half of this and it is correct:
src_to_suites()mapsingress-nginxto suite names that do not exist as directories, so thegatewaysuite — whereingress-hostname-policylives — is unreachable from the package it covers. That is a real hole, the one-line fix is right, and it is worth landing regardless of what caused this. Same for #3985.On the diagnosis, though, the evidence does not support the two headline claims, and I think the issue is mis-scoped.
"Every Ingress creation fails admission." All 302 occurrences in the 2026-08-28 log fall inside 16 consecutive seconds, 05:26:34 through 05:26:49, at a steady ~20/s — one test in a retry loop, not a standing failure. Nothing before, nothing in the remaining two hours of the run.
"The full E2E suite is red on main because of it." The suite failed on
chainsaw/kubernetes-previousandchainsaw/kubernetes-latest, 2 failed tests. The 2026-08-27 nightly failed on exactly the same two, also 2 failed. That run was red before #3798 existed, for the same reason.And the decisive one: the test you identify as the only coverage passed.
It hit the retry burst at 05:26:34, got
CREATE OK/APPLY DONEat 05:26:49, cleaned up, and the suite moved on tovminstance.What I think happened. #3798 changed the controller image digest, so the HelmRelease moved to a new revision and the controller rolled. During the roll the admission Service had no ready endpoints, and a Service with no backends is exactly what returns
EHOSTUNREACH— a Cilium policy drop gives a timeout instead, and would not clear itself after 16 seconds. The nights before changed no digest, so nothing rolled, so zero occurrences. Supporting facts from the same log:deploy/root-ingress-controller condition metat 05:17, the LB at192.168.123.200reachable, and bothingressandingress-nginx-systemHelmReleases intenant-rootTrue.I have not proven this — the log keeps no pod-event dump for 05:26, so it is a consistent reading rather than a demonstration.
On the five-minute check. Disabling
admissionWebhookPolicyand creating an Ingress will succeed, but it would also succeed under the rollout explanation, since nothing is rolling now. It cannot separate the two.What does separate them is a rerun on the same commit with no digest change. I started one before this issue was filed: run 33158462012,
workflow_dispatchonc8e4d97df, identical to the failing run. Zero occurrences means the window came from the roll; a repeat means the policy is denying the API server, and then yourfromEntities: worldreview comment — which I left unanswered when the PR merged, my fault — is the cause and the rule comes out.Separately, and independent of all this: that comment asked what the
worlddeny buys when the Service is ClusterIP with no override. I agree it buys little, and I am happy to remove it either way.kvaps commentedon Aug 28, 2026
My rollout explanation was wrong, and my own experiment refutes it.
Run 33158462012, same commit
c8e4d97df, no digest change, nothing to roll: 304 occurrences, against 302 in the scheduled run. Same ~16-second shape, 10:04:47 → 10:05:02.The log shows what precedes it:
The window opens immediately after a tenant namespace is deleted. The policy's deny carries
fromEndpointswithnamespace NotIn [kube-system], a selector spanning every namespace, so namespace churn forces Cilium to recompute the selector set and regenerate the controller endpoints' policy maps — and traffic to the port is dropped while that happens. That fits the trigger, the duration, and the self-healing, and it is specific to this policy, which is why the nights before show zero.So you were right that the policy is implicated, and I was wrong about the mechanism.
Two things from my earlier comment do still hold, for scoping rather than defence:
ingress-hostname-policypassed again in this run (36.14s), and the suite failed onchainsaw/kubernetes-previousandchainsaw/kubernetes-latest— the same two as the 2026-08-27 nightly. The bounded window is real and worth fixing regardless: 16 seconds of admission unavailability on every namespace deletion would reject user Ingress writes in production.I will send a PR removing the
worlddeny and reworking the pod deny so it does not carry a cluster-wide selector. If you would rather revert #3798's policy outright while that is worked out, say so and I will do that first.test(cilium): pin the service-LB assumption both webhook denies rest on
test(e2e): stop the webhook cases passing for the wrong reason
fix(networking): prevent webhook admission outages during namespace c…
fix(networking): enforce the pod admission deny from both ends
test(e2e): cover webhook admission after namespace churn
myasnikovdaniil commentedon Oct 1, 2026
This is fixed already via the #3997
fix(networking): harden webhook policy assumptions (#4058)