Repository navigation
[Backport release-1.6] fix(networking): prevent webhook admission outages during namespace churn - #4052
Closed
github-actions[bot] wants to merge 6 commits into
Closed
[Backport release-1.6] fix(networking): prevent webhook admission outages during namespace churn#4052github-actions[bot] wants to merge 6 commits into
github-actions[bot] wants to merge 6 commits into
Conversation
Author
|
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-3997-to-release-1.6
git worktree add --checkout .worktree/backport-3997-to-release-1.6 backport-3997-to-release-1.6
cd .worktree/backport-3997-to-release-1.6
git reset --hard HEAD^
git cherry-pick -x ad961f6de1b3ec3526f0f56212b9e35dbc83078a f224baae461bafc01504f4f3db60214e8cb356ce 6b2b68a685a828056f2e13c9ae5d819949dc45c3 bdc172a64855ec5af009f9a0c2559ed74a22ec02 f73b54fee963c59f62e26954ce7d19338bfb0d54 9f485a18ad51ce216ec835d20f896e7fb3090d27 042ac2715bfdba51c721e80306cfe5ff94e8e9ae 6a289cf6d068f14b48497e9bbb396d96fdb2418d cbd0e78cadb90560ce1d59df886e9f24466a424a 3daa83f4b5a4b0d50852bdbbd0033b0a7d2077be b78405cede870c48201089588b69921be7793bc5 0335421769af296b274e33e0e10455f1f09a6752 686a81a491f6902d0d1aff9533fa243ced00e087
git push --force-with-lease |
1 of 11 tasks
Backport of #3997 to `release-1.6`, kube-ovn-webhook half. The namespaced CiliumNetworkPolicy/kube-ovn-webhook carried a cluster-spanning fromEndpoints peer on the webhook's own endpoint, so every namespace add or delete rewrote that endpoint's policy map -- the coupling #3983 reported against fail-closed admission webhooks. It also depended on nothing but that one end: while the map is being recomputed or has overflowed, the deny is the thing being rebuilt. What ships now is one cluster-scoped CiliumClusterwideNetworkPolicy/kube-ovn-webhook-c2bdbc146112 carrying two specs that fail in different directions. The source-side spec is an egressDeny on every pod endpoint naming the webhook's backend port, so it holds while the webhook endpoint's own policy is degraded, but it relies on Cilium performing the service DNAT ahead of its egress hook. The target-side spec keeps the pod deny and the world deny on the webhook endpoint, evaluated after whatever DNAT already happened, so it holds where Cilium is not the service load balancer. Neither subsumes the other, so the churn writes stop being load-bearing rather than going away -- the template says so rather than claiming the coupling is removed. `toServices` is deliberately not used to name the Service frontend, and the template carries the reason: Cilium resolves it only in allow rules (pkg/policy/k8s/service.go walks spec.Egress, never spec.EgressDeny), so an egressDeny whose only destination is an unresolved toServices reaches the datapath with an empty L3 selector that Cilium reads as a wildcard when the rule carries toPorts -- a cluster-wide egress outage on that port rather than a tighter deny. The object changes kind and name, so anything keyed on the old namespaced CiliumNetworkPolicy/kube-ovn-webhook goes quiet with no error. The template now also renders only while cilium.io/v2/CiliumClusterwideNetworkPolicy is discoverable, so `enabled: true` is no longer a promise that a policy exists; set `enabled: false` to record a deliberate opt-out. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
hack/helm-unit-tests.sh finds suites by probing `make -n test` per package and skips any package without that target, so this package's two suites had never run in CI -- including the port agreement guard, whose whole purpose is to catch a drift that fails open. The target is declared .PHONY because nothing else declares it: hack/package.mk writes its .PHONY as a variable assignment rather than a target, so a path named `test` next to the Makefile would make the target up to date and skip the recipe silently. The suites now pin the two-spec shape rather than the old single-spec one, and add what review asked for: one deny rule per spec whose destination is a selector Cilium can actually resolve (the toServices regression renders as a plausible ingressDeny and would otherwise pass), the peer of the target-side pod deny, the collision-safe cluster-scoped name, and a capability suite for the render-nothing branch. Checked by mutation rather than by revert. Substituting `fromEntities: [all]` for the target-side pod peer, dropping that rule outright, replacing the source-side toEndpoints with a toServices destination, pointing the deny at a port nothing serves, and removing the capability guard each redden exactly the assertions that name them and nothing else. `matchRegexRaw` and `notMatchRegexRaw` are not used -- helm-unittest 1.0.3 ignores both. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
…thing The kube-ovn-webhook deny names the webhook's backend port only, and that covers a pod dialling the Service ClusterIP for one reason: Cilium translates a service to its backends before it enforces egress policy. That ordering holds only while Cilium performs the DNAT, and the only thing making it so is `cilium.kubeProxyReplacement`, true here against an upstream chart default of "false". Flip it and the ClusterIP path to /mutate-pods reopens while the backend deny keeps working: nothing fails, nothing is logged, and the rendered policy is unchanged. Two halves of a guard. tests/kube_proxy_replacement_test.yaml asserts the value on the RENDERED cilium ConfigMap key -- not on .Values, so an upstream bump that renames the key or stops honouring the override reds too -- once per variant valuesFiles combination that packages/core/platform/sources/networking.yaml installs, plus the defaults case that guest clusters use. Nothing in this repo can observe a live cluster's values, so an operator override in the field is caught only by templates/kube-proxy-replacement-beacon.yaml, a status ConfigMap that renders whenever the value is not true. A beacon and not a fail(): whether Cilium without kube-proxy replacement is a supported cozystack configuration is a platform decision, and refusing to render the cluster's CNI over a weakened defence-in-depth control is not this chart's call. It follows the -awaiting-etcd ConfigMap in packages/apps/kubernetes/templates/cluster.yaml. The package also gains a `test` target, because hack/helm-unit-tests.sh finds suites by probing `make -n test` and silently skips a package without one -- so without this the pin above would never run. Declared .PHONY because hack/package.mk writes its own .PHONY as a variable assignment and therefore declares nothing. Mutation-proven: setting the value false and deleting the umbrella key each red all six variant pins plus the no-beacon case, and neutering the beacon's condition reds exactly the two beacon cases. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
system/ingress-nginx is owned by cozystack.ingress-application, whose derived suite name is `ingress` -- not a suite, so the intersection at the bottom of select-e2e.sh dropped it and an ingress-only change ran only the four tenant-Kubernetes suites. The host cluster's own Ingress coverage lives in the gateway suite, which was therefore never selected by a change to the chart it exercises. cozystack.ingress-nginx owns the same path but cannot carry the mapping on this branch: the caller maps only *-application sources plus external-dns, so a plain system source never reaches src_to_suites at all. The comment says so, because the obvious-looking second arm would be dead code. The new bats case is mutation-proven -- restoring the derived `ingress` name reds it and nothing else -- and sets no EXIT trap, because cozytest.sh runs every @test as a function in one shell, so a test-level trap outlives its test and only the last one registered ever fires. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
Regression for #3983, which reported fail-closed admission webhooks becoming unreachable from the API server while an unrelated namespace was deleted. The mechanism is not confirmed -- the errno in that report cannot come from a policy denial, and namespace deletion churns service backends as well as Cilium identities -- so the case asserts the property the issue is about, that both webhooks still admit across a namespace deletion, rather than claiming to reproduce a cause. Both admission paths are exercised immediately after the deletion with no retry, because a retry would paper over the window this exists to catch. The pod in the churned namespace is what makes it a churn test rather than a smoke test: k8s:io.kubernetes.pod.namespace is an identity-relevant label, so a pod in a brand-new namespace allocates a new Cilium identity and deleting the namespace releases it. An empty namespace drives no identity churn at all. Availability is asserted BEFORE the churn, as a precondition with its own message, because every apply in the case passes through both fail-closed webhooks -- so a cluster whose webhooks were already down fails here in a way that reads exactly like the outage under test. Coordinates come from the webhook registrations themselves, every registration of each, since one being down aborts the apply. Carried unchanged from the upstream change: both webhooks it names exist on this branch and register under the same names. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
…port Companion to the churn case above: that one proves the webhook stays available, this one proves it is still unreachable from pods, which is the property the policy exists for. Rewritten against kube-ovn-webhook rather than ingress-nginx, because this branch ships no ingress-nginx admission policy for the upstream case to measure -- see the PR description. The substance is the same and the structure follows it. The port-scoping assertion is what earns the case. The policy previously expressed the Service-frontend deny with `toServices`, which Cilium resolves only in ALLOW rules, so the deny reached the datapath with an empty L3 selector that Cilium reads as a wildcard when the rule carries toPorts: "deny this Service on 8443" became "deny every destination on 8443" for every pod in the cluster. Every rendered-YAML assertion in the package suites passed on that policy, and so would the two deny verdicts here. So the case stands up its own listener on the denied port and requires that to stay reachable, on both its ClusterIP and its pod IP. Measured on a live kube-ovn+cilium cluster, three ways. With no policy installed the two deny verdicts report BUG, so they are not vacuous. With the policy installed all four verdicts pass, including the ClusterIP path, which is the source-side rule's claim about Cilium translating a service before enforcing egress policy. With the `toServices` form substituted, the two listener verdicts report BUG while the two deny verdicts still pass -- so the positive control is the only thing in the change that separates a scoped deny from a cluster-wide egress outage on that port, and it does separate them. The namespace choice is load-bearing rather than tidiness, and the header carries the measurement: from a tenant namespace both the webhook and an unrelated cozy-kubeovn Service are unreachable with no policy installed at all, because packages/apps/tenant ships a default-deny egress policy per tenant that admits only namespaces carrying tenant.cozystack.io/*. A deny verdict measured from there would hold with the policy uninstalled. The case therefore creates its own cozystack.io/system namespace, where the same cluster reached all four destinations before the policy went on. Every verdict is gated on ready EndpointSlice backends before, during and after the probe and on the webhook pods neither restarting nor being replaced, because "denied by policy" and "no backend to route to" are the same observation from inside a probe. Anything that leaves the question undecidable fails as a setup failure rather than as a verdict. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
myasnikovdaniil
force-pushed
the
backport-3997-to-release-1.6
branch
from
September 3, 2026 10:06
17e38ad to
eb232a4
Compare
Contributor
|
Not needed |
myasnikovdaniil
added a commit
that referenced
this pull request
Sep 24, 2026
…w first start under load (#4433) Backport of #3470 to `release-1.6`. On the E2E run of #4052 install failed because reloader crash-looped (6 restarts) and `failed early due to stalled resources: [Deployment/cozy-reloader/reloader-reloader status: 'Failed']` blocked the whole linstor chain behind it. Same shape as what #3470 fixes on main, slow first start under load killed by the probes. This one changes the chart, not only CI: reloader gets a startupProbe. Cherry-picked clean with `-x`. ### Testing - `make unit-tests` green (includes the reloader helm unit tests from #3470), POSIX sh sweep clean. ```release-note fix(reloader): add startupProbe to survive slow first start under load ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hand-redone backport of the kube-ovn-webhook half of #3997. The automated attempt applied 2 of that PR's 14 commits, committed conflict markers into
hack/select-e2e.shandhack/select-e2e_test.bats, and never attempted the rest, so this replaces it rather than fixing it.Scope: the ingress-nginx half is deliberately not carried
packages/system/ingress-nginxon this branch has notemplates/, notests/, and noadmissionWebhookPolicykey anywhere in the tree. That restriction arrived on main through #3798, which was never backported, so the files #3997 rewrites and deletes there do not exist here. Carrying it would import #3798's behaviour change into a patch release, and it would newly attach a cluster-spanning peer to the ingress controller endpoints - which is the churn coupling #3983 reported, not a fix for it.The kube-ovn-webhook side is different: its
templates/networkpolicy.yamlhere is the same blob as main's copy immediately before #3997, so that fix applies to exactly the code it was written against.Tenant-side beacon (
packages/apps/kubernetes) is not carried either. Its only justification is the guest cluster's ingress-nginx deny, which this branch does not ship, so its rendered message would name a policy that does not exist. The management-side cilium beacon is carried, because the kube-ovn-webhook rule does depend on that value.Consequence worth stating:
packages/apps/kubernetes/templates/helmreleases/cilium.yamlis untouched, so #4033'sstrategy: RetryOnFailurestands byte for byte and there was no collision to resolve.What it does
Pod deny is enforced from both ends: the source-side
egressDenyon the callers, and a target-sideingressDenyon the webhook pods for the backend port, so enforcement does not depend on which component performs the service DNAT. ThekubeProxyReplacementdependency that shape rests on is named in the template and in values, and a render-time beacon inpackages/system/ciliummakes an override visible instead of silent.The enforcement chainsaw case was rewritten against the pod admission webhook - the upstream version targeted the ingress admission webhook and would have failed here - and
hack/select-e2e.shgets only theingress-applicationarm that is reachable on this branch.Testing
Unit:
make unit-testsgreen (1272 helm assertions, 375 bats cases, 0 failures),make test-controllersgreen,make generateno drift with the CI-pinnedcozyvalues-genv1.6.0.make testinpackages/system/kubeovn-webhook3 suites/11 tests and inpackages/system/cilium1 suite/9 tests.helm templateexercised with and without--api-versionsso both branches of the capability gate render, and across all five cilium variant values combinations pluskubeProxyReplacement=false.helm linton cilium fails on a missing chart-metadata dependency identically at base, verified against a clean archive of this branch.POSIX sh gate clean on all five touched surfaces, plus both chainsaw script bodies executed end to end under
dashrather than under the local shell.Live threat run on a kube-ovn plus cilium stand, which is the part worth reading: with no policy installed, both deny verdicts report BUG, so the assertions are not vacuous. With the policy on, four OK including the ClusterIP path. With
toServicessubstituted back in, the two listener verdicts report BUG while both deny verdicts still pass - reproducing the wildcard finding and proving that the deny assertions alone would have passed a broken policy.fromEntities: [all]is left to the unit suite, because live it denies the API server on afailurePolicy: Failwebhook.No migration touched,
targetVersionstill 54.