Skip to content

[Backport release-1.6] fix(networking): prevent webhook admission outages during namespace churn - #4052

Closed
github-actions[bot] wants to merge 6 commits into
release-1.6from
backport-3997-to-release-1.6
Closed

github-actions[bot] wants to merge 6 commits into
release-1.6from
backport-3997-to-release-1.6

Conversation

@github-actions

@github-actions github-actions Bot commented Sep 3, 2026 •

Copy link
Copy Markdown

Hand-redone backport of the kube-ovn-webhook half of #3997. The automated attempt applied 2 of that PR's 14 commits, committed conflict markers into hack/select-e2e.sh and hack/select-e2e_test.bats, and never attempted the rest, so this replaces it rather than fixing it.

Scope: the ingress-nginx half is deliberately not carried

packages/system/ingress-nginx on this branch has no templates/, no tests/, and no admissionWebhookPolicy key anywhere in the tree. That restriction arrived on main through #3798, which was never backported, so the files #3997 rewrites and deletes there do not exist here. Carrying it would import #3798's behaviour change into a patch release, and it would newly attach a cluster-spanning peer to the ingress controller endpoints - which is the churn coupling #3983 reported, not a fix for it.

The kube-ovn-webhook side is different: its templates/networkpolicy.yaml here is the same blob as main's copy immediately before #3997, so that fix applies to exactly the code it was written against.

Tenant-side beacon (packages/apps/kubernetes) is not carried either. Its only justification is the guest cluster's ingress-nginx deny, which this branch does not ship, so its rendered message would name a policy that does not exist. The management-side cilium beacon is carried, because the kube-ovn-webhook rule does depend on that value.

Consequence worth stating: packages/apps/kubernetes/templates/helmreleases/cilium.yaml is untouched, so #4033's strategy: RetryOnFailure stands byte for byte and there was no collision to resolve.

What it does

Pod deny is enforced from both ends: the source-side egressDeny on the callers, and a target-side ingressDeny on the webhook pods for the backend port, so enforcement does not depend on which component performs the service DNAT. The kubeProxyReplacement dependency that shape rests on is named in the template and in values, and a render-time beacon in packages/system/cilium makes an override visible instead of silent.

The enforcement chainsaw case was rewritten against the pod admission webhook - the upstream version targeted the ingress admission webhook and would have failed here - and hack/select-e2e.sh gets only the ingress-application arm that is reachable on this branch.

Testing

Unit: make unit-tests green (1272 helm assertions, 375 bats cases, 0 failures), make test-controllers green, make generate no drift with the CI-pinned cozyvalues-gen v1.6.0. make test in packages/system/kubeovn-webhook 3 suites/11 tests and in packages/system/cilium 1 suite/9 tests. helm template exercised with and without --api-versions so both branches of the capability gate render, and across all five cilium variant values combinations plus kubeProxyReplacement=false. helm lint on cilium fails on a missing chart-metadata dependency identically at base, verified against a clean archive of this branch.

POSIX sh gate clean on all five touched surfaces, plus both chainsaw script bodies executed end to end under dash rather than under the local shell.

Live threat run on a kube-ovn plus cilium stand, which is the part worth reading: with no policy installed, both deny verdicts report BUG, so the assertions are not vacuous. With the policy on, four OK including the ClusterIP path. With toServices substituted back in, the two listener verdicts report BUG while both deny verdicts still pass - reproducing the wildcard finding and proving that the deny assertions alone would have passed a broken policy. fromEntities: [all] is left to the unit suite, because live it denies the API server on a failurePolicy: Fail webhook.

No migration touched, targetVersion still 54.

fix(networking): enforce the Pod admission webhook deny on both the caller and the webhook endpoint, so it no longer depends on Cilium performing service load-balancing

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Author

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-3997-to-release-1.6
git worktree add --checkout .worktree/backport-3997-to-release-1.6 backport-3997-to-release-1.6
cd .worktree/backport-3997-to-release-1.6
git reset --hard HEAD^
git cherry-pick -x ad961f6de1b3ec3526f0f56212b9e35dbc83078a f224baae461bafc01504f4f3db60214e8cb356ce 6b2b68a685a828056f2e13c9ae5d819949dc45c3 bdc172a64855ec5af009f9a0c2559ed74a22ec02 f73b54fee963c59f62e26954ce7d19338bfb0d54 9f485a18ad51ce216ec835d20f896e7fb3090d27 042ac2715bfdba51c721e80306cfe5ff94e8e9ae 6a289cf6d068f14b48497e9bbb396d96fdb2418d cbd0e78cadb90560ce1d59df886e9f24466a424a 3daa83f4b5a4b0d50852bdbbd0033b0a7d2077be b78405cede870c48201089588b69921be7793bc5 0335421769af296b274e33e0e10455f1f09a6752 686a81a491f6902d0d1aff9533fa243ced00e087
git push --force-with-lease

Backport of #3997 to `release-1.6`, kube-ovn-webhook half.

The namespaced CiliumNetworkPolicy/kube-ovn-webhook carried a
cluster-spanning fromEndpoints peer on the webhook's own endpoint, so
every namespace add or delete rewrote that endpoint's policy map -- the
coupling #3983 reported against fail-closed admission webhooks. It also
depended on nothing but that one end: while the map is being recomputed
or has overflowed, the deny is the thing being rebuilt.

What ships now is one cluster-scoped
CiliumClusterwideNetworkPolicy/kube-ovn-webhook-c2bdbc146112 carrying two
specs that fail in different directions. The source-side spec is an
egressDeny on every pod endpoint naming the webhook's backend port, so it
holds while the webhook endpoint's own policy is degraded, but it relies
on Cilium performing the service DNAT ahead of its egress hook. The
target-side spec keeps the pod deny and the world deny on the webhook
endpoint, evaluated after whatever DNAT already happened, so it holds
where Cilium is not the service load balancer. Neither subsumes the
other, so the churn writes stop being load-bearing rather than going
away -- the template says so rather than claiming the coupling is
removed.

`toServices` is deliberately not used to name the Service frontend, and
the template carries the reason: Cilium resolves it only in allow rules
(pkg/policy/k8s/service.go walks spec.Egress, never spec.EgressDeny), so
an egressDeny whose only destination is an unresolved toServices reaches
the datapath with an empty L3 selector that Cilium reads as a wildcard
when the rule carries toPorts -- a cluster-wide egress outage on that
port rather than a tighter deny.

The object changes kind and name, so anything keyed on the old namespaced
CiliumNetworkPolicy/kube-ovn-webhook goes quiet with no error. The
template now also renders only while
cilium.io/v2/CiliumClusterwideNetworkPolicy is discoverable, so
`enabled: true` is no longer a promise that a policy exists; set
`enabled: false` to record a deliberate opt-out.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
hack/helm-unit-tests.sh finds suites by probing `make -n test` per
package and skips any package without that target, so this package's two
suites had never run in CI -- including the port agreement guard, whose
whole purpose is to catch a drift that fails open. The target is declared
.PHONY because nothing else declares it: hack/package.mk writes its
.PHONY as a variable assignment rather than a target, so a path named
`test` next to the Makefile would make the target up to date and skip the
recipe silently.

The suites now pin the two-spec shape rather than the old single-spec
one, and add what review asked for: one deny rule per spec whose
destination is a selector Cilium can actually resolve (the toServices
regression renders as a plausible ingressDeny and would otherwise pass),
the peer of the target-side pod deny, the collision-safe cluster-scoped
name, and a capability suite for the render-nothing branch.

Checked by mutation rather than by revert. Substituting
`fromEntities: [all]` for the target-side pod peer, dropping that rule
outright, replacing the source-side toEndpoints with a toServices
destination, pointing the deny at a port nothing serves, and removing the
capability guard each redden exactly the assertions that name them and
nothing else. `matchRegexRaw` and `notMatchRegexRaw` are not used --
helm-unittest 1.0.3 ignores both.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
…thing

The kube-ovn-webhook deny names the webhook's backend port only, and that
covers a pod dialling the Service ClusterIP for one reason: Cilium
translates a service to its backends before it enforces egress policy.
That ordering holds only while Cilium performs the DNAT, and the only
thing making it so is `cilium.kubeProxyReplacement`, true here against an
upstream chart default of "false". Flip it and the ClusterIP path to
/mutate-pods reopens while the backend deny keeps working: nothing fails,
nothing is logged, and the rendered policy is unchanged.

Two halves of a guard. tests/kube_proxy_replacement_test.yaml asserts the
value on the RENDERED cilium ConfigMap key -- not on .Values, so an
upstream bump that renames the key or stops honouring the override reds
too -- once per variant valuesFiles combination that
packages/core/platform/sources/networking.yaml installs, plus the
defaults case that guest clusters use. Nothing in this repo can observe a
live cluster's values, so an operator override in the field is caught
only by templates/kube-proxy-replacement-beacon.yaml, a status ConfigMap
that renders whenever the value is not true.

A beacon and not a fail(): whether Cilium without kube-proxy replacement
is a supported cozystack configuration is a platform decision, and
refusing to render the cluster's CNI over a weakened defence-in-depth
control is not this chart's call. It follows the -awaiting-etcd ConfigMap
in packages/apps/kubernetes/templates/cluster.yaml.

The package also gains a `test` target, because hack/helm-unit-tests.sh
finds suites by probing `make -n test` and silently skips a package
without one -- so without this the pin above would never run. Declared
.PHONY because hack/package.mk writes its own .PHONY as a variable
assignment and therefore declares nothing.

Mutation-proven: setting the value false and deleting the umbrella key
each red all six variant pins plus the no-beacon case, and neutering the
beacon's condition reds exactly the two beacon cases.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
system/ingress-nginx is owned by cozystack.ingress-application, whose
derived suite name is `ingress` -- not a suite, so the intersection at
the bottom of select-e2e.sh dropped it and an ingress-only change ran
only the four tenant-Kubernetes suites. The host cluster's own Ingress
coverage lives in the gateway suite, which was therefore never selected
by a change to the chart it exercises.

cozystack.ingress-nginx owns the same path but cannot carry the mapping
on this branch: the caller maps only *-application sources plus
external-dns, so a plain system source never reaches src_to_suites at
all. The comment says so, because the obvious-looking second arm would
be dead code.

The new bats case is mutation-proven -- restoring the derived `ingress`
name reds it and nothing else -- and sets no EXIT trap, because
cozytest.sh runs every @test as a function in one shell, so a test-level
trap outlives its test and only the last one registered ever fires.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
Regression for #3983, which reported fail-closed admission webhooks
becoming unreachable from the API server while an unrelated namespace was
deleted. The mechanism is not confirmed -- the errno in that report
cannot come from a policy denial, and namespace deletion churns service
backends as well as Cilium identities -- so the case asserts the property
the issue is about, that both webhooks still admit across a namespace
deletion, rather than claiming to reproduce a cause. Both admission paths
are exercised immediately after the deletion with no retry, because a
retry would paper over the window this exists to catch.

The pod in the churned namespace is what makes it a churn test rather
than a smoke test: k8s:io.kubernetes.pod.namespace is an
identity-relevant label, so a pod in a brand-new namespace allocates a
new Cilium identity and deleting the namespace releases it. An empty
namespace drives no identity churn at all.

Availability is asserted BEFORE the churn, as a precondition with its own
message, because every apply in the case passes through both fail-closed
webhooks -- so a cluster whose webhooks were already down fails here in a
way that reads exactly like the outage under test. Coordinates come from
the webhook registrations themselves, every registration of each, since
one being down aborts the apply.

Carried unchanged from the upstream change: both webhooks it names exist
on this branch and register under the same names.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
…port

Companion to the churn case above: that one proves the webhook stays
available, this one proves it is still unreachable from pods, which is
the property the policy exists for.

Rewritten against kube-ovn-webhook rather than ingress-nginx, because
this branch ships no ingress-nginx admission policy for the upstream case
to measure -- see the PR description. The substance is the same and the
structure follows it.

The port-scoping assertion is what earns the case. The policy previously
expressed the Service-frontend deny with `toServices`, which Cilium
resolves only in ALLOW rules, so the deny reached the datapath with an
empty L3 selector that Cilium reads as a wildcard when the rule carries
toPorts: "deny this Service on 8443" became "deny every destination on
8443" for every pod in the cluster. Every rendered-YAML assertion in the
package suites passed on that policy, and so would the two deny verdicts
here. So the case stands up its own listener on the denied port and
requires that to stay reachable, on both its ClusterIP and its pod IP.

Measured on a live kube-ovn+cilium cluster, three ways. With no policy
installed the two deny verdicts report BUG, so they are not vacuous. With
the policy installed all four verdicts pass, including the ClusterIP
path, which is the source-side rule's claim about Cilium translating a
service before enforcing egress policy. With the `toServices` form
substituted, the two listener verdicts report BUG while the two deny
verdicts still pass -- so the positive control is the only thing in the
change that separates a scoped deny from a cluster-wide egress outage on
that port, and it does separate them.

The namespace choice is load-bearing rather than tidiness, and the header
carries the measurement: from a tenant namespace both the webhook and an
unrelated cozy-kubeovn Service are unreachable with no policy installed
at all, because packages/apps/tenant ships a default-deny egress policy
per tenant that admits only namespaces carrying tenant.cozystack.io/*. A
deny verdict measured from there would hold with the policy uninstalled.
The case therefore creates its own cozystack.io/system namespace, where
the same cluster reached all four destinations before the policy went on.

Every verdict is gated on ready EndpointSlice backends before, during and
after the probe and on the webhook pods neither restarting nor being
replaced, because "denied by policy" and "no backend to route to" are the
same observation from inside a probe. Anything that leaves the question
undecidable fails as a setup failure rather than as a verdict.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
@myasnikovdaniil
myasnikovdaniil force-pushed the backport-3997-to-release-1.6 branch from 17e38ad to eb232a4 Compare September 3, 2026 10:06
@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files labels Sep 3, 2026
@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Not needed

myasnikovdaniil added a commit that referenced this pull request Sep 24, 2026
…w first start under load (#4433)

Backport of #3470 to `release-1.6`.

On the E2E run of #4052 install failed because reloader crash-looped (6
restarts) and `failed early due to stalled resources:
[Deployment/cozy-reloader/reloader-reloader status: 'Failed']` blocked
the whole linstor chain behind it. Same shape as what #3470 fixes on
main, slow first start under load killed by the probes. This one changes
the chart, not only CI: reloader gets a startupProbe.

Cherry-picked clean with `-x`.

### Testing

- `make unit-tests` green (includes the reloader helm unit tests from
#3470), POSIX sh sweep clean.

```release-note
fix(reloader): add startupProbe to survive slow first start under load
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant