Repository navigation
feat(gateway)!: add TLS-passthrough listeners to TenantGateway - #3342
Aleksei Sviridkin (lexfrei) merged 5 commits into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughTenantGateway gains structured TLS passthrough listeners with CRD and controller validation, Gateway rendering, listener-count limits, route selector isolation, generated deepcopy support, comprehensive tests, and updated gateway documentation. ChangesTenant TLS passthrough listeners
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant TenantGateway
participant Admission
participant Reconciler
participant Gateway
TenantGateway->>Admission: submit listener configuration
Admission->>Admission: apply structural schema and CEL rules
Admission-->>TenantGateway: accept or reject specification
TenantGateway->>Reconciler: reconcile accepted specification
Reconciler->>Reconciler: validate and render listeners
Reconciler->>Gateway: publish TLS passthrough listeners and route policies
Possibly related issues
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
IvanHunters
left a comment
There was a problem hiding this comment.
Reviewed hermetically. LGTM with non-blocking notes.
Verified:
- go build/vet/test green;
zz_generated.deepcopy.gois committed and matches the newTLSPassthroughListenertype and CRD markers (nomake generatestaleness). - Tests are non-vacuous: neutralising the reserved-port guard turns the port-collision subtests red. Negative cases cover duplicate name/port, reserved ports, DNS-1123, malformed/wildcard hostname, out-of-apex, service-name collision.
- Validation errors surface on status (
Ready=False, Reason=ReconcileError), not just logs. hostnameWithinApexmatches thecozystack-gateway-hostname-policyVAP CEL; it is a defensive pre-check, the VAP stays authoritative.- Upgrade: additive
+optional/omitemptyfield, no default flip, no migration.
Non-blocking:
- Phased feature: a rendered passthrough listener has no TLSRoute/certificate wiring yet (later phases), so a tenant setting
tlsPassthroughListenersgets a listener that does nothing until they supply their own TLSRoute and backend. Worth a docs note. - One-port-per-listener (duplicate port rejected) is enforced by design rather than SNI-multiplexing; flagging.
acaffb4 to
3958dd1
Compare
|
Both notes were right, thanks. The phased-feature note went further than a docs line: the field description now states that the listener is all it creates, that nothing routes until a TLSRoute attaches by On one-port-per-listener you read it correctly: it is enforced, not inherent. Gateway API keys TLS listeners by (port, protocol, hostname), so same-port SNI selection is legal and the port-443 passthrough listeners already rely on it. It stays for this phase because the field's whole point is the engine's native port, where a second listener means two engines behind one port with SNI deciding, and there is no routing or certificate plumbing yet to make that shape testable. Lifting it is deleting a check — no API or schema change — so it remains available later. That reasoning is now in the code instead of implied. Two things the follow-up pass turned up that are worth flagging: the validation rules initially pushed the CRD past the apiserver's install-time CEL cost budget, making it un-installable on both create and upgrade while every per-request test stayed green — there is now a test running the real CRD validation; and two passthrough listeners could match the same SNI, including a wildcard covering the |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@internal/controller/tenantgateway/crdcel_test.go`:
- Around line 327-340: Add an explicit version-found guard to both CRD schema
validation loops in the relevant tests, including
TestPassthroughListenerCapFitsGatewayAPI. Track whether v1alpha1 was encountered
and fail the test when it is absent, while preserving the existing assertions
for the located version.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 7e2120f5-f9fb-4f81-84f3-1387e95dd346
📒 Files selected for processing (8)
api/gateway/v1alpha1/tenantgateway_types.goapi/gateway/v1alpha1/zz_generated.deepcopy.gointernal/controller/tenantgateway/crdcel_test.gointernal/controller/tenantgateway/reconciler.gointernal/controller/tenantgateway/reconciler_test.gointernal/controller/tenantgateway/renderers.gopackages/extra/gateway/README.mdpackages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
- internal/controller/tenantgateway/renderers.go
- packages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml
- internal/controller/tenantgateway/reconciler.go
- packages/extra/gateway/README.md
- api/gateway/v1alpha1/tenantgateway_types.go
- internal/controller/tenantgateway/reconciler_test.go
myasnikovdaniil
left a comment
There was a problem hiding this comment.
NOT LGTM — the mechanism is sound and phase 1 is unusually well hedged, but this PR adds documentation that the controller does not honor, and the TLSRoute path it exists to enable has no test coverage at all. Separately, it is hard-blocked by the API-owner gate regardless of technical verdict.
Business context: phase 1 of #2815 (external database exposure via SNI passthrough) — adds the tlsPassthroughListeners API field and renders one layer-4 mode: Passthrough listener per entry on a native DB port, with no engine, TLSRoute, or certificate wiring yet.
Process gate — needs an API owner, nothing in the diff can satisfy it
.github/workflows/api-review-gate.yaml:36 sets API_OWNERS: "lllamnyp kvaps", and the approval query at :139-151 counts only each reviewer's latest review on the current head commit (select(.commit_id == $head)) with state APPROVED. So this needs Timofei Larkin (@lllamnyp) or Andrei Kvapil (@kvaps) to approve head 3958dd163 directly; any new push invalidates it. The job re-runs on pull_request_review, so an approval flips the check without needing a push. Flagging so it is not mistaken for something the diff can fix.
Also worth knowing before anyone reads the check list as a code signal: gh pr checks reports E2E Tests as fail, but the API reports "conclusion": "cancelled" for job 88245594808 after 1h2m. That is a concurrency cancellation, not a test failure.
Blockers
B1: the field's documented contract is falsified by the controller once a TLSRoute attaches
api/gateway/v1alpha1/tenantgateway_types.go:331 and :337, internal/controller/tenantgateway/reconciler.go:907
Evidence — verified by execution, not by reading. HTTP-01 TenantGateway, one entry {postgres, 5432, postgres.foo.example.com}, plus a TLSRoute in tenant-foo with sectionName: tls-postgres and spec.hostnames: [postgres.foo.example.com]. Rendered result:
LISTENER name=http port=80 proto=HTTP
LISTENER name=https-postgres-715667a8 port=443 proto=HTTPS host=postgres.foo.example.com
LISTENER name=tls-postgres port=5432 proto=TLS host=postgres.foo.example.com
CERT name=cozystack-postgres-715667a8-tls dnsNames=[postgres.foo.example.com]
collectHostnameClaims (reconciler.go:343-367) folds TLSRoute hostnames into the same claims map as HTTPRoute hostnames; those become winners → dynHostnames (:136-142), which drives both per-app HTTPS listener rendering and reconcilePerListenerCertificates (:175). It never consults sectionName or listener protocol.
Three consequences, each contradicting text this PR adds:
:331says "the backend must present its own certificate … the Gateway neither holds nor issues that certificate". It does issue one — a real Let's Encrypt order against a hostname that needs no Gateway certificate, consuming LE rate limit.:337says the field costs "one per entry here", and:346capsMaxItems=62. A routed entry costs two listener slots, so themaxGatewayListenersguard atreconciler.go:937can trip at roughly half the documented headroom.- A stray HTTPS-terminate listener appears for the database hostname on :443 and will never serve anything useful.
Important qualifier: the underlying defect is pre-existing, not introduced here. I re-ran the identical probe using TLSPassthroughServices — the field already on main, with collectHostnameClaims untouched by this PR — and reproduced the same leak. There it is worse, producing https-postgres-… and tls-postgres both on port 443 with the same hostname, which is a Gateway API protocol conflict.
So this is not "your new code is broken". It is that the PR adds prose asserting a property the codebase does not have, in the exact area phase 2 will build on.
Minimum fix: correct the two claims at :331 and :337 and file a tracking issue — phase 2 is precisely when TLSRoutes get wired and this fires for real. Better fix: filter routeKindTLS refs out of the dynHostnames computation while keeping them in allRefs for conflict and status purposes.
Raised independently by automated review as its sole finding; the pre-existing nature is my addition.
B2: no TLSRoute appears anywhere in the test suite
internal/controller/tenantgateway/reconciler_test.go
Evidence: rg -n "TLSRoute\{" internal/controller/tenantgateway/reconciler_test.go returns zero hits across ~4,500 lines. There is an httpRouteAttached helper (:1648) with no TLSRoute counterpart. The entire TLSRoute branch of collectHostnameClaims is unexercised — which is why B1 survived 22 commits unnoticed.
Given this PR's whole purpose is to make TLSRoute attachment possible, and the area has a substantial existing test convention, shipping the enabling change with no test of the thing being enabled is the gap worth closing before merge.
Fix: add a tlsRouteAttached helper and one test pinning what a TLSRoute attached to a passthrough listener should and should not produce. That test is also what would have caught B1.
Action items (non-blocking)
1. The route-hostname VAP has a hostname-less escape. packages/system/cozystack-basics/templates/route-hostname-policy.yaml:38 reads !has(object.spec.hostnames) || object.spec.hostnames.all(…). A TLSRoute that omits hostnames passes the VAP and, per Gateway API ([email protected]/apis/v1alpha2/tlsroute_types.go:65-67), inherits all the listener's hostnames. Combined with the attach set — packages/apps/tenant/templates/namespace.yaml stamps namespace.cozystack.io/gateway on every inheriting child tenant namespace — a child could in principle attach to a parent's passthrough listener.
Materially downgraded by the README's own note that "tenants in Cozystack do not hold gateway.networking.k8s.io/* RBAC by design, so this is not a tenant-user defense" (packages/extra/gateway/README.md:110), so this is defense-in-depth only and pre-existing. But on a native DB port the failure mode has teeth: PostgreSQL sslmode=require does not verify the server certificate, so a hijacking backend could harvest credentials. Worth closing before phase 2 rather than after.
2. The open nit on crdcel_test.go:327-340 (add a version-found guard to the CRD schema loops) is legitimate and cheap.
3. The renderWildcardCertificate gofmt-only hunk in renderers.go (the seen map realignment) is unrelated noise in an already-oversized diff.
Phase-1 risk assessment: safe, and better hedged than most
Worth stating plainly since it is the main thing a reviewer will worry about — the field is not reachable through any supported path:
packages/extra/gateway/templates/tenantgateway.yamlplumbsattachedNamespacesandtlsPassthroughServicesbut has notlsPassthroughListenersstanza, and the README says so outright: "the chart exposes no corresponding value yet … akubectl editaddingtlsPassthroughListenersis drift that Flux reverts on its next reconcile."- Reaching the field therefore requires cluster-admin RBAC on
TenantGatewayplus a suspended HelmRelease. - If someone does set it, validation is genuinely thorough.
validateTLSPassthroughListeners(renderers.go:233) rejects bad DNS-1123 names, duplicate names, collisions withtlsPassthroughServices, out-of-range ports, reserved 80/443, duplicate ports, malformed hostnames, out-of-apex hostnames, and wildcard-aware SNI overlap (hostnamesOverlap,:341). Four are additionally enforced at admission via CEL. Failures surface asReady=Falseon TenantGateway status rather than a wholesale Gateway rejection — the stated design goal, and it is met. - The listener renders pointing at nothing until a TLSRoute attaches: inert, not half-working — except for B1, which fires only once a TLSRoute exists.
Reversibility is adequate without a feature gate: v1alpha1, +optional, omitempty, no default, chart-unreachable. If phase 2 invalidates the design the field can be removed with no user-visible break.
Dismissed after investigation
- Listener collapse (cilium#45559) regression — the new listeners sit on distinct native ports, so same-port collapse cannot apply, and they correctly carry
Kinds = [TLSRoute]alone rather than the sharedport443Kinds(reconciler.go:907-921), with the reasoning in-code.TestReconcile_Port443ListenersShareKindsfilters onl.Port == 443(reconciler_test.go:4227), so the new listeners are correctly outside its contract and the guard is intact. Bonus:Kinds=[TLSRoute]meanslen(supportedKinds) == len(AllowedRoutes.Kinds), so these listeners will not surface the cosmeticResolvedRefs=False/InvalidRouteKindsthat the :443 ones do on Cilium 1.19.x. - Weakening the GRPCRoute/TCPRoute/UDPRoute exclusion —
reconciler_test.go:4262-4266still asserts the forbidden-kind set for :443, andTestReconcile_TLSPassthroughListenerObjects(:2156-2159) asserts the new listeners carry exactly[TLSRoute]plus the namespace selector. Strictly tighter than the existing bar. - Stale generated artifacts —
zz_generated.deepcopy.gocarries the new type, the CRD YAML matches every marker, andVerify generated code is up to datepasses.go build ./...andgo test ./internal/controller/tenantgateway/...both green locally. - "No negative-path tests" — disproven for the validation layer: eight dedicated rejection-path test functions (
reconciler_test.go:2451, 2480, 2509, 2539;crdcel_test.go:318, 354, 415, 455), includingTestSpecCELMatchesControllerValidationfor CEL/controller parity andTestCRDPassesInstallTimeValidationfor the CEL cost budget — the latter a genuinely good catch. The gap is narrower than "no tests": it is specifically the TLSRoute interaction, which is B2. - The aliasing changes (
slices.Clone(port443Kinds),DeepCopy()) fix a real latent hazard — the oldpassthroughAllowed := *allowedRoutesshared both theNamespacespointer and theKindsslice across every listener. Nothing mutated them, so it was latent rather than active; the fix is correct and covered byTestReconcile_ListenerAllowedRoutesNotAliased(:2238).
Scope drift
The advertised +571/-1 across 6 files was exact at commit 3a60cc72a, the 2nd of 22 — git diff --stat origin/main...3a60cc72a reports precisely that. Current head is +1662/-38 across 8 files, so 20 subsequent commits added roughly 3×:
| Category | Weight |
|---|---|
| New CEL admission layer + cost-budget guard | crdcel_test.go (466 lines, new file) + ~100 lines of markers |
| Wildcard-aware SNI overlap matching | hostnamesOverlap / hostnameCovers in renderers.go |
| AllowedRoutes aliasing fixes | reconciler.go — touches pre-existing :443 and wildcard listeners |
| Listener-cap guard | reconciler.go:937 |
| Docs | README + large doc-comment blocks |
Two observations worth putting on the record rather than blocking on: the aliasing fixes and the tlsPassthroughServices tightening (MaxItems, listType=set, item pattern) are behavior changes to pre-existing fields that the title does not advertise and that the PR body itself notes affect already-deployed objects; and a large share of the added lines is prose inside Go doc comments, with the TenantGatewaySpec block alone running ~35 lines of design rationale. The latter is defensible — the CEL-cost reasoning is genuinely non-obvious — but it inflates the diff against the size/XXL label. Splitting the pre-existing-field hardening into its own PR would have kept phase 1 near its advertised size and made both halves easier to review.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
internal/controller/tenantgateway/crdcel_test.go (1)
352-352: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAvoid duplicating the listener cap magic number.
gatewayAPIListenerCaphardcodes64locally instead of referencing the actualmaxGatewayListenersconstant checked inrenderGateway(reconciler.go). If that constant ever changes, this test silently stops verifying the real relationship it's meant to pin.♻️ Proposed fix
func TestPassthroughListenerCapFitsGatewayAPI(t *testing.T) { - const gatewayAPIListenerCap = 64 + const gatewayAPIListenerCap = maxGatewayListeners field := v1alpha1SpecSchema(t).Properties["tlsPassthroughListeners"]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/tenantgateway/crdcel_test.go` at line 352, Update the test constant gatewayAPIListenerCap to reuse the maxGatewayListeners constant used by renderGateway in reconciler.go instead of defining a separate value of 64, so the test remains aligned with the production listener limit.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@internal/controller/tenantgateway/crdcel_test.go`:
- Line 352: Update the test constant gatewayAPIListenerCap to reuse the
maxGatewayListeners constant used by renderGateway in reconciler.go instead of
defining a separate value of 64, so the test remains aligned with the production
listener limit.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4f93ae2c-6ccf-4a9c-b9a8-448693169592
📒 Files selected for processing (5)
internal/controller/tenantgateway/crdcel_test.gointernal/controller/tenantgateway/reconciler.gointernal/controller/tenantgateway/reconciler_test.gointernal/controller/tenantgateway/renderers.gopackages/extra/gateway/README.md
🚧 Files skipped from review as they are similar to previous changes (2)
- internal/controller/tenantgateway/renderers.go
- packages/extra/gateway/README.md
|
myasnikovdaniil You are right on both, and the reproduction saved me finding it. B1 went the second way you offered, B2 is closed. There is a Action item 2 is in Action item 3 I am keeping. Action item 1 gets its own PR. It changes a VAP that gates every route in the cluster, and your scope-drift point is the reason not to fold it in here. Going the other way, "the Gateway API rejects it wholesale" was wrong in this PR, in a few comments and in the The gate stays red until an API owner approves the head commit. |
78f8d7e to
a8a1ae2
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
internal/controller/tenantgateway/crdcel_test.go (1)
330-330: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueNit: local
fieldshadows the importedfieldpackage.Harmless today (neither function uses
field.NewPath), but a future assertion needingfield.NewPathhere would fail to compile confusingly. ConsiderproporlistenersSchema.Also applies to: 353-353
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/tenantgateway/crdcel_test.go` at line 330, Rename the local field variable in the affected test cases around v1alpha1SpecSchema, including the occurrence near the additional referenced line, to avoid shadowing the imported field package. Use a descriptive name such as listenersSchema or prop and update all references within those test scopes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@internal/controller/tenantgateway/crdcel_test.go`:
- Line 330: Rename the local field variable in the affected test cases around
v1alpha1SpecSchema, including the occurrence near the additional referenced
line, to avoid shadowing the imported field package. Use a descriptive name such
as listenersSchema or prop and update all references within those test scopes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 862767f8-9c7e-4055-8f3c-49c13e7f9cf3
📒 Files selected for processing (8)
api/gateway/v1alpha1/tenantgateway_types.gointernal/controller/tenantgateway/conflict.gointernal/controller/tenantgateway/crdcel_test.gointernal/controller/tenantgateway/reconciler.gointernal/controller/tenantgateway/reconciler_test.gointernal/controller/tenantgateway/renderers.gopackages/extra/gateway/README.mdpackages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
- internal/controller/tenantgateway/renderers.go
- packages/extra/gateway/README.md
- packages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml
- api/gateway/v1alpha1/tenantgateway_types.go
- internal/controller/tenantgateway/reconciler.go
- internal/controller/tenantgateway/reconciler_test.go
a8a1ae2 to
a467c51
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
internal/controller/tenantgateway/renderers.go (1)
356-371: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueComment sits above the wrong check.
Lines 356-368 explain SNI-indistinguishable hostnames across ports — that's the rationale for the overlap loop at Line 376, not for the apex-containment check at Line 369, which has its own unrelated (VAP) justification in the error string. Moving it down to the overlap block would keep each rationale next to the code it defends.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@internal/controller/tenantgateway/renderers.go` around lines 356 - 371, The explanatory comment about listeners sharing hostnames across ports is positioned above the unrelated hostname apex validation. Move that comment so it immediately precedes the overlap-detection loop or check that rejects duplicate hostnames across different ports, while keeping the existing VAP-related rationale adjacent to the hostnameWithinApex validation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@api/gateway/v1alpha1/tenantgateway_types.go`:
- Around line 301-315: The validation markers on TLSPassthroughServices must
preserve updates to existing invalid values through ratcheting rules. Replace
the direct MaxItems, item MaxLength, item Pattern, and set validation with CEL
ratcheting expressions that allow self == oldSelf while enforcing the new
constraints for changed values, preserving duplicate and legacy entries on
unrelated updates.
---
Nitpick comments:
In `@internal/controller/tenantgateway/renderers.go`:
- Around line 356-371: The explanatory comment about listeners sharing hostnames
across ports is positioned above the unrelated hostname apex validation. Move
that comment so it immediately precedes the overlap-detection loop or check that
rejects duplicate hostnames across different ports, while keeping the existing
VAP-related rationale adjacent to the hostnameWithinApex validation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9f046279-bc6e-4469-a718-4a14d387d3fa
📒 Files selected for processing (8)
api/gateway/v1alpha1/tenantgateway_types.gointernal/controller/tenantgateway/conflict.gointernal/controller/tenantgateway/crdcel_test.gointernal/controller/tenantgateway/reconciler.gointernal/controller/tenantgateway/reconciler_test.gointernal/controller/tenantgateway/renderers.gopackages/extra/gateway/README.mdpackages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
- internal/controller/tenantgateway/crdcel_test.go
- internal/controller/tenantgateway/conflict.go
- packages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml
- internal/controller/tenantgateway/reconciler.go
- packages/extra/gateway/README.md
- internal/controller/tenantgateway/reconciler_test.go
a467c51 to
578c355
Compare
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Verdict
NOT LGTM — one blocking item, and it is not in the code. The design is settled, the implementation matches it, and everything the previous two rounds raised is closed at this head. What is missing is the exit gate the accepted design wrote for this exact phase.
Reviewed at head 20abf8768, diffed against merge-base d3559d48e. A note on the review state before anything else: the standing CHANGES_REQUESTED is IvanHunters' from 2026-09-08 05:35Z and this head was pushed 2026-09-08 21:52Z, so the decision field predates the work that answers it. Its MAJOR — the wildcard listener name the Gateway CRD refuses — is addressed here.
Why this exists, and whether the mechanism fits
Taking the design question first, because that is the one an API owner is being asked to sign.
The problem is real and its shape is not this PR's to invent. design-proposals/external-database-exposure in cozystack/community is Accepted, revised 2026-09-03, and it specifies this field. The Go block in the proposal carries the same field names, the same types, Hostname required rather than optional, and the same four list-level markers — +listType=map, +listMapKey=name, MaxItems=62 — that this PR ships. So the API surface is not a new proposal arriving inside an implementation PR; it is the accepted one, and the earlier #2815 sketch with an optional defaulted hostname is superseded rather than departed from. The two passthrough fields staying independent is a recorded decision too: the proposal states that tlsPassthroughServices is too weak for this design and that neither field replaces the other.
The mechanism is proportionate. One mode: Passthrough listener per entry, SNI-matched, on the engine's native port, consuming O(engine types) of the 64-listener budget rather than O(instances). No new CRD, no new controller, no new cross-cutting abstraction — a field on an object that already exists, plus an upstream TLSRoute with no wrapper around it. For a feature whose entire point is that the Gateway must not hold the key, not terminating is the smallest thing that can work.
The breaking half is inside the approved scope as well, which is worth stating because the diff makes it look like scope creep. Lines 183 and 222 of the proposal describe the terminate-listener withdrawal explicitly, including that it is "a behaviour change that reaches clusters which never declare the new field at all".
The blocking item
The proposal's Rollout section gives Phase 1 an exit gate:
Gate: on the deployed Cilium (1.19.x), stand up a
mode: Passthroughlistener on a native database port (for example 5432) with a per-engine subdomain hostname (*.postgres.<apex>) alongside the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain.
That has not been done, and nothing in the PR can do it. The new chainsaw phases settle the object-level facts — the listener reaches Programmed, the withdrawal happens on demand, the route conditions read correctly — and the third phase reads the per-listener readiness flag, which is genuinely the part no fake client could answer. None of that is a handshake. No test in this change opens a TLS connection carrying a passthrough SNI and confirms which backend answers it, or how many filter chains carry the name.
The whole design rests on one datapath fact: that Cilium 1.19.5 folds the Gateway into a single Envoy listener whose filter chains match on SNI without the port. Three load-bearing claims descend from it — the certMode restriction, the README's statement that the declared port is not an access boundary, and the firewall advice in the field doc.
To be fair to the work: I had the reasoning checked independently against the v1.19.5 source rather than taking the PR's word for it, and all of it holds — the single listener (operator/pkg/model/translation/envoy_listener.go:207-239), toFilterChainMatch carrying no destination port (:589-600, shared by both the terminate and passthrough chain builders), the skip for a route with no backends (:408-411), and ComputeHosts substituting the listener hostname for a route that declares none (operator/pkg/model/helpers.go:39-44). So this is not a doubt about the reasoning. What is left is the empirical half — the half the accepted design chose to gate on, presumably because translation, xDS and the datapath have more steps in them than the translator source shows.
It matters more here than it would for an inert field, because the withdrawal reaches clusters that never touch the new field. Closing it is minutes on a disposable cluster: one entry, one TLSRoute, openssl s_client -connect <gw-ip>:443 -servername <hostname>, and a look at the Envoy config dump for the chain count. Given this is the breaking half of a breaking change, and given the gate is one you wrote, I would rather have it before merge than after.
A second one, and the direction of the change is right
Making a failed route-status write fail the reconcile is the correct instinct — logging it away was worse. But the new path does not tell a retryable write apart from one that can never succeed, and it sits ahead of two steps that do not depend on it.
updateRouteStatuses and reconcileWholeApexRouteStatuses now collect their write errors and return them (conflict.go:718, :730, :733), and runReconcileSteps returns on both (reconciler.go:667-672) — where the merge base logged and carried on (git show d3559d48e:internal/controller/tenantgateway/conflict.go, the logger.Error calls at :141 and :153, function returning nil). The two steps that never run after that return are reconcileHTTPToHTTPSRedirect (reconciler.go:672) and reconcileStatus (:677).
For a transient conflict that is fine — one failed pass, a requeue, done. The case that does not resolve is the ceiling: RouteStatus.Parents is MaxItems=32 and parentRefs is MaxItems=32 (gateway-api v1.4.1, apis/v1/shared_types.go:597 and :245), and both this controller and Cilium write one entry per parentRef. A single route carrying seventeen or more parentRefs to the shared Gateway therefore produces a status object the apiserver rejects as Invalid, every pass, forever. The Gateway and the certificates still reconcile — those steps run before the return — but the TenantGateway is pinned at Ready=False, and on a fresh install the http-to-https redirect route is never created at all, which is the one object that guard exists to provide.
The Gateway is shared across the inheriting subtree and the attached cozy-* namespaces, so the route that does this need not belong to whoever notices. Two small changes cover it: treat IsConflict/IsNotFound on a route status as skip-and-requeue rather than fatal, and move the status pass after reconcileHTTPToHTTPSRedirect so a status problem cannot withhold the redirect. Batching a route's refs into one read-modify-write would also cut the conflict rate, since updateRouteParentStatus (conflict.go:915-942) currently re-reads a cached object once per attaching parentRef. Unit tests cannot see any of this — the fake client has no cache lag and no second writer.
What the withdrawal actually changes
Two facts bound this, and the description does not lead with either.
gateway.enabled is false by default (packages/core/platform/values.yaml:479), and all three platform TLSRoutes render only under gateway-enabled=true with the name present in expose-services. So the withdrawal reaches clusters that opted into the Gateway integration, not the installed base.
And on the path those clusters came from, api.<apex> was already passed through. packages/system/cozystack-api/templates/api-ingress.yaml:10-12 sets ssl-passthrough: "true" with backend-protocol: HTTPS to Service kubernetes:443, rendered exactly when gateway-enabled != true, and packages/system/ingress-nginx/values.yaml:4 turns the feature on. The two KubeVirt proxies are the same shape. So this change does not invent passthrough for these names; it makes the Gateway path agree with the ingress-nginx path the platform has always shipped. Meanwhile packages/extra/info/templates/kubeconfig.yaml:21 embeds certificate-authority-data from kube-root-ca, so the kubeconfig Cozystack hands a tenant already validates against the cluster CA — passthrough is what that client expects. Before this change, which of the two port-443 filter chains answered was undetermined and unobservable, since the pinned Cilium has no Conflicted condition to read. Making it deterministic is the fix, not the risk. That framing belongs near the top of the description; right now a reader meets the shed before they meet the reason it is an improvement.
A precondition the release note should carry
Once the ACME certificate is gone, the only certificate on api.<apex> is the kube-apiserver's serving certificate, so it must carry api.<apex> in its certSANs. On the ingress-nginx path that was already true, so most clusters will satisfy it — but a cluster that moved to the Gateway path and has been served by the ACME certificate since will not, and it loses api.<apex> on upgrade. Nothing in this repository states the requirement; the e2e clusters do not meet it either (hack/e2e-container-up.sh:263 and hack/e2e-prepare-cluster.bats:239 list IP SANs only), which is a second reason the suite cannot reach the datapath question above.
Please add one line to the release note and to the README's upgrade section: before upgrading, confirm the management apiserver's serving certificate carries api.<apex>; if it does not, add the SAN, or take the migration path that moves the consumer to another hostname, which the README already names as the one that costs nothing.
While you are in that section — the second migration path reads as more than it delivers. "Switching certMode to dns01 or existingSecret keeps the pre-upgrade shape wholesale" is literally true, and the pre-upgrade shape is the ambiguity: rendersTLSPassthrough (reconciler.go:94-96) excludes edge alone, and both passthrough loops sit outside the certMode switch that ends at reconciler.go:1585, so under dns01 the tls-<svc> listeners still stand beside a wildcard terminate listener covering the same names. Someone picking that path to get termination back gets the undetermined race back instead. Worth saying so where the path is offered, since the README explains elsewhere that the shape is unobservable.
No migration ships with this change and I do not think one is needed given the above — but say that in the release note rather than leaving it inferred.
Notes, none of them blocking
The certMode wedge is real, already documented, and I agree with IvanHunters that it should not block. A stored tlsPassthroughListeners refuses the write that would move certMode, because the CEL rules sit on the spec node and any change there un-suppresses them, and packages/apps/tenant/templates/gateway.yaml:46,52 sets retries: -1, so the gateway release retries the refusal indefinitely. What keeps it out of blocker territory is that the CEL message names the field: the exit is one kubectl edit. The proposal already carries it as an open question under who declares the entry, which is the right place. Worth revisiting when the chart grows a value for the field — an admission rule that can reject a write the platform makes automatically, and that the tenant cannot influence, is the shape to avoid.
tlsRouteForwards (backends.go:54-80) reads Service existence and stops there, which is the right call and reads as deliberate. Worth noting the consequence for the two KubeVirt names: their Services are created by virt-operator and cdi-operator rather than by this tree, so on a cluster where one has not appeared yet the terminate listener simply stays. That is the safe direction.
The new chainsaw phases hand-roll poll() loops with sleep 3 where docs/agents/e2e-testing.md §2 and checklist items 2–3 call for a Chainsaw assert, which already polls to its own timeout and reports a structured diff on failure. The trap cleanup EXIT is fine — self-contained inside one script step, which checklist item 4 permits explicitly. Either move the presence and absence checks onto assert/error operations, or leave a comment saying why the imperative form is needed; the reasoning about not reading a failed kubectl as "absent" is good and belongs beside the code that needs it.
Two tests assert less than the text around them claims, and both are one assertion away from closing.
TestListTypeSetWouldNotRefuseAStoredDuplicate (crdcel_test.go:648-690) is cited for the decision not to put +listType=set on tlsPassthroughServices, and the argument for that decision is the interesting part: that the apiserver's list-type check is gated on the stored object passing it, at whole-object granularity, so one stored duplicate would switch the check off for every list on the TenantGateway at once. The test shows only that the marker refuses a duplicate over a clean stored object, and that a stored duplicate does not block an unrelated write. It never demonstrates the spillover, which is the whole point. I had it checked against the shipped schema and the spillover is real — a stored duplicate in tlsPassthroughServices lets a duplicate name key through in the listType=map list on the same write, guard at k8s.io/[email protected]/pkg/registry/customresource/strategy.go:299-302. The object already carries a listType=map list, so pinning it is one more assertion in a test that is already standing there.
TestCRDPassesInstallTimeValidation (crdcel_test.go:476-497) has no negative control, so it passes whether or not the bounds it exists to justify are doing anything. The claim itself holds — removing MaxLength from apex puts the estimate 3.9x over budget. But the field comment attributes the need to apex, and that undersells it: removing MaxItems from tlsPassthroughServices costs 6.5x and removing it from tlsPassthroughListeners costs over 100x plus the schema's total limit. Worth correcting the comment, since the next person to touch a bound will read it.
The new Service and ReferenceGrant watches map correctly and I found no staleness path — both branches match what Cilium resolves, and the ReferenceGrant branch deliberately over-enqueues rather than under-enqueues. One efficiency note: mapper.go:157-166 lists all TLSRoutes and all TenantGateways on every accepted event, and lists TenantGatewayList even when there are no TLSRoutes at all. serviceExistenceChanged() (mapper.go:126-130) overrides only UpdateFunc, so Creates pass — and controller-runtime replays the whole store as Creates when a handler registers on a running informer, which means every controller start and every leader change runs the mapper once per Service with two full lists each. It is all served from the manager cache rather than the apiserver, and it is tidier than what WorkloadMonitorReconciler already does, so this is not a blocker. An early return when len(routes.Items) == 0, and moving the TenantGateway list inside the match branch, costs nothing.
conflict.go:819 writes the literal "NoMatchingParent" where every other site uses the upstream constant. gatewayv1.RouteReasonNoMatchingParent exists — gateway-api v1.4.1, apis/v1/shared_types.go:374.
Not a finding, but worth recording since this PR touches the file: the two rules added to packages/system/cozystack-controller/templates/rbac.yaml grant nothing today, because the same ClusterRole already ends in apiGroups: ['*'], resources: ['*'], verbs: ["get","list","watch"] (:71-73). Naming the reads is the right thing to do — it is what makes narrowing that catch-all possible later, and rbac_test.go pins it. Narrowing it is separate work and worth scheduling: a cluster-wide read of everything, alongside cluster-wide write on Secrets, is a lot of standing authority for a platform controller in a multi-tenant product.
On IvanHunters' second MINOR: parentRefAttachesTo (reconciler.go:1019-1036) still does not read ref.Port, so the asymmetry he named is still there in that function. The consequence he predicted is not, because judgeHTTPRouteClaim (conflict.go:278-305) now judges the port and refuses anything that is neither the plain-HTTP nor the HTTPS port before it can reach dynHostnames. So the symptom is closed and the trap is not: a helper called parentRefAttachesTo that returns true for a route pinned to a port the Gateway does not publish there will mislead whoever calls it next. Worth a comment on the function at minimum.
Two things I could not settle myself and would rather name than assert. First, the withdrawal is refused for a TLSRoute reaching a native-port listener from a namespace that listener does not admit, so this controller keeps the terminate listener — but if Cilium's TLSRoute ingestion does not consult allowedRoutes and the permissive port-443 listener answering the same hostname satisfies its namespace check, it would build the passthrough chain regardless, and two chains carrying one server name in one Envoy listener is not a graceful failure. It needs a hand-written entry to reach, so nothing in a shipped configuration does it, but it is the same experiment as the Phase 1 gate and worth adding to that run. Second, updateRouteParentStatus merges on (ControllerName, ParentRef) while Cilium's merge keys on the ParentRef alone, and this controller writes ObservedGeneration: 0 where Cilium writes the route's generation — if that means the two rewrite each other's entry indefinitely it would make the conflicts in the section above routine rather than rare. Both predate this change; both are worth ten minutes on the cluster you use for the gate.
One dependency worth a line somewhere, because the rollback story rests on it: the "a re-created Certificate adopts a still-valid Secret rather than ordering" argument holds only while cert-manager runs with enableCertificateOwnerRef: false. It does — packages/system/cert-manager/charts/cert-manager/values.yaml:268, with no override anywhere under packages/ — but that is a chart default this design now depends on, and flipping it turns every backend flap into a fresh order against Let's Encrypt's duplicate-certificate limit. Say so where the rollback is described.
Things I checked and found clean, so nobody re-derives them: the wildcard rename preserves every previously-admissible listener and Certificate name byte for byte (the hash covers the whole hostname including the asterisk, and TestPerListenerName_KeepsEveryNameThatWasAlreadyAdmissible pins the old values rather than recomputing them); the generated CRD is byte-identical to what the pinned controller-gen produces from the markers; packages/core/platform/values.yaml really does change in comments only; and packages/extra/gateway is not a vendored chart, so the patches/*.diff rule does not apply here — cozyvalues-gen regenerates only the ## Parameters table, and all six README hunks sit outside it.
On the description, as a request rather than a finding: +11955/-2088 is misleading in a way that costs reviewers. The production surface is roughly 2,300 lines across six files; reconciler_test.go alone accounts for 9,611 of the rest. Saying that in the first paragraph would set a truer expectation. Relatedly, the body is long enough that its own conclusions are hard to find — #3842 already tracks collapsing the restated Cilium reasoning into one canonical block, and the same treatment would help the body itself. A reader who has to reach the fortieth paragraph to learn that the red check is deliberate is a reader who stopped before forty.
Process
The red Require API owner review for sizeable API changes is the gate working as designed, and no code change clears it: it needs an approving review from Timofei Larkin (@lllamnyp) or Andrei Kvapil (@kvaps) on the current head, and any later push invalidates it. Everything else on this head is green, including the full E2E lane at 2h2m — hack/select-e2e.sh:109 escalates api/, cmd/, internal/ and packages/core/ to the whole suite, so the gateway suite with the new withdrawal phases did run.
Close the Phase 1 gate, add the certSANs line, and correct the second migration path, and I have nothing else.
|
Two more from a second pass over The first is a pre-existing false conflict that this PR builds the machinery to fix and then declines to run. The second is new and small: the withdrawal reads namespace state one pass before it establishes it. While I am here, one correction to sourcing rather than to substance. The three Cilium behaviours I cited in the review — the single Envoy listener, |
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Approving, and being explicit about what that covers
Approving as API owner on 20abf8768, which clears Require API owner review for sizeable API changes. The release is what is driving the timing, and the remaining items below are being carried as follow-ups rather than treated as closed. I would rather write that down than let the approval imply they went away.
On the API surface itself I have no reservations, and did not have any in the previous review. design-proposals/external-database-exposure in cozystack/community is Accepted, revised 2026-09-03, and the Go block there matches this PR field for field — same names, same types, Hostname required, and the same four list-level markers (+listType=map, +listMapKey=name, MaxItems=62). The two passthrough fields staying independent is a recorded decision in that document too. So this is the approved shape arriving as an implementation, not a new proposal arriving inside one, and v1alpha1 plus no chart value writing the field keeps the cost of being wrong low.
On the Phase 1 exit gate: I understand from the maintainer that you did exercise this on your own cluster. That closes it on the merits and I withdraw it as a blocking item. The one thing I would still ask for is that the result goes into the thread or the body — what you stood up, what SNI you dialled, and the filter-chain count — because the design named it as the gate for this phase, and right now nothing in the PR records that it was met. The next person to touch the flat-hostname phase has to re-run the same check against a different shape, and starting from your notes is much cheaper than starting from the design text.
Carried as follow-ups
Listed in the order I would do them, not in the order they appear in the review above.
1. The fatal route-status write is a regression this PR introduces, and it is reachable by a tenant. updateRouteStatuses and reconcileWholeApexRouteStatuses now return their write errors (conflict.go:718, :730, :733) and runReconcileSteps returns on them (reconciler.go:667-672), where the merge base logged and carried on. Failing rather than swallowing is the right instinct; the gap is that a retryable conflict and a permanently invalid write are treated alike. RouteStatus.Parents is MaxItems=32 and both this controller and Cilium write one entry per parentRef, so a single route with seventeen or more parentRefs to the shared Gateway makes the status permanently unwritable, and from then on reconcileHTTPToHTTPSRedirect and reconcileStatus never run — on a fresh install the http-to-https redirect is then never created at all. The Gateway is shared across the inheriting subtree and the attached cozy-* namespaces, so whoever triggers it need not be whoever notices. Treating IsConflict/IsNotFound as skip-and-requeue and moving the status pass after the redirect covers it. This is the one I would not let sit long.
2. The certSANs precondition belongs in the release note. After the shed, the only certificate on api.<apex> is the kube-apiserver's serving certificate, so it must carry api.<apex> in its SANs. The ingress-nginx path already passed through, so most clusters satisfy it — but a cluster that moved to the Gateway path and has been served by the ACME certificate since does not, and loses api.<apex> on upgrade. Nothing in this repository states the requirement today.
3. The second migration path in the README promises more than it delivers. "Switching certMode to dns01 or existingSecret keeps the pre-upgrade shape wholesale" is literally true, and the pre-upgrade shape is the ambiguity: rendersTLSPassthrough excludes edge alone and both passthrough loops sit outside the certMode switch, so under dns01 the tls-<svc> listeners still stand beside a wildcard terminate listener covering the same names. Someone picking that path to get termination back gets the undetermined race back instead.
4. The two small ones from my follow-up comment — the recount gate that declines to fix the ACME-solver false conflict, and the withdrawal reading namespace state a pass before ensureNamespaceLabels writes it.
5. The two Cilium-source items I flagged as unsettled rather than asserted: whether toTLSRoutes consults allowedRoutes during ingestion, and whether mergeStatusConditions keying on the parentRef alone produces a status ping-pong with this controller. Both are cheap to answer on the cluster you already have standing.
Not merging
I am clearing the gate, not pressing the button — the merge decision is not mine to take here, and E2E Tests should be looked at on the current head before anyone does.
A tlsPassthroughServices entry publishes a hostname on port 443, which suits a service whose clients speak HTTP and mis-serves one whose clients expect a database on its own port. The new field takes a name, a native port and the SNI hostname to answer on it, so an engine can be reached where its drivers already look. The cross-field rules are enforced at admission as well as in the controller. renderGateway is the first thing that renders, so a spec the apiserver accepts and the renderer refuses aborts everything behind it, and one mistyped hostname would freeze the Gateway and every route status where they stand. The controller keeps its own copy because it and the CRD roll out separately. Rules with no admission form, and why each one has none, are named on the type. tlsPassthroughServices and apex gain bounds at the same time, for the CEL cost estimator rather than for their own sake: the new rules concatenate apex and scan the services list, and an unbounded string or list makes the install-time estimate exceed the per-CRD budget and the apiserver refuse the whole definition. Every bound is a kind the apiserver ratchets, so a stored value that violates one keeps accepting writes that leave it alone. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
20abf87 to
8af9863
Compare
|
Recording the follow-ups from the approval above as issues, so that "carried as a follow-up" means something a month from now rather than a sentence nobody can act on.
For the record, because the shape of this thread will otherwise be hard to read later: this PR carried a NOT LGTM from me yesterday, and the approval above is not that verdict being withdrawn on the merits. Every item in it is still open — none was answered by a code change, and the head has not moved since. What changed is the release timing, and the judgement that the API surface is the approved design and that the remaining items are better carried as tracked issues than as a held PR. The Phase 1 gate is the one exception: I understand from the maintainer that it was exercised on your own cluster, which closes it on the merits, and the only thing still missing there is the record of it. If any of the five turns out to be wrong on closer reading, say so on its issue and I will close it — I would rather have five issues where one is mistaken than four where the fifth was forgotten. |
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Re-approving on 8af986398
The branch was force-pushed while I was writing the previous approval, so that one sits on 20abf8768 and the gate correctly does not count it. Re-approving here, and the head moving turned out to be worth the round trip: this is not a rebase, it answers the review.
Closed by this push, verified in the delta rather than taken on trust:
The route-status regression is fixed the way I hoped and then some. retryableRouteWrite (conflict.go) classifies IsConflict and IsNotFound as survivable and everything else — an Invalid the apiserver will refuse on every pass, a Forbidden — as fatal, and it recurses into a joined error so one fatal leaf fails the whole batch instead of being retried into silence. routeStatusWriteError tags the failure so Reconcile can tell it from a desired-state failure without matching on message text, and a retryable one now requeues rather than pinning Ready=False on the tenant. reconcileHTTPToHTTPSRedirect moved ahead of the status pass and reconcileStatus now runs before the collected errors are returned, so a route status the apiserver refuses can no longer withhold the redirect or freeze the TenantGateway's own status. That was my one blocking-grade item and it is gone. Closing #4195.
The api.<apex> certSANs precondition is in the README now, ahead of the three migration paths and phrased as a pre-upgrade check. Closing #4196.
conflict.go uses gatewayv1.RouteReasonNoMatchingParent instead of the string literal, and parentRefAttachesTo carries the comment explaining why it answers identity alone and leaves the port to the serving verdict. Both were nits and both are done.
Still open, and still carried as follow-ups rather than as blockers:
- #4197 — the
dns01migration path still reads "keeps the pre-upgrade shape wholesale" with no note that the pre-upgrade shape is the ambiguity itself. - #4198 —
recountis still gated onlen(refused) > 0, andcollectHostnameClaimsstill runs at:208whileensureNamespaceLabelsruns at:673. - #4199 — the two Cilium v1.19.5 assumptions I flagged as read against the branch rather than the pinned tag.
Same framing as before, since the record should stay readable: this PR carried a NOT LGTM from me yesterday, and the approval is not that verdict collapsing on the merits. The API surface is the accepted design and was never the question; what moved is that the one item I would have held on is now fixed in the code, the Phase 1 gate is reported as exercised on your own cluster, and the rest is tracked. If any of the three remaining issues is wrong on closer reading, say so on it and I will close it.
Not merging — that is not my button here, and E2E Tests wants looking at on this head first.
|
Closed the gate on a live cluster, Cilium 1.19.5: terminate listeners on 443, a passthrough listener on 5432, one TLSRoute per name, backends presenting their own certificates. Cilium folds the whole Gateway into a single Envoy listener and builds exactly one filter chain per passthrough name, matching on Your first caveat does not reproduce. A TLSRoute reaching the native-port listener from a namespace that listener does not admit gets no filter chain at all, and the name goes on answering with the terminate certificate. Pinned to the listener it reads Both route-status changes are in. Retryable and permanent are now separated by error type rather than by message text, a retryable batch requeues without touching The precondition is in the README paragraph about what the upgrade sheds and in the release note. A cluster that came from ingress-nginx already meets it, since that path passes the same name through to the same Service; a cluster served by the Gateway's ACME certificate since it moved does not necessarily, and loses the name until the SAN is added. The second migration path now says what it costs instead of implying a clean rollback: under The smaller notes are all in. One number worth quoting since it changes which bound is the load-bearing one: removing each in turn, |
|
Gate report is what I was after, thanks. Checked the rest against the tree at Still open, both in Your first caveat not reproducing is the useful half of #4199: no filter chain built at all, and the name keeps answering with the terminate certificate. I narrowed the issue to the part you could not measure, your own controller writing route status against a live Cilium. The withdrawal case in the suite pins the rule rather than a neighbour: the TLSRoute it attaches has a backend that resolves, and after it the terminate hostname is gone from Approval stands on this head. |
Each tlsPassthroughListeners entry renders a mode=Passthrough listener on its own port, answering the entry's hostname by SNI and admitting TLSRoutes from the publishing tenant's namespace alone. It declares no allowedRoutes.kinds: at the pinned Cilium v1.19.5 a listener that spells its kinds out has them applied to every route on the Gateway rather than to itself, so naming TLSRoute here would reject every HTTPRoute on the Gateway. A hostname a passthrough listener answers no longer earns an HTTPS-terminate listener or a Gateway-issued certificate. That is the break, and it is keyed on service rather than on declaration: the same Cilium builds a passthrough filter chain per route out of the backends that survived resolution, so a listener no TLSRoute attaches to, and a route whose backendRefs resolve to nothing, both match no ClientHello and leave the terminate listener holding the name. Eligibility asks what v1.19.5 asks and no more, the reference allowed and the Service present, and fails closed on a read it cannot complete. Services and ReferenceGrants are watched, so a withdrawal deferred on a missing backend lands on the pass that sees it arrive, and the Service watch takes only the creates and deletes, existence being all that is read. Which hostname a route claims is answered the same way. spec.hostnames is optional on a TLSRoute, and ComputeHosts substitutes the hostname of each TLS listener the route's parentRefs select for a route that declares none, so such a route carries the SNI and takes the name over exactly as one that spells it out. A declared hostname broader than the listener's is that substitution from the other side: the route is served on the listener's name and not on the wildcard it wrote, so a claim covering a reserved name counts as a claim on it. The substitution reaches only the listeners that would admit the route. A native-port listener takes the publishing tenant's namespace alone, so Cilium never puts a route from elsewhere on it and never serves that route on its hostname; borrowing the name anyway refuses the route for a namespace, over a hostname its own object does not carry, in the same pass that sheds a terminate listener because that same route is serving a name it borrowed legitimately. A route that names the listener in its sectionName keeps the borrow, since there the refusal is the answer to what it asked for. A route the Gateway renders no listener for hears it from this controller instead of from Cilium alone: the hostname a passthrough listener took, the hostname nothing answers, the sectionName or port naming a listener that answers something else, the sectionName naming no rendered passthrough listener, and the native-port listener that admits the tenant alone, each with its own Gateway API reason and a message naming the hostnames that lost. An HTTPRoute pinning a rendered passthrough listener by sectionName is refused there too, since such a listener forwards a stream it never terminates and takes TLSRoute alone by protocol. Nothing else tells that route: the port-443 listeners name HTTPRoute in their kinds to keep every port-443 set uniform, which the Gateway reports back as InvalidRouteKinds rather than as a grant, and the pin reads a listener's kinds against every route on the Gateway, so it reports the route accepted. The field is confined to certMode http01, stated as the modes that can carry it so one added later is refused until someone rules on it. The wildcard modes cover the whole apex with one terminate listener the pinned Cilium cannot keep apart from a passthrough listener beneath it, and edge terminates TLS at the class provider and renders no TLS listener at all. An HTTPRoute claiming a wildcard is refused under http01 before the race and the overlap are read. The HTTP-01 challenge issues no wildcard certificate, so the terminate listener rendered for the claim could never carry the name, and the listener name built from the hostname's first label carried the asterisk, which the Gateway CRD admits in a hostname and refuses in a name, so the apiserver refused the whole Gateway write and every other listener on it froze where it stood. The renderer now writes that label as "wildcard", so every hostname the CRD admits yields a name it admits, and no name that was already admissible moves. That refusal is a statement about the terminate listener, so it is judged where that listener is decided rather than ahead of everything else. The port-80 listener carries no hostname and serves a wildcard claim as readily as any other, so a route that selects it, by pinning the port or by naming the listener, never needed the listener HTTP-01 cannot supply and is served rather than refused. Judged first, the refusal reached that route too and reported it unserved while Cilium was serving it in plaintext. A parentRef port on an HTTPRoute is read the way servableOn reads it for a TLSRoute. The terminate listener sits on 443, so a route pinning another port attaches to no listener the render would give it, and it used to earn a listener and an ACME certificate for a route that attached to nothing. Port 80 is the plain-HTTP listener, which admits the publishing tenant alone: a route from there pinning it is served plain and earns neither a listener nor a refusal. A route status write that fails is told apart by kind. A conflict or a route that vanished between the list and the write is a race between the two status writers this Gateway has: the reconcile requeues and retries the write without pinning Ready=False on the tenant, and the status pass runs after the http-to-https redirect, so such a race cannot withhold the one object that guard exists to provide. A write the apiserver will refuse on every pass still fails the reconcile, after every other route has been written, because the condition is the only object saying why a hostname is not served, and a pass that reported success over a lost write left the route reading Accepted=True until the resync. The retryable half is logged on the way out. Declining to pin Ready=False leaves the requeue as the only thing acting on it, and the requeue says nothing: the tenant goes on reading Ready, no event is recorded, and returning no error opts out of the backoff that would widen the interval, so a write failing on every pass retries at a fixed period indefinitely with nothing anywhere to show for it. A TLSRoute claiming a hostname broader than the listener's is pulled into the eligibility question for each name it covers, because the pinned Cilium serves it on the listener's own name. That is right only where the listener answering a name is named exactly that name. Under a wildcard entry the substitution hands such a route the wildcard, so its chain carries "*.db.<apex>" while the terminate chain carries the exact name beneath, and Envoy matches an exact server name ahead of a wildcard, so nothing collides; pulling the route in there withdrew a served endpoint and handed its name to the database behind the passthrough listener. The pull-in is now keyed on an exact listener match. Every condition this controller writes on an HTTPRoute, and the one it writes on a TLSRoute declaring no hostnames, is written under http01, and the whole-apex modes collect no claims, so a switch of certMode left those verdicts standing over listeners the new mode does not render. The whole-apex pass now drops this controller's entry from HTTPRoutes and from hostname-less TLSRoutes; the edge refusal stays confined to routes that declare hostnames, since a refusal written on a lent hostname would have no path back. Within http01 the same verdicts outlive their listeners for a second reason, and no mode change is needed to reach it. The status pass walks the routes that attached rather than the ones that claimed a hostname, and retracts its entry from the difference, because a claim set is not a route's own property once a hostname can be borrowed: a TLSRoute declaring none takes the hostname of every passthrough listener its parentRef selects, so removing that listener empties the claim set while the route stands unchanged, and an HTTPRoute reaches the same state by dropping its hostnames. Judged on claims alone such a route keeps whatever the last pass that did judge it left behind, describing a listener the Gateway no longer has. An HTTPRoute pinned by sectionName to a passthrough listener this Gateway does not render is refused with NoMatchingParent, the reading a TLSRoute in the same shape already got, instead of earning a terminate listener it could never attach to. A route served on the port-80 listener, by parentRef port or by naming it, earns no terminate listener and is refused nothing, but it publishes the name from its namespace all the same, so it stays in the hostname race. Every recount now ranks it with the field it recounts, because a recount that left it out crowned the namespace that had lost to it and dropped a loss the first count recorded: a third route refused in a third namespace, or a passthrough entry declared for the name, flipped the loser to Accepted=True. Selecting the port-80 listener by name is judged as selecting it by port: from a namespace the listener admits the route is served plain, from anywhere else the listener turns it away and the route reads NotAllowedByListeners, and a sectionName and a parentRef port that select different listeners select none and read NoMatchingParent, which is what Gateway API requires when both are given. Before this a route naming the listener from a refused namespace earned a terminate listener and a certificate it could never attach to. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Deciding whether a TLSRoute forwards anywhere reads the backend Service and the ReferenceGrant admitting a cross-namespace reference to it, and both are watched rather than fetched on demand. The read-everything rule at the foot of the ClusterRole covers them today, so nothing is broken. Naming them keeps the feature working the day that rule is narrowed, where an informer starting without a grant fails with Forbidden and takes the manager with it. The kubebuilder markers in the package generate nothing: controller-gen runs over ./api/... alone and writes no rbac artifact, so the shipped role is hand-written and only a test connects the two. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
Three phases on a tenant that owns its Gateway. The shipped passthrough entries render with nothing routed to them and an HTTPRoute on api.<apex> keeps its terminate listener, its certificate and Accepted=True, which is the state a withdrawal keyed on the declaration alone would take away. Attaching a TLSRoute to that listener, pointed at a Service in the tenant's own namespace so it resolves, withdraws all three and moves the route to NoMatchingListenerHostname. A third phase patches in a native-port entry and reads the per-listener readiness flag, which is the claim no test against a fake client can settle. The first two phases are one measurement rather than two: withdrawing on demand is what dates the earlier assertion to settled state instead of to a withdrawal that had not arrived yet. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
The chart README gains the native-port field, the three limitations it carries and where each comes from, the one that is a phase-1 narrowing rather than a consequence of the Cilium pin, and the three ways forward for anything that depended on a terminate listener the change takes away. The withdrawal is stated as what it is, a change of serving path rather than the removal of something idle, because on the pinned version both filter chains carry the SNI today and nothing on the objects says which one answers. The layers section is brought back in step with the listeners: which label each listener class selects on, that the attach set is the whole inheriting tenant subtree rather than the tenant plus the admin list, and the route conditions an operator reads, including the one that says the sectionName itself names nothing. The platform values comment on gateway.enabled said every exposed service renders an HTTPRoute, which the chart value the withdrawal reads contradicts. It now separates them: the services whose TLS the Gateway terminates get an HTTPRoute, while api, vm-exportproxy and cdi-uploadproxy get a TLSRoute pinned to their passthrough listener, one each from the packages that own those services. The overlap paragraph no longer describes an HTTPRoute wildcard as keeping its terminate listener. Such a claim is refused under HTTP-01, which issues no wildcard certificate, and the paragraph says so and what the render used to do with it. The route-condition list gains that shape and the port shapes an HTTPRoute now reaches, and the per-app naming bullets carry the hash suffix the rendered names have. Two sentences that were false against the code are corrected: a TLSRoute on a wildcard entry itself leaves the names beneath it their terminate listeners, and an unusable apex is refused by the containment rule at admission before the controller's own check is reached. The edge section says what happens to the entries the HTTP-01 pass wrote once the mode moves. The NoMatchingParent sentence now separates the two kinds: an HTTPRoute pinned to an absent passthrough section is refused whatever answers its hostname, while a TLSRoute is judged where a passthrough listener answers it. A sectionName outside the passthrough prefix is written Accepted like any attached route rather than left without a verdict, and the text says so; the port-80 listener's refusal joins the NotAllowedByListeners shapes and the name-and-port disagreement the NoMatchingParent ones. The attachedNamespaces comment described the list as reaching every listener, which the native-port ones do not: they select on kubernetes.io/metadata.name and name the publishing tenant alone, so a namespace listed there cannot attach to a database port. It also credited the wrong label with keeping the list honest; what a listed entry gets is the namespace.cozystack.io/gateway label, and what keeps a tenant from stamping it on itself is that no tenant role carries patch on Namespace. The class reference is brought along. Its certificate-mode table promised one Certificate per published hostname under http01, which a hostname served by a passthrough listener no longer earns, and its edge table said every attached TLSRoute is refused, where only a route declaring hostnames is and the rest have this controller's entry removed instead. The refusal-shape list is corrected against the code, without a count of its own so the next correction does not have to move one. A route the plain-HTTP listener turns away on its namespace selector was listed under NoMatchingListenerHostname as well as under NotAllowedByListeners; the controller reports the second. The port shape standing beside it swept port 80 back in by naming every port but 443, where a route selecting the plain-HTTP listener is answered by the namespace shapes and never reaches the port refusal at all. And a TLSRoute pinned to an absent section is said to carry NoMatchingListenerHostname where no passthrough listener answers its hostname, which holds only while no HTTPRoute claims that hostname: with one there a terminate listener stands on the name, and the route is judged like any other attached route. The wildcard shape gains the same qualifier. A wildcard claim is refused because no terminate listener can carry a certificate HTTP-01 will not issue, so the refusal reaches a route that wanted a terminate listener and not one that asked for the plain-HTTP listener, which carries no hostname and serves the claim there. The Helm merge paragraph named the two-way path as what happens to a custom resource Helm has no Go type for. That path is one of two in createPatch, the version deciding which runs, and the version comes from helm-controller rather than from this tree, so the claim was not checkable where it was written. It now names both and says the outcome it rests on holds either way. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
|
Head is A TLSRoute with no Two smaller ones went in with them. A route that attached and then stopped claiming any hostname kept its last verdict forever, since the status pass walked only what claimed; it walks what attached now. And the retryable status-write path had lost its log line, so a write failing every pass retried at a fixed interval with nothing anywhere to show for it. Four sentences this branch adds still misstate the code and I left them in rather than spend another round: the dns01 refusal says one apex listener where two render, the README names the wrong source for |
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
LGTM as of ae5b626c8.
Read the two refusals you pulled back, and both fixes are the narrow ones.
tlsRouteClaims now skips a tenantOnly listener when the route names no section and lives outside the tenant namespace, and keeps the borrow when the route did name it. That second half is the part worth having: a route that asked for the listener still gets an answer to what it asked, instead of falling out with no verdict at all.
The wildcard refusal moved into the !hasPort || port == httpsListenerPort branch, so a route pinned to port 80 or to sectionName: http reaches the plain-served branch first and is not refused for a certificate it never needed. I went looking for a third false refusal the move could have created and did not find one: a wildcard with no port and no section still lands on the terminate branch and is refused, and a wildcard on some other port still reads the port mismatch.
Walking attached instead of what claimed, and retracting the entry for a ref that attached and claims nothing, is right for the same reason. Claiming nothing is not stable, and a stale verdict over a listener set that no longer exists is worse than saying nothing. Both new tests fail on a revert of their own change, and go test ./internal/controller/tenantgateway/... is green here.
The log line on the retryable path matters more than it looks: a nil error opts out of the backoff, so a write failing every pass would have retried at a fixed period with nothing recorded anywhere.
On the five statements in #4207, the godoc one is the one I would rather see fixed in this PR than after it. "Nothing is routed until a TLSRoute attaches to it by sectionName tls-<name>" is not what the code does, since a hostname-less route with no sectionName attaches to every listener it may attach to, and that sentence ships into the CRD description, so it is what kubectl explain prints on every cluster. If you push again for any other reason, fold it in. The other four are internal prose and I am fine with them riding #4207.
Still open from my side, and neither blocks: #4198 for the recount gate and the namespace-label ordering, #4199 for this controller's status writer against a live Cilium, which your run could only half answer.
Approval is on ae5b626c8 alone, and the gate counts it only on that commit.
|
Merged at Every item from your round on The red check is Five statements the branch adds still misstate the code, listed in the comment above and tracked as #4207, one of which ships into the CRD description. If you would rather any of them had been fixed before merge rather than after, say so and I will treat that as the rule next time. |
What this PR does
Phase 1 of external database exposure via Gateway API TLS-passthrough (#2815): adds a
tlsPassthroughListenersfield toTenantGatewayand renders one layer-4mode: Passthroughlistener per entry, on a native database port with a per-engine subdomain hostname, alongside the Gateway's terminate listeners. No engine is wired: no TLSRoute, no certificate or CA changes. The existing layer-7tlsPassthroughServicesfield gains a 62-entry count bound, an item length and a DNS-1123 item pattern, and a repeated entry now fails the reconcile withReady=Falseinstead of rendering the same listener twice.spec.apexgains a 253-character bound and nothing else: the CEL rules concatenate it, so the install-time cost estimator refuses the CRD without a declared bound. It gains no format pattern, and that is not a claim that a mixed-case host works. Such a host reaches the field verbatim from a namespace label, the renderer then composes listener hostnames Gateway API refuses, and that tenant has no Gateway either way; what a pattern would change is only that the write is refused, which fails this HelmRelease where the same fault costs the one Gateway. An apex no listener hostname can sit within is reported by the controller instead, and only whentlsPassthroughListenersis in use. The constraints and what they do to objects that already exist are described further down. The bound on an entry is what the schema can state on its own. An entry also becomes a listener name and a hostname built from the apex, so the controller checks both composed values against the 253 characters Gateway API allows and names whichever one overflows.Validation rejects duplicate listener names or ports, ports reserved by the platform Gateway (80 and 443), out-of-range ports, invalid DNS-1123 names, and hostnames outside the tenant apex (mirroring the Gateway hostname admission policy so an invalid entry cannot make the whole Gateway rejected).
The SNI-overlap rule refuses two passthrough listeners that answer the same name on different ports. That is what cilium#42898 reports: only one of the two routes ever works. It does not extend to a terminate listener and a passthrough listener sharing a name across ports. No failure is reported for that shape, and Cilium counts a terminate/passthrough pair as conflicting only when they share a port.
The per-engine subdomain hostname scheme follows the phase-1 gate. The flat-hostname alternative (pending the Cilium listener-isolation fix targeting 1.20) is a later scheduling decision that does not change this API field.
One shape differs from #2815 on purpose. The ticket has
hostnameoptional with a*.<name>.<apex>default; here it is required. Deriving the SNI from the listener name hides the one value an operator has to get right, and the two drift apart the moment the naming scheme changes.Found while checking that pairing, unrelated to this field: a stock
dns01gateway already renderstls-api(passthrough, 443) besidehttps(terminate, 443,*.<apex>), one port with overlapping SNI, which is the shape Cilium 1.19.6 marksConflicted. That belongs to the Cilium bump and is recorded on #3343.Also named rather than fixed: the controller-side rules on this field stop the whole reconcile when they refuse a spec.
renderGatewayreturns an error, so the Gateway is not updated at all, and the tenant seesReady=FalsewithReconcileErrorwhile new app listeners and every certificate the controller would issue or amend wait behind it. Certificates already issued go on renewing, since cert-manager drives that from the objects already in the cluster and an aborted reconcile writes nothing to them. Dropping only the conflicting listener and reporting it on status would be the better shape, but that changes the error model of the whole renderer rather than one rule, so it is out of scope here. The current behaviour is at least observable:Ready=Falseis a signal, not silent drift.On HTTP-01, a hostname a passthrough listener answers earns no HTTPS-terminate listener once a TLSRoute claiming that hostname can be served on the listener, matched by SNI overlap so a wildcard entry reaches a name beneath it as readily as the name written in the spec, and the field itself is accepted under certMode http01 alone. That refusal is stated as the set of modes that can carry a passthrough listener rather than the set that cannot, because a mode added to the enum would otherwise be admitted by default: dns01 and existingSecret serve the whole apex from one wildcard terminate listener the pinned Cilium cannot keep apart from a passthrough listener beneath it, and edge ends TLS at the class provider and renders no TLS listener for one to sit beside. The route half of that condition is load-bearing rather than incidental. On the pinned v1.19.5, tlsPassthroughFilterChains walks a listener's routes and skips one with no backends, so a declared passthrough listener nothing attaches to contributes no filter chain and matches no ClientHello, while the terminate listener's chain is built from its own Secret and carries the name alone. Keying the withdrawal on the declaration would therefore take a name that is being served and hand it to a listener that forwards nowhere, and that is the ordinary shape rather than a corner: the chart puts api, vm-exportproxy and cdi-uploadproxy on every tenant Gateway, while there is one platform TLSRoute per name cluster-wide and each names one tenant's Gateway as its parent, so every other tenant renders all three entries with a route on none. The backend half is that same rule reached by a different road: the chain is built per route out of the backends that survived resolution, so a route that attached and forwards nowhere puts no chain on the name either, a backendRef naming a Service that does not exist yet because manifests applied out of order, one deleted since, a name with a typo, or one in another namespace with no ReferenceGrant, and without that half an ordinary backend mistake takes a live endpoint offline while Cilium builds nothing to replace it. Resolution asks what v1.19.5 asks and no more: the reference allowed, the Service present, and neither its ports nor its endpoints read, because the translator reads neither. Services and ReferenceGrants are watched alongside routes, so both halves are deferrals rather than exemptions and the withdrawal lands on the pass that sees the missing piece arrive. The withdrawal reaches HTTP-01 alone, because
collectHostnameClaimsreturns nothing under the other two modes and they render no per-hostname listener to withdraw. A TLSRoute that declares nospec.hostnamesclaims the hostname of each rendered passthrough listener its parentRef selects, because that is the name Cilium serves it on:ComputeHostsat v1.19.5 substitutes the listener hostname for a route that names none, so such a route takes the name over exactly as one that spells it out does. The tenant-namespace hostname policy requires a route there to name its hostnames, so this shape reaches a Gateway from the system namespaces where the platform routes live, or from an administrator, not from a tenant chart. A claim broader than the listener's counts the same way from the other side: a TLSRoute claiming*.<apex>is served onapi.<apex>rather than on the wildcard, becauseComputeHostsreturns the listener's own hostname for a route hostname covering it, so a claim covering a reserved name is judged as a claim on that name while the route keeps one entry under the key its owner wrote. Both restrictions are tied to the Cilium pin, currently v1.19.5. That version translates the Gateway into a single Envoy listener whose filter chains match on SNI without the port, so a terminate chain and a passthrough chain carrying one name collide regardless of which Gateway port declared them; upstream states the consequence above NeedsCrossProtocolSplit in operator/pkg/model/model.go as of v1.19.6, the release that starts splitting Envoy listeners per port when NeedsPerPortListeners holds. That is a disjunction rather than a blanket change, and the passthrough side of every one of its three clauses is counted from attached routes, so a listener with no TLSRoute behind it contributes to none of them: the split arrives with the route, not with the bump.The same single-listener translation has a consequence that is not a restriction: the declared port is where the listener is published, not a bound on reach, because the backend answers its hostname's SNI on every port the Gateway exposes. Firewalling a tenant address down to 80 and 443 does not put a database out of reach; the hostname is the access surface, and the field's own doc and the chart README now say so.
Both restrictions are validation-only, and they lift together on the pin plus a configuration the split actually covers rather than on the bump alone.
A route whose hostname is withdrawn this way is told so under this controller's own name: Accepted=False with
NoMatchingListenerHostname, and a message naming the passthrough hostname that answers it, which for a wildcard entry the route cannot derive on its own. It matters most there. A wildcard entry can withdraw the listener from several published names beneath it at once, each one a TLSRoute claims and can be served on, and the certificates are garbage-collected with them, and a rollback is cheaper than that sounds: cert-manager here runs withenableCertificateOwnerRef: false, so deleting a Certificate leaves its Secret behind and a re-created Certificate adopts a still-valid one rather than placing an order. Before this the only signal was Cilium's own RouteParentStatus entry under a different controllerName. A TLSRoute whose hostname no passthrough listener answers is told so as well, on the same condition and for the same reason: before this change that hostname got a terminate listener Cilium refused with ResolvedRefs=False, so withdrawing the listener would otherwise have removed the only object saying the route went nowhere.Two things stay named rather than fixed. A
tlsPassthroughListenershostname of*.<apex>is refused only indirectly: the overlap rule rejects it whiletlsPassthroughServicescarries its defaults, because the wildcard covers api., so emptying that list is enough to let it through, and the terminate listener then goes from every published name a TLSRoute claims under the wildcard. The README states this; refusing the apex and its wildcard on their own terms is a one-line rule worth adding when the field is wired up to chart values.A wildcard hostname on an attached HTTPRoute used to render a listener name the shipped Gateway CRD refuses,
https-*-<hash>, which failed the whole Gateway write while the README said the shape worked. Listener and Certificate names now spell a leading*aswildcard, with the hash still covering the whole hostname, and a test reads both the listener-name and the hostname patterns off the shipped CRD bundle and renders wildcards through them; every name that was already admissible is pinned byte for byte, so nothing is renamed on upgrade. Under http01 an HTTPRoute wildcard claim is refused before the race and the overlap are read, withNoMatchingListenerHostname, because the HTTP-01 challenge issues no wildcard certificate and a terminate listener could never have carried the name.A route status write that fails no longer leaves the route carrying its previous condition:
updateRouteStatusesand the whole-apex pass collect every failed write and return them after the loop, so the reconcile fails, the TenantGateway reportsReady=FalsewithReconcileErrorand controller-runtime requeues, while the loop still runs to the end so one route's conflict does not hold the others back. Conditions this controller wrote under http01 are retracted when the tenant moves to dns01, existingSecret or edge, for HTTPRoutes and hostname-less TLSRoutes alike, so a refusal does not outlive the mode that produced it. And a TLSRoute claiming a wildcard passthrough entry itself leaves the names beneath the wildcard alone, since Envoy ranks an exact server name above the wildcard and nothing collides.One thing this does not cover yet: Accepted is computed from the hostname race and the withdrawals above, so a route is still told it is fine when an HTTPRoute holds the name and the TLSRoute claiming it cannot attach to the terminate listener that answers it. That one predates this change. Two neighbours of it no longer fall in the gap. A
parentRefport that names no listener answering the hostname is compared alongside thesectionName, since Gateway API requires every key a route pins itself with to match the listener it selects. An HTTPRoute'sparentRef.portis judged the same way: the terminate listener sits on 443, so any other pinned port earns no listener and no certificate through that parentRef and reads the port-mismatch cause; port 80, orsectionName: http, from a namespace the plain-HTTP listener admits (the publishing tenant and cozy-cert-manager) is served plain with no terminate listener and no refusal, from any other namespace it is refused withNotAllowedByListeners, and asectionNameand a port that select different listeners select none and readNoMatchingParent. A plain-served route stays in the hostname race on the same footing as a terminating one, in every recount as well as the first count. And asectionNamenaming atls-<name>listener this Gateway does not render now reportsAccepted=FalsewithNoMatchingParent, Gateway API's reason for a parentRef matching no listener: the controller renders every listener in that namespace, so an absent name there is a fact rather than a guess, while asectionNamenaming the plain-HTTP listener is judged like port 80 below, and any other name outside the prefix is written Accepted like an attached route, with the class controller's own status entry answering for the listener it named.The namespace case is off that list. Native-port listeners admit the publishing tenant alone, and a TLSRoute elsewhere on one of their hostnames is refused with
NotAllowedByListeners. An HTTPRoute that pins itself bysectionNameto a renderedtls-<name>passthrough listener is refused with the same reason: that listener forwards the stream it matched by SNI and terminates nothing, so it takes TLSRoute alone by protocol and cannot serve the route whatever answers its hostname. Its claim also leaves the hostname race before ownership is ranked, so the TLSRoute that can be served is not told it lost to a route that could never attach. Cilium does not report this refusal at the pin. The port-443 passthrough listeners name HTTPRoute inallowedRoutes.kindsto keep the port-443 set uniform (cilium#45559), the Gateway reports that entry back asResolvedRefs=False/InvalidRouteKindson the listener, and Cilium's route-side kind check reads a listener's kinds against every route on the Gateway rather than against the routes that named it, so it reports the route Accepted and the condition this controller writes is the only refusal the route gets. A hostname whose every claimant is refused this way earns no HTTPS-terminate listener and no certificate, where before it earned both. The refusal is decided per route rather than per hostname, which matters once a claim overlaps more than one reserved name: a wildcard covering both a port-443 service hostname and a native-port one is attachable through the first and refused through the second, so the answer comes from the listener theparentRefnames, its own hostname and its own attach set. The row that pinned asectionNamenaming no rendered listener as a known gap now pins the verdict instead. On the port-443 listeners the two namespace sets agree by construction, sincecollectHostnameClaimsand the listeners' allowedRoutes selector run off the same list.And the justification for the pin is written out at every site that depends on it rather than once, so the day the pin moves they all have to move together and the one that does not is a false statement no diff can catch. That is not hypothetical here: three claims of exactly that shape were wrong in this branch and were caught by reading the source they described rather than by any diff, so the volume of restated reasoning is itself what produces them. One canonical block with references to it would carry the same knowledge with one place to update, and rewriting the narrative under review would be a documentation change rather than a fix, so it is tracked on #3842 instead.
The gateway chainsaw suite now covers the withdrawal, which it did not before: a tenant owning its own Gateway renders the shipped passthrough entries with nothing routed to them, an HTTPRoute on api. keeps its terminate listener, its certificate and Accepted=True there, and attaching a TLSRoute to the same listener, pointed at a Service in the tenant's own namespace, since a route that forwards nowhere is the case the first phase already measures, withdraws all three and moves the route to NoMatchingListenerHostname. A third phase patches in a native-port entry and reads the per-listener readiness flag, which is the claim no test against a fake client can settle. The first two phases are one measurement: withdrawing on demand is what dates the earlier assertion to settled state rather than to a withdrawal that had not arrived yet.
The
Require API owner review for sizeable API changescheck is red on purpose and does not clear with a code change. A new field plus tightened validation on already-shipped fields counts as a sizeable API change, so.github/workflows/api-review-gate.yamlholds the PR until an API owner approves it. The gate counts each owner's latest review on the current head commit, so the approval has to land on the head as pushed and any later push invalidates it; the job re-runs on review submission, so no push is needed to flip it. Reading that red as "not ready" is the mistake this paragraph exists to prevent.Part of #2815.
Release note
Downstream repositories
Walked the trigger map in
docs/agents/contributing.mdagainst this diff:api/gateway/v1alpha1/{tenantgateway_types.go,zz_generated.deepcopy.go},internal/controller/tenantgateway/{reconciler,renderers,conflict,mapper,status,backends,reconciler_test,crdcel_test,mapper_test,backends_test}.go,internal/controller/tenantgateway/rbac_test.go,cmd/cozystack-controller/main.go,hack/e2e-chainsaw/gateway/chainsaw-test.yaml,packages/system/cozystack-controller/definitions/gateway.cozystack.io_tenantgateways.yaml,packages/system/cozystack-controller/templates/rbac.yaml,packages/core/platform/values.yaml, andpackages/extra/gateway/README.md. No trigger fires, and the two repositories worth ruling out explicitly are below rather than waved off.cozystack/website: no package underpackages/apps/orpackages/extra/is added, renamed or removed.packages/extra/gateway/README.mdis edited in place. The docs generator regenerates a reference page from a package README only for the apps hardcoded in the websiteMakefile, andgatewayis in none ofAPPS,K8S,VMS,NETWORKINGorSERVICES, so there is no generated page for this package to drift out of step with. That gap is the same shape the trigger map already records forpackages/apps/opensearchandpackages/apps/bucket; it predates this change and is neither created nor widened here. None of the other website triggers apply:packages/core/platform/values.yamlchanges only in a comment, which now says that api, vm-exportproxy and cdi-uploadproxy render a TLSRoute rather than an HTTPRoute, with no key added, moved or given a new default, no platform variant or component moves, no release asset is renamed,ApplicationDefinitionsemantics are unchanged, there is no Talos bump and no developer-tooling change.cozystack/terraform-provider-cozystack: this adds an API field, so the field-level triggers are the ones to check, and they are keyed to app charts rather than to this CRD. No app is added underpackages/apps/; novalues.schema.jsongains, loses or renames a field; no version enum,values.yamldefault,kind,plural,release.prefixor output Secret/Service name changes.TenantGatewayis not among the CRDs the provider types by hand. Those arePackageundercozystack.ioandPlan/RestoreJobunderbackups.cozystack.io, and the provider carries no reference togateway.cozystack.ioat all. It therefore falls under the map's last bullet: the spec travels as opaque JSON, so the provider keeps working and simply offers no typed access totlsPassthroughListeners, which is the position it already holds for every other field it does not model, and not a break. The provider's new-top-level-field test guard is scoped toapi/apps/v1alpha1, which this diff does not touch.cozystack/ansible-cozystack: nothing in reach.packages/core/installer/values.yamland thecozy-system/PSA contract are untouched, no platform variant is added or renamed, none of thepackages/core/platform/values.yamlkeys the role sets are involved (the file changes in a comment alone), no object the role waits on is renamed, andhack/e2e-prepare-cluster.batsnode prerequisites are unchanged.The schema constraints this diff adds to already-shipped fields all reach objects that already exist, and every one of them is a kind the apiserver ratchets.
spec.apexgains a 253-character limit (no pattern: a mixed-case apex is admitted and refused by the controller only when passthrough listeners are declared), andspec.tlsPassthroughServicesgains a DNS-1123 item pattern and a length bound. A stored value that violates one still passes any write that leaves it unchanged, so the rule bites only once the value itself changes. A repeatedtlsPassthroughServicesentry is reported by the controller asReady=Falserather than refused at admission, and the reason is not the one this section used to give. Ratcheting does reach a list-type error:customresource.ValidateUpdatevalidates an incoming object's list-type semantics only when the stored object passes them, so a stored duplicate is ratcheted through rather than blocking every write. What makes the marker the wrong tool is the grain of that ratchet, which is the whole object rather than the field, so one stored duplicate would switch list-type checking off for every list on theTenantGatewayat once and say nothing about it. Documented inpackages/extra/gateway/README.md, and pinned byTestTightenedConstraintsOnExistingObjectstogether withTestListTypeSetWouldNotRefuseAStoredDuplicate, which compiles a copy of the shipped schema carrying the marker to show what the apiserver actually does with one. Shipped defaults conform and the chart already fails on an empty host, so this is a no-op in practice.Summary by CodeRabbit
tlsPassthroughListeners) with native port + SNI hostname, rendered as passthrough listeners and limited to TLSRoute attachment.