Repository navigation
Cilium v1.19.6+ marks the default TenantGateway port-443 terminate/passthrough pair ProtocolConflict; the Cilium bump is blocked on a listener layout decision #3912
Description
Summary
Cilium v1.19.6 adds a listener conflict check, and the default TenantGateway layout trips it. Every tenant Gateway with the shipped tlsPassthroughServices (api, vm-exportproxy, cdi-uploadproxy) gets both listeners of each terminate/passthrough pair on port 443 marked Conflicted=True/ProtocolConflict and Accepted=False. On Cilium 1.19.x the datapath keeps working, but the TenantGateway goes Ready=False, the tenant gateway HelmRelease waits on that condition, and every tenant install or upgrade runs into the 10m timeout and a rollback loop. #3343 (the bump) is parked in draft until the layout question is settled. Upstream treats the layout as invalid, so the decision is ours, and this issue exists to hold it. The question at the bottom is open.
Mechanism
samePortCrossProtocolConflictedListeners in operator/pkg/gateway-api/gateway_reconcile.go exists on v1.19.6 and v1.19.7 and not on v1.19.5. It pairs listeners that share a port where one is HTTPS/Terminate, the other TLS/Passthrough, and SNIHostnamesIntersect holds on their hostnames. Both listeners of the pair get Conflicted=True with reason ProtocolConflict and isValid=false, which is Accepted=False. SNIHostnamesIntersect (operator/pkg/gateway-api/helpers/gateway.go) matches *.<apex> against <svc>.<apex> at any label depth, so moving passthrough names to a deeper subdomain does not escape it; only a different domain does.
internal/controller/tenantgateway/reconciler.go renders exactly that pair. With certMode: dns01 or existingSecret, the https listener is *.<apex> on 443 next to one tls-<svc> listener per passthrough service on <svc>.<apex>, also on 443. With certMode: http01 (the default), the per-hostname HTTPS listeners are built from HTTPRoute and TLSRoute hostname claims alike, so api.<apex> gets an HTTPS listener next to tls-api: same port, identical hostname. #3342 removes the http01 half by not rendering a terminate listener for passthrough names. The dns01/existingSecret half stays.
Check, no cluster needed:
for t in v1.19.5 v1.19.6 v1.19.7; do
printf "%s: " "$t"
curl -sL "https://raw.githubusercontent.com/cilium/cilium/$t/operator/pkg/gateway-api/gateway_reconcile.go" |
grep -c samePortCrossProtocolConflictedListeners
done # v1.19.5: 0, v1.19.6: 2, v1.19.7: 2What breaks, by version
On 1.19.6 and 1.19.7 the damage is status only. The reconciler builds the Envoy model from the whole Gateway (ingestion.Input{Gateway: *gw}) before setListenerStatus runs, so conflicted listeners are still programmed into Envoy. That is cilium/cilium#46917, fixed by cilium/cilium#47306 and cilium/cilium#47457, both labelled backport-done/1.20 and nothing for 1.19. The wildcard-vs-exact pair keeps being resolved by filter-chain specificity as on 1.19.5. The identical-SNI http01 pair from #3720 wedges the Envoy listener on 1.19.5 and 1.19.6 alike, for a different reason (duplicate filter chain match).
Status is not cosmetic here. internal/controller/tenantgateway/status.go sets Ready=False/ListenersNotReady when any listener is not both Accepted and Programmed. The tenant gateway HelmRelease renders that single CR, and helm-controller's kstatus wait treats Ready=False as in-progress, so helm upgrade runs to the 10m timeout and the rollback waits on the same CR again. Deleting such a tenant stalls on the same wait.
On 1.20 the conflicted listeners are dropped from Envoy, so the same layout stops serving *.<apex> HTTPS and the passthrough names outright. 1.19.6 is the soft version of the 1.20 failure, and 1.20 is where the platform has to be eventually.
The default TenantGateway stays a single Envoy listener on 1.19.6. NeedsPerPortListeners is false because every HTTPS and passthrough listener sits on 443 (TLSPassthroughPorts() returns unique ports), and NeedsCrossProtocolSplit skips equal ports by design. Nothing in the bump separates the pair on its own.
Upstream position
cilium/cilium#47590 reports exactly this layout, wildcard HTTPS/Terminate plus exact TLS/Passthrough on 443 with both listeners marked ProtocolConflict. It was closed as kind/question after the reporter withdrew the regression claim. The answer there is that the layout is invalid, and the suggested alternatives are two Gateways with different addresses, or terminating at the Gateway and reaching the backend over BackendTLSPolicy. The check arrived with the backport of cilium/cilium#44889, which isolates listeners per port and rejects the same-port case it cannot split. There is nothing to take upstream.
Options
Each keeps the bump unblocked at a different price. Contract cost is spelled out per option so the choice is made on that, not on diff size.
- Drop the wildcard
httpslistener indns01/existingSecretand render per-hostname HTTPS listeners from the claims, as http01 already does, minus the passthrough names. Cost: a hostname under the apex with no route claim stops terminating; the wildcard certificate only covers claimed names. - Move the passthrough listeners off 443 to their native ports (
apion 6443 and so on). Cost:api.<apex>:443stops being the tenant Kubernetes API endpoint; kubeconfigs, docs and anything that hardcodes 443 change. feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342's native-port listeners are the building block, and with two passthrough portsNeedsPerPortListenerssplits Envoy per port on 1.19.6+. - Give the passthrough services their own TenantGateway with a separate address, which is what upstream suggests. Cost: one more LoadBalancer address per tenant, and gateway: a second tenant Gateway steals the platform namespaces from the first #3869 (two tenant Gateways fighting over the platform namespaces) gets a second instance per tenant by construction.
- Terminate at the Gateway and reach the tenant API over
BackendTLSPolicy. Cost: clients see the Gateway's certificate instead of the tenant control plane's, and client-certificate auth to the tenant API stops working through the Gateway. - Keep the pair and change
status.goso aProtocolConflicton a terminate/passthrough pair does not failReady. Cheapest and fully internal. It is honest only on 1.19.x; on 1.20 it would report Ready on a Gateway that does not serve, so it buys time rather than settling anything. - Hold at 1.19.5. Keeps dashboard.xxx.xxx HTTPRoute
.status.parentsreportsHTTPRoute is not allowed to attach to this Gateway due to namespace selector restrictions#3721 (Gateway API: allowedRoutes is incorrectly applied to all listeners when sectionName is omitted cilium/cilium#42159, the motivating fix of the bump) unfixed, unless the cozystack-side workaround of rendering the port-80 listener last is taken instead.
Open question for this thread: which contract change do we accept, and is 5 acceptable as a bridge while one of 1 to 4 lands.
Blind spot
No e2e or chainsaw fixture sets tlsPassthroughServices, and certMode appears in the gateway fixtures once, as http01. The conflicting pair is never rendered under test, so a green e2e on the bump branch says nothing about this. #2478 and #3871 track the missing TLSRoute coverage.
Related
#3343, #3342, #3718, #3720, #3721, #3087, #3869, #2478, #3871, cilium/cilium#46917, cilium/cilium#47306, cilium/cilium#47457, cilium/cilium#47590, cilium/cilium#44889.
Activity
lexfrei commentedon Aug 19, 2026
The layout question here and the open TLS-series PRs (#3475, #3342, #3861) decide each other, so here is the map, and the option I would take.
What each PR changes about the layout
#3475 makes tenant routes declare their hostnames. After it, the hostname claims a Gateway collects are a complete description of what has to be served, and a listener that exists for routes that name nothing has no reader left. That is the precondition for any claim-driven layout in dns01/existingSecret.
#3342 adds native-port passthrough (
tlsPassthroughListeners) and, in http01, stops rendering a terminate listener for a hostname a passthrough listener already answers, which removes the identical-SNI pair from #3720. It keeps the port-443 passthrough trio as it is, and it forbids the new field under dns01/existingSecret by CEL, because the*.<apex>terminate listener of those modes covers every passthrough hostname under the apex. So the wildcard is the one element of the layout that both the new field and this issue trip over.#3861 lets a tenant pick the Gateway class and adds an edge certMode where TLS terminates upstream; edge renders no passthrough listeners at all. It edits
renderGateway, so it matters for merge order, and it has no bearing on the 443 pair.Options against the decisions already taken
Option 3 (a second Gateway for passthrough) goes against the one-address-per-tenant shape the accepted external-exposure design builds on: passthrough backends share the one tenant Gateway address, and per-tenant Gateways cannot share an address through LB-IPAM because they all want 443. #3869 also shows two Gateways in one tenant already fight over the platform namespaces. Option 4 (terminate at the Gateway,
BackendTLSPolicyto the backend) reverses the recorded decision that passthrough is the model and edge termination the exception, and forapi.<apex>it breaks client-certificate kubeconfigs. Option 6 waits on upstream: cilium/cilium#47590 is closed as working as intended, and the Gateway API spec explicitly lets an implementation treat HTTPS and TLS on one port as incompatible even when the listeners are distinct, so there is no conformance argument to bring. Option 5 (relaxReadyin status.go) holds only on 1.19.x, where the conflicted listeners are still programmed; on 1.20 they are not, and the sameReady=Truewould sit on a Gateway that does not serve 443.Option 2 moves
api.<apex>off 443. That hostname is the management cluster API, published bypackages/system/cozystack-apithrough TLS passthrough, the same contract as thessl-passthroughIngress it replaces, and every kubeconfig handed out since points at it on 443. A port change there is a change for every existing client, not a layout change.Option 1 is the one that fits. In dns01/existingSecret, replace the single
*.<apex>terminate listener with per-hostname terminate listeners built from the claims, minus the hostnames a passthrough listener answers, every one referencing the wildcard Secret.https-apexstays. The per-child-apex*.<child-apex>listeners stay, because a child apex and<svc>.<apex>do not intersect. That is the shape http01 has today and the shape #3342 already teaches to subtract passthrough names, so all three cert modes end up with one listener layout and differ only in where the certificate comes from. The 443 pair is gone,tlsPassthroughListenersno longer needs its certMode restriction, and nothing on the public side moves:api.<apex>:443,vm-exportproxy.<apex>:443andcdi-uploadproxy.<apex>:443stay where they are.What option 1 costs, stated up front. The 64-listener cap applies in dns01 as it already does in http01; the platform routes are a handful of hostnames and each child apex is one listener, so a tenant has to publish dozens of distinct hostnames before it matters. An unclaimed hostname under the apex stops answering TLS at all, instead of answering with the wildcard certificate and a 404. A route that claims
*.<apex>itself has nowhere to attach, because that one hostname is exactly the wildcard the passthrough names live under. #3787 has to be closed first or in the same change, since per-hostname rendering of a wildcard claim is what that issue breaks on. And every new route becomes a Gateway spec update, which http01 already does.Order
*.<apex>listener is dropped, the feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342 certMode rule is removed, gateway: a wildcard hostname claim builds a listener name the CRD rejects, failing the whole reconcile #3787 is closed on the way. A reconciler test that renders the defaulttlsPassthroughServicesunder every certMode and asserts that no terminate and passthrough listener share a port with intersecting hostnames is what keeps this from coming back..status.parentsreportsHTTPRoute is not allowed to attach to this Gateway due to namespace selector restrictions#3721 does not have to wait for it: rendering the port-80 listener after the HTTPS listeners fixes it on 1.19.5 as well, and that is a small change inrenderGateway.If the cost list above is acceptable I can open the option-1 PR behind #3342. The layout decision is yours; I am recording the recommendation, not taking it.
lexfrei commentedon Aug 19, 2026
Upstream follow-up: cilium/cilium#48092 asks to narrow the cross-protocol conflict to equal hostnames and apply the spec's exact-before-wildcard precedence for a wildcard superset, the rule the same-protocol branch already uses. Grounds: cilium/cilium#32292 (2024, a maintainer confirmed this exact layout as accepted), the cilium/cilium#44889 review that asked to reject the same-hostname pair only, and no scenario between disjoint and identical hostnames in the author's test matrix. So the 'nothing to take upstream' line under option 6 above is outdated. It does not change the plan here: option 1 is the fix on our side either way, and the upstream change, if it lands, only gives the wildcard listener back later.
lexfrei commentedon Sep 21, 2026
Upstream update. The line above saying there is nothing to take upstream is out of date. I filed cilium/cilium#48092 the same day and it has moved since.
The design question, whether Cilium should serve a wildcard HTTPS listener next to an exact passthrough one, is parked on kubernetes-sigs/gateway-api#4271 and will not resolve soon. The status bug is separate and already filed in cilium/cilium#46917, reported for 1.19.5 up to 1.20.0 and closed by a 1.20-only backport.
That makes option 5 the realistic bridge. Our readiness check is what turns a status bug into a blocked upgrade, and it is the only part of this we control. I am asking for the 1.19 backport upstream in parallel.
feat(gateway)!: reserve a passthrough listener's name by the spec alo…