External database exposure via Gateway API TLS-passthrough (SNI) #2815

Status

Verified against main @ 8a6fe02ee, 2026-09-17.

Rollout step 1 is on main. #3342 added tlsPassthroughListeners to TenantGatewaySpec and the controller renders the listeners. #3475 (tenant routes must declare their hostnames), #3861 (a tenant picks its Gateway class) and #3872 (Gateway API CRDs v1.6.1) landed alongside it. No engine is wired, which is what step 1 was scoped to.

The step-1 gate was run on a test cluster with Cilium 1.19.5, against a hand-built Gateway that mirrors the controller's render. A passthrough listener on 5432 next to the terminate listener on 443 routed by SNI to the right backend, and Envoy built exactly one passthrough filter chain per hostname. Two things came out of that run. The port is not an access boundary: the chain matches on SNI and carries no destination port, so a database hostname also answers on 443. And nobody has opened a connection through the controller's own render yet: the gateway e2e suite renders a passthrough listener and asserts its status, but dials nothing.

What steps 2 to 4 still need, none of it on main:

• Nothing renders a TLSRoute for a database release. The only TLSRoutes in the tree belong to api, vm-exportproxy and cdi-uploadproxy. The ownership question at the bottom is still open, and step 2 cannot start without an answer.
• The charts that add an external name to the certificate add only the flat <release>.<apex> (checked: postgres, qdrant, nats, mariadb, opensearch), and redis carries in-cluster names only. The per-engine name <release>.<engine>.<apex> has to be added before an engine is wired.
• There is no admission policy rejecting external: true without TLS.
• There is no end-to-end test comparing certificate serials, and none showing a client without SNI failing closed.

Step 4 waits on a Cilium carrying the isolation fix, and the Cilium bump itself is blocked on #3912.

Context

The merged design proposal external-database-exposure covers this issue; what follows is the implementation scope it settles.

Today each external: true database gets its own LoadBalancer Service and its own public IP, one per instance. TLS-passthrough already exists in the tenant Gateway (internal/controller/tenantgateway/reconciler.go) for the layer-7 services (api, vm-exportproxy, cdi-uploadproxy), driven by TLSPassthroughServices on TenantGatewaySpec, which renders tls-<service> listeners on port 443. No database is wired through it.

Routing databases by SNI through a non-terminating listener consolidates many databases onto one tenant Gateway IP. And because the Gateway does not terminate, the certificate the external client validates is byte-identical to the operator-issued certificate the database already presents internally: end-to-end TLS is a consequence of not terminating, not a separate feature to build. That is why #2816 is closed rather than tracked separately.

Listener model

One passthrough listener per engine type, on that engine's native port, with a per-engine wildcard hostname: tls-postgres on 5432 with *.postgres.<apex>, tls-redis on 6379 with *.redis.<apex>, tls-mongos on 27017 with *.mongo.<apex>. Each is mode: Passthrough, with AllowedRoutes limited to TLSRoute. Listener consumption is therefore O(engine types), four or five, not O(database instances).

Native ports rather than 443: forcing everything onto 443 buys no additional IP consolidation, since SNI already does that on any port, and it breaks client tooling defaults. All-on-443 remains available as a documented opt-in.

Per-engine subdomain hostnames are the v1 default because they avoid the listener-hostname overlap that Cilium isolates correctly only from 1.20 onward, and the platform runs 1.19.5. The flat *.<apex> scheme is a later additive phase.

API

A new structured field on TenantGatewaySpec, alongside the existing TLSPassthroughServices, which stays, the layer-7 field is not replaced:

type TLSPassthroughListener struct {
    Name     string `json:"name"`     // renders as listener "tls-<name>"
    Port     int32  `json:"port"`     // native port, neither 80 nor 443
    Hostname string `json:"hostname"` // required, exact or left-most-label wildcard inside the apex
}

TLSPassthroughListeners []TLSPassthroughListener `json:"tlsPassthroughListeners,omitempty"`

Validation, as shipped in #3342: Name is a DNS-1123 label, unique within the list and distinct from every tlsPassthroughServices entry; Port is 1..65535, unique within the list, and neither 80 nor 443; Hostname is required rather than defaulted, an exact or wildcard hostname equal to the apex or under it. The field is accepted only under the http01 certificate mode, because the *.<apex> terminate listener of the other two modes would overlap it on SNI.

The route is a standard upstream TLSRoute, no new type. It is rendered once per exposed database release, attaching to the shared tls-<engine> listener by sectionName and carrying <release>.<engine>.<apex> as its hostname. The engine-type listener is created on the first exposure of that engine and removed when its last release goes away; adding or removing a release only touches the route.

Database charts do not render Gateway plumbing. Listener synthesis and route rendering stay in the controller.

Per-engine verdicts

MariaDB is excluded at the protocol level, not on effort grounds. The classic MySQL and MariaDB protocol on 3306 is server-speaks-first: the server sends a plaintext greeting and only then may the client upgrade. A raw passthrough listener waits for a ClientHello that will never come, while the client waits for a greeting the listener will never send. That is a mutual deadlock, not a degraded mode. No TLS-first wire mode exists in any MySQL or MariaDB release (the upstream request, MySQL bug #84849, is still open), and the X Protocol on 33060 is STARTTLS-style as well. MariaDB keeps its dedicated LoadBalancer. This also closes the "MariaDB connector conformance" question the original issue carried: no connector can conform, because the server, not the client, owns the first packet.

Certificate flow

The chart is to inject both hostnames, <release>.<engine>.<apex> and <release>.<apex>, into the operator-issued certificate's SAN list before an engine is wired, so the later flat-hostname phase invalidates no certificate and forces no client reconfiguration. Only the flat name is in the tree today, see Status. External clients validate against the <release>.tenant-ca trust anchor defined by #2814, which makes SNI exposure gated per engine on that engine's convergence, rather than blanket-gated on the contract merely existing.

Security

An explicit tls.enabled: false combined with external: true is rejected at admission, so a tenant cannot stand up a plaintext external endpoint by accident and the value is never silently overridden. The rule ships as a ValidatingAdmissionPolicy bound to the typed apps.cozystack.io kinds. That works even though those kinds are served by the aggregated cozystack-api: the extension apiserver evaluates cluster policies itself through the generic apiserver's delegated admission chain. This was verified at runtime on a live cluster, and the existing cozystack-tenant-host-policy already relies on the same path in production.

A missing or mis-emitted SNI is a hard connection failure, since a passthrough listener has no certificate to fall back on. Fail-closed is the secure default here.

Rollout

1. Done in feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342. API field plus controller listener rendering, with no engine wired. Gate: on the deployed Cilium 1.19.x, stand up a mode: Passthrough listener on a native database port with a per-engine subdomain hostname alongside the Gateway's terminate listeners, and confirm it SNI-routes to the right backend with exactly one Envoy filter chain. No part of this phase depends on an unreleased Cilium.
2. Redis, opt-in.
3. PostgreSQL (direct TLS) and sharded MongoDB, opt-in.
4. Flat-hostname refinement, gated on the platform carrying the Cilium isolation fix. Certificates already carry both SANs, so this phase changes no client.

The default stays today's per-database LoadBalancer. Passthrough is opt-in, and migrating an existing external database onto it is a breaking change: the IP changes, and Postgres clients need direct TLS.

Constraints

• One IP per tenant Gateway on current Cilium: every Gateway claims 443/TCP, so the LB-IPAM sharing-key path is inactive on that collision. ListenerSet consolidates many tenants' listeners onto one shared Gateway; it does not make separate Gateways share an IP. Merged upstream as GatewayAPI: Implement ListenerSets cilium/cilium#46303, expected in Cilium 1.20.
• Gateway.spec.listeners is hard-capped at 64, while TLSRoute objects are not capped, which is exactly why the listener model is per-engine rather than per-database.
• Cilium 1.19.x consumes TLSRoute from the experimental-channel CRD. Pin to whatever API version the targeted Cilium ships, not the upstream v1 graduation.
• Postgres direct TLS requires sslmode=require or stronger, mandates ALPN, and emits SNI only when the client dials by hostname. Driver coverage is broad (pgjdbc 42.7.4+, Npgsql 9.0+, node-postgres, pgx 5.7.5+, lib/pq, psycopg linked against libpq 17), with sqlx and asyncpg as the known gaps.
• redis-cli sends SNI only with an explicit --sni flag; library clients derive it from the hostname.

Open questions

• Settled by feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342, which shipped step 1 on 1.19.5 without waiting; kept for the record. Ship subdomains now on 1.19.x and adopt flat hostnames later as an additive phase, or wait for a Cilium carrying the isolation fix (gateway-api: fix listener isolation for shared hostnames and ports cilium/cilium#44889, targeting 1.20, with an open 1.19 backport) and ship flat-only from day one. The proposal specifies the subdomain path as the no-wait default; this is a scheduling call, not a design fork. Waiting may be free in calendar terms, since engine wiring is gated on Converge the per-app TLS series onto the ca.crt-only consume contract #2814 convergence anyway.
• Who renders the per-release TLSRoute and reference-counts the shared listener down to zero. The natural owner is the orchestration layer that reconciles structured exposure entries; until that lands, the interim trigger is the engine's external-adjacent toggle. Note that the ExposureClass/ServiceExposure API such a layer would build on is itself under reconsideration (Reconsider network.cozystack.io (ExposureClass / ServiceExposure): necessity vs native loadBalancerClass, and coordination with the tenant-networking surface #3164, PR refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass #3218), so this design deliberately does not depend on it.
• Should the controller surface a status condition as the 64-listener budget approaches the cap?
• What fraction of the tenant base predates direct-TLS support on either the Postgres client or the server? There is no graceful downgrade on a passthrough listener.

Acceptance

• A database is reachable externally through a Gateway TLS-passthrough listener routed by SNI, with no termination at the gateway.
• An end-to-end test asserts that the serial of the certificate presented externally equals that of the operator-issued internal certificate, proving reuse rather than re-issuance.
• A client that emits no SNI fails closed.
• tls.enabled: false combined with external: true is rejected at admission.

Part of #2811. Gated per engine on #2814. Subsumes #2816.