Repository navigation
Unified TLS certificate and external-exposure model #2811
Description
Activity
- added
on Jun 3, 2026 While bumping the vendored SeaweedFS chart to 4.31 I found a building block relevant to this epic: the
s3.tlsSecretchart value (added upstream in chart 4.16). It points the S3 gateway at a Kubernetes TLS Secret, so the S3 API can serve a publicly-trusted certificate directly from the pod instead of the internal SeaweedFS CA that clients would otherwise have to trust.It isn't used yet. Today the S3 endpoint is exposed through ingress-nginx with a cert-manager certificate at the edge and re-encrypts to the pod over the internal CA, so
s3.tlsSecretbecomes relevant only once object storage is exposed without that re-encryption hop — for example via Gateway API TLS-passthrough (the mechanism explored in #2815) — or to drop the internal-CA dependency on the backend. It could also consume the operator-provided wildcard Secret from #2812.Recording it here as an available option so it isn't rediscovered later; no change was made in the bump itself.
- added 3 commits that reference this issue
on Jun 21, 2026 A quick architecture review of the unified model — agreeing with the direction, with one
reframing and a few gaps worth pinning down before WS3 lands.1. The "single issuance abstraction" holds at the edge, not inside the engines
cert-manager (or a BYO wildcard) is a clean single abstraction for the ingress/Gateway edge.
But per-app DB TLS has three issuers today: cert-manager (nats/qdrant), CNPG (postgres),
Strimzi (kafka). CNPG and Strimzi mint their own self-signed CA and don't go through
cert-manager — and forcing them to consume an external cert costs operator-managed rotation
(CNPGserverTLSSecretis mutually exclusive withserverAltDNSNames; with Strimzi you'd
bypass its CA lifecycle).So the realistic end-state is two-tier: cert-manager/wildcard at the edge + operator-managed
PKI inside the engines. Suggest rewording the top-line goal from "cert-manager as the single
issuance abstraction" to "a single operator-facing interface to choose the cert source + a
uniform way to consumeca.crt."2. Unify the consume contract, not the CA
CA ownership should stay the operator's choice — one corporate BYO CA for everything,
per-engine self-signed, or a cert-manager issuer; different scopes can legitimately carry
different CAs. Homogenizing the CA isn't a goal and would break the engines that rotate their own.What should be uniform: (a) one place to select the CA source; (b) a
ca.crt-only object the
tenant can read without the private key. Kafka already demonstrates the target shape —
<release>-clients-ca-cert/-cluster-ca-certexpose only the CA cert, no key. That's the
reference for the shape, not a mandate for one shared CA.3. Derive the trust model from the client, not a blanket policy
What a client expects on the wire varies by engine, and this is the axis that decides where an
edge wildcard even applies. A first-pass map (operator-capability column to be confirmed against
each operator's current API):App Authentication What the client expects Operator cert capability kubernetes (Kamaji) mTLS client certs + OIDC, over TLS admin/cluster-issued CA; not a public FQDN — kubeconfig pins the cluster CA, SANs are internal self-signed only; Kamaji owns the control-plane CA, not swappable postgres (CNPG) TLS + password (opt. client-cert) valid chain; public FQDN is fine self-signed default; BYO-CA / BYO-cert ( serverCASecret/serverTLSSecret); one server cert, no edge/internal splitkafka (Strimzi) SASL/SCRAM or mTLS, over TLS valid chain externally; internal broker mTLS on its own CA self-signed default; BYO-CA ( generateCertificateAuthority: false) + per-listener BYO-cert (brokerCertChainAndKey)mongodb (Percona PSMDB) TLS + login/password (opt. x509 mTLS) valid chain; public FQDN is fine cert-manager integration + BYO-cert (confirm knobs) mariadb (mariadb-operator) TLS + login/password valid chain; public FQDN is fine cert-manager issuerRef + BYO-cert (confirm) rabbitmq (cluster-operator) TLS + login/password (AMQPS, mgmt HTTPS) valid chain; public FQDN is fine BYO-cert only — consumes spec.tls.secretName/caSecretName(cert-manager fills the secret)opensearch (opster) TLS + login/password (transport mTLS) valid chain on HTTP API; transport = internal node certs self-signed (generate) or BYO-cert via secret refs (confirm) redis (spotahome fork) password + TLS valid chain; public FQDN is fine BYO-cert — consumes a cert-manager secret (fork, #2729) (confirm) clickhouse (Altinity) TLS + login/password valid chain; public FQDN is fine BYO-cert — mounts a supplied cert, no native issuance (confirm) nats / qdrant TLS + login/password / API key valid chain; public FQDN is fine cert-manager — chart-rendered self-signed chain, no operator The takeaway: kubernetes (Kamaji) is the lone "private-CA-mandatory" case — a public edge cert
is meaningless there. Everything else only needs a valid chain, so an operator wildcard at the
edge applies — but each engine's BYO mechanism differs, which is exactly what WS3/WS4 has to account for.4. What isn't designed yet
- The
ca.crt-only object (the WS3 security deliverable) isn't designed.cozy-tlsis a good
building block but it's cert-manager-only (doesn't touch CNPG/Strimzi), and its CA secret carries
ca.key— so it isn't itself aca.crt-only object and doesn't resolve the coupling. A separate,
CA-agnosticca.crt-only projection is still needed. - "Consume not mint" isn't realized even for the cert-manager engines — the open per-app PRs
(mariadb/rabbitmq/opensearch) each mint a per-app self-signed chain. The open question in the epic
body — "should applications stop minting certs entirely and only consume cert-manager output?" — is
the crux for CNPG/Strimzi and is still open. - CNPG/Strimzi BYO-CA flow isn't designed. Note the asymmetry: Strimzi can serve a public cert
on the external listener while keeping internal mTLS on its own CA; CNPG has a single server cert
(no edge/internal split), so it's all-or-nothing. - Per-tenant wildcard distribution (feat(platform): propagate operator wildcard certificate to per-tenant ingress controllers #2820) has no chosen mechanism yet (cross-namespace replication).
5. One correction to strengthen the WS3 (#2814) evidence
The CA/private-key coupling isn't uniform across engines:
- postgres — the only real candidate:
<release>-credentials(CNPG) is tenant-readable via the
dashboard resource map and carriesca.crt. Whether it also carries a private key depends on the
CNPG secret type — worth confirming at triage; the server key lives in<release>-server, which
isn't exposed. - nats — the credentials secret is passwords-only (no TLS material), so the nats reference in
Converge the per-app TLS series onto the ca.crt-only consume contract #2814 doesn't actually demonstrate coupling. nats/qdrant have the opposite gap:ca.crtisn't
delivered to tenants at all. kafka already exposes CA-cert-only (the desired model). - Worth folding in:
tenantsecretsis a full-data projection (label-filtered, not field-filtered),
so aca.crt-only fix has to cover both the per-app Role and thetenantresourcelabel, not just
the Role.
Verdict
Direction is sound and the epic is honest about its blockers. Main asks before WS3 lands:
(1) reframe "single issuer" → "single interface + uniform consume"; (2) keep CA ownership
operator-chosen and unify the consume contract (kafka's CA-cert-only is the shape); (3) design the
ca.crt-only object as CA-agnostic; (4) decide the mint-vs-consume question explicitly per engine.- The
Follow-up / disclaimer: I wrote the review above before the implementation landed, then
checked the PRs — and it looks like you went the right way.- feat(cozy-lib): add CA-only TLS trust-anchor helper for tenants #2989 (CA-only trust-anchor helper) is exactly the "unify the consume contract, not the CA"
shape this comment argues for: a key-freeca.crtobject delivered through the existing
tenantsecretslabel, with CA ownership left to each engine. So the §4 bullet "theca.crt-only
object isn't designed" is already obsolete — that keystone is in. - feat(platform): ACME wildcard certificate on the default ingress-nginx path #2988 (opt-in ingress wildcard, with the single-label coverage caveat documented) and feat(platform, ingress): propagate operator wildcard certificate to per-tenant termination points #2990
(a dedicated replication controller for feat(platform): propagate operator wildcard certificate to per-tenant ingress controllers #2820, with clustersecret-operator deliberately rejected)
line up with the gaps noted above.
So treat the "what isn't designed yet" section as already largely addressed at the platform layer.
The main thing left is the per-app convergence onto the helper — postgres especially, re #2814.
Nice work.- feat(cozy-lib): add CA-only TLS trust-anchor helper for tenants #2989 (CA-only trust-anchor helper) is exactly the "unify the consume contract, not the CA"
I'd push back on treating the "what isn't designed yet" section as largely closed. What's closed is the edge (#2988 / #2990). The harder half is deferred, not solved — and I think the way we got here is itself the problem.
#2989 fixes the shape of the output object, not where its input comes from. The helper renders a key-free
ca.crtSecret carrying thetenantresourcelabel — good, and provably safe on the output side. But it takes the CA cert as a Helm value, and nothing in the current architecture can feed it that value for a per-release CA:- cert-manager issues the per-release CA asynchronously, so it isn't visible at template-render time;
- we're deliberately removing
lookup(replaced byvaluesFrom), and an app release'svaluesFromis pinned to the singlecozystack-valuesSecret byexpectedValuesFrom()— so a chart can't pull<release>-casideways either.
So for the cert-manager engines (nats/qdrant) there is no "consume via the helper" path today. And for the operator-managed engines (CNPG / Strimzi / redis) the operator publishes the CA-only Secret itself — redis #2729 already does this via
caCertSecretName, and the helper isn't called at all; "converging on the helper" there reduces to "match the shape and the label."The consequence is that the epic's headline premise — consume not mint, cert-manager as the single issuance abstraction — isn't realized in either branch. The architecture review already reframed this into the two-tier model (edge cert-manager/wildcard + operator-owned PKI inside the engines), but the epic body still carries the original framing as a goal, not as a recorded decision. The central open question — should applications stop minting certs entirely and only consume cert-manager output? — is still open, and #2816 lists "a design proposal" as an artifact that doesn't exist yet. In other words, the helper landed before the fork that determines how it's meant to be consumed.
The meta-point: we're growing the model PR by PR without a written target design. Had a design doc been drafted before this fleet of PRs — with an explicit per-engine contract for who owns the PKI and how
ca.crtis delivered — the two-tier reality and the mint-vs-consume fork would have surfaced up front, on paper, instead of being discovered one PR at a time. The edge work is fine because the edge is genuinely same-namespace and value-driven; the engine interior is where the absence of a target design shows.Concretely, I'd propose: pin the engine-interior target design (the per-engine PKI-ownership /
ca.crt-delivery contract) before landing the per-app TLS PRs — postgres #2814 first — rather than continuing to extend the model incrementally and back-filling the rationale.- added a commit that references this issue
on Jun 30, 2026 The written target design this thread asked for is now up as two community design proposals: cozystack/community#19 — the certificate/PKI half (the two-tier model, the per-engine PKI-ownership contract, the
ca.crt-only consume contract, and the extraction-controller delivery mechanism) — and cozystack/community#20 — the external-exposure half covering WS4/WS5 (Gateway TLS-passthrough routed by SNI, with a per-engine fit matrix). Both are in review on their PRs; the per-app TLS series is sequenced to land after the PKI contract is accepted.- added a commit that references this issue
on Jul 15, 2026 The interior half is closed. The CA-extraction controller and its
TenantProjectionCRD are on main (#3407), postgres is converged, and a tenant reads the key-free<release>.tenant-cathrough tenantsecrets. Alongside it: the render-time helper (#3408), a capability gate on the hostname policies (#3409), and dot-free release prefixes (#3411), which is what keeps the projection name unreachable by any operator-generated Secret.The delivery mechanism is not what the accepted proposal describes. The proposal specifies a
publish-ca-certlabel on the CA Secret. API-owner review replaced it with a declared sentinel: a label carries no name, no status and no admission surface, so a broken declaration could only requeue forever. The sentinel gets status conditions instead. Proposal update is cozystack/community#41.That settles the open question here about labelling operator-created CA sources. It does not arise: the chart names the source Secret, so CNPG and PSMDB never need to label their output.
The edge half is closed too, #2820 with it. What remains is #2814 for per-engine convergence and #2815 for external exposure.
- added a commit that references this issue
on Jul 27, 2026 Checkpoint on where the epic stands.
The mechanism is closed and on main. The CA-extraction controller and the
TenantProjectionsentinel CRD landed in #3407, with the render-time helper (#3408), the capability gate on hostname policies (#3409) and dot-free release prefixes (#3411) alongside. #3410 (writer-pinning admission policy) was closed after API-owner review: binding authorization to a service-account name is the wrong layer. #3286 was delivered by #3407. Postgres is fully converged, and its E2E suite exercises the whole runtime chain on every CI run: chart-rendered sentinel, key-free<release>.tenant-caprojection, lineage verdict, tenant read through the tenantsecrets API.Beyond postgres, the three open conversions (#3341, #3340, #2692) now have a live witness. I deployed them on a dedicated test cluster with Flux rendering the charts end to end. Each chart rendered its sentinel, the controller produced the projection holding only
ca.crt, the fingerprint matches the engine CA, and the private key never leaves the source Secret. Once the ApplicationDefinition carries the tenant-ca selector, the lineage webhook stampstenantresource: "true": the projection is readable through tenantsecrets and the source Secret is not. Controller behaviour that previously rested on unit tests is confirmed live as well: a projection deleted without touching its sentinel is recreated by the reverse watch within seconds, a foreign Secret already occupying the canonical name is refused with a Warning event instead of being overwritten, and deleting the sentinel garbage-collects the projection.Queue state: #3341 (qdrant), #3340 (nats) and #2692 (mongodb) are rebased on current main; the review verdicts they carry predate the sentinel rework. Next in line, rabbitmq (#2683), opensearch (#2682) and mariadb (#2680) are TLS features plus the sentinel, since none of them ships TLS on main today. Redis (#2729) stays out of this phase: its blocker is the vendored operator fork, not the trust-anchor contract. Kafka was already converged. Chainsaw coverage for the newly converted engines follows as one PR once they merge.
On the proposal side, the delivery mechanism changed from the accepted text's label leg to the declared sentinel. cozystack/community#36 (canonical name and ownership) and cozystack/community#41 (sentinel mechanism) bring the document in line and are still open; until they merge, the accepted proposal describes a mechanism that never shipped.
Follow-ups filed along the way: #3463 corrects the postgres README to the anchor name a tenant actually sees, and #3464 tracks the tenantsecret registry copying caller-supplied
internal.cozystack.iolabels through.What remains of the epic is #2814 for per-engine convergence and #2815 for external exposure; the edge half is closed.
Checkpoint, 2026-09-02.
The three conversions that had a live witness in the last checkpoint are on main: qdrant (#3341), nats (#3340) and mongodb (#2692) merged on 2026-09-01. Each renders the
TenantProjectionsentinel from its chart, the controller publishes the key-free<release>.tenant-cacopy, and the-rdgrant plus the per-engine bats guard pin that the key-bearing source stays out of the tenant's reach. With postgres and kafka that makes five engines on the contract.The remaining engine series is in review. mariadb (#2680) and rabbitmq (#2683) deliver the trust anchor through the same sentinel instead of the shapes they were written against; opensearch (#2682) still carried the label mechanism that never shipped and is being converted. Every earlier review thread on the three is closed, and they are being rebased onto current main for a fresh CI run. redis (#2729) was rebuilt on the freshworks operator (v3.3.5) with TLS carried as a downstream patch until freshworks-oss/redis-operator#85 ships; the second review round is answered, including an upgrade replay on a live cluster where an existing plaintext instance survived the CRD replace and the operator swap without a restart.
External exposure (#2815) has its phase 1 in #3342: TLS-passthrough listeners on the tenant Gateway, verified on a live cluster (upgrade shed of the old terminate listeners, a route-less listener programmed by Cilium, SNI passthrough end to end). It waits on an API-owner review, as do the hostname policy fix (#3475), the edge/class selection (#3861), the Cloudflare Tunnel controller (#3858) and the Gateway API 1.6.1 bump (#3872), all approved by a reviewer and being rebased.
The accepted proposal is being brought in line with what shipped in cozystack/community#36 (canonical name, sentinel ownership, the x509 guard, the engine table).
Checkpoint, 2026-09-04.
Merged today: community#36, so both proposals now describe the mechanism that shipped and three decision records are frozen on main; #2680 mariadb and #2729 redis, TLS with the sentinel, mariadb also moving its chart-chosen Secret suffixes onto the dot; #4064, the cacert reconciler comments. Engines converged on main: postgres, kafka, qdrant, nats, mongodb, mariadb, redis.
Still open in the series: #3342 (round 4 answered: the terminate listener is withdrawn only for a TLSRoute that forwards, with Service and ReferenceGrant watches), #2682 opensearch and #2683 rabbitmq (reviews answered, E2E green on the new heads), #4063 (the postgres db.yaml comment). #3475 is rebased onto main after #3872 and #3861 landed and waits for re-approval and E2E.
Filed from the reviews and gates this week: #4065 to #4073, plus two upstream issues on the Cloudflare tunnel controller. #4045 was a duplicate of #3347 and is closed.
Checkpoint, 2026-09-17. Body updated to main at 8a6fe02.
Merged since the last one: #3475 and #2683 (09-07), #4152 and #4143 (09-08), #4185 (09-10), #2682 and #3342 (09-11).
Interior is complete, #2814 is closed. All nine engines hand the tenant a key-free CA.
Exposure has step 1 on main and nothing wired, details in #2815. Redis is next, and it needs an owner for per-release
TLSRouterendering first.Closed along the way: #4131, #4073, #3787. Open follow-ups: #4144, #4202, #4210, #4211, #4207, #4208, #4115.
This stays open until #2815 closes.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Summary
Certificate handling grew fragmented across four uncoordinated mechanisms: per-app issuance inside each chart, per-host ACME on the default ingress path, an opt-in Gateway API path that mints a wildcard via DNS-01, and no supported way for an operator to bring an externally-issued wildcard. Separately, every externally-exposed database burns its own public IP.
The original framing of this epic, cert-manager as the single issuance abstraction, with certificates consumed and never minted by applications, is not realizable, and the design work has replaced it. Forcing pure-consume breaks CloudNativePG and Strimzi, whose own-CA rotation is mutually exclusive with an externally-supplied server certificate. The target is instead two-tier: cert-manager or a bring-your-own wildcard at the edge, operator-owned PKI inside the engines. What gets unified is not the certificate authority but two contracts, a single operator-facing interface for choosing the certificate source, and a single uniform way for a tenant to consume
ca.crt. External exposure is routed through a non-terminating gateway, which makes end-to-end TLS a property of the path rather than a feature to build.Accepted design
Both halves are written up in cozystack/community:
unified-tls-pki, the certificate and PKI half: the two-tier model, the per-engine PKI-ownership contract, theca.crt-only consume contract, and the mechanism that delivers it. docs(unified-tls-pki): fix the canonical trust-anchor name and its owner community#36, merged 2026-09-04, brought the delivery-mechanism section in line with what shipped (see below).external-database-exposure, the external-exposure half: SNI routing of databases through Gateway TLS-passthrough, and the end-to-end TLS story that follows from it.Decisions recorded
ca.crt, that is what determines whether it needs any platform machinery at all.<release>.tenant-caholding onlyca.crt, stamped with the tenant-resource label. The dot in the name is load-bearing: it is collision-proof by construction, because application names are DNS-1035 labels (no dot) while Secret names are DNS-1123 subdomains (dot legal), so no operator-generated name can ever equal it. The earlier<release>-ca-certform was rejected because Percona PSMDB already creates a Secret of exactly that name and puts a private key in it.TenantProjectionCRD (internal.cozystack.io) naming the source Secret by name; the controller watches the sentinel and projects a key-free copy. This replaced the label leg from the accepted proposal (apublish-ca-certlabel on the source Secret) after API-owner review: a label carries no name, no status, and no admission surface, so a broken declaration could only requeue silently, whereas the sentinel gets status conditions. The proposal text was brought in line in docs(unified-tls-pki): fix the canonical trust-anchor name and its owner community#36.ca.crt, is accepted openly rather than hidden.*.<engine>.<apex>) in v1, because that is what the deployed Cilium isolates correctly. Flat*.<apex>is a later additive phase, made non-breaking by issuing both SANs from day one.apps.cozystack.iokinds. The aggregated apiserver evaluates cluster policies itself, which was verified at runtime rather than assumed.Current state (verified against
main@8a6fe02ee, 2026-09-17)ca.crt-only trust-anchor helper exists ascozy-lib.tls.caCertSecret(PR feat(cozy-lib): add CA-only TLS trust-anchor helper for tenants #2989, merged); per-tenant propagation of that wildcard is landed (feat(platform): propagate operator wildcard certificate to per-tenant ingress controllers #2820, PR feat(platform, ingress): propagate operator wildcard certificate to per-tenant termination points #2990, merged). East-west encryption is opt-in and landed (Opt-in CNI transparent encryption for in-cluster (east-west) traffic #2977, PR feat(platform): add opt-in in-cluster east-west encryption toggle #2984, merged).TenantProjectionCRD are on main (feat(cozystack-controller): extract application CA into a key-free tenant Secret #3407), alongside the render-time helper (fix(cozy-lib): converge the CA trust-anchor helper on the canonical name #3408), a capability gate on the hostname policies (fix(cozystack-basics): gate the hostname VAP policies on the VAP API #3409), and dot-free release prefixes (feat(api)!: require the application release prefix to be dot-free #3411). All nine engines in scope hand the tenant a key-free CA. Eight do it through the sentinel: postgres (feat(cozystack-controller): extract application CA into a key-free tenant Secret #3407), qdrant (feat(qdrant): publish ca.crt-only trust anchor to tenants #3341), nats (feat(nats): publish ca.crt-only trust anchor to tenants #3340), mongodb (feat(mongodb): add TLS support via PSMDB operator-managed certificates #2692), mariadb (feat(mariadb): add TLS support via cert-manager #2680), redis (feat(redis): add opt-in TLS for Redis and Sentinel #2729), rabbitmq (feat(rabbitmq): add TLS support for AMQPS and management HTTPS #2683) and opensearch (feat(opensearch)!: issue HTTP and Dashboards certificates from cert-manager #2682, the last one, merged 2026-09-11). Kafka does it through the two key-free Secrets Strimzi already publishes. Where the engine's CA exists only when TLS is on, the anchor follows the same condition: redis and rabbitmq publish it oncetls.enabledis set, opensearch undertls.issuer: cert-manager. The mechanism and the first four conversions are in v1.7.0-alpha.1; the rest is on main and not tagged yet. Converge the per-app TLS series onto the ca.crt-only consume contract #2814 is closed.TenantGatewaycarriestlsPassthroughListenersand the controller renders them (feat(gateway)!: add TLS-passthrough listeners to TenantGateway #3342), with the tenant-routes-must-declare-hostnames fix (fix(gateway)!: require tenant routes to declare their hostnames #3475) beside it. No database is exposed through a passthrough listener yet. Nothing renders aTLSRoutefor a database release, the certificates lack the per-engine name, and neither the plaintext-exposure admission policy nor the serial-equality test exists. External database exposure via Gateway API TLS-passthrough (SNI) #2815 carries the detail and the order.Workstreams
Edge, supply or select the certificate source
Interior, the consume contract
External exposure
Complementary
Open questions
TLSRouterendering. The natural owner is the layer that reconciles structured exposure entries, but theExposureClass/ServiceExposureAPI such a layer would build on is itself under reconsideration (Reconsider network.cozystack.io (ExposureClass / ServiceExposure): necessity vs native loadBalancerClass, and coordination with the tenant-networking surface #3164, PR refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass #3218), so External database exposure via Gateway API TLS-passthrough (SNI) #2815 deliberately does not depend on it. The proposal names an interim trigger, a toggle next to the engine'sexternal. Step 2 of External database exposure via Gateway API TLS-passthrough (SNI) #2815 needs this answered first.The subdomains-now-or-wait question is settled: #3342 shipped step 1 on Cilium 1.19.5 without waiting for the isolation fix.
Follow-ups filed along the way
Defects found during the series and left as issues rather than widening the PRs that found them:
vm.max_map_countin the node contract), opensearch: a tenant-chosen users key can construct a Secret name the chart already owns #4202 (auserskey can build a Secret name the chart owns), opensearch: Dashboards authenticates as a user the chart's own security config removes #4210 (Dashboards authenticates as a user the security config removes), opensearch: operator-issued certificates expire after a year and are never reissued #4211 (operator-issued certificates expire after a year and are never reissued).tls: nullaborts the render of nats, qdrant and postgres).