Skip to content

Converge the per-app TLS series onto the ca.crt-only consume contract #2814

Description

Status

Done. Verified against main @ 8a6fe02ee, 2026-09-17: all nine engines in scope hand the tenant a key-free CA, eight through the sentinel and kafka through Strimzi's own Secrets. The last conversion, opensearch (#2682), merged on 2026-09-11. What follows is the record of the contract and of how each engine met it.

Context

The design proposal unified-tls-pki unifies the consume contract, not the CA. The CA stays heterogeneous on purpose: cert-manager or a bring-your-own wildcard at the edge, operator-owned PKI inside the engines. This issue is the per-engine convergence onto that consume contract. Part of #2811. The contract itself landed in #3407.

The problem, restated against main

When the contract landed, exactly two engines handed the tenant a CA object: kafka exports Strimzi's two key-free -ca-cert Secrets, and postgres publishes the canonical projection through the contract. Every other engine exported credentials or an api-key only and delivered no trust anchor at all. So the problem is heterogeneity plus absence, not an existing leak.

That absence is what made the per-app TLS pull requests dangerous. Lacking a key-free object to hand over, several of them reached for a key-bearing one: the CA Secret that also holds ca.key/tls.key, or a leaf key. The contract exists to give every engine one identically-named key-free object, so those PRs had the right thing to deliver.

The contract

Every converged engine publishes its trust anchor as an Opaque Secret named <release>.tenant-ca holding only ca.crt. The engine-agnostic CA-extraction controller (internal/controller/cacert, #3407) produces it; delivery to the tenant rides the existing core.cozystack.io/tenantsecrets API the base tenant roles already grant.

The dot in <release>.tenant-ca is load-bearing, not cosmetic. The dash form <release>-tenant-ca is ambiguous: for a release named foo it renders foo-tenant-ca, which is also CloudNativePG's CA Secret name for a neighbouring cluster named foo-tenant. A dot cannot appear in a generated operator CA Secret name, and #3411 constrains release prefixes to keep it that way.

Two distinct labels — do not conflate them

  • internal.cozystack.io/tenant-ca: "true" is stamped by the CA-extraction controller on the <release>.tenant-ca projection. It both marks the object as the trust anchor and is the single engine-agnostic selector each per-app ResourceDefinition uses to include the projection in the tenant-readable set (matchLabels: internal.cozystack.io/tenant-ca: "true", no per-release name templating).
  • internal.cozystack.io/tenantresource: "true" is applied by the lineage controller by tracing ownership, and is what the aggregated tenant-secret API surfaces to the tenant. It cannot be forged by a static chart stamp, because the lineage controller computes it from ownership rather than reading it off the object.

The convergence shape

There is one shape, not two. An earlier revision of this issue described a label leg (publish-ca-cert on the CA Secret) and a declared leg (spec.caCert on the ApplicationDefinition). Neither shipped: both belonged to #3299, which was closed in favour of #3407. API-owner review replaced them with a declared sentinel, because a label carries no name, no status and no admission surface of its own, so a broken declaration could only requeue silently.

The chart renders one namespaced object naming the source Secret:

apiVersion: internal.cozystack.io/v1alpha1
kind: TenantProjection
metadata:
  name: {{ .Release.Name }}-ca
spec:
  projections:
    - type: CACert
      sourceSecretName: {{ .Release.Name }}-ca
      sourceKey: ca.crt

Naming the source works for every engine regardless of who creates the CA, so the question the two legs used to answer, whether the chart can stamp a label on that Secret, no longer arises. The reference implementation is packages/apps/postgres/templates/tenant-projection.yaml.

Three invariants, each of which breaks silently when violated. The sentinel carries no ownerReferences: it resolves to its HelmRelease through the helm.toolkit.fluxcd.io/name label Flux stamps, and an owner reference makes the lineage walk return early, so the tenant stops seeing the anchor. It carries no namespace or labels of its own. And it must sit inside the chart's TLS conditional wherever the CA only exists when TLS is on, or the sentinel reports Ready=False/SourceNotFound forever on every plaintext release; postgres is ungated only because CloudNativePG mints its CA either way.

Alongside it, one line in the engine's ResourceDefinition:

- matchLabels:
    internal.cozystack.io/tenant-ca: "true"

Per-engine reality (verified against origin/main)

Engine TLS on main? CA source Key-bearing? Who creates it Needs
postgres yes <release>-ca yes (ca.key) CloudNativePG operator done in #3407
kafka yes <release>-cluster-ca-cert, <release>-clients-ca-cert no Strimzi operator nothing, already key-free and already tenant-visible
nats yes, apps/nats/templates/certmanager.yaml <release>-ca yes (tls.key is the CA key) chart (cert-manager) done in #3340 (sentinel, gated on the chart's TLS flag)
qdrant yes, apps/qdrant/templates/certmanager.yaml <release>-ca yes chart (cert-manager) done in #3341 (sentinel, gated)
mongodb yes, since #2692 <release>-ssl, the leaf Secret the chart names in spec.secrets.ssl; its ca.crt also exists on the operator's self-signed fallback and carries both CAs across a rotation, which <release>-ca-cert does not yes (tls.key) PSMDB operator, asynchronously done in #2692 (sentinel, ungated)
opensearch yes, always served <release>.http-ca, a chart-minted CA handed to the operator through spec.security.tls.http.caSecret yes (tls.key is the CA key) chart (cert-manager) done in #2682, merged 2026-09-11 (sentinel, gated on tls.issuer: cert-manager, the default when external is true; under tls.issuer: operator the operator's own CA is used and no anchor is published)
mariadb yes, #2680 merged 2026-09-04 <release>-ca-bundle, reconciled key-free by the operator no mariadb-operator TLS and sentinel landed together in #2680
rabbitmq yes, #2683 merged 2026-09-07 <release>-ca yes chart (cert-manager) TLS and sentinel landed together in #2683 (sentinel, gated on the chart's TLS flag)
redis yes, #2729 merged 2026-09-04 <release>.ca-tls (chart CA; the operator-published <release>.ca-cert copy is not consumed) yes chart (cert-manager) TLS and sentinel landed together in #2729 (merged; operator is freshworks v3.3.5 with a downstream TLS patch until freshworks-oss/redis-operator#85 ships)

Kamaji-backed kubernetes is out of the model: its PKI is Kamaji-owned and non-swappable.

Two corrections to what this issue previously claimed. Kafka is not pending: packages/system/kafka-rd/cozyrds/kafka.yaml already grants both -ca-cert Secrets and hack/check-kafka-rd-secrets.bats pins it. And the redis-operator fork does not publish a key-free CA on main, where caCertSecretName has zero occurrences; that fork ships inside #2729, so redis is a from-scratch TLS feature rather than a convergence.

Order

  1. postgres, with feat(cozystack-controller): extract application CA into a key-free tenant Secret #3407. Done. It exercised the contract against the hardest input: an operator-created, asynchronous, key-bearing CA Secret.
  2. qdrant (feat(qdrant): publish ca.crt-only trust anchor to tenants #3341), nats (feat(nats): publish ca.crt-only trust anchor to tenants #3340) and mongodb (feat(mongodb): add TLS support via PSMDB operator-managed certificates #2692). Done, merged 2026-09-01. qdrant fixed the exact shape the rest copy; mongodb is the reference conversion for an operator-minted CA.
  3. mariadb (feat(mariadb): add TLS support via cert-manager #2680), rabbitmq (feat(rabbitmq): add TLS support for AMQPS and management HTTPS #2683), opensearch (feat(opensearch)!: issue HTTP and Dashboards certificates from cert-manager #2682). Done, merged 2026-09-04, 2026-09-07 and 2026-09-11. Same shape; opensearch was converted from the label leg on the way.
  4. redis (feat(redis): add opt-in TLS for Redis and Sentinel #2729). The operator is now freshworks v3.3.5 with the TLS support carried as a downstream patch until feat(tls): TLS for RedisFailover via cert-manager freshworks-oss/redis-operator#85 ships; the sentinel is in the same PR. Done, merged 2026-09-04.

Acceptance

  • Every converged engine publishes a key-free <release>.tenant-ca holding only ca.crt.
  • A tenant can obtain ca.crt without read access to any Secret containing tls.key or ca.key.
  • Each engine's chart renders the sentinel and its ResourceDefinition carries the tenant-ca selector, with a helm-unittest asserting both, and no key-bearing Secret projected to a tenant.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    on Jun 3, 2026
  2. lexfrei commented on Jun 26, 2026

    @lexfrei
    ContributorAuthor

    Triaged the §2 claim against main — recording the result, since it changes the framing.

    <release>-credentials does not carry a private key. On main it is a chart-rendered Opaque Secret holding only user: password pairs (packages/apps/postgres/templates/init-script.yaml:18-27), and it is the only postgres Secret in the tenant dashboard resource map (packages/apps/postgres/templates/dashboard-resourcemap.yaml). The db.yaml:20-22 comment claiming ca.crt rides in <release>-credentials contradicts what the chart renders — a stale comment, not the behaviour. CNPG's actual CA (<release>-ca, carrying ca.key) and server key (<release>-server) are operator-created and are not tenant-facing. The nats credentials Secret is likewise passwords-only.

    So there is no CA/private-key coupling exposed to tenants on main, and no merged engine leaks a key. The real problem is the inverse: ca.crt is never delivered to the tenant at all — a trust-anchor delivery gap, the same shape across postgres, nats, and qdrant.

    §2 is still a blocker, for a sharper reason. With no key-free trust-anchor object to deliver, the in-flight per-app PRs reach for the wrong one: mariadb #2680 projects the CA private key (<release>-ca-tls), mongodb #2692 projects the key-bearing <release>-ca-cert, and rabbitmq #2683 / opensearch #2682 project the leaf key — each handing the tenant a Secret that also contains tls.key. The fix is unchanged in spirit (a ca.crt-only object), but the motivation is "close the delivery gap so those PRs have a key-free object to deliver", not "stop a current leak".

    The target design is written up in cozystack/community#19.

  3. changed the title [-]Converge the per-app TLS series and fix the CA / private-key coupling[/-] [+]Converge the per-app TLS series onto the ca.crt-only consume contract[/+] on Jul 14, 2026
  4. lexfrei commented on Sep 17, 2026

    @lexfrei
    ContributorAuthor

    Closing, all nine engines are converged on main (checked at 8a6fe02).

    Eight render the TenantProjection sentinel and have the tenant-ca selector in their ResourceDefinition: postgres, qdrant, nats, mongodb, mariadb, redis, rabbitmq, opensearch. Each has a tests/tenant_projection_test.yaml and a guard on what the ResourceDefinition exposes (hack/check-<engine>-rd-secrets.bats, for postgres the chainsaw step). Kafka needed nothing, Strimzi already publishes two key-free -ca-cert Secrets.

    The anchor is conditional in three charts, because the CA it copies exists only then: redis and rabbitmq with tls.enabled, opensearch with tls.issuer: cert-manager (the default when external is true). With tls.issuer: operator the tenant gets no anchor.

    I fixed three rows of the table while closing: the mongodb source is <release>-ssl, opensearch is <release>.http-ca, rabbitmq is <release>-ca.

    Exposure continues in #2815.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/databaseIssues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse)kind/bugCategorizes issue or PR as related to a bugpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasesecuritySecurity-related issues and featuressecurity/triage-neededNeeds security triagetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions