Skip to content

design-proposal: tenant-supplied secrets by reference through External Secrets Operator - #82

Open
Timofei Larkin (lllamnyp) wants to merge 2 commits into
mainfrom
design/user-managed-secrets
Open

Timofei Larkin (lllamnyp) wants to merge 2 commits into
mainfrom
design/user-managed-secrets

Conversation

@lllamnyp

Copy link
Copy Markdown
Member

Adds design-proposals/tenant-supplied-secrets/: how a tenant supplies a secret to a managed application without a plaintext field in the spec and without any grant on core/v1 Secrets.

Tenants get a secret store of their own; application specs reference an entry in it by name; a chart-rendered ExternalSecret materialises the value for the operator. External Secrets Operator becomes a required component. The default store is a single platform OpenBao with one OpenBao namespace per tenant, but the only coupling to any store is one per-tenant SecretStore object, overridable cluster-wide by the operator and per tenant by the tenant, so an external OpenBao, a cloud secrets manager, or the tenant's own managed OpenBao all fit behind the same seam. Tenant Kubernetes clusters consume from the same store through an externalSecrets addon, with a per-cluster auth method the platform provisions.

The store topology is the main open question: the text recommends the in-cluster instance as the installed default, an external OpenBao as the documented production posture, and per-tenant instances as a tenant-level choice. Reviewers who would set a different default should argue it there. The concrete reference shape is deliberately left to the first conversions.

Relationship to existing work

Opening non-draft for direction review. Extends the platform's required components and the Tenant and Kubernetes application schemas, so it sits behind the API-owner gate.

Before review

  • If this revises a merged design proposal: it adds a decision record under that proposal's decisions/ directory, or says below why none is needed. New proposal; no decision record needed.

DCO

  • Commits are signed off (git commit --signoff).

…l Secrets Operator

Tenants hold no verbs on core/v1 Secrets, so every tenant-supplied credential today is a plaintext field in an application spec. This proposal gives tenants a secret store, references entries from application specs by name, and materialises them for operators through External Secrets Operator, with a per-tenant SecretStore as the seam that keeps the store pluggable. It absorbs the SecretRef proposal (#37), inherits cozystack/cozystack#1942, and sits beside the user secrets API proposal (#74), which keeps platform-generated credentials.

Assisted-By: LLM
Signed-off-by: Timofei Larkin <[email protected]>
@coderabbitai

coderabbitai Bot commented Sep 21, 2026 •

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 12b76c38-38ba-42d7-a3a6-36ce2de53d01


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…ies and reorder the rollout

External Secrets Operator and the central OpenBao become default-installed optional packages with the cost of disabling each stated, the way database autoscaling degrades without KEDA. The rollout no longer waits on the central OpenBao: the seam takes any store, so the first consumers are validated against a managed OpenBao application and an external store first. Platform service-account credentials are named as a non-goal belonging to the user secrets API proposal, the write-only TenantSecret alternative is argued concretely, and the phase-2 withdrawal is no longer recorded as settled.

Assisted-By: LLM
Signed-off-by: Timofei Larkin <[email protected]>
@lllamnyp

Copy link
Copy Markdown
Member Author

Revised in response to the discussion on #74, where this proposal was read as making External Secrets Operator and a central OpenBao required components. It never should have said so, and it did. What changed:

  • Capability model instead of required components. Both are default-installed optional packages. Disabling External Secrets Operator makes a chart that receives a reference fail the render naming the capability, the way database autoscaling already fails without KEDA; an application whose only credential field is reference-only is not installable there. Disabling the default store leaves the seam waiting for a provider override. The tenant chart gates its SecretStore on the operator's API being present, so disabling the package does not fail tenant releases on an unknown kind. The expected trajectory, that conversions narrow what an installation without the capability can run, is stated as intended rather than left implied.
  • Rollout reordered so nothing waits on the central OpenBao. The seam takes any store, so the first consumers are validated against a managed OpenBao application and an external store before feat(openbao): central platform instance and per-tenant transit provisioning cozystack#4177 lands. The central instance becomes step 5, not step 2.
  • Platform service-account credentials named as a non-goal. MariaDB root, ClickHouse's backup user, OpenSearch admin and Harbor's redis password are neither tenant-supplied nor tenant-facing; separating them from tenant users belongs to design-proposal: user secrets API, credentials minted on request and shown once #74's closure of the raw-grant route.
  • The write-only TenantSecret alternative argued concretely rather than dismissed: what create, update and delete on tenantsecrets each let a tenant do in a namespace shared with operator secrets, and why fencing that in is building a secrets engine inside an aggregated API. The stopgap of granting those verbs is dropped; the interim for strict installations is design-proposal: user secrets API, credentials minted on request and shown once #74 admitting the inline form as Legacy for reference-only classes until the capability exists.
  • The no-Keycloak write path stated plainly: one ServiceAccount identity per tenant, audit names the tenant.

The phase-2 withdrawal is no longer recorded as done; the scope section says the comparison is what this review settles.

myasnikovdaniil added a commit that referenced this pull request Sep 28, 2026
…r-only storage

Rewrites both documents against the line agreed in the review thread. The
platform now generates a password on an explicit mint call, stores only the
engine's verifier where the engine accepts one, and returns the plaintext once
in the response body.

Removed: the five API kinds and the v1alpha2 move, one-time collection with its
entitlement, window and commitment protocol, the durable journal with its
transactional outbox, exporter and fail-closed threshold, the candidate state
machine with fencing and advisory locks, credential delivery with bindings, and
phase 2 for tenant-supplied secrets, which moves to #82.

Added: a verifier column per coverage row, a service-account split that takes
the platform's own credentials out of the tenant-facing Secret, one small record
per account carrying identity, issuance, initiating identity and applied state,
revocation as three distinct end states with an allowLogin reverse, and the lost
pre-publication check written down as a trade rather than argued away.

Per-engine mechanics now follow what the operators actually do. PostgreSQL moves
the password to CNPG managed.roles, because a Secret change starts no Helm
render and the init Job is a post-upgrade hook; deny-login lives in application
values because a patch on the live Cluster is overwritten and a manual NOLOGIN
is reverted. ClickHouse takes the verifier through a values fragment in
valuesFrom, because its operator watches no Secrets and patching the live CHI
loses a race that hands an account the operator's default password. MariaDB
keeps its native reference and gains the watch label, with its lock behaviour
and its missing applied state named as acceptance requirements. OpenSearch
leaves phase 1 entirely: a tenant-declared user reaches no engine today, which
is a defect to repair before the class can be converted.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
Andrey Kolkov (androndo) added a commit that referenced this pull request Oct 1, 2026
…erence

Folds the review consensus into the proposal: the destination becomes a
storageRef to a typed storage object (apps.cozystack.io/Bucket or a new
backups.cozystack.io/S3Storage) rather than an inline S3 struct in the core
types, keeping core out of storage semantics and giving restore and cleanup a
live anchor. Credentials move to the community#82 model instead of a tenant
write grant on TenantSecret. Records the settled calls in Decisions —
reference-over-inline, #82 credentials, per-strategy integrity opt-in with
single-artifact checksum verification and Velero kept off tenant-writable
storage, and failure-domain separation rather than a platform durability SLA —
and narrows Open questions to storage-object mutability, the durability story
the default relies on, the per-strategy declaration mechanism, enforcement, and
off-platform privileged-apply as a separate track.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <[email protected]>
Aleksei Sviridkin (lexfrei) added a commit to cozystack/cozystack that referenced this pull request Oct 8, 2026
…abel (#4766)

## What this PR does

This continues #4168 by @txmazing, which has been idle since 2026-09-10.
The first commit is the work from #4168 squashed into one, with
@txmazing as author. The other two commits constrain the new values and
complete the documentation.

Five changes to the managed OpenBAO app (`packages/apps/openbao`).

1. **Optional static-key auto-unseal** (#2787, #2483). `seal.type` is
`shamir`, the unchanged default, or `static`. With `static` the chart
mounts the Secret named in `seal.secretName` through the upstream
chart's `server.extraVolumes` and adds OpenBAO's `seal "static"` stanza
to the standalone and the Raft config, so the pods unseal themselves
after a restart. `seal.previousSecretName` and `seal.previousKeyId`
carry the n-1 key during a rotation. The chart never generates the key.
A cluster administrator creates the Secret in the tenant namespace,
because the `cozy:tenant:*` roles grant nothing on Secrets. A way for
tenants to supply secrets by reference is proposed separately in
cozystack/community#82.
2. **No silent seal change.** The template reads the seal in use from
the `<release>-config` ConfigMap and refuses a different `seal.type`
unless `seal.allowMigration: true` acknowledges a migration. Going from
`static` back to `shamir` is refused, because the chart cannot render
the disabled stanza that direction needs.
3. **Apiserver egress label** (#2793). The server pods carry
`policy.cozystack.io/allow-to-apiserver: "true"`, so
`service_registration "kubernetes"` reaches the API server and HA pods
no longer hang at start.
4. **Constrained values.** The upstream chart passes the server config
through `tpl`, and its start script replaces upper case placeholders
such as `HOSTNAME` in it. A key identifier with template delimiters
would run as a template inside helm-controller, which runs as
cluster-admin, so `seal.keyId` and `seal.previousKeyId` take lowercase
alphanumerics, hyphens, dots and underscores. The Secret names become
pod volume names `userconfig-<name>`, so they take DNS labels of at most
52 characters. A previous Secret or key identifier equal to the current
one fails at render. The apps API only defaults against the schema, and
Helm enforces these patterns at install and upgrade.
5. **README.** The trade-off of `static`, the key Secret, the rotation
with the pod replacement that the `OnDelete` strategy requires, the
migration from Shamir step by step after the OpenBAO guide, and the
one-time pod replacement after an upgrade so that existing pods get the
label.

### Validation

- `helm unittest .` in `packages/apps/openbao` passes 28 of 28 with
helm-unittest 1.1.2, the version CI pins, and with 1.2.1.
- Dropping the key identifier patterns, moving the Secret name limit to
51 or 53, or removing either equality guard turns the suite red each
time.
- Rendering this chart and then the vendored `openbao` chart with a
template in `seal.keyId` now stops at the schema. Without the pattern
the same two-stage render executed it, `lookup` included.
- With default values, standalone and with `replicas: 3`, `helm
template` renders the same as `main` apart from the six-line
`extraLabels` block.
- `make generate` in the package with cozyvalues-gen v1.7.0 and at the
repository root with Go 1.27.1 leaves the tree unchanged. `make
unit-tests` passes apart from seven BATS tests that fail the same way on
a clean `main` under macOS, most of them because macOS ships no
`timeout`. In `make test-controllers` only `internal/backupcontroller`
fails, because its tests do not build on current `main` either, which
#4768 fixes. `hack/check-commit-trailers.sh` passes and `cmd/api-gate`
reports no sizeable API change.
- Nothing here was run on a cluster. #4168 reports a Cozystack v1.6.3
cluster where a `replicas: 1` instance with `seal.type: static` came
back `Sealed false` after `kubectl delete pod`.

TLS between pods and an init or bootstrap job are not covered. Both are
on the #2787 list and independent of this change.

### Screenshots

No dashboard code change. The application form gains the `seal` group
from the regenerated `openAPISchema` and `keysOrder`.

### Downstream repositories

Walked the trigger map against the diff. The new `seal` field in
`packages/apps/openbao/values.schema.json` reaches the hand-written
`cozystack_openbao` resource of the Terraform provider. That is tracked
in cozystack/terraform-provider-cozystack#44, which also needs
`allowMigration` and the value constraints. The website reference page
is generated from this `README.md` for the apps listed in its
`Makefile`, and `openbao` is in `APPS`, so the page follows the next
stable tag without a follow-up. The website pages that mention the
apiserver label describe it in general terms and carry no OpenBAO
workaround. There is no `hack/` change, no platform values change, no
variant and no change to ApplicationDefinition semantics.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
cozystack/terraform-provider-cozystack#44
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(openbao): the OpenBAO app can auto-unseal with OpenBAO's static-key seal (`seal.type: static`) from a Secret that a cluster administrator creates in the tenant namespace, with n-1 rotation through `seal.previousSecretName` and `seal.previousKeyId`. Shamir stays the default, and changing the seal of an existing instance needs `seal.allowMigration`. Server pods now carry `policy.cozystack.io/allow-to-apiserver: "true"`, so HA instances no longer hang at start behind the apiserver egress policy.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* OpenBAO supports static sealing with keys stored in an existing
Secret; Shamir sealing remains the default.
* Configure current and previous keys to support key rotation in
standalone and high-availability deployments. Keep the previous key
until stored keys have been re-wrapped.
* Seal-type changes are validated. Shamir-to-static migration requires
explicit acknowledgement; static-to-Shamir migration is not supported.
* Static-seal settings validate required key details and Secret names.
Server pods receive a label needed by the configured API-server egress
policy.
* **Documentation**
* Added guidance for configuring static sealing, managing Secrets,
rotating keys, and migrating seal types.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant