Activity

1. dosubot

   added

   kind/featureCategorizes issue or PR as related to a new featureCategorizes issue or PR as related to a new feature

   on Jan 30, 2026
2. Aleksei Sviridkin (lexfrei)

   changed the title [-]feat: Replace Helm lookup with Vault + ESO + Operator architecture[/-] [+]feat: Replace Helm lookup with OpenBao + ESO + Operator architecture[/+] on Feb 1, 2026
3. 2mind commented on Feb 11, 2026

   

   Nikita Aboltin (2mind)

   on Feb 11, 2026

   More actions

   Aleksei Sviridkin (@lexfrei) Thanks for working on this! I have a question about the proposed architecture -does cozystack-controller really need to generate and write secrets directly to OpenBao?

   Why this feels a bit off:

   • Duplicates logic - ESO already knows how to generate secrets
   • Makes OpenBao mandatory, when it's mostly needed for audit/migration

   Alternative: use ESO Generators

   1. Password generator (declarative)

   apiVersion: generators.external-secrets.io/v1alpha1
   kind: Password
   metadata:
     name: app-password
   spec:
     length: 32
   

   2. ExternalSecret references the generator

   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   spec:
     dataFrom:
     - sourceRef:
         generatorRef:
           kind: Password
           name: app-password
   

   • https://external-secrets.io/latest/guides/targeting-custom-resources/

   Simplified architecture:

   Cozystack → creates Generator + ExternalSecret
                             ↓
                 ESO generates the password
                             ↓
                 K8s Secret for the app
                             ↓
            OpenBao (optional, via PushSecret)
   

   Benefits:
   Fully declarative
   OpenBao becomes optional (can be enabled later)
   Less code in operator
   Leverages ESO which is already there

   👍React with 👍3Reacted by Aleksei Sviridkin, YohanS and Ray Low
4. github-actions commented on May 2, 2026

   

   github-actionsbot

   on May 2, 2026 – with GitHub Actions

   More actions

   This issue has had no activity for 60 days and was marked lifecycle/stale.
   It will be closed in 14 days unless commented or labelled lifecycle/frozen.
5. github-actions

   added

   lifecycle/staleDenotes an issue or PR has remained open with no activity and has become staleDenotes an issue or PR has remained open with no activity and has become stale

   on May 2, 2026
6. myasnikovdaniil commented on Jun 2, 2026

   

   myasnikovdaniil

   on Jun 2, 2026

   Contributor

   More actions

   Reviving this — it's the right north star, and the platform has drifted enough since January that it's worth refreshing the scope. I did a full audit of how credentials are handled across every package today, and I want to reframe the goal slightly around two user-facing properties that matter more than the mechanism (lookup → OpenBao) we use to get there:

   1. Zero-config by default — every stateful app auto-generates its credentials into a Secret on first install, so quick-start / easy-setup just works with no input.
   2. Overridable per app — anyone can pin a specific password without committing plaintext to git, i.e. by referencing a pre-existing Secret they created.

   A single, systemwide OpenBao wired into the cluster is then the backend that makes #2 secure and adds audit/rotation — one platform-managed instance every tenant app can draw from, rather than per-app credential plumbing — but it should be optional, layered on top of a contract that already holds without it. That also resolves the design tension Nikita Aboltin (@2mind) raised in the comment above (don't make OpenBao mandatory; lean on ESO Generators) — see "Backend" below.

   ---

   Current state — the inconsistency is real, and it's the actual problem

   Today there is no shared contract. Each package reinvents credential handling, and the differences are user-visible. Audit of packages/apps/* and packages/system/*:

   Auto-generation: consistent mechanism, inconsistent everything else

   All generation uses the same lookup "v1" "Secret" + randAlphaNum idiom (so upgrades don't rotate — good), but:

   Axis	Reality across packages
   Password length	16 (most), 32 (redis, grafana, opensearch admin, dashboard client), 24 (clickhouse backup api), 10 (seaweedfs jwt) — no standard
   Secret shape	shared <release>-credentials (postgres, mariadb, clickhouse, nats) vs per-user <release>-user-<name> (mongodb, opensearch) vs <release>-auth (redis) vs multi-secret (harbor)
   User override	9 packages expose users.<name>.password; redis exposes only authEnabled: bool — no user-supplied password, no users at all
   Reference an existing Secret	No package supports this. `grep -riE "existingSecret
   Surfacing to UI	via ApplicationDefinition secrets.include selectors in *-rd/cozyrds/*.yaml — also inconsistent (mongodb per-user secrets aren't selected at all, #2678 defect 2)

   The override gap is the sharp edge

   The only way to set a specific password today is inline:

   users:
     app:
       password: "hunter2"   # plaintext, ends up in git / HelmRelease values

   There is no GitOps-safe override — no passwordFromSecret: {name, key} that points at a Secret the user pre-created (or that ESO/OpenBao populated). That single missing field is what blocks property #2 above, independent of whether OpenBao exists.

   Concrete symptoms already filed against this design

   These are all instances of "no shared credential contract", and should be tracked as children of this issue:

   • bug(mongodb): dashboard shows no usable MongoDB credentials on fresh install #2678 — mongodb dashboard shows empty/invisible credentials on fresh install (cross-controller lookup race + per-user Secret not selected for the dashboard)
   • OpenBAO: can't init/unseal without kubectl exec #2483 — OpenBao can't init/unseal without kubectl exec (blocks OpenBao-as-default; the auto-unseal "Open Question 2" below)
   • bug(api): TenantSecrets API ignores labelSelector parameter #2527 — TenantSecrets API ignores labelSelector (surfacing layer is buggy)
   • [harbor] Default secretKey "not-a-secure-key" leaves every stored credential decryptable #2736 / [harbor] Default registry basic-auth credentials (harbor_registry_user/password) bypass Harbor RBAC #2738 / [harbor] Pinned at v2.14.2 — vulnerable to GHSA-prh4-vhfh-24mj (LDAP/OIDC secrets in audit log) #2739 — harbor ships insecure default secrets / basic-auth that bypass RBAC (what a generated-by-default contract would have prevented)
   • Garbage-collect projected cozy-backups-creds Secrets via ownerReferences #2734 / Replace system-namespace projection tick with a watch on the source Secret #2735 — backup-creds Secret projection lifecycle (related secret-distribution plumbing)

   ---

   Proposed: a shared credential contract first, OpenBao backend second

   Layer 1 — the contract (ships without OpenBao)

   Define one helper in cozy-lib that every stateful chart uses, exposing a uniform values block:

   ## @param credentials.generate - auto-generate any credential not explicitly provided (default true → zero-config)
   ## @param credentials.length   - generated length (platform default, e.g. 24)
   ## @param users.<name>.password           - inline override (discouraged; plaintext)
   ## @param users.<name>.passwordFromSecret  - GitOps-safe override: {name, key}
   credentials:
     generate: true
     length: 24
   users:
     app:
       passwordFromSecret:
         name: my-prepared-secret
         key: password

   Resolution order per credential: passwordFromSecret → inline password → existing generated Secret (lookup) → generate. Redis joins the same contract (gains users / override instead of bare authEnabled). Standardize the output Secret shape and the ApplicationDefinition secrets.include selector so the dashboard surfacing is uniform (fixes the #2678 class).

   This alone delivers both user goals and is independently shippable + testable.

   Layer 2 — optional OpenBao + ESO backend

   With the contract in place, passwordFromSecret can be satisfied by a Secret that ESO materializes from OpenBao, and generation can move from Helm randAlphaNum to an ESO Password Generator + ExternalSecret (per Nikita Aboltin (@2mind)'s proposal) rather than the controller writing to OpenBao directly. That keeps OpenBao optional and the operator thin:

   chart → Generator + ExternalSecret  → ESO generates → K8s Secret (app consumes)
                                             ↘ (optional) PushSecret → OpenBao (audit/rotation/migration)
   

   The packages already exist and are wired into the platform bundle:

   • packages/system/openbao + packages/apps/openbao (registered: core/platform/sources/openbao-application.yaml, paas.yaml bundle)
   • packages/system/external-secrets-operator (registered as optional in system.yaml bundle)

   So Layer 2 is "wire + configure", not "introduce".

   Why this actually removes the lookup problem (and the one caveat)

   It's worth being precise, because the existing-value check doesn't disappear — it relocates. The problem with lookup was never "checking existing secrets"; it's that Helm does the check at template-render time, and render must be a pure, deterministic function for GitOps to work.

   Today the chart reads live state and emits the secret value:

   {{- $existing := lookup "v1" "Secret" .Release.Namespace "app-credentials" }}
   {{- $pw := randAlphaNum 24 }}
   {{- if $existing }}{{- $pw = index $existing.data "password" | b64dec }}{{- end }}
   stringData:
     password: {{ $pw }}        # ← the rendered manifest contains the actual value
   

   Three failures follow, all from reading cluster state during render:

   1. helm template / ArgoCD dry-run returns nil for lookup → renders a freshly generated password that differs from what's deployed → perpetual diff, or ArgoCD overwrites the real one. The core GitOps incompatibility.
   2. Charts can't be unit-tested — helm template produces a throwaway value.
   3. If another operator creates the secret asynchronously (Percona/Kamaji), the lookup is nil on first install → empty secret. That is exactly the bug(mongodb): dashboard shows no usable MongoDB credentials on fresh install #2678 mongodb bug.

   With ESO the chart emits a static reference, not a value:

   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   spec:
     refreshInterval: "0"                 # ← load-bearing: generate once, never rotate underneath
     target: { name: app-credentials }
     dataFrom:
       - sourceRef: { generatorRef: { kind: Password, name: app-pw } }

   This manifest is byte-identical on every render — no lookup, no randAlphaNum, nothing environment-dependent — so helm template is deterministic, ArgoCD sees no drift, unit tests pass. The "preserve the existing value" job moves out of render and into a reconcile loop, where reading live state is exactly what a controller is supposed to do:

   • ESO Password generator: ESO generates once, writes the target Secret; with refreshInterval: "0" later reconciles see it exists and leave it.
   • OpenBao backend: the value's source of truth is OpenBao; ESO just syncs it down, so re-syncing re-reads the same value and the cross-controller race (bug(mongodb): dashboard shows no usable MongoDB credentials on fresh install #2678) is gone — one authoritative source instead of "whoever lookup'd first."

   	lookup (today)	ESO
   Who preserves the value	Helm template, at render	a controller, at reconcile
   What the chart emits	the secret value	a static reference
   helm template / ArgoCD	broken (nil → drift)	deterministic
   Async cross-controller secret	races → empty (#2678)	single source of truth

   Caveat — refreshInterval: "0" is load-bearing. With a non-zero interval, ESO re-runs the Password generator on each refresh and overwrites the live password — worse than lookup. The contract must pin 0 for generated credentials, and that should be enforced in the cozy-lib helper, not left to each chart author.

   This only cleanly solves the secret-generation lookups. Non-secret lookups (tenant UID, VM/MachineTemplate spec preservation) stay — out of scope here.

   ---

   Refreshed plan

   • P0 — Contract: cozy-lib credential helper (resolution order above); add passwordFromSecret; standardize length + Secret shape + AppDef secrets.include. Migrate redis to the contract. Land symptom fixes bug(mongodb): dashboard shows no usable MongoDB credentials on fresh install #2678.
   • P1 — ESO generation: swap Helm randAlphaNum for ESO Password Generator + ExternalSecret behind a platform flag (the cozy-lib helper pins refreshInterval: "0" — see caveat above); keep lookup fallback when ESO is absent (GitOps/ArgoCD compatibility — the original P1 win).
   • P2 — OpenBao backend (optional): ClusterSecretStore → OpenBao; resolve auto-unseal (OpenBAO: can't init/unseal without kubectl exec #2483) before it can be a default; PushSecret for audit/rotation.
   • P3 — Cleanup & migration: remove secret-generation lookups, migration guide, docs. (Non-secret lookup uses — tenant uid, VM/kubernetes spec preservation — stay out of scope; this issue is credentials only.)

   Open questions (refined)

   1. Default backend: ship the contract with Helm/ESO generation by default and OpenBao strictly opt-in? (I lean yes — OpenBAO: can't init/unseal without kubectl exec #2483 makes OpenBao-by-default a non-starter until auto-unseal lands.)
   2. Auto-unseal: transit-seal via a tiny bootstrap OpenBao, or cloud-KMS where available? Tracked in OpenBAO: can't init/unseal without kubectl exec #2483.
   3. Migration: for existing installs, do we import current Secret values into OpenBao, or treat the K8s Secret as source-of-truth and only PushSecret forward?
   4. Rotation UX: is rotation operator-driven (annotation/CR) or dashboard-button? Out of P0 but shapes the contract.

   cc Aleksei Sviridkin (@lexfrei) Nikita Aboltin (@2mind) — proposing we keep this as the canonical tracking issue and label it lifecycle/frozen so it stops aging out.
7. myasnikovdaniil

   added this to Cozystack Roadmapon Jun 2, 2026
8. myasnikovdaniil

   moved this to Todo in Cozystack Roadmapon Jun 2, 2026
9. myasnikovdaniil

   removed

   lifecycle/staleDenotes an issue or PR has remained open with no activity and has become staleDenotes an issue or PR has remained open with no activity and has become stale

   on Jun 8, 2026
10. github-actions

    added

    triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

    on Aug 12, 2026
11. Aleksei Sviridkin (lexfrei)

    added

    area/platformIssues or PRs related to platform infrastructure (bundle, flux, talos, installer)Issues or PRs related to platform infrastructure (bundle, flux, talos, installer)

    triage/acceptedIndicates an issue is ready to be actively worked onIndicates an issue is ready to be actively worked on

    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completeImportant over the long term, but may not be staffed and/or may need multiple releases to complete

    lifecycle/frozenIndicates that an issue or PR should not be auto-closed due to stalenessIndicates that an issue or PR should not be auto-closed due to staleness

    and removed

    triage/needs-triageIndicates an issue needs triage by a maintainerIndicates an issue needs triage by a maintainer

    on Sep 23, 2026