Make managed OpenBAO production-mature: auto-unseal, TLS, init/bootstrap, guardrails #2787

Summary

The managed OpenBAO app (packages/apps/openbao) and the system OpenBAO
(packages/system/openbao) are missing the pieces needed to run OpenBAO safely in
production. Per the OpenBAO Helm guide,
a production deployment needs end-to-end TLS, an unseal strategy, and a bootstrap/init
flow. Today we have none of these.

Current gaps

1. No auto-unseal / no init automation. Tenant instances come up sealed and
   uninitialized. The readiness probe checks seal status, so pods stay NotReady
   indefinitely until a human runs bao operator init + bao operator unseal, and they
   re-seal on every restart/reschedule. The only place init/unseal happens is the e2e test
   (hack/e2e-apps/openbao.bats), by hand.
2. TLS hard-disabled everywhere. global.tlsDisable: true, listener tls_disable = true
   on 8200 and cluster_address 8201, Raft retry_join over http://. So client traffic
   and inter-node Raft replication are plaintext. external: true publishes a plaintext
   Vault API via LoadBalancer.
3. System OpenBAO is a dormant stub. packages/system/openbao only sets
   injector.enabled: false / csi.enabled: false; no server runs. It's wired into the
   PaaS bundle but cannot serve as anything yet.
4. Misleading defaults / unguarded destructive op. Default replicas: 1 uses
   non-production storage "file"; changing replicas across the 1↔N boundary silently
   switches the storage backend (file↔raft) with no migration and no guard.
5. No audit logging option exposed.

Proposed direction

Bring the system OpenBAO to life as a central, platform-wide HA instance exposing a
transit secrets engine. Tenant OpenBAO instances use seal "transit" against it, so
they auto-unseal on every restart and need only a one-time init. This is the canonical
mature pattern and matches the system stub's evident intent.

Phase 1 — Central platform OpenBAO (packages/system/openbao, wrapper only)

• Enable HA (3 replicas) + Raft + TLS (cert-manager self-signed CA→leaf, modeled on
  packages/apps/qdrant/templates/certmanager.yaml); TLS on 8200 + cluster 8201, retry_join
  over https://.
• Solve the central instance's own unseal on bare metal (no guaranteed KMS): Shamir seal +
  an idempotent one-time init Job (keys stored in a Secret) + a long-running unsealer that
  re-unseals all pods on restart (bank-vaults style). Optional KMS seal for cloud installs.
• Enable the transit engine + Kubernetes auth + a base per-tenant policy.

Phase 2 — Per-tenant transit provisioning (internal/operator)

• A reactive reconciler creates a per-tenant transit key + scoped credential and delivers a
  <name>-transit-unseal Secret into the tenant namespace, owner-referenced for GC.

Phase 3 — Tenant OpenBAO maturity (packages/apps/openbao)

• seal "transit" stanza in both standalone + HA configs, consuming <name>-transit-unseal.
• Internal TLS via cert-manager (listener + Raft + retry_join over https); surface CA.
• Idempotent auto-init Job → store root token + recovery keys in <name>-credentials
  (modeled on the postgres/mariadb/mongodb patterns).
• Guardrails (helm fail): refuse external: true while TLS disabled; refuse
  standalone↔HA backend switch (detect via existing config).
• Optional audit logging (auditStorage + bao audit enable file).
• Update the tenant-facing schema (packages/system/openbao-rd/cozyrds/openbao.yaml) and
  rewrite the e2e test to assert self-init + auto-unseal instead of manual steps.

Key risks

• The central root token + unseal keys live in a Kubernetes Secret (etcd) — unavoidable for
  full automation without KMS; lock down RBAC and document it.
• Init-Job vs Helm ownership of the credentials Secret must use split ownership + lookup
  preservation or the root token is lost on upgrade.
• Verify the vendored OpenBAO version supports ${env "VAR"} interpolation / Kubernetes
  auth in the seal "transit" stanza — it determines whether the operator must manage
  token renewal.
• Transit-Secret delivery must precede (or race-tolerate) tenant pod start.

References

• OpenBAO Helm run guide: https://openbao.org/docs/platform/k8s/helm/run/
• Existing patterns to reuse: packages/apps/qdrant/templates/certmanager.yaml (TLS),
  packages/apps/postgres/templates/init-{script,job}.yaml (bootstrap Job),
  packages/apps/mongodb/templates/credentials.yaml (credential surfacing).