[Epic] Adopt the kstatus readiness model platform-wide (Flux 2.8 / helm-controller v1.5.0) #2642

Epic. Flux 2.8 (helm-controller v1.5.0) flips HelmRelease health assessment to the kstatus model platform-wide. The version bump landed in #2602 (merged), so this epic tracks the consequential work: an API change to ComponentInstall/ApplicationDefinition (healthCheckExprs, waitStrategy) plus per-package adoption across the three readiness classes below.

Status as of 2026-08-13

The plumbing is finished and adoption has barely begun. Task 1 landed in #2602 via Option B, and the typed API plus its threading through both HR builders landed in #3273, but no package on origin/main sets waitStrategy or healthCheckExprs through that typed API, so it is a live and entirely unconsumed surface. Only two HelmRelease templates carry a kstatus gate anywhere in the tree, packages/extra/seaweedfs/templates/seaweedfs-db.yaml and packages/extra/monitoring/templates/helmrelease.yaml, and both got there through unrelated PRs rather than through this epic. Phase 1 of the template work (#3264) is a stalled draft: CHANGES_REQUESTED since 2026-07-20 over dead failed predicates, with five unticked dev-cluster verification boxes and an E2E run that never reached the changed code. Two corrections to the original plan are recorded in Task 2 and Task 5 below. The "add healthCheckExprs to all 12 umbrellas" framing is wrong for umbrellas that wrap only native workloads, and packages/apps/kubernetes plus packages/apps/kubernetes-nodes are structurally excluded from the model as long as they carry release.cozystack.io/helm-install-disable-wait.

Context

PR #2602 (feat(flux): upgrade to v2.8.0 — tenant chart v0.48.0 + management-cluster embed, web UI opt-in) upgrades Flux from v2.7.3 → v2.8.0 across both planes:

• tenant clusters via flux-operator / flux-instance charts v0.33.0 → v0.48.0 (now at 0.50.0, still tracking distribution.version: 2.8.x in packages/system/fluxcd/values.yaml).
• management cluster via the embedded internal/fluxinstall/manifests/fluxcd.yaml regenerated from the flux-aio timoni bundle (helm-controller:v1.5.0, internal/fluxinstall/manifests/fluxcd.yaml:8097).

In particular helm-controller goes v1.4.3 → v1.5.0. Beyond the version bump, v1.5.0 flips two defaults to align with Helm v4 (verbatim from the v1.5.0 CHANGELOG):

The controller now uses Helm v4, and, with this change, new default behaviors are being introduced (breaking changes) to keep Flux and Helm aligned:

• Apply method is now defaulting to server-side apply for new HelmReleases.

• Health checks now default to using kstatus for assessing readiness and failures of applied resources.

waitStrategy.name now defaults to poller (kstatus polling of every applied resource, including managed HelmReleases), and can be reverted to legacy (Helm v3 wait, core kinds only) via the UseHelm3Defaults controller feature gate. Cozystack does not enable that gate.

The kstatus default makes HelmRelease.Ready mean "every applied resource is healthy" instead of "helm install returned". That is correct for the end-to-end semantics we want between packages, spec.dependsOn becomes a real readiness gate instead of an ordering hint, but it interacts badly with existing umbrella patterns and is currently unreachable for leaf packages.

Symptoms

After applying #2602 on dev10, tenant-root/kubernetes-test HR enters Reconciling and its child HRs report DependencyNotReady. Cycle shape: parent waits on each child via kstatus → each child has dependsOn: [parent] → deadlock. Same shape (different chains) already fixed in 60aecd5f (vpa vpa-for-vpa nested HR) and unrelated yaml-merge bug in a23b5d27 (seaweedfs dependsOn collision). Retrospective write-up in docs/release.md:565-571.

Tasks

1. Unblock the Flux 2.8 bump (Class 3, single package), DONE

packages/apps/kubernetes is the only package in the monorepo whose child HRs carry dependsOn: [{Release.Name}] (guarded by a lookup). Pick one:

• Option A, operator-side knob. Extend internal/operator/package_reconciler.go::buildHelmReleaseSpec to set spec.waitStrategy.name: legacy (or spec.install.disableWait: true + spec.upgrade.disableWait: true) on the parent HR for cozystack.kubernetes-application. Not taken, superseded by Option B.
• Option B, child-side, remove the guard. Delete the {{- if lookup … Release.Name }} block from the 19 child HR templates in packages/apps/kubernetes/templates/helmreleases/. Landed in feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) #2602 (f6c737983). No child HR references the bare parent any more, all remaining dependsOn entries point at siblings, and packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml:93 keeps the comment explaining why no render-time guard is wired.

2. Class 2 umbrellas adopt healthCheckExprs (template edits only)

Correction (#3264 readiness audit, unmerged, needs sign-off). The original "add healthCheckExprs to every umbrella" framing is wrong. kstatus already assesses Deployments, StatefulSets and child HelmReleases natively, and a built-in workload has no empty-status race to close, so an expression on such an umbrella is a no-op. A gate is required only where the HR's manifest is a custom resource. Rows below are annotated with that verdict, and the 8 marked no-op are candidates for closing rather than pending work.

• packages/apps/openbao (inner HR → OpenBao CR), no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/apps/qdrant, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/apps/bucket, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/apps/nats, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/apps/harbor, needed (CNPG Cluster), drafted in feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264, blocked on a dead failed predicate. The bundled RedisFailover has no status subresource and cannot be gated.
• packages/apps/tenant, the package now has 7 HR children, not the 5 originally listed (computeplane, etcd, gateway, info, ingress, monitoring, seaweedfs). Needed for etcd (EtcdCluster) and gateway (TenantGateway), both drafted in feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264, no-op for the other five.
• packages/extra/external-dns, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/extra/ingress, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/extra/monitoring, landed in fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName #3215 (f9afc7309): packages/extra/monitoring/templates/helmrelease.yaml:30 sets waitStrategy.name: poller, :51 gates on VLCluster.status.updateStatus, field pins in packages/extra/monitoring/tests/helmrelease_test.yaml:22-43. feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 would extend it to VMCluster/VMAgent/VMAlert/VMAlertmanager, Grafana and the backing CNPG Cluster.
• packages/extra/seaweedfs, landed in refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system #2601 (29c6afc84): packages/extra/seaweedfs/templates/seaweedfs-db.yaml:24-25 sets waitStrategy.name: poller, :30-35 gates on the CNPG Cluster. The sibling -system child (packages/extra/seaweedfs/templates/seaweedfs.yaml:195) is intentionally ungated (native workloads). See Task 5 for the contested failed predicate.
• packages/system/bootbox, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit
• packages/system/cozystack-basics, no-op per feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 audit

3. Class 1 leaves, operator support for per-package knobs

These packages render the underlying CR directly, the wrapping HR is built in Go. Adopting Model B requires plumbing new fields through the operator and through the apiserver. The epic originally named only the operator, but every catalog app's HR is built by pkg/registry/apps/application/rest.go::convertApplicationToHelmRelease.

• Extend cozystack.io/v1alpha1.ComponentInstall (or Package) in api/v1alpha1/ with healthCheckExprs, optionally waitStrategy and disableWait. Landed in feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273: api/v1alpha1/packagesource_types.go:154,164 on ComponentInstall and api/v1alpha1/applicationdefinitions_types.go:97,111 on ApplicationDefinitionRelease, both typed as kustomize.CustomHealthCheck and gated by a poller;legacy enum. Prerequisite go.mod bump of helm-controller/api v1.4.3 → v1.5.1 landed in the same PR (83f0d3bc9, go.mod:11). disableWait was not added, the pre-existing release.cozystack.io/helm-install-disable-wait annotation (pkg/config/config.go:59) already covers it.
• Thread them into internal/operator/package_reconciler.go::buildHelmReleaseSpec. Landed in feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273 at internal/operator/package_reconciler.go:100-106, plus the apiserver builder at pkg/registry/apps/application/rest.go:1645-1646 fed from pkg/cmd/server/start.go:274-275. Shared coupling helper config.ResolveWaitStrategy (pkg/config/config.go:220-228) defaults to poller whenever expressions are set without an explicit strategy. Tests: internal/operator/package_reconciler_test.go:164, pkg/registry/apps/application/rest_helmrelease_spec_test.go:140, pkg/config/waitstrategy_test.go:30.
• Adopt per-package via values. Zero packages consume the typed API today. No ApplicationDefinition in packages/system/*-rd/cozyrds/ and no PackageSource in packages/core/platform/sources/ sets either field. Candidate leaves and their natural readiness signals:
  • packages/apps/postgres → CNPG Cluster.status.phase == 'Cluster in healthy state'
  • packages/apps/kafka → Strimzi Kafka Ready condition
  • packages/apps/mongodb → Percona PerconaServerMongoDB.status.state
  • packages/apps/mariadb → mariadb-operator MariaDB Ready
  • packages/apps/clickhouse → ClickHouseInstallation.status.status
  • packages/apps/redis, rabbitmq, foundationdb, opensearch (valkey dropped, no valkey-rd package exists on main)
  • packages/apps/vm-instance, vm-disk → KubeVirt VirtualMachine.status.printableStatus

4. Out of scope (explicit)

• Do not enable --feature-gates=UseHelm3Defaults=true on helm-controller. The global flip would undo every Model B gain from tasks 2 and 3 and force the project back onto Helm v3 wait semantics. Verified holding: UseHelm3Defaults appears on main only inside CRD description text in internal/fluxinstall/manifests/fluxcd.yaml, and the only feature gates actually set there are ExternalArtifact=true (:8014, :8079).

5. Follow-ups discovered after the epic was written

• The merged CNPG failed predicate in packages/extra/seaweedfs/templates/seaweedfs-db.yaml:34 may be dead code, tracked in its own issue, Verify whether the seaweedfs-db CNPG failed health-check predicate is reachable #3807. The EtcdCluster status.brokenMembers > 0 predicate flagged by the same review stays inside feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264's scope, since that predicate is not merged.
• Decide what to do about helm-install-disable-wait. packages/apps/kubernetes and packages/apps/kubernetes-nodes carry release.cozystack.io/helm-install-disable-wait: "true" (packages/system/kubernetes-rd/cozyrds/kubernetes.yaml:28, packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml:16) to break a real chicken-and-egg with the talos-reconcile Job. Upstream evaluates healthCheckExprs only when the Helm action has wait enabled, so any gate on those two kinds is a silent no-op (documented at api/v1alpha1/applicationdefinitions_types.go:105-109). Either accept them as permanently outside the kstatus model or find a wait-compatible way to break the cycle.
• Document the readiness contract for package authors. git grep -il kstatus -- docs/ hits only docs/changelogs/v1.5.0.md and the retrospective at docs/release.md:565-571. There is no guidance on when a gate is required, how to write the three predicates, or that disableWait defeats them.
• [API proposal] Typed healthCheckExprs / waitStrategy on ApplicationDefinition and ComponentInstall (kstatus readiness, #2642) #3263 was closed as implemented by feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273. The API-proposal issue asked for typed healthCheckExprs / waitStrategy on ApplicationDefinition and ComponentInstall. That surface and its threading landed in #3273, so anyone arriving from a link to [API proposal] Typed healthCheckExprs / waitStrategy on ApplicationDefinition and ComponentInstall (kstatus readiness, #2642) #3263 should read feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273 instead.

References

• PR feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) #2602: feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) #2602
• PR refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system #2601 (seaweedfs-db reference gate): refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system #2601
• PR fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName #3215 (monitoring-system reference gate): fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName #3215
• PR feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273 (typed API + threading): feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273
• PR feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264 (Phase 1 template gates, blocked draft): feat(flux): kstatus health gates on CR-wrapping umbrella HelmReleases #3264
• Issue [API proposal] Typed healthCheckExprs / waitStrategy on ApplicationDefinition and ComponentInstall (kstatus readiness, #2642) #3263 (API proposal, implemented by feat(api): typed waitStrategy/healthCheckExprs for kstatus readiness (#2642) #3273): [API proposal] Typed healthCheckExprs / waitStrategy on ApplicationDefinition and ComponentInstall (kstatus readiness, #2642) #3263
• helm-controller v1.5.0 CHANGELOG: https://github.com/fluxcd/helm-controller/blob/v1.5.0/CHANGELOG.md
• HelmRelease API spec v2: https://github.com/fluxcd/helm-controller/blob/v1.5.0/docs/spec/v2/helmreleases.md
• Flux 2.8.0 release notes: https://github.com/fluxcd/flux2/releases/tag/v2.8.0
• CEL health-checks cheatsheet: https://fluxcd.io/flux/cheatsheets/cel-healthchecks/