Repository navigation
Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus #2653
Description
Activity
Audit results — 22
hack/e2e-apps/*.batsfilesWalked all 22 files (not ~18). 82 total
kubectl waitinvocations:Bucket Count KEEP — HelmRelease --for=condition=readygate24 KEEP — CR / operator-async (CRD-Established, CR conditions, functional Job/probe) 13 KEEP — teardown --for=delete11 DROP candidate — post-HR wait on a standard resource kstatus subsumes 23 The DROP-candidate count (23) matches the ~21 estimate. Total is 82 rather than ~36 because the estimate excluded all of
gateway.bats(13 waits) and every--for=deleteteardown wait.⚠️ Key finding: only 11 of the 23 are safe to drop on this issue's premiseA cozystack app HR's inventory is the CR, so
HR Ready == CR Ready. kstatus health-checks the CR's condition, not the operator-created StatefulSet/PVC downstream of it. Andetcd.bats:175already documents in-tree that the etcd operator flips its CR Ready early (Available=TrueatClusterDiscovered, before all members are up) — exactly why that test gates onreadyMembers=3rather than the HR. So a post-HR wait on an operator-created workload is not guaranteed redundant.Tier 1 — high-confidence DROP (chart renders the resource → in HR inventory → kstatus genuinely subsumes it):
kuberture.bats:90 deployment kuberture (controller) qdrant.bats:26,28 sts + pvc (no operator; chart-rendered sts) openbao.bats:59,61 sts + pvc (chart-rendered sts) harbor.bats:84,86,88 deploy core/registry/portal (chart Deployments) mariadb.bats:49,108 deployment mariadb-$name-metrics (confirm chart-rendered)Tier 2 — verify-first DROP (operator-created downstream of a chart CR; remove only after confirming the operator gates CR Ready on the workload):
mariadb.bats:45,104 sts (mariadb-operator) redis.bats:26,28,30 pvc/deploy/sts (spotahome) clickhouse.bats:44,48 sts (Altinity) mongodb.bats:40 sts (Percona) kafka.bats:49,51 pvc (Strimzi) postgres.bats:48 job init-job (only if in HR inventory, not a Helm hook)Tier 3 — redundant with a sibling wait, not with the HR:
vminstance.bats:33,68 pvc Bound (redundant with the preceding `dv --for=condition=ready`)Recommendation
Ship a first PR removing Tier 1 + the 2 Tier-3 vminstance PVCs (each with a one-line "chart-renders-it / redundant-with-dv" justification). Follow up on Tier 2 only after a live
describeconfirms each operator sets CR Ready after its rollout. Leave all 24 HR gates, 13 CR/functional waits, and 11 teardown deletes untouched.- changed the title
[-]Audit e2e bats post-HR-Ready waits for redundancy under Flux v2.8 kstatus[/-][+]Audit e2e Chainsaw post-HR-Ready asserts for redundancy under Flux v2.8 kstatus[/+]on Oct 1, 2026 Re-checked this against
main(8799a4e). App suites moved fromhack/e2e-apps/*.batsto Chainsaw inhack/e2e-chainsaw/(only two monitoring bats files left ine2e-apps), so the 82 waits and file:line references in the audit above don't point at anything anymore.New baseline is 23 suites, 56 Test documents and 312
tryoperations (counted per document withyq): 184 assert, 66 script, 41 apply, 14 error, 7 delete. 34 of the asserts are on HelmRelease, 33 wait forReady=Trueand one expectsReady=False(redis TLS flip rejection). Imperative waits left in suite YAML are only in gateway, 7until kubectlloops and 1kubectl wait.Candidates for this issue are 72 asserts on Deployment, StatefulSet, PVC, Pod, Job, Service and Endpoints, 69 of them after an HR assert (gateway and securitygroup have none). Service ports and Endpoints (25) plus Job and Pod (7) check shape or function that kstatus doesn't see, so they stay. The other 37 are workload readiness waits:
- 10 in a nested
-systemHR: external-dns Deployment (2), harbor Deployment (3), kuberture Deployment, qdrant sts and PVC, openbao sts and PVC - 22 operator-created from a CR: mariadb (4), redis (7), valkey (4), clickhouse (3), mongodb, kafka PVC (2), rabbitmq
- 2 vminstance PVC asserts that follow a DataVolume
Ready=Trueassert in the same step - 3 that check rollout state and not readiness (opensearch sts twice, second assert in redis-tls-flip-rejected), these stay
qdrant, openbao and external-dns charts render a nested HelmRelease and no workload (checked with
helm template, nodisableWaiton it), so they are redundant through the nested HR's own health check, not because the chart renders them. I didn't confirm on a live cluster that the parent HR waits for the nested one, this is the hop to check before dropping anything. Harbor doesn't render with default values and kuberture is a system package, didn't check those two.mariadb metrics Deployment was Tier 1 pending a chart check and it is not chart-rendered (chart has only the
MariaDBCR), so it moves to the operator group. Also mariadb sts, mariadb metrics Deployment and clickhouse replica sts assertstatus.replicas, that is pod count and not readiness, so dropping them loses an existence check.First PR then is the 10 nested-HR asserts plus the 2 vminstance PVC asserts, after the live check. Operator group only after
describeon each, same as before.- 10 in a nested
Checked the nested HR hop on a dev cluster (helm-controller v1.5.0). Installed a Qdrant app with a non-existent
storageClass, so the PVC staysPendingand the nested-systemHR can't become Ready.Parent HR stayed
Ready=Unknown/Progressingfor the whole 10m install timeout, then failed withtimeout waiting for: [HelmRelease/tenant-root/qdrant-nested-wait-probe-system status: 'InProgress']. Nested HR was cycling betweenProgressingandInstallFailed(failed early due to stalled resources: StatefulSet ... Failed) the whole time. The message lists only the nested HR, so that is what the parent waits on, and on a normal install parent HR Ready covers the nested HR's StatefulSet and PVC.openbao and external-dns render the same nested HR and no workload (
helm template), I didn't run them. harbor and kuberture are still unchecked. Upgrade path is not covered (9 of 101 upgraded HRs on that cluster have emptystatus.inventory, didn't check why), e2e only installs so it doesn't matter here.Correction to what I wrote here first: the qdrant sts and PVC asserts can't just go. The sts assert also pins the
app.kubernetes.io/instancelabel that the catch selector relies on, and the PVC assert is what shows the PVC existed before the reclaim-on-delete check looks for its absence (same in kafka). For qdrant onlyreadyReplicas: 1is redundant.First step is up: #4652. It drops the
readyReplicaschecks in qdrant and external-dns (the instance-label pins stay) and the two vminstance PVCBoundasserts. The three suites pass on CI.Correction to the baseline comment above: openbao sts and PVC asserts don't belong in the redundant group. They run after the init and unseal step, and that step also carries the teardown
finally, so they stay. The numbers there are from 8799a4e, several suites changed onmainsince and akafka-metadatasuite was added.What is left here: the operator-created workloads (mariadb, redis, valkey, clickhouse, mongodb, kafka, rabbitmq), each after a live
describe, and harbor and kuberture, where I haven't checked the nested HR.- added a commit that references this issue
on Oct 2, 2026
Flux helm-controller v1.5.0 (Flux v2.8) uses kstatus-based health checking by default. For HRs whose chart applies standard k8s resources (Deployment, StatefulSet, Service, PVC), HR Ready=True already implies the resource is rollout-ready. Several e2e Chainsaw suites in
hack/e2e-chainsaw/still have explicit asserts on those same resources after the HR Ready assert, and they may be redundant now.Scope is asserts that only wait for readiness of Deployment, StatefulSet and PVC (
readyReplicas,Available,Bound). Asserts on Service ports, Endpoints, Jobs and Pods, and asserts that check rollout state (revision, mounted volumes), check shape or function that kstatus doesn't cover, so they stay.CR-based apps (mariadb, postgres CNPG, kafka Strimzi, etc.) likely still need the asserts, because kstatus sees only the CR condition, not the operator-created workload behind it. Apps that render a nested
-systemHelmRelease are covered only through that nested HR, so check that the parent really waits for it before dropping anything.Triage: walk each
hack/e2e-chainsaw/*/chainsaw-test.yamland decide per assert whether it is still meaningful or covered by kstatus. The suites were ported fromhack/e2e-apps/*.bats, the first audit of those files is in the first comment. Surfaced from PR #2619.