Repository navigation
testing: 14 test cases in 7 suites pin documents by index with only negative asserts, so they cannot see index drift #3634
Description
Summary
14 test cases across 7 helm-unittest suites address a document by documentIndex and then make only negative assertions (notExists / notContains) about it. That combination cannot notice when the index stops pointing at the intended document: a negative assert is satisfied by any document that merely lacks the forbidden content, so inserting or reordering documents in the template leaves the whole case green while the property it names goes unguarded. This is the exact shape just measured and fixed in the harbor suites, where inserting one document at the top of the template left two of four jobservice cases green for the wrong reason, and in the seaweedfs cleanup suite, where the merge-base tests stayed fully green while the Client role held delete on another release's Postgres cluster.
The catalog
Pure form, documentIndex plus negatives only, no positive anchor: packages/apps/mongodb (4 cases), packages/apps/nats certmanager (3), packages/apps/opensearch (3), packages/apps/qdrant certmanager (2), packages/apps/kubernetes cluster (1), packages/system/seaweedfs cluster_scoped_names (1).
Wider context: 19 suites mix documentIndex with negative assertions, 6 suites carry no hasDocuments count at all. The nearest twin of the already-fixed harbor pair is packages/apps/bucket/tests/cleanup_acme_test.yaml. Tree-wide there are roughly 691 documentIndex uses in about 30 suites; this issue tracks only the silent-form subset above, which is where the risk is concentrated.
Fix pattern, measured on the first wave
Select the document by an identifying field (documentSelector on metadata.name, spec.secretName, or similar) instead of by position, and pair the negatives with at least one positive assert on the same document. A selector that matches nothing fails loudly (document not found) even for pure-notExists cases, and a selector that matches two documents fails loudly too (multiple indexes found), so the conversion removes both the index-drift hole and the vacuous-negative hole at once.
Two traps for whoever does the conversion, both hit during the first wave: helm-unittest numbers documents in template SOURCE order while helm template prints them kind-sorted, so never diagnose or verify indexes against helm template output; and prove each conversion by mutation with the mutation confirmed visible in the render first, since a mock or a wrong-place mutation reads as coverage.
Activity
lexfrei commentedon Aug 8, 2026
Wider catalog, measured at 605030b after the first-wave fixes landed: 707
documentIndexoccurrences across 40 suite files, all underpackages/*/*/tests/. This issue tracks the silent subset (index plus negatives only), but the whole class shares the fragility: any insertion or reordering in the template shifts every index after it, and positive asserts then fail on the wrong document with a misleading message rather than a "document not found".The densest single case is
packages/apps/tenant/tests/cleanup_role_watch_test.yaml, which pinsdocumentIndex: 3nineteen times in one suite, so one inserted document above it invalidates all of them at once. That file is the natural start for a second conversion wave.One more data point from the first wave worth recording here: in the tenant suite, deleting the selected document turned only the positive anchor assert red while the
notExistsassert stayed green on the empty render. A bare negative under a drifted index is satisfied vacuously, which is the failure mode this issue exists to remove.lexfrei commentedon Aug 8, 2026
Scale update from a repo-wide sweep across all 258 suites: 66
it:blocks assert ONLY absence (notExists/notContains with no equal/isKind/hasDocuments in the same block), spread over core/platform, core/installer, most of system/* and apps/*, including the vendored ingress-nginx suites. The heuristic is coarse in the safe direction, a block over a template that renders unconditionally cannot see zero documents, so a share of the 66 is fine as-is; but the surface is real and sits at a different granularity than this issue's current catalog, blocks rather than suites.Two refinements worth carrying into the conversion wave. First, the silent forms differ by direction: index drift silences INDEXED negatives (the wrong document satisfies them), total disappearance silences only the BARE negative, since an indexed one fails loudly with "document index out of range"; an
equal: metadata.nameanchor covers both. Second, the tenant chart is now clean, its remaining bare negatives are anchored by hasDocuments/equal in the same blocks, so the wave can skip it.lexfrei commentedon Aug 8, 2026
One more measured instance for the catalog, notable because the nulling switch already exists in-tree:
packages/apps/kubernetes/tests/cilium_gateway_alpn_test.yamlopens with a single barenotExistsand no anchor ("renders no gatewayAPI block when the gatewayAPI addon is disabled"), while the template it inspects renders behind{{- if .Values._namespace.etcd }}and the chart shipstests/values-ci-no-etcd.yamlwithetcd: "". Point the suite at that values file, or rename the key, and the test stays green on an empty render. Found by spot-check rather than sweep (notExistsappears in 81 files), which supports the pattern that instances of this class surface from a different chart on every look.lexfrei commentedon Aug 8, 2026
Narrowed target list for the wave, produced by filtering the wide scan: of 26 unanchored
notExistsacross the tree, 20 sit on templates that render unconditionally and are safe as-is; the remaining SIX are real instances, each on a template wholly wrapped in a top-levelif, so a values change nulls the render and the test stays green:apps/mongodb/tests/restore_test.yaml,system/linstor-scheduler/tests/admission_test.yaml,system/seaweedfs/tests/s3_traffic_distribution_test.yaml, and three tests insystem/vertical-pod-autoscaler/tests/vpa_test.yaml. The mongodb one was confirmed by hand. Together with the earlier alpn instance this is the concrete starting set; the tenant chart is clean after #3661 and the wave can skip it.test(tests): make absence-only helm-unittest cases fail when they che…