testing: negated containsDocument without any: true passes on any multi-document render #3864

Seven assertions in the platform chart's helm-unittest suites are written as a negated containsDocument without any: true. In that form the assertion cannot fail, so each of them is green regardless of what the template emits. Every one of them is the only assertion standing between a rendered Package and the regression it was written to catch.

Mechanism

helm-unittest evaluates containsDocument against every rendered document unless any: true narrows it to "some document". Negating the all-documents form asserts "not every document matches", which any render of more than one distinct document satisfies. The intended meaning — "no document matches" — is the negation of the any: true form.

documentIndex: -1 does not change this. -1 is the default sentinel that already means "every document", so writing it explicitly restates the broken form rather than repairing it.

Sites

• packages/core/platform/tests/bundles_velero_default_test.yaml:40 and :45 — the opt-out case for cozystack.velero and cozystack.backupstrategy-controller.
• packages/core/platform/tests/bundles_proxy_protocol_test.yaml:21, :153, :170 — the three cases pinning that cozystack.ouroboros is not emitted.
• packages/core/platform/tests/bundles_proxy_protocol_lookup_test.yaml:59 — same guard under the lookup-populated variant.
• packages/core/platform/tests/bundles_proxy_protocol_lookup_nil_test.yaml:31 — same guard under the lookup-nil variant.

The correct form is already in the tree at packages/core/platform/tests/bundles_isp_hosted_dependency_closure_test.yaml:132 and :141, with a comment at :118 stating exactly why any: true is load-bearing. The sites above predate or missed that comment.

Observing it

Copy packages/core/platform to a scratch directory, replace its tests/ with the file below, and run helm unittest . (helm-unittest 1.1.1). Cases B and C assert that a document which provably exists is absent, and both pass. Case D is the same claim in the any: true form and fails, which is the correct behaviour.

suite: negated containsDocument without any true
templates:
  - templates/bundles/system.yaml
release:
  name: cozystack
  namespace: cozy-system
tests:
  - it: A positive control velero IS rendered
    set:
      bundles: {system: {enabled: true, variant: isp-full}}
    asserts:
      - containsDocument: {apiVersion: cozystack.io/v1alpha1, kind: Package, name: cozystack.velero, any: true}
  - it: B negated WITHOUT any true on a document that provably exists
    set:
      bundles: {system: {enabled: true, variant: isp-full}}
    asserts:
      - containsDocument: {apiVersion: cozystack.io/v1alpha1, kind: Package, name: cozystack.velero}
        not: true
  - it: C negated WITHOUT any true plus documentIndex minus one
    set:
      bundles: {system: {enabled: true, variant: isp-full}}
    asserts:
      - containsDocument: {apiVersion: cozystack.io/v1alpha1, kind: Package, name: cozystack.velero}
        not: true
        documentIndex: -1
  - it: D negated WITH any true on a document that provably exists
    set:
      bundles: {system: {enabled: true, variant: isp-full}}
    asserts:
      - containsDocument: {apiVersion: cozystack.io/v1alpha1, kind: Package, name: cozystack.velero, any: true}
        not: true

Result: Tests: 1 failed, 3 passed, 4 total, with D as the only failure.

Impact

No user-visible defect today — the templates these cases guard are currently correct, so the tests report the right answer for the wrong reason. The cost is that the guards are inert: disabledPackages could stop working for cozystack.velero, or the proxyProtocol=false branch could start emitting cozystack.ouroboros, and the suite would stay green. The velero case matters most, because its own header comment explains that cozystack.backupstrategy-controller hard-depends on cozystack.velero and a regression there wedges the cozystack-platform HelmRelease at DependenciesNotReady.

Fix shape

Add any: true inside the containsDocument map at each of the seven sites and drop the redundant documentIndex: -1. Prove each conversion by mutation: make the template emit the forbidden Package, confirm the case now fails, revert. Without that step a fixed assertion is indistinguishable from a still-broken one.

Worth considering alongside the fix: a lint over packages/**/tests/*.yaml that rejects a containsDocument carrying not: true without any: true, so the shape cannot come back. Tree-wide there are 26 containsDocument uses outside vendored charts, 18 of which already carry any: true.

This is adjacent to #3634, which catalogues a different silent-green shape (documentIndex addressing paired with only negative asserts). The two share a root cause — an assertion that cannot fail reads exactly like one that passes — but the mechanisms and the fixes are separate.