helm-unittest's isNotNullOrEmpty does not fail when the path it checks is missing from the rendered manifest. It fails only when the path exists and holds an empty value. So a test that uses it to say "this key is set" keeps passing after the key is renamed or dropped.
The cause is in the validator. isNotNullOrEmpty is isNullOrEmpty with context.Negative set. In pkg/unittest/validators/is_nullorempty_validator.go at v1.1.2, the version CI installs, the "unknown path" error is returned only when !context.Negative. For the negated form the result is len(actual) == 0 && context.Negative, which is true for a path that resolves to nothing.
Checked locally with v1.1.1 against a ConfigMap template:
asserts:
- isNotNullOrEmpty:
path: data.nope # passes
- isNotEmpty:
path: data.nope # passes
- isNotNull:
path: data.nope # failsexists also fails on a missing path, as expected.
Three test files here use isNotNullOrEmpty:
packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml, lines 44 and 297packages/apps/clickhouse/tests/backup_api_contract_test.yaml, line 89packages/apps/vpn/tests/vpn_test.yaml, line 24
None of them is vacuous today. I swapped each one for exists and the suites stayed green, so every path is present in the current render. The risk is a future rename that these asserts will not notice.
The fix is to add an exists assert next to each of them. It is probably also worth asking upstream whether the negated form is supposed to accept a missing path.
helm-unittest's
isNotNullOrEmptydoes not fail when the path it checks is missing from the rendered manifest. It fails only when the path exists and holds an empty value. So a test that uses it to say "this key is set" keeps passing after the key is renamed or dropped.The cause is in the validator.
isNotNullOrEmptyisisNullOrEmptywithcontext.Negativeset. Inpkg/unittest/validators/is_nullorempty_validator.goat v1.1.2, the version CI installs, the "unknown path" error is returned only when!context.Negative. For the negated form the result islen(actual) == 0 && context.Negative, which is true for a path that resolves to nothing.Checked locally with v1.1.1 against a ConfigMap template:
existsalso fails on a missing path, as expected.Three test files here use
isNotNullOrEmpty:packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml, lines 44 and 297packages/apps/clickhouse/tests/backup_api_contract_test.yaml, line 89packages/apps/vpn/tests/vpn_test.yaml, line 24None of them is vacuous today. I swapped each one for
existsand the suites stayed green, so every path is present in the current render. The risk is a future rename that these asserts will not notice.The fix is to add an
existsassert next to each of them. It is probably also worth asking upstream whether the negated form is supposed to accept a missing path.