Repository navigation
Assertions written as ! command cannot fail a test #3565
Description
An assertion written as ! command cannot fail a test under hack/cozytest.sh. It passes whether the thing it forbids happened or not. There are 75 of them in hack/*.bats today, and they are the negative half of the contract on scripts that migrate etcd and object storage.
Two mechanisms compound, and each alone would be enough.
POSIX exempts a pipeline whose status is inverted with ! from set -e, in every position, not only as the last statement. So a failing ! command does not abort the test:
$ sh -c 'set -e; ! true; echo "still here"'
still here
$ sh -c 'set -e; false; echo "not printed"'; echo "rc=$?"
rc=1On top of that, the generator in hack/cozytest.sh rewrites each test body and emits return 0 immediately before the closing }, so the status of the final statement is discarded as well:
/^}$/ {
print " return 0"
print "}"
next
}A scratch suite with three deliberately-false assertions shows both halves:
✅ Test OK: negated assertion mid function is vacuous
✅ Test OK: negated assertion as last line is vacuous
❌ Test failed: plain false assertion does fail as expected (exit 1)
The two negated ones assert ! true, which is false by construction. They report success.
Where this lands:
| File | ! assertions |
|---|---|
hack/migration-50-etcd-adopt.bats |
46 |
hack/migration-seaweedfs-db-adopt.bats |
9 |
hack/promote-retag_test.bats |
6 |
hack/release-changelog-contract.bats |
5 |
hack/nightly-mirror_test.bats |
5 |
hack/seaweedfs-guard-parity.bats |
4 |
These are the assertions that say a migration must not do something: ! echo "$apply" | grep -qF -- "--skip-backup", ! grep -qF -- "SCALE 0" "$FAKE_CMDLOG" and similar. They read as guards on destructive paths and currently hold nothing: the suite is green whether the script skips the backup or not.
The mechanical fix per assertion is a counted form, which is an ordinary command and so is subject to set -e:
[ "$(echo "$out" | grep -c 'forbidden')" -eq 0 ]Worth pairing with something that stops the shape coming back, since it reads as correct and the suite gives no feedback either way: either the generator stops injecting return 0, or cozytest.sh refuses a test body containing a !-prefixed statement.
One caveat for whoever picks this up: converting these will surface assertions that were not merely dead but wrong, and the suite will go red on unmutated code. That is the point — a dead assertion hides its own incorrectness, so expect to fix the claim as well as the form.
Activity
test(tests): make negated BATS assertions able to fail (cozystack#3790)