Skip to content

Assertions written as ! command cannot fail a test #3565

Closed

Description

An assertion written as ! command cannot fail a test under hack/cozytest.sh. It passes whether the thing it forbids happened or not. There are 75 of them in hack/*.bats today, and they are the negative half of the contract on scripts that migrate etcd and object storage.

Two mechanisms compound, and each alone would be enough.

POSIX exempts a pipeline whose status is inverted with ! from set -e, in every position, not only as the last statement. So a failing ! command does not abort the test:

$ sh -c 'set -e; ! true; echo "still here"'
still here
$ sh -c 'set -e; false; echo "not printed"'; echo "rc=$?"
rc=1

On top of that, the generator in hack/cozytest.sh rewrites each test body and emits return 0 immediately before the closing }, so the status of the final statement is discarded as well:

/^}$/ {
    print "  return 0"
    print "}"
    next
}

A scratch suite with three deliberately-false assertions shows both halves:

✅ Test OK: negated assertion mid function is vacuous
✅ Test OK: negated assertion as last line is vacuous
❌ Test failed: plain false assertion does fail as expected (exit 1)

The two negated ones assert ! true, which is false by construction. They report success.

Where this lands:

File ! assertions
hack/migration-50-etcd-adopt.bats 46
hack/migration-seaweedfs-db-adopt.bats 9
hack/promote-retag_test.bats 6
hack/release-changelog-contract.bats 5
hack/nightly-mirror_test.bats 5
hack/seaweedfs-guard-parity.bats 4

These are the assertions that say a migration must not do something: ! echo "$apply" | grep -qF -- "--skip-backup", ! grep -qF -- "SCALE 0" "$FAKE_CMDLOG" and similar. They read as guards on destructive paths and currently hold nothing: the suite is green whether the script skips the backup or not.

The mechanical fix per assertion is a counted form, which is an ordinary command and so is subject to set -e:

[ "$(echo "$out" | grep -c 'forbidden')" -eq 0 ]

Worth pairing with something that stops the shape coming back, since it reads as correct and the suite gives no feedback either way: either the generator stops injecting return 0, or cozytest.sh refuses a test body containing a !-prefixed statement.

One caveat for whoever picks this up: converting these will surface assertions that were not merely dead but wrong, and the suite will go red on unmutated code. That is the point — a dead assertion hides its own incorrectness, so expect to fix the claim as well as the form.

Activity

  1. added
    kind/bugCategorizes issue or PR as related to a bug
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  2. added a commit that references this issue on Oct 5, 2026
    9485f91
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/bugCategorizes issue or PR as related to a bugpriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions