Repository navigation
build(make): bats-unit-tests stops at the first failing file, so the tail of the suite never runs #3692
Description
make bats-unit-tests stops at the first failing file, so every file sorting after it is never executed and the run reports on a prefix of the suite rather than on the suite.
The recipe at Makefile:163 loops over BATS_UNIT_FILES and exits the loop on the first non-zero status:
@for f in $(BATS_UNIT_FILES); do \
echo "--- running $$f ---"; \
hack/cozytest.sh "$$f" || exit 1; \
doneOn Linux CI this is invisible, because nothing in the list fails there and the loop always reaches the end. It bites locally on macOS, where hack/migration-seaweedfs-db-adopt.bats fails for a platform reason unrelated to any branch: the container it starts runs as --user 501:20 and cannot write into a bind mount taken from the macOS temp directory. That file sorts under m, so roughly two dozen files after it -- run-*, seaweedfs-*, talos-*, tenant-*, including the suites that hold cross-cutting guards -- do not run at all.
The practical cost is not the missing coverage, it is that the output looks complete. A developer who runs the target, sees one familiar red and concludes "only the known platform failure" is making a statement about the files before the letter m and has no information about the rest. The same applies to comparing a branch against origin/main by the set of failing targets: both sides truncate at the same point, so the difference is honest only on the prefix, and a guard newly imported by a rebase is precisely the kind of thing that sits in the tail.
make --keep-going does not help. That flag works between targets, and the exit here happens inside a single recipe.
The smallest fix keeps running and still reports honestly:
@rc=0; for f in $(BATS_UNIT_FILES); do \
echo "--- running $$f ---"; \
hack/cozytest.sh "$$f" || rc=1; \
done; exit $$rcWorth deciding separately whether the target should also print a count of files executed against files listed. Right now a green run and a run that stopped after three files are distinguishable only by reading the whole log.
Activity
lexfrei commentedon Aug 9, 2026
The same truncation happens one level down, inside a single file.
hack/cozytest.shruns each test throughrun_one, and on a non-zero result that function prints the captured output and then exits:The first failing test ends the whole file, so every later test in it never runs. Together with the recipe-level abort described above, a run stops twice: at the first failing test of the first failing file.
This surfaced during mutation testing, which is where it costs something. Mutating the source to check that a pin bites can knock out an early test, and the later test you were actually aiming at then does not execute. Its silence is indistinguishable from passing, so you record "the mutation is caught" on the strength of a test that was never the subject. Isolating the intended test needs a second, differently shaped mutation, which is only obvious once you know the runner stops early.
batsdoes not behave this way. It runs every test in the file and reports each, so the two runners disagree not only on how failures are reported but on how many tests ran at all.Worth stating the intent along with whatever fix lands for the recipe. Stopping at the first failure is defensible for a fast local loop, but it makes "the file passed" and "the file ran" the same sentence only when nothing fails.
fix(ci): run every bats unit file after a failure and cover build-*.s…