test: cozytest.sh matches test names by substring, and a pattern that selects nothing exits 0 #3717

hack/cozytest.sh selects tests by substring, not by glob, and a pattern that matches nothing exits 0 with no output. A filtered run that silently executed zero tests is indistinguishable from a run where everything passed.

The selection is one line, hack/cozytest.sh:281:

if (pat == "*" || index(title, pat) > 0)

index() searches for a literal substring. Metacharacters are compared as characters, so *, ? and [ never expand. The single special case is a pattern equal to exactly *, handled by the left branch and meaning "everything". Any other glob goes into index() and finds nothing, because no test title contains a literal asterisk.

Measured on origin/main against a suite of six tests:

The exit code is the reason this is worth filing rather than documenting. The awk output feeds while IFS=' ' read -r fn title; do run_one "$fn" "$title"; done; on empty input the loop body never runs, and the status of the pipeline is the status of while, which is 0. Nothing in the selection path guards against an empty selection, so a mistyped filter and a fully green run report the same thing.

Two directions for a fix, and the second matters more than the first. Matching with a glob operator instead of index() would make *foo* behave the way its shape suggests. Independently of that, printing how many tests were selected, and failing when a non-empty pattern selects none, would make the miss visible whatever the matching rule is.

Not established: whether any past run was affected. I read this out of the selection code and reproduced it by invoking the script, not from an observed incident.