Our unit test scripts emit output on stdout in TAP format:
1..3
ok First thing
ok Second thing
ok Third thing
and some of our unit test scripts are very long: test-run.sh is now up to 32 test cases.
Each time we add new test coverage to a long script like test-run.sh, whether it's for a security fix or a new feature or anything else, we have to increase the number of tests in the "plan":
-echo "1..3"
+echo "1..4"
…
+run_some_commands
+echo "ok Fourth thing"
This almost guarantees that any time we cherry-pick new test coverage to a stable-branch, as we often want to do for security fixes in particular, there will be a merge conflict on the "plan".
We can avoid this by making use of TAP's alternative syntax that emits the "plan" at the end, turning it into less of a plan and more of a confirmation that testing has finished:
ok First thing
ok Second thing
ok Third thing
1..3
or (canonically)
ok 1 - First thing
ok 2 - Second thing
ok 3 - Third thing
1..3
This is fairly simple to do. It just requires tracking which test number we're on, optionally emitting that number in the ok function, and eventually emitting the number of tests done at the end.
In Perl (the origin of the TAP format) the way this works is that you call done_testing at the end of the test script. I suggest that we should reuse this naming, and make all our test scripts end with that.
Our unit test scripts emit output on stdout in TAP format:
and some of our unit test scripts are very long:
test-run.shis now up to 32 test cases.Each time we add new test coverage to a long script like
test-run.sh, whether it's for a security fix or a new feature or anything else, we have to increase the number of tests in the "plan":This almost guarantees that any time we cherry-pick new test coverage to a stable-branch, as we often want to do for security fixes in particular, there will be a merge conflict on the "plan".
We can avoid this by making use of TAP's alternative syntax that emits the "plan" at the end, turning it into less of a plan and more of a confirmation that testing has finished:
or (canonically)
This is fairly simple to do. It just requires tracking which test number we're on, optionally emitting that number in the
okfunction, and eventually emitting the number of tests done at the end.In Perl (the origin of the TAP format) the way this works is that you call
done_testingat the end of the test script. I suggest that we should reuse this naming, and make all our test scripts end with that.