Skip to content

Cherry-picking new test coverage usually causes conflicts #6793

Description

@smcv

Our unit test scripts emit output on stdout in TAP format:

1..3
ok First thing
ok Second thing
ok Third thing

and some of our unit test scripts are very long: test-run.sh is now up to 32 test cases.

Each time we add new test coverage to a long script like test-run.sh, whether it's for a security fix or a new feature or anything else, we have to increase the number of tests in the "plan":

-echo "1..3"
+echo "1..4"
…
+run_some_commands
+echo "ok Fourth thing"

This almost guarantees that any time we cherry-pick new test coverage to a stable-branch, as we often want to do for security fixes in particular, there will be a merge conflict on the "plan".

We can avoid this by making use of TAP's alternative syntax that emits the "plan" at the end, turning it into less of a plan and more of a confirmation that testing has finished:

ok First thing
ok Second thing
ok Third thing
1..3

or (canonically)

ok 1 - First thing
ok 2 - Second thing
ok 3 - Third thing
1..3

This is fairly simple to do. It just requires tracking which test number we're on, optionally emitting that number in the ok function, and eventually emitting the number of tests done at the end.

In Perl (the origin of the TAP format) the way this works is that you call done_testing at the end of the test script. I suggest that we should reuse this naming, and make all our test scripts end with that.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions