test(e2e): the LB not-recorded-as-reachable test races the machine instead of the collector's decision #3685

hack/capture-dataplane.bats, the test an LB whose probe never ran is not recorded as reachable, races the machine it runs on. It invokes the collector with COZY_DATAPLANE_LIST_TIMEOUT=1 COZY_DATAPLANE_READ_TIMEOUT=1 and then asserts that a per-LB output file exists:

PATH="$d/bin:$PATH" COZY_DATAPLANE_LIST_TIMEOUT=1 COZY_DATAPLANE_READ_TIMEOUT=1 \
  ...
[ -f "$d/out/lb-tenant-web.txt" ]

One second is enough on an idle machine and not always enough on a loaded one. Observed once in five consecutive runs of the same tree on a developer machine; repeated runs of that same tree passed 71/71. Nothing in the assertion distinguishes "the collector decided not to record this LB", which is what the test is about, from "the collector had not finished writing the file yet", which is what a slow machine produces.

The one-second budgets are deliberate: they are what makes the read fail fast so the test can exercise the not-recorded path without waiting out a real timeout. The problem is the assertion, not the budget. A file-existence check is satisfied or not by wall-clock timing, while the property under test is a decision the collector makes.

Two directions, either works. Wait for the collector's own completion signal before asserting, so the check runs against a finished output directory rather than a snapshot of one in progress. Or assert on something the collector emits when it makes the decision, rather than on the artifact's presence, so a slow write cannot masquerade as a different verdict.

Worth fixing rather than tolerating: a test that fails once in five on a loaded machine will fail on CI under contention, and it fails in the direction that reads as a real regression in the code under test.

A second test in the same file behaves the same way: the script survives a hung apiserver and still reaches its own end. It predates the change that surfaced it and fails under load for the same reason, a wall-clock assumption standing in for a decision the collector makes.

That one showed a cost the first did not. During a mutation run it failed first, and since the runner stops at the first failing test, it masked the test that was supposed to catch the mutation. The mutation looked survived. Re-checking that mutation on its own showed the target test red with it and green without, so nothing was actually wrong with the pin, but the signal had been destroyed by an unrelated flake.

That is worth stating because it changes who pays. A flaky test costs a rerun; a flaky test that aborts the run costs the verdict of every test after it, including the ones being used to prove something else. Any fix here should also consider whether the suite ought to keep going after a failure, so one timing-dependent test cannot decide what the rest of the file reports.