e2e test selection: a package that maps to some suites never escalates, so suites that also cover it are dropped silently #3985

The hole

Test selection escalates to the full suite when a changed package maps to no suite. It does not notice when a package maps to some suites but not to all the ones that cover it — those are dropped silently, and the run looks like a normal targeted run.

Partial coverage is therefore worse than none: no coverage is loud, partial coverage is quiet.

The case that showed it

#3798 changed packages/system/ingress-nginx and broke the ingress-nginx admission webhook (#3983). Its E2E was green, because:

The run selected exactly those four and nothing else. The test that catches the breakage is ingress-hostname-policy — one of 15 tests inside the gateway suite directory — and gateway maps only from gateway-application. So the package reached a suite, escalation never fired, and the suite that actually covers it was not selected. It failed the next night in the full suite instead.

The contrast is in the tree already: packages/system/cozystack-basics, which owns the Ingress-hostname VAP itself, maps to no suite at all and therefore escalates to the full suite — pinned by a test in hack/select-e2e_test.bats. The package with worse coverage gets the safer behaviour.

Two contributing shapes

Suite directories are named after one of the things they cover. gateway holds 15 tests, one of which is about legacy Ingress on ingress-nginx. Nothing in the name or the mapping says so, and selection works on directory names.

The mapping is one-way. src_to_suites() derives suite names from PackageSource names, so a suite is reachable only from a source whose name matches after the transform. A suite covering more than one package can only ever be reached from one of them.

Also worth recording while looking: the backup suite directory is reachable from no PackageSource at all, so it runs only in the full suite today.

Suggested fixes

Immediate, one line in hack/select-e2e.sh:

ingress-application|ingress-nginx) echo gateway ;;

Structural: let a suite declare what it covers instead of deriving it from a name, and pin the round-trip in both directions — every suite reachable from every package it covers, not just from the one it is named after. The existing round-trip test covers suite → source → suite, which is why this direction went unnoticed.