test(e2e): backup round-trips take a third of every full PR e2e run #4821

Problem

Backup round-trips take about a third of every full PR e2e run. Measured on pull-requests.yaml runs from 2026-10-02 to 2026-10-08: 98 runs where the full chainsaw matrix ran (98 of the 119 runs that reached the chainsaw step), container lane with local storage class, runs of the branch behind #3427 left out. Median of one full run:

• e2e job 179 min (job cap is 215)
• install 28 min
• Run E2E tests step 141 min, 128 min of it chainsaw tests
• backup/restore tests 60 min (p90 62, max 80), so 47% of chainsaw time and 34% of the job

Per test, median minutes:

• postgres-2-backup-roundtrip 24.7
• kafka-2-backup-roundtrip 10.2
• kafka-3-metadata-roundtrip 6.6
• mariadb, rabbitmq, clickhouse, mongodb and redis round-trips 3.2 to 3.9 each
• clickhouse-1-backup-contracts 0.6
• etcd-2-backup-roundtrip 0, it exits 0 in CI because it needs a real cluster

Postgres, kafka and kafka metadata are 41.5 of those 60 minutes. Postgres alone does source cluster, base backup and four restores (to-copy, PITR, unreachable target, in-place) in one Test. At about 16 full runs a day it is about 16 hours of job time a day in backup tests.

Every PR pays this because most PRs escalate to the full suite (docs/agents/e2e-testing.md section 7), and round-trips are Tests inside the per-app suites (postgres-2-backup-roundtrip sits in hack/e2e-chainsaw/postgres/chainsaw-test.yaml next to postgres), so suite selection can't leave them out.

Nightly has the same cost on a lane with less room: E2E (full suite) hit the 3h35m job cap on the three nightly runs I checked (2026-10-05, 2026-10-06, 2026-10-08), and both kubernetes suites wait 29m for tenant nodes there before failing.

What we get for the minutes, same window, about 100 runs per test: postgres and kafka metadata round-trips failed 0 times, kafka once (renovate digest bump). The clickhouse and rabbitmq round-trips failed together once on https://github.com/cozystack/cozystack/actions/runs/37437778419 (backup client image PR), clickhouse contracts once on https://github.com/cozystack/cozystack/actions/runs/37373899072 (cert-manager PR). I did not check which of these were real regressions and which flakes.

#3427 adds two more round-trips, vmdisk 5.0 and vminstance 3.2 min on its branch, about 8 min per full run.

Proposed solution

Keep cheap backup checks in the PR gate (five round-trips of 3 to 4 min and clickhouse contracts, about 19 min) and run postgres, kafka and kafka metadata round-trips on schedule, plus on PRs that touch backup controllers or examples/backups. That removes about 41 min from 179. The vm round-trips from #3427 go to the same scheduled set.

Round-trips must be selectable apart from their app suite (own suite directories or chainsaw test selection by name), not sized yet. Nightly must fit its cap first, otherwise tests moved there run nowhere.

Additional context

Related: #3278 (A9 and A12 are about e2e time too, nothing there is about backup), #4642, #3236.

Numbers come from the chainsaw-report artifact of each run plus job step timings, parsed per testcase. Chainsaw runs with parallel: 1, so testcase minutes add up.