Skip to content

Redis backup demo drops kubectl stderr, so its E2E failures carry no cause #4395

Closed

Description

The redis backup demo that redis-2-backup-roundtrip runs throws away stderr on its kubectl calls, so when the suite goes red the log cannot say why. It went red twice on 2026-09-22, on PRs that do not touch redis, with two different messages.

  1. test(e2e): pin the release-history read against the upstream type #3607, run 35782020678, 21:43 UTC: x corruption step did not take (got '') from examples/backups/redis/03-restore-in-place.sh:20. The backup step just before it had succeeded.
  2. docs(api): stop listing helm-install-timeout carriers in rest.go #4381, run 35783065898, 21:39 UTC: x cozy-default-redis strategy not found from examples/backups/redis/run-all.sh:118, about 30 minutes after the platform install finished.

Both messages come from a read that returned an empty string. redis_cmd in examples/backups/redis/00-helpers.sh:124-137 runs kubectl run --rm -i ... 2>/dev/null, and the strategy lookup in run-all.sh:116-117 is kubectl get ... 2>/dev/null || true. A missing object, an API error, a failed image pull, a sentinel with no master, and a kubectl run -i that lost the output of a short-lived pod all end up as the same empty string. Paths and line numbers are at origin/main a759190.

I don't know yet which of these happened in either run, and I have not measured how often the suite fails this way.

Proposed fix: keep stderr in both places, for example by sending it to the job log instead of /dev/null. The next red then names its cause, and the retry or wait it needs can be decided from that. Adding a retry now would only hide whatever the cause is.

Activity

  1. lexfrei commented on Sep 22, 2026

    @lexfrei
    ContributorAuthor

    A third one, run 35785043547 on #4387 (docs only), 22:08 UTC: x in-place restore mismatch: got '', expected 'roundtrip-4f1c9a2b' after the RestoreJob reached Succeeded. Same shape again, an empty read through redis_cmd, so it cannot be told apart from a restore that actually lost the key.

  2. lexfrei commented on Sep 23, 2026

    @lexfrei
    ContributorAuthor

    A fourth: the rerun of #3607 (job 106981668453, 00:15 UTC) failed at the same step as the first attempt, corruption step did not take (got ''). At this rate reruns are not a way through, so I am working on a fix here.

  3. added
    kind/flakeCategorizes issue or PR as related to a flaky test
    area/testingIssues or PRs related to testing (e2e, bats, unit tests)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next release
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/testingIssues or PRs related to testing (e2e, bats, unit tests)kind/flakeCategorizes issue or PR as related to a flaky testpriority/important-soonMust be staffed and worked on either currently, or very soon, ideally in time for the next releasetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions