Repository navigation
redis-backup e2e image repository is private in OCIR: 403 on pull, Redis suite red fleet-wide #4218
Description
redis-2-backup-roundtrip fails on main, and it has never had a green run.
The suite arrived with the Redis backup strategy driver in #3992. That PR's in-tree e2e never executed: the Finalize job died fetching the flux install script (curl exit 22), E2E (in-tree) gates on it and reported skipped, and I merged with admin rights while that was the state. So the first run that actually reached this test is the first evidence anyone has about it, and it is red.
Seen in run 34516348714, on a PR that touches neither Redis nor the backup controller. From the chainsaw-report artifact rather than a log read: redis tests=4 failures=1, the failing case being redis-2-backup-roundtrip. Eighteen other suites in the same run are green, so this is not a broken environment taking everything down with it.
Two other suites failed in that run and are tracked separately: kafka (the base app suite) and kafka-3-metadata-roundtrip (from an open PR).
What I cannot say yet is whether it fails every time. One run is one observation, and I have not found a second run that reached the test. What makes it worth filing anyway is the asymmetry: there is no green run to weigh against the red one. A suite that has never passed is not a flake until someone shows it passing.
Worth fixing in this order: get one run of the suite in isolation to establish whether it is deterministic, then read the step it dies at. If it turns out environmental, the finding is still useful — it means the suite needs to be re-run and seen green before the next merge that depends on it.
Activity
lexfrei commentedon Sep 11, 2026
Mechanism found, and it is not the test. Correcting my own framing above.
The pod never ran. From the job log of run 34516348714:
redis-backupis a brand-new first-party image, so that run was its first push to the e2e registry, and OCIR creates a repository private on auto-create. The build job pushes with write credentials and succeeds; the lower-privilege puller is then authenticated but not authorized to read. The fix is a maintainer flipping repository visibility, and nothing in the PR or in a rerun can do it.Three things make the status the answer rather than a guess. It is 403, not 404 or
MANIFEST_UNKNOWN, which would have meant the image is absent. The reference is by digest and that digest is what the build job pushed, so the artifact is there. And there is exactly oneunexpected status from HEAD requestin the whole job log: same puller, same credentials, same registry, same run, every other image resolved. That leaves per-repository visibility and rules out both credentials and a transient.Two corrections to what I wrote above. I said the chart points at an image that does not exist, having checked ghcr and found only a
nightlytag. Wrong registry: e2e pulls from OCIR (REGISTRY: iad.ocir.io/idyksih5sir9/cozystackin the build and e2e workflows), and the image is present there. And the scope is wider than this suite having a bad run. Every PR whose e2e selects the Redis suite pulls that same private repository and gets the same 403, so this is red fleet-wide until visibility is flipped. Thepr-NNNN-tag on the image is just whichever PR's build of main content ran first, and it misreads as ownership.What stands from the original report: the suite has never had a green run. That is still true and still worth a green run once the repository is readable, because a pull failure has told us nothing about whether the backup roundtrip works.
[-]redis-2-backup-roundtrip fails on main and has never had a green run[/-][+]redis-backup e2e image repository is private in OCIR: 403 on pull, Redis suite red fleet-wide[/+]lexfrei commentedon Sep 11, 2026
Repository visibility is flipped.
cozystack/redis-backupnow readsis-public: true, stateAVAILABLE.It was the only outlier: after the change, a listing of every private repository in the registry returns nothing outside the
cozystack-cache-release-*namespaces, which are private by design. So this was a first-push artifact rather than a policy difference, exactly as the auto-create behaviour predicts.Leaving this open. The 403 told us nothing about whether the backup roundtrip works, so the suite still needs one green run before the original claim in this issue is retired. Any PR whose e2e selects the Redis suite will now pull the image; the next such run is the measurement.
lexfrei commentedon Sep 14, 2026
Fixed, and now verified by a run rather than by the registry alone.
redis-2-backup-roundtrippassed in run34818275885, 4 tests in theredissuite with this one green. That is the first time this suite has had a green run since it arrived.The visibility flip was the whole fix. The anonymous manifest HEAD that used to answer 403 answers 200, and the cluster-side pull evidently works too, since the test reached the roundtrip instead of sitting in ImagePullBackOff.
redis-tlsfailed in that same run. Different suite, unrelated to this, not tracked here.Closing.