backups: the e2e lane runs a backupstrategy-controller older than the tree, so four round-trip suites fail on kinds it supports

Describe the bug

packages/system/backupstrategy-controller pins an image tagged v1.6.0, cut on 2026-07-22. Four backup strategy drivers have landed since. The running controller therefore refuses the very Kinds the tree in the same commit knows how to dispatch, and every backup round-trip suite for those engines fails in e2e with a message that reads like a missing feature rather than a stale binary.

Environment

Cozystack: main
In-tree e2e lane

To Reproduce

Run any e2e lane that selects the backup round-trip suites. Four of them fail.

Expected behaviour

The controller the lane runs supports what the tree it was built from supports.

Actual behaviour

strategy Kind "Redis" is not supported by this controller (supported: ...)

The same shape appears for Rabbitmq, MongoDB and Kafka. The guard producing it is in internal/backupcontroller/backupjob_controller.go:127-135, and the list it checks against is right below:

$ git grep -n -A14 "func supportedBackupStrategyKinds" origin/main -- internal/backupcontroller/
192:func supportedBackupStrategyKinds() []string {
193:	return []string{
194:		strategyv1alpha1.JobStrategyKind,
...
202:		strategyv1alpha1.RabbitmqStrategyKind,
203:		strategyv1alpha1.RedisStrategyKind,
204:		strategyv1alpha1.KafkaStrategyKind,
205:	}

Eleven Kinds in the tree; the binary in the lane named seven in its own error message.

Logs

The pin:

$ git show origin/main:packages/system/backupstrategy-controller/values.yaml | head -2
backupStrategyController:
  image: "ghcr.io/cozystack/cozystack/backupstrategy-controller:v1.6.0@sha256:e8bf0521..."

The tag, and one of the drivers it cannot contain:

$ git log -1 --format="%ad" --date=short v1.6.0
2026-07-22

$ git log -1 --format="%h %ad %s" --date=short b0eb10427
b0eb10427 2026-08-28 feat(backups): add Redis backup/restore strategy driver

$ git merge-base --is-ancestor b0eb10427 v1.6.0; echo "exit $?"
exit 1

The driver postdates the tag by five weeks and is not an ancestor of it, so no image built from that tag can carry it.

Additional context

Why it is worth filing separately rather than living inside whichever PR notices it: the failure lands on unrelated pull requests. Four of the seven chainsaw failures on #4251's head are this, and #4242 and #4252 carry the same set on their own heads. Each author then has to establish, independently, that the red lane is not about their change — three separate investigations of one stale line.

It also degrades the lane's value while it stands. A backup round-trip that fails before reaching the round trip proves nothing about backups, so the suites are burning e2e minutes to report a version skew.

Adjacent but not the same: #3508 is about which components the round-trip suites need installed. This is about the version of one that is installed.

The fix is a pin bump, but the durable question is why a platform-version tag gates a component whose drivers land on main between releases. A release-line image for a controller that the tree keeps extending will drift again on the next driver.

Checklist

I have checked the documentation
I have searched for similar issues
I have included all required information
I have provided clear steps to reproduce
I have included relevant logs

Correcting my own framing above, and narrowing it. The pin is real, but "bump it" is not the fix, and the stale binary is not the whole story.

No tag on the 1.6 line carries the driver. I assumed a newer release tag would, and it does not:

$ for t in v1.6.0 v1.6.1 v1.6.2 v1.6.3; do
    printf '%-8s ' "$t"
    git merge-base --is-ancestor b0eb10427 "$t" && echo contains || echo "does not ($(git log -1 --format=%ad --date=short $t))"
  done
v1.6.0   does not (2026-07-22)
v1.6.1   does not (2026-08-05)
v1.6.2   does not (2026-08-19)
v1.6.3   does not (2026-09-04)

So release-1.6 does not carry what lands on main, and moving the pin to the newest release tag changes nothing. Any fix has to reach a main-built image.

The image overlay works; it skipped this package specifically. From the cozyreport of the E2E (in-tree) run on 4878250a1 (PR #4252), two controllers in cozy-backup-controller, side by side:

backup-controller-597b56cd6b-2nngc        iad.ocir.io/idyksih5sir9/cozystack/backup-controller:main
backupstrategy-controller-5cf66566b-hc    ghcr.io/cozystack/cozystack/backupstrategy-controller:v1.6.0

One took the current-main image, the other kept the committed release ref. Neither package is touched by that PR, so hack/overlay-main-images.sh should have treated them alike. That asymmetry, not the age of the pin, is what I would chase first.

Which binary refuses the Kind. Both cmd/backup-controller/main.go and cmd/backupstrategy-controller/main.go import internal/backupcontroller, so both carry supportedBackupStrategyKinds(). With backup-controller on main and backupstrategy-controller on v1.6.0, the seven-Kind list in the error came from the second.

Why this package is awkward to pin by hand. Two of its three image keys are stamped by a different package's Makefile (packages/core/platform), and the file says so:

# DO NOT hand-edit: packages/core/platform/Makefile stamps this key with
# the same ref it writes to .migrations.image there, so the two stay in
# lockstep by construction. They previously did not — this key had no
# producer at all and froze at v1.4.0-rc.2 while the other advanced to
# v1.5.0.

That is the same failure shape as this one, one layer down, and hack/image-pin-consistency.bats exists because of it. So a hand-written digest here risks re-opening the defect that guard was added for.

What stands from the original report: the lane runs a backupstrategy-controller older than the tree, four round-trip suites fail on Kinds the tree supports, and three unrelated pull requests each had to establish independently that the red lane was not theirs. What I got wrong was the remedy.

One more suite falls to this, and it dates the running binary more precisely than the pin does.

postgres-2-backup-roundtrip fails with what reads like a CNPG webhook problem:

failed to apply barmanObjectStore to Cluster: admission webhook "vcluster.cnpg.io" denied the request:
Cluster.cluster.cnpg.io "postgres-pg-src" is invalid: spec.plugins:
Invalid value: ["barman-cloud.cloudnative-pg.io"]:
Cannot enable a WAL archiver plugin when barmanObjectStore is configured

The chart puts a barman-cloud.cloudnative-pg.io plugin on the Cluster with isWALArchiver: true (packages/apps/postgres/templates/db.yaml:172-183). Something then tries to add spec.backup.barmanObjectStore beside it, and CNPG refuses the combination. The webhook is doing its job; the writer is the problem.

That writer is not in the tree. The message does not exist on main, and it does not exist at the pinned tag either:

$ for ref in origin/main v1.6.0; do
    printf '%-12s ' "$ref"
    git grep -l "failed to apply barmanObjectStore" "$ref" | head -2 | tr '\n' ' '; echo
  done
origin/main
v1.6.0

Both empty. It was removed on 2026-07-15, a week before v1.6.0 was cut:

$ git log --all -S "failed to apply barmanObjectStore" --format="%h %ad %s" --date=short
13ff34027 2026-07-15 fix(backups): migrate CNPG backups to the barman-cloud plugin
ed5ae16e3 2026-05-07 feat(platform): implement backup strategy for postgres application

After that commit the controller wires the plugin instead, which applyClusterPluginBackup states in as many words: "This replaces the deprecated native spec.backup.barmanObjectStore path".

So the binary answering in that namespace predates not just main but the tag this package pins. Whatever is serving it is older than the committed ref suggests, which is worth knowing before anyone fixes this by moving the pin one notch.

What this adds up to. Of the seven chainsaw failures that had three separate PR authors investigating red lanes, five now trace to this one cause: four backup round-trips refused on Kinds the tree supports, plus this. They are not five defects. Reading them as five is exactly the cost of the drift being silent — Finalize logs drift (non-ref change) in ... -> keeping committed ref and nothing downstream repeats it, so the next person starts from the symptom again.

Two candidate explanations for the drift, both ruled out, so the next person does not spend the time again.

Not a stale artifact. Every drifted file was last edited well before the artifact was built:

backupstrategy-controller    2026-09-08
cozystack-api                2026-09-07
dashboard                    2026-09-07
kubeovn-webhook              2026-09-02
linstor-gui                  2026-09-07
core/platform                2026-09-11

last successful build-main: 2026-09-15 08:09, 2026-09-14 18:45, 2026-09-14 15:21

So the artifact carries those edits; the committed tree cannot be ahead of it on config.

Not yq normalising the file. make image rewrites the ref with yq -i, and a file this comment-heavy looked like a plausible victim of reformatting. It is not — the rewrite touches exactly the one line:

$ git show origin/main:packages/system/backupstrategy-controller/values.yaml > v.orig
$ cp v.orig v.yaml
$ REPOSITORY=iad.ocir.io/x/cozystack/backupstrategy-controller \
    yq -i '.backupStrategyController.image = strenv(REPOSITORY) + ":main@sha256:deadbeef"' v.yaml
$ diff v.orig v.yaml | grep -cE '^[<>]'
2

Two lines, both halves of the same image: change. No comment, quote style or indentation moved.

So the question stands as: what, in these six files specifically, differs from the artifact copy beyond image lines? Answering it needs the artifact itself — flux pull artifact oci://$REGISTRY/cozystack-packages:main and a plain diff against the committed file. I do not have registry access, so I am leaving that step rather than guessing at it.

Worth noting the set is not random: all six are packages whose values.yaml was edited in the last two weeks, while backup-controller, untouched since the v1.6.0 release, overlaid cleanly in the same run. Whatever the mechanism, recent editing is what correlates.

Separately, #4261 does not fix any of this. It only names the drifted packages in one annotation at the end of the overlay, so the next occurrence is legible from the job summary instead of requiring this investigation. The components still run release images until the cause is found.

fix(ci): read a suffixed image key as a ref, not a config change

Follow-up, deliberately left out of #4261: the classifier there stops these refs from costing their file the overlay, but it does not make them reproducible, and they are invisible to the release chain either way.

Two first-party refs carry no digest:

$ grep -n redisBackupClientImage packages/system/backupstrategy-controller/values.yaml
57:  redisBackupClientImage: "ghcr.io/cozystack/cozystack/redis-backup:v1.6.0"
$ head -2 packages/system/migration-controller/values.yaml
migrationController:
  image: "ghcr.io/cozystack/cozystack/migration-controller:latest"

Neither reaches the collector every consumer reads:

$ . hack/lib/image-refs.sh && collect_image_refs packages | wc -l
70
$ . hack/lib/image-refs.sh && collect_image_refs packages | grep -cE 'redis-backup|migration-controller'
0

collect_image_refs matches on @sha256:, so promote-retag, the nightly mirror and verify-promoted-packages all step over these two. Nothing in make unit-tests fails on it either, which is why they have sat there.

docs/agents/image-refs.md rules both out under Invariants: "Every first-party reference is tag+digest. A bare tag with no digest is not reproducible and must not be committed." The :latest one is the worse of the two — it names no version at all, so what a cluster pulls depends on when it pulled.

Both files say DO NOT hand-edit on the keys around them, and the producer is packages/core/platform/Makefile, so the fix is to teach a producer to stamp them rather than to paste a digest. That is a different change from #4261 and a maintainer call, hence this note rather than a commit.

Worth noting the same document already lists chBackupClientImage in that same file as a live violation of the one-producer invariant, still pinned to v1.4.0-rc.2. Three of the four image keys in backupstrategy-controller/values.yaml are named in an invariant list one way or another, which suggests the file wants one pass rather than three.

ci(e2e): name the packages that kept release refs instead of current …

fix(ci): read a suffixed image key as a ref, not a config change

ci(e2e): name the packages that kept release refs instead of current …

fix(ci): read a suffixed image key as a ref, not a config change

ci(e2e): surface overlay drift and remove both causes of it (#4261)

Fixed by #4261 (merged as c983390).

The cause was the classifier, not the pin. redisBackupClientImage is pinned by tag alone, and the old key branch accepted image, repository, registry, tag and digest only, so that one line read as a config change and cost the whole file its overlay. backupstrategy-controller then kept its committed v1.6.0 ref in every lane that did not rebuild it, which is what failed four round-trip suites on kinds the tree already dispatches.

The branch added in #4261 accepts a prefixed key whose value carries a repository path, so all four ref lines in packages/system/backupstrategy-controller/values.yaml classify as refs now.

Confirmed on the first lane to run after the merge (run 35190292154, Finalize):

overlay: packages/system/backupstrategy-controller/values.yaml -> current-main
Overlay current-main images: overlaid=35 same=136 skipped(rebuilt/owned/edited)=3 drift=0 failed=0

Navigation Menu

Repository navigation

backups: the e2e lane runs a backupstrategy-controller older than the tree, so four round-trip suites fail on kinds it supports #4257

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

backups: the e2e lane runs a backupstrategy-controller older than the tree, so four round-trip suites fail on kinds it supports #4257

Description

Activity

IvanHunters commented on Sep 15, 2026

IvanHunters commented on Sep 15, 2026

IvanHunters commented on Sep 15, 2026

IvanHunters commented on Sep 15, 2026

IvanHunters commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions