Repository navigation
ci(e2e): a blank line dropped by yq on stamping costs five packages their image overlay #4265
Description
Describe the bug
Six packages kept their committed release refs instead of current-main images on the run behind #4257. One of them, backupstrategy-controller, is explained: a suffixed image key pinned by tag alone read as a config change, which #4261 fixes. The other five are not explained, and nothing currently watches them.
Environment
- Cozystack:
main - In-tree e2e lane,
Finalize (merge patches + installer)job
To Reproduce
Read the overlay decisions in a Finalize job log:
$ gh run view --job <finalize-job-id> --log | grep 'drift (non-ref change)'
drift (non-ref change) in packages/core/platform/values.yaml -> keeping committed ref
drift (non-ref change) in packages/system/backupstrategy-controller/values.yaml -> keeping committed ref
drift (non-ref change) in packages/system/kubeovn-webhook/values.yaml -> keeping committed ref
drift (non-ref change) in packages/system/linstor-gui/values.yaml -> keeping committed ref
drift (non-ref change) in packages/system/cozystack-api/values.yaml -> keeping committed ref
drift (non-ref change) in packages/system/dashboard/values.yaml -> keeping committed ref
Expected behaviour
A package whose config genuinely matches current main takes the current-main image. Where it does not, the reason is known rather than assumed.
Actual behaviour
Five packages run release images in a lane whose purpose is to exercise the tree under review, for reasons nobody has established. cozystack-api and dashboard are not peripheral: an e2e result about either is a result about the last release, not about main.
Additional context
What rules out the two obvious explanations.
Not "the artifact lags main": none of the five changed after the last successful build-main:
$ for p in core/platform system/cozystack-api system/dashboard system/kubeovn-webhook system/linstor-gui; do
printf '%-26s %s commit(s)\n' "$p" "$(git log 37e22dc35..origin/main --oneline -- packages/$p/values.yaml | wc -l)"
done
core/platform 0 commit(s)
system/cozystack-api 0 commit(s)
system/dashboard 0 commit(s)
system/kubeovn-webhook 0 commit(s)
system/linstor-gui 0 commit(s)
Not yq reformatting on stamping: round-tripping a values file through the same yq -i assignment the build uses produces no non-ref difference.
Why I could not finish it. The comparison that found the backupstrategy-controller cause reads the mirrored artifact (ghcr.io/cozystack/cozystack/cozystack-packages:nightly), whose refs are rewritten to ghcr by the nightly mirror. That rewrite hides registry-only differences, so the method is blind to exactly the class of difference these five may carry. Settling it needs the :main artifact from the CI registry, which I cannot pull.
Whoever picks this up has one advantage I did not: once #4261 lands, every run names the drifted packages in an annotation, so the set can be watched over time instead of reconstructed from one job log.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity
IvanHunters commentedon Sep 15, 2026
Established, and the explanation this issue ruled out is the one that was right.
yq -ire-emits the document when the build stamps a digest, and a blank line separating two top-level keys does not survive that. A blank line carries no image reference, so the overlay counts it as a config change and the whole file keeps its committed ref. That is all five packages, and nothing else is involved.The claim above that this was not
yqreformatting was mine and it was wrong. The round-trip check behind it did not cover the blank line, so it produced a clean answer for the wrong reason.Two signals. First, the artifact diffed against the tree at the artifact's own revision:
One blank line and one ref, in each of the five. Second, running the stamping assignment over the committed file reproduces that diff exactly, which is a different kind of evidence rather than the same check twice:
The registry-rewrite objection in the issue body does not apply here: a blank line is not a ref, so the mirror does not touch it, and the public artifact carries the difference intact. That is why this was findable without the CI registry after all.
Fixed in #4261 by discounting blank lines before the drift test. A line with content still counts, so a genuine config difference keeps the committed ref as before. Measured against the public artifact with the tree at its revision:
Worth recording for whoever meets this class again: the check that produced the wrong answer here and the check that produced the right one differ only in whether the artifact was compared as bytes or as parsed YAML. The blank line is invisible to the second.
[-]ci(e2e): five packages keep release refs in the image overlay for reasons not yet established[/-][+]ci(e2e): a blank line dropped by yq on stamping costs five packages their image overlay[/+]ci(e2e): surface overlay drift and remove both causes of it (#4261)
lexfrei commentedon Sep 23, 2026
Fixed by #4261.
hack/overlay-main-images.shnow drops blank lines before the drift check, and drift on the public artifact went from 5 to 0.