Skip to content

E2E on a stacked PR tests stale committed image refs because the base-branch overlay tag is invalid #4391

Open
Open
E2E on a stacked PR tests stale committed image refs because the base-branch overlay tag is invalid#4391
Labels
area/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)kind/bugCategorizes issue or PR as related to a bugpriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

Description

@lexfrei
Aleksei Sviridkin (lexfrei)
opened on Sep 22, 2026

When a PR targets a branch other than main, its E2E run does not test current main images. It falls back to the image refs committed in the repo, and some of those are too old for the charts that use them today. The visible symptom is Grafana failing to start in "Configure Tenant and wait for applications" with failed to download plugin archive: 404 from http://grafana-dashboards.cozy-grafana-operator.svc/plugins/<id>-<ver>.zip.

There are two separate defects here. Either one alone would not produce the failure.

Base-branch overlay builds an invalid tag for a stacked PR

The Finalize job pulls the current packages tree for the PR base and overlays its image refs onto packages the PR did not rebuild. The tag is the base branch name as is, see .github/workflows/pull-requests.yaml line 648 at a759190:

if ! flux pull artifact "oci://${REGISTRY}/cozystack-packages:${BASE_REF}" \

A branch name like fix/cdi-update-keeps-local-templates has a /, which is not valid in an OCI tag, so the pull fails at parse time:

✗ 'oci://iad.ocir.io/idyksih5sir9/cozystack/cozystack-packages:fix/cdi-update-keeps-local-templates' invalid URL: could not parse reference: ...
No current-main packages tree at _out/mainpkgs — unbuilt packages keep committed refs

The job prints a warning and continues, so every unbuilt package runs its committed ref. For a stacked PR that means E2E tests the last release, not the tip of main. The overlay from the base branch came in with #3471.

Committed grafana-dashboards ref predates plugin mirroring

packages/system/grafana-operator/images/grafana-dashboards.tag on main points to grafana-dashboards:v1.6.0@sha256:cb22416b34d4385e4df58314de390920af805c03358fe00734c8fa44ba505fe9. It was last stamped in 235d38d (2026-07-22). Plugin mirroring into that image landed two days later in 606ffd5, and grafana.yaml on main now installs plugins from it. I exported the pinned image: it has the dashboards under /var/www/dashboards/ and no plugins/ directory at all. So any path that falls back to committed refs gets the 404.

Affected runs

In PR runs created 2026-09-12 to 2026-09-22 UTC, 211 reached E2E. Only two had a base other than main, and both hit this 404. None of the 207 main-based runs did. The two runs, both with base fix/kafka-hook-credentials:

  • 34991666797, 2026-09-15 15:55 UTC
  • 35190300009, 2026-09-17 06:32 UTC

The same failure happened earlier in run 34543762473 (2026-09-11), twice on retry.

Possible fixes

For the overlay, fall back to main when the base is neither main nor a release-* line, or sanitize the ref into a valid tag. Either way, a stacked PR would get current main images, not the last release.

For the grafana-dashboards ref, re-stamp it so the committed image includes the plugins its chart asks for.

I did not pull the cozystack-packages:main artifact myself, so I have not checked directly that its grafana-dashboards image has the plugins. The 207 clean main-based runs are the only evidence for that.

Activity

  1. Aleksei Sviridkin (lexfrei)
    added
    kind/bugCategorizes issue or PR as related to a bug
    area/ciIssues or PRs related to CI workflows, GitHub Actions, automation
    on Sep 22, 2026
  2. github-actions
    added
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 22, 2026
  3. Aleksei Sviridkin (lexfrei)
    added
    area/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)
    triage/acceptedIndicates an issue is ready to be actively worked on
    priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete
    and removed
    triage/needs-triageIndicates an issue needs triage by a maintainer
    on Sep 23, 2026
  4. Aleksei Sviridkin (lexfrei)
    added a commit that references this issue on Sep 24, 2026

    fix(ci): overlay the right line's refs for a PR on a feature branch (#…

    6748014

  5. IvanHunters commented on Sep 25, 2026

    @IvanHunters
    IvanHunters
    on Sep 25, 2026
    Collaborator

    Hit the same failure on a fresh run and captured the server side, which confirms the grafana-dashboards half directly (this issue notes the main artifact was not pulled to check).

    Run 36069518943 (PR #4471, base test/wait-budget-guard-margin, a stacked PR). From that run's cozyreport, the grafana-dashboards mirror pod runs the committed pin ghcr.io/cozystack/cozystack/grafana-dashboards:v1.6.0@sha256:cb22416b34d4385e4df58314de390920af805c03358fe00734c8fa44ba505fe9 and is healthy the whole time (phase: Running, ready: true, restartCount: 0, darkhttpd listening). It serves every dashboard .json with 200 and answers every /plugins/*.zip with 404:

    "GET /main/controller.json HTTP/1.1" 200 166566 "" "Go-http-client/1.1"
"GET /plugins/grafana-worldmap-panel-1.0.6.zip HTTP/1.1" 404 400 "" "Go-http-client/1.1"

    So the pinned image has the dashboards and no plugins/, exactly as reasoned here. Grafana stops on the first plugin and crashloops, and Configure Tenant and wait for applications times out.

    One addition for the fix scope: 606ffd5b1 (the plugin mirroring) is only on the v1.7.0 line, not in v1.6.1..v1.6.3. If releases carry the committed digest through the promote/retag path rather than a rebuild, a v1.7.0 release would ship the pre-plugin grafana-dashboards image to production as well, not just E2E, which is the air-gapped case the mirror exists for. So re-stamping the ref closes a production risk too, not only the stacked-PR symptom.

  6. IvanHunters
    added a commit that references this issue on Sep 25, 2026

    test(monitoring): scope the parity guard's comment to what it proves

    2c9e939

  7. lexfrei commented on Sep 28, 2026

    @lexfrei
    Aleksei Sviridkin (lexfrei)
    on Sep 28, 2026
    ContributorAuthor

    I checked whether the stale grafana-dashboards pin reaches a release. It doesn't, so the defect is confined to CI runs that fall back to committed refs.

    The v1.7.0-alpha.1 tag still carries the pre-mirror pin in the tree (grafana-dashboards:v1.6.0@sha256:cb22416b…). The packages artifact the release actually installs from is cozystack-packages@sha256:ea73f9c03dafa3012789087b187c40ff518ca39de05a3c9729144b712e758a5a, taken from cozystack-operator-generic.yaml in the release assets. That artifact has grafana-dashboards:v1.7.0-alpha.1@sha256:41a87524e84624cd5b816f7539d30cd5ec9d73d1899b024481cd447a3381b87c, and that image does ship the plugins:

    var/www/dashboards/plugins/grafana-worldmap-panel-1.0.6.zip
var/www/dashboards/plugins/marcusolsson-dynamictext-panel-6.2.0.zip
var/www/dashboards/plugins/natel-discrete-panel-0.1.1.zip
var/www/dashboards/plugins/victoriametrics-logs-datasource-0.29.0.zip

    The v1.6.x line is not affected either. v1.6.4 pins its own v1.6.4 image, and its grafana.yaml does not install plugins from the mirror at all.

    One more stacked-PR run with the same failure: 36059722762 (PR #4453, base feat/kubevirt-csi-vmi-hotplug-fallback-flag). Its cozyreport shows the mirror on the v1.6.0@sha256:cb22416b… pin answering GET /plugins/grafana-worldmap-panel-1.0.6.zip with 404 72 times, and Grafana exits with failed to download plugin archive: 404.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/ciIssues or PRs related to CI workflows, GitHub Actions, automationarea/monitoringIssues or PRs related to the monitoring stack (vlogs, vmstack, grafana, workloadmonitor)kind/bugCategorizes issue or PR as related to a bugpriority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to completetriage/acceptedIndicates an issue is ready to be actively worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions