The nightly workflow has been red since 2026-09-15, and since then it stops at the "Mirror images to GHCR" job, so the full e2e suite has not run on main for nine nights. Before that, 2026-09-04 to 2026-09-14 failed in "E2E (full suite)".
The mirror step fails on a ghcr.io write, not on the source side. On 2026-09-15 to 2026-09-18 it was opensearch-operator, and from 2026-09-20 it is cluster-api-provider-kubevirt:
level=fatal msg="writing blob: initiating layer upload to /v2/cozystack/cozystack/cluster-api-provider-kubevirt/blobs/uploads/ in ghcr.io: denied: permission_denied: write_package"
2026-09-19 failed differently, with authenticating creds for ..., so that night has its own cause.
The package changing from one night to the next fits a per-package permission problem: the workflow token has no write access to a package that hack/nightly-mirror.sh pushes to, and once one package is fixed, the next one in the list fails. I have not checked the package settings on ghcr.io, so this is the likely explanation, not a confirmed one. If it holds, the fix is to give the repository's Actions write access on each first-party package under ghcr.io/cozystack/cozystack, and to make the mirror step list every package it can't write in one run instead of stopping at the first.
Runs: 34920721643 (09-15), 35415239321 (09-19), 35483655482 (09-20), 35809888223 (09-23).
The nightly workflow has been red since 2026-09-15, and since then it stops at the "Mirror images to GHCR" job, so the full e2e suite has not run on main for nine nights. Before that, 2026-09-04 to 2026-09-14 failed in "E2E (full suite)".
The mirror step fails on a ghcr.io write, not on the source side. On 2026-09-15 to 2026-09-18 it was
opensearch-operator, and from 2026-09-20 it iscluster-api-provider-kubevirt:2026-09-19 failed differently, with
authenticating creds for ..., so that night has its own cause.The package changing from one night to the next fits a per-package permission problem: the workflow token has no write access to a package that
hack/nightly-mirror.shpushes to, and once one package is fixed, the next one in the list fails. I have not checked the package settings on ghcr.io, so this is the likely explanation, not a confirmed one. If it holds, the fix is to give the repository's Actions write access on each first-party package underghcr.io/cozystack/cozystack, and to make the mirror step list every package it can't write in one run instead of stopping at the first.Runs: 34920721643 (09-15), 35415239321 (09-19), 35483655482 (09-20), 35809888223 (09-23).