Skip to content

fix(ci): resolve nightly packages digest without image-specific skopeo ops - #3332

Merged
myasnikovdaniil merged 1 commit into
mainfrom
fix/nightly-resolve
Jul 20, 2026
Merged

myasnikovdaniil merged 1 commit into
mainfrom
fix/nightly-resolve

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

The Nightly workflow has never completed a run. Every scheduled run since NIGHTLY_ENABLED was set has failed in mirror at "Resolve source commit and version", ~80s in, with:

level=fatal msg="unsupported image-specific operation on artifact with type \"application/vnd.cncf.flux.config.v1+json\""

Nothing has ever been published — cozy-installer:nightly and cozystack-packages:nightly both 404 on GHCR, and cozystack-nocloud does not exist.

This fixes that failure and the next two behind it.

1. The digest lookup could never have worked

cozystack-packages is a Flux OCI artifact (config mediaType application/vnd.cncf.flux.config.v1+json), not a container image. A plain skopeo inspect parses the config blob as an image config and fatals on it, so skopeo inspect --format '{{.Digest}}' was never viable against this ref — the --raw call on the very next line already acknowledged this.

Replaced with skopeo inspect --raw + skopeo manifest-digest, which hashes the manifest bytes — exactly what the registry addresses the artifact by, with no image-specific parsing.

This also collapses two registry fetches into one: the digest and the org.opencontainers.image.revision annotation now come from the same bytes, which strengthens the race the original comment describes — there is no longer any window between pinning the tag and reading it.

2. flux pull artifact rejects a non-existent output path (×2)

flux pull artifact --output _baked fails with invalid output path "_baked": no such file or directory. _baked is neither tracked nor created by anything. pull-requests.yaml does rm -rf + mkdir -p first; nightly.yaml omitted it. Since the workflow always died at step 1, this was never reached.

Same bug for _rw in the e2e staging step. Both now create the directory first.

Verification

  • Reproduced the original fatal against the live artifact with skopeo 1.22.2.
  • Ran the fixed resolve step end-to-end against the real registry: returns the digest the registry reports, and a sha that matches the current main HEAD.
  • Reproduced the _baked failure and confirmed the fix extracts the tree.
  • actionlint clean; zizmor --offline clean; shellcheck clean on changed lines.

Also checked and not changed, each disproven rather than assumed:

  • skopeo copy --multi-arch all on a Flux artifact — works; copy is mediaType-agnostic.
  • The awk -F@ '/artifact successfully pushed/' digest capture — matches current flux CLI output, and is the same idiom packages/core/installer/Makefile already uses in production.
  • nightly-mirror.sh's inspect --format Digest — all mirrored refs there are genuine image manifests, so it is correct as written.
  • make -C packages/core/talos talos-nocloud — exists and is CI-proven via tags.yaml.

Honest scope

This fixes the mirror job's blockers. It does not promise a green nightly on the first run: every step after mirror is still unproven, having never executed. The most likely next wall is build-disk — cozystack-nocloud is a brand-new package, so the first oras push creates it, and the follow-up oras tag can hit 403 permission_denied until the package is linked (the risk the workflow already documents in-line). The first-ever full e2e run may well surface its own issues.

Reviewing that follow-on separately seems better than growing this PR.

Summary by CodeRabbit

  • Bug Fixes
    • Improved nightly artifact mirroring and staging reliability.
    • Prevented artifact pull failures by ensuring required output directories are created before use.
    • Enhanced retrieval of artifact version and source revision metadata.

…o ops

The Nightly workflow has never completed a run. Every scheduled run since
NIGHTLY_ENABLED was set has failed in `mirror` at "Resolve source commit
and version", ~80s in:

  level=fatal msg="unsupported image-specific operation on artifact with
  type \"application/vnd.cncf.flux.config.v1+json\""

cozystack-packages is a Flux OCI artifact, not a container image: its
config mediaType is application/vnd.cncf.flux.config.v1+json. Plain
`skopeo inspect` parses the config blob as an image config and fatals on
any non-image artifact, so resolving the digest with
`inspect --format '{{.Digest}}'` could never work.

Fetch the :main manifest once with `inspect --raw` and derive both the
digest (`skopeo manifest-digest`, which just hashes the manifest bytes —
no image-specific parsing) and the revision annotation from those exact
bytes. This also tightens the race the original comment describes: the
digest and the annotation now come from a single fetch, so there is no
window at all between pinning the tag and reading its revision, and one
registry round-trip is saved.

Because the workflow never ran past this step, the rest was unproven.
Also fix both `flux pull artifact` calls, which passed an --output
directory that does not exist and would have failed with "invalid output
path": create it first, matching the established idiom in
pull-requests.yaml ("Pull current-main packages tree").

Verified against the live OCIR artifact: the fixed step resolves
packages_ref to the same digest the registry reports and yields main's
current HEAD sha; the fixed pull extracts the tree. skopeo copy on Flux
artifacts, the flux push digest awk parse, the cozy-installer chart name
and nightly-mirror.sh's per-image digest verification were each checked
and are correct as-is.

Signed-off-by: Myasnikov Daniil <[email protected]>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Note

Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.

@github-actions github-actions Bot added area/ci Issues or PRs related to CI workflows, GitHub Actions, automation kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Jul 17, 2026
@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3470d12b-8c22-4a87-94f5-cd587ba4841c

📥 Commits

Reviewing files that changed from the base of the PR and between df541b9 and 90254ef.

📒 Files selected for processing (1)
  • .github/workflows/nightly.yaml

📝 Walkthrough

Walkthrough

The nightly workflow now reads OCI package metadata from a single manifest and creates required output directories before pulling baked and E2E package trees.

Changes

Nightly artifact flow

Layer / File(s) Summary
OCI manifest metadata resolution
.github/workflows/nightly.yaml
The resolve step stores the cozystack-packages:main manifest once, derives its digest, and extracts the revision annotation from the same data.
Artifact output directory staging
.github/workflows/nightly.yaml
The baked package and E2E install tree steps recreate _baked and _rw before running flux pull artifact.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • cozystack/cozystack#3148: Both changes create output directories before pulling OCI artifacts with flux pull artifact.

Suggested reviewers: lexfrei

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main workflow fix: resolving the nightly packages digest via raw OCI artifact metadata instead of image-specific skopeo inspection.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/nightly-resolve

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@myasnikovdaniil
myasnikovdaniil merged commit 8020656 into main Jul 20, 2026
17 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the fix/nightly-resolve branch July 20, 2026 11:49
Timur Tukaev (tym83) added a commit that referenced this pull request Aug 10, 2026
A second internal pass found that the previous commit's disclosure argument did
not hold together and that its support test could not be evaluated by a reader.

The document had nominated the monthly aggregate summaries as safe-to-publish
evidence on the grounds that they hold no vulnerability detail. They do: they
name accepted-risk findings individually by CVE ID, severity and package, some
of them critical, next to a count of confirmed findings awaiting a fix. State
the line where it actually falls — aggregate counts by severity are
publishable, naming an individual unfixed finding is not, because a CVE ID plus
a package plus this project's public pinned component list locates unpatched
exposure — and note that the monthly summary needs reducing to counts before it
can be published. Present the resulting posture as a proposal rather than as a
position the maintainers have taken.

The scanner SLA paragraph said a triage backlog was running against the clock.
The pipeline's own output says something more specific and worse: triage is
keeping up, remediation is not, with the last monthly period closing on no
confirmed findings fixed. Say that instead.

The supported-versions table keyed support to having an active release branch.
Release branches are never deleted, so every line back to the earliest has one
and the test tells a reader nothing; read literally it also committed the
project to every 1.x line. Key it to patch activity within a stated window
instead, restore an end-of-life category the previous commit had removed
entirely, and put the v0.41 cutoff back rather than extending discretionary
backports across all of 0.x.

Also: drop an unevidenced claim that the per-PR checks are healthy, correct the
nightly attribution — the current red run is a regression after #3332 and is
not tracked by an issue, unlike the older failure in #3278 — and qualify how
zizmor gates, through the required pre-commit hook rather than through its own
path-filtered workflow.

Assisted-By: Claude <[email protected]>
Signed-off-by: Timur Tukaev <[email protected]>
Timur Tukaev (tym83) added a commit that referenced this pull request Aug 10, 2026
…timeline

The previous commit narrowed a true statement into a false one. It said the
scanner gap was remediation throughput and that triage ran on schedule; triage
is also behind, and the backlog awaiting a triage decision includes findings at
critical severity well past the one-business-day target. Both tracks are now
stated as behind. Figures stay out of this repository for the disclosure reason
the document already gives and belong in the private channel, so the admission
is qualitative rather than rounded down.

The nightly characterisation was also wrong. It read as fixed-then-regressed,
which understates it: after the fix in #3332 the scheduled run passed for four
days, went red for five, passed for three, and has failed every run since
2026-08-02 — seven of the last twenty-four passed. It is flapping, not newly
broken, and the current failure is untriaged rather than provably unfiled.

Carry the target-attainment caveat into SECURITY.md, which stated the triage
SLAs with no indication they are not being met, and resync the self-assessment
on the monthly report: it names findings individually and so is not publishable
until reduced to counts by severity.

Also note in SECURITY.md what the activity test actually classifies today —
every 1.x line from 1.0 to 1.6 — and how backport effort is prioritised across
that set, since supporting seven lines in parallel otherwise invites the
question it does not answer.

Assisted-By: Claude <[email protected]>
Signed-off-by: Timur Tukaev <[email protected]>
Andrei Kvapil (kvaps) pushed a commit that referenced this pull request Sep 7, 2026
A second internal pass found that the previous commit's disclosure argument did
not hold together and that its support test could not be evaluated by a reader.

The document had nominated the monthly aggregate summaries as safe-to-publish
evidence on the grounds that they hold no vulnerability detail. They do: they
name accepted-risk findings individually by CVE ID, severity and package, some
of them critical, next to a count of confirmed findings awaiting a fix. State
the line where it actually falls — aggregate counts by severity are
publishable, naming an individual unfixed finding is not, because a CVE ID plus
a package plus this project's public pinned component list locates unpatched
exposure — and note that the monthly summary needs reducing to counts before it
can be published. Present the resulting posture as a proposal rather than as a
position the maintainers have taken.

The scanner SLA paragraph said a triage backlog was running against the clock.
The pipeline's own output says something more specific and worse: triage is
keeping up, remediation is not, with the last monthly period closing on no
confirmed findings fixed. Say that instead.

The supported-versions table keyed support to having an active release branch.
Release branches are never deleted, so every line back to the earliest has one
and the test tells a reader nothing; read literally it also committed the
project to every 1.x line. Key it to patch activity within a stated window
instead, restore an end-of-life category the previous commit had removed
entirely, and put the v0.41 cutoff back rather than extending discretionary
backports across all of 0.x.

Also: drop an unevidenced claim that the per-PR checks are healthy, correct the
nightly attribution — the current red run is a regression after #3332 and is
not tracked by an issue, unlike the older failure in #3278 — and qualify how
zizmor gates, through the required pre-commit hook rather than through its own
path-filtered workflow.

Assisted-by: LLM
Signed-off-by: Timur Tukaev <[email protected]>
Andrei Kvapil (kvaps) pushed a commit that referenced this pull request Sep 7, 2026
…timeline

The previous commit narrowed a true statement into a false one. It said the
scanner gap was remediation throughput and that triage ran on schedule; triage
is also behind, and the backlog awaiting a triage decision includes findings at
critical severity well past the one-business-day target. Both tracks are now
stated as behind. Figures stay out of this repository for the disclosure reason
the document already gives and belong in the private channel, so the admission
is qualitative rather than rounded down.

The nightly characterisation was also wrong. It read as fixed-then-regressed,
which understates it: after the fix in #3332 the scheduled run passed for four
days, went red for five, passed for three, and has failed every run since
2026-08-02 — seven of the last twenty-four passed. It is flapping, not newly
broken, and the current failure is untriaged rather than provably unfiled.

Carry the target-attainment caveat into SECURITY.md, which stated the triage
SLAs with no indication they are not being met, and resync the self-assessment
on the monthly report: it names findings individually and so is not publishable
until reduced to counts by severity.

Also note in SECURITY.md what the activity test actually classifies today —
every 1.x line from 1.0 to 1.6 — and how backport effort is prioritised across
that set, since supporting seven lines in parallel otherwise invites the
question it does not answer.

Assisted-by: LLM
Signed-off-by: Timur Tukaev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci Issues or PRs related to CI workflows, GitHub Actions, automation kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants