RFC: immutable release tags & rc-promotion flow #2677

TL;DR

Replace the current "tag first, force-retag after digest bake" release flow with one where every git tag is write-once. Stable vX.Y.Z tags are created exclusively by the release bot by promoting an existing rc. Maintainers and a nightly job push only rc tags. api/apps/v1alpha1/vX.Y.Z is created only with stable, never for rc.

Current state (problem)

Today (see docs/release.md:70, 164) the maintainer pushes the final vX.Y.Z tag up front. CI builds, creates a "Prepare release" commit on release-X.Y.Z with vendored image digests, and on PR merge force-moves vX.Y.Z from the maintainer's commit to the merge commit. Same flow force-creates api/apps/v1alpha1/vX.Y.Z.

This breaks tag immutability:

• Go module proxy / pkg.go.dev cache poisoning on api/apps/v1alpha1/*
• SBOM and provenance attestation drift
• Registry/scanner alerts on mutated tags

Force-retag sites in CI:

• .github/workflows/tags.yaml:147-148
• .github/workflows/tags.yaml:200-201
• .github/workflows/pull-requests-release.yaml:63-64, 97-103
• .github/workflows/auto-release.yaml:170-187

Goals

1. No tag is ever force-pushed or moved. Once created, the commit it points at is permanent.
2. Stable vX.Y.Z is created exclusively by the bot, exclusively by promoting an existing rc.
3. api/apps/v1alpha1/vX.Y.Z is created only with stable. RCs do not get Go module tags.
4. Nightly rc tags are produced automatically from main and active maintenance branches.
5. Preparing a release (push rc) and blessing a release (promote rc → stable) are decoupled operations.

Non-goals

• Changing release tarball / OCI artifact / GHCR layout.
• Changing semver.
• Migrating already-published tags. New rules apply to tags created after rollout.

Tag taxonomy

Enforced by GitHub tag protection: no human or workflow may push -f to refs/tags/v* or refs/tags/api/apps/v1alpha1/*.

Lifecycle

Nightly rc

gitGraph
  commit id: "main HEAD"
  commit id: "later commit" tag: "v0.43.0-nightly.20260520"

Cron job. For each tracked target (main and active release-X.Y): skip if HEAD unchanged since last nightly; otherwise create <version>-nightly.<date> and push. Idempotent — re-runs on the same day are a no-op.

Promote rc → stable

gitGraph
  commit id: "main HEAD"
  commit id: "feature" tag: "v0.42.0-rc.3"
  branch promote-v0.42.0
  commit id: "Prepare release v0.42.0 (bake digests)"
  checkout main
  merge promote-v0.42.0 id: "PR merge" tag: "v0.42.0"

1. Maintainer triggers Promote RC workflow with rc_tag=v0.42.0-rc.3.
2. Bot validates: rc exists, stable does not, rc check suites green, optional approval gate.
3. Bot checks out the rc commit, runs make build to bake digests, commits "Prepare release vX.Y.Z" on promote-vX.Y.Z, opens PR (squash disabled).
4. Maintainer reviews/merges.
5. On merge: bot creates vX.Y.Z and api/apps/v1alpha1/vX.Y.Z at the merge commit (git tag, no -f). Bot publishes the GitHub Release.

Because the stable tag attaches to a commit that did not exist before the promotion PR opened, there is no prior SHA to overwrite — force is impossible by construction.

Workflow inventory

Retired/changed:

• auto-release.yaml — old "auto-bump patch stable" behavior dropped; stable always requires explicit promote.
• tags.yaml — split into tag-build.yaml + promote-rc.yaml.
• pull-requests-release.yaml — replaced by finalize-stable.yaml; force-tag step removed.

Retention

Repo settings

1. Tag protection rule: pattern v* — only cozystack-ci[bot] can create; no one can delete/update.
2. Tag protection rule: pattern api/apps/v1alpha1/* — same.
3. "Limit branches/tags updated in a single push" = 1.

Rollout plan

Land in stages, each independently revertible.

• Stage 1 — defensive cleanup (no flow change). Add compare-before-force guards to existing workflows. Independently useful.
• Stage 2 — nightly rc + retention. Add nightly-rc.yaml (disabled flag for first week) and retention.yaml (dry-run mode first).
• Stage 3 — split build from retag. Extract image build into tag-build.yaml; leave existing stable-tag path intact for now.
• Stage 4 — promotion path. Add promote-rc.yaml + finalize-stable.yaml; add tag protection rules; update docs/release.md.
• Stage 5 — retire old paths. Delete digest-bake-and-retag steps from tags.yaml; clean up auto-release.yaml, pull-requests-release.yaml.

Open questions

1. Squash vs merge commits. Promotion flow needs a real merge commit. Either disallow squash on promote-* PRs, or tag the rebased commit produced by squash. Recommend: disallow squash on these PRs.
2. What make build writes to source. If digests can be moved out of source entirely (release manifest only), the promotion step shrinks to "tag the rc commit directly" — no PR, no merge commit. Worth investigating in parallel; not required to ship this design.
3. api/apps/v1alpha1 during rc. Consumers testing against an rc would have to vendor by commit SHA. Acceptable, since publishing rc Go module tags would re-introduce the immutability problem.
4. In-flight release-1.x. Don't migrate. New flow applies from the next minor onward.

Risks

Related

• Move E2E tests out of PR pipeline to release/nightly #2397 — move E2E tests out of PR pipeline to release/nightly (overlaps with nightly-rc.yaml build cost)
• API stabilization and release v1.0.0 #1251 — API stabilization & release v1.0.0 (prior release-process epic)
• Use OCI container registry for non-release builds #1151 — OCI container registry for non-release builds