Skip to content

docs(release): expand release.md into release-engineer playbook - #2732

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
release-engineer-guide
Jun 9, 2026
Merged

Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
release-engineer-guide

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented May 25, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Expands docs/release.md from a ~165-line process-design doc into a ~573-line single source covering both the process design (kept verbatim — mermaid diagrams and the RC/regular/patch numbered walkthroughs) and the operational playbook for cutting a release.

New sections added under the existing diagrams:

  • What CI does on a tag push — phase-by-phase walk through tags.yaml (prepare-release, generate-changelog, update-website-docs), pull-requests-release.yaml (finalize), and update-releasenotes.yaml (sync release body).
  • Automated patch tag — auto-release.yaml's nightly behavior.
  • Backports — bot mechanism, recursive-backport regression history (fix fix(ci): stop backport workflow from cascading on its own PRs #2584), manual fallback, cherry-pick triage include/skip rule, monotonic-counter trap for migration backports.
  • Changelogs — pointer to agents/changelog.md plus the recurring failure modes that have shipped in real changelogs, with a verification snippet (chart-bump coverage, range-vs-changelog PR sweep, URL liveness, website PR-author check).
  • Release-time fires — three customer-impacting regressions (v1.3.0 cert-manager ingressClassName, v1.4.0 kubevirt-instancetypes upgrade, v1.4.0 Flux 2.7 → 2.8 readiness deadlock) with "what would have caught this" notes.
  • Pre-release verification checklist, CI failures release engineers commonly hit, Force-retagging (with open RFC RFC: immutable release tags & rc-promotion flow #2677 context), Splitting a release-blocking bundle PR, Cleanup after release.

Example version numbers across the walkthrough switched from the upstream v0.42.x legacy placeholders to simple semver-style (v1.2.0, v1.2.1, release-1.2) consistent with semver.org examples and applicable at any point in v1.x.

Release note

NONE

Summary by CodeRabbit

  • Documentation
    • Enhanced release process documentation with clarified CI workflows, comprehensive verification checklists, expanded troubleshooting guidance, and case studies to ensure more reliable releases.

Review Change Stack

Adds operational sections under the existing process-design walkthrough:
CI phase-by-phase behavior, backport bot mechanics + cherry-pick triage,
monotonic-counter migration trap, changelog failure modes + verification,
three release-time war stories, pre-release verification checklist,
force-retagging discussion + RFC link, bundle-PR splitting strategy.

Mermaid diagrams and numbered RC/regular/patch walkthroughs are kept
verbatim. Example versions normalized to simple semver-style (v1.2.0,
v1.2.1, release-1.2) consistent across the doc.

Assisted-By: Claude <[email protected]>
Signed-off-by: Myasnikov Daniil <[email protected]>
@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/documentation Categorizes issue or PR as related to documentation size/XL This PR changes 500-999 lines, ignoring generated files labels May 25, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the project's release documentation by transforming the existing release workflow document into a robust operational playbook. By centralizing CI workflow explanations, backporting procedures, and historical incident data, the update provides release engineers with the necessary context and tools to manage releases more reliably and effectively.

Highlights

  • Documentation Expansion: Transformed docs/release.md from a basic process document into a comprehensive operational playbook for release engineers.
  • CI and Automation Details: Added in-depth sections covering CI workflows, automated patch tagging, and the backport bot mechanism.
  • Incident Prevention: Introduced a 'Release-time fires' section documenting historical regressions and specific verification steps to prevent similar issues.
  • Operational Readiness: Included a pre-release verification checklist and a troubleshooting guide for common CI failures encountered during the release process.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@coderabbitai

coderabbitai Bot commented May 25, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

This PR updates docs/release.md to modernize the release process documentation with newer version examples (v1.2.0 instead of v0.42.x), significantly expands operational guidance for CI automation, backport mechanics, changelog verification, and release troubleshooting, while maintaining the existing playbook structure.

Changes

Release Process Documentation Refresh

Layer / File(s) Summary
Release guide overview and regular release workflow
docs/release.md
"When to use this guide" updated with concrete tag/release/cherry-pick scenarios and tag-push requirements for tags.yaml. "Regular Releases" section rewritten with v1.2.0 example, updated CI tag-push → prepare-release → merge sequence, and adjusted mermaid diagram.
Patch release workflow and process
docs/release.md
"Patch Releases" introductory scenario and step-by-step process rewritten for v1.2.1-style patches, including release branch creation, cherry-pick constraints, prepare-release PR review expectations, and updated mermaid flow.
CI automation on tag push and patch automation
docs/release.md
Substantially expanded "What CI does on a tag push" section: enumerated workflows and phases, detailed phase semantics (prepare-release, generate-changelog, update-website-docs, finalize as hard gate or non-blocking), changelog token/scoping behavior, retry/verification expectations, and automated nightly patch tag behavior.
Backport automation mechanics and triage
docs/release.md
Expanded "Backports" section: backport bot triggers and label routing, recursion/guard fix approach (gating on base ref), conflict-resolution handling commands, cherry-pick triage rules (include/skip/borderline for patches), and new section on monotonic-counter backport hazards with migration targetVersion forward-upgrade mitigations.
Changelog process and pre-publish verification
docs/release.md
Substantially expanded changelog guidance: canonical process reference, detailed common failure modes, extensive pre-publish verification checklist with chart bump mention checks, PR existence/merge validation, URL liveness tests, and website author attribution, plus Cozystack-specific URL quirks and platform-component coverage rules.
Release verification, troubleshooting, and post-release operations
docs/release.md
Added "Release-time fires" regression writeups and pre-release verification checklist with cluster/upgrade/E2E assertions; updated CI failure matrix with release-engineer heuristics and workflow-regression guidance; refreshed force-retagging touchpoints, bundle PR splitting guidance, post-release cleanup steps, and "See also" links to workflows and agent docs.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 The releases now flow with clarity bright,
From v1.2.0 through patches done right,
With CI phases and backport guards in place,
The playbook's refreshed—a better pace!
Checklists and fires now guide the way,
Release day's smoother, hooray hooray! 🎉

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR expands release.md documentation but does not implement the CI workflow fixes required by #2584 (gating backport.yaml on base.ref == 'main' and removing pr-labeler backport label logic). Update backport.yaml and pr-labeler.yaml workflows with the required CI fixes from #2584 to prevent cascading backport PRs, or clarify if this PR is documentation-only and those fixes are addressed in a separate PR.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: expanding docs/release.md into a comprehensive release-engineer playbook.
Out of Scope Changes check ✅ Passed The PR is strictly limited to expanding docs/release.md with documentation; all changes are in-scope documentation updates aligned with the stated objective.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch release-engineer-guide

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates docs/release.md to provide a comprehensive release workflow guide, detailing CI phases, automated patch tags, backports, changelogs, and pre-release verification checklists. The review feedback identifies several issues in the documented shell commands and scripts: a grep command needs to filter out external repository references to prevent validation errors, a kubectl apply command is missing -f - to read from standard input, a git push -f command should explicitly specify the remote and tag to avoid force-pushing the current branch, and a curl command should include the -L flag to follow redirects during URL liveness checks.

Comment thread docs/release.md
git diff --name-only $PREV..$NEW -- 'images/*/Dockerfile'

# C. Every cited #NNNN exists and is a merged PR (not an issue)
grep -oE '#[0-9]+' docs/changelogs/$NEW.md | sort -u | sed 's/#//' | xargs -P 20 -I{} sh -c '

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The grep -oE '#[0-9]+' command extracts all PR/issue numbers from the changelog. However, this also matches external repository references like cozystack/website#456, extracting #456 and incorrectly querying it against the main cozystack/cozystack repository. This leads to false positives or validation errors during the release check.

Filtering out cozystack/website# references before running grep resolves this issue.

Suggested change
grep -oE '#[0-9]+' docs/changelogs/$NEW.md | sort -u | sed 's/#//' | xargs -P 20 -I{} sh -c '
sed 's|cozystack/website#[0-9]*||g' docs/changelogs/$NEW.md | grep -oE '#[0-9]+' | sort -u | sed 's/#//' | xargs -P 20 -I{} sh -c '

Comment thread docs/release.md
- [ ] **Upgrade from previous minor** (`v(X-1).Y.0`) for RC and minor releases. Same assertions.
- [ ] **At least one tenant with `publishing.ingressName != tenant-root`** and a real Let's Encrypt-staging cert issuance.
- [ ] **Apply-mutate-reapply** on the kubernetes umbrella HelmRelease (catches second-reconcile bugs).
- [ ] **For every CRD-source bump** (kubevirt, cert-manager, flux, cilium, kamaji): `helm template | kubectl apply --dry-run=server` against the freshly-installed CRDs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The kubectl apply --dry-run=server command is missing the -f - argument to read the template from standard input (piped from helm template). Without it, the command will fail with an error indicating that no file or directory was specified.

It should be:
helm template | kubectl apply --dry-run=server -f -

Comment thread docs/release.md

Fires on merge of a PR with the `release` label and head branch matching `release-X.Y.Z[-suffix]`. Three steps:

1. **Force-move the tag** to the merge commit (`git tag -f <vX.Y.Z> <github.sha> && git push -f`). Intentional and load-bearing for the digest-bake flow — see [Force-retagging](#force-retagging).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

The command git push -f without specifying the remote and tag name might fail or force-push the current branch instead of the tag, depending on the user's local git configuration (e.g., push.default).

Specifying the remote and tag explicitly is safer and more robust:
git push origin -f <vX.Y.Z>

Comment thread docs/release.md

# D. URL liveness
grep -oE 'https://[^)" ]+' docs/changelogs/$NEW.md | sort -u | xargs -P 10 -I{} sh -c '
code=$(curl -s -o /dev/null -w "%{http_code}" {})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

The curl command used to check URL liveness does not follow redirects by default. Since some valid URLs (such as cozystack.io paths or external documentation links) may redirect (e.g., from HTTP to HTTPS or due to path normalization), they will return a 3xx status code instead of 200 and be falsely flagged as broken.

Adding the -L (or --location) flag to curl ensures that redirects are followed and only truly broken links are reported.

Suggested change
code=$(curl -s -o /dev/null -w "%{http_code}" {})
code=$(curl -sL -o /dev/null -w "%{http_code}" {})

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/release.md`:
- Line 125: Fix the typo in the branch name shown as `release-1.2,` by removing
the trailing comma so it reads `release-1.2`; update the text where the branch
name appears (the backticked literal `release-1.2,`) to the corrected
`release-1.2` to prevent copy/paste errors during release prep.
- Around line 240-241: Update the release docs to state that the maintenance
branch named release-X.Y is force-updated unconditionally to the tag commit when
it already exists (rather than “if newer”) so the documented behavior matches
the workflow: replace the phrase “force-updated if newer” with language that the
workflow will force-update the existing release-X.Y branch to point to the tag
commit every run, and mention the exact branch name pattern (`release-X.Y`) and
the unconditional force-update to the tag commit to avoid operator confusion
during triage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f27052c3-7188-4694-a6d6-20f04df6c981

📥 Commits

Reviewing files that changed from the base of the PR and between 2cb16d5 and 6d8634e.

📒 Files selected for processing (1)
  • docs/release.md

Comment thread docs/release.md

1. The maintainer creates a release branch, `release-0.42,` and cherry-picks patch commits from `main` to `release-0.42`.
These must be only patches to features that were present in version `v0.42.0`.
1. The maintainer creates a release branch, `release-1.2,` and cherry-picks patch commits from `main` to `release-1.2`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Fix branch name typo in patch workflow step.

release-1.2, includes a trailing comma inside the branch name and can cause copy/paste mistakes during release prep.

Suggested edit
-1. The maintainer creates a release branch, `release-1.2,` and cherry-picks patch commits from `main` to `release-1.2`.
+1. The maintainer creates a release branch, `release-1.2`, and cherry-picks patch commits from `main` to `release-1.2`.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
1. The maintainer creates a release branch, `release-1.2,` and cherry-picks patch commits from `main` to `release-1.2`.
1. The maintainer creates a release branch, `release-1.2`, and cherry-picks patch commits from `main` to `release-1.2`.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/release.md` at line 125, Fix the typo in the branch name shown as
`release-1.2,` by removing the trailing comma so it reads `release-1.2`; update
the text where the branch name appears (the backticked literal `release-1.2,`)
to the corrected `release-1.2` to prevent copy/paste errors during release prep.

Comment thread docs/release.md
Comment on lines +240 to +241
2. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing, force-updated if newer.
3. **Publish the draft release.** `make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Correct maintenance-branch update semantics to match workflow behavior.

The docs say release-X.Y is force-updated “if newer,” but the workflow force-updates unconditionally to the tag commit when the branch exists. This should be documented precisely to avoid wrong operator assumptions during release triage.

Suggested edit
-2. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing, force-updated if newer.
+2. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing; if it already exists, it is force-updated to the tag commit.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
2. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing, force-updated if newer.
3. **Publish the draft release.** `make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`).
2. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Created if missing; if it already exists, it is force-updated to the tag commit.
3. **Publish the draft release.** `make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/release.md` around lines 240 - 241, Update the release docs to state
that the maintenance branch named release-X.Y is force-updated unconditionally
to the tag commit when it already exists (rather than “if newer”) so the
documented behavior matches the workflow: replace the phrase “force-updated if
newer” with language that the workflow will force-update the existing
release-X.Y branch to point to the tag commit every run, and mention the exact
branch name pattern (`release-X.Y`) and the unconditional force-update to the
tag commit to avoid operator confusion during triage.

@myasnikovdaniil myasnikovdaniil self-assigned this May 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — large, well-organized expansion of the release doc into an operational playbook; the process description and walkthroughs are accurate. A few embedded copy-paste commands have minor bugs worth fixing, but they are docs-only and do not affect the repo.

Business context: docs/release.md was a process-design doc with no operational runbook; this adds the phase-by-phase CI walk-through, backport mechanics, changelog verification, real release-time regressions, and a pre-release checklist.

Non-blocking follow-ups (snippet hygiene — a release engineer runs these verbatim):

  1. The CRD checklist item helm template | kubectl apply --dry-run=server needs -f -; without it kubectl apply errors with "must specify one of -f and -k" instead of reading stdin.
  2. The URL-liveness check curl -s -o /dev/null -w "%{http_code}" does not pass -L, so any 301/302 redirect (common on docs hosts) reports as non-200 and yields false "broken URL" hits — add -L.
  3. The #NNNN sweep grep -oE '#[0-9]+' also matches numbers inside code blocks and URLs in the changelog; harmless (extra refs resolve to ISSUE/null), but a tighter match gives a cleaner list.
  4. The branch-name and git push -f snippets flagged on the thread are worth the same copy-paste-correctness pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/documentation Categorizes issue or PR as related to documentation size/XL This PR changes 500-999 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants