Skip to content

fix(platform): make *-rd sources depend on cozystack-engine - #3041

Merged
IvanHunters merged 2 commits into
mainfrom
ray/fix-applicationdefinition-crd-ordering
Jun 25, 2026
Merged

IvanHunters merged 2 commits into
mainfrom
ray/fix-applicationdefinition-crd-ordering

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On an unlucky Flux reconcile order, a *-application PackageSource that ships a system/*-rd component can install before the applicationdefinitions.cozystack.io CRD exists. Every *-rd chart renders an ApplicationDefinition (cozystack.io/v1alpha1); that CRD is installed by the application-definition-crd component of cozystack.cozystack-engine. When an *-rd HelmRelease renders first, it hard-fails with no matches for kind "ApplicationDefinition" in version "cozystack.io/v1alpha1", and that cascades into a broad install timeout — surfacing even on changes that have nothing to do with ordering, because it is a latent race.

Fix

Add - cozystack.cozystack-engine to the dependsOn of every *-application source that ships a system/*-rd component (29 sources). gateway-application already carried this edge — it was added earlier for the TenantGateway CRD, the same bug class, but only for that one source. A new invariant test fails if any source with a system/*-rd component is missing the edge.

Why

The ordering was previously implicit, relying on Flux happening to reconcile cozystack-engine before the *-application sources. Making the edge explicit removes the race deterministically. No dependency cycle is introduced: cozystack-engine depends only on networking and cert-manager (plus keycloak in its oidc variant), none of which reach back to any *-application.

Test

  • New generic guard hack/check-applicationdefinition-crd-ordering.bats (auto-discovered by make bats-unit-tests, runs in CI via make unit-tests): parses every source and fails if any PackageSource variant with a system/*-rd component lacks the cozystack.cozystack-engine edge — so future sources are covered, not just today's set.
  • helm unittest packages/core/platform: 13 suites / 66 tests pass; the source edits do not disturb existing assertions.

Summary by CodeRabbit

  • Bug Fixes

    • Improved dependency ordering across several platform packages to ensure related components install in the correct sequence.
    • Prevented end-to-end test selection from being overly broad when only ordering-related dependencies change.
  • Tests

    • Added checks to catch missing dependency requirements and to verify test selection behavior stays accurate.

Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.

Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@github-actions github-actions Bot added area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Jun 24, 2026
@coderabbitai

coderabbitai Bot commented Jun 24, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: bf59de21-ed8b-4ae1-8722-6d6b8cc95c5e

📥 Commits

Reviewing files that changed from the base of the PR and between 29bf56f and 02eeb9e.

📒 Files selected for processing (32)
  • hack/check-applicationdefinition-crd-ordering.bats
  • hack/select-e2e.sh
  • hack/select-e2e_test.bats
  • packages/core/platform/sources/bootbox-application.yaml
  • packages/core/platform/sources/bucket-application.yaml
  • packages/core/platform/sources/clickhouse-application.yaml
  • packages/core/platform/sources/etcd-application.yaml
  • packages/core/platform/sources/external-dns-application.yaml
  • packages/core/platform/sources/foundationdb-application.yaml
  • packages/core/platform/sources/harbor-application.yaml
  • packages/core/platform/sources/http-cache-application.yaml
  • packages/core/platform/sources/info-application.yaml
  • packages/core/platform/sources/ingress-application.yaml
  • packages/core/platform/sources/kafka-application.yaml
  • packages/core/platform/sources/kubernetes-application.yaml
  • packages/core/platform/sources/mariadb-application.yaml
  • packages/core/platform/sources/mongodb-application.yaml
  • packages/core/platform/sources/monitoring-application.yaml
  • packages/core/platform/sources/nats-application.yaml
  • packages/core/platform/sources/openbao-application.yaml
  • packages/core/platform/sources/opensearch-application.yaml
  • packages/core/platform/sources/postgres-application.yaml
  • packages/core/platform/sources/qdrant-application.yaml
  • packages/core/platform/sources/rabbitmq-application.yaml
  • packages/core/platform/sources/redis-application.yaml
  • packages/core/platform/sources/seaweedfs-application.yaml
  • packages/core/platform/sources/tcp-balancer-application.yaml
  • packages/core/platform/sources/tenant-application.yaml
  • packages/core/platform/sources/virtualprivatecloud-application.yaml
  • packages/core/platform/sources/vm-disk-application.yaml
  • packages/core/platform/sources/vm-instance-application.yaml
  • packages/core/platform/sources/vpn-application.yaml

📝 Walkthrough

Walkthrough

PackageSource manifests now add cozystack.cozystack-engine to dependsOn for variants with system/*-rd components, select-e2e.sh filters that edge from reverse-dependency traversal, and new BATS tests cover the selection behavior and the dependency invariant.

Changes

Engine dependency ordering

Layer / File(s) Summary
PackageSource engine dependencies
packages/core/platform/sources/*.yaml
PackageSource variants add cozystack.cozystack-engine to dependsOn; postgres-application.yaml updates an existing engine entry and rabbitmq-application.yaml removes it.
Reverse-dependency selection
hack/select-e2e.sh, hack/select-e2e_test.bats
select-e2e.sh excludes cozystack.cozystack-engine from reverse-dependency edges, and the new BATS case checks that postgres and harbor are selected while kafka is not.
CRD ordering invariant
hack/check-applicationdefinition-crd-ordering.bats
A new BATS script scans packages/core/platform/sources/*.yaml, counts system/*-rd variants, and fails when any such variant lacks cozystack.cozystack-engine in dependsOn.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • cozystack/cozystack#2842: Also updates packages/core/platform/sources/* dependsOn relationships around cozystack.cozystack-engine.

Suggested reviewers

  • kvaps
  • androndo
  • sircthulhu
  • lllamnyp
  • IvanHunters
  • myasnikovdaniil

Poem

A bunny hops through YAML lanes,
With engine crumbs and tidy chains.
The *-rd moon now shines so bright,
And tests go nibble-gently right.
🐇✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: platform *-rd sources now depend on cozystack-engine.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ray/fix-applicationdefinition-crd-ordering

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…election

Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.

The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.

Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a race condition during cluster installation where *-application sources could attempt to render ApplicationDefinition resources before the cozystack-engine component had installed the necessary CRD. By explicitly defining this dependency across all relevant sources, the installation order is now deterministic. Additionally, the PR adds automated guardrails to prevent future regressions and optimizes CI test selection logic to avoid redundant test runs.

Highlights

  • Dependency Ordering Fix: Added cozystack.cozystack-engine as a dependency to all *-application sources that include a system/*-rd component to ensure the ApplicationDefinition CRD is present before rendering.
  • Invariant Testing: Introduced a new BATS test (hack/check-applicationdefinition-crd-ordering.bats) to automatically verify that all future *-application sources with system/*-rd components include the required dependency.
  • Test Selection Optimization: Updated hack/select-e2e.sh to exclude cozystack.cozystack-engine from reverse dependency calculations, preventing unnecessary test fan-out while maintaining safety nets.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request ensures that all application PackageSources containing a system/*-rd component explicitly declare a dependency on cozystack.cozystack-engine to guarantee the ApplicationDefinition CRD is installed prior to reconciliation. A new BATS test is added to enforce this invariant. To prevent this ordering dependency from triggering E2E tests for all applications, hack/select-e2e.sh is updated to exclude cozystack.cozystack-engine from test-impact propagation, backed by a new test. The reviewer feedback suggests using optional chaining (?.) in the jq queries of the new BATS test to handle potentially malformed PackageSource files more robustly.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +31 to +37
rd_total="$(printf '%s' "$json" | jq -s '
[ .[]
| select(.kind == "PackageSource")
| (.spec.variants // [])[]
| select((.components // []) | any(.path // "" | test("^system/[a-z0-9-]+-rd$")))
] | length
')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

To make the test more robust and prevent potential jq execution failures if a malformed or incomplete PackageSource file is present in the directory, consider using optional chaining (?.) for accessing .spec.variants. This aligns with defensive programming best practices.

  rd_total="$(printf '%s' "$json" | jq -s '
    [ .[]
      | select(.kind == "PackageSource")
      | (.spec?.variants // [])[]
      | select((.components // []) | any(.path // "" | test("^system/[a-z0-9-]+-rd$")))
    ] | length
  ')"

Comment on lines +43 to +52
offenders="$(printf '%s' "$json" | jq -rs '
[ .[]
| select(.kind == "PackageSource")
| .metadata.name as $name
| (.spec.variants // [])[]
| select((.components // []) | any(.path // "" | test("^system/[a-z0-9-]+-rd$")))
| select(((.dependsOn // []) | any(. == "cozystack.cozystack-engine")) | not)
| "\($name) [variant: \(.name)]"
] | .[]
')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

Similarly, use optional chaining (?.) when extracting the metadata name and variants to ensure the test does not crash on malformed files missing these fields.

  offenders="$(printf '%s' "$json" | jq -rs '
    [ .[]
      | select(.kind == "PackageSource")
      | .metadata?.name as $name
      | (.spec?.variants // [])[]
      | select((.components // []) | any(.path // "" | test("^system/[a-z0-9-]+-rd$")))
      | select(((.dependsOn // []) | any(. == "cozystack.cozystack-engine")) | not)
      | "\\($name) [variant: \\(.name?)]"
    ] | .[]
  ')"

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the diff. Edge correctly added to 29 *-application sources, no cycle (engine depends only on networking/cert-manager, and on keycloak/keycloak-operator in the oidc variant; none of those route back to *-application). bats invariant prevents future drift. Excluding engine from the propagation hub in test-impact analysis is correct (this is a lifecycle edge, not a behavioral one) and the new "engine-dependency change does not fan out" test pins that. LGTM.

@IvanHunters
IvanHunters merged commit e0597e3 into main Jun 25, 2026
14 checks passed
@IvanHunters
IvanHunters deleted the ray/fix-applicationdefinition-crd-ordering branch June 25, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants