[cozystack] Major Architecture Refactoring - #1641

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

I'm not sure how well-supported the flux-aio distribution is. (For sure it will make the distribution incompatible with Flux Operator.) Are there other reasons for this switch than the CNI dependency?

I'm working on getting to be a Control Plane customer, to purchase Flux Enterprise - with Flux Operator it's very easy to drop in the Zero-CVE images distribution link. New flux patch releases (even without the Flux Enterprise) can be installed automatically and safely as soon as they become available, and we already do this with the 2.6.x, 2.7.x wildcard in our FluxInstance spec today. The old versions of Flux Operator can usually install newer Flux patch releases, I think, without upgrading Flux Operator first. (have to double check - this might not actually be true)

Still, Control Plane updates flux-operator with every Flux release, and it's the fastest way to get the latest version of Flux. With flux-aio, we will have to wait for Stefan Prodan to do the flux-aio release - which sometimes comes a bit later. For example, Flux 2.7.4 was released two days ago, but flux-aio is still on Flux 2.7.3. I don't think this is necessarily a deal-breaker but I want to take a breath and make some contact so I confirm that Flux-AIO is really going to be supported into the future before I let you make this change without comment :)

In reality, Flux 2.7 has been released for over a month and we're (Cozystack is) still on Flux 2.6, and that's partially conservativeness, I heard there were some regressions in the earlier 2.7 releases, but also partially just my fault, (been busy lately!) - I don't think Cozystack users have a high expectation of getting the latest Flux release instantly after it comes out, so maybe this is all fine.

If there are other reasons than CNI, I would like to understand them better before this architecture change is finalized.

I think that dropping the Flux Operator will also lose us the new (upcoming) Flux Status Page feature, that was announced at FluxCon.

Also, I was excited to see the inclusion of ExternalArtifacts and the source-watcher in the updated architecture 💪

I should also mention:

Upgrade Procedure
Before upgrading to Flux v2.7, you need to run flux migrate on all clusters.

In case you haven't seen it in the Flux 2.7 upgrade notes:
https://github.com/stefanprodan/flux-aio/releases/tag/v2.7.3-0

https://github.com/fluxcd/flux2/releases/tag/v2.7.0#:~:text=S3%2Dcompatible%20storage.-,Upgrade%20procedure,-%E2%9A%A0%EF%B8%8F

The Flux Operator simplifies upgrades, see this section "Upgrade Procedure"

This is about migrating etcd storage. It isn't likely to have any impact on the way that Flux is used in Cozystack, based on the way it's currently used. Flux 2.7 is the first release to have this upgrade procedure requirement, the CLI now adds flux migrate - if you are using Flux Operator, it migrates the storage automatically. If you do not use it, you need to be aware of the new flux migrate command when doing a Flux upgrade. (Cozystack installer can run this in the migrate step, as it typically happens)

Anyway, in the general case, if your storage is in Git, and you upgrade your API versions on each resource by changing the API version in the file in the Git repository and pushing the change to Flux, that doesn't necessarily upgrade the storage version of the Flux resource in Etcd - until there is a change in resource version, Kubernetes will not touch the storage in etcd and the release will still be the same version, it is a "migrate-on-upgrade."

The flux migrate command forces the upgrade.

This is important when old beta APIs are finally deleted (really something that is maybe happening for the first time in Flux 2.7), in case the resources as they're defined in the cluster haven't changed since the API version moved from v1betaX to v1, then the storage won't be automatically migrated, and the upgrade will likely fail. It's just one extra step, not really a reason to not use flux-aio. I think that Stefan is going to continue building timoni and flux-aio - I think I can support this move. I just want to double-check and confirm everything so we're doing this as responsibly as possible.

Ref:

• Deploy Flux AIO with hostNetwork controlplaneio-fluxcd/flux-operator#489

One major negative downside to removing Flux Operator, is it removes also the opportunity for Flux Enterprise users. (I just got my Flux Enterprise subscription, and am about to test enabling it on Cozystack)

This distro enables faster CVE responses, FIPS encryption, and other compliance checkboxes that would potentially be a blocker for example at my employer if they can't be covered. Flux Enterprise only covers these needs, Flux Upstream does just what the open source project supports. I'm not sure if your mind is made up about this already Andrei Kvapil (@kvaps)

I guess this is free for the user to customize, if you will adopt flux-aio then I will support in my fork flux-operator and that is fine A-OK (it is a reversible decision and it will be easy enough for the user to switch it for themselves, whether they used a hypothetical distro-A or distro-B of Cozystack)

Kingdon Barrett (@kingdonb) I'd like to preserve the opportunity of using flux enterprise with Cozystack.

Right now you have the option to disable flux management by cozystack-operator:

This way, you can install Flux manually via flux-operator and just let Cozystack use it.

Another idea is to provide oportunity to the user to specify kustomize patches for flux-aio. It could be preserved in Secret or ConfigMap. Would it be enough to install flux enterprise?

I think so, yes. Have to come back to this when I can put hands on it. I had some troubles upgrading to 0.38 but they're all resolved now! The changelog is big this time 😅

BYO Flux sounds like a very reasonable interface! I think that flux-aio is probably also compatible with the images from the Flux Enterprise distribution, but that's something I'll need to check out for myself yet as well.

We have decided to not merge this PR as-is, and go apply changes one-by-one.

The plan:

• Migrate to flux-aio (kubectl apply from a script) + migration to remove the old Flux
• Move the asset-server into a separate Deployment
• Add labels to HelmReleases for CozystackResourceDefinitions, cozystack-controller, webhook, and the API server + migration (patch existing resources)
• Get rid of lookups and drop hr reconcilers
• Extract Talos into a separate component
• cozystack-operator
• remove the asset-server and switch to artifactgenerators and ocirepository/gitrepository + move Grafana dashboards into a separate Deployment
• start installing the platform chart via Flux
• move migrations into the platform chart

closed in favor #1816