cozystack-operator: workload scan issues five uncached etcd LISTs per system namespace per reconcile #3977

findRequestAboveDefaultLimit in internal/operator/package_reconciler.go lists Deployments, StatefulSets, DaemonSets, CronJobs and Jobs through APIReader, so all five bypass manager cache and hit etcd as quorum reads. This runs for every system namespace on every Package reconcile. Pod list is skipped once namespace settled at configured limit, these five are not, nothing gates creation of a Deployment.

Package reconciles follow HelmRelease status churn, so on ~30 system namespaces that is around 150 direct etcd LISTs per churn event. Workqueue coalesces events so it is not a storm, but it is constant background cost growing with number of system namespaces.

Bypassing cache is right for pods, cluster-wide pod informer costs the operator more memory than the whole feature saves. That argument doesn't transfer to workload kinds, they are few and small.

Problem is neither scoping mechanism fits:

• routing five kinds through manager cache starts five lazy cluster-wide informers, and jobs accumulate
• scoping them with cache.Options.ByObject needs namespace set at cache construction time, operator discovers it from Packages at reconcile time and it grows as packages get installed
• cache label selector filters cached objects and not their namespaces, and a Deployment carries no label about namespace it sits in

So what is left is bigger than swapping a reader. Either manage informer lifecycle by hand as system namespace set changes, or list each kind once cluster-wide per reconcile pass and bucket by namespace, which is cheapest but turns a namespaced read into cluster-scoped one, or drive the scan from workload events instead of Package reconciles.

Before any of that someone needs to measure real reconcile rate on a busy cluster with realistic package set. Choice between those depends on it, and it may well show current behaviour is fine as is.

Raised in review of #3494 and deferred there on purpose, bounded and namespaced, not worth guessing a caching strategy in the same change that introduced the scan.