Repository navigation
fix(postgres): follow-up review remarks from #3954 (KEDA autoscaling) - #4188
Conversation
… query The read-load trigger query joined on kube_pod_labels without pinning job="kube-state-metrics", while the sibling DatabaseAutoscaler* alert rules already pin it. A second exporter emitting a kube_pod_labels series (or a duplicate scrape) would multiply the join and distort the desired-count arithmetic. Pin the job on every kube_pod_labels selector in the query for parity with the alerts. Follow-up from #3954 (#4187). Signed-off-by: Alexey Artamonov <[email protected]>
…tions parameter cozy-lib.keda.scaledObject only checked max >= min, so an all-negative or mixed-sign pair (min -5, max -3) passed the ordering guard and rendered a ScaledObject with a nonsensical floor. Reject minReplicaCount < 0 before the ordering check. Also add a cozy-lib-tests case for the documented annotations parameter, which had no caller and no coverage. Follow-up from #3954 (#4187). Signed-off-by: Alexey Artamonov <[email protected]>
… client-side rollout
The KEDA-missing fail-closed guard's comment still described the replaced
drop-instances design ("db.yaml drops spec.instances ... CNPG collapses to 1");
under the shipped seed design db.yaml seeds instances at max(replicas,
effectiveMin), so a skipped ScaledObject leaves a fixed-size cluster with no HPA
- autoscaling silently absent, not a collapse. Correct the rationale. Also note
on the cozyrds apply-strategy annotation that the client-side switch rolls out
per-app as each Application is next written through the aggregated API, not
fleet-wide at upgrade (the ApplicationDefinition reconciler syncs only
chartRef/valuesFrom/labels onto a live HelmRelease). Follow-up from #3954 (#4187).
Signed-off-by: Alexey Artamonov <[email protected]>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (7)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe changes pin PostgreSQL PromQL joins to kube-state-metrics, validate non-negative KEDA replica bounds, test annotation merging, and update autoscaling and apply-strategy documentation. ChangesPostgreSQL autoscaling
KEDA helper contracts
Rollout documentation
Priority: ⬇️ Low Estimated code review effort: 2 (Simple) | ~10 minutes Severity of issue fixed: Low Merge Risk: ⚪ Minimal · up to This change scopes PostgreSQL autoscaling metric joins, rejects invalid negative minimum replica counts, expands helper coverage, and corrects operational documentation. No concrete current-head merge-blocking risk remains. Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
IvanHunters
left a comment
There was a problem hiding this comment.
Verdict
LGTM with non-blocking notes
Summary
All five to-fix items from #4187 landed, and both behaviour changes survive a close read.
The job pin is correct for this platform. packages/system/monitoring-agents/templates/kube-state-metrics-scrape.yaml:20 relabels job to kube-state-metrics outright, so every selector keeps matching, and the query now lines up with the DatabaseAutoscaler* alerts.
The negative-floor guard can't break anything that exists today: packages/apps/postgres/templates/scaledobject.yaml:128 is the helper's only production caller, and it passes max(minReplicas, maxSyncReplicas+1, 2).
Both comment corrections match the code they describe, the rollout note included. internal/controller/applicationdefinition_helmreconciler.go:127 syncs chartRef, valuesFrom and labels onto a live HelmRelease and nothing else, and the apply strategy is set only on the aggregated-API write path at pkg/registry/apps/application/rest.go:1683.
Everything below is non-blocking. Two of the notes predate this PR and aren't yours to fix in this branch; they're written up here because this is where the reasoning about them lives.
Findings
- [MINOR]
packages/apps/postgres/tests/autoscaling_test.yaml:190, the new assertion pins one selector out of four - [MINOR]
packages/apps/postgres/templates/_autoscaling.tpl:177, the dashboard panels join the same series unpinned - [MINOR]
packages/library/cozy-lib/templates/_keda.tpl:96, a fractional negative floor still renders as 0 - [MINOR]
packages/apps/postgres/templates/_autoscaling.tpl:221, the job label does not separate the platform's own second producer - [NIT]
packages/tests/cozy-lib-tests/tests/keda_scaledobject_test.yaml:213, the annotations case exercises the pass-through branch only
What came out clean
helm unittest is green as claimed (postgres 66/66, cozy-lib-tests 45/45), and reverting the whole pin turns it red, so the fix isn't vacuously tested. The feature is in no release tag yet, so no upgraded fleet carries a ScaledObject and nothing here can regress an existing install.
I also re-checked the six findings still open from my #3954 rounds against this head. Four are closed. The two that remain, the keda-hpa-.* alert selector and the package-removal teardown hang, sit outside this diff in the deferred section of #4187.
One correction to my own issue while I'm here. #4187 defers the alert-selector item on the grounds that database autoscaling is KEDA's only consumer in cozystack. It isn't: packages/system/ingress-nginx/charts/ingress-nginx/templates/controller-keda.yaml renders a second ScaledObject behind controller.keda.enabled, off by default but reachable by editing values. I'll fix the rationale there.
Caveats
Nothing ran against a live cluster, vmselect, KEDA or an HPA. Renders, unittests and mutations in a throwaway clone, that's all.
| # multiply the join. | ||
| - matchRegex: | ||
| path: spec.triggers[0].metadata.query | ||
| pattern: kube_pod_labels\{job="kube-state-metrics" |
There was a problem hiding this comment.
[MINOR] the new assertion pins one selector out of four
pattern: kube_pod_labels\{job="kube-state-metrics" matches anywhere in the query string, so one pinned occurrence satisfies it no matter what the other three selectors do. Reverting the pin one line at a time and re-running the suite each time:
unpin _autoscaling.tpl:177 ($joinReplica) -> GREEN (gap) [Tests: 66 passed, 66 total]
unpin _autoscaling.tpl:190 ($idleFloor) -> GREEN (gap) [Tests: 66 passed, 66 total]
unpin _autoscaling.tpl:203 ($joinCluster) -> GREEN (gap) [Tests: 66 passed, 66 total]
unpin _autoscaling.tpl:221 ($frozen) -> RED (covered) [Tests: 1 failed, 65 passed, 66 total]
The fix is not vacuously tested: dropping all four pins at once gives Tests: 2 failed, 64 passed. So this is granularity, not a hole. It matters most for $joinReplica at _autoscaling.tpl:177, which carries the read-load signal in both metric modes: unpin only that one in a later edit and CI stays green while the join goes back to matching any exporter's series. One assertion per selector closes it, each pairing the job pin with the label that identifies its site.
| {{- $rel := .Release.Name -}} | ||
| {{- $target := .Values.autoscaling.target -}} | ||
| {{- $joinReplica := printf "* on(namespace,pod) group_left() kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=\"replica\"}" $ns $rel -}} | ||
| {{- $joinReplica := printf "* on(namespace,pod) group_left() kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=\"replica\"}" $ns $rel -}} |
There was a problem hiding this comment.
[MINOR] the dashboard panels join the same series unpinned
dashboards/db/database-autoscaling.json:56,67,89 joins kube_pod_labels with group_left(label_cnpg_io_cluster) and no job pin, across the read-connections, replica-CPU and replication-lag panels. It ships wherever monitoring does, registered at packages/system/monitoring/dashboards.list:10.
That dashboard is not incidental to this feature: the autoscaling.dryRun field description tells the operator to read the scaling recommendation off "the matching read-load dashboard panel for your metric". Under the exact duplicate-exporter condition this PR defends against, the trigger query is now correct while the panel either double counts or fails outright on the ambiguous match group, so the number the operator sizes maxReplicas from stops agreeing with what KEDA acts on.
Not introduced here, and not in the scope #4187 set, which asked for the query only. No objection to leaving it out of this branch; worth pinning the three expressions in a follow-up, and I will add it to the issue.
| all-negative or mixed-sign pair (e.g. min -5, max -3, which satisfies max >= min) | ||
| cannot render a ScaledObject with a nonsensical bound. A negative max with a | ||
| non-negative min is already caught by the ordering check below. */ -}} | ||
| {{- if lt $min 0 -}} |
There was a problem hiding this comment.
[MINOR] a fractional negative floor still renders as 0
The guard reads $min, which is already int .minReplicaCount, and truncation toward zero lets a fractional negative through. On a probe chart wrapping the helper:
$ helm template p ./probe --set-json 'minr=-0.5'
minReplicaCount: 0
$ helm template p ./probe --set-json 'minr=-1'
Error: execution error at (probe/templates/so.yaml:1:3): cozy-lib.keda.scaledObject: minReplicaCount must be >= 0
So -0.5 produces exactly the scale-to-zero-capable ScaledObject that the type check two blocks above exists to prevent. Latent rather than live: the only production caller passes max(minReplicas, maxSyncReplicas+1, 2), so postgres cannot reach it. Testing the untruncated value closes it, verified on the same probe:
$ sed -i 's/if lt $min 0/if lt (float64 .minReplicaCount) 0.0/' charts/cozy-lib/templates/_keda.tpl
-0.5 -> Error: ... minReplicaCount must be >= 0
-1 -> Error: ... minReplicaCount must be >= 0
0 -> minReplicaCount: 0
2 -> minReplicaCount: 2
| GC. Without this filter a lingering join Job pod inflates the frozen count | ||
| while the brake is engaged, nudging desired up instead of holding it. */ -}} | ||
| {{- $frozen := printf "(count(kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}} | ||
| {{- $frozen := printf "(count(kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}} |
There was a problem hiding this comment.
[MINOR] the job label does not separate the platform's own second producer
A tenant's guest Kubernetes cluster installs the same monitoring-agents chart and remote-writes into the tenant's store (packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml:47 points vmagent at vminsert-shortterm.<tenant>), and that chart relabels job to kube-state-metrics too. What tells the two producers apart is the cluster/tenant external labels set alongside that remote-write, and on(namespace,pod) ignores both. A guest namespace and pod name colliding with the management-side ones therefore still land in the match group, job pin or not.
Contrived to hit, and the failure is fail-safe (an ambiguous match group errors the query rather than mis-scaling), so it does not change my read of the PR. Parity with the alerts is worth having on its own terms. The wording to watch is the commit message's "a duplicate kube_pod_labels series from another exporter cannot multiply the join": within one cluster, yes.
The documentation half is more actionable. A tenant pointing autoscaling.serverAddress at their own vmselect now needs kube_pod_labels carried under that specific job name, and the field description at packages/apps/postgres/values.yaml:148-149 only says the vmselect must hold the series.
| asserts: | ||
| - failedTemplate: {errorMessage: "cozy-lib.keda.scaledObject: minReplicaCount must be >= 0"} | ||
|
|
||
| - it: merges a caller-supplied annotation onto the ScaledObject |
There was a problem hiding this comment.
[NIT] the annotations case exercises the pass-through branch only
The fixture sets wantAnnotations but never paused, so $annotations comes straight from default (dict) .annotations and the merge at _keda.tpl:123 never runs. That merge is the part worth pinning, since the pause pair is the destination and therefore wins a key collision. #4187 asked for this case specifically as "extra annotations merged under the pause pair". Setting paused on the same fixture and asserting both the caller key and autoscaling.keda.sh/paused-scale-in on one render would cover it.
…e pause-merge collision The negative-floor guard read $min, already int .minReplicaCount, and int truncates toward zero, so a fractional negative such as -0.5 became 0 and slipped past into exactly the scale-to-zero-capable ScaledObject the type check above exists to prevent. Test the untruncated value instead. The annotations test only set an extra key with no pause, so it exercised the pass-through branch and never the merge where the pause pair is the destination and must win a key collision. Pass a colliding paused-scale-in="false" alongside a distinct key and assert the pause value overrides it on the rendered object. Assisted-by: LLM Signed-off-by: Alexey Artamonov <[email protected]>
…nce for the query The pin check matched job="kube-state-metrics" anywhere in the trigger query, so one pinned selector satisfied it no matter what the other three did: unpinning $joinReplica, $idleFloor or $joinCluster individually left the suite green while the join went back to matching any exporter's series. Anchor one assertion per selector on the label that identifies its site so a single dropped pin fails. Assisted-by: LLM Signed-off-by: Alexey Artamonov <[email protected]>
IvanHunters
left a comment
There was a problem hiding this comment.
Verdict
LGTM
Re-review of 2d1e425..9033d41
Three of the five notes from my last pass are closed. I checked each by reverting the fix rather than by reading it.
Unpinning any one of the four kube_pod_labels selectors in _autoscaling.tpl now reddens the suite, where before only $frozen did:
unpin line 177 ($joinReplica) -> 1 failed, 65 passed
unpin line 190 ($idleFloor) -> 1 failed, 65 passed
unpin line 203 ($joinCluster) -> 1 failed, 65 passed
unpin line 221 ($frozen) -> 1 failed, 65 passed
Putting the guard back to the $min form reddens exactly the new fractional-negative case, and reversing the merge so the caller's dict becomes the destination reddens exactly the new pause-collision case. Both suites are green at this head: postgres 66/66, cozy-lib-tests 47/47.
The two notes left open are the ones I said aren't yours to fix in this branch: the unpinned dashboard panels, and the serverAddress description that doesn't state the job-name requirement. Neither blocks this.
Re-approving, since the push dismissed the earlier approval.
Follow-up to #3954 (postgres read-replica autoscaling via KEDA), closing the non-blocking review remarks tracked in #4187. Each change is small, isolated, and covered by a red→green test where it is a behaviour change.
Changes
job="kube-state-metrics"on the autoscaler read-load query. The trigger query joined onkube_pod_labelswithout the job pin the siblingDatabaseAutoscaler*alerts already carry; a duplicatekube_pod_labelsseries from another exporter would multiply the join. Pinned on every selector; unittest asserts it in both the base and lag-brake query shapes.minReplicaCountincozy-lib.keda.scaledObject. Onlymax >= minwas checked, so a mixed-sign pair (min -5, max -3) passed and rendered a nonsensical floor. Now failed closed before the ordering check, with a unit case. Also added coverage for the previously-untestedannotationsparameter.Testing
helm unittestgreen: postgres 66/66, cozy-lib-tests 45/45, keda 12/12. Both behaviour changes are mutation-verified (reverting the fix reddens the suite). No generated artifacts affected (novalues.yaml/values.schema.json/Chart.yaml/README.mdchanges).Release note
Closes #4187
Summary by CodeRabbit
Bug Fixes
Documentation