Skip to content

fix(postgres): follow-up review remarks from #3954 (KEDA autoscaling) - #4188

Merged
scooby87 merged 5 commits into
mainfrom
postgres-keda-followup
Sep 10, 2026
Merged

scooby87 merged 5 commits into
mainfrom
postgres-keda-followup

Conversation

@scooby87

@scooby87 scooby87 commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #3954 (postgres read-replica autoscaling via KEDA), closing the non-blocking review remarks tracked in #4187. Each change is small, isolated, and covered by a red→green test where it is a behaviour change.

Changes

  • fix(postgres): pin job="kube-state-metrics" on the autoscaler read-load query. The trigger query joined on kube_pod_labels without the job pin the sibling DatabaseAutoscaler* alerts already carry; a duplicate kube_pod_labels series from another exporter would multiply the join. Pinned on every selector; unittest asserts it in both the base and lag-brake query shapes.
  • fix(cozy-lib): reject a negative minReplicaCount in cozy-lib.keda.scaledObject. Only max >= min was checked, so a mixed-sign pair (min -5, max -3) passed and rendered a nonsensical floor. Now failed closed before the ordering check, with a unit case. Also added coverage for the previously-untested annotations parameter.
  • docs(postgres): correct the KEDA-missing guard rationale and note the gradual client-side rollout. The guard comment still described the replaced drop-instances design; corrected to the seed design (a skipped ScaledObject leaves a fixed-size cluster with no HPA, not a collapse to 1). The cozyrds apply-strategy annotation now notes that the client-side switch rolls out per-app as each Application is next written through the aggregated API, not fleet-wide at upgrade.

Testing

helm unittest green: postgres 66/66, cozy-lib-tests 45/45, keda 12/12. Both behaviour changes are mutation-verified (reverting the fix reddens the suite). No generated artifacts affected (no values.yaml/values.schema.json/Chart.yaml/README.md changes).

Release note

NONE

Closes #4187

Summary by CodeRabbit

  • Bug Fixes

    • Improved PostgreSQL autoscaling metrics by ensuring pod data is sourced from the correct metrics exporter, preventing duplicate data from distorting scaling decisions.
    • Added validation to reject negative minimum replica counts.
    • Preserved caller-provided annotations on generated autoscaling resources.
  • Documentation

    • Clarified PostgreSQL autoscaling fallback behavior when the autoscaling controller is unavailable.
    • Documented the gradual transition of existing PostgreSQL resources to the updated apply behavior.

… query

The read-load trigger query joined on kube_pod_labels without pinning
job="kube-state-metrics", while the sibling DatabaseAutoscaler* alert rules
already pin it. A second exporter emitting a kube_pod_labels series (or a
duplicate scrape) would multiply the join and distort the desired-count
arithmetic. Pin the job on every kube_pod_labels selector in the query for
parity with the alerts. Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <[email protected]>
…tions parameter

cozy-lib.keda.scaledObject only checked max >= min, so an all-negative or
mixed-sign pair (min -5, max -3) passed the ordering guard and rendered a
ScaledObject with a nonsensical floor. Reject minReplicaCount < 0 before the
ordering check. Also add a cozy-lib-tests case for the documented annotations
parameter, which had no caller and no coverage. Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <[email protected]>
… client-side rollout

The KEDA-missing fail-closed guard's comment still described the replaced
drop-instances design ("db.yaml drops spec.instances ... CNPG collapses to 1");
under the shipped seed design db.yaml seeds instances at max(replicas,
effectiveMin), so a skipped ScaledObject leaves a fixed-size cluster with no HPA
- autoscaling silently absent, not a collapse. Correct the rationale. Also note
on the cozyrds apply-strategy annotation that the client-side switch rolls out
per-app as each Application is next written through the aggregated API, not
fleet-wide at upgrade (the ApplicationDefinition reconciler syncs only
chartRef/valuesFrom/labels onto a live HelmRelease). Follow-up from #3954 (#4187).

Signed-off-by: Alexey Artamonov <[email protected]>
@coderabbitai

coderabbitai Bot commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 0aa032e8-c8f7-4fe8-8825-77035aa6f089

📥 Commits

Reviewing files that changed from the base of the PR and between afc003c and 2d1e425.

📒 Files selected for processing (7)
  • packages/apps/postgres/templates/_autoscaling.tpl
  • packages/apps/postgres/templates/scaledobject.yaml
  • packages/apps/postgres/tests/autoscaling_test.yaml
  • packages/library/cozy-lib/templates/_keda.tpl
  • packages/system/postgres-rd/cozyrds/postgres.yaml
  • packages/tests/cozy-lib-tests/templates/tests/keda-scaledobject-arg.yaml
  • packages/tests/cozy-lib-tests/tests/keda_scaledobject_test.yaml

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.


📝 Walkthrough

Walkthrough

The changes pin PostgreSQL PromQL joins to kube-state-metrics, validate non-negative KEDA replica bounds, test annotation merging, and update autoscaling and apply-strategy documentation.

Changes

PostgreSQL autoscaling

Layer / File(s) Summary
Pin autoscaling metric joins
packages/apps/postgres/templates/_autoscaling.tpl, packages/apps/postgres/tests/autoscaling_test.yaml
All kube_pod_labels selectors use job="kube-state-metrics". Tests verify the rendered selectors.
Align fail-closed documentation
packages/apps/postgres/templates/scaledobject.yaml
The comment describes the static instance seed used when KEDA is unavailable.

KEDA helper contracts

Layer / File(s) Summary
Validate and render ScaledObject arguments
packages/library/cozy-lib/templates/_keda.tpl, packages/tests/cozy-lib-tests/templates/tests/keda-scaledobject-arg.yaml, packages/tests/cozy-lib-tests/tests/keda_scaledobject_test.yaml
Negative minReplicaCount values now fail. Tests cover the failure and caller-supplied annotation merging.

Rollout documentation

Layer / File(s) Summary
Document apply-strategy rollout
packages/system/postgres-rd/cozyrds/postgres.yaml
Comments document gradual per-Application switching to client-side apply.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to 2d1e4

This change scopes PostgreSQL autoscaling metric joins, rejects invalid negative minimum replica counts, expands helper coverage, and corrects operational documentation. No concrete current-head merge-blocking risk remains.

Suggested reviewers: sircthulhu

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies a Postgres KEDA autoscaling follow-up and matches the primary changes in the pull request.
Linked Issues check ✅ Passed The pull request addresses all coding objectives in issue #4187: it scopes all relevant kube_pod_labels selectors, rejects negative minReplicaCount values, adds annotations coverage, updates the KEDA-…
Out of Scope Changes check ✅ Passed The changes remain within the scope of issue #4187. They update related templates, tests, and documentation without modifying deferred CRD validation, unrelated KEDA alerts, or generated artifacts.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch postgres-keda-followup

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@scooby87
scooby87 marked this pull request as ready for review September 9, 2026 18:38
@github-actions github-actions Bot added area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Sep 9, 2026
IvanHunters
IvanHunters previously approved these changes Sep 10, 2026

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

LGTM with non-blocking notes

Summary

All five to-fix items from #4187 landed, and both behaviour changes survive a close read.

The job pin is correct for this platform. packages/system/monitoring-agents/templates/kube-state-metrics-scrape.yaml:20 relabels job to kube-state-metrics outright, so every selector keeps matching, and the query now lines up with the DatabaseAutoscaler* alerts.

The negative-floor guard can't break anything that exists today: packages/apps/postgres/templates/scaledobject.yaml:128 is the helper's only production caller, and it passes max(minReplicas, maxSyncReplicas+1, 2).

Both comment corrections match the code they describe, the rollout note included. internal/controller/applicationdefinition_helmreconciler.go:127 syncs chartRef, valuesFrom and labels onto a live HelmRelease and nothing else, and the apply strategy is set only on the aggregated-API write path at pkg/registry/apps/application/rest.go:1683.

Everything below is non-blocking. Two of the notes predate this PR and aren't yours to fix in this branch; they're written up here because this is where the reasoning about them lives.

Findings

  • [MINOR] packages/apps/postgres/tests/autoscaling_test.yaml:190, the new assertion pins one selector out of four
  • [MINOR] packages/apps/postgres/templates/_autoscaling.tpl:177, the dashboard panels join the same series unpinned
  • [MINOR] packages/library/cozy-lib/templates/_keda.tpl:96, a fractional negative floor still renders as 0
  • [MINOR] packages/apps/postgres/templates/_autoscaling.tpl:221, the job label does not separate the platform's own second producer
  • [NIT] packages/tests/cozy-lib-tests/tests/keda_scaledobject_test.yaml:213, the annotations case exercises the pass-through branch only

What came out clean

helm unittest is green as claimed (postgres 66/66, cozy-lib-tests 45/45), and reverting the whole pin turns it red, so the fix isn't vacuously tested. The feature is in no release tag yet, so no upgraded fleet carries a ScaledObject and nothing here can regress an existing install.

I also re-checked the six findings still open from my #3954 rounds against this head. Four are closed. The two that remain, the keda-hpa-.* alert selector and the package-removal teardown hang, sit outside this diff in the deferred section of #4187.

One correction to my own issue while I'm here. #4187 defers the alert-selector item on the grounds that database autoscaling is KEDA's only consumer in cozystack. It isn't: packages/system/ingress-nginx/charts/ingress-nginx/templates/controller-keda.yaml renders a second ScaledObject behind controller.keda.enabled, off by default but reachable by editing values. I'll fix the rationale there.

Caveats

Nothing ran against a live cluster, vmselect, KEDA or an HPA. Renders, unittests and mutations in a throwaway clone, that's all.

# multiply the join.
- matchRegex:
path: spec.triggers[0].metadata.query
pattern: kube_pod_labels\{job="kube-state-metrics"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] the new assertion pins one selector out of four

pattern: kube_pod_labels\{job="kube-state-metrics" matches anywhere in the query string, so one pinned occurrence satisfies it no matter what the other three selectors do. Reverting the pin one line at a time and re-running the suite each time:

unpin _autoscaling.tpl:177 ($joinReplica)  -> GREEN (gap)    [Tests: 66 passed, 66 total]
unpin _autoscaling.tpl:190 ($idleFloor)    -> GREEN (gap)    [Tests: 66 passed, 66 total]
unpin _autoscaling.tpl:203 ($joinCluster)  -> GREEN (gap)    [Tests: 66 passed, 66 total]
unpin _autoscaling.tpl:221 ($frozen)       -> RED (covered)  [Tests: 1 failed, 65 passed, 66 total]

The fix is not vacuously tested: dropping all four pins at once gives Tests: 2 failed, 64 passed. So this is granularity, not a hole. It matters most for $joinReplica at _autoscaling.tpl:177, which carries the read-load signal in both metric modes: unpin only that one in a later edit and CI stays green while the join goes back to matching any exporter's series. One assertion per selector closes it, each pairing the job pin with the label that identifies its site.

{{- $rel := .Release.Name -}}
{{- $target := .Values.autoscaling.target -}}
{{- $joinReplica := printf "* on(namespace,pod) group_left() kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=\"replica\"}" $ns $rel -}}
{{- $joinReplica := printf "* on(namespace,pod) group_left() kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=\"replica\"}" $ns $rel -}}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] the dashboard panels join the same series unpinned

dashboards/db/database-autoscaling.json:56,67,89 joins kube_pod_labels with group_left(label_cnpg_io_cluster) and no job pin, across the read-connections, replica-CPU and replication-lag panels. It ships wherever monitoring does, registered at packages/system/monitoring/dashboards.list:10.

That dashboard is not incidental to this feature: the autoscaling.dryRun field description tells the operator to read the scaling recommendation off "the matching read-load dashboard panel for your metric". Under the exact duplicate-exporter condition this PR defends against, the trigger query is now correct while the panel either double counts or fails outright on the ambiguous match group, so the number the operator sizes maxReplicas from stops agreeing with what KEDA acts on.

Not introduced here, and not in the scope #4187 set, which asked for the query only. No objection to leaving it out of this branch; worth pinning the three expressions in a follow-up, and I will add it to the issue.

all-negative or mixed-sign pair (e.g. min -5, max -3, which satisfies max >= min)
cannot render a ScaledObject with a nonsensical bound. A negative max with a
non-negative min is already caught by the ordering check below. */ -}}
{{- if lt $min 0 -}}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] a fractional negative floor still renders as 0

The guard reads $min, which is already int .minReplicaCount, and truncation toward zero lets a fractional negative through. On a probe chart wrapping the helper:

$ helm template p ./probe --set-json 'minr=-0.5'
  minReplicaCount: 0
$ helm template p ./probe --set-json 'minr=-1'
Error: execution error at (probe/templates/so.yaml:1:3): cozy-lib.keda.scaledObject: minReplicaCount must be >= 0

So -0.5 produces exactly the scale-to-zero-capable ScaledObject that the type check two blocks above exists to prevent. Latent rather than live: the only production caller passes max(minReplicas, maxSyncReplicas+1, 2), so postgres cannot reach it. Testing the untruncated value closes it, verified on the same probe:

$ sed -i 's/if lt $min 0/if lt (float64 .minReplicaCount) 0.0/' charts/cozy-lib/templates/_keda.tpl
-0.5 -> Error: ... minReplicaCount must be >= 0
-1   -> Error: ... minReplicaCount must be >= 0
0    -> minReplicaCount: 0
2    -> minReplicaCount: 2

GC. Without this filter a lingering join Job pod inflates the frozen count
while the brake is engaged, nudging desired up instead of holding it. */ -}}
{{- $frozen := printf "(count(kube_pod_labels{namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}}
{{- $frozen := printf "(count(kube_pod_labels{job=\"kube-state-metrics\",namespace=%q,label_cnpg_io_cluster=%q,label_cnpg_io_instance_role=~\".+\"}) * %v)" $ns $rel $target -}}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MINOR] the job label does not separate the platform's own second producer

A tenant's guest Kubernetes cluster installs the same monitoring-agents chart and remote-writes into the tenant's store (packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml:47 points vmagent at vminsert-shortterm.<tenant>), and that chart relabels job to kube-state-metrics too. What tells the two producers apart is the cluster/tenant external labels set alongside that remote-write, and on(namespace,pod) ignores both. A guest namespace and pod name colliding with the management-side ones therefore still land in the match group, job pin or not.

Contrived to hit, and the failure is fail-safe (an ambiguous match group errors the query rather than mis-scaling), so it does not change my read of the PR. Parity with the alerts is worth having on its own terms. The wording to watch is the commit message's "a duplicate kube_pod_labels series from another exporter cannot multiply the join": within one cluster, yes.

The documentation half is more actionable. A tenant pointing autoscaling.serverAddress at their own vmselect now needs kube_pod_labels carried under that specific job name, and the field description at packages/apps/postgres/values.yaml:148-149 only says the vmselect must hold the series.

asserts:
- failedTemplate: {errorMessage: "cozy-lib.keda.scaledObject: minReplicaCount must be >= 0"}

- it: merges a caller-supplied annotation onto the ScaledObject

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[NIT] the annotations case exercises the pass-through branch only

The fixture sets wantAnnotations but never paused, so $annotations comes straight from default (dict) .annotations and the merge at _keda.tpl:123 never runs. That merge is the part worth pinning, since the pause pair is the destination and therefore wins a key collision. #4187 asked for this case specifically as "extra annotations merged under the pause pair". Setting paused on the same fixture and asserting both the caller key and autoscaling.keda.sh/paused-scale-in on one render would cover it.

…e pause-merge collision

The negative-floor guard read $min, already int .minReplicaCount, and int
truncates toward zero, so a fractional negative such as -0.5 became 0 and slipped
past into exactly the scale-to-zero-capable ScaledObject the type check above
exists to prevent. Test the untruncated value instead.

The annotations test only set an extra key with no pause, so it exercised the
pass-through branch and never the merge where the pause pair is the destination
and must win a key collision. Pass a colliding paused-scale-in="false" alongside a
distinct key and assert the pause value overrides it on the rendered object.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>
…nce for the query

The pin check matched job="kube-state-metrics" anywhere in the trigger query, so
one pinned selector satisfied it no matter what the other three did: unpinning
$joinReplica, $idleFloor or $joinCluster individually left the suite green while
the join went back to matching any exporter's series. Anchor one assertion per
selector on the label that identifies its site so a single dropped pin fails.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>
@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files and removed size/M This PR changes 30-99 lines, ignoring generated files labels Sep 10, 2026

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

LGTM

Re-review of 2d1e425..9033d41

Three of the five notes from my last pass are closed. I checked each by reverting the fix rather than by reading it.

Unpinning any one of the four kube_pod_labels selectors in _autoscaling.tpl now reddens the suite, where before only $frozen did:

unpin line 177 ($joinReplica)  -> 1 failed, 65 passed
unpin line 190 ($idleFloor)    -> 1 failed, 65 passed
unpin line 203 ($joinCluster)  -> 1 failed, 65 passed
unpin line 221 ($frozen)       -> 1 failed, 65 passed

Putting the guard back to the $min form reddens exactly the new fractional-negative case, and reversing the merge so the caller's dict becomes the destination reddens exactly the new pause-collision case. Both suites are green at this head: postgres 66/66, cozy-lib-tests 47/47.

The two notes left open are the ones I said aren't yours to fix in this branch: the unpinned dashboard panels, and the serverAddress description that doesn't state the job-name requirement. Neither blocks this.

Re-approving, since the push dismissed the earlier approval.

@scooby87
scooby87 enabled auto-merge September 10, 2026 14:05
@scooby87
scooby87 merged commit f7cb9e7 into main Sep 10, 2026
79 of 81 checks passed
@scooby87
scooby87 deleted the postgres-keda-followup branch September 10, 2026 16:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Follow-up: non-blocking review remarks from #3954 (postgres KEDA autoscaling)

2 participants