Repository navigation
fix(db): raise pgxpool MaxConns + PG max_connections + set PG resources - #1221
Merged
Merged
Conversation
Today's 17:35-17:40 UTC and 18:17 UTC alerts were caused by scraper-driven concurrency on /v0/servers (~15 req/s sustained). Individual queries are fast post-#1215 (mean 42ms) but pgxpool MaxConns=30 per pod (60 total) saturates under that rate; queue grows at the Go HTTP layer, nginx-side latencies hit 20–35s while server-side tail max stays around 10s. Three coupled changes: - pgxpool: MaxConns 30→60, MinConns 5→10 per pod. 2 pods × 60 = 120 total app connections (was 60). Cuts queue depth roughly in half at current load and gives headroom for genuine bursts. - PG max_connections: 100→200. Required to support the larger pool plus PG's own ~10 internal connections (autovacuum, replication, admin) plus buffer for ad-hoc psql sessions. Triggers a PG postmaster restart (postmaster-level setting) — same in-pod restart shape as the pg_stat_statements deploy yesterday, ~30s of registry unavailability covered by v1.7.1's DB-retry budget. - PG pod resources: explicit requests/limits (512Mi/200m → 4Gi/1500m). The pod previously had no resource block at all, which made node-level OOM behaviour unpredictable. With max_connections=200 the worst-case per-connection memory (~10–15MB each) + shared_buffers + overhead lands around 3–4GiB; 4Gi limit gives headroom for query workspaces. Resource budget: PG node (dy89) currently 39% mem (~2.4GiB / 6GiB allocatable). Worst-case PG memory growth of ~1.5GiB lands at ~65% — fits. Out of scope but discussed: - ILIKE substring search (server_name ILIKE '%foo%') is unindexable and takes 4–7s under load. Could use pg_trgm GIN or full-text search. - Per-IP nginx rate limiting + Cache-Control on /v0/servers — defends against scraper retry storms regardless of pool size. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2 of 6 tasks
rdimitrov
added a commit
that referenced
this pull request
Apr 28, 2026
## Summary Promotes [v1.7.3](https://github.com/modelcontextprotocol/registry/releases/tag/v1.7.3) to production. Contents: - **#1220** — `RemoteURL` filter SQL rewritten to use JSONB containment (`@>`) instead of `EXISTS jsonb_array_elements`. The new form is GIN-indexable; the old form forced a full table scan. Local benchmark: 40,398 → 294 buffer reads, 17 ms → 1.6 ms. Maps to prod's 10,005 ms cold-cache observation. - **#1221** — pgxpool `MaxConns 30→60`, `MinConns 5→10`. PG `max_connections 100→200`. Explicit PG `resources:` block (was unset). ## What this addresses - **#1220**: yesterday's 17:08 UTC `Publish Endpoint Latency` alert (`dev.storage/mcp` publish took 14.8s with `remotes_ms=10980` — pinpointed by the per-phase slog from #1215). - **#1221**: yesterday's 17:35–17:40 UTC `Availability dropped below 95%` alert and 18:17 UTC `Publish Endpoint Latency` re-fire. Both were scraper-driven concurrency on `/v0/servers` (~15 req/s sustained from ServiceNow + others). With the bumped pool, the queue at the Go HTTP layer should clear faster instead of blowing up to 20–35s nginx-level latencies. ## Deployment caveat The PG `max_connections` change is a postmaster-level setting → CNPG triggers a PG restart on the next prod Pulumi run. With `instances: 1` this is brief downtime — staging took **~30s** during the equivalent restart, with **one registry pod bouncing once** on its 8-attempt DB-retry budget before recovering on the next kubelet restart. **Time the merge for a low-traffic UTC window.** Alert history suggests very early UTC (02:00–04:00) is quietest. ## Resource impact PG node memory is currently 39% (~2.4 GiB / 6 GiB allocatable). Worst-case PG memory growth with `max_connections=200` lands around 3–4 GiB, putting the node at ~65% — fits with headroom. Empirically, prod PG has peaked at **413 MiB** in the last 30h of incident data, so the proposed 4 GiB limit is ~10× the historical max — guardrail not constraint. ## Post-merge CNPG handles `pg_stat_statements` extension creation automatically (no manual `CREATE EXTENSION` step needed — it was already done in v1.7.2's deploy). Verify after deploy: ```bash PATH=/opt/homebrew/share/google-cloud-sdk/bin:$PATH # max_connections actually changed kubectl exec -i registry-pg-1 -c postgres \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -- psql -U postgres -tAc "SHOW max_connections" # expect: 200 # resources block applied kubectl get pod registry-pg-1 \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -o jsonpath='{.spec.containers[?(@.name=="postgres")].resources}{"\n"}' # pgxpool MaxConns reflected (registry app uses 60 per pod after restart) kubectl exec -i registry-pg-1 -c postgres \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -- psql -U postgres -tAc "SELECT count(*) FROM pg_stat_activity WHERE datname='app'" ``` ## Test plan - [x] v1.7.3 release built and pushed (`ghcr.io/modelcontextprotocol/registry:1.7.3`) - [x] Staging deployed cleanly; PG restarted and came back with `max_connections=200`; one staging pod bounced as expected - [ ] Prod Pulumi run applies cleanly; brief PG restart - [ ] Confirm `SHOW max_connections` returns 200 on prod - [ ] Confirm `publish complete` events show `remotes_ms` < 10ms - [ ] Watch for any "too many connections" errors during the rollout window (none expected — Pulumi orders CNPG cluster before Deployment, so PG accepts the new conn limit before pgxpool tries to use it) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2 of 4 tasks
rdimitrov
added a commit
that referenced
this pull request
Apr 29, 2026
…1225) ## Summary Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per pod without bumping the registry pod memory limit. pgxpool keeps per-connection state in the Go process (prepared statement cache, scratch buffers, row iteration state), so memory grew roughly proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB against a 512 MiB limit. ## What surfaced it Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`, `reason: OOMKilled`). Memory history from Cloud Monitoring confirms the regression: | Pods | Peak 30-min memory | |------|-------------------:| | Pre-#1221 (`MaxConns=30`) | 178–333 MB | | **Post-#1221** (`MaxConns=60`) | **437–449 MB** | | Old limit | **512 MiB** ← right at the edge | Other pod was at 394 MB at the time of inspection — could OOM again. ## Fix `deploy/pkg/k8s/registry.go:217-225`: | | Before | After | |---|-------:|------:| | `requests.memory` | 256 MiB | **512 MiB** | | `limits.memory` | 512 MiB | **1 GiB** | 1 GiB is ~2× the current peak under load — gives room for traffic spikes without immediate OOM risk. Either node has 4–5 GiB free allocatable memory, so the new request fits comfortably. ## Test plan - [x] `go build ./...` clean - [x] `golangci-lint run` clean - [ ] After release + prod promotion: pod memory stabilises around 450 MB peak with no OOMKills - [ ] After release + prod promotion: tx49f restartCount stops climbing (currently at 5) ## Deployment Zero downtime — Deployment template change only, rolling update with `maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no `max_connections` change, no shared_preload_libraries change). Same shape as a normal v1.7.x image promotion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2 of 5 tasks
rdimitrov
added a commit
that referenced
this pull request
Apr 29, 2026
## Summary Promotes [v1.7.4](https://github.com/modelcontextprotocol/registry/releases/tag/v1.7.4) (#1225) to production: registry pod memory request `256Mi → 512Mi`, limit `512Mi → 1Gi`. ## Why #1221 doubled pgxpool `MaxConns` 30→60 per pod without bumping the registry pod memory limit. pgxpool keeps per-connection state in the Go process (prepared statement cache, scratch buffers, row iteration state) so peak per-pod memory shifted from ~330 MB to ~450 MB against the 512 MiB limit. Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29. ## Validation so far - Staging deployed cleanly via `deploy-staging.yml` after #1225 merged - Both staging pods running with `req=512Mi limit=1Gi` ✓ - Rolling update was clean — no failed health checks during the transition - Both pods spread across nodes (q716 + 7exo) ## Deployment **Zero downtime** — Deployment template change only, rolling update with `maxSurge: 1, maxUnavailable: 0`. No PG involvement (no `max_connections` change, no shared_preload_libraries change). ## Test plan - [x] v1.7.4 release built and pushed (`ghcr.io/modelcontextprotocol/registry:1.7.4`) - [x] Staging deployed cleanly with new limits - [ ] Prod Pulumi run applies cleanly; rolling update completes - [ ] Prod pods come up with `req=512Mi limit=1Gi` - [ ] After 24h: no OOMKills on either pod, peak memory stays under ~600 MB 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #1215 and #1220. Both of those addressed individual slow queries; this PR addresses concurrency — under today's scraper load the cursor query is fast on average (mean 42ms) but the connection pool saturates and queue depth blows up at the Go HTTP layer.
Diagnosis
pg_stat_statements(added in #1215) made this possible to see:Mean times are healthy. But
max_exec_timeof 7–10s on the cursor query, combined with sustained ~15 req/s from scrapers (ServiceNow's 148.139.x.x range, anonymousnodeuser-agent, etc) saturatesMaxConns=30 × 2 pods = 60. New requests queue at the Go HTTP layer; nginx-side p99 hits 35s; eventually scrapers time out at 60s, retry, and amplify the queue.Today's two ongoing alerts are both this pattern:
Availability dropped below 95%— 4,526 GET requests in 5 min, hundreds of 504sPublish Endpoint Latencyre-fire — same scraper concurrency dragging the publish pathCritically, the symptoms today were also visible during yesterday's incident, but yesterday's broken cursor (#1215) was the dominant cause. After #1215 the cursor is fast individually; concurrency now becomes the next bottleneck.
Changes
pgxpool (
internal/database/postgres.go)MaxConnsMinConnsMaxConnIdleTimeMaxConnLifetime2 pods × 60 = 120 total app connections. Cuts queue depth roughly in half at current scraper load.
PG cluster (
deploy/pkg/k8s/postgres.go)max_connections: 100 → 200— required to support the larger pool. 120 (app) + ~10 (PG internals: autovacuum, replication, admin) + 70 headroom.resources:block — previously the pod had no resource limits, making node-level OOM behaviour unpredictable.Resource budget
Both nodes well within capacity. CPU usage <50% on both, plenty of headroom.
PG worst-case memory math:
shared_buffers128 MiBmaintenance_work_mem, OS overhead, etc.Deployment caveat
max_connectionsis a postmaster-level setting → CNPG triggers a PG restart on the change. Same in-pod restart shape as the pg_stat_statements deploy yesterday — registry pods see ~30s of DB unavailability, covered by v1.7.1's retry-with-backoff (8 attempts, 1→8s capped). One registry pod may bounce once before recovering, like yesterday.Time the merge for a low-traffic UTC window.
Order of operations matters within the deploy itself:
max_connections=200MaxConns=60configPulumi's standard ordering does step 1 before step 2 in this scenario (Pulumi resource graph: CNPG cluster precedes Deployment). If for any reason it doesn't, the worst case is
too many connectionserrors during a small window — Self-correcting once the rollout completes.Test plan
go build ./...clean for app + deploymake lintcleango test -race ./internal/database/...greentoo many connectionserrors during the rollout windowSHOW max_connectionsreturns 200; query pg_stat_statements after a scraper burst, confirmmax_exec_timefor cursor query no longer hits 10sOut of scope
server_name ILIKE '%foo%') is unindexable. Three pg_stat_statements variants run with means 141–333ms and max 4–7s. Worth replacing with apg_trgmGIN index or full-text search in a separate PR.Cache-Controlon/v0/servers— let nginx absorb scraper-repeated cursors.superfluous WriteHeaderwarnings — separate Huma framework issue.🤖 Generated with Claude Code