Skip to content

fix(db): raise pgxpool MaxConns + PG max_connections + set PG resources - #1221

Merged
rdimitrov merged 2 commits into
mainfrom
raise-pool-and-pg-resources
Apr 28, 2026
Merged

rdimitrov merged 2 commits into
mainfrom
raise-pool-and-pg-resources

Conversation

@rdimitrov

Copy link
Copy Markdown
Member

Summary

Follow-up to #1215 and #1220. Both of those addressed individual slow queries; this PR addresses concurrency — under today's scraper load the cursor query is fast on average (mean 42ms) but the connection pool saturates and queue depth blows up at the Go HTTP layer.

Diagnosis

pg_stat_statements (added in #1215) made this possible to see:

max_ms   mean_ms  calls    template
10,755     41.8   94,009   plain cursor pagination, no filter
 7,277    214.7   14,770   ILIKE substring filter
 7,638    330.8    7,656   ILIKE substring + is_latest filter

Mean times are healthy. But max_exec_time of 7–10s on the cursor query, combined with sustained ~15 req/s from scrapers (ServiceNow's 148.139.x.x range, anonymous node user-agent, etc) saturates MaxConns=30 × 2 pods = 60. New requests queue at the Go HTTP layer; nginx-side p99 hits 35s; eventually scrapers time out at 60s, retry, and amplify the queue.

Today's two ongoing alerts are both this pattern:

  • 17:35–17:40 UTC Availability dropped below 95% — 4,526 GET requests in 5 min, hundreds of 504s
  • 18:17 UTC Publish Endpoint Latency re-fire — same scraper concurrency dragging the publish path

Critically, the symptoms today were also visible during yesterday's incident, but yesterday's broken cursor (#1215) was the dominant cause. After #1215 the cursor is fast individually; concurrency now becomes the next bottleneck.

Changes

pgxpool (internal/database/postgres.go)

Setting Before After
MaxConns 30 60 per pod
MinConns 5 10 per pod
MaxConnIdleTime 30 min (unchanged)
MaxConnLifetime 2 h (unchanged)

2 pods × 60 = 120 total app connections. Cuts queue depth roughly in half at current scraper load.

PG cluster (deploy/pkg/k8s/postgres.go)

  • max_connections: 100 → 200 — required to support the larger pool. 120 (app) + ~10 (PG internals: autovacuum, replication, admin) + 70 headroom.
  • Added explicit resources: block — previously the pod had no resource limits, making node-level OOM behaviour unpredictable.
Request Limit
memory 512Mi 4Gi
cpu 200m 1500m

Resource budget

Node Now After
dy89 (PG node) 39% mem (~2.4 GiB / 6 GiB) ~65% mem worst-case (~3.9 GiB)
2yxm 36% mem unchanged

Both nodes well within capacity. CPU usage <50% on both, plenty of headroom.

PG worst-case memory math:

  • 200 conns × ~15 MB per backend = 3 GiB
    • shared_buffers 128 MiB
    • maintenance_work_mem, OS overhead, etc.
  • ≈ 3–4 GiB total

Deployment caveat

max_connections is a postmaster-level setting → CNPG triggers a PG restart on the change. Same in-pod restart shape as the pg_stat_statements deploy yesterday — registry pods see ~30s of DB unavailability, covered by v1.7.1's retry-with-backoff (8 attempts, 1→8s capped). One registry pod may bounce once before recovering, like yesterday.

Time the merge for a low-traffic UTC window.

Order of operations matters within the deploy itself:

  1. Pulumi applies the new CNPG spec → PG restarts with max_connections=200
  2. Rolling deploy of registry pods picks up MaxConns=60 config
  3. New conns are accepted up to 200 limit

Pulumi's standard ordering does step 1 before step 2 in this scenario (Pulumi resource graph: CNPG cluster precedes Deployment). If for any reason it doesn't, the worst case is too many connections errors during a small window — Self-correcting once the rollout completes.

Test plan

  • go build ./... clean for app + deploy
  • make lint clean
  • go test -race ./internal/database/... green
  • On merge: staging deploy applies the spec change cleanly; PG restarts; pool reaches 60 max
  • On prod deploy: same; verify no too many connections errors during the rollout window
  • After deploy: query SHOW max_connections returns 200; query pg_stat_statements after a scraper burst, confirm max_exec_time for cursor query no longer hits 10s

Out of scope

  • ILIKE substring search (server_name ILIKE '%foo%') is unindexable. Three pg_stat_statements variants run with means 141–333ms and max 4–7s. Worth replacing with a pg_trgm GIN index or full-text search in a separate PR.
  • Per-IP nginx rate limiting — defends against scraper retry storms regardless of pool size.
  • Cache-Control on /v0/servers — let nginx absorb scraper-repeated cursors.
  • Pre-existing superfluous WriteHeader warnings — separate Huma framework issue.

🤖 Generated with Claude Code

rdimitrov and others added 2 commits April 28, 2026 21:59
Today's 17:35-17:40 UTC and 18:17 UTC alerts were caused by scraper-driven
concurrency on /v0/servers (~15 req/s sustained). Individual queries are
fast post-#1215 (mean 42ms) but pgxpool MaxConns=30 per pod (60 total)
saturates under that rate; queue grows at the Go HTTP layer, nginx-side
latencies hit 20–35s while server-side tail max stays around 10s.

Three coupled changes:

- pgxpool: MaxConns 30→60, MinConns 5→10 per pod. 2 pods × 60 = 120 total
  app connections (was 60). Cuts queue depth roughly in half at current
  load and gives headroom for genuine bursts.

- PG max_connections: 100→200. Required to support the larger pool plus
  PG's own ~10 internal connections (autovacuum, replication, admin) plus
  buffer for ad-hoc psql sessions. Triggers a PG postmaster restart
  (postmaster-level setting) — same in-pod restart shape as the
  pg_stat_statements deploy yesterday, ~30s of registry unavailability
  covered by v1.7.1's DB-retry budget.

- PG pod resources: explicit requests/limits (512Mi/200m → 4Gi/1500m). The
  pod previously had no resource block at all, which made node-level OOM
  behaviour unpredictable. With max_connections=200 the worst-case
  per-connection memory (~10–15MB each) + shared_buffers + overhead lands
  around 3–4GiB; 4Gi limit gives headroom for query workspaces.

Resource budget: PG node (dy89) currently 39% mem (~2.4GiB / 6GiB
allocatable). Worst-case PG memory growth of ~1.5GiB lands at ~65% — fits.

Out of scope but discussed:

- ILIKE substring search (server_name ILIKE '%foo%') is unindexable and
  takes 4–7s under load. Could use pg_trgm GIN or full-text search.
- Per-IP nginx rate limiting + Cache-Control on /v0/servers — defends
  against scraper retry storms regardless of pool size.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@rdimitrov
rdimitrov merged commit 736d376 into main Apr 28, 2026
5 checks passed
@rdimitrov
rdimitrov deleted the raise-pool-and-pg-resources branch April 28, 2026 19:21
@rdimitrov rdimitrov mentioned this pull request Apr 28, 2026
2 of 6 tasks
rdimitrov added a commit that referenced this pull request Apr 28, 2026
## Summary

Promotes
[v1.7.3](https://github.com/modelcontextprotocol/registry/releases/tag/v1.7.3)
to production. Contents:

- **#1220** — `RemoteURL` filter SQL rewritten to use JSONB containment
(`@>`) instead of `EXISTS jsonb_array_elements`. The new form is
GIN-indexable; the old form forced a full table scan. Local benchmark:
40,398 → 294 buffer reads, 17 ms → 1.6 ms. Maps to prod's 10,005 ms
cold-cache observation.
- **#1221** — pgxpool `MaxConns 30→60`, `MinConns 5→10`. PG
`max_connections 100→200`. Explicit PG `resources:` block (was unset).

## What this addresses

- **#1220**: yesterday's 17:08 UTC `Publish Endpoint Latency` alert
(`dev.storage/mcp` publish took 14.8s with `remotes_ms=10980` —
pinpointed by the per-phase slog from #1215).
- **#1221**: yesterday's 17:35–17:40 UTC `Availability dropped below
95%` alert and 18:17 UTC `Publish Endpoint Latency` re-fire. Both were
scraper-driven concurrency on `/v0/servers` (~15 req/s sustained from
ServiceNow + others). With the bumped pool, the queue at the Go HTTP
layer should clear faster instead of blowing up to 20–35s nginx-level
latencies.

## Deployment caveat

The PG `max_connections` change is a postmaster-level setting → CNPG
triggers a PG restart on the next prod Pulumi run. With `instances: 1`
this is brief downtime — staging took **~30s** during the equivalent
restart, with **one registry pod bouncing once** on its 8-attempt
DB-retry budget before recovering on the next kubelet restart.

**Time the merge for a low-traffic UTC window.** Alert history suggests
very early UTC (02:00–04:00) is quietest.

## Resource impact

PG node memory is currently 39% (~2.4 GiB / 6 GiB allocatable).
Worst-case PG memory growth with `max_connections=200` lands around 3–4
GiB, putting the node at ~65% — fits with headroom. Empirically, prod PG
has peaked at **413 MiB** in the last 30h of incident data, so the
proposed 4 GiB limit is ~10× the historical max — guardrail not
constraint.

## Post-merge

CNPG handles `pg_stat_statements` extension creation automatically (no
manual `CREATE EXTENSION` step needed — it was already done in v1.7.2's
deploy).

Verify after deploy:

```bash
PATH=/opt/homebrew/share/google-cloud-sdk/bin:$PATH

# max_connections actually changed
kubectl exec -i registry-pg-1 -c postgres \
  --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \
  -- psql -U postgres -tAc "SHOW max_connections"  # expect: 200

# resources block applied
kubectl get pod registry-pg-1 \
  --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \
  -o jsonpath='{.spec.containers[?(@.name=="postgres")].resources}{"\n"}'

# pgxpool MaxConns reflected (registry app uses 60 per pod after restart)
kubectl exec -i registry-pg-1 -c postgres \
  --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \
  -- psql -U postgres -tAc "SELECT count(*) FROM pg_stat_activity WHERE datname='app'"
```

## Test plan

- [x] v1.7.3 release built and pushed
(`ghcr.io/modelcontextprotocol/registry:1.7.3`)
- [x] Staging deployed cleanly; PG restarted and came back with
`max_connections=200`; one staging pod bounced as expected
- [ ] Prod Pulumi run applies cleanly; brief PG restart
- [ ] Confirm `SHOW max_connections` returns 200 on prod
- [ ] Confirm `publish complete` events show `remotes_ms` < 10ms
- [ ] Watch for any "too many connections" errors during the rollout
window (none expected — Pulumi orders CNPG cluster before Deployment, so
PG accepts the new conn limit before pgxpool tries to use it)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
rdimitrov added a commit that referenced this pull request Apr 29, 2026
…1225)

## Summary

Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per
pod without bumping the registry pod memory limit. pgxpool keeps
per-connection state in the Go process (prepared statement cache,
scratch buffers, row iteration state), so memory grew roughly
proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB
against a 512 MiB limit.

## What surfaced it

Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`,
`reason: OOMKilled`). Memory history from Cloud Monitoring confirms the
regression:

| Pods | Peak 30-min memory |
|------|-------------------:|
| Pre-#1221 (`MaxConns=30`) | 178–333 MB |
| **Post-#1221** (`MaxConns=60`) | **437–449 MB** |
| Old limit | **512 MiB** ← right at the edge |

Other pod was at 394 MB at the time of inspection — could OOM again.

## Fix

`deploy/pkg/k8s/registry.go:217-225`:

| | Before | After |
|---|-------:|------:|
| `requests.memory` | 256 MiB | **512 MiB** |
| `limits.memory` | 512 MiB | **1 GiB** |

1 GiB is ~2× the current peak under load — gives room for traffic spikes
without immediate OOM risk. Either node has 4–5 GiB free allocatable
memory, so the new request fits comfortably.

## Test plan

- [x] `go build ./...` clean
- [x] `golangci-lint run` clean
- [ ] After release + prod promotion: pod memory stabilises around 450
MB peak with no OOMKills
- [ ] After release + prod promotion: tx49f restartCount stops climbing
(currently at 5)

## Deployment

Zero downtime — Deployment template change only, rolling update with
`maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no
`max_connections` change, no shared_preload_libraries change). Same
shape as a normal v1.7.x image promotion.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
@rdimitrov rdimitrov mentioned this pull request Apr 29, 2026
2 of 5 tasks
rdimitrov added a commit that referenced this pull request Apr 29, 2026
## Summary

Promotes
[v1.7.4](https://github.com/modelcontextprotocol/registry/releases/tag/v1.7.4)
(#1225) to production: registry pod memory request `256Mi → 512Mi`,
limit `512Mi → 1Gi`.

## Why

#1221 doubled pgxpool `MaxConns` 30→60 per pod without bumping the
registry pod memory limit. pgxpool keeps per-connection state in the Go
process (prepared statement cache, scratch buffers, row iteration state)
so peak per-pod memory shifted from ~330 MB to ~450 MB against the 512
MiB limit. Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29.

## Validation so far

- Staging deployed cleanly via `deploy-staging.yml` after #1225 merged
- Both staging pods running with `req=512Mi limit=1Gi` ✓
- Rolling update was clean — no failed health checks during the
transition
- Both pods spread across nodes (q716 + 7exo)

## Deployment

**Zero downtime** — Deployment template change only, rolling update with
`maxSurge: 1, maxUnavailable: 0`. No PG involvement (no
`max_connections` change, no shared_preload_libraries change).

## Test plan

- [x] v1.7.4 release built and pushed
(`ghcr.io/modelcontextprotocol/registry:1.7.4`)
- [x] Staging deployed cleanly with new limits
- [ ] Prod Pulumi run applies cleanly; rolling update completes
- [ ] Prod pods come up with `req=512Mi limit=1Gi`
- [ ] After 24h: no OOMKills on either pod, peak memory stays under ~600
MB

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant