Repository navigation
fix(list): use JSONB containment for RemoteURL filter (GIN-indexable) - #1220
Merged
Merged
Conversation
Same shape of bug as #1215's cursor fix, different code path. The RemoteURL filter built its WHERE predicate as EXISTS (SELECT 1 FROM jsonb_array_elements(value->'remotes') AS remote WHERE remote->>'url' = $1) which is logically equivalent to a GIN containment lookup, but the planner can't translate the per-row array unfolding into a GIN search. It falls back to scanning the entire table with the EXISTS subplan applied per row — measured at ~10s on prod's 21K-row table during the 2026-04-28 17:08 UTC slow publish. Rewrote as: value -> 'remotes' @> jsonb_build_array(jsonb_build_object('url', $1)) which the planner uses idx_servers_json_remotes (GIN) for. Local benchmark with 20K rows and prod-shaped JSONB (~1KB/row): | Form | Buffer hits | Index | Time | |------|-------------|---------|--------| | OLD | 40,398 | none | 17.0ms | | NEW | 294 | GIN | 1.6ms | ~140x fewer buffer reads. Maps to the prod EXPLAIN we ran during the incident (10,005ms → expected low single-digit ms). This filter is called from validateNoDuplicateRemoteURLs in the publish path. The fix removes a 10-second-per-call hit on the connection pool that was responsible for today's 17:08 UTC latency alert and the 17:35-17:40 UTC pool-exhaustion availability dip. Found via the per-phase publish slog added in #1215 — remotes_ms=10980 on a single publish was the smoking gun. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
3 of 6 tasks
rdimitrov
added a commit
that referenced
this pull request
Apr 28, 2026
…es (#1221) ## Summary Follow-up to #1215 and #1220. Both of those addressed *individual* slow queries; this PR addresses *concurrency* — under today's scraper load the cursor query is fast on average (mean 42ms) but the connection pool saturates and queue depth blows up at the Go HTTP layer. ## Diagnosis `pg_stat_statements` (added in #1215) made this possible to see: ``` max_ms mean_ms calls template 10,755 41.8 94,009 plain cursor pagination, no filter 7,277 214.7 14,770 ILIKE substring filter 7,638 330.8 7,656 ILIKE substring + is_latest filter ``` Mean times are healthy. But `max_exec_time` of 7–10s on the cursor query, combined with sustained ~15 req/s from scrapers (ServiceNow's 148.139.x.x range, anonymous `node` user-agent, etc) saturates `MaxConns=30 × 2 pods = 60`. New requests queue at the Go HTTP layer; nginx-side p99 hits 35s; eventually scrapers time out at 60s, retry, and amplify the queue. Today's two ongoing alerts are both this pattern: - 17:35–17:40 UTC `Availability dropped below 95%` — 4,526 GET requests in 5 min, hundreds of 504s - 18:17 UTC `Publish Endpoint Latency` re-fire — same scraper concurrency dragging the publish path Critically, the symptoms today were also visible during yesterday's incident, but yesterday's broken cursor (#1215) was the dominant cause. After #1215 the cursor is fast individually; concurrency now becomes the next bottleneck. ## Changes ### pgxpool (`internal/database/postgres.go`) | Setting | Before | After | |---------|-------:|------:| | `MaxConns` | 30 | **60** per pod | | `MinConns` | 5 | **10** per pod | | `MaxConnIdleTime` | 30 min | (unchanged) | | `MaxConnLifetime` | 2 h | (unchanged) | 2 pods × 60 = 120 total app connections. Cuts queue depth roughly in half at current scraper load. ### PG cluster (`deploy/pkg/k8s/postgres.go`) - `max_connections: 100 → 200` — required to support the larger pool. 120 (app) + ~10 (PG internals: autovacuum, replication, admin) + 70 headroom. - Added explicit `resources:` block — previously the pod had no resource limits, making node-level OOM behaviour unpredictable. | | Request | Limit | |---|--------:|------:| | memory | 512Mi | **4Gi** | | cpu | 200m | 1500m | ## Resource budget | Node | Now | After | |------|----:|------:| | dy89 (PG node) | 39% mem (~2.4 GiB / 6 GiB) | ~65% mem worst-case (~3.9 GiB) | | 2yxm | 36% mem | unchanged | Both nodes well within capacity. CPU usage <50% on both, plenty of headroom. PG worst-case memory math: - 200 conns × ~15 MB per backend = 3 GiB - + `shared_buffers` 128 MiB - + `maintenance_work_mem`, OS overhead, etc. - ≈ 3–4 GiB total ## Deployment caveat `max_connections` is a postmaster-level setting → CNPG triggers a PG restart on the change. Same in-pod restart shape as the pg_stat_statements deploy yesterday — registry pods see ~30s of DB unavailability, covered by v1.7.1's retry-with-backoff (8 attempts, 1→8s capped). One registry pod may bounce once before recovering, like yesterday. **Time the merge for a low-traffic UTC window.** Order of operations matters within the deploy itself: 1. Pulumi applies the new CNPG spec → PG restarts with `max_connections=200` 2. Rolling deploy of registry pods picks up `MaxConns=60` config 3. New conns are accepted up to 200 limit Pulumi's standard ordering does step 1 before step 2 in this scenario (Pulumi resource graph: CNPG cluster precedes Deployment). If for any reason it doesn't, the worst case is `too many connections` errors during a small window — Self-correcting once the rollout completes. ## Test plan - [x] `go build ./...` clean for app + deploy - [x] `make lint` clean - [x] `go test -race ./internal/database/...` green - [ ] On merge: staging deploy applies the spec change cleanly; PG restarts; pool reaches 60 max - [ ] On prod deploy: same; verify no `too many connections` errors during the rollout window - [ ] After deploy: query `SHOW max_connections` returns 200; query pg_stat_statements after a scraper burst, confirm `max_exec_time` for cursor query no longer hits 10s ## Out of scope - **ILIKE substring search** (`server_name ILIKE '%foo%'`) is unindexable. Three pg_stat_statements variants run with means 141–333ms and max 4–7s. Worth replacing with a `pg_trgm` GIN index or full-text search in a separate PR. - **Per-IP nginx rate limiting** — defends against scraper retry storms regardless of pool size. - **`Cache-Control` on `/v0/servers`** — let nginx absorb scraper-repeated cursors. - **Pre-existing `superfluous WriteHeader` warnings** — separate Huma framework issue. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
2 of 6 tasks
rdimitrov
added a commit
that referenced
this pull request
Apr 28, 2026
## Summary Promotes [v1.7.3](https://github.com/modelcontextprotocol/registry/releases/tag/v1.7.3) to production. Contents: - **#1220** — `RemoteURL` filter SQL rewritten to use JSONB containment (`@>`) instead of `EXISTS jsonb_array_elements`. The new form is GIN-indexable; the old form forced a full table scan. Local benchmark: 40,398 → 294 buffer reads, 17 ms → 1.6 ms. Maps to prod's 10,005 ms cold-cache observation. - **#1221** — pgxpool `MaxConns 30→60`, `MinConns 5→10`. PG `max_connections 100→200`. Explicit PG `resources:` block (was unset). ## What this addresses - **#1220**: yesterday's 17:08 UTC `Publish Endpoint Latency` alert (`dev.storage/mcp` publish took 14.8s with `remotes_ms=10980` — pinpointed by the per-phase slog from #1215). - **#1221**: yesterday's 17:35–17:40 UTC `Availability dropped below 95%` alert and 18:17 UTC `Publish Endpoint Latency` re-fire. Both were scraper-driven concurrency on `/v0/servers` (~15 req/s sustained from ServiceNow + others). With the bumped pool, the queue at the Go HTTP layer should clear faster instead of blowing up to 20–35s nginx-level latencies. ## Deployment caveat The PG `max_connections` change is a postmaster-level setting → CNPG triggers a PG restart on the next prod Pulumi run. With `instances: 1` this is brief downtime — staging took **~30s** during the equivalent restart, with **one registry pod bouncing once** on its 8-attempt DB-retry budget before recovering on the next kubelet restart. **Time the merge for a low-traffic UTC window.** Alert history suggests very early UTC (02:00–04:00) is quietest. ## Resource impact PG node memory is currently 39% (~2.4 GiB / 6 GiB allocatable). Worst-case PG memory growth with `max_connections=200` lands around 3–4 GiB, putting the node at ~65% — fits with headroom. Empirically, prod PG has peaked at **413 MiB** in the last 30h of incident data, so the proposed 4 GiB limit is ~10× the historical max — guardrail not constraint. ## Post-merge CNPG handles `pg_stat_statements` extension creation automatically (no manual `CREATE EXTENSION` step needed — it was already done in v1.7.2's deploy). Verify after deploy: ```bash PATH=/opt/homebrew/share/google-cloud-sdk/bin:$PATH # max_connections actually changed kubectl exec -i registry-pg-1 -c postgres \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -- psql -U postgres -tAc "SHOW max_connections" # expect: 200 # resources block applied kubectl get pod registry-pg-1 \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -o jsonpath='{.spec.containers[?(@.name=="postgres")].resources}{"\n"}' # pgxpool MaxConns reflected (registry app uses 60 per pod after restart) kubectl exec -i registry-pg-1 -c postgres \ --context gke_mcp-registry-prod_us-central1-b_mcp-registry-prod \ -- psql -U postgres -tAc "SELECT count(*) FROM pg_stat_activity WHERE datname='app'" ``` ## Test plan - [x] v1.7.3 release built and pushed (`ghcr.io/modelcontextprotocol/registry:1.7.3`) - [x] Staging deployed cleanly; PG restarted and came back with `max_connections=200`; one staging pod bounced as expected - [ ] Prod Pulumi run applies cleanly; brief PG restart - [ ] Confirm `SHOW max_connections` returns 200 on prod - [ ] Confirm `publish complete` events show `remotes_ms` < 10ms - [ ] Watch for any "too many connections" errors during the rollout window (none expected — Pulumi orders CNPG cluster before Deployment, so PG accepts the new conn limit before pgxpool tries to use it) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Same shape of bug as #1215's cursor fix, different filter. The RemoteURL filter built its WHERE predicate using
jsonb_array_elements+->>extraction, which the planner can't translate into a GIN search. Replaced with a JSONB containment predicate@>that usesidx_servers_json_remotes.Diagnosis (from the per-phase publish slog #1215 added)
Today's first publish-latency alert at 17:08 UTC pointed straight at this:
remotes_ms=10,980on a server with one remote URL pointed atvalidateNoDuplicateRemoteURLs, which runs the RemoteURL filter against the database. EXPLAIN ANALYZE on prod confirmed:Fix
internal/database/postgres.go:99rewritten:Semantically equivalent for our schema (
urlis always a string, NOT NULL).Local benchmark — 20K rows, prod-shaped JSONB (~1KB/row)
idx_servers_json_remotesGIN bitmapMaps to prod's 10,005 ms cold-cache observation — same shape of speedup we got from the cursor fix.
Scope honesty
This PR addresses the publish slowness path (
validateNoDuplicateRemoteURLsin the publish transaction). It directly resolves today's 17:08 UTCPublish Endpoint Latencyalert.It does not address:
Availability dropped below 95%alert. Per-investigation: only 4 publishes happened in the entire 17:00–17:50 window and zero during the burst minutes — so my initial "concurrent slow publishes starved the pool" hypothesis was wrong. The actual cause was scraper-driven concurrency (4,500+ requests in 5 min) on plain cursor pagination + ILIKE substring search, where individual queries are fast (mean 42 ms) but tail latency under concurrency reaches 10 s server-side and 20–35 s nginx-level due to pool queueing.Publish Endpoint Latencyre-fire (same scraper-driven concurrency).A follow-up PR will raise pool size + PG
max_connections+ add explicit PG resource limits to absorb concurrency without queue blowup. That fix has a different shape (config) and a different risk profile (PG restart formax_connections); kept separate so this surgical SQL change can ship cleanly.Test plan
go build ./...cleanmake lintcleango test -race ./internal/database/... ./internal/service/...green — existingTestPostgreSQL_ListServers/filter by remote URLcovers semantic correctnessEXPLAIN ANALYZE; confirmremotes_msinpublish completeslog drops to single-digit msOut of scope (separate follow-ups)
MaxConns/MinConnsbump + PGmax_connectionsraise + PGresourceslimits — addresses the cursor-tail and ILIKE-search concurrency that this PR doesn't touchCache-Controlheaders on/v0/servers— let nginx absorb scraper repeatssuperfluous WriteHeaderwarnings — separate Huma framework issueserver_nameis unindexable — could be replaced withpg_trgmGIN index or full-text search🤖 Generated with Claude Code