Repository navigation
fix(db): bound asyncpg statement and connect time so a dead socket recovers (port of #1552 to main) - #1679
Conversation
| "that were reset server-side. Internal default is False." | ||
| ), | ||
| ) | ||
| command_timeout: float | None = Field( |
There was a problem hiding this comment.
Add these two new fields to the sample configuration
There was a problem hiding this comment.
Done in e02d300. Both fields now sit beside the pool_* settings in deployments/helm/values.yaml and are rendered into the server config by memmachine-configmaps.yaml, following the existing pattern:
pool_pre_ping: false
command_timeout: 60 # asyncpg only: seconds a statement may run before it is cancelled; null for unbounded
connect_timeout: 10 # asyncpg only: seconds to wait for a new connection; null to wait indefinitelyOne thing I checked rather than assumed: that the documented “set to null to restore the unbounded behaviour” is actually reachable from the chart. Setting either value to null renders an empty value, which YAML reads as null, which the config model accepts and which then omits connect_args entirely. Confirmed by rendering the chart both ways and feeding the output through SqlAlchemyConf and _sql_engine_kwargs:
null -> command_timeout=None connect_timeout=None -> NO connect_args (unbounded)
60/10 -> {"command_timeout": 60.0, "timeout": 10.0}
helm lint is clean.
There was a problem hiding this comment.
add them to the sample config too.
There was a problem hiding this comment.
You are right — I had only added them to the Helm chart, which is a different file. Now added to the sample configs proper in f9b7268: the profile_storage block of all three of episodic_memory_config.cpu.sample, .gpu.sample and .nebula.sample, beside the pool_* settings and in the same comment style.
pool_pre_ping: false # test connections for liveness before use (default: false)
command_timeout: 60.0 # max seconds a statement may run; ~ = unbounded (asyncpg only, default: 60)
connect_timeout: 10.0 # max seconds to establish a connection; ~ = wait forever (asyncpg only, default: 10)Checked rather than assumed: each sample still parses, its profile_storage config still validates against SqlAlchemyConf, and the ~ = unbounded in the comment resolves to null and drops connect_args entirely, so the documented escape hatch works from the sample as written.
… name Review follow-ups on MemMachine#1679. Shu: add both fields to the sample configuration. They now sit beside the pool_* settings in the chart's values and are rendered into the server config, following the same pattern. Setting either to null in values.yaml renders an empty value, which YAML reads as null and the config model accepts, so the documented "null restores the unbounded behaviour" is reachable from the chart and not just from a hand-written config. Checked by rendering the chart both ways and feeding the result through SqlAlchemyConf and _sql_engine_kwargs. Edwin: rename the unqualified "timeout" key. It cannot be renamed - it is asyncpg's own parameter name for the connect deadline, and asyncpg has no connect_timeout parameter, so the key is fixed by their API. Our config field is already qualified as connect_timeout; only the wire name is bare. Added a comment so the next reader does not have to check asyncpg's signature to find that out. ruff, ty (3.12) and helm lint clean; server unit suite unchanged at 1869 passed. Signed-off-by: Haiyan Wang <[email protected]>
…covers (MemMachine#1552) A pooled connection whose peer stops responding is not an error, it is a wait. The kernel retransmits with exponential backoff - observed at tcp_retries2 attempt 13, Send-Q stuck at 34 bytes and unchanged over 20 seconds - and asyncpg has no deadline of its own, so the request blocks for the whole of it. pool_pre_ping does not help: its liveness SELECT is written into the same dead socket and waits with everything else. The failure this produces is worse than a slow request. The pool never discards the dead connections, so the process stays wedged after the network recovers and only a restart clears it. Seen twice on a benchmark deployment: searches timing out indefinitely while /api/v2/health answered in about a millisecond, four uvicorn workers idle in select(), no lock contention in Postgres, and a fresh connection from the same pod completing in 18 ms. create_async_engine was passed no connect_args, so nothing bounded either the statement or the connect. This adds command_timeout and connect_timeout, defaulted to 60 s and 10 s and settable to null for the old behaviour. Both are gated on driver == "asyncpg": they reach asyncpg.connect() and mean nothing to aiosqlite or aiomysql. The keyword assembly moves into a helper, because adding a branch to async_get_sql_engine pushed it past ruff's complexity limit and the repeated "if not None" lines were asking for it. Verified by fault injection on a deployed platform - iptables DROP on the pod's traffic to Postgres:5432, so pooled connections wedge exactly as they did in the incident: under the fault after the fault is removed stock no response at 150 s (cap) still hung, no response at 90 s with fix 500 after 194 s 200 in 1.5 s The recovery is the point: the stock build stays wedged once the network is healthy again, which is what forced the restarts. Under a total outage the request still takes ~194 s to fail rather than ~60 s. A search touches three separate stores, so it pays a timeout per store; the bound is roughly command_timeout times the number of stores touched, not command_timeout. Lowering the default would shorten it proportionally - 60 s is chosen as generous against real query times here, which are milliseconds, and is left tunable rather than tuned. ruff check and ruff format both pass. Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP Co-authored-by: Claude Opus 5 <[email protected]> (cherry picked from commit 10b3706) Signed-off-by: Haiyan Wang <[email protected]>
… name Review follow-ups on MemMachine#1679. Shu: add both fields to the sample configuration. They now sit beside the pool_* settings in the chart's values and are rendered into the server config, following the same pattern. Setting either to null in values.yaml renders an empty value, which YAML reads as null and the config model accepts, so the documented "null restores the unbounded behaviour" is reachable from the chart and not just from a hand-written config. Checked by rendering the chart both ways and feeding the result through SqlAlchemyConf and _sql_engine_kwargs. Edwin: rename the unqualified "timeout" key. It cannot be renamed - it is asyncpg's own parameter name for the connect deadline, and asyncpg has no connect_timeout parameter, so the key is fixed by their API. Our config field is already qualified as connect_timeout; only the wire name is bare. Added a comment so the next reader does not have to check asyncpg's signature to find that out. ruff, ty (3.12) and helm lint clean; server unit suite unchanged at 1869 passed. Signed-off-by: Haiyan Wang <[email protected]>
e02d300 to
471ecec
Compare
Shu asked for these in the sample config as well; the earlier commit only reached the Helm chart, which is a different file. Both fields now sit beside the pool_* settings in the profile_storage block of all three episodic_memory_config samples, with the same comment style. Checked rather than assumed: each sample still parses, its profile_storage config still validates against SqlAlchemyConf, and the documented "~ = unbounded" resolves to null and drops connect_args entirely. Signed-off-by: Haiyan Wang <[email protected]>
A pooled connection whose peer stops responding is not an error, it is a wait.
The kernel retransmits with exponential backoff — observed at
tcp_retries2attempt 13,
Send-Qstuck at 34 bytes and unchanged over 20 seconds — andasyncpg has no deadline of its own, so the request blocks for the whole of it.
pool_pre_pingdoes not help: its livenessSELECTis written into the samedead socket and waits with everything else.
The failure this produces is worse than a slow request. The pool never
discards the dead connections, so the process stays wedged after the network
recovers and only a restart clears it. Seen twice on a benchmark deployment:
searches timing out indefinitely while
/api/v2/healthanswered in about amillisecond, four uvicorn workers idle in
select(), no lock contention inPostgres, and a fresh connection from the same pod completing in 18 ms.
create_async_enginewas passed noconnect_args, so nothing bounded either thestatement or the connect. This adds
command_timeoutandconnect_timeout,defaulted to 60 s and 10 s and settable to null for the old behaviour. Both are
gated on
driver == "asyncpg": they reachasyncpg.connect()and mean nothing toaiosqlite or aiomysql.
The keyword assembly moves into a helper, because adding a branch to
async_get_sql_enginepushed it past ruff’s complexity limit and the repeated"if not None" lines were asking for it.
Verified by fault injection
iptables DROPon the pod’s traffic to Postgres:5432, so pooled connections wedgeexactly as they did in the incident:
The recovery is the point. The stock build stays wedged once the network is
healthy again, which is what forced the restarts.
What this does not claim
Under a total outage the request still takes ~194 s to fail rather than ~60 s.
A search touches three separate stores, so it pays a timeout per store: the bound
is roughly
command_timeout× stores touched, notcommand_timeout. Lowering thedefault would shorten it proportionally.
60 s is chosen, not tuned. It is generous against real query times here, which
are milliseconds, and is left configurable rather than optimised. Reviewers who
know the workload better should push back on it.
Caveats for review
injection carried this change and the startup-warming fix together, on top of
the Qdrant wiring from fix(metrics): wire the Qdrant vector store, and let a build include it #1532.
ruff checkandruff format --checkpass.