Skip to content

fix(db): bound asyncpg statement and connect time so a dead socket recovers (port of #1552 to main) - #1679

Merged
malatewang merged 3 commits into
MemMachine:mainfrom
wanghy73:port-main/asyncpg-timeouts-1552
Sep 19, 2026
Merged

malatewang merged 3 commits into
MemMachine:mainfrom
wanghy73:port-main/asyncpg-timeouts-1552

Conversation

@wanghy73

@wanghy73 wanghy73 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Port of #1552 from speedkick to main.

Cherry-pick of 10b3706d5574080153a62b678196989dbc24f85f, the squash commit #1552 merged as on
speedkick. It applied to main with no conflicts, and the diff is
byte-identical to the one reviewed on #1552 (checked with git patch-id), so the
review there still stands.

Re-checked on this branch, based on main:

The body below is #1552's, unchanged.


A pooled connection whose peer stops responding is not an error, it is a wait.
The kernel retransmits with exponential backoff — observed at tcp_retries2
attempt 13, Send-Q stuck at 34 bytes and unchanged over 20 seconds — and
asyncpg has no deadline of its own, so the request blocks for the whole of it.

pool_pre_ping does not help: its liveness SELECT is written into the same
dead socket and waits with everything else.

The failure this produces is worse than a slow request. The pool never
discards the dead connections, so the process stays wedged after the network
recovers
and only a restart clears it. Seen twice on a benchmark deployment:
searches timing out indefinitely while /api/v2/health answered in about a
millisecond, four uvicorn workers idle in select(), no lock contention in
Postgres, and a fresh connection from the same pod completing in 18 ms.

create_async_engine was passed no connect_args, so nothing bounded either the
statement or the connect. This adds command_timeout and connect_timeout,
defaulted to 60 s and 10 s and settable to null for the old behaviour. Both are
gated on driver == "asyncpg": they reach asyncpg.connect() and mean nothing to
aiosqlite or aiomysql.

The keyword assembly moves into a helper, because adding a branch to
async_get_sql_engine pushed it past ruff’s complexity limit and the repeated
"if not None" lines were asking for it.

Verified by fault injection

iptables DROP on the pod’s traffic to Postgres:5432, so pooled connections wedge
exactly as they did in the incident:

under the fault after the fault is removed
stock no response at 150 s (cap) still hung, no response at 90 s
with this change 500 after 194 s 200 in 1.5 s

The recovery is the point. The stock build stays wedged once the network is
healthy again, which is what forced the restarts.

What this does not claim

Under a total outage the request still takes ~194 s to fail rather than ~60 s.
A search touches three separate stores, so it pays a timeout per store: the bound
is roughly command_timeout × stores touched, not command_timeout. Lowering the
default would shorten it proportionally.

60 s is chosen, not tuned. It is generous against real query times here, which
are milliseconds, and is left configurable rather than optimised. Reviewers who
know the workload better should push back on it.

Caveats for review

"that were reset server-side. Internal default is False."
),
)
command_timeout: float | None = Field(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add these two new fields to the sample configuration

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in e02d300. Both fields now sit beside the pool_* settings in deployments/helm/values.yaml and are rendered into the server config by memmachine-configmaps.yaml, following the existing pattern:

  pool_pre_ping: false
  command_timeout: 60   # asyncpg only: seconds a statement may run before it is cancelled; null for unbounded
  connect_timeout: 10   # asyncpg only: seconds to wait for a new connection; null to wait indefinitely

One thing I checked rather than assumed: that the documented “set to null to restore the unbounded behaviour” is actually reachable from the chart. Setting either value to null renders an empty value, which YAML reads as null, which the config model accepts and which then omits connect_args entirely. Confirmed by rendering the chart both ways and feeding the output through SqlAlchemyConf and _sql_engine_kwargs:

null  -> command_timeout=None connect_timeout=None -> NO connect_args (unbounded)
60/10 -> {"command_timeout": 60.0, "timeout": 10.0}

helm lint is clean.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add them to the sample config too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right — I had only added them to the Helm chart, which is a different file. Now added to the sample configs proper in f9b7268: the profile_storage block of all three of episodic_memory_config.cpu.sample, .gpu.sample and .nebula.sample, beside the pool_* settings and in the same comment style.

        pool_pre_ping: false  # test connections for liveness before use (default: false)
        command_timeout: 60.0  # max seconds a statement may run; ~ = unbounded (asyncpg only, default: 60)
        connect_timeout: 10.0  # max seconds to establish a connection; ~ = wait forever (asyncpg only, default: 10)

Checked rather than assumed: each sample still parses, its profile_storage config still validates against SqlAlchemyConf, and the ~ = unbounded in the comment resolves to null and drops connect_args entirely, so the documented escape hatch works from the sample as written.

wanghy73 added a commit to wanghy73/MemMachine that referenced this pull request Sep 17, 2026
… name

Review follow-ups on MemMachine#1679.

Shu: add both fields to the sample configuration. They now sit beside the
pool_* settings in the chart's values and are rendered into the server config,
following the same pattern.

Setting either to null in values.yaml renders an empty value, which YAML reads
as null and the config model accepts, so the documented "null restores the
unbounded behaviour" is reachable from the chart and not just from a hand-written
config. Checked by rendering the chart both ways and feeding the result through
SqlAlchemyConf and _sql_engine_kwargs.

Edwin: rename the unqualified "timeout" key. It cannot be renamed - it is
asyncpg's own parameter name for the connect deadline, and asyncpg has no
connect_timeout parameter, so the key is fixed by their API. Our config field is
already qualified as connect_timeout; only the wire name is bare. Added a comment
so the next reader does not have to check asyncpg's signature to find that out.

ruff, ty (3.12) and helm lint clean; server unit suite unchanged at 1869 passed.

Signed-off-by: Haiyan Wang <[email protected]>
wanghy73 and others added 2 commits September 18, 2026 00:35
…covers (MemMachine#1552)

A pooled connection whose peer stops responding is not an error, it is a
wait. The kernel retransmits with exponential backoff - observed at
tcp_retries2 attempt 13, Send-Q stuck at 34 bytes and unchanged over
20 seconds - and asyncpg has no deadline of its own, so the request blocks
for the whole of it. pool_pre_ping does not help: its liveness SELECT is
written into the same dead socket and waits with everything else.

The failure this produces is worse than a slow request. The pool never
discards the dead connections, so the process stays wedged after the network
recovers and only a restart clears it. Seen twice on a benchmark deployment:
searches timing out indefinitely while /api/v2/health answered in about a
millisecond, four uvicorn workers idle in select(), no lock contention in
Postgres, and a fresh connection from the same pod completing in 18 ms.

create_async_engine was passed no connect_args, so nothing bounded either
the statement or the connect. This adds command_timeout and connect_timeout,
defaulted to 60 s and 10 s and settable to null for the old behaviour. Both
are gated on driver == "asyncpg": they reach asyncpg.connect() and mean
nothing to aiosqlite or aiomysql. The keyword assembly moves into a helper,
because adding a branch to async_get_sql_engine pushed it past ruff's
complexity limit and the repeated "if not None" lines were asking for it.

Verified by fault injection on a deployed platform - iptables DROP on the
pod's traffic to Postgres:5432, so pooled connections wedge exactly as they
did in the incident:

                        under the fault          after the fault is removed
  stock        no response at 150 s (cap)    still hung, no response at 90 s
  with fix     500 after 194 s               200 in 1.5 s

The recovery is the point: the stock build stays wedged once the network is
healthy again, which is what forced the restarts.

Under a total outage the request still takes ~194 s to fail rather than
~60 s. A search touches three separate stores, so it pays a timeout per
store; the bound is roughly command_timeout times the number of stores
touched, not command_timeout. Lowering the default would shorten it
proportionally - 60 s is chosen as generous against real query times here,
which are milliseconds, and is left tunable rather than tuned.

ruff check and ruff format both pass.

Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP

Co-authored-by: Claude Opus 5 <[email protected]>
(cherry picked from commit 10b3706)
Signed-off-by: Haiyan Wang <[email protected]>
… name

Review follow-ups on MemMachine#1679.

Shu: add both fields to the sample configuration. They now sit beside the
pool_* settings in the chart's values and are rendered into the server config,
following the same pattern.

Setting either to null in values.yaml renders an empty value, which YAML reads
as null and the config model accepts, so the documented "null restores the
unbounded behaviour" is reachable from the chart and not just from a hand-written
config. Checked by rendering the chart both ways and feeding the result through
SqlAlchemyConf and _sql_engine_kwargs.

Edwin: rename the unqualified "timeout" key. It cannot be renamed - it is
asyncpg's own parameter name for the connect deadline, and asyncpg has no
connect_timeout parameter, so the key is fixed by their API. Our config field is
already qualified as connect_timeout; only the wire name is bare. Added a comment
so the next reader does not have to check asyncpg's signature to find that out.

ruff, ty (3.12) and helm lint clean; server unit suite unchanged at 1869 passed.

Signed-off-by: Haiyan Wang <[email protected]>
@wanghy73
wanghy73 force-pushed the port-main/asyncpg-timeouts-1552 branch from e02d300 to 471ecec Compare September 18, 2026 00:36
Shu asked for these in the sample config as well; the earlier commit only
reached the Helm chart, which is a different file.

Both fields now sit beside the pool_* settings in the profile_storage block of
all three episodic_memory_config samples, with the same comment style.

Checked rather than assumed: each sample still parses, its profile_storage
config still validates against SqlAlchemyConf, and the documented "~ = unbounded"
resolves to null and drops connect_args entirely.

Signed-off-by: Haiyan Wang <[email protected]>
@malatewang
malatewang merged commit 73937ac into MemMachine:main Sep 19, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants