Wait for a pending delete before re-creating a session by wanghy73 · Pull Request #1739 · MemMachine/MemMachine

wanghy73 · 2026-10-01T23:35:50Z

Problem

Deleting a project and creating it again right away returned 500 with no
handler entry in the log (found by mm-test functional FP-18 and FM-62).

delete_session marks the session row Deleted and queues the purge, which
removes the row only when it finishes. During that window
create_or_validate_session matches the row by key, finds the config equal,
and returns as if it created the session. The Active read-back then finds
nothing, and create_session raised a bare RuntimeError.

Fix

create_session waits for the pending purge before creating, polling the
row (up to 10s). It polls the database rather than waiting on an in-process
event, because the purge may run in another worker.
If the purge has not finished in time, it raises the new
SessionDeletionPendingError, which the API maps to 503 with
Retry-After: 10, the wait the server already did. The message says the
delete may have failed if this persists.
Not 409: the Python client's get_or_create_project reads 409 as "created
concurrently" and fetches the project, which would 404 while the delete
runs.
RestError takes an optional headers argument to carry Retry-After.
A purge that raises is no longer dropped: the delete worker re-queues it
with backoff (1s, doubling to 60s) until it succeeds. Before, the row stayed
Deleted until restart and every create of that name returned 503.
stop() cancels pending retries; start() re-queues Deleted rows as
before.

Flipping the Deleted row back to Active was rejected: the in-flight purge
would then delete the new session's data and its row.

Tests

Create waits out a purge, then succeeds, and never writes while the old row
exists.
A purge that outlasts the wait fails the create without touching the row.
Router maps the error to 503 with Retry-After rounded up from the wait.
A purge that fails twice is retried and succeeds on the third attempt.
stop() cancels a pending retry instead of waiting out its backoff.

Both new MemMachine tests fail with the wait disabled.

Follow-up (not in this PR)

A purge that fails every time (for example #1575, deleting with semantic
memory disabled) is retried every 60s and the name stays blocked. Partial
deletes, and a terminal state for failed deletes, are tracked in #1577, with
the lifecycle redesign in #1579. Partly addresses #1577 (the retry).

delete_session marks the row Deleted and queues the purge, which removes the row when it finishes. Until then create_or_validate_session matches the key, sees the configuration agrees and returns as though it created the session -- and the Active read-back then finds nothing and raised a bare RuntimeError, so deleting a project and creating it again answered 500 with no handler entry in the log. create_session now waits for that purge, polling the row rather than an in-process event because the purge may belong to another worker. If it does not finish inside ten seconds the caller gets SessionAlreadyExistsError, which the API maps to 409: true, actionable and logged, where the 500 was none of those. Found by mm-test functional FP-18 and FM-62. Signed-off-by: Haiyan Wang <[email protected]>

The first commit turned an unfinished delete into SessionAlreadyExistsError, a 409. The Python client reads 409 as "created concurrently" and fetches the project, which 404s while the delete runs. The wait now raises the new SessionDeletionPendingError, which the API maps to 503 with Retry-After: temporary, and retryable. RestError gains an optional headers argument to carry it. The wait's timeout parameter is renamed max_wait to satisfy ruff ASYNC109. Adds tests for waiting out a purge, for a purge that outlasts the wait, and for the 503 mapping. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Haiyan Wang <[email protected]>

edwinyyyu · 2026-10-02T00:25:16Z

Fix CI.

ty rejects assigning a functools.partial over the bound method because the partial's signature differs; monkeypatch.setattr sets it without the check. Signed-off-by: Haiyan Wang <[email protected]>

leomem

Two comments inline, both about what happens when the pending delete never finishes.

leomem · 2026-10-02T08:35:38Z

+        # alone, so creating during that window would "succeed" against the
+        # dying row. Wait for the purge instead. Poll the row rather than an
+        # in-process event, because the purge may run in another worker.
+        await self._await_pending_delete(session_key)


If the purge fails, this key stays blocked until the server restarts.

_delete_session_worker catches and logs any exception from _delete_queued_session, so the row stays Deleted and manager.delete_session never runs. start() only re-queues Deleted rows at startup, so nothing clears the key until a restart.

Until then, every POST /api/v2/projects for this key polls the DB about 200 times over max_wait=10s, then returns 503 with Retry-After: 1 and "retry shortly". A client that follows the header repeats the 10s wait indefinitely. One way to hit this: the vector store is unreachable during the purge and the user retries creating the project.

Could the worker re-queue the row or mark it as failed when the purge raises? Or, at minimum, the error could say the delete is stuck instead of suggesting a short retry.

Good catch. The delete worker now re-queues a failed delete with backoff (1s, doubling to 60s) instead of dropping it, so a temporary failure like an unreachable vector store clears on its own once the store is back, and the next create succeeds. stop() cancels pending retries; start() re-queues Deleted rows as before. The 503 message also now says the delete may have failed if it persists, instead of "retry shortly". Fixed in 972e0d9.

I left out the "mark as failed" state. A delete that fails every time (e.g. #1575) is still retried every 60s and keeps the name blocked. That and partial deletes are already tracked in #1577, and the lifecycle redesign (#1579) covers a proper failed or reconciled state, so I'd rather fix it there than add a new status here.

leomem · 2026-10-02T08:35:38Z

+            code=503,
+            message="Project is still being deleted; retry shortly",
+            ex=e,
+            headers={"Retry-After": "1"},


This 503 only goes out after _await_pending_delete has already polled for 10s. At that point the purge is long-running (a large session, or a queue of deletes ahead of it on the single worker), and Retry-After: 1 invites tight retry loops that each hold a connection for another 10s. A value tied to max_wait would fit better.

Agreed. SessionDeletionPendingError now carries the wait the server already did, and the router sends that rounded up as Retry-After (10s by default), so clients back off for as long as the server waited instead of retrying after 1s. Fixed in 972e0d9.

A purge that raised was logged and dropped, leaving the row Deleted until restart; every create of that key then waited 10s and returned 503. The delete worker now re-queues a failed purge with backoff (1s doubling to 60s); stop() cancels pending retries and start() re-queues the rows. The 503 now carries Retry-After equal to the wait the server already did, instead of 1s, and its message says the delete may have failed. Signed-off-by: Haiyan Wang <[email protected]>

wanghy73 · 2026-10-02T18:26:55Z

Fixed in 820d59b (the ty failure in the new tests); CI is green on that commit.

edwinyyyu · 2026-10-02T23:32:22Z

Tracked under #1755: the wait-and-503 semantics stay; the in-process retry moves into a durable delete job (#1749) in step (c) of the PR split there, so the retry survives a crash and does not fan out across replicas at boot.

🤖 Written by Claude Code (Claude Fable 5.1) on behalf of @edwinyyyu.

wanghy73 · 2026-10-05T15:20:25Z

Closing this in favour of Shu's fix, which takes a better approach to the same
problem.

Recording what the bug is before this closes, so it is not lost with the PR —
it is still present on main as of d6068cd, verified 2026-10-03:

POST /api/v2/projects        {"org_id":"o","project_id":"p"}   -> 201
POST /api/v2/projects/delete {"org_id":"o","project_id":"p"}   -> 204
POST /api/v2/projects        {"org_id":"o","project_id":"p"}   -> 500

The server side:

api_v2/router.py:179        create_project
main/memmachine.py:544      create_session
  raise RuntimeError(f"Failed to create session {session_key}")

create_session calls create_or_validate_session, which does not raise, then
reads the session back with get_session — and get_session asks for
status=SessionStatus.Active only. A deleted project's row is still Deleted
with its purge queued, so the read finds nothing and the bare RuntimeError
fires.

Two things worth keeping separate for whichever fix lands:

The behaviour — a project name cannot be reused after a delete.
The diagnosis — RuntimeError("Failed to create session <key>") reaches
the client as a 500 with no cause. It names the session and nothing about
why, so an operator sees "failed to create" for a row that was created and
has to read the source to find that a status filter two calls away is what
failed. Worth addressing even if the status handling changes.

For reference, this is a regression against the release: a functional run on
0.3.9-post1 passes the three cases that fail on main
(FP-18, FM-62, FT-04 — delete then re-create under the same name). The
change in that window that touches session lifecycle is the segment-store
overhaul (#1548 / #1661), which is where I would look if the cause is still
open.

wanghy73 and others added 2 commits October 1, 2026 23:27

wanghy73 requested a review from leomem October 1, 2026 23:39

Patch the delete wait in tests with monkeypatch to satisfy ty

820d59b

ty rejects assigning a functools.partial over the bound method because the partial's signature differs; monkeypatch.setattr sets it without the check. Signed-off-by: Haiyan Wang <[email protected]>

leomem reviewed Oct 2, 2026

View reviewed changes

wanghy73 requested a review from malatewang October 2, 2026 17:32

wanghy73 added this to the v0.4.0 milestone Oct 2, 2026

This was referenced Oct 2, 2026

Session deletion runs in a per-process queue: a crash loses the job until a restart, and every replica re-enqueues every Deleted session at boot #1749

Open

Tracking: make the session lifecycle safe across replicas with a registry row and durable jobs #1755

Open

edwinyyyu mentioned this pull request Oct 3, 2026

Deleting a project that is in use returns 204 and then never completes: the deletion worker gives up on SessionInUseError, leaves a partial deletion, and only a restart retries #1577

Open

wanghy73 modified the milestones: v0.4.0, v0.4.1 Oct 5, 2026

wanghy73 closed this Oct 5, 2026

Navigation Menu

Repository navigation

Wait for a pending delete before re-creating a session - #1739

Wait for a pending delete before re-creating a session#1739
wanghy73 wants to merge 4 commits into
MemMachine:mainfrom
wanghy73:fix/recreate-after-delete

wanghy73 commented Oct 1, 2026 •

edited

Loading

Uh oh!

edwinyyyu commented Oct 2, 2026

Uh oh!

leomem left a comment

Uh oh!

leomem Oct 2, 2026

Uh oh!

wanghy73 Oct 2, 2026

Uh oh!

leomem Oct 2, 2026

Uh oh!

wanghy73 Oct 2, 2026

Uh oh!

wanghy73 commented Oct 2, 2026

Uh oh!

edwinyyyu commented Oct 2, 2026

Uh oh!

wanghy73 commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wanghy73 commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Tests

Follow-up (not in this PR)

Uh oh!

edwinyyyu commented Oct 2, 2026

Uh oh!

leomem left a comment

Choose a reason for hiding this comment

Uh oh!

leomem Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

wanghy73 Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

leomem Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

wanghy73 Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

wanghy73 commented Oct 2, 2026

Uh oh!

edwinyyyu commented Oct 2, 2026

Uh oh!

wanghy73 commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wanghy73 commented Oct 1, 2026 •

edited

Loading