Support scaleout session deletion - #1774

Reviewed at f9da45a. The lease + recheck shape looks right to me; one ask.

1. stop() can hang if it lands while the worker is in its idle scan. The scan runs inside the worker task, so if stop() queues its None sentinel while _queue_pending_session_deletions is awaiting get_sessions_by_status, the scan's keys go into the queue behind the sentinel. The worker takes None and returns, those items never get task_done(), and self._deletion_queue.join() never returns, so resources.close() isn't reached either. It needs at least one Deleted row at that moment (a peer mid-delete, or a deletion that keeps failing) and the window is one DB round trip per 30 s of idle, so it's rare, but when it hits, shutdown hangs until the process gets killed. I reproduced it with mocks against this head: stop() times out with the worker task done and one unfinished item; with the scan returning no keys it returns normally. Checking if not self._started: return between the query and the put_nowait loop would close it (no await in between, so no race), plus a test along the lines of the repro.

Observation, not an ask: Fixes #1749 will close the issue on merge. This covers the boot fan-out and picking up a crashed replica's work, and the 30 s rescan also retries the deletions #1577 describes as stuck until restart. It doesn't add the durable job row, backoff, or terminal failed state from the issue's Expected section, so if those are planned as a follow-up, Part of #1749 would keep it open.

What I checked: the lock is a 30 s lease that #1722's service renews every 10 s and cancels the body with LeaseLostError if renewal can't keep up, which the worker's except Exception logs and the next scan retries. wait_timeout=timedelta(0) collapses the boot re-enqueue on N replicas to one deleter per session. The service shares the session DB engine, and the lease store supports both relational providers the config allows (PostgreSQL, SQLite). init_global_memory always passes key_to_session to start(), so the idle scan is live in the server.

Reviewed at a095d3f, against the merged lease lock from #1722.

Blocking: none.

The stop() hang from the earlier review is fixed: the _started check sits after the scan's last await and before the enqueue loop. With a real SQLite-backed SQLLeaseLockService shared by two MemMachine instances, only one instance runs the store deletes for a pending session, and the session row is deleted once.

Note, pre-existing (#1655). An episodic memory instance checks the session row only when it is opened. With session_manager.instance_cache_size at its default of 0, that happens on every request, so new requests see the deletion. A request already running on another replica keeps its instance, and so does an idle cached instance when the cache size is above 0, for up to max_life_time. Below the instance, only the segment store rejects writes from a stale handle (SegmentStorePartitionHandleStaleError), and the event backend writes segments before vectors. The declarative backend, the episode store, and semantic memory have no liveness check, and add_episodes writes the episode store in parallel with the episodic status check. A write in flight when the deletion finishes can therefore land under the old session key and appear in a project re-created with the same org and project IDs. The deletion itself is correct for the data present when it runs; this is the cross-replica session lifecycle gap in #1655, noted here because scale-out deletion relies on it.

Non-blocking, optional. Five inline comments. Also:

• When the process dies after the session row is deleted and before the lease is released, the session-deletion:<key> lease row stays, because that key is never acquired again. The lease service's docstring asks workloads with one-time keys for periodic expiry cleanup.
• Every grant goes through the lease counter row, and on each scan every idle replica loads all Deleted sessions and probes one lease per key.
• The scan runs only after DELETION_IDLE_SECONDS of empty queue, so a replica that receives a steady stream of deletions never picks up work left by another replica.
• Replicas that first use the lease service at the same time race on table creation (#1570). The losing replica's deletion fails and is retried on the next scan.
• stop() processes everything queued ahead of its sentinel, so a stop right after a scan works through the whole backlog first.

🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.