Tracking: make the session lifecycle safe across replicas with a registry row and durable jobs #1755

Priority 1 of the horizontal-scaling work (#1574). Read this before any of the issues it tracks.

Why one issue

The session layer's defects are one defect seen from several sides: the session row is not the arbiter of the lifecycle, so each operation re-arbitrates in process memory (locks, ref counts, an in-process queue). Fixing them one lock at a time, a lease here and a poll there, leaves the semantics where they are. This issue states the target semantics once and says which open issue each piece closes.

The storage layer below is done or in review. The segment store (#1661) and the collection registry chain (#1733 to #1736) give every store a registry row per key, strict create by primary key, an incarnation per life, fenced handles, logical delete and a concurrent-safe purge. The session layer needs the same shape one level up, and nothing else.

At a glance

Target semantics

1. The sessions row is the registry. States provisioning, active, deleting. Create is one INSERT on the primary key; the loser gets SessionAlreadyExistsError or the idempotent accept, never a driver error. Config updates are one statement or carry a version.
2. Storage is provisioned with the row by a job, not on a request. [session storage 1/2] Create a session's storage with the session, never on a request #1622 creates the partition and collection at create time; the provisioning state plus a job row make a crash between row and storage repairable from any replica. No memory request creates a session.
3. Delete is a state flip plus a durable job. The job row is written in the same transaction as the flip, claimed by any replica with FOR UPDATE SKIP LOCKED, retried with backoff, and lands in a terminal failed state the API can show. Store deletes are O(1) logical; the row goes when the job completes; a create during deletion answers as Wait for a pending delete before re-creating a session #1739 does. Boot does not fan out.
4. No per-process arbitration. _session_locks, _close_lock, the in-use check and ref counts go. In-flight writers on another replica are fenced by the stores; the stale-handle errors are mapped at the API and evict the cached instance. Deleting a session creates nothing.
5. Instances hold no per-session state. Once short-term memory is a process-scoped component (Tracking: declare a concurrency scope per component so single-process components stay usable without blocking multi-replica deployments #1757), an EpisodicMemory is handles plus a segmenter and a deriver, cheap to build per request. The LRU cache, its janitor and the lifetime checker can go, which removes the cross-replica cache coherence gap that Tracking: move multitenancy from sharded (one process per collection, per-tenant tables and shard keys) to non-sharded (shared structures served by any process) #1574 listed as out of scope.
6. Deletion skips the semantic seam at the call site when semantic memory is disabled, and the composition root refuses to build the semantic manager. No component below reads the flag (semantic_memory.enabled is honored only at startup: search, add and list still build and query the semantic stack whenever a request names the semantic type #1600).

Not part of this

A general lock service (#1722 has no consumer here; the arbitration above is rows and constraints). Request routing or session affinity. The full #1579 redesign, which stays the reference for the vocabulary (registry, job, fence) along with #1734.

PR split

Each is reviewable alone; (c) and (e) are the only two that touch the same files.

• (a) create and update by constraint (Session data manager loses concurrent config updates and leaks IntegrityError on concurrent create #1543)
• (b) job table and claim primitive, no consumer change (shared with Tracking: background work across replicas with one claimable-job primitive (ingestion, deletion, purge) #1756)
• (c) deletion on the job table, replacing the queue (Session deletion runs in a per-process queue: a crash loses the job until a restart, and every replica re-enqueues every Deleted session at boot #1749, Deleting a project that is in use returns 204 and then never completes: the deletion worker gives up on SessionInUseError, leaves a partial deletion, and only a restart retries #1577)
• (d) storage with the session ([session storage 1/2] Create a session's storage with the session, never on a request #1622, [session storage 2/2] Remove open-or-create from both stores, and close from the segment store #1625 as they stand) plus the provisioning state
• (e) remove locks, in-use check and ref counts; map stale errors (Session lifecycle is single-process: locks, in-use checks and row-vs-storage creation are not arbitrated across replicas #1655, Stale segment-store handles need cache eviction and an API status mapping #1571)
• (f) no lazy create (A search or add on a nonexistent project creates it (session row, partition and collection), so deletion is not final and the search 404 path is unreachable #1576, Make no memory request create a project #1624)
• (g) delete without create ([Bug]: Deleting a session whose memory is not open creates its storage first, only to delete it #1728)

Tracked

• Session data manager loses concurrent config updates and leaks IntegrityError on concurrent create #1543
• Project deletion never completes when semantic memory is disabled: the deletion worker requests the semantic service unconditionally and gives up #1575
• semantic_memory.enabled is honored only at startup: search, add and list still build and query the semantic stack whenever a request names the semantic type #1600
• Session deletion runs in a per-process queue: a crash loses the job until a restart, and every replica re-enqueues every Deleted session at boot #1749
• Deleting a project that is in use returns 204 and then never completes: the deletion worker gives up on SessionInUseError, leaves a partial deletion, and only a restart retries #1577
• A refused session delete leaks an instance reference: the session stays pinned in that process and every later delete there is refused #1765
• Session lifecycle is single-process: locks, in-use checks and row-vs-storage creation are not arbitrated across replicas #1655
• Stale segment-store handles need cache eviction and an API status mapping #1571
• A search or add on a nonexistent project creates it (session row, partition and collection), so deletion is not final and the search 404 path is unreachable #1576
• [Bug]: Deleting a session whose memory is not open creates its storage first, only to delete it #1728

🤖 Written by Claude Code (Claude Fable 5.1) on behalf of @edwinyyyu.