fix(vector-store): create payload indexes even when the collection exists (port of #1578 to main) - #1681

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard points the registry at its SQLite
database.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard and the sample configurations' Qdrant
blocks point the registry at their SQLite database, and the docs' parameter
table and databases page describe the two keys and which stores serve any
process.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard and the sample configurations' Qdrant
blocks point the registry at their SQLite database, and the docs' parameter
table and databases page describe the two keys and which stores serve any
process.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard and the sample configurations' Qdrant
blocks point the registry at their SQLite database, and the docs' parameter
table and databases page describe the two keys and which stores serve any
process.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard and the sample configurations' Qdrant
blocks point the registry at their SQLite database, and the docs' parameter
table and databases page describe the two keys and which stores serve any
process.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard and the sample configurations' Qdrant
blocks point the registry at their SQLite database, and the docs' parameter
table and databases page describe the two keys and which stores serve any
process.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

…y; delete logically, reclaim by purge

QdrantVectorStore keeps its catalog in SqlCollectionRegistry instead of the
per-namespace `__registry` collections, and the per-process creation locks
go with them. A logical collection is identified to callers by its
(namespace, name) and inside the store by the incarnation the registry mints
when it is created: every point carries the incarnation in the payload field
that carried the name (the `is_tenant` index stays), and a handle is bound to
one incarnation. A collection deleted and re-created under the same name
starts empty, its predecessor's points are never seen by it and never
reclaimed out from under it, and the old handle raises
VectorStoreCollectionHandleStaleError on every operation, `get` included.

Creation creates the native collection first (idempotent, as MemMachine#1681 left it)
and registers last, so a crash between the two leaves an empty native
collection the next creation of the same configuration adopts, never a row
whose points have nowhere to go; the registry's primary key arbitrates, so a
racing creator on any process gets AlreadyExists and open-or-create adopts
the winner's row. Deletion is one registry transaction: the collection is
unreachable when it commits and its points wait on the queue. The new
`purge_deleted_collections` does one round on the oldest tombstone due: it
looks for one point under the incarnation in the native collection the
tombstone names and, if there is one, deletes by filter in a single
server-side operation, `wait=True`; the registry keeps or removes the
tombstone by what the round found.

A handle checks liveness before an operation, to refuse a handle known to be
dead, and after it, so an operation completed under an incarnation that died
meanwhile raises instead of reporting success. No lock spans the remote call:
a write that lands under a dead incarnation is the purge's to reclaim, which
is what the tombstone's retention is for.

The VectorStore contract states this: the ABC's "at most one process per
collection" sentence becomes each store's own statement (QdrantVectorStore
serves any process sharing the backend and the registry database; the SQLite
stores keep their bound in their class docstrings), a handle's staleness is
part of the collection contract, and `purge_deleted_collections` is part of
the store contract, returning False on the stores whose deletion reclaims
physically (both SQLite stores, and Milvus until its own commit).

QdrantConf gains `registry_database`, the name of a relational database
under `resources.databases`, required, and `tombstone_retention_seconds`
(default 86400); `registry_replication_factor` goes with the registry
collections it sized. DatabaseManager hands the store the engine and the
backend's id, which scopes its rows in the tables every Qdrant store on one
relational database shares. The wizard and the sample configurations' Qdrant
blocks point the registry at their SQLite database, and the docs' parameter
table and databases page describe the two keys and which stores serve any
process.

Tests: the collection lifecycle contract (stale handles, empty re-creation,
open-or-create adopting the live incarnation, idempotent deletion, purge
reclaiming what deletion deferred and sparing live collections, a write
landing under a dead incarnation) runs against the local-mode client and,
as integration tests, the REST and gRPC clients; the cross-worker tests use
two clients over one registry and pin that a strict create is created once.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn