[vector store 2/13] Remove custom sharding from the Qdrant store (speedkick) - #1654

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 408b0a9)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 408b0a9)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…edkick) (MemMachine#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in MemMachine#1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
(cherry picked from commit 779362c)

…tore (port of #1654) (#1671)

[vector store 2/13] Remove custom sharding from the Qdrant store (speedkick) (#1654)

Remove custom sharding from the Qdrant store

The Qdrant store could shard its native collection by logical collection
(`QdrantConf.is_distributed`, CUSTOM sharding, one shard key per logical
collection, a shard-key selector on every operation) so that a logical
collection could be deleted by dropping its shard. The payload partition
key was written and filtered in both modes, with the shard key on top, so
no query changes here; what changes is the cost of a tenant.

A shard key is a physical structure, and its cost is per tenant. Measured
in #1564 (Qdrant 1.19.0, one hundred tenants): admitting a tenant is an
explicit `create_shard_key` call of about 450 ms, 45 s for the hundred
against 0.3 s with payload partitioning alone, including 10,000 points;
505 segments against 5, and still 5 after deleting and reusing tenants;
and cluster mode is required, with a bootstrap `--uri`. At ten thousand
tenants that is over an hour of shard-key creation and some 50,000
segments before a point is written. Qdrant's own guidance says the same:
a physical structure per tenant is for the few oversized ones, the long
tail is a payload value.

What the shard bought was the O(1) delete, and with it a kind of fencing
(a write to a dropped shard fails). Deletion is about to become a
registry write that is O(1) and atomic as seen by every reader, a stale
handle is fenced by that registry, and the points are reclaimed afterward
by a filter-delete off the request path, so a shard per logical
collection would only add its cost; it goes now, on the current shape, so
the later changes do not carry it. Promoting a single oversized tenant to
its own custom-sharded collection later, Qdrant's tiered arrangement,
stays open: it is a separate collection, not a flag on this one.

`is_distributed` was never documented; a configuration naming it is
rejected.

Claude-Session: https://claude.ai/code/session_01ESpWYTmCR7X3bJEpoA8SAn


(cherry picked from commit 779362c)

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>