Vector store indexing and mapping decision: tenant-defined property indexes, runtime-created native resources, and the logical-to-native mapping (folds #1528, #1534, #1535, #1562, #1564, #1565, #1572, #1573) #1648

Status 2026-10-02. The decision this issue frames is being made in code: #1670 removes per-project filterable properties, #1702 keeps user properties out of the vector store, and #1627 makes a vector store one native collection with string-keyed partitions and deployment-declared indexed properties. The comment below (2026-09-15) that the description contains inaccuracies stands for the text that follows; read those three PR bodies for the decisions as taken, and use this issue as the index to the eight it folds.

Why this issue exists

Eight issues filed between 2026-08-26 and 2026-09-02 each argue one piece of a single decision about the vector store, which has four axes:

• Who declares the keys the vector store indexes: a tenant, per collection (the pre-[vector store 1/12] Remove per-project filterable properties (speedkick) #1606 per-project properties_schema, merged into the collection's indexed_properties_schema), or the deployment, once per store (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573, indexed_properties_schema is honoured on Qdrant, ignored on Milvus, and part of collection identity either way #1535, [Bug]: Semantic memory indexes free-form content as filterable vector-store properties #1528).
• Whether the vector store holds and filters keys nobody declared: store and filter them unindexed, store them write-only, or reject them on upsert and query. A separate axis from the first; either declaration model combines with either answer (VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534).
• Whether native resources are created at runtime per logical collection: a native collection per distinct config and a shard key per logical collection, or containers provisioned before serving and bounded by deployment configuration (Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572, VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573, Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564).
• How a logical collection maps to its native container and discriminator: a name re-derived from sha256(config) on every open or an address pinned at creation; the logical name or an incarnation as the discriminator; and what reclaims whatever the mapping leaves behind (QdrantVectorStore re-derives the native collection name on every open, so any config-serialization change silently orphans existing data #1562, Qdrant integration never reclaims native collections or records under dead partition keys #1565, Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564).

The two stacks below are corners of the first two axes: A is deployment-declared and rejects undeclared keys; B is tenant-declared and filters undeclared keys unindexed. speedkick before #1606 was tenant-declared with undeclared keys stored, and refused in filters only once a project declared a non-empty schema; and #1627 before #1628 is deployment-declared with undeclared keys stored and filtered.

Read one at a time they look like eight defects; read together they are one design choice, being made now, in code, on speedkick. This issue states the choice once, carries every argument, measurement and citation from the eight (their bodies are appended unchanged under "Folded issues"), records where the two candidate resolutions stand, and replaces the eight, which are closed as folded into it. Its scope is exactly the four axes above. The rest of the multitenancy move stays on #1574 (see "Related, not folded").

State on speedkick today (a8322a7, 2026-09-15)

Everything the folded issues describe is still in the tree, except where noted:

• VectorStore.create_collection(namespace, name, config) still takes a per-collection VectorStoreCollectionConfig carrying indexed_properties_schema (common/vector_store/vector_store.py:147-166), and the VectorStoreCollection docstring still mandates storing and filtering undeclared, possibly mixed-type properties (:29-33; the "and returning" half that VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534 quotes was removed by Answer with cosine scores and uuids, not vectors and stale properties (speedkick) #1598).
• QdrantVectorStore._build_native_collection_name still hashes config.model_dump_json() on every open (qdrant_vector_store.py:535-539); is_distributed, the shard key per logical collection and the CUSTOM sharding method are still there (:444, :646, :673); so is _name_locks (:495).
• Milvus still creates no index for any property (its two add_index calls, milvus_vector_store.py:428 and :562, are both on the vector field) and still compares the whole config on reopen (:654).
• Semantic memory: the write side of [Bug]: Semantic memory indexes free-form content as filterable vector-store properties #1528 is fixed by Answer with cosine scores and uuids, not vectors and stale properties (speedkick) #1598 (vector records carry a vector and no properties); the eleven-key indexed_properties_schema, value included, is still declared in common/resource_manager/semantic_manager.py:155-167, so the indexes are still created for keys nothing writes.
• [vector store 1/12] Remove per-project filterable properties (speedkick) #1606 (merged 2026-09-14) removed per-project properties_schema from the server configuration, project API, SDK, sample configurations, docs and OpenAPI document: a project can no longer name keys for a backend to index. The one part of the decision already landed, and the part the alternative stack reverts.

The decision and the two candidate resolutions

Resolution A: declared per deployment, provisioned, no tenant-created resources

What #1572, #1573, #1534 and #1535 argue for, what design/server_redesign.md on #1579 specifies ("Properties and filtering": nothing a caller sends creates an index; "Vector store": one container per embedder, provisioned by the schema command, never by a request), and what the [vector store N/13] chain implements:

• [vector store 1/12] Remove per-project filterable properties (speedkick) #1606 (merged): per-project filterable properties removed.
• Add session, source and expansion to EventMemory (speedkick) #1597: system fields (since, until, session_ids, source_ids, block_kinds) are typed search parameters the vector store evaluates; property_filter is the segment store's alone. Took the place of [vector store 2/12] Route system filters to the vector store and user-defined ones to the segment store (speedkick) #1621 (closed).
• [vector store scale-out 6/6] Make a vector store one collection, with string-keyed partitions #1627: a store is one collection, VectorStore(collection, vector_dimensions, indexed_properties); provision() creates its durable resources idempotently before startup; partitions are string keys inside it; hash-derived native names, VectorStoreCollectionConfig and the per-partition config go.
• [vector store scale-out] Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection #1631: an incarnation per partition life, logical delete, purge sweeper; Qdrant custom sharding removed.
• [declared schema 1/2] Make a vector store filter only on the properties it declares #1628: a partition stores and filters only the keys its store declares (UndeclaredPropertyKeyError on upsert and on query); Qdrant collections are created in strict mode, so a filter on an unindexed key is refused by the server.
• [vector store 10/13] Let a deployment declare the properties a vector store indexes (speedkick) #1629 (closed): a deployment-declared user key had no consumer once property_filter never reaches the vector store, so the declared set is the system keys. This settles the question left open on 2026-09-14, whether "at most static config-defined filterable properties" survive: none do.

Under A, "filterable" and "efficiently filterable" are separate tiers: an undeclared key is filtered in the segment store. The design has the deployment name, in settings, the segment-store keys it wants expression indexes on, created by the schema command without touching the vector index or any co-tenant; the chain routes the filter but does not build those indexes yet.

Resolution B: tenant-defined indexes retained, incarnations layered under the existing mapping

Built on 2026-09-15 as the [vector store alt N/9] chain (#1636 to #1644) to see what the retained capability costs once its limits are written where a tenant sees them. All nine were closed the same day with "Reopen if stack chosen": B is parked, not rejected.

• [vector store alt 1/9] Restore per-project filterable properties (speedkick) #1636 reverts [vector store 1/12] Remove per-project filterable properties (speedkick) #1606's removal. A project's properties_schema is fixed for its life (a change is a delete and a create); Qdrant and Milvus map every distinct (namespace, vector dimensions, properties_schema) to one physical collection that all projects declaring it share, created the first time a project declares it and never dropped by this code; co-located projects therefore have identical schemas, types included, and no project can change its schema in place.
• [vector store alt 7/9] Remove custom sharding from the Qdrant store, and disable strict mode on its collections (speedkick) #1642 removes custom sharding and creates every Qdrant collection with strict mode off, so an unindexed filter is evaluated rather than refused.
• [vector store alt 8/9] Mint an incarnation per collection life on Qdrant; delete logically, reclaim by purge (speedkick) #1643 mints an incarnation per collection life on Qdrant, with a per-store purge queue and uuid5(incarnation, record uuid) point ids (bare record-uuid ids let a stale handle or a co-tenant of the shared physical collection overwrite a live point). SQLite and Milvus keep immediate reclamation.

B's documented limits are reproduced under "Limits of resolution B" below, verbatim, since the PRs carrying them are closed.

Neutral, shared by both

Session-owned storage, examples creating the project first, no memory request creating a project, removing open-or-create and close, the open -> get rename, sharding removal and request timeouts (#1622 to #1626, #1630, #1631's sharding half; #1637 to #1642, #1644). The incarnation slices cannot be made neutral: the registry row and the data key are the route.

Resolution of this issue

This issue closes when one stack has merged and the other is closed for good; the closing comment names which and why. Until then the argument record below is the basis for the choice.

Arguments for A (against tenant-defined indexes and runtime-created native resources)

Source per point; the appended bodies carry the full text and citations.

Index cost under shared-container multitenancy

1. A payload or scalar index is a structure per (container, field) spanning every record in the container, not only the declaring collection's. N collections declaring k distinct fields each is up to N x k structures, index builds run over all co-located data, and nothing can be dropped without tracking which collections still need it; the cost grows with churn and never shrinks. (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573)
2. That is a vendor constraint, not a MemMachine one: Milvus partition-key tenancy, the only tier that reaches millions of tenants, requires a shared schema; per-tenant schemas exist only at collection level (65,536 by default) or database level (64). Isolating index sets by giving each collection a container is the collection-per-tenant model Qdrant's guidance warns against. (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573)
3. Qdrant caps payload indexes per collection (max_payload_index_count, 100 on Qdrant Cloud) and documents degradation as the count grows (from 1.16.0). In a container carrying the union of its tenants' fields, roughly 100 tenants each adding one field exhaust it, permanently, for everyone on it. (VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534, indexed_properties_schema is honoured on Qdrant, ignored on Milvus, and part of collection identity either way #1535)
4. The distinction that decides is who defines an indexed property. Server-defined properties are identical for every collection a server serves, so co-located collections carry the same index set by construction: no union accumulates, no collection pays memory and build time for another's fields, a new collection costs no migration. Tenant-defined properties (invoice_id here, patient_id there) break that invariant. What A removes is tenant-defined indexes, not indexes. (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573)

The undeclared-property mandate

5. The contract mandates storing and filtering undeclared, mixed-type properties. Qdrant strict mode (unindexed_filtering_retrieve=false, which Qdrant Cloud activates for new collections) is a first-class, documented switch that makes the shipped backend non-conformant, and the contract gives an operator no reason not to flip it. (VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534)
6. Mixed types are unimplementable where attribute types are fixed (turbopuffer errors on a type change; Weaviate properties are typed), and the tree already diverges: the SQL stores type-tag values and gate every comparison on the tag, Milvus stores untagged and inherits its JSON comparison, and != versus Not(=) compile to different predicates only because of the allowance. (VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534)
7. Key cardinality is unbounded. On JSON-payload stores the cost is the unaccelerated per-point check, paid on shared nodes; on schema-mapped stores (Elasticsearch: 1,000 fields per index; ignore_dynamic_beyond_limit silently stops mapping fields so filters match nothing) it is a shared budget one tenant exhausts for its neighbors. (VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534)
8. A vector-store property has one purpose, to be filtered on: queries return ids and scores, never properties, so "store undeclared, filter best-effort" is incoherent, an unfilterable property there being write-only data. Under A the store rejects undeclared keys on both paths, so an unindexed filter is unrepresentable at the store instead of a silent scan. (design/server_redesign.md; VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534's suggested direction)

Runtime-created native resources and the create path

9. Runtime-defined config makes the container set unknowable before serving, so containers are created on demand and creation spans the registry and the backend. That can be made to appear atomic, and the current design does (native-first ordering plus idempotent, content-addressed creation adopts a crashed create's orphan), so the objection is cost on three axes. Scalability: container count becomes a consequence of caller behavior against hard caps (5 to 200 serverless indexes per Pinecone project, 1000 collections per Qdrant Cloud cluster, 65,536 on Milvus), and creation lands on the request path. Safety: the appearance rests on an unstated adoptability precondition that a backend naming containers per collection violates by leaking one per crashed create; config in the registry entry is a second source of truth that drifts from deployment configuration silently (QdrantVectorStore re-derives the native collection name on every open, so any config-serialization change silently orphans existing data #1562's family). Maintainability: ordering rationale, crash-window reasoning, an intent ledger with a sweeper and grace periods, all existing only to reclaim what a failed create left. (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573, Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572)
10. Provisioning is the same problem and answer shape as SQL schema provisioning: one step reads configuration and idempotently ensures the resources a deployment needs, run before serving rather than from a request; create_collection becomes one registry insert arbitrated by a primary key. It should share Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570's entry point rather than invent a second mechanism. (Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572)
11. Content-addressed containers are never reclaimed: one appears per distinct config, including every revision of any project's properties_schema; nothing enumerates them and nothing can tell whether one is referenced; Qdrant Cloud caps a cluster at 1000. (Qdrant integration never reclaims native collections or records under dead partition keys #1565)
12. A shard key per logical collection (Qdrant is_distributed) is the same per-tenant physical structure at another level. Measured on Qdrant 1.19.0: 450 ms per tenant admission, 45.0 s for 100 tenants against 0.3 s with payload partitioning, 505 segments against 5, cluster mode required. The O(1) shard drop it buys is paid per tenant; if a container needs splitting, shard_number > 1 with automatic sharding is a provisioning knob with no per-tenant cost. (Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564)

The mapping

13. Deriving the native name from sha256(config.model_dump_json()) on every open makes a collection's identity a function of how the config serializes today. Adding an optional field, changing a default, reordering fields, or a pydantic upgrade silently repoints every collection at a name that does not exist; reads return nothing, writes succeed into a new empty collection, the data stays on disk with nothing pointing at it. Identity must be pinned at creation. (QdrantVectorStore re-derives the native collection name on every open, so any config-serialization change silently orphans existing data #1562)
14. Because indexed_properties_schema is inside that hash, declaring one new index changes the collection's address and forces a full re-ingest, even on Milvus where the schema buys nothing (no scalar index is ever created; the schema is read only to pre-null dynamic fields; open_or_create compares the whole config on reopen). The cost is invisible: two identically configured deployments perform differently for a reason a caller cannot see without reading backend source. (indexed_properties_schema is honoured on Qdrant, ignored on Milvus, and part of collection identity either way #1535)
15. Semantic memory declared free-form content (value) and every scalar of caller-supplied metadata as indexed properties: index builds over per-record unique text only exact-string equality could use, every search transferring up to 10,000 full payloads to read one field, every metadata-only update rewriting the vector record (Answer with cosine scores and uuids, not vectors and stale properties (speedkick) #1598 removed the payload and the read; the declaration remains). The remaining declared keys are real filter dimensions, unused only because filtering routes to SQL; a pushed-down pre-filter must cover only attributes immutable after ingest, since a stale pre-filter turns staleness into unrecoverable false negatives. ([Bug]: Semantic memory indexes free-form content as filterable vector-store properties #1528)

Raised on this issue, 2026-09-15

16. Under B the declared set is immutable for the project's life: the schema is part of the physical collection's address, so adding a key would move the project to another physical collection, a migration nothing builds, and the documented remedy is delete and recreate with memories starting empty. A tenant must therefore name at creation every key it will ever filter efficiently, which is the one need per-tenant declaration exists to serve (an application whose filters evolve) and the one it cannot. Under A the declared set is equally fixed per container, but changing it is one deployment operation for all tenants, owned by the provisioning step (Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570), and a key a tenant did not anticipate stays filterable through the segment store, where the design adds an index online.
17. Whether a backend can hold two types under one key decides whether undeclared keys are safe in a shared container. Declared keys cannot collide under either stack: under B co-located projects share one schema, types included ({foo: int} and {foo: str} are two physical collections, one more way the count grows); under A the deployment declares once. Undeclared keys are stored per point on the JSON-payload backends, so mixed types coexist and a filter of the other type matches nothing: Qdrant ("if the stored value type does not fit the filtering condition - it will be considered not satisfied", payload docs, checked 2026-09-15), Milvus $meta (untagged, its own JSON comparison; VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534), the SQLite stores (type-tagged), Postgres JSONB. On schema-first backends the first write fixes the key's type for the whole container: Weaviate auto-schema throws on a conflicting type, Elasticsearch "in most cases" cannot change a mapped field (both checked 2026-09-15), turbopuffer errors on a type change (VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534). There an undeclared key is a cross-tenant collision channel: the first tenant to write foo as an integer makes every co-tenant's string foo an error, with no per-tenant type namespace. The only portable rule for undeclared keys is VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534's (one type per key per container; a write that changes it may be rejected), and under shared-container tenancy that rule is itself a coupling between tenants. A never asks the question at the vector store (undeclared keys are rejected; a declared key of another type is PropertyTypeMismatchError) and leaves mixed types to the segment store, whose compiler gates every comparison on a type tag.

Arguments against A, and B's case

18. Per-tenant dimensions or metric need separate containers, since no provider indexes mixed dimensions together; that is bounded by embedder count, not tenant count, and survives A as one container per embedder. (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573; design/server_redesign.md)
19. Under A a schema change stops silently landing in a new container and demands an explicit migration, and provisioning moves rather than vanishes: nothing for the SQLite stores, startup reconciliation for Qdrant and Milvus, infrastructure-as-code for Pinecone. A needs Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570's provisioning owner before it is usable outside SQLite. (Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572, VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573)
20. An application that later wants a key efficiently filterable needs a deployment change under A. Mitigations, all short of tenant-defined indexes: the two tiers (a new property costs nothing until it is indexed); declaring a generous superset up front (a field absent from a record costs nothing on Qdrant); a small server-published menu of collection profiles, bounded by profiles rather than tenants. (VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support #1573; VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534; the profile menu is a further option stated in neither)
21. Rejecting undeclared properties at the vector store is less flexible than the backends that accept them (Qdrant payload, Milvus $meta, Postgres JSONB are unbounded), and it makes property propagation load-bearing: segments and derivatives must carry a verbatim copy of the event's properties for segment-store filtering to be correct, which becomes a clause of the segmenter and deriver contracts. (design/server_redesign.md, "Propagation")
22. Filtering undeclared keys outside the vector store needs two query plans, chosen by selectivity: resolve candidates in the segment store and score that allowlist, or query the vector store on the declared part, overfetch, and post-filter; the middle band pays overfetch. That routing is DEFERRED (until lifecycle/DDL changes): Route filtered queries by selectivity, and score allowlists exactly (speedkick) #1602, deferred until the lifecycle and DDL changes land.
23. Deleting a tenant inside a shared container is linear in its rows (9 ms filter-delete for 100 points, Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564) rather than an O(1) drop, and records under a dead discriminator are found only by scanning (Qdrant's facet is approximate and capped at 10 values unless exact and limit are raised; Milvus has no listing short of a query), so a purge queue written at deletion time and a sweeper survive under A and B alike; a backend that cannot reject a write to a dead tenant value leaves a bounded leak that only reclamation closes. (Qdrant integration never reclaims native collections or records under dead partition keys #1565, Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation #1564)
24. B's case is that a per-project schema is the capability the interface promised and the first thing a tenant reaches for, and that its costs can be stated where the tenant sees them instead of removed. What it costs, verified against vendor docs and Qdrant's planner source on 2026-09-15, is the next section.

Limits of resolution B

Verbatim from #1636 and #1643 ("Tenant-created physical resources: limits and cross-tenant effects"); also carried as a warning in the configuration docs and on the field descriptions in that stack.

Every project (tenant) declares its own properties_schema, fixed for the life of the project (to change it, delete the project and create it again; its memories start empty). Filters may name any m.<key>: a declared key is indexed where the backend supports it, an undeclared key is evaluated without an index. On Qdrant and Milvus the vector store maps every distinct (namespace, vector dimensions, properties_schema) to one physical collection that all projects declaring that schema share, created the first time a project declares it and never dropped by this code. One tenant's choices therefore change what other tenants get. Verified 2026-09-15 against the vendor docs and, for the Qdrant planner, Qdrant's dev source (read_view/filtering.rs, read_view/dispatch.rs, sample_estimation.rs).

• Physical collections grow with distinct schemas. A project declaring a schema nobody else uses creates a physical collection for the whole deployment. Qdrant Cloud allows 1000 collections per cluster by default and every collection carries its own resource overhead; Milvus allows 65,536 per instance. No reclaim of an empty physical collection is built.
• Qdrant indexes every declared key as a payload index on the shared physical collection, next to ~10 system keys MemMachine indexes itself. Each index costs memory for every tenant's points, and Qdrant (>= 1.16) degrades with large index counts. The store disables strict mode on every collection it creates, so Qdrant Cloud's default caps (max_payload_index_count 100, rejection of unindexed filters) do not apply; collections created by an earlier MemMachine keep their own setting.
• Qdrant evaluates an undeclared key per point, within the project. Every query carries the project's tenant condition on an is_tenant keyword index, which is the planner's primary clause, so the candidates are the project's own points, never the physical collection. An undeclared condition estimates as unknown (min 0), so the planner brute-forces the project's posting list with one payload read per point when the project's count in a segment is below full_scan_threshold (10,000 KB / vector bytes: ~1,700 points at 1536 dims), and otherwise samples 1,000 points to choose between that brute force and HNSW traversal with a payload read per visited node, where the missing payload_m links for the undeclared value fragment traversal under a restrictive condition (fewer results). Payload is in the cold tier (disk) by default. The cost lands on the querying project; other tenants feel it only as load on the shared node.
• Milvus indexes nothing (stated, not mitigated). The store creates no scalar index, so declared and undeclared keys cost the same: all keys are written as dynamic fields ($meta, 65,536 bytes per row, all keys together) and again into the properties JSON field, and the filter, tenant term included, is evaluated per row over the project's hash partition (1 of 16 by default), which holds every tenant hashing there. Milvus allows 64 fields and 1,024 partitions per collection, VARCHAR <= 65,535.
• SQLite stores share nothing. Each project gets its own tables (and, for the search-engine store, its own index file). sqlite_vec takes the limit nearest neighbors and applies the filter afterwards (k = min(limit, 4096)), so a selective filter returns fewer results than limit, declared or not; the search-engine store evaluates the filter per visited candidate with one SQL statement each and indexes no key.
• Cross-tenant effects, plainly: (1) the physical-collection count grows with every distinct schema ever declared and is bounded by the backend; (2) on Qdrant, index memory and query load are shared per physical collection, while undeclared filters cost the querying tenant, not its co-tenants' data; (3) on Milvus every filter scans the shared hash partition per row; (4) a schema cannot change in place.

Corrections to the quoted text (2026-09-15). "Each index costs memory for every tenant's points" and effect (2) overstate the cross-tenant cost. Under B a physical collection is addressed by (namespace, vector dimensions, properties_schema), so every project in it declared the same keys with the same types and can never change them: the index set is identical across co-located projects by construction, there is no union and no accumulation (the reason the schema stays in the address), and each payload index is one every co-located project asked for. What is cross-tenant on Qdrant under B is (1) the physical-collection count, growing with distinct schemas, type variants included, and (2) node-level memory, CPU and IO shared by every collection on the node, the unindexed per-point checks a tenant runs over its own posting list included. The "~10 system keys" are the ten the event backend declares on speedkick (EVENT_BACKEND_SYSTEM_FIELDS plus the timestamp) and, on Qdrant, the store's tenant index, the same on every physical collection.

Related, not folded

• Tracking: move multitenancy from sharded (one process per collection, per-tenant tables and shard keys) to non-sharded (shared structures served by any process) #1574: the tracking issue for the sharded-to-non-sharded move; its status table names the eight issues folded here, and this issue stands in for them there.
• [Feat]: Cross-process collection registry for vector store metadata #1524, [Feat]: Redesign QdrantVectorStore collection registry for multi-process collection management #1525: cross-process arbitration of collection lifecycle (the registry primitive and Qdrant's use of it). [Feat]: Redesign QdrantVectorStore collection registry for multi-process collection management #1525 also records the mapping hazard: two open_or_create_collection calls with different configs create two native collections, one silently wins the registry point, and the loser's records become unreachable.
• Handles held across delete and recreate resurrect records in both SQLite vector stores #1536, Handles held across delete and recreate resurrect records in QdrantVectorStore #1563: stale handles reaching a recreated collection on the SQLite stores and on Qdrant; fixed by the incarnation slice of whichever stack lands ([vector store scale-out] Mint an incarnation per collection life in a SQL-arbitrated registry, so any process may serve any Qdrant or Milvus collection #1631 covers all four stores; [vector store alt 8/9] Mint an incarnation per collection life on Qdrant; delete logically, reclaim by purge (speedkick) #1643 covers Qdrant, with SQLite and Milvus to follow).
• Schema provisioning runs from every process's boot path: create_all races on cold boot and never evolves a table, Alembic runs destructive migrations on first use #1570: SQL schema provisioning, the mechanism Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path #1572 says the vector containers should share.
• DEFERRED (until lifecycle/DDL changes): Route filtered queries by selectivity, and score allowlists exactly (speedkick) #1602: selectivity-based routing of filtered queries, deferred; the two-plan cost in point 22.
• Design: MemMachine server redesign (tenants, event memory API, configuration, lifecycle, schema management) #1579: the design proposal (design/server_redesign.md, design/components/vector_store.md, design/components/filters_and_properties.md) that specifies resolution A.

Folded issues

The eight bodies as filed, unchanged except that headings are demoted below this issue's, each body's attribution footer is dropped in favor of the one at the end, and claims found inaccurate on 2026-09-15 are struck with a bracketed note.

#1573: VectorStore.create_collection takes a per-collection config, promising per-tenant schemas that shared-container multitenancy cannot support (filed 2026-09-02)

What happened

VectorStore.create_collection(namespace, name, config) takes a VectorStoreCollectionConfig per logical collection, so the interface promises that two collections in the same store may have different vector dimensions, similarity metrics, and indexed property schemas.

Per-collection dimensions and metric are bounded in a way schemas are not — they mean a native container per embedding model, so the container count follows model count rather than collection count. They are still not free, for the reason in the next section.

Per-collection indexed property schemas are not, and cannot be made so under shared-container multitenancy. A payload or scalar index is a structure per (container, field) that spans every record in the container, not only the declaring collection's. N collections declaring k distinct fields each gives up to N×k structures, index builds run over all co-located data, and nothing can be dropped without tracking which collections still need it. The cost grows with collection churn and never shrinks.

This is not a MemMachine limitation. Milvus documents the same tradeoff directly: partition-key tenancy, the only tier that scales to millions of tenants, requires a shared schema; per-tenant schemas are available only at collection level (65,536 by default) or database level (64 by default). Isolating index sets by giving each collection its own container is the collection-per-tenant model that Qdrant's guidance also warns against.

[Struck 2026-09-15: a paragraph saying no caller exercised the capability because properties_schema was server-global. The memory-configuration API set it per project (config_service.py), so per-project schemas were reachable.]

The broader reason: runtime config makes the container set unknowable

The index-cost argument above applies only to schemas. This one applies to any per-collection configuration, dimensions and metric included.

A native container's properties are fixed when it is created. If callers supply configuration at collection-creation time, the set of containers a deployment needs becomes a function of what callers happen to ask for, discovered at runtime. A deployment cannot provision ahead what it cannot enumerate ahead, so container creation lands on the request path.

That can be made to appear atomic to the caller, and the current design does so: native-first ordering plus idempotent, content-addressed creation means a crashed create leaves an empty container that the next matching create adopts. The objection is not that it is impossible. It is what it costs on three axes.

Scalability. The number of native containers stops being a deployment decision and becomes a consequence of caller behaviour. Providers cap containers hard — 5 to 200 serverless indexes per project on Pinecone, 1000 collections per Qdrant Cloud cluster, 65,536 on Milvus — so an unbounded caller-driven set turns a capacity limit into a runtime failure triggered by what someone asked for. [Struck: an unverified claim that a Pinecone index takes minutes to create.]

Safety. Appearing atomic rests on a precondition that is nowhere written down: creation must be idempotent and content-addressed, so the orphan is adoptable rather than leaked. A backend that names containers per collection leaks one per crashed create, and nothing in the code says why that differs. Preconditions that are not stated are violated by the next implementation. Separately, config in the entry is a second source of truth for something that also lives in deployment configuration, and the two can drift apart silently — the same family as #1562.

Maintainability. The machinery that sustains the appearance is real and permanent: ordering rationale, crash-window reasoning, an adoptability precondition, and — for any backend where creation is not adoptable — an intent ledger with a sweeper and grace periods, purely to reclaim what a failed create left behind. All of it exists to manage a problem the interface introduces. Moving configuration to deployment configuration deletes that class of code instead of perfecting it: the container set becomes finite and provisionable, and creating a collection becomes one registry insert.

A smaller benefit follows. With no config in the entry there is nothing to compare, so get_or_register's "never compares entries" policy, the config-mismatch error, and the question of what config equality means across backends all leave the interface with it — questions the interface currently invites and cannot answer.

What is being removed is tenant-defined indexes, not indexes

The distinction that decides this is who defines an indexed property, not whether one exists.

Server-defined properties are declared once by the deployment and are therefore identical for every collection that server serves. That is precisely what makes container sharing work: co-located collections carry the same index set by construction, so there is no union to accumulate, no collection paying memory and build time for another's fields, and a new collection costs no migration.

Tenant-defined properties — one collection indexing invoice_id while another indexes patient_id — are the ones that break the sharing invariant, because the container must then carry the union of whatever its occupants happened to ask for. That union is what grows with churn and cannot be reclaimed.

Removing the config parameter removes the second, not the first. Indexed properties keep existing and keep being declared; they move to deployment configuration, where being identical across co-located collections becomes a property of the design rather than a coincidence of every caller passing the same thing. What disappears is a capability no vector database offers under shared-container tenancy.

Expected

The interface should promise what the architecture can support: one schema per native container, configured per deployment.

Suggested direction

Drop the config parameter from create_collection and move collection configuration into deployment configuration, alongside the decision about which native containers exist (#1572). Registry entries then record where a collection's records live, not how it is shaped.

Callers that need a property they did not declare are not blocked: both Qdrant and Milvus accept undeclared fields and filter them, so "filterable" and "efficiently filterable" are separate tiers and only the second needs declaring. Declaring a generous superset per deployment covers the rest.

This needs confirmation from whoever intended caller-defined indexes, since removing the parameter forecloses it.

Notes

Related to #1535, which covers indexed_properties_schema being honoured inconsistently across backends and forming part of collection identity. This issue is the narrower question of whether per-collection configuration should be in the interface at all.

#1572: Collection creation spans the registry and the vector backend, so it cannot be atomic; native containers belong in provisioning, not on the create path (filed 2026-09-02)

What happened

create_collection writes to two systems that share no transaction: it creates a native collection in the vector database, then records the collection in the registry. No ordering makes that atomic, so every choice is a choice about which garbage to leave.

Today's ordering is native-first, register-last, which is the safe one — a crash leaves an empty native collection rather than a registered collection with nowhere to put records. It is safe, though, only because native names are content-addressed and creation is idempotent, so the orphan is adopted by the next creation with the same config rather than leaked. That precondition is unstated, and it is load-bearing: a backend whose containers are named per-collection rather than per-config leaks one container per crashed creation, and nothing in the code says why that is different.

The deeper problem is that the dual write exists at all. Native containers are heavyweight and few — providers cap them: 5 to 200 serverless indexes per Pinecone project, 1000 collections per Qdrant Cloud cluster, 65,536 on Milvus — while logical collections are numerous and created at runtime. These are control-plane and data-plane concerns being done in one call.

Expected

Creating a logical collection should touch one transactional system, so it is atomic by construction and needs no ordering argument. Native containers should already exist when a collection is created.

Suggested direction

Move native container creation off the create path and into provisioning: a step that reads configuration and idempotently ensures the containers a deployment needs, run before serving rather than from a request. create_collection then becomes a single registry insert, arbitrated by a primary key.

This is the same problem and the same answer shape as #1570, which covers SQL schema provisioning running from every process's boot path. The two should share one provisioning entry point rather than inventing separate mechanisms — "create the tables" and "create the containers" are the same step for different resources.

Whatever is left over is then reclaimable rather than merely harmless: see #1565.

Notes

Related to #1524 and #1525, which cover the cross-process races in collection metadata. This issue is about the create path spanning two systems at all, which remains true however well the metadata side is arbitrated.

#1535: indexed_properties_schema is honoured on Qdrant, ignored on Milvus, and part of collection identity either way (filed 2026-08-28)

Summary

indexed_properties_schema is documented as a suggestion, so Milvus ignoring it is not strictly a contract violation. The problem is that it is acted on by Qdrant and ignored entirely by Milvus, a caller has no way to learn which, and the value is part of a collection's identity either way — so revising it forces a migration even on the backend where it does nothing.

common/vector_store/data_types.py:27-28:

indexed_properties_schema (dict[str, type[PropertyValue]]):
    Schema suggesting which properties should be indexed for filtering.

Qdrant acts on it

common/vector_store/qdrant_vector_store.py:767-775 iterates the schema and creates a payload index per property:

for prop_name, prop_type in config.indexed_properties_schema.items():
    index_type = QdrantVectorStore._PROPERTY_TYPE_TO_INDEX_TYPE.get(
        prop_type
    )
    if index_type is not None:
        await self._client.create_payload_index(
            collection_name=native_collection_name,
            field_name=prop_name,
            field_schema=index_type,

Milvus does not

common/vector_store/milvus_vector_store.py contains exactly two add_index( calls, at :557 and :692, and both are field_name=_VECTOR_FIELD. No index is created for any property.

The schema is read only to clear stale dynamic fields on upsert, at :192-194:

#### Explicit nulls clear stale dynamic fields during native Milvus upserts.
for key in self._config.indexed_properties_schema:
    entity[_property_field(key)] = None

The collection is created with enable_dynamic_field=True (:662), and per Milvus's documentation "any undefined scalar fields are stored as key-value pairs in JSON format" in the reserved $meta field. Declared and undeclared properties are therefore stored identically, and every property filter is evaluated over that JSON. The schema has no effect on Milvus beyond the pre-null.

Why it still matters

The cost is invisible. Queries return correct results, just at scan cost. Two deployments configured identically perform differently for a reason a caller cannot see without reading backend source, and cross-backend benchmarks silently compare different amounts of work.

Revising it is a migration regardless. open_or_create_collection compares the whole config on reopen, milvus_vector_store.py:788-792:

existing_config = MilvusVectorStore._parse_entry(entry)
if existing_config != config:
    raise VectorStoreCollectionConfigMismatchError(
        namespace, name, existing_config, config
    )

So indexed_properties_schema is part of a collection's identity. A deployment that revises it must migrate to a new collection — paying the full cost of a schema change on Milvus for a value that buys nothing there.

The budget is finite where the suggestion is honoured. Qdrant's max_payload_index_count "caps the maximum number of payload index that can exist on a collection"; Qdrant Cloud sets it to 100 and notes that "Larger numbers of payload indexes lead to performance degradation (starting with Qdrant v1.16.0)". Deciding how to spend a budget that small requires knowing which backends spend it at all.

Options

1. Create the indexes. Milvus supports indexing a dynamic-field key: "Milvus supports creating an index on such an undefined scalar field, effectively by building a JSON path index", using json_path plus json_cast_type with INVERTED ("Currently, only INVERTED type is supported for JSON path indexing").

Two caveats worth weighing before choosing this:

• json_cast_type accepts only bool, double and varchar, so the int and datetime property types in PropertyValue need a deliberate cast choice.
• "If parsing or casting fails (for example, trying to parse a string as double), those rows will be skipped in the index." Combined with the mixed-type allowance in VectorStoreCollection (see VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality #1534), a key holding more than one type would be silently and partially indexed.

2. Say so where a caller will see it. Note in VectorStoreCollectionConfig.indexed_properties_schema and in the Milvus backend docstring which backends act on the suggestion and which ignore it, so the difference is discoverable without reading source.

Either is fine. The problem is that a caller currently cannot find out, while still paying the migration cost of the schema being treated as identity.

Sources. Code read at 231ce171. External claims quoted from
Milvus 2.5 dynamic field documentation and
Qdrant Cloud cluster configuration.

Related: #1534.

#1534: VectorStoreCollection's undeclared-property mandate: refusable by Qdrant strict mode, unimplementable where attribute types are fixed, unbounded in key cardinality (filed 2026-08-28)

Summary

VectorStoreCollection's docstring requires implementations to support undeclared, mixed-type record properties. Three problems follow: a shipped backend has a supported configuration in which it cannot satisfy the clause, the mixed-type half is unimplementable on stores that fix attribute types, and nothing bounds how many distinct property keys a collection accumulates.

packages/server/src/memmachine_server/common/vector_store/vector_store.py:29-33:

Implementations must support storing, filtering on, and returning
record properties not declared in the configured indexed properties schema.

The schema exists to support indexing on fixed-type record properties.
Record properties not declared in the schema may have mixed-type values.

Problem 1: Qdrant can be configured to reject exactly what the clause mandates

Filtering on an undeclared property is, by definition, filtering on a non-indexed payload key. Qdrant's strict mode has a setting for refusing that: setting unindexed_filtering_retrieve to false "prevents retrieving points by filtering on a non indexed payload key which can be very slow".

To be precise about the default, because it matters: strict mode is "Available as of v1.13.0" in open-source Qdrant, and on Qdrant Cloud "strict mode is enabled by default for new collections" — but the same page notes that an unset restriction "does not have any effect. You need to explicitly set the restrictions you want to enforce." So this is not a claim that Qdrant Cloud rejects the mandate out of the box. It is that an operator protecting a shared cluster has a first-class, documented switch that makes the shipped Qdrant backend non-conformant, and the contract gives them no reason to think they shouldn't flip it.

The filtering docs show the same behaviour applied to a filter condition: without an index a condition "still returns correct results but is not accelerated (it is checked per point rather than served by the index)", and "when strict mode is enabled with unindexed_filtering_retrieve or unindexed_filtering_update set to false, a prefix condition is rejected unless the field has a prefix-enabled keyword index".

Problem 2: mixed types are unimplementable where attribute types are fixed

turbopuffer infers an attribute's type from its first occurrence, and "Changing the attribute type of an existing attribute is currently an error." Schema changes of that kind "cannot be done in-place"; the documented remedy is "exporting documents and upserting into a new namespace". A key that arrives as an int on one record and a str on the next is therefore a hard error there, so a conforming turbopuffer backend cannot be written — it will not be attempted, or it will be written non-conformant and diverge silently. (turbopuffer is not a backend in this repo; it is cited as a representative store this clause excludes.)

It already diverges, in-tree. The two shipped remote backends disagree today:

• SQL stores write type-tagged values ({"t": …, "v": …}) and gate every comparison on the tag, so a filter never matches a value of another type — _compile_properties_json_leaf, common/filter/sql_filter_util.py:160-167.
• Milvus writes untagged dynamic fields — _normalize_property_filter_value at milvus_vector_store.py:92-96 converts only datetime, returning every other value unchanged — and inherits whatever Milvus's own JSON comparison does.

Same filter, same data, two answers, and nothing says which is correct.

It also creates an undocumented semantic split. Because the SQL compiler must gate on a type tag, Comparison(field, "!=", v) compiles at sql_filter_util.py:167 to and_(type_check, col != v) — matching only records holding a value of v's type that differs — while Not(Comparison(field, "=", v)) compiles at :203 to ~and_(type_check, col == v), which additionally matches records that do not carry the field or carry it with another type. Both are reasonable predicates; they are not the same predicate; and the difference is discoverable only by reading the compiler. That subtlety exists solely because the contract permits mixed types.

Finally, the allowance imposes machinery no caller asked for: the type-tagged {"t", "v"} encoding and its per-leaf type_check exist to make cross-type comparison well defined for undeclared keys.

Problem 3: key cardinality is unbounded, and on several stores it is a shared budget

Because undeclared keys must work, nothing bounds how many distinct property keys a collection accumulates. Severity depends on how the backend stores keys.

On the shipped backends it is a cost paid by neighbours. Qdrant, Milvus and the SQL stores hold undeclared keys in a JSON payload, so there is no hard cap — but an unaccelerated filter is "checked per point", against a payload that grows without bound, and because the clause also requires returning undeclared properties that cost is paid on ordinary reads too. On a collection shared across tenants (Qdrant's is_tenant=True partition-key index at qdrant_vector_store.py:759-765, Milvus partition keys) one caller's key sprawl is paid for by everyone co-located with it.

The indexed side is explicitly capped. max_payload_index_count is a Qdrant strict-mode parameter that "caps the maximum number of payload index that can exist on a collection". Qdrant Cloud documents "The maximum number of payload indexes per collection is set to 100 (max_payload_index_count is set to 100)" and that "Larger numbers of payload indexes lead to performance degradation (starting with Qdrant v1.16.0)". Since indexed_properties_schema is compared for equality when reopening a collection, that is a hard ceiling on how much of the property space can ever be made efficient — and it appears nowhere in the contract.

On a store that maps keys into a schema it becomes exhaustion. Elasticsearch is the canonical precedent (not a backend here; cited for the failure mode): index.mapping.total_fields.limit has "default value is 1000", and "Beyond this limit, Elasticsearch returns the error Limit of total fields [X] has been exceeded". The opt-in escape, ignore_dynamic_beyond_limit (default false), is worse for a filter contract: "the index request will not fail. Instead, fields that would exceed the limit are not added to the mapping... The fields that were not added to the mapping will be added to the _ignored field." Writes succeed, filters on those fields match nothing, and nobody is told.

Suggested direction

1. Make the type rule a caller contract rather than an implementation mandate:

A property key holds values of a single type within a collection. An implementation may reject a write that changes a key's type.

This is what turbopuffer already does (type inferred from first occurrence, later change is an error), so it makes the strictest backend the reference rather than the non-conformant one. It also lets the SQL stores' type tagging be defence in depth rather than load-bearing, and removes the != / Not(=) subtlety along with its cause.

2. State a minimum key-cardinality guarantee — an implementation must support at least N distinct property keys per collection — so callers get a portable number instead of discovering a database limit in production.

3. Separate the bundled promises, since they are not equally portable:

• storing undeclared properties — universal, keep as required
• filtering undeclared properties — normally available, but the contract must accommodate a store configured to refuse it (Qdrant strict mode), and an implementation that can only offer it by creating an index should say so, because on a shared collection that cost lands on co-located tenants
• returning undeclared properties — worth stating separately, since it makes payload size a per-read cost
• mixed types — replaced by the single-type contract in (1)

Sources. Code read at 231ce171. External claims quoted from:
Qdrant strict mode ·
Qdrant filtering ·
Qdrant Cloud cluster configuration ·
turbopuffer schema ·
Elasticsearch mapping limit settings

Related: #1535.

#1528: [Bug]: Semantic memory indexes free-form content as filterable vector-store properties (filed 2026-08-26)

Describe the bug

VectorStoreSemanticStorage declares a semantic feature's free-form content as indexed, filterable vector-store properties.

_vector_properties (vector_store_semantic_storage.py:821) writes value -- the feature's content -- and every scalar entry of caller-supplied metadata into the vector record's properties, and semantic_manager.py:156 declares value in indexed_properties_schema. Declaring a property in that schema is what causes each backend to build an index for it: a keyword payload index per field in Qdrant (qdrant_vector_store.py:_create_native_collection), a JSON expression index per field in sqlite-vec (sqlite_vec_vector_store.py:_ensure_collection_tables).

value is not a filter dimension. It is unique per record, free-form, and is precisely what the record's vector already encodes -- the collection is storing and indexing each record's content inside its own index, in a form that can only ever answer exact-string equality. Caller-supplied metadata is unbounded and unenumerated in the same way; it is written into the payload for whatever keys a caller happens to pass.

Consequences, all on the hot paths:

• Index build and maintenance over high-cardinality free text, per backend, that no query can use selectively.
• _vector_search_features (:635) queries with return_properties=True and a limit of _DEFAULT_VECTOR_QUERY_LIMIT = 10_000 (:54, :646), so every semantic search transfers each matched record's full property set -- content included -- for up to 10,000 records, in order to read one field, feature_id (:657).
• Every metadata-only update re-reads the stored vector and rewrites the whole record to keep the copy current (:295-311), so content duplication costs a vector round trip on writes that never touched the vector.

The authority for all of this is already relational: VectorSemanticFeature (:83) holds value, json_metadata, and the rest, with composite indexes for the real lookup shapes.

[Steps to reproduce struck: #1598 (merged 2026-09-10) removed the properties from the vector record and the payload read; the indexed_properties_schema declaration in semantic_manager.py remains.]

Expected behavior

Content does not become an indexed vector-store property. A record's payload carries what the vector store itself needs -- here, the feature_id pointer back to VectorSemanticFeature -- and content stays in the relational authority that already holds it.

Concretely: drop value and the merged caller metadata from _vector_properties, and drop value from indexed_properties_schema. feature_id is read back but never filtered on, and VectorStoreCollection already contracts to store and return properties that are not declared in the schema, so it needs no schema entry.

Additional context

Separate from this defect, and deliberately not folded into it: the remaining declared properties (set_id/set, semantic_category_id/category_name/category, tag_id/tag, feature/feature_name) are legitimate filter dimensions -- enumerated, repeated, and exactly what a pushed-down pre-filter would use. They are unused today only because filtering is routed to SQL: no call site passes property_filter to the collection, and _resolve_feature_field (:774) maps every caller-facing filter field, these included, onto a VectorSemanticFeature column or json_metadata[key]. Whether they stay in the payload is a routing decision about where filtering happens, not a category error, and should be decided on its own terms.

If pre-filtering is ever pushed down to the vector store, it should cover only attributes that are immutable after ingest. Post-filtering at the relational authority can drop candidates but cannot recover ones a stale pre-filter wrongly excluded, so a mutable attribute used as a pre-filter converts staleness into unrecoverable false negatives.

Note on landing this: indexed_properties_schema is part of VectorStoreCollectionConfig, and both remote stores derive native collection names as sha256(config.model_dump_json()), so changing the schema repoints the logical collection at a new, empty native collection and orphans the existing vectors. This needs a reindex, or it should land together with storing the resolved native collection name in the collection registry (#1524) so that config evolution stops re-deriving storage identity.

#1562: QdrantVectorStore re-derives the native collection name on every open, so any config-serialization change silently orphans existing data (filed 2026-09-01)

What happened

QdrantVectorStore never stores the native collection name it resolved. It re-derives it from the config on every open:

def _build_collection_handle(self, namespace, name, config):
    return QdrantVectorStoreCollection(
        collection_name=QdrantVectorStore._build_native_collection_name(namespace, config),
        ...

and the name is f"{namespace}__{sha256(config.model_dump_json())}".

So the native collection a logical collection resolves to is a function of how VectorStoreCollectionConfig serializes today, not of what it resolved to when the data was written. Any change to that serialization silently repoints every existing collection at a different name — which does not exist, so the next write creates it empty.

Changes that do this, none of which look dangerous at review time:

• adding a field to VectorStoreCollectionConfig, even an optional one with a default
• changing an existing field's default
• reordering fields
• anything that alters pydantic's model_dump_json() output, including a pydantic upgrade

The symptom is not an error. Reads return nothing, writes succeed into a new empty collection, and the original data is still on disk under the old name with nothing pointing at it.

Expected

A collection's identity is fixed when it is created. Later changes to config serialization must not be able to move it.

Fix

Store the resolved native collection name in the registry entry at creation and use the stored value on open, instead of recomputing a hash. #1527 does this — the entry gains native_collection_name, pinned at registration.

Notes

Related but distinct from #1535, which is about indexed_properties_schema being part of collection identity at all. This issue is narrower: even if the identity inputs are agreed to be right, deriving the name at open time rather than storing it makes the mapping unstable across versions.

#1565: Qdrant integration never reclaims native collections or records under dead partition keys (filed 2026-09-01)

What happened

Nothing in the Qdrant integration ever reclaims a resource, and two kinds of garbage accumulate permanently.

Native collections. The native name is {namespace}__{sha256(config)}, so a new collection appears for every distinct config, including every revision of the server-global properties_schema. They are never deleted, nothing enumerates them, and nothing can tell whether one is still referenced. Qdrant Cloud caps a cluster at 1000 collections.

Records under dead partition keys. After delete_collection, records written through a handle held across the deletion land under the old partition key. With #1527 they are correctly invisible and can never be resurrected — the code comment says so — but they are also unreclaimable, because nothing records that the partition key ever existed. Payload values can be enumerated only by a facet with exact=true and a raised limit (approximate and 10 values by default), a scan of the collection.

A failed creation leaves a third, milder kind: an empty native collection, and in distributed mode an unused shard key. That one is self-healing when the same config recurs, since creation is idempotent and content-addressed.

Expected

Every resource that exists should be either referenced by a live registry entry or recorded somewhere that a reclaimer can find it. Leakage should be bounded and, ideally, actively reclaimed.

Suggested direction

Two mechanisms, because the two kinds of garbage are discoverable in different ways:

• Container-level garbage is enumerable — Qdrant can list its collections — so a periodic reconcile that removes collections matching no live entry handles it, with a grace period so an in-flight creation is not mistaken for garbage.
• Record-level garbage is discoverable only by an exact facet scan, so it should be written down when it is created: a queue row committed in the same transaction as the registry deregistration, drained by a purger that filter-deletes by partition key. That also makes delete_collection O(1) for the caller instead of O(rows), which matters since deleting a large tenant is proportional to its record count.

The reclaim verb belongs on the store, since only it knows what deleting means for its backend; the queue belongs with the registry, which already has the transaction.

Notes

Not addressed by any open PR. #1526's design doc records the absence of enumeration as a dated omission.

#1564: Commit Qdrant to payload-partitioned multitenancy: drop per-collection shard keys, retire the in-process lock table, recover fencing and reclamation (filed 2026-09-01)

Qdrant offers two ways to separate tenants, and they have opposite cost profiles. This issue tracks committing to the scalable one and finishing it, rather than leaving both half-supported.

Measured against Qdrant 1.19.0, one hundred tenants each way:

Payload partitioning is the only one that scales, because a tenant is a value rather than a physical structure. Everything below follows from choosing it.

1. Stop creating a shard key per logical collection

QdrantConf.is_distributed switches to custom sharding and gives every logical collection its own shard key:

sharding_method=models.ShardingMethod.CUSTOM if self._is_distributed else None
...
if self._is_distributed:
    await self._ensure_shard_key(entry.native_collection_name, entry.partition_key)

The cost is per logical collection. A deployment serving thousands of tenants — a tenant being a user, or a group of users — has at least that many logical collections and in practice more, since a tenant's memory is typically partitioned further. At a thousand logical collections that is ~7.5 minutes of shard-key creation and ~5,000 segments; at ten thousand, ~75 minutes and ~50,000 segments, before any data is written.

Qdrant's own guidance recommends against per-tenant physical structures: "Creating a separate collection for each tenant is rarely the most efficient approach. Each collection carries its own resource overhead."

The flag's docstring sells the O(1) delete, which is genuine, but it is paid for per tenant. Note the two mechanisms are layered correctly — the payload partition key is written and filtered in both modes, with the shard key passed additionally — so this is a policy problem, not a correctness one.

If a container ever needs splitting, the right lever is shard_number > 1 with automatic sharding: a provisioning knob, no custom sharding, no per-tenant cost.

2. Retire the per-process lock table

_name_locks serialises lifecycle operations within one process, which was the only arbitration available before a cross-process registry existed. It also leaks:

#### Keyed by client so locks are garbage-collected when the client is.
_name_locks: ClassVar[
    WeakKeyDictionary[AsyncQdrantClient, defaultdict[tuple[str, str], asyncio.Lock]]
] = WeakKeyDictionary()

The outer mapping is weak, but the inner defaultdict gains a lock per (namespace, name) the process ever touches and never drops one, for the client's lifetime. Small individually, unbounded in the number of logical collections a process serves, and easy to miss because the comment reads as though the lifetime question is settled.

Once the registry arbitrates lifecycle across processes (#1524, #1525, delivered by #1526 and #1527), these locks are vestigial and should go rather than be bounded.

3. Recover what payload partitioning gives up

Choosing it means giving up the two things the shard-key path provided, both of which have to be built back:

• Fencing. Qdrant will not reject a write carrying a retired tenant value. Incarnation-scoped partition keys (Handles held across delete and recreate resurrect records in QdrantVectorStore #1563, Feat: Back QdrantVectorStore collection metadata with SQL collection registry (collection registry stack 2/5) #1527) make such writes invisible rather than harmful; closing the window entirely needs the handle to be lease-validated against the registry.
• Reclamation. Records under a dead partition key are discoverable only by an exact facet scan (facet is approximate and capped at 10 values by default), so they should be recorded at deletion time and purged. Tracked in Qdrant integration never reclaims native collections or records under dead partition keys #1565.

Tiered multitenancy stays available

Promoting a single oversized tenant to its own shard key, while the long tail stays payload-partitioned, is Qdrant's third documented approach and remains open. sharding_method is immutable per collection — verified, creating a shard key on an auto-sharded collection fails with "Shard Key cannot be created with Auto sharding method" — but since registry entries pin their native collection, promotion later means provisioning a new custom-sharded collection and directing new collections to it, leaving existing ones untouched. Nothing here forecloses it.

🤖 Written by Claude Code (Opus 5) on behalf of @edwinyyyu.