Repository navigation
fix(vector-store): create payload indexes even when the collection exists - #1578
Merged
Merged
Conversation
…ists _create_native_collection wrapped create_collection and every create_payload_index in one try and swallowed "already exists" for the whole block. A collection that already existed therefore raised on the first call, took the already-exists path, and was left with no payload indexes at all - despite the docstring promising both were created idempotently. Two creators are easy to arrive at. The guarding lock is keyed on the AsyncQdrantClient object, so it serialises callers inside one process and nothing across them; with MEMMACHINE_WORKERS above 1 each worker has its own client and its own lock. A crash between the two calls leaves the same state. The collection and the indexes now sit under separate guards, and each index is created individually and tolerant of already-exists. Verified against Qdrant 1.19 in testcontainers. Before the change, test_indexes_are_created_when_the_collection_already_exists fails with an empty payload_schema - not a missing index, none of the twelve. After it, both new tests pass, along with the rest of the vector store suite: 293 unit and 142 integration. Scope, checked rather than assumed: a filtered query on an unindexed collection returns only the matching tenant's points, so what a missing sys-partition_key index costs is the multitenant storage layout and query speed, not isolation. Note also that only the already-exists path reproduces; two clients creating simultaneously both succeed, which the second test pins. The tests need a real server and are marked integration - local-mode Qdrant ignores payload indexes, so the defect is invisible there and CI, which runs with `-m "not integration"`, will not exercise them. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP
edwinyyyu
approved these changes
Sep 2, 2026
edwinyyyu
left a comment
Contributor
There was a problem hiding this comment.
Heads up: The implementation needs to change completely to properly support horizontal scalability without sharding. Also having dynamic schema/index creation is not good, so that should likely change as well.
This was referenced Sep 16, 2026
Merged
Closed
Merged
[session storage 2/2] Remove open-or-create from both stores, and close from the segment store
#1625
Draft
Closed
This was referenced Sep 16, 2026
wanghy73
added a commit
to wanghy73/MemMachine
that referenced
this pull request
Sep 18, 2026
…ists (MemMachine#1578) _create_native_collection wrapped create_collection and every create_payload_index in one try and swallowed "already exists" for the whole block. A collection that already existed therefore raised on the first call, took the already-exists path, and was left with no payload indexes at all - despite the docstring promising both were created idempotently. Two creators are easy to arrive at. The guarding lock is keyed on the AsyncQdrantClient object, so it serialises callers inside one process and nothing across them; with MEMMACHINE_WORKERS above 1 each worker has its own client and its own lock. A crash between the two calls leaves the same state. The collection and the indexes now sit under separate guards, and each index is created individually and tolerant of already-exists. Verified against Qdrant 1.19 in testcontainers. Before the change, test_indexes_are_created_when_the_collection_already_exists fails with an empty payload_schema - not a missing index, none of the twelve. After it, both new tests pass, along with the rest of the vector store suite: 293 unit and 142 integration. Scope, checked rather than assumed: a filtered query on an unindexed collection returns only the matching tenant's points, so what a missing sys-partition_key index costs is the multitenant storage layout and query speed, not isolation. Note also that only the already-exists path reproduces; two clients creating simultaneously both succeed, which the second test pins. The tests need a real server and are marked integration - local-mode Qdrant ignores payload indexes, so the defect is invisible there and CI, which runs with `-m "not integration"`, will not exercise them. Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP Co-authored-by: Claude Opus 5 <[email protected]> (cherry picked from commit a2a1754) Signed-off-by: Haiyan Wang <[email protected]>
malatewang
pushed a commit
that referenced
this pull request
Sep 18, 2026
…ists (port of #1578 to main) (#1681) * fix(vector-store): create payload indexes even when the collection exists (#1578) _create_native_collection wrapped create_collection and every create_payload_index in one try and swallowed "already exists" for the whole block. A collection that already existed therefore raised on the first call, took the already-exists path, and was left with no payload indexes at all - despite the docstring promising both were created idempotently. Two creators are easy to arrive at. The guarding lock is keyed on the AsyncQdrantClient object, so it serialises callers inside one process and nothing across them; with MEMMACHINE_WORKERS above 1 each worker has its own client and its own lock. A crash between the two calls leaves the same state. The collection and the indexes now sit under separate guards, and each index is created individually and tolerant of already-exists. Verified against Qdrant 1.19 in testcontainers. Before the change, test_indexes_are_created_when_the_collection_already_exists fails with an empty payload_schema - not a missing index, none of the twelve. After it, both new tests pass, along with the rest of the vector store suite: 293 unit and 142 integration. Scope, checked rather than assumed: a filtered query on an unindexed collection returns only the matching tenant's points, so what a missing sys-partition_key index costs is the multitenant storage layout and query speed, not isolation. Note also that only the already-exists path reproduces; two clients creating simultaneously both succeed, which the second test pins. The tests need a real server and are marked integration - local-mode Qdrant ignores payload indexes, so the defect is invisible there and CI, which runs with `-m "not integration"`, will not exercise them. Claude-Session: https://claude.ai/code/session_01Nr9kacmpFVTTfkZRw6esxP Co-authored-by: Claude Opus 5 <[email protected]> (cherry picked from commit a2a1754) Signed-off-by: Haiyan Wang <[email protected]> * style: reformat an assert ruff 0.15.14 formats differently (speedkick) (#1592) style: reformat an assert ruff 0.15.14 formats differently `ruff format --check` fails on speedkick as it stands: the pinned ruff (0.15.14, from the root pyproject the lint workflow reads its version from) moves the message of a multi-line `assert` onto its own line, and one assert in the Qdrant store tests predates that. No behavior change. Every PR targeting speedkick inherits the red format check until this lands, which is why it is on its own rather than folded into one of them. Claude-Session: https://claude.ai/code/session_01NKmF9xNph9QH3ozNw3ZnJL Co-authored-by: Claude Opus 5 (1M context) <[email protected]> (cherry picked from commit c2d234b) Signed-off-by: Haiyan Wang <[email protected]> --------- Signed-off-by: Haiyan Wang <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: Edwin Yu <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
_create_native_collectionwrappedcreate_collectionand everycreate_payload_indexin onetry, then swallowed "already exists" for the wholeblock. A collection that already existed therefore raised on the first call, took the
already-exists path, and was left with no payload indexes at all — despite the
docstring promising both were created idempotently.
Two creators are easy to arrive at. The guarding lock is keyed on the
AsyncQdrantClientobject:so it serialises callers inside one process and nothing across them. With
MEMMACHINE_WORKERSabove 1 each worker has its own client and its own lock. A crashbetween the two calls leaves the same state.
The change
The collection and the indexes now sit under separate guards, and each index is created
individually and tolerant of already-exists.
Evidence
Against Qdrant 1.19 in testcontainers.
Before —
test_indexes_are_created_when_the_collection_already_existsfails, and notwith one index missing:
payload_schemais empty — none of the twelve.After — both new tests pass, along with the rest of the suite: 293 unit, 142
integration. ruff and ruff format clean.
Scope, checked rather than assumed
A filtered query on an unindexed collection returns only the matching tenant's points, so
what a missing
sys-partition_keyindex costs is the multitenant storage layout andquery speed, not isolation. I verified this directly rather than reasoning from
is_tenant=True.Only the already-exists path reproduces. Two clients creating simultaneously both
succeed, which the second test pins.
What a reviewer needs to know
These tests are marked
integrationand need a real server — local-mode Qdrant ignorespayload indexes, so the defect is invisible there. CI will not exercise them, since
addopts = ["-m", "not integration"]. To run them:Docker required; the fixture starts its own Qdrant.
Not affected
The mm-tb6 benchmark collection was checked and holds all twelve payload indexes
including
sys-partition_key, covering every point — no measurement ran against anunindexed collection.