Skip to content

Feat: Remove open-or-create semantics from VectorStore and SegmentStore (collection registry stack 3/5) - #1530

Closed
edwinyyyu wants to merge 1 commit into
MemMachine:mainfrom
edwinyyyu:feat/remove-open-or-create
Closed

edwinyyyu wants to merge 1 commit into
MemMachine:mainfrom
edwinyyyu:feat/remove-open-or-create

Conversation

@edwinyyyu

@edwinyyyu edwinyyyu commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the change

Remove adopt-on-exists (open_or_create_*) semantics from the storage lifecycle APIs. With collection management becoming multi-process-capable (#1524, #1525), the long-term shape is a strict control plane — create fails on an existing resource, open fails on a missing one — with ensure semantics confined to bootstrap composition at the call sites that genuinely need it. An audit found exactly two consumers of these methods, neither of which needs adopt-on-exists as a primitive.

Stacked on #1527 (its qdrant lifecycle is what this PR edits); only the last commit is new to this PR.

Description

Removed from the ABCs and every implementation:

  • VectorStore.open_or_create_collection (Qdrant, Milvus, SQLite, sqlite-vec) and VectorStoreCollectionConfigMismatchError — the method was its only raiser.
  • SegmentStore.open_or_create_partition (SQLAlchemy) and SegmentStorePartitionConfigMismatchError, along with the implementation's mismatch-check helper.

EventMemory itself has no ensure semantics (it receives already-opened handles), so nothing changes there.

Consumers restructured to strict create/open with concurrent-creation tolerance:

  • semantic_manager.get_semantic_storage: open; on miss create (suppressing AlreadyExistsError — a concurrent creator winning is fine) and re-open.
  • service_locator (episodic event backend): the partition path gets the same open -> create -> open shape its collection path already used; the collection path additionally gains AlreadyExistsError tolerance, which multi-process deployments make reachable.

Config-mismatch detection goes away with the methods: with strict create, a caller that loses a creation race adopts the winner's collection via open, and its config is available on the handle for any consumer that wants to enforce its own equality policy. No current consumer does — configs are derived from code, not user input, at every call site.

The collection registry's get_or_register (from #1526) is unaffected: it is the registry-level primitive these APIs no longer surface.

Design doc: design/storage_lifecycle_strictness.md.

Fixes/Closes

Related to #1524 / #1525.

Type of change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactor (does not change functionality, e.g., code style improvements, linting)

How Has This Been Tested?

  • Unit Test
  • Integration Test

Removed the open_or_create-specific tests; remaining usages in test suites converted to explicit create+open helpers. The segment-store concurrency tests now create their partition once up front (the previous version exercised concurrent adopt-on-exists). Semantic-manager wiring test updated to the open -> create -> open flow. Full server suite plus the qdrant/registry/segment-store integration matrix pass.

Test Results: 1891 passed (default run); 574 passed (targeted integration run: qdrant http/grpc/distributed containers + PostgreSQL); ruff check, ruff format --check, ty check packages clean (no new diagnostics).

Checklist

  • I have signed the commit(s) within this pull request
  • My code follows the style guidelines of this project (See STYLE_GUIDE.md)
  • I have performed a self-review of my own code
  • My changes generate no new warnings
  • New and existing unit tests pass locally with my changes

Further comments

EpisodicMemoryManager.open_or_create_episodic_memory is deliberately untouched: it is an in-process instance-cache accessor (get-or-construct a session's memory object), not a durable-storage ensure, so it is a different kind of API. Flagging it in case the naming alignment is wanted separately.

@edwinyyyu edwinyyyu changed the title Feat: Remove open-or-create semantics from VectorStore and SegmentStore Feat: Remove open-or-create semantics from VectorStore and SegmentStore (3/4) Aug 26, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/remove-open-or-create branch 3 times, most recently from 9f5e463 to b1faa00 Compare August 27, 2026 17:27
@edwinyyyu edwinyyyu changed the title Feat: Remove open-or-create semantics from VectorStore and SegmentStore (3/4) Feat: Remove open-or-create semantics from VectorStore and SegmentStore (3/5) Aug 27, 2026
@edwinyyyu edwinyyyu changed the title Feat: Remove open-or-create semantics from VectorStore and SegmentStore (3/5) Feat: Remove open-or-create semantics from VectorStore and SegmentStore (collection registry stack 3/5) Aug 27, 2026
@edwinyyyu
edwinyyyu force-pushed the feat/remove-open-or-create branch 11 times, most recently from 828ceba to 6c6261c Compare August 28, 2026 00:25
@edwinyyyu
edwinyyyu force-pushed the feat/remove-open-or-create branch 4 times, most recently from 6b740f4 to e98835f Compare September 1, 2026 22:17
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 3, 2026
From the registry session's review of the draft. Store creates are
strict and raise on any row under the key; idempotency is the
component's ensure, which knows the key's provenance, and a row in a
non-live state is a reused key that raises. A table states what every
operation does with a non-live key. Tombstones are kept by default;
pruning is an operator's trade gated on a clean sweep, with what it
gives up stated. The fence's cost is stated for sizing: a pooled
connection held across each remote write, one row read per query.
Containers are retired by the schema command once undeclared and
unreferenced. maintain runs without exclusion and says why that is
safe. MemMachine#1530 is recorded as agreeing.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MbYdqGZsuws6Z2WHYfCCR5
@edwinyyyu
edwinyyyu force-pushed the feat/remove-open-or-create branch from e98835f to e71664c Compare September 3, 2026 16:57
edwinyyyu added a commit to edwinyyyu/MemMachine that referenced this pull request Sep 3, 2026
Same signature and reason; MemMachine#1530's names are reused and separated by
incarnations, so its create raising means exists, while here a row
under a never-reused key is a violated invariant and kept tombstones
are load-bearing.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MbYdqGZsuws6Z2WHYfCCR5
@edwinyyyu

Copy link
Copy Markdown
Contributor Author

#1579 outlines a full design so it would be churn to merge this.

@edwinyyyu edwinyyyu closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant