Streaming SPOT diff for graph sync — memory bounded by the delta, not the graph #1691

Follow-up to #1682, which ships graph sync with the same memory profile as CLEAR/COPY/MOVE: staging materializes every currently-asserted flake of the target graph, plus a hash-map copy in the accumulator, before netting.

Problem

The delta computation is memory-bounded by the graph, not the delta. An identical resync of a large graph materializes the whole graph, nets to zero, and commits nothing — peak memory is ~2 copies of every fact (accumulator key + bucket survivor, ~200 bytes inline per flake plus heap for literal strings) regardless of how little changed. Rough order: 10M triples ≈ 4–6 GB peak; 100M is tens of GB and an OOM kill. Under Raft the branch-owning worker node is the one that dies, mid-stage.

None of the existing guards bound it: at_max_novelty runs before the scan but measures only current novelty; NoveltyWouldExceed measures only the surviving delta, after materialization; fuel charges land after the scan returns its Vec.

#1682 adds a pre-flight size cap as a stopgap — a clear resource-limit error instead of an OOM — but the cap refuses large graphs rather than handling them.

Design sketch: streaming SPOT diff

The pieces are already in place for a two-pointer diff that never materializes the current graph:

1. The whole-graph scan (scan_graph_flakes) reads SPOT order — the cursor yields sorted batches natively, and the overlay merge already resolves novelty lifecycles inline.
2. Sort the payload's assertion flakes SPOT once (O(|payload| log |payload|) — the payload is already in memory from JSON-LD parsing, so this is the floor).
3. Walk both sides with two pointers on (s, p, o, dt, m) identity (the accumulator's fact key, which deliberately excludes t/op/g): equal → skip; scan-only → emit retraction; payload-only → emit assertion.

Memory drops to O(|payload| + |delta|).

What makes it more than a loop

• The accumulator is bypassed, so the netting semantics it provides must be preserved explicitly: per-graph scoping, duplicate collapse within the payload, and the m (language tag / list position) component of identity. finalize()'s SPOT sort of the delta comes free from the merge order.
• Modify-policy enforcement currently sees the full staged set after cancellation; the streaming path must feed it the emitted delta only, which is the same set — but SHACL validation reads the resulting state and needs to keep working against a delta produced incrementally.
• One commit, not chunked commits. The verb's contract is one atomic commit holding exactly the delta. The delta itself is bounded by novelty backpressure (NoveltyWouldExceed), so emitting it into one commit stays safe; it is the scan that must stream, not the commit.
• Blank nodes: the deterministic graph-scoped skolem key already makes payload bnodes comparable to stored ones; the streaming diff inherits that unchanged, but RDFC-style canonicalization (the other deferred item) would change payload ordering and should be sequenced first or together.
• CLEAR/DROP/COPY/MOVE share scan_graph_flakes and inherit whatever it becomes; COPY/MOVE additionally hold source and destination simultaneously and need their own look.

Acceptance

• An identical resync of a graph far above the pre-flight cap completes as a no-op within a bounded memory envelope (the cap becomes a guard on the payload, or is retired).
• Delta counts, policy, SHACL, bnode stability, and the no-op path match the accumulator implementation — the feat: graph sync — whole-graph replacement committing only the delta #1682 test suite should pass unchanged against the streaming path.

Origin

#1682 review discussion; the materialization profile is called out in that PR's body as CLEAR's known cliff, with chunked staging named as the follow-up.