Repository navigation
lite→Origin document sync: pushed CRDT deltas are acked but never materialize into a shape-servable collection (ShapeSnapshot doc_count=0) #146
Description
Activity
Note on layer scope, since pagedb#6 was just fixed with a related-sounding symptom:
NodeDB-Lab/pagedb#6's fix #2 ("incremental apply never advanced the trees — applied segments and key/value data were unreachable on the follower even after promotion") rhymes with this issue's symptom (pushed deltas accepted/promoted but unreachable on read). But I checked:
nodedb(Origin) does not depend onpagedb— that fix is lite-side. So it does not carry over to this Origin-side document-serve gap; this remains open.Flagging it anyway in case the same class of bug exists in the Origin's own materialization path — i.e. pushed document CRDT deltas are committed/checkpointed but the target collection root the shape snapshot reads from is never advanced, so
ShapeSnapshot.doc_countstays 0. If the Origin's snapshot/materialize path has an analogous "never advanced the target root" defect, the fix shape may mirror pagedb#6's.Still reproduces on current
main(Origin LSN advances to 3577 from the pushes, 20MB crdt-ckpt persisted, but shape round-trip = 0 docs in all tenants).Resolved. Root cause was two distinct gaps in the lite→Origin document sync path; both are now fixed and proven by a cross-repo end-to-end test.
1. Collection was never registered on Origin. A collection created only on lite pushed CRDT deltas, but Origin had no catalog entry for it, so the deltas had nowhere to land as a shape-servable collection. Fixed by a standardized collection-schema announce: lite now emits a
CollectionSchemaframe (opcode 0x13) before the first delta for a collection, and Origin materializes it create-only viaPutCollectionIfAbsent.2. Synced CRDT deltas were acked but never materialized into the queryable document store. Even once registered, applying a Loro delta on Origin updated CRDT state (and the FTS index, delivered via a separate frame) but never wrote the document into the sparse
DOCUMENTSbtree thatDocumentScan/ShapeSnapshotread — so a plain scan and the shape snapshot returned 0. Fixed on the Data Plane apply path: after a clean CRDT apply, the merged row is now materialized into the sparse document store (keyed by surrogate, matching the native put path; bitemporal-aware), making the collection shape-servable.Proof (E2E, no Origin pre-create): a schemaless document collection created only on lite, written to, and synced now becomes visible in Origin's
pg_class, is returned by bothtext_matchand a plainSELECT id, and yields a non-zero ShapeSnapshot document set.Follow-ups tracked separately (out of scope here): the synced-doc apply path does not yet emit CDC WriteEvents, update column stats, or maintain secondary/vector/spatial indexes the way the native write path does.
Re-ran this end-to-end against
main(incl.3a06321e) under mae8's real workload, and the fix definitely moved the needle: the collection now registers and the round-trip returns non-zero, exactly as your E2E shows. But it lands at 1, not N — only the first document of a synced collection becomes servable. So the literaldoc_count=0is gone, but the round-trip still doesn't carry the full set. Reopening since it's the same round-trip, just one layer deeper.Repro
- Origin on
main(trust auth), sync WS:9090. - From a Lite replica, register collection
entriesviacreate_collection(so it registers cleanly — see the companion nodedb-lite announce issue) and write N > 1 distinct documents with sync enabled. Throttle to stay under the sync rate limit so nothing is dropped. - On Origin:
SELECT count(*) FROM entries;→ 1.SELECT id FROM entries;→ only the first-written doc. AShapeSnapshotreturns a 1-document set, not N.
Measured at N = 5 / 300 / 1000 — deterministic, always the first-written doc.
What I ruled out (so this isn't a repro artifact)
- Not the client write path. The local Lite replica holds all N distinct docs (local scan), and Origin's own logs show each delta arriving with a unique per-document UUID (
collection=entries doc=019f…). - Not delta loss. Throttled run: session close
mutations=15 rejected=0 silent_dropped=0for N=5; all applied cleanly, none rejected/fenced. - Not bitemporal-specific. Announcing the collection as non-bitemporal (so the merged row goes through the plain
put, notversioned_put) still yields count=1. - Not a materialize-lag. Waited through multiple 30s clone-materializer sweeps; stays at 1.
The sparse store does grow during the run (
core-N.redbgrows,CRDT engines checkpointed … dirty_pages=955), so bytes are landing — but only the first document surfaces toDocumentScan/ShapeSnapshot.Where I think it lives (your call to confirm)
Data-Plane apply materialize:
data/executor/handlers/control/crdt.rs::execute_crdt_apply→crdt_materialize.rs::{encode_crdt_row, materialize_synced_document}. The applies reportCleanand each carries a unique surrogate, so the gate (Clean && surrogate != ZERO) should fire per doc. Working hypothesis: the Lite deltas are incremental Loro updates (ExportMode::updates(&version_before)), and when applied independently to Origin's Loro replica,engine.read_row(collection, document_id)only resolves the first document into the merged map — soencode_crdt_rowreturnsNonefor docs 2..N and they never reach the sparse btree. If that's right, the fix shape is on the Origin's incremental-apply → row-reconstruction, not the materialize write itself.Happy to hand over the exact harness + logs (it's mae8's
bruteforce_origin_synctest).- Origin on
- added and removed
on Jul 8, 2026 Reopening to investigate under v0.4.0 — the round-trip now returns non-zero but caps at 1 (only the first-written doc surfaces). Digging into the Origin incremental-apply → row-reconstruction path (
crdt_materialize.rs::{encode_crdt_row,materialize_synced_document}) as you pointed to; will report root cause + fix.Fixed on
mainin 222ad32.Root cause: Origin acked pushed CRDT deltas but never materialized them into a servable collection, so
DocumentScan/ShapeSnapshotreporteddoc_count=0.Fix:
crdt_materialize.rsmaterializes appliedDocUpsert/DocDeletedeltas into the sparse DOCUMENTS store via the native put path, so scans and shape snapshots observe them end-to-end.Coverage:
document_collection_registers_on_origin_via_announce(programmatic path) plus new SQL-DDL round-trip coverage — create → sync → Origin serves all rows via the exact entry point from the report.
Summary
A Lite replica can push document CRDT deltas to Origin and they are accepted (acked, server LSN advances, a CRDT checkpoint is persisted) — but the data never becomes servable: a fresh Lite shape-subscribing the same collection gets an initial
ShapeSnapshotwithdoc_count=0. So the lite->Origin push path works, but the Origin->lite serve path returns nothing. The round-trip (write on one replica -> read on another via Origin) does not complete.Repro
:9090.entrieswith sync enabled. Observe in the Lite log:pushed deltas to Origin,delta acknowledged rtt_ms~8,FtsIndexAck ... accepted=truefor every doc.ShapeSubscribe { ShapeType::Document { collection: "entries", predicate: [] } }.Observed:
ShapeSnapshot { doc_count: 0, snapshot_lsn: 3577, data: [1 byte] }— identical fortenant_id0, 1, and 2.Evidence the pushes were received
snapshot_lsn= 3577 (advanced from the pushes)./data/<inst>/crdt-ckpt/tenant-tenant:0.ckpt~= 20 MB, written right after the sync.Evidence the collection is not served
SHOW COLLECTIONSon Origin -> 0 rows.SELECT ... FROM entries-> "table not found".ShapeSnapshot.doc_count= 0 in all tenants.Is this expected / known?
The repo's own interop test acknowledges the serve side is stubbed —
nodedb-lite/tests/sync_interop_shape.rs:137:Two questions:
Scope note
The CRDT protocol itself is solid — handshake, real Loro delta push, ack, reconnect, and the
sync_interop_livesuite all pass against a real Origin. This is specifically about the serve/materialize half of lite<->Origin document sync.