Origin sync: interleaved KV deferred-flush delta silently halts document materialization (rejected=0, silent_dropped=0)

Version / build tested against

origin/main @ 81169d3c7 (release build, SELECT version() → NodeDB 0.4.0, wire format 1). Originally observed on the 38bfc3084 pre-release build; re-verified on current main 2026-07-22.

Deployment mode

Origin — single node (local), with a NodeDB-Lite embedded client syncing to it.

Engine(s) involved

Document (schemaless), Key-Value — the interaction between the two on one sync stream is the trigger.

Summary

When a lite client's sync stream interleaves KV deferred-flush deltas (kv_put + kv_flush) with valid per-document CRDT deltas, Origin silently stops materializing the document deltas after the first one — while session counters report mutations=N rejected=0 silent_dropped=0. Documents are acked but never become visible to SELECT/ShapeSnapshot. Silent-correctness class (same family as #202/#156/#170/#171): all counters green, rows missing.

Steps to reproduce

Controlled A/B, one variable, from a fresh Origin data dir (trust mode, single_node_calvin = false to keep single-node raft noise out; reproduces with defaults too). Client: nodedb-lite main @ c913b6b built against the 0.4 workspace crates.

-- both variants: on lite —
--   execute_sql: CREATE COLLECTION probe WITH (bitemporal=true)
--   start sync (SyncClient + run_sync_loop), wait Connected
--   3× document_put with distinct ids
--   drain 15s, then on Origin over pgwire:

SELECT count(*) FROM probe;

Variant A (documents only): each document_put emits one per-document delta.

Variant B (documents + per-write KV bookkeeping — after each document_put):

-- lite client, after each document_put:
--   kv_put("signals", <entry_id>, <bytes>)
--   kv_flush()          -- emits a coalesced deferred-flush delta

Expected result

Variant A: 3 rows. Variant B: 3 rows — the KV deltas should either be rejected loudly (counted in rejected, DLQ'd) without derailing subsequent valid document deltas, or never enter the document sync stream at all.

Actual result

Variant A: count(*) = 3 ✅ (re-verified on origin/main @ 81169d3c7)
Variant B: count(*) = 1 ❌ — only the FIRST document materializes (re-verified on origin/main @ 81169d3c7, 2026-07-22)
At N=50 with real workload code: session closed mutations=221 rejected=0 silent_dropped=0, exactly 1 row materialized.

Logs / evidence

Session close line (Origin, info level — nothing else logged about the loss):

sync: session closed session=sync-127.0.0.1:59532-1 mutations=221 rejected=0 silent_dropped=0

Mechanism pointers

Lite's KV deferred path flushes as a coalesced multi-row delta (engine/crdt/engine.rs flush_deltas → collection="deferred", document_id="{count}_ops") — violates the one-document-per-delta contract.
Origin's contract enforcement (nodedb/src/data/executor/handlers/control/crdt_apply.rs:50 single_document_write_set; reject constraint crdt_single_document_delta at :174 — unchanged between 38bfc3084 and current main) is loud in the direct path, but in this scenario rejected stays 0 and materialization of subsequent valid deltas halts instead.

Notes

Self-contained repro tests (~60 lines per variant) exist and can be PR'd into the lite interop suite on request.

Triage proposal (no label rights): type:bug sev:2-high area:crdt-sync engine:document — silent data loss on the sync stream; client-side workaround exists (keep KV bookkeeping off the synced stream), which is what keeps it out of sev-1 for us. A/B repro tests are self-contained and ready to PR into the interop suite on request.

Re-verified on origin/main @ 81169d3c7 (2026-07-22): still reproduces — variant A (documents only) materializes 3/3, variant B (documents + per-write kv_put+kv_flush) materializes 1/3 with rejected=0 silent_dropped=0. crdt_apply.rs is unchanged between the pre-release build and current main. Body restructured to the bug_report.yml template shape with the main version pin.

Root cause found and fixed. The controlled A/B repro was exactly right and made this straightforward to trace — thank you.

What was happening

Lite keeps one Loro document for the whole database and exports each delta as an incremental slice of that shared oplog. Origin keeps one document per collection. So a delta labelled probe, whose causal predecessors were routed into a different collection's document (here the deferred KV flush), arrives at Origin without its history.

Loro accepts such an update and buffers the operations as causally pending: the import call returns success, but the applied state never advances. Every authoritative path discarded that signal:

CrdtState::import_with_limits threw away Loro's ImportStatus, including .pending
write_set_since then diffed an unchanged frontier and returned an empty write-set
the apply reported Clean { write_set: [] }, which the one-document-per-delta guard accepts as a legal no-op (a delete writes no rows)

Net effect: high-water-mark advanced, AckStatus::Applied returned, row never materialized. Notably the read-only preview path already enforced this invariant (PreviewPendingDependencies) — the durable write path did not.

The all-green counters had a second, independent cause. mutations_rejected was only incremented during the session's envelope validation. Refusals decided downstream — including the KV delta's genuine rejection — never touched a session counter, so the close line reported rejected=0 regardless of what actually happened.

What changed

nodedb@1d4a0a8fc, nodedb-lite@e878e78:

A pending import is now a typed error (ImportPendingDependencies) rather than silent success, enforced at the single chokepoint every path funnels through.
New ValidatedApplyOutcome::PendingDependencies, kept distinct from Malformed — malformed bytes can never apply and are safely skipped; these are well-formed operations that simply arrived without their history.
A retryable refusal no longer advances the high-water-mark. Advancing it would turn the client's re-push into a Duplicate and lose the write permanently. This also corrected the pre-existing ConstraintVersionPending path, which had the same defect.
The same guard now covers the Raft-committed apply, snapshot restore, transaction rollback, and WAL replay. All four previously reported success on a partial apply; on the post-consensus path that meant a replica could silently diverge from a committed log entry.
Session accounting reports admitted / applied / rejected / not_applied, counted where the outcome is actually decided.
New AckStatus::Accepted, so the session's provisional ack stops claiming Applied before the durable apply has run.
Lite no longer retires un-applied deltas on Gap / Fenced, and acknowledge now retires a single mutation instead of everything at or below it — one ack was discarding the entire backlog behind it.
Separately, each collection's document now derives a distinct Loro peer id. Previously every collection shared the node's peer id, so unrelated writes in different collections minted identical (peer, counter) operation ids, and any peer merging two collections into one document silently dropped one of the rows.

Regression coverage was added first, across both repos, including the interleaved-collection case from this report.

Current behaviour on your repro

Variant A — 3 rows, unchanged.

Variant B — still 1 row, but nothing is silent any more. The KV deferred-flush delta is rejected loudly and counted, and the subsequent document deltas are refused as retryable with the high-water-mark held, instead of being acked and dropped. Session counters now reflect reality.

So the silent-correctness half of this report is closed: no acknowledged write is lost, and no refusal is invisible. The halt itself is not yet fixed — getting Variant B to 3 rows requires the remaining structural piece, moving Lite to one CRDT document per collection so its deltas are self-contained. That removes the asymmetry at the source rather than detecting it after the fact, and is tracked separately.

The remaining halt is now tracked in #220 — Lite moving to one CRDT document per collection so its deltas are self-contained. Closing this one: the silent-loss behaviour it reports (acknowledged-but-never-applied writes, rejected=0 while rows go missing) is fixed and covered by regression tests. Please reopen if anything here still reproduces silently.

Navigation Menu

Repository navigation

Origin sync: interleaved KV deferred-flush delta silently halts document materialization (rejected=0, silent_dropped=0) #208

Version / build tested against

Deployment mode

Engine(s) involved

Summary

Steps to reproduce

Expected result

Actual result

Logs / evidence

Mechanism pointers

Notes

What was happening

What changed

Current behaviour on your repro

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Origin sync: interleaved KV deferred-flush delta silently halts document materialization (rejected=0, silent_dropped=0) #208

Description

Version / build tested against

Deployment mode

Engine(s) involved

Summary

Steps to reproduce

Expected result

Actual result

Logs / evidence

Mechanism pointers

Notes

Activity

emanzx commented on Jul 22, 2026

emanzx commented on Jul 22, 2026

farhan-syah commented on Jul 25, 2026

What was happening

What changed

Current behaviour on your repro

farhan-syah commented on Jul 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions