Deduplicate repeated statements in bulk import - #1754

Summary

Bulk import stored every input statement verbatim — no deduplication existed on either import path (the merge ran with skip_dedup: true), so duplicated input lines became duplicate index rows, violating RDF set semantics and silently inflating COUNT, aggregates, and join cardinalities (#1651). This PR makes the index the set view while the commit blobs keep the raw ops: duplicates are squashed where the data is already sorted, at no measurable cost to clean imports.

Closes #1651

Design

Two dedup lanes, both O(1) memory, both operating on the existing five-field identity (s_id, p_id, o_type, o_key, o_i) — so language tags (folded into o_type), datatypes, and list positions all keep distinct facts distinct:

• Chunk sort: an in-place dedup_by pass immediately after the per-chunk graph-prefixed SPOT sort collapses within-chunk copies before the sorted-commit artifact is written — upstream of the stats hooks and every downstream artifact. Guarded (same t, both asserts) so anything unexpected is kept rather than collapsed.
• K-way merges: a new KWayMerge::next_unique_assert() collapses cross-chunk copies adjacently in the SPOT and secondary-order merges. Earliest t wins (a fact is present from its first assertion; later copies are no-op re-asserts). Retracts pass through verbatim; cascade intermediates stay lossless; the rebuild arm (next_deduped_with_history) is untouched. The now-misleading skip_dedup flag is renamed import_unique_asserts.

Stats stay tallied in the parallel remap workers exactly where they are today (no pipeline restructuring): the chunk-lane dedup lands upstream of the hooks so within-chunk copies never get counted, and the SPOT merge tallies cross-chunk squashes into a lazily-allocated (g, p, o_type) map — never touched on clean data — applied as signed-delta discounts at stats finalize via the hook's existing delta machinery. HLL sketches are duplicate-insensitive and need no correction. Per-property counts and stats.flakes now describe the set view.

Counters: flake_count / cumulative_flakes / total_asserts / fuel keep describing raw commit ops (the commit chain deliberately retains every op). A new ImportResult.duplicates_removed reports the collapsed copies, and the CLI prints both when non-zero:

Imported 49.2M flakes in 33.77s (1.46 M flakes/s) across 9 commits (t=9)
Collapsed 18,938 duplicate statement(s); the index holds 49,156,066 distinct flakes

Latent language-remap ordering bug (fixed here; adjacency dedup depends on it)

Adjacent-identity dedup requires remapped chunk streams to stay sorted, which exposed a pre-existing hazard: the unified language dictionary was built in first-seen order across chunks while chunk-local ids were parse-order, so a local→global lang remap could be non-monotone — un-sorting a remapped stream within (s, p) (lang_id is the langString o_type payload, part of the sort key) so equal langString records never meet in the merge. Both sides now assign language ids lexically (normalized): the chunk sort reassigns local ids and rewrites records, build_lang_remap_from_vocabs builds the unified dict lexically, making every remap monotone. Cost: O(distinct tags).

Semantics preserved by construction

• Anonymous blank nodes and anonymous reifiers mint per-occurrence ids and intentionally never collapse (RDF-correct: two reification events of one fact stay two reifiers, annotating the single deduped triple).
• Labeled blank nodes are document-scoped, so duplicated labeled-bnode lines produce identical records and collapse.
• Edge-annotation (f:reifies*) bundles are separate records keyed by the reifier subject — base-triple dedup cannot drop annotation data.
• Vector literals are intentionally out of scope: the vector arena appends per occurrence, so copies carry distinct handles and are invisible to identity dedup. Status quo is preserved (nothing deduped them before either). Deduping them would require value interning in the arena; vectors are a Fluree extension rather than RDF set semantics, so this is a deliberate non-goal, not a deferred follow-up.

Benchmarks

DBLP N-Triples, first 50,000,000 lines of the 2026-06-01 dump (6.0 GB), fresh ledger per run, interleaved runs, Apple M4 Max 16c/48GB. Three binaries: A = released v4.1.6 (homebrew), C = origin/main unpatched, B = this branch (C + this patch, isolating its cost):

Speed and peak memory are identical within ±1% run noise across all three (B's mean is nominally lowest). Correctness reconciles to the digit: the prefix holds 49,175,004 triple statements (plus 824,996 blank lines) of which 49,156,066 are distinct; A/C store all 49,175,004, B collapses exactly 18,938 and stores exactly the distinct count, and the A−B row delta equals duplicates_removed.

Two findings from the ground-truthing worth recording:

• The issue's claim that the chunked path already dedups IRI-only lines was an artifact of blank-line counting in its input analysis — empirically, the pre-patch import dedups nothing (A and C store every duplicate).
• The published DBLP-core report's "561,544,658 distinct triples" is the duplicate-inflated count; true distinct is 561,477,456. A post-patch full-dump import should report duplicates_removed ≈ 67,202 and store the true count.

Testing

Every dedup lane is pinned by a test proven to fail with its fix reverted (mutation-tested):

• Chunk lane: spool unit test (sorted_commit_collapses_duplicate_records).
• Merge lane: merge unit tests (min-t winner, dropped counting, o_i near-miss, retract passthrough) + cross-chunk e2e — including the issue's 6-line repro asserting COUNT(*) = 3, and a chunked variant asserting per-property stats reflect the set view.
• Lang monotonicity: remap unit test + an adversarial e2e where chunk 2 introduces a lexically-smaller tag before a duplicated @en statement under one (s, p).

Additional coverage: near-miss e2e (same literal under @en / @de / plain / xsd:byte / integer all preserved, duplicates_removed == 0); one pre-existing overlap fixture updated whose seam records — identical-comparing by design — now correctly collapse (spot_hierarchical_matches_flat, CID flat/hierarchical parity assertion intact).

Green: indexer lib + integration (389+), grp_import (105), grp_misc (263), grp_transact (190); clippy clean on touched crates; cargo fmt --all --check clean.