fix(stats): reconcile novelty asserts against the base index for user-facing counts - #1699

Partially addresses #1391 — see What's left at the bottom for exactly what still drifts and why. (Started life as Fixes; the residuals below are real enough that closing the issue on this would be overclaiming.)

The mechanism

assemble_fast_stats_inner (fluree-db-novelty/src/runtime_stats.rs) folds novelty onto the indexed counts as a blind delta log — let delta = if flake.op { 1 } else { -1 }; — with no idea what the base index already holds. Novelty's own set-semantics dedup (NoveltyFactState) is window-scoped by construction, so once a fact has been reindexed into the base and dropped from the window, asserting it again is kept as a fresh novelty flake, and that flake is charged a second +1 on top of the base count it duplicates. The mirror case has the same root and nobody had noticed it: apply_commit's gate is if flake.op && …, which short-circuits, so a retraction is never examined and never dropped — a DELETE that matched nothing charged -1 against a count that never included it.

Consequence: any count rendered from those stats over- or under-reports until the next reindex. The issue framed this as planner-only imprecision, and for the planner it is — but two surfaces put those numbers in front of people. apoc.meta.data is what LangChain-style tooling reads to learn a graph's schema, and fluree info prints class/property counts as if they were facts about the ledger. On the fixture in the new test, one re-inserted document makes apoc.meta.data report ex:Widget ex:name = 6 where a scan of exactly those facts returns 5, and fluree info report ex:Widget instance count 6 where a scan returns 4.

Worth flagging for anyone reading the issue's "self-corrects at the next reindex" line: the sweep that re-verified #1391 found that in the pure-duplicate case the per-class stats are wiped on the incremental reindex rather than settling back to truth. Separate mechanism, not fixed here.

The fix

New NoveltyDeltaResolver in runtime_stats.rs. Drive it over a novelty walk in IndexType::Post order and it hands back each flake's current-state delta instead of flake.op's sign.

It folds presence, not ops. Per fact identity it carries the op that has won lifecycle resolution so far — the same rule NoveltyFactState::record applies, highest t wins and at equal t a retract beats an assert — and charges each flake the change in presence it causes. The first flake of an identity costs one base-index probe and charges op − base_present. Those deltas telescope, so their sum over an identity is exactly final_present − base_present however many flakes the window holds and in whatever order they arrive.

It deliberately does not assume kept flakes alternate assert/retract, because they don't. apply_commit's gate short-circuits on flake.op &&, so it suppresses redundant asserts only; and cmp_post sorts op ascending at equal t while fact_state resolves a same-t pair retract-wins, so the losing assert reaches the fold last. Both (retract, retract) and (retract, assert, assert) are reachable — the second deliberately, since bulk_apply_commits replays raw persisted flakes on cold load, which is exactly why the crate keeps same_t_assert_retract_keeps_later_reassert around to pin that state as legal. All three shapes are regression tests ([0, -1], [0, 1, 1], [-1, 1, 1]), along with two same-t pairs back to back, where a fold that compared against the previous op rather than resolving presence credits a trailing assert that lost.

The probe is one bounded SPOT scan with s and p both bound, cached per (graph, subject, predicate) for the assembly, read against an empty overlay at the indexed t — the question is strictly about the base, not about the merged view. It is skipped for commit-metadata flakes, whose subject is the commit's own CID digest: no base fact can share the identity, and include_in_runtime_stats discards them from every per-class and per-property count anyway. That was 7–10 guaranteed cache misses per commit in the window, buying nothing.

That t has to be the same one novelty was flushed to, or a partial flush leaves a retraction in the window whose matching assert has already moved into a base this probe reads as of an earlier t — charging 0 where the truth is -1. It is, by construction: both production callers of Novelty::clear_up_to (fluree-db-ledger/src/lib.rs) pass new_snapshot.t, so the cutoff and the snapshot installed alongside it are the same value, and both refuse a snapshot whose t is below the current index t. Recorded on the field rather than left to be re-derived.

base_contains binary-searches the cached facts, which scan_base sorts on (o, dt). o and dt compare with Ord because that is the ordering the search needs and the one the index comparators and NoveltyFactState's keys already use — not because Ord is stricter. It isn't: FlakeValue's cmp and its eq both route cross-representation numerics through numeric_cmp, so Long(3) and Double(3.0) compare Equal under either. What keeps them from being one identity is the dt tiebreak alongside o. m is matched with == rather than cmp, since it isn't part of the sort key — it filters the block the search lands on.

Deliberately the same shape as the indexer-side correction from 9d4026b32 (StatsIdHook::on_record_with_base_presence vs on_record): derive the count delta from the materialized state transition, not the raw op.

NoveltyMerge::Reconciled { site } is wired on at the four surfaces that render counts to users — ledger_info.rs (both the realtime_property_details arm and the fast one), cypher_procedures.rs merged_stats (the four db.* catalog shims), and cypher_procedures.rs meta_data_rows, which had its own inline if flake.op { 1 } else { -1 } and never went through merged_stats at all. site is required rather than optional: a stamp label shared across entry points is useless as a routing assertion, because a must-fire check on it is satisfied by any reconciling assembly rather than by the one under test. The labels live as named constants in stats_merge_site, so adding a reconciling surface is a visible edit there.

assemble_fast_stats / assemble_full_stats keep their signatures and their existing (estimate) behavior; reconciliation is opt-in via the new *_with variants.

Presence is not attribution

A base-presence probe answers "is this fact in the base index?". Per-class attribution needs a different question — "did the base rollup count this fact under this class?" — and the two diverge on exactly one shape: a subject whose class membership is new in the window.

Take a subject imported untyped and then re-upserted whole with an @type. Its name is base-present, so the fold charges it zero; but the base rollup filed that name under the classes the subject held at index time, and the new class was not one of them. The restatement is the only contribution that row could ever have, so charging it zero doesn't make the number low — apoc.meta.data drops zero-count datatypes, so the row disappears. That is #1391's own shape, inverted, on the surfaces this change exists to fix.

delta > 0 on an rdf:type flake is the signal, and it costs nothing extra: the resolver has already probed that flake. A restatement is charged +1 under the classes its subject gained in this window and nothing under the ones it already had, so the class the base rollup already counted it under is not charged a second time. reconcile_credits_only_the_gained_class_not_the_ones_base_already_filed is the pin for that half — an indexed ex:Employee that also becomes a ex:Person must not have its name counted twice.

Exactly one fold may make up the difference or the breakdown doubles instead, so assemble_fast_stats_inner now takes a RestatedAttribution saying whether a class-attributing second pass follows. The single-pass arm does the attribution from its intra-pass rdf:type side table; the two-pass assembly defers to the second pass, which resolves classes through lookup_subject_classes and so doesn't depend on where rdf:type happens to land in POST order. meta_data_rows reaches the same answer from its own two passes — pass A drives the resolver to learn which memberships are new, pass B charges a restatement under those.

None of this is reachable from the estimate lane: nothing resolves to zero there, so the gained-class table isn't even built.

The measurement

#1391 asked for this specifically, so: worst case by construction — every novelty flake a duplicate on its own (s, p), so the probe cache misses on every one — release build, local file-backed ledger.

novelty= 10,008 flakes   estimate 1.6ms   reconciled  42.0ms   (26.4x)
novelty= 48,008 flakes   estimate 8.1ms   reconciled 204.2ms   (25.1x)

~4.3µs per novelty flake, ~205ms at the ceiling MAX_RECONCILED_NOVELTY_FLAKES = 50_000 permits. Reproducible: cargo test --release -p fluree-db-api --features native --test it_fast_stats_1391_regression -- --ignored --nocapture.

Two things that measurement surfaced. The cap works and is visible — a 50,008-flake window declines and the two lanes converge to the same time and the same (wrong) count. And this is a per-request metadata cost that needs a deliberately raised reindex threshold to reach at all: the default is low enough that server_defaults.rs describes it as reindexing roughly every commit. Real windows also share (s, p) across values and carry new subjects, whose dictionary lookup misses cheaply, so they land well under the worst case.

One thing worth knowing that this doesn't fix: RangeProvider::range is synchronous and called from an async path without spawn_blocking, so that 205ms is blocking a runtime worker. It isn't new — ledger_info's existing assemble_full_stats already drives lookup_subject_classes through the same synchronous provider on the same path — but the cost is larger now than it was.

The differential test

fluree-db-api/tests/it_fast_stats_1391_regression.rs builds an indexed ledger, then a second commit mixing duplicates with real news (ex:w1 restated verbatim; ex:w2 restating type and name while adding one new name; ex:w3 entirely new; ex:w4 gaining a @type it never had while restating an already-indexed name; ex:g1 restating type and one partOf edge while adding another), then a third commit with two retractions — one real, one matching nothing. Every (class, property) count is compared against a scan of exactly those facts.

Four things keep it from passing for the wrong reason:

1. Every count is asserted against both a hand-pinned number and a scan, so a generic-pipeline regression fails as loudly as a fast-stats one.
2. Everything runs twice — fast paths on, then under set_fast_paths_disabled(true) — and the two must agree. That pins the issue's read-path claim (query answers were never affected; set semantics make the duplicate idempotent at read time) and would catch a future COUNT lane that started trusting the drifted stats.
3. Per-surface must-fire stamps. Each of the four reconciling sites must stamp reconciled on its own site label with at least the duplicate count the fixture plants (8 for the whole-window walks, 5 for apoc.meta.data, whose property pass counts rdf:type separately).
4. db.propertyKeys() must drop ex:size, whose every fact was retracted — that is merged_stats' one count-sensitive output, and under a blind merge the duplicate re-assert cancels the real retraction and keeps it listed.

Reverting all four Reconciled call sites produces 18 failures, and each site fails its own assertions when reverted alone (6 / 6 / 4 / 2, verified one at a time).

Plus 16 unit tests in runtime_stats.rs against a stub base index: duplicate assert → 0, new assert → +1, retract of an indexed fact → −1, retract of an absent fact → 0, the reassert-then-retract and retract-then-reassert folds, the three adversarial non-alternating runs, two consecutive same-t pairs, language tags and @list positions as distinct identities, the per-(s,p) probe cache, cross-graph resolution, the estimate lane reading zero leaflets, and the three gained-class attribution cases (gained class gets the restatement; a restated type is not a gained one; an existing class isn't charged twice).

No CLI-level assertion, deliberately. The API pin covers fluree.ledger_info(&ledger_id).execute(), which is the exact call fluree-db-cli/src/commands/graph.rs makes, and the CLI adds only rendering, which this PR doesn't touch — and fluree-db-cli isn't built by this PR's CI jobs, so a test there wouldn't gate the change anyway. A CLI test that pins rendered output is worth having on its own merits, not as a condition of this fix.

Scope-outs

Deliberately not reconciled, each documented in place rather than left as a silent gap: stats_cache.rs (the planner — reconciling there would make accumulating a window quadratic in its own size, and no COUNT answer reads it); policy_builder.rs (uncached per governed query, and enforcement reads property identity rather than counts); fluree_db_novelty::stats::current_stats (its throwaway genesis snapshot carries no range provider, so it cannot reconcile; also attributes every flake to graph 0); fluree-db-indexer/src/stats/{class_property,id_hook}.rs (on_record is the replay-from-scratch lane where the raw op is the transition; class_property.rs has no on_record at all — it has a base-seeded from_prior driving blind ±1, i.e. the #1391 shape, and what saves it is that its only consumer has no caller in the workspace, which is now noted on the function itself so nobody wiring it up later inherits the bug silently).

One scope-out is worth arguing for rather than just listing. The property_ref_only trust gate. The hazard is real and I traced the chain: merge_property_datatypes drops any datatype whose merged count reaches zero, so a spurious -1 can drop a predicate's last literal tag, flip ref_only to true, and license folding FILTER(?x = ?y) into a term-equality join where SPARQL value equality was required. But the repair has to land in the estimate lane, which is the planner hot path, and the two available shapes are both worse than the disease inside this PR: gating on empty novelty disables the fold for the ~50% of queries that carry novelty (filter_fold's own docstring cites BSBM BI-2 at ~28s → ~0.03s), and keeping base-observed datatype tags changes what PropertyStatEntry.datatypes emits for every consumer that sums it (fast_count.rs:151, count_plan_exec.rs:2061, stats_view.rs:265, index_stats.rs:250). It's latent, pre-existing, and not demonstrated end to end. Changing a hot lane on an unproven mechanism is the error this change was written to avoid, so it's documented in place at stats_cache.rs and filed as #1721, with both candidate fix directions and a note that reconciling the planner lane is not one of them. Happy to be overruled.

The stats_cache.rs comment there is also corrected on two counts it previously got wrong: several COUNT lanes deliberately do not gate on fast_path_store (count_plan_exec.rs, count_rows.rs, both on allow_cursor_fast_path) — what makes them safe is that they read through an overlay-merging BinaryCursor, not that they decline; and "these numbers only order joins" is false, since StatsView::property_ref_only feeds filter_fold's node-only soundness guard. policy_builder's comment likewise now says that "never reads a count" isn't "count drift can't matter" — the assembler prunes zero-count entries before policy sees them.

What's left — why this is Partially addresses

Real residual miscounts after this change:

1. Above the 50k cap, the fixed surfaces revert to the blind delta log and drift exactly as before. The stamp records it (declined:novelty_too_large) but the number a user sees is still wrong.
2. The planner lane (stats_cache) and policy_builder still carry the drift by design, with the reasons written down. Defensible — fast-stats over-count: novelty asserts not reconciled against the base index #1391's own closing note says planning tolerates approximation — but it is the over-count the issue names.
3. stats.size is still indexed.size + novelty.size, charging a duplicate re-assert's bytes unconditionally. It's an estimate of storage rather than a fact count, but fluree info prints it.
4. LedgerState::current_stats() — public, documented as "the canonical way to get up-to-date statistics", still estimate-grade and structurally unable to reconcile. Two test callers in-repo today; fixing it properly means fixing its graph attribution too.
5. class_property.rs retains the shape, unreachable today.
6. assemble_full_stats attributes a property twice when its subject's rdf:type lands in the same novelty window: once from the first pass's intra-pass graph_subject_classes side table, once from the second pass's lookup_subject_classes + subject_props. One fact reports a class-property count of 2 under both merge modes, so this drifts a reconciled surface — fluree info's class→property block on the default arm. Pre-existing and untouched here; closing it means deciding which of the two passes owns class attribution, which is a larger change than reconciliation and wants its own PR. Named at the site.
7. A class gained in the window attracts only the facts the window carries. The gained-class rule above can credit a restatement, because a restatement is a flake in the window. A base fact that isn't restated has no flake to attribute, so a subject that gains an @type and nothing else picks up an instance count but none of its indexed properties. Pre-existing under both merge modes — the estimate lane has the same gap, for the same reason — and not made worse here, but adjacent enough to the fix that it should be written down rather than discovered later.

Closing #1391 on this would mean claiming counts no longer drift, and (1) alone makes that untrue. The issue should stay open against those residuals.

Gates

• cargo test -p fluree-db-novelty — 90 green (22 in runtime_stats, incl. 16 new).
• cargo test -p fluree-db-api --test it_fast_stats_1391_regression — green; 18 failures with all four Reconciled sites reverted, and 6 / 6 / 4 / 2 when reverted one at a time.
• cargo test -p fluree-db-api --features native — all binaries green (grp_query 422, grp_ledger 145, grp_index 89, grp_misc 262, grp_policy 96, grp_transact 188, grp_import 99, grp_query_sparql 349, lib 775, and the rest).
• cargo test -p fluree-db-query --lib 1421 green; cargo test -p fluree-db-indexer --lib 380 green.
• cargo clippy -p fluree-db-novelty -p fluree-db-api --features fluree-db-api/native --all-targets --no-deps — clean.
• cargo fmt --all — clean, run last.