Skip to content

refactor: consolidate staging and commit helpers - #1929

Merged
bplatz merged 5 commits into
mainfrom
refactor/staging-helper-dedup
Sep 26, 2026
Merged

bplatz merged 5 commits into
mainfrom
refactor/staging-helper-dedup

Conversation

@bplatz

@bplatz bplatz commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR follows the staging and graph-routing work in #1926. It moves the write path's common steps into shared helpers, so every entry point runs the same staging, checks and commit logic. Apart from the no-op change under "Behavior", the effect of each write is unchanged.

Staging

  • stage_and_check runs staging and the post-staging checks for the API's write entry points. It covers SHACL or plain staging, novelty-backpressure handling, uniqueness, and reasoning modes. Each entry point keeps its own authoring context and error mapping.
  • stage_options builds stage options from the index config, the policy, and an enabled tracker.

Commit

  • StageResult::is_noop is the rule for skipping a commit. It applies to an update, upsert or sync that staged no flakes and registers no new graph. A registration-only stage such as SPARQL CREATE GRAPH still commits.
  • CommitReceipt::no_op builds the receipt for a skipped commit.
  • commit_stage_result is the stage-to-commit step shared by the transact paths and the transaction builder.
  • GraphRegistry::has_unregistered answers whether a commit would register a graph.

SHACL and uniqueness

  • StagedLedger::staged_flakes_by_graph exposes the ledger graph id staging assigned to each flake. SHACL validation and uniqueness enforcement read it directly, so they always use the same routing as staging. The SHACL context therefore drops its separate graph-to-Sid input.
  • namespace_prefix_map and CrossLedgerShapesModel::membership build the cross-ledger sh:class membership handle for transaction staging, branch-operation validation, and validate.

Removed

  • stage_with_shacl, which the API's SHACL-aware staging superseded, along with the report formatter only it used.

Behavior

The tracked policy path now uses the shared no-op rule. A policy-carrying update whose WHERE matches nothing succeeds without writing a commit, as it already did on the other write paths. Before, it returned a 500 "Empty transaction".

  • tracked_update_matching_nothing_is_a_noop covers this, and it fails without the change.
  • Two policy tests had checked that error to confirm a view-filtered update wrote nothing. They now expect the no-op and assert that the ledger did not advance, so they fail if the tracked path regresses.

The shared rule also changes the Fluree::transact() paths, which never had the registration clause. A no-match update whose INSERT template names a graph the ledger has not registered now commits that registration, with zero flakes, and advances t. Before, it skipped the commit. The transaction builder already behaved this way; the two entry points now agree. no_match_update_naming_a_new_graph_commits_its_registration pins it and fails if the registration clause is removed.

Build

stage_and_check boxes its staging future. Every write entry point awaits it, and after rebasing onto main, two test targets whose async helpers nest a few layers deep failed to compile with "queries overflow the depth limit". Boxing that one future bounds the depth for every caller.

@bplatz
bplatz added this pull request to stack #1922 September 25, 2026 09:55
@bplatz bplatz added the area:transact Staging, commit, import/bulk-import, novelty, retraction semantics label Sep 25, 2026
@bplatz bplatz changed the title refactor/staging helper dedup refactor: consolidate staging and commit helpers Sep 25, 2026
@bplatz
bplatz requested review from aaj3f and zonotope September 25, 2026 09:55
@bplatz
bplatz marked this pull request as ready for review September 25, 2026 09:55
@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch from 57db27d to 6bb1f7a Compare September 25, 2026 10:24
@bplatz
bplatz removed this pull request from stack #1922 September 25, 2026 18:27
@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch from 6bb1f7a to 24743ea Compare September 25, 2026 19:15
@bplatz
bplatz added this pull request to stack #1947 September 25, 2026 19:15

@aaj3f aaj3f left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bplatz I love abstraction-cleanup or dedup refactors like this one. On the whole, no notes, though there are some small findings Claude made below in its review assistance that are worth some consideration. Full review below:


The diff here makes the PR's own case sound: seven copies of the no-op rule that had already diverged (the three in tx.rs were missing the registration guard the three in tx_builder.rs carried, and the tracked path had no rule at all), plus three places independently re-deriving which graph a staged flake landed in. Naming each of those once is exactly right, and StagedLedger::staged_flakes_by_graph() is the pick of the bunch — it doesn't just deduplicate, it deletes a silent fallback where a named-graph flake whose IRI didn't resolve back through the staged namespace registry got validated and uniqueness-checked against the default graph with nothing to say so. I treated "behaviour preserved" as a claim to break rather than a claim to accept, and it held everywhere I pushed: the commit_transfer routing is byte-identical to what it replaced, StageOptions::new() really is Self::default(), the tracker is_enabled() gating survived at all three sites that had it, and stage_with_shacl / format_shacl_report have zero remaining references anywhere including docs.

The one thing I'd like before this merges is a note in the "Behavior" section, because there are two changes there and the body describes one. Alongside the documented tracked-path no-op fix, folding registers_new_graph into is_noop also adds that guard to the three tx.rs commit paths that never had it — so on Fluree::transact(), a zero-flake update naming a not-yet-registered graph now writes an empty registration commit and bumps t where before it didn't. I verified it (probe and exact repro in the inline note on tx.rs:2122): t=1 → t=2 at HEAD, t=1 → t=1 with the has_unregistered clause neutralised. I think it's the right direction and it's what the builder already did, so this is "document it and pin it", not "change it" — but nothing in the suite distinguishes HEAD from base on that path right now, which is a gap worth closing in the PR that opens it.

Two smaller ones, both in the inline notes: it_policy_verbs.rs:855's relaxed Err(_) | Ok(_) match passes under both the old and the new behaviour (I confirmed it stays green under the mutation that reverts the no-op rule, while the other three touched tests go red), so it isn't pinning anything any more; and tx.rs:3192's hardcoded sync_graph: None was inert before this PR and is load-bearing now that is_noop reads it — worth a derivation or a comment, especially with #1932 stacking stage_sync_transaction_tracked on top.

Adherence to repo commitments:

  • Patterns/abstractions: ✔ Extends the existing mechanisms rather than adding parallel ones — StageResult, CommitReceipt, GraphRegistry and StagedLedger each grew the method that belonged on them, and the graph_sids parameter drop genuinely narrows two contracts instead of just moving them.
  • Performance (speed first, memory second): ✔ Net improvement, no regression risk. Both per-flake consumers (enforce_unique_constraints, validate_staged_nodes) swap a HashMap<Sid, GraphId> lookup for a Vec<u16> index, and two per-write map builds disappear — graph_ids_by_sid, which scaled with the whole graph registry rather than with the transaction, and build_reverse_graph_lookup. No new allocations, clones or locks anywhere in the diff. No bench covers staging or uniqueness, so the win won't surface in the numbers; that's pre-existing, not yours.
  • Testing: ⚠️ The routing consolidation is genuinely pinned — collapsing staged_flakes_by_graph() to g_id = 0 turns it_config_graph::shacl_rejects_violation_in_any_named_graph_{sparql,jsonld}, shacl_per_graph_mode_applies_to_graph_variable_targets and both it_constraints_inline::inline_unique_property_enforced_in_second_named_graph_* red, so both the SHACL and uniqueness sides are covered. The new tracked_update_matching_nothing_is_a_noop is wired into grp_transact and goes red under mutation. The ⚠️ is only for the two gaps above: the registration change is uncovered, and it_policy_verbs.rs:855 no longer discriminates.
  • Conventions: ✔ Self-describing subjects, substantial multi-line bodies that carry the rationale (the commit bodies are better than the PR body here — the registration point is implicit in commit 2's first paragraph). Clippy and fmt green at this head. W3C registers untouched.

Verified locally at 6bb1f7a15: cargo nextest run -p fluree-db-api --features shacl --test grp_transact --test grp_policy (4/4 pass, new test present by name); three mutation runs (no-op rule → 3 of 4 red; graph routing → 5 red across it_config_graph + it_constraints_inline; registration guard neutralised → the probe flips); all four StagedShaclContext sites and every commit_stage_result call site traced against the inline rule it replaced. CI at this exact head (run 36123814997, event=pull_request): test, clippy, fmt, testsuite-sparql, wasm32 all green. --all-features wouldn't build locally (usearch → cxx can't find <algorithm> on my machine) — unrelated to this PR, and CI's --all-features run covers it.

Approving so you can merge when ready — but I'd really like the registration change documented and pinned before it goes, since this is the PR that introduces it.

Comment thread fluree-db-api/src/tx.rs
.base()
.snapshot
.graph_registry
.has_unregistered(self.graph_delta.values().map(String::as_str))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional, but I'd want it in this PR — there's a second behaviour change here that the body doesn't call out.

Folding registers_new_graph into the shared is_noop doesn't just move that guard — it adds it to the three tx.rs commit paths (execute_transaction, transact_with_trig_meta, transact_with_named_graphs) that never had it. Their old inline rule was !view.has_staged() && (Update|Upsert || sync_graph.is_some()), with no registration clause at all; only the three tx_builder.rs copies carried one.

The effect: on Fluree::transact(), a zero-flake update whose graph_delta names a graph the registry doesn't know now writes an empty registration commit and advances t, where before it returned without one. I verified rather than inferred it — a temporary probe against a ledger seeded at t=1:

let update = json!({
    "@context": {"ex": "http://example.org/"},
    "where":  {"@id": "?s", "ex:absent": "?o"},          // matches nothing
    "insert": {"@id": "?s", "@graph": "http://example.org/graphs/brandnew", "ex:q": "y"}
});
fluree.update_with_opts(ledger, &update, TxnOpts::default(), Default::default(), &index_cfg).await
// HEAD:                              receipt.t == 2, flake_count == 0   ← empty registration commit
// `.has_unregistered(std::iter::empty())`:  receipt.t == 1, flake_count == 0   ← the base's rule on this path

I actually think this is the right direction — it's what the builder already did, and a transact() that disagrees with stage(&handle) about whether a graph got registered is exactly the drift this PR exists to kill. But the rule is broader than the intent the old comment stated ("CREATE GRAPH: zero staged flakes, but a graph_delta IRI the registry doesn't know yet"), and a no-match update that merely mentions a graph in an INSERT template isn't a CREATE GRAPH.

So two things would settle it: a line in the "Behavior" section saying this path changed too, and a test — nothing in the suite distinguishes HEAD from base here today, which is why I had to write one. If instead you'd rather the rule stay narrow, the distinguishing signal is presumably txn.graph_mgmt / the SPARQL CREATE GRAPH lowering rather than "any unregistered IRI in the delta".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. I kept the broad rule: the point of this PR is that transact() and the builder stop disagreeing, and the builder's rule was the one already in use. Narrowing it to CREATE GRAPH raises a wider question: should a graph a template names but writes nothing to be registered at all? Commits with other flakes do that today on every path, so that belongs in its own decision, not the no-op case alone.

Documented in the Behavior section and pinned in 8321b52 by no_match_update_naming_a_new_graph_commits_its_registration, which is your probe: t advances, zero flakes, graph registered. It fails with has_unregistered neutralised.

);
// The profile cannot see a Person, so the WHERE matches nothing: the
// update is refused or is a no-op, and either way writes nothing.
let before_t = ledger.t();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional — this test now passes whichever behaviour is in effect, so it isn't a net for the change any more.

Under the mutation that reverts the no-op rule (TxnType::Update | TxnType::Upsert → TxnType::Upsert), the other three tests this PR touches go red and this one stays green — the Err(_) => {} arm absorbs the old 500.

I get why it was relaxed: what the test cares about is "nothing was written", and asserting after.t() == before_t says that far more directly than asserting an error string did. So this isn't wrong, it's just no longer load-bearing. Since tracked_update_matching_nothing_is_a_noop covers the no-op rule directly, maybe the cleanest thing is to drop the Err(_) arm and let this test assert the outcome it now expects:

// The profile cannot see a Person, so the WHERE matches nothing: the
// update is a no-op that writes nothing and does not advance the ledger.
let before_t = ledger.t();
let after = try_txn(&fluree, ledger, TxnType::Update, &update_person, &ctx)
    .await
    .expect("a view-filtered no-match update is a no-op, not an error");
assert_eq!(
    after.t(),
    before_t,
    "Lead write profile must NOT edit a Person"
);

Minor and non-blocking — but if you agree it's right, I'd rather see it folded in now than lost in the backlog.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8321b52, using your version: it expects the no-op and asserts t did not advance. With the no-op rule reverted it now goes red (it stayed green before).

Comment thread fluree-db-api/src/tx.rs Outdated
txn_meta,
graph_delta,
graph_delta: staged.graph_delta,
sync_graph: None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More of a question than a suggestion — sync_graph: None is hardcoded here while every sibling StageResult derives it from txn.sync_graph.

It's accurate today — this entry point parses through parse_transaction, and only parse_sync_transaction ever sets sync_graph — but it was inert before this PR and isn't any more: is_noop reads that field, so if a sync ever reaches this door it silently loses the no-op and lands back on the 500 this PR just fixed.

Given #1932 is adding stage_sync_transaction_tracked immediately on top of this branch, it seems worth either deriving it (let sync_graph = txn.sync_graph.clone(); alongside txn_meta, one line up) or leaving a comment saying why None is structurally guaranteed here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Derived it in 8321b52: let sync_graph = txn.sync_graph.clone(); alongside txn_meta, so a sync reaching this entry point keeps its no-op.

}

/// Each staged flake with the ledger graph id staging routed it to.
pub fn staged_flakes_by_graph(&self) -> impl Iterator<Item = (GraphId, &Flake)> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Praise — reading the routing from flake_graph_ids instead of re-deriving it is the best decision in the PR, and it deletes a silent failure mode.

The old flake_graph_id's unwrap_or(0) meant a named-graph flake whose IRI didn't resolve through staged_ns.lookup_sid_for_iri got validated and uniqueness-checked against the default graph, with nothing to announce it. StagedOverlay::from_flakes is strict about the same question, so reading the answer from there can't fail quietly.

A coverage note while we're on this routing, because it's more useful to you than to me. I mutation-checked it — collapsed staged_flakes_by_graph() to g_id = 0, which is the old unwrap_or(0) fallback firing on every flake — and it is pinned: five tests go red, in it_config_graph (shacl_rejects_violation_in_any_named_graph_{sparql,jsonld} and shacl_per_graph_mode_applies_to_graph_variable_targets) and it_constraints_inline (both inline_unique_property_enforced_in_second_named_graph_*). So SHACL and uniqueness are both genuinely covered. What surprised me is where it isn't: that same mutation left all 86 tests green across it_sync_graph, it_branch_shacl, it_policy_named_graphs, it_enforce_unique*, it_named_graphs and it_named_graph_isolation — i.e. every suite whose name suggests it would catch a graph-routing regression. Nothing for this PR to fix, but worth knowing that the net under this code is narrower, and sits somewhere other than where you'd reach for it.

The one thing I'd flag for a future reader: zip truncates to the shorter side, so if flake_graph_ids ever fell out of lock-step with store.flakes the consequence is silently dropped flakes from SHACL and uniqueness, not a panic. It's correct today — both are built in the one pass at staged.rs:83-97 and neither is mutated after — and honestly a debug_assert_eq! on the two lengths is probably over-engineering for a private field with one writer. Raising it only because the failure shape is "validation quietly stops seeing some of your data", which is the worst one to discover late.

Comment thread fluree-db-api/src/tx.rs
Ok((templates, graph_iris))
}

/// Stage options for a write: backpressure against `index_config`, modify

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Praise — stage_and_check keeping txn_context and the error mapping at the caller is the right seam.

Those two really are the only things that legitimately differ per entry point, and the tracked path at tx.rs:3178 deliberately not going through stage_options (because it attaches its tracker unconditionally rather than gating on is_enabled()) is the kind of detail a consolidation like this usually flattens by accident. Nice.

@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch 2 times, most recently from ea1b176 to fc6574f Compare September 25, 2026 21:07
@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch 2 times, most recently from 8c0cf99 to 7d4b313 Compare September 25, 2026 21:31
@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch 4 times, most recently from f1615b7 to 98c1548 Compare September 25, 2026 23:31
@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch 2 times, most recently from 090daaf to fcdfb20 Compare September 25, 2026 23:52
Base automatically changed from fix/sparql-protocol-dataset-params to main September 26, 2026 00:45
The API's write entry points each assemble stage options, a per-transaction
ResolveCtx, SHACL or plain staging, novelty-backpressure handling, and the
uniqueness and reasoning-mode checks. `stage_and_check` now carries that
sequence and `stage_options` builds the options, so every entry point shares
one implementation while keeping its own authoring context and error mapping.

Also retires `stage_with_shacl` and the report formatter it used, which the
API's staging path superseded, and adds `GraphRegistry::has_unregistered`
for the commit and the transaction builder.
…w graph routing

Commit tail. `StageResult::is_noop`, `CommitReceipt::no_op` and
`commit_stage_result` give every write path the same stage-to-commit step,
including the rule for skipping a commit when a stage changes nothing. The
tracked policy path now follows that rule too: a policy-carrying update whose
WHERE matches nothing succeeds without a commit, as it does elsewhere, where
it previously surfaced a 500 "Empty transaction". Two policy tests now assert
directly that nothing was written.

Cross-ledger SHACL membership. `namespace_prefix_map` and
`CrossLedgerShapesModel::membership` build the membership handle for
transaction staging, branch-operation validation and `validate`.

Graph routing. `StagedLedger::staged_flakes_by_graph` exposes the ledger
graph id staging assigned to each flake. SHACL validation and uniqueness
enforcement read it directly, so the SHACL context no longer takes a
separate graph -> Sid map.
@bplatz
bplatz force-pushed the refactor/staging-helper-dedup branch from fcdfb20 to 819b3a5 Compare September 26, 2026 00:45
Folding the registration guard into StageResult::is_noop also changed the
Fluree::transact() paths, which never had it: a no-match update whose
INSERT template names an unregistered graph now commits the registration,
with zero flakes, as the builder already did. A new test pins that.

The policy test that tolerated either an error or a no-op now expects the
no-op, so it fails if the tracked path regresses to the old 500.

The tracked policy path also takes sync_graph from the Txn instead of
hardcoding None, since is_noop now reads it.
After the rebase onto main, clippy and the test build failed with "queries
overflow the depth limit" in it_absent_subject_scan_narrowing and
it_exists_semijoin_correlation. Every write entry point now awaits
stage_and_check, and the staging future inlined into each of them made the
future types of callers a few async layers deep exceed rustc's layout
depth limit. Boxing that one future bounds the depth for every caller.
@bplatz

bplatz commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author

@aaj3f thanks. All three are in 8321b52 (replies on each thread): the transact() registration change is documented in Behavior and pinned by a test, it_policy_verbs asserts the no-op again, and the tracked path derives sync_graph.

One more thing turned up. After the stack was rebased onto main, CI failed to compile two newer test targets (it_absent_subject_scan_narrowing, it_exists_semijoin_correlation) with "queries overflow the depth limit". stage_and_check added a layer to the future every write entry point awaits. 68ed2c7 boxes the staging future inside it, which clears it for every caller. The branch also has the latest main merged in, including #1919.

@bplatz
bplatz merged commit 204fa25 into main Sep 26, 2026
16 checks passed
@bplatz
bplatz deleted the refactor/staging-helper-dedup branch September 26, 2026 01:06
@aaj3f aaj3f mentioned this pull request Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:transact Staging, commit, import/bulk-import, novelty, retraction semantics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants