Repository navigation
refactor: consolidate staging and commit helpers - #1929
Conversation
57db27d to
6bb1f7a
Compare
6bb1f7a to
24743ea
Compare
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz I love abstraction-cleanup or dedup refactors like this one. On the whole, no notes, though there are some small findings Claude made below in its review assistance that are worth some consideration. Full review below:
The diff here makes the PR's own case sound: seven copies of the no-op rule that had already diverged (the three in tx.rs were missing the registration guard the three in tx_builder.rs carried, and the tracked path had no rule at all), plus three places independently re-deriving which graph a staged flake landed in. Naming each of those once is exactly right, and StagedLedger::staged_flakes_by_graph() is the pick of the bunch — it doesn't just deduplicate, it deletes a silent fallback where a named-graph flake whose IRI didn't resolve back through the staged namespace registry got validated and uniqueness-checked against the default graph with nothing to say so. I treated "behaviour preserved" as a claim to break rather than a claim to accept, and it held everywhere I pushed: the commit_transfer routing is byte-identical to what it replaced, StageOptions::new() really is Self::default(), the tracker is_enabled() gating survived at all three sites that had it, and stage_with_shacl / format_shacl_report have zero remaining references anywhere including docs.
The one thing I'd like before this merges is a note in the "Behavior" section, because there are two changes there and the body describes one. Alongside the documented tracked-path no-op fix, folding registers_new_graph into is_noop also adds that guard to the three tx.rs commit paths that never had it — so on Fluree::transact(), a zero-flake update naming a not-yet-registered graph now writes an empty registration commit and bumps t where before it didn't. I verified it (probe and exact repro in the inline note on tx.rs:2122): t=1 → t=2 at HEAD, t=1 → t=1 with the has_unregistered clause neutralised. I think it's the right direction and it's what the builder already did, so this is "document it and pin it", not "change it" — but nothing in the suite distinguishes HEAD from base on that path right now, which is a gap worth closing in the PR that opens it.
Two smaller ones, both in the inline notes: it_policy_verbs.rs:855's relaxed Err(_) | Ok(_) match passes under both the old and the new behaviour (I confirmed it stays green under the mutation that reverts the no-op rule, while the other three touched tests go red), so it isn't pinning anything any more; and tx.rs:3192's hardcoded sync_graph: None was inert before this PR and is load-bearing now that is_noop reads it — worth a derivation or a comment, especially with #1932 stacking stage_sync_transaction_tracked on top.
Adherence to repo commitments:
- Patterns/abstractions: ✔ Extends the existing mechanisms rather than adding parallel ones —
StageResult,CommitReceipt,GraphRegistryandStagedLedgereach grew the method that belonged on them, and thegraph_sidsparameter drop genuinely narrows two contracts instead of just moving them. - Performance (speed first, memory second): ✔ Net improvement, no regression risk. Both per-flake consumers (
enforce_unique_constraints,validate_staged_nodes) swap aHashMap<Sid, GraphId>lookup for aVec<u16>index, and two per-write map builds disappear —graph_ids_by_sid, which scaled with the whole graph registry rather than with the transaction, andbuild_reverse_graph_lookup. No new allocations, clones or locks anywhere in the diff. No bench covers staging or uniqueness, so the win won't surface in the numbers; that's pre-existing, not yours. - Testing:
⚠️ The routing consolidation is genuinely pinned — collapsingstaged_flakes_by_graph()tog_id = 0turnsit_config_graph::shacl_rejects_violation_in_any_named_graph_{sparql,jsonld},shacl_per_graph_mode_applies_to_graph_variable_targetsand bothit_constraints_inline::inline_unique_property_enforced_in_second_named_graph_*red, so both the SHACL and uniqueness sides are covered. The newtracked_update_matching_nothing_is_a_noopis wired intogrp_transactand goes red under mutation. The⚠️ is only for the two gaps above: the registration change is uncovered, andit_policy_verbs.rs:855no longer discriminates. - Conventions: ✔ Self-describing subjects, substantial multi-line bodies that carry the rationale (the commit bodies are better than the PR body here — the registration point is implicit in commit 2's first paragraph). Clippy and fmt green at this head. W3C registers untouched.
Verified locally at 6bb1f7a15: cargo nextest run -p fluree-db-api --features shacl --test grp_transact --test grp_policy (4/4 pass, new test present by name); three mutation runs (no-op rule → 3 of 4 red; graph routing → 5 red across it_config_graph + it_constraints_inline; registration guard neutralised → the probe flips); all four StagedShaclContext sites and every commit_stage_result call site traced against the inline rule it replaced. CI at this exact head (run 36123814997, event=pull_request): test, clippy, fmt, testsuite-sparql, wasm32 all green. --all-features wouldn't build locally (usearch → cxx can't find <algorithm> on my machine) — unrelated to this PR, and CI's --all-features run covers it.
Approving so you can merge when ready — but I'd really like the registration change documented and pinned before it goes, since this is the PR that introduces it.
| .base() | ||
| .snapshot | ||
| .graph_registry | ||
| .has_unregistered(self.graph_delta.values().map(String::as_str)) |
There was a problem hiding this comment.
Optional, but I'd want it in this PR — there's a second behaviour change here that the body doesn't call out.
Folding registers_new_graph into the shared is_noop doesn't just move that guard — it adds it to the three tx.rs commit paths (execute_transaction, transact_with_trig_meta, transact_with_named_graphs) that never had it. Their old inline rule was !view.has_staged() && (Update|Upsert || sync_graph.is_some()), with no registration clause at all; only the three tx_builder.rs copies carried one.
The effect: on Fluree::transact(), a zero-flake update whose graph_delta names a graph the registry doesn't know now writes an empty registration commit and advances t, where before it returned without one. I verified rather than inferred it — a temporary probe against a ledger seeded at t=1:
let update = json!({
"@context": {"ex": "http://example.org/"},
"where": {"@id": "?s", "ex:absent": "?o"}, // matches nothing
"insert": {"@id": "?s", "@graph": "http://example.org/graphs/brandnew", "ex:q": "y"}
});
fluree.update_with_opts(ledger, &update, TxnOpts::default(), Default::default(), &index_cfg).await
// HEAD: receipt.t == 2, flake_count == 0 ← empty registration commit
// `.has_unregistered(std::iter::empty())`: receipt.t == 1, flake_count == 0 ← the base's rule on this pathI actually think this is the right direction — it's what the builder already did, and a transact() that disagrees with stage(&handle) about whether a graph got registered is exactly the drift this PR exists to kill. But the rule is broader than the intent the old comment stated ("CREATE GRAPH: zero staged flakes, but a graph_delta IRI the registry doesn't know yet"), and a no-match update that merely mentions a graph in an INSERT template isn't a CREATE GRAPH.
So two things would settle it: a line in the "Behavior" section saying this path changed too, and a test — nothing in the suite distinguishes HEAD from base here today, which is why I had to write one. If instead you'd rather the rule stay narrow, the distinguishing signal is presumably txn.graph_mgmt / the SPARQL CREATE GRAPH lowering rather than "any unregistered IRI in the delta".
There was a problem hiding this comment.
Good catch, thanks. I kept the broad rule: the point of this PR is that transact() and the builder stop disagreeing, and the builder's rule was the one already in use. Narrowing it to CREATE GRAPH raises a wider question: should a graph a template names but writes nothing to be registered at all? Commits with other flakes do that today on every path, so that belongs in its own decision, not the no-op case alone.
Documented in the Behavior section and pinned in 8321b52 by no_match_update_naming_a_new_graph_commits_its_registration, which is your probe: t advances, zero flakes, graph registered. It fails with has_unregistered neutralised.
| ); | ||
| // The profile cannot see a Person, so the WHERE matches nothing: the | ||
| // update is refused or is a no-op, and either way writes nothing. | ||
| let before_t = ledger.t(); |
There was a problem hiding this comment.
Optional — this test now passes whichever behaviour is in effect, so it isn't a net for the change any more.
Under the mutation that reverts the no-op rule (TxnType::Update | TxnType::Upsert → TxnType::Upsert), the other three tests this PR touches go red and this one stays green — the Err(_) => {} arm absorbs the old 500.
I get why it was relaxed: what the test cares about is "nothing was written", and asserting after.t() == before_t says that far more directly than asserting an error string did. So this isn't wrong, it's just no longer load-bearing. Since tracked_update_matching_nothing_is_a_noop covers the no-op rule directly, maybe the cleanest thing is to drop the Err(_) arm and let this test assert the outcome it now expects:
// The profile cannot see a Person, so the WHERE matches nothing: the
// update is a no-op that writes nothing and does not advance the ledger.
let before_t = ledger.t();
let after = try_txn(&fluree, ledger, TxnType::Update, &update_person, &ctx)
.await
.expect("a view-filtered no-match update is a no-op, not an error");
assert_eq!(
after.t(),
before_t,
"Lead write profile must NOT edit a Person"
);Minor and non-blocking — but if you agree it's right, I'd rather see it folded in now than lost in the backlog.
There was a problem hiding this comment.
Done in 8321b52, using your version: it expects the no-op and asserts t did not advance. With the no-op rule reverted it now goes red (it stayed green before).
| txn_meta, | ||
| graph_delta, | ||
| graph_delta: staged.graph_delta, | ||
| sync_graph: None, |
There was a problem hiding this comment.
More of a question than a suggestion — sync_graph: None is hardcoded here while every sibling StageResult derives it from txn.sync_graph.
It's accurate today — this entry point parses through parse_transaction, and only parse_sync_transaction ever sets sync_graph — but it was inert before this PR and isn't any more: is_noop reads that field, so if a sync ever reaches this door it silently loses the no-op and lands back on the 500 this PR just fixed.
Given #1932 is adding stage_sync_transaction_tracked immediately on top of this branch, it seems worth either deriving it (let sync_graph = txn.sync_graph.clone(); alongside txn_meta, one line up) or leaving a comment saying why None is structurally guaranteed here.
There was a problem hiding this comment.
Derived it in 8321b52: let sync_graph = txn.sync_graph.clone(); alongside txn_meta, so a sync reaching this entry point keeps its no-op.
| } | ||
|
|
||
| /// Each staged flake with the ledger graph id staging routed it to. | ||
| pub fn staged_flakes_by_graph(&self) -> impl Iterator<Item = (GraphId, &Flake)> { |
There was a problem hiding this comment.
Praise — reading the routing from flake_graph_ids instead of re-deriving it is the best decision in the PR, and it deletes a silent failure mode.
The old flake_graph_id's unwrap_or(0) meant a named-graph flake whose IRI didn't resolve through staged_ns.lookup_sid_for_iri got validated and uniqueness-checked against the default graph, with nothing to announce it. StagedOverlay::from_flakes is strict about the same question, so reading the answer from there can't fail quietly.
A coverage note while we're on this routing, because it's more useful to you than to me. I mutation-checked it — collapsed staged_flakes_by_graph() to g_id = 0, which is the old unwrap_or(0) fallback firing on every flake — and it is pinned: five tests go red, in it_config_graph (shacl_rejects_violation_in_any_named_graph_{sparql,jsonld} and shacl_per_graph_mode_applies_to_graph_variable_targets) and it_constraints_inline (both inline_unique_property_enforced_in_second_named_graph_*). So SHACL and uniqueness are both genuinely covered. What surprised me is where it isn't: that same mutation left all 86 tests green across it_sync_graph, it_branch_shacl, it_policy_named_graphs, it_enforce_unique*, it_named_graphs and it_named_graph_isolation — i.e. every suite whose name suggests it would catch a graph-routing regression. Nothing for this PR to fix, but worth knowing that the net under this code is narrower, and sits somewhere other than where you'd reach for it.
The one thing I'd flag for a future reader: zip truncates to the shorter side, so if flake_graph_ids ever fell out of lock-step with store.flakes the consequence is silently dropped flakes from SHACL and uniqueness, not a panic. It's correct today — both are built in the one pass at staged.rs:83-97 and neither is mutated after — and honestly a debug_assert_eq! on the two lengths is probably over-engineering for a private field with one writer. Raising it only because the failure shape is "validation quietly stops seeing some of your data", which is the worst one to discover late.
| Ok((templates, graph_iris)) | ||
| } | ||
|
|
||
| /// Stage options for a write: backpressure against `index_config`, modify |
There was a problem hiding this comment.
Praise — stage_and_check keeping txn_context and the error mapping at the caller is the right seam.
Those two really are the only things that legitimately differ per entry point, and the tracked path at tx.rs:3178 deliberately not going through stage_options (because it attaches its tracker unconditionally rather than gating on is_enabled()) is the kind of detail a consolidation like this usually flattens by accident. Nice.
ea1b176 to
fc6574f
Compare
8c0cf99 to
7d4b313
Compare
f1615b7 to
98c1548
Compare
090daaf to
fcdfb20
Compare
The API's write entry points each assemble stage options, a per-transaction ResolveCtx, SHACL or plain staging, novelty-backpressure handling, and the uniqueness and reasoning-mode checks. `stage_and_check` now carries that sequence and `stage_options` builds the options, so every entry point shares one implementation while keeping its own authoring context and error mapping. Also retires `stage_with_shacl` and the report formatter it used, which the API's staging path superseded, and adds `GraphRegistry::has_unregistered` for the commit and the transaction builder.
…w graph routing Commit tail. `StageResult::is_noop`, `CommitReceipt::no_op` and `commit_stage_result` give every write path the same stage-to-commit step, including the rule for skipping a commit when a stage changes nothing. The tracked policy path now follows that rule too: a policy-carrying update whose WHERE matches nothing succeeds without a commit, as it does elsewhere, where it previously surfaced a 500 "Empty transaction". Two policy tests now assert directly that nothing was written. Cross-ledger SHACL membership. `namespace_prefix_map` and `CrossLedgerShapesModel::membership` build the membership handle for transaction staging, branch-operation validation and `validate`. Graph routing. `StagedLedger::staged_flakes_by_graph` exposes the ledger graph id staging assigned to each flake. SHACL validation and uniqueness enforcement read it directly, so the SHACL context no longer takes a separate graph -> Sid map.
fcdfb20 to
819b3a5
Compare
Folding the registration guard into StageResult::is_noop also changed the Fluree::transact() paths, which never had it: a no-match update whose INSERT template names an unregistered graph now commits the registration, with zero flakes, as the builder already did. A new test pins that. The policy test that tolerated either an error or a no-op now expects the no-op, so it fails if the tracked path regresses to the old 500. The tracked policy path also takes sync_graph from the Txn instead of hardcoding None, since is_noop now reads it.
After the rebase onto main, clippy and the test build failed with "queries overflow the depth limit" in it_absent_subject_scan_narrowing and it_exists_semijoin_correlation. Every write entry point now awaits stage_and_check, and the staging future inlined into each of them made the future types of callers a few async layers deep exceed rustc's layout depth limit. Boxing that one future bounds the depth for every caller.
|
@aaj3f thanks. All three are in 8321b52 (replies on each thread): the One more thing turned up. After the stack was rebased onto |
Summary
This PR follows the staging and graph-routing work in #1926. It moves the write path's common steps into shared helpers, so every entry point runs the same staging, checks and commit logic. Apart from the no-op change under "Behavior", the effect of each write is unchanged.
Staging
stage_and_checkruns staging and the post-staging checks for the API's write entry points. It covers SHACL or plain staging, novelty-backpressure handling, uniqueness, and reasoning modes. Each entry point keeps its own authoring context and error mapping.stage_optionsbuilds stage options from the index config, the policy, and an enabled tracker.Commit
StageResult::is_noopis the rule for skipping a commit. It applies to an update, upsert or sync that staged no flakes and registers no new graph. A registration-only stage such as SPARQLCREATE GRAPHstill commits.CommitReceipt::no_opbuilds the receipt for a skipped commit.commit_stage_resultis the stage-to-commit step shared by the transact paths and the transaction builder.GraphRegistry::has_unregisteredanswers whether a commit would register a graph.SHACL and uniqueness
StagedLedger::staged_flakes_by_graphexposes the ledger graph id staging assigned to each flake. SHACL validation and uniqueness enforcement read it directly, so they always use the same routing as staging. The SHACL context therefore drops its separate graph-to-Sid input.namespace_prefix_mapandCrossLedgerShapesModel::membershipbuild the cross-ledgersh:classmembership handle for transaction staging, branch-operation validation, andvalidate.Removed
stage_with_shacl, which the API's SHACL-aware staging superseded, along with the report formatter only it used.Behavior
The tracked policy path now uses the shared no-op rule. A policy-carrying update whose WHERE matches nothing succeeds without writing a commit, as it already did on the other write paths. Before, it returned a 500 "Empty transaction".
tracked_update_matching_nothing_is_a_noopcovers this, and it fails without the change.The shared rule also changes the
Fluree::transact()paths, which never had the registration clause. A no-match update whose INSERT template names a graph the ledger has not registered now commits that registration, with zero flakes, and advancest. Before, it skipped the commit. The transaction builder already behaved this way; the two entry points now agree.no_match_update_naming_a_new_graph_commits_its_registrationpins it and fails if the registration clause is removed.Build
stage_and_checkboxes its staging future. Every write entry point awaits it, and after rebasing ontomain, two test targets whose async helpers nest a few layers deep failed to compile with "queries overflow the depth limit". Boxing that one future bounds the depth for every caller.