Repository navigation
feat(sparql): # PRAGMA request options, the SPARQL twin of JSON-LD opts - #2002
Conversation
A SPARQL request can now carry Fluree request options in `# PRAGMA name: value` comments, so it no longer depends on `fluree-*` headers for them and its text stays valid SPARQL. Before this, only the reasoning pragmas existed. New pragmas: - queries: include-system-facts, min-t - queries and updates: meta, max-fuel, identity, policy-class, policy-values, default-allow - updates: event-time, validation-mode, unique-properties Each pragma is applied where its JSON-LD `opts` counterpart is: - lowering sets include-system-facts on the query IR, and the SPARQL UPDATE lowering sets validation-mode and unique-properties on TxnOpts unless the caller already set them; - the API derives fuel limits and tracking from pragmas, and the query builders take policy selection from them; - the server merges pragmas into FlureeHeaders after bind_authorization. A pragma wins over the header that names the same option, and policy selection is still held to the caller's credential by bound_governance. A multi-query SPARQL alias treats its pragmas as its body opts, and the server checks that they cannot replace the authorized selection. The MCP sparql_query tool refuses policy and min-t pragmas, because the connection's identity and the `t` argument decide those. Pragmas are strict: an unknown name, a malformed value, or a pragma that doesn't apply to the request's form (query vs update) fails the parse with F012 and a 400. Previously an unknown pragma was ignored, and a reasoning pragma on an UPDATE was parsed and then dropped. IRI values expand against the request's own PREFIX declarations. The lexer now records each comment's span, so the diagnostic points at the offending line. Docs: a request-options section in the SPARQL reference. The policy, tracking, streaming and header pages that described SPARQL as having no opts are updated. unique-constraints.md also no longer claims that unknown IRIs are dropped silently; the transaction fails on them.
bplatz
added
enhancement
aaj3f
left a comment
aaj3f
left a comment
There was a problem hiding this comment.
@bplatz -- this is nice, and a nice parity improvement. The one headline note I'd make from reviewing this is that I think the pragma config vs header config should be unified / abstracted over / precedence-consolidated in the same ways that we already do this for, say, the various opts config in JSON-LD. As the PR stands now, PRAGMA config can disagree with headers config in a way that could be either/both a matter of permission vulnerability and/or at least confusing and an operator footgun. See more details in full review below:
Every pragma lands in the carrier its JSON-LD twin already uses (GovernanceOptions, TrackingOptions, TxnOpts, Query::include_system_facts), so there's no second execution path, and making unknown or misplaced pragmas a 400 instead of silently dropping them is exactly right. The multi-query same_policy_selection check, which refuses to just trust that the merge came out right, is a nice touch too.
The two that need to land first:
fluree-db-server/src/extract/headers.rs:408— under an application credential ("fluree.policy": "request"), a policy pragma in forwarded SPARQL replaces the identity/policy the application sent in headers. I reproduced it: headeremployee-userplus# PRAGMA identity: <manager-user>returnsExecutive Salaries. Fix path: a policy pragma fills what headers left unset and a conflicting one is a 403 (as fixed credentials already behave), take the tightermax-fuel, and updatedocs/security/policy-authorization.md.fluree-db-api/src/tx_builder.rs:135—validation-mode/unique-propertiesonly apply throughparse_and_lower_sparql_update; the publiclower_sparql_update_astthat embedders (solo's transact Lambda among them) use drops them, sounique-propertiesis accepted and not enforced. Fix path: apply them insidelower_sparql_update_ast.
The rest:
fluree-db-api/src/query/builder.rs:964— the caller-wins guard has no test; I mutated it away and the wholeit_policygroup stayed green.fluree-db-sparql/src/parse/query/mod.rs:139— pragma-bearing requests parse two or three extra times.- Embedded hosts that call
execute_tracked()without.tracking(…)now let the SPARQL text choose its tracking (# PRAGMA meta: timeturns fuel tracking off), asopts.metaalready does for JSON-LD. Probably worth a line under "Behavior changes" for embedders who meter on fuel.
Adherence to repo commitments:
- Patterns/abstractions: ✔ pragmas are a second front-end onto the shared carriers, not a parallel path;
⚠️ three separate translators ofPragmas(server headers,GovernanceOptions::from_sparql_pragmas,sparql_pragma_opts) have to stay in step as pragmas are added. - Performance (speed first, memory second): ✔ no engine hot path touched; pragma-free requests pay one byte scan, pragma-bearing ones re-parse (🟡).
- Deployment targets:
⚠️ reaches the embedded host (builders, trackers, update lowering), the server and wasm32 (CI green). Evidence is server- and builder-shaped; the embedded update entry is where pragmas are dropped (item 2). No Lambda fence involved. - Testing: ✔ broad server and API twins plus parser units, wired into
grp_query/grp_policy/grp_httpand thefluree-db-apigroups;⚠️ no request-credential case and no caller-wins case. - Conventions: ✔ thorough multi-line commit, docs updated across the policy, tracking and header pages;
⚠️ policy-authorization.mdstill says headers are SPARQL's policy transport.
Verified locally: CI is green at this head. I trial-merged the stack onto origin/main (61b836e9a, clean) and ran policy_integration (59), override_control_identity (9), sparql_pragmas (7), stream_query_integration (23), mcp_agent_json (8), and the API it_policy_graph_builder (21), it_fuel_floor (11), it_edge_annotations (119) and it_constraints_inline (7) groups — all green with the new tests present by name — plus fluree-db-sparql --lib pragma (29) at this head. I also probed the parser's boundary cases (#PRAGMA, ## PRAGMA, after the body, inside strings and IRIs, #), and they behave as documented.
Approving now so you can merge without waiting on another pass from me — just be sure the first two are in before you do.
| if pragmas.min_t.is_some() { | ||
| self.min_t = pragmas.min_t; | ||
| } | ||
| if pragmas.identity.is_some() { |
There was a problem hiding this comment.
🔴 Must address before merge: under a request-selected credential, a pragma in forwarded SPARQL text replaces the identity and policy the application sent in headers.
with_sparql_pragmas lays each policy pragma over the header that names the same option, and bound_governance then resolves that against the bound credential. For a fixed credential that's a 403 on conflict (I confirmed # PRAGMA default-allow: true under an identity token comes back 403 … conflicting default-allow). For an application credential ("fluree.policy": "request"), though, CredentialPolicy::Request.resolve_options (extract/credential_policy.rs:32) accepts whatever selection arrives, so what runs is the selection in the SPARQL text, not the one the application put in the headers.
That's the deployment docs/security/policy-authorization.md describes — the application authenticates its users, resolves their grants and supplies the context — and that page still says the fluree-identity / fluree-policy-class / … headers carry policy options for SPARQL. I reproduced it on the policy_integration.rs fixture with a request-credential token and fluree-identity: http://example.org/employee-user: the plain query returns Internal Memo, Public Post; the same query with # PRAGMA identity: <http://example.org/manager-user> returns Executive Salaries as well, and so does # PRAGMA policy-class: ex:ManagerClass sent over a fluree-policy-class: …/EmployeeClass header. A JSON-LD body's opts.identity does the same thing today, but there the application builds the body; with SPARQL the text is usually the end user's, and until this PR the headers were the one channel the application controlled.
I think the cleanest fix is for a policy pragma (identity, policy-class, policy-values, default-allow) to fill what the headers left unset and to be refused with a 403 when it conflicts with a header the caller did send, the way the fixed-credential path already behaves. That keeps the anonymous and pragma-only cases working and makes the forwarding case fail closed; sparql_pragma_policy_class_wins_over_header would flip to assert the refusal, and a request-credential case belongs next to it. The same reasoning applies to max-fuel as a cap: taking the tighter of header and pragma means a pragma can't lift a limit the application set. Whichever way you go, policy-authorization.md should say what an application holding a request credential needs to do about policy pragmas in text it forwards.
There was a problem hiding this comment.
Addressed in 84ae3f1.
One change from the suggested fix: when the headers select policy, a pragma that fills a field they left unset is also a 403. A policy-class beside a header identity selects the rules, and default-allow: true widens them. Repeating the selection or narrowing default-allow to false is accepted. The check applies only under a bound credential; unauthenticated requests keep pragma-wins, because an inline fluree-policy header paired with policy-values pragmas is documented usage. So sparql_pragma_policy_class_wins_over_header stays, and the request-credential cases sit beside it. It covers SPARQL UPDATE and multi-query aliases too. In 45cddc6, a meta pragma also stopped being able to switch off tracking the headers ask for.
| return Err(ApiError::sparql(message, errors)); | ||
| } | ||
|
|
||
| // Transaction options a `# PRAGMA` names, under the same precedence as |
There was a problem hiding this comment.
🔴 Must address before merge: update pragmas take effect only through parse_and_lower_sparql_update; the public lower_sparql_update_ast drops them without a word.
validation-mode and unique-properties are folded into TxnOpts here, and event-time / max-fuel / the policy pragmas are picked up by the server route and TransactCore::tracker. fluree_db_transact::lower_sparql_update_ast (re-exported from fluree_db_api) is the entry an embedded host uses to lower an update itself and hand a pre-built Txn to the stage builder, and nothing on that path reads ast.pragmas — git grep '\.pragmas' finds only this block, the parser that fills them, the MCP tool's refusal and the SPARQL query lowering.
So # PRAGMA unique-properties: ex:email on an update lowered that way parses cleanly and commits without the constraint. I checked it on your fixture: the AST carries unique_properties: Some(["http://example.org/ns/email"]), and a duplicate email commits through lower_sparql_update_ast + .txn(txn), while inline_unique_property_via_sparql_pragma rejects the same duplicate through the builder in the same run. That's exactly the outcome the new comment in parse/query/mod.rs:105-107 says strictness is there to prevent ("running the request without the option it asked for … would silently weaken it"), and it isn't hypothetical: solo's transact Lambda lowers SPARQL UPDATE this way (lower_sparql_update_ast(&ast, &mut ns, TxnOpts::default()), then .txn(txn)), so after a pin bump the pragma would be accepted and ignored there.
Minimally we could move this block into lower_sparql_update_ast itself (the AST is in hand there, and the caller-set-wins rule moves with it), so every lowering entry honors validation-mode and unique-properties, and say on that function that event-time and max-fuel ride ast.pragmas for the caller to apply. Commenting here because fluree-db-transact/src/lower_sparql_update.rs is not in this diff.
| /// no room for, unless the caller chose policy programmatically. Under | ||
| /// [`Self::authorization`] they are constrained like any request selection. | ||
| fn adopt_sparql_pragma_policy(&mut self) { | ||
| if self.connection_opts.is_some() || self.policy.is_some() { |
There was a problem hiding this comment.
🟠 Should address: nothing pins that a caller's own policy selection wins over the pragmas.
This early return is what embedded hosts lean on: a host that sets connection_opts (solo's query Lambda does, for every request with a bearer) gets the text's policy pragmas ignored, so a user can't pick their own policy through a comment. I replaced the condition with if false and all 139 tests in grp_policy it_policy stayed green, including the new pragma twins, so a refactor that loses it would ship quietly.
One test — .connection_opts(<identity A>) plus # PRAGMA identity: <B> returning A's rows, and the same with .policy(…) — would hold it. It's small — but if you agree it's right, I'd rather see it folded in now than lost in the backlog.
| let (head, rest) = trimmed.split_at(keyword.len()); | ||
| if !head.eq_ignore_ascii_case(keyword) { | ||
| return None; | ||
| let output = parse_sparql(input); |
There was a problem hiding this comment.
🟡 Optional: a request that mentions "pragma" now parses its whole text two or three extra times.
request_pragmas is a full parse_sparql whenever the text contains "pragma", and it's called independently by the server's header merge, sparql_tracking_options (tracked queries), the builders' policy adoption and the multi-query alias merge, each ahead of the parse that runs the query. For ordinary queries that's noise; for the multi-MB VALUES payloads people do send it's a few redundant parses. I don't think this needs anything clever — parsing once and carrying the Pragmas alongside the text would cover it. Minor and non-blocking, but if you agree it's right, I'd rather see it folded in now than lost in the backlog.
There was a problem hiding this comment.
Partially addressed in ee5cd2e: request_pragmas now parses only when a # comment could be a directive, and the view and dataset paths read the pragmas from the AST they already hold. Parsing once end to end is #2014. The same commit adds every_pragma_reaches_its_translators for the translator-drift point in your summary.
…date pragmas apply on every lowering entry Policy pragmas under a bound credential. An application holding a request-selected credential forwards its users' SPARQL and selects policy in headers, but `with_sparql_pragmas` laid each policy pragma over its header, so text like `# PRAGMA identity: <manager>` replaced the application's `fluree-identity`. When a credential is bound and the headers select policy, a policy pragma may now only repeat that selection or narrow `default-allow` to false (`validate_pragma_selection`, reusing `validate_selection`); anything else is a 403. That includes a pragma naming an option the headers left unset: a `policy-class` beside a header identity selects the rules, and `default-allow: true` widens them. When the headers select nothing, the pragmas select. The same check runs for a SPARQL UPDATE, where the pragma identity also became the commit author, and for a multi-query SPARQL alias against the envelope and alias opts. Unauthenticated requests keep pragma-wins, so an inline `fluree-policy` header can still pair with `policy-values` pragmas. max-fuel is a cap: a request now runs under the smaller of the pragma and `fluree-max-fuel`, and a multi-query alias's pragma cannot raise its `opts.max-fuel` (`cap_pragma_max_fuel`, in both the server's alias authorization and the dispatcher). Update pragmas on embedded lowering. `validation-mode` and `unique-properties` were applied only in `parse_and_lower_sparql_update`, so a host that lowers with `lower_sparql_update_ast` and stages the Txn accepted `# PRAGMA unique-properties` without enforcing it. Both public entries, `lower_sparql_update_request` and `lower_sparql_update_ast`, now apply them; a value the caller set still wins. Tests: request-credential query, update and multi-query cases in policy_integration; `max_fuel_pragma_only_tightens_the_header` and `multi_query_alias_max_fuel_pragma_only_tightens_its_opts`; `inline_unique_property_via_sparql_pragma_lowered_by_caller` and a transact unit test for both entries; `sparql_pragma_policy_yields_to_caller_selection` pins that a builder's `connection_opts` or prebuilt `policy` wins over the text's pragmas. Each fails with its fix reverted. Docs: policy-authorization.md tells applications with request-selection credentials to send their selection in headers; the precedence rules are updated in sparql.md, headers.md, policy-in-queries.md, the policy cookbook and tracking-and-fuel.md.
…ng it `with_sparql_pragmas` replaced the `fluree-track-*` flags with the pragma's, so `# PRAGMA meta: time` in a request's text switched off fuel tracking its headers asked for, and an application reading fuel from the response lost it. The pragma's flags now add to the headers': a request reports what either asks for. A multi-query SPARQL alias's `meta` pragma likewise adds to its `opts.meta` (`hold_pragma_tracking`, which also carries the alias's max-fuel cap). The `sparql_pragmas.rs` fuel helper matched "fuel" anywhere in a body, which a tracked success contains too; it now reads only the error message. Tests: `meta_pragma_adds_to_header_tracking` and `multi_query_alias_meta_pragma_adds_to_its_opts`, each failing with the fix reverted.
…a pragma comment; guard the pragma translators `request_pragmas` parsed any text that mentioned "pragma", so a large VALUES payload containing the word paid a full parse. It now parses only when some `#` comment could be a directive (`may_carry_pragma`, a superset of what the parser reads as one). The view and dataset query paths parsed the request, then parsed it again to build their fuel and tracking trackers. `tracked_query_tracker`, `tracker_for_input_limits` and the new `input_fuel_limit` take the AST when the caller holds it and read `ast.pragmas`; the residency retry loop computes the limit once instead of re-parsing every round. Four translators turn pragmas into options: `FlureeHeaders:: with_sparql_pragmas`, `GovernanceOptions::from_sparql_pragmas`, `sparql_pragma_tracking` (now public) and `sparql_pragma_opts`. `every_pragma_reaches_its_translators` walks `fluree_db_sparql:: pragma_names()` and checks each translator against a table, so a new pragma that is not listed fails the test with what to wire up. Tests: `test_request_pragmas_skips_only_text_without_a_directive`, `tracked_sparql_meta_pragma_selects_tracking`, `untracked_sparql_max_fuel_pragma_caps_a_within_ledger_dataset_query`, and the translator guard; each fails with its change reverted.
…owering The pragma guard from #2002 requires every pragma to be listed. union-default-graph lowers into Query::union_default_graph, where its opts.unionDefaultGraph twin also lands, so no header, governance, tracking or alias-opts translator carries it.
2 participants
Summary
SPARQL requests can now carry Fluree request options in
# PRAGMA name: valuecomments, the counterpart of a JSON-LD request'sopts. Until now, a SPARQL request got these options only fromfluree-*headers; the reasoning pragmas were the one exception. Pragmas are comments, so the query text stays valid SPARQL for any other tool.include-system-facts,min-tmeta,max-fuel,identity,policy-class,policy-values,default-allowevent-time,validation-mode,unique-propertiesThe inline policy document has no pragma. It stays on the
fluree-policyheader.Where each pragma is applied
Each pragma takes effect at the point where its JSON-LD
optscounterpart does:fluree-db-sparql):Pragmasgains typed fields, and the extraction lives inparse/query/pragma.rs.request_pragmas(text)lets a host read the options before it parses the request. It skips the parse entirely unless some#comment could be a directive.pragma_names()lists every name the parser accepts.include-system-factssetsQuery::include_system_facts.validation-modeandunique-propertiesare set onTxnOptsby both public lowering entries,lower_sparql_update_requestandlower_sparql_update_ast, unless the caller already set them. So a host that lowers the AST itself and stages theTxngets them too, as does every server surface, Raft included.max-fuelandmeta; they previously gotTracker::disabled()or all tracking. Where the view and dataset paths already hold the parsed AST, the trackers readast.pragmasrather than parsing the text again.GraphQueryBuilderandFromQueryBuildertake policy selection from pragmas when the caller supplies none. A caller's ownconnection_optsor.policy(…)wins over them. Under.authorization(), those values are constrained the same way any request selection is.max-fuelpragma (TransactCore::tracker), as a JSON-LD body'smax-fuelalready does.FlureeHeaders::with_sparql_request_pragmasruns straight after the SPARQL text is resolved, on the query, explain, stream and update routes. A pragma wins over the header that names the same option, with three exceptions, because the headers may be an application's while the text is its end user's:metaadds to the tracking thefluree-track-*headers ask for; it cannot switch it off.max-fueltakes the smaller of the pragma andfluree-max-fuel.default-allowtofalse(validate_pragma_selection). Anything else is a 403, including a pragma naming an option the headers left unset: apolicy-classbeside a header identity would select the rules. When the headers select nothing, the pragmas select. Unauthenticated requests keep pragma-wins, so an inlinefluree-policyheader still pairs withpolicy-valuespragmas.bind_authorization, sobound_governancethen holds the selection to the caller's credential exactly as it holds a header. A conflicting selection gets a 403.event-timeshares theopts.eventTimevalidation, through the new helperwith_event_time.opts, so they win oversub.optsand the envelopeopts, with the same exceptions:metaadds to the alias's tracking andmax-fuelcannot raise its cap (hold_pragma_tracking, applied by both the server and the dispatcher), and on an authenticated request a policy pragma may only repeat the selection the envelope and aliasoptsmake. The server includes the pragmas in the authorization merge. It then checks that laying them back over the authorized selection changes nothing (same_policy_selection), instead of relying on that being true.sparql_queryrefuses policy andmin-tpragmas, because the connection's identity and thetargument decide those.Behaviour changes
min-ton an update,event-timeon a query) fails the parse with the new diagnosticF012and returns a 400. Previously an unknown pragma was ignored silently; the oldtest_unknown_pragma_ignoredused# PRAGMA timeout: 30, a pragma that never existed.reasoningpragma on an UPDATE is now a 400. Before, it was parsed and then dropped.PRAGMAmakes a comment a directive. Any comment whose first word isPRAGMAis now read as one; the diagnostic's help text says so.Docs
docs/query/sparql.mdhas a new "Request options (# PRAGMA)" section, including how pragmas combine with headers.policy-authorization.mdtells an application holding a request-selection credential to send its policy selection in headers when it forwards its users' SPARQL.optsand are updated:eventTime,validationMode,uniquePropertiesandincludeSystemFactson the pages that document them.unique-constraints.mdclaimed that unknown IRIs inuniquePropertiesare dropped silently. The transaction actually fails on them, andinline_property_unknown_to_ledger_fails_loudlypins that.Testing
Parser: unit tests cover parsing, rejection, query-vs-update applicability, prefix expansion and
request_pragmas. Lowering tests showinclude-system-factsreaching all four query forms.Server, pragma versions of existing header tests:
policy_integration.rs: 12 policy tests, including bearer-conflict 403s on query, update and multi-query aliases, and request-selection-credential cases showing a pragma cannot replace or extend the headers' (or envelope's) selection.override_control_identity.rs:validation-modeunder a verified vs unverified identity.sparql_pragmas.rs: fuel,max-fuelandmetacombining with headers (and with a multi-query alias'sopts),min-t, malformed pragmas, update fuel andmeta, andevent-time.max-fuel, and the MCP refusals.API: builder policy selection, compared against a count derived from the fixture, through both builders, and a caller's
connection_opts/.policy(…)winning over pragmas. Also untrackedmax-fuel,include-system-factsalongside its JSON-LD twin, andunique-propertiesthrough the update builder and throughlower_sparql_update_ast+.txn(), plus a transact unit test for both lowering entries.Checked against
main: with the source changes stashed and only the tests kept, 25 of the 27 original integration tests fail. The 2 that pass are guards by design: a pragma that matches the credential must not be refused, and an unverified identity must not soften SHACL validation. Their doc comments say so. Each test added for the review fixes fails with its fix reverted.Translator guard:
every_pragma_reaches_its_translatorswalkspragma_names()and checks each of the four translators (FlureeHeaders::with_sparql_pragmas,GovernanceOptions::from_sparql_pragmas,sparql_pragma_tracking,sparql_pragma_opts) against a table. A new pragma that isn't listed fails it.Not covered
fluree.query(&view, …)) ignore policy pragmas, just as they already ignore JSON-LDopts.Follow-up: #2014