Repository navigation
Conversation
|
|
Contributor
|
@jobez Thank you for your contribution! Currently the minimum required version of clojure tools-deps is 1.10.1.697. Is it difficult to upgrade to that version (or later) for you? At the very least we should document this requirement more explicitly. |
Author
|
Documenting it would be lovely! Updating is a bit more involved for me unfortunately, but no worries I can keep my changes stashed away while I tinker about ⛵ |
dpetran
added a commit
that referenced
this pull request
May 17, 2024
Three approaches considered: 1) just document that :values needs to appear first 2) discard solutions that don't match the given solution 3) during the parse step, sort the patterns so :values is first I don't like #1, though that's the most expedient. #2 is a bit wasteful as it forces us to generate useless solutions. #3 is doable, and may be a minimal requirement in basic query planning. This implements solution #2.
dpetran
added a commit
that referenced
this pull request
May 31, 2024
Three approaches considered: 1) just document that :values needs to appear first 2) discard solutions that don't match the given solution 3) during the parse step, sort the patterns so :values is first I don't like #1, though that's the most expedient. #2 is a bit wasteful as it forces us to generate useless solutions. #3 is doable, and may be a minimal requirement in basic query planning. This implements solution #2.
dpetran
added a commit
that referenced
this pull request
Jun 12, 2024
Three approaches considered: 1) just document that :values needs to appear first 2) discard solutions that don't match the given solution 3) during the parse step, sort the patterns so :values is first I don't like #1, though that's the most expedient. #2 is a bit wasteful as it forces us to generate useless solutions. #3 is doable, and may be a minimal requirement in basic query planning. This implements solution #2.
JaceRockman
pushed a commit
that referenced
this pull request
Jul 2, 2024
Three approaches considered: 1) just document that :values needs to appear first 2) discard solutions that don't match the given solution 3) during the parse step, sort the patterns so :values is first I don't like #1, though that's the most expedient. #2 is a bit wasteful as it forces us to generate useless solutions. #3 is doable, and may be a minimal requirement in basic query planning. This implements solution #2.
Closed
5 tasks done
5 tasks
bplatz
added a commit
that referenced
this pull request
May 17, 2026
… diagnostics, nits Selected items from the additional review feedback. Each call explained. **#1 — Retry-fragility comment on `take(opts.shapes)`.** No retry exists on the staging path today, but `take()` moves the inline SHACL JSON off the txn, so a future retry would silently skip inline-shapes validation on the second attempt. Hard invariant comment added at the move site; if retry is ever added, defer the take until after `stage_txn` returns successfully. **#3 — `UnsupportedFeature` errors no longer carry an empty `ledger_id`.** `resolve_graph_ref` rejected `f:atT` / `f:trustPolicy` / `f:rollbackGuard` *before* verifying that the ref was actually cross-ledger, so a misrouted same-ledger ref hitting this path would have produced a confusing error with `ledger_id: ""`. Reordered: the same-ledger-ref guard runs first, binding `raw_ledger_ref` once; the unsupported-feature checks now carry the real ledger id in every diagnostic. **#4 — `f:atT` rejection breadth: audited, no gap.** All five `f:GraphRef`-shaped predicates already reject `f:atT` at their same-ledger entry points: - `policy_builder::resolve_policy_source_g_ids` - `tx::resolve_shapes_source_g_ids` (SHACL) - `ontology_imports::resolve_schema_bundle` - `tx::resolve_constraint_source_g_ids_for` - `view::fluree_ext::resolve_local_rules_source_g_id` Plus the cross-ledger `resolver` (now with non-empty ledger_id per #3). Audit-only change; no code modification beyond the doc note. **#2 — Cross-ledger policy target IRI loss is now logged.** `wire_to_restrictions` previously `filter_map`'d unresolvable target IRIs and `for_classes` IRIs silently. A cross-ledger policy could quietly narrow its scope on D if M referenced IRIs D had never registered. Two `tracing::warn!` calls now surface both cases with `restriction_id`, requested count, and resolved count — operators can spot the silent narrowing in logs. Kept the restriction even when targets become empty (rather than dropping the whole restriction): same-ledger via `load_policy_restriction` also produces empty target sets for the same IRI-not-in-snapshot scenario, and dropping the restriction would diverge cross-ledger from same-ledger semantics. The warn log is the parity-preserving signal. `fluree-db-policy` gains a `tracing` dependency for this. **Nits taken:** - `TAtUnavailable` doc-commented as Phase 3 reserved (unreachable today because `UnsupportedFeature { feature: "f:atT" }` fires first). - Empty `uniqueProperties: []` over HTTP is now documented as an intentional "no inline constraints" treatment, not a silent bug. - Cache read/write race in `resolver` documented as benign: losers materialize the same value structurally; single-flight is a future optimization, not a correctness need. - `f:AccessPolicy` IRI promoted from local `const` / duplicated-literal to `fluree_vocab::policy_iris::ACCESS_POLICY`. Two call sites updated; nothing else used the local consts. **Nits skipped per "don't make changes that don't seem needed":** - HTTP-level negative test for malformed `opts.shapes` / `uniqueProperties` (#5) — unit-level coverage exists; HTTP harness setup cost not justified. - Same-ledger `rules_source_g_id` end-to-end test (#6) — already covered by `it_rules_source::rules_source_in_named_graph_is_honored`, which runs an actual query against the named-graph rule. - DynamoDB hard-drop failure handling (#7) — storage layer, unrelated to this PR. - Duplicate pattern match in `shapes_materializer` — cosmetic. - `snapshot.ledger_id` redundancy in inline-shapes / inline- ontology `txn_id` — cosmetic. - `is_last_live_branch` WARN log — unrelated path.
mwatts
pushed a commit
to mwatts/fluree-db
that referenced
this pull request
Jun 12, 2026
Addresses four review findings against the multi-query handler and snapshot application. Bearer ledger scope (High fluree#1): The handler now walks the validated distinct_ledgers set after validation and rejects with 404 (existence-leak avoidance) if any ledger is outside the bearer's read scope. Matches the single-query /query and /query/:ledger behavior exactly: an unsigned bearer that can't read one of the envelope's ledgers gets the whole envelope rejected, not partial results that would reveal whether the ledger exists. Signed requests bypass this check, same as single-query. Identity + default policy-class threading (High fluree#2): The handler builds a MultiQueryIdentityContext containing the effective bearer identity (signed DID wins over bearer identity) and the server-configured default policy class, threaded through to the dispatcher. The dispatcher applies it per JSON-LD sub-query via apply_auth_identity_to_opts — the same code path the single-query JSON-LD handler uses, including the root-identity impersonation semantic. The impersonation check binds to each sub-query's primary ledger (first entry of its from clause), conservative for multi- ledger sub-queries. Headers are now injected into envelope opts BEFORE validation runs, so a client supplying max-fuel or maxConcurrency via fluree-* headers hits the same envelope-level rejections the body opts get. The fluree-* headers (policy-class, policy, policy-values, identity, default-allow, max-fuel) ride on the envelope opts and merge into every sub-query's opts as defaults via merged_opts — sub-query opts still win on key conflict, matching single-query behavior. SPARQL identity threading is explicitly deferred: the existing connection-scoped SPARQL path (/query with SPARQL FROM clauses) also doesn't currently thread identity. v1 multi-query matches that for parity; the docs flag this as a v1 limitation so the two paths can land identity threading together. SPARQL fragment-aware snapshot pinning (Medium fluree#1): apply_snapshot_to_sparql previously skipped any IRI whose value differed from its bare-ledger form via the value_str != bare check, which fired on both real temporal pins AND on named-graph fragments (ledger#txn-meta). So an envelope-pinned SPARQL like FROM <mq:frag#txn-meta> would run against current head instead of the envelope's t. Fixed by gating the skip on actual temporal-marker presence (has_temporal_marker checks for @t: / @iso: / @commit:), and splicing @t:N BEFORE any fragment when rewriting: <ledger#txn-meta> -> <ledger@t:42#txn-meta>. The dataset parser separates fragment from temporal suffix and reattaches the fragment after time-spec resolution, so the spliced form is what it expects. Per-sub-query response size cap (Medium fluree#4): DispatchConfig now carries max_subquery_response_bytes. Each sub-query's serialized result is sized once after dispatch returns; if a single alias's result exceeds the cap it's downgraded to an error outcome (code: "response_too_large") and the data dropped before assembly. This catches a single runaway query before it adds to envelope-wide memory pressure. The envelope-level cap in assemble_response is still the strict guarantee; the per-sub-query cap is the early-exit. The doc now spells out honestly that this is best-effort, not an OOM guard — peak memory is bounded by max_concurrency * max_subquery_response_bytes, not the envelope cap alone — and per-sub-query streaming serialization is queued for v1.1. Tests: - 4 new auth integration tests (multi_query_auth_integration.rs): unauthenticated request when data auth required returns 401, in-scope bearer succeeds, out-of-scope JSON-LD returns 404, out-of-scope SPARQL returns 404. - 1 new fragment-ledger SPARQL integration test confirms a SPARQL FROM <ledger#txn-meta> envelope alias runs successfully (regression against the unpinned-fragment bug). - 2 new unit tests cover the SPARQL fragment splicing (apply_snapshot_to_sparql) for both FROM and FROM NAMED clauses. 47 server integration + 21 policy + 4 multi-query auth + 16 multi-query core + 73 envelope/snapshot/dispatcher unit tests all green.
bplatz
added a commit
that referenced
this pull request
Jun 27, 2026
CachedOverlaySegment::byte_size now counts the HashMap table capacity and each ephemeral predicate Sid's Arc<str> name heap (dominant for novelty-only-predicate-heavy overlays) plus the Arc<[T]> control blocks, instead of a flat len*size_of that under-counted them - tightening the shared-budget weighing. Reconcile the design doc: the cursor key window is applied AFTER the merge (at cursor attach), not per-segment during assembly - the full merged product is cached per epoch for warm-repeat reuse and must be window-independent. The selective-cold-query O(total) copy is a known cost; eliminating it needs per-predicate-scoped assembly (deferred gap #1), not window-during-assembly.
aaj3f
added a commit
that referenced
this pull request
Jul 6, 2026
Address the review's findings — all verified real; all make the harness's green trustworthy rather than changing what it covers: - (review #1) the 'no unexpected named graph' guard was dead code: the engine binds GRAPH ?g as a plain literal, so the Iri-only filter in list_named_graphs always produced []. Accept literal and IRI bindings (excluding the alias-named default graph) so the guard survives the eventual engine fix. Verified live: an update leaving a stray graph with no expected graphs now fails with 'Unexpected non-empty named graph'. - (review #2) bail when a manifest yields zero tests — a submodule restructure or manifest-parser regression must not report green. - (review #3) a registered test that dies by timeout/subprocess crash now hard-fails: the register excuses a known wrong answer, not an infra death masking a new hang or panic. - (review #4) UpdateEvaluationTest now rejects an mf:result blank node exposing none of ut:data/ut:graphData/ut:result, instead of degrading to a trivially satisfiable 'expected empty store'. - (review #5) register entries matching no discovered test now fail the suite, so dead entries (typos, upstream renames) cannot accumulate. - notes: tightened the TSV bare-integer heuristic to a single optional leading sign; documented the line-based directive-hoisting assumption; fixed the stale 'CI runs make ci' Makefile comment. Full suite remains green: 36 suites, ~1420 tests, 0 failed, 0 ignored.
aaj3f
added a commit
that referenced
this pull request
Jul 13, 2026
…1473) Every SPARQL entry method re-lexed the query string: query_with_options built the within-ledger dataset (parse #1) and then lowered it (parse #2); the FROM-default single-ledger route parsed a third time inside the dataset impl to test for a dataset clause. That is 2x for all SPARQL and up to 3x on the FROM path. Parse + validate once, then reuse the AST for both dataset-clause resolution and IR lowering. helpers gains lower_sparql_ast (inject-prefixes-then-lower from an owned AST), inject_default_prefixes, and sparql_ast_has_dataset; parse_sparql_to_ir is now a thin parse-then-lower wrapper. build_within_ledger _dataset_from_ast resolves the clause from an already-parsed AST. The four SPARQL entry methods (query_with_options, query_tracked_with_options, query_tracked_with_r2rml_options, query_view_with_r2rml_options) parse once and thread the owned AST into the dataset path via new prepared entries on query_dataset_with_options / query_dataset_tracked_with_options (their public signatures, with many external callers, stay unchanged). Behavior is identical. The boxed-future boundary that breaks the mutual- recursion Send auto-trait cycle is preserved: only owned data (the AST) is threaded through it, never opaque async recursion. A cfg(test) parse counter plus helpers unit tests assert the single-parse guarantee by construction; the existing dataset/tracked/r2rml suites and the W3C testsuite cover behavioral equivalence. No register change.
aaj3f
added a commit
that referenced
this pull request
Jul 14, 2026
…1473) Every SPARQL entry method re-lexed the query string: query_with_options built the within-ledger dataset (parse #1) and then lowered it (parse #2); the FROM-default single-ledger route parsed a third time inside the dataset impl to test for a dataset clause. That is 2x for all SPARQL and up to 3x on the FROM path. Parse + validate once, then reuse the AST for both dataset-clause resolution and IR lowering. helpers gains lower_sparql_ast (inject-prefixes-then-lower from an owned AST), inject_default_prefixes, and sparql_ast_has_dataset; parse_sparql_to_ir is now a thin parse-then-lower wrapper. build_within_ledger _dataset_from_ast resolves the clause from an already-parsed AST. The four SPARQL entry methods (query_with_options, query_tracked_with_options, query_tracked_with_r2rml_options, query_view_with_r2rml_options) parse once and thread the owned AST into the dataset path via new prepared entries on query_dataset_with_options / query_dataset_tracked_with_options (their public signatures, with many external callers, stay unchanged). Behavior is identical. The boxed-future boundary that breaks the mutual- recursion Send auto-trait cycle is preserved: only owned data (the AST) is threaded through it, never opaque async recursion. A cfg(test) parse counter plus helpers unit tests assert the single-parse guarantee by construction; the existing dataset/tracked/r2rml suites and the W3C testsuite cover behavioral equivalence. No register change.
aaj3f
added a commit
that referenced
this pull request
Jul 17, 2026
…flag slice-1.5) The fused fact⋈dim join declined on ANY folded constant-object constraint (star_constraints) — the blanket O1 guard — so a dim flag like `?prod ex:isCurrent true` forced the whole join to materialize the fact. It is the round-2 deployed timeout for the two headline shapes (#1/#2: OrderLine/SupportTicket ⋈ Product with a dim isCurrent flag). E2 replaces the blanket decline with constraint APPLICATION: - dim-side constraints are enforced while building the FK→GKey map — a dim row failing its flag is skipped, so its join key never enters the map and fact rows probing it drop (inner-join + constraint semantics); - fact-side constraints are enforced per fact row in the fold (next_batch); - resolve_star_constraint_checks resolves each (predicate, constant) to a scalar column PredicateObjectMap and DECLINES (falls back to materialize) any constraint that is a RefObjectMap object or absent — never a silent over-count; - the constraint columns are projected into the respective scans; the equality reuses the normal scan's exact primitives (materialize_object_from_batch + object_column_is_numeric + rdf_term_eq_object_constant_cached, now pub(crate)), so the fused constrained count is byte-parity with the materialized answer. The single-table O1 decline is unchanged (slice-1.75). The COUNT(*) manifest shortcut gains an explicit fact_constraints guard (belt-and-suspenders; the join path always groups, so it never coincided). Hermetics: e2_resolve_star_constraint_checks_scalar_vs_ref (scalar admits, ref / missing declines), e2_row_satisfies_boolean_flag (isCurrent true/false/null under the normal-scan equality). Corpus q065 (Order⋈Customer, dim isCurrent flag, group by segment, FROM path): fused == materialized CONSTRAINED oracle, 0 hash mismatches — the join-path over-count trap. Live: fusion-OFF materialize 52s → fusion-ON E2 0.45s (~115x), same constrained count. fluree-db-query lib 1273 green; corpus 64->65; scoped clippy clean.
aaj3f
added a commit
that referenced
this pull request
Jul 31, 2026
Review #1529 fold-in #1 (id=3690216219). The O6 sub-2GB chunk-sizing rewrite resizes chunks for EVERY text import (Turtle/TriG/JSON-LD) whose memory budget is below 2048MB, not just materialize: at 512MB it goes 128MB -> ~51MB chunks (~2.5x the commit count, a different ledger shape). Name that shared-path blast radius at the site. Sizing only, never correctness. (The PR-body native-impact appendix is corrected separately.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello! I am excited to explore this project.
Following the readme to build locally, when I tried entering
make installI was met with
To which adding the aliases for the extra deps resolves (in the case of having clojure-1.10.1.645 installed)