Repository navigation
perf(gfql/cypher): multi-seed WHERE a.id IN [...] hop 4,004 ms → 10 ms (vectorized IN, index reaches the row pipeline, IN seeds the pattern, kernel takes a seed set) - #2117
Merged
Conversation
…ment Python loop The row pipeline evaluated `IN` by calling _gfql_cypher_value_equal for every (row, element) pair: 100k alias rows x 50 seeds = 5M Python steps, ~4 s, on a query whose answer the native is_in chain produces in 8 ms. A constant list of plain scalars now goes through Series.isin with the loop's three-valued rules (null row, null element, empty list, NaN as a value) re-derived as masks; lists and maps keep the structural-equality loop. The rules are pinned against the loop they replace in test_in_list_literal_lane.py. Found while verifying the multi-seed example for the index_adjacency docs (#2116). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…sident adjacency index The bindings builder tags the base edge frame with a per-edge identity column before hopping. That derivation keeps every row in place, but the new frame object missed the index registry's identity guard, so every hop under a Cypher query scanned while the same native hop was served. Migrate the adjacency indexes onto the identified frame (the chain executor's rebind_edges idiom); gfql_explain now reports the hop as `index` for int and string ids, and an edge frame the index was not built over still scans. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…TCH pattern A literal equality on a MATCH alias already lowers onto the pattern's filter dict; a literal list did not, so `WHERE a.id IN [...]` reached the executor as a post-join row filter and the middle `(a)-[e]->(b)` ran unseeded over every edge before the bindings path recomputed it. Each top-level AND conjunct of the form `alias.prop IN [scalars]` (node or edge alias, $param lists included) now becomes `is_in([...])` on the pattern and leaves the row WHERE. Lists holding null keep their three-valued evaluation; OR, NOT and nested lists stay put. With the vectorized IN lane and the index migration this takes the 50-seed hop on 100k nodes / 500k edges (pandas) from 4,004 ms on master to 20 ms with the indexes resident and 62 ms without. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…ed on the node-id column The kernel admitted one integral id as the seed and declined is_in / list seeds, so a Cypher `WHERE a.id IN [...]` (now lowered onto the pattern) still ran the unseeded middle before the bindings path recomputed it. A membership set of integral ids on the node-id key now resolves through the same node_id lookup; every other op keeps the scalar-only gate and the cost gates are unchanged, so a seed set covering most of the graph still scans. String ids keep declining (integer-key kernel) and are served by the indexed hop instead. 100k nodes / 500k edges, pandas, 50 seeds: 19.7 ms -> 10.5 ms with the indexes resident; gfql_explain reports connected_bindings as served. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
lmeyerov
changed the title
WHERE a.id IN [...] hop 4,004 ms → 20 ms (vectorized IN, index reaches the row pipeline, IN seeds the pattern)WHERE a.id IN [...] hop 4,004 ms → 10 ms (vectorized IN, index reaches the row pipeline, IN seeds the pattern, kernel takes a seed set)
…he element loop Validated on dgx (cuDF 26.02): cudf.Series([True, False]).isin([1]) is all False where pandas and the element loop say [True, False] (Python True == 1). A probe that mixes bool and non-bool, or whose bool-ness differs from the column's dtype, now falls back to the loop; bool against a bool column keeps the lane. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…ship.py lowering.py is under a line-count ratchet (cypher-frontend-surface-guard); the pass only needs a one-line hook there. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…age baseline The parser always hands the pass a PropertyAccessExpr and a ListLiteral, so the Identifier and parameter-list fallbacks were unreachable and are gone; the two remaining declines (a list held by another alias, an alias outside the pattern) are pinned. 97% under the pin files alone; floor 96. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
4 of 5 tasks
… pins are route_engaged
`datetime('...')` lowers to zoned ISO text before the membership pass saw it, so
`n.ts IN [datetime('...')]` was pushed as a string is_in and matched nothing where
the row path compares instants (test_b4_in_list_of_temporal_literals). The pass
now keeps zoned ISO text in the WHERE, exactly where `=` already keeps it.
The new served-by pins assert that a route serves, so they carry
@pytest.mark.route_engaged and skip in the gfql-routes-off replay instead of
reporting as divergences.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…e-add the IN bullet under [Development] after the 0.59.1 cut)
…bench #283) The docs policy caps graphistry/compute commit drift at 12 since a published run's measured commit; this branch adds 8 on top of master's 8. Instead of a waiver, the GraphBench 20k/100k and SNB boards were re-measured at 2d64913 under the published protocol and republished from pyg-bench (#283, same contract v3). Drift for the four re-measured runs is now 0; the bench-provenance directives name the new run ids. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
This was referenced Oct 3, 2026
Merged
…duces, across engines Peel: a bool element and a text/number mixed list stay in the WHERE (True == 1 and coercion differ between pandas and cuDF isin); an empty list stays too (NeverMatch has no wire form for the bindings op). Positive pins for float lists, duplicate ids, text ids; negative pins for each decline. End-to-end parity now runs on pandas, polars and cuDF. Kernel: parity on every engine with served-by asserted where the kernel is dispatched; absent seed ids are simply unmatched; a label beside the seed set still goes through. Lane: shape parity against the element loop on cudf.Series, and the bool-vs-int decline, when cuDF is runnable (probe exercises cupy: a CPU box can import cuDF and still not run the index kernels). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…hem in the polars lane polars is installed only in the test-polars lane, so a polars param must importorskip there (the repo idiom in test_indexed_bindings.py), and a module that mentions polars must be in bin/test-polars.sh's file list (test_polars_lane_completeness). cuDF params skip unless cupy can run, which is what the index kernels need. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
…efore indexing An index is built for the frame it sees; querying pandas-built indexes with engine='cudf' is a foreign-index decline by design (dgx run showed served=[] with parity intact). Build engine-native frames first, as test_indexed_bindings.py does. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
Contributor
Author
|
Real-GPU receipt (dgx-spark,
Cause (verified on my own pins, which failed identically and pass after the change): the graph is built from pandas frames and then run with |
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud
Contributor
Author
|
At 9731d49 two routes-off cells are red —
Each asserts |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Four composable changes found while chasing #2116 (the
index_adjacencydocs page's multi-seed example scanning instead of using the index). Measured on 100k nodes / 500k edges, pandas, 50 seeds,MATCH (a)-[e]->(b) WHERE a.id IN [...] RETURN b, local dev box, median of 7:37d7d9b6aINlane (commit 1)IN [literals]seeds the pattern (commit 3)row/pipeline.py::_gfql_eval_in_exprevaluatedx IN [...]with a Python loop over rows × list elements (_gfql_cypher_value_equal, 5M calls for 100k rows × 50 seeds). A constant list of plain scalars now goes throughSeries.isinplus two masks that reproduce the loop's three-valued table (null row, null element, empty list, NaN as a value). Lists/maps keep the structural-equality loop. Pinned against the loop it replaces on 22 dtype/null shapes._gfql_connected_bindings_statetags the edge frame with a per-edge identity column; the new frame object missed the registry's identity guard, so every hop under a Cypher query scanned while the same native hop was index-served. The adjacency indexes now migrate viarebind_edges(chain.py's idiom).gfql_explainreports the hop asindexfor int and string ids; a permuted edge frame the index was not built over still scans.WHERE alias.prop IN [scalar literals](node or edge alias,$paramlists) becomesis_in([...])on the pattern, the wayalias.prop = literalalready does, and leaves the row WHERE. Lists with null, and anything under OR/NOT/nested, stay aswhere_rows. Parity pinned on IN+AND, edge IN, OR, null lists, inline-eq ∧ IN hit/miss, params,RETURN a, b, OPTIONAL MATCH null row, int and string ids.Verified locally
bin/lint.sh(ruff + type-hygiene + comment-density guards),bin/ci/ci_cypher_surface_guard.py(the pass lives inwhere_membership.pysolowering.pystays under its line ratchet), mypy: clean.tests/compute/gfql/{cypher,index,row}+test_lowering.py(non-cudf): 3495 + 1443 passed; the cudf-parametrized and polars-conformance cases fail identically on master here (no GPU / local polars) — not touched.test_in_list_literal_lane.py(37),test_cypher_row_pipeline_reaches_index.py(3),test_in_list_seeds_the_pattern.py(12),test_indexed_bindings_membership_seed.py(12). Fulltests/compute/gfql/{cypher,index,row}+chain_specializations(non-cudf): 5273 passed.index/bindings.py): a membership set of integral ids on the node-id key is a seed, resolved through the samenode_idlookup as a scalar; the cost gates are unchanged (a seed set covering 90% of the graph still scans), string ids keep declining and take the indexed hop.gfql_explainreportsconnected_bindingsserved and the unseeded middle no longer runs. This is the piece GFQL: Cypher multi-seed hop (WHERE a.id IN [...]) does not use the resident adjacency index; single-seed does #2116 asked for.cuDF / polars validation (dgx-spark,
graphistry/test-rapids-official:26.02-gfql, cuDF 26.02, polars 1.35)cudf.Series: int, int+null, str+null, float NaN, empty list, only-null list all identical. One divergence found and closed:cudf.Series([True, False]).isin([1])is all-False where pandas and the loop sayTrue == 1; the lane now leaves any bool/non-bool mix to the loop (commit 5).test_lowering.pycudf+polars params and the index suites inside the image: 463 passed.nullkeeps the prefilter route, which polars rejects (NotImplementedError: polars engine cannot natively honour rows(alias_prefilters=...)) on master and here alike.Drift budget: re-measured, not waived
#2017 added
policy.max_compute_commit_drift = 12(graphistry/compute commits since a run's measured commit). Master sat at 8; this branch adds 8. The shipped GraphBench 20k/100k and SNB boards were therefore re-measured at this tree (2d64913b5) under the published protocol — pinned cpuset 5-9,15-19 / 20 polars threads / harness e6dbf8f8 for GraphBench; 3 process runs × {polars,pandas} × {SF0.1,SF1} on harness 684c0b409 for SNB, competitor arms carried byte-verified — and republished from pyg-bench (graphistry/pyg-bench#283). The vendored artifact's drift for those four runs is 0;docs/test_bench_numbers.py40 passed.Numbers vs the f283a30 board: GraphBench within run-to-run spread (100k polars q1 26.00 vs 25.74, q5 7.66 vs 7.57, q8 still the one LOSE cell); SNB every GFQL cell within ±2.5 % or faster (polars SF0.1 message-replies −18.7 %, tag-cooccurrence −20.5 %, new-topics −9.8 %). The export needed no
accepted_regressions. The 20k lane is slot-merged (7 published slots re-measured after a foreign campaign contended them;VALIDITY-MERGED.jsonin the artifact).Not in this PR
WHERE NOT (b.id IN [...])raises "AST evaluator unsupported" (noted on GFQL: Cypher multi-seed hop (WHERE a.id IN [...]) does not use the resident adjacency index; single-seed does #2116).🤖 Generated with Claude Code
https://claude.ai/code/session_017ropeBMLJUuy6ViYwy15ud