Skip to content

perf(gfql): serve seeded node lookups and two-alias projections on the fast paths - #2035

Merged
lmeyerov merged 8 commits into
masterfrom
perf/gfql-seeded-lookup-fastpaths
Sep 5, 2026
Merged

lmeyerov merged 8 commits into
masterfrom
perf/gfql-seeded-lookup-fastpaths

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Stacked on #2031 (contract pins) → #2030 (dtype memo). Closes the two fast-path coverage gaps behind the SNB point-lookup regression (plan T2.10.2).

What changes

  • New seeded_node_lookup fast path: MATCH (p {id}) RETURN p / RETURN p.a AS x, p.b resolves the seed through the resident node-id index, a resident node-property index (gfql_index_node_props), or one scalar scan, and projects the matched rows directly. Before, this shape ran the full two-pass chain over the node table with no index consulted (the alias on the node op disqualified the native single-op shortcut).
  • Seeded typed hop: a bag RETURN may project from both pattern aliases (RETURN m.id, p.firstName, the LDBC IS1 form), one row per matched edge, built from the seed and destination rows the reduction already holds. Before, it fell to the row pipeline's connected-bindings frame builder.
  • Shared seed resolution (chain_fast_paths._seed_node_rows): node-id index → property index → scan, with the ORIGINAL filter dict re-applied through the canonical filter (filter_by_dict / filter_by_dict_polars) so typed errors, temporal and mixed-type comparisons match the full path exactly. The seeded hop now also benefits from a property-indexed seed.
  • Dtype parity rule for the lean projections: the pandas rows-pivot upcast (int/float → float64, bool → object) is applied only where the canonical path pivots — destination alias only, empty results always, and never when the indexed connected-bindings kernel would have served (_indexed_kernel_admits mirrors its seed admission and frontier/gather cost gates).

Evidence (SNB SF0.1 fixture, local box, not published; pyg-bench sentinel #245): pandas IS1 official 19.1 → 4.0 ms, node-only lookup 13.7 → 1.2 ms; polars IS1 7.9 → 5.7 ms; cuDF IS1 78 → 39 ms; every basic shape now reports fast_path served on pandas, polars and cuDF. Contract pins in test_gfql_latency_contract.py flip from strict-xfail to pass.

Tests: test_seeded_node_lookup_fastpath.py (lookup shapes × scan/indexed × 3 engines, independent oracle, property-index engagement with the node_lookup trace seam, out-of-shape declines, policy decline, duplicate-id rows, two-alias shapes incl. int/bool from both aliases, no-match dtypes, non-binding indexed seeds, bag multiplicity, multi-seed, hub seed over the frontier gate); cross-alias pins in test_seeded_typed_hop_fastpath.py flip from decline to engage-with-parity; engagement map extended; test_use_policy_sparse_serves_dense_declines moved to a 2-hop shape. Full graphistry/tests/compute/gfql sweep on pandas/polars/cuDF: 11,896 passed, 5 failures = pre-existing local libcublas.so.12 import errors (identical on master). Guards (ruff, mypy, comment density, type hygiene, Cypher surface) green.

Found on the way: #2034 (full path multiplies rows for duplicate node ids and joins null ids), #2033 (polars native chain skips the seeded fast path).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WwMmVFo44ADiRRj5cxh1i1

Base automatically changed from test/gfql-latency-contract to master September 5, 2026 05:45
lmeyerov and others added 2 commits September 5, 2026 01:02
…e fast paths

- seeded_node_lookup fast path: MATCH (p {id}) RETURN p / RETURN p.a, p.b resolves the
  seed via the resident node-id index, a resident node-property index, or one scalar
  scan and projects the matched rows directly (pandas, polars, cuDF)
- seeded typed hop: RETURN may project from both pattern aliases (IS1 shape), one row
  per matched edge, built from the seed and destination rows the reduction holds
- shared seed resolution (_seed_node_rows) so property-indexed seeds engage on the
  seeded hop as well
- pins: engagement map, cross-alias cases flip from decline to engage, contract gaps
  cleared, new test_seeded_node_lookup_fastpath.py (parity, oracle, index, declines)

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01WwMmVFo44ADiRRj5cxh7i1
…; gfql_explain no longer reports a served fast path as an index

Negative pins: whole-row RETURN with a suffix, expression items, mixed
whole-row and property items, LazyFrame nodes, a requested engine that
is not the frames' engine, and a destination-side extension dtype all
decline to the full path with parity; seed-side extension and datetime
columns keep parity; the scan reason (index_missing / served /
index_stale) and policy off are visible in gfql_explain. Under
index_policy="off" a fast path may still serve, so gfql_explain now
derives used_index and the decision from index steps only and reports
policy_off.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Ready for review at f8f2e73 (rebased on master after #2024/#2028; 77/77 checks green incl. tck-gfql, RTD green).

Hardening per the review bar (audit table in the private plan, reviews/pygraphistry-2035-hardening/audit.md):

  • Every admission check in _execute_seeded_node_lookup_fast_path and the two-alias projection now has a pin on both sides: whole-row RETURN with a suffix, expression items, mixed whole-row + property items, LazyFrame nodes, requested engine ≠ frames' engine, and a destination-side extension dtype all decline to the full path with parity; seed-side extension and datetime columns keep parity; the scan reason (index_missing / served / index_stale) is visible in gfql_explain.
  • One finding fixed here (explain.py): under index_policy="off" a served Cypher fast path was reported as used_index: True / index_selected; gfql_explain now derives used_index and the decision from index steps only and reports policy_off. Pinned on pandas / polars / cuDF.
  • Cross-platform: dgx-spark, RAPIDS 26.02 image (cudf 26.02.01), lookup + typed-hop + engagement + latency-contract + index + indexed-bindings + polars-gpu suites: 727 passed / 0 failed / 1 opt-in skip. mypy clean on the changed file; no new Any/ignore/cast.
  • pyg-bench lock-ins: the SNB point-latency sentinel (is1/is5/seeded-hop/node-only on pandas + polars, < 10 ms absolute, drift-gated vs the committed baseline that contains this PR) and the docs-numbers acceptance gate.

Suggested order after this: #2036 next (green; it is what keeps the #2028 scaling pin stable — a loaded runner just measured 30.6x vs the 30x bound on a branch without it), then #2037 → #2038 → #2040.

@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

READY (f8f2e73): 77/77 checks + RTD; boundary audit above; real-GPU 727/0/1 opt-in skip; mypy clean; sentinel + acceptance gate lock-ins.

…hop, not only its endpoints

Edge-alias properties now ride the per-matched-edge frame with the seed and
destination lookups, keeping their source dtypes like the full path; a
projection that touches any non-destination alias takes that frame, so a
no-match seed with an edge property projects an empty frame instead of
reading node columns. Pins: all three aliases, edge-only incl. endpoints,
twelve properties across three aliases, parallel edges, no match; two hops
and an absent edge property decline with parity.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Fair challenge on "two projections". In the code, "two" was never a projection count: it meant the two node endpoints of the single hop, and the path already took any number of properties from either endpoint (an eight-property RETURN is served). The benchmark-shaped gap was real though: an edge-alias property (RETURN r.since) declined to the full path, and that is the natural n. Pushed in e2db9cb:

  • the seeded typed-hop fast path now projects any number of properties from any alias of the hop (seed node, edge, destination node), one row per matched edge; edge properties keep their source dtypes like the full path;
  • a projection that touches any non-destination alias takes the per-edge frame, so a no-match seed with an edge property projects an empty frame instead of reading node columns (a bug the new pins caught);
  • pins: all three aliases, edge-only including endpoints, twelve properties across three aliases, parallel edges, no match; two hops and an absent edge property decline with parity;
  • CHANGELOG wording now states the general contract rather than the IS1 shape.

What still declines by construction: anything beyond one hop (a different fast path's job) and anything with side channels (policy, WHERE, OPTIONAL). Local: fast-path + index + latency + lowering suites 548 + 1765 passed (the two cuDF t6 lowering failures are the dev-box libcublas environment, identical on master); ruff/guards/mypy clean, no new Any/ignore/cast. Real-GPU receipt for the new shapes follows on this PR. #2037/#2038/#2040 are restacked on this head.

lmeyerov and others added 3 commits September 5, 2026 06:36
…alias marker

The full path's chain output puts the edge alias' boolean marker first on
the edge frame; the seeded projection now emits the same empty frame. The
connected-bindings kernel parity test disables the seeded fast paths so the
kernel stays the route under test.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
…w a column of their frame

An edge alias named like an edge column (or a node alias named like a node
column) keeps the full path's collision contract instead of the fast path
inserting a marker over the column. Pins on pandas/polars/cuDF for edge
and node collisions, projected or not, and the whole-row RETURN.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
… too

The guard now sits after the shape checks and before the projection
branches, so a colliding alias declines on every RETURN form.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Scenario review (include/exclude, per your note; matrix in the private plan reviews/pygraphistry-2035-hardening/scenarios.md): 16 shapes probed for served-or-declined plus fast-vs-full parity.

Real-GPU at 37ada01: 766 passed / 0 failed; rerun on 7e3f983 in flight. #2037/#2038/#2040 restacked on this head.

@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Real-GPU receipt at 7e3f983 (dgx-spark, RAPIDS 26.02 image, cudf 26.02.01 / polars 1.35.2): lookup + typed-hop + engagement + latency-contract + index + indexed-bindings + polars-gpu suites = 784 passed, 0 failed, 1 opt-in skip. CI on this head and the restacked #2037/#2038/#2040 is running.

@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

READY (7e3f983): 77/77 checks + RTD; real-GPU 784 passed / 0 failed; n-alias generalization, alias/column collision declines, scenario matrix (#2048 for the two include candidates). #2037 e08a059 / #2038 88ff625 / #2040 c4d2e9a restacked on this head, 75/75 each.

…endpoint bindings are not edge properties

A random-shape differential (aliases, property subsets from every side,
seed hit or miss, suffixes) found that projecting an edge's endpoint
binding served the ids where the full path resolves an absent property;
the fast path now declines those items. The latency contract gains the
all-aliases shape against a two-lookup plain-frame floor.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Quality pass per the review checklist, landed as aad106c:

  • Error amplification: a random-shape differential (test_random_projection_shapes_match_the_full_path: random aliases, property subsets from seed/edge/destination, seed hit or miss, ORDER BY/LIMIT suffixes, indexed or not, 12 seeds × 3 engines) found one more real divergence: projecting an edge's endpoint binding (r.src) served the ids where the full path resolves an absent property (null). The fast path now declines those items (pinned). On polars, shapes the full path declares unsupported (NotImplementedError for the row-op select) are held to parity-or-error, and served shapes are checked against the pandas full path as the oracle.
  • pyg-bench / contract: the latency contract gains the all-aliases shape (m.lastName, r.type, p.firstName) against a faithful two-lookup plain-frame floor: pandas 1.1×, polars 2.5×, cuDF 4.4× locally. The sentinel's IS1/IS5/hop/node-only routes are unchanged by the later commits, so the cdbcc64 baseline stands; the master re-measure covers the cells.
  • DRY: the select-item admission loop is duplicated between the two fast paths; noted on perf(gfql): seeded fast paths could admit single-alias WHERE predicates and LIMIT-only whole-row RETURNs #2048 to fold with the WHERE-pushdown work rather than churn this PR again.

Suites: lookup + typed-hop + indexed-bindings + engagement + latency-contract + index + alias-scoping = 825 passed locally; guards/mypy clean. #2037 ab3a32e / #2038 6312e18 / #2040 ab74105 restacked. Real-GPU run of aad106c in flight; CI running.

@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

Real-GPU receipt at aad106c (dgx-spark, RAPIDS 26.02 image, cudf 26.02.01 / polars 1.35.2): lookup + typed-hop + engagement + latency-contract + index + indexed-bindings + polars-gpu suites = 829 passed, 0 failed, 1 opt-in skip.

@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

READY (aad106c): 77/77 checks + RTD; real-GPU 829 passed / 0 failed; random-shape differential + curated boundary pins on 3 engines; latency-contract pin for the all-aliases shape; #2037 ab3a32e / #2038 6312e18 / #2040 ab74105 restacked, 75/75 each.

…as categorical rows

Round-002 amplification of the seeded fast paths found two dtype
divergences against the full Cypher path:

- an empty projection kept the node id column at int64 where the full
  path widens it to float64 (no-match seed, destination label mismatch)
- a whole-row RETURN on pandas frames with a categorical column kept the
  neighbouring bool/int columns narrow where the full path widens them;
  the typed-hop path now declines those frames (polars/cuDF keep parity
  and stay on the fast path)

Pins: two empty shapes in the two-alias parity matrix and a
categorical matrix over pandas/polars/cudf x scan/indexed x four shapes.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA
@lmeyerov

lmeyerov commented Sep 5, 2026

Copy link
Copy Markdown
Contributor Author

READY + MERGING (7db7968) — owner-authorized after the round-002 test-amplification pass.

Round-002 (timeboxed 1–1.5 h, hypotheses not pre-constrained): a 90-cell differential matrix (30 shapes × pandas/polars/cuDF, default route vs fast paths disabled vs pandas oracle) over 13 hypothesis families (categorical / string / nullable dtypes, stale indexes, $param seeds, naming collisions, WITH/UNION/OPTIONAL/count/DISTINCT composition, parallel-edge multiplicity, policy force/decline, whole-row variants, seed value types, empty results). Found 2 new classes, both fixed here with pins:

  • an empty projection kept the node id column at int64 where the full path widens it to float64 (no-match seed, destination label mismatch; pandas + cuDF)
  • a whole-row RETURN on pandas next to a categorical column kept neighbouring bool/int columns narrow; the typed-hop lane now declines pandas extension-dtype frames (polars/cuDF keep parity and stay served)

Post-fix matrix: 0 DIFF / 81 parity / 9 both-raise (same exception class on both routes). Artifacts and probe outputs in the plan's rounds/round-002/.

Receipts at 7db7968

  • CI: 77/77 check-runs success, incl. tck-gfql, changed-line coverage, polars-lane completeness and the three hygiene guards.
  • Real GPU (dgx-spark, graphistry/test-rapids-official:26.02-gfql-polars, cudf 26.02.01 / cupy 13.6.0 / polars 1.35.2, safe_run): 865 passed, 1 skipped (opt-in wall-clock gate), 2 xfailed (recorded gaps) — +36 cells over the previous head's 829 with the same skip/xfail set.
  • Local broad GFQL suite (graphistry/tests/compute/gfql + chain/hop): 12152 passed, 0 failed.
  • mypy on the changed product files: no new errors vs master; no new Any / type: ignore / typing cast in product code.
  • pyg-bench sentinel (snb_point_latency, SF0.1, dgx-spark release runner, pyg-bench main 5cf81ae, run snb-sentinel-7db7968f6-r1): every Cypher point served and within the gate vs the 2026-09-04 baseline (pandas IS1 1.97 ms vs 1.97; IS5 1.07 vs 1.05; seeded-hop props 1.03 vs 1.02; entity 3.03 vs 3.01; node-only props 0.59 vs 0.58; entity 2.47 vs 2.49; polars 1.7–5.3 ms, all ≤ baseline except node-only-entity 2.74 vs 2.26, inside the 1.5× gate). The three native op-list points read as regressions only because the baseline was measured at the perf(gfql): use resident seed indexes in native Polars chains #2038 head, which stacks perf(gfql): native chain seeds resolve through the resident indexes; named patterns served (#2027) #2037/perf(gfql): use resident seed indexes in native Polars chains #2038 on this PR; those lanes do not exist at this head and land next, after which the sentinel is re-run on master.

Restacked on this head with identical patch-ids: #2037 ff818e1, #2038 0664c3a (also drops a duplicated #2033 CHANGELOG bullet a union rebase had left), #2040 682c072.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant