perf(gfql): use resident seed indexes in native Polars chains by lmeyerov · Pull Request #2038 · graphistry/pygraphistry

lmeyerov · 2026-09-05T01:37:08Z

Native Polars op-list queries with named nodes, scalar edge filters, or a filtered destination fell through to the full traversal even when resident indexes could resolve the seed. Single-node native queries also scanned their seed predicates.

Reuse the shared seeded reduction for these directed one-hop shapes and the shared node resolver for single-node queries. The seeded lane serves only when the resident node-id and adjacency indexes cover the hop (validated once, passed into the reduction), runs after the polars index-policy consult so index_policy off/force/use and the engine-mismatch diagnostics are unchanged, and records an indexed_traversal step so gfql_explain reports the engagement. Indexed gathers preserve input table order; native aliases match the full Polars path. Unsupported shapes, alias/property collisions, duplicate node ids, and graphs without a usable index keep the existing full-path behavior; scalar filters raise the same structured schema errors as the full path.

Stacked on #2037. Fixes #2033.

Validation (head cdbcc64): polars lane bin/test-polars.sh 8,238 passed (4 failures are the local libcublas/dask environment, identical on the base); index contract + narrow-combine + seed-resolution + seeded-fast-path suites 436 passed; 33 pins in the new test file (explain/index_trace engagement, policy-off and no-index declines with full-path parity, duplicate-id decline, schema-error parity). Ruff, changed-file mypy, comment-density, type-hygiene, Cypher surface guard and diff checks pass. DGX measurements follow from a dedicated frozen checkout. Found while amplifying: #2039 (polars full path raises on an edge alias colliding with its filtered edge column).

…index-resolved seeds Vendors pyg-bench published/docs-numbers.json at main 5cf81ae (byte-identical): the SF0.1 indexed GFQL cells come from the #2038 head with the pre-landing disclosure, SF1 stays the release measurement. The SNB prose states the native op-list arm, where the resident indexes now engage, the hop rows they do not change, and the open issues behind the remaining gaps. Release-notes Performance paragraph names the native seed resolution and the sentinel's native op-list points. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_014RhboAkmjP5eXbCCkX1BMt

lmeyerov · 2026-09-05T13:12:07Z

READY (479a421): 75/75 + RTD incl. tck-gfql; index-gated native polars seeded lane with engagement trace; start_nodes negative pin; real-GPU polars-gpu suites green. Stacked on #2037.

lmeyerov · 2026-09-05T16:57:06Z

READY (4a80847) — restacked on the new #2037 head after #2035 landed (patch-id identical to the previous head). This restack also drops a duplicated #2033 CHANGELOG bullet that an earlier union rebase had left (one bullet now).

Round-003 amplification (owner ask: reflect on the review flow before this lands) ran at this head: 51 shapes × pandas/polars/cuDF, default route vs the polars seed lane and the chain fast path forced to decline, plus the pandas full path as oracle. Polars cells: every served shape (_try_seeded_chain_polars engaged and the native_seeded_hop trace recorded) is value- and dtype-identical to the polars full path, including seed forms (None, absent, float, bool, label-only, two scalars, absent column), NULL-endpoint and dangling edges, string keys, categorical props, stale index, empty edge table, and declines for undirected / multi-hop / fixed-point / edge-first shapes. The only polars-vs-pandas differences (alias/column collisions rendered with _right suffixes; repeated node rows answered once per row) are identical on master and are not this lane.

Cost receipt (local box, medians of 7, ms; polars seeded typed hop [n({"id": k}, name="p"), e_forward({"type": ...}, name="e"), n(name="q")]): indexed 4.00 / 6.24 / 14.22 on master at 10k / 100k / 1M rows vs 1.88 / 1.65 / 1.74 here (flat in table size); unindexed 4.80 / 7.81 / 29.87 vs 4.88 / 8.14 / 26.26 (unchanged). Node lookup unchanged (~0.3 ms).

Receipts at 4a80847

Real GPU (dgx-spark, RAPIDS 26.02 image, 13 suites incl. test_chain.py): 1331 passed, 3 failed, 1 skipped (opt-in wall-clock gate), 2 xfailed. The 3 failures (test_chain.py:730 ×2, :1277) are identical on master in the same image: the cudf-26.02-only divergences tracked as gfql cuDF 26.02 (real GPU): chain default route and policy-forced full path disagree on prune_to_endpoints hops and duplicate node ids; tests use the removed cudf.DataFrame.from_pandas #2043, not this PR.
CI: 75/75 check-runs success (incl. tck-gfql, changed-line coverage, polars-lane completeness, the three hygiene guards).
pyg-bench: the sentinel's polars native points are this PR's lock-in; re-run on master after landing.

…exes and records its engagement The seeded shortcut ran before the polars index-policy consult and served un-indexed shapes without a trace step, so gfql_explain lost the policy off/force diagnostics and the engine-mismatch reason, the i1913 engagement pin saw no index step, and a named seeded shape skipped the collect-once combine. The lane now requires the resident node-id + adjacency indexes, runs after the consult, and records an indexed_traversal step. Structured schema errors and duplicate-id declines from the amplification pass are kept and pinned. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_014RhboAkmjP5eXbCCkX1BMt

…olars seed lane Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA

Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA

lmeyerov · 2026-09-05T18:55:27Z

CI at the rebased head a3991bb (on master 4c5e252 after #2037 landed): 77/77 check-runs success (tck-gfql, changed-line coverage, polars lane, hygiene guards included). Patch-id identical to the previous head 4a80847, so the GPU receipt (1331 passed, 3 failures identical to master, #2043) and the round-003 parity sweep carry over. Ready to merge; #2040 auto-retargets to master afterwards and I rebase it.

…index-resolved seeds Vendors pyg-bench published/docs-numbers.json at main 5cf81ae (byte-identical): the SF0.1 indexed GFQL cells come from the #2038 head with the pre-landing disclosure, SF1 stays the release measurement. The SNB prose states the native op-list arm, where the resident indexes now engage, the hop rows they do not change, and the open issues behind the remaining gaps. Release-notes Performance paragraph names the native seed resolution and the sentinel's native op-list points. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_014RhboAkmjP5eXbCCkX1BMt

lmeyerov mentioned this pull request Sep 5, 2026

gfql polars: an edge alias that collides with the edge column its own filter uses raises incompatible-column-type (pandas serves it) #2039

Open

lmeyerov mentioned this pull request Sep 5, 2026

perf(gfql): undirected k-hop chain re-runs the seed-rediscovery rule and the endpoint-resolvability filter on every pass #2026

Closed

lmeyerov mentioned this pull request Sep 5, 2026

perf(gfql): eager native seed lanes record their index engagement in gfql_explain #2040

Merged

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from 3a90313 to 47da761 Compare September 5, 2026 08:03

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from daa7348 to 994f4bd Compare September 5, 2026 08:03

lmeyerov mentioned this pull request Sep 5, 2026

gfql cuDF 26.02 (real GPU): chain default route and policy-forced full path disagree on prune_to_endpoints hops and duplicate node ids; tests use the removed cudf.DataFrame.from_pandas #2043

Closed

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 994f4bd to 479a421 Compare September 5, 2026 08:36

This was referenced Sep 5, 2026

perf(gfql): serve seeded node lookups and two-alias projections on the fast paths #2035

Merged

gfql: rows(table=nodes, source=alias) multiplies rows for duplicate node ids and joins null ids to each other #2034

Open

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from 9b190d2 to 2eeaf99 Compare September 5, 2026 13:28

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 479a421 to 530afa4 Compare September 5, 2026 13:28

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from 2eeaf99 to bf9acb7 Compare September 5, 2026 13:29

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 530afa4 to 059b2a8 Compare September 5, 2026 13:29

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from bf9acb7 to 14be070 Compare September 5, 2026 13:36

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 059b2a8 to bf46a8a Compare September 5, 2026 13:37

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from 14be070 to 7f4e7a7 Compare September 5, 2026 13:42

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from bf46a8a to 86b7017 Compare September 5, 2026 13:42

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from 7f4e7a7 to e08a059 Compare September 5, 2026 13:44

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 86b7017 to 88ff625 Compare September 5, 2026 13:44

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from e08a059 to ab3a32e Compare September 5, 2026 14:23

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 88ff625 to 6312e18 Compare September 5, 2026 14:23

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from ab3a32e to ff818e1 Compare September 5, 2026 15:42

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 6312e18 to 0664c3a Compare September 5, 2026 15:42

lmeyerov force-pushed the perf/gfql-native-seed-resolution branch from ff818e1 to a45374b Compare September 5, 2026 16:33

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 0664c3a to 4a80847 Compare September 5, 2026 16:33

Base automatically changed from perf/gfql-native-seed-resolution to master September 5, 2026 18:29

lmeyerov and others added 4 commits September 5, 2026 11:33

perf(gfql): resolve native Polars chain seeds through resident indexes

65ff05b

test(gfql): a seeded chain with start_nodes never enters the native p…

56a2f03

…olars seed lane Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA

docs(changelog): drop the duplicated #2033 bullet left by a union rebase

a3991bb

Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01QztW7jYsDd66e8rb8pJNQA

lmeyerov force-pushed the perf/gfql-polars-native-seed-resolution branch from 4a80847 to a3991bb Compare September 5, 2026 18:33

lmeyerov merged commit 8b208a6 into master Sep 5, 2026
78 checks passed

lmeyerov deleted the perf/gfql-polars-native-seed-resolution branch September 5, 2026 19:12

lmeyerov mentioned this pull request Sep 5, 2026

perf(gfql): undirected hop skips the whole-frame dedup of the doubled edge frame (#2026) #2036

Merged

Navigation Menu

Repository navigation

perf(gfql): use resident seed indexes in native Polars chains - #2038

perf(gfql): use resident seed indexes in native Polars chains#2038
lmeyerov merged 4 commits into
masterfrom
perf/gfql-polars-native-seed-resolution

lmeyerov commented Sep 5, 2026 •

edited

Loading

Uh oh!

lmeyerov commented Sep 5, 2026

Uh oh!

lmeyerov commented Sep 5, 2026

Uh oh!

lmeyerov commented Sep 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

lmeyerov commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lmeyerov commented Sep 5, 2026

Uh oh!

lmeyerov commented Sep 5, 2026

Uh oh!

lmeyerov commented Sep 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

lmeyerov commented Sep 5, 2026 •

edited

Loading