Skip to content

perf(gfql): project count-path polars filters to id columns (round 4) - #1853

Merged
lmeyerov merged 5 commits into
masterfrom
perf/gfql-q8-proj-r4
Aug 6, 2026
Merged

lmeyerov merged 5 commits into
masterfrom
perf/gfql-q8-proj-r4

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Round 4 of the q8 arc, thread-free. Every consumer in the two-hop count fast path reads only node_col/src/dst, so on polars the node/edge filters run as one fused lazy filter+select each instead of materializing full-width frames. The filter expr is built against the full schema via the same validated builder, so predicates may reference projected-away columns and the typed error/NIE contract is unchanged; pandas/cuDF keep full width.

Boundary finding the pins caught: a PAYLOAD edge column named like a reserved degree counter (__in_count__) no longer forces the fused distinct-domain lane to decline — projection removes it, and a [src,dst] frame structurally cannot collide. A reserved SRC/DST binding survives projection and still declines. Both sides pinned with identical values.

Tests: projection widths + value on the decoy-column fixture (both engines), narrow-vs-full row parity, E302 error parity through the narrow lane, both reserved-column sides. 1,643 green in the DGX container; mypy clean.

Receipted lane numbers follow once the shared DGX host frees up (the pending 100k rerun measures master; an r4 lane comes after).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi

lmeyerov and others added 4 commits August 5, 2026 15:48
Count-shaped consumers read only node_col/src/dst; polars filters become
one fused lazy filter+select each, gathering only those columns. Expr is
built against the full schema via the same validated builder (predicates
may reference projected-away columns; typed error/NIE contract intact).
pandas/cuDF keep full width. Pins: projection widths + value on the
decoy fixture both engines, narrow-vs-full row parity, E302 parity
through the narrow lane.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Payload column named like a degree counter is projected away, so the
fused distinct-domain lane serves (collision structurally impossible on
a [src,dst] frame); a SRC binding carrying the reserved name survives
projection and must still decline. Values identical on both sides.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
… prologue, deep pins

Owner review on #1853, point by point: (1) project honored on every
engine via one _filter_project helper (polars fused lazy; pandas/cudf
pre-narrow to project+match keys, filter, cut -- same typed errors);
(3) reserved-name hazard pinned per engine, payload and endpoint-binding
shapes; (4) getattr prologues in both fast paths replaced with typed
Plottable attribute access; (5) insight moved to the helper docstring,
call sites reference it; (6) differential matrix engines x filter shapes
(incl. error-type parity), cache-key separation pin, per-engine width
pins. Hygiene guard back under baseline (fixes the red CI).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov

lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Review rework pushed — point by point:

  1. polars-only projection → generalized: one _filter_project(frame, match, project, engine=...) serves every engine. polars keeps the fused lazy filter+select; pandas/cudf pre-narrow to project + the match keys present on the frame, filter, then cut to project — missing match keys stay missing so filter_by_dict raises its usual typed error. Same at-most-these-columns contract everywhere.
  2. CI red → it was the type-hygiene ratchet (my new helper's bare dict annotation, 23>22). Fixed (Dict[str, Any]); guard green locally; CI re-dispatched.
  3. alias issue on other engines → yes, the hazard class existed beyond polars. Generalized projection removes the payload-column shape uniformly (a __in_count__ data column never reaches any counting path now); the endpoint-binding shape (src/dst literally named like a counter) survives projection by design and is value-pinned per engine — pandas/cudf/polars all return the degree-product oracle on both shapes.
  4. getattr prologue → replaced with typed Plottable attribute access (base_graph._nodes etc. are declared on the protocol); the identical prologue existed in a sibling fast path and got the same fix. mypy clean, no new casts.
  5. insight placement → the count-shape⇒id-columns knowledge now lives in _filter_project's docstring with call sites referencing it. The general home is logical-plan column pruning (compute per-op column needs once in the IR so every path inherits narrow scans, subsuming per-call-site project=) — filed as question 9 in the typed-ontology design doc for your call, since it's planner work beyond this PR.
  6. shallow tests → added: a differential matrix (pandas/polars/cudf × 7 filter shapes incl. predicate-on-decoy, missing-column and E302 error paths — narrow must equal full-cut-to-columns, and both arms must raise the same exception type), per-engine projection-width pins, per-engine reserved-name value pins (both shapes), and a cache-key separation pin (same match, different projection, one store → two entries, no cross-contamination). 1,692 green in the DGX container including the cudf and polars-gpu arms.

🤖 Generated with Claude Code

… on pandas/cudf

Owner review point 1 follow-up: WHEN to project must be static, so both
arms are made monotone (never costlier than the plain filter). polars
was already (fused lazy). pandas/cudf now build the boolean mask without
materializing anything (filter_by_dict's mask extracted as
filter_mask_by_dict -- same resolution, same typed errors, same 3VL
membership) and gather .loc[mask, project]: identical mask work,
strictly fewer gathered columns. The prior pre-narrow arm paid an
upfront full-height copy that LOST on selective filters over narrow
frames -- a data-dependent gamble, now gone. Admission rule documented
in _filter_project: project iff the plan provably reads only those
columns.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
@lmeyerov

lmeyerov commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Re: when to project — the answer shipped in the follow-up commit: WHEN must be a static plan property, never a runtime cost call, and that's only sound if projection is monotone (never costlier than the plain filter). The first pandas/cudf arm wasn't: pre-narrowing paid an upfront full-height copy of the projected columns, which loses on selective filters over narrow frames (exactly the q8 edge shape). Now:

  • polars: fused lazy filter+select (was already monotone).
  • pandas/cudf: filter_by_dict's mask logic is extracted as filter_mask_by_dict (same column resolution, same typed errors, same 3VL membership — filter_by_dict itself now composes from it), and the narrow arm does .loc[mask, project]: identical mask work, strictly fewer gathered columns.

With both arms monotone, the admission rule is simply: a caller passes project iff its plan provably reads only those columns — documented in _filter_project's docstring. The count path qualifies today; grouped-aggregate paths could declare ids + referenced props next; and the general home (per-op column requirements computed once during logical-plan compilation, so every path and the generic executor inherit narrow scans) is filed as typed-ontology question 9 for your call.

🤖 Generated with Claude Code

@lmeyerov
lmeyerov merged commit d888ba7 into master Aug 6, 2026
69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant