Repository navigation
perf(gfql): project count-path polars filters to id columns (round 4) - #1853
Conversation
Count-shaped consumers read only node_col/src/dst; polars filters become one fused lazy filter+select each, gathering only those columns. Expr is built against the full schema via the same validated builder (predicates may reference projected-away columns; typed error/NIE contract intact). pandas/cuDF keep full width. Pins: projection widths + value on the decoy fixture both engines, narrow-vs-full row parity, E302 parity through the narrow lane. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
Payload column named like a degree counter is projected away, so the fused distinct-domain lane serves (collision structurally impossible on a [src,dst] frame); a SRC binding carrying the reserved name survives projection and must still decline. Values identical on both sides. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
… prologue, deep pins Owner review on #1853, point by point: (1) project honored on every engine via one _filter_project helper (polars fused lazy; pandas/cudf pre-narrow to project+match keys, filter, cut -- same typed errors); (3) reserved-name hazard pinned per engine, payload and endpoint-binding shapes; (4) getattr prologues in both fast paths replaced with typed Plottable attribute access; (5) insight moved to the helper docstring, call sites reference it; (6) differential matrix engines x filter shapes (incl. error-type parity), cache-key separation pin, per-engine width pins. Hygiene guard back under baseline (fixes the red CI). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
|
Review rework pushed — point by point:
🤖 Generated with Claude Code |
… on pandas/cudf Owner review point 1 follow-up: WHEN to project must be static, so both arms are made monotone (never costlier than the plain filter). polars was already (fused lazy). pandas/cudf now build the boolean mask without materializing anything (filter_by_dict's mask extracted as filter_mask_by_dict -- same resolution, same typed errors, same 3VL membership) and gather .loc[mask, project]: identical mask work, strictly fewer gathered columns. The prior pre-narrow arm paid an upfront full-height copy that LOST on selective filters over narrow frames -- a data-dependent gamble, now gone. Admission rule documented in _filter_project: project iff the plan provably reads only those columns. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi
|
Re: when to project — the answer shipped in the follow-up commit: WHEN must be a static plan property, never a runtime cost call, and that's only sound if projection is monotone (never costlier than the plain filter). The first pandas/cudf arm wasn't: pre-narrowing paid an upfront full-height copy of the projected columns, which loses on selective filters over narrow frames (exactly the q8 edge shape). Now:
With both arms monotone, the admission rule is simply: a caller passes 🤖 Generated with Claude Code |
Round 4 of the q8 arc, thread-free. Every consumer in the two-hop count fast path reads only node_col/src/dst, so on polars the node/edge filters run as one fused lazy filter+select each instead of materializing full-width frames. The filter expr is built against the full schema via the same validated builder, so predicates may reference projected-away columns and the typed error/NIE contract is unchanged; pandas/cuDF keep full width.
Boundary finding the pins caught: a PAYLOAD edge column named like a reserved degree counter (
__in_count__) no longer forces the fused distinct-domain lane to decline — projection removes it, and a [src,dst] frame structurally cannot collide. A reserved SRC/DST binding survives projection and still declines. Both sides pinned with identical values.Tests: projection widths + value on the decoy-column fixture (both engines), narrow-vs-full row parity, E302 error parity through the narrow lane, both reserved-column sides. 1,643 green in the DGX container; mypy clean.
Receipted lane numbers follow once the shared DGX host frees up (the pending 100k rerun measures master; an r4 lane comes after).
🤖 Generated with Claude Code
https://claude.ai/code/session_01MF7uRZLKZaD6Q9FGWSmyXi