Repository navigation
perf(gfql): row-evaluator text sniffers decide on a sample before scanning a column (#2116 3c) - #2128
Conversation
…nning a column (3c) A string equality on a 100k-row object column paid ~670 ms regex-fullmatching every value against eight temporal patterns plus ~240 ms in the list-like and mapping-like probes, for both operands. Each probe is an all-rows conjunction, so a failing 16-value sample settles it exactly; the full scan runs only when the sample passes. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp
74a5e3d to
a47a332
Compare
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp
Read-only review (parallel session) — sample-first sniffers, head 853ec1aNothing here touches the branch; findings only, fixes deferred. What it changes: Correctness check (by reading the full functions on the head, not just the diff):
Tests ( Findings
Recommendation: merge once CI is green; nothing blocking found. 🤖 Generated with Claude Code |
…elog states the one changed answer Review found the entry filed under the shipped 0.59.1 section, the temporal sniffer still rendering the whole column before sampling it, the list and mapping probes still rendering columns their sample already rules out, and the "identical answers" claim false for one shape. Rendering is the cost, so each probe now renders its sample first: on 100k rows an object column of timestamps goes from 331 ms to 6 ms for the temporal check and 230 ms to 6 ms for the list check. Answers match master on 30 adversarial shapes; the one correction -- a column mixing arrays with list-looking text is no longer read as a list column -- is pinned and stated in the changelog, now under Development. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp
…ers-early-negative-3c # Conflicts: # CHANGELOG.md
|
Your three suggestions plus my own pass found six items; all six are fixed at 0c741d0, and two of them were larger than reported. The "identical answers" claim was false, and the CHANGELOG entry was in a shipped release. Both confirmed. The entry has moved from Master answered True only because Sample-first was incomplete, and on the expensive half. The temporal sniffer still ran
No cuDF pin for a List and mapping engagement rested on timing. Now mechanism-pinned the way the temporal one was: Bench attribution. Standing: the only arm with an A/A control is the combined 3a+3c run (1908.8 ms against master's 3297.8 and 3244.8, A/A spread 1.00 to 1.03), and the threshold is a master times 1.5 regression bound rather than an improvement pin. I am not claiming an isolated 3c number from it; the four local before/after pairs above are what this PR stands on. Also fixed from your list: the comment run the density guard flagged in |
… receipt measured it My cuDF parametrization asserted pandas' answer for a mapping column. cuDF holds mappings as a struct column and cudf 26.02 cannot render a struct to text, so the temporal and list probes raise there; master raises identically. The divergence is now its own pin instead of a shared row that could never pass. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp
|
GPU receipt at the current head. dgx-spark, GB10, The first pass at 0c741d0 failed one cell, and it was my own test's fault rather than the code's: the cuDF parametrization asserted pandas' answer for a mapping column. Probing a master-tip tree and this head side by side on the same GPU gives the identical result on both: cuDF holds mappings as a struct column and cudf 26.02 cannot render a struct to text, so the text probes raise there on master too. The shared row could never pass, so at c9e75d5 it is replaced by a per-engine pin of the real divergence: pandas renders and declines, cuDF raises, and the mapping probe answers True on both. |
…ers-early-negative-3c
…ers-early-negative-3c # Conflicts: # CHANGELOG.md
Summary
#2116 item 3c reported
cuDF IN ... ORat ~670 ms. Profiled before touching anything (20k nodes / 100k edges, pandas, warmed medians, local = direction only): the cost is not cuDF-specific and not theIN.WHERE a.id IN [50 ids] OR b.kind = 'zzz'takes 545 ms on pandas and 402 ms on cuDF (plainIN26 / 55 ms): the cross-aliasORkeeps the predicate out of the alias prefilter, and then the string equalityb.kind = 'zzz'paysorder_detect_temporal_mode— eight regex full-matches over every value of a 100k-row object column (1.6Mfullmatchcalls), and_gfql_series_is_list_like/_gfql_series_is_mapping_like(astype(str)+ regex over all rows),for both operands (the broadcast scalar too), before the compare itself (
INis 26 ms of it).All three probes are all-rows conjunctions (
.all()), so a failing 16-value sample settles them exactly; the full scan runs only when the sample passes. For the list/mapping probes, a sampled non-nullstrvalue is already forcedFalseby the probe's ownactual_stringrule, so it is an immediateFalse.Measured on 100k rows: temporal 151 ms → 5.5 ms, list-like 34 → 2.7 ms, mapping-like 34 → 2.7 ms per probe; a 100k all-dates column is still detected (
date, 33 vs 34 ms — the full scan still runs when the sample passes). Old-vs-new equivalence: 176 series (temporal text, constructors, list/map text, real lists/maps, words, nulls, mixed families, failures only past the sample, native dtypes) — 0 mismatches.Not in this PR: the unseeded
MATCH (a)-[e]->(b) RETURN b LIMIT 5(81 ms vs 3.6 ms native) is the full 100k-row binding table built beforeLIMIT— a pushdown/lazy-table design item, filed separately.Test plan
graphistry/tests/compute/gfql/row/test_sniffers_sample_first_3c.py(18): answers preserved per family, failure past the sample still counted, native dtypes, and a counting pin that a 100k string column sniffs on ≤16 valuesbin/lint.sh, mypy clean🤖 Generated with Claude Code
https://claude.ai/code/session_012Me1E7ZdDuGqJGu3mMEzhp