Repository navigation
fix(gfql): show_indexes reports engine usability, not just fingerprint validity - #1841
Merged
Merged
Conversation
…t validity show_indexes() claimed valid=True for an index the resolved query engine cannot use. Indexes are engine-specific, so a fingerprint-fresh index built for another engine (e.g. polars-built, inspected on a graph whose AUTO queries resolve to pandas) silently declines to a scan at query time while the inspection surface said everything was fine. This is the follow-up proposed in #1767's disposition comment ("stop show_indexes() reporting valid=True for an index the resolved query engine cannot use"). #1838 made the decline loud in gfql_explain via _engine_mismatch_reason; this completes the pair on show_indexes by reusing the same mechanism and wording rather than inventing a parallel one. - valid keeps its fingerprint-only meaning (backward compatible) - new columns: query_engine (what engine= resolves to for THIS graph, the same resolution a query makes; default 'auto'), usable (fresh AND engine-matched), reason (shared #1838 mismatch wording + stale reason) - show_indexes(engine=...) previews an explicit engine choice; SHOW GFQL INDEXES respects the gfql(engine=...) of the call; gfql_explain passes its engine through - reporting-only: no query routing changes (#1743 AUTO-routing stays held) Tests: 9 new (polars index vs AUTO/pandas query not-usable with exact reason, matching-engine usable, explicit-engine preview both directions, stale-but-engine-matched not usable, stale+mismatched combined reason, per-kind reasons across all four index kinds, DDL surface, cudf lane with import skip-guard). Index suite: 219 passed, 1 skipped locally (GPU lanes deselected, no GPU on this box). Lint + scoped mypy clean. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
Resolve CHANGELOG.md conflict by keeping both Fixed entries: this PR's show_indexes engine-usability entry and master's gfql_clear_caches registry entry (#1836). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
lmeyerov
added a commit
that referenced
this pull request
Aug 12, 2026
, stacked on #1743) resolve_engine(AUTO) maps polars input frames to PANDAS (legacy input-format policy), so gfql_index_all()/create_index() with the default engine coerce-and-REPLACED a user's polars frames with pandas copies: every later polars-engine query then paid a full-frame pandas->polars re-conversion (O(E) per call) and the resident pandas index could never fingerprint-match. New resolve_index_engine(): when the caller said AUTO and BOTH resident frames are eager polars, the index layer resolves POLARS and indexes the frames in place (the layer is already engine-polymorphic: numpy sidecar arrays + polars row-gather). Wired into create_index, _is_resident_index_valid, show_indexes, and the DDL wire's NODE_PROP reuse check. Explicit engines are unchanged. Why this is safe NOW and was not in 2026-07: the retracted #1767 shipped this without query-side AUTO routing, so an AUTO-built polars index met an AUTO(pandas) query -> every hop declined to the scan floor, a measured 125-534x DEFAULT-path regression. This branch is STACKED ON #1743 (fix/gfql-auto-engine-polars-native), whose gfql() guard routes AUTO on all-polars-frame graphs to the native polars engine -- so the cliff inverts into the win: AUTO builds the polars index, AUTO queries route polars, index serves. Pinned by test_inversion_auto_index_auto_gfql_serves_polars_index (index_trace() must show path=index engine=polars with NO engine argument anywhere, plus value parity vs both explicit-engine spellings). show_indexes() under AUTO now reports the routed truth through the #1841 usable/reason columns: the previously not-usable "polars index + AUTO query" combination flips to usable=True (test renamed to test_polars_index_auto_query_usable_the_1841_flip); explicit mismatched-engine previews keep the #1838 decline wording. Deliberately narrower than #1743's query gate, self-consistent either way because create_index coerces the frames it indexes: LazyFrame graphs keep the legacy pandas build (an index cannot gather from a lazy plan), and edges-only graphs keep it too (materialize_nodes() does not yet produce polars nodes from polars edges -- pre-existing gap, also reachable via explicit engine='polars'). Pandas/cuDF graphs, mixed frames, and nodes-only graphs are unchanged, each pinned. Verification (local CPU lane, cudf/gpu params env-excluded on this box): index suite 229 passed; polars cypher conformance + cache coverage lock 177 passed / 3 skipped; lint.sh clean; scoped mypy (api.py, wire.py) clean; type-hygiene guard OK (no growth). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
lmeyerov
added a commit
that referenced
this pull request
Aug 12, 2026
, stacked on #1743) resolve_engine(AUTO) maps polars input frames to PANDAS (legacy input-format policy), so gfql_index_all()/create_index() with the default engine coerce-and-REPLACED a user's polars frames with pandas copies: every later polars-engine query then paid a full-frame pandas->polars re-conversion (O(E) per call) and the resident pandas index could never fingerprint-match. New resolve_index_engine(): when the caller said AUTO and BOTH resident frames are eager polars, the index layer resolves POLARS and indexes the frames in place (the layer is already engine-polymorphic: numpy sidecar arrays + polars row-gather). Wired into create_index, _is_resident_index_valid, show_indexes, and the DDL wire's NODE_PROP reuse check. Explicit engines are unchanged. Why this is safe NOW and was not in 2026-07: the retracted #1767 shipped this without query-side AUTO routing, so an AUTO-built polars index met an AUTO(pandas) query -> every hop declined to the scan floor, a measured 125-534x DEFAULT-path regression. This branch is STACKED ON #1743 (fix/gfql-auto-engine-polars-native), whose gfql() guard routes AUTO on all-polars-frame graphs to the native polars engine -- so the cliff inverts into the win: AUTO builds the polars index, AUTO queries route polars, index serves. Pinned by test_inversion_auto_index_auto_gfql_serves_polars_index (index_trace() must show path=index engine=polars with NO engine argument anywhere, plus value parity vs both explicit-engine spellings). show_indexes() under AUTO now reports the routed truth through the #1841 usable/reason columns: the previously not-usable "polars index + AUTO query" combination flips to usable=True (test renamed to test_polars_index_auto_query_usable_the_1841_flip); explicit mismatched-engine previews keep the #1838 decline wording. Deliberately narrower than #1743's query gate, self-consistent either way because create_index coerces the frames it indexes: LazyFrame graphs keep the legacy pandas build (an index cannot gather from a lazy plan), and edges-only graphs keep it too (materialize_nodes() does not yet produce polars nodes from polars edges -- pre-existing gap, also reachable via explicit engine='polars'). Pandas/cuDF graphs, mixed frames, and nodes-only graphs are unchanged, each pinned. Verification (local CPU lane, cudf/gpu params env-excluded on this box): index suite 229 passed; polars cypher conformance + cache coverage lock 177 passed / 3 skipped; lint.sh clean; scoped mypy (api.py, wire.py) clean; type-hygiene guard OK (no growth). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
lmeyerov
added a commit
that referenced
this pull request
Aug 13, 2026
, stacked on #1743) resolve_engine(AUTO) maps polars input frames to PANDAS (legacy input-format policy), so gfql_index_all()/create_index() with the default engine coerce-and-REPLACED a user's polars frames with pandas copies: every later polars-engine query then paid a full-frame pandas->polars re-conversion (O(E) per call) and the resident pandas index could never fingerprint-match. New resolve_index_engine(): when the caller said AUTO and BOTH resident frames are eager polars, the index layer resolves POLARS and indexes the frames in place (the layer is already engine-polymorphic: numpy sidecar arrays + polars row-gather). Wired into create_index, _is_resident_index_valid, show_indexes, and the DDL wire's NODE_PROP reuse check. Explicit engines are unchanged. Why this is safe NOW and was not in 2026-07: the retracted #1767 shipped this without query-side AUTO routing, so an AUTO-built polars index met an AUTO(pandas) query -> every hop declined to the scan floor, a measured 125-534x DEFAULT-path regression. This branch is STACKED ON #1743 (fix/gfql-auto-engine-polars-native), whose gfql() guard routes AUTO on all-polars-frame graphs to the native polars engine -- so the cliff inverts into the win: AUTO builds the polars index, AUTO queries route polars, index serves. Pinned by test_inversion_auto_index_auto_gfql_serves_polars_index (index_trace() must show path=index engine=polars with NO engine argument anywhere, plus value parity vs both explicit-engine spellings). show_indexes() under AUTO now reports the routed truth through the #1841 usable/reason columns: the previously not-usable "polars index + AUTO query" combination flips to usable=True (test renamed to test_polars_index_auto_query_usable_the_1841_flip); explicit mismatched-engine previews keep the #1838 decline wording. Deliberately narrower than #1743's query gate, self-consistent either way because create_index coerces the frames it indexes: LazyFrame graphs keep the legacy pandas build (an index cannot gather from a lazy plan), and edges-only graphs keep it too (materialize_nodes() does not yet produce polars nodes from polars edges -- pre-existing gap, also reachable via explicit engine='polars'). Pandas/cuDF graphs, mixed frames, and nodes-only graphs are unchanged, each pinned. Verification (local CPU lane, cudf/gpu params env-excluded on this box): index suite 229 passed; polars cypher conformance + cache coverage lock 177 passed / 3 skipped; lint.sh clean; scoped mypy (api.py, wire.py) clean; type-hygiene guard OK (no growth). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lineage: the one unlanded piece of #1767's disposition
#1767 was retracted after measurement showed it regressed the default path 125–534×. Its disposition comment proposed replacing it with a diagnostic instead:
#1838 landed the first half —
gfql_explain/index_tracenow name the mismatch via_engine_mismatch_reason. This PR lands the second half: the inspection surface itself. On master today,show_indexes()still saysvalid=Truefor every index in the disposition's measured cells, including the ones being declined at query time — the user inspecting their own index state is told it is fine while paying a silent 100–500× cliff (the disposition's words, not a new measurement; no perf claims are made here).What it does
validkeeps its fingerprint-only meaning, so nothing existing breaks. Three columns are added:query_engineengineargument (default'auto') resolves to for THIS graph — the sameresolve_enginecall a query makesusablereasonresident edge_out_adj index engine=polars, requested engine=pandas -> scan), plus a stale-fingerprint reasonSo the disposition's headline case now reads honestly: a polars-built index on a graph whose AUTO queries resolve to pandas shows
valid=True, usable=Falsewith the same reason stringgfql_explaingives for the decline.Surfaces wired through:
g.show_indexes(engine=...)previews an explicit engine choice;SHOW GFQL INDEXESrespects thegfql(engine=...)of the call;gfql_explainpasses its engine into the resident-index table it embeds.Mechanism reuse, not a parallel one
The per-row reason and the #1838 trace diagnostic share one formatter (
_engine_mismatch_text), so the two surfaces can never drift apart in wording; the existing exact-string pins on the explain side (test_explain_reports_bidirectional_engine_mismatch) keep guarding both.Explicitly NOT a routing change
No query planning or engine resolution is altered anywhere — reporting only. #1743 (AUTO → native routing) remains on hold, untouched.
Tests
9 new in
TestShowIndexesEngineUsability:engine='polars'preview usable; cudf lane (importorskip-guarded, mirrors the suite's existing cudf gating)SHOW GFQL INDEXEShonors AUTO vs explicit engineLocal (CPU, GPU lanes deselected — box has no GPU):
graphistry/tests/compute/gfql/index/219 passed, 1 skipped../bin/lint.sh(incl. type-hygiene guard) andMYPY_CMD="uvx mypy==2.3.0" ./bin/mypy.shon the touched files: clean.CHANGELOG: entry under Development/Fixed; docs:
indexing.rstliveness section now explainsusable/reason.🤖 Generated with Claude Code
https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr