Skip to content

GFQL/Cypher: ORDER BY on stringified-list columns uses Cypher list-orderability (#1359) - #1362

Merged
lmeyerov merged 5 commits into
masterfrom
issue-1359-with-orderby-wrong-rows
May 9, 2026
Merged

lmeyerov merged 5 commits into
masterfrom
issue-1359-with-orderby-wrong-rows

Conversation

@lmeyerov

@lmeyerov lmeyerov commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes pygraphistry-side semantic gap from #1359 (item #1 of meta-issue #1353 — Comparator + ORDER BY semantics tranche).

When a list-valued property is stored as a string column (e.g. round-tripped through CSV / Arrow string columns), ORDER BY <col> previously fell back to lex string sort, which mishandles negative numbers because '-' (0x2D) sorts before '2' (0x32) — so '[1, -20]' came before '[1, 2]', putting the wrong row in LIMIT N results.

What changed

graphistry/compute/gfql/row/ordering.py

  • New helper order_detect_stringified_list_series(series) — returns True when every non-null entry is an actual string matching ^\[.*\]$. Complements order_detect_list_series (which only matches Python-list values).
  • New helper parse_stringified_list_series(series) — parses the column via ast.literal_eval, returning a Python-list Series (or None on parse failure).

graphistry/compute/gfql/row/pipeline.py

  • order_by: after the existing list_candidate check, if the column is a stringified-list series with no nulls, parse into a fresh __gfql_sort_listparsed_*__ aux column and re-enter the existing list-sort path (build_list_sort_columns).
  • Track aux columns in aux_drop_cols and drop them after sort_values so they don't leak into downstream stages (regression-tested via existing test_string_cypher_supports_labels_projection_* and test_string_cypher_supports_graph_functions_on_list_wrapped_entities).

Python-list-typed columns continue through the existing path unchanged.

Tests

graphistry/tests/compute/gfql/cypher/test_lowering.py

  • test_string_cypher_order_by_python_list_column_uses_list_orderability — pins ASC/DESC top-3 on a Python-list column.
  • test_string_cypher_order_by_stringified_list_column_uses_list_orderability — same query, dtype="string" list column; pre-fix produced E ([2,-2,100]) in top-3 instead of A ([2,-2]); post-fix the SET matches openCypher.

Test plan

  • ./bin/lint.sh clean
  • ./bin/mypy.sh graphistry/compute/gfql/row/ordering.py graphistry/compute/gfql/row/pipeline.py clean
  • pytest graphistry/tests/compute/gfql/cypher/ — 1224 passed, 90 skipped, 15 xfailed (no regressions)
  • Direct cypher xfail-outcome verification for the 14 with-orderby keys: Cluster A with-orderby1-32-{1,2} (list ORDER BY DESC) flip to success_matches_expected; remaining 12 are pygraphistry-SET-correct, blocked at runner level
  • Once tck-gfql#36 lands the matching fixture/runner fixes, all 14 keys flip to success_matches_expected

Out of scope (handled in tck-gfql#36)

  • tests/cypher_tck/parse_cypher.py::_parse_properties doesn't parse list literals (lists become strings in fixtures).
  • tests/cypher_tck/test_tck_runner.py::_rows_ordered enforces strict order even when the openCypher feature uses Then the result should be, in any order:.

Cross-reference

🤖 Generated with Claude Code

lmeyerov and others added 2 commits May 8, 2026 18:05
…ability (#1359)

Issue #1359 (meta #1353 item #1) — when a list-valued property is stored
as a string column (CSV/Arrow round-trip), `ORDER BY <col>` previously
fell back to lex string sort, mishandling negatives ("-" < "2" in ASCII
puts "[1, -20]" before "[1, 2]").

Add `order_detect_stringified_list_series` + `parse_stringified_list_series`
in graphistry/compute/gfql/row/ordering.py; route the row pipeline through
the existing `build_list_sort_columns` after `ast.literal_eval` parsing
when the column is fully list-shaped. Track the parsed aux column in
`aux_drop_cols` and drop it after `sort_values` so it doesn't leak into
downstream stages.

Python-list-typed columns continue through the existing list-aware path
unchanged. Add pygraphistry-side regression tests on both Python-list
and stringified-list inputs.

Matching TCK port-level fixture/runner fixes that flip the 14 wrong-row
scenarios to success_matches_expected are tracked in tck-gfql#36.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…1359)

Apply review-wave findings inline:

- W1-I1: bridge cuDF index in parse_stringified_list_series via
  host_index.to_pandas(), mirroring pipeline.py
  _gfql_eval_dynamic_list_subscript canonical pattern.

- W2-A1: validate_order_series_vector_safe now splits the str family
  into list_string (list-shaped) vs plain str and raises mixed-family
  on the boundary, instead of silently lex-sorting partial-list-shape
  columns. Also extended to dtype="string" (StringDtype) columns.

- W1-I4/I5/I6: add 4 behavioral tests pinning null-row tolerance,
  malformed-string fallback, multi-key ORDER BY (stringified-list +
  scalar) with aux-col leak guard, and partial-list-shape mixed-family
  raise.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@lmeyerov
lmeyerov force-pushed the issue-1359-with-orderby-wrong-rows branch from f3bb6a2 to de10711 Compare May 9, 2026 01:18
Wave-4 review surfaced a misleading docstring on
test_string_cypher_order_by_stringified_list_with_nulls — it claimed the
parsed-list path runs through, but actually pipeline.py:3955-3967 skips
that branch when has_null=True (mirroring the Python-list path), and we
fall through to lex sort. Rewrite the docstring to describe what really
happens and rename to ..._returns_top_k_without_error to remove ambiguity.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@lmeyerov

lmeyerov commented May 9, 2026

Copy link
Copy Markdown
Contributor Author

Audit follow-ups filed as #1373 — (1) latent cuDF host-bridge gap in order_by list-path at pipeline.py:3964 (the select callsite at pipeline.py:3651-3658 has the canonical consumer-side to_pandas() guard; the new order_by callsite emits a pd.Series from parse_stringified_list_series but doesn't mirror the consumer half — unreachable in production today, latent break); (2) test-placement debt — 6 new tests at test_lowering.py:7781-7972 belong in row-pipeline unit tests, ideally under a new tests/compute/gfql/row/ mirror that matches the source layout. Neither is blocking. Audit details at plans/1359-orderby-wrong-rows/audits/cudf-flow-audit.md (local).

@lmeyerov
lmeyerov merged commit 4ea9336 into master May 9, 2026
137 checks passed
@lmeyerov
lmeyerov deleted the issue-1359-with-orderby-wrong-rows branch May 9, 2026 04:55
lmeyerov added a commit that referenced this pull request May 9, 2026
[CHORE] GFQL row tests: relocate #1362 ordering cases into row mirror (#1377)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant