Skip to content

feat(arrow): add opt-in mixed type coercion - #1591

Merged
lmeyerov merged 1 commit into
masterfrom
issue-867-auto-coerce-arrow
May 21, 2026
Merged

lmeyerov merged 1 commit into
masterfrom
issue-867-auto-coerce-arrow

Conversation

@lmeyerov

@lmeyerov lmeyerov commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #867.

Summary

  • Add auto_coerce=False to plot(), upload(), and to_arrow() so Arrow conversion remains strict by default.
  • When auto_coerce=True, retry failed pandas/cuDF Arrow conversion by probing object columns one at a time and coercing only Arrow-incompatible columns to strings.
  • Emit an info log naming coerced columns instead of warning or silently coercing.
  • Update docs, changelog, Plottable protocol, and anchored regression tests.

Coercion heuristic

Graphistry first runs the normal Arrow conversion. If pandas/cuDF conversion fails with ArrowTypeError/ArrowInvalid and auto_coerce=True, it tests object columns individually with the same Arrow conversion path. Only columns that fail individual Arrow conversion are coerced to strings; Arrow-compatible object columns are left alone. Pandas coercion uses nullable string dtype so missing values remain Arrow nulls. cuDF uses its string dtype path. This keeps the fallback narrow and mirrors the user workaround from #867 while preserving strict default behavior.

Info-log sample:

Auto-coerced mixed-type columns to string for Arrow conversion: ['amount']

LOC buckets

  • Production: +61 / -60, net +1 LOC
  • Tests / fixtures / baselines: +129 / -64, net +65 LOC
  • Comments / docs / changelog: +17 / -10, net +7 LOC

Compiler-plan surface touched: no. This is public Arrow-conversion API surface only; no GFQL compiler-plan routes, dispatch contracts, error-code/defer-code context, source spans, pass/verifier hooks, schema/type-system hooks, remote schema hooks, or compat executor paths were touched.

Validation

  • python3 -m pytest -q graphistry/tests/test_plotter.py::TestPlotterArrowConversions -> 25 passed, 7 skipped, 1 xfailed
  • python3 -m pytest -q graphistry/tests/test_plotter.py graphistry/tests/test_validate_settings.py graphistry/tests/test_schema_artifacts.py graphistry/tests/test_viz_settings.py -> 83 passed, 10 skipped, 4 xfailed
  • python3 bin/changed_line_coverage.py --data-file /tmp/pr1591.coverage --base-ref origin/master --head-ref HEAD --min-percent 80 --output-dir /tmp/pr1591-changed-coverage -> passed, 19/23 = 82.61%
  • python3 -m graphistry.devschemas.export --check
  • ./bin/ruff.sh graphistry/PlotterBase.py graphistry/Plottable.py graphistry/exceptions.py graphistry/tests/test_plotter.py
  • ./bin/typecheck.sh graphistry/PlotterBase.py graphistry/Plottable.py graphistry/exceptions.py
  • git diff --check
  • ssh dgx-spark 'cd repos/pygraphistry-issue-867-dgx && RAPIDS_VERSION=25.02 PROFILE=basic WITH_GPU=1 WITH_IMAGE_BUILD=0 TEST_FILES="graphistry/tests/test_plotter.py::TestPlotterArrowConversions::test_table_to_arrow_cudf_auto_coerce_smoke graphistry/tests/test_plotter.py::TestPlotterArrowConversions::test_api3_cudf_to_arrow_memoization graphistry/tests/test_plotter.py::TestPlotterArrowConversions::test_api3_cudf_to_arrow_memoization_forgets" docker/test-rapids-official-local.sh' -> cudf 25.02.02, cuml 25.02.01, Python 3.12.9, 3 passed
  • ssh dgx-spark 'cd repos/pygraphistry-issue-867-dgx && RAPIDS_VERSION=26.02 PROFILE=basic WITH_GPU=1 WITH_IMAGE_BUILD=0 TEST_FILES="graphistry/tests/test_plotter.py::TestPlotterArrowConversions::test_table_to_arrow_cudf_auto_coerce_smoke graphistry/tests/test_plotter.py::TestPlotterArrowConversions::test_api3_cudf_to_arrow_memoization graphistry/tests/test_plotter.py::TestPlotterArrowConversions::test_api3_cudf_to_arrow_memoization_forgets" docker/test-rapids-official-local.sh' -> cudf 26.02.01, cuml 26.02.00, Python 3.13.12, 3 passed
  • Review skill converged: W1 fixed nullable-string coercion; W2/W3 clean; W4 clean after cuDF smoke addition; W5 clean after CI test mock fix; W6 clean after changed-line coverage test amplification

Typecheck note: ./bin/typecheck.sh ... graphistry/tests/test_plotter.py still hits existing test typing gaps (pytest stubs missing and unrelated COMPLEX_EMPTY annotation). Production touched files typecheck cleanly.

@lmeyerov
lmeyerov force-pushed the issue-867-auto-coerce-arrow branch 4 times, most recently from 668ccce to ab99fc5 Compare May 21, 2026 06:16
@lmeyerov
lmeyerov force-pushed the issue-867-auto-coerce-arrow branch from ab99fc5 to 07cc9dc Compare May 21, 2026 06:36
@lmeyerov
lmeyerov merged commit 6be1ac9 into master May 21, 2026
73 checks passed
@lmeyerov
lmeyerov deleted the issue-867-auto-coerce-arrow branch May 21, 2026 06:50
lmeyerov added a commit that referenced this pull request May 21, 2026
@lmeyerov

Copy link
Copy Markdown
Contributor Author

Backed out from master with a normal revert commit, no force-push:

#867 has been reopened for renewed oversight/review.

lmeyerov added a commit that referenced this pull request May 22, 2026
lmeyerov added a commit that referenced this pull request May 22, 2026
lmeyerov added a commit that referenced this pull request May 22, 2026
lmeyerov added a commit that referenced this pull request May 22, 2026
* Reapply "feat(arrow): add opt-in mixed type coercion (#1591)"

This reverts commit 15893a6.

* Revert "Reapply "feat(arrow): add opt-in mixed type coercion (#1591)""

This reverts commit 2eae715.

* fix(arrow): preserve nulls in validate autofix

* test(arrow): tolerate pandas fallback null repr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add auto_coerce option to handle mixed-type columns during Arrow conversion

1 participant