Skip to content

fix(arrow): preserve nulls in validate autofix - #1598

Merged
lmeyerov merged 4 commits into
masterfrom
issue-867-auto-coerce-arrow-review
May 22, 2026
Merged

lmeyerov merged 4 commits into
masterfrom
issue-867-auto-coerce-arrow-review

Conversation

@lmeyerov

@lmeyerov lmeyerov commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Closes #867.

Summary

  • Rewrites the fresh Add auto_coerce option to handle mixed-type columns during Arrow conversion #867 lane away from the reverted auto_coerce API.
  • Keeps the existing public validate='autofix' / warn contract on plot(), upload(), and to_arrow().
  • Improves pandas mixed-type Arrow autofix to use pandas nullable string dtype when available, preserving missing values as Arrow nulls.
  • Keeps a compatibility fallback to standard str coercion for older pandas or nullable-string coercion failures.
  • Leaves cuDF behavior on the existing string coercion path.

Design

master already has the right public API shape:

g.plot(validate='autofix', warn=True)
g.upload(validate='autofix', warn=True)
g.to_arrow(validate='autofix', warn=True)

This PR does not add auto_coerce. validate='strict' / validate='strict-fast' continue to raise ArrowConversionError, while validate='autofix' retries Arrow conversion by probing object columns and coercing only columns that fail individual Arrow conversion.

Pandas autofix now prefers nullable string dtype when available so None/missing values stay Arrow nulls. If that dtype is unavailable or fails, it falls back to the prior str behavior.

Review / Oversight

  • Review skill converged after 2 waves with 0 blockers, 0 important findings, 0 suggestions.
  • Local artifacts: plans/review-pr-1598/.
  • Rebased over latest origin/master after it moved to 49e8dd02; final PR head is eee9d399ec4e193b34f283b55890bb8dd952cd48.

Local validation

  • python3 -m pytest -q graphistry/tests/test_plotter.py::TestPlotterArrowConversions -> 28 passed, 6 skipped, 1 xfailed
  • ./bin/ruff.sh graphistry/PlotterBase.py graphistry/tests/test_plotter.py
  • ./bin/typecheck.sh graphistry/PlotterBase.py
  • git diff --check

DGX / RAPIDS

  • DGX checkout: /home/lmeyerov/repos/pygraphistry-issue-867-dgx
  • RAPIDS 26.02: RAPIDS_VERSION=26.02 PROFILE=basic WITH_GPU=1 ... docker/test-rapids-official-local.sh
    • cudf DataFrame _table_to_arrow(..., validate_mode='autofix') smoke passed
    • focused Arrow autofix tests passed: 4 passed
  • RAPIDS 25.02: RAPIDS_VERSION=25.02 PROFILE=basic WITH_GPU=1 ... docker/test-rapids-official-local.sh
    • RAPIDS import smoke passed (cudf 25.02.02, cuml 25.02.01)
    • focused Arrow autofix tests passed: 4 passed
    • actual cudf DataFrame construction smoke was attempted first and hit cudaErrorMemoryAllocation out of memory; nvidia-smi showed VLLM::EngineCore using 81870MiB. This PR leaves the cuDF coercion behavior on the existing str path, and 26.02 covers the actual cudf Arrow smoke.

CI

  • GitHub Actions run 26262954992 completed success on head eee9d399ec4e193b34f283b55890bb8dd952cd48.
  • Green coverage includes python-lint-types py3.8-3.14, test-pandas-compat legacy/latest/rapids-aligned, test-pandas-compat-gfql legacy/latest, test-minimal-python, test-gfql-core, test-docs, test-core-python, test-spark, test-neo4j, test-build, changed-line-coverage, and CodeQL.

Compiler-plan surface touched: no.

@lmeyerov lmeyerov changed the title feat(arrow): add opt-in mixed type coercion fix(arrow): preserve nulls in validate autofix May 22, 2026
@lmeyerov
lmeyerov force-pushed the issue-867-auto-coerce-arrow-review branch from dc37ae7 to eee9d39 Compare May 22, 2026 01:24
@lmeyerov
lmeyerov merged commit 62cc48c into master May 22, 2026
74 checks passed
@lmeyerov
lmeyerov deleted the issue-867-auto-coerce-arrow-review branch May 22, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add auto_coerce option to handle mixed-type columns during Arrow conversion

1 participant