Repository navigation
Parquet v2.0 deprecated #12298
Description
Description
FutureWarning: Parquet format '2.0' pseudo version is deprecated, use '2.4' or '2.6' for fine-grained feature selection
See https://github.com/astropy/astropy/runs/4012372350?check_suite_focus=true
@erykoff, could you take a look at this. Thanks!
Steps to Reproduce
Run CI on any PR.
System Details
tox -e py38-test-alldeps-numpy118-cov-clocale -- --remote-data=astropy
Activity
pllim commentedon Oct 26, 2021
Well, that was fast. #12215 was only merged recently... This is blocking CI on other unrelated PRs, so hopefully a quick fix is possible.
pllim commentedon Oct 26, 2021
Maybe it is as simple as changing this line to use
'2.4'?astropy/astropy/io/misc/parquet.py
Line 307 in 3353a7d
erykoff commentedon Oct 26, 2021
pyarrow 6.0.0 just came out but isn't on conda forge yet. And I can't even find any documentation on pyarrow 6 about what these version strings mean, or if you can use them with pyarrow 5.0.0 ...
WilliamJamieson commentedon Oct 26, 2021
In local testing this appears to fix some (all but 2) of the test failures.
WilliamJamieson commentedon Oct 26, 2021
Looks like there are some changes in tests along this line too. These map to the remaining tests which were failing.
astropy/astropy/io/misc/tests/test_parquet.py
Line 647 in 3353a7d
astropy/astropy/io/misc/tests/test_parquet.py
Line 674 in 3353a7d
erykoff commentedon Oct 26, 2021
Problem is that
2.4doesn't work with pyarrow 5.0.0. I think we need to pin pyarrow, or do a noqa mark.erykoff commentedon Oct 26, 2021
Checking the version of pyarrow is also possible, but I'm not sure if that's wise.
WilliamJamieson commentedon Oct 26, 2021
Unfortunately, you are correct
'2.4'fails when pyarrow is 5.0.WilliamJamieson commentedon Oct 26, 2021
@erykoff it looks like we will have to check the
pyarrowversion to set theversionfor parquet or pinpyarrow. Is there a preference?erykoff commentedon Oct 26, 2021
My preference is to pin
pyarrowfor now, and exploration on updating support for both can come later.pllim commentedon Oct 26, 2021
Once we pin a minversion, we usually do not go back and try to support the older one in the future. FYI.
erykoff commentedon Oct 26, 2021
I think we need to pin a min/max version right now at 5, since 6 isn't widely available.
pllim commentedon Oct 26, 2021
Ah, ok...
WilliamJamieson commentedon Oct 26, 2021
PR #12300 fixes the issue via a
minversionforpyarrow. I am happy to change this, but 6 is now on pypy and is whatpip install pyarrownow installs.