Repository navigation
Error reading a FITS table into a pandas DataFrame via astropy #1156
Description
I'm trying to read a FITS table into a pandas DataFrame via astropy.
Test data and script is here:
https://github.com/cdeil/test_fits_to_pandas
This is the error I get (same error from read_via_table.py):
$ python read_via_fits.py
nfev statname numpoints
ROI Solution
HESS_J1023m575 4_big 273 cash 75000
5 316 cash 75000
3 131 cash 75000
4 273 cash 75000
2_elongated 111 cash 75000
ERROR: ValueError: Big-endian buffer not supported on little-endian compiler [unknown]
Traceback (most recent call last):
File "read_via_fits.py", line 8, in <module>
print(table[['nfev', 'statname']])
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2017, in __getitem__
return self._getitem_array(key)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2054, in _getitem_array
return self.take(indexer, axis=1, convert=True)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2998, in take
return self.reindex(columns=new_columns)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2630, in reindex
fill_value, limit)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2722, in _reindex_columns
copy, fill_value)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/frame.py", line 2739, in _reindex_with_indexers
fill_value=fill_value)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/internals.py", line 1959, in reindex_indexer
return self._reindex_indexer_items(new_axis, indexer, fill_value)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/internals.py", line 1993, in _reindex_indexer_items
allow_fill=False)
File "/Users/deil/Library/Python/2.7/lib/python/site-packages/pandas-0.11.1.dev_cf47a42-py2.7-macosx-10.8-intel.egg/pandas/core/common.py", line 540, in take_nd
func(arr, indexer, out, fill_value)
File "generated.pyx", line 2842, in pandas.algos.take_2d_axis0_int16_int16 (pandas/algos.c:69040)
ValueError: Big-endian buffer not supported on little-endian compiler
@iguananaut @astrofrog Am I doing something wrong (passing a FITSrec instead of a recarray)?
Is there a workaround?
Activity
embray commentedon Jun 6, 2013
It looks like Pandas will only accept data in native byte order. FITS data is big-endian. You would have to swap the byte order to native first. I can't tell whether there's otherwise an issue with pyfits here.
cdeil commentedon Jun 6, 2013
I think reading FITS data into a pandas DataFrame is a common use case, so i've asked in a pandas issue whether they can make it possible to accept a structured array with non-native byte order.
cdeil commentedon Jun 6, 2013
So as per the advice in #3778 the solution is to call
array.byteswap().newbyteorder()before passing the structured array to the pandas DataFrame constructor.Is it worth adding
astropy.table.Table.to_dataframe()andastropy.table.Table.from_dataframe()methods to make it easy to go back and forth between astropyTableand pandasDataFrameand take care of the byte swapping (and mask, I guess there's no way to preserve the FITS header info) there?Would this be a welcome addition and acceptable to have a pandas import in that method?
And if you don't think it's worth it: how about adding this info to the astropy docs somewhere?
astrofrog commentedon Jun 6, 2013
I wonder whether we should suggest to the Pandas team that they support a
__panda__interface (like__array__) so that one could just doDataFrame(table). We had discussed having conversion methods to/from pandas before in a previous issue, and I think it's still a good idea, though I'd call itto_pandasandfrom_pandasotherwisedataframeis a too generic word.taldcroft commentedon Jun 7, 2013
On the input side I would prefer to just add support for initializing from a dataframe like
Table(dataframe). I guess the only caveat is if there are pandas-specific options that would be required for the conversion. Hopefully not.I agree with @astrofrog that the better solution for
DataFrame(astropy_table)is a__panda__interface. But even if the pandas project implements this immediately we would need a back-compatible mechanism for older pandas versions. So something liketo_pandas()sounds fine.For the outputter you would need a pandas import in that method. That would be fine, assuming the import is in a proper try / except block.
Cadair commentedon Mar 3, 2014
We are having this issue in SunPy at the moment sunpy/sunpy#871. What is the best way to flip the byte order of a FITS table? Convert to a astropy.Table first and then do what? or some other way?
andrevitorelli commentedon Apr 27, 2014
Strangely, the X.byteswap().newbyteorder() works in one computer but not in another. when I ask for X.dtype it says all fields have been swapped, but when I ask for X.field('fieldname').dtype, it shows as big-endian. My solution was to make the conversion inside the constructor...
embray commentedon Apr 28, 2014
I'm working on some experimental custom dtype magic in Numpy that will allow data in non-native byte order to be "seen" as native by always byteswapping on demand when individual values are needed (such as going in and out of a ufunc, or simply displaying the data). This is actually what Numpy already does with data in non-native byte order--I'm just trying to make it a little more transparent. Slower? Yes, possibly. But it's a tradeoff. Still not 100% sure if it will work, but it might make for an interesting approach to this. Though it won't work with any code that tries to access the data buffer directly.
astrofrog commentedon Jan 14, 2015
It sounds like this will not be fixed by 1.0 so removing the milestone.
rasmi commentedon Feb 18, 2015
Any updates on this, @embray? Maybe @jreback can chime in on the different approaches mentioned by you and @astrofrog.
Cadair commentedon Feb 18, 2015
@astrofrog does your work in #3504 deal with this?
astrofrog commentedon Feb 18, 2015
There are two issues here. The conversion from Table to DataFrame works ok with #3504:
It is the next line that causes an error:
So weirdly, pandas is accepting the data regardless of endianness. I get the right result if I print out the DataFrame:
So somehow maybe it's just at the
set_indexphase that there is an endianness bug.Either way, I find it weird that pandas would accept the data if it wasn't valid. So to me, there is definitely a bug in pandas, even if it's just that it should complain when big-endian data is first passed.
I am short on time but if someone else can investigate further this would be very helpful.
9 remaining items
cdeil commentedon Feb 24, 2015
+1 to converting to native format by default!
(I think the number of users that will be bitten by this backwards-incompatible change will be very small compared to the number of users that will run into issues they don't understand because they got big-endian arrays.)
How do other packages (e.g. h5py, ....?) deal with this?
taldcroft commentedon Feb 24, 2015
@astrofrog - I was also thinking that the endianness conversion could be isolated to
Table.readwhileio.fitswould still just give the values straight from the file (which would need to be the case for mem mapping). So usingio.fitsdirectly would represent a path for the rare case of requiring the original endianness, so maybeTable.readjust always does it as the "high-level" user-friendly interface.But yeah, wrong endianness is really subtle and annoying to trace down.
astrofrog commentedon Feb 24, 2015
@taldcroft - that sounds good to me. Now we could go one step further and build it in to
Table(notTable.read) so that users don't accidentally add columns with the wrong endian-ness, the idea being that we don't have to fix endian-ness on a per-format basis. We wouldn't fail if the user specifies an array with the wrong endian-ness, we'd just silently fix it.Tablecould also include an option in__init__that would beallow_non_native_endiandefaulting toFalse, or something similar.taldcroft commentedon Feb 24, 2015
@astrofrog sounds good to me!
rasmi commentedon Feb 24, 2015
Makes sense. Would values have to be re-converted if data was being saved to a FITS file, or is that already built in?
astrofrog commentedon Feb 24, 2015
If I'm understanding @embray correctly, all FITS data is stored in big endian? (so in that sense, yes the conversion back to big endian would be automatic when writing to a FITS file). Basically, everything would be seamless from the point of view of a user who doesn't care about endian-ness.
mhvk commentedon Feb 24, 2015
👍 to doing endian-conversion in the I/O parts (i.e., in
Table.read(...)). However, 👎 to also do things as part ofTableorColumn-- that seems overkill since one already has to be pretty explicit to get the endianness wrong in non-I/O case.mwcraig commentedon Apr 11, 2015
👍 to automatic conversion of some sort. Just got bit by this. Thanks to @taldcroft for the
+0workaround!astrofrog commentedon Apr 25, 2015
Just for info, we've now merged #3504 which makes it easy to convert between astropy Tables and pandas DataFrame (and should deal properly with endian-ness). So once can now do:
Should we close this?
cdeil commentedon Apr 25, 2015
@astrofrog Thanks for implementing the Table / DataFrame converters.
I still see this error though ...
astrofrog commentedon Apr 25, 2015
Oops, indeed I can reproduce that by adding a single line to a unit test
astrofrog commentedon Apr 25, 2015
I actually think this is a pandas bug - if you access fields individually, it works fine. I've opened an issue in the panda repo: pandas-dev/pandas#9989
astrofrog commentedon Apr 26, 2015
Ok so as recommended by the pandas dev, we should just flip the byte order ourselves if it's not native, though they are planning on trying to make that seamless in pandas because it's a common issue. So #3729 will fix that for Astropy.