Repository navigation
io.fits errors with tables containing strings #5280
Copy link
Copy link
Closed
Description
The following code contains a set of unit tests for creating FITS binary tables with strings. They pass under python 2.7 but three of the five fail under python 3.5. This is also available as a public gist but I don't promise to keep that around unchanged forever, so the code is also copy and pasted below.
The tests that fail are:
- writing a numpy structured array with a column of strings using
astropy.io.fits.writetoand reading back in withastropy.io.fits.getdata: the data are munged. This works when writing the same data converted to a Table object, and also works when creating the BinTableHDU from columns by hand. This test is a demonstration of the failure reported in fits.io fails to write numpy structured arrays with unicode columns #5268 . - writing then reading a numpy structured array containing a 2D column of strings using
astropy.io.fits.writetoand reading back in withastropy.io.fits.getdata: the dtype seems to have mixed up the number of unicode bytes vs. the number of characters to write. - writing then reading an astropy.table.Table containing a 2D column of strings: same problem with the dtype
import unittest
from uuid import uuid4
import os
import numpy as np
from astropy.io import fits
from astropy.table import Table, Column
class _TestTable(unittest.TestCase):
'''
Base class for testing FITS tables.
Subclass this and add setupClass to define attributes to test:
data (numpy structured array)
table (astropy Table)
'''
def setUp(self):
self.testfile = 'test-{}.fits'.format(uuid4())
def tearDown(self):
if os.path.exists(self.testfile):
os.remove(self.testfile)
def test_fits(self):
fits.writeto(self.testfile, self.data)
dx = fits.getdata(self.testfile)
self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
def test_table(self):
self.table.write(self.testfile)
tx = Table.read(self.testfile)
self.assertEqual(self.table['x'].dtype, tx['x'].dtype)
self.assertEqual(self.table['y'].dtype, tx['y'].dtype)
self.assertTrue(np.all(self.table['x'] == tx['x']), 'x: {} != {}'.format(self.table['x'], tx['x']))
self.assertTrue(np.all(self.table['y'] == tx['y']), 'y: {} != {}'.format(self.table['y'], tx['y']))
class TestTable1DColumn(_TestTable):
@classmethod
def setUpClass(cls):
dtype = [
('x', (str, 5)), #- 1D column of 5-character strings
('y', (str, 3)), #- !D column of 3-character strings
]
data = np.zeros(2, dtype=dtype)
data['x'] = ['abcde', 'xyz']
data['y'] = ['A', 'BCX']
table = Table(data)
cls.data = data
cls.table = table
def test_manual_fits(self):
'''Construct the BinTableHDU by hand'''
col1 = fits.Column(name='x', format='5A', array=self.data['x'])
col2 = fits.Column(name='y', format='3A', array=self.data['y'])
cols = fits.ColDefs([col1, col2])
tbhdu = fits.BinTableHDU.from_columns(cols)
tbhdu.writeto(self.testfile)
dx = fits.getdata(self.testfile)
self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
class TestTable2DColumn(_TestTable):
@classmethod
def setUpClass(cls):
dtype = [
('x', (str, 5)), #- 1D column of 5-character strings
('y', (str, 3), (4,)), #- 2D column; each row is four 3-char strings
]
data = np.zeros(2, dtype=dtype)
data['x'] = ['abcde', 'xyz']
data['y'][0] = ['A', 'BC', 'DEF', '123']
data['y'][1] = ['X', 'YZ', 'PQR', '999']
table = Table(data)
cls.data = data
cls.table = table
#- I'm not sure how to create 2D column of strings by hand analogous to
#- the 1D test case; see https://github.com/astropy/astropy/issues/5279
# def test_manual_fits(self):
# '''Construct the BinTableHDU by hand
#
# When these data are successfully written under py2.7, they have keywords
# TTYPE1 = 'x '
# TFORM1 = '5A '
# TTYPE2 = 'y '
# TFORM2 = '12A '
# TDIM2 = '(3,4) '
#
# Trying to replicate that logic here with dim=(3,4) doesn't work
# '''
# col1 = fits.Column(name='x', format='5A', array=self.data['x'])
# col2 = fits.Column(name='y', format='12A', array=self.data['y'], dim=(3,4))
# cols = fits.ColDefs([col1, col2])
# tbhdu = fits.BinTableHDU.from_columns(cols)
# tbhdu.writeto(self.testfile)
# dx = fits.getdata(self.testfile)
# self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
# self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
# self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
# self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
if __name__ == '__main__':
### unittest.main()
t1 = unittest.defaultTestLoader.loadTestsFromTestCase(TestTable1DColumn)
t2 = unittest.defaultTestLoader.loadTestsFromTestCase(TestTable2DColumn)
unittest.TextTestRunner(verbosity=2).run(unittest.TestSuite([t1,t2]))Running these pass under python 2.7 but fail under python 3.5 with:
(py3) py3 $ python test-fits-table.py
test_fits (__main__.TestTable1DColumn) ... WARNING: File may have been truncated: actual file length (8636) is smaller than the expected size (8640) [astropy.io.fits.file]
FAIL
test_manual_fits (__main__.TestTable1DColumn)
Construct the BinTableHDU by hand ... ok
test_table (__main__.TestTable1DColumn) ... ok
test_fits (__main__.TestTable2DColumn) ... WARNING: File may have been truncated: actual file length (8566) is smaller than the expected size (8640) [astropy.io.fits.file]
FAIL
test_table (__main__.TestTable2DColumn) ... FAIL
======================================================================
FAIL: test_fits (__main__.TestTable1DColumn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test-fits-table.py", line 30, in test_fits
self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
AssertionError: False is not true : x: ['abcde' 'xyz'] != ['abcde' 'zBCX']
======================================================================
FAIL: test_fits (__main__.TestTable2DColumn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test-fits-table.py", line 29, in test_fits
self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
AssertionError: dtype('<U3') != dtype('<U12')
======================================================================
FAIL: test_table (__main__.TestTable2DColumn)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test-fits-table.py", line 37, in test_table
self.assertEqual(self.table['y'].dtype, tx['y'].dtype)
AssertionError: dtype('<U3') != dtype('<U12')
----------------------------------------------------------------------
Ran 5 tests in 0.100s
FAILED (failures=3)EDIT: Added syntax highlighting.
Reactions are currently unavailable
Activity
pllim commentedon Aug 30, 2016
Do they also fail in Python 2 when you add
from __future__ import unicode_literalsat the beginning of your test script?sbailey commentedon Aug 30, 2016
Adding
from __future__ import unicode_literalsalso required wrapping all column names withstr(), because bizarrely numpy requires unicode columns names under python 3.5 but forbids it under python 2.7 (see numpy/numpy#2407). e.g.After making that update, the tests continue to pass under python 2.7 even with unicode_literals. However; unicode_literals only makes literals like 'abc' into unicode; the
strtype remains bytes and thus the example numpy arrays being created are using bytes not unicode, which I think is the key difference for these tests.sbailey commentedon Aug 30, 2016
Using the hint from #5279, I was able to fix the commented out test:
This passes for both python 2.7 and 3.5, demonstrating that the low level tools are able to write these 2D columns of strings, but something is going wrong in the conversion of Table/ndarray objects into the
fits.Columnwhen usingTable.writeandfits.writeto.pllim commentedon Aug 30, 2016
@sbailey , please see my comments on #5268. Basically, I think
test_fits()is not the right way to construct a proper FITS table.sbailey commentedon Aug 30, 2016
I've updated the public gist to test 4 ways of writing/reading these tables:
io.fits.writetoandio.fits.getdataio.fits.BinTableHDU(data)and opening the file, extracting the data, closing the fileio.fits.BinTableHDU.from_columns()+io.fits.getdataTable(data).write()andTable.read()They all pass on python 2.7, but 5 of the 8 tests fail under 3.5. From the documentation and other usage of these functions within astropy, I believe all of these are supposed to be valid ways of creating a FITS table.
MSeifert04 commentedon Aug 30, 2016
I've tested it and there seems happen some dtype conversion for
from_columns:The different ways:
I'm not sure which ways are supposed to work but those which do seem to convert the unicodes to strings. I suspect that the convenience functions and BinTableHDU try to make educated guesses about the data but can't handle those. I haven't looked at the internals yet.
weaverba137 commentedon Aug 30, 2016
Please see my analysis of
data.view(FITS_rec)in #5268, wheredatais a numpy array. I believe that, internally, the methods likefits.BinTableHDUorfits.writeto(filename, numpy_array)are both usingdata.view(FITS_rec)to create the data payload. I believe my test in #5268 is consistent with @MSeifert04's result.weaverba137 commentedon Aug 30, 2016
I also invite you to take a look at
astropy.io.fits.util.decode_ascii(). Here's the snippet from that module:So, when exactly is this going to be replaced with a function for Python 3?
MSeifert04 commentedon Aug 30, 2016
@weaverba137 here?
weaverba137 commentedon Aug 30, 2016
Wow. Just. Dang. Cripes. No wonder no one wants to take over this code.
weaverba137 commentedon Aug 31, 2016
I can go into truly exhaustive detail if needed, but I think I have a clue as to what is going on.
Short version:
Writing numpy arrays with unicode columns (
kind == 'U') directly to FITS binary tables in a file skips over a conversion process that (inadvertently) preserves the correct width of the column in the binary output.Longer version:
Columns are created explicitly, combined into aColDefsobject, and a HDU is created withBinTableHDU.from_columns(), any unicode columns get converted to Numpy bytes, that is, columns withkind == 'S'. This happens in both 2.7 & 3.5.BinTableHduis created directly from a numpy array (recarray, etc), this conversion does not happen, so there are some columns that havekind == 'U'..dataattribute) has unicode columns (kind == 'U'), the data are written row-by-row, and column-by-column within a row.encoded_item = np.char.encode(item, 'ascii'), and then written to the output stream withencoded_item.tofile(fileobj).'foo'will be written to the output stream, instead of'foo\0\0'.Exhaustive version: by request.
cmarmo commentedon Feb 15, 2022
I have run the test of this issue against the current version on main.
The exact same error is thrown.
Test script
Here is the traceback:
I could fix the
AssertionError: dtype('<U5') != dtype('S5')modifyingtest_tableas followingIf I understand correctly, this is due to the fact that string dtypes are written in unicode by numpy, but the astropy
read_table_fitsin connect.py reads by default in binary. Addingcharacter_as_bytes=Falsewill read in unicode.This is not a bug, perhaps this could be made more explicit in the documentation.
I was unable to fix the
AssertionError: dtype('<U3') != dtype('<U12')but I tracked it to table.py.This is where columns move from the format
[('x', 'S5'), ('y', 'S3', (4,))]toI guess, there is some automatic mechanism to convert to ColDefs that I couldn't understand.
Hope this can help in moving forward.