Skip to content

io.fits errors with tables containing strings #5280

Closed

Description

@sbailey

The following code contains a set of unit tests for creating FITS binary tables with strings. They pass under python 2.7 but three of the five fail under python 3.5. This is also available as a public gist but I don't promise to keep that around unchanged forever, so the code is also copy and pasted below.

The tests that fail are:

  • writing a numpy structured array with a column of strings using astropy.io.fits.writeto and reading back in with astropy.io.fits.getdata: the data are munged. This works when writing the same data converted to a Table object, and also works when creating the BinTableHDU from columns by hand. This test is a demonstration of the failure reported in fits.io fails to write numpy structured arrays with unicode columns #5268 .
  • writing then reading a numpy structured array containing a 2D column of strings using astropy.io.fits.writeto and reading back in with astropy.io.fits.getdata: the dtype seems to have mixed up the number of unicode bytes vs. the number of characters to write.
  • writing then reading an astropy.table.Table containing a 2D column of strings: same problem with the dtype
import unittest
from uuid import uuid4
import os

import numpy as np
from astropy.io import fits
from astropy.table import Table, Column

class _TestTable(unittest.TestCase):
    '''
    Base class for testing FITS tables.

    Subclass this and add setupClass to define attributes to test:
        data (numpy structured array)
        table (astropy Table)
    '''

    def setUp(self):
        self.testfile = 'test-{}.fits'.format(uuid4())

    def tearDown(self):
        if os.path.exists(self.testfile):
            os.remove(self.testfile)

    def test_fits(self):
        fits.writeto(self.testfile, self.data)
        dx = fits.getdata(self.testfile)
        self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
        self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
        self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
        self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))

    def test_table(self):
        self.table.write(self.testfile)
        tx = Table.read(self.testfile)
        self.assertEqual(self.table['x'].dtype, tx['x'].dtype)
        self.assertEqual(self.table['y'].dtype, tx['y'].dtype)
        self.assertTrue(np.all(self.table['x'] == tx['x']), 'x: {} != {}'.format(self.table['x'], tx['x']))
        self.assertTrue(np.all(self.table['y'] == tx['y']), 'y: {} != {}'.format(self.table['y'], tx['y']))

class TestTable1DColumn(_TestTable):
    @classmethod
    def setUpClass(cls):
        dtype = [
            ('x', (str, 5)),    #- 1D column of 5-character strings
            ('y', (str, 3)),    #- !D column of 3-character strings
        ]
        data = np.zeros(2, dtype=dtype)
        data['x'] = ['abcde', 'xyz']
        data['y'] = ['A', 'BCX']
        table = Table(data)
        cls.data = data
        cls.table = table

    def test_manual_fits(self):
        '''Construct the BinTableHDU by hand'''
        col1 = fits.Column(name='x', format='5A', array=self.data['x'])
        col2 = fits.Column(name='y', format='3A', array=self.data['y'])
        cols = fits.ColDefs([col1, col2])
        tbhdu = fits.BinTableHDU.from_columns(cols)
        tbhdu.writeto(self.testfile)
        dx = fits.getdata(self.testfile)
        self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
        self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
        self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
        self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))


class TestTable2DColumn(_TestTable):
    @classmethod
    def setUpClass(cls):
        dtype = [
            ('x', (str, 5)),        #- 1D column of 5-character strings
            ('y', (str, 3), (4,)),  #- 2D column; each row is four 3-char strings
        ]
        data = np.zeros(2, dtype=dtype)
        data['x'] = ['abcde', 'xyz']
        data['y'][0] = ['A', 'BC', 'DEF', '123']
        data['y'][1] = ['X', 'YZ', 'PQR', '999']
        table = Table(data)
        cls.data = data
        cls.table = table

    #- I'm not sure how to create 2D column of strings by hand analogous to
    #- the 1D test case; see https://github.com/astropy/astropy/issues/5279
    # def test_manual_fits(self):
    #     '''Construct the BinTableHDU by hand
    #     
    #     When these data are successfully written under py2.7, they have keywords
    #     TTYPE1  = 'x       '
    #     TFORM1  = '5A      '
    #     TTYPE2  = 'y       '
    #     TFORM2  = '12A     '
    #     TDIM2   = '(3,4)   '
    #     
    #     Trying to replicate that logic here with dim=(3,4) doesn't work
    #     '''
    #     col1 = fits.Column(name='x', format='5A', array=self.data['x'])
    #     col2 = fits.Column(name='y', format='12A', array=self.data['y'], dim=(3,4))
    #     cols = fits.ColDefs([col1, col2])
    #     tbhdu = fits.BinTableHDU.from_columns(cols)
    #     tbhdu.writeto(self.testfile)
    #     dx = fits.getdata(self.testfile)
    #     self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
    #     self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
    #     self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
    #     self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))


if __name__ == '__main__':
    ### unittest.main()
    t1 = unittest.defaultTestLoader.loadTestsFromTestCase(TestTable1DColumn)
    t2 = unittest.defaultTestLoader.loadTestsFromTestCase(TestTable2DColumn)
    unittest.TextTestRunner(verbosity=2).run(unittest.TestSuite([t1,t2]))

Running these pass under python 2.7 but fail under python 3.5 with:

(py3) py3 $ python test-fits-table.py 
test_fits (__main__.TestTable1DColumn) ... WARNING: File may have been truncated: actual file length (8636) is smaller than the expected size (8640) [astropy.io.fits.file]
FAIL
test_manual_fits (__main__.TestTable1DColumn)
Construct the BinTableHDU by hand ... ok
test_table (__main__.TestTable1DColumn) ... ok
test_fits (__main__.TestTable2DColumn) ... WARNING: File may have been truncated: actual file length (8566) is smaller than the expected size (8640) [astropy.io.fits.file]
FAIL
test_table (__main__.TestTable2DColumn) ... FAIL

======================================================================
FAIL: test_fits (__main__.TestTable1DColumn)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test-fits-table.py", line 30, in test_fits
    self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
AssertionError: False is not true : x: ['abcde' 'xyz'] != ['abcde' 'zBCX']

======================================================================
FAIL: test_fits (__main__.TestTable2DColumn)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test-fits-table.py", line 29, in test_fits
    self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
AssertionError: dtype('<U3') != dtype('<U12')

======================================================================
FAIL: test_table (__main__.TestTable2DColumn)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test-fits-table.py", line 37, in test_table
    self.assertEqual(self.table['y'].dtype, tx['y'].dtype)
AssertionError: dtype('<U3') != dtype('<U12')

----------------------------------------------------------------------
Ran 5 tests in 0.100s

FAILED (failures=3)

EDIT: Added syntax highlighting.

Activity

  1. pllim commented on Aug 30, 2016

    @pllim
    Member

    Do they also fail in Python 2 when you add from __future__ import unicode_literals at the beginning of your test script?

  2. sbailey commented on Aug 30, 2016

    @sbailey
    ContributorAuthor

    Adding from __future__ import unicode_literals also required wrapping all column names with str(), because bizarrely numpy requires unicode columns names under python 3.5 but forbids it under python 2.7 (see numpy/numpy#2407). e.g.

            dtype = [
                (str('x'), (str, 5)),    #- 1D column of 5-character strings
                (str('y'), (str, 3)),    #- !D column of 3-character strings
            ]

    After making that update, the tests continue to pass under python 2.7 even with unicode_literals. However; unicode_literals only makes literals like 'abc' into unicode; the str type remains bytes and thus the example numpy arrays being created are using bytes not unicode, which I think is the key difference for these tests.

  3. sbailey commented on Aug 30, 2016

    @sbailey
    ContributorAuthor

    Using the hint from #5279, I was able to fix the commented out test:

        def test_manual_fits(self):
            '''Construct the BinTableHDU by hand
    
            When these data are successfully written under py2.7, they have keywords
            TTYPE1  = 'x       '
            TFORM1  = '5A      '
            TTYPE2  = 'y       '
            TFORM2  = '12A     '
            TDIM2   = '(3,4)   '
    
            Trying to replicate that logic here with dim=(3,4) doesn't work
            '''
            col1 = fits.Column(name=str('x'), format='5A', array=self.data['x'])
            col2 = fits.Column(name=str('y'), format='12A', array=self.data['y'], dim='(3,4)')
            cols = fits.ColDefs([col1, col2])
            tbhdu = fits.BinTableHDU.from_columns(cols)
            tbhdu.writeto(self.testfile)
            dx = fits.getdata(self.testfile)
            self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
            self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
            self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
            self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))

    This passes for both python 2.7 and 3.5, demonstrating that the low level tools are able to write these 2D columns of strings, but something is going wrong in the conversion of Table/ndarray objects into the fits.Column when using Table.write and fits.writeto.

  4. pllim commented on Aug 30, 2016

    @pllim
    Member

    @sbailey , please see my comments on #5268. Basically, I think test_fits() is not the right way to construct a proper FITS table.

  5. sbailey commented on Aug 30, 2016

    @sbailey
    ContributorAuthor

    I've updated the public gist to test 4 ways of writing/reading these tables:

    • convenience functions io.fits.writeto and io.fits.getdata
    • io.fits.BinTableHDU(data) and opening the file, extracting the data, closing the file
    • io.fits.BinTableHDU.from_columns() + io.fits.getdata
    • Table(data).write() and Table.read()

    They all pass on python 2.7, but 5 of the 8 tests fail under 3.5. From the documentation and other usage of these functions within astropy, I believe all of these are supposed to be valid ways of creating a FITS table.

  6. MSeifert04 commented on Aug 30, 2016

    @MSeifert04
    Contributor

    I've tested it and there seems happen some dtype conversion for from_columns:

    import numpy as np
    from astropy.io import fits
    from astropy.table import Table, Column
    
    dtype = [('x', (str, 5)),
             ('y', (str, 3)),]
    data = np.empty(2, dtype=dtype)
    data['x'] = ['abcde', 'xyz']
    data['y'] = ['A', 'BCX']
    

    The different ways:

    >>> col1 = fits.Column(name='x', format='5A', array=data['x'])
    >>> col2 = fits.Column(name='y', format='3A', array=data['y'])
    >>> cols = fits.ColDefs([col1, col2])
    >>> tbhdu = fits.BinTableHDU.from_columns(cols)
    >>> tbhdu.data
    FITS_rec([('abcde', 'A'), ('xyz', 'BCX')], 
          dtype=(numpy.record, [('x', 'S5'), ('y', 'S3')]))
    
    >>> fits.BinTableHDU.from_columns(data).data
    FITS_rec([('abcde', 'A'), ('xyz', 'BCX')], 
          dtype=(numpy.record, [('x', 'S5'), ('y', 'S3')]))
    
    >>> hdu = fits.convenience._makehdu(data, None) # equivalent to writeto-convenience
    >>> hdu.data
    FITS_rec([('abcde', 'A'), ('xyz', 'BCX')], 
          dtype=[('x', '<U5'), ('y', '<U3')])
    
    >>> fits.BinTableHDU(data, fits.Header()).data
    FITS_rec([('abcde', 'A'), ('xyz', 'BCX')], 
          dtype=[('x', '<U5'), ('y', '<U3')])
    

    I'm not sure which ways are supposed to work but those which do seem to convert the unicodes to strings. I suspect that the convenience functions and BinTableHDU try to make educated guesses about the data but can't handle those. I haven't looked at the internals yet.

  7. weaverba137 commented on Aug 30, 2016

    @weaverba137
    Member

    Please see my analysis of data.view(FITS_rec) in #5268, where data is a numpy array. I believe that, internally, the methods like fits.BinTableHDU or fits.writeto(filename, numpy_array) are both using data.view(FITS_rec) to create the data payload. I believe my test in #5268 is consistent with @MSeifert04's result.

  8. weaverba137 commented on Aug 30, 2016

    @weaverba137
    Member

    I also invite you to take a look at astropy.io.fits.util.decode_ascii(). Here's the snippet from that module:

    def decode_ascii(s):
        """
        In Python 2 this is a no-op.  Strings are left alone.  In Python 3 this
        will be replaced with a function that actually decodes ascii bytes to
        unicode.
        """
    
        return s

    So, when exactly is this going to be replaced with a function for Python 3?

  9. MSeifert04 commented on Aug 30, 2016

    @MSeifert04
    Contributor

    So, when exactly is this going to be replaced with a function for Python 3?

    @weaverba137 here?

  10. weaverba137 commented on Aug 30, 2016

    @weaverba137
    Member

    Wow. Just. Dang. Cripes. No wonder no one wants to take over this code.

  11. weaverba137 commented on Aug 31, 2016

    @weaverba137
    Member

    I can go into truly exhaustive detail if needed, but I think I have a clue as to what is going on.

    Short version:

    Writing numpy arrays with unicode columns (kind == 'U') directly to FITS binary tables in a file skips over a conversion process that (inadvertently) preserves the correct width of the column in the binary output.

    Longer version:

    1. When Columns are created explicitly, combined into a ColDefs object, and a HDU is created with BinTableHDU.from_columns(), any unicode columns get converted to Numpy bytes, that is, columns with kind == 'S'. This happens in both 2.7 & 3.5.
    2. When a BinTableHdu is created directly from a numpy array (recarray, etc), this conversion does not happen, so there are some columns that have kind == 'U'.
    3. When writing to a file, if a BinTableHDU data payload (the .data attribute) has unicode columns (kind == 'U'), the data are written row-by-row, and column-by-column within a row.
    4. Unicode "cells" (remember we're writing one column within one row at a time) get converted to bytes using the equivalent of encoded_item = np.char.encode(item, 'ascii'), and then written to the output stream with encoded_item.tofile(fileobj).
    5. The number of bytes written is not necessarily the same as the actual width of the column. For example, say a column is supposed to have width '5A' (FITS), equivalently, 'S5' (Numpy), but one particular item within that column is only three bytes wide, say 'foo'. With the row-by-row, item-by-item method, only three bytes are written instead of five. That is 'foo' will be written to the output stream, instead of 'foo\0\0'.
    6. Since the columns don't have the correct width, the resulting file won't have the correct size.
    7. Coming full circle back to item 1, because the explicitly-created object doesn't have unicode columns, it does not go through the row-by-row writing process, so the byte-widths of the columns are preserved correctly.
    8. Note however, that if the explicitly-created object had somehow been forced to go through the row-by-row writing method, it too would have incorrectly-sized columns. That's why I used the word "inadvertently" in the short version.

    Exhaustive version: by request.

  12. cmarmo commented on Feb 15, 2022

    @cmarmo
    Member

    I have run the test of this issue against the current version on main.
    The exact same error is thrown.

    Test script
    #- unit tests of astropy.io.fits
    #- Stephen Bailey, Lawrence Berkeley Lab
    
    from __future__ import unicode_literals
    
    import unittest
    from uuid import uuid4
    import os
    
    import numpy as np
    from astropy.io import fits
    from astropy.table import Table, Column
    
    class _TestTable(unittest.TestCase):
        '''
        Base class for testing FITS tables.
        Subclass this and add setupClass to define attributes to test:
            data (numpy structured array)
            table (astropy Table)
        '''
    
        def setUp(self):
            self.testfile = 'test-{}.fits'.format(uuid4())
    
        def tearDown(self):
            if os.path.exists(self.testfile):
                os.remove(self.testfile)
    
        def test_fits_writeto(self):
            '''test convenience functions io.fits.writeto / getdata'''
            fits.writeto(self.testfile, self.data)
            dx = fits.getdata(self.testfile)
            self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
            self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
            self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
            self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
        
        def test_fits_bintable(self):
            '''Test fits.BinTableHDU(data) and avoid convenience functions'''
            hdu0 = fits.PrimaryHDU()
            hdu1 = fits.BinTableHDU(self.data)
            hx = fits.HDUList([hdu0, hdu1])
            hx.writeto(self.testfile)
            # dx = fits.getdata(self.testfile)
            fx = fits.open(self.testfile)
            dx = fx[1].data
            fx.close()
            self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
            self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
            self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
            self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
    
        def test_table(self):
            self.table.write(self.testfile)
            tx = Table.read(self.testfile)
            self.assertEqual(self.table['x'].dtype, tx['x'].dtype)
            self.assertEqual(self.table['y'].dtype, tx['y'].dtype)
            self.assertTrue(np.all(self.table['x'] == tx['x']), 'x: {} != {}'.format(self.table['x'], tx['x']))
            self.assertTrue(np.all(self.table['y'] == tx['y']), 'y: {} != {}'.format(self.table['y'], tx['y']))
    
    class TestTable1DColumn(_TestTable):
        @classmethod
        def setUpClass(cls):
            dtype = [
                (str('x'), (str, 5)),    #- 1D column of 5-character unicode strings (Python3)
                (str('y'), (str, 3)),    #- !D column of 3-character unicode strings (Python3)
            ]
            data = np.zeros(2, dtype=dtype)
            data['x'] = ['abcde', 'xyz']
            data['y'] = ['A', 'BCX']
            table = Table(data)
            cls.data = data
            cls.table = table
    
        def test_fits_columns(self):
            '''Test BinTableHDU.from_columns()'''
            col1 = fits.Column(name=str('x'), format='5A', array=self.data['x'])
            col2 = fits.Column(name=str('y'), format='3A', array=self.data['y'])
            cols = fits.ColDefs([col1, col2])
            tbhdu = fits.BinTableHDU.from_columns(cols)
            tbhdu.writeto(self.testfile)
            dx = fits.getdata(self.testfile)
            self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
            self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
            self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
            self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
            
    
    class TestTable2DColumn(_TestTable):
        @classmethod
        def setUpClass(cls):
            dtype = [
                (str('x'), (str, 5)),        #- 1D column of 5-character unicode strings
                (str('y'), (str, 3), (4,)),  #- 2D column; each row is four 3-char unicode strings
            ]
            data = np.zeros(2, dtype=dtype)
            data['x'] = ['abcde', 'xyz']
            data['y'][0] = ['A', 'BC', 'DEF', '123']
            data['y'][1] = ['X', 'YZ', 'PQR', '999']
            table = Table(data)
            cls.data = data
            cls.table = table
    
        def test_fits_columns(self):
            '''Test BinTableHDU.from_columns()'''
            col1 = fits.Column(name=str('x'), format='5A', array=self.data['x'])
            col2 = fits.Column(name=str('y'), format='12A', array=self.data['y'], dim='(3,4)')
            cols = fits.ColDefs([col1, col2])
            tbhdu = fits.BinTableHDU.from_columns(cols)
            tbhdu.writeto(self.testfile)
            dx = fits.getdata(self.testfile)
            self.assertEqual(self.data['x'].dtype, dx['x'].dtype)
            self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
            self.assertTrue(np.all(self.data['x'] == dx['x']), 'x: {} != {}'.format(self.data['x'], dx['x']))
            self.assertTrue(np.all(self.data['y'] == dx['y']), 'y: {} != {}'.format(self.data['y'], dx['y']))
    
                    
    if __name__ == '__main__':
        ### unittest.main()
        t1 = unittest.defaultTestLoader.loadTestsFromTestCase(TestTable1DColumn)
        t2 = unittest.defaultTestLoader.loadTestsFromTestCase(TestTable2DColumn)
        unittest.TextTestRunner(verbosity=2).run(unittest.TestSuite([t1, t2]))
    Here is the traceback:
    (astropydevenv) [cmarmo@2603-800c-2300-a5d0-d22b-4854-594d-178b tests]$ python 5280-astropy.py 
    test_fits_bintable (__main__.TestTable1DColumn)
    Test fits.BinTableHDU(data) and avoid convenience functions ... ok
    test_fits_columns (__main__.TestTable1DColumn)
    Test BinTableHDU.from_columns() ... ok
    test_fits_writeto (__main__.TestTable1DColumn)
    test convenience functions io.fits.writeto / getdata ... ok
    test_table (__main__.TestTable1DColumn) ... FAIL
    test_fits_bintable (__main__.TestTable2DColumn)
    Test fits.BinTableHDU(data) and avoid convenience functions ... FAIL
    test_fits_columns (__main__.TestTable2DColumn)
    Test BinTableHDU.from_columns() ... ok
    test_fits_writeto (__main__.TestTable2DColumn)
    test convenience functions io.fits.writeto / getdata ... FAIL
    test_table (__main__.TestTable2DColumn) ... FAIL
    
    ======================================================================
    FAIL: test_table (__main__.TestTable1DColumn)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cmarmo/software/tests/5280-astropy.py", line 56, in test_table
        self.assertEqual(self.table['x'].dtype, tx['x'].dtype)
    AssertionError: dtype('<U5') != dtype('S5')
    
    ======================================================================
    FAIL: test_fits_bintable (__main__.TestTable2DColumn)
    Test fits.BinTableHDU(data) and avoid convenience functions
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cmarmo/software/tests/5280-astropy.py", line 49, in test_fits_bintable
        self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
    AssertionError: dtype('<U3') != dtype('<U12')
    
    ======================================================================
    FAIL: test_fits_writeto (__main__.TestTable2DColumn)
    test convenience functions io.fits.writeto / getdata
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cmarmo/software/tests/5280-astropy.py", line 34, in test_fits_writeto
        self.assertEqual(self.data['y'].dtype, dx['y'].dtype)
    AssertionError: dtype('<U3') != dtype('<U12')
    
    ======================================================================
    FAIL: test_table (__main__.TestTable2DColumn)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/home/cmarmo/software/tests/5280-astropy.py", line 56, in test_table
        self.assertEqual(self.table['x'].dtype, tx['x'].dtype)
    AssertionError: dtype('<U5') != dtype('S5')
    
    ----------------------------------------------------------------------
    Ran 8 tests in 0.067s
    
    FAILED (failures=4)

    I could fix the AssertionError: dtype('<U5') != dtype('S5') modifying test_table as following

        def test_table(self):
            self.table.write(self.testfile)
            tx = Table.read(self.testfile, character_as_bytes=False)
            self.assertEqual(self.table['x'].dtype, tx['x'].dtype)
            self.assertEqual(self.table['y'].dtype, tx['y'].dtype)
            self.assertTrue(np.all(self.table['x'] == tx['x']), 'x: {} != {}'.format(self.table['x'], tx['x']))
            self.assertTrue(np.all(self.table['y'] == tx['y']), 'y: {} != {}'.format(self.table['y'], tx['y']))

    If I understand correctly, this is due to the fact that string dtypes are written in unicode by numpy, but the astropy read_table_fits in connect.py reads by default in binary. Adding character_as_bytes=False will read in unicode.
    This is not a bug, perhaps this could be made more explicit in the documentation.

    I was unable to fix the AssertionError: dtype('<U3') != dtype('<U12') but I tracked it to table.py.
    This is where columns move from the format [('x', 'S5'), ('y', 'S3', (4,))] to

    ColDefs(
        name = 'x'; format = '5A'
        name = 'y'; format = '48A'; dim = '(12,4)'
    )
    

    I guess, there is some automatic mechanism to convert to ColDefs that I couldn't understand.

    Hope this can help in moving forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions