Numpy 1.10 issues for io.fits

@embray

@embray - sadly, it appears in some recent change in numpy-dev again broke fits. I very much suspect numpy/numpy#5947.

But it may be good to check whether numpy/numpy#5943 resolves it.

Maybe Numpy should include astropy.test() into their test suite ;)

For the record, numpy/numpy#5943 does not resolve the issues.

@charris

@charris commented (numpy/numpy#5962 (comment)) that

Fixing the astropy.io.fits problem looks like a 1.10 blocker. Could you spend some time on that?

I'll try, but I fear I may need help! @embray?

Could you provide some more details? What test is broken?

I don't see why one of our tests failing is necessarily a blocker for Numpy. astropy.io.fits does some funky stuff anyways that may be stretching the intended use (or at the very least has workarounds).

@embray

@embray I'll let you folks decide on the severity. My thought was that io.fits is probably one of the more used packages in astropy at this time.

I just looked at the latest test build on Travis against numpy-dev (https://travis-ci.org/astropy/astropy/jobs/67227038) and didn't see any io.fits failures, so I'm not sure what this is referring to.

Oh, I get some failures locally though. I guess maybe whatever was in that Travis build wasn't bleeding edge.

Oh, scratch that. I was on another branch of Astropy that didn't have my more recent fixes for Numpy. On Python 2 all the tests pass locally with Numpy-dev. However, I can confirm that there are test failures on Python 3 (which would explain why they didn't show up in Travis since we only have it testing against numpy-dev on Python 2).

I don't think it's anything too severe though. It appears to be an issue with string handling.

@embray

@embray - thanks for checking. It does seem to be just a python3 issue.

But a very odd one: lots of strings just "disappear"! The least convoluted test that shows the problem is
test_table.TestTableFunctions.test_string_column_padding. Minimal test case

from astropy.io import fits; from numpy import char as chararray
acol = fits.Column(name='MEMNAME', format='A10', array=chararray.array(['img1', 'p']))
acol.array
# chararray(['img1', 'p'], dtype='<U10')
ahdu = fits.BinTableHDU.from_columns([acol])
ahdu.data
# FITS_rec([(''), ('')], dtype=(numpy.record, [('MEMNAME', 'S10')]))

I quickly checked whether it mattered whether I passed unicode=False to the chararray definition, but while this changes acol.array it does not affect the result.

The problem seems to be that in FITS_rec.from_columns, outarr = field[:n] leads to an object which is not a view of field (np.may_share_memory(field[:], field) returns False). Not quite sure why that is, though.

The lack of memory sharing happens in recarray.__getitem__ --
https://github.com/numpy/numpy/blob/master/numpy/core/defchararray.py#L1850
Specifically,in the temp = val.rstrip() stanza...

But, darn, that seems to be true for numpy 1.8 already. So, perhaps the problem is that field is not quite the right type of object? Well, too sleepy now to investigate further; hopefully, I at least managed to narrow it down a bit...

p.s. Obviously, it may well be better to handle this further up, in ensuring a single item from the field names is always returned as a string. But hopefully the above localization helps.

I figured that was the "one" that still failed :)

@mhvk

@mhvk

Obviously, it may well be better to handle this further up, in ensuring a single item from the field names is always returned as a string. But hopefully the above localization helps.

Indeed, I think this is actually indicating something I need to fix further up. To demonstrate, if I do the following I can create an ASCII table from a binary table like so:

>>> h = fits.open('astropy/io/fits/tests/data/tb.fits')
>>> ascii_table = fits.TableHDU(data=h[1].data)

However, if you compare the resulting data in the ASCII table with the original binary table, you see that in the ASCII table the string field is returned as bytes instead of strings:

FITS_rec([(1, 'abc', 3.7000000715255736, False),
       (2, 'xy', 6.6999997138977054, True)], 
      dtype=[('c1', '>i4'), ('c2', 'S3'), ('c3', '>f4'), ('c4', 'i1')])
>>> ascii_table.data
FITS_rec([(1, b'abc', 1.1, 'F'), (2, b'xy ', 2.0999999, 'T')], 
      dtype=(numpy.record, [('c1', 'S11'), ('c2', 'S3'), ('c3', 'S15'), ('c4', 'S1')]))

A further oddity is that if I open the original binary table, print the data first, and then create the ASCII table it outright crashes:

>>> h = fits.open('astropy/io/fits/tests/data/tb.fits')
>>> h[1].data
FITS_rec([(1, 'abc', 3.7000000715255736, False),
       (2, 'xy', 6.6999997138977054, True)], 
      dtype=[('c1', '>i4'), ('c2', 'S3'), ('c3', '>f4'), ('c4', 'i1')])
>>> ascii_table = fits.TableHDU(data=h[1].data)
ERROR: AttributeError: '_AsciiColumnFormat' object has no attribute 'p_format' [astropy.io.fits.fitsrec]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/internal/1/root/src/astropy/astropy/astropy/io/fits/hdu/table.py", line 702, in __init__
    super(TableHDU, self).__init__(data, header, name=name)
  File "/internal/1/root/src/astropy/astropy/astropy/io/fits/hdu/table.py", line 303, in __init__
    self.data = data
  File "/internal/1/root/src/astropy/astropy/astropy/utils/decorators.py", line 522, in __set__
    ret = self.fset(obj, val)
  File "/internal/1/root/src/astropy/astropy/astropy/io/fits/hdu/table.py", line 438, in data
    data = FITS_rec.from_columns(new_columns)
  File "/internal/1/root/src/astropy/astropy/astropy/io/fits/fitsrec.py", line 469, in from_columns
    columns._arrays[idx] = data.field(idx)
  File "/internal/1/root/src/astropy/astropy/astropy/io/fits/fitsrec.py", line 654, in field
    converted = self._convert_other(column, field, recformat)
  File "/internal/1/root/src/astropy/astropy/astropy/io/fits/fitsrec.py", line 832, in _convert_other
    if not column.ascii and column.format.p_format:
AttributeError: '_AsciiColumnFormat' object has no attribute 'p_format'

I don't understand why it does this in this case, and not in the previous case. But I suspect the bugs are related.

@mhvk

@mhvk
Meanwhile, even with your patch to the .view() method, I'm getting a bunch of this exception when I run the tests:

astropy/io/fits/tests/test_connect.py:127: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
astropy/io/fits/hdu/base.py:1766: in writeto
    checksum=checksum)
astropy/io/fits/hdu/hdulist.py:691: in writeto
    hdu._writeto(hdulist._file)
astropy/io/fits/hdu/base.py:743: in _writeto
    data_offset, data_size = self._writedata(fileobj)
astropy/io/fits/hdu/base.py:680: in _writedata
    size += self._writedata_internal(fileobj)
astropy/io/fits/hdu/table.py:858: in _writedata_internal
    self._writedata_by_row(fileobj)
astropy/io/fits/hdu/table.py:908: in _writedata_by_row
    item = field[idx]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = chararray(['a', 'b', 'c', 'd'], 
      dtype='<U1'), obj = 0

    def __getitem__(self, obj):
>       val = ndarray.__getitem__(self, obj)
E       UnicodeDecodeError: 'utf-32-le' codec can't decode bytes in position 0-3: codepoint not in range(0x110000)

granted, I'm still actually on an older dev version. So let me update to Numpy 1.10 final and see what I get...

No, I still get those failures with Numpy 1.10 final. But it occurs to me now that I'm working on a branch where I've made other changes, so this failure might actually be related to other changes I've made in the effort to improve unicode handling in tables. I'll figure it out...

It seems likely that is indeed related to the other changes, since I got zero failures with the two changes mentioned above (where I agree the second is probably treating the symptom rather than the cause).

I figured it out. It was an issue related to byte swapping

Should be fixed by #4228

Changed title so that people will find there is an existing issue with numpy 1.10.

See #4266 for one outstanding issue still.

@mhvk

@mhvk @embray - just to be clear, do we think it's ok to go ahead with v1.1 without #4266 fixed? (Or does anyone know how to fix #4266 now?) I see that the skip mentioned in #4266 was already backported to v1.1, so it should work, but the real question is if there's a genuine issue that needs to be addressed... If we're ok with delaying that to 1.1.1, I'd say we can close this, right?

@embray

Not sure - I'll leave this for @embray ...

I'll try to fix it ASAP, but it's not a blocker. It's fine to have the test marked xfail for now.

Navigation Menu

Repository navigation

Numpy 1.10 issues for io.fits #3854

39 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Numpy 1.10 issues for io.fits #3854

Description

Activity

mhvk commented on Jun 14, 2015

astrofrog commented on Jun 15, 2015

mhvk commented on Jun 17, 2015

mhvk commented on Jun 17, 2015

embray commented on Jun 17, 2015

charris commented on Jun 17, 2015

embray commented on Jun 17, 2015

embray commented on Jun 17, 2015

embray commented on Jun 17, 2015

mhvk commented on Jun 17, 2015

mhvk commented on Jun 18, 2015

mhvk commented on Jun 18, 2015

mhvk commented on Jun 18, 2015

39 remaining items

mhvk commented on Oct 6, 2015

embray commented on Oct 6, 2015

embray commented on Oct 6, 2015

embray commented on Oct 6, 2015

embray commented on Oct 6, 2015

mhvk commented on Oct 6, 2015

embray commented on Oct 7, 2015

embray commented on Oct 7, 2015

mhvk commented on Oct 19, 2015

mhvk commented on Oct 23, 2015

eteq commented on Nov 9, 2015

mhvk commented on Nov 9, 2015

embray commented on Nov 12, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions