astropy.io.fits failures with Numpy dev

@embray

@embray - FYI there are currently four test failures for astropy.io.fits with the developer version of Numpy:

https://jenkins.shiningpanda-ci.com/astropy/view/MacOS%20X/job/astropy-master-numpydev-osx-10.8-multiconfig/

Those are already fixed in PyFITS. I just need to merge the changes over.

@embray

@embray - were those changes merged over? At the moment, the astropy.io.fits tests are crashing and causing a segfault with Numpy dev (on MacOS 10.8) - see above link. Any idea if this is related?

I actually have no memory of what this issue might have been. It doesn't help that the output from those builds doesn't show what the actual failure is. Can you reproduce this more simply?

I don't remember ever seeing a segfault though.

Let me see if I can reproduce it outside Jenkins.

The segfault is in

astropy/io/fits/tests/test_table.py:1974: TestTableFunctions.test_array_broadcasting

Investigating some more.

That's a pretty odd one for it to segfault on. I have no idea why I said this was fixed. I do have some vague recollection of fixing something that was happening with Numpy-dev, but now I have no idea what it was so it's a good thing you pinged me again on it. I'll see if I can reproduce this on Loonix as well.

Er, actually a different test is failing every time...

Well.

The good news is I can reproduce this on Linux so it's not just OSX. Though I do see it dying in test_array_broadcasting every time.

Right, most of the time it's dying there. I think it's just a stdout issue when it doesn't (i.e. it doesn't have time to print to stdout). Great that you can reproduce it!

But if I remove all methods before test_array_broadcasting then it actually passes test_array_broadcasting and crashes on test_dump_load_round_trip

Well there's also the matter of one test, test_mask_array that is failing. And then it segfaults in that later test. So I suspect it has something to do with that file. Or maybe more specifically to do with ASCII tables (since that is one).

I can confirm the segfault is fixed now. For the record, the remaining failure is:

=================================== FAILURES ===================================
______________________ TestTableFunctions.test_mask_array ______________________

self = <astropy.io.fits.tests.test_table.TestTableFunctions object at 0x109af4090>

    def test_mask_array(self):
        t = fits.open(self.data('table.fits'))
        tbdata = t[1].data
        mask = tbdata.field('V_mag') > 12
        newtbdata = tbdata[mask]
        hdu = fits.BinTableHDU(newtbdata)
>       hdu.writeto(self.temp('newtable.fits'))

astropy/io/fits/tests/test_table.py:926: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astropy.io.fits.hdu.table.BinTableHDU object at 0x109af4ed0>
name = '/var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/fits-test-UJJ2f4/newtable.fits'
output_verify = 'exception', clobber = False, checksum = False

    def writeto(self, name, output_verify='exception', clobber=False,
                checksum=False):
        """
            Works similarly to the normal writeto(), but prepends a default
            `PrimaryHDU` are required by extension HDUs (which cannot stand on
            their own).
            """

        from .hdulist import HDUList
        from .image import PrimaryHDU

        hdulist = HDUList([PrimaryHDU(), self])
        hdulist.writeto(name, output_verify, clobber=clobber,
>                       checksum=checksum)

astropy/io/fits/hdu/base.py:1705: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x109af4c10>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x109af4ed0>]
fileobj = <astropy.io.fits.file._File <open file '/var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/fits-test-UJJ2f4/newtable.fits', mode 'wb' at 0x108233420>>
output_verify = 'exception', clobber = False, checksum = False

    def writeto(self, fileobj, output_verify='exception', clobber=False,
                checksum=False):
        """
            Write the `HDUList` to a new file.

            Parameters
            ----------
            fileobj : file path, file object or file-like object
                File to write to.  If a file object, must be opened in a
                writeable mode.

            output_verify : str
                Output verification option.  Must be one of ``"fix"``,
                ``"silentfix"``, ``"ignore"``, ``"warn"``, or
                ``"exception"``.  See :ref:`verify` for more info.

            clobber : bool
                When `True`, overwrite the output file if exists.

            checksum : bool
                When `True` adds both ``DATASUM`` and ``CHECKSUM`` cards
                to the headers of all HDU's written to the file.
            """

        if (len(self) == 0):
            warnings.warn("There is nothing to write.", AstropyUserWarning)
            return

        self.verify(option=output_verify)

        # make sure the EXTEND keyword is there if there is extension
        self.update_extend()

        # make note of whether the input file object is already open, in which
        # case we should not close it after writing (that should be the job
        # of the caller)
        closed = fileobj_closed(fileobj)

        # writeto is only for writing a new file from scratch, so the most
        # sensible mode to require is 'ostream'.  This can accept an open
        # file object that's open to write only, or in append/update modes
        # but only if the file doesn't exist.
        fileobj = _File(fileobj, mode='ostream', clobber=clobber)
        hdulist = self.fromfile(fileobj)

        for hdu in self:
>           hdu._prewriteto(checksum=checksum)

astropy/io/fits/hdu/hdulist.py:653: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astropy.io.fits.hdu.table.BinTableHDU object at 0x109af4ed0>
checksum = False, inplace = False

    def _prewriteto(self, checksum=False, inplace=False):
        if self._has_data:
>           self.data._scale_back()

astropy/io/fits/hdu/table.py:386: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = FITS_rec([('NGC1002', 12.3), ('NGC1003', 15.2)], 
      dtype=[('target', 'S20'), ('V_mag', '>f4')])

    def _scale_back(self):
        """
            Update the parent array, using the (latest) scaled array.
            """

        # Running total for the new heap size
        heapsize = 0

        for indx in range(len(self.dtype.names)):
            recformat = self._coldefs._recformats[indx]
            field = super(FITS_rec, self).field(indx)

            # add the location offset of the heap area for each
            # variable length column
            if isinstance(recformat, _FormatP):
                # Irritatingly, this can return a different dtype than just
                # doing np.dtype(recformat.dtype); but this returns the results
                # that we want.  For example if recformat.dtype is 'a' we want
                # an array of characters.
                dtype = np.array([], dtype=recformat.dtype).dtype

                if self._convert[indx] is not None:
                    # The VLA has potentially been updated, so we need to
                    # update the array descriptors
                    field[:] = 0  # reset
                    npts = [len(arr) for arr in self._convert[indx]]

                    field[:len(npts), 0] = npts
                    field[1:, 1] = (np.add.accumulate(field[:-1, 0]) *
                                    dtype.itemsize)
                    field[:, 1][:] += heapsize

                heapsize += field[:, 0].sum() * dtype.itemsize
                # Even if this VLA has not been read, we need to include
                # the size of its constituent arrays in the heap size total

            if self._convert[indx] is None:
                continue

            if isinstance(recformat, _FormatX):
                _wrapx(self._convert[indx], field, recformat.repeat)
                continue

            _str, _bool, _number, _scale, _zero, bscale, bzero, _ = \
                self._get_scale_factors(indx)

            # conversion for both ASCII and binary tables
            if _number or _str:
                column = self._coldefs[indx]
                if _number and (_scale or _zero) and column._physical_values:
                    dummy = self._convert[indx].copy()
                    if _zero:
                        dummy -= bzero
                    if _scale:
                        dummy /= bscale
                    # This will set the raw values in the recarray back to
                    # their non-physical storage values, so the column should
                    # be mark is not scaled
                    column._physical_values = False
                elif _str:
                    dummy = self._convert[indx]
                elif isinstance(self._coldefs, _AsciiColDefs):
                    dummy = self._convert[indx]
                else:
                    continue

                # ASCII table, convert numbers to strings
                if isinstance(self._coldefs, _AsciiColDefs):
                    starts = self._coldefs.starts[:]
                    spans = self._coldefs.spans
                    format = self._coldefs.formats[indx].strip()

                    # The the index of the "end" column of the record, beyond
                    # which we can't write
                    end = super(FITS_rec, self).field(-1).itemsize
                    starts.append(end + starts[-1])

                    if indx > 0:
                        lead = (starts[indx] - starts[indx - 1] -
                                spans[indx - 1])
                    else:
                        lead = 0

                    if lead < 0:
                        warnings.warn(
                            'Column %r starting point overlaps the '
                            'previous column.' % (indx + 1))

                    trail = starts[indx + 1] - starts[indx] - spans[indx]

                    if trail < 0:
                        warnings.warn(
                            'Column %r ending point overlaps the next '
                            'column.' % (indx + 1))

                    # TODO: It would be nice if these string column formatting
                    # details were left to a specialized class, as is the case
                    # with FormatX and FormatP
                    if 'A' in format:
                        _pc = '%-'
                    else:
                        _pc = '%'

                    fmt = ''.join([_pc, format[1:], ASCII2STR[format[0]],
                                   (' ' * trail)])

                    # not using numarray.strings's num2char because the
                    # result is not allowed to expand (as C/Python does).
                    for jdx in xrange(len(dummy)):
                        x = fmt % dummy[jdx]
                        if len(x) > starts[indx + 1] - starts[indx]:
                            raise ValueError(
                                "Value %r does not fit into the output's "
                                "itemsize of %s." % (x, spans[indx]))
                        else:
                            field[jdx] = x
                    # Replace exponent separator in floating point numbers
                    if 'D' in format:
                        field.replace(encode_ascii('E'), encode_ascii('D'))
                # binary table
                else:
                    if len(field) and isinstance(field[0], np.integer):
                        dummy = np.around(dummy)
                    elif isinstance(field, np.chararray):
                        # Ensure that blanks at the end of each string are
                        # converted to nulls instead of spaces, see Trac #15
                        # and #111
                        itemsize = dummy.itemsize
                        if dummy.dtype.kind == 'U':
                            pad = self._coldefs._padding_byte
                        else:
                            pad = self._coldefs._padding_byte.encode('ascii')

                        for idx in xrange(len(dummy)):
                            val = dummy[idx]
>                           dummy[idx] = val + (pad * (itemsize - len(val)))
E                           ValueError: assignment destination is read-only

astropy/io/fits/fitsrec.py:986: ValueError
==================== 1 failed, 431 passed in 43.81 seconds =====================

Right, I'm aware of that one. It's a result from a change in Numpy in how the readonly flag is applied to views/slices of an array subclass. I'm still not entirely sure that the change is intentional though so I need to go through Numpy again and see when this changed and why.

With #2693 resolved I think there are no further issues here. Feel free to reopen, or open a new ticket if any arise.

Navigation Menu

Repository navigation

astropy.io.fits failures with Numpy dev #1987

19 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

astropy.io.fits failures with Numpy dev #1987

Description

Activity

embray commented on Jan 22, 2014

astrofrog commented on Mar 12, 2014

embray commented on Mar 12, 2014

embray commented on Mar 12, 2014

astrofrog commented on Mar 12, 2014

astrofrog commented on Mar 12, 2014

embray commented on Mar 12, 2014

astrofrog commented on Mar 12, 2014

embray commented on Mar 12, 2014

embray commented on Mar 12, 2014

astrofrog commented on Mar 12, 2014

astrofrog commented on Mar 12, 2014

embray commented on Mar 12, 2014

19 remaining items

astrofrog commented on Apr 5, 2014

embray commented on Apr 7, 2014

embray commented on Sep 18, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions