Issue with certain compressed FITS files with astropy.io.fits

I am using astropy.io.fits to read in many FITS files from the UKIDSS GPS survey, which uses RICE compression to reduce their file size. They warn on their website that a recent version of cftisio is required:

When trying to view or uncompress our FITS files, in some rare cases you may come across one or both of the following error messages:

FITSIO status = 1: non-CFITSIO program error

decompression error: hit end of compressed byte stream
This is because our FITS file images have been compressed using a newer version of fpack & CFITSIO. The only solution is to first uncompress the FITS file using a version of CFITSIO >= 3.1 (fpack >= 1.0) prior to viewing.

Here is an example file that is causing an issue (these represent 5% of all files I've been using):

http://mpia.de/~robitaille/astropy/w20070425_02461_sf_st.fit

The error occurs if I try and access the 4th HDU:

In [8]: fits.getdata('w20070425_02461_sf_st.fit', 4)
ERROR: ValueError: mmap length is greater than file size [numpy.core.memmap]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/Volumes/Raptor/tmp/ukidss_bug/<ipython-input-8-6ac880274a55> in <module>()
----> 1 fits.getdata('w20070425_02461_sf_st.fit', 4)

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/convenience.pyc in getdata(filename, *args, **kwargs)
    186     hdulist, extidx = _getext(filename, mode, *args, **kwargs)
    187     hdu = hdulist[extidx]
--> 188     data = hdu.data
    189     if data is None and extidx == 0:
    190         try:

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/utils/misc.pyc in __get__(self, obj, owner)
    256         key = self._fget.func_name
    257         if key not in obj.__dict__:
--> 258             val = self._fget(obj)
    259             obj.__dict__[key] = val
    260             return val

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/compressed.pyc in data(self)
    901     def data(self):
    902         # The data attribute is the image data (not the table data).

--> 903         data = compression.decompress_hdu(self)
    904 
    905         # Scale the data if necessary


/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/utils/misc.pyc in __get__(self, obj, owner)
    256         key = self._fget.func_name
    257         if key not in obj.__dict__:
--> 258             val = self._fget(obj)
    259             obj.__dict__[key] = val
    260             return val

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/compressed.pyc in compData(self)
    946         # First we will get the table data (the compressed

    947         # data) from the file, if there is any.

--> 948         compData = super(BinTableHDU, self).data
    949         if isinstance(compData, np.rec.recarray):
    950             del self.data

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/utils/misc.pyc in __get__(self, obj, owner)
    256         key = self._fget.func_name
    257         if key not in obj.__dict__:
--> 258             val = self._fget(obj)
    259             obj.__dict__[key] = val
    260             return val

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/table.pyc in data(self)
    229     @lazyproperty
    230     def data(self):
--> 231         data = self._get_tbdata()
    232         data._coldefs = self.columns
    233         data.formats = self.columns.formats

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/table.pyc in _get_tbdata(self)
     81             self._datSpan > self._theap):
     82             # We have a heap; include it in the raw_data

---> 83             raw_data = self._get_raw_data(self._datSpan, np.byte, self._datLoc)
     84             data = raw_data[:self._theap].view(dtype=dtype,
     85                                                type=np.rec.recarray)

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/base.pyc in _get_raw_data(self, shape, code, offset)
    366                               offset=offset)
    367         else:
--> 368             return self._file.readarray(offset=offset, dtype=code, shape=shape)
    369 
    370     # TODO: Rework checksum handling so that it's not necessary to add a


/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/file.pyc in readarray(self, size, offset, dtype, shape)
    269             return Memmap(self.__file, offset=offset,
    270                           mode=MEMMAP_MODES[self.mode], dtype=dtype,
--> 271                           shape=shape).view(np.ndarray)
    272         else:
    273             count = reduce(lambda x, y: x * y, shape)

/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/numpy/core/memmap.pyc in __new__(subtype, filename, dtype, mode, offset, shape, order)
    251             bytes -= start
    252             offset -= start
--> 253             mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
    254         else:
    255             mm = mmap.mmap(fid.fileno(), bytes, access=acc)

ValueError: mmap length is greater than file size

However, if I download and compile the latest cfitsio, then do make imcopy and use imcopy to decompress the file, it works fine. Therefore, I think there must be some bug/issue in astropy.io.fits - I'm not sure if it's to do with pyfits or the bundling of cfitsio in astropy, so I just thought I'd report it here.

When I open the file I get the warning:

File may have been truncated: actual file length (51388416) is smaller than the expected size (63106560)

Do you get this too? The other HDUs decompress fine, and the error you're getting strongly suggests that the header is reporting a larger size for the file than it actually contains data-wise. So it's strange that imcopy doesn't complain about it.

I think the file may be corrupt - I will double check tomorrow, and will close the issue if so - sorry!

It's still strange that CFITSIO had no problem with it. It's possible that it's just being more permissive with truncated images. I think, if that's the case, PyFITS should try to be too, and could be fairly easily. If nothing else a more useful error message should be raised.

And yeah, after going over the header and manipulating the file a bit I'm also convinced that it's truncated. There are several thousand bytes missing from the heap that should be there.

@iguananaut

@iguananaut - weird, I went back to re-download this specific file, and it has a different checksum and works, so it's possible that the original file got truncated during download. There is still the question of how cfitsio can manage to read in this file anyway. But I wouldn't worry about it too much. Certainly not 0.2-worthy, so I'm tagging it as 'Future'.

I guess so. I would like to see what went on here. You say that CFITSIO could read the file, but did you get it to actually extract the data from that last HDU? And if so did you display it to confirm that it was being decompressed successfully?

If it's using tile compression it's conceivable that CFITSIO could still compress up to a certain number of tiles before hitting EOF, but you should still get a truncated image. If you are still getting the full image out of CFITSIO even with the truncated file, I say wizards did it.

I'm going to go ahead an accelerate this a bit because now, with some recent changes in PyFITS, when trying to decompress a truncated image like this I'm actually getting a segfault +_+ Obviously it would have something to do with trying to access memory past the end of the mmap, but I'm not sure why it's not bombing out on creating the mmap to begin with like the exception you were getting previously.

It's possible that the exception is just happening while in the C extension now but isn't being handled. I'll have to take a look...

Fixed the segfault issue (as I suspected it was just an unhandled exception). But I do want to see if I can get some nicer default behavior for truncated arrays. Like imcopy, funpack will also gladly decompress the truncated data (without even outputting a warning, troublingly) and fills the missing bits with zeros. But I think that I will put back to the backburner again after all. The fact that PyFITS doesn't do this isn't a bug, but it would be a nice to have feature.

Meant to reopen this as a placeholder for the issue of adding better support for truncated data

Fix unhandled exception when getting the HDU data fails (typically in…

I'm getting these warnings when building astropy master on my Mac (with a recent clang compiler):

cextern/cfitsio/imcompress.c:7253:58: warning: comparison of constant -2147483646 with expression of type 'unsigned char' is always false
      [-Wtautological-constant-out-of-range-compare]
                if (dither_method == SUBTRACTIVE_DITHER_2 && input[ii] == ZERO_VALUE)
                                                             ~~~~~~~~~ ^  ~~~~~~~~~~

https://gist.github.com/cdeil/9228460

Related or worth reporting to the CFITSIO maintainer?

Well I don't think that would be related, but that does look a bit odd.

Does that warning also occur on line 7331? It should...

@embray

@embray Yes, that warning also appears on line 7331:
https://gist.github.com/cdeil/9228460

I'll write an email to the CFITSIO maintainer then, because to me this compiler warning looks like it might cause real errors (which I have not observed).

Indeed, it looks like a bug to me. I haven't seen too many files in the wild using SUBTRACTIVE_DITHER_2, but I think I have seen some?! Further more it's not likely they would be using fewer than 32-bit integers for quantization. That said, as written, the code makes no sense.

Merge pull request astropy#541 from dhomeier/olobole-maintainer

Navigation Menu

Repository navigation

Issue with certain compressed FITS files with astropy.io.fits #541

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with certain compressed FITS files with astropy.io.fits #541

Description

Activity

embray commented on Dec 8, 2012

astrofrog commented on Dec 8, 2012

embray commented on Dec 8, 2012

astrofrog commented on Dec 8, 2012

embray commented on Dec 10, 2012

embray commented on Aug 9, 2013

embray commented on Aug 9, 2013

embray commented on Aug 12, 2013

cdeil commented on Feb 26, 2014

embray commented on Feb 26, 2014

cdeil commented on Feb 27, 2014

embray commented on Feb 27, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions