Repository navigation
Issue with certain compressed FITS files with astropy.io.fits #541
Description
I am using astropy.io.fits to read in many FITS files from the UKIDSS GPS survey, which uses RICE compression to reduce their file size. They warn on their website that a recent version of cftisio is required:
When trying to view or uncompress our FITS files, in some rare cases you may come across one or both of the following error messages:
FITSIO status = 1: non-CFITSIO program error
decompression error: hit end of compressed byte stream
This is because our FITS file images have been compressed using a newer version of fpack & CFITSIO. The only solution is to first uncompress the FITS file using a version of CFITSIO >= 3.1 (fpack >= 1.0) prior to viewing.
Here is an example file that is causing an issue (these represent 5% of all files I've been using):
http://mpia.de/~robitaille/astropy/w20070425_02461_sf_st.fit
The error occurs if I try and access the 4th HDU:
In [8]: fits.getdata('w20070425_02461_sf_st.fit', 4)
ERROR: ValueError: mmap length is greater than file size [numpy.core.memmap]
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/Volumes/Raptor/tmp/ukidss_bug/<ipython-input-8-6ac880274a55> in <module>()
----> 1 fits.getdata('w20070425_02461_sf_st.fit', 4)
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/convenience.pyc in getdata(filename, *args, **kwargs)
186 hdulist, extidx = _getext(filename, mode, *args, **kwargs)
187 hdu = hdulist[extidx]
--> 188 data = hdu.data
189 if data is None and extidx == 0:
190 try:
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/utils/misc.pyc in __get__(self, obj, owner)
256 key = self._fget.func_name
257 if key not in obj.__dict__:
--> 258 val = self._fget(obj)
259 obj.__dict__[key] = val
260 return val
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/compressed.pyc in data(self)
901 def data(self):
902 # The data attribute is the image data (not the table data).
--> 903 data = compression.decompress_hdu(self)
904
905 # Scale the data if necessary
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/utils/misc.pyc in __get__(self, obj, owner)
256 key = self._fget.func_name
257 if key not in obj.__dict__:
--> 258 val = self._fget(obj)
259 obj.__dict__[key] = val
260 return val
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/compressed.pyc in compData(self)
946 # First we will get the table data (the compressed
947 # data) from the file, if there is any.
--> 948 compData = super(BinTableHDU, self).data
949 if isinstance(compData, np.rec.recarray):
950 del self.data
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/utils/misc.pyc in __get__(self, obj, owner)
256 key = self._fget.func_name
257 if key not in obj.__dict__:
--> 258 val = self._fget(obj)
259 obj.__dict__[key] = val
260 return val
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/table.pyc in data(self)
229 @lazyproperty
230 def data(self):
--> 231 data = self._get_tbdata()
232 data._coldefs = self.columns
233 data.formats = self.columns.formats
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/table.pyc in _get_tbdata(self)
81 self._datSpan > self._theap):
82 # We have a heap; include it in the raw_data
---> 83 raw_data = self._get_raw_data(self._datSpan, np.byte, self._datLoc)
84 data = raw_data[:self._theap].view(dtype=dtype,
85 type=np.rec.recarray)
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/hdu/base.pyc in _get_raw_data(self, shape, code, offset)
366 offset=offset)
367 else:
--> 368 return self._file.readarray(offset=offset, dtype=code, shape=shape)
369
370 # TODO: Rework checksum handling so that it's not necessary to add a
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/astropy/io/fits/file.pyc in readarray(self, size, offset, dtype, shape)
269 return Memmap(self.__file, offset=offset,
270 mode=MEMMAP_MODES[self.mode], dtype=dtype,
--> 271 shape=shape).view(np.ndarray)
272 else:
273 count = reduce(lambda x, y: x * y, shape)
/Volumes/Raptor/Library/Python/2.7/lib/python/site-packages/numpy/core/memmap.pyc in __new__(subtype, filename, dtype, mode, offset, shape, order)
251 bytes -= start
252 offset -= start
--> 253 mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
254 else:
255 mm = mmap.mmap(fid.fileno(), bytes, access=acc)
ValueError: mmap length is greater than file size
However, if I download and compile the latest cfitsio, then do make imcopy and use imcopy to decompress the file, it works fine. Therefore, I think there must be some bug/issue in astropy.io.fits - I'm not sure if it's to do with pyfits or the bundling of cfitsio in astropy, so I just thought I'd report it here.
Activity
embray commentedon Dec 8, 2012
When I open the file I get the warning:
Do you get this too? The other HDUs decompress fine, and the error you're getting strongly suggests that the header is reporting a larger size for the file than it actually contains data-wise. So it's strange that imcopy doesn't complain about it.
astrofrog commentedon Dec 8, 2012
I think the file may be corrupt - I will double check tomorrow, and will close the issue if so - sorry!
embray commentedon Dec 8, 2012
It's still strange that CFITSIO had no problem with it. It's possible that it's just being more permissive with truncated images. I think, if that's the case, PyFITS should try to be too, and could be fairly easily. If nothing else a more useful error message should be raised.
And yeah, after going over the header and manipulating the file a bit I'm also convinced that it's truncated. There are several thousand bytes missing from the heap that should be there.
astrofrog commentedon Dec 8, 2012
@iguananaut - weird, I went back to re-download this specific file, and it has a different checksum and works, so it's possible that the original file got truncated during download. There is still the question of how cfitsio can manage to read in this file anyway. But I wouldn't worry about it too much. Certainly not 0.2-worthy, so I'm tagging it as 'Future'.
embray commentedon Dec 10, 2012
I guess so. I would like to see what went on here. You say that CFITSIO could read the file, but did you get it to actually extract the data from that last HDU? And if so did you display it to confirm that it was being decompressed successfully?
If it's using tile compression it's conceivable that CFITSIO could still compress up to a certain number of tiles before hitting EOF, but you should still get a truncated image. If you are still getting the full image out of CFITSIO even with the truncated file, I say wizards did it.
embray commentedon Aug 9, 2013
I'm going to go ahead an accelerate this a bit because now, with some recent changes in PyFITS, when trying to decompress a truncated image like this I'm actually getting a segfault +_+ Obviously it would have something to do with trying to access memory past the end of the mmap, but I'm not sure why it's not bombing out on creating the mmap to begin with like the exception you were getting previously.
It's possible that the exception is just happening while in the C extension now but isn't being handled. I'll have to take a look...
embray commentedon Aug 9, 2013
Fixed the segfault issue (as I suspected it was just an unhandled exception). But I do want to see if I can get some nicer default behavior for truncated arrays. Like imcopy, funpack will also gladly decompress the truncated data (without even outputting a warning, troublingly) and fills the missing bits with zeros. But I think that I will put back to the backburner again after all. The fact that PyFITS doesn't do this isn't a bug, but it would be a nice to have feature.
embray commentedon Aug 12, 2013
Meant to reopen this as a placeholder for the issue of adding better support for truncated data
Fix unhandled exception when getting the HDU data fails (typically in…
Fix unhandled exception when getting the HDU data fails (typically in…
cdeil commentedon Feb 26, 2014
I'm getting these warnings when building astropy master on my Mac (with a recent clang compiler):
https://gist.github.com/cdeil/9228460
Related or worth reporting to the CFITSIO maintainer?
embray commentedon Feb 26, 2014
Well I don't think that would be related, but that does look a bit odd.
Does that warning also occur on line 7331? It should...
cdeil commentedon Feb 27, 2014
@embray Yes, that warning also appears on line 7331:
https://gist.github.com/cdeil/9228460
I'll write an email to the CFITSIO maintainer then, because to me this compiler warning looks like it might cause real errors (which I have not observed).
embray commentedon Feb 27, 2014
Indeed, it looks like a bug to me. I haven't seen too many files in the wild using
SUBTRACTIVE_DITHER_2, but I think I have seen some?! Further more it's not likely they would be using fewer than 32-bit integers for quantization. That said, as written, the code makes no sense.Merge pull request astropy#541 from dhomeier/olobole-maintainer