Repository navigation
Segfault with CompImageHDU #9715
Description
Activity
On further inspection this probably isn't related to #9641 because it doesn't fail for non-PLIO compressions. 🤷♂
Reacted by P. L. Lim@pllim Given the label I assume you tested it on a non-windows machine? Did it always work correctly there - even when executed multiple times in a row?
Well, it could be windows-specific but I haven't had time to run it in docker 😄
Reacted by P. L. LimIn my linux docker environment with python&numpy debug:
free(): invalid next size (fast) Fatal Python error: Aborted Current thread 0x00007fa2ce726740 (most recent call first): File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/compressed.py", line 1663 in _update_compressed_data File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/compressed.py", line 1793 in _prewriteto File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 935 in writeto File "/usr/local/lib/python3.8/site-packages/astropy/utils/decorators.py", line 521 in wrapper File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 1581 in writeto File "/usr/local/lib/python3.8/site-packages/astropy/utils/decorators.py", line 521 in wrapper File "/io/test1.py", line 11 in <module> AbortedSo probably not windows-specific.
Reacted by P. L. LimOr inside CFITSIO 🤷♂
Reacted by P. L. LimThanks for checking, @MSeifert04 !
I think the error happens during the call to
fits_write_img(fileptr, datatype, 1, PyArray_SIZE(indata), PyArray_DATA(indata), &status);. At least that's what my print-debugging indicates 😓So it is
cfitsio. 😞Or we're passing invalid parameters... I really hate C debugging in docker 😟
Reacted by P. L. LimI get this with your code, which probably means that some type detection is not done correctly on all platforms ?
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-1-223acca8c7ff> in <module> 9 hdu = fits.CompImageHDU(arr, compression_type="PLIO_1") 10 buffer = BytesIO() ---> 11 hdu.writeto(buffer) 12 print(buffer.getvalue()) 13 buffer.seek(0) ~/dev/astropy/astropy/utils/decorators.py in wrapper(*args, **kwargs) 519 kwargs[new_name[i]] = value 520 --> 521 return function(*args, **kwargs) 522 523 return wrapper ~/dev/astropy/astropy/io/fits/hdu/base.py in writeto(self, name, output_verify, overwrite, checksum) 1587 hdulist = HDUList([PrimaryHDU(), self]) 1588 hdulist.writeto(name, output_verify, overwrite=overwrite, -> 1589 checksum=checksum) 1590 1591 def _verify(self, option='warn'): ~/dev/astropy/astropy/utils/decorators.py in wrapper(*args, **kwargs) 519 kwargs[new_name[i]] = value 520 --> 521 return function(*args, **kwargs) 522 523 return wrapper ~/dev/astropy/astropy/io/fits/hdu/hdulist.py in writeto(self, fileobj, output_verify, overwrite, checksum) 933 with _free_space_check(self, dirname=dirname): 934 for hdu in self: --> 935 hdu._prewriteto(checksum=checksum) 936 hdu._writeto(hdulist._file) 937 hdu._postwriteto() ~/dev/astropy/astropy/io/fits/hdu/compressed.py in _prewriteto(self, checksum, inplace) 1791 1792 if self._has_data: -> 1793 self._update_compressed_data() 1794 1795 # Use methods in the superclass to update the header with ~/dev/astropy/astropy/io/fits/hdu/compressed.py in _update_compressed_data(self) 1661 # compress_hdu returns the size of the heap for the written 1662 # compressed image table -> 1663 heapsize, self.compressed_data = compression.compress_hdu(self) 1664 finally: 1665 # if data was byteswapped return it to its original order RuntimeError: writing TLONGLONG to compressed image is not supportedOr you are using a 32bits system ?
So I can reproduce with int32 data:
(gdb) bt #0 0x00007ffff7c5df25 in raise () from /usr/lib/libc.so.6 #1 0x00007ffff7c47897 in abort () from /usr/lib/libc.so.6 #2 0x00007ffff7ca1258 in __libc_message () from /usr/lib/libc.so.6 #3 0x00007ffff7ca877a in malloc_printerr () from /usr/lib/libc.so.6 #4 0x00007ffff7caa1cc in _int_free () from /usr/lib/libc.so.6 #5 0x00007fffea41eb22 in imcomp_compress_tile (outfptr=0x555555e04ae0, row=1, datatype=31, tiledata=0x555555fe2440, tilelen=6, tilenx=6, tileny=1, nullcheck=0, nullflagval=0x7fffffff48bc, status=0x7fffffff494c) at cextern/cfitsio/lib/imcompress.c:2136 #6 0x00007fffea422733 in fits_write_compressed_img (fptr=0x555555e04ae0, datatype=31, infpixel=0x7fffffff47f0, inlpixel=0x7fffffff4820, nullcheck=0, array=0x555555f756d0, nullval=0x7fffffff48bc, status=0x7fffffff494c) at cextern/cfitsio/lib/imcompress.c:3725 #7 0x00007fffea422bc8 in fits_write_compressed_pixels (fptr=0x555555e04ae0, datatype=31, fpixel=1, npixel=6, nullcheck=0, array=0x555555f756d0, nullval=0x7fffffff48bc, status=0x7fffffff494c) at cextern/cfitsio/lib/imcompress.c:3839 #8 0x00007fffea3c39b1 in ffpprk (fptr=0x555555e04ae0, group=1, firstelem=1, nelem=6, array=0x555555f756d0, status=0x7fffffff494c) at cextern/cfitsio/lib/putcolk.c:40 #9 0x00007fffea43a495 in ffppr (fptr=0x555555e04ae0, datatype=31, firstelem=1, nelem=6, array=0x555555f756d0, status=0x7fffffff494c) at cextern/cfitsio/lib/putcol.c:452 #10 0x00007fffea2fe922 in compression_compress_hdu ( self=<module at remote 0x7fffea659410>, args=(<CompImageHDU(_header=<Header(_cards=[<Card(_keyword='XTENSION', _value='BINTABLE', _comment='binary table extension', _valuestring=None, _image="XTENSION= 'BINTABLE' / binary table extension ", _verified=True, _hierarch=False, _invalid=False, _field_specifier=None, _rawkeyword=None, _rawvalue=None, _modified=False, _valuemodified=False) at remote 0x7fffea2bfc30>, <Card(_keyword='BITPIX', _value=8, _comment='array data type', _valuestring=None, _image='BITPIX = 8 / array data type ', _verified=True, _hierarch=False, _invalid=False, _field_specifier=None, _rawkeyword=None, _rawvalue=None, _modified=False, _valuemodified=False) at remote 0x7fffe97a4f50>, <Card(_keyword='NAXIS', _value=2, _comment='number of array dimensions', _valuestring=None, _image='NAXIS = 2 / number of array dimensions ', _verified=True, _hierarch=False, _invalid=False, _field_specifier=None, _rawkeyword=None, _raw...(truncated)) at astropy/io/fits/src/compressionmodule.c:1052The crash happens here (in cfitsio):
#5 0x00007fffea41eb22 in imcomp_compress_tile (outfptr=0x555555e04ae0, row=1, datatype=31, tiledata=0x555555fe2440, tilelen=6, tilenx=6, tileny=1, nullcheck=0, nullflagval=0x7fffffff48bc, status=0x7fffffff494c) at cextern/cfitsio/lib/imcompress.c:2136 2136 free(cbuf); /* finished with this buffer */ (gdb) l 2131 bscale, status); 2132 ffpcld (outfptr, (outfptr->Fptr)->cn_zzero, row, 1, 1, 2133 bzero, status); 2134 } 2135 2136 free(cbuf); /* finished with this buffer */ 2137 2138 /* =========================================================================== */ 2139 } else { /* if flag == 0., floating point data couldn't be quantized */ 2140So is this bug still in
cfitsioor inastropy.io.fits?So is this bug still in
cfitsioor inastropy.io.fits?for what it's worth at this point (when we are going to fix it in a soon-to-be opened pr), but this is a bug in
astropy.io.fits.Reacted by P. L. Lim
The following code (sometimes) segfaults on my machine.
The faulthandler reports varying stack-traces, for example:
or
or
It could be windows-specific as in PR #9641 but could also be something different.
It could be an issue with an upstream dependency like NumPy, cfitsio or even CPython. I wasn't able to get to the root-cause of the issue.