Segfault with CompImageHDU

The following code (sometimes) segfaults on my machine.

import numpy as np
from astropy.io import fits
from io import BytesIO
import faulthandler

faulthandler.enable()

arr = np.array([1,5123124,2,1,2,1]).astype(int)
hdu = fits.CompImageHDU(arr, compression_type="PLIO_1")
buffer = BytesIO()
hdu.writeto(buffer)
print(buffer.getvalue())
buffer.seek(0)
with fits.open(buffer, memmap=False) as hdulist:
    print(hdulist)

The faulthandler reports varying stack-traces, for example:

Current thread 0x0000344c (most recent call first):
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1663 in _update_compressed_data
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1793 in _prewriteto
  File "...\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 935 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 1582 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper

or

Current thread 0x000000f4 (most recent call first):
  File "...\lib\site-packages\astropy\io\fits\util.py", line 674 in _array_to_file_like
  File "...\lib\site-packages\astropy\io\fits\util.py", line 651 in _array_to_file
  File "...\lib\site-packages\astropy\io\fits\file.py", line 380 in writearray
  File "...\lib\site-packages\astropy\io\fits\hdu\table.py", line 931 in _writedata_internal
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 610 in _writedata
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1839 in _writedata
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 680 in _writeto_internal
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 674 in _writeto
  File "...\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 936 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 1582 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper

or

Current thread 0x00002a60 (most recent call first):
  File "...\lib\site-packages\astropy\io\fits\column.py", line 1953 in __new__
  File "...\lib\site-packages\astropy\io\fits\fitsrec.py", line 782 in _convert_p
  File "...\lib\site-packages\astropy\io\fits\fitsrec.py", line 702 in field
  File "...\lib\site-packages\astropy\io\fits\hdu\table.py", line 522 in _prewriteto
  File "...\lib\site-packages\astropy\io\fits\hdu\compressed.py", line 1821 in _prewriteto
  File "...\lib\site-packages\astropy\io\fits\hdu\hdulist.py", line 935 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper
  File "...\lib\site-packages\astropy\io\fits\hdu\base.py", line 1582 in writeto
  File "...\lib\site-packages\astropy\utils\decorators.py", line 521 in wrapper

It could be windows-specific as in PR #9641 but could also be something different.

It could be an issue with an upstream dependency like NumPy, cfitsio or even CPython. I wasn't able to get to the root-cause of the issue.

On further inspection this probably isn't related to #9641 because it doesn't fail for non-PLIO compressions. 🤷‍♂

@pllim

@pllim Given the label I assume you tested it on a non-windows machine? Did it always work correctly there - even when executed multiple times in a row?

Oh, I just assumed since this was first mentioned in #9641 that dealt with Windows specific segfault. Sorry!

Well, it could be windows-specific but I haven't had time to run it in docker 😄

In my linux docker environment with python&numpy debug:

free(): invalid next size (fast)
Fatal Python error: Aborted

Current thread 0x00007fa2ce726740 (most recent call first):
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/compressed.py", line 1663 in _update_compressed_data
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/compressed.py", line 1793 in _prewriteto
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py", line 935 in writeto
  File "/usr/local/lib/python3.8/site-packages/astropy/utils/decorators.py", line 521 in wrapper
  File "/usr/local/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py", line 1581 in writeto
  File "/usr/local/lib/python3.8/site-packages/astropy/utils/decorators.py", line 521 in wrapper
  File "/io/test1.py", line 11 in <module>
Aborted

So probably not windows-specific.

Or inside CFITSIO 🤷‍♂

@MSeifert04

Thanks for checking, @MSeifert04 !

I think the error happens during the call to fits_write_img(fileptr, datatype, 1, PyArray_SIZE(indata), PyArray_DATA(indata), &status);. At least that's what my print-debugging indicates 😓

So it is cfitsio. 😞

Or we're passing invalid parameters... I really hate C debugging in docker 😟

I get this with your code, which probably means that some type detection is not done correctly on all platforms ?

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-223acca8c7ff> in <module>
      9 hdu = fits.CompImageHDU(arr, compression_type="PLIO_1")
     10 buffer = BytesIO()
---> 11 hdu.writeto(buffer)
     12 print(buffer.getvalue())
     13 buffer.seek(0)

~/dev/astropy/astropy/utils/decorators.py in wrapper(*args, **kwargs)
    519                             kwargs[new_name[i]] = value
    520 
--> 521             return function(*args, **kwargs)
    522 
    523         return wrapper

~/dev/astropy/astropy/io/fits/hdu/base.py in writeto(self, name, output_verify, overwrite, checksum)
   1587         hdulist = HDUList([PrimaryHDU(), self])
   1588         hdulist.writeto(name, output_verify, overwrite=overwrite,
-> 1589                         checksum=checksum)
   1590 
   1591     def _verify(self, option='warn'):

~/dev/astropy/astropy/utils/decorators.py in wrapper(*args, **kwargs)
    519                             kwargs[new_name[i]] = value
    520 
--> 521             return function(*args, **kwargs)
    522 
    523         return wrapper

~/dev/astropy/astropy/io/fits/hdu/hdulist.py in writeto(self, fileobj, output_verify, overwrite, checksum)
    933         with _free_space_check(self, dirname=dirname):
    934             for hdu in self:
--> 935                 hdu._prewriteto(checksum=checksum)
    936                 hdu._writeto(hdulist._file)
    937                 hdu._postwriteto()

~/dev/astropy/astropy/io/fits/hdu/compressed.py in _prewriteto(self, checksum, inplace)
   1791 
   1792         if self._has_data:
-> 1793             self._update_compressed_data()
   1794 
   1795             # Use methods in the superclass to update the header with

~/dev/astropy/astropy/io/fits/hdu/compressed.py in _update_compressed_data(self)
   1661             # compress_hdu returns the size of the heap for the written
   1662             # compressed image table
-> 1663             heapsize, self.compressed_data = compression.compress_hdu(self)
   1664         finally:
   1665             # if data was byteswapped return it to its original order

RuntimeError: writing TLONGLONG to compressed image is not supported

Or you are using a 32bits system ?

So I can reproduce with int32 data:

(gdb) bt
#0  0x00007ffff7c5df25 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff7c47897 in abort () from /usr/lib/libc.so.6
#2  0x00007ffff7ca1258 in __libc_message () from /usr/lib/libc.so.6
#3  0x00007ffff7ca877a in malloc_printerr () from /usr/lib/libc.so.6
#4  0x00007ffff7caa1cc in _int_free () from /usr/lib/libc.so.6
#5  0x00007fffea41eb22 in imcomp_compress_tile (outfptr=0x555555e04ae0, row=1, 
    datatype=31, tiledata=0x555555fe2440, tilelen=6, tilenx=6, tileny=1, nullcheck=0, 
    nullflagval=0x7fffffff48bc, status=0x7fffffff494c)
    at cextern/cfitsio/lib/imcompress.c:2136
#6  0x00007fffea422733 in fits_write_compressed_img (fptr=0x555555e04ae0, datatype=31, 
    infpixel=0x7fffffff47f0, inlpixel=0x7fffffff4820, nullcheck=0, array=0x555555f756d0, 
    nullval=0x7fffffff48bc, status=0x7fffffff494c)
    at cextern/cfitsio/lib/imcompress.c:3725
#7  0x00007fffea422bc8 in fits_write_compressed_pixels (fptr=0x555555e04ae0, 
    datatype=31, fpixel=1, npixel=6, nullcheck=0, array=0x555555f756d0, 
    nullval=0x7fffffff48bc, status=0x7fffffff494c)
    at cextern/cfitsio/lib/imcompress.c:3839
#8  0x00007fffea3c39b1 in ffpprk (fptr=0x555555e04ae0, group=1, firstelem=1, nelem=6, 
    array=0x555555f756d0, status=0x7fffffff494c) at cextern/cfitsio/lib/putcolk.c:40
#9  0x00007fffea43a495 in ffppr (fptr=0x555555e04ae0, datatype=31, firstelem=1, nelem=6, 
    array=0x555555f756d0, status=0x7fffffff494c) at cextern/cfitsio/lib/putcol.c:452
#10 0x00007fffea2fe922 in compression_compress_hdu (
    self=<module at remote 0x7fffea659410>, 
    args=(<CompImageHDU(_header=<Header(_cards=[<Card(_keyword='XTENSION', _value='BINTABLE', _comment='binary table extension', _valuestring=None, _image="XTENSION= 'BINTABLE'           / binary table extension                         ", _verified=True, _hierarch=False, _invalid=False, _field_specifier=None, _rawkeyword=None, _rawvalue=None, _modified=False, _valuemodified=False) at remote 0x7fffea2bfc30>, <Card(_keyword='BITPIX', _value=8, _comment='array data type', _valuestring=None, _image='BITPIX  =                    8 / array data type                                ', _verified=True, _hierarch=False, _invalid=False, _field_specifier=None, _rawkeyword=None, _rawvalue=None, _modified=False, _valuemodified=False) at remote 0x7fffe97a4f50>, <Card(_keyword='NAXIS', _value=2, _comment='number of array dimensions', _valuestring=None, _image='NAXIS   =                    2 / number of array dimensions                     ', _verified=True, _hierarch=False, _invalid=False, _field_specifier=None, _rawkeyword=None, _raw...(truncated))
    at astropy/io/fits/src/compressionmodule.c:1052

The crash happens here (in cfitsio):

#5  0x00007fffea41eb22 in imcomp_compress_tile (outfptr=0x555555e04ae0, row=1, datatype=31, tiledata=0x555555fe2440, tilelen=6, tilenx=6, tileny=1, nullcheck=0, nullflagval=0x7fffffff48bc, 
    status=0x7fffffff494c) at cextern/cfitsio/lib/imcompress.c:2136
2136	        free(cbuf);  /* finished with this buffer */
(gdb) l
2131	                      bscale, status);
2132		      ffpcld (outfptr, (outfptr->Fptr)->cn_zzero,  row, 1, 1,
2133	                      bzero,  status);
2134	        }
2135	
2136	        free(cbuf);  /* finished with this buffer */
2137	
2138	    /* =========================================================================== */
2139	    } else {    /* if flag == 0., floating point data couldn't be quantized */
2140

So is this bug still in cfitsio or in astropy.io.fits?

So is this bug still in cfitsio or in astropy.io.fits?

for what it's worth at this point (when we are going to fix it in a soon-to-be opened pr), but this is a bug in astropy.io.fits.

Navigation Menu

Repository navigation

Segfault with CompImageHDU #9715

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Segfault with CompImageHDU #9715

Description

Activity

MSeifert04 commented on Dec 3, 2019

MSeifert04 commented on Dec 3, 2019

pllim commented on Dec 3, 2019

MSeifert04 commented on Dec 3, 2019

MSeifert04 commented on Dec 3, 2019

pllim commented on Dec 3, 2019

MSeifert04 commented on Dec 3, 2019

pllim commented on Dec 3, 2019

MSeifert04 commented on Dec 4, 2019

pllim commented on Dec 4, 2019

MSeifert04 commented on Dec 4, 2019

saimn commented on Dec 13, 2019

saimn commented on Dec 13, 2019

saimn commented on Dec 13, 2019

pllim commented on Dec 13, 2019

Cadair commented on Dec 15, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions