Skip to content

test_comp_image_hcompress_image_stack with cfitsio 3.380 #4647

Closed
#4659
Closed
test_comp_image_hcompress_image_stack with cfitsio 3.380#4647
#4659
Labels
Bugio.fits

Description

@olebole
olebole
opened on Feb 29, 2016

This is the second failure when compiling with cfitsio 3.380 (and numpy 1.11b3):

____ TestCompressedImage.test_comp_image_hcompress_image_stack ___________

args = (<astropy.io.fits.tests.test_image.TestCompressedImage object at 0x7fcf3a85e810>,)
kwargs = {}

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        # Originally this just reused self, but that doesn't work if the
        # function is called more than once so we need to make a new
        # context manager instance for each call
        with self.__class__(category=self.category):
>           return func(*args, **kwargs)

astropy/tests/helper.py:361: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <astropy.io.fits.tests.test_image.TestCompressedImage object at 0x7fcf3a85e810>

    @ignore_warnings(AstropyPendingDeprecationWarning)
    def test_comp_image_hcompress_image_stack(self):
        """[...]"""

        cube = np.arange(300, dtype=np.float32).reshape((3, 10, 10))
        hdu = fits.CompImageHDU(data=cube, name='SCI',
                                compressionType='HCOMPRESS_1',
                                quantizeLevel=16, tileSize=[5, 5, 1])
        hdu.writeto(self.temp('test.fits'))

        with fits.open(self.temp('test.fits')) as hdul:
>           assert (hdul['SCI'].data == cube).all()
E           assert <built-in method all of numpy.ndarray object at 0x7fcf3a662210>()
E            +  where <built-in method all of numpy.ndarray object at 0x7fcf3a662210>
              = array([[[   0...dtype=float32) == array([[[   0....dtype=float32)
E               Detailed information truncated, use "-vv" to show.all

This may be the same problem as #4646; however it looks different to me (and numpy is involved here as well).
Full log here.
This also happens on all architectures.

Activity

  1. astrofrog commented on Feb 29, 2016

    @astrofrog
    astrofrog
    on Feb 29, 2016
    Member

    @olebole do you see the same issue with cfitsio 3.380 and numpy 1.10? And what about with the previous cfitsio and numpy 1.11b3?

  2. olebole commented on Feb 29, 2016

    @olebole
    olebole
    on Feb 29, 2016
    MemberAuthor

    It did work with numpy 1.11b3 and the old (3.370) cfitsio. cfitsio 3.380 and numpy 1.10 is not tested yet. I could try this manually if needed.

  3. astrofrog commented on Feb 29, 2016

    @astrofrog
    astrofrog
    on Feb 29, 2016
    Member

    @olebole - that would be great, to narrow it down - thanks!

  4. olebole commented on Feb 29, 2016

    @olebole
    olebole
    on Feb 29, 2016
    MemberAuthor

    cfitsio 3.380 and numpy 1.10 has the same issue, so it is clearly a problem caused by the cfitsio update.

  5. pllim
    added
    io.fits
    Bug
    on Feb 29, 2016

  6. olebole commented on Mar 1, 2016

    @olebole
    olebole
    on Mar 1, 2016
    MemberAuthor

    Re-running it with more verbosity gives

    E           assert <built-in method all of numpy.ndarray object at 0x7fd7d3a18080>()
E            +  where <built-in method all of numpy.ndarray object at 0x7fd7d3a18080> = array([[[   0...dtype=float32) == array([[[   0....dtype=float32)
E               Full diff:
E               + array([[[   0.,    1.,    2.,    3.,    4.,    5.,    6.,    7.,    8.,
E               +             9.],
E               +         [  10.,   11.,   12.,   13.,   14.,   15.,   16.,   17.,   18.,
E               +            19.],
E               +         [  20.,   21.,   22.,   23.,   24.,   25.,   26.,   27.,   28.,
E               +            29.],
E               +         [  30.,   31.,   32.,   33.,   34.,   35.,   36.,   37.,   38.,
E               +            39.],
E               +         [  40.,   41.,   42.,   43.,   44.,   45.,   46.,   47.,   48.,
E               +            49.],
E               +         [  50.,   51.,   52.,   53.,   54.,   55.,   56.,   57.,   58.,
E               +            59.],
E               +         [  60.,   61.,   62.,   63.,   64.,   65.,   66.,   67.,   68.,
E               +            69.],
E               +         [  70.,   71.,   72.,   73.,   74.,   75.,   76.,   77.,   78.,
E               +            79.],
E               +         [  80.,   81.,   82.,   83.,   84.,   85.,   86.,   87.,   88.,
E               +            89.],
E               +         [  90.,   91.,   92.,   93.,   94.,   95.,   96.,   97.,   98.,
E               +            99.]],
E               - array([[[   0.        ,    1.04835784,    1.96567106,    3.01402903,
E               -             4.06238699,    4.97970009,    6.02805805,    6.94537115,
E               -             7.99372911,    9.0420866 ],
E               -         [   9.95940018,   11.00775814,   12.0561161 ,   12.97342873,
E               -            14.02178669,   14.93910027,   15.98745823,   17.03581619,
E               -            17.95312881,   19.00148773],
E               -         [  20.04984474,   20.96715736,   22.01551628,   23.06387329,
E               -            23.98118782,   25.02954483,   25.94685745,   26.99521637,
E               -            28.04357338,   28.96088791],
E               -         [  30.00924492,   31.05760193,   31.97491646,   33.02327347,
E               -            33.94058609,   34.98894501,   36.03730392,   36.95461655,
E               -            38.00297546,   39.05133057],
E               -         [  39.96864319,   41.01700211,   42.06536102,   42.98267365,
E               -            44.03103256,   44.94834518,   45.9967041 ,   47.0450592 ,
E               -            47.96237564,   49.01073074],
E               -         [  50.05908966,   50.97640228,   52.0247612 ,   52.94207382,
E               -            53.99043274,   55.03879166,   55.95610428,   57.00445938,
E               -            58.0528183 ,   58.97013092],
E               -         [  60.01848984,   60.93580246,   61.98416138,   63.03252029,
E               -            63.94983292,   64.99819183,   66.04654694,   66.96385956,
E               -            68.01222229,   69.06057739],
E               -         [  69.97789001,   71.02624512,   71.94355774,   72.99192047,
E               -            74.04027557,   74.9575882 ,   76.00595093,   77.05430603,
E               -            77.97161865,   79.01997375],
E               -         [  79.93728638,   80.98564911,   82.03400421,   82.95131683,
E               -            83.99967957,   85.04803467,   85.96534729,   87.01370239,
E               -            88.06206512,   88.97937775],
E               -         [  90.02773285,   90.94504547,   91.9934082 ,   93.04176331,
E               -            93.95907593,   95.00743103,   96.05579376,   96.97310638,
E               -            98.02146149,   98.93877411]],
E               
E               +        [[ 100.,  101.,  102.,  103.,  104.,  105.,  106.,  107.,  108.,
E               +           109.],
E               +         [ 110.,  111.,  112.,  113.,  114.,  115.,  116.,  117.,  118.,
E               +           119.],
E               +         [ 120.,  121.,  122.,  123.,  124.,  125.,  126.,  127.,  128.,
E               +           129.],
E               +         [ 130.,  131.,  132.,  133.,  134.,  135.,  136.,  137.,  138.,
E               +           139.],
E               +         [ 140.,  141.,  142.,  143.,  144.,  145.,  146.,  147.,  148.,
E               +           149.],
E               +         [ 150.,  151.,  152.,  153.,  154.,  155.,  156.,  157.,  158.,
E               +           159.],
E               +         [ 160.,  161.,  162.,  163.,  164.,  165.,  166.,  167.,  168.,
E               +           169.],
E               +         [ 170.,  171.,  172.,  173.,  174.,  175.,  176.,  177.,  178.,
E               +           179.],
E               +         [ 180.,  181.,  182.,  183.,  184.,  185.,  186.,  187.,  188.,
E               +           189.],
E               +         [ 190.,  191.,  192.,  193.,  194.,  195.,  196.,  197.,  198.,
E               +           199.]],
E               -        [[  99.98713684,  101.03549194,  101.95280457,  103.0011673 ,
E               -           104.0495224 ,  104.96683502,  106.01519012,  107.06355286,
E               -           107.98086548,  109.02922058],
E               -         [ 109.9465332 ,  110.99489594,  112.04325104,  112.96056366,
E               -           114.00891876,  115.05728149,  115.97459412,  117.02294922,
E               -           117.94026184,  118.98862457],
E               -         [ 120.03697968,  120.9542923 ,  122.0026474 ,  123.05101013,
E               -           123.96832275,  125.01667786,  126.06504059,  126.98235321,
E               -           128.03071594,  128.94802856],
E               -         [ 129.99638367,  131.04473877,  131.96205139,  133.01040649,
E               -           134.0587616 ,  134.97607422,  136.02444458,  136.9417572 ,
E               -           137.9901123 ,  139.03846741],
E               -         [ 139.95578003,  141.00413513,  142.05249023,  142.96980286,
E               -           144.01817322,  144.93548584,  145.98384094,  147.03219604,
E               -           147.94950867,  148.99786377],
E               -         [ 150.04621887,  150.96353149,  152.01190186,  153.06025696,
E               -           153.97756958,  155.02592468,  155.9432373 ,  156.99159241,
E               -           158.03994751,  158.95726013],
E               -         [ 160.00563049,  161.0539856 ,  161.97129822,  163.01965332,
E               -           163.93696594,  164.98532104,  166.03367615,  166.95098877,
E               -           167.99935913,  169.04771423],
E               -         [ 169.96502686,  171.01338196,  172.06173706,  172.97904968,
E               -           174.02740479,  174.94471741,  175.99308777,  177.04144287,
E               -           177.95875549,  179.0071106 ],
E               -         [ 180.0554657 ,  180.97277832,  182.02113342,  182.93844604,
E               -           183.98681641,  185.03517151,  185.95248413,  187.00083923,
E               -           188.04919434,  188.96650696],
E               -         [ 190.01486206,  191.06323242,  191.98054504,  193.02890015,
E               -           193.94621277,  194.99456787,  196.04292297,  196.9602356 ,
E               -           198.00860596,  199.05696106]],
E               
E               +        [[ 200.,  201.,  202.,  203.,  204.,  205.,  206.,  207.,  208.,
E               +           209.],
E               +         [ 210.,  211.,  212.,  213.,  214.,  215.,  216.,  217.,  218.,
E               +           219.],
E               +         [ 220.,  221.,  222.,  223.,  224.,  225.,  226.,  227.,  228.,
E               +           229.],
E               +         [ 230.,  231.,  232.,  233.,  234.,  235.,  236.,  237.,  238.,
E               +           239.],
E               +         [ 240.,  241.,  242.,  243.,  244.,  245.,  246.,  247.,  248.,
E               +           249.],
E               +         [ 250.,  251.,  252.,  253.,  254.,  255.,  256.,  257.,  258.,
E               +           259.],
E               +         [ 260.,  261.,  262.,  263.,  264.,  265.,  266.,  267.,  268.,
E               +           269.],
E               +         [ 270.,  271.,  272.,  273.,  274.,  275.,  276.,  277.,  278.,
E               +           279.],
E               +         [ 280.,  281.,  282.,  283.,  284.,  285.,  286.,  287.,  288.,
E               +           289.],
E               +         [ 290.,  291.,  292.,  293.,  294.,  295.,  296.,  297.,  298.,
E               +           299.]]], dtype=float32)
E               -        [[ 199.97427368,  201.02262878,  201.93994141,  202.98829651,
E               -           204.03665161,  204.95396423,  206.00233459,  207.0506897 ,
E               -           207.96800232,  209.01635742],
E               -         [ 210.06471252,  210.98202515,  212.03038025,  212.94769287,
E               -           213.99606323,  215.04441833,  215.96173096,  217.01008606,
E               -           218.05844116,  218.97575378],
E               -         [ 220.02410889,  220.94142151,  221.98979187,  223.03814697,
E               -           223.95545959,  225.0038147 ,  226.0521698 ,  226.96948242,
E               -           228.01783752,  228.93515015],
E               -         [ 229.98352051,  231.03187561,  231.94918823,  232.99754333,
E               -           234.04589844,  234.96321106,  236.01156616,  237.05993652,
E               -           237.97724915,  239.02560425],
E               -         [ 239.94291687,  240.99127197,  242.03962708,  242.9569397 ,
E               -           244.0052948 ,  245.05366516,  245.97097778,  247.01933289,
E               -           247.93664551,  248.98500061],
E               -         [ 250.03335571,  250.95066833,  251.99902344,  253.0473938 ,
E               -           253.96470642,  255.01306152,  256.06143188,  256.97872925,
E               -           258.02709961,  258.94439697],
E               -         [ 259.99276733,  261.04110718,  261.95843506,  263.0067749 ,
E               -           264.05514526,  264.97247314,  266.02081299,  266.93814087,
E               -           267.98648071,  269.03485107],
E               -         [ 269.95214844,  271.0005188 ,  272.04888916,  272.96618652,
E               -           274.01455688,  275.06289673,  275.98022461,  277.02856445,
E               -           277.94589233,  278.99423218],
E               -         [ 280.04260254,  280.95993042,  282.00827026,  283.05664062,
E               -           283.97393799,  285.02230835,  285.93960571,  286.98797607,
E               -           288.03634644,  288.9536438 ],
E               -         [ 290.00201416,  291.050354  ,  291.96768188,  293.01602173,
E               -           294.06439209,  294.98168945,  296.03005981,  296.9473877 ,
E               -           297.99572754,  299.0440979 ]]], dtype=float32).all

  7. olebole commented on Mar 1, 2016

    @olebole
    olebole
    on Mar 1, 2016
    MemberAuthor

    After reading the astropy documentation for CompImageHDU I wonder why the test ever worked? Without checking the detailed computation, I would expect that the image is not exactly the same after writing to disk and reading back. Just the quantization seems to be different from exact integer. However, I don't know whether the difference is within the expectations: -0.065 ... 0.065 (~ 1./16.).

  8. olebole
    added a commit that references this issue on Mar 2, 2016

    Loose identity test of a HCOMPRESSed image

    4a0b7c6

  9. aurel32 commented on Mar 2, 2016

    @aurel32
    aurel32
    on Mar 2, 2016

    The change on the cfitsio side which caused this regression is the following one:

     modified the 4 FnNoise5_(type) routines in quantize.c to correctly
 count the number of non-null pixels in the input array.  Previously the
 count could be inaccurate if the image mainly consisted of null pixels.
 This could have caused certain floating point image tiles to be
 quantized during the image compression process, when in fact the tile
 did not satisfy all the criteria to be safely quantized.

    Basically the quantization was not done correctly in some cases (including the one in astropy), so in practice the data where left unchanged.

    Therefore the suggested change looks to me the right way to fix the issue.

  10. olebole
    added 3 commits that reference this issue on May 18, 2016

    Loose identity test of a HCOMPRESSed image

    62fdf22

    Loose identity test of a HCOMPRESSed image

    3518cae

    Loose identity test of a HCOMPRESSed image

    39b8390

  11. cdeil commented on Jan 6, 2017

    @cdeil
    cdeil
    on Jan 6, 2017
    Member

    Macports was updated to Astropy 1.3 just now: https://trac.macports.org/ticket/53162

    This fail for TestCompressedImage.test_comp_image_hcompress_image_stack shows up there as well (for Python 2.7, 3.4, 3.5, 3.6), and is the only fail in the Astropy test suite.

    In Macports, Astropy uses the separate port for CFITSIO, which at the moment is at version cfitsio @3.390_1, numpy is at @1.11.3_0+gfortran.

    cc @MSeifert04 @pllim as astropy.io.fits maintainers - What to do? Adapt the test?

  12. pllim commented on Jan 6, 2017

    @pllim
    pllim
    on Jan 6, 2017
    Member

    If the test result is supposed to be different on Macports, we can either mark it as xfail for that setting or, like you said, adapt it.

  13. olebole
    added a commit that references this issue on Feb 8, 2017

    Loose identity test of a HCOMPRESSed image

    f476436
  14. pllim
    closed this as completedin #4659on Feb 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bugio.fits

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions