FITS headers don't update immediately?

This is an issue noted on #5396, but is independent of that issue.

In short, given a FITS header with an invalid entry (e.g., a newline within the 'ORIGIN' value), the following will happen:

fh = fits.open(fn)
fh[0].header.tostring() # produces exception
fh[0].header.tostring() # does not produce exception

>>> h = fits.open('test1n.fits')
>>> h[0].header['origin'] = 'goodval'
>>> w = wcs.WCS(h[0].header)  # <- crashes
# a repeat attempt here would (probably) not crash

see: #5396 (comment), #5396 (comment), #5396 (comment), #5396 (comment)

@keflavich

No idea how to fix this, but thanks for the workaround @keflavich -- it just saved me having to reprocess several thousand images from a couple years ago.

@mwcraig

For clarification, the workaround is to do exactly as above, but with a try/except around it

fh = fits.open(fn)
try:
    fh[0].header.tostring() # produces exception
except ValueError:
    fh[0].header.tostring() # does not produce exception

(@mwcraig - let me know if I'm misremembering and there was some other component that is the "actual" workaround)

That is indeed the fix. I had several hundred images with newline characters in the header, fixed them with this snippet (your workaround is in the last few lines):

from ccdproc import ImageFileCollection
from astropy.io.fits import VerifyError

ic = ImageFileCollection('.')

for hdu in ic.hdus(save_location='foo'):
    h = hdu.header
    bads = []
    for i, c in enumerate(h.cards):
        if isinstance(c[1], str) and '\n' in c[1]:
           bads.append(i)
    for b in bads:
        new = h[b].replace('\n', '~~')
        h[b] = new
    try:
        h.tostring()
    except VerifyError:
        # Yes, seriously, this.
        h.tostring()
    h.update()

Just to confirm that there is clearly an issue :

In [1]: %astropy
Numpy 1.12.1
Astropy 2.0.dev18422

In [2]: c = fits.Card.fromstring('FOO     = BAR\n   ')

In [3]: c.value
---------------------------------------------------------------------------
VerifyError                               Traceback (most recent call last)
<ipython-input-3-d335c3962011> in <module>()
----> 1 c.value

/home/simon/dev/astropy/astropy/io/fits/card.py in value(self)
    279             value = self._value
    280         elif self._valuestring is not None or self._image:
--> 281             self._value = self._parse_value()
    282             value = self._value
    283         else:

/home/simon/dev/astropy/astropy/io/fits/card.py in _parse_value(self)
    738         if m is None:
    739             raise VerifyError("Unparsable card ({}), fix it first with "
--> 740                               ".verify('fix').".format(self.keyword))
    741 
    742         if m.group('bool') is not None:

VerifyError: Unparsable card (FOO), fix it first with .verify('fix').

In [4]: c.value = 'FIXED'

In [5]: c.value
Out[5]: 'FIXED'

In [6]: c
Out[6]: ('FOO', 'FIXED', '')

In [7]: c.verify('fix')
WARNING: VerifyWarning: Verification reported errors: [astropy.io.fits.verify]
WARNING: VerifyWarning: Card 'FOO' is not FITS standard (invalid value string: 'BAR').  Fixed 'FOO' card to meet the FITS standard. [astropy.io.fits.verify]
WARNING: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.
 [astropy.io.fits.verify]

In [8]: c.value
Out[8]: 'BAR'

It looks like the private Card._image is not updated when setting a new value, and this attribute is used at least when verifying the card which brings back the old value ! 😨

@yannick1974

I was able to reproduce this much in the same way as @yannick1974 by creating a FITS header containing an invalid card value (the literal INF in this case).

If I open the file, call header.tostring() first (which implicitly verifies and fixes the broken card--another aspect of FITS verification that's a bit weird and broken), then manually update the value and write it out, it works:

>>> f = fits.open('bogus.fits')
>>> f[0].header.tostring()
WARNING: VerifyWarning: Verification reported errors: [astropy.io.fits.verify]
WARNING: VerifyWarning: Card 'KW' is not FITS standard (invalid value string: 'INF').  Fixed 'KW' card to meet the FITS standard. [astropy.io.fits.verify]
WARNING: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.
 [astropy.io.fits.verify]
"SIMPLE  =                    T / conforms to FITS standard                      BITPIX  =                    8 / array data type                                NAXIS   =                    0 / number of array dimensions                     EXTEND  =                    T                                                  KW      = 'INF     '                                                            END                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             "
>>> f[0].header['KW'] = -1
>>> f.writeto('bogus_fixed.fits', overwrite=True)
>>> fits.getheader('bogus_fixed.fits')
SIMPLE  =                    T / conforms to FITS standard                      
BITPIX  =                    8 / array data type                                
NAXIS   =                    0 / number of array dimensions                     
EXTEND  =                    T                                                  
KW      =                   -1

But if I just open the file, try to update the value of KW before the the header has been fully parsed/verified, and then try to write it out, it doesn't take the update and produces an exception:

>>> f = fits.open('bogus.fits')
>>> f[0].header['KW'] = -1
>>> f.writeto('bogus_fixed.fits', overwrite=True)
Traceback (most recent call last):
  File "<ipython-input-101-d81d9e87de59>", line 1, in <module>
    f.writeto('bogus_fixed.fits', overwrite=True)
  File "astropy/utils/decorators.py", line 535, in wrapper
    return function(*args, **kwargs)
  File "astropy/io/fits/hdu/hdulist.py", line 928, in writeto
    self.verify(option=output_verify)
  File "astropy/astropy/io/fits/verify.py", line 119, in verify
    raise VerifyError('\n' + '\n'.join(messages))
VerifyError: 
Verification reported errors:
HDU 0:
    Card 4:
        Card 'KW' is not FITS standard (invalid value string: 'INF').
Note: astropy.io.fits uses zero-based indexing.