Repository navigation
ValueError FITS header values must contain standard printable ASCII characters. Unicode in FITS header entry. #3497
Description
I am encountering this problem with unicode in a FITS header entry. I have reproduced the problem with astropy 4.2, 4.4, and now the development version. I'm using Mac OSX 10.6, with anaconda. The problem is that unicode is not supported in FITS headers (apparently by design). Is this this a bug for an intentional limitation? Forgive me if this is a n00b question, but I thought the behavior was weird enough (since somehow the unicode got into the FITS header in the first place), and I could not find any similar issue either open or closed. Thank you for your time to review this issue.
sp = fits.open("filename.fits")
sp[0].header[20] #same as sp[0].header['SPT']VerifyError: Unparsable card (SPT), fix it first with .verify('fix').
sp.verify('fix')ValueError: FITS header values must contain standard printable ASCII characters; "'L0\xce\xb3 ' /Spectral type (visual classification)" contains characters/bytes that do not represent printable characters in ASCII.
Meanwhile, here is some abbreviated output.
In [21]: header
Out[21]:
[...]
RA = '01:24:45.990' /Right ascension J2000 at the epoch of 2MASS
DEC = '-57:45:37.90' /Declination J2000 at the epoch of 2MASS
SPT = 'L0γ ' /Spectral type (visual classification)
COMMENT
COMMENT OBSERVING INFORMATION
[...]
Activity
embray commentedon Feb 12, 2015
Yep. FITS predates unicode and has never been updated to support anything beyond the ASCII printable characters for data. It is impossible to encode non-ASCII characters in FITS headers.
also does nothing in the case. It's similar to some of the examples on this page that use an invalid keyword name 'P.I.' No unicode involved in that, but the basic principle is the same--there is no way to automatically 'fix' it without losing information. It could outright remove the offending characters, but better to let the user know there's something wrong and that the data needs to be manually updated (i.e. by manually assigning a new value to the 'SPT' keyword in this case).
That second
ValueErroryou reported could be better though. It doesn't say what keyword it was in, and could make it clearer that it encountered an unfixable error.embray commentedon Feb 12, 2015
Do you know who produced the file or what software they used?
embray commentedon Feb 12, 2015
I'm surprised you could read in the file at all actually due to this issue, which I still encounter when I try to read in a fake file I made with bad characters in it.
embray commentedon Feb 12, 2015
Actually, now I'm not sure if that was the issue I meant to link to. But I think there was another one somewhere that specifically had to do with handling illegal characters during the first pass read-in of the file.
embray commentedon Feb 12, 2015
I see--it does work on Python 2. But then when I try to display the full header I get:
If I try to display that specific card I get:
which is of course misleading because it's not a fixable error. All stuff to keep in mind when reworking header validation...
saimn commentedon May 19, 2017
Closing since the issue is quite old, and more details would be needed to test with the recent changes (#6084). Please reopen if needed.