Repository navigation
Silence warnings by default when reading in VO Tables #8028
Description
TL;DR
Users often are given files they don't have control over, and those files aren't always standard-compliant. This is especially true of VO Tables. I'd like to suggest that we make the VO Table reader more forgiving, although the writer should continue to emit warnings. Obviously we should discuss this first before doing, but I just want to put the proposal out there.
Details
Taking the example of VO Tables, the following is an example of reading in one of the files in our test suite (which wasn't there to test warnings):
In [5]: parse('gemini.xml')
WARNING: W49: gemini.xml:37:12: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
WARNING: W49: gemini.xml:49:12: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
WARNING: W49: gemini.xml:61:12: W49: Empty cell illegal for integer fields. [astropy.io.votable.converters]
WARNING: W48: gemini.xml:78:10: W48: Unknown attribute 'value' on OPTION [astropy.io.votable.tree]
WARNING: W48: gemini.xml:79:10: W48: Unknown attribute 'value' on OPTION [astropy.io.votable.tree]
WARNING: W06: gemini.xml:98:6: W06: Invalid UCD 'obs.field': Secondary word 'obs.field' is not valid as a primary word [astropy.io.votable.tree]
WARNING: W06: gemini.xml:99:6: W06: Invalid UCD 'obs.field': Secondary word 'obs.field' is not valid as a primary word [astropy.io.votable.tree]
WARNING: E02: gemini.xml:99:6: E02: Incorrect number of elements in array. Expected multiple of 3, got 1 [astropy.io.votable.converters]
WARNING: W06: gemini.xml:100:6: W06: Invalid UCD 'obs.field': Secondary word 'obs.field' is not valid as a primary word [astropy.io.votable.tree]
WARNING: W06: gemini.xml:101:6: W06: Invalid UCD 'em.wl;stat.interval': Unknown word 'stat.interval' [astropy.io.votable.tree]
WARNING: E02: gemini.xml:101:6: E02: Incorrect number of elements in array. Expected multiple of 2, got 1 [astropy.io.votable.converters]
WARNING: W06: gemini.xml:102:6: W06: Invalid UCD 'time;stat.interval': Unknown word 'stat.interval' [astropy.io.votable.tree]
WARNING: E02: gemini.xml:102:6: E02: Incorrect number of elements in array. Expected multiple of 2, got 1 [astropy.io.votable.converters]
WARNING: W06: gemini.xml:112:6: W06: Invalid UCD 'obs.field': Secondary word 'obs.field' is not valid as a primary word [astropy.io.votable.tree]
WARNING: W06: gemini.xml:113:6: W06: Invalid UCD 'obs.field': Secondary word 'obs.field' is not valid as a primary word [astropy.io.votable.tree]
WARNING: E02: gemini.xml:113:6: E02: Incorrect number of elements in array. Expected multiple of 3, got 1 [astropy.io.votable.converters]
WARNING: W06: gemini.xml:114:6: W06: Invalid UCD 'obs.field': Secondary word 'obs.field' is not valid as a primary word [astropy.io.votable.tree]
WARNING: W06: gemini.xml:115:6: W06: Invalid UCD 'em.wl;stat.interval': Unknown word 'stat.interval' [astropy.io.votable.tree]
WARNING: E02: gemini.xml:115:6: E02: Incorrect number of elements in array. Expected multiple of 2, got 1 [astropy.io.votable.converters]
WARNING: W06: gemini.xml:116:6: W06: Invalid UCD 'time;stat.interval': Unknown word 'stat.interval' (suppressing further warnings of this type...) [astropy.io.votable.tree]
WARNING: E02: gemini.xml:116:6: E02: Incorrect number of elements in array. Expected multiple of 2, got 1 [astropy.io.votable.converters]
WARNING: E02: gemini.xml:127:6: E02: Incorrect number of elements in array. Expected multiple of 3, got 1 [astropy.io.votable.converters]
WARNING: E02: gemini.xml:137:6: E02: Incorrect number of elements in array. Expected multiple of 2, got 1 [astropy.io.votable.converters]
WARNING: E02: gemini.xml:151:6: E02: Incorrect number of elements in array. Expected multiple of 3, got 1 [astropy.io.votable.converters]
WARNING: E02: gemini.xml:161:6: E02: Incorrect number of elements in array. Expected multiple of 2, got 1 (suppressing further warnings of this type...) [astropy.io.votable.converters]
Out[5]: <VOTABLE>... 1 tables ...</VOTABLE>
This is a pretty typical number of warnings in my experience with VO Tables. I've never done anything about any of the warnings though...
Note that there is actually a way to be even more pedantic:
In [6]: parse('gemini.xml', pedantic=True)
---------------------------------------------------------------------------
W49 Traceback (most recent call last)
<ipython-input-6-70047e7af5ca> in <module>()
----> 1 parse('gemini.xml', pedantic=True)
~/Dropbox/Code/Astropy/astropy/astropy/io/votable/table.py in parse(source, columns, invalid, pedantic, chunk_size, table_number, table_id, filename, unit_format, datatype_mapping, _debug_python_based_parser)
135 _debug_python_based_parser=_debug_python_based_parser) as iterator:
136 return tree.VOTableFile(
--> 137 config=config, pos=(1, 1)).parse(iterator, config)
138
139
...
~/Dropbox/Code/Astropy/astropy/astropy/io/votable/exceptions.py in vo_raise(exception_class, args, config, pos)
96 if config is None:
97 config = {}
---> 98 raise exception_class(args, config, pos)
99
100
W49: gemini.xml:37:12: W49: Empty cell illegal for integer fields.
But actually no way to be less pedantic and ignore the warnings (short of using warnings.catch_warnigns. I'd like to suggest that we add a verify key to the VO Table parse which can take different options as for FITS, including ignore, warn, exception (and possibly deprecate pendantic).
Furthermore, I think we might want to consider defaulting to 'ignore'.
We could also do something similar with FITS files - ignore warnings when reading but show them when writing?
Activity
pllim commentedon Oct 30, 2018
Good enough for FITS, good enough for me. This would address #7928 (comment) .
Do you have any strong opinions about this, @tomdonaldson and @theresadower ?
hamogu commentedon Nov 29, 2018
👍 to having a
verifykey and 👍 toignoreas the default.eteq commentedon May 15, 2019
👍to this, and 4.0 is a good time to do it, @astrofrog. But perhaps there should be a configuration item or other global-ish state that can be turned on? My thinking is that this might be good as a tool to test what's valid and what is not in particular workflows.
pllim commentedon May 15, 2019
For the record, we also discussed the possibility of adding a new
pedanticoption for this behavior and make it default. But at the same time, also implementverifyto do this "properly". We will removepedanticfrom docs (though it would still work for backward compatibility), then in a near future, deprecate it, and then only remove it from code. This is because having bothpedanticandverifyto control validation behavior is too confusing, but we also don't want to make people cry by suddenly removing support forpedantic.