Repository navigation
[1.3] test_fortran_* failure #5693
Description
We have a group that tests every package for bitwise reproducibility of the packaging, and they found a built time test failure in test_fortran_reader and in test_fortran_invalid_exp (on x86_64):
_________________________ test_fortran_reader[True] ___________________________
parallel = True
@pytest.mark.parametrize("parallel", [
pytest.mark.xfail(os.name == 'nt', reason=
"Multiprocessing is currently unsupported on Windows")(True),
False])
def test_fortran_reader(parallel):
"""
Make sure that ascii.read() can read Fortran-style exponential notation
using the fast_reader.
"""
text = 'A B C\n100.01{:s}+99 2.0 3\n 4.2{:s}-1 5.0{:s}-1 0.6{:s}4'
expected = Table([[1.0001e101, 0.42], [2, 0.5], [3.0, 6000]],
names=('A', 'B', 'C'))
expstyles = { 'e': 4*('E'), 'D': ('D', 'd', 'd', 'D'), 'Q': 2*('q', 'Q'),
'fortran': ('D', 'E', 'Q', 'd') }
# C strtod (not-fast converter) can't handle Fortran exp
with pytest.raises(FastOptionsError) as e:
ascii.read(text.format(*(4*('D'))), format='basic', guess=False,
fast_reader={'use_fast_converter': False,
'parallel': parallel, 'exponent_style': 'D'})
assert 'fast_reader: exponent_style requires use_fast_converter' in str(e)
# enable multiprocessing and the fast converter
# iterate over all style-exponent combinations
for s, c in expstyles.items():
table = ascii.read(text.format(*c), format='basic', guess=False,
fast_reader={'parallel': parallel,
'exponent_style': s})
> assert_table_equal(table, expected)
astropy/io/ascii/tests/test_c_reader.py:1087:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
astropy/io/ascii/tests/test_c_reader.py:37: in assert_table_equal
assert_equal(t1[name].dtype.kind, t2[name].dtype.kind)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
a = 'S', b = 'f'
def assert_equal(a, b):
> assert a == b
E AssertionError
astropy/io/ascii/tests/common.py:39: AssertionError
________________________ test_fortran_invalid_exp[True] ________________________
parallel = True
@pytest.mark.parametrize("parallel", [
pytest.mark.xfail(os.name == 'nt', reason=
"Multiprocessing is currently unsupported on Windows")(True),
False])
def test_fortran_invalid_exp(parallel):
"""
Test Fortran-style exponential notation in the fast_reader with invalid
exponent-like patterns (no triple-digits) to make sure they are returned
as strings instead, as with the standard C parser.
"""
if parallel and TRAVIS:
pytest.xfail("Multiprocessing can sometimes fail on Travis CI")
fields = [ '1.0001+1', '.42d1', '2.3+10', '0.5', '3+1001', '3000.',
'2', '4.56e-2.3', '8000', '4.2-122' ]
values = [ '1.0001+1', 4.2, '2.3+10', 0.5, '3+1001', 3.e3,
2, '4.56e-2.3', 8000, 4.2e-122 ]
t = ascii.read(StringIO(' '.join(fields)), format='no_header', guess=False,
fast_reader={'parallel': parallel, 'exponent_style': 'A'})
read_values = [col[0] for col in t.itercols()]
> assert read_values == values
E assert ['1.0001+1', ..., 3000.0, ...] == ['1.0001+1', 4..., 3000.0, ...]
E At index 1 diff: '.42d1' != 4.2
E Full diff:
E ['1.0001+1',
E - '.42d1',
E + 4.2,
E '2.3+10',
E 0.5,
E '3+1001',
E 3000.0,
E 2,
E '4.56e-2.3',
E 8000,
E - '4.2-122']
E ? - -
E + 4.2e-122]
E ? +
astropy/io/ascii/tests/test_c_reader.py:1123: AssertionError
I am not sure whether this is a regression, and I also don't know what they do different from the normal setup. Full build log here. This is independent of whether external (3.0.5) or internal pytest ist used.
Activity
pllim commentedon Jan 13, 2017
This was added in #5552 by @dhomeier . In both failures, I notice that parallel processing is enabled.
dhomeier commentedon Jan 13, 2017
Yes, and at least the second fails on the automatic detection of accepted exponent styles. I'd bet that it's also failing on the auto-detection pass of
test_fortran_reader- could you check if the test passes when you remove the'fortran'key from theexpstylesdict?I don't know exactly how the parallel reader works, but I could imagine how it interferes with the auto-detection if different threads find different input formats. Should probably mark the parallel test version as
xfail, but perhaps beyond that the fast reader should in fact raise an exception when these two options are combined.dhomeier commentedon Jan 14, 2017
Actually it's not that clear to me how the parallel read interferes with the
auto/Fortranstyles, since the exponent styles are detected on a per-item base (i.e. independently in each call ofxtrtod). It seems a bit as ifself.tokenizeris initialised differently in a parallel read.I am wondering if some of the problems with
guessing formats come into play here as well. @olebole, could you test if these errors still occur with PR #5578?olebole commentedon Jan 25, 2017
I applied #5578 (putting to Debian experimental as 1.3-8~exp2) and it works. However, also in the unpatched variant (currently 1.3.7), the failure disappeared... So, the problem appears to be non-reproducible for the moment (and unfortunately, the build log referenced above disappeared as well), and I am unable to confirm that the changes fixes it.
bsipocz commentedon Jul 6, 2017
@olebole - Is this still an issue?
olebole commentedon Jul 6, 2017
This disappeared almost everywhere, except for GNU Hurd.
dhomeier commentedon Jul 6, 2017
That is without #5578 then? I noticed four patches in the build, but none of them looked like they addressed the
fast_readerissues.olebole commentedon Jul 6, 2017
That is 2.0rc1. Since #5578 is not merged, I would guess it is without it?
dhomeier commentedon Jul 6, 2017
Yes, just wanted to check it was not patched in the way, I think, you did with Debian
1.3.8~exp2; but from the names of the patches I would also guess it's not.olebole commentedon Jul 6, 2017
1.3-8~exp2 was a test build, I removed the patch afterwards.
I also rechecked the build 1.3-8~exp2 mentioned above on GNU Hurd. It seems that 1.3-8~exp2 did not have this Problem on Hurd; however there were a bunch of other problems with the fast reader, and I am unable to separate them now.
dhomeier commentedon Jul 6, 2017
Those seem to be mostly (if not all) show up with
parallel=True, at least this looks multithreading-related:dhomeier commentedon Jul 7, 2017
Yes, all
CParserErrors in parallel mode except for a singleInconsistentTableErrorwhere the reader seems to stumble overQ-style exponents.But in
2.0~rc1-1also only two of the Fortran-reader failures are left.bsipocz commentedon Mar 18, 2019
@olebole - next time you run tests, could you please check whether you still see this on any of the platforms and close if it's gone?
olebole commentedon Mar 18, 2019
This is probably fixed, at least for Linux. For GNU Hurd, we don't build it in the moment, since some of the dependencies are not there yet.
I would propose to close it, I will re-open when it seems to re-appear.
bsipocz commentedon Mar 18, 2019
Thank you for the feedback @olebole!