You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, essentially all PRs are continuously failing travis because of URL time-outs. This is huge pain. Should we remove the remote-data tests from travis? Might they run on circle or appveyor instead?
Perhaps 3-second remote timeout is too short. I will try an experimental PR to increase timeout to 10 seconds and see what happens when I get the chance.
But they would stop molesting innocent PRs in case they are failing (no matter why). On the other hand the tests do serve a purpose especially in case they always fail: It means either out remote-test setup is wrong, the test case itself isn't working correctly or the "real" code is. 😄
I thought @bsipocz had mentioned there was a known travis issue; this is partially why I wondered about trying to run remote data tests on appveyor or circle.
_______________________ TestUfuncCoverage.test_coverage ________________________
self = <astropy.units.tests.test_quantity_ufuncs.TestUfuncCoverage object at 0x7f749fc24f98>
def test_coverage(self):
all_np_ufuncs = set([ufunc for ufunc in np.core.umath.__dict__.values()
if type(ufunc) == np.ufunc])
from .. import quantity_helper as qh
all_q_ufuncs = (qh.UNSUPPORTED_UFUNCS |
set(qh.UFUNC_HELPERS.keys()))
> assert all_np_ufuncs - all_q_ufuncs == set([])
E assert {<ufunc 'positive'>} == set()
E Extra items in the left set:
E <ufunc 'positive'>
E Use -v to get the full diff
astropy/units/tests/test_quantity_ufuncs.py:25: AssertionError
@mhvk - I was away for a few days, and now trying to catch up with things. I suspected this to be related to travis as everything worked perfectly when running the test locally and as @jhunkeler didn't see anything in the stsci logs. However I haven't raised this to travis support as I wanted to come up with some good comparisons of passing and timeouting builds but hasn't got the time for it yet.
I'm hesitant to remove the remote tests, maybe we can try to move it to appveyor if it keeps being horridly unreliable. However the current circle setup (32 bit with old numpy etc) doesn't seem to be appropriate enough for this.
I would like to keep the remote tests from the astropy data server, otherwise WCSAxes won't get properly tested. Looks like this was only a temporary issue and indeed with the mirror functionality now merged, this should be a rarer issue.
The full remote tests are an allowed failure, which is fine. Since we have two servers for astropy data and since the timeout is now larger, I'm closing this issue.
Activity
pllim commentedon May 2, 2017
Ops, is data.astropy.org down? @jhunkeler maintains that but he is on vacation. @rendinam, are you Joe's backup for this as well?
FWIW, I think we should solve the underlying timeout problem rather than remove the tests. We don't want to throw out baby with bath water.
rendinam commentedon May 2, 2017
I am not a maintainer of data.astropy.org, but it loads for me both from off-site and via the STScI network.
pllim commentedon May 2, 2017
Thanks, Matt! Sorry to bother you.
Perhaps 3-second remote timeout is too short. I will try an experimental PR to increase timeout to 10 seconds and see what happens when I get the chance.
MSeifert04 commentedon May 2, 2017
Wasn't there some discussion to make the remote tests cronjob-only with some email notifications in case one failed? I can't find it right now...
pllim commentedon May 2, 2017
Cron job or not, the tests are useless if they always fail.
MSeifert04 commentedon May 2, 2017
But they would stop molesting innocent PRs in case they are failing (no matter why). On the other hand the tests do serve a purpose especially in case they always fail: It means either out remote-test setup is wrong, the test case itself isn't working correctly or the "real" code is. 😄
mhvk commentedon May 2, 2017
I thought @bsipocz had mentioned there was a known travis issue; this is partially why I wondered about trying to run remote data tests on appveyor or circle.
pllim commentedon May 2, 2017
Huh... This is weird... The tests are magically working again... (Even the Sphinx one from #5992). Only numpy-dev test fails now.
https://travis-ci.org/astropy/astropy/builds/227884308
bsipocz commentedon May 4, 2017
@mhvk - I was away for a few days, and now trying to catch up with things. I suspected this to be related to travis as everything worked perfectly when running the test locally and as @jhunkeler didn't see anything in the stsci logs. However I haven't raised this to travis support as I wanted to come up with some good comparisons of passing and timeouting builds but hasn't got the time for it yet.
pllim commentedon May 9, 2017
Is this still relevant? Looks like the failures not so bad anymore. And I think with addition of mirror by #5547, things will get even better?
bsipocz commentedon May 9, 2017
I'm hesitant to remove the remote tests, maybe we can try to move it to appveyor if it keeps being horridly unreliable. However the current circle setup (32 bit with old numpy etc) doesn't seem to be appropriate enough for this.
astrofrog commentedon May 10, 2017
I would like to keep the remote tests from the astropy data server, otherwise WCSAxes won't get properly tested. Looks like this was only a temporary issue and indeed with the mirror functionality now merged, this should be a rarer issue.
astrofrog commentedon May 10, 2017
The full remote tests are an allowed failure, which is fine. Since we have two servers for astropy data and since the timeout is now larger, I'm closing this issue.
mhvk commentedon May 10, 2017
OK, that's fine. If it turns out that this is mostly a travis issue, we can still consider moving the tests to appveyor.