Astropy data mirror download infrastructure is not robust

Even with #5547, I still see timeouts. Will more mirrors fix this problem or should we revive #5994?

@astrofrog

Here are another examples of the URL issues: https://circleci.com/gh/astropy/astropy/8682?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

https://travis-ci.org/astropy/astropy/jobs/389046646

@astrofrog as already mentioned somewhere that it's very unlikely that both the astropy data server and the github mirror fail at the same time, so this may be a symptom of a bug with the download infrastructure.

Come to think of it, I wonder if it is the test itself that is flawed. Consider this:

    def test_download_parallel():
        # ...
        main_url = 'http://data.astropy.org/intersphinx/README'
        mirror_url = 'http://www.astropy.org/astropy-data/intersphinx/README'
        fnout = download_files_in_parallel([
>           'http://data.astropy.org', main_url, mirror_url])

This test is set up to hit data.astropy.org twice and also www.astropy.org at the same time. Maybe that sometimes trigger some server-side panic? Will have to think about this some more...

Yes, have been seeing fairly regular timeouts again...

I agree that there must be some bug hiding in the mirror infrastructure. Note that the easiest way to simulate an issue with the primary server is to edit your /etc/hosts file and add:

127.0.0.1       data.astropy.org

@mhvk

@mhvk do you mean just the test_download_parallel or some other ones as well?

@pllim

@pllim - I fear I have not looked carefully. It is just that regularly there is a failing travis tests, and scrolling to the bottom I see a URL timeout, so I restart. It would be very nice if this were not necessary.

One thing that is odd is that I only see this on travis - do the others not do any remote tests? Or does that mean it is the all-remote tests that usually fail.

do the others not do any remote tests?

Appveyor does not. CircleCI only does remote tests for visualization.

And circleCI fails with those downloads regularly, too.

I'll try and investigate this

@pllim

See #7573 for a coupe of failures (including the one from @pllim in #6061 (comment)) that happen consistently if you 'turn off' one of the servers. The issue with that specific test is that it's flawed - it will only pass if both the main server and the mirror work.

Yeah... I think that is my bad (#6987)... I'll fix it...

Everyone... Hopefully this is fixed. Please re-open if the problem resurfaces. Thanks for your patience!

Navigation Menu

Repository navigation

Astropy data mirror download infrastructure is not robust #6061

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Astropy data mirror download infrastructure is not robust #6061

Description

Activity

bsipocz commented on Jun 7, 2018

pllim commented on Jun 7, 2018

mhvk commented on Jun 13, 2018

astrofrog commented on Jun 13, 2018

pllim commented on Jun 14, 2018

mhvk commented on Jun 14, 2018

pllim commented on Jun 14, 2018

bsipocz commented on Jun 15, 2018

astrofrog commented on Jun 18, 2018

astrofrog commented on Jun 18, 2018

pllim commented on Jun 18, 2018

pllim commented on Jun 21, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions