Skip to content

BUG: Ignore mirror caching when no internet - #8163

Merged
bsipocz merged 2 commits into
astropy:masterfrom
pllim:no-internet-cache
Dec 4, 2018
Merged

bsipocz merged 2 commits into
astropy:masterfrom
pllim:no-internet-cache

Conversation

@pllim

@pllim pllim commented Nov 20, 2018

Copy link
Copy Markdown
Member

Address a bug reported by @parejkoj in #7653 where obtaining cached download fails when there is no internet. This does not address the original feature request in #7653 (i.e., providing a user-defined sites.json).

@eteq , can you please review, given you also reviewed #6987 that introduced this bug?

@bsipocz , not sure if this can be cleanly backported to 2.x though #6987 was put in 2.0.4. What should I milestone this to? Do you need a separate PR straight to 2.x branch? Please advise.

Thanks!

@astropy-bot

This comment has been minimized.

2 similar comments
@astropy-bot

This comment has been minimized.

@astropy-bot

This comment has been minimized.

@astropy-bot

astropy-bot Bot commented Nov 20, 2018 •

Copy link
Copy Markdown

Hi there @pllim 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

@codecov

codecov Bot commented Nov 20, 2018 •

Copy link
Copy Markdown

Codecov Report

Merging #8163 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8163      +/-   ##
==========================================
+ Coverage   86.92%   86.92%   +<.01%     
==========================================
  Files         383      383              
  Lines       57860    57863       +3     
  Branches     1056     1056              
==========================================
+ Hits        50295    50299       +4     
+ Misses       6951     6950       -1     
  Partials      614      614
Impacted Files Coverage Δ
astropy/utils/data.py 80.93% <100%> (+0.35%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 168db06...26223ac. Read the comment docs.

@parejkoj

Copy link
Copy Markdown
Contributor

Is there any way to write a unittest for this? It looks like there's a pytest-socket plugin to disable the internet during tests: https://github.com/miketheman/pytest-socket

@pllim

pllim commented Nov 20, 2018

Copy link
Copy Markdown
Member Author

Is there any way to write a unittest for this?

@drdavella , is there some kind of magical thingies 🦄 I can use from pytest-remotedata?

@pllim

pllim commented Nov 20, 2018

Copy link
Copy Markdown
Member Author

FWIW, I did test this patch on my laptop with internet turned off. 😅

@bsipocz bsipocz added this to the v2.0.10 milestone Nov 20, 2018
@drdavella

Copy link
Copy Markdown
Contributor

pytest-remotedata should disable internet access for all tests if --remote-data=none.

@bsipocz

bsipocz commented Nov 20, 2018

Copy link
Copy Markdown
Member

This looks backportable, so I've milestoned as 2.0.10 and will try to remember to double check the behaviour (but having a test doing that for me would be even more awesome)

@pllim
pllim force-pushed the no-internet-cache branch from 9c8d5cb to 26223ac Compare November 20, 2018 22:11
@pllim

pllim commented Nov 20, 2018

Copy link
Copy Markdown
Member Author

Incidentally, the test I refactored back in #7575 was not quite right (it wasn't exactly testing the thing I designed for it to test). So I fixed it here and also incidentally, that test also somehow tests this issue, because urlopen() emits URLError for local file URI that is a directory, just as if there is no internet. Win, win!

p.s. Given how careless I was at #7575, extra pairs of eyes to check are appreciated!

@pllim

pllim commented Nov 20, 2018

Copy link
Copy Markdown
Member Author

p.s. @bsipocz , I have a feeling that this might need a manual backport. Just let me know if that is indeed the case, I'll be happy to open another PR straight to 2.x. Thanks!

@bsipocz

bsipocz commented Nov 20, 2018

Copy link
Copy Markdown
Member

Yeah, the diff got much larger. I'll definitely ping you if we get there and if that's a bridge we need to cross.

@bsipocz

bsipocz commented Nov 27, 2018

Copy link
Copy Markdown
Member

@eteq - any chance for a review here, so this can land in RC2?

@eteq eteq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although I have one minor suggestion of giving a warning so the user isn't surprised by using the cache. (Approving on the basis that it just needs a 👍 or 👎 from @pllim on that one suggestion - I can live with it either way.)

Comment thread astropy/utils/data.py
with urllib.request.urlopen(conf.dataurl, timeout=timeout) as remote:
_dataurls_to_alias[conf.dataurl] = [conf.dataurl, remote.geturl()]
except urllib.error.URLError: # Host unreachable
_dataurls_to_alias[conf.dataurl] = [conf.dataurl]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
_dataurls_to_alias[conf.dataurl] = [conf.dataurl]
msg = ('url "{}" was requested in download_file, but was not found.'
' Will try cached file if it exists.')
warn(AstropyWarning(msg.format(conf.dataurl)))
_dataurls_to_alias[conf.dataurl] = [conf.dataurl]

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eteq , personally, I would prefer not emitting a warning. This is because this line was only to cache an alias of the original URL. Hence, user will not find the warning make any sense unless user knows the internal mechanics of this function. If this warning turns out to be necessary, we can always follow up in a future PR. Thank you for your understanding!

@bsipocz
bsipocz merged commit 09d6a45 into astropy:master Dec 4, 2018
@bsipocz

bsipocz commented Dec 4, 2018

Copy link
Copy Markdown
Member

Thanks @pllim!

@pllim
pllim deleted the no-internet-cache branch December 4, 2018 06:39
@bsipocz bsipocz modified the milestones: v2.0.10, v3.1 Dec 5, 2018
bsipocz added a commit that referenced this pull request Dec 5, 2018
BUG: Ignore mirror caching when no internet
@bsipocz

bsipocz commented Dec 5, 2018

Copy link
Copy Markdown
Member

f2f decision was to not backport this to LTS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants