Repository navigation
Even when a cached downloaded file exists, urlopen() is called #8676
Description
I'm tracking down a problem using a cached coordinates/sites.json on a read-only filesystem.
An issue I am finding is this:
in astropy/utils/data.py,
get_pkg_data_fileobj
ends up calling
get_readable_fileobj
with a URL, which calls
download_file()
Now, even if the file is already in the cache, on data.py:1005 it ends up calling
with urllib.request.urlopen(conf.dataurl, timeout=timeout) as remote:
_dataurls_to_alias[conf.dataurl] = [conf.dataurl, remote.geturl()]
to find out if the URL is an alias.
Now, okay, this only gets called once per session, but really? Please can you avoid doing this somehow? Pre-fill known aliases for data.astropy.org, maybe? ie, initialize:
_dataurls_to_alias = {'http://data.astropy.org/': ['http://data.astropy.org/', 'https://astropy.stsci.edu/data/']}
Activity
pllim commentedon May 8, 2019
Did you pass in a
cache=Trueoption?dstndstn commentedon May 8, 2019
Yes, it gets passed in from get_pkg_data_contents. And it also doesn't matter, this code fragment gets called regardless of the cache setting.
https://github.com/astropy/astropy/blob/master/astropy/utils/data.py#L1001
pllim commentedon May 8, 2019
Ah, okay, yes. That is not related to the
cache. The way it is designed currently is because the URL can actually be configured by user (if they want to) and we have no idea if user provided URL even have alias or not. And if we start to hardcode all the known aliases that might be used by users, where do we stop? Once per session is not that high of an overhead, is it? What is your use case that this is a blocker?dstndstn commentedon May 8, 2019
Well.... you could make the simple case simple :) and pre-populate the _dataurls_to_alias with the single alias of the default URL.
OR, have a config option to not resolve aliases. This would allow us to pre-download the file to the cache, and then the user does not need internet access to use it.
aarchiba commentedon Aug 29, 2019
If I understand the code correctly, the failure of the alias lookup doesn't cause a problem, it just gets the URL listed as not having an alias and the function proceeds fine (possibly after a timeout, which you could set small if the Internet is never accessible on the machine). So this one call poses no particular problem to offline use of astropy.
dstndstn commentedon Aug 29, 2019
I'm sorry, but I have to object to this line of reasoning. We have the data in the cache!! But the code still insists on, just in case, hitting the URL to check whether it is an alias! Setting a timeout to be small is a work-around that should not be necessary.
dstndstn commentedon Aug 29, 2019
PS, to answer a previous question, this came up in some code where we need to compute the airmass of an observation, and that required looking up the site coordinates. This is in a Docker container, so we want to pre-populate the container with cached files as much as possible, because we're going to run this on millions of observations on a cluster. On this particular cluster, the compute nodes cannot see the internet. (They do communicate with other nodes on the cluster, so we can't just mess with timeouts without potentially introducing side effects.) And also, I mean, come on, you went to all the work of writing a caching system, and this issue makes that system not nearly as useful as it should be.
Did I mention there's a one-line fix?
pllim commentedon Aug 29, 2019
@aarchiba , is your PR #9182 an attempt to address this issue?
pllim commentedon Aug 29, 2019
@dstndstn , I understand the frustration, but one of the downside of open source relying on volunteer time is that things might not get done in time or to everyone's satisfaction. If you do not want your code to access the internet, here are a couple of quick hacks that I can think of:
astropy/astropy/utils/data.py
Lines 47 to 52 in b6b1f36
dstndstn commentedon Aug 30, 2019
I just sent PR #9187 with my one-line fix
pllim commentedon Oct 30, 2019
With #9182 merged, this is no longer an issue in dev and will not be an issue in v4.0. Thank you for your patience!