Repository navigation
Resource error when accessing IERS data using multiprocessing #10114
Description
From https://stackoverflow.com/questions/61043112/astropy-multiprocessing-access-to-iers-data, this looks like a IERS issue related to multiprocessing:
We are using astropy to perform large SKA simulations running in Dask on single nodes or cluster. We use both Time and astroplan.Observer in our calculations. Some fraction of the time, we see errors in access of the IERS data. For example:
rascil/processing_components/visibility/base.py:275: in create_blockvisibility
570 from astroplan import Observer
571 /usr/local/lib/python3.7/site-packages/astroplan/__init__.py:32: in <module>
572 get_IERS_A_or_workaround()
573 /usr/local/lib/python3.7/site-packages/astroplan/utils.py:57: in get_IERS_A_or_workaround
574 if IERS_A_in_cache():
575 /usr/local/lib/python3.7/site-packages/astroplan/utils.py:78: in IERS_A_in_cache
576 with _open_shelve(urlmapfn, True) as url2hash:
577 /usr/local/lib/python3.7/site-packages/astroplan/utils.py:322: in _open_shelve
578 shelf = shelve.open(shelffn, protocol=2)
579 /usr/local/lib/python3.7/shelve.py:243: in open
580 return DbfilenameShelf(filename, flag, protocol, writeback)
581 /usr/local/lib/python3.7/shelve.py:227: in __init__
582 Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
583 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
584 > return mod.open(file, flag, mode)
585 E _gdbm.error: [Errno 11] Resource temporarily unavailable
586 /usr/local/lib/python3.7/dbm/__init__.py:94: error
Our interpretation is that this arises from multiple Dask workers trying to access the same cache file. It does occur whenever using multiple processes on the same node or processes spread across a cluster accessing a shared mount point. The same functions performed serially do not cause this error.
For workarounds we have tried the following:
iers.conf.auto_max_age = None
iers.conf.remote_timeout = 100.0
data.conf.download_cache_lock_attempts = 10
None of these have helped. We still see the same gdbm errors.
Activity
bsipocz commentedon Apr 5, 2020
Are they using the latest versions? (I mostly think about astropy, a lot has happened in the recent releases that may be related to this, but also what astroplan is being used? It's is quite often a hit and miss compatibility-wise.)
mhvk commentedon Apr 6, 2020
Adding the general
utils, as this likely has to do with caching and thusdownload_file, etc. cc @aarchibatimcornwell commentedon Apr 6, 2020
We are using the following:
astroplan==0.6
astropy==4.0
pllim commentedon Apr 6, 2020
Please try with the 4.1.dev version from
master, or at least 4.0.1, just in case.timcornwell commentedon Apr 7, 2020
I'm trying 4.0.1 and will try the dev version afterwards.
timcornwell commentedon Apr 7, 2020
4.0.1post1 didn't fix it. I can try the dev version but it'll take a little time to set it up.
I should add a bit more info - we see this error intermittently, mostly during our CI on GitLab. There are hundreds of unit tests and maybe a few failures are seen per run. At the moment, this means that almost none of our builds succeed, which really gets in the way of development.
pllim commentedon Apr 7, 2020
If 4.0.1 didn't fix it, dev is unlikely to fix it either, if I remember what patches went where correctly. 😞
I don't have any good ideas right now, but maybe @aarchiba has. It is probably coming from
shelve-- https://stackoverflow.com/questions/52381091/resource-temporarily-unavailable-on-python-shelve-openpllim commentedon Apr 7, 2020
And just to be sure this is not already fixed in
astroplan, did you try their dev version? I see that a few patches had gone in theirmastersince 0.6 release.timcornwell commentedon Apr 7, 2020
@pllim We see it in astropy.time.Time as well as astroplan but I'll check it out anyway.
By the way, I appreciate that it's not a straightforward fix :)
pllim commentedon Apr 7, 2020
Coincidentally, the
Timestuff with cluster computing is being discussed in #9479.bsipocz commentedon Apr 7, 2020
If you see it with
Timethen there is no need to check astroplan dev. In fact better to just stick to sort this issue out with astropy only test case.pllim commentedon Apr 7, 2020
I was wondering if
astroplangot a hack in it to work around this. But it was a stretch on my part.aarchiba commentedon Jun 2, 2020
Pre-4.0 astropy had this problem sporadically - the gdbm backend for shelve has a reader-writer lock built-in, and it did not live behind a reliable locking mechanism, so when contention occurred you got exactly this error.
In fact it's unclear how this error can be produced from 4.0-plus astropy, since as far as I can tell it catches and handles all dbm exceptions. Are you sure the error is coming from astropy 4.0 and not an earlier version?
pllim commentedon Jun 5, 2020
Thanks, @aarchiba ! Does #10437 also fix this issue?
aarchiba commentedon Jun 15, 2020
Definitely fixed by #10437 - gdbm is no longer used.