Repository navigation
Test failures if data.astropy.org is down #7573
Description
It's possible to simulate what would happen if data.astropy.org is down by adding the following row to /etc/hosts on a Linux/MacOS X machine:
127.0.0.1 data.astropy.org
Running tests with --remote-data and the above results in:
============================================================================== FAILURES ===============================================================================
__________________________________________________________ [doctest] astropy.utils.data.get_pkg_data_fileobj __________________________________________________________
238 ... fcontents = fobj.read()
239 ...
240
241 This next example would download a data file from the astropy data server
242 because the ``allsky/allsky_rosat.fits`` file is not present in the
243 source distribution. It will also save the file locally so the
244 next time it is accessed it won't need to be downloaded.::
245
246 >>> from astropy.utils.data import get_pkg_data_fileobj
247 >>> with get_pkg_data_fileobj('allsky/allsky_rosat.fits',
Expected:
Downloading http://data.astropy.org/allsky/allsky_rosat.fits [Done]
Got:
Downloading http://www.astropy.org/astropy-data/allsky/allsky_rosat.fits [Done]
/private/var/folders/zy/t1l3sx310d3d6p0kyxqzlrnr0000gr/T/astropy-test-7szyaoca/lib/python3.6/site-packages/astropy/utils/data.py:247: DocTestFailure
_______________________________________________________________________ test_download_parallel ________________________________________________________________________
@pytest.mark.remote_data(source='astropy')
def test_download_parallel():
import shelve
from ..data import (download_file, download_files_in_parallel,
_get_download_cache_locs, get_cached_urls)
main_url = 'http://data.astropy.org/intersphinx/README'
mirror_url = 'http://www.astropy.org/astropy-data/intersphinx/README'
fnout = download_files_in_parallel([
> 'http://data.astropy.org', main_url, mirror_url])
astropy/utils/tests/test_data.py:55:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
astropy/utils/data.py:1191: in download_files_in_parallel
multiprocess=True)
astropy/utils/console.py:745: in map
ipython_widget=ipython_widget)
astropy/utils/console.py:818: in map_unordered
p.imap_unordered(function, items, chunksize=chunksize)):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <multiprocessing.pool.IMapUnorderedIterator object at 0x1c2df96588>, timeout = None
def next(self, timeout=None):
with self._cond:
try:
item = self._items.popleft()
except IndexError:
if self._index == self._length:
raise StopIteration
self._cond.wait(timeout)
try:
item = self._items.popleft()
except IndexError:
if self._index == self._length:
raise StopIteration
raise TimeoutError
success, value = item
if success:
return value
> raise value
E multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x1c2df962b0>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'
/Users/tom/miniconda3/envs/dev/lib/python3.6/multiprocessing/pool.py:735: MaybeEncodingError
================================================== 2 failed, 10574 passed, 94 skipped, 58 xfailed in 306.22 seconds ===================================================
The first failure can be prevented - we shouldn't be checking the output of that doctest since otherwise it only works when the primary data server works. The second failure is occurring because it expects both the primary server and the mirror to work. I think the files in parallel should indeed be multiple files on the main server, and if that fails multiple files on the mirror.
Activity