Unable to disable remote_data test option

This is a forward of a bug in the Debian bug database:

The test option remote_data of Astropy now seems to be enabled by default and cannot be disabled (at least easily). This causes the tests of some Astropy affiliated packages to fail on Debian, e.g. astroplan.

@olebole

@olebole - hmm it should indeed still be disabled by default even with the changes I made recently. I will investigate.

In the mean time, you should be able to do --remove-data=none - does that help?

@olebole

@olebole - I think it's probably an astroplan issue rather than an astropy one, I saw this remote-data problem when testing with astropy dev (before the 1.3 release, so probably it also shows up for 1.3), but didn't have enough time to investigate the details. Also I'm afraid that there are other issues with astroplan as neither the latest release, nor the current dev works with astropy 1.3, there are quite a few test failures (see the recent PRs).

@bsipocz

@bsipocz is this due to some undocumented incompabilities? It would be nice to get them fixed soon, since we are going to prepare Debian Stretch and it would be useful to have a current set of astropy + affilated packages there.

@olebole

@olebole - Most of it is because of the recent coordinate fixes (e.g. #5254). There are a few WIP enhancement PRs from @StuartLittlefair in astroplan regarding coordinate handling, I expect that those might help, but he is the one who can comment further details.
I've done a few workaround PRs, but there are still a handful of broadcasting related failures, and this remote-data issue left unaddressed.

Just a small note: I suspect this being an astroplan bug also because we don't see similar issues with other packages, etc astroquery has loads of remote-data tests, etc.
I guess the core issue is that astroplan needs the IERS tables, and probably it tries to download it even when it should not access any remote-data.

However, it seems that it did not happen with astropy 1.2.1, but only with astropy 1.3 -- the difference between the two builds is just the new astropy.

In our latest Astropy CI tests, I also see that an access to some google APIs fails:

_______________________________ test_of_address ________________________________

    @remote_data
    def test_of_address():
        # [...]
>       loc = EarthLocation.of_address("New York, NY")

/usr/lib/python2.7/dist-packages/astropy/coordinates/tests/test_earth.py:280: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/dist-packages/astropy/coordinates/earth.py:341: in of_address
    geo_result = _get_json_result(geo_url, err_str=err_str)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

url = 'https://maps.googleapis.com/maps/api/geocode/json?address=New+York%2C+NY'
err_str = "Unable to retrieve coordinates for address 'New York, NY'; {msg}"

    def _get_json_result(url, err_str):
        try: [...]
        except urllib.error.URLError as e:
        [...]
>               raise NameResolveError(err_str.format(msg=e.reason))
E               NameResolveError: Unable to retrieve coordinates for address 'New York, NY'; [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)

/usr/lib/python2.7/dist-packages/astropy/coordinates/earth.py:65: NameResolveError

which suggests that also astropy has some problems with disabling remote access...
This happens only in the CI test, not in the built time test.

I just now updated to Astropy 1.3 in Macports, and now I'm getting this deprecation warning (on any Python version):

$ python2.7 -c 'import astropy; astropy.test()'
WARNING: AstropyDeprecationWarning: The remote_data option should be one of none/astropy/any. For backward-compatibility, assuming 'any', but you should change the option to be one of the supported ones to avoid issues in future. [astropy.tests.runner]

Looks like the default needs to be updated?

This issue also affects pyregion, and the option --remote-data=none does not change anything, as far as I can tell.

@ViviCoder

@ViviCoder - There was a similar issue in astroplan that is fixed with astropy/astroplan#281, maybe something similar goes on in pyregion? (I don't have time today to check it)

@olebole

@olebole - looking into this now. How exactly do you run the tests for the Debian CI? (e.g. python setup.py, or astropy.test(), etc.)

@bsipocz

Ah I see @bsipocz figured it out already :) (though not sure pyregion has the issue, will check)

@cdeil

Ah but @cdeil's issue is still a problem, investigating

I patched the Debian version with #5689, but the remote_data option seems still to be somehow ignored from affiliated packages in the doctests. This gives the warning (Debian Bug #851437, for astroplan):

IOError: An attempt was made to connect to the internet by a test 
that was not marked `remote_data`. The requested host was: data.astropy.org

I am not sure whether this belongs here or better to #5670, however.
Full build log of astroplan here

@olebole

@olebole , any chance #5688 also fixed this problem here? Just wondering since I am pretty sure #5688 can run doctest in pytest 3.0.5 now (see its latest Travis CI logs).

@plim

@plim I can't say that currently, since this involves the test of other affiliated packages. I will go through this tomorrow.

@ViviCoder

I just run the astroplan tests with the patches astropy (version 1.3 with patches from #5688). Unfortunately, it does not fix the remote_data issue on Python 2.7:

collected 92 items / 2 errors

==================================== ERRORS ====================================
___ ERROR collecting lib.linux-x86_64-2.7/astroplan/tests/test_scheduling.py ___
astroplan/tests/test_scheduling.py:25: in <module>
    apo = Observer.at_site('apo')
astroplan/observer.py:256: in at_site
    return cls(location=EarthLocation.of_site(site_name), name=name, **kwargs)
/usr/lib/python2.7/dist-packages/astropy/coordinates/earth.py:283: in of_site
    registry = cls._get_site_registry()
/usr/lib/python2.7/dist-packages/astropy/coordinates/earth.py:423: in _get_site_registry
    reg = get_downloaded_sites()
/usr/lib/python2.7/dist-packages/astropy/coordinates/sites.py:130: in get_downloaded_sites
    jsondb = json.loads(get_file_contents(jsonurl, show_progress=False))
/usr/lib/python2.7/dist-packages/astropy/utils/data.py:372: in get_file_contents
    with get_readable_fileobj(*args, **kwargs) as f:
/usr/lib/python2.7/contextlib.py:17: in __enter__
    return self.gen.next()
/usr/lib/python2.7/dist-packages/astropy/utils/data.py:197: in get_readable_fileobj
    timeout=remote_timeout)
/usr/lib/python2.7/dist-packages/astropy/utils/data.py:1030: in download_file
    remote_url, timeout=timeout)) as remote:
/usr/lib/python2.7/urllib2.py:154: in urlopen
    return opener.open(url, data, timeout)
[...]
/usr/lib/python2.7/httplib.py:821: in connect
    self.timeout, self.source_address)
/usr/lib/python2.7/dist-packages/astropy/tests/disable_internet.py:75: in new_function
    "requested host was: {0}".format(host))
E   IOError: An attempt was made to connect to the internet by a test that was not marked `remote_data`. The requested host was: data.astropy.org

For Python 3, I get slightly different exceptions:

astroplan/tests/test_scheduling.py:25: in <module>
    apo = Observer.at_site('apo')
astroplan/observer.py:256: in at_site
    return cls(location=EarthLocation.of_site(site_name), name=name, **kwargs)
/usr/lib/python3/dist-packages/astropy/coordinates/earth.py:287: in of_site
    raise UnknownSiteException(e.site, 'EarthLocation.get_site_names', close_names=e.close_names)
E   astropy.coordinates.errors.UnknownSiteException: "Site 'apo' not in database. Use EarthLocation.get_site_names to see available sites."

These look like for Python 3, the remote_data option is somehow acknowledged, but the package failed anyway. It worked however under astropy 1.2.
I am however not very familar with astroplan, maybe @ViviCoder can have a look as well? The new astropy package is in Debian experimental.

@olebole

@olebole - astroplan has a bug of it's own related to remote-data (as well as several others against astropy1.3). Could you try it with the patch in astropy/astroplan#281, that should with the remote data issue.

@bsipocz

@bsipocz - I tried the patch in astropy/astroplan#281, but the remote-data option is still enabled and tests fail, whatever I put in setup.cfg (remote_data = none) or in the command line for the tests (--remote-data=none).
@olebole - The version of astropy in Debian experimental does not help. I get the same results for Python 2 and 3. The tests do not even finish: one test in test_constraints.py takes forever to run.

@ViviCoder

@ViviCoder Similar result here, except that than one test finished after quite a while (30 min or so). However, with a number of errors, like:

_____________________________ test_at_night_basic ______________________________

    def test_at_night_basic():
        subaru = Observer.at_site("Subaru")
        time_ranges = [Time(['2001-02-03 04:05:06', '2001-02-04 04:05:06']),  # 1 day
                       Time(['2007-08-09 10:11:12', '2007-08-09 11:11:12'])]  # 1 hr
        targets = [vega, rigel, polaris]
        for time_range in time_ranges:
            # Calculate constraint using methods on astroplan.Observer:
            observer_is_night = [subaru.is_night(time)
>                                for time in time_grid_from_range(time_range)]

astroplan/tests/test_constraints.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
astroplan/tests/test_constraints.py:39: in <listcomp>
    for time in time_grid_from_range(time_range)]
/usr/lib/python3/dist-packages/astropy/utils/decorators.py:860: in is_night
    func = make_function_with_signature(func, name=name, **wrapped_args)
/usr/lib/python3/dist-packages/astropy/units/decorators.py:130: in wrapper
    return wrapped_function(*func_args, **func_kwargs)
astroplan/observer.py:1601: in is_night
    solar_altitude = self.altaz(time, target=get_sun(time), obswl=obswl).alt
astroplan/observer.py:464: in altaz
    list(map(get_coord, target)))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <SkyCoord (GCRS: obstime=2451943.6702083335, obsgeoloc=( 0.,  0.,  0.) m, obsgeovel=( 0.,  0.,  0.) m / s): (ra, dec, distance) in (deg, deg, AU)
    ( 316.84673779, -16.51666611,  0.98567647)>

    def __iter__(self):
        if self.isscalar:
            raise TypeError('scalar {0!r} object is not iterable.'
>                           .format(self.__class__.__name__))
E           TypeError: scalar 'SkyCoord' object is not iterable.

/usr/lib/python3/dist-packages/astropy/utils/misc.py:952: TypeError

Summary line is

==== 34 failed, 69 passed, 1 skipped, 1 pytest-warnings in 1788.51 seconds =====

:-( No idea what to do here -- it is a degression, but I don't know whether this is astroplan or astropy.

@olebole

@olebole - I remember fixing that one, probably this is the relevant PR astropy/astroplan#274. In summary the current release (or even master) of astroplan is not compatible with astropy 1.3.

@bsipocz

@bsipocz I didn't realize that astropy 1.3 has so many incompatible changes. Is there any chance to get this fixed in the next days? Then we could release Debian Stretch including astroplan, which would be quite nice.

Is this still relevant here?

I think any issues with remote-data have been fixed in master - there may still be some issues in astroplan, but these should be discussed in the astroplan issue tracker.

If anyone continues to have issues with packages other than astroplan, and you suspect a bug in astropy itself, feel free to re-open or open a new issue.

Navigation Menu

Repository navigation

Unable to disable remote_data test option #5651

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Unable to disable remote_data test option #5651

Description

Activity

astrofrog commented on Dec 28, 2016

astrofrog commented on Dec 28, 2016

bsipocz commented on Dec 29, 2016

olebole commented on Dec 29, 2016

bsipocz commented on Dec 29, 2016

bsipocz commented on Dec 29, 2016

olebole commented on Dec 29, 2016

olebole commented on Dec 29, 2016

cdeil commented on Jan 6, 2017

ViviCoder commented on Jan 12, 2017

bsipocz commented on Jan 12, 2017

astrofrog commented on Jan 12, 2017

astrofrog commented on Jan 12, 2017

astrofrog commented on Jan 12, 2017

olebole commented on Jan 15, 2017

pllim commented on Jan 17, 2017

olebole commented on Jan 19, 2017

olebole commented on Jan 20, 2017

bsipocz commented on Jan 20, 2017

ViviCoder commented on Jan 20, 2017

olebole commented on Jan 20, 2017

bsipocz commented on Jan 20, 2017

olebole commented on Jan 21, 2017

pllim commented on May 12, 2017

astrofrog commented on May 12, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions