Skip to content

TST: Fix parallel data download test for v2.x - #7586

Merged
pllim merged 1 commit into
astropy:v2.0.xfrom
pllim:fix-data-parallel-test-backport
Jul 10, 2018
Merged

pllim merged 1 commit into
astropy:v2.0.xfrom
pllim:fix-data-parallel-test-backport

Conversation

@pllim

@pllim pllim commented Jun 21, 2018

Copy link
Copy Markdown
Member

Backport straight to v2.x for #7575

@astropy-bot

astropy-bot Bot commented Jun 21, 2018 •

Copy link
Copy Markdown

Hi there @pllim 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

@pllim pllim mentioned this pull request Jun 21, 2018
1 task done
@pllim

pllim commented Jun 21, 2018

Copy link
Copy Markdown
Member Author

p.s. Ignore 32bit_py2 build. Failure is unrelated.

@pllim

pllim commented Jul 9, 2018

Copy link
Copy Markdown
Member Author

@bsipocz , is this ok to merge?

@pllim
pllim requested a review from bsipocz July 9, 2018 17:43
@bsipocz

bsipocz commented Jul 9, 2018

Copy link
Copy Markdown
Member

@pllim - If it passes the tests then good to go. I need to restart the failing circleCI one though, as that's the only one that actually does parallel testing on python2...

@bsipocz bsipocz added the zzz 💤 merge-when-ci-passes Do not use: We have auto-merge option now. label Jul 9, 2018
@bsipocz

bsipocz commented Jul 9, 2018

Copy link
Copy Markdown
Member

@pllim - I'm afraid the failure is related:

........
=================================== FAILURES ===================================
__________________________ test_download_mirror_cache __________________________
[gw0] linux2 -- Python 2.7.15 /opt/python/cp27-cp27m/bin/python

    @pytest.mark.skipif(not HAS_PATHLIB and sys.platform.startswith('win'),
                        reason="URI incorrect for Windows without pathlib")
    def test_download_mirror_cache():
        from ..data import (_find_pkg_data_path, download_file, get_cached_urls,
                            _open_shelve)
    
        main_path = _find_pkg_data_path(os.path.join('data', 'dataurl'))
        mirror_path = _find_pkg_data_path(os.path.join('data', 'dataurl_mirror'))
    
        if HAS_PATHLIB:
            main_url = pathlib.Path(main_path).as_uri()
            mirror_url = pathlib.Path(mirror_path).as_uri()
        else:  # Falls back to brute force URI
            main_url = 'file://' + main_path
            mirror_url = 'file://' + mirror_path
    
        main_file = main_url + '/index.html'
        mirror_file = mirror_url + '/index.html'
    
        # "Download" files by rerouting URLs to local URIs.
        download_file(main_file, cache=True)
        download_file(mirror_file, cache=True)
    
        # Now test that download_file looks in mirror's cache before download.
        # https://github.com/astropy/astropy/issues/6982
        dldir, urlmapfn = _get_download_cache_locs()
        with _open_shelve(urlmapfn, True) as url2hash:
>           del url2hash[main_file]

astropy/utils/tests/test_data.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/python/cp27-cp27m/lib/python2.7/shelve.py:136: in __delitem__
    del self.dict[key]
/opt/python/cp27-cp27m/lib/python2.7/bsddb/__init__.py:288: in __delitem__
    _DeadlockWrap(wrapF)  # del self.db[key]
/opt/python/cp27-cp27m/lib/python2.7/bsddb/dbutils.py:68: in DeadlockWrap
    return function(*_args, **_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def wrapF():
>       del self.db[key]
E       TypeError: String or Integer object expected for key, unicode found

/opt/python/cp27-cp27m/lib/python2.7/bsddb/__init__.py:287: TypeError
----------------------------- Captured stdout call -----------------------------
 [Done]
 [Done]
=============================== warnings summary ===============================
<undetermined location>
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
  Module already imported so cannot be rewritten: xdist
  Module already imported so cannot be rewritten: xdist
  Module already imported so cannot be rewritten: pytest_remotedata
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.
  [pytest] section in setup.cfg files is deprecated, use [tool:pytest] instead.

-- Docs: http://doc.pytest.org/en/latest/warnings.html
 1 failed, 12562 passed, 383 skipped, 69 xfailed, 1 xpassed, 8 warnings in 106.62 seconds 
Exited with code 1

@pllim

pllim commented Jul 9, 2018 •

Copy link
Copy Markdown
Member Author

😱 OK thx for alerting me to the failure. I'll have a look soon.

Update: I can't reproduce this error in a local PY2 env. I'll just have to guess at the fix and push it here and see... 🤞

@pllim
pllim force-pushed the fix-data-parallel-test-backport branch from cff74b6 to fd6b2a9 Compare July 10, 2018 02:03
@pllim
pllim force-pushed the fix-data-parallel-test-backport branch from fd6b2a9 to 44b0765 Compare July 10, 2018 02:12
@pllim

pllim commented Jul 10, 2018

Copy link
Copy Markdown
Member Author

All tests pass now, so merging. Thanks for the review!

@pllim
pllim merged commit 1da8a98 into astropy:v2.0.x Jul 10, 2018
@pllim
pllim deleted the fix-data-parallel-test-backport branch July 10, 2018 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants