MNT: Bump minversion of Python to 3.7 (also Scipy, PyYAML, Matplotlib) - #10900

Would be good to clear APE18 for 4.2 anyway tbh, it seems pretty uncontroversial.

Re: urgency -- No clear urgency except that #10888 really wants in for 4.2. 🤷

I think we can't accept this until astropy/astropy-APEs#63 is accepted (or not do it if it isn't). But I for one have no objection towards saying this (and #10888 in consequence) are something that in principle should have been ready but are awaiting the APE and therefore can go in a post-freeze backport (do @taldcroft and/or @astrofrog agree?)

(And I agree the APE doesn't seem controversial, I just don't want to have to deal with that as a blocker to feature freeze)

I suggest to follow the pre-APE procedure here, there is no benefit of holding this up. There is a wide support for this already amongst the maintainers and no objections so far, which is along with the previous approach.

IMO getting this on the branch as soon as there is a branch is useful, and nothing like this should be hold back for backport to ensure downstream is testing this as much as possible (I think it's already quite late in the dev cycle to get these kinds of cleanups in, ideally these should happen in the middle of the cycle, just as the drop for numpy 1.16 happened earlier (which also didn't required the APE to be accepted)).

There is certainly precedent for what @bsipocz suggests - APE 9 on timeseries! (not yet accepted). It's true that there haven't been any objections so it looks pretty certain it will be accepted and we might just refine some of the wording at most.

Thanks for the support! Still need @taldcroft or @mhvk to comment on the table changes before merge. If I don't hear anything back in 12 hours or so though, I might just drop the table patch so we can proceed.

@astrofrog , how do I fix that image-tests-mpl222 failure that is trying to use astropy/image-tests-py36-mpl222:1.10 Docker image? (And also astropy/image-tests-py36-mpl212)

some weird travis failures, I wonder maybe those got fixed since this was rebased? Would you mind a new rebase?

Rebased, but still not sure how to fix the problem with CircleCI pulling down Python 3.6 Docker images.

We might have to also bump minversion of scipy here (currently 0.18):

  Building wheel for scipy (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /home/travis/build/astropy/astropy/.tox/py37-test-oldestdeps/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-06_2g6hh/scipy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-06_2g6hh/scipy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-j49f3wtu
       cwd: /tmp/pip-install-06_2g6hh/scipy/
  Complete output (9 lines):
  /tmp/pip-install-06_2g6hh/scipy/setup.py:115: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-06_2g6hh/scipy/setup.py", line 415, in <module>
      setup_package()
    File "/tmp/pip-install-06_2g6hh/scipy/setup.py", line 395, in setup_package
      from numpy.distutils.core import setup
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for scipy

OK, so having the minimum python version bumped also means that we need to bump the version of other dependencies, otherwise they are in contradiction, no being compatible with python 3.7.

This means:

• pyyaml has to be bumped to 1.13
• scipy has to be bumped to 1.1
• matplotlib most probably has to be 3.0, I updated it without it failing explicitely on an install

If this is all right with you, then I'll come back and clean up the commits/etc here.

Thanks for taking this over, @bsipocz ! It's all fine by me, as long as others are okay with this. I also see a parallel discussion over at the APE proposal, but I haven't seen any objections.

If you don't get to the clean-ups, I will on Monday. Just a note on what is to be done:

• Update change log. (Might need a rebase before this.)
• Recursive grep for the package + old minversion combo to smoke out other files to update.
• Also grep to see if any PACKAGE_LT_X_Y need updating.
• Squash? Not sure if this is necessary, as I think the commits are atomic enough thus far, so it is a matter of personal taste.
• Let the full CI run to completion, just in case.

Bumping the minimum version of those dependencies seems sensible to me!

OK, so having the minimum python version bumped also means that we need to bump the version of other dependencies, otherwise they are in contradiction, no being compatible with python 3.7.

This means:

• pyyaml has to be bumped to 1.13
• scipy has to be bumped to 1.1
• matplotlib most probably has to be 3.0, I updated it without it failing explicitely on an install

I am a bit confused by the logic, shouldn't the minimum version be determined by the maximum Python version we are running on (e.g. if we officially support python 3.9, the minimum package version to support that)?
Anyway matplotlib 3.0[.3] is the last to support py35, py36 supported everything up to 3.3.2; conversely 2.2 still builds on py38 and has wheels at least up to py37.

This is finally all green, so I'm merging for the sake of being able to proceed with branching. The "what" is very much agreed to, all over, if there is any issues in the "how", please open follow-up PRs that will/can possibly be backported.

Thanks, @bsipocz !

And thank you, @pllim!

I think for the doc build we need matplotlib>=3.1.0. Otherwise, with sphinx 3.2 I get:

sphinx-build -b html -d _build/doctrees   . _build/html
Running Sphinx v3.2.1

Exception occurred:
  File "matplotlib/sphinxext/plot_directive.py", line 262, in setup
    app.add_directive('plot', plot_directive, True, (0, 2, False), **options)
TypeError: add_directive() got an unexpected keyword argument 'alt'
The full traceback has been saved in /tmp/sphinx-err-x34uejnl.log, if you want to report the issue to the developers.

This is due to use of deprecated interfaces in matplotlib.sphinxext that were only reworked in matplotlib 3.1 (matplotlib/matplotlib#13138). Should I open a new PR?

Should I open a new PR?

Probably, but I don't know what is the policy to have minversion for matplotlib to build docs higher than official required version. Seems a bit excessive to bump required version for the whole package to 3.1 just for the docs. What do you think, people?

@embray , how about a PR to change one line here to start a conversation? Thanks!

It would resolve #10954 as well, though that is only about an unexpected warning and could also be easily fixed without the version bump.

How the mpl version didn't cause an issue for the docs before?

RTD picks up latest releases from PyPI, I think. (Its log is a bit of a mess, but if you scroll half way down, you can see that it gets the latest releases.) So, we probably missed that problematic combo by luck. We don't have exhaustive matrix for doc builds.

As for #10954 , not sure why it didn't pop up over at CircleCI image tests. (Ah, okay, right, we don't test against Python 3.9 yet, never mind.)

Agreed that we should not bump versions of optional dependencies just for a doc build. So, would it be fine to just leave it as is? (Or at a line to the matplotlib requirement in the documentation noting that with matplotlib 3.0 the docs cannot be build?)

@embray et al., see #10956