Repository navigation
Consider moving away from docker based figure test builds #10483
Description
Activity
Why is this not a PR? 😄
I would love to be able to get rid of the astropy-docker repo, one thing less to maintain
Reacted by P. L. LimWhy is this not a PR? smile
Because normally when I suggest 🔥 on some of the infra it ends up in a long discussion 🤣
Reacted by Simon Conseil, Thomas Robitaille and Larry BradleySounds good to me, too!
Can we close this now? We could still set up the hash-based system suggested above (maybe that should be a new issue instead), but at least we're not building our own dockers anymore.
Before we close this, is this statement now true?
The upshot of this is that, as long as you run the figure tests with tox, you should always get the right version of matplotlib and freetype to be able to accurately reproduce the reference figures.
Can we now run the image tests locally in tox? If so, I think we can indeed close it - if not I think it would be good to understand why.
I do not run tox locally. I am not one of the cool kids.
tox does not safeguard architecture dependent changes, right? We still rely on Docker but we use the base image provided by CircleCI. There is some dev docs on CircleCI on how to use their images but it involves accounts and their CLI software, yadda yadda.
Yeah, it doesn't work cross-platform, but any linux system should do it.
I recently spent a day (yes really) looking into the figure test builds for sunpy, specifically surrounding how we pin the version of freetype to minimize differences with the reference images (for sunpy they are hashes of the figures so have no tolerance unlike the currently astropy test setup). This is documented in sunpy/sunpy#4271.
What I learned is that matplotlib actually does a very good job of managing its own freetype versions under normal (pip) operations. When you install matplotlib from wheels a version of freetype is included, which is determined by matplotlib. More interestingly though is that is is also true when compiling matplotlib from a git checkout, and optionally true when compiling matplotlib from a release tarball (controlled by an environment variable).
Therefore, what I think is a good compromise between reliability and simplicity is to rely on these versions of freetype as provided by matplotlib, and not use docker etc on the CI, but to have tox install a known version of matplotlib, which will have an enclosed known version of freetype. To support this approach in sunpy we also check the version of freetype at runtime, and then stop the test run before the figure comparison fails.
The upshot of this is that, as long as you run the figure tests with tox, you should always get the right version of matplotlib and freetype to be able to accurately reproduce the reference figures.
@larrybradley @astrofrog as visualization maintainers.