Proposal for the baseline images problem

This is a proposal to help with the difficulty in adding/modifying tests which require a baseline image. As a brief reminder, baseline images are problematic because

they cause the repo size to grow rather quickly (matplotlib's .git is now ~400Mb, compared e.g. to numpy's which is 72Mb), even though we've been exceedingly careful in recent years to avoid adding more baseline images (the problems with the merge of Fix mplot3d projection #8896 got me to think about this again).
they force us to pin to a somewhat old version of freetype, because nearly every release of freetype causes tiny rasterization changes that would entail regenerating all baseline images (and thus cause even more repo size growth).

The idea is to actually not store baseline images in the repo at all.

When generating a sdist or wheels for a new release, we do generate baseline images (from the tests) and store them in the released sdist or wheel. The validity of the baseline images is guaranteed by what comes below.
During normal development, the baseline images should not change. If someone wants to run the test suite from a git checkout, matplotlib would first generate a set of "known" baseline images either by grabbing the last release, or just installing the last tagged release into a temporary venv and running the test suite there (of course, they would be cached into ~/.cache/matplotlib so this is only done once -- this caching also needs to be done so that CI time doesn't double...).
If a baseline image needs to change (we should still try to keep this as rare as possible), the commit message needs to include a special string e.g. [baseline-change]: path/to/image. The process described in 2) would also consult the git log to know if there's any images that need to be gen'd in such a way. Likewise, if a baseline image needs to be added for a new test, the commit message needs to be tagged accordingly. Possibly, we could also require that the filename also be changed in that case so that even if you're not in a git repo, you at least don't end up using the wrong baseline image, and instead just get a skip.

Note that this means that if you're using a non-git-repo, non-sdist/wheel, you cannot run the full test suite anymore, because you at least need to know what was the last release, and if any image changed since then, you'd like to know when (however, if you want to write a patch and check that it's good, this still works because you can compare the baselines with what was present before your patch). OTOH distro/conda packagers, which should be working from sdists, would still be able to run the test suite fine, because the sdists/wheels will include baseline images per 1).

There's a bit of machinery involved in making this work, but I don't think it's overly hard to implement. Thoughts?

At least as far as getting new baseline images with other FreeType, you can grab them from https://github.com/QuLogic/mpl-images.

I have not yet completely thought about the rest of the proposal, but I suspect if caching is not sufficient, it might be possible to do the remaining work in a GitHub Action.

This is the best proposal for handling this problem yet!

In summary:

on release, generate the 'blessed' version of the test images for that tag (these could even be thrown onto github as artifacts with the release)
on a dev machine, generate and/or download and cache the test images running against the last released version and/or current master
some logic to flag what images have been changed from last version to the current branch tip and do something about them
for CI we can rely on CI cache or stand up a store on github/DO/AWS/... managing that cache can be done via github actions

As part of this we should publish mpl-test-data as a wheel separately.

If we have the tools for CI to automatically grab the latest baseline images based on the current commit, might as well have that be the recommended way for devs to get the test images as well (with the generate-in-place being an option if you have local CPU/time, but poor internet)?

One thing we might lose with this is that we currently use the exactly same version of the images on all OS, all python versions, all machines etc. I think we can maintain this, but it is a bit trickier to be 100% sure that we have it right (as opposed to "git says it is these bits")

A thing that this would give us is a nice way to generate forward and backwards compatibility reports (run the next tag with the baseline images from the previous tag, list what fails).

This would also get us around the problem that some of our latex baselines are different with newer versions of latex.

Are we re-inventing git LFS?

Are we re-inventing git LFS?

If you add the machinery to grab the latest test images, perhaps (but then you also need to decide where to host them and may run into bandwidth issues -- similar to why we haven't adopted git-lfs on github); but the intent was really that CI uses CI cache to roll over one run's generated test images as baselines for the next run, and that devs do the same locally. Which is really the fundamental idea here: instead of comparing against a fixed baseline, compare a commit to the next (with some additional shortcuts based on reference points at releases, etc.).

I'm not fully following this, but it sounds painful for contributors. If we are going to be painful to contributors, it seems to me that just having a separate images repository on github would be simpler. Someone would have to tag it to releases, so maybe that makes the release work harder? But then folks could just do shallow fetches if they do not need the full history of the images. But perhaps I'm missing something, or maybe thats more complicated.

This should transparent for contributors (except that their first test run is going to be a bit slower if some baseline images changed since the last release, given that mpl will automatically rebuild itself at the correct commit to generate the corresponding baseline).
Likewise, the release work should be automated as much as possible.

@anntzer

I am interested in this issue . This is the kind of problem i would like to work on my summer time . As i Know testing is an important part of a development process . From what i understand we can actually store the binary of baseline images and their format in the release . I need to work a lot more than that. @anntzer please guide me
Thanks

See some suggestions at https://discourse.matplotlib.org/t/gsoc-due-march-31-18-00utc/20975/10?u=anntzer.lee as well.

@anntzer

See some suggestions at https://discourse.matplotlib.org/t/gsoc-due-march-31-18-00utc/20975/10?u=anntzer.lee as well.

yeah i have looked at it . i have on python testing in one of internships but i have to get comfortable with the codebase and then i can do a pull request .
I need some help in my proposal . can you please help @anntzer

See some suggestions at https://discourse.matplotlib.org/t/gsoc-due-march-31-18-00utc/20975/10?u=anntzer.lee as well.

Okay, I am looking towards them. Thanks

@greglucas

@greglucas Do you know what that looks like?

I think our somewhat draconian tolerances arose because a missing glyph could pass looser tolerances. Would we have the same problem here? If we continually update FreeType and thousands of images are changing, no one is likely to go through and check them all?

@greglucas

@greglucas Do you know what that looks like?

No. I have not played around with these things at all which is why I looped Bill in here because I believe he has :) Maybe @rcomer knows if they are using it on Iris too?

Maybe there is a way for us to figure out what the tolerance was for an entirely missing glyph and set the threshold just below that. That would theoretically give us multiple minor version updates of freetype with minor kerning adjustments and only once enough of the kerning changes all add up does it go over that threshold.

I am probably not the one to explicitly ask about the tolerances as I am more OK with looser tolerances. One idea is that we could try to make a distinction between categories of tests, freetype-specific / font-specific where I do care about tolerances, and plots where I am less concerned about minor glyph kerning and care more about the placement of a scatter marker in the right spot.

One thing that isn't clear to me is whether I need internet access to do these things?

If we store the test images in a repository, you could either point the location of the baseline images to be that remote repository, or you could clone it (before starting your internet free time) and point to the local checkout as the location of the baseline images.

That would theoretically give us multiple minor version updates of freetype

The idea of using hashes is that updating the images is (almost) zero cost in terms of effort and main repository size, so I would not worry too much about having to update all the images for different versions of freetype to start with. I think to keep things simple here we should try out storing the images in a separate repository and using hashes - if that works, we could then explore whether there's still a need to reduce the frequency that we're updating test images using a different comparison metric (presumably to keep the size of the test image respository low?)

Ahh I have just understood the motivation for using a perceptual difference instead of a hash - it would allow us to replicate the current behaviour of using tolerances in the tests, where using a cryptographic hash wouldn't allow us to do that.

I think our somewhat draconian tolerances arose because a missing glyph could pass looser tolerances.

Yes, we were using \cdot (one dot in the center) instead of \ddots (three dots diagonal) or something similar. Because the contrast is black/white for text even minor shifts give huge RMS and the number of pixels "wrong" for dropping some dots is less than from shifting a lowercase l by 2 pixels. I do not think there exists a tolerance that will reliably catch that sort of error but will ignore the sort of version-to-version drift we see in freetype.

https://github.com/tacaswell/mpl-imageset-demo and #25734 are working but lack documentation and sorting out how to make the required caching work on CI.

I also have general concerns about updating images too often and view it as a high cost (even if they are in a second repo) or high risk. We have a lot of test images and if we are on the regular regenerating most of them, it will either require a lot of manual work to validate that the "small" changes are in fact small-from-a-communication point of view or we are going to be lax and end up committing very wrong baseline images again.

An additional concern with hashes is that we put the version + date in the files as metadata so we can not hash the files. Additionally for the vector backends we want the tests to be insensitive to things like the order of things in the file, local identifiers, and other non-meaningful changes to the files (e.g. dropping the number of digits we write out for floats). Currently the way we do this is to checking the svg/pdf/ps and then at test time use the same viewer to rasterize the files and compare the pixels of the rasterization. If we were to track the hash of that rasterization then we would add a dependency on the version of the viewer we used (and any platform dependent variation they have).

One idea is that we could try to make a distinction between categories of tests, freetype-specific / font-specific where I do care about tolerances, and plots where I am less concerned about minor glyph kerning and care more about the placement of a scatter marker in the right spot.

If we are going to go through and re-generate a lot of test images it should be coupled with an audit of if that image actually needs text in it and switching them to the new defaults.

An additional concern with hashes is that we put the version + date in the files as metadata so we can not hash the files.

That's not a fundamental issue. Worst-case, we hash the RGB image values.

@pelson

Iris’s decision to use perceptual image hashes was before I got involved, so I’m not really up on the pros and cons. I guess @pelson would have been involved back then so might have an opinion whether they’d be appropriate here.

For any kind of hash, we can use them as a pre-test for an image test if the hash is reasonably stable. If the hash is "known-good", we are done, if not we have to fall back to the image test (possibly downloading the reference). If the image test is successful, we add the hash to the "known-good" hashes.

This works as long as long as there is a limited number of image variants. It wouldn't work if there's a small randomness in every generated image.

@greglucas

@greglucas Thanks for the nudge and looping me in ...

I'm still very keen to close matplotlib/pytest-mpl#146. The original pull-request matplotlib/pytest-mpl#150 unfortunately stalled as I needed some pytest-mpl core dev support to help me cross the line. Also my proposal was a major breaking change in behaviour and some big decisions still required to be agreed and ironed out. Anyways, I kinda lost momentum, life got in the way (as it does), and I closed the PR to keep things honest. That's all totally on me btw.

In terms of https://github.com/SciTools/iris, it was my decision to implement and introduce perceptual image hashing, and thankfully it's really worked out for us, in that it's given us the stability and control that we required. We've been happily using it for many years now to perform our mpl image testing.

Anyways, I'd love to blow the dust off matplotlib/pytest-mpl#150 and resurrect it.

Perhaps what would give me more confidence and conviction to actually follow through and deliver would be if I could work with someone from the matplotlib dev community? Someone who could be a technical point of contact so that we could kick the tyres of perceptual image hashing to determine whether it addresses some/all/none of your issues. That would really give me the confidence to push the breaking change to pytest-mpl and more importantly it might help mitigate risk on your side, give you guys clarity on the impact and implications, and ultimately help decide whether this might be useful to you or not.

My proposal would be to work with @rcomer, who's also at the Met Office, and logistically that would make a lot of sense. I know this suggestion has come pretty much from left field, so have a think and let me know what you guys decide. In the meantime I'm going to tentatively see whether it's feasible to secure the time and space to do this work during my 9-to-5; I know of several groups at the Met Office that would like to see this happen, so that gives me some leverage to hard-sell the idea.

Anyways, just a thought ...

@timhoffm

I also have general concerns about updating images too often and view it as a high cost (even if they are in a second repo) or high risk.

Yes, I suppose moving to another repo just lowers the cost, but doesn't make it zero. Thinking about this a bit more, I guess typically updates fall into either 1) updating or adding a single or a handful of images 2) wholesale updating a lot of images (e.g., freetype updates). The nice thing about hashes is it would automatically download only the images you need from the secondary repo when tests fail, which for most contributors results in a big saving in MB downloaded when they're just updating a few images.

An additional concern with hashes is that we put the version + date in the files as metadata so we can not hash the files.

We can easily strip metadata - indeed, pytest-mpl already does this. Or as @timhoffm suggests, hash the RGB values instead of the whole file.

for the vector backends we want the tests to be insensitive to things like the order of things in the file, local identifiers, and other non-meaningful changes to the files (e.g. dropping the number of digits we write out for floats). Currently the way we do this is to checking the svg/pdf/ps and then at test time use the same viewer to rasterize the files and compare the pixels of the rasterization. If we were to track the hash of that rasterization then we would add a dependency on the version of the viewer we used (and any platform dependent variation they have).

Is there not a dependency on the version of the viewer used anyway to do the rasterization?

I tried to use pytest-mpl to convert one test to hash-based at #29823, and it didn't go great - the hashes on different platforms were different compared to the one I generated locally. Some of them matched, but some didn't - ironically, I'm on macOS, and the hash I generated only matched some Linux runs and no macOS runs 😆 I will pick away and try and work out why there are differences.

Anyways, I'd love to blow the dust off matplotlib/pytest-mpl#150 and resurrect it. Perhaps what would give me more confidence and conviction to actually follow through and deliver would be if I could work with someone from the matplotlib dev community?

@bjlittle I'd be super happy to 'pair' on this and resurrect it! I think regardless of the potential use case for our figure tests here, it's a neat little feature that I'd support getting into pytest-mpl anyway. Let me know what the best way forward is for you - perhaps re-opening a PR, and we can discuss/iterate from there?

Is there not a dependency on the version of the viewer used anyway to do the rasterization?

We run the rasterization of both the baseline and the machine running the test with (presumably) the same version of the viewer so variation between how the veiwers render them should was out.

@dstansby

@dstansby FYI matplotlib/pytest-mpl#150 has been superseeded by matplotlib/pytest-mpl#259

@timhoffm

@timhoffm no review is provided yet; does this mean this is in the scope of mpl?

I played with Claude Sonnet 4.5 on this problem, as I admit I found it pretty hard to follow the ins and outs without more guidance.

It seems possible to do a lot of things I'd not have thought of.. Overall, the basic (AI-generated) summary is:

Hash-Based Baseline Image Proposal Summary:

Core Approach: Replace storing 2,330 baseline images (40MB + 507MB Git history) with a lightweight baseline_hashes.json file (~200KB) containing perceptual hashes for each test's expected output. Actual baseline images are stored separately (GitHub release artifacts or dedicated repo) and downloaded on-demand only during test failures for visual inspection. This prevents repository bloat while maintaining full test coverage—if hash comparison passes, the test passes without downloading any images; only mismatches trigger image downloads for debugging.

Platform Handling & Workflow: Perceptual hashes support configurable Hamming distance tolerance (default: 5 bits) to handle minor platform rendering differences automatically (antialiasing, slight font variations). For legitimate platform-specific rendering (macOS vs Windows), the JSON stores multiple valid hash variants per test. Contributors run pytest --accept-new-baseline locally to update the hash file; CI automatically uploads actual images to storage, verifies hash matches, and posts images for reviewer inspection. This eliminates manual image uploads, requires only single-PR contributions, and maintains visual review capability while drastically reducing repository size.

This all sounds pretty reasonable to me. However, it definitely requires pytest and CI infrastructure to make work. I particularly like the idea of the CI posting image diffs to Github for visual acceptance. This still means that there is image storage somewhere, but the user never has to download them all going forward, only the ones for tests that they are running.

I think this can be implemented on an incremental basis, in that we could start the hash file, and the "hashed_images" repository while still allowing the old method, and convert old files as needed.

Details at at https://gist.github.com/jklymak/cb5ace18dd4d9b2359ec5f9aca1b2fda

Navigation Menu

Repository navigation

Proposal for the baseline images problem #16447

72 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal for the baseline images problem #16447

Description

Activity

QuLogic commented on Feb 9, 2020

tacaswell commented on Feb 9, 2020

anntzer commented on Feb 9, 2020

jklymak commented on Feb 10, 2020

anntzer commented on Feb 10, 2020

SidharthBansal commented on Mar 31, 2020

SaraLatif99 commented on Mar 31, 2020

anntzer commented on Mar 31, 2020

SidharthBansal commented on Mar 31, 2020

SidharthBansal commented on Mar 31, 2020

laksh9950 commented on Mar 31, 2020

anntzer commented on Mar 31, 2020

laksh9950 commented on Mar 31, 2020

SidharthBansal commented on Mar 31, 2020

72 remaining items

jklymak commented on Mar 28, 2025

greglucas commented on Mar 28, 2025

dstansby commented on Mar 28, 2025

dstansby commented on Mar 28, 2025

tacaswell commented on Mar 28, 2025

tacaswell commented on Mar 28, 2025

timhoffm commented on Mar 28, 2025

rcomer commented on Mar 28, 2025

timhoffm commented on Mar 28, 2025

bjlittle commented on Mar 28, 2025

dstansby commented on Mar 31, 2025

tacaswell commented on Apr 1, 2025

timhoffm commented on Feb 25, 2026

cvanelteren commented on Mar 7, 2026

jklymak commented on Mar 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions