Repository navigation
Add diffTolerance parameter to matchesGolden #76337
Description
Currently, the goldens tests will fail if there is any pixel difference. I'd like to be able to pass a tolerance in via matchesGolden and have it be used by the comparator.
Example:
matchesGoldenFile(goldenPath, diffTolerance: 0.01)
\cc @Piinks
Activity
Piinks commentedon Feb 18, 2021
Related to #71437
Golden file testing in flutter/tests has been causing difficulty with rolls when an image changes.
The two times this has occurred so far, the changes were minuscule diffs most likely caused by skia changes. This proposal will let customers specify a pass-through for tiny diffs like this.
lukepighetti commentedon Mar 9, 2021
We're seeing tests fail due to 0.09% difference from font rendering on macOS vs Linux. Being able to set epsilon on goldens seems obvious and low required effort.
To quote one of our web devs: "the pipeline failed because of 0.09% diff?!?!?!" 😅
I'm going to take a stab at a PR.
lukepighetti commentedon Mar 9, 2021
Just saw this PR,
will stop work on my PR#76545Edit: I was nearly done with mine so I pushed it up in case it's useful to someone, can be found here #77703
Piinks commentedon Mar 10, 2021
We went a slightly different route with this, see #77014
Instead of adding a parameter to
matchesGoldenFile, which requires every individual golden file test to set a tolerance level, we've made it easier to access the diff and factor in a tolerance level in sub classed comparators. I think all that remains here is adding some examples of how to do this to the docs.lukepighetti commentedon Mar 10, 2021
Great! I'll close my PR. Thanks Kate!
Piinks commentedon Mar 15, 2021
I think that we have done all of the work we decided to do to address this issue, so I am going to close it. If there is more to do, let me know and we'll re-open! :)
github-actions commentedon Aug 4, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.alestiago commentedon Jun 17, 2024
We are still missing a clear reference to
goldenFileComparatorin thematchesGoldenFiledocumentation and some examples that show how to change the tolerant difference with the current approach.There are still some users that are finding difficulties on finding documentation to alter the tolerant difference, for example see: VeryGoodOpenSource/very_good_cli#972 or from StackOverflow.