Repository navigation
The output when there's a Skia Gold error is inscrutable #47898
Description
One error I got said:
The following NonZeroExitCode was thrown while running async test code:
Exit code 1: Skia Gold tryjobAdd failure.
stdout: Given image with hash 382fa8b67631fa1fb565a14393c90dbb for test
cupertino.date_picker_test.datetime.initial
Expectation for test: 6799e6f0fc81d60d1fe72afab17eddd5 (positive)
Expectation for test: 68b87a3d1557212339b24f7023c42b19 (positive)
Expectation for test: 6ee9fed2051d189bfaebcf461ae0e82c (positive)
Expectation for test: 87599f3dec5b56dc110f1b63dc747182 (positive)
Expectation for test: 0eeb47eef65dca3d8a64fb6e65f2ca4d (positive)
Expectation for test: 47fc581696de17734c8d65d0d58336f8 (positive)
Untriaged or negative image:
https://flutter-gold.skia.org/detail?test=cupertino.date_picker_test.datetime.initial&digest=382fa8b67631fa1fb565a14393c90dbb&issue=47875
stderr: ERROR: logging before flag.Parse: I1227 11:30:32.855240 23375 auth.go:364] auth.go:364
Read from file /tmp/flutter
sdk/bin/cache/pkg/skia_goldens7583/packages/flutter/test/cupertino/serviceAccount.json
ERROR: logging before flag.Parse: I1227 11:30:32.967221 23375 auth.go:364] auth.go:364 Read from
file /tmp/flutter
sdk/bin/cache/pkg/skia_goldens7583/packages/flutter/test/cupertino/serviceAccount.json
Test: cupertino.date_picker_test.datetime.initial FAIL
Another said:
The following NonZeroExitCode was thrown while running async test code:
Exit code 1: Skia Gold tryjobAdd failure.
stdout: Given image with hash 24d0b1b915250fc26be3aca836f44bc5 for test
cupertino.date_picker_test.datetime.drag
Expectation for test: 7d04fc1ef547a8e092495dab4294b4cd (positive)
Expectation for test: aa748136c70cefdda646df5be0ae189d (positive)
Expectation for test: c3312ad1479f50caf3754b42a42740c5 (positive)
Expectation for test: f0a6cea41931cb63d141a7eb67b7f8ca (positive)
Expectation for test: 007c679a276885ad84777935ac0b63ce (positive)
Expectation for test: 0b9795b218a8e367b552dcea55c8d589 (positive)
Untriaged or negative image:
https://flutter-gold.skia.org/detail?test=cupertino.date_picker_test.datetime.drag&digest=24d0b1b915250fc26be3aca836f44bc5&issue=47875
stderr: ERROR: logging before flag.Parse: I1227 11:30:34.748231 23505 auth.go:364] auth.go:364
Read from file /tmp/flutter
sdk/bin/cache/pkg/skia_goldens7583/packages/flutter/test/cupertino/serviceAccount.json
ERROR: logging before flag.Parse: I1227 11:30:34.857654 23505 auth.go:364] auth.go:364 Read from
file /tmp/flutter
sdk/bin/cache/pkg/skia_goldens7583/packages/flutter/test/cupertino/serviceAccount.json
Test: cupertino.date_picker_test.datetime.drag FAIL
Yet another said:
The following NonZeroExitCode was thrown while running async test code:
Exit code 1: Skia Gold tryjobAdd failure.
stdout: Given image with hash 5ddf1e33c70bfc68c2353f229a3f7871 for test
cupertino.timer_picker_test.datetime.drag
Expectation for test: 511a4cea88085911d1aa5c4db9686998 (positive)
Expectation for test: 981d20adfaeebfcd139ae761336a3202 (positive)
Expectation for test: 9d01038ff6dc288eeb14b084b4649421 (positive)
Untriaged or negative image:
https://flutter-gold.skia.org/detail?test=cupertino.timer_picker_test.datetime.drag&digest=5ddf1e33c70bfc68c2353f229a3f7871&issue=47875
stderr: ERROR: logging before flag.Parse: I1227 11:30:36.974149 23674 auth.go:364] auth.go:364
Read from file /tmp/flutter
sdk/bin/cache/pkg/skia_goldens7583/packages/flutter/test/cupertino/serviceAccount.json
ERROR: logging before flag.Parse: I1227 11:30:37.118697 23674 auth.go:364] auth.go:364 Read from
file /tmp/flutter
sdk/bin/cache/pkg/skia_goldens7583/packages/flutter/test/cupertino/serviceAccount.json
Test: cupertino.timer_picker_test.datetime.drag FAIL
These are unactionable and intimidating error messages. We should replace them with something that is clear, actionable, and friendly.
cc @Piinks
Activity
kjlubick commentedon Jan 2, 2020
Of note, there's a --failure-file which you can provide and then read from to have just the list of links which could be easier to parse than the raw stdout
Piinks commentedon Jan 2, 2020
Thanks! That's super helpful!
I want to still be able to provide the logging information when the test fails for reasons beyond improper drawing, like when we were trying to diagnose the upload issues in November. Any suggestions there? I don't want the output to be intimidating, but I want to make sure it's there when we need it.
Hixie commentedon Jan 3, 2020
The message right now just has lot of incomplete sentences and it's not at all clear to someone who hasn't dealt with Skia Gold what any of it means. Consider:
Does the number mean anything? We already know it's not zero, since there was an error. Does reporting the number help? If so, what does the number mean?
What is "tryjobAdd" and why does it matter? What are the other possibilities? What does this mean to the person who wrote the failing PR?
Why do we suddenly mention "stdout"?
What is "cupertino.date_picker_test.datetime.initial"? The string given in the test is "date_picker_test.datetime.initial.png", why is the string here different? Is that meaningful?
What are these hash codes? What does "positive" mean? What am I supposed to do with this information?
What does "untriaged or negative" mean here? (Again, consider this from the perspective of someone who has never contributed before.)
I have no idea what this means. How do I make use of it?
...etc
A better message might be:
github-actions commentedon Aug 13, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.