Repository navigation
flutter/packages release sometimes fails due to slow task check-in #137299
Description
This code in the flutter/packages CI release step was added to fix a problem where the following section would run before any of the LUCI checks had checked in (especially when there was an incident causing slow webhooks), causing release to run before any of the tests. It's fixed that problem, but the same basic problem now manifests in a different way: sometimes the new code runs before Linux ci_yaml packages roller has checked in, causing the step to immediately fail.
This is a lot better than publishing plugins that haven't passed any tests, but it's now a flaky failure, which is not ideal since it then requires manual intervention to re-run to open the tree. Ideally what we want is a step that waits until that step both exits and completes, rather than failing if it's not there yet. I don't see a way to do that with the wait action we're using though, so we may well need to build our own replacement to get that behavior.
Activity
flutter-triage-bot commentedon Mar 5, 2024
This issue is marked P1 but has had no recent status updates.
The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.
Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!
stuartmorgan-g commentedon Mar 5, 2024
stuartmorgan-g commentedon Mar 11, 2024
7 remaining items
keyonghan commentedon Mar 14, 2024
How about adding a sleep step to the action: https://cardinalby.github.io/blog/post/github-actions/implementing-deferred-steps? This is not a real fix, but could help mitigate the issue when github webhook regression happens (before we have a long term fix).
stuartmorgan-g commentedon Mar 15, 2024
Yes, we could do that. Do we have historical stats on how long checkin generally takes when it has these slowdown regressions? I don't want to make it sleep for too long, because while we wouldn't notice for standard triggers (since we have to wait for other tests to run anyway), it could make re-runs annoying if we sleep for too long.
keyonghan commentedon Mar 18, 2024
That really depends on how long GitHub issue gets resolved. Whenever the issue is mitigated, the task status should be posted to GitHub in seconds. We can start with a small number like 1 min to mitigate the too-fast github action. If working well, we can readjust the number further.
ditman commentedon Mar 26, 2024
I just saw this happen again, I'll try adding a 60s delay to mitigate as suggested by @keyonghan!
[ci] Adds sleep 60s to release action. (#6405)
ditman commentedon Mar 28, 2024
I landed the timeout, but should we keep this open to track the fine-tuning of the timeout, or landing a more proper fix?
ditman commentedon Mar 28, 2024
(The extra minute gets completely lost in the CI run; it's barely noticeable)
stuartmorgan-g commentedon Apr 1, 2024
Let's leave it closed for now, and we can re-open if we have more issues later.
github-actions commentedon Apr 15, 2024
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.[ci] Adds sleep 60s to release action. (flutter#6405)
[ci] Adds sleep 60s to release action. (flutter#6405)