Skip to content

flutter/packages release sometimes fails due to slow task check-in #137299

Description

@stuartmorgan-g

This code in the flutter/packages CI release step was added to fix a problem where the following section would run before any of the LUCI checks had checked in (especially when there was an incident causing slow webhooks), causing release to run before any of the tests. It's fixed that problem, but the same basic problem now manifests in a different way: sometimes the new code runs before Linux ci_yaml packages roller has checked in, causing the step to immediately fail.

This is a lot better than publishing plugins that haven't passed any tests, but it's now a flaky failure, which is not ideal since it then requires manual intervention to re-run to open the tree. Ideally what we want is a step that waits until that step both exits and completes, rather than failing if it's not there yet. I don't see a way to do that with the wait action we're using though, so we may well need to build our own replacement to get that behavior.

Activity

  1. added
    c: contributor-productivityTeam-specific productivity, code health, technical debt.
    packageflutter/packages repository. See also p: labels.
    fyi-infraFor the attention of Infrastructure team
    on Oct 25, 2023
  2. added
    P1High-priority issues at the top of the work list
    on Oct 31, 2023
  3. flutter-triage-bot commented on Mar 5, 2024

    @flutter-triage-bot

    This issue is marked P1 but has had no recent status updates.

    The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.

    Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!

  4. stuartmorgan-g commented on Mar 5, 2024

    @stuartmorgan-g
    Author
  5. stuartmorgan-g commented on Mar 11, 2024

    @stuartmorgan-g
    Author
  6. 7 remaining items

  7. keyonghan commented on Mar 14, 2024

    @keyonghan
    Contributor

    How about adding a sleep step to the action: https://cardinalby.github.io/blog/post/github-actions/implementing-deferred-steps? This is not a real fix, but could help mitigate the issue when github webhook regression happens (before we have a long term fix).

  8. stuartmorgan-g commented on Mar 15, 2024

    @stuartmorgan-g
    ContributorAuthor

    Yes, we could do that. Do we have historical stats on how long checkin generally takes when it has these slowdown regressions? I don't want to make it sleep for too long, because while we wouldn't notice for standard triggers (since we have to wait for other tests to run anyway), it could make re-runs annoying if we sleep for too long.

  9. keyonghan commented on Mar 18, 2024

    @keyonghan
    Contributor

    Do we have historical stats on how long checkin generally takes when it has these slowdown regressions?

    That really depends on how long GitHub issue gets resolved. Whenever the issue is mitigated, the task status should be posted to GitHub in seconds. We can start with a small number like 1 min to mitigate the too-fast github action. If working well, we can readjust the number further.

  10. ditman commented on Mar 26, 2024

    @ditman
    Member

    I just saw this happen again, I'll try adding a 60s delay to mitigate as suggested by @keyonghan!

  11. ditman commented on Mar 28, 2024

    @ditman
    Member

    I landed the timeout, but should we keep this open to track the fine-tuning of the timeout, or landing a more proper fix?

  12. ditman commented on Mar 28, 2024

    @ditman
    Member

    (The extra minute gets completely lost in the CI run; it's barely noticeable)

  13. stuartmorgan-g commented on Apr 1, 2024

    @stuartmorgan-g
    ContributorAuthor

    Let's leave it closed for now, and we can re-open if we have more issues later.

  14. github-actions commented on Apr 15, 2024

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  15. locked as resolved and limited conversation to collaborators on Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority issues at the top of the work listc: contributor-productivityTeam-specific productivity, code health, technical debt.packageflutter/packages repository. See also p: labels.team-ecosystemOwned by Ecosystem teamtriaged-ecosystemTriaged by Ecosystem team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions