Skip to content

[cocoon] Alert on GitHub API quota getting low #79467

Closed
Closed
[cocoon] Alert on GitHub API quota getting low#79467
Assignees
keyonghan
Labels
P2Important issues not at the top of the work listc: contributor-productivityTeam-specific productivity, code health, technical debt.from: postmortemIssues identified as part of a postmorteminfra: metricsInfrastructure metrics-related issuesteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

Description

@CaseyHillers
CaseyHillers
opened on Mar 31, 2021

AI from postmortem142046

GitHub API quota is reset hourly. If Cocoon uses all 15k requests in the first minute, services will not operate correctly for 59 minutes.

We should also find a way to escalate and manually reset the app's API quota in these cases.

Activity

  1. CaseyHillers
    added
    team-infraOwned by Infrastructure team
    P2Important issues not at the top of the work list
    infra: metricsInfrastructure metrics-related issues
    on Mar 31, 2021
  2. CaseyHillers
    self-assigned this
    on Mar 31, 2021
  3. pedromassangocode
    added
    c: contributor-productivityTeam-specific productivity, code health, technical debt.
    passed first triage
    on Apr 1, 2021
  4. godofredoc
    added
    passed secondary triage
    on Apr 1, 2021
  5. CaseyHillers
    added
    postmortem
    on Apr 5, 2021

  6. CaseyHillers commented on Apr 6, 2021

    @CaseyHillers
    CaseyHillers
    on Apr 6, 2021
    ContributorAuthor

    There is now a dashboard in go/flutter-infra-metrics to track GitHub quota usage. Next steps are to also create a BigQuery monitoring alert to notify on this going out of SLO (50% is probably a good starting point)

  7. godofredoc commented on Apr 22, 2021

    @godofredoc
    godofredoc
    on Apr 22, 2021
    Contributor

    ETA end of q2

  8. CaseyHillers commented on Apr 29, 2021

    @CaseyHillers
    CaseyHillers
    on Apr 29, 2021
    ContributorAuthor

    Met with Dart infra team and learned how they handle these types of issues. My plans are to add their cloud metric code to cocoon, and move to that instead of BigQuery.

  9. godofredoc commented on Jun 24, 2021

    @godofredoc
    godofredoc
    on Jun 24, 2021
    Contributor

    PR in progress, ETA before end of quarter.

  10. CaseyHillers
    removed their assignment
    on Mar 8, 2022

  11. ricardoamador commented on Feb 2, 2023

    @ricardoamador
    ricardoamador
    on Feb 2, 2023
    Contributor

    @CaseyHillers any update you can provide on this?

  12. flutter-triage-bot
    added
    from: postmortemIssues identified as part of a postmortem
    on Jul 6, 2023
  13. Hixie
    removed
    postmortem
    passed secondary triage
    passed first triage
    on Jul 6, 2023
  14. ricardoamador
    added
    triaged-infraTriaged by Infrastructure team
    on Aug 22, 2023
  15. keyonghan
    self-assigned this
    on Nov 30, 2023

  16. keyonghan commented on Nov 30, 2023

    @keyonghan
    keyonghan
    on Nov 30, 2023
    Contributor

    This has been implemented. Infra gardener is expected to receive such alerts when quota gets low.
    Screenshot 2023-11-30 at 12 56 28 PM

  17. keyonghan
    closed this as completedon Nov 30, 2023

  18. github-actions commented on Dec 14, 2023

    @github-actions
    github-actionsbot
    on Dec 14, 2023 – with GitHub Actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  19. github-actions
    locked as resolved and limited conversation to collaborators on Dec 14, 2023
  20. github-project-automation
    added this to Infra - pay technical debton Aug 28, 2024
  21. github-project-automation
    moved this to Done in Infra - pay technical debton Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Important issues not at the top of the work listc: contributor-productivityTeam-specific productivity, code health, technical debt.from: postmortemIssues identified as part of a postmorteminfra: metricsInfrastructure metrics-related issuesteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions