Skip to content

[cocoon] API rate limit exceeded #58627

Closed
Closed
[cocoon] API rate limit exceeded#58627
Assignees
godofredoc
Labels
P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure team

Description

@godofredoc
godofredoc
opened on Jun 3, 2020

GitHub Error: API rate limit exceeded for user ID 52682268.

#0      GitHub.handleStatusCode (package:github/src/common/github.dart:435:5)
#1      GitHub.request (package:github/src/common/github.dart:372:7)
<asynchronous suspension>
#2      PaginationHelper.fetchStreamed (package:github/src/common/util/pagination.dart:36:33)
<asynchronous suspension>
#3      PaginationHelper.jsonObjects (package:github/src/common/util/pagination.dart:99:34)
<asynchronous suspension>
#4      PaginationHelper.objects (package:github/src/common/util/pagination.dart:132:12)
#5      RepositoriesService.listStatuses (package:github/src/common/repos_service.dart:1221:10)
#6      LuciStatusHandler._setPendingStatus (package:cocoon_service/src/request_handlers/luci_status.dart:244:38)
<asynchronous suspension>
#7      LuciStatusHandler.post (package:cocoon_service/src/request_handlers/luci_status.dart:82:15)
<asynchronous suspension>
#8      RequestHandler.service.<anonymous closure> (package:cocoon_service/src/request_handling/request_handler.dart:49:28)
#9      _rootRun (dart:async/zone.dart:1184:13)
#10     _CustomZone.run (dart:async/zone.dart:1077:19)
#11     _runZoned (dart:async/zone.dart:1619:10)
#12     runZoned (dart:async/zone.dart:1539:10)
#13     RequestHandler.service (package:cocoon_service/src/request_handling/request_handler.dart:39:12)
#14     main.<anonymous closure>.<anonymous closure> (file:///app/bin/server.dart:83:23)
#15     runAppEngine.<anonymous closure> (package:appengine/appengine.dart:80:12)
#16     runAppEngine.<anonymous closure>.<anonymous closure>.<anonymous closure> (package:appengine/src/appengine_internal.dart:59:18)
#17     _ServiceScope._fork.<anonymous closure> (package:gcloud/service_scope.dart:209:19)
#18     _rootRun (dart:async/zone.dart:1184:13)
#19     _CustomZone.run (dart:async/zone.dart:1077:19)
#20     _runZoned (dart:async/zone.dart:1619:10)
#21     runZonedGuarded (dart:async/zone.dart:1608:12)
#22     runZoned (dart:async/zone.dart:1536:12)
#23     _ServiceScope._fork (package:gcloud/service_scope.dart:208:12)
#24     fork (package:gcloud/service_scope.dart:104:30)
#25     runAppEngine.<anonymous closure>.<anonymous closure> (package:appengine/src/appengine_internal.dart:57:9)
#26     AppEngineHttpServer.run.<anonymous closure>.<anonymous closure> (package:appengine/src/server/server.dart:83:16)
#27     _rootRunUnary (dart:async/zone.dart:1192:38)
#28     _CustomZone.runUnary (dart:async/zone.dart:1085:19)
#29     _CustomZone.runUnaryGuarded (dart:async/zone.dart:987:7)
#30     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)
#31     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:266:7)
#32     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:779:19)
#33     _StreamController._add (dart:async/stream_controller.dart:655:7)
#34     _StreamController.add (dart:async/stream_controller.dart:597:5)
#35     _HttpServer._handleRequest (dart:_http/http_impl.dart:2837:19)
#36     new _HttpConnection.<anonymous closure> (dart:_http/http_impl.dart:2596:19)
#37     _rootRunUnary (dart:async/zone.dart:1192:38)
#38     _CustomZone.runUnary (dart:async/zone.dart:1085:19)
#39     _CustomZone.runUnaryGuarded (dart:async/zone.dart:987:7)
#40     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)
#41     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:266:7)
#42     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:779:19)
#43     _StreamController._add (dart:async/stream_controller.dart:655:7)
#44     _StreamController.add (dart:async/stream_controller.dart:597:5)
#45     _HttpParser._headersEnd (dart:_http/http_parser.dart:413:17)
#46     _HttpParser._doParse (dart:_http/http_parser.dart:739:15)
#47     _HttpParser._parse (dart:_http/http_parser.dart:328:7)
#48     _HttpParser._onData (dart:_http/http_parser.dart:850:5)
#49     _rootRunUnary (dart:async/zone.dart:1192:38)
#50     _CustomZone.runUnary (dart:async/zone.dart:1085:19)
#51     _CustomZone.runUnaryGuarded (dart:async/zone.dart:987:7)
#52     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)
#53     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:266:7)
#54     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:779:19)
#55     _StreamController._add (dart:async/stream_controller.dart:655:7)
#56     _StreamController.add (dart:async/stream_controller.dart:597:5)
#57     _Socket._onData (dart:io-patch/socket_patch.dart:1982:41)
#58     _rootRunUnary (dart:async/zone.dart:1196:13)
#59     _CustomZone.runUnary (dart:async/zone.dart:1085:19)
#60     _CustomZone.runUnaryGuarded (dart:async/zone.dart:987:7)
#61     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:339:11)
#62     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:266:7)
#63     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:779:19)
#64     _StreamController._add (dart:async/stream_controller.dart:655:7)
#65     _StreamController.add (dart:async/stream_controller.dart:597:5)
#66     new _RawSocket.<anonymous closure> (dart:io-patch/socket_patch.dart:1527:33)
#67     _NativeSocket.issueReadEvent.issue (dart:io-patch/socket_patch.dart:1019:14)
#68     _microtaskLoop (dart:async/schedule_microtask.dart:43:21)
#69     _startMicrotaskLoop (dart:async/schedule_microtask.dart:52:5)
#70     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:118:13)
#71     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:169:5)

Activity

  1. godofredoc
    self-assigned this
    on Jun 3, 2020
  2. godofredoc
    added
    team-infraOwned by Infrastructure team
    P0Critical issues such as a build break or regression
    on Jun 3, 2020

  3. keyonghan commented on Jun 3, 2020

    @keyonghan
    keyonghan
    on Jun 3, 2020
    Contributor

    The increased release branches may increase the github API requests.
    PR: flutter/cocoon#795 removed one unused branch. More to come.

  4. keyonghan commented on Jun 4, 2020

    @keyonghan
    keyonghan
    on Jun 4, 2020
    Contributor

    Moving the conversation about filtering out release branches here.

    One possible way is to add more accurate regexps here: https://github.com/flutter/cocoon/blob/master/app_dart/dev/branch_regexps.txt

    Instead of using general ^flutter-[0-9]+\.[0-9]+-candidate\.[0-9]+$, we use more accurate ones like '^flutter-1+.17+-candidate.3+$'. Whenever new release branch is created, we need to update this file as well. The same when we decide to skip branches.

    Another way is keep the above file as it is, and simply add a new file which keeps track of all unwanted branches. Again, it needs to be updated when we decide to skip branches.

    Both cases will face the same issue when the wanted branches keep accumulating in the long run. If some day the wanted branch number hit a threshold where the Github API requests already hit the limit, it would be a problem.

    So the ideal case is that we keep track of a limited number of release branches.

    /cc @godofredoc @pcsosinski

  5. pcsosinski commented on Jun 4, 2020

    @pcsosinski
    pcsosinski
    on Jun 4, 2020

    FYI I'm working on a proposal that will reduce the number of branches we create.

  6. Piinks commented on Jun 4, 2020

    @Piinks
    Piinks
    on Jun 4, 2020
    Contributor

    I've recently seen this show up once on the gold checks cocoon runs. Since the gold check is kinda moot on branches other than master we could make a reduction there.

  7. pcsosinski commented on Jun 4, 2020

    @pcsosinski
    pcsosinski
    on Jun 4, 2020

    @christopherfujino may have created an issue for removing gold checks on release branches

  8. digiter
    added
    passed secondary triage
    on Jun 4, 2020

  9. christopherfujino commented on Jun 4, 2020

    @christopherfujino
    christopherfujino
    on Jun 4, 2020
    Contributor

    Instead of using general ^flutter-[0-9]+\.[0-9]+-candidate\.[0-9]+$, we use more accurate ones like '^flutter-1+.17+-candidate.3+$'. Whenever new release branch is created, we need to update this file as well. The same when we decide to skip branches.

    We already have to update regexes in flutter/infra for each beta/stable release, as documented in the wiki, so we could add a step to update a file in cocoon.

  10. godofredoc commented on Jun 5, 2020

    @godofredoc
    godofredoc
    on Jun 5, 2020
    ContributorAuthor

    Thank you all for the information keeping only the current branches in the file sounds like a very good short term solution. Ideally if we only keep <10 branches we can directly add the branch names and we don't need to call the github apis at all.

    @christopherfujino do you have at hand the bug to remove gold checks from branches?

  11. christopherfujino commented on Jun 5, 2020

    @christopherfujino
    christopherfujino
    on Jun 5, 2020
    Contributor

    @christopherfujino do you have at hand the bug to remove gold checks from branches?

    I had not previously created the issue, so I created one now: #58814

  12. pcsosinski commented on Jun 5, 2020

    @pcsosinski
    pcsosinski
    on Jun 5, 2020

    sgtm

  13. keyonghan commented on Jun 5, 2020

    @keyonghan
    keyonghan
    on Jun 5, 2020
    Contributor

    With using hardcode branches instead of regexp, we didn't hit the rate limit error for the past two hours. Closing it for now.

  14. keyonghan
    closed this as completedon Jun 5, 2020

  15. github-actions commented on Aug 20, 2021

    @github-actions
    github-actionsbot
    on Aug 20, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  16. github-actions
    locked as resolved and limited conversation to collaborators on Aug 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions