Skip to content

flutter-devicelab-mac-31 bot seems unhealthy #189274

Closed
Closed
flutter-devicelab-mac-31 bot seems unhealthy#189274
Assignees
kentnguyen99
Labels
P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

Description

@b-luk
b-luk
opened on Jul 10, 2026

https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-31

Of the most recent 30 tasks, there are 29 failures.

I examined a few of the failing logs, and the failures seem to be related to flutter run reporting that it can't find the expected mokey device.

Not sure if it's relevant, but I compared logs on a successful task on a different machine with one of this machine's failures. There might be something funky going on with parsing the device identifier:

Image

Activity

  1. b-luk
    added
    team-infraOwned by Infrastructure team
    P0Critical issues such as a build break or regression
    on Jul 10, 2026
  2. jtmcdole
    assigned
    kentnguyen99
    on Jul 10, 2026

  3. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    @kentnguyen99 - can we checkout what's wrong with the mokey device?

  4. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    I shutdown the bot.

  5. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    Will take a look of Mokey phone device. We have many issues with Mokey's phone hardware (not stable).

  6. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    We'll start planning to replace them, its just one of those opportune times to change some configurations in the tree as well.

  7. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    Can you give another try? The phone device has been reset and erased all data to factory reset. It is back online and ready for service.

    flutter-devicelab-mac-31:~ flutter$ adb devices
    List of devices attached
    IN7XM7VOROQ47DRW device

  8. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    can we reboot the mac?

  9. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    Yes, it has been rebooted as well, it is up for 14mins.

    flutter-devicelab-mac-31:log flutter$ uptime
    10:46 up 14 mins, 3 users, load averages: 1.35 1.55 1.53

  10. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    It seems flutter-mac-31's hardware has issued, machine keeps rebooting every hour. https://chromium-swarm.appspot.com/bot?e=true&id=flutter-devicelab-mac-31

  11. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    I think that Mokey might be dead. I see mostly "No device found", which as I recall causes the bot to reboot.

    [2026-07-10 13:03:34.127626] [STDOUT] stderr: JVM crash log found: file:///opt/s/w/ir/x/w/rc/tmp8fn9oj1y/flutter%20sdk/examples/hello_world/android/hs_err_pid7123.log
[2026-07-10 13:03:34.131118] [STDOUT] stderr: 
[2026-07-10 13:03:34.131307] [STDOUT] stderr: FAILURE: Build failed with an exception.
[2026-07-10 13:03:34.131469] [STDOUT] stderr: 
[2026-07-10 13:03:34.131768] [STDOUT] stderr: * What went wrong:
[2026-07-10 13:03:34.132015] [STDOUT] stderr: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

  12. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    @b-luk this might be relevant to you highlighted parts: 629df07

    @bkonyi fyi

  13. jtmcdole commented on Jul 10, 2026

    @jtmcdole
    jtmcdole
    on Jul 10, 2026
    Member

    Lets hold off on nuke the machine till @bkonyi has a chance to look at it.

  14. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    Can you run another test on flutter-mac-31? I connected a different phone device to machine.

  15. b-luk commented on Jul 10, 2026

    @b-luk
    b-luk
    on Jul 10, 2026
    ContributorAuthor

    This is continuing to fail in the same way:

    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_mokey%20run_debug_test_android/8691/overview:

    [2026-07-10 16:01:01.764250] [STDOUT] uninstall:stdout: [  +61 ms] List of devices attached
[2026-07-10 16:01:01.764277] [STDOUT] uninstall:stdout:            PZ6P5HLJNZFQJZSC       device 20-3 product:mokey model:mokey device:mokey transport_id:1
[2026-07-10 16:01:01.769078] [STDOUT] uninstall:stdout: [   +5 ms] /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s PZ6P5HLJNZFQJZSC       device shell getprop
[2026-07-10 16:01:01.803939] [STDOUT] uninstall:stderr: [  +32 ms] Error 1 retrieving device properties for mokey:
[2026-07-10 16:01:01.803970] [STDOUT] uninstall:stderr: [   +1 ms] adb: device 'PZ6P5HLJNZFQJZSC       device' not found

    I think it'll keep failing until the new issue caused by #187943 is fixed.

    I don't know how to shut this machine down (The "Stop the bot gracefully" button at https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-31 is disabled for me). But if someone can shut it down, we should do it to keep the tree green.

  16. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    I have been monitoring the task job is running on flutter-mac-31. It is definitely that something is wrong with hardware of flutter-mac-31 because machine starts rebooting in middle while the job still running.

  17. kentnguyen99 commented on Jul 10, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 10, 2026

    I put flutter-mac-31 in quarantine mode until look into further.

  18. bkonyi commented on Jul 13, 2026

    @bkonyi
    bkonyi
    on Jul 13, 2026
    Contributor

    Thanks for the heads up @jtmcdole! Fix up for review here: #189369.

  19. jtmcdole
    added
    triaged-infraTriaged by Infrastructure team
    on Jul 13, 2026
  20. loic-sharma
    added a commit that references this issue on Jul 13, 2026

    [flutter_tools] Fix ADB device listing output parsing regression (flu…

    43ddd61

  21. b-luk commented on Jul 13, 2026

    @b-luk
    b-luk
    on Jul 13, 2026
    ContributorAuthor

    #189369 has been merged, so the original issue that we saw should be fixed. I think we should re-enable the machine to how it's behaving now.

    I don't know if that fix also addresses the other issue of the machine unexpectedly rebooting. Maybe the rebooting was caused by the original issue, so fixing the original issue may resolve the reboots too? We can re-enable and find out.

  22. kentnguyen99 commented on Jul 13, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 13, 2026

    flutter-mac-31 has been re-enabled and back into production pool. Will monitoring status and see issue still persist.

  23. auto-submit
    added a commit that references this issue on Jul 15, 2026

    [beta] [flutter_tools] Fix ADB device listing output parsing regressi…

    c7edcb6

  24. aam commented on Jul 24, 2026

    @aam
    aam
    on Jul 24, 2026
    Member

    This machine is still causing problems dart-lang/sdk#63881. Please take it out of the pool permanently.

  25. kentnguyen99 commented on Jul 24, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 24, 2026

    I see one job completed successfully yesterday in the display logs https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-31. Can someone submit more test jobs on this machine?

  26. kentnguyen99 commented on Jul 27, 2026

    @kentnguyen99
    kentnguyen99
    on Jul 27, 2026

    I see another job test ran successfully on Friday 7/24 without errors - https://chromium-swarm.appspot.com/bot?id=flutter-devicelab-mac-31. This service ticket can be closed.

  27. kentnguyen99
    closed this as completedon Jul 27, 2026

  28. github-actions commented on Aug 10, 2026

    @github-actions
    github-actionsbot
    on Aug 10, 2026 – with GitHub Actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  29. github-actions
    locked as resolved and limited conversation to collaborators on Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions