Skip to content

💥 ☂️ Solution to fleet-wide Android emulator crashes on CI #153445

Closed
Closed
💥 ☂️ Solution to fleet-wide Android emulator crashes on CI#153445
Assignees
matanlurey
Labels
P1High-priority issues at the top of the work listc: crashStack traces logged to the consolec: tech-debtTechnical debt, code quality, testing, etc.platform-androidAndroid applications specificallyteam-infraOwned by Infrastructure team

Description

@matanlurey
matanlurey
opened on Aug 14, 2024

Forked from #152769 as it's unrelated to my specific test.

As of 2024-08-14, we have systemic fleet-wide Android emulator crashes that appear unrelated with any specific test:

In nearly every case, the following occurs (note that every test uses a slightly different setup, the logs are not uniform):

  1. The emulator successfully starts (as part of a CI task job)
  2. The test, typically using flutter drive ... installs the test APK, and starts executing
  3. An ambient print message appears in the console: getIsolate: (-32000) Service connection disposed
  4. The test (driver script) ends (usually you'll see adb: device offline in the logs by this point) with a failure

Confusingly, these tests work run fine roughly 95% - 99% of the time.

My plan this week is roughly the following:

  • Try LED builds of the Android emulator try-jobs, trying to extract crash logs, twiddle scripts
  • See what the memory usage of the emulator is locally (we have a max of 2gb right now)
  • Meet with the maintainer of the Chromium Android emulator tools (we use) on GVC

/cc @reidbaker @zanderso @johnmccutchan @christopherfujino @chunhtai

TODOs

  • Consider removing --writable-system:

    ✅ https://flutter-review.googlesource.com/c/recipes/+/59180/1

    W 15:30:04.302    0.034s Main  Emulator will be slow to start, as "writable_system=True" but system snapshot not found.

  • Try meeting with the Chromium Android emulator team on any obvious next steps.

    ✅ Upgrading to the latest avd_cipd_version:
    Roll avd_cipd_verison to latest to use the crashreport tool. #153520

  • Try getting a copy of the emu-crash.db log entries, if any.

    ❌ Did not result in anything.

    During emulator startup, the console reads:

    INFO    | Storing crashdata in: /tmp/android-unknown/emu-crash-34.2.14.db, detection is enabled for process: 124450

    I wrote a small script to try and scrape this directory when a test crashes, but it resulted in nothing:

    Could not read file: /tmp/android-unknown/emu-crash-34.2.14.db/settings.dat

    It's possible that nothing is being logged, the logs are actually elsewhere, but seems like a dead end for now.

  • Try running the same tests on Github Actions VM and see if I observe similar flakes:

    ✅ Ran overnight on a 15m cron, API v35 seems too unstable to use as non-bringup task:

    https://github.com/matanlurey/flutter-renegade-gha

  • [ ] Try running a demo Android app (or even just the emulator itself) without any Flutter

  • [ ] Try SSHing into a bot and running steps manually to try and detect the crash

Activity

  1. matanlurey
    added
    c: crashStack traces logged to the console
    platform-androidAndroid applications specifically
    team-infraOwned by Infrastructure team
    P1High-priority issues at the top of the work list
    c: tech-debtTechnical debt, code quality, testing, etc.
    on Aug 14, 2024
  2. matanlurey
    self-assigned this
    on Aug 14, 2024
  3. matanlurey
    added this to Flutter Driver + Nativeon Aug 14, 2024
  4. matanlurey
    moved this to In Progress in Flutter Driver + Nativeon Aug 14, 2024

  5. zanderso commented on Aug 14, 2024

    @zanderso
    zanderso
    on Aug 14, 2024
    Member

    Also FYI @camsim99

  6. matanlurey commented on Aug 14, 2024

    @matanlurey
    matanlurey
    on Aug 14, 2024
    ContributorAuthor

    One interesting note so far is this log message at the top of every Android emulator run:

    W 15:30:04.302    0.034s Main  Emulator will be slow to start, as "writable_system=True" but system snapshot not found.

  7. reidbaker commented on Aug 14, 2024

    @reidbaker
    reidbaker
    on Aug 14, 2024
    Contributor

    Related internal bugs https://b.corp.google.com/issues/326109113 https://b.corp.google.com/issues/344730009

  8. reidbaker commented on Aug 14, 2024

    @reidbaker
    reidbaker
    on Aug 14, 2024
    Contributor

    Do we know the size of the vms that are hosting the emulators?
    According to android they can crash if there is less than 5gb of disk space. https://developer.android.com/studio/run/emulator-troubleshooting#disk-space

  9. zanderso commented on Aug 14, 2024

    @zanderso
    zanderso
    on Aug 14, 2024
    Member

    These are running on n1-standard-8 instances. https://cloud.google.com/compute/docs/general-purpose-machines#n1_machine_types. I'll try to find the config file for our pool and share the link internally to double check.

  10. matanlurey commented on Aug 14, 2024

    @matanlurey
    matanlurey
    on Aug 14, 2024
    ContributorAuthor

    Didn't solve the flakiness issue, but got rid of an error message and sped up startup time a bit:

    https://flutter-review.googlesource.com/c/recipes/+/59180/1

  11. reidbaker commented on Aug 15, 2024

    @reidbaker
    reidbaker
    on Aug 15, 2024
    Contributor

    These are running on n1-standard-8 instances. https://cloud.google.com/compute/docs/general-purpose-machines#n1_machine_types. I'll try to find the config file for our pool and share the link internally to double check.

    Based on what I read about those instances the space is measured in Terabytes so we are probably ok on disk space.

  12. johnmccutchan commented on Aug 15, 2024

    @johnmccutchan
    johnmccutchan
    on Aug 15, 2024
    Contributor
    • Can we get the Android emulator configuration and create a local version for testing?
    • Are we passing the -verbose flag to the emulator?

  13. johnmccutchan commented on Aug 15, 2024

    @johnmccutchan
    johnmccutchan
    on Aug 15, 2024
    Contributor

    Log of AVD configuration:

    I0812 18:49:08.002179    9101 main.cpp:333] Content of hardware configuration file:
I0812 18:49:08.002190    9101 main.cpp:335] 	hw.cpu.arch = x86_64
I0812 18:49:08.002194    9101 main.cpp:335] 	hw.cpu.ncore = 4
I0812 18:49:08.002198    9101 main.cpp:335] 	hw.ramSize = 2048
I0812 18:49:08.002201    9101 main.cpp:335] 	hw.screen = multi-touch
I0812 18:49:08.002205    9101 main.cpp:335] 	hw.mainKeys = false
I0812 18:49:08.002216    9101 main.cpp:335] 	hw.trackBall = true
I0812 18:49:08.002219    9101 main.cpp:335] 	hw.keyboard = true
I0812 18:49:08.002223    9101 main.cpp:335] 	hw.keyboard.lid = true
I0812 18:49:08.002226    9101 main.cpp:335] 	hw.keyboard.charmap = qwerty2
I0812 18:49:08.002230    9101 main.cpp:335] 	hw.dPad = true
I0812 18:49:08.002233    9101 main.cpp:335] 	hw.rotaryInput = false
I0812 18:49:08.002237    9101 main.cpp:335] 	hw.gsmModem = true
I0812 18:49:08.002240    9101 main.cpp:335] 	hw.gps = true
I0812 18:49:08.002244    9101 main.cpp:335] 	hw.battery = true
I0812 18:49:08.002247    9101 main.cpp:335] 	hw.accelerometer = true
I0812 18:49:08.002251    9101 main.cpp:335] 	hw.accelerometer_uncalibrated = true
I0812 18:49:08.002255    9101 main.cpp:335] 	hw.gyroscope = true
I0812 18:49:08.002258    9101 main.cpp:335] 	hw.audioInput = true
I0812 18:49:08.002261    9101 main.cpp:335] 	hw.audioOutput = true
I0812 18:49:08.002265    9101 main.cpp:335] 	hw.sdCard = true
I0812 18:49:08.002268    9101 main.cpp:335] 	disk.cachePartition = true
I0812 18:49:08.002276    9101 main.cpp:335] 	disk.cachePartition.path = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/avd/../avd/android_35_google_apis_x64.avd/cache.img
I0812 18:49:08.002280    9101 main.cpp:335] 	disk.cachePartition.size = 66m
I0812 18:49:08.002283    9101 main.cpp:335] 	test.quitAfterBootTimeOut = -1
I0812 18:49:08.002287    9101 main.cpp:335] 	test.delayAdbTillBootComplete = 0
I0812 18:49:08.002291    9101 main.cpp:335] 	test.monitorAdb = 0
I0812 18:49:08.002294    9101 main.cpp:335] 	hw.lcd.width = 1080
I0812 18:49:08.002297    9101 main.cpp:335] 	hw.lcd.height = 1920
I0812 18:49:08.002301    9101 main.cpp:335] 	hw.lcd.depth = 16
I0812 18:49:08.002304    9101 main.cpp:335] 	hw.lcd.circular = false
I0812 18:49:08.002308    9101 main.cpp:335] 	hw.lcd.density = 480
I0812 18:49:08.002311    9101 main.cpp:335] 	hw.lcd.backlight = true
I0812 18:49:08.002315    9101 main.cpp:335] 	hw.lcd.vsync = 60
I0812 18:49:08.002318    9101 main.cpp:335] 	hw.gltransport = pipe
I0812 18:49:08.002322    9101 main.cpp:335] 	hw.gltransport.asg.writeBufferSize = 1048576
I0812 18:49:08.002325    9101 main.cpp:335] 	hw.gltransport.asg.writeStepSize = 4096
I0812 18:49:08.002329    9101 main.cpp:335] 	hw.gltransport.asg.dataRingSize = 32768
I0812 18:49:08.002333    9101 main.cpp:335] 	hw.gltransport.drawFlushInterval = 800
I0812 18:49:08.002336    9101 main.cpp:335] 	hw.displayRegion.0.1.xOffset = -1
I0812 18:49:08.002340    9101 main.cpp:335] 	hw.displayRegion.0.1.yOffset = -1
I0812 18:49:08.002363    9101 main.cpp:335] 	hw.displayRegion.0.1.width = 0
I0812 18:49:08.002370    9101 main.cpp:335] 	hw.displayRegion.0.1.height = 0
I0812 18:49:08.002375    9101 main.cpp:335] 	hw.displayRegion.0.2.xOffset = -1
I0812 18:49:08.002381    9101 main.cpp:335] 	hw.displayRegion.0.2.yOffset = -1
I0812 18:49:08.002386    9101 main.cpp:335] 	hw.displayRegion.0.2.width = 0
I0812 18:49:08.002391    9101 main.cpp:335] 	hw.displayRegion.0.2.height = 0
I0812 18:49:08.002396    9101 main.cpp:335] 	hw.displayRegion.0.3.xOffset = -1
I0812 18:49:08.002402    9101 main.cpp:335] 	hw.displayRegion.0.3.yOffset = -1
I0812 18:49:08.002407    9101 main.cpp:335] 	hw.displayRegion.0.3.width = 0
I0812 18:49:08.002412    9101 main.cpp:335] 	hw.displayRegion.0.3.height = 0
I0812 18:49:08.002416    9101 main.cpp:335] 	hw.display1.width = 0
I0812 18:49:08.002420    9101 main.cpp:335] 	hw.display1.height = 0
I0812 18:49:08.002429    9101 main.cpp:335] 	hw.display1.density = 0
I0812 18:49:08.002432    9101 main.cpp:335] 	hw.display1.xOffset = -1
I0812 18:49:08.002436    9101 main.cpp:335] 	hw.display1.yOffset = -1
I0812 18:49:08.002439    9101 main.cpp:335] 	hw.display1.flag = 0
I0812 18:49:08.002443    9101 main.cpp:335] 	hw.display2.width = 0
I0812 18:49:08.002446    9101 main.cpp:335] 	hw.display2.height = 0
I0812 18:49:08.002450    9101 main.cpp:335] 	hw.display2.density = 0
I0812 18:49:08.002453    9101 main.cpp:335] 	hw.display2.xOffset = -1
I0812 18:49:08.002457    9101 main.cpp:335] 	hw.display2.yOffset = -1
I0812 18:49:08.002460    9101 main.cpp:335] 	hw.display2.flag = 0
I0812 18:49:08.002464    9101 main.cpp:335] 	hw.display3.width = 0
I0812 18:49:08.002467    9101 main.cpp:335] 	hw.display3.height = 0
I0812 18:49:08.002471    9101 main.cpp:335] 	hw.display3.density = 0
I0812 18:49:08.002475    9101 main.cpp:335] 	hw.display3.xOffset = -1
I0812 18:49:08.002478    9101 main.cpp:335] 	hw.display3.yOffset = -1
I0812 18:49:08.002482    9101 main.cpp:335] 	hw.display3.flag = 0
I0812 18:49:08.002485    9101 main.cpp:335] 	hw.multi_display_window = false
I0812 18:49:08.002489    9101 main.cpp:335] 	hw.hotplug_multi_display = false
I0812 18:49:08.002492    9101 main.cpp:335] 	hw.gpu.enabled = true
I0812 18:49:08.002496    9101 main.cpp:335] 	hw.gpu.mode = swiftshader_indirect
I0812 18:49:08.002499    9101 main.cpp:335] 	hw.initialOrientation = portrait
I0812 18:49:08.002503    9101 main.cpp:335] 	hw.camera.back = emulated
I0812 18:49:08.002506    9101 main.cpp:335] 	hw.camera.front = none
I0812 18:49:08.002510    9101 main.cpp:335] 	vm.heapSize = 512
I0812 18:49:08.002514    9101 main.cpp:335] 	hw.sensors.light = true
I0812 18:49:08.002517    9101 main.cpp:335] 	hw.sensors.pressure = true
I0812 18:49:08.002521    9101 main.cpp:335] 	hw.sensors.humidity = true
I0812 18:49:08.002524    9101 main.cpp:335] 	hw.sensors.proximity = true
I0812 18:49:08.002528    9101 main.cpp:335] 	hw.sensors.magnetic_field = true
I0812 18:49:08.002531    9101 main.cpp:335] 	hw.sensors.magnetic_field_uncalibrated = true
I0812 18:49:08.002535    9101 main.cpp:335] 	hw.sensors.gyroscope_uncalibrated = true
I0812 18:49:08.002539    9101 main.cpp:335] 	hw.sensors.orientation = true
I0812 18:49:08.002542    9101 main.cpp:335] 	hw.sensors.temperature = true
I0812 18:49:08.002546    9101 main.cpp:335] 	hw.sensors.rgbclight = false
I0812 18:49:08.002549    9101 main.cpp:335] 	hw.sensor.hinge = false
I0812 18:49:08.002553    9101 main.cpp:335] 	hw.sensor.hinge.count = 0
I0812 18:49:08.002556    9101 main.cpp:335] 	hw.sensor.hinge.type = 0
I0812 18:49:08.002560    9101 main.cpp:335] 	hw.sensor.hinge.sub_type = 0
I0812 18:49:08.002563    9101 main.cpp:335] 	hw.sensor.hinge.resizable.config = 1
I0812 18:49:08.002567    9101 main.cpp:335] 	hw.sensor.hinge.fold_to_displayRegion.0.1_at_posture = 1
I0812 18:49:08.002571    9101 main.cpp:335] 	hw.sensor.roll = false
I0812 18:49:08.002574    9101 main.cpp:335] 	hw.sensor.roll.count = 0
I0812 18:49:08.002578    9101 main.cpp:335] 	hw.sensor.roll.resize_to_displayRegion.0.1_at_posture = 6
I0812 18:49:08.002582    9101 main.cpp:335] 	hw.sensor.roll.resize_to_displayRegion.0.2_at_posture = 6
I0812 18:49:08.002585    9101 main.cpp:335] 	hw.sensor.roll.resize_to_displayRegion.0.3_at_posture = 6
I0812 18:49:08.002589    9101 main.cpp:335] 	hw.sensors.heart_rate = false
I0812 18:49:08.002592    9101 main.cpp:335] 	hw.sensors.wrist_tilt = false
I0812 18:49:08.002596    9101 main.cpp:335] 	hw.useext4 = true
I0812 18:49:08.002600    9101 main.cpp:335] 	hw.arc = false
I0812 18:49:08.002603    9101 main.cpp:335] 	hw.arc.autologin = false
I0812 18:49:08.002607    9101 main.cpp:335] 	kernel.path = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/system-images/android-35/google_apis/x86_64//kernel-ranchu
I0812 18:49:08.002611    9101 main.cpp:335] 	kernel.newDeviceNaming = yes
I0812 18:49:08.002614    9101 main.cpp:335] 	kernel.supportsYaffs2 = no
I0812 18:49:08.002618    9101 main.cpp:335] 	disk.ramdisk.path = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/system-images/android-35/google_apis/x86_64//ramdisk.img
I0812 18:49:08.002624    9101 main.cpp:335] 	disk.systemPartition.initPath = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/system-images/android-35/google_apis/x86_64//system.img
I0812 18:49:08.002628    9101 main.cpp:335] 	disk.systemPartition.size = 12000m
I0812 18:49:08.002632    9101 main.cpp:335] 	disk.vendorPartition.initPath = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/system-images/android-35/google_apis/x86_64//vendor.img
I0812 18:49:08.002635    9101 main.cpp:335] 	disk.vendorPartition.size = 12000m
I0812 18:49:08.002639    9101 main.cpp:335] 	disk.dataPartition.path = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/avd/../avd/android_35_google_apis_x64.avd/userdata-qemu.img
I0812 18:49:08.002643    9101 main.cpp:335] 	disk.dataPartition.size = 12000m
I0812 18:49:08.002646    9101 main.cpp:335] 	disk.encryptionKeyPartition.path = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/avd/../avd/android_35_google_apis_x64.avd/encryptionkey.img
I0812 18:49:08.002651    9101 main.cpp:335] 	PlayStore.enabled = false
I0812 18:49:08.002655    9101 main.cpp:335] 	avd.name = android_35_google_apis_x64
I0812 18:49:08.002661    9101 main.cpp:335] 	avd.id = android_35_google_apis_x64
I0812 18:49:08.002665    9101 main.cpp:335] 	fastboot.forceColdBoot = false
I0812 18:49:08.002669    9101 main.cpp:335] 	userdata.useQcow2 = true
I0812 18:49:08.002672    9101 main.cpp:335] 	firstboot.bootFromDownloadableSnapshot = true
I0812 18:49:08.002676    9101 main.cpp:335] 	firstboot.bootFromLocalSnapshot = true
I0812 18:49:08.002679    9101 main.cpp:335] 	firstboot.saveToLocalSnapshot = true
I0812 18:49:08.002683    9101 main.cpp:335] 	android.sdk.root = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64
I0812 18:49:08.002687    9101 main.cpp:335] 	android.avd.home = /b/s/w/ir/cache/avd/src/.android_emulator/android_35_google_apis_x64/avd
I0812 18:49:08.002694    9101 main.cpp:337] .

  14. zanderso commented on Aug 15, 2024

    @zanderso
    zanderso
    on Aug 15, 2024
    Member

    These are running on n1-standard-8 instances. https://cloud.google.com/compute/docs/general-purpose-machines#n1_machine_types. I'll try to find the config file for our pool and share the link internally to double check.

    Based on what I read about those instances the space is measured in Terabytes so we are probably ok on disk space.

    Found the config. Disks are 300 GB on the Linux VMs: https://chrome-internal.googlesource.com/infradata/config/+/refs/heads/main/configs/chromium-swarm/starlark/bots/flutter.star#161

  15. auto-submit
    added a commit that references this issue on Aug 15, 2024

    Roll `avd_cipd_verison` to latest to use the `crashreport` tool. (#15…

    af815f0
  16. zanderso
    added a commit that references this issue on Aug 16, 2024

    Move Android API level 35 emulator tests to staging

    63444e8

  17. matanlurey commented on Aug 16, 2024

    @matanlurey
    matanlurey
    on Aug 16, 2024
    ContributorAuthor

    Unfortunately the solution here is, at the moment, "don't use Android 35 on emulators".

    We will revisit adding them about 1 quarter after the public release, and keep the tasks bringup: true for now.

  18. matanlurey commented on Aug 16, 2024

    @matanlurey
    matanlurey
    on Aug 16, 2024
    ContributorAuthor

    Forked a P2 issue here: #153570.

  19. matanlurey
    closed this as completedon Aug 16, 2024
  20. github-project-automation
    moved this from In Progress to Done in Flutter Driver + Nativeon Aug 16, 2024
  21. auto-submit
    added a commit that references this issue on Aug 16, 2024

    Move Android API level 35 emulator tests to staging (#153568)

    69ff9ea
  22. DBowen33
    added 2 commits that reference this issue on Aug 16, 2024

    Roll `avd_cipd_verison` to latest to use the `crashreport` tool. (flu…

    89c8872

    Move Android API level 35 emulator tests to staging (flutter#153568)

    d144133

  23. github-actions commented on Aug 30, 2024

    @github-actions
    github-actionsbot
    on Aug 30, 2024 – with GitHub Actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  24. github-actions
    locked as resolved and limited conversation to collaborators on Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listc: crashStack traces logged to the consolec: tech-debtTechnical debt, code quality, testing, etc.platform-androidAndroid applications specificallyteam-infraOwned by Infrastructure team

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions