Skip to content

Devicelab tasks flaking due to insufficient storage #135035

Closed

Description

@jonahwilliams

Is there an existing issue for this?

Type of Request

bug

Infrastructure Environment

Devicelab

What is happening?

Many Device lab tasks are flaking due to insufficient storage errors, see: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8769511480680570273/+/u/run_opacity_peephole_grid_of_alpha_savelayers_perf__e2e_summary/test_stdout

[2023-09-19 09:05:37.901796] [STDOUT] stderr: 
[2023-09-19 09:05:37.901814] [STDOUT] stderr:            adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Package couldn't be installed in /data/app/com.example.macrobenchmarks-1: Creating application package com.example.macrobenchmarks failed]
[2023-09-19 09:05:37.901836] [STDOUT] stdout: [        ] Warning: Failed to install APK.
[2023-09-19 09:05:37.902070] [STDOUT] stderr: [        ] Application failed to start on attempt: 2
[2023-09-19 09:05:37.902695] [STDOUT] stdout: [        ] Stopping app 'app-profile.apk' on Moto G 4.
[2023-09-19 09:05:37.902745] [STDOUT] stdout: [        ] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s ZY223HK9P7 shell am force-stop com.example.macrobenchmarks
[2023-09-19 09:05:38.788481] [STDOUT] stdout: [ +885 ms] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s ZY223HK9P7 shell pm list packages com.example.macrobenchmarks
[2023-09-19 09:05:39.733880] [STDOUT] stdout: [ +945 ms] Installing APK.
[2023-09-19 09:05:39.734219] [STDOUT] stdout: [        ] Installing build/app/outputs/flutter-apk/app-profile.apk...
[2023-09-19 09:05:39.734266] [STDOUT] stdout: [        ] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s ZY223HK9P7 install -t -r /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk
[2023-09-19 09:05:54.183736] [STDOUT] stdout: [+14448 ms] Performing Streamed Install
[2023-09-19 09:05:54.183857] [STDOUT] stdout: 
[2023-09-19 09:05:54.183881] [STDOUT] stdout:             adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Package couldn't be installed in /data/app/com.example.macrobenchmarks-1: Creating application package com.example.macrobenchmarks failed]
[2023-09-19 09:05:54.183906] [STDOUT] stdout: [        ] Installing build/app/outputs/flutter-apk/app-profile.apk... (completed in 14.4s)
[2023-09-19 09:05:54.183982] [STDOUT] stderr: [        ] Error: ADB exited with exit code 1
[2023-09-19 09:05:54.184001] [STDOUT] stderr: [        ] Performing Streamed Install
[2023-09-19 09:05:54.184018] [STDOUT] stderr: 
[2023-09-19 09:05:54.184037] [STDOUT] stderr:            adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Package couldn't be installed in /data/app/com.example.macrobenchmarks-1: Creating application package com.example.macrobenchmarks failed]
[2023-09-19 09:05:54.184073] [STDOUT] stdout: [        ] Warning: Failed to install APK.
[2023-09-19 09:05:54.184351] [STDOUT] stderr: [        ] Application failed to start on 

Steps to reproduce

No response

Expected results

No response

Activity

  1. added
    P1High-priority issues at the top of the work list
    on Sep 19, 2023
  2. drewroengoogle commented on Sep 19, 2023

    @drewroengoogle
    Contributor

    Many of them seem to be occurring on linux-15 link, @yusuf-goog Would it make sense to quarantine this bot?

  3. yusuf-goog commented on Sep 19, 2023

    @yusuf-goog
    Contributor

    let me look at it and free up space on the attached device.

  4. self-assigned this
    on Sep 19, 2023
  5. yusuf-goog commented on Sep 19, 2023

    @yusuf-goog
    Contributor

    quarantined the host. Looking at freeing up space on the device.

  6. yusuf-goog commented on Sep 19, 2023

    @yusuf-goog
    Contributor

    Uninstalled all third party packages:
    adb shell pm list packages -3 | cut -d':' -f2 | tr '\r' ' ' | xargs -r -n1 -t adb uninstall

    Not much direct use of space. Before:

    Filesystem                             1K-blocks    Used Available Use% Mounted on
    tmpfs                                     945452     692    944760   1% /dev
    tmpfs                                     945452       0    945452   0% /mnt
    /dev/block/bootdevice/by-name/system     2499580 1851308    631888  75% /system
    /dev/block/bootdevice/by-name/userdata  11349888 3141948   8122052  28% /data
    /dev/block/bootdevice/by-name/cache       253920     296    248384   1% /cache
    /dev/block/bootdevice/by-name/persist      28144     468     27024   2% /persist
    /dev/block/bootdevice/by-name/dsp          12016    3740      7952  32% /dsp
    /dev/block/bootdevice/by-name/modem        96688   71244     23396  76% /firmware
    /dev/block/bootdevice/by-name/fsg           1968    1956         0 100% /fsg
    /dev/block/bootdevice/by-name/oem         650656   97864    539360  16% /oem
    /data/media                             11349888 3141948   8122052  28% /storage/emulated
    

    after:

    Filesystem                             1K-blocks    Used Available Use% Mounted on
    tmpfs                                     945452     692    944760   1% /dev
    tmpfs                                     945452       0    945452   0% /mnt
    /dev/block/bootdevice/by-name/system     2499580 1851308    631888  75% /system
    /dev/block/bootdevice/by-name/userdata  11349888 2736248   8527752  25% /data
    /dev/block/bootdevice/by-name/cache       253920     296    248384   1% /cache
    /dev/block/bootdevice/by-name/persist      28144     468     27024   2% /persist
    /dev/block/bootdevice/by-name/dsp          12016    3740      7952  32% /dsp
    /dev/block/bootdevice/by-name/modem        96688   71244     23396  76% /firmware
    /dev/block/bootdevice/by-name/fsg           1968    1956         0 100% /fsg
    /dev/block/bootdevice/by-name/oem         650656   97864    539360  16% /oem
    /data/media                             11349888 2736248   8527752  25% /storage/emulated
    
  7. yusuf-goog commented on Sep 19, 2023

    @yusuf-goog
    Contributor
  8. yusuf-goog commented on Sep 19, 2023

    @yusuf-goog
    Contributor

    @jonahwilliams the device has ~8gb in /data, which should be enough to install any app?

    I think we might be hitting a different issue which just manifests as INSTALL_FAILED_INSUFFICIENT_STORAGE :

    https://stackoverflow.com/questions/4709137/solution-to-install-failed-insufficient-storage-error-on-android

    Looking into this.

  9. yusuf-goog commented on Sep 19, 2023

    @yusuf-goog
    Contributor

    Ran:
    pm uninstall com.example.macrobenchmarks-1
    rm -rf /data/app/com.example.macrobenchmarks*

    Unquarantined the host, looking to see if we still get the failure.

  10. yusuf-goog commented on Sep 20, 2023

    @yusuf-goog
    Contributor

    builds are running successfully on linux-15 now, including a few that had previously failed with out of space errors. Closing this, reopen if it happens again.

  11. 74 remaining items

  12. godofredoc commented on Oct 27, 2023

    @godofredoc
    Contributor

    Things to check:

    • is cleanup logic running? which version?
    • are there any tests not removing the app after the test?
  13. godofredoc commented on Oct 27, 2023

    @godofredoc
    Contributor

    Cleanup logic is running on_after_task with a task failure. The following is a sample:

     7774 755 2023-10-27 16:41:44.485 I: device doctor recovery: b'Device recovery: deleting package caches...
    Uninstalling package com.android.cts.priv.ctsshim : STDOUT(Success)
    Uninstalling package com.google.android.youtube : STDOUT(Success)
    Uninstalling package com.google.android.ext.services : STDOUT(Success)
    Uninstalling package com.motorola.motocare : STDOUT(Success)
    Uninstalling package com.android.providers.telephony : STDOUT(Success)
    Uninstalling package com.google.android.gallery3d : STDOUT(Success)
    Uninstalling package cci.usage : STDOUT(Success)
    Uninstalling package com.android.sdm.plugins.connmo : STDOUT(Success)
    Uninstalling package com.google.android.googlequicksearchbox : STDOUT(Success)
    Uninstalling package com.android.providers.calendar : STDOUT(Success)
    Uninstalling package com.android.providers.media : STDOUT(Success)
    Uninstalling package com.qti.service.colorservice : STDOUT(Success)
    Uninstalling package com.google.android.onetimeinitializer : STDOUT(Success)
    Uninstalling package com.google.android.ext.shared : STDOUT(Success)
    Uninstalling package com.motorola.brapps : STDOUT(Success)
    Uninstalling package com.motorola.bug2go : STDOUT(Success)
    Uninstalling package com.android.wallpapercropper : STDOUT(Success)
    Uninstalling package com.quicinc.cne.CNEService : STDOUT(Success)
    Uninstalling package com.motorola.ccc.devicemanagement : STDOUT(Success)
    Uninstalling package com.motorola.android.fmradio : STDOUT(Success)
    Uninstalling package com.android.documentsui : STDOUT(Success)
    Uninstalling package com.motorola.android.settings.modemdebug : STDOUT(Success)
    Uninstalling package com.android.externalstorage : STDOUT(Success)
    Uninstalling package com.android.htmlviewer : STDOUT(Success)
    Uninstalling package com.android.mms.service : STDOUT(Success)
    Uninstalling package com.android.providers.downloads : STDOUT(Success)
    Uninstalling package com.motorola.coresettingsext : STDOUT(Success)
    Uninstalling package com.motorola.wappushsi : STDOUT(Success)
    Uninstalling package com.google.android.apps.messaging : STDOUT(Success)
    Uninstalling package com.motorola.android.settings.diag_mdlog : STDOUT(Success)
    Uninstalling package com.motorola.entitlement : STDOUT(Success)
    Uninstalling package com.motorola.att.phone.extensions : STDOUT(Success)
    Uninstalling package com.motorola.ccc.checkin : STDOUT(Success)
    Uninstalling package com.motorola.programmenu : STDOUT(Success)
    Uninstalling package com.motorola.ccc.mainplm : STDOUT(Success)
    Uninstalling package com.motorola.motgeofencesvc : STDOUT(Success)
    Uninstalling package com.google.android.configupdater : STDOUT(Success)
    Uninstalling package com.motorola.ccc.ota : STDOUT(Success)
    Uninstalling package com.motorola.ccc.notification : STDOUT(Success)
    Uninstalling package com.android.defcontainer : STDOUT(Success)
    Uninstalling package com.android.providers.downloads.ui : STDOUT(Success)
    Uninstalling package com.android.vending : STDOUT(Success)
    Uninstalling package com.android.pacprocessor : STDOUT(Success)
    Uninstalling package com.motorola.pgmsystem2 : STDOUT(Success)
    Uninstalling package com.motorola.motodisplay.env : STDOUT(Success)
    Uninstalling package com.motorola.ptt.prip : STDOUT(Success)
    Uninstalling package com.android.certinstaller : STDOUT(Success)
    Uninstalling package com.android.carrierconfig : STDOUT(Success)
    Uninstalling package com.google.android.marvin.talkback : STDOUT(Success)
    Uninstalling package com.motorola.digitalpersonalization : STDOUT(Success)
    Uninstalling package android : STDOUT(Success)
    Uninstalling package com.android.contacts : STDOUT(Success)
    Uninstalling package com.motorola.android.provisioning : STDOUT(Success)
    Uninstalling package com.google.android.apps.inputmethod.hindi : STDOUT(Success)
    Uninstalling package com.lenovo.FileBrowser2 : STDOUT(Success)
    Uninstalling package com.android.egg : STDOUT(Success)
    Uninstalling package com.android.mtp : STDOUT(Success)
    Uninstalling package com.android.stk : STDOUT(Success)
    Uninstalling package com.android.backupconfirm : STDOUT(Success)
    Uninstalling package com.google.android.launcher : STDOUT(Success)
    Uninstalling package com.motorola.timeweatherwidget : STDOUT(Success)
    Uninstalling package com.google.android.deskclock : STDOUT(Success)
    Uninstalling package org.codeaurora.ims : STDOUT(Success)
    Uninstalling package com.android.statementservice : STDOUT(Success)
    Uninstalling package com.republicwireless.tel : STDOUT(Success)
    Uninstalling package com.google.android.gm : STDOUT(Success)
    Uninstalling package com.motorola.motocit : STDOUT(Success\n)\nUninstalling package com.android.sdm.plugins.sprintdm : STDOUT(Success)
    Uninstalling package com.google.android.apps.inputmethod.zhuyin : STDOUT(Success)
    Uninstalling package com.google.android.setupwizard : STDOUT(Success)
    Uninstalling package com.qualcomm.qcrilmsgtunnel : STDOUT(Success)
    Uninstalling package com.android.providers.settings : STDOUT(Success)
    Uninstalling package com.android.sharedstoragebackup : STDOUT(Success)
    Uninstalling package com.google.android.music : STDOUT(Success)
    Uninstalling package com.android.printspooler : STDOUT(Success)
    Uninstalling package com.motorola.android.sepolicyupdate : STDOUT(Success)
    Uninstalling package com.motorola.colorprofiles : STDOUT(Success)
    Uninstalling package com.android.sdm.plugins.diagmon : STDOUT(Success)
    Uninstalling package com.android.dreams.basic : STDOUT(Success)
    Uninstalling package com.android.inputdevices : STDOUT(Success)
    Uninstalling package com.motorola.android.nativedropboxagent : STDOUT(Success)
    Uninstalling package android.oem.overlay : STDOUT(Success)
    Uninstalling package com.motorola.android.jvtcmd : STDOUT(Success)
    Uninstalling package com.motorola.frameworks.singlehand : STDOUT(Success)
    Uninstalling package com.motorola.motosignature.app : STDOUT(Success)
    Uninstalling package com.google.android.apps.cloudprint : STDOUT(Success)
    Uninstalling package com.google.android.apps.docs : STDOUT(Success)
    Uninstalling package com.google.android.apps.maps : STDOUT(Success)
    Uninstalling package com.android.cellbroadcastreceiver : STDOUT(Success)
    Uninstalling package com.motorola.invisiblenet : STDOUT(Success)
    Uninstalling package com.google.android.webview : STDOUT(Success)
    Uninstalling package com.motorola.contacts.preloadcontacts : STDOUT(Success)
    Uninstalling package com.android.server.telecom : STDOUT(Success)
    Uninstalling package com.google.android.syncadapters.contacts : STDOUT(Success)
    Uninstalling package com.motorola.android.providers.settings : STDOUT(Success)
    Uninstalling package com.android.facelock : STDOUT(Success)
    Uninstalling package com.android.keychain : STDOUT(Success)
    Uninstalling package ca.bell.wt.android.tunesappswidget : STDOUT(Success)
    Uninstalling package com.google.android.calculator : STDOUT(Success)
    Uninstalling package com.android.chrome : STDOUT(Success)
    Uninstalling package com.android.dialer : STDOUT(Success)
    Uninstalling package com.google.android.packageinstaller : STDOUT(Success)
    Uninstalling package com.motorola.ptt.prip.br : STDOUT(Success)
    Uninstalling package com.google.android.gms : STDOUT(Success)
    Uninstalling package com.google.android.gsf : STDOUT(Success)
    Uninstalling package com.google.android.ims : STDOUT(Success)
    Uninstalling package com.google.android.tts : STDOUT(Success)
    Uninstalling package com.verizon.omadm : STDOUT(Success)
    Uninstalling package com.android.calllogbackup : STDOUT(Success)
    Uninstalling package com.google.android.partnersetup : STDOUT(Success)
    Uninstalling package com.motorola.genie : STDOUT(Success)
    Uninstalling package com.fingerprints.serviceext : STDOUT(Success)
    Uninstalling package com.motorola.setup : STDOUT(Success)
    Uninstalling package com.google.android.videos : STDOUT(Success)
    Uninstalling package com.motorola.retailconfig : STDOUT(Success)
    Uninstalling package com.motorola.lifetimedata : STDOUT(Success)
    Uninstalling package com.motorola.appdirectedsmsproxy : STDOUT(Success)
    Uninstalling package com.android.proxyhandler : STDOUT(Success)
    Uninstalling package com.google.android.feedback : STDOUT(Success)
    Uninstalling package com.google.android.printservice.recommendation : STDOUT(Success)
    Uninstalling package com.google.android.apps.photos : STDOUT(Success)
    Uninstalling package com.google.android.calendar : STDOUT(Success)
    Uninstalling package com.motorola.onetimeinitializer : STDOUT(Success)
    Uninstalling package com.android.managedprovisioning : STDOUT(Success)
    Uninstalling package com.motorola.launcherconfig : STDOUT(Success)
    Uninstalling package com.motorola.demo : STDOUT(Success)
    Uninstalling package com.motorola.hide : STDOUT(Success)
    Uninstalling package com.motorola.moto : STDOUT(Success)
    Uninstalling package com.motorola.slpc : STDOUT(Success)
    Uninstalling package com.android.sdm.plugins.dcmo : STDOUT(Success)
    Uninstalling package com.android.providers.partnerbookmarks : STDOUT(Success)
    Uninstalling package com.google.android.gsf.login : STDOUT(Success)
    Uninstalling package com.android.wallpaper.livepicker : STDOUT(Success)
    Uninstalling package com.motorola.carriersettingsext : STDOUT(Success)
    Uninstalling package com.motorola.motocare.internal : STDOUT(Success)
    Uninstalling package com.motorola.storageoptimizer : STDOUT(Success)
    Uninstalling package com.google.android.inputmethod.korean : STDOUT(Success)
    Uninstalling package com.motorola.fmplayer : STDOUT(Success)
    Uninstalling package com.google.android.backuptransport : STDOUT(Success)
    Uninstalling package com.motorola.audioeffects : STDOUT(Success)
    Uninstalling package com.motorola.msimsettings : STDOUT(Success)
    Uninstalling package com.android.bookmarkprovider : STDOUT(Success)
    Uninstalling package com.android.settings : STDOUT(Success)
    Uninstalling package com.google.android.inputmethod.pinyin : STDOUT(Success)
    Uninstalling package com.android.sprint.hiddenmenuapp : STDOUT(Success)
    Uninstalling package com.qualcomm.location : STDOUT(Success)
    Uninstalling package com.android.omadm.service : STDOUT(Success)
    Uninstalling package com.android.cts.ctsshim : STDOUT(Success)
    Uninstalling package com.lmi.motorola.rescuesecurity : STDOUT(Success)
    Uninstalling package com.motorola.slpc_sys : STDOUT(Success)
    Uninstalling package com.qualcomm.qti.tetherservice : STDOUT(Success)
    Uninstalling package com.motorola.actions : STDOUT(Success)
    Uninstalling package com.android.vpndialogs : STDOUT(Success)
    Uninstalling package com.google.android.talk : STDOUT(Success)
    Uninstalling package com.motorola.bach.modemstats : STDOUT(Success)
    Uninstalling package com.android.phone : STDOUT(Success)
    Uninstalling package com.android.shell : STDOUT(Success)
    Uninstalling package com.android.wallpaperbackup : STDOUT(Success)
    Uninstalling package com.android.providers.blockednumber : STDOUT(Success)
    Uninstalling package com.android.providers.userdictionary : STDOUT(Success)
    Uninstalling package com.android.emergency : STDOUT(Success)
    Uninstalling package com.motorola.cameraone : STDOUT(Success)
    Uninstalling package com.google.android.inputmethod.japanese : STDOUT(Success)
    Uninstalling package com.android.location.fused : STDOUT(Success)
    Uninstalling package com.android.systemui : STDOUT(Success)
    Uninstalling package com.motorola.motodisplay : STDOUT(Success)
    Uninstalling package com.motorola.easyprefix : STDOUT(Success)
    Uninstalling package com.android.bluetoothmidiservice : STDOUT(Success)
    Uninstalling package com.example.platform_channels_benchmarks : STDOUT(Success)
    Uninstalling package com.android.bluetooth : STDOUT(Success)
    Uninstalling package com.qualcomm.timeservice : STDOUT(Success)
    Uninstalling package com.qualcomm.atfwd : STDOUT(Success)
    Uninstalling package com.motorola.config.wifi : STDOUT(Success)
    Uninstalling package com.android.providers.contacts : STDOUT(Success)
    Uninstalling package com.android.captiveportallogin : STDOUT(Success)
    Uninstalling package com.google.android.inputmethod.latin : STDOUT(Success)
    Uninstalling package com.motorola.android.providers.chromehomepage : STDOUT(Success)
    Uninstalling package com.motorola.imagertuning_athene : STDOUT(Success)
    Device recovery: deleting 3P packages...
    Device recovery: rebooting...'
     7775 755 2023-10-27 16:41:44.486 E: device doctor recovery error: b'* daemon not running; starting now at tcp:5037\n* daemon started successfully'
    
  14. godofredoc commented on Oct 27, 2023

    @godofredoc
    Contributor

    The problem is here https://cs.opensource.google/flutter/cocoon/+/main:cipd_packages/device_doctor/lib/src/android_device.dart;l=431?q=deleting%20package%20caches&ss=flutter%2Fcocoon

    The apps are not being uninstalled because they are returning package names like package:com.example.platform_channels_benchmarks and the filter is using file paths. Furthermore as we are requesting only third party packages we can delete all of them without filters.

  15. godofredoc commented on Oct 28, 2023

    @godofredoc
    Contributor

    Tests installing the following packages are not cleaning up the app:

    • package:com.example.image_list
    • package:com.yourcompany.integration_ui
    • package:com.example.app
    • package:com.example.platform_channels_benchmarks
  16. godofredoc commented on Oct 30, 2023

    @godofredoc
    Contributor

    Bug to track cleaning up on tests: #137555

  17. added
    P2Important issues not at the top of the work list
    and removed
    P1High-priority issues at the top of the work list
    on Oct 30, 2023
  18. yusuf-goog commented on Oct 30, 2023

    @yusuf-goog
    Contributor

    Recovery code is running and cleaning up devices. closing.

  19. github-actions commented on Nov 13, 2023

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  20. locked as resolved and limited conversation to collaborators on Nov 13, 2023
  21. moved this from Fixed to Verified in [DEPRECATED] Infra Ticket Queueon Nov 27, 2023
  22. moved this from Verified to To be archived in [DEPRECATED] Infra Ticket Queueon Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P2Important issues not at the top of the work listinfra: device labInfrastructure device lab problems

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions