Repository navigation
Devicelab tasks flaking due to insufficient storage #135035
Copy link
Copy link
Closed
Labels
P2Important issues not at the top of the work listImportant issues not at the top of the work listinfra: device labInfrastructure device lab problemsInfrastructure device lab problems
Description
Is there an existing issue for this?
- I have searched the existing infra issues
Type of Request
bug
Infrastructure Environment
Devicelab
What is happening?
Many Device lab tasks are flaking due to insufficient storage errors, see: https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8769511480680570273/+/u/run_opacity_peephole_grid_of_alpha_savelayers_perf__e2e_summary/test_stdout
[2023-09-19 09:05:37.901796] [STDOUT] stderr:
[2023-09-19 09:05:37.901814] [STDOUT] stderr: adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Package couldn't be installed in /data/app/com.example.macrobenchmarks-1: Creating application package com.example.macrobenchmarks failed]
[2023-09-19 09:05:37.901836] [STDOUT] stdout: [ ] Warning: Failed to install APK.
[2023-09-19 09:05:37.902070] [STDOUT] stderr: [ ] Application failed to start on attempt: 2
[2023-09-19 09:05:37.902695] [STDOUT] stdout: [ ] Stopping app 'app-profile.apk' on Moto G 4.
[2023-09-19 09:05:37.902745] [STDOUT] stdout: [ ] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s ZY223HK9P7 shell am force-stop com.example.macrobenchmarks
[2023-09-19 09:05:38.788481] [STDOUT] stdout: [ +885 ms] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s ZY223HK9P7 shell pm list packages com.example.macrobenchmarks
[2023-09-19 09:05:39.733880] [STDOUT] stdout: [ +945 ms] Installing APK.
[2023-09-19 09:05:39.734219] [STDOUT] stdout: [ ] Installing build/app/outputs/flutter-apk/app-profile.apk...
[2023-09-19 09:05:39.734266] [STDOUT] stdout: [ ] executing: /opt/s/w/ir/cache/android/sdk/platform-tools/adb -s ZY223HK9P7 install -t -r /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk
[2023-09-19 09:05:54.183736] [STDOUT] stdout: [+14448 ms] Performing Streamed Install
[2023-09-19 09:05:54.183857] [STDOUT] stdout:
[2023-09-19 09:05:54.183881] [STDOUT] stdout: adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Package couldn't be installed in /data/app/com.example.macrobenchmarks-1: Creating application package com.example.macrobenchmarks failed]
[2023-09-19 09:05:54.183906] [STDOUT] stdout: [ ] Installing build/app/outputs/flutter-apk/app-profile.apk... (completed in 14.4s)
[2023-09-19 09:05:54.183982] [STDOUT] stderr: [ ] Error: ADB exited with exit code 1
[2023-09-19 09:05:54.184001] [STDOUT] stderr: [ ] Performing Streamed Install
[2023-09-19 09:05:54.184018] [STDOUT] stderr:
[2023-09-19 09:05:54.184037] [STDOUT] stderr: adb: failed to install /opt/s/w/ir/x/w/recipe_cleanup/tmp8szcrs63/flutter sdk/dev/benchmarks/macrobenchmarks/build/app/outputs/flutter-apk/app-profile.apk: Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Package couldn't be installed in /data/app/com.example.macrobenchmarks-1: Creating application package com.example.macrobenchmarks failed]
[2023-09-19 09:05:54.184073] [STDOUT] stdout: [ ] Warning: Failed to install APK.
[2023-09-19 09:05:54.184351] [STDOUT] stderr: [ ] Application failed to start on
Steps to reproduce
No response
Expected results
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P2Important issues not at the top of the work listImportant issues not at the top of the work listinfra: device labInfrastructure device lab problemsInfrastructure device lab problems
Type
Projects
- StatusShow more project fieldsTo be archived
Activity
drewroengoogle commentedon Sep 19, 2023
Many of them seem to be occurring on
linux-15link, @yusuf-goog Would it make sense to quarantine this bot?yusuf-goog commentedon Sep 19, 2023
let me look at it and free up space on the attached device.
yusuf-goog commentedon Sep 19, 2023
quarantined the host. Looking at freeing up space on the device.
yusuf-goog commentedon Sep 19, 2023
Uninstalled all third party packages:
adb shell pm list packages -3 | cut -d':' -f2 | tr '\r' ' ' | xargs -r -n1 -t adb uninstallNot much direct use of space. Before:
after:
yusuf-goog commentedon Sep 19, 2023
cc @reidbaker
yusuf-goog commentedon Sep 19, 2023
@jonahwilliams the device has ~8gb in /data, which should be enough to install any app?
I think we might be hitting a different issue which just manifests as
INSTALL_FAILED_INSUFFICIENT_STORAGE:https://stackoverflow.com/questions/4709137/solution-to-install-failed-insufficient-storage-error-on-android
Looking into this.
yusuf-goog commentedon Sep 19, 2023
Ran:
pm uninstall com.example.macrobenchmarks-1rm -rf /data/app/com.example.macrobenchmarks*Unquarantined the host, looking to see if we still get the failure.
yusuf-goog commentedon Sep 20, 2023
builds are running successfully on linux-15 now, including a few that had previously failed with out of space errors. Closing this, reopen if it happens again.
74 remaining items
godofredoc commentedon Oct 27, 2023
Things to check:
godofredoc commentedon Oct 27, 2023
Cleanup logic is running on_after_task with a task failure. The following is a sample:
godofredoc commentedon Oct 27, 2023
The problem is here https://cs.opensource.google/flutter/cocoon/+/main:cipd_packages/device_doctor/lib/src/android_device.dart;l=431?q=deleting%20package%20caches&ss=flutter%2Fcocoon
The apps are not being uninstalled because they are returning package names like
package:com.example.platform_channels_benchmarksand the filter is using file paths. Furthermore as we are requesting only third party packages we can delete all of them without filters.Remove all third_party apps in recovery mode. (#3197)
godofredoc commentedon Oct 28, 2023
Tests installing the following packages are not cleaning up the app:
godofredoc commentedon Oct 30, 2023
Bug to track cleaning up on tests: #137555
yusuf-goog commentedon Oct 30, 2023
Recovery code is running and cleaning up devices. closing.
github-actions commentedon Nov 13, 2023
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.