flutter_gallery's driver test doesn't show image resources

To reproduce, run flutter drive --profile -t test_driver/transitions_perf.dart inside flutter_gallery on iPhone6S. Pay attention to Shrine and Contact inside Studies.

This issue makes our benchmark inaccurate as it doesn't capture the possibly slow image loading jank in #28113 (comment)

Below is a screenshot of Contact when the flutter_gallery is opened by flutter drive:

For comparison, here's a screenshot if I open flutter_gallery normally without flutter_drive:

Seems to be related with #21931. Here's my flutter doctor -v:

liyuqian-macbookpro2:flutter_gallery liyuqian$ flutter doctor -v
[✓] Flutter (Channel master, v1.2.1-pre.717, on Mac OS X 10.13.6 17G5019, locale en-US)
    • Flutter version 1.2.1-pre.717 at /Users/liyuqian/flutter/flutter
    • Framework revision d0f89c1190 (2 days ago), 2019-04-05 18:55:33 -0700
    • Engine revision 6bc33b5e14
    • Dart version 2.2.1 (build 2.2.1-dev.3.0 None)


[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/liyuqian/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
    • All Android licenses accepted.

[✓] iOS toolchain - develop for iOS devices (Xcode 10.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 10.1, Build version 10B61
    • ios-deploy 1.9.4
    • CocoaPods version 1.5.3

[✓] Android Studio (version 3.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 29.0.1
    • Dart plugin version 173.4700
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)

[✓] IntelliJ IDEA Community Edition (version 2018.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 31.3.1
    • Dart plugin version 181.4203.498

[✓] Connected device (1 available)
    • SixDegreesOfSeparation • 2b398993d86f42b05511b330a89ae65289832852 • ios • iOS 12.1.4

• No issues found!

I just tested it with MotoG4 on Linux. The images are missing too. Here's my flutter doctor -v:

liyuqian@liyuqian:~/flutter/flutter/packages/flutter$ flutter doctor -v
[✓] Flutter (Channel revert_warm_up_check, v1.2.1-pre.678, on Linux, locale en_US.UTF-8)
    • Flutter version 1.2.1-pre.678 at /usr/local/google/home/liyuqian/flutter/flutter
    • Framework revision 2b1cb9b607 (4 days ago), 2019-04-03 13:32:21 -0700
    • Engine revision cc115500b9
    • Dart version 2.2.1 (build 2.2.1-dev.3.0 None)

 
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /usr/local/google/home/liyuqian/android/Sdk/
    • Android NDK at /usr/local/google/home/liyuqian/android/Sdk/ndk-bundle
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /usr/local/google/home/liyuqian/android/Sdk/
    • ANDROID_SDK_ROOT = /usr/local/google/home/liyuqian/android/Sdk/
    • Java binary at: /opt/android-studio-with-blaze-3.3/jre/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)
    • All Android licenses accepted.

[✓] Android Studio (version 3.3)
    • Android Studio at /opt/android-studio-with-blaze-3.3
    • Flutter plugin version 32.0.1
    • Dart plugin version 182.5124
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1248-b01)

[✓] IntelliJ IDEA Community Edition (version 2018.3.backup)
    • IntelliJ at /opt/intellij-ce-2018.3
    • Flutter plugin version 32.0.3
    • Dart plugin version 183.5153.38

[✓] IntelliJ IDEA Community Edition (version 2018.2)
    • IntelliJ at /opt/intellij-ce-2018.2
    • Flutter plugin version 31.1.3
    • Dart plugin version 182.5124

[✓] VS Code (version 1.32.3)
    • VS Code at /usr/share/code
    • Flutter extension version 2.25.0

[✓] Connected device (1 available)
    • Moto G 4 • ZY2245H6KS • android-arm • Android 7.0 (API 24)

@liyuqian

@liyuqian Has it worked now with latest flutter SDK version?

@jonahwilliams

Unfortunately no. I just confirmed that the issue is still there in 3b945da

@jonahwilliams : do you have any idea why flutter run has the images while flutter drive don't?

No, I don't. This will need some time spent investigating. Possibly related to the reports that isolates don't work also.

I thought the reason is Isolated.spawn can NOT work in flutter drive.
referring to: #35720

@leishuai

Thank you @leishuai for your detailed explanation in #35720 (comment) !

CC @chunhtai from #35720

Apparently you can unpause the isolate in the observatory and then the image should show up:
#24703 (comment)

That's not the fix though.

I think this whole issue is part of #24703 . The flutter team should fix this issue immediately.

show image resources when using gallery's driver

Here's workaround that makes those images appear in the aforementioned tests:

vishna@2f858fc

Flutter team didn't have time to fix the issue #24703 since Nov 2018 so unsure how critical working instrumentation tests are.

It is a really bad bug to have in my opinion and it seems to be only caused by an isolate being paused for whatever reasons 🤷‍♀️

@CareF

CC @CareF . If you're going to write a host-independent test for flutter_gallery, keep an eye on whether it shows the image and avoids this issue.

@liyuqian

I can confirm that the test app generated by e2e doesn't have this issue if directly run from the device, but if we are running e2e test using flutter drive we still go into this. @liyuqian

A demo in
#62064 by dev/integration_tests/flutter_gallery/test/transitions_perf_e2e.dart. It's not yet a complete migration and error at some step due to #61458, but is good enough to show this problem.

Running the test by flutter drive command will produce the same result as the first figure in the description, lacking some figures. This means that with the e2e solution, we still can't do this on device lab. firebase test lab may be fine.

Running the test app generated by flutter drive on device manually works.

@CareF

@CareF in that case, I wonder if flutter-team-archive/plugins#2873 can only be used with flutter drive, or could it also be used with flutter run? In the worst case, I think we can just let the app print out the result, and parse it either through the stdout of flutter run, or from something like adb logcat. I believe all the test platforms (e.g., Firebase test lab) must persist the running log in some way?

@CareF

@CareF in that case, I wonder if flutter/plugins#2873 can only be used with flutter drive, or could it also be used with flutter run? In the worst case, I think we can just let the app print out the result, and parse it either through the stdout of flutter run, or from something like adb logcat. I believe all the test platforms (e.g., Firebase test lab) must persist the running log in some way?

Yes flutter-team-archive/plugins#2873 is just an adaptor to flutter drive, and may be used in the future for Timeline collection on e2e. If we are to print the result we don't need flutter-team-archive/plugins#2873.
My plan for the place of adding other metric report channel is https://github.com/flutter/flutter/pull/61509/files#diff-a64ae307a67762bf902f9a7a714b602eR84-R96

#61841 Will fix this.

My fix for this in #61841 was reverted with #62239. So this bug is still open.

If anyone has tips on how to run the devicelab tests, I'm happy to try to debug the issues so that this fix can land.

There are some instructions in https://github.com/flutter/flutter/tree/master/dev/devicelab

two of the specific tests that failed were new_gallery_ios__transition_perf and new_gallery__transition_perf

Fix relanded in #65703

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

flutter_gallery's driver test doesn't show image resources #30641

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

flutter_gallery's driver test doesn't show image resources #30641

Description

Activity

liyuqian commented on Apr 7, 2019

liyuqian commented on Apr 7, 2019

leishuai commented on Jul 17, 2019

liyuqian commented on Jul 17, 2019

jonahwilliams commented on Jul 17, 2019

leishuai commented on Jul 18, 2019

liyuqian commented on Jul 18, 2019

vishna commented on Aug 26, 2019

The-Redhat commented on Aug 26, 2019

vishna commented on Aug 26, 2019

4 remaining items

liyuqian commented on Jul 22, 2020

CareF commented on Jul 22, 2020

liyuqian commented on Jul 22, 2020

CareF commented on Jul 22, 2020

dnfield commented on Jul 23, 2020

speaking-in-code commented on Jul 27, 2020

jonahwilliams commented on Jul 27, 2020

dnfield commented on Aug 17, 2022

github-actions commented on Aug 31, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions