Skip to content

E2E iOS Documentation unclear #48600

Closed
#74428
Closed
E2E iOS Documentation unclear#48600
#74428
Labels
f: integration_testThe flutter/packages/integration_test pluginpackageflutter/packages repository. See also p: labels.

Description

@gaaclarke
gaaclarke
opened on Jan 10, 2020

Document: https://github.com/flutter/plugins/blob/master/packages/e2e/README.md

  1. The docs say "Create a test target" but it doesn't specify if it should be a UI test target or Unit Test target.
  2. It isn't specified when you should have to call flutter build ios -t, anytime the test dart file is edited? Anytime any code changes anywhere?

Activity

  1. gaaclarke
    added
    plugin
    documentation
    on Jan 10, 2020

  2. gaaclarke commented on Jan 10, 2020

    @gaaclarke
    gaaclarke
    on Jan 10, 2020
    MemberAuthor

    cc @mklim @collinjackson

  3. mklim
    added
    f: integration_testThe flutter/packages/integration_test plugin
    on Jan 10, 2020

  4. SiTianyu commented on May 14, 2020

    @SiTianyu
    SiTianyu
    on May 14, 2020

    Yes, I have same trouble on here.

  5. gaaclarke commented on May 14, 2020

    @gaaclarke
    gaaclarke
    on May 14, 2020
    MemberAuthor

    @SiTianyu I have a working example here: https://github.com/flutter/packages/tree/master/packages/pigeon/e2e_tests/test_objc

    I don't know if that will help you.

  6. workerbee22 commented on Jan 8, 2021

    @workerbee22
    workerbee22
    on Jan 8, 2021

    @gaaclarke Both this e2e and the newer integration_test documentation for iOS setup is the same everywhere ... and it is all unclear and very very brief.

    Related issue here, where documentation is followed, but the IntegrationTestIosTest.h file not found: #72876

  7. letsar commented on Jan 21, 2021

    @letsar
    letsar
    on Jan 21, 2021
    Contributor

    Hi,
    I tried to create an instrumented version of my app to send it to Firebase but it didn't work, and as I'm not an iOS developer, I'm kind of lost here.

    The first issue I encountered was the #72876. I resolved this issue by writing the following code in my RunnerTests.m :

    #import <Foundation/Foundation.h>
#import <XCTest/XCTest.h>

@interface IntegrationTestIosTest : NSObject

- (BOOL)testIntegrationTest:(NSString **)testResult;

@end

@interface RunnerTests : XCTestCase

@end

@implementation RunnerTests

-(void)testIntegrationTest {
  NSString *testResult;
  IntegrationTestIosTest *integrationTestIosTest = [[IntegrationTestIosTest alloc] init];
  BOOL testPass = [integrationTestIosTest testIntegrationTest:&testResult];
  XCTAssertTrue(testPass, @"%@", testResult);
}

@end

    Then I was able to do Product > Test, but the test failed with an unrecognized selector error. I suspect that Xcode did not link my integration test target file in the process.
    I was able to run

    flutter build ios integration_test/foo_test.dart

    but I don't know what to do after that in order to create the . xctestrun file necessary to run or upload the instrumented app to Firebase Test Lab.

    @dnfield, @jiahaog you seemed to have work on this plugin, have you a better idea on what we would need to do here ? I could make a PR to the Readme in order to improve the doc if I can make it work on my side.

  8. clement-buchart commented on Jan 21, 2021

    @clement-buchart
    clement-buchart
    on Jan 21, 2021

    Hello,

    It did the setup for the project I'm working on this Monday.
    Not beeing an iOS dev I too found the documentation, succinct :)
    Here are some pointer that helped me, in addition to the doc at https://pub.dev/packages/integration_test :

    #import <XCTest/XCTest.h>
#import <integration_test/IntegrationTestIosTest.h>

INTEGRATION_TEST_IOS_RUNNER(IntegrationTests)
    • I set the iOS Deployment Target to iOS 13.0 in the Build Settings tab of the "Unit Testing Bundle" target in Xcode
      image

    That for the set-up, then to run the tests (using firebase test lab), I do :
    In flutter directory :

    • flutter build ios integration_test/my_test.dart -release

    In ios directory

    • xcodebuild -workspace Runner.xcworkspace -scheme Runner -config Flutter/Release.xcconfig -derivedDataPath /tmp/build -sdk iphoneos build-for-testing
    • zip -r MyTests.zip /tmp/build/Build/Products/Release-iphoneos /tmp/build/Build/Products/Runner_iphoneos14.3-arm64.xctestrun
    • gcloud firebase test ios run --test MyTests.zip --device model=iphone11pro,version=14.1,locale=fr_FR,orientation=portrait

    You need to build the tests with -config Flutter/Release.xcconfig to run them on iOS >= 14 because of https://flutter.dev/docs/development/ios-14

    You can of course change the third command to run the tests on a device or just use Xcode GUI to run the tests.

    All this was done on a M1 Mac Mini, if anyone is wondering if that's possible.

  9. letsar commented on Jan 21, 2021

    @letsar
    letsar
    on Jan 21, 2021
    Contributor

    Thanks @clement-buchart for this amazing answer, I need to try the last commands :-)

  10. letsar commented on Jan 21, 2021

    @letsar
    letsar
    on Jan 21, 2021
    Contributor

    I follow your tutorial @clement-buchart and it worked perfectly! Thank you very much.
    I also chose UI Testing Bundle the first time, that's why it certainly didn't worked!
    Do you mind if I make a PR using what you said, so that it can profit to every one?

  11. clement-buchart commented on Jan 21, 2021

    @clement-buchart
    clement-buchart
    on Jan 21, 2021 via email

  12. workerbee22 commented on Jan 21, 2021

    @workerbee22
    workerbee22
    on Jan 21, 2021

    Thanks @clement-buchart for debugging this for us all. I also made that fundamental mistake of selecting in Xcode the "UI Testing Bundle" (because the official documentation doe not specify which sort of test in Xcode).

    This is the sort of detail we need for iOS in the official documentation 👍

  13. dnfield
    closed this as completedin #74428on Jan 22, 2021

  14. adityadroid commented on Apr 2, 2021

    @adityadroid
    adityadroid
    on Apr 2, 2021

    Hello,

    It did the setup for the project I'm working on this Monday.
    Not beeing an iOS dev I too found the documentation, succinct :)
    Here are some pointer that helped me, in addition to the doc at pub.dev/packages/integration_test :

    * I ignored the part about "linking all of the plugins dynamically" (but my Podfile already contained the use_framework! bit)

* I ran flutter build ios integration_test/foo_test.dart as instructed

* I added a "Unit Testing Bundle" target in XCode, as instructed (initially went with a "UI Testing Bundle" as it sounded closer to what i wanted, but after checking out the example here [https://github.com/flutter/plugins/tree/master/packages/integration_test/example](https://github.com/flutter/plugins/tree/master/packages/integration_test/example?rgh-link-date=2021-01-21T16%3A37%3A06Z) it appeared I was wrong). I named mine "Integration Test"

* I then hit the import problem. The key here was to setup build configuration as blaugold mention in #72876. You can do this in the _Info_ tab of the Runner project in Xcode. By default the Test target config is _None_, you need to use the same than the main flutter project.
  ![image](https://user-images.githubusercontent.com/30914745/105380671-7492b000-5c0e-11eb-9aa4-8b6d40eca914.png)

* Once you do this the code specified in the doc works fine (you need to change RunnerTests to the name you gave the "Unit Testing Bundle" Target you created just before )

    #import <XCTest/XCTest.h>
#import <integration_test/IntegrationTestIosTest.h>

INTEGRATION_TEST_IOS_RUNNER(IntegrationTests)

    * I set the iOS Deployment Target to iOS 13.0 in the _Build Settings_ tab of the "Unit Testing Bundle" target in Xcode
  ![image](https://user-images.githubusercontent.com/30914745/105380877-a9066c00-5c0e-11eb-9ab3-d07cd6fa3b58.png)

    That for the set-up, then to run the tests (using firebase test lab), I do :
    In flutter directory :

    * `flutter build ios integration_test/my_test.dart -release`

    In ios directory

    * `xcodebuild -workspace Runner.xcworkspace -scheme Runner -config Flutter/Release.xcconfig -derivedDataPath /tmp/build -sdk iphoneos build-for-testing`

* `zip -r MyTests.zip /tmp/build/Build/Products/Release-iphoneos /tmp/build/Build/Products/Runner_iphoneos14.3-arm64.xctestrun`

* `gcloud firebase test ios run --test MyTests.zip --device model=iphone11pro,version=14.1,locale=fr_FR,orientation=portrait`

    You need to build the tests with -config Flutter/Release.xcconfig to run them on iOS >= 14 because of flutter.dev/docs/development/ios-14

    You can of course change the third command to run the tests on a device or just use Xcode GUI to run the tests.

    All this was done on a M1 Mac Mini, if anyone is wondering if that's possible.

    While this did help me I was unable to make it work with flavors. Flutter uses iOS schemes as flavors and I had a separate scheme integration for our test flavor. Here's how I managed to create the test artifacts and run it on Firebase Test Lab

    output="../build/ios_integ"
product="Build/Products"
dev_target="14.3"

#First generate the test flavor build
flutter build ios --release --flavor integration -t integration_test/main_test.dart

#Generate iOS XCTest Package
pushd ios
xcodebuild -workspace Runner.xcworkspace -scheme integration -configuration Release-integration -derivedDataPath $output -sdk iphoneos build-for-testing
popd
cd build/ios_integ/Build/Products
zip -r "ios_tests.zip" "Release-integration-iphoneos" "integration_iphoneos$dev_target-arm64-armv7.xctestrun"

gcloud firebase test ios run --test "ios_tests.zip" --device model=iphone11pro,version=14.1,locale=en_US,orientation=portrait --xcode-version=12.3 --timeout 2m

    • First thing to note here is that while building you have to make sure that you pass the correct scheme via -scheme and also the correct configuration which would be Release-<SCHEME_NAME> based on what the name of your scheme is.

    • Also for some reason the zip command for me kept zipping the artifacts in a subdirectory /Build/Products, causing the tests to fail on Test Lab. That's why I had to use the cd command to get into the directory first.

  15. github-actions commented on Aug 3, 2021

    @github-actions
    github-actionsbot
    on Aug 3, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  16. github-actions
    locked as resolved and limited conversation to collaborators on Aug 3, 2021
  17. flutter-triage-bot
    added
    packageflutter/packages repository. See also p: labels.
    on Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    f: integration_testThe flutter/packages/integration_test pluginpackageflutter/packages repository. See also p: labels.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions