Skip to content

Mac_staging plugin_test_ios keeps failing in staging #88794

Closed
#90288
Closed
Mac_staging plugin_test_ios keeps failing in staging#88794
#90288
Assignees
keyonghan
Labels
P1High-priority issues at the top of the work listpackageflutter/packages repository. See also p: labels.platform-macosBuilding on or for macOS specificallyteam-infraOwned by Infrastructure team

Description

@keyonghan
keyonghan
opened on Aug 24, 2021

Mac_staging plugin_test_ios was recently enabled in staging, but keeps failing with the following log.

Related: #86802
Example build: https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_staging%20plugin_test_ios/58/overview

Seems we need a higher version of xcode:

[plugin_test_ios] [STDOUT] stderr: [  +20 ms] Found "Xcode 12.0, Build version 12A7209". Xcode 12.0.1 or greater is required to develop for iOS.
[plugin_test_ios] [STDOUT] stdout: [   +3 ms] "flutter ios" took 5,064ms.
[plugin_test_ios] [STDOUT] stderr: [   +3 ms] Encountered error while building for device.
[plugin_test_ios] [STDOUT] stderr: [        ] 
[plugin_test_ios] [STDOUT] stderr:            #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
[plugin_test_ios] [STDOUT] stderr:            #1      _BuildIOSSubCommand.runCommand (package:flutter_tools/src/commands/build_ios.dart:277:7)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #2      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1125:27)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #3      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #4      CommandRunner.runCommand (package:args/command_runner.dart:209:13)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #5      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:288:9)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #7      FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:236:5)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #8      run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #10     main (package:flutter_tools/executable.dart:92:3)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>

Activity

  1. keyonghan commented on Aug 24, 2021

    @keyonghan
    keyonghan
    on Aug 24, 2021
    ContributorAuthor

    which xcode version shall we use here? /cc @jmagman @stuartmorgan

  2. keyonghan
    added
    plugin
    team-infraOwned by Infrastructure team
    on Aug 24, 2021

  3. jmagman commented on Aug 24, 2021

    @jmagman
    jmagman
    on Aug 24, 2021
    Member

    @keyonghan
    12c* https://github.com/flutter/infra/pull/358

    The engine needs a different version than the other builders. Maybe we need an extra tag on the Xcode cipd package to make that clearer?

  4. keyonghan commented on Aug 24, 2021

    @keyonghan
    keyonghan
    on Aug 24, 2021
    ContributorAuthor

    Thanks. Yeah, seems now we use only specific version numbers. It would be helpful at least we put notes somewhere (not sure if we already have that info somewhere though).

    I will submit a cl to include the 12c version.

  5. keyonghan commented on Aug 24, 2021

    @keyonghan
    keyonghan
    on Aug 24, 2021
    ContributorAuthor

    Does this test need a device?
    https://ci.chromium.org/ui/p/flutter/builders/staging/Mac_staging%20plugin_test_ios/65/overview

    [plugin_test_ios] [STDOUT] stdout: [   +5 ms] executing: security find-identity -p codesigning -v
[plugin_test_ios] [STDOUT] stdout: [  +21 ms]      0 valid identities found
[plugin_test_ios] [STDOUT] stderr: [   +3 ms] ════════════════════════════════════════════════════════════════════════════════
[plugin_test_ios] [STDOUT] stderr:            No valid code signing certificates were found
[plugin_test_ios] [STDOUT] stderr:            You can connect to your Apple Developer account by signing in with your Apple ID
[plugin_test_ios] [STDOUT] stderr:            in Xcode and create an iOS Development Certificate as well as a Provisioning 
[plugin_test_ios] [STDOUT] stderr:            Profile for your project by:
[plugin_test_ios] [STDOUT] stderr:              1- Open the Flutter project's Xcode target with
[plugin_test_ios] [STDOUT] stderr:                   open ios/Runner.xcworkspace
[plugin_test_ios] [STDOUT] stderr:              2- Select the 'Runner' project in the navigator then the 'Runner' target
[plugin_test_ios] [STDOUT] stderr:                 in the project settings
[plugin_test_ios] [STDOUT] stderr:              3- Make sure a 'Development Team' is selected under Signing & Capabilities > Team. 
[plugin_test_ios] [STDOUT] stderr:                 You may need to:
[plugin_test_ios] [STDOUT] stderr:                     - Log in with your Apple ID in Xcode first
[plugin_test_ios] [STDOUT] stderr:                     - Ensure you have a valid unique Bundle ID
[plugin_test_ios] [STDOUT] stderr:                     - Register your device with your Apple Developer Account
[plugin_test_ios] [STDOUT] stderr:                     - Let Xcode automatically provision a profile for your app
[plugin_test_ios] [STDOUT] stderr:              4- Build or run your project again
[plugin_test_ios] [STDOUT] stderr:              5- Trust your newly created Development Certificate on your iOS device
[plugin_test_ios] [STDOUT] stderr:                 via Settings > General > Device Management > [your new certificate] > Trust
[plugin_test_ios] [STDOUT] stderr: 
[plugin_test_ios] [STDOUT] stderr:            For more information, please visit:
[plugin_test_ios] [STDOUT] stderr:              https://developer.apple.com/library/content/documentation/IDEs/Conceptual/
[plugin_test_ios] [STDOUT] stderr:              AppDistributionGuide/MaintainingCertificates/MaintainingCertificates.html
[plugin_test_ios] [STDOUT] stderr: 
[plugin_test_ios] [STDOUT] stderr:            Or run on an iOS simulator without code signing
[plugin_test_ios] [STDOUT] stderr:            ════════════════════════════════════════════════════════════════════════════════
[plugin_test_ios] [STDOUT] stdout: [   +3 ms] "flutter ios" took 6,186ms.
[plugin_test_ios] [STDOUT] stderr: [   +3 ms] No development certificates available to code sign app for device deployment
[plugin_test_ios] [STDOUT] stderr: [        ] 
[plugin_test_ios] [STDOUT] stderr:            #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
[plugin_test_ios] [STDOUT] stderr:            #1      _chooseSigningIdentity (package:flutter_tools/src/ios/code_signing.dart:217:5)
[plugin_test_ios] [STDOUT] stderr:            #2      getCodeSigningIdentityDevelopmentTeam (package:flutter_tools/src/ios/code_signing.dart:153:41)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #3      buildXcodeProject (package:flutter_tools/src/ios/mac.dart:188:26)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #4      _BuildIOSSubCommand.runCommand (package:flutter_tools/src/commands/build_ios.dart:264:37)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #5      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1125:27)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #7      CommandRunner.runCommand (package:args/command_runner.dart:209:13)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #8      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:288:9)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #10     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:236:5)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #11     run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #12     AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr:            #13     main (package:flutter_tools/executable.dart:92:3)
[plugin_test_ios] [STDOUT] stderr:            <asynchronous suspension>
[plugin_test_ios] [STDOUT] stderr: 
[plugin_test_ios] [STDOUT] stderr:

  6. jmagman commented on Aug 25, 2021

    @jmagman
    jmagman
    on Aug 25, 2021
    Member

    I guess this was only ever run in the devicelab which has the provisioning profiles set up. That test is missing a --no-codesign. I'll put that out for review (sorry about that!)

  7. jmagman commented on Aug 25, 2021

    @jmagman
    jmagman
    on Aug 25, 2021
    Member

    It got further...

    [plugin_test_ios] [STDOUT] stdout: 00:09 +0 -1: loading /opt/s/w/ir/x/t/flutter_devicelab_plugin_test.hMgce9/plugintestapp/test/widget_test.dart [E]                                                                                      
[plugin_test_ios] [STDOUT] stdout:   Failed to load "/opt/s/w/ir/x/t/flutter_devicelab_plugin_test.hMgce9/plugintestapp/test/widget_test.dart": Shell subprocess ended cleanly. Did main() call exit()?
[plugin_test_ios] [STDOUT] stdout: 
[plugin_test_ios] [STDOUT] stdout: 00:09 +0 -1: Some tests failed.                                                                                                                                                                        
[plugin_test_ios] [STDOUT] "/opt/s/w/ir/x/w/recipe_cleanup/tmpfkpQPr/flutter sdk/bin/flutter" exit code: 1

    https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8837907446638483457/+/u/run_plugin_test_ios/test_stdout

  8. jmagman commented on Aug 27, 2021

    @jmagman
    jmagman
    on Aug 27, 2021
    Member

    I'm able to reproduce this. It's only on the macOS part of the test.

    flutter/dev/devicelab/bin/tasks/plugin_test_ios.dart

    Line 12 in 0c81077

    PluginTest('macos', <String>['--platforms=macos']),

    It only happens when the app is built right before flutter test is run. If I comment out the app.build(buildTarget) lines the tests pass.

    flutter/dev/devicelab/lib/tasks/plugin_tests.dart

    Lines 53 to 56 in 413f88f

    section('Build app');
    await app.build(buildTarget);
    section('Test app');
    await app.test();

    If I flutter clean after the build and then flutter test it also passes.

  9. jmagman commented on Aug 27, 2021

    @jmagman
    jmagman
    on Aug 27, 2021
    Member

    When it works the PACKAGE_CONFIG is .packages. When it fails, the PACKAGE_CONFIG is /absolute/path/to/plugintestapp/.dart_tool/package_config.json.

  10. TahaTesser
    added
    passed first triage
    on Sep 6, 2021
  11. jmagman
    added
    platform-macosBuilding on or for macOS specifically
    on Sep 8, 2021
  12. jmagman
    changed the title [-]Mac_staging plugin_test_ios keeps failing in staging[/-] [+]Mac_staging plugin_test_ios keeps failing in staging building and testing a new macOS app[/+] on Sep 8, 2021

  13. stuartmorgan-g commented on Sep 9, 2021

    @stuartmorgan-g
    stuartmorgan-g
    on Sep 9, 2021
    Contributor

    It only happens when the app is built right before flutter test is run. If I comment out the app.build(buildTarget) lines the tests pass.

    Does the app include path_provider by any chance? This sounds suspiciously like #87991 (comment)

  14. jmagman commented on Sep 9, 2021

    @jmagman
    jmagman
    on Sep 9, 2021
    Member

    Does the app include path_provider by any chance?

    Wow, yes it does!

    flutter/dev/devicelab/lib/tasks/plugin_tests.dart

    Line 52 in 413f88f

    await app.addPlugin('path_provider');

  15. 15 remaining items

  16. stuartmorgan-g
    changed the title [-]Mac_staging plugin_test_ios keeps failing in staging building and testing a new macOS app[/-] [+]Mac_staging plugin_test_ios keeps failing in staging[/+] on Oct 1, 2021
  17. stuartmorgan-g
    reopened this on Oct 1, 2021
  18. stuartmorgan-g
    added a commit that references this issue on Oct 1, 2021

    Update minimum iOS version in plugin_test

    b7db947

  19. keyonghan commented on Oct 1, 2021

    @keyonghan
    keyonghan
    on Oct 1, 2021
    ContributorAuthor

    That's a new unrelated failure:

     Task result:
 {
   "success": false,
   "reason": "Update this test to match plugin minimum ios deployment version"
 }

    I'll restore the original issue title and re-open.

    FWIW: checked commits prior to the above fix, this error seems have been there consistently for a while.

  20. jmagman commented on Oct 5, 2021

    @jmagman
    jmagman
    on Oct 5, 2021
    Member

    Another one: #91324

  21. jmagman commented on Oct 7, 2021

    @jmagman
    jmagman
    on Oct 7, 2021
    Member

    It went green! https://ci.chromium.org/p/flutter/builders/staging/Mac_staging%20plugin_test_ios/451

    @keyonghan How can we move this into prod?

  22. keyonghan
    self-assigned this
    on Oct 11, 2021

  23. keyonghan commented on Oct 11, 2021

    @keyonghan
    keyonghan
    on Oct 11, 2021
    ContributorAuthor

    It went green! https://ci.chromium.org/p/flutter/builders/staging/Mac_staging%20plugin_test_ios/451

    @keyonghan How can we move this into prod?

    I will wait for 50 runs to make sure no flake happens before moving it to prod. Assigning to myself to monitor this and move later.

  24. keyonghan commented on Oct 12, 2021

    @keyonghan
    keyonghan
    on Oct 12, 2021
    ContributorAuthor

    Test has passed 50 consecutive runs.
    Bring this up in prod as bringup: true first to make sure no failure breaking the tree: #91713

    This later can be promoted as non-flaky automatically, or we can push manually whenever we see a successful run as we have validated the flakiness in staging.

  25. keyonghan commented on Oct 15, 2021

    @keyonghan
    keyonghan
    on Oct 15, 2021
    ContributorAuthor

    This test has been enabled in prod. Closing.

  26. keyonghan
    closed this as completedon Oct 15, 2021
  27. clocksmith
    added a commit that references this issue on Oct 29, 2021

    Fix Dart plugin registrant interaction with 'flutter test' (flutter#9…

    5530976

  28. github-actions commented on Oct 29, 2021

    @github-actions
    github-actionsbot
    on Oct 29, 2021

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  29. github-actions
    locked as resolved and limited conversation to collaborators on Oct 29, 2021
  30. flutter-triage-bot
    added
    packageflutter/packages repository. See also p: labels.
    on Jul 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listpackageflutter/packages repository. See also p: labels.platform-macosBuilding on or for macOS specificallyteam-infraOwned by Infrastructure team

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions