Skip to content

Mac tests are flaking due to missing runtime #99429

Closed
Closed
Mac tests are flaking due to missing runtime#99429
Labels
P0Critical issues such as a build break or regressionc: flakeTests that sometimes, but not always, incorrectly passteam-infraOwned by Infrastructure team

Description

@keyonghan
keyonghan
opened on Mar 2, 2022

The bot build807-m9 starts causing test flake after a release build failure: https://luci-milo.appspot.com/ui/p/flutter/builders/prod/Mac%20beta%20verify_binaries_codesigned/400/overview

Screen Shot 2022-03-02 at 12 44 14 PM

All these failures are missing Runtimes package, e.g. https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/5288/overview

It is possible the cache need a clean up.

Activity

  1. keyonghan
    added
    P1
    c: flakeTests that sometimes, but not always, incorrectly pass
    team: flakes
    team-infraOwned by Infrastructure team
    on Mar 2, 2022

  2. keyonghan commented on Mar 2, 2022

    @keyonghan
    keyonghan
    on Mar 2, 2022
    ContributorAuthor

    Marking as P1 as it causes multiple flakes across different tests.

  3. godofredoc commented on Mar 2, 2022

    @godofredoc
    godofredoc
    on Mar 2, 2022
    Contributor

    aaaah, if those tests are not using the default runtimes it is failing because we don't have support from that yet. We'll need to map a .ci.yaml property to the runtimes property in the xcode module.

  4. keyonghan commented on Mar 3, 2022

    @keyonghan
    keyonghan
    on Mar 3, 2022
    ContributorAuthor

    aaaah, if those tests are not using the default runtimes it is failing because we don't have support from that yet. We'll need to map a .ci.yaml property to the runtimes property in the xcode module.

    I don't think we are using customized runtime for existing tests. All should be using the default ones? /cc @jmagman

    The issue is happening on build811 as well. Almost all flakes on the recent runs are due to these two bots
    Screen Shot 2022-03-03 at 9 59 24 AM

    /cc @CaseyHillers can we move these two bots to staging during investigation?

  5. CaseyHillers
    self-assigned this
    on Mar 3, 2022

  6. jmagman commented on Mar 3, 2022

    @jmagman
    jmagman
    on Mar 3, 2022
    Member

    aaaah, if those tests are not using the default runtimes it is failing because we don't have support from that yet. We'll need to map a .ci.yaml property to the runtimes property in the xcode module.

    I don't think we are using customized runtime for existing tests. All should be using the default ones? /cc @jmagman

    Correct, the framework repo is only using the default iOS 15 runtime. The download of which should be handled by the mac_toolchain populating the Xcode cache. This looks the same as #99039 ?

  7. keyonghan
    added
    passed secondary triage
    on Mar 3, 2022

  8. keyonghan commented on Mar 4, 2022

    @keyonghan
    keyonghan
    on Mar 4, 2022
    ContributorAuthor

    Same issue is happening on build810 too: https://chromium-swarm.appspot.com/bot?id=build810-m9

    All these three bots started hitting runtine package issue after a failure on a releaser builder, e.g.
    Screen Shot 2022-03-03 at 4 32 30 PM

  9. jmagman commented on Mar 4, 2022

    @jmagman
    jmagman
    on Mar 4, 2022
    Member

    When you download a vanilla version of Xcode, there's a default iOS simulator located in Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes (iOS 15 in this case). However when mac_toolchain uploads the Xcode bundle, it strips that runtime out. The runtime is uploaded as a separate cipd package ios_runtime. See https://source.chromium.org/chromium/infra/infra/+/main:go/src/infra/cmd/mac_toolchain/README.md and https://g3doc.corp.google.com/company/teams/chrome/ios/docs/infra/xcode_upload.md?cl=head#upload-ios-runtimes

    So how this should work is when the Xcode cache is first by

      '/opt/s/w/ir/cache/osx_sdk/mac_toolchain',
  'install',
  '-kind',
  'ios',
  '-xcode-version',
  '12d4e',
  '-output-dir',
  '/opt/s/w/ir/cache/osx_sdk/XCode.app',

    it should download three packages, flutter_internal/ios/xcode/mac, flutter_internal/ios/xcode/ios (one of those the Xcode bundle with nothing in that directory), and flutter_internal/ios/xcode/ios_runtime (which just has the runtime, and puts it in the correct Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes directory.

    [P92769 18:56:42.815 action_plan.go:160 I] In root:
[P92769 18:56:42.815 action_plan.go:166 I]   to install:
[P92769 18:56:42.815 action_plan.go:168 I]     flutter_internal/ios/xcode/mac:UCaNcftiBhJwpRpc5DUfKSIw8COAV-Tys6P58EzPEykC
[P92769 18:56:42.815 action_plan.go:168 I]     flutter_internal/ios/xcode/ios:rWdxdSMwqPafHwtN9Uh77do6HVmD6iVatO_XNPbfHvEC
...
[P92815 19:01:13.007 action_plan.go:160 I] In root:
[P92815 19:01:13.007 action_plan.go:166 I]   to install:
[P92815 19:01:13.007 action_plan.go:168 I]     flutter_internal/ios/xcode/ios_runtime:xbExNWHTJfpNf8vDNXhmLNx9sjfFvAl4ed74gTrXl3cC

    (copied from a led run but it should look like this everywhere when the Xcode cache is populated). https://logs.chromium.org/logs/flutter/led/magder_google.com/34b0add45b8fe97daadbfb8f634b2426c6fdea1e39c03af96488ca5ff59eb6f2/+/u/install_xcode/stdout

    So either the ios_runtime part isn't being downloaded, or it is but it's somehow being stripped out of the cache, or maybe there's something wrong with ios-15 runtime package as @keyonghan suggested in #99039 (comment) (pretty sure these are the same issues). However if there was something wrong with the runtime I would expect these tests to fail on every builder, not just some.

    Some questions to answer:

    1. On a machine where this is happening, is Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes empty?
    2. In the original build where the Xcode cache was populated, does it show that ios_runtime download in the install_xcode step?
    3. Is the mac_toolchain version correct on these machines? The runtime feature was introduced not that long ago. See Engine builders running Xcode 13 do not have any valid iOS simulator runtimes #94780 and https://flutter-review.googlesource.com/c/recipes/+/23980
    4. The framework does not need custom runtimes, it can use the default iOS 15, but is it possible the changes to support that in the engine like https://flutter-review.googlesource.com/c/recipes/+/24944 separated out the caches, and the framework needs to add the xcode_runtime_ios-15-0 cache explicitly somewhere? @godofredoc may have insights here.

  10. godofredoc commented on Mar 4, 2022

    @godofredoc
    godofredoc
    on Mar 4, 2022
    Contributor

    If a build using custom runtimes ran on the failing test bed it deletes all the runtimes and installs the one passed explicitly. Because the next executions do not pass a custom runtime and the original xcode was already cached it does not install the cipd packages again and only the custom runtime installed previously is available.

    https://cs.opensource.google/flutter/recipes/+/main:recipe_modules/osx_sdk/api.py;l=214

  11. CaseyHillers
    changed the title [-]Prod bot `build807-m9` is causing test flake due to missing `runtime`[/-] [+]Mac tests are flaking due to missing `runtime`[/+] on Mar 4, 2022

  12. jmagman commented on Mar 4, 2022

    @jmagman
    jmagman
    on Mar 4, 2022
    Member

    If a build using custom runtimes ran on the failing test bed it deletes all the runtimes and installs the one passed explicitly.

    This is happening in prod, and there aren't any custom runtimes in prod, only the "default" one.

    only the custom runtime installed previously is available.

    Though it looks like no runtimes are available, though still want to know what's in Contents/Developer/Platforms/iPhoneOS.platform/Library/Developer/CoreSimulator/Profiles/Runtimes to know if the runtime is there and bad somehow, or if the directory is empty.

  13. 28 remaining items

  14. godofredoc commented on Mar 16, 2022

    @godofredoc
    godofredoc
    on Mar 16, 2022
    Contributor

    Not sure how codesigned is impacting xcode, specially if they are using different caches.

  15. christopherfujino commented on Mar 16, 2022

    @christopherfujino
    christopherfujino
    on Mar 16, 2022
    Contributor

    Just merged a recipe change to fix this, however, I believe the bots may need to be manually recovered: https://flutter-review.googlesource.com/c/recipes/+/28064

  16. zanderso
    added
    P0Critical issues such as a build break or regression
    and removed
    P1
    on Mar 17, 2022

  17. zanderso commented on Mar 17, 2022

    @zanderso
    zanderso
    on Mar 17, 2022
    Member

    The tree is closed on this, so I have increased to P0.

  18. godofredoc commented on Mar 17, 2022

    @godofredoc
    godofredoc
    on Mar 17, 2022
    Contributor

    All the new failures are coming from these bots:

    [build797-m9](https://chromium-swarm.appspot.com/bot?id=build797-m9)
[build798-m9](https://chromium-swarm.appspot.com/bot?id=build798-m9)
[build803-m9](https://chromium-swarm.appspot.com/bot?id=build803-m9)
[build805-m9](https://chromium-swarm.appspot.com/bot?id=build805-m9)
[build809-m9](https://chromium-swarm.appspot.com/bot?id=build809-m9)

  19. godofredoc commented on Mar 17, 2022

    @godofredoc
    godofredoc
    on Mar 17, 2022
    Contributor

    Caches for those builders have been cleaned.

  20. godofredoc commented on Mar 17, 2022

    @godofredoc
    godofredoc
    on Mar 17, 2022
    Contributor

    This is fixed after @christopherfujino's change and cleaning up the caches.

  21. godofredoc
    closed this as completedon Mar 17, 2022

  22. github-actions commented on Mar 31, 2022

    @github-actions
    github-actionsbot
    on Mar 31, 2022

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  23. github-actions
    locked as resolved and limited conversation to collaborators on Mar 31, 2022
  24. keyonghan
    moved this to Done in [DEPRECATED] Infra Ticket Queueon May 1, 2023
  25. keyonghan
    added this to [DEPRECATED] Infra Ticket Queueon May 1, 2023
  26. ricardoamador
    moved this from Verified to To be archived in [DEPRECATED] Infra Ticket Queueon Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical issues such as a build break or regressionc: flakeTests that sometimes, but not always, incorrectly passteam-infraOwned by Infrastructure team

    Type

    No type

    Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions