Mac tool_integration_tests_4_4 flaky on multiple unit tests

(part of #97036)

Since 03/14/2022, Mac tool_integration_tests_4_4 has been flaky 4 times, each hit same multiple unit tests errors:

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3475/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3484/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3486/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20tool_integration_tests_4_4/3487/overview

01:19 +0 -1: test/integration.shard/ios_content_validation_test.dart: iOS app validation builds all plugin architectures for simulator [E]                                                             
  Expected: <0>
    Actual: <1>
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 246:7  main.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>

01:35 +0 -2: test/integration.shard/ios_content_validation_test.dart: iOS app validation build for simulator with all available architectures [E]                                                      
  Expected: <0>
    Actual: <1>
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 283:7  main.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>

02:28 +0 -3: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode validate obfuscation [E]                                                                  
  Expected: <1>
    Actual: <2>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 228:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>

02:28 +0 ~1 -4: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode flutter build ios builds a valid app [E]                                               
  Expected: a file system entity that exists
    Actual: LocalFile:<LocalFile: '/opt/s/w/ir/x/t/flutter_ios_content_validation.ocq7dS/hello/example/build/ios/iphoneos/Runner.app/Frameworks/hello.framework/hello'>
     Which: does not exist
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 97:11  main.<fn>.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>
  

02:28 +0 ~1 -4: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode Info.plist dart observatory Bonjour service                                            
02:28 +0 ~1 -5: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in debug mode Info.plist dart observatory Bonjour service [E]                                        
  Expected: <true>
    Actual: <false>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 133:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>

03:35 +1 ~1 -6: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode validate obfuscation [E]                                                             
  Expected: <1>
    Actual: <2>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 228:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>
  

03:35 +1 ~1 -6: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode xcode_backend embed_and_thin                                                         
03:35 +1 ~1 -7: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode xcode_backend embed_and_thin [E]                                                     
  FileSystemException: Deletion failed, path = '/opt/s/w/ir/x/t/flutter_ios_content_validation.ocq7dS/hello/example/build/ios/iphoneos/Runner.app/Frameworks/Flutter.framework' (OS Error: No such file or directory, errno = 2)
  dart:io                                                               FileSystemEntity.deleteSync
  package:file/src/forwarding/forwarding_file_system_entity.dart 70:16  ForwardingFileSystemEntity.deleteSync
  test/integration.shard/ios_content_validation_test.dart 165:34        main.<fn>.<fn>.<fn>
  dart:async                                                            runZoned
  test/src/common.dart 197:14                                           testWithoutContext.<fn>
  test/src/common.dart 196:18                                           testWithoutContext.<fn>
  test/src/common.dart 166:18                                           test.<fn>
  test/src/common.dart 161:5                                            test.<fn>
  

03:35 +1 ~1 -7: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode flutter build ios builds a valid app                                                 
03:35 +1 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode flutter build ios builds a valid app [E]                                             
  Expected: a file system entity that exists
    Actual: LocalFile:<LocalFile: '/opt/s/w/ir/x/t/flutter_ios_content_validation.ocq7dS/hello/example/build/ios/iphoneos/Runner.app/Frameworks/hello.framework/hello'>
     Which: does not exist
  
  package:test_api                                               expect
  test/integration.shard/ios_content_validation_test.dart 97:11  main.<fn>.<fn>.<fn>
  dart:async                                                     runZoned
  test/src/common.dart 197:14                                    testWithoutContext.<fn>
  test/src/common.dart 196:18                                    testWithoutContext.<fn>
  test/src/common.dart 166:18                                    test.<fn>
  test/src/common.dart 161:5                                     test.<fn>
  

03:35 +1 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode Info.plist dart observatory Bonjour service                                          
03:35 +2 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode Info.plist dart observatory Bonjour service                                          
03:35 +2 ~1 -8: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode check symbols                                                                        
03:35 +2 ~1 -9: test/integration.shard/ios_content_validation_test.dart: iOS app validation build in release mode check symbols [E]                                                                    
  Expected: <true>
    Actual: <false>
  
  package:test_api                                                expect
  test/integration.shard/ios_content_validation_test.dart 161:11  main.<fn>.<fn>.<fn>
  dart:async                                                      runZoned
  test/src/common.dart 197:14                                     testWithoutContext.<fn>
  test/src/common.dart 196:18                                     testWithoutContext.<fn>
  test/src/common.dart 166:18                                     test.<fn>
  test/src/common.dart 161:5                                      test.<fn>

18:32 +46 ~1 -10: test/integration.shard/analyze_size_test.dart: --analyze-size flag produces expected output on hello_world for iOS [E]                                                               
  Expected: contains 'Dart AOT symbols accounted decompressed size'
    Actual: 'Warning: Building for device with codesigning disabled. You will have to manually codesign before deploying to device.\n'
              'Building io.flutter.examples.hello-world for device (ios-release)...\n'
              'Warning: Missing build name (CFBundleShortVersionString).\n'
              'Warning: Missing build number (CFBundleVersion).\n'
              'Running Xcode build...                                          \n'
              'Xcode build done.                                           25.2s\n'
              'Failed to build iOS app\n'
              'Error output from Xcode build:\n'
              '↳\n'
              '    2022-03-15 21:51:43.660 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk; its version (10.15.6) is below required minimum (11.0) for the macosx platform.\n'
              '    2022-03-15 21:51:43.662 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator13.6.sdk; its version (13.6) is below required minimum (15.0) for the iphonesimulator platform.\n'
              '    2022-03-15 21:51:43.662 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.0.sdk; its version (14.0) is below required minimum (15.0) for the iphonesimulator platform.\n'
              '    2022-03-15 21:51:43.663 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS13.6.sdk; its version (13.6) is below required minimum (15.0) for the iphoneos platform.\n'
              '    2022-03-15 21:51:43.663 xcodebuild[12578:73409] [MT] DVTSDK: Skipped SDK /opt/s/w/ir/cache/osx_sdk/XCode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.0.sdk; its version (14.0) is below required minimum (15.0) for the iphoneos platform.\n'
              '    ** BUILD FAILED **\n'
              '\n'
              '\n'
              'Xcode\'s output:\n'
              '↳\n'
              '    Writing result bundle at path:\n'
              '    \t/opt/s/w/ir/x/t/flutter_tools.6CBhPi/flutter_ios_build_temp_diru69dKJ/temporary_xcresult_bundle\n'
              '\n'
              '    2022-03-15 21:52:08.614 ibtoold[12136:70583] [MT] IBPlatformTool: *** Failed to launch tool with description <IBCocoaTouchPlatformToolDescription: 0x600002532ce0> System content for IBCocoaTouchFramework-EightAndLater <IBScaleFactorDeviceTypeDescription: 0x600002533fc0> scaleFactor=2x, renderMode.identifier=(null): Failed to find or create execution context for description \'<IBCocoaTouchPlatformToolDescription: 0x600002532ce0> System content for IBCocoaTouchFramework-EightAndLater <IBScaleFactorDeviceTypeDescription: 0x600002533fc0> scaleFactor=2x, renderMode.identifier=(null)\'.\n'
              '    Device type: IBSimDeviceTypeiPad2x (com.apple.dt.Xcode.IBSimDeviceType.iPad-2x)\n'
              '    Sim runtime: (null)\n'
              '    ** Please also include the output of `xcrun simctl list` and `xcode-select -p`.: Failed to find newest available Simulator runtime: Could not find any supported simulator runtimes\n'
              '    /* com.apple.ibtool.errors */\n'
              '    /opt/s/w/ir/x/w/flutter/examples/hello_world/ios/Runner/Base.lproj/LaunchScreen.storyboard: error: Failed to find or create execution context for description \'<IBCocoaTouchPlatformToolDescription: 0x600002532ce0> System content for IBCocoaTouchFramework-EightAndLater <IBScaleFactorDeviceTypeDescription: 0x600002533fc0> scaleFactor=2x, renderMode.identifier=(null)\'.\n'
              '    Device type: IBSimDeviceTypeiPad2x (com.apple.dt.Xcode.IBSimDeviceType.iPad-2x)\n'
              '    Sim runtime: (null)\n'
              '    ** Please also include the output of `xcrun simctl list` and `xcode-select -p`.\n'
              '        Underlying Errors:\n'
              '            Description: Failed to find newest available Simulator runtime\n'
              '            Underlying Errors:\n'
              '                Description: Could not find any supported simulator runtimes\n'
              '    note: Using new build system\n'
              '    note: Planning\n'
              '    note: Build preparation complete\n'
              '    note: Building targets in parallel\n'
              '\n'
              '    Result bundle written to path:\n'
              '    \t/opt/s/w/ir/x/t/flutter_tools.6CBhPi/flutter_ios_build_temp_diru69dKJ/temporary_xcresult_bundle\n'
              '\n'
              '\n'
              ''
  
  package:test_api                                    expect
  test/integration.shard/analyze_size_test.dart 72:5  main.<fn>

@jmagman

Routing to @jmagman to triage.

Removing infra for now, please add it back if this is caused by infrastructure.

These are all issues with the iOS simulator runtimes missing.

== Runtimes ==
== Devices ==

Lowering the priority to P3.

@keyonghan

Synced with @keyonghan over chat. The reason this is in the ticket queue as the current infra state allows the tests to corrupt the bot, which will cause it to fail until it is quarantined and wiped. CP'ing the ToT fixes into the release branch would prevent it getting into a bad state.

@keyonghan since this requires doing a major rev upgrade to xcode, it's unlikely for this to be approved as a CP. Is there a way we can update device doctor to check for this state, or catch the failure in the recipes?

This happens on host only bots, where we are not running device doctor.
One thing we can do is fail the test earlier as infra failure from recipe side when no runtime in list simulators step.

Are there any blockers to running device doctor on the host only bots?

This needs device doctor package installed, but these bots are maintained by chromium team. Also, we need to separate the bot config file from fuchsia.py. Another blocker: this needs xcode preinstalled.

Instead, could you help check why the release build failure caused runtime not available? The issue in #99429 is due to xcode dependency not consistent before tot and release builders. is it same here?

This is also on bot 842-m9. Duplicate of #99039

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Mac tool_integration_tests_4_4 flaky on multiple unit tests #100245

Description

Activity

zanderso commented on Mar 16, 2022

godofredoc commented on Mar 17, 2022

jmagman commented on Mar 22, 2022

11 remaining items

keyonghan commented on Mar 25, 2022

CaseyHillers commented on Mar 28, 2022

keyonghan commented on Mar 28, 2022

CaseyHillers commented on Mar 28, 2022

keyonghan commented on Mar 28, 2022

keyonghan commented on Apr 2, 2022

github-actions commented on Apr 16, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions