Skip to content

The module_test_ios flakes, crashing at dart::SimpleHashMap::Lookup > dart::TimelineEventRecorder::AddAsyncTrackMetadataBasedOnEvent #125425

Closed
Closed
The module_test_ios flakes, crashing at dart::SimpleHashMap::Lookup > dart::TimelineEventRecorder::AddAsyncTrackMetadataBasedOnEvent#125425
Assignees
jmagman
Labels
P1High-priority issues at the top of the work listc: flakeTests that sometimes, but not always, incorrectly passdependency: dartDart team may need to help usengineflutter/engine related. See also e: labels.

Description

@keyonghan
keyonghan
opened on Apr 24, 2023

Both Mac module_test_ios and Mac_arm64_ios module_test_ios are severely flaky with 6 and 5 flakes each from recent 30 runs.

Examples:
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/12040/overview
https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20module_test_ios/5367/overview

[2023-04-21 13:18:36.230191] [STDOUT] stdout: Test Case '-[FlutterUITests testHybridViewWarm]' passed (11.399 seconds).
[2023-04-21 13:18:36.230598] [STDOUT] stdout: Test Suite 'FlutterUITests' failed at 2023-04-21 13:18:36.226.
[2023-04-21 13:18:36.230650] [STDOUT] stdout: 	 Executed 5 tests, with 1 failure (0 unexpected) in 63.441 (63.443) seconds
[2023-04-21 13:18:36.232730] [STDOUT] stdout: Test Suite 'FlutterUITests.xctest' failed at 2023-04-21 13:18:36.227.
[2023-04-21 13:18:36.232786] [STDOUT] stdout: 	 Executed 5 tests, with 1 failure (0 unexpected) in 63.441 (63.445) seconds
[2023-04-21 13:18:36.232820] [STDOUT] stdout: Test Suite 'All tests' failed at 2023-04-21 13:18:36.228.
[2023-04-21 13:18:36.232854] [STDOUT] stdout: 	 Executed 5 tests, with 1 failure (0 unexpected) in 63.441 (63.446) seconds
[2023-04-21 13:18:36.675545] [STDOUT] stderr: 2023-04-21 13:18:36.675 xcodebuild[42059:798874]  IDETestOperationsObserverDebug: Failure collecting logarchive: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.apple.testmanagerd.control was invalidated: failed at lookup with error 3 - No such process." UserInfo={NSDebugDescription=The connection to service named com.apple.testmanagerd.control was invalidated: failed at lookup with error 3 - No such process.}
[2023-04-21 13:18:43.522143] [STDOUT] stderr: 2023-04-21 13:18:43.521 xcodebuild[42059:794082] [MT] IDETestOperationsObserverDebug: 72.501 elapsed -- Testing started completed.
[2023-04-21 13:18:43.522241] [STDOUT] stderr: 2023-04-21 13:18:43.521 xcodebuild[42059:794082] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[2023-04-21 13:18:43.522284] [STDOUT] stderr: 2023-04-21 13:18:43.521 xcodebuild[42059:794082] [MT] IDETestOperationsObserverDebug: 72.501 sec, +72.501 sec -- end
[2023-04-21 13:18:43.529354] [STDOUT] stdout: 
[2023-04-21 13:18:43.529438] [STDOUT] stdout: Test session results, code coverage, and logs:
[2023-04-21 13:18:43.529481] [STDOUT] stdout: 	/opt/s/w/ir/x/t/flutter_module_test_ios_xcresult.No1wOx/result
[2023-04-21 13:18:43.529520] [STDOUT] stdout: 
[2023-04-21 13:18:43.529560] [STDOUT] stderr: Failing tests:
[2023-04-21 13:18:43.529611] [STDOUT] stderr: 	-[FlutterUITests testDualCold]
[2023-04-21 13:18:43.529650] [STDOUT] stderr: 
[2023-04-21 13:18:43.529688] [STDOUT] stderr: ** TEST FAILED **

Activity

  1. keyonghan
    added
    toolAffects the "flutter" command-line tool. See also t: labels.
    P1
    c: flakeTests that sometimes, but not always, incorrectly pass
    on Apr 24, 2023
  2. keyonghan
    assigned
    jmagman
    on Apr 24, 2023

  3. jmagman commented on Apr 24, 2023

    @jmagman
    jmagman
    on Apr 24, 2023
    Member

    When the test failed the app wasn't running:

    (screenshots are in the xcresult log links)
    Screenshot 2023-04-24 at 11 30 13 AM

    However it didn't find a crash log? This recent flaking corresponded to the recent Xcode version/simulator bump.

    cc @vashworth

  4. jmagman
    changed the title [-]The `module_test_ios` has been severely flaky[/-] [+]The `module_test_ios` flakes, crashing at `dart::SimpleHashMap::Lookup > dart::TimelineEventRecorder::AddAsyncTrackMetadataBasedOnEvent`[/+] on Apr 24, 2023
  5. jmagman
    added
    engineflutter/engine related. See also e: labels.
    dependency: dartDart team may need to help us
    and removed
    toolAffects the "flutter" command-line tool. See also t: labels.
    on Apr 24, 2023

  6. jmagman commented on Apr 24, 2023

    @jmagman
    jmagman
    on Apr 24, 2023
    Member

    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20module_test_ios/5367/overview shows crash logs in the xcresult:

    crash_log.txt
    Crashing thread:

    Thread 31 Crashed:: io.flutter.3.ui
0   libsystem_kernel.dylib        	       0x1b0549e60 __pthread_kill + 8
1   libsystem_pthread.dylib       	       0x1b059c3c0 pthread_kill + 256
2   libsystem_c.dylib             	       0x180129e28 abort + 124
3   libsystem_malloc.dylib        	       0x18019b284 malloc_vreport + 912
4   libsystem_malloc.dylib        	       0x18019b500 malloc_zone_error + 96
5   libsystem_malloc.dylib        	       0x1801841a0 free_small_botch + 36
6   Flutter                       	       0x1092804d4 dart::SimpleHashMap::Lookup(void*, unsigned int, bool) + 340
7   Flutter                       	       0x10945cd38 dart::TimelineEventRecorder::AddAsyncTrackMetadataBasedOnEvent(dart::TimelineEvent const&) + 108
8   Flutter                       	       0x10945da6c dart::TimelineEventFixedBufferRecorder::CompleteEvent(dart::TimelineEvent*) + 36
9   Flutter                       	       0x10945958c dart::TimelineEvent::Complete() + 32
10  Flutter                       	       0x1096659e8 Dart_TimelineEvent + 484
11  Flutter                       	       0x108f8e0b8 flutter::Pipeline<flutter::LayerTreeItem>::ProducerContinuation::ProducerContinuation(std::_LIBCPP_ABI_NAMESPACE::function<flutter::PipelineProduceResult (std::_LIBCPP_ABI_NAMESPACE::unique_ptr<flutter::LayerTreeItem, std::_LIBCPP_ABI_NAMESPACE::default_delete<flutter::LayerTreeItem> >, unsigned long)> const&, unsigned long) + 80
12  Flutter                       	       0x108f8bdb0 flutter::Pipeline<flutter::LayerTreeItem>::Produce() + 156
13  Flutter                       	       0x108f8bb28 flutter::Animator::BeginFrame(std::_LIBCPP_ABI_NAMESPACE::unique_ptr<flutter::FrameTimingsRecorder, std::_LIBCPP_ABI_NAMESPACE::default_delete<flutter::FrameTimingsRecorder> >) + 268
14  Flutter                       	       0x108f8e7ac std::_LIBCPP_ABI_NAMESPACE::__function::__func<flutter::Animator::AwaitVSync()::$_0, std::_LIBCPP_ABI_NAMESPACE::allocator<flutter::Animator::AwaitVSync()::$_0>, void (std::_LIBCPP_ABI_NAMESPACE::unique_ptr<flutter::FrameTimingsRecorder, std::_LIBCPP_ABI_NAMESPACE::default_delete<flutter::FrameTimingsRecorder> >)>::operator()(std::_LIBCPP_ABI_NAMESPACE::unique_ptr<flutter::FrameTimingsRecorder, std::_LIBCPP_ABI_NAMESPACE::default_delete<flutter::FrameTimingsRecorder> >&&) + 68
15  Flutter                       	       0x108fb70c4 std::_LIBCPP_ABI_NAMESPACE::__function::__func<flutter::VsyncWaiter::FireCallback(fml::TimePoint, fml::TimePoint, bool)::$_0, std::_LIBCPP_ABI_NAMESPACE::allocator<flutter::VsyncWaiter::FireCallback(fml::TimePoint, fml::TimePoint, bool)::$_0>, void ()>::operator()() + 112
16  Flutter                       	       0x108c58480 fml::MessageLoopImpl::FlushTasks(fml::FlushType) + 156
17  Flutter                       	       0x108c60060 fml::MessageLoopDarwin::OnTimerFire(__CFRunLoopTimer*, fml::MessageLoopDarwin*) + 32
18  CoreFoundation                	       0x18037383c __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 28
19  CoreFoundation                	       0x1803734f8 __CFRunLoopDoTimer + 868
20  CoreFoundation                	       0x180372be4 __CFRunLoopDoTimers + 284
21  CoreFoundation                	       0x18036d2d4 __CFRunLoopRun + 1816
22  CoreFoundation                	       0x18036c7a4 CFRunLoopRunSpecific + 584
23  Flutter                       	       0x108c60190 fml::MessageLoopDarwin::Run() + 88
24  Flutter                       	       0x108c58394 fml::MessageLoopImpl::DoRun() + 40
25  Flutter                       	       0x108c5e90c void* std::_LIBCPP_ABI_NAMESPACE::__thread_proxy[abi:v15000]<std::_LIBCPP_ABI_NAMESPACE::tuple<std::_LIBCPP_ABI_NAMESPACE::unique_ptr<std::_LIBCPP_ABI_NAMESPACE::__thread_struct, std::_LIBCPP_ABI_NAMESPACE::default_delete<std::_LIBCPP_ABI_NAMESPACE::__thread_struct> >, fml::Thread::Thread(std::_LIBCPP_ABI_NAMESPACE::function<void (fml::Thread::ThreadConfig const&)> const&, fml::Thread::ThreadConfig const&)::$_0> >(void*) + 208
26  libsystem_pthread.dylib       	       0x1b059c6c8 _pthread_start + 116
27  libsystem_pthread.dylib       	       0x1b0597910 thread_start + 8

    This recent flaking corresponded to the recent Xcode version/simulator bump.

    Or not, this is a dart crash. Looking to see when this started.

  7. jmagman commented on Apr 24, 2023

    @jmagman
    jmagman
    on Apr 24, 2023
    Member

    Flakes started on the same engine roll 7f11020 which contained dart roll https://dart.googlesource.com/sdk.git/+log/50b96abe9f6f..4cc2e637a315

    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20module_test_ios/12023/overview
    https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_arm64_ios%20module_test_ios/5359/overview

    cc @chinmaygarde engine sheriff

  8. keyonghan commented on Apr 24, 2023

    @keyonghan
    keyonghan
    on Apr 24, 2023
    ContributorAuthor

    This recent flaking corresponded to the recent Xcode version/simulator bump.

    The recent xcode bump happened on host only bots. I would not expect this devicelab task affected though.

  9. auto-submit
    added a commit that references this issue on Apr 24, 2023

    Mark module_test_ios flaky (#125426)

    624bdd3

  10. jmagman commented on Apr 24, 2023

    @jmagman
    jmagman
    on Apr 24, 2023
    Member

    AsyncTimelineTrackMetadata was added in dart-lang/sdk@6fb71e7 was in flutter-team-archive/engine#41379 and #125271

  11. a-siva commented on Apr 24, 2023

    @a-siva
    a-siva
    on Apr 24, 2023
    Contributor

    Derek is going to disable perfetto which should fix the crash and then investigate the cause.

  12. 1 remaining item

  13. jmagman commented on Apr 25, 2023

    @jmagman
    jmagman
    on Apr 25, 2023
    Member

    https://dart-review.googlesource.com/c/sdk/+/297821 to turn off SUPPORT_PERFETTO is in dart 3.1.0-39.0.dev, should be picked up in the next dart -> engine roll.

  14. jmagman commented on Apr 25, 2023

    @jmagman
    jmagman
    on Apr 25, 2023
    Member

    Turning off SUPPORT_PERFETTO is in flutter-team-archive/engine#41475 - #125472. Will mark these tests as unflaky. Downgrading priority while @derekxu16 investigates.

  15. jmagman
    added
    P1High-priority issues at the top of the work list
    and removed
    P1
    on Apr 25, 2023
  16. auto-submit
    added a commit that references this issue on Apr 27, 2023

    Mark module_test_ios tests unflaky (#125498)

    28e1717

  17. derekxu16 commented on May 1, 2023

    @derekxu16
    derekxu16
    on May 1, 2023
    Contributor

    I have landed a CL that fixes the crash and re-enables Perfetto-related features: dart-lang/sdk@c4f31a9.

  18. jmagman commented on May 2, 2023

    @jmagman
    jmagman
    on May 2, 2023
    Member

    Fix in dart -> engine roll flutter-team-archive/engine#41654

  19. jmagman commented on May 2, 2023

    @jmagman
    jmagman
    on May 2, 2023
    Member

    Fix is in #125858, thanks @derekxu16. Closing.

  20. jmagman
    closed this as completedon May 2, 2023

  21. github-actions commented on May 16, 2023

    @github-actions
    github-actionsbot
    on May 16, 2023

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  22. github-actions
    locked as resolved and limited conversation to collaborators on May 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High-priority issues at the top of the work listc: flakeTests that sometimes, but not always, incorrectly passdependency: dartDart team may need to help usengineflutter/engine related. See also e: labels.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions