runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker

Soon after #164554, we've hit some postsubmit failures that I haven't seen before:

26:42 +6876 ~120 -1: loading /b/s/w/ir/x/w/flutter sdk/packages/flutter/test/semantics/semantics_binding_test.dart [E]                                                                                 
  Failed to load "/b/s/w/ir/x/w/flutter sdk/packages/flutter/test/semantics/semantics_binding_test.dart":
  Unexpected character (at character 1)
  ../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorke...
  ^

@a-siva

Found this crash among flaky failures on Dart CI:

** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart --pause-isolates-on-start --pause-isolates-on-exit --disable-service-auth-codes --profiler --sound-null-safety -Dtest_runner.configuration=vm-linux-release-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json --enable-vm-service:0 /b/s/w/ir/pkg/vm_service/test/step_through_await_for_test.dart

>testee>err> Could not start the VM service:
>testee>err> ../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker: 11 (Resource temporarily unavailable)
>testee>err> version=3.8.0-edge (main) (Unknown timestamp) on "linux_x64"
>testee>err> pid=513175, thread=513178, isolate_group=(nil)((nil)), isolate=(nil)((nil))
>testee>err> os=linux, arch=x64, comp=no, sim=no
>testee>err> isolate_instructions=0, vm_instructions=557802a835a0
>testee>err> fp=7fcedd3be770, sp=7fcedd3be640, pc=557802d08fac
>testee>err>   pc 0x0000557802d08fac fp 0x00007fcedd3be770 dart::Profiler::DumpStackTrace+0x7c
>testee>err>   pc 0x0000557802a83794 fp 0x00007fcedd3be850 dart::Assert::Fail+0x84
>testee>err>   pc 0x0000557802d06159 fp 0x00007fcedd3bec80 dart::OSThread::Start+0x69
>testee>err>   pc 0x0000557802d53c3d fp 0x00007fcedd3becb0 dart::ThreadPool::RunImpl+0x6d
>testee>err>   pc 0x0000557802c7dfd8 fp 0x00007fcedd3becf0 dart::MessageHandler::Run+0x78
>testee>err>   pc 0x0000557802d3f878 fp 0x00007fcedd3bed80 dart::RunServiceTask::Run+0x1f8
>testee>err>   pc 0x0000557802d54087 fp 0x00007fcedd3bee00 dart::ThreadPool::WorkerLoop+0x127
>testee>err>   pc 0x0000557802d542e2 fp 0x00007fcedd3bee30 dart::ThreadPool::Worker::Main+0x72
>testee>err>   pc 0x0000557802d06309 fp 0x00007fcedd3beef0 dart::ThreadStart+0xd9
>testee>err> -- End of DumpStackTrace
>testee>err>

--- Re-run this test:
python3 tools/test.py -n vm-linux-release-x64 pkg/vm_service/test/step_through_await_for_test

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8721372083109890177/+/u/test_results/ignored_flaky_test_failure_logs
(Mar 03)

** Launching SERVICE_TEST_TESTEE=true out/ReleaseX64/dart --disable-service-auth-codes --profiler --timeline_streams=Dart --sound-null-safety -Dtest_runner.configuration=vm-linux-release-x64 --ignore-unrecognized-flags --packages=/b/s/w/ir/.dart_tool/package_config.json --enable-vm-service:0 /b/s/w/ir/pkg/vm_service/test/get_http_profile_test.dart

>testee>err> Could not start the VM service:
>testee>err> ../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker: 11 (Resource temporarily unavailable)
>testee>err> version=3.8.0-edge (main) (Unknown timestamp) on "linux_x64"
>testee>err> pid=1680188, thread=1680190, isolate_group=(nil)((nil)), isolate=(nil)((nil))
>testee>err> os=linux, arch=x64, comp=no, sim=no
>testee>err> isolate_instructions=0, vm_instructions=563f2bceea20
>testee>err> fp=7f69c7bae770, sp=7f69c7bae640, pc=563f2bf7431c
>testee>err>   pc 0x0000563f2bf7431c fp 0x00007f69c7bae770 dart::Profiler::DumpStackTrace+0x7c
>testee>err>   pc 0x0000563f2bceec14 fp 0x00007f69c7bae850 dart::Assert::Fail+0x84
>testee>err>   pc 0x0000563f2bf714c9 fp 0x00007f69c7baec80 dart::OSThread::Start+0x69
>testee>err>   pc 0x0000563f2bfbefad fp 0x00007f69c7baecb0 dart::ThreadPool::RunImpl+0x6d
>testee>err>   pc 0x0000563f2bee9418 fp 0x00007f69c7baecf0 dart::MessageHandler::Run+0x78
>testee>err>   pc 0x0000563f2bfaabf8 fp 0x00007f69c7baed80 dart::RunServiceTask::Run+0x1f8
>testee>err>   pc 0x0000563f2bfbf3f7 fp 0x00007f69c7baee00 dart::ThreadPool::WorkerLoop+0x127
>testee>err>   pc 0x0000563f2bfbf652 fp 0x00007f69c7baee30 dart::ThreadPool::Worker::Main+0x72
>testee>err>   pc 0x0000563f2bf71679 fp 0x00007f69c7baeef0 dart::ThreadStart+0xd9
>testee>err> -- End of DumpStackTrace
>testee>err> 
>testee>out>

--- Re-run this test:
python3 tools/test.py -n vm-linux-release-x64 pkg/vm_service/test/get_http_profile_test

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8721847336512224753/+/u/test_results/ignored_flaky_test_failure_logs
(Feb 26)

/cc @a-siva @rmacnak-google

The earliest crash I can find happened at dart-lang/sdk@7a02fa5.

@rmacnak-google

Assuming this problem didn't happen before the roll #164168, and it happened at dart-lang/sdk@7a02fa5, here is the list of suspects: https://dart.googlesource.com/sdk.git/+log/b8292dfeaa67..7a02fa5d3d2924bcc2d38c1463c0c74788d99534

@rmacnak-google Could https://dart-review.googlesource.com/c/sdk/+/410641 be the culprit?

I would expect https://dart-review.googlesource.com/c/sdk/+/406404 to increase resource exhaustion failures and https://dart-review.googlesource.com/c/sdk/+/410641 to reduce them back to normal.

Adding a note to remember to check in next week.

Looks like this just happened on dart->flutter roll https://github.com/flutter/flutter/pull/165026/checks?check_run_id=38609915783

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8720634007130983409/+/u/run_test.dart_for_customer_testing_shard_and_subshard_None/stdout

| 00:03 +3: /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.DDXMJM/tests/packages/rfw/test/argument_decoders_test.dart: String example
| 00:03 +3 -1: loading /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.DDXMJM/tests/packages/rfw/test/material_widgets_test.dart [E]
|   Failed to load "/b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.DDXMJM/tests/packages/rfw/test/material_widgets_test.dart":
|   Unexpected character (at character 1)
|   ../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorke...
|   ^
|
|
| To run this test again: /b/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/dart test /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.DDXMJM/tests/packages/rfw/test/material_widgets_test.dart -p vm --plain-name 'loading /b/s/w/ir/x/t/flutter_customer_testing.flutter_packages.DDXMJM/tests/packages/rfw/test/material_widgets_test.dart'

We haven't seen this error in a while now (more than 2 weeks).
Closing the issue, please re-open if the error is observed again.

We will wait and see if dart-lang/sdk@326e32a solves this problem.

Despite dart-lang/sdk@326e32a landing, we still are getting
https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8716593502530708801/+/u/build_ci_android_embedder_debug_unopt_flutter_shell_platform_embedder:flutter_engine/stdout

[58/5147] ACTION //flutter/third_party/dart/runtime/observatory:build_observatory(//build/toolchain/android:clang_arm)
FAILED: gen/flutter/third_party/dart/runtime/observatory/observatory/web/main.dart.js gen/flutter/third_party/dart/runtime/observatory/observatory/web/main.dart.js.map 
vpython3 ../../../flutter/third_party/dart/build/gn_run_binary.py compiled_action ../../../flutter/third_party/dart/tools/sdks/dart-sdk/bin/dart --dfe=../../../flutter/third_party/dart/tools/sdks/dart-sdk/bin/snapshots/kernel-service.dart.snapshot compile js -o gen/flutter/third_party/dart/runtime/observatory/observatory/web/main.dart.js --packages=../../../flutter/third_party/dart/.dart_tool/package_config.json ../../../flutter/third_party/dart/runtime/observatory/web/main.dart --enable-asserts
Command failed: /b/s/w/ir/cache/builder/engine/src/flutter/third_party/dart/tools/sdks/dart-sdk/bin/dart --dfe=../../../flutter/third_party/dart/tools/sdks/dart-sdk/bin/snapshots/kernel-service.dart.snapshot compile js -o gen/flutter/third_party/dart/runtime/observatory/observatory/web/main.dart.js --packages=../../../flutter/third_party/dart/.dart_tool/package_config.json ../../../flutter/third_party/dart/runtime/observatory/web/main.dart --enable-asserts
exitCode: -6
output: pthread_create failed
/proc/self/limits: Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             514952               514952               processes 
Max open files            50000                50000                files     
Max locked memory         8388608              8388608              bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       514952               514952               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

/proc/sys/kernel/threads-max: 1029905

/proc/sys/kernel/pid_max: 4194304

/sys/fs/cgroup/user.slice/memory.current: 23109632

/sys/fs/cgroup/user.slice/memory.max: max

/sys/fs/cgroup/user.slice/memory.peak: 25108480

/sys/fs/cgroup/user.slice/pids.current: 5

/sys/fs/cgroup/user.slice/pids.max: max

/sys/fs/cgroup/user.slice/pids.peak: 5

../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker: 11 (Resource temporarily unavailable)
version=3.9.0-27.0.dev (dev) (Wed Apr 16 21:03:19 2025 -0700) on "linux_x64"
pid=1029466, thread=1029537, isolate_group=(nil)((nil)), isolate=(nil)((nil))
os=linux, arch=x64, comp=no, sim=no
isolate_instructions=0, vm_instructions=62e3b6df8e00
fp=7f1127eee2c0, sp=7f1127eee190, pc=62e3b70801fc
  pc 0x000062e3b70801fc fp 0x00007f1127eee2c0 dart::Profiler::DumpStackTrace+0x7c
  pc 0x000062e3b6df8ff4 fp 0x00007f1127eee3a0 dart::Assert::Fail+0x84
  pc 0x000062e3b707d2a9 fp 0x00007f1127eee7d0 dart::OSThread::Start+0x69
  pc 0x000062e3b70cb9fd fp 0x00007f1127eee800 dart::ThreadPool::RunImpl+0x6d
  pc 0x000062e3b70de191 fp 0x00007f1127eeed50 dart::BackgroundCompiler::Run+0x1f1
  pc 0x000062e3b70cbe47 fp 0x00007f1127eeedd0 dart::ThreadPool::WorkerLoop+0x127
  pc 0x000062e3b70cc0a2 fp 0x00007f1127eeee00 dart::ThreadPool::Worker::Main+0x72
  pc 0x000062e3b707d559 fp 0x00007f1127eeeec0 dart::ThreadStart+0xd9
  pc 0x00007f112869caa4 fp 0x00007f1127eeef70 /lib/x86_64-linux-gnu/libc.so.6+0x9caa4
-- End of DumpStackTrace

The prints are still present due to the fact that prebuilt tools/sdks/dart-sdk has not been moved to the version with the fix 3.9.0-44.0.dev, still on 3.9.0-27.0.dev. But the change to limiting those actions to the pool size should have gone into effect, yet without desired outcome.

@alexmarkov

@alexmarkov regarding your question about why the background compiler is running if js compilation is happening in AOT mode, the reason is we first invoke the dart CLI tool which is still running in JIT mode, parse the command 'compile js' and then exec the dartaotruntime process to do the actual compilation. It looks like the above failure happened even before the exec step.

I am thinking of modifying the observatory build step to directly invoke dartaotruntime instead of first invoking a shell script (dart) which then invokes the dart CLI and finally results in invoking the dartaotruntime process.

https://dart-review.googlesource.com/c/sdk/+/425420 should invoke dart2js directly using dartaotruntime instead of using 'compile js'

Saw this crash recently on Flutter head-head bot while it was running Flutter tests:

00:58 +1103 ~13 -1: loading /b/s/w/ir/cache/builder/flutter/packages/flutter/test/rendering/simple_semantics_test.dart [E]
  Failed to load "/b/s/w/ir/cache/builder/flutter/packages/flutter/test/rendering/simple_semantics_test.dart":
  Unexpected character (at character 1)
  ../../flutter/third_party/dart/runtime/vm/os_thread.cc: 348: error: Could n...
  ^
  
Could not parse shell VM Service port message: DartDevelopmentServiceException: Failed to start Dart Development Service

Log: https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8713069049797598785/+/u/Run_framework_coverage_tests/stdout

Mark Linux coverage as bringup (flutter#166144)

I hit the same bug today. Running under the debugger from Android Studio. Please reopen this issue.

Connecting to: localhost:2925
../../../flutter/third_party/dart/runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker: 22 (The device does not recognize the command.

)
version=3.8.1 (stable) (Wed May 28 00:47:25 2025 -0700) on "windows_x64"
pid=3092, thread=13644, isolate_group=main(00000196F9E971F0), isolate=main(00000196F9E9DF20)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=7fffb962ab80, vm_instructions=7fffb962aba0
fp=8ce37fdff0, sp=8ce37fdfc8, pc=7fffb9400a81
pc 0x00007fffb9400a81 fp 0x0000008ce37fdff0 InternalFlutterGpu_Texture_AsImage+0x1356c1
-- End of DumpStackTrace
pc 0x0000000000000000 fp 0x0000008ce37feba0 sp 0x0000000000000000 [Stub] CallBootstrapNative
pc 0x00000196fcb0f862 fp 0x0000008ce37febe8 sp 0x0000008ce37febb0 [Unoptimized] _SendPort@1026248._sendInternal@1026248
pc 0x0000019690b40509 fp 0x0000008ce37fec98 sp 0x0000008ce37febf8 [Optimized] _IOService@15069316._dispatch@15069316
pc 0x000001968cec53a6 fp 0x0000008ce37fed00 sp 0x0000008ce37feca8 [Unoptimized] [email protected]
pc 0x000001968cec4f78 fp 0x0000008ce37fed80 sp 0x0000008ce37fed10 [Unoptimized] [email protected]..lookupAddresses
pc 0x000001968cec4af2 fp 0x0000008ce37fedc8 sp 0x0000008ce37fed90 [Unoptimized] [email protected].
pc 0x00000196887a6cb3 fp 0x0000008ce37fee50 sp 0x0000008ce37fedd8 [Optimized] _runGuarded@5048458
pc 0x0000019690b5069c fp 0x0000008ce37fee80 sp 0x0000008ce37fee60 [Optimized] _StreamController@5048458._subscribe@5048458.
pc 0x0000019690b4d140 fp 0x0000008ce37feec8 sp 0x0000008ce37fee90 [Optimized] _BufferingStreamSubscription@5048458._guardCallback@5048458
pc 0x0000019690b48537 fp 0x0000008ce37fef20 sp 0x0000008ce37feed8 [Optimized] _StreamController@5048458._subscribe@5048458
pc 0x0000019690b53d2e fp 0x0000008ce37fef68 sp 0x0000008ce37fef30 [Optimized] _ControllerStream@5048458._createSubscription@5048458
pc 0x000001968cec20f0 fp 0x0000008ce37ff000 sp 0x0000008ce37fef78 [Unoptimized] [email protected]
pc 0x000001968cec1ce7 fp 0x0000008ce37ff060 sp 0x0000008ce37ff010 [Unoptimized] [email protected]
pc 0x000001968cebfec4 fp 0x0000008ce37ff0d8 sp 0x0000008ce37ff070 [Unoptimized] [email protected].
pc 0x000001969085aab3 fp 0x0000008ce37ff150 sp 0x0000008ce37ff0e8 [Optimized] [email protected]
pc 0x0000019690b3c7cd fp 0x0000008ce37ff1e8 sp 0x0000008ce37ff160 [Optimized] _Future@5048458[email protected]
pc 0x0000019690164a87 fp 0x0000008ce37ff250 sp 0x0000008ce37ff1f8 [Optimized] _Future@5048458._propagateToListeners@5048458
pc 0x000001969082e56b fp 0x0000008ce37ff2a0 sp 0x0000008ce37ff260 [Optimized] _Future@5048458._completeWithValue@5048458
pc 0x0000019690b51314 fp 0x0000008ce37ff2d8 sp 0x0000008ce37ff2b0 [Optimized] _Future@5048458._asyncCompleteWithValue@5048458.
pc 0x0000019690b51ba7 fp 0x0000008ce37ff348 sp 0x0000008ce37ff2e8 [Optimized] _startMicrotaskLoop@5048458
pc 0x0000019690b45d33 fp 0x0000008ce37ff370 sp 0x0000008ce37ff358 [Optimized] _startMicrotaskLoop@5048458
pc 0x00000196f96830df fp 0x0000008ce37ff490 sp 0x0000008ce37ff380 [Stub] InvokeDartCode
Lost connection to device.

Got it again, this time doing a build. (Solved with Flutter clean, closed Android Studio, killed two lingering java processes, restarted, cleared caches, restarted again, and restarted build.)

Launching lib\main.dart on Windows in profile mode...
Building Windows application...
../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker: 22 (The device does not recognize the command.
)
version=3.8.1 (stable) (Wed May 28 00:47:25 2025 -0700) on "windows_x64"
pid=21052, thread=2504, isolate_group=main(0000015C502AE9D0), isolate=main(0000015C54C5DFD0)
os=windows, arch=x64, comp=no, sim=no
isolate_instructions=15c53f00000, vm_instructions=7ff782fa99c0
fp=4556ffcb20, sp=4556ffcaf8, pc=7ff78325dacc
pc 0x00007ff78325dacc fp 0x0000004556ffcb20 Dart_DetectNullSafety+0x2aba1c
-- End of DumpStackTrace
pc 0x0000000000000000 fp 0x0000004556ffd678 sp 0x0000000000000000 [Stub] CallToRuntime
pc 0x0000015c54582fcd fp 0x0000004556ffd6b0 sp 0x0000004556ffd688 [Stub] OptimizeFunction
pc 0x0000015c53f03efe fp 0x0000004556ffd708 sp 0x0000004556ffd6c0 [Unoptimized] [email protected]
pc 0x0000015c5ba31d6b fp 0x0000004556ffd770 sp 0x0000004556ffd718 [Unoptimized] [email protected].
pc 0x0000015c53f90520 fp 0x0000004556ffd7f0 sp 0x0000004556ffd780 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90877 fp 0x0000004556ffd858 sp 0x0000004556ffd800 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90fd6 fp 0x0000004556ffd8f0 sp 0x0000004556ffd868 [Unoptimized] [email protected]
pc 0x0000015c5a108eed fp 0x0000004556ffd978 sp 0x0000004556ffd900 [Unoptimized] [email protected]
pc 0x0000015c5a10e5db fp 0x0000004556ffd9c8 sp 0x0000004556ffd988 [Optimized] _BufferingStreamSubscription@5048458._sendData@5048458
pc 0x0000015c5a10879a fp 0x0000004556ffda08 sp 0x0000004556ffd9d8 [Unoptimized] _BufferingStreamSubscription@5048458._add@5048458
pc 0x0000015c5ba31aa2 fp 0x0000004556ffda48 sp 0x0000004556ffda18 [Unoptimized] _ForwardingStreamSubscription@5048458._add@5048458
pc 0x0000015c5ba3150a fp 0x0000004556ffdab8 sp 0x0000004556ffda58 [Unoptimized] _WhereStream@5048458._handleData@5048458
pc 0x0000015c5ba3120f fp 0x0000004556ffdb00 sp 0x0000004556ffdac8 [Unoptimized] _ForwardingStreamSubscription@5048458._handleData@5048458
pc 0x0000015c5ba31043 fp 0x0000004556ffdb40 sp 0x0000004556ffdb10 [Unoptimized] _ForwardingStreamSubscription@5048458._handleData@5048458
pc 0x0000015c53f90520 fp 0x0000004556ffdbc0 sp 0x0000004556ffdb50 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90877 fp 0x0000004556ffdc28 sp 0x0000004556ffdbd0 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90fd6 fp 0x0000004556ffdcc0 sp 0x0000004556ffdc38 [Unoptimized] [email protected]
pc 0x0000015c5a108eed fp 0x0000004556ffdd48 sp 0x0000004556ffdcd0 [Unoptimized] [email protected]
pc 0x0000015c5a10e5db fp 0x0000004556ffdd98 sp 0x0000004556ffdd58 [Optimized] _BufferingStreamSubscription@5048458._sendData@5048458
pc 0x0000015c5a10879a fp 0x0000004556ffddd8 sp 0x0000004556ffdda8 [Unoptimized] _BufferingStreamSubscription@5048458._add@5048458
pc 0x0000015c5a10ccaa fp 0x0000004556ffde18 sp 0x0000004556ffdde8 [Unoptimized] _SinkTransformerStreamSubscription@5048458._add@5048458
pc 0x0000015c5a10c9cc fp 0x0000004556ffde58 sp 0x0000004556ffde28 [Unoptimized] [email protected]
pc 0x0000015c5ba30008 fp 0x0000004556ffde98 sp 0x0000004556ffde68 [Unoptimized] [email protected]
pc 0x0000015c5ba30ac7 fp 0x0000004556ffdf08 sp 0x0000004556ffdea8 [Unoptimized] _LineSplitterSink@11003594._addLines@11003594
pc 0x0000015c5ba3054e fp 0x0000004556ffdf60 sp 0x0000004556ffdf18 [Unoptimized] [email protected]
pc 0x0000015c5ba30274 fp 0x0000004556ffdfb8 sp 0x0000004556ffdf70 [Unoptimized] StringConversionSink.add
pc 0x0000015c5a10c1c5 fp 0x0000004556ffe020 sp 0x0000004556ffdfc8 [Unoptimized] _SinkTransformerStreamSubscription@5048458._handleData@5048458
pc 0x0000015c5a10bfc3 fp 0x0000004556ffe060 sp 0x0000004556ffe030 [Unoptimized] _SinkTransformerStreamSubscription@5048458._handleData@5048458
pc 0x0000015c53f90520 fp 0x0000004556ffe0e0 sp 0x0000004556ffe070 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90877 fp 0x0000004556ffe148 sp 0x0000004556ffe0f0 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90fd6 fp 0x0000004556ffe1e0 sp 0x0000004556ffe158 [Unoptimized] [email protected]
pc 0x0000015c5a108eed fp 0x0000004556ffe268 sp 0x0000004556ffe1f0 [Unoptimized] [email protected]
pc 0x0000015c5a10e5db fp 0x0000004556ffe2b8 sp 0x0000004556ffe278 [Optimized] _BufferingStreamSubscription@5048458._sendData@5048458
pc 0x0000015c5a10879a fp 0x0000004556ffe2f8 sp 0x0000004556ffe2c8 [Unoptimized] _BufferingStreamSubscription@5048458._add@5048458
pc 0x0000015c5a10ccaa fp 0x0000004556ffe338 sp 0x0000004556ffe308 [Unoptimized] _SinkTransformerStreamSubscription@5048458._add@5048458
pc 0x0000015c5a10c9cc fp 0x0000004556ffe378 sp 0x0000004556ffe348 [Unoptimized] [email protected]
pc 0x0000015c5ba30008 fp 0x0000004556ffe3b8 sp 0x0000004556ffe388 [Unoptimized] [email protected]
pc 0x0000015c5ba2fd5f fp 0x0000004556ffe400 sp 0x0000004556ffe3c8 [Unoptimized] [email protected]
pc 0x0000015c5ba2da25 fp 0x0000004556ffe460 sp 0x0000004556ffe410 [Unoptimized] [email protected]
pc 0x0000015c5ba2d764 fp 0x0000004556ffe4b8 sp 0x0000004556ffe470 [Unoptimized] [email protected]
pc 0x0000015c5a10c4cc fp 0x0000004556ffe4f8 sp 0x0000004556ffe4c8 [Unoptimized] [email protected]
pc 0x0000015c5a10c1c5 fp 0x0000004556ffe560 sp 0x0000004556ffe508 [Unoptimized] _SinkTransformerStreamSubscription@5048458._handleData@5048458
pc 0x0000015c5a10bfc3 fp 0x0000004556ffe5a0 sp 0x0000004556ffe570 [Unoptimized] _SinkTransformerStreamSubscription@5048458._handleData@5048458
pc 0x0000015c53f90520 fp 0x0000004556ffe620 sp 0x0000004556ffe5b0 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90877 fp 0x0000004556ffe688 sp 0x0000004556ffe630 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90fd6 fp 0x0000004556ffe720 sp 0x0000004556ffe698 [Unoptimized] [email protected]
pc 0x0000015c5a108eed fp 0x0000004556ffe7a8 sp 0x0000004556ffe730 [Unoptimized] [email protected]
pc 0x0000015c5a10e5db fp 0x0000004556ffe7f8 sp 0x0000004556ffe7b8 [Optimized] _BufferingStreamSubscription@5048458._sendData@5048458
pc 0x0000015c5a10879a fp 0x0000004556ffe838 sp 0x0000004556ffe808 [Unoptimized] _BufferingStreamSubscription@5048458._add@5048458
pc 0x0000015c5a10853c fp 0x0000004556ffe878 sp 0x0000004556ffe848 [Unoptimized] _SyncStreamController@5048458._sendData@5048458
pc 0x0000015c5a10615e fp 0x0000004556ffe8b8 sp 0x0000004556ffe888 [Unoptimized] _StreamController@5048458._add@5048458
pc 0x0000015c5a105e1d fp 0x0000004556ffe8f8 sp 0x0000004556ffe8c8 [Unoptimized] [email protected]
pc 0x0000015c5a109264 fp 0x0000004556ffe948 sp 0x0000004556ffe908 [Unoptimized] _Socket@15069316._onData@15069316
pc 0x0000015c5a109023 fp 0x0000004556ffe988 sp 0x0000004556ffe958 [Unoptimized] _Socket@15069316._onData@15069316
pc 0x0000015c53f9066c fp 0x0000004556ffea08 sp 0x0000004556ffe998 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90877 fp 0x0000004556ffea70 sp 0x0000004556ffea18 [Unoptimized] _rootRunUnary@5048458
pc 0x0000015c53f90fd6 fp 0x0000004556ffeb08 sp 0x0000004556ffea80 [Unoptimized] [email protected]
pc 0x0000015c5a108eed fp 0x0000004556ffeb90 sp 0x0000004556ffeb18 [Unoptimized] [email protected]
pc 0x0000015c5a10e5db fp 0x0000004556ffebe0 sp 0x0000004556ffeba0 [Optimized] _BufferingStreamSubscription@5048458._sendData@5048458
pc 0x0000015c5a10879a fp 0x0000004556ffec20 sp 0x0000004556ffebf0 [Unoptimized] _BufferingStreamSubscription@5048458._add@5048458
pc 0x0000015c5a10853c fp 0x0000004556ffec60 sp 0x0000004556ffec30 [Unoptimized] _SyncStreamController@5048458._sendData@5048458
pc 0x0000015c5a10615e fp 0x0000004556ffeca0 sp 0x0000004556ffec70 [Unoptimized] _StreamController@5048458._add@5048458
pc 0x0000015c5a105e1d fp 0x0000004556ffece0 sp 0x0000004556ffecb0 [Unoptimized] [email protected]
pc 0x0000015c5a108365 fp 0x0000004556ffed20 sp 0x0000004556ffecf0 [Unoptimized] new _RawSocket@15069316..
pc 0x0000015c580546f3 fp 0x0000004556ffed78 sp 0x0000004556ffed30 [Unoptimized] [email protected]
pc 0x0000015c53f8ddf2 fp 0x0000004556ffedc0 sp 0x0000004556ffed88 [Unoptimized] _microtaskLoop@5048458
pc 0x0000015c53f8df42 fp 0x0000004556ffee00 sp 0x0000004556ffedd0 [Unoptimized] _startMicrotaskLoop@5048458
pc 0x0000015c53f8e173 fp 0x0000004556ffee28 sp 0x0000004556ffee10 [Unoptimized] _startMicrotaskLoop@5048458
pc 0x0000015c54030328 fp 0x0000004556ffee68 sp 0x0000004556ffee38 [Unoptimized] _runPendingImmediateCallback@1026248
pc 0x0000015c5405b856 fp 0x0000004556ffeea8 sp 0x0000004556ffee78 [Unoptimized] _RawReceivePort@1026248._handleMessage@1026248
pc 0x0000015c545830df fp 0x0000004556ffefd0 sp 0x0000004556ffeeb8 [Stub] InvokeDartCode

seems to have happened on flutter-linux-framework-coverage luci bot during flutter test run:

RUNNING: cd packages/flutter; ../../bin/flutter test --reporter=expanded --file-reporter=json:/b/s/w/ir/x/t/metrics_1752606916920934.json --test-randomize-ordering-seed=20250715 --fatal-warnings --coverage
...
09:47 +12980 ~42: /b/s/w/ir/cache/builder/flutter/packages/flutter/test/widgets/row_test.dart: Row with MainAxisAlignment.center - RTL
09:47 +12980 ~42 -1: loading /b/s/w/ir/cache/builder/flutter/packages/flutter/test/widgets/scrollable_animations_test.dart [E]
  Failed to load "/b/s/w/ir/cache/builder/flutter/packages/flutter/test/widgets/scrollable_animations_test.dart":
  Unexpected character (at character 1)
  ../../flutter/third_party/dart/runtime/vm/os_thread.cc: 348: error: Could n...
  ^
  
Could not parse shell VM Service port message: DartDevelopmentServiceException: WebSocketChannelException: HttpException: Connection reset by peer, uri = http://127.0.0.1:39979/_eYPxEbHy0A=/ws

We still see this error happen but it is less frequent than before, lowering priority to P2 to reflect that, efforts to reproduce this locally in a consistent manner is still ongoing.

Mark Linux coverage as bringup (flutter#166144)

Since we upgraded the Flutter SDK in our project from 3.29.3 to 3.35.5 we see this error very often (more than once a day) in our CI pipeline builds.
We use the following command to execute tests: flutter test --concurrency 4 --coverage --reporter json
Here is an example showing the error:

{"testID":106,"error":"Failed to load \"/builds/autovio/autovio-app/mobile-app/model/test/business_logic/create_theory_exam_simulation_test.dart\":\nUnexpected character (at character 1)\n../../runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorke...\n^\n","stackTrace":"dart:convert-patch/convert_patch.dart 1467:5   _ChunkedJsonParser.fail\ndart:convert-patch/convert_patch.dart 1333:9   _ChunkedJsonParser.parseNumber\ndart:convert-patch/convert_patch.dart 935:22   _ChunkedJsonParser.parse\ndart:convert-patch/convert_patch.dart 1555:13  _JsonStringDecoderSink.addSlice\ndart:convert-patch/convert_patch.dart 1560:5   _JsonStringDecoderSink.add\ndart:convert/chunked_conversion.dart 70:18     _ConverterStreamEventSink.add\ndart:async/stream_transformers.dart 115:24     _SinkTransformerStreamSubscription._handleData\ndart:async/zone.dart 1538:47                   _rootRunUnary\ndart:async/zone.dart 1429:19                   _CustomZone.runUnary\ndart:async/zone.dart 1329:7                    _CustomZone.runUnaryGuarded\ndart:async/stream_impl.dart 381:11             _BufferingStreamSubscription._sendData\ndart:async/stream_impl.dart 312:7              _BufferingStreamSubscription._add\ndart:async/stream_transformers.dart 67:11      _SinkTransformerStreamSubscription._add\ndart:async/stream_transformers.dart 13:11      _EventSinkWrapper.add\ndart:convert/string_conversion.dart 228:11     _StringAdapterSink.add\ndart:convert/string_conversion.dart 233:7      _StringAdapterSink.addSlice\ndart:convert/string_conversion.dart 313:20     _Utf8ConversionSink.addSlice\ndart:convert/string_conversion.dart 306:5      _Utf8ConversionSink.add\ndart:convert/chunked_conversion.dart 70:18     _ConverterStreamEventSink.add\ndart:async/stream_transformers.dart 115:24     _SinkTransformerStreamSubscription._handleData\ndart:async/zone.dart 1538:47                   _rootRunUnary\ndart:async/zone.dart 1429:19                   _CustomZone.runUnary\ndart:async/zone.dart 1329:7                    _CustomZone.runUnaryGuarded\ndart:async/stream_impl.dart 381:11             _BufferingStreamSubscription._sendData\ndart:async/stream_impl.dart 312:7              _BufferingStreamSubscription._add\ndart:async/stream_controller.dart 798:19       _SyncStreamControllerDispatch._sendData\ndart:async/stream_controller.dart 663:7        _StreamController._add\ndart:async/stream_controller.dart 618:5        _StreamController.add\ndart:io-patch/socket_patch.dart 2912:41        _Socket._onData\ndart:async/zone.dart 1546:13                   _rootRunUnary\ndart:async/zone.dart 1429:19                   _CustomZone.runUnary\ndart:async/zone.dart 1329:7                    _CustomZone.runUnaryGuarded\ndart:async/stream_impl.dart 381:11             _BufferingStreamSubscription._sendData\ndart:async/stream_impl.dart 312:7              _BufferingStreamSubscription._add\ndart:async/stream_controller.dart 798:19       _SyncStreamControllerDispatch._sendData\ndart:async/stream_controller.dart 663:7        _StreamController._add\ndart:async/stream_controller.dart 618:5        _StreamController.add\ndart:io-patch/socket_patch.dart 2331:31        new _RawSocket.<fn>\ndart:io-patch/socket_patch.dart 1655:14        _NativeSocket.issueReadEvent.issue\ndart:async/schedule_microtask.dart 40:35       _microtaskLoop\ndart:async/schedule_microtask.dart 49:5        _startMicrotaskLoop\ndart:isolate-patch/isolate_patch.dart 127:13   _runPendingImmediateCallback\ndart:isolate-patch/isolate_patch.dart 194:5    _RawReceivePort._handleMessage\n","isFailure":false,"type":"error","time":18860}

This is very annoying, because our CI build used to be very stable and now they are so flaky that it is nearly impossible to get a green pipeline without retrying a failed build. :(

I would really like to help here, what can I do?

Without even knowing the cause, an acceptable solution might be to simply add a retry after a 20 ms pause.

Navigation Menu

Repository navigation

`runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker` #164591

17 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker #164591

Description

Activity

alexmarkov commented on Mar 5, 2025

alexmarkov commented on Mar 5, 2025

alexmarkov commented on Mar 5, 2025

rmacnak-google commented on Mar 5, 2025

jonahwilliams commented on Mar 10, 2025

aam commented on Mar 12, 2025

a-siva commented on Mar 26, 2025

17 remaining items

a-siva commented on Apr 23, 2025

aam commented on Apr 25, 2025

a-siva commented on Apr 28, 2025

a-siva commented on Apr 29, 2025

alexmarkov commented on Jun 4, 2025

apulsifer commented on Jun 19, 2025

apulsifer commented on Jun 24, 2025

aam commented on Jul 15, 2025

a-siva commented on Jul 21, 2025

MichaelTamm commented on Oct 14, 2025

apulsifer commented on Oct 14, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`runtime/vm/os_thread.cc: 348: error: Could not start thread DartWorker` #164591