Repository navigation
DevTools + Flutter + Dart SDK memory polling race condition causing flakey tests. #157296
Description
This flake has happened I believe on other tests as well. But a good example is Linux_mokey large_image_changer_perf_android:
[2024-10-13 17:03:25.219495] [STDOUT] stderr: Bad state: StreamSink is closed
[2024-10-13 17:03:25.219572] [STDOUT] stderr: #0 _StreamSinkImpl.add (dart:_http/http_impl.dart:919:7)
[2024-10-13 17:03:25.219598] [STDOUT] stderr: #1 _WebSocketImpl.add (dart:_http/websocket_impl.dart:1223:11)
[2024-10-13 17:03:25.219627] [STDOUT] stderr: #2 VmService._processRequest (package:vm_service/src/vm_service.dart:2030:18)
[2024-10-13 17:03:25.219662] [STDOUT] stderr: <asynchronous suspension>
What appears to be happening is on the device, devtools launches a task that polls for memory information forever. It's pretty difficult sussing out exactly the flow here because the code crossed 3+ repositories (flutter/devtools/dart-lang); but a stab maybe:
FlutterTools DriverService -> startTest
if (profileMemory) -> unawaited(_devtoolsLauncher.launch(_vmServiceUri, --record-memory-profile=file)
--record-memory-profile is handled in DDS, dart-lang/sdk's devtools_server
serveDevTools -> _hookupMemoryProfiling -> connect to vm service && MemoryProfile.startPolling ->
devTools memory_profile -> startPolling ->
timer loop (_pollMemory) -> _fetchAdbInfo -> getAdbMemoryInfo() -> flutterMemory
flutterMemory is defined: packages/devtools_shared/lib/src/service_utils.dart -> flutterMemoryInfo string ->
packages/flutter_tools/lib/src/vmservice.dart -> kFlutterMemoryInfoServiceName
setUpVmService() registers a service callback for kFlutterMemoryInfoServiceName, which calls device.queryMemoryInfo and returns <String,Object>{}. If this is android, queryMemoryInfo() executes adb shell dumpsys meminfo
- Nothing calls MemoryProfile.stopProfiling. It probably should. It should also wait on the work in
_pollMemoryto complete. - I don't know where to signal this.
await driver.close?
Activity
jtmcdole commentedon Oct 21, 2024
Plausible timeline:
service -> host: please collect meminfo
host -> adb: start meminfo
host: test end - shut'er down and uninstall
host -> service: here's the info 💣
zanderso commentedon Oct 21, 2024
cc @bkonyi
bkonyi commentedon Oct 22, 2024
This looks like it could happen if we're in the middle of responding to a service extension request and the connection goes down before the response can be sent. In that case, the invoking client should have already thrown an
RPCError, but we fail to check if the connection has been disposed on the service side before trying to respond.I think this should be as simple as checking the
_disposedflag before invoking_writeMessagefrom_processRequestinpackage:vm_service.bkonyi commentedon Oct 22, 2024
Fix up for review here.
[ package:vm_service ] Don't try to send service extension response a…
a-siva commentedon Oct 28, 2024
Can this issue be closed ?
bkonyi commentedon Oct 28, 2024
The fix hasn't rolled into the Flutter repo yet so we should probably keep it open for now.
bkonyi commentedon Nov 4, 2024
This is still blocked on the Pub roller breakage: #158135
a-siva commentedon Nov 6, 2024
Fix has rolled into Flutter repo and hence closing issue.
github-actions commentedon Nov 20, 2024
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.