Repository navigation
flutter_gallery_ios32__transition_perf test consistently failing with VMServiceFlutterDriver error #57756
Description
Full stack trace for the error is at: https://gist.github.com/iskakaushik/fef32af4d559ee0f5450ec65f07f6a16
2020-05-18T14:28:09.819332: stderr: VMServiceFlutterDriver: Instance of '_WebSocketImpl' is closed with an unexpected code 1005
2020-05-18T14:28:09.819559: stderr: VMServiceFlutterDriver: Instance of '_WebSocketImpl' is closed with an unexpected code 1005
This seems to be happening on slower/older devices, i.e, iPhone 4S in our dashboard and @jason-simmons' Moto G4. (also referenced in #57586 (comment))
These tests started to become consistently red from consistently green starting: #57402
Metadata
Metadata
Assignees
Labels
Critical issues such as a build break or regression"flutter test", flutter_test, or one of our testsTeam-specific productivity, code health, technical debt.Stack traces logged to the consoleTests that sometimes, but not always, incorrectly passIt was better in the past than it is nowflutter/engine related. See also e: labels.
Activity
jason-simmons commentedon May 26, 2020
I tried looking at some
Instance of '_WebSocketImpl' is closed with an unexpected code 1005failures that I saw while running this benchmark on a Linux/Android setup.The socket closure happened while the driver script was invoking
getVMTimeline. Downloading the timeline data was taking over 30 seconds.On the device side, the VM service in the app process was successfully generating the timeline JSON and writing it to the socket as expected. But it looked like the app process was being terminated by a SIGKILL before the driver script received all the data. I don't know who is sending the SIGKILL.
zanderso commentedon May 27, 2020
@jason-simmons Could the app be getting sent SIGKILL by the OS for some perceived misbehavior?
zanderso commentedon Jun 1, 2020
It looks like the tree is closed on this flake again. I noted in chat that it became noticeably flakier after the engine roll here: 68073a0. Team velocity is affected by this issue, so I am bumping the priority to P1 and assigning @jason-simmons to investigate further.
/cc @godofredoc in case he has some hints about where the SIGKILL might be coming from.
@jason-simmons please assign back to me if you won't be able to get to this sometime this week. Thanks!
(@kf6gpe sorry if I'm using the new priorities wrong!)
zanderso commentedon Jun 1, 2020
Ah, sorry, it's
new_gallery__transition_perfwhose failure is currently closing the tree on this issue, which I guess we can possibly conclude is affecting multiple benchmarks.jonahwilliams commentedon Jun 1, 2020
I'm going to make an educated guess that Android itself is sending the SIGKILL, it has similar behavior if you attempt to send "a lot" of data over the VM Service or even through print. In the Flutter framework, the print wrapper throttles output for this reason: https://github.com/flutter/flutter/blob/master/packages/flutter/lib/src/foundation/print.dart#L43
If this is the case, the solution is to add pagination to the timeline request somehow and download in chunks.
jonahwilliams commentedon Jun 1, 2020
Either that, or some component in the Dart start is being overwhelmed and doing something similar
zanderso commentedon Jun 1, 2020
@bkonyi Do you know where a good place would be to experiment with throttling the VM's response to a getVMTimeline VM service request?
jason-simmons commentedon Jun 2, 2020
Looked some more at the
new_gallery__transition_perfbenchmark. I think the app process on the device is being killed because it is reaching a memory limit. The payload of thegetVMTimelineservice RPC response is 110MB, and the app process' VmSize can approach 2GB while serializing the response.The timeline chunking patch landed by @jonahwilliams reduces peak memory consumption and appears to make the benchmark much more likely to run to completion.
zanderso commentedon Jun 2, 2020
@jason-simmons thanks for digging deeper, and thanks @jonahwilliams for the fix. Since neither flutter_gallery_ios32__transition_perf nor new_gallery__transition_perf has flaked since Jonah's change, I think we should leave the chunking in place, and remove the flaky mark from flutter_gallery_ios32__transition_perf.
github-actions commentedon Aug 13, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.