You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
☂️ [Flake] Device lab tests and benchmarks sometimes don't start the test or benchmark. #79097
The logs indicate "Last layer tree was null when screenshotting" after a successful isolate launch while paused with a subsequent resume. I don't think see how the could possible tell when the raster thread has received the layer tree to rasterize. So, IMO, there is a race between when the driver is attempting to screenshot the layer tree and the engine getting the request to render that tree. Or, it could just be the isolate was never unpaused.
Yet another instance #78805. The more we run into this kind of error, the more I think it is the driver not waiting for the successful resumption of the isolate as described in this VM service procotol method.
changed the title [-]☂️ Better logs from ios tests in devicelab[/-][+]☂️ [Flake] Device lab tests and benchmarks sometimes don't start the test or benchmark.[/+]on Mar 29, 2021
The 8 is fine, and the 11 is fine after a restart. I wonder if there's something going on where the phone(s) get clogged up and take longer to start sometimes, perhaps longer than the 30s timeout allows.
Updating the timeline API here so that we always write the raw timeline before trying to summarize it, and a summary failure will leave us at least with the raw timeline: #79310. This covers at least cases where we get a failure in summarization that blows up the test, especially if the test itself tried to summarize before dumping the timeline.
As of now, all the tests affected should be uploading at least a timeline in the LUCI UI. For the failures where the VM is starting but we get a white screen, that will hopefully give us a little more information about what point we're getting to before it hangs up.
@jmagman I think this is related to what we were chatting about last week, for driver tests, whether on the side to also just dump syslog to stdout or to another file for more debugging information.
Ok, we should now pretty much always be uploading a timeline if the VM is available - if we're still seeing failures where the timeline doesn't get uploaded we can look at beefing that up too.
Since I'm not regularly seeing failures around this right now, I'm going to deprioritize this. And since I don't have any further action steps to take I'm going to unassign. Leaving open for now in case we continue to see this happen over the next week.
Since @dnfield has landed a change to get better logs and timelines, this probably makes more sense to track on a case-by-case basis, now, for example in #77193.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
Activity
zanderso commentedon Mar 25, 2021
Also #79095
chinmaygarde commentedon Mar 29, 2021
The logs indicate "Last layer tree was null when screenshotting" after a successful isolate launch while paused with a subsequent resume. I don't think see how the could possible tell when the raster thread has received the layer tree to rasterize. So, IMO, there is a race between when the driver is attempting to screenshot the layer tree and the engine getting the request to render that tree. Or, it could just be the isolate was never unpaused.
chinmaygarde commentedon Mar 29, 2021
cc @xster for routing.
[-]Better logs from ios tests in devicelab[/-][+]☂️ Better logs from ios tests in devicelab[/+]chinmaygarde commentedon Mar 29, 2021
Another instance is at #78920
chinmaygarde commentedon Mar 29, 2021
Yet another instance #78805. The more we run into this kind of error, the more I think it is the driver not waiting for the successful resumption of the isolate as described in this VM service procotol method.
[-]☂️ Better logs from ios tests in devicelab[/-][+]☂️ [Flake] Device lab tests and benchmarks sometimes don't start the test or benchmark.[/+]chinmaygarde commentedon Mar 29, 2021
Another one #77102. Bumping up the priority as this seems like a widely prevalent issue with multiple harnesses.
3 remaining items
dnfield commentedon Mar 29, 2021
The 8 is fine, and the 11 is fine after a restart. I wonder if there's something going on where the phone(s) get clogged up and take longer to start sometimes, perhaps longer than the 30s timeout allows.
dnfield commentedon Mar 29, 2021
Updating the recipes here so that we always upload whatever timelines we get: https://flutter-review.googlesource.com/c/recipes/+/11622
Updating the timeline API here so that we always write the raw timeline before trying to summarize it, and a summary failure will leave us at least with the raw timeline: #79310. This covers at least cases where we get a failure in summarization that blows up the test, especially if the test itself tried to summarize before dumping the timeline.
dnfield commentedon Mar 31, 2021
As of now, all the tests affected should be uploading at least a timeline in the LUCI UI. For the failures where the VM is starting but we get a white screen, that will hopefully give us a little more information about what point we're getting to before it hangs up.
dnfield commentedon Mar 31, 2021
#79310 will fix potential bugs where we'd miss a timeline because we try to summarize a timeline that had no frames.
xster commentedon Mar 31, 2021
@jmagman I think this is related to what we were chatting about last week, for driver tests, whether on the side to also just dump syslog to stdout or to another file for more debugging information.
dnfield commentedon Mar 31, 2021
Ok, we should now pretty much always be uploading a timeline if the VM is available - if we're still seeing failures where the timeline doesn't get uploaded we can look at beefing that up too.
Since I'm not regularly seeing failures around this right now, I'm going to deprioritize this. And since I don't have any further action steps to take I'm going to unassign. Leaving open for now in case we continue to see this happen over the next week.
zanderso commentedon Apr 28, 2021
Since @dnfield has landed a change to get better logs and timelines, this probably makes more sense to track on a case-by-case basis, now, for example in #77193.
github-actions commentedon Aug 2, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.