Repository navigation
stick-movie-plank-alien is flaky #42241
Description
Last week it wouldn't pave - took it to my desk and it paved fine, plugged it back in and it paved again.
Last night, it had an ErrInternal when trying to serve packages: https://ci.chromium.org/p/flutter/builders/prod/Linux%20Fuchsia/1005
Activity
dnfield commentedon Jan 17, 2020
I've unplugged this badly behaved bot - it's having issues paving again.
chinmaygarde commentedon Jan 17, 2020
Is there a playbook for determining the health of these bots before they are added to the pool? This was the cause of 9 flakes in the past two days and has held up autorollers before because manual intervention was necessary over the weekend or over holidays.
godofredoc commentedon Jan 17, 2020
There is no playbook yet for fuchsia devices on luci infra but we are creating one during this quarter. I've been monitoring those devices daily but I haven't found the reason why stick-movie-plank-alien is flaking more frequently than the others. I changed the ethernet cable yesterday and run manually multiple times and it passed 100% of the times.
I'll experiment a couple more things tomorrow to see if I can find the reason is failing
dnfield commentedon Jan 17, 2020
@godofredoc - how about we just replace the machine? :) It seems like we've lost a number of hours at this point on this machine in particular and something is just off with it.
godofredoc commentedon Jan 17, 2020
We definitely can replace it but the same problem is happening in the other bots although less frequently. Having this one replicating the problem almost 100% will make it easier to replicate. I'll try to find the problem and if I can't will replace it with a new machine by the end of the day.
chinmaygarde commentedon Jan 21, 2020
2: Thanks for working on the playbook. In the meantime, I'd like to minimize tree closure due to misbehaving bots while also allowing you to tinker on the bots. Can we create a separate pool for such bots that runs all jobs? Once we know the bots are reasonably cooked in this pool, we can migrate then to the
prodortestspools.godofredoc commentedon Jan 21, 2020
The remaining issues are related to timing in tooling which make the bots unstable:
https://github.com/flutter/packages/blob/master/packages/fuchsia_ctl/lib/src/dev_finder.dart#L87
The dev_finder utility is trying to get the ip address after paving with 30 retries and 2 seconds delay(1 minute). Seems like 1 minute is not always enough for the device to come up and advertise itself after paving.
When this happens the devices are not rebooted in zedboot mode and then they get quarantined
dnfield commentedon Jan 21, 2020
We should be able to extend that to 5 minutes. But that also seems long.
Either way it seems like stick has some other issue
chinmaygarde commentedon Jan 21, 2020
Sounds good. Is anyone chasing this right now?
godofredoc commentedon Jan 21, 2020
Uploading a CL to increase the timeout to 2 mins
godofredoc commentedon Feb 27, 2020
Device was removed from pool and getting replaced with a new one.
github-actions commentedon Aug 22, 2021
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.