Repository navigation
stick-movie-plank-alien is flaky #42241
Description
Activity
I've unplugged this badly behaved bot - it's having issues paving again.
Is there a playbook for determining the health of these bots before they are added to the pool? This was the cause of 9 flakes in the past two days and has held up autorollers before because manual intervention was necessary over the weekend or over holidays.
There is no playbook yet for fuchsia devices on luci infra but we are creating one during this quarter. I've been monitoring those devices daily but I haven't found the reason why stick-movie-plank-alien is flaking more frequently than the others. I changed the ethernet cable yesterday and run manually multiple times and it passed 100% of the times.
I'll experiment a couple more things tomorrow to see if I can find the reason is failing
@godofredoc - how about we just replace the machine? :) It seems like we've lost a number of hours at this point on this machine in particular and something is just off with it.
We definitely can replace it but the same problem is happening in the other bots although less frequently. Having this one replicating the problem almost 100% will make it easier to replicate. I'll try to find the problem and if I can't will replace it with a new machine by the end of the day.
Reacted by Dan Field- If the unreliability is happening at all with any frequency on other machines, can we file a separate issue (or just amend this one) with detailed information about the machine, peripherals in use (vendors, date of purchase, etc.), network configuration, etc.. I'll go chase it down. Infra reliability has severe knock on issues and I'd like to treat tree closures (due for any reason) with utmost severity. I suspect, speculatively tinkering on the bots and manually testing for reliability is probably not the best use of your time either.
2: Thanks for working on the playbook. In the meantime, I'd like to minimize tree closure due to misbehaving bots while also allowing you to tinker on the bots. Can we create a separate pool for such bots that runs all jobs? Once we know the bots are reasonably cooked in this pool, we can migrate then to theprodortestspools.
- If the unreliability is happening at all with any frequency on other machines, can we file a separate issue (or just amend this one) with detailed information about the machine, peripherals in use (vendors, date of purchase, etc.), network configuration, etc.. I'll go chase it down. Infra reliability has severe knock on issues and I'd like to treat tree closures (due for any reason) with utmost severity. I suspect, speculatively tinkering on the bots and manually testing for reliability is probably not the best use of your time either.
The remaining issues are related to timing in tooling which make the bots unstable:
https://github.com/flutter/packages/blob/master/packages/fuchsia_ctl/lib/src/dev_finder.dart#L87
The dev_finder utility is trying to get the ip address after paving with 30 retries and 2 seconds delay(1 minute). Seems like 1 minute is not always enough for the device to come up and advertise itself after paving.
When this happens the devices are not rebooted in zedboot mode and then they get quarantined
We should be able to extend that to 5 minutes. But that also seems long.
Either way it seems like stick has some other issue
The remaining issues are related to timing in tooling which make the bots unstable:
Sounds good. Is anyone chasing this right now?
Uploading a CL to increase the timeout to 2 mins
Reacted by Chinmay GardeDevice was removed from pool and getting replaced with a new one.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.- locked as resolved and limited conversation to collaborators
on Aug 22, 2021
Last week it wouldn't pave - took it to my desk and it paved fine, plugged it back in and it paved again.
Last night, it had an ErrInternal when trying to serve packages: https://ci.chromium.org/p/flutter/builders/prod/Linux%20Fuchsia/1005
/cc @godofredoc @digiter @keyonghan