Skip to content

some devicelab win machines offline #107872

Closed

Description

@yusuf-goog
No description provided.

Activity

  1. self-assigned this
    on Jul 18, 2022
  2. yusuf-goog commented on Jul 18, 2022

    @yusuf-goog
    ContributorAuthor

    win-3 had windows update which needed the dialogs to be accepted.

    win-3,4,6 had password for swarming account got changed to the alternate one. Changed it back to our flutter password and it works.

  3. keyonghan commented on Jul 19, 2022

    @keyonghan
    Contributor

    Seems win 3, 4, 6 are offline again.

  4. yusuf-goog commented on Jul 19, 2022

    @yusuf-goog
    ContributorAuthor

    possibly the password is being overwritten by the password stored in the salt state, which is different from our standard password thats been added to the local machine, causing it to not login automatically.

    Will try to remotely fix, or will look at it when I go into the lab tomorrow.

  5. yusuf-goog commented on Jul 19, 2022

    @yusuf-goog
    ContributorAuthor

    can ssh and vnc into win-3.

    Everything seems to be looking fine. Command prompt is showing swarming bot is started.

    Swarming bot log extract here: https://paste.googleplex.com/6031265670103040

    Seems to indicate it had issues contacting auth server?

  6. keyonghan commented on Aug 2, 2022

    @keyonghan
    Contributor

    @yusuf-goog what would be the next step here? Now 3 out of 4 are down in prod, and 1 out of 2 is down in staging.

  7. keyonghan commented on Aug 2, 2022

    @keyonghan
    Contributor

    Restart win3 and win4, and they are back.
    win2 is not sshable though.

  8. yusuf-goog commented on Aug 2, 2022

    @yusuf-goog
    ContributorAuthor

    Maybe it needed some time. Win-2 is sshable now.

    Logs from win-2 show problem renewing certs: https://paste.googleplex.com/5164869304188928

    Reaching out to LUCI team for guidance.

  9. yusuf-goog commented on Aug 4, 2022

    @yusuf-goog
    ContributorAuthor

    Win-3 hit windows task scheduler error 2147943726, which is caused by a password change. Reset the password in task scheduler job properties and bot is back up.

  10. yusuf-goog commented on Aug 4, 2022

    @yusuf-goog
    ContributorAuthor

    win-4 and win-6, windows "require username/password option" under command netplwiz was reset and so needed login when the host was rebooted. Unchecked the box, entered password and bot is now up.

  11. yusuf-goog commented on Aug 4, 2022

    @yusuf-goog
    ContributorAuthor

    all windows hosts are back up.

  12. github-actions commented on Aug 18, 2022

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  13. locked as resolved and limited conversation to collaborators on Aug 18, 2022
  14. added
    P0Critical issues such as a build break or regression
    and removed on Jun 28, 2023
  15. moved this from Verified to To be archived in [DEPRECATED] Infra Ticket Queueon Jul 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure team

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions