Skip to content

Updating progress pane using timer causes race condition that would randomly crash powershell #2800

Description

With PR #2640, a timer is used to trigger updating progress pane every 100 ms to improve progress record rendering performance (commit: 760a9c2). Before the change, _progPane.Show and _pendingProgress.Update used to be called on the same thread, and now they will be called on different threads. This causes race conditions -- for all type members that are used by both methods, they might be changed at the same time and the states may be corrupted.

The screenshot below is from a recent CI build on master branch. The exception was raised from RenderHelper, and that was because it was actually enumerating over PendingProgress._topLevelNodes and at the same time new node was being added to/removed from _topLevelNodes by Update.

image

/cc Ilya (@iSazonov) Jason Shirk (@lzybkr)

Activity

  1. iSazonov commented on Nov 29, 2016

    @iSazonov
    Collaborator

    Dongbo Wang (@daxian-dbw) Is the #2771 addressed this?

  2. iSazonov commented on Nov 29, 2016

    @iSazonov
    Collaborator

    Dongbo Wang (@daxian-dbw) I specially made restart the timer after completing _progPane.Show to exclude collisions but missed the need to control in HandleIncomingProgressRecord.
    Perhaps it would be easier to add a lock control in the HandleIncomingProgressRecord and ProgressPaneUpdateTimerElapsed.
    /cc Steve Lee (@SteveL-MSFT)

  3. lzybkr commented on Nov 29, 2016

    @lzybkr
    Contributor

    Does #2771 fix this?

  4. daxian-dbw commented on Nov 29, 2016

    @daxian-dbw
    MemberAuthor

    Ah, I didn't know we already had an issue for this: #2756.
    However, _topLevelNodes is not the only type member that is shared between the main pipeline thread and _progPane.Show running on the timer thread, _rawui and _progressRegion are another 2 examples. I think we need to analyze all those members in case any of them have similar race conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions