Async cmdlets

I've recently tried to write a cmdlet that had to use async APIs. Doing so turned out to be incredibly difficult, and with help I found an AsyncCmdlet class that lets you override processRecordAsync etc. as async functions that get CancellationTokens:
https://github.com/felixfbecker/PSKubectl/blob/master/src/AsyncCmdlet.cs
https://github.com/felixfbecker/PSKubectl/blob/master/src/ThreadAffinitiveSynchronizationContext.cs

Could PowerShell provide this out of the box?

Felix Becker (@felixfbecker)

Felix Becker (@felixfbecker) So if I understand you correctly, you'd like to see an in-box variant of the PSCmdlet base class possibly called PSAsyncCmdlet that lets you write synchronous cmdlets that use asynchronous APIs and the await keyword including built-in cancellation support? (Note: cmdlets are always synchronous WRT the pipeline but may have asynchronous implementations). I'm currently working on some stuff to let you do write asynchronous task-based scripts but I don't think this overlaps that. This is really an SDK request instead of an end-user feature - right?

Yep

I just ran into the exact same thing. When I tried to implement a solution, it got pretty messy with Tasks blocking when awaiting.

I'd like to see this feature as well since more and more libraries are async-only...

Felix Becker (@felixfbecker)

Just came across this issue today. Thanks for posting Felix Becker (@felixfbecker).

Ditto. In my case I am finding weird situations where the pipeline exits when I invoke a method with await.

This seems to work well. I just submitted a PR to it to use dotnetcore.

https://github.com/ttrider/PowerShellAsync

Maybe this could be a starting point for official support in PS?

GitHub
ttrider/PowerShellAsync
create async cmdlets for PowerShell. Contribute to ttrider/PowerShellAsync development by creating an account on GitHub.

This seems to work well. I just submitted a PR to it to use dotnetcore.

https://github.com/ttrider/PowerShellAsync

Maybe this could be a starting point for official support in PS?

GitHub**ttrider/PowerShellAsync**create async cmdlets for PowerShell. Contribute to ttrider/PowerShellAsync development by creating an account on GitHub.

I take it back. I'm finding ttrider/PowerShellAsync actually doesn't work as expected. I'm not savvy enough to debug it. It's pretty easy to repro what I'm experiencing by doing something like this:

        protected override async Task EndProcessingAsync() {
            _cancellationToken = new CancellationTokenSource();
            var version = await ServiceLocator.Current.UpdateService.GetLatestStableVersionAsync(_cancellationToken.Token);
            Debug.WriteLine($"Version: {version}");
            var path = await ServiceLocator.Current.UpdateService.StartUpgradeAsync();
            Debug.WriteLine($"path: {path}");
            ShouldContinue("my message.", "Exit this Powershell instance?");
            Debug.WriteLine($"done.");
            await base.EndProcessingAsync();
        }

ShouldContinue wedges and "done" is never written to the debug console.

The two async methods I call are really just wrappers around WebClient, so nothing fancy.

GitHub
ttrider/PowerShellAsync
create async cmdlets for PowerShell. Contribute to ttrider/PowerShellAsync development by creating an account on GitHub.

Tig (@tig)

Tig (@tig) are you seeing the same problem with https://github.com/felixfbecker/PSKubectl/blob/master/src/AsyncCmdlet.cs too? If not, maybe we could use that as a starting point instead.

GitHub
felixfbecker/PSKubectl
kubectl with the power of the object pipeline. Contribute to felixfbecker/PSKubectl development by creating an account on GitHub.

Tig (@tig)

Tig (@tig) are you seeing the same problem with https://github.com/felixfbecker/PSKubectl/blob/master/src/AsyncCmdlet.cs too? If not, maybe we could use that as a starting point instead.

I'll take a look at.

For now, FWIW, a bunch of debugging last night gave me SOME insight into the problem with PowerShellAsync, but not enough for me to actually suggest how to fix it.

If I do this:

 var path = await ServiceLocator.Current.UpdateService.StartUpgradeAsync();
 ShouldContinue("my message.", "Exit this Powershell instance?");

The deadlock happens here:

https://github.com/ttrider/PowerShellAsync/blob/66eb99e33516532749cdd81ced3faf210361a9ee/PowerShellAsync/AsyncCmdlet.cs#L294

However, if I do this:

 var path = await ServiceLocator.Current.UpdateService.StartUpgradeAsync();
 await Task.Run(() => ShouldContinue("my message.", "Exit this Powershell instance?"));

That Wait doesn't block, but instead the task gets pulled off the queue and invoked via:

https://github.com/ttrider/PowerShellAsync/blob/66eb99e33516532749cdd81ced3faf210361a9ee/PowerShellAsync/AsyncCmdlet.cs#L200

I think there's probably something that could be done to how methods like ShouldContinue are wrapped by PowerShellAsync to fix this:

https://github.com/ttrider/PowerShellAsync/blob/66eb99e33516532749cdd81ced3faf210361a9ee/PowerShellAsync/AsyncCmdlet.cs#L117

For now, I'm moving forward by just wrapping CmdLet calls like ShouldContinue and WriteProgress which appear to not be correctlyl wrapped by PowerShellAsync with Task.Run calls.

Felix Becker (@felixfbecker)

Felix Becker (@felixfbecker) I need help. I just can't get my head around this stuff enough to actually fix the problem. I'm currently using your version and things are mostly working. However, WriteDebug/Error etc... need to run on the main cmdlet thread and I'm not smart enough to modify your code to do so.

Not sure if this Issue is the right place for me to get educated. Please send me elsewhere if there's a better forum for this topic. I'm putting it here for now because a) you are here and b) this really is all about building async cmdlets....

To set context, I've built a Serilog sink for PowerShell CmdLets. My libraries use Serilog logging throughout for other clients (mainly a Winforms app) and I want to redirect Log.Debug -> WriteDebug, Log.Information -> WriteVerbose, etc... in my PowerShell cmdlet.

I discovered along the way that Serilog's singleton pattern makes running within a shared host like Powershell challenging, but I've worked around it via a registration model from my CmdLet.

See https://github.com/tig/winprint/blob/master/src/WinPrint.Console/PowerShellSink.cs

Emit is often called from threads other than the cmdlet thread. Thus I get exceptions like:

Exception thrown: 'System.Management.Automation.PSInvalidOperationException' in System.Management.Automation.dll
The WriteObject and WriteError methods cannot be called from outside the overrides of the BeginProcessing, ProcessRecord, and EndProcessing methods, and they can only be called from within the same thread. Validate that the cmdlet makes these calls correctly, or contact Microsoft Customer Support Services.

And WriteDebug just fails silently when on the wrong thread (apparently).

It seems the solution should be simple:

Wrap PSCmdLet.WriteDebug et. al. in SynchronizationContext.Post calls. If I'm understanding this right, the message pump in ThreadAffinitiveSynchronizationContext will pick these up and run them on the right thread.

I don't think the combination of your AsyncCmdlet and ThreadAffinitiveSynchronizationContext are fully baked because when I do this:

    public new void WriteDebug(string text) {
        
        ThreadAffinitiveSynchronizationContext.Current.Post(new SendOrPostCallback((o) => {
            base.WriteDebug(text);
        }), null);
    }

Current is null on those calls and I obviously need to prime the pump somehow, this is where I need a hint.

I won't feel too bad if y'all just tell me to go really study how to do threading right. I'm actually doing all this specifically to learn. So...

GitHub
tig/winprint
WinPrint 2.0 - A modern take on the the classic source code printing app from 1988. WinPrint is the perfect tool for printing source code, web pages, reports generated by legacy systems, documenta...

Tig (@tig)

Any updates on async cmdlet support?

Tig (@tig) Have you ever found a solution to the problem you described?

or admit that Powershell binary module writing can't truly be combined with modern Microsoft software development practices and provide recommendations on how to avoid async problems and still write useful, reliable programs using what that leaves us with.

Currently that is the case.

One solution is for a Cmdlet to spawn its own thread to run async tasks in and wait for that thread to complete, but those tasks can't use any other PowerShell API, including WriteObject. Any writing to output streams needs to be done by the Cmdlet itself in the original thread/context.

Perhaps PowerShell 8 will be the async aware one, which uses a special compatibility mode to run PowerShell 5 and PowerShell 7 modules.

I stumbled into this problem when writing my https://github.com/Luiz-Monad/PowershellProcess/blob/9172154e707cf01f1d3eb054b1c88beed9939d9b/PowerProcess/InvokeProcessFast.cs#L439 library, which is a faster text streaming re-implementation of Start-Process which I use with CMake for tracing.
The solution for the problem was using Nito.AsyncEx.Context.

The problem is as follows.
Any and all interaction with the powershell Engine has to be done via its only thread, think of it like a Win32 UI Thread.
Every single async task that runs should run in the thread pool and NEVER block the main thread (preferentially never blocking any of the threads of the pool, but that's not a problem as it will spawn more for us and will do work stealing).
That automatically leads us to something like ConfigureAwait(true) to capture the sync-context (another outstanding dotnet gotcha because people will blindly set it to false, thus breaking the synchronization context, but this is the case we actually want it, but we can ensure that by scheduling the task manually)

So the solution is the other way around. You don't need a new thread to be blocked waiting the async to finish while the powershell thread is also blocked.
What you need to do is to block the Powershell thread pumping messages from a queue of actions, any new Task that will run is going to run in the powershell engine thread. You basically serialize the execution of the Async and block the thread until all async Tasks finished.
Async is for I/O not for parallelism, so having the concurrency set to 1 in this case is appropriate. If you need anything else in Async you should do use fork/join and a different synchronization context for that and do that on your own. It is not a problem for an async task to block the powershell thread waiting for things in another synchronization context, as long as you NEVER do reentrancy, aka, you have to use fork/join and only access the powershell engine things (like WriteObject) after a join, or use a synchronized queue.

Doing what the PowerShellAsync did will deadlock because you shouldn't wait on the result Tasks, you have to wait on the Task Scheduler and run the actual async tasks in the Powershell thread, you have to serialize the async, not to block it, that's ripe for deadlock.

Also, what Windows Forms did was sprinkle cross-threading checks everywhere, and ConfigureAwait is true by default (which is kinda bad, but that's another problem), then the call always go back to the UI thread.
But then Windows Forms has its own SynchronizationContext.

So the solution for powershell is to do the same, have its own SynchronizationContext, and then the implementation of the AsyncCmdLet can use a tactic similar to Nito.AsyncEx.Context.
That will cover most of the needs for AsyncCmdLet, but the problem is that its impossible to ensure that the user of AsyncCmdLet will do the right thing, but not having anything on the SDK ensures that 100% they will do the wrong thing, which is bad.
So why not have an AsyncCmdLet that works fine for 90% of the cases, which can be covered by using the AsyncEx and having a SynchonizationContext ?

I stumbled back on this because I'm implementing a CmdLet to call SignalR and create a Job2 for me.
So, to conclude.

push the async start to the synccontext and run the taskpool
any callback to the engine should be an action post to the taskpool
never use any other engine api without posting to the taskpool (no way to enforce that, crossthreading exceptions is the better than can be done)
in the endprocessing, block the powershell engine thread and wait for the taskpool to get empty, that's the only blocking ever allowed
the AsyncEx context will run all async tasks synchronously in the engine thread, thus it guarantees thread safety, everything runs in one thread
serializing async code might introduce deadlocks, but that's the problem in the async code because people might have used locking synchonization primitives in async code, which is bad

GitHub
PowershellProcess/PowerProcess/InvokeProcessFast.cs at 9172154e707cf01f1d3eb054b1c88beed9939d9b · Luiz-Monad/PowershellProcess
Contribute to Luiz-Monad/PowershellProcess development by creating an account on GitHub.

[edit] the proposed implementation:
https://github.com/Luiz-Monad/PowerShellAsync/blob/f9c6e738dbf6f51ca44f967b0e9369c8d0b213f9/PowerShellAsync/AsyncCmdlet.cs#L17

I published my package to do that here https://www.nuget.org/packages/MonadicSoftware.PowerShellAsync

MonadicSoftware.PowerShellAsync 2.0.0
Base class for async-supported PowerShell cmdlets.

I published my package to do that here

What happens when different cmdlets use different versions of your PowerShellAsync package and conflict at runtime? Either one will get a different version to what it expected or fail to load. With dotnet in a single assembly load context there can be only one version of a given assembly.

I published my package to do that here

What happens when different cmdlets use different versions of your PowerShellAsync package and conflict at runtime? Either one will get a different version to what it expected or fail to load. With dotnet in a single assembly load context there can be only one version of a given assembly.

Good question.
What happens in this case is that only the first assembly is able to be loaded, whoever was loaded first because Powershell uses Assembly.LoadFrom to load assemblies manually.
All of the assemblies are going to be subject to the Powershell module resolution because it has a custom Assembly Context handler.

PowerShell/src/System.Management.Automation/engine/ExecutionContext.cs

Line 1388 in 311674b

loadedAssembly = Assembly.LoadFrom(filePath);

PowerShell/src/System.Management.Automation/CoreCLR/CorePsAssemblyLoadContext.cs

Line 98 in 311674b

AssemblyLoadContext.Default.Resolving += Resolve;

But if you load the most recent one first, then it works.

Because powershell only cares about the name.

PowerShell/src/System.Management.Automation/CoreCLR/CorePsAssemblyLoadContext.cs

Line 436 in 311674b

if (s_assemblyCache.TryGetValue(assemblyName.Name, out asmLoaded))

I don't know how to solve this problem sort of having the LoadFrom in the Microsoft.PowerShell_profile.ps1 user profile.
Usually when its a dotnet application and you have problems like that you use something like a binding policy

<configuration>
 <runtime>
   <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
     <dependentAssembly>
       <assemblyIdentity name="MyAssembly" publicKeyToken="..." culture="neutral" />
       <bindingRedirect oldVersion="1.0.0.0-2.0.0.0" newVersion="2.0.0.0" />
     </dependentAssembly>
   </assemblyBinding>
 </runtime>
</configuration>

Is there something similar I could use in a "psd1" ? I don't remember seeing anything.
When creating my PowerSignalR.dll maybe I should list a RequiredModules for another powershell module which provides the actual DLL for PowershellAsync.

The way I'm doing is using nuget and just deploying both libraries.
edit: I'm still finishing my other module, I'm going to publish both the PowershellAsync in the powershellgallery and the PowerSignalR, and let the module installer do its job, I think that solves the problem. I never used RequiredModules before, all my powershell modules were always self-contained. But in this case it wont work.

When the class is not the PSCmdlet class then you can use an assembly load context and load the classes in a new context.

When the class is the PSCmdlet and loaded by PowerShell I think it has to be a unique class name in the module's root module dll.

Which really means each module needs to copy the source to AsyncCmdlet and give it a new unique namespace to avoid conficts.

That's one way of solving that problem. Previously I used NitoAsyncEx and copied its source.
https://github.com/Luiz-Monad/PowershellProcess/tree/master/PowerProcess/Nito.AsyncEx.Context

GitHub
PowershellProcess/PowerProcess/Nito.AsyncEx.Context at master · Luiz-Monad/PowershellProcess
Contribute to Luiz-Monad/PowershellProcess development by creating an account on GitHub.

But lets see if PowershellGallery is able to solve that problem by installing a newer dependency as long as I don't specify versions in the RequiredModules on the psd1, it should work.

I'll try that tomorrow, I'm doing this in my spare time outside commercial projects of mine. I spent the time I had today, but thanks for the questions, It helped me think better about how I can integrate this sort of thing. Learning was the purpose of the exercise.

Luiz (@Luiz-Monad)

Luiz (@Luiz-Monad) This is a document left by a former PowerShell team member, it doesn't contain an Async Cmdlet part, but it has everything else. https://gist.github.com/rjmholt/7c1072a24a51e7b43d1e7df02265fe18

Gist
Using the PowerShell API
Using the PowerShell API. GitHub Gist: instantly share code, notes, and snippets.

@Luiz-Monad This is a document left by a former PowerShell team member, it doesn't contain an Async Cmdlet part, but it has everything else. https://gist.github.com/rjmholt/7c1072a24a51e7b43d1e7df02265fe18

Gist**Using the PowerShell API**Using the PowerShell API. GitHub Gist: instantly share code, notes, and snippets.

Thank you, that would have been helpful when I started writing my https://www.nuget.org/packages/MonadicSoftware.PowerShellAsync , I had to figure all of that by reading the Powershell source code and debugging it.

Gist
Using the PowerShell API
Using the PowerShell API. GitHub Gist: instantly share code, notes, and snippets.

MonadicSoftware.PowerShellAsync 2.0.0
Base class for async-supported PowerShell cmdlets.

The Engine WG discussed this today, and since the underlying framework is moving in an async direction, we think that PS should ensure that this is also available to cmdlet developers. We believe that this behavior is very useful, we would want to see a detailed RFC which fully investigates the behavior. We also believe that the PS team should own both the RFC and the implementation because this is so core to the engine.

Yay!!!

Navigation Menu

Repository navigation

Async cmdlets #7690

30 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Async cmdlets #7690

Description

Activity

BrucePay commented on Sep 2, 2018

felixfbecker commented on Sep 2, 2018

michsco commented on Apr 12, 2019

henning-krause commented on Oct 14, 2019

pcgeek86 commented on Dec 4, 2019

tig commented on Mar 17, 2020

tig commented on Mar 18, 2020

tig commented on Apr 3, 2020

felixfbecker commented on Apr 3, 2020

tig commented on Apr 3, 2020

tig commented on Apr 12, 2020

chausner commented on Mar 16, 2021

30 remaining items

Luiz-Monad commented on Dec 20, 2024

Luiz-Monad commented on Jan 10, 2025

rhubarb-geek-nz commented on Jan 11, 2025

Luiz-Monad commented on Jan 13, 2025

Luiz-Monad commented on Jan 13, 2025

rhubarb-geek-nz commented on Jan 13, 2025

Luiz-Monad commented on Jan 13, 2025

fMichaleczek commented on Jan 14, 2025

Luiz-Monad commented on Feb 2, 2025

JamesWTruher commented on Sep 22, 2025

tig commented on Sep 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions