Feature Request: Async Cmdlets

Summary of the new feature/enhancement

Start-ThreadJob is a great tool, although it's a bit cumbersome if you just need to invoke single cmdlet with parameters. Would be nice just to indicate this with some switch, e.g. "Async". Somewhat similar to AsJob switch, but returning task or thread job.

Example

Invoke-WebRequest -Uri $Url -Method $Method -Headers $h -Async

This should return task object, or be a shortcut to this:

Start-ThreadJob -ScriptBlock { param($Url, $Method, $h)

    Invoke-WebRequest -Uri $Url -Method $Method -Headers $h

} -ArgumentList ($Url, $Method, $h)

If not possible as general mechanism, this could be implemented for some selected cmdlets. I think this could be useful for iwr/irm, test-connection and GetWMI-Object

You can do this with the & suffix operator:

$job = Invoke-WebRequest $Url -Method $Method -Headers $headers &
$job | Receive-Job -Wait

vexx

vexx I believe this is shortcut for start-job. I was thinking about runspace/task mechanism.

Isn't this a dupe of #9873?

Steve Lee (@SteveL-MSFT)

It is in intent, Steve Lee (@SteveL-MSFT), but not in suggested implementation.

mikeTWC1984, #9873 suggests a syntax that is part of the "plumbing" rather than relying on parameters supported by individual cmdlets (or even a new common parameter), which to me is the better way to go.

I believe the ask is to have ubiquitous parameter to run as a threadjob, like -AsThreadJob. This would be more discoverable than a new background operator. We DO currently have both -AsJob and the background operator, so I guess we COULD have both.

Paul Higinbotham (@PaulHigin)

Making -AsThreadJob a common parameter would be... interesting. Paul Higinbotham (@PaulHigin), any thoughts on that?

Yes, making it a common parameter would be an alternative way to make the functionality available to every cmdlet, but I don't think it belongs there conceptually:

It's awkward to specify an invocation method - something that is extrinsic to the invoked cmdlet - as one of its parameters.

Note to how the existing -AsJob parameter is sensibly not a common parameter, because it only applies to cmdlets that themselves have invocation functionality, such as Invoke-Command.

Also, an operator such as the postpositional & is more comprehensive in that it also covers expressions (to use an overly simplistic example: 1..10 & ).

As for discoverability: yes, all symbol-based functionality is not easy to discover, but the reason for choosing a symbol to begin with is that the feature is so common (or is expected to become that) that it warrants maximum concision.

With frequent use and exposure comes familiarity.

Also, to help with discoverability, the feature should be documented alongside its verbose equivalent, i.e. Start-ThreadJob, just as postpositional use of & should be documented in the Start-Job topic - but currently isn't: see MicrosoftDocs/PowerShell-Docs#4965

Michael Klement (@mklement0)

There's another problem with using -AsJob as a common parameter.

In PowerShell 6.x and earlier, we have -AsJob as a parameter on some commands (most notably: Invoke-Command), and it returns a PSRemotingJob for the job that is being run in the background.

In PowerShell 7, we have -AsJob as a parameter on the new ForEach-Object -parallel parameter set, and it returns a PSTaskJob.

We also have ThreadJob as a third type for jobs that are launched using Start-ThreadJob.
(Aside: I find it quite confusing that we have ThreadJob and PSTaskJob types -- do we need two types of jobs here or can we move forward with just one?)

If we were to try going the common parameter route, do we provide a common parameter for each type of job? Or have a default job type for -AsJob and another common parameter to define the job type to use?

Plus, as Michael Klement (@mklement0) pointed out, using common parameters here feels awkward. I'd rather we control invocation of commands using post-positional operators.

For those who are interested, I opened an RFC for this a while back when I felt this was missing and would add value to the PowerShell async story. You can read that proposal and add comments in PowerShell/PowerShell-RFC#205.

I actually came to this from .net's async methods (like DoSmth()/DoSmthAsync()). As I understand from above there is already a mechanism to invoke cmdlet asynchronously without creating extra scopes (which was my main concern) and we just discuss implementation. I'm good with either operator or switch, latter sounds more intuitive to me, but I agree this could be awkward to see it on every cmdlet. Probably switch appearance could be indicated inside CmdletBinding attribute.

Btw, what is a PSTaskJob? Thread job seem to create a decent overhead, unlike task that came from async methods. Wondering if PSTaskJob can improve this. Below are some dummy examples.

This eats/cloggs 100MB+ of memory on my machine (on average)

1..100 | foreach {Start-ThreadJob -ScriptBlock {sleep 1} }

This takes 1-2 MB

1..100 | foreach {([System.Net.NetworkInformation.Ping]::new()).SendPingAsync("www.example.com") }

This is all a great discussion.

Cmdlets aren't really designed to run asynchronously. There are threading, data streaming, and state sharing issues. Over the years PowerShell has added various ways to run script/commands asynchronously: PowerShell remoting, Start-Job, Workflows, asynchronous API.

More recently I have tried to make the async API easier to use through ThreadJobs and the new ForEach-Object -Parallel experimental feature. This has been mostly successful but has also accentuated PowerShell's inherent limitations, in particular the large overhead required to run PowerShell scripts concurrently (see my blog post for more information: (https://devblogs.microsoft.com/powershell/powershell-foreach-object-parallel-feature/).

I am a bit ambivalent about adding syntactical ways to automatically invoke cmdlets in parallel. The feedback we have received from the new ForEach -Parallel experimental feature makes clear that it does not meet many users expectations. The large overhead, often slower performance, and lack of awareness of basic threading safety makes ThreadJobs and ForEach -Parallel potentially dangerous to use, or at minimum a disappointment for some users (again see blog link above for more explanation).

I would love to build something that 'just works'. But unfortunately, due to inherent PowerShell design limitations, this is not possible and each existing concurrent model has its own benefits and side effects.

PowerShell
PowerShell ForEach-Object Parallel Feature | PowerShell
PowerShell ForEach-Object Parallel Feature PowerShell 7.0 Preview 3 is now available with a new ForEach-Object Parallel Experimental feature. This feature is a great new tool for parallelizing work, but like any tool, it has its uses and drawbacks. This article describes this new feature,

Adding -AsJob to commands that can natively support it is the right way to go. It is the existing idiom for running things asynchronously in PowerShell. And regardless of the underlying implementation, -AsJob should always return a Job2 object. PowerShell jobs are the abstraction for asynchronous operations. As long as you return a Job object, the user can use the standard *-Job cmdlets to manipulate it. If you break this abstraction, then you'll end up rewriting these cmdlets for each type of asynchronous "handle". In the OP's example Invoke-WebRequest -Uri $Url -Method $Method -Headers $h -AsJob would probably return a subclass of Job2 that wrapped a Task. Not a background job object, not a threadjob but still a job object that can be manipulated with standard patterns.

Bruce Payette (@BrucePay)

Bruce Payette (@BrucePay)

Adding -AsJob to commands that can natively support it is the right way to go.

It is, but only for "meta cmdlets" that themselves invoke other commands - but that isn't the issue at hand, which is about a general-purpose, cmdlet-agnostic way to invoke any cmdlet in a separate thread.

Thanks for the background information, Paul Higinbotham (@PaulHigin) - I think it would make sense to create a conceptual help topic based on your blog post.

I am a bit ambivalent about adding syntactical ways to automatically invoke cmdlets in parallel.

There's no way around users needing to understand the constraints and appropriate use case, and burdening those that do with verbose syntax doesn't sound like the best way forward.

We already have postpositional & to make implicit Start-Job calls easy, so there's no reason not to provide the same convenience for Start-ThreadJob - see Kirk Munro (@KirkMunro)'s RFC at PowerShell/PowerShell-RFC#205

Regarding documentation: I've created MicrosoftDocs/PowerShell-Docs#4977, asking that guidance be added to about_Jobs, along with extending the topic to cover Start-ThreadJob and ForEach-Object -Parallel -AsJob too.

Paul Higinbotham (@PaulHigin)

Paul Higinbotham (@PaulHigin) Thanks for the info. So far I'm more than happy with thread jobs, it's trade-offs are quite reasonable. Question - is PSTaskJob an attempt to reduce thread overheads, or it's still something based on runspaces?

PSTaskJob is similar to ThreadJob except that it is specific to ForEach-Object -Parallel when run with the -AsJob switch. Unlike a ThreadJob, which encompasses only one running job at a time, PSTaskJob contains child jobs for all scriptblocks invoked by ForEach -Parallel (one script block is invoked for each piped input object). So if you run 1..10 | foreach -parallel { $_ }, there will be 10 child jobs within the PSTaskJob. Note that throttling still applies so only 5 child jobs will run at a time if the throttle limit is set to 5.

But basically it allows you to use PowerShell job cmdlets on a foreach -parallel running asynchronously as a job.

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

That would be so useful that I implemented a base class to allow for the implementation of async cmdlets, it is indeed possible to implement it.

Here is how you do it #7690 (comment)

I wish this issue is taken more seriously, it was not that hard to fix the problem myself in a couple of days.

Navigation Menu

Repository navigation

Feature Request: Async Cmdlets #10841

Summary of the new feature/enhancement

Example

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature Request: Async Cmdlets #10841

Description

Summary of the new feature/enhancement

Example

Activity

vexx32 commented on Oct 19, 2019

mikeTWC1984 commented on Oct 19, 2019

SteveL-MSFT commented on Oct 19, 2019

mklement0 commented on Oct 19, 2019

SteveL-MSFT commented on Oct 20, 2019

vexx32 commented on Oct 20, 2019

mklement0 commented on Oct 20, 2019

KirkMunro commented on Oct 20, 2019

mikeTWC1984 commented on Oct 21, 2019

PaulHigin commented on Oct 21, 2019

BrucePay commented on Oct 21, 2019

mklement0 commented on Oct 21, 2019

mklement0 commented on Oct 22, 2019

mikeTWC1984 commented on Oct 22, 2019

PaulHigin commented on Oct 22, 2019

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 23, 2023

Luiz-Monad commented on Dec 28, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions