Repository navigation
Provide option in ForEach-Object -parallel to transfer current runspace state #12240
Description
Summary of the new feature/enhancement
Foreach-Object -parallel currently runs each loop iteration in a runspace initialized to its default state. But users might reasonably expect the runspace state to be the same as the runspace state running the foreach-object cmdlet.
This feature idea is to add a new -UseCurrentState (or something) switch that transfers the current runspace state to each parallel loop iteration runspace.
This would ensure that any modules imported or defined functions are available to each foreach -parallel script.
It is not clear how much of the current runspace state can or should be transferred to the loop runspaces, for example defined global variables are problematic since they are likely not thread safe.
Example:
New-PSDrive -Name ZZ -PSProvider FileSystem -Root c:\temp
1..1 | ForEach-Object -Parallel -UseCurrentState {
# Should have access to ZZ drive
dir ZZ:
}
Import-Module -Name c:\temp\Modules\MyModule.psd1
1..1 | ForEach-Object -Parallel -UseCurrentState {
# Should have access to MyModule functions
Get-MyInfo -Name Hello
}This needs to be opt in since it will be a performance hit. Some users will likely be unhappy in the performance degradation for simple scenarios.
Activity
PaulHigin commentedon Apr 1, 2020
This is a response to other issues created against foreach -parallel, where there is a reasonable expectation that the script block is run in the current state.
#12239
iSazonov commentedon Apr 2, 2020
Perhaps current state should be default and new switch "-UseDefaultRunspaceState".
somescout commentedon Apr 7, 2020
May I suggest that if a -Parallel switch is specified, it is expected that only one thing has changed - that For-EachObject will become parallel, and not that the code block will work differently.
PaulHigin commentedon Apr 7, 2020
somescout That would be ideal, but is not possible. There are too many issues. One is performance; replicating runspace state is time and resource intensive, and many times not even necessary. Another is multi-threaded issues. Much single thread code simply doesn't work in multi-threaded environments.
Like anything, there are trade offs to be made. I feel the best solution is to provide options along with guidance to help the user know and understand the optional benefits and trade offs.
somescout commentedon Apr 7, 2020
Premature optimization. If there will be significant performance issues, it can be solved by "-Isolated" switch. But if isolation is a default - it will cause too much unpredictable behavior in parallelized code (you add -Parallel switch and your code silently produce wrong result - worst possible behavior).
Then there should be separated parallelization commandlet, like Start-Job.
Is it really worth adding a another feature, the problems with which can be found only by trial and error? (People Don't Read Instructions)
PaulHigin commentedon Apr 14, 2020
Linking in related issue: #12313
PaulHigin commentedon Apr 20, 2020
Per, #12378, we should also consider allowing passing in script block variables if the client runspace state is replicated. The script block would be re-created/re-parsed within the new runspace so that it will have affinity to that runspace and run correctly. We still have potential threading issues of any shared variable referenced in the script block, but variables and multi-threading are potential problems anyway, which a user needs to be aware of.
iSazonov commentedon Apr 21, 2020
So we could have three options to cover all scenarios:
powercode commentedon May 12, 2020
There will also be issues with PowerShell classes defined in the current runspace. They will not exist in the new runspaces. Hard problem, for sure.
theJasonHelmick commentedon Jul 29, 2020
Ilya (@iSazonov) and somescout - I've been thinking about the default sunspace - should by default Foreach -parallel use current (dirty) runspace or, by default, use default (clean) runspace. It seems to me that as an admin, I would expect my current (dirty) runspace to be available when I use the -parallel switch. While this makes intuitive sense for an admin - It just works - I wonder if the performance impact (which may be incurred for no real reason) have such a negative impact that admins would no longer see a benefit to -parallel. With this in mind, I tend to agree with Paul Higinbotham (@PaulHigin) that the sunspace is clean. Thoughts?
PaulHigin commentedon Jul 29, 2020
I think I like
sunspacebetter thanrunspace:). But this is a great question. The current runspace running ForEach-Object -Parallel could have accumulated a lot of state that is not needed for running the parallel script blocks, and transferring that state to each parallel runspace could have a major performance impact. So my inclination is to make state transfer opt-in. I haven't yet thought about exactly what state should be transferred.iSazonov commentedon Jul 29, 2020
I think
opt-inin right direction. We could have options that transfer - from fastest clean runspace from a pool to slowest one with full features.As side notice, we have an issue for runspace performance creation without any progress but we could investigate this too.
3 remaining items
PaulHigin commentedon Jan 12, 2021
This work should include ensuring that experimental features are correctly enabled in the parallel running runspaces.
#14602
mcascone commentedon Apr 21, 2021
I vote for providing the option of runspace (sunspace) copying, perhaps not as default, so that we can use it, and take on the responsibility of dealing with the known gaps. Other issues aside, not being able to use the working runspace is worse than having to be aware of edge cases where it might be unsafe/unreliable. In most cases, especially when the user is aware of the caveats, the benefits would far outweigh the downsides. Thanks for reading.
PaulHigin commentedon Apr 26, 2021
Also consider extending PSScriptRoot into each foreach scriptblock (Issue #15302). Not sure yet if this makes sense for all cases.
PaulHigin commentedon Apr 28, 2021
Also consider built-in variables (Issue #14672).
whitTech commentedon Jan 29, 2022
As another admin, I also expect my runspace to be (very) dirty. And while I love performance just as much as the next admin (I mean it, please don't stop optimising for performance!), I've noticed parallelisation performance concerns around these parts tend to be significantly exaggerated (I've personally found it to be incredibly fast in basically all my admin-y scenarios, despite the abundance of caution stated that the man himself would have you believe. Paul Higinbotham (@PaulHigin), you don't give yourself enough credit for your amazing work!).
These days NVMe SSD's are ubiquitous and we have more cores/threads than we know what to do with. What kind of real-world performance hits are we actually talking about in each given scenario? The hits would have to be galactic in impact to have me switching away from parallelisation when I'm applying WSUS patches to computers across my domain that can take an hour to apply (and yes, I use parallelisation for very small things too, it's equally awesome).
With regard to the opt-in/opt-out debate, when I'm solving a problem, the priority order is as follows: Efficacy > Reliability > Performance, not the other way around. I want to build an effective solution as simply as possible, then I will mess about with performance and other aspects if necessary. Therefore, assuming performance isn't genuinely crippled, I do hope end-user simplicity prevails and we opt for an opt-out approach, if possible.
PaulHigin commentedon Feb 4, 2022
Thanks for you input and kind words. Unfortunately, this task has been deprioritized and I have been busy working on other things. But I still hope to get back to this in the Spring.
whitTech commentedon Feb 5, 2022
No worries at all.
The issue forced me to properly develop my modules and ensure they're properly distributed throughout my environments. More work, but I'm glad I have finally gotten around to it.
Thanks again, Paul Higinbotham (@PaulHigin). You and the other Powershell developers deserve much more than a few kind words for the effort you contribute to this wonderful tool.
Luiz-Monad commentedon Oct 19, 2023
What I would find most useful when using something like
$using:scriptBlock, was if parallel-for-each would copy the scriptblock as a pure function in a new closure to the new state.That way using parallel would be more useful in a lot of scenarios when you just want to reuse functions, like saving them to a script file and then importing the script file to the new runspace, but automatically.
The modules those functions use would have to be loaded, but I don't see that as a big of a deal from a point of view of performance.
The serialization/deserialization system ought to be able to handle that.
Its not the entire Runspace state, you keep starting them clean, its only a helper "sugar" step.
Gives us:
But something like this works:
{ function rel ($base, $path) { resolve-path $path -Relative -RelativeBasePath $base } } | out-file "script.ps1" $cwd = $pwd.ProviderPath ls ./ -rec | select -exp fullname | foreach -parallel { Import-Module "$using:cwd/script.ps1" "vendor/$(rel ./lib $_)" } }It would be useful even if it was capped only to pure functions that don't use the global state. I think this is the most useful scenario I would actually need to use
foreach -paralleland have to bring$using:scriptblock.If I left some functions in my scope, but they're pure functions that only use things from loaded modules, and now I'm using the shell and want to plop a
-parallelin a foreach command, I expect basically$using:function:funfunction's AST to be copied over to the new Scope. I don't need any state at all, just copy the AST of the function, and load the modules it uses automatically for me.That's would be very useful. Powershell is a shell, that's from a shell point of view of using the interactive shell. if I or anyone ever need something more complex, then we would just create modules.
noamper commentedon Apr 17, 2024
Commenting to keep this open. I would enthusiastically opt-in to a dirty runspace.