Repository navigation
[Feature Request] Parallel Dependency installations #69
Description
Activity
It makes sense to install in parallel and resolve dependencies so they aren't installed multiple times (like in the case of Az modules all depending on Az.Accounts).
My recommendation would be to basically just execute a NuGet restore, it already has all the logic in there to do this in parallel and handle all the dependencies. I'll reference my Modulefast POC again https://gist.github.com/JustinGrote/ecdf96b4179da43fb017dccbd1cc56f6 which installs all the Az modules (which are a really good test case due to their metapackages and dependencies) in less than 5 seconds
I did some experimentation with runspace pools yesterday. Pseudocode with the Az module:
- Find dependencies first:
$Parent = Find-PSResource -Name 'Az' - List of modules to install =
$Parent.Name+$Parent.Dependencies - Parallelize
foreachwith runspace pools, useInstall-PSResourcewith-SkipDependencyChecksince we already have all dependencies.
This drastically sped up installation of a module with many dependecies, and should be backwards compatible to Windows PowerShell >= 3.
This is a low hanging fruit for great speed improvements IMO.
Edit: I created a function that uses runspace pools to install modules in parallel, and tested it on
Az,Microsoft.Graph,Microsoft.Graph.Betaand all their dependencies, 166 unique modules in total (as of writing).- Original
Save-PSResource -Repository 'PSGallery' -TrustRepository -IncludeXml -SkipDependencyCheck -Path $SavePath -Name $ListOfModulestook 110 seconds. Save-PSResourceInParallel -Repository 'PSGallery' -Path $SavePath -Name $ListOfModules -ThrottleLimit 16using runpsace factory took around 20 seconds.- Justin Grote ModuleFast
Install-ModuleFast -ModulesToInstall 'Az','Microsoft.Graph','Microsoft.Graph.Beta' -Destination $SavePath -Credential ([PSCredential]::Empty) -NoPSModulePathUpdate -NoProfileUpdate -Updatetook about 35 seconds.
To my surprise, by parallalizing with runspace factory one can go significantly faster than even ModuleFast. And it's backwards compatible with Windows PowerShell.
Here is the code if anyone want to experiment further
# Function function Save-PSResourceInParallel { <# .SYNOPSIS Speed up PSResourceGet\Save-PSResource by parallizing using PowerShell native runspace factory. .NOTES Author: Olav Rønnestad Birkeland | github.com/o-l-a-v Created: 231116 Modified: 231116 .EXAMPLE Save-PSResourceInParallel -Type 'Module' -Name (Find-PSResource -Repository 'PSGallery' -Type 'Module' -Name 'Az').'Dependencies'.'Name' #> [CmdletBinding()] [OutputType([Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]])] Param( [Parameter(Mandatory)] [string[]] $Name, [Parameter()] [bool] $IncludeXml = [bool] $true, [Parameter()] [ValidateNotNullOrEmpty()] [string] $Path, [Parameter()] [ValidateNotNullOrEmpty()] [string] $PSResourceGetPath = (Get-Module -Name 'Microsoft.PowerShell.PSResourceGet').'Path', [Parameter()] [ValidateNotNullOrEmpty()] [string] $Repository = 'PSGallery', [Parameter()] [bool] $SkipDependencyCheck = [bool] $true, [Parameter()] [byte] $ThrottleLimit = 10, [Parameter()] [bool] $TrustRepository = [bool] $true ) # Begin Begin { # Assets $ScriptBlock = [scriptblock]{ [OutputType([System.Void])] Param( [Parameter()] [bool] $IncludeXml = $true, [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string] $Name, [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string] $Path, [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string] $PSResourceGetPath, [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string] $Repository, [Parameter()] [bool] $SkipDependencyCheck = $true, [Parameter()] [bool] $TrustRepository = $true ) $ErrorActionPreference = 'Stop' $null = Import-Module -Name $PSResourceGetPath Microsoft.PowerShell.PSResourceGet\Save-PSResource -Repository $Repository -TrustRepository:$TrustRepository ` -IncludeXml:$IncludeXml -Path $Path -SkipDependencyCheck:$SkipDependencyCheck -Name $Name } # Initilize runspace pool $RunspacePool = [runspacefactory]::CreateRunspacePool(1,$ThrottleLimit) $RunspacePool.Open() } # Process Process { # Start jobs in the runspace pool $RunspacePoolJobs = [PSCustomObject[]]( $( foreach ($ModuleName in $Name) { $PowerShellObject = [powershell]::Create().AddScript($ScriptBlock).AddParameters( @{ 'IncludeXml' = [bool] $IncludeXml 'Name' = [string] $ModuleName 'Path' = [string] $Path 'PSResourceGetPath' = [string] $PSResourceGetPath 'Repository' = [string] $Repository 'SkipDependencyCheck' = [bool] $SkipDependencyCheck 'TrustRepository' = [bool] $TrustRepository } ) $PowerShellObject.'RunspacePool' = $RunspacePool [PSCustomObject]@{ 'ModuleName' = $ModuleName 'Instance' = $PowerShellObject 'Result' = $PowerShellObject.BeginInvoke() } } ) ) # Wait for jobs to finish $PrettyPrint = [string]('0'*$RunspacePoolJobs.'Count'.ToString().'Length') while ($RunspacePoolJobs.Where{-not $_.'Result'.'IsCompleted'}.'Count' -gt 0) { Write-Verbose -Message ( '{0} / {1} jobs finished, {2} / {0} was successfull.' -f ( $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted'}.'Count'.ToString($PrettyPrint), $RunspacePoolJobs.'Count'.ToString(), $RunspacePoolJobs.Where{$_.'Result'.'IsCompleted' -and -not $_.'Instance'.'HadErrors'}.'Count'.ToString($PrettyPrint) ) ) Start-Sleep -Milliseconds 250 } # Get success state of jobs Write-Verbose -Message ( $RunspacePoolJobs.ForEach{ [PSCustomObject]@{ 'Name' = [string] $_.'ModuleName' 'IsCompleted' = [bool] $_.'Result'.'IsCompleted' 'HadErrors' = [bool] $_.'Instance'.'HadErrors' } } | Sort-Object -Property 'ModuleName' | Format-Table | Out-String ) # Collect results $Results = [Microsoft.PowerShell.PSResourceGet.UtilClasses.PSResourceInfo[]]( $RunspacePoolJobs.ForEach{ $_.'Instance'.EndInvoke($_.'Result') } ) } # End End { # Terminate runspace pool $RunspacePool.Close() $RunspacePool.Dispose() # Output results $Results } } # Testing if ($false) { # Import module Import-Module -Name 'Microsoft.PowerShell.PSResourceGet' -RequiredVersion '1.0.1' # Assets ## List of modules $ListOfModules = [string[]]('Az','Microsoft.Graph','Microsoft.Graph.Beta') $ListOfModules += [string[]]((Find-PSResource -Name $ListOfModules).'Dependencies'.'Name') $ListOfModules = [string[]]($ListOfModules | Sort-Object -Unique) ## Temp save path $SavePath = [string][System.IO.Path]::Combine([System.Environment]::GetFolderPath('Desktop'),'Modules',[datetime]::Now.ToString('yyyyMMddHHmmss')) # Prepare temp save path if (-not [System.IO.Directory]::Exists($SavePath)) { $null = [System.IO.Directory]::CreateDirectory($SavePath) } # Install modules ## Original Measure-Command -Expression { Save-PSResource -Repository 'PSGallery' -TrustRepository -IncludeXml -SkipDependencyCheck -Path $SavePath -Name $ListOfModules } ## Parallel with Runspace Factory Measure-Command -Expression { Save-PSResourceInParallel -Repository 'PSGallery' -Path $SavePath -Name $ListOfModules -ThrottleLimit 16 } ## ModuleFast ### Load if not already loaded & ( [scriptblock]::Create( ( Invoke-RestMethod -Method 'Get' -Uri 'https://raw.githubusercontent.com/JustinGrote/ModuleFast/main/ModuleFast.ps1' ) ) ) ### Run Measure-Command -Expression { Install-ModuleFast -ModulesToInstall 'Az','Microsoft.Graph','Microsoft.Graph.Beta' -Destination $SavePath -Credential ([PSCredential]::Empty) -NoPSModulePathUpdate -NoProfileUpdate -Update } # Clean up ## For testing again [System.IO.Directory]::Delete($SavePath,$true);$null=[System.IO.Directory]::CreateDirectory($SavePath) ## For good [System.IO.Directory]::Delete($SavePath,$true) }
- Find dependencies first:
One of the big reasons why this was requested was that the dependency tree of the AzureRM & Az Modules built a list of cross dependencies between each sub module that was incredibly painful when not working with older & slower internet connections and there was no
Ooh we should only download this particular module version oncecheck built into PowerShellGet, oh and some Az Modules had depenencies on different versions of one or two of thier core modules. I recall profile/account being the ones that caused the most pain.I previously tested this all using PoshRSJob for parallelising which essentially is a wrapper around Runspaces, and has been superseeded with the Microsoft.PowerShell.ThreadJobs module. Taking that as a dependency in PSResourceGet is for me a sensible solution as we won't just be supporting the use of nuget backed repositories so we can't just call out to nuget for this. I wish I had written up the testing that I had done at the time but this was done multiple times between 2016-2020 & I'm still using the device I was running some of those tests on as much as I'd like to retire it.
The logic to build the full dependency tree must also be contained in PSResourceGet so that
Install-PSResource Azjust works in the most optimal way possible for your current device and current set of Registered package registries and the priority ordering of those registries.It was one reason why I repeatedly asked the Azure PowerShell team to release a condensed package, which annoyingly they repeatedly refused to do, as part of each release that instead of requiring you to download each dependency seperately when they could easily bundle them as part of the package so that you could download only that one package and for it to also contain a simple post-install extraction script to place all these modules and versions into one of the two main correct PSModulePath locations user or system and not like some MSI installs do and add a new path to PSModulePath. This still makes sense for many orgs especially if they are on bad connections. It also makes sense for many orgs that are conscious & hyper aware of bad package releases and will disallow installation from a non-internallised package repository where each version has been scrutinised prior to being made available to their users as allows them to test and then repackage to thier users as they require. But this negates the need for Parallel Dependency Installs as part of PSResourceGet.
I also suggested for PowerShell to possibly also release a torrent file or magnet links for releases as this gives another mechanism for obtaining releases. I can understand why many would not see this as suitable for many orgs but this reduces load on places like GitHub by sharing the packages across multiple nodes.
Same with providing something like a blob in Azure Storage like Steve mentioned or now we are getting Microsoft Artifact Registry. I still think using a torrent file/magnet link could have benefit in future especially as you can point those downloads to download straight to the particular PSModulePath entry that you desire. The added benefit from seeding those downloads is that those closest to you also get the downloads quicker, bypassing service issues or network conjestion in internet providers that don't have enough bandwidth and regularly have outages.Parallel Dependency Installations is something that I feel needs to be in the next release of PSResourceGet. We can't really start 2026 without something as useful as this being part of the next release. Like we really really can't as it's silly that this still isn't a part of the PSResourceGet featureset and have to have external workarounds like modulefast or countless individual wrappers for something so important.
Also Justin Grote (@JustinGrote) & Olav Rønnestad Birkeland (@o-l-a-v) you don't mention at all what specs the devices are running, what load they where under or what bandwidth the connection you are running on has which all drastically changes the time to complete this task so these results are good to see but don't provide enough context to be truly valuable overall as not everyone can test on devices like an M128 Azure Virtual Machine that isn't under any load ( I use that as a silly example) or in network conditions where there aren't massive bandwidth constriants or where there are volitile connections with lots of network drops (like on trains/planes/mobile networks/ships/submarines etc) and these senarios need equating for too.
But yeah I hope we see this in the next release of PSResourceGet so that we can get it in PowerShell 7.6.0 GA too.
When installing meta modules like Az / AzureRM each of the required modules gets installed sequentially which is painfully slow and could be greatly improved.
Therefore I would ask that as a feature request for PSGet v3 that dependencies could be installed via some form of parallelisation as to reduce overall installation time