Repository navigation
Progress bar can significantly impact cmdlet performance #2138
Description
Repro Steps :
Compare perf of WebClient, BITS and Invoke-WebRequest:
Measure-Command { $wc = New-Object System.Net.WebClient; $wc.DownloadFile("https://github.com/dotnet/corefx/archive/master.zip", "C:\Users\crispop\Desktop\master.zip") }
Days : 0
Hours : 0
Minutes : 0
Seconds : 3
Milliseconds : 992
Ticks : 39928426
TotalDays : 4.62134560185185E-05
TotalHours : 0.00110912294444444
TotalMinutes : 0.0665473766666667
TotalSeconds : 3.9928426
TotalMilliseconds : 3992.8426
Vs
Measure-Command { Invoke-WebRequest "https://github.com/dotnet/corefx/archive/master.zip" -OutFile "C:\Users\crispop\Desktop\master.zip" }
Days : 0
Hours : 0
Minutes : 0
Seconds : 50
Milliseconds : 821
Ticks : 508212825
TotalDays : 0.000588209288194444
TotalHours : 0.0141170229166667
TotalMinutes : 0.847021375
TotalSeconds : 50.8212825
TotalMilliseconds : 50821.2825
Vs
Measure-Command { Start-BitsTransfer -Source "https://github.com/dotnet/corefx/archive/master.zip" -Destination "C:\Users\crispop\Desktop\masterBits2.zip" }
Days : 0
Hours : 0
Minutes : 0
Seconds : 4
Milliseconds : 506
Ticks : 45062594
TotalDays : 5.21557800925926E-05
TotalHours : 0.00125173872222222
TotalMinutes : 0.0751043233333333
TotalSeconds : 4.5062594
TotalMilliseconds : 4506.2594Expected: Transfer speeds don't differ too much (maybe Invoke-WebRequest should be a bit faster than BITS which performs the transfer in background.)
Actual: 10x slowdown for Invoke-WebRequest
Activity
GeeLaw commentedon Aug 31, 2016
How can you be sure that it's the progress bar? With
$ProgressPreference = 'SilentlyContinue', you can suppress the progress bar UI, which will surely boostInvoke-WebRequest. Also, use-UseBasicParsingwhen downloading files withiwr(which you shouldn't do since that's not whatiwris for...). You cannot suppress parsing, but you can suppress DOM parsing.Moreover, I did your tests and found that the standard deviation is rather great (poor network here in China).
Please do observe Task Manager for the network speed and also do more statistics. If the network is perfect (I hosted a file locally and tried to download via
localhost, all three were superfast even the file is 80MB+, with progress bar suppressed.)And, could you include the
$PSVersionTable?SteveL-MSFT commentedon Aug 31, 2016
PSEdition Core
CLRVersion
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
BuildVersion 3.0.0.0
GitCommitId v6.0.0-alpha.9
PSRemotingProtocolVersion 2.3
PSVersion 6.0.0-alpha
WSManStackVersion 3.0
SerializationVersion 1.1.0.1
jasongin commentedon Oct 3, 2016
I also see the progress bar slowing down Invoke-WebRequest by >50X on a fast internet connection. Every time I write a script that uses Invoke-WebRequest I need to include code to temporarily change
$ProgressPreference, then restore it later (because I do want progress for other things).Here's my output from using Invoke-WebRequest to download the same file as Steve Lee (@SteveL-MSFT), without and then with progress:
It would be nice to be able to have reasonable progress when downloading a file. Perhaps if there were a way to adjust the progress reporting interval to several KB instead of every byte (!!) then the overhead wouldn't be so bad.
jasongin commentedon Oct 4, 2016
This issue also affects Expand-Archive. For a large archive (10 MB with 3495 files), expanding it takes over 4X long with progress compared to without.
[-]Invoke-webrequest slow at downloading files due to progress bar perf[/-][+]Progress bar can significantly impact cmdlet performance[/+]SteveL-MSFT commentedon Oct 4, 2016
Changed title to be a generic issue with the progress bar.
Jaykul commentedon Oct 7, 2016
Yeah, this is pretty bad ...
oising commentedon Oct 18, 2016
I remember noticing how much of a blocker progress writing was when I wrote the PSCX GAC provider back in 2007:
https://github.com/Pscx/Pscx/blob/master/Src/Pscx/Providers/AssemblyCache/AssemblyNameCache.cs#L79
We decided to only write a record on every fifth iteration.
iSazonov commentedon Nov 2, 2016
It's very simple (write a record on every fifth iteration). 👍 Just seems the compromise is not always the optimal.
May be possible to write asynchronously every 3-5 seconds?
oising commentedon Nov 2, 2016
Ilya (@iSazonov) Yeah. I'm just saying that it's a problem that's been there since the beginning. Skipping records doesn't solve all issues.
iSazonov commentedon Nov 3, 2016
Oisin Grehan (@oising) This is not a criticism from me. I just asked your opinion on "async" since you have experience with the problem.
22 remaining items
iSazonov commentedon Jul 12, 2017
Niclas Lindgren (@NiclasLindgren) Thanks for your comment!
ISE is not in the PowerShell Core repo, it is not ported and no plans is.
NiclasLindgren commentedon Jul 12, 2017
Yes, my point was, that "powershell for windows" (v4.0) has the same problem, When run under ISE the "console" does not slow down the performance, I thought it might be interesting for those with access to ISE to check what they do. Either way, the solution discussed here helped and I am happy =)
iSazonov commentedon Jul 13, 2017
Niclas Lindgren (@NiclasLindgren) Welcome to test PowerShell Core Beta.4! We have some console fixes in last time from Jason Shirk (@lzybkr) so you may be surprised how much the console is faster. Also we wait a feedback for loading and using Windows PowerShell modules in PowerShell Core Beta.4.
oising commentedon Jul 13, 2017
Niclas Lindgren (@NiclasLindgren) the long and short of it is that rendering progress bars in WPF is orders of magnitude fast than drawing progress bars in the windows console. It's not about the code path per-se.
SteveL-MSFT commentedon Jul 13, 2017
Niclas Lindgren (@NiclasLindgren) haven't peeked at the ISE code, but I suspect the progress bar is being rendered on a separate thread while the console uses the same thread
Suppress progress bars whenever Invoke-WebRequest is used to workaround
Rollup merge of rust-lang#61304 - lzybkr:iwr_progress, r=alexcrichton
Rollup merge of rust-lang#61304 - lzybkr:iwr_progress, r=alexcrichton
Disable the progress bar while downloading in the Windows wrapper scr…
PowerShellForGitHub: Disable Progress Bar for Invoke-WebRequest (#229)
max06 commentedon Sep 30, 2022
Hello folks!
I'm very sorry for digging up this old issue again, but I had to notice it still seem to be unresolved 6 years later.
I ran my test using
Invoke-WebRequestto download a bigger file. With no additional flags or settings, the download speed gets limited to 2-3 MBit/s on my machine - compared to 100 MBit/s+ without the progress bar. This was tested in the stock powershell console delivered with the latest Win11 22H2 update (PSVersion: 5.1.22621.169).As I read above, there is, or used to be a fix. Is it still there?
ThomasNieto commentedon Sep 30, 2022
Flo (@max06) this has already been fixed in PowerShell. Windows PowerShell most likely will never get that fix.
You can get around it on Windows PowerShell by using setting https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_preference_variables?view=powershell-7.2#progresspreference to SilentlyContinue.
max06 commentedon Oct 1, 2022
Thomas Nieto (@ThomasNieto) Thank you for your quick response!
So, Windows Powershell is different from this one? I never knew that 😲
I'm also not using PS very often, a colleague reported this issue while setting up a new server - I'll forward your hint to him.
Progress bar can significantly impact cmdlet performance
Stop paying the PowerShell 5.x progress-bar tax on large downloads