Repository navigation
Simplify usage of Multipart Uploads with WebRequestPSCmdlet #2112
Description
Activity
This is horrendous in Powershell and hasn't moved in an age. I personally would love to see some movement here, as uploading large file is very, very difficult.
As some pointer around the topic see my comments here (albeit for PS 5.1) which like to some other sites, namely http://blog.majcica.com/2016/01/13/powershell-tips-and-tricks-multipartform-data-requests/
Mark Kraus (@markekraus) Could you please look this if you have free time?
Ilya (@iSazonov) sure. I will take a look at it. I have seen other requests in the past few years for multipart forms in general as several IoT APIs seem to only work with multipart forms.
Feel free to ping MSFT experts here if there is compatibility issues.
FWIW, we mainly Python, Django and Apache for our Web interfaace stuff. This is all RESTfull APIs. I am not developer, just attempting to interact with the API as we are seeing more sysadmins wanting to do this.
swinster if you could provide me an exact scenario that would be very useful. Right now, all I have to go off of is to build a out a test API that takes multipart forms and files and hope that fits most scenarios.
Mark Kraus (@markekraus) sure. I work for Pexip, and the entire platform a can be automated via API calls. This specific example is required with regard to uploading an upgrade file to the platform, which will be in excess of 1GB. The API documentation can be found here - https://docs.pexip.com/api_manage/api_configuration.htm?Highlight=upgrade
I can even post here the resultant function created to achieve this, although this is based on the information post by Mario Majčica in his blog above.
swinster awesome. If I need more I'll reach out.
swinster Can you share your function with me? i want to make sure I'm on the right path.
This presents some interesting implementation issues. To truly support Multipart form data, the Cmdlets would need to accept multiple files, a field name for each file, possibly a content type for each file, and the ability to accept a dictionary of field/value pairs at the same time. Additionally, It appears some APIs are particular about the boundary used, so an optional boundary would need to be supplied. Attempting to make that mesh with the current Cmdlets would be a significant rework almost to the point of justifying separate cmdlets for Multipart support.
The issue is in Multipart form data being extremely flexible and radically different from other requests. It would be easy to tack on support for a single file or to convert a
-Bodydictionary, but it would not be simple to mix a file with form data or to support multiple files. Fitting all the Multipart use-cases into the current cmdlets would touch a large portion of code to accommodate a valid but somewhat less common usage of the cmdlets.IMO, It would also lead to a somewhat confusing UX. If
-InFilewas used for Multipart files and only accepted multiple files on a Multipart request it would lead to issues with users supplying multiple paths on a standard request, getting errors, and wondering why it accepts an array of paths. Also, explaining that-Bodyand-InFileare mutually exclusive except when doing Multipart requests would be somewhat confusing confusing. If files were split into a separate and new parameter like-MultipartFile, that might cause confusion as to why-InFiledoesn't work on Multipart requests or why there needs to be a difference in the first place.The problem I see with simplifying this is that many of the requests I've read for sending Multipart form data through PowerShell have only their own use-case in mind. It might seem simple from that perspective, but when you look a dozen or so of these requests you begin to see they almost all are distinctly different needs falling under the broad category of Multipart form data. A Narrow implementation would leave many use-cases in their current state, a broad implementation would impact a lot of code but for feature addition that is less commonly used.
With that in mind, I think a fair compromise and better approach would be to accept a
-Bodyobject/collection that can be used by the Cmdlets to create the Multipart Form-Data Request. This could be simplified by exposing new cmdlet(s) and new classes to simplify generating that object/collection. Body could simply be adapted to acceptMultipartFormDataContentat least easing the burden of the user managing anHttpClient.I could start by adding support for
MultipartFormDataContentin-Body. From there it could be decided to add convenience cmdlets or possibly even wrapper classes to easy creation.James Truher (@JamesWTruher) Dan Travison (@dantraMSFT) Paul Higinbotham (@PaulHigin) Steve Lee (@SteveL-MSFT) What are your thoughts?
Mark Kraus (@markekraus) FWIW, here is the modified function from Mario's blog.
The main addition was the Basic Authentication headers which also required a Base64 encoded version of the authentication credentials. Initially, I decrypted the
PSCredentialpassword via a separate function, however this was changed to a single line. The Basic Authroziation header is required as although you can provide simple credentials to the theHTTPClient(and indeed theInvoke-WebRequest), the first request is sent without this header, which is then challenged, thus a second request is made. This means that then entire 1 GB file is uploaded twice - a huge waste of time and resources.As an aside, I have a similar issue (WRT the Basic Authorisation Header) when downloading a large file (3-4 GB), again again leads to a two request issue, and the download of the file is actioned twice), However the
Invoke-WebRequestis even worse in that case as the entire file is streamed into memory and cause the remote server to lock up when I tried to use this CmdLet. I eventually changed to use theWebClient(I couldn't get theHTTPClientto work with my lacking programming knowledge) , however,WebClientdoesn't implement atimeoutproperty so you have to build a class to inherit and extend the WebClient. All very, very complex and confusing, especially to someone like me with little programming expertise. As mentioned, this is an aside to this particular issue, but is similar in nature.function Invoke-MultipartFormDataUpload { [CmdletBinding()] PARAM ( [string][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$InFile, [string]$ContentType, [Uri][parameter(Mandatory = $true)][ValidateNotNullOrEmpty()]$Uri, [System.Management.Automation.PSCredential]$Credential ) BEGIN { if (-not (Test-Path $InFile)) { $errorMessage = ("File {0} missing or unable to read." -f $InFile) $exception = New-Object System.Exception $errorMessage $errorRecord = New-Object System.Management.Automation.ErrorRecord $exception, 'MultipartFormDataUpload', ([System.Management.Automation.ErrorCategory]::InvalidArgument), $InFile $PSCmdlet.ThrowTerminatingError($errorRecord) } if (-not $ContentType) { Add-Type -AssemblyName System.Web $mimeType = [System.Web.MimeMapping]::GetMimeMapping($InFile) if ($mimeType) { $ContentType = $mimeType } else { $ContentType = "application/octet-stream" } } } PROCESS { Add-Type -AssemblyName System.Net.Http $httpClientHandler = New-Object System.Net.Http.HttpClientHandler if ($Credential) { $networkCredential = New-Object System.Net.NetworkCredential @($Credential.UserName, $Credential.Password) $httpClientHandler.Credentials = $networkCredential $httpClientHandler.PreAuthenticate = $true $httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler #$password = Get-PlainText -SecureString $Credential.Password $Base64Auth = [System.Convert]::ToBase64String([System.Text.Encoding]::GetEncoding("iso-8859-1").GetBytes([String]::Format( "{0}:{1}", $Credential.UserName, $Credential.GetNetworkCredential().Password))) #$Base64Auth = [Convert]::ToBase64String([Text.Encoding]::GetEncoding("iso-8859-1").Getbytes("$($Credential.UserName):$($password)")) $httpClient.DefaultRequestHeaders.Add("Authorization", "Basic $Base64Auth") } else { $httpClient = New-Object System.Net.Http.Httpclient $httpClientHandler } $httpClient.Timeout = 18000000000 #$httpClient.DefaultRequestHeaders.Add("AUTHORIZATION", "Basic YTph") $packageFileStream = New-Object System.IO.FileStream @($InFile, [System.IO.FileMode]::Open) $contentDispositionHeaderValue = New-Object System.Net.Http.Headers.ContentDispositionHeaderValue "form-data" $contentDispositionHeaderValue.Name = "package" $contentDispositionHeaderValue.FileName = (Split-Path $InFile -leaf) $streamContent = New-Object System.Net.Http.StreamContent $packageFileStream $streamContent.Headers.ContentDisposition = $contentDispositionHeaderValue $streamContent.Headers.ContentType = New-Object System.Net.Http.Headers.MediaTypeHeaderValue $ContentType $content = New-Object System.Net.Http.MultipartFormDataContent $content.Add($streamContent) try { $response = $httpClient.PostAsync($Uri, $content).Result if (!$response.IsSuccessStatusCode) { $responseBody = $response.Content.ReadAsStringAsync().Result $errorMessage = "Status code {0}. Reason {1}. Server reported the following message: {2}." -f $response.StatusCode, $response.ReasonPhrase, $responseBody throw [System.Net.Http.HttpRequestException] $errorMessage } #return $response.Content.ReadAsStringAsync().Result return $response } catch [Exception] { $PSCmdlet.ThrowTerminatingError($_) return $response } finally { if($null -ne $httpClient) { $httpClient.Dispose() } if($null -ne $response) { $response.Dispose() } } } END { } }I have also hardcoded the
ContentDispositionHeaderValuevalue (package) which is all that was required for my needs.47 remaining items
if
[file] -eq [system.io.fileinfo], I think that will cause confusion, e.g.using namespace System.IO [File]::WriteAllLines($path, $line)
Clearly that's not meant to be
FileInfo, but if you don't see theusing, you might be confused if you get used to see[File]in other places.For me, the name isn't so important as just having easy accelerators for files and directories. I'm not a huge fan of
[system.io.fileinfo]not throwing on the file not existing. ideally this accelerator would be used like thisfunction Get-Widget { param([PSFile]$File) # Do something with $File } Get-Widget -File 'x:\no\lo\existo.nope' # parameter binding error for non existent file Get-Widget -File 'c:\some\real\file.txt' # works with existing file $form = @{ StringField = 'stringValue' FileField = [PSFile]'x:\no\lo\existo.nope' } # type conversion error on non existing file $form = @{ StringField = 'stringValue' FileField = [PSFile]'c:\some\real\file.txt' } # no error
also... I'd like to note that the accelerators should not be blocking to the multipart implementation. I can implement this now with out
[file]and add support for that later. The accelerators should have been tied to a separate issue. while having them would make the UX for the-Formfeature easier, for now the user can do$form = @{ FileField = Get-Item 'c:\some\real\file.txt' } iwr -Form $form -Method POST -Uri $uri
The accelerators should be a separate issue and PR. You can still use [System.IO.FileInfo] for now.
Just a quick question. with regard to this issue, has anything made it to PowerShell v6 as yet ?
Not sure if the underlying mechanism to retrieve a file will be altered, but it came to light the other day that the way I have the "working" at the moment in v5 is a little on the slow side. An HTTP download of a 4GB file from a browser from a server based on the same network takes approximately 30 seconds. In PowerShell v5, this takes around 1 hour !!!
swinster PowerShell 6.0.0 includes the ability to supply a
System.Net.Http.MultipartFormDataContentobject to the-Bodyparameter. You can read about this in detail here https://get-powershellblog.blogspot.com/2017/09/multipartform-data-support-for-invoke.html . The simplified-Formparameter will be added in 6.1.0 (assuming nothing comes up or blocks it). It is the first feature on my TODO list, but I have a few bugs and some code refactoring I would like to resolve first.Awesome work, thanks Mark Kraus (@markekraus) . Can't wait to be able to retrieve files with
-outfileCan't wait to be able to retrieve files with -outfile
swinster Can you elaborate on this? You can already download files and use the
-OutFileparameter to save them. This has been around for several versions at least.The blog post I linked has an earlier version of what was planned for the simplified multipart/form-data support. The planned implimentation has since been revised. You can see a "demo" of the currently planned simplified multipart/form-data support here https://get-powershellblog.blogspot.com/2017/12/powershell-core-web-cmdlets-in-depth_24.html#L21 and you can see the proposal here #2112 (comment)
Mark Kraus (@markekraus) apologies, the comment above was not in the correct place as it related to the underlying method of data retrieval from a simple GET method on the
Invoke-WebRequest, which was in the order of 200 time slower that a browser request, but actually nothing to do the MultiPart uploads. However, I have just run the same basic tests usingInvoke-WebRequestto GET a large file in v6 and v5 simultaneously and honestly there is a HUGE difference - whoever made these changes to the implementation here need some applause as well - even if this is the wrong place!An update for anyone following this issue. I am currently working on implementing the solution as approved by the PowerShell Committee with some minor adjustments. I hope to have a PR submitted in the coming days.
After reading RFC-7578 it became clear that a single field name can be supplied multiple times with different field values. The use case for this is an array of files (multiple file selection) or values for a single form field. To accommodate that, I'm adjusting the implementation slightly to treat collections as multiple field values for the same field name.
for example:
$Form = @{ Files = Get-ChildItem c:\temp -File Names = "Bill", "Sue", "Jane" Cars = [System.Collections.Generic.List[String]]::new([String[]]@("Smart", "Honda")) Description = "Multiple and single value support." Image = [System.IO.FileInfo]"c:\picture\me.png" } Invoke-WebRequest -Method POST -Uri $uri -Form $Form
In the approved implementation, this would result in a collection being converted to a single string. In the implementation I am working on,
Nameswould be supplied 3 times withBillas the first value,Sueas the second, andJaneas the third. It doesn't add much complexity and initial testing shows it works for endpoints that support it. It provides more flexibility at minimal cost. After checking, I personally have an internal use case for this in a form that requires multiple values for a single field.
Doing multipart requests with PowerShell is quite complicated and requires several extra steps for large file uploads. An easy example for a small file can be found on StackOverflow. An in depth discussion can be found in this blog post.
It would be a huge improvement if the WebRequestPSCmdlets (Invoke-RestMethod and Invoke-WebRequest) could be enhanced so that they support Multipart messages directly.
For an implementation I would expect the following parameters:
.NET Core seems to have support for MultipartContent which may simplify the implementation.