Repository navigation
webcmdlets should support resuming downloads #5964
Description
wget -c equivalent. If told to resume we should be able to read the length of the current file and use the HTTP Range header to resume the download. In the case where Range is not supported by the server, we would revert to restarting the download and output a warning.
Steps to reproduce
iwr http://someurl -outfile bigfile.zip
// assume download gets stopped prematurelyExpected behavior
iwr http://someurl -outfile bigfile.zip -resume
Actual behavior
Need to restart the download from the beginning
Activity
markekraus commentedon Jan 20, 2018
On remote endpoints where resume is not supported, do we want to over write the OutFile from the beginning? or do we want to skip the response stream forward and append? We might not get network IO gains, but we may save on Storage IO. It's not a trivial implementation, but could be done.
SteveL-MSFT commentedon Jan 22, 2018
Mark Kraus (@markekraus) I think we can start with the 80% case (which I assume is in this day and age resume is supported) and based on demand consider fallback later based on feedback
markekraus commentedon Jan 29, 2018
Steve Lee (@SteveL-MSFT) What should we do in the following scenario:
User has supplied
iwr -uri http://test.io/file -OutFile c:\temp\out.txt -Resume.c:\temp\out.txtis 100 bytes buthttp://test.io/fileis 50 bytes. Server responds with416 Range Not Satisfiable.Should we error as usual? or should we "special case" this and provide the user something more palatable like "The file supplied to -OutFile was unable to resume because the remote server was unable to comply. This can happen when the local file is not the same as the remote file."? The special case will add some complexity, but
416 Range Not Satisfiableis a really unhelpful message.Either of the error scenarios would be in some conflict with how
-OutFilecurrently works. Currently it overwrites the supplied file without complaint. So another option would be to have the cmdlets clear the range header, resubmit the request, download the whole file, and overwrite-OutFilewithout error, This would add complexity. Also, this action may not be what the user is expecting. It could warn on this action.Another option is to have 2 params
-Resumeand-ForceResume(I hesitate to add just-Forcefor this feature as it has implications for other things that would not be supported by-Force).-ForceReumsewould overwrite the supplied-OutFileon416and-Resumewould error.SteveL-MSFT commentedon Jan 30, 2018
Mark Kraus (@markekraus) I think having a descriptive error would be good. I think
-ForceResumeis a good switch as I'm not a fan of the generic-Forcewhich doesn't provide context on what it actually does so, in my opinion, this is setting a good pattern we should adopt in the future.SteveL-MSFT commentedon Jan 31, 2018
Mark Kraus (@markekraus) upon further consideration, I think
-AllowClobberis better than-ForceResumewith the expectation that what the user wants is: download this file, resume if possible, but restart download from beginning if necessary.However, if most usage is
-Resume -AllowClobberthen perhaps it should be-NoClobber.markekraus commentedon Jan 31, 2018
Having
-AllowClobberwill be confusing. For example.. someone will runiwr -OutFile c:\ExstingFile.txt -AllowClobber $uri(without-Resume) and expect it to overwrite (which is the current behavior) but theniwr -OutFile c:\ExstingFile.txt $uri(without-Resume) to not overwrite, which is not the case.. the file would be overwritten.We have the the opposite problem
-NoClobber,iwr -OutFile c:\ExstingFile.txt -NoClobber $uri(without-Resume) one would expect it to not overwrite the existing file but it would. Also,iwr -OutFile c:\ExstingFile.txt -Resume $uriwould now be, by default, overwriting the file which would probably unexpected (the original issue which prompted my asking).We could always validate that
-NoClobberor-AllowClobberare supplied only when-Resumeis supplied... but we have to rely on run time checks so this isn't apparent in the syntax.I think both
-NoClobberand-AllowClobberare undesirable unless we break how-OutFileworks now (without-Resume) which we obviously can't do such a major breaking change until the next major version or until the Web Cmdlets are a separate project.-NoClobberand-AllowClobberare just generic and have implication for features outside of-Resumethat they would not actual have any affect on.Currently, we don't have
-Resumeso if we want to have it act like-OutFiledoes now where it overwrites all the time without warning we can. This will only happen a) the remote server does not support Resume, b) the local file is larger than the remote file. So long as we document the behavior, it might be fine.I should note we have no way to protect against a scenario where a user supplies an
-OutFilethat is not the same as the remote file but is smaller. For examplec:\ExstingFile.txtis 50 byteshttp://test.io/fileis 100 bytes. user doesiwr -outfile c:\ExstingFile.txt -Resume -uri http://test.io/file. In that scenario, since we hav no way of knowing whether remote and local are the same file, we would end up appending the remote file to the unrelated local file. Only way around this is to do what browsers do and track the downloaded files and only allow resume on file it knows has downloaded. or download random ranges from the remote file and check them with the local file to see if they match... all of which is way too complex to implement for what it's worth.I swear I'm not trying to make this complicated.... but this is a feature that once we put it in will be really hard to change in the future.
SteveL-MSFT commentedon Jan 31, 2018
Mark Kraus (@markekraus) you bring up some good points. Let's keep this simple for now and only have
-resumeand document the behavior that:markekraus commentedon Mar 21, 2018
PR #6447
Add -Resume Feature to Web Cmdlets (#6447)
Keep uptodate (#1)
Add -Resume Feature to Web Cmdlets (PowerShell#6447)