Repository navigation
webcmdlets should support resuming downloads #5964
Description
Activity
On remote endpoints where resume is not supported, do we want to over write the OutFile from the beginning? or do we want to skip the response stream forward and append? We might not get network IO gains, but we may save on Storage IO. It's not a trivial implementation, but could be done.
Mark Kraus (@markekraus) I think we can start with the 80% case (which I assume is in this day and age resume is supported) and based on demand consider fallback later based on feedback
Steve Lee (@SteveL-MSFT) What should we do in the following scenario:
User has supplied
iwr -uri http://test.io/file -OutFile c:\temp\out.txt -Resume.c:\temp\out.txtis 100 bytes buthttp://test.io/fileis 50 bytes. Server responds with416 Range Not Satisfiable.Should we error as usual? or should we "special case" this and provide the user something more palatable like "The file supplied to -OutFile was unable to resume because the remote server was unable to comply. This can happen when the local file is not the same as the remote file."? The special case will add some complexity, but
416 Range Not Satisfiableis a really unhelpful message.Either of the error scenarios would be in some conflict with how
-OutFilecurrently works. Currently it overwrites the supplied file without complaint. So another option would be to have the cmdlets clear the range header, resubmit the request, download the whole file, and overwrite-OutFilewithout error, This would add complexity. Also, this action may not be what the user is expecting. It could warn on this action.Another option is to have 2 params
-Resumeand-ForceResume(I hesitate to add just-Forcefor this feature as it has implications for other things that would not be supported by-Force).-ForceReumsewould overwrite the supplied-OutFileon416and-Resumewould error.Mark Kraus (@markekraus) I think having a descriptive error would be good. I think
-ForceResumeis a good switch as I'm not a fan of the generic-Forcewhich doesn't provide context on what it actually does so, in my opinion, this is setting a good pattern we should adopt in the future.Mark Kraus (@markekraus) upon further consideration, I think
-AllowClobberis better than-ForceResumewith the expectation that what the user wants is: download this file, resume if possible, but restart download from beginning if necessary.However, if most usage is
-Resume -AllowClobberthen perhaps it should be-NoClobber.Having
-AllowClobberwill be confusing. For example.. someone will runiwr -OutFile c:\ExstingFile.txt -AllowClobber $uri(without-Resume) and expect it to overwrite (which is the current behavior) but theniwr -OutFile c:\ExstingFile.txt $uri(without-Resume) to not overwrite, which is not the case.. the file would be overwritten.We have the the opposite problem
-NoClobber,iwr -OutFile c:\ExstingFile.txt -NoClobber $uri(without-Resume) one would expect it to not overwrite the existing file but it would. Also,iwr -OutFile c:\ExstingFile.txt -Resume $uriwould now be, by default, overwriting the file which would probably unexpected (the original issue which prompted my asking).We could always validate that
-NoClobberor-AllowClobberare supplied only when-Resumeis supplied... but we have to rely on run time checks so this isn't apparent in the syntax.I think both
-NoClobberand-AllowClobberare undesirable unless we break how-OutFileworks now (without-Resume) which we obviously can't do such a major breaking change until the next major version or until the Web Cmdlets are a separate project.-NoClobberand-AllowClobberare just generic and have implication for features outside of-Resumethat they would not actual have any affect on.Currently, we don't have
-Resumeso if we want to have it act like-OutFiledoes now where it overwrites all the time without warning we can. This will only happen a) the remote server does not support Resume, b) the local file is larger than the remote file. So long as we document the behavior, it might be fine.I should note we have no way to protect against a scenario where a user supplies an
-OutFilethat is not the same as the remote file but is smaller. For examplec:\ExstingFile.txtis 50 byteshttp://test.io/fileis 100 bytes. user doesiwr -outfile c:\ExstingFile.txt -Resume -uri http://test.io/file. In that scenario, since we hav no way of knowing whether remote and local are the same file, we would end up appending the remote file to the unrelated local file. Only way around this is to do what browsers do and track the downloaded files and only allow resume on file it knows has downloaded. or download random ranges from the remote file and check them with the local file to see if they match... all of which is way too complex to implement for what it's worth.I swear I'm not trying to make this complicated.... but this is a feature that once we put it in will be really hard to change in the future.
Mark Kraus (@markekraus) you bring up some good points. Let's keep this simple for now and only have
-resumeand document the behavior that:- it doesn't check if the existing file is actual a partial download of the target, it just resumes after the last byte
- it overwrites if there is inconsistency or if the server doesn't support resume
PR #6447
- added a commit that references this issue
on Mar 26, 2018 - added a commit that references this issue
on Apr 24, 2018 - added a commit that references this issue
on Aug 6, 2026
wget -cequivalent. If told toresumewe should be able to read the length of the current file and use the HTTP Range header to resume the download. In the case where Range is not supported by the server, we would revert to restarting the download and output a warning.Steps to reproduce
Expected behavior
Actual behavior