Repository navigation
serve sftp, serve webdav: fix stale hashes reported while a file is being uploaded - fixes #4956 - #9946
Open
hsdfat wants to merge 3 commits into
Open
serve sftp, serve webdav: fix stale hashes reported while a file is being uploaded - fixes #4956#9946hsdfat wants to merge 3 commits into
hsdfat wants to merge 3 commits into
Conversation
File.Hash returns the hash of the cached copy while the file is modified in the VFS cache and waiting to be written back, and the hash of the remote object otherwise, in the same way that File.Size already reports the size of a dirty cache item. For rclone#4956
…fixes rclone#4956 md5sum and friends read the hash from the remote object, which still describes the old contents for the whole time the new contents are sitting in the VFS cache waiting to be written back. Use File.Hash instead so the hash describes the data which will be uploaded. This also replaces the special case for files which have no remote object yet, as File.Hash now covers it.
The ETag and the owncloud checksum were read from the remote object, which still describes the old contents while the new contents are in the VFS cache waiting to be written back, so a PROPFIND or HEAD could return the size of the new file next to the ETag of the old one. For rclone#4956
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change do?
With
--vfs-cache-mode writesorfull, a file uploaded throughrclone serve sftporrclone serve webdavsits inthe VFS cache until writeback uploads it. Until then the sftp
md5sum/sha1sumcommands and the webdavETagandchecksumsproperty report the hash of the old object still on the remote. A client which verifies after upload -rclone syncto an sftp remote does - sees a mismatch and uploads the file again, as diagnosed in the issue.Following the approach suggested in the issue, this adds a
Hashmethod tovfs.File. It returns the remoteobject's hash unless the file is dirty in the cache, in which case it hashes the cached data read back through the
VFS.
serve sftpandserve webdavcall it instead of asking the remote object directly.Files which are not dirty in the cache behave exactly as before: the hash still comes from the remote object, and
a clean copy left in the cache by a read is not used in its place. For dirty files the hash is computed from the
cached copy, so a webdav
PROPFINDwith--etag-hashreads any files in that directory which are waiting to beuploaded. A hash type the remote doesn't support is refused for dirty files just as the remote refuses it for clean
ones.
A file with neither a remote object nor a dirty cache item - one still being streamed up with
--vfs-cache-mode off- returnsENOENTstraight away, so webdav leaves out its ETag and checksums as it did before, without logging anerror for it. The sftp
handler used to open such a file to hash it, which waits up to 5 seconds for the upload and then usually fails
with the same "file does not exist";
File.Hashreplaces that fallback, somd5sumnow fails without the wait.Testing
vfs/file_test.goTestFileHash, for cache modes off, minimal, writes and full: an unmodified file returns theremote hash, including after a read has left a clean copy in the cache; while a writer has new data in the cache
(checked to be dirty) it returns the hash of the new data (the remote hash with no cache); after writeback, with
the item checked clean, it returns the new remote hash.
TestFileHashUploading: a file being streamed up with the cache off returnsENOENTwithout waiting.cmd/serve/sftpTestHashsumWhileUploadingandcmd/serve/webdavTestEtagWhileUploadingoverwriteonewith
twousing--vfs-write-back 1hand check thatmd5sum/ theETagreport md5 oftwo. With the servechanges reverted both fail, reporting md5 of
one.go build ./...,make quicktest,go test -raceandgolangci-lint runonvfs,cmd/serve/sftpandcmd/serve/webdavpass.Linked issue
Fixes #4956
Checklist
test_allpasses for this backend - not a backend change.