Skip to content

serve sftp, serve webdav: fix stale hashes reported while a file is being uploaded - fixes #4956 - #9946

Open
hsdfat wants to merge 3 commits into
rclone:masterfrom
hsdfat:fix-vfs-hash-4956
Open

hsdfat wants to merge 3 commits into
rclone:masterfrom
hsdfat:fix-vfs-hash-4956

Conversation

@hsdfat

@hsdfat hsdfat commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

What does this change do?

With --vfs-cache-mode writes or full, a file uploaded through rclone serve sftp or rclone serve webdav sits in
the VFS cache until writeback uploads it. Until then the sftp md5sum/sha1sum commands and the webdav ETag and
checksums property report the hash of the old object still on the remote. A client which verifies after upload -
rclone sync to an sftp remote does - sees a mismatch and uploads the file again, as diagnosed in the issue.

Following the approach suggested in the issue, this adds a Hash method to vfs.File. It returns the remote
object's hash unless the file is dirty in the cache, in which case it hashes the cached data read back through the
VFS. serve sftp and serve webdav call it instead of asking the remote object directly.

Files which are not dirty in the cache behave exactly as before: the hash still comes from the remote object, and
a clean copy left in the cache by a read is not used in its place. For dirty files the hash is computed from the
cached copy, so a webdav PROPFIND with --etag-hash reads any files in that directory which are waiting to be
uploaded. A hash type the remote doesn't support is refused for dirty files just as the remote refuses it for clean
ones.

A file with neither a remote object nor a dirty cache item - one still being streamed up with --vfs-cache-mode off - returns ENOENT straight away, so webdav leaves out its ETag and checksums as it did before, without logging an
error for it. The sftp
handler used to open such a file to hash it, which waits up to 5 seconds for the upload and then usually fails
with the same "file does not exist"; File.Hash replaces that fallback, so md5sum now fails without the wait.

Testing

  • vfs/file_test.go TestFileHash, for cache modes off, minimal, writes and full: an unmodified file returns the
    remote hash, including after a read has left a clean copy in the cache; while a writer has new data in the cache
    (checked to be dirty) it returns the hash of the new data (the remote hash with no cache); after writeback, with
    the item checked clean, it returns the new remote hash.
  • TestFileHashUploading: a file being streamed up with the cache off returns ENOENT without waiting.
  • cmd/serve/sftp TestHashsumWhileUploading and cmd/serve/webdav TestEtagWhileUploading overwrite one
    with two using --vfs-write-back 1h and check that md5sum / the ETag report md5 of two. With the serve
    changes reverted both fail, reporting md5 of one.

go build ./..., make quicktest, go test -race and golangci-lint run on vfs, cmd/serve/sftp and
cmd/serve/webdav pass.

Linked issue

Fixes #4956

Checklist

  • This change is trivial OR it has been discussed and agreed in the linked issue.
  • I have read the contribution guidelines.
  • (If I used AI tools to help write this code) I have read and understood the AI-assisted contributions guidance, and I have tested and take ownership of this change myself.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate. (No doc change needed - this makes the existing hash and ETag features report the right value.)
  • All commit messages are in house style.
  • (Backend changes only) test_all passes for this backend - not a backend change.
  • This Pull Request is ready for review.

File.Hash returns the hash of the cached copy while the file is modified
in the VFS cache and waiting to be written back, and the hash of the
remote object otherwise, in the same way that File.Size already reports
the size of a dirty cache item.

For rclone#4956
…fixes rclone#4956

md5sum and friends read the hash from the remote object, which still
describes the old contents for the whole time the new contents are
sitting in the VFS cache waiting to be written back. Use File.Hash
instead so the hash describes the data which will be uploaded.

This also replaces the special case for files which have no remote
object yet, as File.Hash now covers it.
The ETag and the owncloud checksum were read from the remote object,
which still describes the old contents while the new contents are in
the VFS cache waiting to be written back, so a PROPFIND or HEAD could
return the size of the new file next to the ETag of the old one.

For rclone#4956
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vfs: serve sftp is reporting incorrect hashes when used with --vfs-cache-mode full

1 participant