vfs: serve sftp is reporting incorrect hashes when used with --vfs-cache-mode full #4956

Activity

1. ncw

   added

   bug

   VFS / mount

   on Jan 25, 2021
2. ncw

   added this to the v1.55 milestone on Jan 25, 2021
3. ncw

   changed the title [-]rclone serve sftp needs 2 uploads to save over an existing file[/-] [+]vfs: serve sftp is reporting incorrect hashes when used with --vfs-cache-mode full[/+] on Jan 25, 2021
4. ncw commented on Jan 25, 2021

   

   ncw

   on Jan 25, 2021

   Member

   More actions

   I reproduced this with

   rclone serve -vv sftp --user x --pass x --vfs-cache-mode writes /tmp/sftptest/
   

   And

   rclone sync -P -vv --sftp-host localhost --sftp-port 2022 --bwlimit 100k src :sftp:
   

   I think I see what is going on here.

   • first upload uploads a partial file
   • partial file is uploaded from vfs cache to /tmp/sftptest/
   • retry uploads a full file - this goes to the vfs cache /path/to/vfscache/1M
   • MD5SUM is checked from /tmp/sftptest/1M not /path/to/vfscache/1M
   • Checksums don't match so a retry is initiated

   You can see this is the problem if you use this on the sending command

   --sftp-md5sum-command none  --sftp-sha1sum-command none
   

   Then it will only do 1 transfer not 2.

   What needs to be done here is implement a Hash() method on VFS objects which can read either from the remote object, or from the local cached copy if present.

   This needs to be used in the Etag method in cmd serve webdav

   rclone/cmd/serve/webdav/webdav.go

   Line 372 in 127f48e

   hash, err := o.Hash(ctx, hashType)

   And in the md5sum/sha1sum commands in cmd serve sftp

   rclone/cmd/serve/sftp/connection.go

   Line 121 in 127f48e

   hashSum, err = o.Hash(ctx, ht)
5. ncw

   modified the milestones: v1.55, v1.56 on Apr 3, 2021
6. ncw

   modified the milestones: v1.56, v1.57 on Jul 20, 2021
7. ivandeex

   added

   Serving

   Remote: SFTP

   on Oct 22, 2021
8. ncw

   modified the milestones: v1.57, Known Problem on Nov 1, 2021