Repository navigation
[DAV] Download fails for big files: stream_copy_to_stream() throws an error #1707
Description
Activity
Update: According to sabre-io/http#61, the issue is located somewhere else. The file size seems to be invalid (NaN) which leads to that error. However, if i apply that patch, it works - but is not solving the root cause as noted in the comments.
Update2: The file breaks at exact 4 GB (real size is 9,1 GB). So there must be an issue somewhere.
Can anybody please test large file downloads in an environment with 32 bit an php 7 ? thx !
Well you said you use 32 bits, the max int there is 2147483647.
From min to max, the number is 4294967294, which is exactly 4GB.
So I'd say try to switch to 64 bits if you want to have so huge files.I guess the problem is, that before your cast to int (the intval from the patch), the size is an invalid string. casting to int seems to use the max int, when the value is too high:
See that the following 3 numeric strings all result in the same integer:
https://3v4l.org/OMMWpvar_dump((int) '9223372036854775807'); var_dump((int) '9223372036854775808'); var_dump((int) '92233720368547758081231');
So not sure if there is anything we can do.
There seems to be a general issue with files bigger than 2GB on 32-bit systems. 🙈
There has been a discussion about that topic, generally, stating that as an alternative either floats may be used (which are essentially 64 bits, even on the 32bit arm platform) or numeric strings are an option:
https://github.com/fruux/sabre-dav/issues/408var_dump((int) '9223372036854775807');var_dump((int) '9223372036854775808');var_dump((float) '92233720368547758081231');results in:
int(9223372036854775807) int(9223372036854775807) float(9.2233720368548E+22)... but as far as i understood, floats are not an option for sabre dav, so it possibly won´t change. Nextcloud shall catch the error and throw a warnings prior downloading instead, i guess.
Can you try, whether #1740 fixes the issue?
And by the way, you should not run any nextcloud 11 alpha version yet. We didn't make any release for that Oo
No, does not seem to solve it. Download is still 0 Byte. Log:
{"reqId":"cIm0o8yJYIzbld34vhUX","remoteAddr":"127.0.0.1","app":"PHP","message":"stream_copy_to_stream() expects parameter 3 to be integer, string given at /media/data/www/nextcloud/3rdparty/sabre/http/lib/Sapi.php#78","level":3,"time":"October 15, 2016 11:24:15","method":"GET","url":"/remote.php/webdav/Media-Fernsehaufnahmen/arte_HD/arte_HD_Der_Tag_der_Kr%C3%A4hen20160615_135900.ts","user":"me","version":"9.2.0.4"}
-> I will investigate and test further by the end of tomorrow.
The proposed fix did unfortunately not help. So is there really no chance to download files > 4 GB in an 32 Bit Environment ? I am not into the internals of NC, but i think a download is "streamed" in some kind of chunks, so Filesize should not matter for the download itself, correct ? Is it needed to know the actual filesize (as variable) before downloading ?
Well that seems to exactly be the problem. Before streaming the size is given. But in your case the size is not a number, but a string (because its bigger than the max int) and therefor the error occures.
The problem is, that the log is not really helpful. Let me check if I can find out where this comes from and whether you can add some debug calls for us, that help us finding the real issue.
Can you make the following changes, try an download and then revert the changes again (otherwise your log will become very huge quickly), and then post the log here?
File
3rdparty/sabre/dav/lib/DAV/CorePlugin.php
find (line 795):$propFind->handle('{DAV:}getcontentlength', [$node, 'getSize']);
before add:
\OC::$server->getLogger()->error('Debugging propFind(): ' . json_encode($node->getSize()));
File
apps/dav/lib/Connector/Sabre/Node.php
find (line 201):return $this->info->getSize();
before add:
\OC::$server->getLogger()->error('Debugging getSize(): ' . json_encode($this->info->getSize()));
File
lib/private/Files/FileInfo.php
find:return isset($this->data['size']) ? $this->data['size'] : 0;
before add:
\OC::$server->getLogger()->error('Debugging FileInfo::getSize(): ' . json_encode($this->data['size']));
27 remaining items
Filed a php bug https://bugs.php.net/bug.php?id=74395
Oh nice work @rikmeijer
Why do you use 32bit php in the first place?
Most SBCs such as Raspberry Pi and Odroid devices still rely on 32 Bit. Therefore this change is a real big thing for those.
Reacted by Helder M. and githuleYEP! Just stumbled in this right now. Got really sad :(
just released https://github.com/fruux/sabre-http/releases/tag/4.2.3 which contains a workaround for the 32bit file size problems
Reacted by Joas Schilling, Morris Jobke, Michael New and Saiful KhanI just checked on daily, big downloads on 32 Bit work perfectly now (6,8 GB file in my testcase), the md5 sum matches and network speed did not slow down (tested in my local lan).
Thank you - Let consider this fixed with Nextcloud 13 💯 👍
i use Nextcloud 12.0.3 on a 32Bit 16.04 Ubuntu (from a Webhoster) with PHP 7.0.22
i am facing the issue that the zip generated by the Download-Button crashed downloading arround 4GB.i also tried the sabre-http fix (copied the lib to nextcloud/3rdparty/sabre/http/lib) without success 😢
what did i do wrong?!
Steps to reproduce
Expected behaviour
File with size of 9 GB should be downloaded
Actual behaviour
0 Byte file created, no error message but logged error
Server configuration
Operating system:
Ubuntu 16.04
Web server:
Apache
Database:
MySQL
PHP version:
PHP 7
Nextcloud version: (see Nextcloud admin page)
Nextcloud 11.0.1
Updated from an older Nextcloud/ownCloud or fresh install:
Updated from Owncloud 9 -> Nextcloud 10 -> Nexcloud 11 alpha
Where did you install Nextcloud from:
Git
Signing status:
List of activated apps:
The content of config/config.php:
Are you using external storage, if yes which one: local/smb/sftp/...
local
Are you using encryption: yes/no
no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...
no
Client configuration
Browser: Chrome
Operating system: MacOS 10.12
Logs
Nextcloud log (data/nextcloud.log)
Nextcloud log