Repository navigation
IO.copy_stream doesn't seem to limit the chunk size #4701
Description
Environment
$ jruby -v
jruby 9.1.12.0 (2.3.3) 2017-06-15 33c6439 Java HotSpot(TM) 64-Bit Server VM 25.40-b25 on 1.8.0_40-b27 +jit [darwin-x86_64]
$ uname -a
Darwin Jankos-MacBook-Pro-2.local 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64
Expected Behavior
When I use IO.copy_stream on MRI, it limits the chunk size to 16KB.
require "stringio"
class FakeIO
def write(data)
p(data.bytesize)
end
end
io = StringIO.new(100*1024)
IO.copy_stream(io, FakeIO.new)16384
16384
16384
16384
16384
16384
4096
Actual Behavior
On JRuby IO.copy_stream appears not to limit the chunk size, it just always reads the entire content at once, because the output of the above script is
102400
I tried with a StringIO up to 10MB of size, and the behaviour is still the same, all 10MB of content is read at once. The behaviour is the same if I change the source IO into a File object.
Activity
Fix tests on JRuby
Fix tests on JRuby
headius commentedon Sep 7, 2017
If you're going from a file to a file we use the builtin Java implementation of FileChannel.transfer, which will do whatever's efficient on that platform. For non-file copying, we do it manually via a loop, and this is likely where we're not choosing a reasonable block size. I'll have a look.
janko commentedon Nov 13, 2017
This snippet shows that
IO.copy_streamin latest JRuby (9.1.14.0) passesnilas thelengthargument when calling#readon the source IO object, which is the same as not passing thelengthargument at all and will read the entire content of the IO object into a string.# output on JRuby [nil]I think this is problematic because the reason people use
IO.copy_streamis because it will copy the contents from source to destination in a memory-efficient way. If JRuby's implementation effectively doesdestination.write source.read, then for larger files the server could easily run out of memory.MRI's
IO.copy_streampasses both alengthargument (of 16KB) and a buffer string to#read, so MRI the output of the previous script is the following:To provide background, I'm a maintainer of two libraries which handle file uploads and use
IO.copy_streamwith non-file IO objects (so JRuby cannot switch to the built-in Java implementation). The second library implements the tus resumable upload protocol, and that one is expected to handle very large file uploads (multiple GB), so I rely onIO.copy_streamfor memory-efficient streaming.headius commentedon Nov 14, 2017
Fixed along with #4842.
janko commentedon Nov 14, 2017
Awesome, thank you!