Repository navigation
IO.copy_stream doesn't seem to limit the chunk size #4701
Description
Activity
If you're going from a file to a file we use the builtin Java implementation of FileChannel.transfer, which will do whatever's efficient on that platform. For non-file copying, we do it manually via a loop, and this is likely where we're not choosing a reasonable block size. I'll have a look.
This snippet shows that
IO.copy_streamin latest JRuby (9.1.14.0) passesnilas thelengthargument when calling#readon the source IO object, which is the same as not passing thelengthargument at all and will read the entire content of the IO object into a string.source = StringIO.new("foo") destination = File::NULL def source.read(*args) p args super end IO.copy_stream(source, destination)
# output on JRuby [nil]I think this is problematic because the reason people use
IO.copy_streamis because it will copy the contents from source to destination in a memory-efficient way. If JRuby's implementation effectively doesdestination.write source.read, then for larger files the server could easily run out of memory.MRI's
IO.copy_streampasses both alengthargument (of 16KB) and a buffer string to#read, so MRI the output of the previous script is the following:# output on MRI [16384, ""] [16384, "foo"]
To provide background, I'm a maintainer of two libraries which handle file uploads and use
IO.copy_streamwith non-file IO objects (so JRuby cannot switch to the built-in Java implementation). The second library implements the tus resumable upload protocol, and that one is expected to handle very large file uploads (multiple GB), so I rely onIO.copy_streamfor memory-efficient streaming.Awesome, thank you!
Environment
Expected Behavior
When I use
IO.copy_streamon MRI, it limits the chunk size to 16KB.Actual Behavior
On JRuby
IO.copy_streamappears not to limit the chunk size, it just always reads the entire content at once, because the output of the above script isI tried with a
StringIOup to 10MB of size, and the behaviour is still the same, all 10MB of content is read at once. The behaviour is the same if I change the source IO into aFileobject.