Repository navigation
System.ArgumentOutOfRangeException when resizing large image #636
Description
Prerequisites
- I have written a descriptive issue title
- I have verified that I am running the latest version of ImageSharp
- I have verified if the problem exist in both
DEBUGandRELEASEmode - I have searched open and closed issues to ensure it has not already been reported
Description
When resizing large image, under certain conditions the following exception is thrown:
SixLabors.ImageSharp.ImageProcessingException: An error occurred when processing the image using ResizeProcessor1. See the inner exception for more detail. ---> System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: minimumLength
at System.Buffers.ConfigurableArrayPool1.Rent(Int32 minimumLength)
at SixLabors.ImageSharp.Memory.ArrayPoolMemoryManager.Allocate[T](Int32 length, Boolean clear)
at SixLabors.ImageSharp.Memory.MemoryManagerExtensions.Allocate2D[T](MemoryManager memoryManager, Int32 width, Int32 height, Boolean clear)
at SixLabors.ImageSharp.Processing.Transforms.Processors.ResizeProcessor1.OnFrameApply(ImageFrame1 source, ImageFrame1 destination, Rectangle sourceRec tangle, Configuration configuration)
at SixLabors.ImageSharp.Processing.Processors.CloningImageProcessor1.CloneAndApply(Image1 source, Rectangle sourceRectangle)
The reason is integer overflow when trying to allocate memory inside ResizeProcessor. In OnFrameApply it tries to allocate "width resizing to" x "source height" x "size of Vector4" array. I have image with dimensions 27400x10138, resizing to 13700x5069. That means it tries to allocate 13700x10138x16 (size of Vector4) block, which is 2222249600, and that is bigger than int.MaxValue, so overflows to negative number. Trying to allocate buffer with negative length results in mentioned exception.
Steps to Reproduce
Simple code to reproduce:
using (var image = new Image<Rgba32>(27400, 10138)) {
image.Mutate(c => c.Resize(13700, 5069));
}
System Configuration
Full .NET on Windows 10, .NET Core on Windows 10, .NET Core on linux (CentOS).
ImageSharp version: 1.0.0-beta0004 (last available on nuget).
Activity
antonfirsov commentedon Jul 4, 2018
We can probably handle this specific case with resize, but it looks like there is a is an upper limit for contiguous (managed?) buffers (
4GBor2GB, not sure yet).If we want to support images bigger than
30,000 x 30,000 [Rgba32 pixels] = 4GB(or maybe22,000 * 22,000 [Rgba32 pixels] = 2GB) we need to integrate advanced memory utilities like ReadOnlySequence into our memory management subsystem. This might be beyond our goals for1.0.@EvK @DanStout regarding your use case: isn't it sufficient to throw a specific exception in those extreme cases?
@JimBobSquarePants your thoughts on the topic?
antonfirsov commentedon Jul 4, 2018
Maybe there is no theoretical limit (at least not for native OS virtual memory) & it's just my system throwing
OutOfMemoryExceptionfornew Vector4[Int32.MaxValue].Nevertheless, we should avoid allocations of this kind.
DanStout commentedon Jul 5, 2018
I'll have to double check what specific size it was, but I don't think it was that large. I thought it was about a 3k x 4k image resized 3-4x in each dimension. In any case it didn't seem like a very extreme case to me!
JimBobSquarePants commentedon Jul 5, 2018
@antonfirsov Looks like there a couple of problems to solve here.
ResizeOptimize memory consumption of ResizeProcessor #642,WuFrameQuantizer<T>,JpegPostProcessorfamilyGetPixelSpan<T>()andGetPixelMemory<T>()API's as it looks like the limit is 2GB https://github.com/dotnet/corefx/issues/26603 which mearns we fall far short of theoretical limits for jpeg for example at 65535 x 65535.ReadOnlySequence<T>does look to me like what we should expose but I don't know yet how we interop that with things like System.Drawing.I'd additionally consider dropping the
RgbaVectorpixel format as that of limited use with no codecs able to preserve the information anyway.When do we do part 2 though? 1.0 seems the safest bet but how long would that take?
vpenades commentedon Jul 30, 2018
Years ago, I had the need to handle very large images, with resize and all. At the time, NetFX3.5 was only able to handle single arrays of up to 2GB, but you got out of memory exceptions much sooner due to memory fragmentation.
The solution was to have a virtualized image over an IBitmap interface. The interface allowed to access the pixels as a continuous matrix. Under the hood, the image was stored as chunks of 256x256 pixels; this allowed better use of fragmented memory, since the image data was sparsely distributed filling the gaps of free memory.
With this approach, operations are, obviously, much slower, but hey, you cannot have big and fast at the same time.
Maybe, for those that require handling extremely large images you could add a "BigSlowImage" class with limited functionality.
antonfirsov commentedon Jul 30, 2018
@vpenades check out
IReadOnlySequence<T>, it's a relatively simple, and standard way to achieve a very similar solution. (Yet I think we won't be able to make it into 1.0)JimBobSquarePants commentedon Apr 26, 2019
Closing. Should be fixed now with #888
antonfirsov commentedon May 2, 2019
The exception message has been improved by SixLabors/Core#25.
If you are really interested in very large (arbitrarily-sized) images, it's tracked in #898 from now.