Repository navigation
System.ArgumentOutOfRangeException when resizing large image #636
Description
Activity
We can probably handle this specific case with resize, but it looks like there is a is an upper limit for contiguous (managed?) buffers (
4GBor2GB, not sure yet).If we want to support images bigger than
30,000 x 30,000 [Rgba32 pixels] = 4GB(or maybe22,000 * 22,000 [Rgba32 pixels] = 2GB) we need to integrate advanced memory utilities like ReadOnlySequence into our memory management subsystem. This might be beyond our goals for1.0.@EvK @DanStout regarding your use case: isn't it sufficient to throw a specific exception in those extreme cases?
@JimBobSquarePants your thoughts on the topic?Maybe there is no theoretical limit (at least not for native OS virtual memory) & it's just my system throwing
OutOfMemoryExceptionfornew Vector4[Int32.MaxValue].Nevertheless, we should avoid allocations of this kind.
I'll have to double check what specific size it was, but I don't think it was that large. I thought it was about a 3k x 4k image resized 3-4x in each dimension. In any case it didn't seem like a very extreme case to me!
@antonfirsov Looks like there a couple of problems to solve here.
- We need to avoid allocating large buffers and conduct a thorough review of various known allocation hotspots.
ResizeOptimize memory consumption of ResizeProcessor #642,WuFrameQuantizer<T>,JpegPostProcessorfamily
- Review our
GetPixelSpan<T>()andGetPixelMemory<T>()API's as it looks like the limit is 2GB https://github.com/dotnet/corefx/issues/26603 which mearns we fall far short of theoretical limits for jpeg for example at 65535 x 65535.ReadOnlySequence<T>does look to me like what we should expose but I don't know yet how we interop that with things like System.Drawing. - Should we still have issues after the above, provide a neat way to warn when those limits are exceeded during allocation.
I'd additionally consider dropping the
RgbaVectorpixel format as that of limited use with no codecs able to preserve the information anyway.When do we do part 2 though? 1.0 seems the safest bet but how long would that take?
Years ago, I had the need to handle very large images, with resize and all. At the time, NetFX3.5 was only able to handle single arrays of up to 2GB, but you got out of memory exceptions much sooner due to memory fragmentation.
The solution was to have a virtualized image over an IBitmap interface. The interface allowed to access the pixels as a continuous matrix. Under the hood, the image was stored as chunks of 256x256 pixels; this allowed better use of fragmented memory, since the image data was sparsely distributed filling the gaps of free memory.
With this approach, operations are, obviously, much slower, but hey, you cannot have big and fast at the same time.
Maybe, for those that require handling extremely large images you could add a "BigSlowImage" class with limited functionality.
@vpenades check out
IReadOnlySequence<T>, it's a relatively simple, and standard way to achieve a very similar solution. (Yet I think we won't be able to make it into 1.0)Closing. Should be fixed now with #888
The exception message has been improved by SixLabors/Core#25.
If you are really interested in very large (arbitrarily-sized) images, it's tracked in #898 from now.
Reacted by James Jackson-South
Prerequisites
DEBUGandRELEASEmodeDescription
When resizing large image, under certain conditions the following exception is thrown:
The reason is integer overflow when trying to allocate memory inside
ResizeProcessor. InOnFrameApplyit tries to allocate "width resizing to" x "source height" x "size of Vector4" array. I have image with dimensions 27400x10138, resizing to 13700x5069. That means it tries to allocate 13700x10138x16 (size of Vector4) block, which is 2222249600, and that is bigger than int.MaxValue, so overflows to negative number. Trying to allocate buffer with negative length results in mentioned exception.Steps to Reproduce
Simple code to reproduce:
System Configuration
Full .NET on Windows 10, .NET Core on Windows 10, .NET Core on linux (CentOS).
ImageSharp version: 1.0.0-beta0004 (last available on nuget).