Maintain Xamarin iOS support by keeping the library "AOT-proof"

@JimBobSquarePants

Description

Xamarin iOS users need to be able to "seed" open-generic library API-s with the help of AotCompilerTools, otherwise it will lead to runtime exceptions on that platform.

Related: #767, #785, #776, #800

Problems

Currently we are struggling to maintain this feature because of the following reasons:

There is no one in the core team having experience with Xamarin iOS or any other AOT platform.
We don't have any regression tests for this, we never know when a change breaks the "AOT-proofness" of an existing API
We have introduced a massive change with Introduce a non-generic Image base class #904, which will most likely break AOT & Xamarin iOS for many existing API-s in RC1. I have defined a point in epic Epic: introduce pixel-agnostic API across the whole library #907 to investigate this, however in practice it turned out that I hardly have the resources to do so during this summer. (Neither I think @JimBobSquarePants can make it.) Therefore it will most likely not happen for RC1.

Conclusion

With our current prioritites and resources, the only way to maintain this feature is community support. If you can provide any help (add missing AOT seeds, or fix regressions), feel free to open a Pull Request!

Long term solution

If someone could define a mono AOT test configuration for appveyor, that would be ace.

/CC @vicfergar @davilovick @gameleon-dev @Lapinou42 @dmanning23 @swoolcock

EDIT
PS: Could someone do a favor and chack the latest MyGet dev builds from on Xamarin iOS? (JPEG, PNG, Resize, Quantization etc.)

Cool, super appreciate the work yall have been putting into ImageSharp. I have some downtime this week and can take a look at the iOS build.

@dmanning23

@dmanning23 That would be absolutely amazing if you can, thank you very much!

SixLabors#946: AoT compiler fixes and hid Seed methods

Opened a pull-request with a few changes to fix ImageSharp on iOS. I've verified with these changes that it works for jpg, gif, and image resizing.
I also rolled up some changes I've had in mind for the #800 ticket. I've verified that the AoTCompilerTools.Seed methods never actually have to be called, that code just needs to be present for the AoT compiler to pick it up.

Since these AoT/JIT errors only pop up at runtime, and only on physical iOS devices (doesn't occur on simulator!), it'll be extremely difficult to come up with a CI solution to detect them before they pop up. I've started asking around if any other Xamarin developers know how to detect & fix these issues preemptively.

@dmanning23

@dmanning23
Thanks for looking into this!

I've verified with these changes that it works for jpg, gif, and image resizing.

Would these changes also fix issues with PNG images? PNG was the format giving the most iOS based JIT/AOT issues in beta-0006.

Would these changes also fix issues with PNG images? PNG was the format giving the most iOS based JIT/AOT issues in beta-0006.

I don't think so.

Suggested follow-up step:

Run Image.Load(...) for all the test images within our test suite, inside a Xamarin iOS app (running on a device ...).
Collect the failing ones into a checklist of missing seeds, so we can address the issue in a systematic way.

Ok, I can take a look at .png images also.
Are you having problems loading or saving to .png?

Mostly loading from PNG bytes. I haven't been able to properly test encoding to PNG yet in an iOS environment.

In beta0006 I can consistently reproduce the following error by loading png image bytes using Image.Load(pngbytes):

Unhandled managed exception: Attempting to JIT compile method 'SixLabors.ImageSharp.Formats.Png.PngScanlineProcessor:ProcessRgbaScanline<SixLabors.ImageSharp.PixelFormats.Rgba32> (SixLabors.ImageSharp.Configuration,SixLabors.ImageSharp.Formats.Png.PngHeader&,System.ReadOnlySpan`1<byte>,System.Span`1<SixLabors.ImageSharp.PixelFormats.Rgba32>,int,int)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.
 (System.ExecutionEngineException)
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ProcessDefilteredScanline[TPixel] (System.ReadOnlySpan`1[T] defilteredScanline, SixLabors.ImageSharp.ImageFrame`1[TPixel] pixels, SixLabors.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x00135] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.DecodePixelData[TPixel] (System.IO.Stream compressedStream, SixLabors.ImageSharp.ImageFrame`1[TPixel] image, SixLabor
s.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x000e5] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ReadScanlines[TPixel] (System.IO.Stream dataStream, SixLabors.ImageSharp.ImageFrame`1[TPixel] image, SixLabors.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x00018] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.Decode[TPixel] (System.IO.Stream stream) [0x00171] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoder.Decode[TPixel] (SixLabors.ImageSharp.Configuration configuration, System.IO.Stream stream) [0x00008] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Decode[TPixel] (System.IO.Stream stream, SixLabors.ImageSharp.Configuration config) [0x00015] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image+<>c__DisplayClass55_0`1[TPixel].<Load>b__0 (System.IO.Stream s) [0x00000] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLa
bors.ImageSharp.Image.WithSeekableStream[T] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream, System.Func`2[T,TResult] action) [0x0002b] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream, SixLabors.ImageSharp.Formats.IImageFormat& format) [0x00022] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream) [0x00000] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.Byte[] data) [0x00007] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load (System.Byte[] data) [0x00005] in <7430a37e6bca4100b1763856ff8f7867>:0

A similar error occurs when loading a png image from other sources such as a stream.

Someone has to call all the PNG scanline decoder methods from AotCompilerTools.

I expect similar issues with gif, and probably even certain jpegs.

The following steps do not require any knowledge related of our codebase, and it would be really helpful if someone could do them:

Upload our whole test image suite to an iOS device.
Run this code from inside a Xamarin.iOS app. (If running into something like OutOfMemoryException: just delete the giant jpegs.)
Send back Report.txt

This will help getting a nice overview about how bad the situation is. I want to avoid doing the discussion overhead image-by-image.

SixLabors#946: fixed StyleCop errors

Awesome, I will have some extra time next week and will create a quick iOS app with those images & code.

Merge pull request #949 from dmanning23/AoT_iOS_compiler

Ok wrote a quick&dirty iOS app that loads all the images in the test suite, and didn't run into any AoT/JIT errors. Whatever error it was hitting must have been fixed in #949

@gameleon-dev If you want to send me the specific .png I can double check that it will work going forward.

Here is the sample app with output:
https://github.com/dmanning23/ImageSharpTest.iOS

@dmanning23

@dmanning23 Legend! 👑

Too high memory usage.

What if you upscale a different gif? (One that did noes not produce this issue originally)
If it also hangs, can you incrementally find a size where it starts to hang? (or maybe slow down very noticeably)

It's probably worth to try seeding FromRgb24 and similar calls in Seed:

Tried that as well, somehow not working, it's only work if i seed directly for Rgba32.FromRgb24

the difference between the "small" and "large" gifs might be actually a difference in transFlag here:

The transFlag are the same for both images since the width and height ratio is the same.

What if you upscale a different gif? (One that did noes not produce this issue originally)

I've resize the gif to like your suggestion and here is the result:
250x250 gif, it's working fine

400x400 gif, it's crashing without stacktrace

Though these gif are very light, it crashes when only 400x400 pixels, so i don't think that could be high memory usage?

EDIT:
Sorry for wrong debugging information, actually it was Image.Save method that throw nothing with high dimension gif, it was crashing inside the Encode method of GifEncoderCore. And it was crashing at QuantizeFrame method.

And after further debugging, it was this method which crash silently

ErrorDither.ApplyQuantizationDither

So far it looks like the crash is a completely separate problem. Does it happen on emulator as well?

Can you help us by filing a new GitHub issue for it (with templates filled) so we can track it?

@unamed000

Tried that as well, somehow not working, it's only work if i seed directly for Rgba32.FromRgb24

Just to clarify @unamed000 you are saying the following did not work when added to AotCompilerTools?

TPixel pixel = default;

Rgb24 rgb24 = default;
pixel.FromRgb24(rgb24);

@antonfirsov if that is the case then we should migrate to using T4 for generating the tools. Messier to extend but if it's the only thing that works we have little choice.

I would like to try this though before we go that route.

private static void AotPixelInterface<TPixel>()
    where TPixel : unmanaged, IPixel<TPixel>
{
    TPixel pixel = default;
    Rgba32 rgba32 = default;
    pixel.ToRgba32(ref rgba32);
    pixel.FromRgba32(rgba32);
    pixel.FromScaledVector4(pixel.ToScaledVector4());
    pixel.FromVector4(pixel.ToVector4());

    pixel.FromArgb32(default);
    pixel.FromBgr24(default);
    pixel.FromBgra32(default);
    pixel.FromBgra5551(default);
    pixel.FromL16(default);
    pixel.FromL8(default);
    pixel.FromLa16(default);
    pixel.FromLa32(default);
    pixel.FromRgb24(default);
    pixel.FromRgb48(default);
    pixel.FromRgba64(default);
    pixel.FromRgb24(default);
}

I've added them to https://github.com/SixLabors/ImageSharp/tree/cc84f6f66a73febd3c72ec93a932ac6868662acb for testing.

@JimBobSquarePants

@JimBobSquarePants
yes, i tried that before and i didn't workout.
But could you give me a little of explanation why we need to seed these? Even though i copied the entire code into my project seem like i need to seed them as well (I don't have much knowledge about these "seeding" things)

@unamed000

@unamed000 I just updated my branch as I realized I made a mistake in the loader code and was calling the Rgba32 version only. I would pull the latest from that branch and build/reference from that source code.

The reason we seed the code is to attempt to ensure the AOT compiler doesn't try to delete code branches it incorrectly thinks are not being used. By explicitly calling troublesome methods we try to get the compiler to recognize their use.

You shouldn't have to copy the code to your system I wouldn't think.

@JimBobSquarePants

@JimBobSquarePants would you indicate which branch you did the update?

https://github.com/SixLabors/ImageSharp/tree/js/aot-experiments

@unamed000

@unamed000 How did you get on building from the source in the branch?

We need someone to raise an issue upstream here for us to track. We can't fix issues with the AOT compiler.

I had a similar problem with Unity Android IL2CPP, but I found a solution.
Xamarin iOS and Unity Android IL2CPP both work with Mono, so I think you can solve it the same way.

According to the Unity manual, you need to write a specific code to make a particular generic method work.

This should probably cache all interface methods that have generic methods.
For example, this part.

I wanted to shrink a large PNG image, so I created the following code.

/// <summary>
/// Because it calls an internal class, it needs to be placed inside the ImageSharp project.
/// </summary>
static class UnityAoT<TPixel> where TPixel : unmanaged, IPixel<TPixel>
{
    /// <summary>
    /// call from <see cref="SixLabors.ImageSharp.Advanced.AotCompilerTools.Seed{TPixel}"/>
    /// </summary>
    internal static void UnitySeed()
    {
        AotCompileImageProcessor();
        AotCompileResampler();
        AotCompileEncoder();
    }
    private static void AotCompileEncoder()
    {
        default(PngEncoderCore).Encode<TPixel>(default, default, default);
    }
    private static void AotCompileImageProcessor()
    {
        AotCompilerPixelSpecifics<ResizeProcessor>();
    }
    private static void AotCompilerPixelSpecifics<TProc>()
        where TProc : class, ICloningImageProcessor
    {
        default(TProc).CreatePixelSpecificCloningProcessor<TPixel>(default, default, default);
        default(TProc).CreatePixelSpecificProcessor<TPixel>(default, default, default);
    }
    private static void AotCompileResampler()
    {
        AotCompileResamplers<BicubicResampler>();
    }
    private static void AotCompileResamplers<TResampler>()
        where TResampler : struct, IResampler
    {
        // TODO : Methodize it.
        default(ResizeProcessor<TPixel>).ApplyTransform<TResampler>(default);
        default(TResampler).ApplyTransform(default(ResizeProcessor<TPixel>));
    }
}

This is working fine, but doesn't allow me to do anything other than shrink.
I think we need to cache more methods to do all the processing.

@UltraNamahage

@UltraNamahage Thanks for the additional info.

We have a class AotCompilerTools which we have used to for AOT issues previously. Perhaps you can update that?

ImageSharp/src/ImageSharp/Advanced/AotCompilerTools.cs

Line 26 in 36018bd

internal static class AotCompilerTools

@JimBobSquarePants

@JimBobSquarePants Please wait a week.

Navigation Menu

Repository navigation

Maintain Xamarin iOS support by keeping the library "AOT-proof" #946

Description

Problems

Conclusion

Long term solution

34 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Maintain Xamarin iOS support by keeping the library "AOT-proof" #946

Description

Description

Problems

Conclusion

Long term solution

Activity

dmanning23 commented on Jul 15, 2019

JimBobSquarePants commented on Jul 16, 2019

dmanning23 commented on Jul 19, 2019

leonluc-dev commented on Jul 19, 2019

antonfirsov commented on Jul 19, 2019

dmanning23 commented on Jul 19, 2019

leonluc-dev commented on Jul 20, 2019

antonfirsov commented on Jul 26, 2019

dmanning23 commented on Jul 27, 2019

dmanning23 commented on Aug 7, 2019

JimBobSquarePants commented on Aug 7, 2019

34 remaining items

antonfirsov commented on Sep 3, 2020

unamed000 commented on Sep 4, 2020

antonfirsov commented on Sep 4, 2020

JimBobSquarePants commented on Sep 7, 2020

unamed000 commented on Sep 8, 2020

JimBobSquarePants commented on Sep 8, 2020

unamed000 commented on Sep 9, 2020

JimBobSquarePants commented on Sep 9, 2020

JimBobSquarePants commented on Sep 17, 2020

JimBobSquarePants commented on Oct 13, 2020

UltraNamahage commented on Mar 8, 2021

JimBobSquarePants commented on Mar 9, 2021

UltraNamahage commented on Mar 9, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions