Repository navigation
Maintain Xamarin iOS support by keeping the library "AOT-proof" #946
Description
Activity
- changed the title
[-]Keep the library "AOT-proof" and iOS-compatible[/-][+]Maintain Xamarin iOS support by keeping the library "AOT-proof"[/+]on Jul 14, 2019 Cool, super appreciate the work yall have been putting into ImageSharp. I have some downtime this week and can take a look at the iOS build.
Reacted by Anton FirszovReacted by James Jackson-South@dmanning23 That would be absolutely amazing if you can, thank you very much!
- added a commit that references this issue
on Jul 19, 2019 Opened a pull-request with a few changes to fix ImageSharp on iOS. I've verified with these changes that it works for jpg, gif, and image resizing.
I also rolled up some changes I've had in mind for the #800 ticket. I've verified that the AoTCompilerTools.Seed methods never actually have to be called, that code just needs to be present for the AoT compiler to pick it up.Since these AoT/JIT errors only pop up at runtime, and only on physical iOS devices (doesn't occur on simulator!), it'll be extremely difficult to come up with a CI solution to detect them before they pop up. I've started asking around if any other Xamarin developers know how to detect & fix these issues preemptively.
@dmanning23
Thanks for looking into this!I've verified with these changes that it works for jpg, gif, and image resizing.
Would these changes also fix issues with PNG images? PNG was the format giving the most iOS based JIT/AOT issues in beta-0006.
Would these changes also fix issues with PNG images? PNG was the format giving the most iOS based JIT/AOT issues in beta-0006.
I don't think so.
Suggested follow-up step:
- Run
Image.Load(...)for all the test images within our test suite, inside a Xamarin iOS app (running on a device ...). - Collect the failing ones into a checklist of missing seeds, so we can address the issue in a systematic way.
- Run
Ok, I can take a look at .png images also.
Are you having problems loading or saving to .png?Mostly loading from PNG bytes. I haven't been able to properly test encoding to PNG yet in an iOS environment.
In beta0006 I can consistently reproduce the following error by loading png image bytes using
Image.Load(pngbytes):Unhandled managed exception: Attempting to JIT compile method 'SixLabors.ImageSharp.Formats.Png.PngScanlineProcessor:ProcessRgbaScanline<SixLabors.ImageSharp.PixelFormats.Rgba32> (SixLabors.ImageSharp.Configuration,SixLabors.ImageSharp.Formats.Png.PngHeader&,System.ReadOnlySpan`1<byte>,System.Span`1<SixLabors.ImageSharp.PixelFormats.Rgba32>,int,int)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information. (System.ExecutionEngineException) at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ProcessDefilteredScanline[TPixel] (System.ReadOnlySpan`1[T] defilteredScanline, SixLabors.ImageSharp.ImageFrame`1[TPixel] pixels, SixLabors.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x00135] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.DecodePixelData[TPixel] (System.IO.Stream compressedStream, SixLabors.ImageSharp.ImageFrame`1[TPixel] image, SixLabor s.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x000e5] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ReadScanlines[TPixel] (System.IO.Stream dataStream, SixLabors.ImageSharp.ImageFrame`1[TPixel] image, SixLabors.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x00018] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.Decode[TPixel] (System.IO.Stream stream) [0x00171] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Formats.Png.PngDecoder.Decode[TPixel] (SixLabors.ImageSharp.Configuration configuration, System.IO.Stream stream) [0x00008] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Image.Decode[TPixel] (System.IO.Stream stream, SixLabors.ImageSharp.Configuration config) [0x00015] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Image+<>c__DisplayClass55_0`1[TPixel].<Load>b__0 (System.IO.Stream s) [0x00000] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLa bors.ImageSharp.Image.WithSeekableStream[T] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream, System.Func`2[T,TResult] action) [0x0002b] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream, SixLabors.ImageSharp.Formats.IImageFormat& format) [0x00022] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream) [0x00000] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.Byte[] data) [0x00007] in <7430a37e6bca4100b1763856ff8f7867>:0 at SixLabors.ImageSharp.Image.Load (System.Byte[] data) [0x00005] in <7430a37e6bca4100b1763856ff8f7867>:0A similar error occurs when loading a png image from other sources such as a stream.
Someone has to call all the PNG scanline decoder methods from
AotCompilerTools.I expect similar issues with gif, and probably even certain jpegs.
The following steps do not require any knowledge related of our codebase, and it would be really helpful if someone could do them:
- Upload our whole test image suite to an iOS device.
- Run this code from inside a Xamarin.iOS app. (If running into something like OutOfMemoryException: just delete the giant jpegs.)
- Send back
Report.txt
This will help getting a nice overview about how bad the situation is. I want to avoid doing the discussion overhead image-by-image.
Reacted by Dan Manning- added a commit that references this issue
on Jul 27, 2019 Awesome, I will have some extra time next week and will create a quick iOS app with those images & code.
Reacted by James Jackson-South- added a commit that references this issue
on Aug 6, 2019 Ok wrote a quick&dirty iOS app that loads all the images in the test suite, and didn't run into any AoT/JIT errors. Whatever error it was hitting must have been fixed in #949
@gameleon-dev If you want to send me the specific .png I can double check that it will work going forward.
Here is the sample app with output:
https://github.com/dmanning23/ImageSharpTest.iOSReacted by Anton Firszov, James Jackson-South and leonluc-dev@dmanning23 Legend! 👑
Reacted by Dan Manning34 remaining items
Too high memory usage.
What if you upscale a different gif? (One that did noes not produce this issue originally)
If it also hangs, can you incrementally find a size where it starts to hang? (or maybe slow down very noticeably)It's probably worth to try seeding FromRgb24 and similar calls in Seed:
Tried that as well, somehow not working, it's only work if i seed directly for Rgba32.FromRgb24
the difference between the "small" and "large" gifs might be actually a difference in transFlag here:
The transFlag are the same for both images since the width and height ratio is the same.
What if you upscale a different gif? (One that did noes not produce this issue originally)
I've resize the gif to like your suggestion and here is the result:
250x250 gif, it's working fine
400x400 gif, it's crashing without stacktrace
Though these gif are very light, it crashes when only 400x400 pixels, so i don't think that could be high memory usage?
EDIT:
Sorry for wrong debugging information, actually it was Image.Save method that throw nothing with high dimension gif, it was crashing inside the Encode method of GifEncoderCore. And it was crashing at QuantizeFrame method.And after further debugging, it was this method which crash silently
ErrorDither.ApplyQuantizationDither
Reacted by Anton FirszovSo far it looks like the crash is a completely separate problem. Does it happen on emulator as well?
Can you help us by filing a new GitHub issue for it (with templates filled) so we can track it?
Tried that as well, somehow not working, it's only work if i seed directly for Rgba32.FromRgb24
Just to clarify @unamed000 you are saying the following did not work when added to
AotCompilerTools?TPixel pixel = default; Rgb24 rgb24 = default; pixel.FromRgb24(rgb24);
@antonfirsov if that is the case then we should migrate to using T4 for generating the tools. Messier to extend but if it's the only thing that works we have little choice.
I would like to try this though before we go that route.
private static void AotPixelInterface<TPixel>() where TPixel : unmanaged, IPixel<TPixel> { TPixel pixel = default; Rgba32 rgba32 = default; pixel.ToRgba32(ref rgba32); pixel.FromRgba32(rgba32); pixel.FromScaledVector4(pixel.ToScaledVector4()); pixel.FromVector4(pixel.ToVector4()); pixel.FromArgb32(default); pixel.FromBgr24(default); pixel.FromBgra32(default); pixel.FromBgra5551(default); pixel.FromL16(default); pixel.FromL8(default); pixel.FromLa16(default); pixel.FromLa32(default); pixel.FromRgb24(default); pixel.FromRgb48(default); pixel.FromRgba64(default); pixel.FromRgb24(default); }
I've added them to https://github.com/SixLabors/ImageSharp/tree/cc84f6f66a73febd3c72ec93a932ac6868662acb for testing.
@JimBobSquarePants
yes, i tried that before and i didn't workout.
But could you give me a little of explanation why we need to seed these? Even though i copied the entire code into my project seem like i need to seed them as well (I don't have much knowledge about these "seeding" things)@unamed000 I just updated my branch as I realized I made a mistake in the loader code and was calling the
Rgba32version only. I would pull the latest from that branch and build/reference from that source code.The reason we seed the code is to attempt to ensure the AOT compiler doesn't try to delete code branches it incorrectly thinks are not being used. By explicitly calling troublesome methods we try to get the compiler to recognize their use.
You shouldn't have to copy the code to your system I wouldn't think.
@JimBobSquarePants would you indicate which branch you did the update?
@unamed000 How did you get on building from the source in the branch?
We need someone to raise an issue upstream here for us to track. We can't fix issues with the AOT compiler.
I had a similar problem with Unity Android IL2CPP, but I found a solution.
Xamarin iOS and Unity Android IL2CPP both work with Mono, so I think you can solve it the same way.According to the Unity manual, you need to write a specific code to make a particular generic method work.
This should probably cache all interface methods that have generic methods.
For example, this part.I wanted to shrink a large PNG image, so I created the following code.
/// <summary> /// Because it calls an internal class, it needs to be placed inside the ImageSharp project. /// </summary> static class UnityAoT<TPixel> where TPixel : unmanaged, IPixel<TPixel> { /// <summary> /// call from <see cref="SixLabors.ImageSharp.Advanced.AotCompilerTools.Seed{TPixel}"/> /// </summary> internal static void UnitySeed() { AotCompileImageProcessor(); AotCompileResampler(); AotCompileEncoder(); } private static void AotCompileEncoder() { default(PngEncoderCore).Encode<TPixel>(default, default, default); } private static void AotCompileImageProcessor() { AotCompilerPixelSpecifics<ResizeProcessor>(); } private static void AotCompilerPixelSpecifics<TProc>() where TProc : class, ICloningImageProcessor { default(TProc).CreatePixelSpecificCloningProcessor<TPixel>(default, default, default); default(TProc).CreatePixelSpecificProcessor<TPixel>(default, default, default); } private static void AotCompileResampler() { AotCompileResamplers<BicubicResampler>(); } private static void AotCompileResamplers<TResampler>() where TResampler : struct, IResampler { // TODO : Methodize it. default(ResizeProcessor<TPixel>).ApplyTransform<TResampler>(default); default(TResampler).ApplyTransform(default(ResizeProcessor<TPixel>)); } }This is working fine, but doesn't allow me to do anything other than shrink.
I think we need to cache more methods to do all the processing.Reacted by Anton Firszov, James Jackson-South and unamed000@UltraNamahage Thanks for the additional info.
We have a class
AotCompilerToolswhich we have used to for AOT issues previously. Perhaps you can update that?@JimBobSquarePants Please wait a week.
Reacted by James Jackson-South
Description
Xamarin iOS users need to be able to "seed" open-generic library API-s with the help of AotCompilerTools, otherwise it will lead to runtime exceptions on that platform.
Related: #767, #785, #776, #800
Problems
Currently we are struggling to maintain this feature because of the following reasons:
Conclusion
With our current prioritites and resources, the only way to maintain this feature is community support. If you can provide any help (add missing AOT seeds, or fix regressions), feel free to open a Pull Request!
Long term solution
If someone could define a mono AOT test configuration for appveyor, that would be ace.
/CC @vicfergar @davilovick @gameleon-dev @Lapinou42 @dmanning23 @swoolcock
EDIT
PS: Could someone do a favor and chack the latest MyGet dev builds from on Xamarin iOS? (JPEG, PNG, Resize, Quantization etc.)