Feature Request for a Synchronous way to Rasterize Images / Draw Pixels to Canvas

@andrewackerman

Use case

This feature request is about the Canvas class in engine/painting.dart.

As extensively discussed in #31598, using regular canvas operations to draw pixel data is extremely slow.
This makes the only option we have decoding an image from our raw pixel data, e.g. using decodeImageFromPixels. This has another problem, pointed out by @andrewackerman in the mentioned issue, which is that creating an Image in Dart is an async operations.
This means that we will have to postpone drawing our raw pixel data to the next frame if we wish to use real-time operations.

The `async` `Image` problem

Images in Flutter currently have the inherent problem that rasterizing / decoding them is always an asynchronous operation. This is why using them for drawing pixel data does not work synchronously, but it also creates a whole nother class of problems:

Any other canvas operation in Flutter that involves Images is also asynchronous. So not only can you not draw pixel data synchronously, but you cannot rasterize any Picture synchronously, which is necessary for achieving various effects (e.g. this). Also Canvas.drawAtlas suffers from this problem.

`drawPixels` proposal

Here is an example proposal (written by @andrewackerman in #31598) for directly drawing pixel data.

Note that this might not cover all use cases because often we do need to have a decoded image (dart:ui.Image) available to us, e.g. when using Canvas.drawAtlas. It should, however, give an idea as to what we are looking for.

[...] it should have a similar signature to decodImageFromPixels [sic], so something like:

class Canvas {
  ...

  void drawPixels(
    Offset c, 
    Uint8List bytes,
    int width,
    int height,
    PixelFormat pixelFormat, {
    int rowBytes,
    int targetWidth,
    int targetHeight,
  }) {
    assert(c != null && bytes != null && width != null && height != null && pixelFormat != null);
    assert(bytes.length == width * height * 4);
    if (rowBytes == null) rowBytes = width * 4;
    if (targetWidth == null && targetHeight == null) {
      targetWidth = width;
      targetHeight = height;
    } else if (targetWidth == null) {
      targetWidth = (targetHeight * (width / height)).toInt();
    } else if (targetHeight == null) {
      targetHeight = (targetWidth * (height / width)).toInt();
    }
    _drawPixels(c, pixelData, width, height, pixelFormat, rowBytes, targetWidth, targetHeight);
  }
}

Click to expand details

c is the position that the image will be drawn at.

bytes is the pixel data that will be drawn to the canvas, represented as byte groups of 4 bytes per pixel. The color channels will be extrapolated based on the value of pixelFormat.

width and height are the dimensions of the image represented by the pixel data. If width * height * 4 is not equal to pixelData.length, an error will be thrown.

rowBytes is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided format.

The targetWidth and targetHeight` arguments specify the size of the output image, in image pixels. If they are not equal to the intrinsic dimensions of the image, then the image will be draw. If exactly one of these two arguments is specified, then the aspect ratio will be maintained while forcing the image to match the other given dimension. If neither is specified, then the image maintains its real size. (Debatable as to whether these parameters should be supported as they don't really fall into the whole "just draw these pixels" mentality of this method.)**

@andrewackerman assumed that the byte data will always have 32 bits per pixel, which makes sense because the only PixelFormat values that currently exist are bgra8888 and rgba8888, which use 32 bits per pixel.
The decodeImageFromPixels function is open to any PixelFormat that might be supported in the future:

rowBytes is the number of bytes consumed by each row of pixels in the data buffer. If unspecified, it defaults to width multiplied by the number of bytes per pixel in the provided format.

I think that this approach would make sense to also be used for the Canvas method (if it will use PixelFormat). Consequently, the assertion for width * height * 4 should probably be width * height * bytesPerPixel and similarly, bytes should be pixel data, represented as byte groups of bytesPerPixel bytes per pixel.

int width, int height & int targetWidth, int targetHeight should be replaced with Size class

Ideally, though this signature matches the style of the next closest existing method for this sort of thing (decodeImageFromPixels). I could see arguments both for leaving it "low-levelish" using only primitives and for standardizing it with Size.

I get the idea but this method would not only be used in conjunction with decodeImageFromPixels and it can feel strange to have different pattern that other methods in Canvas that use Data Object (here it's Rect but it can be any other base class)

void drawImageRect ( Image image, Rect src, Rect dst, Paint paint )

I may be interested to draw pixel directly in a buffer without having Image ever exposed.

Just thinking loudly but I may be interested in having access to a Buffer object with low level methods and then high level methods under Canvas to draw pixels.

Also I think there's I miss about the title of this ticket. I want access to Buffer which is different from synchronous access. I don't know enough the framework but isn't this feature against the principles of the rendering pipeline and impossible (very bad idea) to implement ?

We should never block the main thread so I guess behind the future is the conversion to raster model in another Isolate.

@Solido

@Solido Well, the other canvas methods are synchronous. It would just have to be a little faster..

Others methods are native. Not sure that what we're asking for is available is the same way else
drawPoints would be enough

@Solido

@Solido Well, adding this definitely requires an engine adjustment.

My personal hope is that the implementation of this would be something along the lines of Create internal texture from pixel data >> Paint texture to canvas/painting context. There are currently several methods to draw texture data to the screen (though exact methods escape me at the moment), but precious little documentation on what a "texture" is as those methods take a texture id as a parameter. If a texture is a Skia texture, then I can't imagine it would take that much effort to do it this way (relatively speaking, of course).

It's also possible that low-level painting methods would require waiting for FFI to be finished so that the Skia rendering functions can be invoked directly, bypassing the Flutter engine altogether.

@andrewackerman

@andrewackerman If you mean Texture, that is unrelated to Skia. It is native textures, so not what you are looking for.

@andrewackerman

@andrewackerman There seems to be a lot of consideration done before integrating any new feature and make it visible from skia to flutter.

See Dotted lines #4858
Implemented in Skia since ever and still now in the API.

I let you imagine about exposing Texture ...

@creativecreatorormaybenot

@creativecreatorormaybenot Actually that's exactly what I was looking for. The texture used by that widget is a native texture that can be, among other things, a texture registered to the internal OpenGL context. This is very nearly what would need to happen in order for raw pixel data rendering to get implemented, so it's comforting that it already exists. The only problem with it is that the actual supplying of the texture is done exclusively through native code, so there would need to be some kind of Dart-facing interface to expose the functionality, and doing that through platform channels would run into the same asynchronous and/or performance issues.

If anyone knows of a faster way than manipulating the byte arrays using C++ and dart:ffi, then getting images via decodeImageFromPixels asynchronously, and then rendering using Canvas.drawImage, please let me know - I feel like this is severely limiting Flutter.

I recently also read a very nicely written article touching upon exactly this problem - just this section hurt me:

Sometimes it happens that you just can’t find some things on the Internet, no matter how hard you try

considering all my efforts online regarding this topic.

Any updates on this? Trying to dynamically generate a perlin noise background, and there's no clear way from within Flutter.

@andy98725

@andy98725 Not sure whether shaders can be used for that noise. e.g. https://gist.github.com/patriciogonzalezvivo/670c22f3966e662d2f83#perlin-noise

@andy98725

@andy98725 @fzyzcjy I actually did something very similar when I was trying out the new shader features.

https://github.com/Abion47/flutter-book-of-shaders/blob/main/shaders/src/3-generative-designs/noise-1.glsl

There are no updates because the feature itself is vague and if interpreted literally, technically impossible

How about adding a render thread callback? Unity does precisely that via GL.IssuePluginEvent, and it allows thread-safe custom rendering.

Rendering in Unity can be multithreaded if the platform and number of available CPUs will allow for it. When multithreaded rendering is used, the rendering API commands happen on a thread which is completely separate from the one that runs the scripts. Consequently, it is not possible for your plugin to start rendering immediately, since it might interfere with what the render thread is doing at the time.

A demonstration of the API can be found in Plug-in callbacks on the rendering thread section here:
https://docs.unity3d.com/Manual/NativePluginInterface.html

C++ side:

// Plugin function to handle a specific rendering event
static void UNITY_INTERFACE_API OnRenderEvent(int eventID)
{
    // User rendering code
}
    
// Freely defined function to pass a callback to plugin-specific scripts
extern "C" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API GetRenderEventFunc()
{
    return OnRenderEvent;
}

C# side:

private static extern IntPtr GetRenderEventFunc();

// Queue a specific callback to be called on the render thread
GL.IssuePluginEvent(GetRenderEventFunc(), 1);

A full example of a rendering plugin can be found here
https://github.com/Unity-Technologies/NativeRenderingPlugin/blob/master/PluginSource/source/RenderingPlugin.cpp

@razenpok

@razenpok Implementing the feature as a multithreaded callback strikes me as being idiomatically bizarre for Flutter. Dart uses Isolates for its multithreading which by definition don't share any memory with any other thread, so even if the work was done to somehow wrap the rendering thread in an Isolate to be able to invoke Dart code with it, compared to either a custom shader or the asynchronous options we already have, it sounds like far more work for the same or worse result. I'd imagine this feature would at the very least have to be implemented either as a native-code option or as an FFI plugin in order to even have a chance.

It might be useful to look at some prior art here. When Flash added the BitmapData class to AS3, it was a complete game-changer in how you could work with bitmaps in Flash. It has fast-copy methods for one Bitmap to another, and fast syncronous draw methods where we could blit any DisplayObject (widget) to a BitmapData.
https://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/display/BitmapData.html

We abused this to do a ton of cool effects, it was useful for all kinds of usecases: advanced image filtering like blur/sharpen /other pixel level effects, caching/exporting complex vectors as a raster images etc. You can also combine multiple bitmapdata using blend-modes to create depth of field and tilt-shift style effects.

I think this is all "possible" now in Flutter, but it's slow and requires a lot of work. BitmapData was fast and low-level, which made an excellent foundation to build on. Instead of jumping through a million hoops at the Widget level, just give us a blazing fast path to manipulate textures directly.

If we look at the use-cases asked for above they would all be trivial with bitmapData API. You could just set a single bitmap to be displayed, and directly update it's pixels from dart,

bitmap.lock();
// manipulate pixels as needed
bitmap.unlock(); // bitmap object in memory is updated

It has lots of utility APIs that make things very easy for developers to work with efficiently:

BitmapData(width:int, height:int, transparent:Boolean = true, fillColor:uint = 0xFFFFFFFF)

// Takes a source image and a filter object and generates the filtered image.
applyFilter(sourceBitmapData:BitmapData, sourceRect:Rectangle, destPoint:Point, filter:BitmapFilter):void
  
// Provides a fast routine to perform pixel manipulation between images
copyPixels(sourceBitmapData:BitmapData, sourceRect:Rectangle ... )

// Draws the source display object onto the bitmap image, using the Flash runtime vector renderer.
draw(...)

// Fills a rectangular area of pixels with a specified ARGB color.
fillRect

// Fills a rectangular area of pixels with a specified ARGB color.
floodFill

getPixel / setPixel
getPixels / setPixels

perlinNoise

// Performs pixel-level hit detection between one bitmap image and a point, rectangle, or other bitmap image.
hitTest

Compresses this BitmapData object tp PNG/JPG etc
// encode

The core idea here is

a fast way to capture a Widget to bitmap
a fast way to manipulate the bitmap pixels directly
fast way to apply filters
fast way to encode the bitmap

There is a decodeImageFromPixelsSync now: https://api.flutter.dev/flutter/dart-ui/decodeImageFromPixelsSync.html

This should be resolved.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Feature Request for a Synchronous way to Rasterize Images / Draw Pixels to Canvas #37180

Use case

The `async` `Image` problem

`drawPixels` proposal

58 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature Request for a Synchronous way to Rasterize Images / Draw Pixels to Canvas #37180

Description

Use case

The async Image problem

drawPixels proposal

Activity

Solido commented on Aug 17, 2019

andrewackerman commented on Aug 18, 2019

Solido commented on Aug 18, 2019

creativecreatorormaybenot commented on Aug 18, 2019

Solido commented on Aug 18, 2019

creativecreatorormaybenot commented on Aug 18, 2019

andrewackerman commented on Aug 18, 2019

creativecreatorormaybenot commented on Aug 18, 2019

Solido commented on Aug 18, 2019

andrewackerman commented on Aug 19, 2019

creativecreatorormaybenot commented on Dec 18, 2019

58 remaining items

andy98725 commented on May 8, 2023

fzyzcjy commented on May 8, 2023

Abion47 commented on May 8, 2023

jonahwilliams commented on May 8, 2023

razenpok commented on Oct 5, 2023

Abion47 commented on Oct 5, 2023

esDotDev commented on Jul 13, 2024

iterationharold commented on Oct 16, 2025

gaaclarke commented on Feb 24, 2026

github-actions commented on Mar 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The `async` `Image` problem

`drawPixels` proposal