[Flutter GPU] Read textures created by another graphics API without a copy #194142

Problem

Flutter GPU can only use textures it created itself (GpuContext.createTexture) or took from a ui.Image (Texture.fromImage, PR #188605 for issue #188246). An image that another GPU API in the same process produced can't be used at all.

Where this comes up:

• Compute or simulation in another library. wgpu-native running in the same process writes its results into a texture, and the only way to get them into Flutter GPU today is a CPU readback followed by Texture.overwrite.
• Video and camera frames that arrive as IOSurfaces, AHardwareBuffers or GL textures.
• Engines and runtimes that hand out images they own.

Impeller already wraps external images internally: TextureMTL::Wrapper/WrapTextureMTL for embedder backing stores, TextureGLES::WrapTexture, AHBTextureSourceVK, and WrapTextureSourceVK from the #187525 series (PR #193001). Flutter GPU can't reach any of them.

Scope

This proposal covers reading external images: sampling them, using them as storage textures in compute, and copying to and from them in blit passes. An imported texture can't be a render pass attachment.

Rendering into externally owned targets (OpenXR swapchain images, embedder surfaces) is #192059, where @adrian-moisa has a working GpuExternalSurface prototype. That work needs a frame acquire/present protocol and probably scheduling changes, as discussed there, and I'm keeping this proposal out of it. If it's useful, the import primitive here could sit under that surface: a surface frame would be an imported image plus the acquire/present rules.

Proposal

Impeller

One entry point on impeller::Context that doesn't depend on the backend:

struct ExternalTextureDescriptor {
  ExternalTextureType type;      // kMTLTexture, kIOSurface, kVkImage,
                                 // kAHardwareBuffer, kGLTexture, kEGLImage
  void* handle;                  // borrowed for the call
  uint64_t param;                // IOSurface plane, VkImageLayout, GL target
  TextureDescriptor desc;        // what the caller says it is
  std::function<void()> release_proc;  // after Impeller's last reference
};

absl::StatusOr<std::shared_ptr<Texture>> Context::ImportTexture(
    ExternalTextureDescriptor external);   // non-virtual: null handle, desc.Validate()
virtual absl::StatusOr<std::shared_ptr<Texture>> OnImportTexture(...);
                                           // ContextMTL, ContextVK, ContextGLES

SurfaceContextVK forwards to its parent. A type the backend doesn't handle fails with kFailedPrecondition.

What each backend checks

The caller describes the texture the same way as for createTexture: format, size, sample count, mip count, storage mode and usage. The backend checks that description against the native object wherever the object can be queried. Usages are requirements, not hints.

• Metal, id<MTLTexture>. The handle has to conform to MTLTexture and belong to the context's device. Type, pixel format, size, sample count, mip count, array length and storage mode must match exactly, and the texture's usage has to allow every requested usage. Memoryless and framebuffer-only textures are rejected.
• Metal, IOSurfaceRef. The handle's CFGetTypeID has to be IOSurfaceGetTypeID(). The surface's pixel format code is mapped per plane against the declared format ('BGRA' to b8g8r8a8UNormInt or its sRGB variant, 'RGBA', 'L008', '2C08', 'RGhA', 'RGfA', 'L00f', and the two planes of '420v'/'420f'), then plane size and bytes per element are checked. Compressed, depth and stencil formats are rejected up front.
• Vulkan, VkImage on Impeller's device. A VkImage can't be queried, so its size and format are taken on trust. What is checked: a single-sampled 2D image with one mip and one layer (the limits of WrapTextureSourceVK), a known current layout, and that the physical device supports the requested usages for the format with optimal tiling. This builds on WrapTextureSourceVK from Move the Vulkan wrapped-texture helper out of the GPU surface #193001.
• Vulkan, AHardwareBuffer (Android). Through the existing AHBTextureSourceVK. Checked: size and layer count from AHardwareBuffer_describe, the format Impeller derives from the buffer, and the usage bits (GPU_SAMPLED_IMAGE for sampling). The import takes its own reference.
• OpenGL ES, texture name. The target has to be GL_TEXTURE_2D. Neither the name nor its storage can be queried off the GL thread, so the description is taken on trust. Impeller never deletes the name.

Designed but not implemented:

• Vulkan, an image from another VkDevice (wgpu-native has its own device). A raw VkImage from another device can't be used. The import would take an opaque fd from vkGetMemoryFdKHR with the allocation size, the memory type bits and the VkImageCreateInfo the producer used, create Impeller's own VkImage with VkExternalMemoryImageCreateInfo, import the memory with VkImportMemoryFdInfoKHR and bind it. That needs VK_KHR_external_memory_fd on Impeller's device: add it to OptionalDeviceExtensionVK next to the external fence and semaphore extensions Impeller already enables on Android, enable it where it's available, and fail with kFailedPrecondition where it isn't. I couldn't test this locally (SwiftShader on macOS has no external memory fd support), so it isn't in the first PRs.
• OpenGL ES, EGLImageKHR (Android). Impeller would own a texture name, bind the image to it with glEGLImageTargetTexture2DOES on the reactor thread (as a 2D or external OES texture, whichever the image allows), and call the release proc after deleting its own name. That needs a small reactor change: a cleanup hook that runs after the GL object is deleted, not instead of deleting it.

Release and ownership

release_proc runs once, after Impeller has dropped its last reference to the native object, on an arbitrary thread. It never runs when the import fails.

• Metal: after the TextureMTL wrapper is deleted. TextureMTL::Wrapper used to call its deletion proc before deleting the wrapper and now deletes first. Committed command buffers keep their own Metal retain.
• Vulkan: from the texture source's deleter, after the image view is destroyed. Command buffers keep the source alive until their fence signals, so this happens after the last GPU use.
• OpenGL ES: from the reactor's cleanup callback, when the handle is collected on the GL thread with the context current. The callback must not call back into Impeller.

Flutter GPU (Dart)

final texture = gpu.gpuContext.importTexture(
  gpu.IOSurfaceHandle(surfacePointer),   // or MetalTextureHandle, VulkanImageHandle,
                                         // AHardwareBufferHandle, GLTextureHandle
  gpu.StorageMode.hostVisible, width, height,
  format: gpu.PixelFormat.b8g8r8a8UNormInt,
  enableShaderReadUsage: true,           // default; the minimal set
  enableShaderWriteUsage: false,
  releaseCallback: nativeRelease,        // Pointer<NativeFunction<Void Function(Pointer<Void>)>>
  releaseUserData: userData,
);

There's no render target usage parameter. NativeTextureHandle is abstract final, so new kinds can be added without breaking exhaustive switches.

The release callback is a native function pointer plus user data. It runs on an arbitrary thread, possibly after the isolate is gone, so a Dart closure isn't safe there; the intended source is a C function exported by the plugin that owns the native object. It's required for the kinds without a reference count (VkImage, GL names), since otherwise the producer can't know when it may destroy the object, and optional for the counted ones (Metal textures, IOSurfaces, hardware buffers), which the backend retains.

Raw handles from Dart, or registration on the native side?

This is the main question for maintainers. There are two shapes:

1. Raw handles from Dart, which is what the branch does. The plugin gives Dart an address and Dart calls importTexture. It's simple and needs no new C API, but Flutter GPU's Dart API would then take raw pointers, which goes against the "safe API" goal in [Flutter GPU] Discovery: Is Flutter GPU's rasterization workflow safe? #150816, and the release callback has to be a native function.
2. Registration on the native side. Native code registers a handle and its release function with the engine and gets an id back; Dart imports by id. Pointers and the release path never cross into Dart. @adrian-moisa's ExternalSurfaceRegistry in Flutter GPU: external render targets for OpenXR — prototype and request for upstream guidance #192059 works this way for render targets and is a good model. The cost is a C API that plugins have to reach (an exported symbol or a registrar method), and an id namespace that has to agree with his.

I don't have a strong preference. (2) is safer and (1) is smaller. If you prefer (2), I'd share one registry with #192059 rather than build a second one.

Synchronization

The first version has no fence. It documents a contract instead: the producer's writes must be complete before a Flutter GPU command buffer that uses the texture is submitted, and the producer must not write again until that command buffer completes (CommandBuffer.submit(completionCallback:) reports when). That works for CPU-paced producers but stalls the pipeline. The next step would be a GpuSharedEvent:

final event = gpu.gpuContext.importSharedEvent(handle);
commandBuffer.waitForEvent(event, value: n);     // before the work that reads
commandBuffer.signalEvent(event, value: n + 1);  // after the work that writes

• Metal: MTLSharedEvent, with encodeWaitForEvent:value: and encodeSignalEvent:value: on the command buffer. It works across queues and devices.
• Vulkan: a timeline semaphore imported with VK_KHR_external_semaphore_fd (opaque fd) and added to the wait and signal lists of the queue submit with VkTimelineSemaphoreSubmitInfo. On Android, where sync fds are what everyone passes around, a binary semaphore imported from a sync fd for the wait, and a VkFence exported with VK_KHR_external_fence_fd for the signal (Impeller already enables both extensions there).
• OpenGL ES: EGL_ANDROID_native_fence_sync to turn a sync fd into an EGLSync and wait on it with eglWaitSyncKHR, and eglDupNativeFenceFDANDROID to export one; glFenceSync/glWaitSync within the share group.

All of these need a small HAL addition: wait and signal lists on CommandQueue::Submit, or CommandBuffer::AddWait/AddSignal. I'd like agreement on that shape before writing it.

Buffers

importDeviceBuffer (an MTLBuffer, a VkBuffer on Impeller's device or from external memory, an AHardwareBuffer blob) would follow the same pattern and matters for compute interop. It's only useful once storage buffer bindings (#188479) land, though, so it isn't part of this. DeviceBufferMTL's constructor is private to AllocatorMTL today, so it would need a wrapper function like WrapTextureMTL.

Status

This is on a local branch, with tests that run on macOS (Metal, Vulkan through SwiftShader, OpenGL ES through ANGLE):

• Context::ImportTexture and the Metal, Vulkan (VkImage, AHardwareBuffer) and OpenGL ES (texture name) backends;
• a fix in BlitCopyTextureToBufferCommandGLES, which treated every wrapped texture as the default framebuffer, so reading back an imported GL texture read the screen instead;
• GpuContext.importTexture in Flutter GPU.

The AHardwareBuffer path compiles only on Android and wasn't built or run locally.

Proposed PR split

1. Impeller: Context::ImportTexture, ExternalTextureDescriptor, and the Metal implementation (with the TextureMTL::Wrapper release order fix).
2. Impeller Vulkan: VkImage on Impeller's device (needs Move the Vulkan wrapped-texture helper out of the GPU surface #193001) and AHardwareBuffer.
3. Impeller OpenGL ES: the blit fix (which could go first on its own) and texture name import.
4. Flutter GPU: GpuContext.importTexture, the handle classes and the Dart tests.
5. Later: external memory from another VkDevice, EGLImage, shared events and buffers.

Questions

1. Raw handles from Dart, or registration on the native side (possibly shared with Flutter GPU: external render targets for OpenXR — prototype and request for upstream guidance #192059)?
2. Is the "producer must have finished" contract acceptable for the first PRs, with shared events as the next step? And which HAL shape for wait and signal?
3. Is a context-level ImportTexture hook the right layering, or would you rather keep backend-specific free functions and dispatch in lib/gpu?