Repository navigation
[Embedder] Support render texture for vulkan - #188855
Conversation
|
It looks like this pull request may not have tests. Please make sure to add tests or get an explicit test exemption before merging. If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix? Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing. If you believe this PR qualifies for a test exemption, contact "@test-exemption-reviewer" in the #hackers channel in Discord (don't just cc them here, they won't see it!). The test exemption team is a small volunteer group, so all reviewers should feel empowered to ask for tests, without delegating that responsibility entirely to the test exemption group. |
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
d00520d to
b55309e
Compare
b55309e to
a4618a8
Compare
a4618a8 to
00f4631
Compare
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
There was a problem hiding this comment.
Code Review
This pull request introduces support for external Vulkan textures in the Flutter embedder API, adding the necessary structures, callbacks, and resolver classes to handle texture rendering across both Skia and Impeller backends, including YUV/YCbCr conversion support. Feedback on the changes highlights several critical issues and improvements: correcting a function signature mismatch for the destruction callback to prevent undefined behavior and CFI crashes, respecting the freeze parameter during painting to avoid resolving new frames, adding a null check for the command buffer in the Impeller texture resolver, simplifying the YUV conversion check, and querying vkGetPhysicalDeviceFeatures2KHR as a fallback on Vulkan 1.0 devices to ensure compatibility.
|
FYI I'm out on vacation for the next two weeks. cc @andywolff @gaaclarke @mboetger This PR - which adds support for external Vulkan textures to the embedder API - is ready for review. This is a prerequisite for Android to migrate to the embedder API. If there are any ABI concerns for the embedder API, please also get a review from @cbracken. |
Awesome. This actually aligns with the work that I have done: prototype embedder api. The biggest divergences in the Embedder API:
I'm curious what @andywolff and @gaaclarke have to say though. |
andywolff
left a comment
There was a problem hiding this comment.
I read through the ABI setup, struct versioning, and lifecycle management, and those parts make sense to me. Using struct_size alongside SAFE_ACCESS handles versioning cleanly. This will also allow appending Android-specific fields like external_format later for the embedder migration @mboetger mentioned without breaking backward compatibility. Keeping embedder.h self-contained without direct <vulkan/vulkan.h> includes avoids header leakage across the C ABI. Using VoidCallback with user data batons keeps the cleanup callback signatures consistent. The RAII destruction callbacks ensure resources are freed across normal rendering paths and early failure exits. The physical device feature query in Skia also properly enables samplerYcbcrConversion.
I have a few comments with suggestions on specific areas. I suggest clarifying texture usage and layout expectations to avoid per-frame queue submission overhead in Impeller. I also suggest ensuring parity between Skia and Impeller on multi-planar YUV format handling and BGRA color type mapping, along with adding unit tests for those paths.
andywolff
left a comment
There was a problem hiding this comment.
Thanks for responding to my feedback. I have a few more small comments
There was a problem hiding this comment.
Two things worth resolving before this lands. Both are in the public API rather than the implementation, so they are cheaper to change now than after embedders ship against it.
1. No synchronization contract for the embedder→engine hand-off.
The doc states the image must be in VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL when handed over, but nothing says when the embedder's writes to it become visible. There is no semaphore or fence in the struct, and no sync in either resolve path.
The producers this exists for are cameras and video decoders, writing on their own queue or in another process. Without an ordering guarantee the engine can sample a frame mid-write.
This engine's Vulkan API already states such a contract — in the other direction:
/// The callback invoked when a VkImage has been written to and is ready for
/// use by the embedder. Prior to calling this callback, the engine performs
/// a host sync, and so the VkImage can be used in a pipeline by the embedder
/// without any additional synchronization.
FlutterVulkanPresentCallback present_image_callback;
and for compositor layers:
/// The engine will perform a host sync for all layers prior to calling the
/// compositor present callback, and so the written layer images can be freely
/// bound by the embedder without any additional synchronization.
So engine→embedder is an explicit, documented host sync. This PR is the reciprocal direction and documents no obligation: the embedder is told which layout to leave the image in, but not that its writes must have retired before it returns.
Two ways to close it:
- Document the reciprocal contract — the embedder must host-sync before returning from the callback — matching the wording already used for
present_image_callback. Costs nothing in the API and makes correct embedders possible today. - Carry an acquire semaphore in the struct, letting the producer stay on the GPU.
(1) is consistent with what the rest of the Vulkan embedder API promises and is probably the right scope here. (2) is the better long-term answer for camera and video, and is the reason to at least reserve the field now rather than break the struct later.
For the record, Metal is not a counterexample: FlutterMetalExternalTexture carries no synchronization primitive either and EmbedderExternalTextureMetal::ResolveTexture does no synchronization, so it has the same gap rather than a solution to borrow.
2. struct_size is never validated.
The struct carries struct_size, but the returned texture is read unconditionally on both paths — texture->width, ->height, ->format, ->destruction_callback. SAFE_ACCESS appears twice in the diff and both are on vulkan_config/args, never on the callback's result.
An embedder compiled against an earlier header returns a shorter struct and the engine reads past it. That is the case struct_size exists for, and the rest of embedder.h guards it consistently. Suggest SAFE_ACCESS on each field read, matching the other resolvers.
Minor, while here:
ResolveTextureImpellerbindsauto& impeller_context = impeller::ContextVK::Cast(...)and never uses it; the code below goes throughaiks_context->GetContext()directly.- The PR description is still the template (
*Replace this paragraph...*).
Nice to see both ResolveTextureSkia and ResolveTextureImpeller implemented, and NV12 covered with a fixture — the Vulkan paths in the embedder have a habit of landing Skia-only.
cbracken
left a comment
There was a problem hiding this comment.
LGTM for the embedder ABI side of things! (Sorry, was out last week)
I haven't looked at any of the rest, purely just the usual ABI-stability/evolvability suspects :)
|
Thanks! Looks like fuchia tests are failing because they think two of the variables are unused. Please fix |
andywolff
left a comment
There was a problem hiding this comment.
See my previous comment about unused variables breaking CI
Done |
andywolff
left a comment
There was a problem hiding this comment.
google testing was stuck in a failure loop again so I overwrote it. But Dashboard Checks are all passing now, so LGTM
|
@gaaclarke this is ready for you to take another look, please do |
flutter/flutter@e89fd0a...d03768e 2026-10-02 [email protected] Roll Skia from 7b7326917e77 to 9e88bf828078 (8 revisions) (flutter/flutter#193694) 2026-10-02 [email protected] [Widget Preview] Provide descriptive error when widget preview is unconstrained (flutter/flutter#193005) 2026-10-02 [email protected] Roll Dart SDK from 0e7642b85457 to ac1a97aae47d (26 revisions) (flutter/flutter#193692) 2026-10-02 [email protected] Remove unused Fuchsia sysmem header files (flutter/flutter#193244) 2026-10-02 [email protected] [Windows] Preserve composing extent in setEditingState (flutter/flutter#189968) 2026-10-02 [email protected] [docs] Make Border.symmetric docs more explicit about their arguments (flutter/flutter#193222) 2026-10-02 [email protected] [Embedder] Support render texture for vulkan (flutter/flutter#188855) 2026-10-02 [email protected] Updated Remaining Engine Defaults to SDK 37 (flutter/flutter#190429) 2026-10-02 [email protected] Document that enableSuggestions: false can disable keyboard languages on Android (flutter/flutter#192714) 2026-10-02 [email protected] [flutter_tools] Explicitly track host CPU architecture in command result analytics (flutter/flutter#191836) 2026-10-02 [email protected] [web] Preserve DOM focus on role update and honor isAccessibilityFocusBlocked (flutter/flutter#192963) 2026-10-02 [email protected] [flutter_tools] Include base href in web hot reload script paths (flutter/flutter#193678) 2026-10-02 [email protected] [Impeller] Deduplicate GLES render pass state (flutter/flutter#193427) 2026-10-02 [email protected] Roll Skia from f2d68e0b8863 to 7b7326917e77 (16 revisions) (flutter/flutter#193676) 2026-10-01 [email protected] Fix analysis failures due to missing `const` (flutter/flutter#193685) 2026-10-01 [email protected] Removes a11y_assessment app (flutter/flutter#193671) 2026-10-01 [email protected] test: configure Xvfb and openbox for windowing_test (flutter/flutter#193529) 2026-10-01 [email protected] Sync CHANGELOG.md from stable (flutter/flutter#193666) 2026-10-01 [email protected] Avoid using relative path in Process.start (flutter/flutter#193664) 2026-10-01 [email protected] [AGP 9.1.0 Migration #6] Deliver Flutter assets as a generated assets source directory on the app path (flutter/flutter#192488) 2026-10-01 [email protected] ci(bringup): android_java17_build_android_host_app_with_module_aar is green (flutter/flutter#193580) 2026-10-01 [email protected] [flutter_tools] Fix Use dependency graph to determine plugin initialization order (flutter/flutter#191591) 2026-10-01 [email protected] [flutter_tools] Migrate DaemonCommand and Daemon domains to constructor DI (flutter/flutter#193542) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Adopt the Vulkan external texture embedder API from flutter/flutter#188855 so plugins can hand the engine a VkImage on the Vulkan backends, the counterpart of kFlutterDesktopGpuSurfaceTypeGlTexture2D on EGL. API - embedder.h: add FlutterVulkanExternalTexture, FlutterVulkanTextureFrameCallback and FlutterVulkanRendererConfig::external_texture_frame_callback, verbatim from the merged upstream header. - flutter_texture_registrar.h: new kFlutterDesktopGpuSurfaceTypeVkImage; the descriptor's handle points to a FlutterDesktopVulkanImage {VkImage, VkFormat}. The GPU-surface callback runs on the raster thread on every resolve, so a producer can cycle images. The header documents the full contract: device, layout, host sync, image lifetime, supported formats, and asynchronous unregistration. Resolve and lifetime (flutter_desktop_vk_texture.{h,cc}) - The plugin callback runs with no registrar lock held, so it may host-sync or (un)register textures without stalling or deadlocking other plugins. - Each frame handed to the engine carries an embedder token instead of the plugin's release context. The plugin's release_callback fires exactly once even if the engine reports a frame twice (its Skia path can: BorrowTextureFrom releases on failure and the engine releases again). - Unregistration completes only after the engine has released every frame and no plugin callback is running, so a plugin cannot free an image the engine is still sampling. With a completion callback it never blocks; without one it waits for an in-flight callback (not a re-entrant one) and warns if the engine still holds frames. - Rejects formats the engine cannot sample on both renderers, YCbCr formats unless the device enabled samplerYcbcrConversion (creating a conversion without it is undefined behavior), and sizes beyond maxImageDimension2D. A rejected frame goes straight back to the plugin. - The token table is leaked so engine threads releasing frames during exit() never touch a destroyed map; tokens skip 0 and any still in use after a 32-bit wrap. Backends - wayland_vulkan, drm_kms_vulkan, headless_vulkan: wire the callback. BackendVulkanContext reports sampler_ycbcr_conversion (wayland_vulkan enables it when supported; the others do not request it). - wayland_vulkan set vulkan.struct_size to sizeof(FlutterRendererConfig); it now uses sizeof(FlutterVulkanRendererConfig). Tests - vk_external_texture-test (21 cases, no GPU or GL): resolve, validation, ABI bounds, idempotent release, deferred and re-entrant unregistration, lock-free callback. Clean under TSan and ASan+UBSan. On an engine without #188855 the config field is ignored and VkImage textures register but never resolve. Pixel-buffer textures still resolve only on GL backends. Co-Authored-By: Claude Fable 5.1 <[email protected]>
Adopt the Vulkan external texture embedder API from flutter/flutter#188855 so plugins can hand the engine a VkImage on the Vulkan backends, the counterpart of kFlutterDesktopGpuSurfaceTypeGlTexture2D on EGL. API - embedder.h: add FlutterVulkanExternalTexture, FlutterVulkanTextureFrameCallback and FlutterVulkanRendererConfig::external_texture_frame_callback, verbatim from the merged upstream header. - flutter_texture_registrar.h: new kFlutterDesktopGpuSurfaceTypeVkImage; the descriptor's handle points to a FlutterDesktopVulkanImage {VkImage, VkFormat}. The GPU-surface callback runs on the raster thread on every resolve, so a producer can cycle images. The header documents the full contract: device, layout, host sync, image lifetime, supported formats, and asynchronous unregistration. Resolve and lifetime (flutter_desktop_vk_texture.{h,cc}) - The plugin callback runs with no registrar lock held, so it may host-sync or (un)register textures without stalling or deadlocking other plugins. - Each frame handed to the engine carries an embedder token instead of the plugin's release context. The plugin's release_callback fires exactly once even if the engine reports a frame twice (its Skia path can: BorrowTextureFrom releases on failure and the engine releases again). - Unregistration completes only after the engine has released every frame and no plugin callback is running, so a plugin cannot free an image the engine is still sampling. With a completion callback it never blocks; without one it waits for an in-flight callback (not a re-entrant one) and warns if the engine still holds frames. - Rejects formats the engine cannot sample on both renderers, YCbCr formats unless the device enabled samplerYcbcrConversion (creating a conversion without it is undefined behavior), and sizes beyond maxImageDimension2D. A rejected frame goes straight back to the plugin. - The token table is leaked so engine threads releasing frames during exit() never touch a destroyed map; tokens skip 0 and any still in use after a 32-bit wrap. Backends - wayland_vulkan, drm_kms_vulkan, headless_vulkan: wire the callback. BackendVulkanContext reports sampler_ycbcr_conversion (wayland_vulkan enables it when supported; the others do not request it). - wayland_vulkan set vulkan.struct_size to sizeof(FlutterRendererConfig); it now uses sizeof(FlutterVulkanRendererConfig). Tests - vk_external_texture-test (21 cases, no GPU or GL): resolve, validation, ABI bounds, idempotent release, deferred and re-entrant unregistration, lock-free callback. Clean under TSan and ASan+UBSan. On an engine without #188855 the config field is ignored and VkImage textures register but never resolve. Pixel-buffer textures still resolve only on GL backends.
Adopt the Vulkan external texture embedder API from flutter/flutter#188855 so plugins can hand the engine a VkImage on the Vulkan backends, the counterpart of kFlutterDesktopGpuSurfaceTypeGlTexture2D on EGL. API - embedder.h: add FlutterVulkanExternalTexture, FlutterVulkanTextureFrameCallback and FlutterVulkanRendererConfig::external_texture_frame_callback, verbatim from the merged upstream header. - flutter_texture_registrar.h: new kFlutterDesktopGpuSurfaceTypeVkImage; the descriptor's handle points to a FlutterDesktopVulkanImage {VkImage, VkFormat}. The GPU-surface callback runs on the raster thread on every resolve, so a producer can cycle images. The header documents the full contract: device, layout, host sync, image lifetime, supported formats, and asynchronous unregistration. Resolve and lifetime (flutter_desktop_vk_texture.{h,cc}) - The plugin callback runs with no registrar lock held, so it may host-sync or (un)register textures without stalling or deadlocking other plugins. - Each frame handed to the engine carries an embedder token instead of the plugin's release context. The plugin's release_callback fires exactly once even if the engine reports a frame twice (its Skia path can: BorrowTextureFrom releases on failure and the engine releases again). - Unregistration completes only after the engine has released every frame and no plugin callback is running, so a plugin cannot free an image the engine is still sampling. With a completion callback it never blocks; without one it waits for an in-flight callback (not a re-entrant one) and warns if the engine still holds frames. - Rejects formats the engine cannot sample on both renderers, YCbCr formats unless the device enabled samplerYcbcrConversion (creating a conversion without it is undefined behavior), and sizes beyond maxImageDimension2D. A rejected frame goes straight back to the plugin. - The token table is leaked so engine threads releasing frames during exit() never touch a destroyed map; tokens skip 0 and any still in use after a 32-bit wrap. Backends - wayland_vulkan, drm_kms_vulkan, headless_vulkan: wire the callback. BackendVulkanContext reports sampler_ycbcr_conversion (wayland_vulkan enables it when supported; the others do not request it). - wayland_vulkan set vulkan.struct_size to sizeof(FlutterRendererConfig); it now uses sizeof(FlutterVulkanRendererConfig). Tests - vk_external_texture-test (21 cases, no GPU or GL): resolve, validation, ABI bounds, idempotent release, deferred and re-entrant unregistration, lock-free callback. Clean under TSan and ASan+UBSan. On an engine without #188855 the config field is ignored and VkImage textures register but never resolve. Pixel-buffer textures still resolve only on GL backends. Signed-off-by: Joel Winarske <[email protected]>
This PR adds external texture support for the Vulkan embedder API.
Embedders using Flutter in Vulkan mode can now register a texture frame callback and have the engine directly sample embedder-owned
VkImages during composition.Both the Impeller and Skia Vulkan backends are supported, including RGBA/BGRA formats and multi-planar YUV formats (e.g. NV12) via YCbCr conversion.
API changes (
embedder.h)FlutterVulkanExternalTexture, describing an embedder-ownedVkImagehanded to the engine:image: handle to theVkImage, which must be in theVK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMALlayout when provided to the engine.format: theVkFormatof the image (for exampleVK_FORMAT_R8G8B8A8_UNORM).width/height(non-zero specifies the texture size).destruction_callback+user_data, invoked on an engine-managed thread when the texture can be collected.FlutterVulkanTextureFrameCallback, invoked by the engine when a texture marked with a new frame available needs to be resolved. The embedder must perform a host sync before returning so the engine can sample theVkImagewithout additional synchronization.external_texture_frame_callbacktoFlutterVulkanRendererConfig.Engine changes
EmbedderExternalTextureVulkan(embedder_external_texture_vulkan.{h,cc}), aflutter::Textureimplementation that resolves embedder textures on both backends:EmbedderExternalTextureSourceVulkanimplementsimpeller::TextureSourceVK, creating the image view and aYUVConversionVKwhen the format requires YCbCr conversion.VkImageinto a Skia image, dynamically selecting theSkColorTypebased on theVkFormat.EmbedderExternalTextureResolverto create Vulkan external textures.embedder.ccandembedder_surface_vulkan.cc.Tests
embedder_vk_unittests.cccovering RGBA, BGRA and NV12 textures on both Impeller and Skia, plus destruction-callback variants (RenderTextureWith*Vulkan,Render*TextureWith*Vulkan,RenderTextureWith*VulkanDestructCallback).texture.nv12,external_texture_nv12.png).TestVulkanContextandEmbedderTestContextVulkanto support external texture rendering and sampler YCbCr conversion.Addresses: #117937
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.