Repository navigation
Speed up transforming assets #168629
Description
Hey!
In our project, we have a Design System package with many many icons, the SVG files. We wanted to use the vector_graphics_compiler to transform them into compiled form and save on SVG parsing on the device, but there is one problem.
The asset transformation happens for every flutter run for every asset. It's not cached in the build/ directory or anything like that. Transforming them once takes a good 5-10 minutes (on a not-bad M1 Pro MacBook). Well, that's the cost of this transformation, sure. But if the assets were cached, and e.g. their checksum checked first if they're not already transformed, it would save this time on hot-reloads, consequent builds/runs, and also we could put this cache directory under cache on CI/CD. Right now it's unusable with asset transformers.
The flutter_tool also spawns a big amount of processes to run on each asset. Maybe that could be optimized somehow to run as many as the machine's CPU has or something along these lines, if that could help the performance. The other solution would be to support transforming whole directories (which we too can pass in assets in pubspec, not only files one by one), so that the flutter_tool would spawn one process per asset definition and not file, hence saving at least on spinning-up the Dart executable and the script. That would of course also require a change of the specification of executables for transformers and a change in vector_graphics_compiler package. Just thinking out loud.
Thanks
Flutter 3.29.3 • channel stable • https://github.com/flutter/flutter.git
Framework • revision ea121f8859 (4 weeks ago) • 2025-04-11 19:10:07 +0000
Engine • revision cf56914b32
Tools • Dart 3.7.2 • DevTools 2.42.3
Activity
Albert221 commentedon May 10, 2025
cc @andrewkolos, as I'm aware you're the one behind the transformers (thanks for the work on them!)
jonahwilliams commentedon May 13, 2025
There is caching for the generated assets, but If I remember the way it works is that the entire bundle is cached as one unit. So its possible that small changes are invalidating it.
That said, I've found that its often simpler to make the process faster than it is to get an effective caching strategy. The first thing I'd do is check to make sure that we're limiting the compilation process to something like half of available concurrency, then make sure that we're using compiled dart snapshots instead of running from source.
If both of those are true, i'd look at more finely grained caching. Its possible that may not be much of a win - I'd expect that individually each vector graphic asset is only a bit slower (10x) to compile than reading the file and hashing it.
jonahwilliams commentedon May 13, 2025
From checking it looks like we're using a max concurrency of 64, which is probably way too high. That number was based on only doing copying during asset transformation, we need to use a smaller limit for process execution.
jonahwilliams commentedon May 13, 2025
@Albert221 here is a patch to try to see if makes things go a bit faster for you: https://github.com/flutter/flutter/compare/master...jonahwilliams:flutter:lower_transform_limit?expand=1
Albert221 commentedon May 16, 2025
With this addition, the asset transformation took 10 minutes (whole
flutter runtook 680,563ms). And my CPU usage was at 60%, and not 100%. Fans didn't even turn on :)Without this change, the asset transformation took less than 9 minutes (8:51; whole
flutter runtook 565,477ms).Both time measurements were after
flutter clean, runningflutter runand Cmd+C right when I started seeing in verbose output it started to asset transform, then once againflutter runthat I measured.So looks like a pool of 4 transformers is making it longer
andrewkolos commentedon May 16, 2025
If I recall correctly, implementing caching for transformers would require a way to check whether the package doing the transformation has changed since the last build. Said another way, if a user uses a package as a transformer and they modify that package between builds1, how do we make sure that the asset is re-transformed?
Perhaps we could calculate a checksum of the transformer package's directory? This would definitely not work for some edge cases2 but should work fine in the vast majority of scenarios.
Footnotes
This scenario seems unlikely but could be relevant in the case that a local path package is being used as a transformer (e.g. in a monorepo project) ↩
Consider a hypothetical transformer that depends on machine state or some other directory, etc. ↩
matanlurey commentedon May 23, 2025
I think it would be reasonable to:
dart run)... before investing in any (expensive to get right/support) caching strategies.
[-][asset_transformers] Cache transformed assets[/-][+]Speed up transforming assets[/+]matanlurey commentedon May 23, 2025
@Albert221 Would you mind giving #169386 a shot? It should use 1/2 the CPUs - which might be higher than 4 for your tests.
Use at most `PROC~/2` tasks to transform assets. (#169386)
Use at most `PROC~/2` tasks to transform assets. (flutter#169386)
Better null-safe implementation of `AssetTransformerEntry`. (#169645)
davidmorgan commentedon May 30, 2025
build_runnerdoes exactly this: caching/invalidation over the Dart source code that is doing the transformation and the input files.But, it does not today support running multiple processes. So it would be able to do the invalidation/caching but you'd get much lower throughput.
Scaling up to multiple processes according to what's available on the machine is exactly something I want it to be able to do. So, perhaps it will be a good option at some point.
Filed dart-lang/build#4023 to look into it ... but not soon ... maybe later this year / some time next year :) ... could push that a bit sooner if there is particular interest from your side.
matanlurey commentedon May 30, 2025
@davidmorgan Thanks! To be clear, however confusing this is is - asset transformers shares no code with/does not use
build.davidmorgan commentedon Jun 2, 2025
I know :) sorry, I should have been clearer: the reason I mention
buildis because it already does one of the things Jonah+Andrew were discussing: it can determine if your transformer source changed. So instead of writing new code to do that, you could just usebuild. But, there are probably reasons today why it's not a fit. That's potentially good for me, as it gives me a list of things it would be useful to support, and I'm looking for things that would be useful to add/improve inbuild:)mosuem commentedon Jun 2, 2025
@Albert221 We are currently in the process of shipping hooks for data assets to Flutter - this is experimental for now, but the intention is to replace asset transformers with this more general mechanism.
So in the (hopefully near) future, you should be able to use a
hook/build.dartfile which will do the work for you, including caching and such, and of course using only one "run" thus eliminating performance issues.Use at most `PROC~/2` tasks to transform assets. (#169386)
Better null-safe implementation of `AssetTransformerEntry`. (#169645)
vm-px commentedon Oct 8, 2026
Additional data point from Windows (Flutter 3.44.8): with 181 SVGs using the vector_graphics_compiler transformer, flutter widget-preview start (and any web build) drives memory usage to 70–90% of 32 GB and freezes the machine. Web asset bundling in bundle_builder.dart runs transformations through Pool(64), so up to 64 dart run vector_graphics_compiler processes run at once, while the native build target (build_system/targets/assets.dart) limits it to numberOfProcessors ~/ 2. Concurrent runs also fail on Windows with "file is used by another process" when reading the compiler snapshot from .dart_tool/pub/bin. Applying the same CPU-based limit to bundle_builder.dart would already make web builds and widget previews usable.
PR #169386 applied the CPU-based limit only to build_system/targets/assets.dart; bundle_builder.dart (web builds, widget previews) still uses Pool(64)