Repository navigation
Native assets on pub #3693
Description
We've been discussing this for a while, but I wanted to open an issue on the pub repo to track this.
In the context of
we should support uploading precompiled native assets to pub, so that pub can download them for the required target architectures.
If package authors precompile binaries, the current way of working is to either
- host them on a third party server and download them using
bin/setup.dartfor Dart standalone and in the native build scripts for FFI plugins in Flutter, or - push them all to pub (if you're under the 100MB package limit).
This has several drawbacks.
- If the third party server goes down, packages don't run anymore (for option 1).
- The package developer needs to have the binaries for all target architectures on a single machine when doing
dart pub publish(copying over Mac binaries over to Linux for example) (for option 2). pub getdownloads for all target architectures (for option 2).
It would be better if binaries can be uploaded to pub.
The best workflow would be if the binaries can be build on the (Github) CI, and then automatically uploaded to pub. Together with the commit-tagging-leading-to-releases-on-pub. This way package authors don't have to have access to MacOS, Linux and Windows machines to cover all target architectures.
In order to work in this direction we need to solve many design questions. Some questions that come to mind:
- How does authentication/security work?
- What is the list of target architectures. In the current prototype, it is OS x Architecture, but distinguishing iOS device from iOS simulator due to the different SDKs being linked in.
Activity
mosuem commentedon Feb 1, 2024
Any updates on this?
For
package:intl4x, we need to download the correct binary file in thebuild.dart. pub.dev would be the obvious candidate to host these binaries needed to run a package. Currently, I use a 3P solution from @oprypin.Ideally, these binaries could be tied to the package version, and include some sort of signing or similar to increase trust in the downloaded binaries.
mosuem commentedon Mar 4, 2024
Conclusion of an offline talk with @sigurdm and @jonasfj: This will not be supported anytime soon, as adding potentially large files to pub.dev and associating them with releases poses many challenges.
There is a viable alternative: Use Github to create binaries with a release and check their hash in the consuming Dart package.
dcharkes commentedon Mar 4, 2024
That's reason for giving it a p4 or p3 label, not for closing it correct?
faern commentedon Jul 2, 2025
We are currently not using dart, but we are evaluating it for potential use. This issue came up in our supply chain security and dependency auditability evaluation criteria. Having opaque binaries in your dependency tree is a big security issue. Has this part been considered? What prevents a package maintainer from uploading malicious binaries, that then become very hard to spot.
If binaries were forced to be reproducibly built, and have metadata pointing to the source code it was built from, it could mitigate the issue. Because then a downstream user of the package can audit the code and have automatic checks that verify that the binaries were built from that exact code.
At least for us as an organization, it's a hard no to ship products containing code from third parties that we are not able to audit.
dcharkes commentedon Jul 2, 2025
Ideally we'd support something like https://slsa.dev/ for prebuilt binaries.
(Most open source packages should have a user-define for whether libraries are built locally or prebuilt. See an example here: https://github.com/dart-lang/i18n/blob/ab9032794ad7b14a383b4f8883bc7ca0c84c0932/pkgs/intl4x/hook/build.dart#L28. So until we implement a supply chain security solution, you should stick to locally built solutions for your use case.)
simolus3 commentedon Nov 26, 2025
I assume the 100MB size limit is there for a good reason, so wouldn't this just weaken that? I'm not sure if we really need the binaries to be hosted on pub.dev, it sounds to me like assets on GitHub releases are equaly stable. And if package authors have CI/CD infra to compile binaries when publishing, they might as well create an automated GitHub release with all the assets (at least that's what I'm doing for sqlite3).
What if a package declared hashes of assets it might want to download upfront? E.g. something like this in
pubspec.yaml:And then we could have an API like the following:
This would verify that the hash is mentioned in the pubspec, download the asset from the given source URI, verify its hash, then save it in say
.pub-cache/hosted-assets/pub.dev/$packageName-$version/$hashand return the file path. This would allow:.pub-cacheis commonly cached).dcharkes commentedon Nov 26, 2025
I like the idea!
I'm no sure the hashes verification helps a ton, if we don't also tell people to pin all dependencies. E.g. a package could simply make a patch release with new hashes and you'd be downloading unverified binaries. I explored whether it would be possible for example to publish
pubspec.locks on pub.dev inside packages that have abin/so thatdart install,dart pub global activate, anddart run https://pub.dev/<...>wont resolve to newer dependencies. However, this is problematic because (a) locks are explicitly ignored on pub upload, and (b) you can't upload a lock that has local dependencies - which means it doesn't work at all with pub workspaces. Long story short, I'd like https://slsa.dev/ support for the whole Dart ecosystem, but we need to design that holistically, not only check hashes for prebuilt code assets. (Build repeatability is also a problem right now with publishing pub packages as the tar ball has a timestamp in it which changes the hash if you were to reupload exactly the same package at a different timestamp.)If we can't verify hashes ... then I think we can do this idea as a community package, without any connection to pub.
package:github_assets(or a more creative name).$DART_DATA_HOME/github_assets/(see dart-lang/sdk@2af3166 for more details).$DART_DATA_HOME.hook/build.dart(or a yaml file that it references, and add that yaml file to the dependencies).The only thing missing then is a way to purge the assets based on package versions. (I believe we don't even delete old packages from the pub cache right now.)
(Note: I'd still like to do SLSA and artifacts on pub, but having a well paved-way via GitHub actions and artifacts in the meantime is a good state before we go there.)
simolus3 commentedon Nov 26, 2025
I don't understand why we treat binaries differently than Dart source code. If a compromised Dart package has a patch version published to introduce malware written in Dart, would anyone catch that? That's equally powerful and untrusted code that would get downloaded in that case, it's not like binaries can do something Dart code couldn't.
Sure, Dart code is easier to read than precompiled binaries - but that doesn't matter because virtually no one bothers to validate downloaded packages at all. So I think "applications can't really be published to pub.dev" is a different issue.
It would help to ensure that a known good version of a package doesn't get bad (e.g. because an external file URI gets replaced). So it's really more of a feature to guarantee correctness (hooks downloading assets should verify hashes already, but this is a neat way to make unverified downloads impossible and to ensure a package version is immutable).
jonasfj commentedon Nov 26, 2025
jonasfj commentedon Nov 26, 2025
For the record, I'm increasingly convinced that storing binary artifacts on pub.dev is not a bad idea.
There is merit in everything being locked by pubspec.lock -- and pub.dev ensuring files remain available in the future.
But a abusing a service that allows for 1GB tarballs is a lot more attractive, than one allowing 100MB.
Also as mentioned, if we're downloading packages that take up 1-3GB, users will need progressbars and other good UI in pub client, showing outstanding bytes, progress, estimated ETA, restartable downloads, etc.
So I think it might be preferable to only download architectures you intend to use.
(Correct me if my numbers are off, I was just guessing people might want to do multi-GB packages).
As for right now. I think we're still just getting started with native assets.
We should let the community experiment, and patterns evolve before we try to formalize something in pub.
Of the top of me head, I suggest that for now it might be a good idea to:
package:github_assets(as Daco suggested) for making it easy to download artifacts from GitHub releases..dart_tool/...(yes, redownload for each project).This way, when/if we ever build something into pub, we can find users by looking at people using
package:github_assets.We can discontinue such package with link to instructions on how to use the new system. Or even just migrate the package.
simolus3 commentedon Nov 26, 2025
Definitely 👍 I assume most hooks would use one asset per ABI, so there should be a way to download individual assets on demand.
The caching logic in
hook_runnermeans that this is what most hooks are doing already.... Maybe a first step could be to add a global shared cache directory for hooks that is in
DART_DATA_HOME, which could then be used to build a package likegithub_assetson top of it.sigurdm commentedon Nov 27, 2025
Won't pubspec.lock prevent this?
dcharkes commentedon Nov 28, 2025
Only for doing
dart runon a checked out source code from git. It does not prevent it fordart installanddart run https://pub.dev/....I believe the only benefit of doing that over
github_assetsdoing that itself would be that we could potentially provide a command indart clean --globalthat would delete global caches? (Maybe it should also delete the pub cache?!) So far we've not been willing to introduce a global clean. Hence we closed dart-lang/native#1962 saying that a package can make their own global cache and provide abin/clean.dartto clean it.