Repository navigation
[new feature] build_dependencies #3794
Description
Introduce build_dependencies: in pubspec.yaml, and let dart files in build/native.dart only depend on these dependencies.
Consider the following scenario:
package:adepends onpackage:b, but thebuild/native.dartinpackage:adoes not depend onpackage:b.package:candpackage:dcircularly depending on each other, but both of theirbuild/native.dartbuilds do not depend on the other package.- All four packages depend on
package:clangand use it in theirbuild/native.dart.
Specifying build_dependencies has the following advantages:
- Native builds can be run more in parallel, all packages (a, b,c, and d) can be built in parallel.
- Native builds are less often invalidated. Updating
package:ato a newer version does not trigger a rebuild ofpackage:bs’ native assets. - Native builds can work for circular dependencies. Packages c and d have no valid build order if we don’t know that both of their native builds do not depend on each other's native dependencies.
- Native builds can depend on different versions of packages for their native builds. For example
package:acould use a different major version ofpackage:clangthanpackage:b. This would not be possible with their dependencies being resolved in one go. Dependency resolution would fail.
This means that the .dart_tool/package_config.json would contain a resolution set per package for its build_dependencies.
This means we probably need another dart <...> command, because dart run resolves to the bin/ folder, not the build/ folder.
This would require the IDE to understand having different versions of dependencies in a project in the build/.
It is unclear whether any of the benefits outweigh the downsides, so we leave this for when this becomes relevant in the ecosystem.
Workaround: split your package into two packages, and have the native assets in a small package that the rest of the code in the other package depends on.
Context:
Taken from go/dart-native-asset-caching to have a tracking bug.
Activity
dcharkes commentedon Feb 22, 2023
This design would be similar to Rust/Cargo's
build.rs, it also has a separate list of build dependencies: https://doc.rust-lang.org/cargo/reference/build-scripts.html#build-dependencies[pkg][vm] Native Assets builder
devoncarew commentedon Dec 5, 2023
Is this issue mostly about decoupling the build dependencies (the package deps in
dependencies:contributed from the build script) from the regular packages deps (those consumed by code inlib/)?If so, an option to support this that does not require a new
xx_dependencies:section in the pubspec is to putbuild.dartscripts in a new, well-known sub-folder, and optionally, for that sub-folder to have its own pubspec (similar to theexample/directory).Call this directory
build/for now (irrespective of flutter's use of that directory name). So, the build script would go inbuild/build.dartand any link script would go inbuild/link.dart. If they needed improved dependency hygiene - separate deps from what they would get from the parent package - that directory could have its own, separate pubspec file. That file would just use regulardependencies:deps.Perhaps
assemble/would be a good name for the dir?script/?packaging/?build_scripts/?dev/? Dart Asset Collation and Organization -daco/?dcharkes commentedon Jan 25, 2024
Haha, I completely missed this one earlier. 😆 How'd you come up with this! 🤣
Considering the developments in dart-lang/sdk#54334, this should be
hook/.I like it, but we'd need to do a
pub getin a subdir in the pub cache though, before running some of your root-package dependency'shook/build.dart. I don't think @jonasfj would approve modifying the pub cache.jonasfj commentedon Jan 25, 2024
Yeah, modifying
$PUB_CACHEis a big no no.But also you really want your build-dependencies covered by
pubspec.lock.Otherwise, you risk that your build dependencies change between dev machines, CI env, QA builds and release builds. As a result application developers might accidentally push untested code into production -- this what
pubspec.lockaims to prevent.I think there are other reasons for not wanting entirely separate resolutions. If some of your dependencies are in
dependenciesand other dependencies are inbuild_dependencies(which makes sense) -- after all that's howdev_dependencieswork. Then you might be surprised if at build-time packages from yourdependenciessection have a different version than at runtime.IMO, a safer bet here would be
private_dependencies(my proposal is not ready yet, and there is no SDK support for rewriting import-paths / renaming packages).jonasfj commentedon Jan 25, 2024
In particular, I can imagine that given how packages can pass meta-data and build assets between each other, it could cause problems if
foodepends onbar, and both have build-dependencies, butbaruses different package versions at build-time thanfoodoes.Because
foocan consume the binary assets frombar. And those binary assets might be affected by transitive dependencies ofbar, or there may be Dart libraries inbarused at build-time byfooto read the binary assets frombar. This is very reasonable.It's very reasonable for
footo assume that the transitive dependencies underbarat build-time forfooandbarare the same (whenfoohas a build-time dependency onbar).dcharkes commentedon Jan 25, 2024
Right we talked about a similar thing yesterday. I tried to remember it this morning but couldn't come up with it.
So there are multiple places where version skew between different packages
build.dart/link.dartshould not happen.build.dartto another if one package uses a helper package to unparse an object into json and the other package parses that metadata again. Version skew between the twobuild.dartscripts depending on that metadata parsing fails.build.dartto another packageslink.dartif they are for example json serialization of a format comes from a helper package. These two package should not have a different package version for this helper package. @mosuemdcharkes commentedon Jan 25, 2024
@jonasfj Private dependencies is not entirely safe either. Because you could not leak your dependency types to the public API but leak the json-serialized format to metadata or a json encoded asset. So if we introduce private_dependencies, we need to document clearly that you cannot add packages to private_dependencies when you rely on serialized formats (assuming that the serialized formats are versioned by semver).
jonasfj commentedon Jan 25, 2024
You can rely on private_dependencies, you must not re-export anything from a private-dependency.
But yes, there be dragons here 🐉 -- including assetIds that might be hard to figure out.
I think it's important to recognize that multiple versions of the same package is difficult to do. Even if things run in a different context, data or JSON meta-data might flow between them.
It kind of only works, if you promise that packages (and transitive dependencies) of every in
build_dependencieswon't affect your public interface -- the public interface being API, but also assets, meta-data for said assets, etc.That can work, if we make people explicitly promise this. It's complicated. Not perfect. And it doesn't not alleviate us from having to lock everything in the
pubspec.lock. But could be made to work one day.I think all of this is something we can explore if/when version conflicts become a huge issue.
mosuem commentedon Jun 21, 2024
Reviving this discussion.
My worry is that by not having
build_dependencies, we pin the version of any package used link/build hooks for all users of the package. For example, we might want to store the treeshaking information as protobuf, which would mean that we would pin protobuf (at least the major version) for all users of link or build hooks, which might be many users if looking at the transitive closure of package deps.Is there any plan to move forward with some kind of different dependencies for hooks?
jonasfj commentedon Jun 21, 2024
@mosuem Why would you pin dependencies?
sigurdm commentedon Jun 21, 2024
In my understanding the point is that a dependency used for building will constrain the same dependency for run-time while they really are running in separate contexts, and therefore could use separate versions.
jonasfj commentedon Jun 21, 2024
Okay, I understood it as pinning a dependency to the same version.
In the case of protobuf (or similar), I would suspect that there easily is a runtime component to it (some library used by the generated code). And said runtime component would need to be compatible.
mosuem commentedon Jun 21, 2024
Exactly :) Thanks for the clarification.
In the case of storing tree-shaking information, this is only used in the link hook, so no runtime components.
I would even argue that hooks can't have runtime components, as they consume a configuration and produce assets, so there can be no code leakage. So the dependencies are actually separate.