Separate engine builds to build, test archive components

We are planning to split the Mac iOS builds into multiple smaller pieces to optimize resource usage and be able to build/test/archive in less than 30 mins.

In order to start splitting the build it would be great if we can get a dependency graph to help us identify how can we split the gn, ninja, test, archive steps of the recipe.

@zanderso

\cc @zanderso

@gw280

@gw280 is this something you can help us with?

Definitely. When are you going to need this by?

I'm planning to start working on this on Thu | Fri this week.

OK, I'll get this sorted out today/tomorrow.

@gw280

@gw280 thanks for creating the dependency graph. To follow up on this topic we will need to create three different recipes:

Note: I'll refer to a single combination of build flags as flavor. eg ios_debug, ios_debug_arm

manager, will read the build config, trigger subbuilds, wait for completion, collect artifacts, trigger tests and finally create the bundle.
builder, will build a single flavor.
tester, will use the artifacts generated by a builder to run tests.

So far this is the configuration used by the builder recipe:

[
   {
         "gn" : ["--ios", "--runtime-mode", "debug", "--simulator", "--no-lto"],
          "ninja": {"config": "ios_debug_sim", "targets": ["ios_test_flutter"]}
   }
]

The next step is to create the artifacts packager, to optimize the execution time we'll need to package only what is required.

@gw280 can you please help us identify which directories from out are needed for each flavor?

@gw280

The next step is to create the artifacts packager, to optimize the execution time we'll need to package only what is required.

@gw280 can you please help us identify which directories from out are needed for each flavor?

I suspect it's easier to identify the directories that aren't needed, like obj/ and exe.unstripped/. I'm not sure there would be other unneeded ones at the top level beyond those two, but if we need to cut down on space, we can pick out the exact things under gen/ that are needed.

Removing those two directories are probably a good enough optimization, I'll start with that.

@zanderso

So for packaging itself, I think we all need is the .framework files in out/`:

https://github.com/flutter/engine/blob/master/sky/tools/create_ios_framework.py#L32

In particular, this is the actual packaging script and it looks for Flutter.framework only.

For running tests, ultimately we just need the *_unittests, *_tests and *_benchmarks executables and their library and asset dependencies. This is going to be harder to enumerate because the assets and dependencies will differ with each binary and could change when people update the tests etc. In this case I'd say go with @zanderso's suggestion of just removing the directories we are pretty sure aren't needed. obj should be fine, gen is needed because some of the assets like kernel_blob.bin for shell_unittests (for example) are needed from there.

I wonder if we can use gn to list the runtime deps and package them in the most efficient way possible, I've seen that approach used by different teams.

Add gen_snapshot to flutter_gtk.

Add post-processed dart files to web_sdk.

Gn target to generate gen_snapshot for windows.

https://flutter-review.googlesource.com/c/infra/+/34500 adding prod drone builders.

Move the two remaining engine v2 builds to prod. (#37274)

Archive windows gen_snapshot.exe. (#35414)

Add archives to android aot engine. (#37601)

Move the two remaining engine v2 builds to prod. (flutter-team-archiv…

Archive windows gen_snapshot.exe. (flutter-team-archive#35414)

Add archives to android aot engine. (flutter-team-archive#37601)

Move windows engine_v2 to prod. (#37680)

An initial implementation of a tester recipe has been created, that runs Flutter tests on the engine builds uploaded by the engine_v2 recipes. Further work will be done on deciding whether this recipe can be combined with the existing flutter/flutter_drone recipe that is currently used to run tests, using an alternate way of downloading artifacts.

Follow this on the tracking issue: #116906

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Separate engine builds to build, test archive components #81855

29 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Separate engine builds to build, test archive components #81855

Description

Activity

godofredoc commented on May 4, 2021

godofredoc commented on May 4, 2021

gw280 commented on May 4, 2021

godofredoc commented on May 4, 2021

gw280 commented on May 4, 2021

godofredoc commented on May 21, 2021

zanderso commented on May 21, 2021

godofredoc commented on May 21, 2021

gw280 commented on May 24, 2021

godofredoc commented on May 24, 2021

29 remaining items

godofredoc commented on Sep 30, 2022

whesse commented on Dec 12, 2022

github-actions commented on Jun 1, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions