Use the Dart SDK built from Flutter engine rather than from dartlang.org - #14610

@tvolkert, this is the follow-up to today's conversation regarding using engine's dart sdk instead of downloaded one.

Immediate thing that gets broken with this switch is building of the flutter

╰─➤  bash -x bin/flutter run -v 
...
+ cd $FH/flutter/packages/flutter_tools
+ $FH/flutter/bin/cache/dart-sdk/bin/pub upgrade --verbosity=error --no-packages-dir
Package file has no versions that match 2.3.5 derived from:
- flutter_tools depends on version 2.3.5
+ echo Error: Unable to 'pub upgrade' flutter tool. Retrying in five seconds...
Error: Unable to pub upgrade flutter tool. Retrying in five seconds...
+ sleep 5
...

This is happening I believe because file pub package has sdk dependency on "up to" dev version:

environment:
  sdk: '>=1.19.0 <2.0.0-dev.infinity'

but engine's dart sdk is tagged as edge (since it's built from bleeding edge of dart sdk).
Do you know the reason why file pub package has <2.0.0-dev.infinity rather than <2.0.0 constraint?

Looking at environment/sdk constraint in packages in my .pub-cache I see a mix of <2.0.0-dev.infinity and <2.0.0. I think in order to make pub upgrade work with edge dart sdk all of <2.0.0-dev.infinity should be changed to <2.0.0.

/.pub-cache/hosted/pub.dartlang.org/typed_data-1.1.4/pubspec.yaml:9:  sdk: ">=1.8.0 <2.0.0-dev.infinity"
/.pub-cache/hosted/pub.dartlang.org/async-2.0.3/pubspec.yaml:7:  sdk: ">=2.0.0-dev.15.0 <2.0.0"
/.pub-cache/hosted/pub.dartlang.org/platform-2.1.1/pubspec.yaml:12:  sdk: '>=1.24.0-dev.0.0 <2.0.0-dev.infinity '
/.pub-cache/hosted/pub.dartlang.org/front_end-0.1.0-alpha.6/pubspec.yaml:7:  sdk: '>=1.12.0 <2.0.0-dev.infinity'
/.pub-cache/hosted/pub.dartlang.org/file-2.3.5/pubspec.yaml:19:  sdk: '>=1.19.0 <2.0.0-dev.infinity'
/.pub-cache/hosted/pub.dartlang.org/mime-0.9.6/pubspec.yaml:7:  sdk: '>=2.0.0-dev.20.0 <2.0.0'
/.pub-cache/hosted/pub.dartlang.org/watcher-0.9.7+7/pubspec.yaml:9:  sdk: '>=2.0.0-dev.20.0 <2.0.0'
/.pub-cache/hosted/pub.dartlang.org/kernel-0.3.0-alpha.3/pubspec.yaml:7:  sdk: ">=1.8.0 <2.0.0-dev.infinity"
/.pub-cache/hosted/pub.dartlang.org/meta-1.1.1/pubspec.yaml:10:  sdk: '>=1.12.0 <2.0.0-dev.infinity'
/.pub-cache/hosted/pub.dartlang.org/utf-0.9.0+4/pubspec.yaml:8:  sdk: '>=2.0.0-dev.20.0 <2.0.0'
/.pub-cache/hosted/pub.dartlang.org/analyzer-0.31.0+1/pubspec.yaml:7:  sdk: '>=2.0.0-dev <2.0.0'
/.pub-cache/hosted/pub.dartlang.org/vm_service_client-0.2.4+1/pubspec.yaml:8:  sdk: '>=2.0.0-dev.20.0 <2.0.0'

thanks @tvolkert , I left the best for the last!

/cc @goderbauer for Windows changes

I don't see a description. Can you clarify, please, what this change does?

@mit-mit, flutter engine has dart sdk built as an artifact. This change switches from using dart sdk downloaded based on dart sdk tag(that has to be advanced expliclty) from general dart sdk repository, to using dart sdk that is built as part of flutter engine build.

Got it. Two follow-up questions then:

1. I't my understanding that Engine has often rolled bleeding edge versions of the Dart SDK, whereas the Flutter has has rolled to dev or stable builds. With this change, are we going to be more conservative and also use more stable SDK builds in Engine?

2. As a flutter developer, it's been very useful to be able to roll a new Dart SDK to flutter simply by editing the https://github.com/flutter/flutter/blob/master/bin/internal/dart-sdk.version file. Will that still be possible?

I don't think anything will change in regards to frequency of dart rolls into the engine. So flutter tools will be on same bleeding edge version of dart sdk as engine is.

Yes, you will lose ability to override dart sdk version. Part of the reason is that dart sdk used by flutter tools has to produce kernel file that is compatible, can be consumed by the engine. That means at this point that both flutter tools and engine have to be on the same dart sdk.

OK, that's sad, but I understand

Would be nice to update the description of this PR to explain more what this does. :) Not clear from the comments what the consequences of this are.

Yes, you will lose ability to override dart sdk version. Part of the reason is that dart sdk used by flutter tools has to produce kernel file that is compatible, can be consumed by the engine. That means at this point that both flutter tools and engine have to be on the same dart sdk.

What then is the new process for rolling flutter framework to a new SDK?

What then is the new process for rolling flutter framework to a new SDK?

Flutter tools get the same bleeding edge dart sdk that Flutter engine is on as soon Flutter engine is rolled into Flutter tools.

Right, but presumably the framework doesn't. That is, the file bin/internal/dart-sdk.version still exists, and presumably still means something? Or is this file now residual and should be deleted?

Right, the file should be deleted.

Interesting. Couple of questions:

• This all works with pub? That is, it correctly matches up whatever bleeding edge commit hash you're using against release versions?
• We used to have a constraint that flutter stayed behind google3. Clearly that's no longer operative?
• This means that we no longer "season" things in google3 before rolling them to flutter. Are we ok with that?

This all works with pub? That is, it correctly matches up whatever bleeding edge commit hash you're using against release versions?

Yep

We used to have a constraint that flutter stayed behind google3. Clearly that's no longer operative?

Correct. We used to refuse flutter_tools changes until they could be seamlessly rolled into Google. But the cognitive overhead benefits of having one Dart SDK version for all of Flutter was significant enough that we relaxed this constraint. It does mean that from time to time, we'll need to submit things like #14599 to work around this disparity in Dart SDK versions externally and internally.

This means that we no longer "season" things in google3 before rolling them to flutter. Are we ok with that?

Keep in mind that this only applies to the flutter_tools code. The Flutter app code (and correspondingly, the Flutter framework [i.e. packages/flutter]) was always running inside the Flutter engine and thus always close to tip-of-tree Dart SDK. This change makes the code in packages/flutter_tools follow this same pattern, but inside Google, the flutter_tools code is generally only used to launch Flutter apps -- once they're launched, the app Dart code remains the purview of the Flutter engine.

Keep in mind that this only applies to the flutter_tools code. The Flutter app code (and correspondingly, the Flutter framework [i.e. packages/flutter]) was always running inside the Flutter engine and thus always close to tip-of-tree Dart SDK.

I think this is the source of confusion. You are talking about what is used to run flutter framework (and client) code, I'm talking about what is used to analyze flutter framework (and client) code.

The flutter framework was always running inside flutter engine, but it was always being analyzed by something that was pinned to an earlier version. I understand from the comments above that this is no longer the case.

Correct, that's no longer the case.

The fact that it used to behave like that was itself a source of confusion since (a) whether it was analyzed inside the Flutter engine or using the host SDK was actually dependent upon whether you passed a --watch flag, and (b) you generally want the analysis of your code to match the behavior of the runtime VM.

Ok, interesting. So are flutter dev/stable releases going to try to use Dart dev/stable releases, or will y'all just go with whatever version of Dart is live at the time?

We're not gonna match our dev/stable releases to Dart dev/stable releases.

Our dev releases get cut every time we successfully roll Flutter into Google (which vets the commit as passing level-1 muster), with whatever Dart SDK commit happened to be live in that Flutter build. Each dev release is then a candidate to get promoted to a beta release (as long as it hasn't since been discovered to have release-blocking issues). And then every beta release is a candidate to get promoted to a more stable (e.g. release) channel.

Ok, thanks for the clarification. This has a pretty large impact on how we roll out breaking changes, since it means that you guys will now need to be fixed up first

Our dev releases get cut every time we successfully roll Flutter into Google (which vets the commit as passing level-1 muster)

Just one more comment on this - your level-1 muster here is probably substantially less validation than you're getting now, since you will only be testing out the Dart SDK that you roll into Google on Flutter code in Google, rather than on all Google Dart code, as is done when we roll dev releases .

you will only be testing out the Dart SDK that you roll into Google on Flutter code in Google, rather than on all Google Dart code, as is done when we roll dev releases.

That's a good point, though the non-Flutter Google Dart code is for the most part AngularDart, so to some extent it's exercising different bits of the sdk.

Nonetheless, we could choose to adopt a policy of only rolling Dart into the Flutter engine at dev-tagged milestones... thoughts @Hixie @cbracken ?

Alternatively, I suppose you could make a point of always rolling our dev releases (in addition to the usual bleeding edge rolls), and then trying to choose your own dev releases from among those releases? I guess that constrains your release candidates pretty substantially though.

By the way, just to be clear, I definitely agree that having just one version of the Dart sdk in play in flutter at a time is a good thing.

We've always been on a bleeding edge build of the Dart SDK for the code running on the user device, which is what I'm particularly concerned about; this doesn't change that. I'm not aware of that ever having caused a problem.

Just one more comment on this - your level-1 muster here is probably substantially less validation than you're getting now, since you will only be testing out the Dart SDK that you roll into Google on Flutter code in Google, rather than on all Google Dart code, as is done when we roll dev releases .

Ok, just one more one more comment... :) You're actually not even getting your level-1 checkout for the analyzer from this, since analysis of internal flutter code is done with the standard Dart SDK rather than the flutter engine. I personally really think you would be well-advised to work with keerti to get some kind of automated presubmit run with your release candidates.