FlutterEngineGroup: Hot Reloads hang and Hot Restarts start isolates in different IsolateGroups

Problem

When using a FlutterEngineGroup to provide multiple engines, both hot reload and hot restart appear broken.

In my case, I use a FlutterEngineGroup to provide an engine much like the multiple_flutters example.

I create one engine and provide the result to provideFlutterEngine in my main FlutterActivity. This utilizes createAndRunDefaultEngine method of FlutterEngineGroup to run the main engine.

A secondary engine is created via createAndRunEngine to another entrypoint in the same bundle of code.

Everything works as expected when initially starting, but:

Saving a file and hot reloading causes hot reloading to hang
Running hot restart does work, but creates the isolates in two different IsolateGroup - which impacts cross Isolate communication.

Steps to Reproduce

Steps to reproduce the behavior:

Setup two flutter engines much like the flutter/samples/multiple_flutters. In my instance, I add a FlutterEngineGroup via App.kt and then provide two flutter engines. One in my FlutterActivity via provideFlutterEngine and another in a foreground service (effectively running the isolate "headless"
Run the app via vscode OR flutter run
Observe that there is a single IsolateGroup in the VM Observatory
Click "restart" in vscode or ctrl-shift-r if running via flutter run
Observe that there are now two IsolateGroups in the VM Observatory (the once grouped Isolates are now running in separate groups)

Optionally - to trigger a failure in Hot Reload - simply change any file and trigger a reload. The reload will hang.

Expected results: Hot Reload works, and Hot Restarts start isolates in the same IsolateGroup

Actual results: Hot Reload hangs. Hot Restart works, but starts the isolates in different IsolateGroup, which is different than non-debug runtime behavior.

Code sample

Adapted from the `flutter/samples/mutliple_flutters` sample.

main.dart

@pragma('vm:entry-point')
void main() {
   print('Main running');
   runApp(const MyApp(color: Colors.green));
}

@pragma('vm:entry-point')
void serviceMain() {
   print('Service running'); 
}

App.kt

class App : Application() {
    lateinit var engines: FlutterEngineGroup

    override fun onCreate() {
        super.onCreate()
        engines = FlutterEngineGroup(this)
    }
}

MainActivity.kt

class MainActivity: FlutterActivity() {

  override fun provideFlutterEngine(context: Context): FlutterEngine? {
    return (applicationContext as App).engines.createAndRunDefaultEngine(context);
  }


  override fun configureFlutterEngine(flutterEngine: FlutterEngine) {
    GeneratedPluginRegister.registerGeneratedPlugins(flutterEngine)
    
    // Start foreground service and isolate
    ForegroundService.startService(this@MainActivity)

  }
}

ForegroundService.kt

/// Somewhere in the Foreground Service
val appBundlePath =
            FlutterInjector.instance().flutterLoader().findAppBundlePath()

val entryPoint = DartExecutor.DartEntrypoint(appBundlePath, "package:multiple_flutter_module/main.dart", "serviceMain")

flutterEngine = (applicationContext as App).engines.createAndRunEngine(this, entryPoint)

In the below logs - I reloaded, it succeeded. Then I reloaded with a change - and `Reassembling application` is never logged. I then terminated the app (as the UI was non-responsive).

Logs

[   +2 ms] Flutter run key commands.
[   +1 ms] r Hot reload. 🔥🔥🔥
[        ] R Hot restart.
[        ] h List all available interactive commands.
[        ] d Detach (terminate "flutter run" but leave application running).
[        ] c Clear the screen
[        ] q Quit (terminate the application on the device).
[        ] 💪 Running with sound null safety 💪
[   +1 ms] An Observatory debugger and profiler on Pixel 5 is available at: http://127.0.0.1:34907/UaJa0wTOFA0=/
[  +61 ms] The Flutter DevTools debugger and profiler on Pixel 5 is available at: http://127.0.0.1:9102?uri=http://127.0.0.1:34907/UaJa0wTOFA0=/
[+4348 ms] Skipping target: gen_localizations
[        ] Skipping target: gen_dart_plugin_registrant
[        ] Skipping target: _composite
[        ] complete
[        ] Performing hot reload...
[  +17 ms] Scanned through 592 files in 4ms
[        ] Syncing files to device Pixel 5...
[        ] Compiling dart to kernel with 0 updated files
[        ] Processing bundle.
[        ] <- recompile package:tip_test_isolates/main.dart 44d0762b-9ab5-4789-8c92-3ad83e05dbfa
[        ] <- 44d0762b-9ab5-4789-8c92-3ad83e05dbfa
[        ] Bundle processing done.
[  +18 ms] Updating files.
[ +128 ms] DevFS: Sync finished
[        ] Syncing files to device Pixel 5... (completed in 149ms)
[        ] Synced 0.0MB.
[  +61 ms] Reassembling application
[ +156 ms] Hot reload performed in 372ms.
[   +1 ms] Performing hot reload... (completed in 387ms)
[   +2 ms] Reloaded 0 libraries in 392ms (compile: 19 ms, reload: 0 ms, reassemble: 214 ms).
[+8372 ms] Skipping target: gen_localizations
[        ] Skipping target: gen_dart_plugin_registrant
[        ] Skipping target: _composite
[        ] complete
[        ] Performing hot reload...
[  +13 ms] Scanned through 592 files in 4ms
[        ] Syncing files to device Pixel 5...
[   +2 ms] Compiling dart to kernel with 1 updated files
[        ] Processing bundle.
[        ] <- recompile package:tip_test_isolates/main.dart b2190e91-93d1-4657-8306-e09af61ea038
[        ] package:tip_test_isolates/foo.dart
[        ] <- b2190e91-93d1-4657-8306-e09af61ea038
[        ] Bundle processing done.
[  +70 ms] Updating files.
[  +20 ms] DevFS: Sync finished
[        ] Syncing files to device Pixel 5... (completed in 94ms)
[        ] Synced 0.0MB.
[+4405 ms] Application finished.

No issues found! (ran in 1.0s)

[✓] Flutter (Channel stable, 3.7.10, on Linux Mint 20.3 5.4.0-146-generic, locale en_US.UTF-8)
    • Flutter version 3.7.10 on channel stable at /opt/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 4b12645012 (7 days ago), 2023-04-03 17:46:48 -0700
    • Engine revision ec975089ac
    • Dart version 2.19.6
    • DevTools version 2.20.1

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at /home/dballance/Android/Sdk
    • Platform android-33, build-tools 30.0.3
    • ANDROID_SDK_ROOT = /home/dballance/Android/Sdk
    • Java binary at: /opt/android-studio/jbr/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • CHROME_EXECUTABLE = /opt/brave.com/brave/brave-browser

[✗] Linux toolchain - develop for Linux desktop
    ✗ clang++ is required for Linux development.
      It is likely available from your distribution (e.g.: apt install clang), or can be downloaded from https://releases.llvm.org/
    • cmake version 3.16.3
    • ninja version 1.10.0
    • pkg-config version 0.29.1

[✓] Android Studio (version 2022.1)
    • Android Studio at /opt/android-studio
    • Flutter plugin version 73.0.1
    • Dart plugin version 221.6103.1
    • Java version OpenJDK Runtime Environment (build 11.0.15+0-b2043.56-8887301)

[✓] VS Code (version 1.77.1)
    • VS Code at /usr/share/code
    • Flutter extension version 3.62.0

[✓] Connected device (3 available)
    • Pixel 5 (mobile) • 13291FDD4000UB • android-arm64  • Android 13 (API 33)
    • Linux (desktop)  • linux          • linux-x64      • Linux Mint 20.3 5.4.0-146-generic
    • Chrome (web)     • chrome         • web-javascript • Brave Browser 112.1.50.114

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 1 category.

If I remove the second isolate instantiation (the createAndRunEngine instantiated entrypoint) - reload works appropriately.

Details

[        ] Performing hot reload...
[  +37 ms] Scanned through 592 files in 23ms
[   +1 ms] Syncing files to device Pixel 5...
[   +5 ms] Compiling dart to kernel with 1 updated files
[        ] Processing bundle.
[        ] <- recompile package:tip_test_isolates/main.dart 6ece343c-3c3a-44ff-9187-30ce6304037a
[        ] package:tip_test_isolates/foo.dart
[        ] <- 6ece343c-3c3a-44ff-9187-30ce6304037a
[        ] Bundle processing done.
[  +34 ms] Updating files.
[  +25 ms] DevFS: Sync finished
[        ] Syncing files to device Pixel 5... (completed in 67ms)
[        ] Synced 0.0MB.
[ +164 ms] <- accept
[   +2 ms] reloaded 2 of 825 libraries
[  +37 ms] Reassembling application
[ +114 ms] Hot reload performed in 411ms.
[   +1 ms] Performing hot reload... (completed in 427ms)
[        ] Reloaded 2 of 825 libraries in 435ms (compile: 36 ms, reload: 167 ms, reassemble: 150 ms).

Adds more context and separates the issue from: #119589

@dballance

Thanks for the detailed report @dballance
Please check this related / similar issue and underlying comments which might help for further reference:

#114005 (comment)
#114005 (comment)
#114005 (comment)

@darshankawar

@darshankawar - a few things here:

#114005 (comment)

I have verified that all isolates are being restarted on a restart - just in the wrong IsolateGroup. Also, hot reload doesn't complete. So, unable to verify anything assumed in this comment - simply because the reload doesn't work.

#114005 (comment)

In our usecase, we've been operating by passing a SendPort pair via IsolateNameServer for ~3 years now. One isolate acts as a "server" and advertises a "connect" SendPort used to initiate a connection. But then both Isolates place a "connection specific" SendPort into the IsolateNameServer to create an IsolateChannel to allow a stream of events between both isolates. This works fine in 3.0.5, and is broken (in with hot restarts and hot reloads) in development ONLY in 3.7.x+. If I start the app in a release build - everything works appropriately. So the issue lies in flutter_tools and / or dart-lang/sdk and how it hot reloads and hot restarts isolates from the same FlutterEngineGroup.

From my viewpoint - this is a regression.

However, we did move to FlutterEngineGroup with 3.7.x since the previously working method of simply starting another FlutterEngine no longer starts Isolates in the same IsolateGroup.

#114005 (comment)

This is the rub. I don't believe that this is a tenable workaround. Isolate communication for Isolates in the same IsolateGroup has clearly defined constraints and benefits around memory. See: https://api.dart.dev/stable/2.19.6/dart-isolate/SendPort/send.html

If the sender and receiver isolate share the same code (e.g. isolates created via Isolate.spawn), the transitive object graph of message can contain any object, with the following exceptions:

Further - this all works appropriately at runtime. If I was to deploy this app, without utilizing the "serializing" workaround from the above comment, it will work as expected. I can stop / start the same MainActivity hosted Isolate over and over and it will correctly be started again in the same IsolateGroup.

It's only hot restarts / hot reloads that don't seem to handle FlutterEngineGroup semantics and start the Isolates again in the same IsolateGroup.

Thanks for the detailed feedback and update. Based on the report and above comment, I am going ahead and keep this issue open for team's thoughts.

@aam

/cc @aam @mkustermann

Adjusting labels for new triage process.

Looks like most of the work here is in the engine and not the Dart VM, so removing dart:dependency label.

@gaaclarke

I think this has to do with the call to RuntimeController::Clone instead of RuntimeController::Spawn in Engine::Restart.

We probably need some way to check if an Engine object was spawned from another or not, and if it was we need to Spawn the runtime controller instead of cloning it when restarting.

@gaaclarke might know better.

Specifically, the code here: https://github.com/flutter/engine/blob/1df6e462d333c37a11c6b46f20ae2480e75f85b3/shell/common/engine.cc#L206 seems wrong if the engine was spawned.

Navigation Menu

Repository navigation

`FlutterEngineGroup`: Hot Reloads hang and Hot Restarts start isolates in different `IsolateGroups` #124546

Problem

Steps to Reproduce

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

FlutterEngineGroup: Hot Reloads hang and Hot Restarts start isolates in different IsolateGroups #124546

Description

Problem

Steps to Reproduce

Activity

dballance commented on Apr 10, 2023

dballance commented on Apr 10, 2023

darshankawar commented on Apr 11, 2023

dballance commented on Apr 11, 2023

darshankawar commented on Apr 12, 2023

mraleph commented on Apr 12, 2023

21 remaining items

zanderso commented on Aug 1, 2023

a-siva commented on Jan 25, 2024

dnfield commented on Feb 12, 2024

dnfield commented on Feb 12, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`FlutterEngineGroup`: Hot Reloads hang and Hot Restarts start isolates in different `IsolateGroups` #124546