A cold-start "flutter test" takes multiple seconds

@mraleph

Running "flutter test" used to be pretty fast (a few hundred milliseconds until the test runs — not great, but workable). Now though it takes entire seconds to start up the test. This makes iterating on a test an unbearably slow process.

cc @mraleph @aam @a-siva @tvolkert

dart-lang/sdk#32953 might be related.
#15767 is similar
#4719 is about flutter test --watch to improve performance.

We can reduce script start-up time 0.3 seconds by switching to app jit (Switch flutter_tools from script to app-jit snapshot. #27749).
Overall test performance is also influenced by the flutter_tester (see Flutter Tester binaries should be optimized with debug assertions enabled #22873). I noticed that running from a locally built profile build reduced overall test execution time to ~70% of the normal time.
We also spend about 1-2 seconds compiling to kernel for a test (more if there are more dependencies). We might be able to speed this up by incrementally compiling the dependencies into kernel once and reusing them for subsequent test runs. Currently we use the frontend server to keep the kernel in memory which helps for multiple tests, but everything gets thrown away on subsequent test runs. This will require Tracking Issue for landing code generation support in flutter tool #27141 to integrated into the tool.

We spend a couple hundred ms in the tool before even getting to the test execution:

jonahwilliams-macbookpro2:hello_world jonahwilliams$ flutter test -v
[  +25 ms] executing: [/Users/jonahwilliams/Documents/flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +32 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[        ] upstream/master
[        ] executing: [/Users/jonahwilliams/Documents/flutter/] git rev-parse --abbrev-ref HEAD
[   +8 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[        ] initialize_from_dill
[        ] executing: [/Users/jonahwilliams/Documents/flutter/] git ls-remote --get-url upstream
[   +9 ms] Exit code 0 from: git ls-remote --get-url upstream
[        ] [email protected]:flutter/flutter.git
[        ] executing: [/Users/jonahwilliams/Documents/flutter/] git log -n 1 --pretty=format:%H
[  +12 ms] Exit code 0 from: git log -n 1 --pretty=format:%H
[        ] 216a65b6697f2ca5cf3501e50c3f9592334fbe4b
[        ] executing: [/Users/jonahwilliams/Documents/flutter/] git log -n 1 --pretty=format:%ar
[  +12 ms] Exit code 0 from: git log -n 1 --pretty=format:%ar
[        ] 2 days ago
[   +1 ms] executing: [/Users/jonahwilliams/Documents/flutter/] git describe --match v*.*.* --first-parent --long --tags
[  +52 ms] Exit code 0 from: git describe --match v*.*.* --first-parent --long --tags
[        ] v1.2.0-140-g216a65b66
[ +314 ms] switching to directory LocalDirectory: 'test' to run tests
[   +1 ms] running test package with arguments: [-r, compact, --concurrency=6, --, /Users/jonahwilliams/Documents/flutter/examples/hello_world/test/hello_test.dart]

From investigating we spend approximately 0.4 seconds gathering analytics data. Times below on hello_world test

All usage and version gathering removed:

time flutter test
00:01 +1: All tests passed!                                                                                                                                                                                
DONE: 1815 ms

real	0m2.428s
user	0m2.568s
sys	0m0.572s

master:

jonahwilliams-macbookpro2:hello_world jonahwilliams$ time flutter test
00:01 +1: All tests passed!                                                                                                                                                                                
DONE: 1955 ms

real	0m2.803s
user	0m2.754s
sys	0m0.689s

We also spend a couple hundred milliseconds checking pub get. We could replace that by first checking if the pubspec.yaml was modified after the .packages.

without analytics, pub get, and with app jit snapshot we're almost fast:

jonahwilliams-macbookpro2:hello_world jonahwilliams$ time flutter test
00:01 +1: All tests passed!                                                                                                                                                                                
DONE: 1916 ms
real	0m2.200s
user	0m2.334s
sys	0m0.540s

And to back up my claims about the host_profile, flutter test in the flutter gallery takes
default: 101463 ms
profile: 54277 ms

So actually about half the time

Currently we use the frontend server to keep the kernel in memory which helps for multiple tests, but everything gets thrown away on subsequent test runs.

This is not quite right - on subsequent runs incremental compiler gets bootstrapped from previously generated dill file: https://github.com/flutter/flutter/blob/master/packages/flutter_tools/lib/src/test/flutter_platform.dart#L274

Understood, we might be lumping in some additional start/warm-up time in the first compile then. I see If I can get a better breakdown of that time

We've moved the needle on the benchmarks a bit by switching to an app-jit snapshot: #27749

More improvements from the last change:

flutter_test_performance without_change_elapsed_time_ms:

From 2945 ms down to 2420 ms

flutter_test_performance6266 interface_change_elapsed_time_ms:

From 6800 ms down to 6200 ms

Even with an empty test directory it's slow

And yet sometimes it takes up to 10 seconds

My one test file with 7 empty tests is taking 5+ seconds to start running. I've used both flutter test and the Dart test runner in VS Code, and they both take about the same amount of time.

My whole test suite of ~150 tests takes over 30 seconds to complete, because each test file is taking 3-7 seconds just to start running.

launch.json (for Dart test runner)

    {
      "name": "Dart: Run all tests",
      "type": "dart",
      "request": "launch",
      "program": "./test/"
    },

Timed test run:

00:05 +1 -7: Some tests failed.
real    0m9.532s
user    0m0.075s
sys     0m0.138s

I am using nested group blocks, but I don't believe this should cause such an enormous delay that makes TDD extremely slow

Example section of code

void main() {
  group('AuthCubit:', () {

    late AuthCubit          authCubit;
    late MockLogIn          logIn;
    late MockLogOut         logOut;
    late MockRestoreSession restoreSession;

    setUp(() {
      logIn         = MockLogIn();
      logOut        = MockLogOut();
      restoreSession= MockRestoreSession();
      authCubit     = AuthCubit(
        logIn         : logIn,
        logOut        : logOut,
        restoreSession: restoreSession,
      );
    });

    tearDown(() {
      authCubit.close();
    });

    test('initial state is unauthenticated', () async {
      expect(authCubit.state, equals(const AuthStateUnauthenticated()));
    });
    group('logIn():', () {
      //? Create late variables that all stages use, if needed
      setUp(() {});
      //? Create subgroups for stages this function will step through
      group('On failure:', () {
        setUp(() {});
        test('emits unauthenticated state', () async { throw UnimplementedError(); });
      });
      group('On success:', () {
        setUp(() {});
        test('emits authenticated state', () async { throw UnimplementedError(); });
      });
    });
  });
  // ... + similar empty tests for other functions ...
}

I've tried even testing two empty example test files both separately and combined (see below code) both take 4+ seconds to run individually. If I combine both test files into one file, the execution time is the almost the same as just one test.

The problem seems to be around launching each test file, not the tests themselves.

Tests

test_test_1.dart

import 'package:flutter_test/flutter_test.dart';

//? Create Mocks, Fakes, etc, if needed

void main() {
  group('Test File 1:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
}

test_test_2.dart

import 'package:flutter_test/flutter_test.dart';

//? Create Mocks, Fakes, etc, if needed

void main() {
  group('Test File 2:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
}

test_test_1_2.dart

import 'package:flutter_test/flutter_test.dart';

//? Create Mocks, Fakes, etc, if needed

void main() {
  group('Test File 1:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
  group('Test File 2:', () {
    //? Create late variables that all functions use, if needed
    test('empty test 1', () async { throw UnimplementedError(); });
    test('empty test 2', () async { throw UnimplementedError(); });
    test('empty test 3', () async { throw UnimplementedError(); });
    test('empty test 4', () async { throw UnimplementedError(); });
    test('empty test 5', () async { throw UnimplementedError(); });
  });
}

Results

test_test_1.dart individual result

00:04 +0 -5: Some tests failed.

real    0m8.743s
user    0m0.060s
sys     0m0.167s

`test_test_2.dart` individual result

00:05 +0 -5: Some tests failed.

real    0m8.982s
user    0m0.137s
sys     0m0.106s

`test_test_1_2.dart` individual result

00:04 +0 -10: Some tests failed.

real    0m8.602s << Note this is actually FASTER than the smaller test files
user    0m0.045s << same ^
sys     0m0.200s

All 3 test file results (in one run)

00:08 +0 -20: Some tests failed.

real    0m12.696s
user    0m0.015s << Weirdly, this is the smallest value out of all test cases
sys     0m0.152s

Questions

What could be the issue with my code, setup (software or hardware), with Dart/Flutter, or anything else?

Pretend I know nothing about Flutter, Dart, or VS Code; what questions would help find the potential causes and solutions for this?

@SabianF

@SabianF how do you run your example files?
When I run from the terminal I get this:

$ flutter clean
[...]
$ time flutter test test/test_test_*
[...]
00:09 +0 -20: Some tests failed.                                                                                                                                                               

real    0m10.940s
user    0m15.826s
sys     0m3.289s

$ time flutter test test/test_test_*
00:02 +0 -20: Some tests failed.                                                                                                                                                               

real    0m3.340s
user    0m4.926s
sys     0m1.925s

On the first run (after I've cleaned) it has to compile the flutter framework etc again, so it takes longer. In this case 9-10 seconds to compile and run the tests.
On the subsequent run it can reuse the existing compile (it still has to read all the files though, to make sure things haven't changed) so it skips the compilation of the flutter framework etc and the test runs are thus much faster. 2-3 seconds in this case.
The number-examples you gave did not show this. I wonder if either caching is - for whatever reason - not in effect, or if caching is in effect (i.e. neither of the runs was "the first run") and it's just slow.

A number of questions that might help:

(again) How did you run your examples?
What are your numbers with/without a flutter clean first?
What CPU does your computer have?
How much CPU is used by other processes?
What version of flutter are you using?
What's your operating system?
What type of disk and/or how fast is the disk in your machine? (the one having flutter on it and the one with the project on it)
Whats the output from e.g. flutter test -v test/test_test_1.dart
If your on Linux (or possibly mac, although I'm not sure --- but give it a try) what's the output from DART_CFE_ENABLE_INCREMENTAL_COMPILER_BENCHMARKING=true flutter test -v test/test_test_1.dart

Another interesting thing, though, is that your reported user time is so low compared to mine. So also

what does /usr/bin/time -v flutter test test/test_test_1.dart print?

an also @mraleph might know more about the output for user and sys that I do...

@jensjoha

Details

Quoted reply by @jensjoha ("@SabianF how do you run your example files...")

@SabianF how do you run your example files?
When I run from the terminal I get this:
$ flutter clean
[...]
$ time flutter test test/test_test_*
[...]
00:09 +0 -20: Some tests failed.                                                                                                                                                               

real    0m10.940s
user    0m15.826s
sys     0m3.289s

$ time flutter test test/test_test_*
00:02 +0 -20: Some tests failed.                                                                                                                                                               

real    0m3.340s
user    0m4.926s
sys     0m1.925s
On the first run (after I've cleaned) it has to compile the flutter framework etc again, so it takes longer. In this case 9-10 seconds to compile and run the tests.
On the subsequent run it can reuse the existing compile (it still has to read all the files though, to make sure things haven't changed) so it skips the compilation of the flutter framework etc and the test runs are thus much faster. 2-3 seconds in this case.
The number-examples you gave did not show this. I wonder if either caching is - for whatever reason - not in effect, or if caching is in effect (i.e. neither of the runs was "the first run") and it's just slow.

A number of questions that might help:
(again) How did you run your examples?
What are your numbers with/without a flutter clean first?
What CPU does your computer have?
How much CPU is used by other processes?
What version of flutter are you using?
What's your operating system?
What type of disk and/or how fast is the disk in your machine? (the one having flutter on it and the one with the project on it)
Whats the output from e.g. flutter test -v test/test_test_1.dart
If your on Linux (or possibly mac, although I'm not sure --- but give it a try) what's the output from DART_CFE_ENABLE_INCREMENTAL_COMPILER_BENCHMARKING=true flutter test -v test/test_test_1.dart
Another interesting thing, though, is that your reported user time is so low compared to mine. So also
what does /usr/bin/time -v flutter test test/test_test_1.dart print?
an also @mraleph might know more about the output for user and sys that I do...

How did you run your examples?
- To be as specific as possible, I did them with
  - The Run button directly above the test/group blocks inside the test files
  - VS Code's Run with debugging and Run without debugging according to the launch.json I showed
  - time flutter test <dir>

What are your numbers with/without a flutter clean first?

Everything in my previous comment was with NO flutter clean
I just retested all files using flutter clean (the times are much longer)

Details

Commands

flutter clean && time flutter test test/test_tests/test_test_1_test.dart
flutter clean && time flutter test test/test_tests/test_test_2_test.dart
flutter clean && time flutter test test/test_tests/test_test_1_2_test.dart
flutter clean && time flutter test test/test_tests/*

Details

Results

# test_test_1
00:15 +0 -5: Some tests failed.

real    0m22.504s
user    0m0.030s
sys     0m0.121s

# test_test_2
00:18 +0 -5: Some tests failed.

real    0m25.694s
user    0m0.091s
sys     0m0.090s

# test_test_1_2
00:19 +0 -10: Some tests failed.

real    0m31.579s
user    0m0.045s
sys     0m0.213s

# all 3 files in one run
00:19 +0 -20: Some tests failed.

real    0m26.931s
user    0m0.030s
sys     0m0.152s

Also, after running flutter clean I ran the same tests again without flutter clean 3 consecutive times. The 2nd time was much faster (by ~75%), and the 3rd time was slightly faster than that (~11%), but the fastest time to run these empty tests was still over 4 seconds

What CPU does your computer have?
- i5 4590 (4-cores) @ 3.3GHz
How much CPU is used by other processes?
- 20% at most, usually less

What version of flutter are you using?

Flutter 3.0.1 • channel stable • https://github.com/flutter/flutter
Framework • revision fb57da5f94 (13 days ago) • 2022-05-19 15:50:29 -0700
Engine • revision caaafc5604
Tools • Dart 2.17.1 • DevTools 2.12.2

What's your operating system?
- Windows 10.0 build 19041 x64
What type of disk and/or how fast is the disk in your machine? (the one having flutter on it and the one with the project on it)
- Samsung 860 EVO 250GB Sata3 SSD

Whats the output from e.g. flutter test -v test/test_test_1.dart

Details

Verbose results

$ flutter test -v test/test_tests/test_test_1_test.dart 
[  +93 ms] executing: [C:\src\flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[ +571 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[   +2 ms] fb57da5f945d02ef4f98dfd9409a72b7cce74268
[        ] executing: [C:\src\flutter/] git tag --points-at fb57da5f945d02ef4f98dfd9409a72b7cce74268
[  +74 ms] Exit code 0 from: git tag --points-at fb57da5f945d02ef4f98dfd9409a72b7cce74268
[   +1 ms] 3.0.1
[  +16 ms] executing: [C:\src\flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +61 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[   +1 ms] origin/stable
[   +1 ms] executing: [C:\src\flutter/] git ls-remote --get-url origin
[ +111 ms] Exit code 0 from: git ls-remote --get-url origin
[   +1 ms] https://github.com/flutter/flutter
[ +228 ms] executing: [C:\src\flutter/] git rev-parse --abbrev-ref HEAD
[  +51 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[   +1 ms] stable
[ +157 ms] Found 1 files which will be executed as Widget Tests.
[  +54 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[  +12 ms] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[  +28 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'WindowsUwpEngineArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[   +7 ms] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[   +3 ms] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[   +2 ms] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[ +194 ms] Artifact Instance of 'MaterialFonts' is not required, skipping update.
[        ] Artifact Instance of 'GradleWrapper' is not required, skipping update.
[        ] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'FlutterSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'WindowsUwpEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[   +1 ms] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'PubDependencies' is not required, skipping update.
[ +128 ms] Skipping pub get: version match.
[ +238 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[  +54 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[  +13 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +8 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[  +29 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[  +11 ms] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[ +224 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[  +13 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[  +70 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +9 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[  +15 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[   +7 ms] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +9 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[  +23 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[  +11 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[ +295 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[   +5 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[   +2 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +2 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +8 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[   +6 ms] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\        
[  +46 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +5 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[  +20 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +3 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[   +8 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[   +2 ms] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +2 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[  +17 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[   +7 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[ +235 ms] Generating C:\src\Ugeddit\ugeddit_v2\android\app\src\main\java\io\flutter\plugins\GeneratedPluginRegistrant.java
[ +558 ms] running test package with arguments: [-r, compact, --timeout, 30s, --concurrency=2, --chain-stack-traces, --,
C:\src\Ugeddit\ugeddit_v2\test\test_tests\test_test_1_test.dart]
00:00 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_tests\test_test_1_test.dart                                                                                     [ +262 ms] test 0: starting test C:\src\Ugeddit\ugeddit_v2\test\test_tests\test_test_1_test.dart
[  +11 ms] Stopping scan for flutter_test_config.dart; found project root at C:\src\Ugeddit\ugeddit_v2
[   +5 ms] Compiler will use the following file as its incremental dill file: C:\Temp\flutter_tools.41da03de\flutter_test_compiler.9b840528\output.dill
[        ] Listening to compiler controller...
[  +18 ms] Compiling file:///C:/Temp/flutter_tools.41da03de/flutter_test_listener.410f33b3/listener.dart
[  +66 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[  +13 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[   +3 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +2 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +9 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[   +2 ms] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +49 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[  +13 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[  +20 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +4 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[   +5 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[   +3 ms] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +2 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +4 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[   +7 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[  +56 ms] C:\src\flutter\bin\cache\dart-sdk\bin\dart.exe --disable-dart-dev C:\src\flutter\bin\cache\dart-sdk\bin\snapshots\frontend_server.dart.snapshot --sdk-root
C:\src\flutter\bin\cache\artifacts\engine\common\flutter_patched_sdk/ --incremental --no-print-incremental-dependencies --target=flutter --debugger-module-names       
--experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=true --output-dill C:\Temp\flutter_tools.41da03de\flutter_test_compiler.9b840528\output.dill --packages   
C:\src\Ugeddit\ugeddit_v2\.dart_tool\package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --track-widget-creation
--initialize-from-dill C:\src\Ugeddit\ugeddit_v2\build\test_cache\build\c075001b96339384a97db4862b8ab8db.cache.dill.track.dill --source
C:\src\Ugeddit\ugeddit_v2\.dart_tool\flutter_build\dart_plugin_registrant.dart --source package:flutter/src/dart_plugin_registrant.dart
-Dflutter.dart_plugin_registrant=file:///C:/src/Ugeddit/ugeddit_v2/.dart_tool/flutter_build/dart_plugin_registrant.dart
[  +98 ms] <- compile file:///C:/Temp/flutter_tools.41da03de/flutter_test_listener.410f33b3/listener.dart
00:09 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_tests\test_test_1_test.dart                                                                                     [+8990 ms] <- accept
[   +1 ms] <- reset
[        ] Compiling file:///C:/Temp/flutter_tools.41da03de/flutter_test_listener.410f33b3/listener.dart took 9353ms
[   +1 ms] test 0: starting test device
[   +9 ms] test 0: awaiting connection to test device
[   +5 ms] test 0: Observatory uri is not available
[   +9 ms] test 0: test harness socket server is running at port:52014
[   +4 ms] Using this directory for fonts configuration: C:\Temp\flutter_tools.41da03de\flutter_test_fonts.bf76ecff
[   +3 ms] test 0: Starting flutter_tester process with command=[C:\src\flutter\bin\cache\artifacts\engine\windows-x64\flutter_tester.exe, --disable-observatory,      
--enable-checked-mode, --verify-entry-points, --enable-software-rendering, --skia-deterministic-rendering, --enable-dart-profiling, --non-interactive,
--use-test-fonts, --packages=C:\src\Ugeddit\ugeddit_v2\.dart_tool\package_config.json,
C:\Temp\flutter_tools.41da03de\flutter_test_listener.410f33b3\listener.dart.dill], environment={FLUTTER_TEST: true, FONTCONFIG_FILE:
C:\Temp\flutter_tools.41da03de\flutter_test_fonts.bf76ecff/fonts.conf, SERVER_PORT: 52014, APP_NAME: ugeddit_v2, UNIT_TEST_ASSETS:
C:\src\Ugeddit\ugeddit_v2\build\unit_test_assets}
[  +85 ms] test 0: Started flutter_tester process at pid 4352
00:10 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_tests\test_test_1_test.dart                                                                                     [ +878 ms] test 0: connected to test device, now awaiting test result
[   +2 ms] test 0: Waiting for test harness or tests to finish
00:10 +0 -1: empty test 1 [E]
  UnimplementedError
  test\test_tests\test_test_1_test.dart 7:35  main.<fn>
  test\test_tests\test_test_1_test.dart 7:24  main.<fn>

00:10 +0 -2: empty test 2 [E]
  UnimplementedError
  test\test_tests\test_test_1_test.dart 8:35  main.<fn>
  test\test_tests\test_test_1_test.dart 8:24  main.<fn>

00:10 +0 -3: empty test 3 [E]
  UnimplementedError
  test\test_tests\test_test_1_test.dart 9:35  main.<fn>
  test\test_tests\test_test_1_test.dart 9:24  main.<fn>

00:10 +0 -4: empty test 4 [E]
  UnimplementedError
  test\test_tests\test_test_1_test.dart 10:35  main.<fn>
  test\test_tests\test_test_1_test.dart 10:24  main.<fn>

00:11 +0 -5: empty test 5 [E]
  UnimplementedError
  test\test_tests\test_test_1_test.dart 11:35  main.<fn>
  test\test_tests\test_test_1_test.dart 11:24  main.<fn>

[ +608 ms] test 0: Test harness is no longer needed by test process
[  +10 ms] test 0: finished
[   +3 ms] test 0: cleaning up...
[  +20 ms] test 0: ensuring test device is terminated.
[   +8 ms] test 0: Terminating flutter_tester process
[   +3 ms] test 0: Shutting down test harness socket server
[  +29 ms] test 0: Test process is no longer needed by test harness
[  +23 ms] test 0: flutter_tester process at pid 4352 exited with code=-1
[   +1 ms] test 0: deleting temporary directory
[  +14 ms] test 0: finished
00:11 +0 -5: Some tests failed.
[  +34 ms] Deleting C:\Temp\flutter_tools.41da03de\flutter_test_compiler.9b840528...
[  +24 ms] killing pid 7352
[ +106 ms] Deleting C:\Temp\flutter_tools.41da03de\flutter_test_fonts.bf76ecff...
[  +10 ms] test package returned with exit code 1
[  +13 ms] "flutter test" took 14,051ms.
[  +20 ms] 
           #0      throwToolExit (package:flutter_tools/src/base/common.dart:10:3)
           #1      TestCommand.runCommand (package:flutter_tools/src/commands/test.dart:471:7)
           <asynchronous suspension>
           #2      FlutterCommand.run.<anonymous closure> (package:flutter_tools/src/runner/flutter_command.dart:1183:27)
           <asynchronous suspension>
           #3      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #4      CommandRunner.runCommand (package:args/command_runner.dart:209:13)
           <asynchronous suspension>
           #5      FlutterCommandRunner.runCommand.<anonymous closure> (package:flutter_tools/src/runner/flutter_command_runner.dart:281:9)
           <asynchronous suspension>
           #6      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #7      FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:229:5)
           <asynchronous suspension>
           #8      run.<anonymous closure>.<anonymous closure> (package:flutter_tools/runner.dart:62:9)
           <asynchronous suspension>
           #9      AppContext.run.<anonymous closure> (package:flutter_tools/src/base/context.dart:150:19)
           <asynchronous suspension>
           #10     main (package:flutter_tools/executable.dart:94:3)
           <asynchronous suspension>


[ +117 ms] ensureAnalyticsSent: 102ms
[   +4 ms] Running shutdown hooks
[   +3 ms] Shutdown hooks complete
[   +4 ms] exiting with code 1

If your on Linux (or possibly mac, although I'm not sure --- but give it a try) what's the output from DART_CFE_ENABLE_INCREMENTAL_COMPILER_BENCHMARKING=true flutter test -v test/test_test_1.dart
- Not using Linux or Mac

Thanks a ton for all the questions and detail, Joha.

What do you think of these answers? Any bottlenecks you see from my setup here, or is it an issue with VS Code, Flutter/Dart, etc?

Alright.

I'm not familiar with a Windows time command, but it probably explains why the user and sys is so different from mine.

I'm not sure exactly how tests are run when clicking the Run or Debug button on specific tests and/or groups in the IDE, so lets concentrate about the command line runs --- and those commands looks right to me, so far so good :)
That the times are much longer with flutter clean at least means that the caching is working. So no funny business there.
This naturally gives us a whole new set of numbers --- looking at the "real" numbers it takes from between 22 and 32 seconds on a clean build. Yikes.
Another interesting point here, though, is that test_test_1 and test_test_2 are the same test, i.e. really the time for compiling and running should be the same. Naturally we can't expect timings to be rock stable, but here we're talking 22.504s vs 25.694s (real, or 15s vs 18s as reported by flutter test). That's quite a bit of difference.
The same thing goes for test_test_1_2 which is included in the last run of all 3 tests, and yet the real time is 31.579s vs 26.931s (i.e. compiling and running more is more than 4.5 seconds faster).
This - I think - indicates that there's something interfering. Combined with Windows, antivirus software (and the like) comes to mind, but I surely can't say anything with certainty.
A i5 4590 (4-cores) @ 3.3GHz CPU looks to be pretty powerful (e.g. comparing to the one in my machine the "single threaded rating" is only 20% slower than mine (or mine is 25% faster) --- and with running only one tests having more cores probably doesn't matter much). I don't think this is the issue.
20% as reported by Windows, if I recall correctly, is - on a 4-core machine - almost an entire core being used for something else. In all likelihood it's not a single core being used though, but rather ~20% of all of them meaning that the new "single threaded rating" might again be ~20% slower. Said another way, something taking 3.340 seconds on my machine (running all the tests) could reasonably be expected to take 5.18 seconds on yours. Your timing said 12.696 seconds, so nowhere close.
Up-to-date, super!
Windows --- I've touched a bit on this above --- it probably indicates that you have some security software which could influence performance too.
The disk itself seems fast when I look it up.
This is interesting. Was this a clean build or not? (the self-reported 11 seconds before saying all tests failed doesn't seem to fit with any of the previous numbers --- if it's not a clean build this might indicate that printing the verbose output takes a very long time). Let me try to sum it up a bit:
Flutter launches git a few times before doing anything else. This takes a combined total of 1.2 seconds (!).
It then finds the test, does some Artifact Instance of (I have no idea what that is) and something with plugins (which it seems to do twice?). This takes a combined total of 1.75 seconds (!).
It then takes ~0.25 seconds generating some java file, and another ~0.5 seconds before it actually runs the test package. We're now at ~3.7 seconds
It does something with plugins again -- again seemingly the same ones (that's three times now) - this time it "only" takes ~0.2 seconds though. As a side note, for my runs (a new project with just the test files in it) I don't have any plugins.
We're at ~3.9 seconds --- now it launches the compilation process. This takes something like 100-150 ms. Let's say we're at 4 seconds before the compiler is asked to do anything.
It then takes almost 9.5 seconds compiling (well, sort of --- this step covers a lot of stuff --- for instance, if it starts from a previous compile it first has to read all the files to see if anything has changed etc.). This surely isn't great either. We're at ~13.5 seconds.
We're starting the test process (~100 ms) and running the tests (seemingly ~600 ms), does some cleanup (shuts down processes, deletes temporary files etc) (~300 ms). We're at ~15.5 seconds.
It says "flutter test" took 14,051ms.. This fits approximately with flutter test not spending the first ~1.2 seconds on git stuff (~300 ms is missing somewhere, I'm not sure where, but relative to the overall numbers it hardly matters).
I then sends some analytics, taking another ~100 ms.
On windows, try running set DART_CFE_ENABLE_INCREMENTAL_COMPILER_BENCHMARKING=true in the terminal first. Then run "as normal" (possibly with -v though).

On a higher level note I should probably add that for each file you have tests in, running tests are going to be slower because it has to do a compile, launch a new process etc. Similarly - as we saw - getting to the point where it actually starts doing something with the tests takes quite a while, so running flutter test specificfile.dart (at least for many specific files) isn't great.
As somewhat of a workaround - when you want to run all tests - you could try to create one test file like done in #86722 (comment) and only run that.

@jensjoha

@jensjoha I'm going to reply to the rest of your points when I can, but just wanted to let you know quickly that the literal very last thing you mentioned (about combining tests via imports) made my entire (real) test suite run seamlessly

time flutter test test/app_combined_test.dart
...
00:07 +144 -4: Some tests failed.

real    0m11.258s
user    0m0.075s
sys     0m0.121s

flutter clean && time flutter test test/app_combined_test.dart
...
00:22 +144 -4: Some tests failed.

real    0m33.431s
user    0m0.045s 
sys     0m0.228s

I'll review your other points when I have time, but thank you tons for sharing that. I'm using Clean Architecture, so it's simple to create "test repositories" and a "app all features" test, and run them using launch.json configs for more granular testing if needed.

You rock.

Edit: answering your points below (if there are any Qs in them):

👍
Yikes indeed. Even before posting my initial comment, I had already run the tests on a completely different PC (laptop) to compare, and the general time it takes is around the same monster of 25-30 seconds (maybe it would've helped if I mentioned that earlier, but now you know)
The 20% CPU usage is a non-issue. I know what's using it, and I agree with you that it's not the cause of such an enormous time difference
^
👍
While MS Security Essentials is irritating at times, I keep a close eye on CPU, memory, disk, network, etc usage every time I notice something off, and I haven't suspected anything from any particular software interfering, but I might've missed something
Yup

That one was NOT with flutter clean before the verbose command. Here is one WITH flutter clean

Verbose results with `flutter clean`

$ flutter clean && flutter test -v test/test_test_1_test.dart

[  +84 ms] executing: [C:\src\flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[ +581 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[   +1 ms] fb57da5f945d02ef4f98dfd9409a72b7cce74268
[        ] executing: [C:\src\flutter/] git tag --points-at fb57da5f945d02ef4f98dfd9409a72b7cce74268
[  +72 ms] Exit code 0 from: git tag --points-at fb57da5f945d02ef4f98dfd9409a72b7cce74268
[        ] 3.0.1
[  +12 ms] executing: [C:\src\flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +37 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[        ] origin/stable
[        ] executing: [C:\src\flutter/] git ls-remote --get-url origin
[  +32 ms] Exit code 0 from: git ls-remote --get-url origin
[        ] https://github.com/flutter/flutter
[ +123 ms] executing: [C:\src\flutter/] git rev-parse --abbrev-ref HEAD
[  +36 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[        ] stable
[  +80 ms] Found 1 files which will be executed as Widget Tests.
[  +10 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[   +5 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'WindowsUwpEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[  +77 ms] Artifact Instance of 'MaterialFonts' is not required, skipping update.
[        ] Artifact Instance of 'GradleWrapper' is not required, skipping update.
[        ] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'FlutterSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'WindowsUwpEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'PubDependencies' is not required, skipping update.
[  +55 ms] Running "flutter pub get" in ugeddit_v2...
[   +4 ms] executing: [C:\src\Ugeddit\ugeddit_v2/] C:\src\flutter\bin\cache\dart-sdk\bin\dart __deprecated_pub --verbose get --no-precompile
[  +69 ms] MSG : Resolving dependencies...
[  +11 ms] MSG :   async 2.8.2 (2.9.0 available)
[        ]     |   characters 1.2.0 (1.2.1 available)
[        ]     |   convert 3.0.1 (3.0.2 available)
[        ]     |   ffi 1.2.1 (2.0.0 available)
[        ]     |   material_color_utilities 0.1.4 (0.1.5 available)
[        ]     |   meta 1.7.0 (1.8.0 available)
[        ]     |   path 1.8.1 (1.8.2 available)
[        ]     |   path_provider_linux 2.1.6 (2.1.7 available)
[        ]     |   path_provider_windows 2.0.6 (2.1.0 available)
[        ]     |   provider 6.0.2 (6.0.3 available)
[        ]     |   source_span 1.8.2 (1.9.0 available)
[        ]     |   string_scanner 1.1.0 (1.1.1 available)
[        ]     |   win32 2.6.1 (2.7.0 available)
[        ]     |   xml 6.0.1 (6.1.0 available)
[   +1 ms] MSG : Got dependencies!
[        ] MSG : Logs written to C:\Users\Sabian\AppData\Local\Pub\Cache\log\pub_log.txt.
[  +51 ms] Running "flutter pub get" in ugeddit_v2... (completed in 6.2s)
[  +90 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[  +13 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[   +1 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +3 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +3 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[   +2 ms] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +29 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +2 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[   +9 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +2 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[   +2 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[  +11 ms] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +2 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +2 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[   +1 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[ +166 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[   +3 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[        ] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[        ] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[        ] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +10 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +1 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[   +4 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +1 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[        ] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[        ] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[        ] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +1 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[        ] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[  +77 ms] Generating C:\src\Ugeddit\ugeddit_v2\android\app\src\main\java\io\flutter\plugins\GeneratedPluginRegistrant.java
[ +335 ms] running test package with arguments: [--no-color, -r, compact, --timeout, 30s, --concurrency=2, --chain-stack-traces, --, C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart]

00:00 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 [ +240 ms] test 0: starting test C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart
[   +8 ms] Stopping scan for flutter_test_config.dart; found project root at C:\src\Ugeddit\ugeddit_v2
[   +3 ms] Compiler will use the following file as its incremental dill file: C:\Temp\flutter_tools.44cb397b\flutter_test_compiler.7d297c62\output.dill
[        ] Listening to compiler controller...
[  +13 ms] Compiling file:///C:/Temp/flutter_tools.44cb397b/flutter_test_listener.ac21d5f3/listener.dart
[  +33 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[   +4 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[   +1 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[        ] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +13 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +2 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[   +5 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +1 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[   +1 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[        ] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +9 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +2 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[   +1 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[  +43 ms] C:\src\flutter\bin\cache\dart-sdk\bin\dart.exe --disable-dart-dev C:\src\flutter\bin\cache\dart-sdk\bin\snapshots\frontend_server.dart.snapshot --sdk-root C:\src\flutter\bin\cache\artifacts\engine\common\flutter_patched_sdk/ --incremental --no-print-incremental-dependencies --target=flutter --debugger-module-names --experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=true --output-dill C:\Temp\flutter_tools.44cb397b\flutter_test_compiler.7d297c62\output.dill --packages C:\src\Ugeddit\ugeddit_v2\.dart_tool\package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --track-widget-creation --initialize-from-dill C:\src\Ugeddit\ugeddit_v2\build\test_cache\build\c075001b96339384a97db4862b8ab8db.cache.dill.track.dill --source C:\src\Ugeddit\ugeddit_v2\.dart_tool\flutter_build\dart_plugin_registrant.dart --source package:flutter/src/dart_plugin_registrant.dart -Dflutter.dart_plugin_registrant=file:///C:/src/Ugeddit/ugeddit_v2/.dart_tool/flutter_build/dart_plugin_registrant.dart
[  +10 ms] <- compile file:///C:/Temp/flutter_tools.44cb397b/flutter_test_listener.ac21d5f3/listener.dart

00:01 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:02 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:03 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:04 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:05 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:06 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:07 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:08 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:09 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:10 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:11 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:12 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:13 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 [+13770 ms] <- accept
[        ] <- reset
[        ] Compiling file:///C:/Temp/flutter_tools.44cb397b/flutter_test_listener.ac21d5f3/listener.dart took 13904ms
[        ] test 0: starting test device
[   +4 ms] test 0: awaiting connection to test device
[   +1 ms] test 0: Observatory uri is not available
[   +5 ms] test 0: test harness socket server is running at port:59197
[   +1 ms] Using this directory for fonts configuration: C:\Temp\flutter_tools.44cb397b\flutter_test_fonts.3ac46c6a
[   +1 ms] test 0: Starting flutter_tester process with command=[C:\src\flutter\bin\cache\artifacts\engine\windows-x64\flutter_tester.exe, --disable-observatory, --enable-checked-mode, --verify-entry-points, --enable-software-rendering, --skia-deterministic-rendering, --enable-dart-profiling, --non-interactive, --use-test-fonts, --packages=C:\src\Ugeddit\ugeddit_v2\.dart_tool\package_config.json, C:\Temp\flutter_tools.44cb397b\flutter_test_listener.ac21d5f3\listener.dart.dill], environment={FLUTTER_TEST: true, FONTCONFIG_FILE: C:\Temp\flutter_tools.44cb397b\flutter_test_fonts.3ac46c6a/fonts.conf, SERVER_PORT: 59197, APP_NAME: ugeddit_v2, UNIT_TEST_ASSETS: C:\src\Ugeddit\ugeddit_v2\build\unit_test_assets}

00:14 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 [   +4 ms] test 0: Started flutter_tester process at pid 3204
[ +792 ms] test 0: connected to test device, now awaiting test result
[        ] test 0: Waiting for test harness or tests to finish

00:14 +0: empty test 1                                                                                                                                                                                 
00:14 +0 -1: empty test 1 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 5:35  main.<fn>
  test\test_test_1_test.dart 5:24  main.<fn>
  

00:14 +0 -1: empty test 2                                                                                                                                                                              
00:14 +0 -2: empty test 2 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 6:35  main.<fn>
  test\test_test_1_test.dart 6:24  main.<fn>
  

00:14 +0 -2: empty test 3                                                                                                                                                                              
00:14 +0 -3: empty test 3 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 7:35  main.<fn>
  test\test_test_1_test.dart 7:24  main.<fn>
  

00:14 +0 -3: empty test 4                                                                                                                                                                              
00:15 +0 -4: empty test 4                                                                                                                                                                              
00:15 +0 -4: empty test 4 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 8:35  main.<fn>
  test\test_test_1_test.dart 8:24  main.<fn>
  

00:15 +0 -4: empty test 5                                                                                                                                                                              
00:15 +0 -5: empty test 5 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 9:35  main.<fn>
  test\test_test_1_test.dart 9:24  main.<fn>
  
[ +276 ms] test 0: Test harness is no longer needed by test process
[        ] test 0: finished
[   +1 ms] test 0: cleaning up...
[        ] test 0: ensuring test device is terminated.
[        ] test 0: Terminating flutter_tester process
[        ] test 0: Shutting down test harness socket server
[        ] test 0: Test process is no longer needed by test harness
[  +30 ms] test 0: flutter_tester process at pid 3204 exited with code=-1
[        ] test 0: deleting temporary directory
[   +4 ms] test 0: finished

00:15 +0 -5: Some tests failed.                                                                                                                                                                        
[  +18 ms] Deleting C:\Temp\flutter_tools.44cb397b\flutter_test_compiler.7d297c62...
[   +8 ms] killing pid 17328
[ +143 ms] Deleting C:\Temp\flutter_tools.44cb397b\flutter_test_fonts.3ac46c6a...
[   +5 ms] test package returned with exit code 1
[   +2 ms] "flutter test" took 22,655ms.
[ +112 ms] ensureAnalyticsSent: 100ms
[   +1 ms] Running shutdown hooks
[        ] Shutdown hooks complete
[        ] exiting with code 1

Oh, doing set DART_CFE_ENABLE_INCREMENTAL_COMPILER_BENCHMARKING=true appears to have made it slightly faster

$ time flutter test test/test_test_1_test.dart

00:03 +0 -5: Some tests failed.

real    0m6.757s
user    0m0.045s
sys     0m0.091s

Verbose output

$ flutter test -v test/test_test_1_test.dart

[  +80 ms] executing: [C:\src\flutter/] git -c log.showSignature=false log -n 1 --pretty=format:%H
[ +597 ms] Exit code 0 from: git -c log.showSignature=false log -n 1 --pretty=format:%H
[   +1 ms] fb57da5f945d02ef4f98dfd9409a72b7cce74268
[        ] executing: [C:\src\flutter/] git tag --points-at fb57da5f945d02ef4f98dfd9409a72b7cce74268
[  +63 ms] Exit code 0 from: git tag --points-at fb57da5f945d02ef4f98dfd9409a72b7cce74268
[        ] 3.0.1
[   +9 ms] executing: [C:\src\flutter/] git rev-parse --abbrev-ref --symbolic @{u}
[  +36 ms] Exit code 0 from: git rev-parse --abbrev-ref --symbolic @{u}
[        ] origin/stable
[        ] executing: [C:\src\flutter/] git ls-remote --get-url origin
[  +38 ms] Exit code 0 from: git ls-remote --get-url origin
[        ] https://github.com/flutter/flutter
[ +134 ms] executing: [C:\src\flutter/] git rev-parse --abbrev-ref HEAD
[  +40 ms] Exit code 0 from: git rev-parse --abbrev-ref HEAD
[        ] stable
[  +76 ms] Found 1 files which will be executed as Widget Tests.
[  +15 ms] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[   +5 ms] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'WindowsUwpEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[  +71 ms] Artifact Instance of 'MaterialFonts' is not required, skipping update.
[        ] Artifact Instance of 'GradleWrapper' is not required, skipping update.
[        ] Artifact Instance of 'AndroidGenSnapshotArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'AndroidInternalBuildArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterWebSdk' is not required, skipping update.
[        ] Artifact Instance of 'FlutterSdk' is not required, skipping update.
[        ] Artifact Instance of 'WindowsEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'WindowsUwpEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxEngineArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'LinuxFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'MacOSFuchsiaSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerSDKArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FlutterRunnerDebugSymbols' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'IosUsbArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'FontSubsetArtifacts' is not required, skipping update.
[        ] Artifact Instance of 'PubDependencies' is not required, skipping update.
[  +57 ms] Skipping pub get: version match.
[  +51 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[  +10 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[   +1 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +2 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[   +1 ms] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +21 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +2 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[  +10 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +1 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[   +8 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[   +1 ms] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +1 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +2 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[   +1 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[ +103 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[   +3 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[   +1 ms] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[   +1 ms] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +11 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +1 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[   +6 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[   +1 ms] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[        ] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[   +1 ms] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[   +1 ms] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +1 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[   +1 ms] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[  +79 ms] Generating C:\src\Ugeddit\ugeddit_v2\android\app\src\main\java\io\flutter\plugins\GeneratedPluginRegistrant.java
[ +238 ms] running test package with arguments: [--no-color, -r, compact, --timeout, 30s, --concurrency=2, --chain-stack-traces, --, C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart]

00:00 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 [ +189 ms] test 0: starting test C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart
[   +8 ms] Stopping scan for flutter_test_config.dart; found project root at C:\src\Ugeddit\ugeddit_v2
[   +3 ms] Compiler will use the following file as its incremental dill file: C:\Temp\flutter_tools.99f2dc17\flutter_test_compiler.6889da96\output.dill
[        ] Listening to compiler controller...
[  +11 ms] Compiling file:///C:/Temp/flutter_tools.99f2dc17/flutter_test_listener.93021bea/listener.dart
[  +22 ms] Found plugin flutter_appauth at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_appauth-4.0.0\
[   +3 ms] Found plugin flutter_secure_storage at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage-5.0.2\
[        ] Found plugin flutter_secure_storage_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_linux-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_macos-1.1.0\
[   +1 ms] Found plugin flutter_secure_storage_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_web-1.0.2\
[        ] Found plugin flutter_secure_storage_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\flutter_secure_storage_windows-1.1.2\
[  +10 ms] Found plugin path_provider_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_linux-2.1.6\
[   +1 ms] Found plugin path_provider_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\path_provider_windows-2.0.6\
[   +4 ms] Found plugin shared_preferences at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences-2.0.15\
[        ] Found plugin shared_preferences_android at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_android-2.0.12\
[   +1 ms] Found plugin shared_preferences_ios at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_ios-2.1.1\
[        ] Found plugin shared_preferences_linux at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_linux-2.1.1\
[        ] Found plugin shared_preferences_macos at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_macos-2.0.4\
[   +1 ms] Found plugin shared_preferences_web at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_web-2.0.4\
[        ] Found plugin shared_preferences_windows at C:\Users\Sabian\AppData\Local\Pub\Cache\hosted\pub.dartlang.org\shared_preferences_windows-2.1.1\
[  +26 ms] C:\src\flutter\bin\cache\dart-sdk\bin\dart.exe --disable-dart-dev C:\src\flutter\bin\cache\dart-sdk\bin\snapshots\frontend_server.dart.snapshot --sdk-root C:\src\flutter\bin\cache\artifacts\engine\common\flutter_patched_sdk/ --incremental --no-print-incremental-dependencies --target=flutter --debugger-module-names --experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=true --output-dill C:\Temp\flutter_tools.99f2dc17\flutter_test_compiler.6889da96\output.dill --packages C:\src\Ugeddit\ugeddit_v2\.dart_tool\package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts --track-widget-creation --initialize-from-dill C:\src\Ugeddit\ugeddit_v2\build\test_cache\build\c075001b96339384a97db4862b8ab8db.cache.dill.track.dill --source C:\src\Ugeddit\ugeddit_v2\.dart_tool\flutter_build\dart_plugin_registrant.dart --source package:flutter/src/dart_plugin_registrant.dart -Dflutter.dart_plugin_registrant=file:///C:/src/Ugeddit/ugeddit_v2/.dart_tool/flutter_build/dart_plugin_registrant.dart
[  +43 ms] <- compile file:///C:/Temp/flutter_tools.99f2dc17/flutter_test_listener.93021bea/listener.dart

00:01 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 
00:02 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 [+2111 ms] <- accept
[        ] <- reset
[        ] Compiling file:///C:/Temp/flutter_tools.99f2dc17/flutter_test_listener.93021bea/listener.dart took 2233ms
[        ] test 0: starting test device
[   +4 ms] test 0: awaiting connection to test device
[   +1 ms] test 0: Observatory uri is not available
[   +4 ms] test 0: test harness socket server is running at port:61867
[   +1 ms] Using this directory for fonts configuration: C:\Temp\flutter_tools.99f2dc17\flutter_test_fonts.2ac5ee52
[   +1 ms] test 0: Starting flutter_tester process with command=[C:\src\flutter\bin\cache\artifacts\engine\windows-x64\flutter_tester.exe, --disable-observatory, --enable-checked-mode, --verify-entry-points, --enable-software-rendering, --skia-deterministic-rendering, --enable-dart-profiling, --non-interactive, --use-test-fonts, --packages=C:\src\Ugeddit\ugeddit_v2\.dart_tool\package_config.json, C:\Temp\flutter_tools.99f2dc17\flutter_test_listener.93021bea\listener.dart.dill], environment={FLUTTER_TEST: true, FONTCONFIG_FILE: C:\Temp\flutter_tools.99f2dc17\flutter_test_fonts.2ac5ee52/fonts.conf, SERVER_PORT: 61867, APP_NAME: ugeddit_v2, UNIT_TEST_ASSETS: C:\src\Ugeddit\ugeddit_v2\build\unit_test_assets}
[   +3 ms] test 0: Started flutter_tester process at pid 6184

00:03 +0: loading C:\src\Ugeddit\ugeddit_v2\test\test_test_1_test.dart                                                                                                                                 [ +771 ms] test 0: connected to test device, now awaiting test result
[        ] test 0: Waiting for test harness or tests to finish

00:03 +0: empty test 1                                                                                                                                                                                 
00:03 +0 -1: empty test 1 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 5:35  main.<fn>
  test\test_test_1_test.dart 5:24  main.<fn>
  

00:03 +0 -1: empty test 2                                                                                                                                                                              
00:03 +0 -2: empty test 2 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 6:35  main.<fn>
  test\test_test_1_test.dart 6:24  main.<fn>
  

00:03 +0 -2: empty test 3                                                                                                                                                                              
00:03 +0 -3: empty test 3 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 7:35  main.<fn>
  test\test_test_1_test.dart 7:24  main.<fn>
  

00:03 +0 -3: empty test 4                                                                                                                                                                              
00:03 +0 -4: empty test 4 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 8:35  main.<fn>
  test\test_test_1_test.dart 8:24  main.<fn>
  

00:03 +0 -4: empty test 5                                                                                                                                                                              
00:03 +0 -5: empty test 5 [E]                                                                                                                                                                          
  UnimplementedError
  test\test_test_1_test.dart 9:35  main.<fn>
  test\test_test_1_test.dart 9:24  main.<fn>
  
[ +234 ms] test 0: Test harness is no longer needed by test process
[        ] test 0: finished
[   +1 ms] test 0: cleaning up...
[   +1 ms] test 0: ensuring test device is terminated.
[        ] test 0: Terminating flutter_tester process
[   +1 ms] test 0: Shutting down test harness socket server
[        ] test 0: Test process is no longer needed by test harness
[  +43 ms] test 0: flutter_tester process at pid 6184 exited with code=-1
[        ] test 0: deleting temporary directory
[   +5 ms] test 0: finished

00:03 +0 -5: Some tests failed.                                                                                                                                                                        
[  +12 ms] Deleting C:\Temp\flutter_tools.99f2dc17\flutter_test_compiler.6889da96...
[   +5 ms] killing pid 2252
[  +77 ms] Deleting C:\Temp\flutter_tools.99f2dc17\flutter_test_fonts.2ac5ee52...
[   +4 ms] test package returned with exit code 1
[  +10 ms] "flutter test" took 4,389ms.
[ +144 ms] ensureAnalyticsSent: 137ms
[   +1 ms] Running shutdown hooks
[        ] Shutdown hooks complete
[   +1 ms] exiting with code 1

After doing what you mentioned setting the incremental benchmarking, my whole combined test suite (that I also created thanks to you) is even faster!

$ flutter clean && time flutter test test/app_combined_test.dart 

00:20 +144 -4: Some tests failed.

real    0m27.944s
user    0m0.045s
sys     0m0.122s

2nd run

$ time flutter test test/app_combined_test.dart

00:06 +144 -4: Some tests failed.

real    0m9.740s
user    0m0.015s
sys     0m0.106s

Now this looks much more reasonable for testing. I was afraid a moderately-sized app would start taking minutes for every incremental TDD change, but you just helped a crap ton and saved ~75% (24 seconds) for every run of these tests.

I say again, you rock, I hope my results help in some way, and I'll be excited for Flutter's watch mode ("hot reload" for tests) they're working on.

Edit: if there's a way to further reduce the 3-4 second test startup time to the OP's 100 millisecond timings, I would love to know some ways that may do this, if you have more ideas? :)

I'm glad combining the tests seems to be a workaround.

With regard to 8) it's funny --- that seems to take ~exactly as long as without verbose. I wonder what happened with the previous verbose run which was slower than the non-verbose one.

As for 9), setting that won't make it faster. That it appears faster is coincidence. In fact, it appears not to have done the wanted thing at all. It was meant to enable printing extra timings from the compilation, but it's not in the output. I'm not quite sure why.

As for the OP's 100 ms startup that's surely not going to happen anytime soon (if at all --- not directly anyway). For instance we've seen how git runs when starting the flutter command on your machine takes ~1 second (1000 ms).
(that being said, having a watcher of sorts with a compiler already present and initialized etc we may conceivably get a lot closer).

@jensjoha

Details

Quoted reply from @jensjoha

I'm glad combining the tests seems to be a workaround.

With regard to 8) it's funny --- that seems to take ~exactly as long as without verbose. I wonder what happened with the previous verbose run which was slower than the non-verbose one.

As for 9), setting that won't make it faster. That it appears faster is coincidence. In fact, it appears not to have done the wanted thing at all. It was meant to enable printing extra timings from the compilation, but it's not in the output. I'm not quite sure why.

As for the OP's 100 ms startup that's surely not going to happen anytime soon (if at all --- not directly anyway). For instance we've seen how git runs when starting the flutter command on your machine takes ~1 second (1000 ms). (that being said, having a watcher of sorts with a compiler already present and initialized etc we may conceivably get a lot closer).

For (8), I'm thinking you mean it seems to have taken exactly as long without flutter clean (not verbose)? But either way, no idea.

For (9) is there a distinct place or command I can check to verify the setting's current state?

We'll see 👍. Thanks again, Joha!

I'm not sure exactly how tests are run when clicking the Run or Debug button on specific tests and/or groups in the IDE

FWIW there are some instances where in VS Code I would expect running tests to be slower:

When running tests from multiple files with debugging active
I don't know the specific reason here, but flutter test --machine --start-paused reports "When using --start-paused, you must specify a single test file to run" so VS Code runs each test individually (we need --start-paused in order to set breakpoints before code starts running, to avoid races)
When running a selection of tests across multiple files
When running a subset of tests (eg. a selection from the test tree) that spans multiple files, we have to run each file individually to pass the names of those tests only to that particular file

If there are other cases that are significantly slower when running from VS Code (for example just running all tests without debugging I would expect to always be nearly as fast as the CLI) then please file issues at https://github.com/Dart-Code/Dart-Code with repros so I can take a look.

Hello folks!
I'm also trying to understand possible improvements on how long it takes for a flutter test to run.

Here, a single call to flutter test path-to-test-file.dart is taking ~10s.

One thing that caught my attention is that is seems like the flutter tester is being re-created and re-compiled every time I run the flutter test command.

For example, when I run with the --verbose flag, one part of the output is:

...
/Users/philippe.ehlert/sdk-flutter/bin/cache/artifacts/engine/common/flutter_patched_sdk/ --incremental --no-print-incremental-dependencies --target=flutter --debugger-module-names --experimental-emit-debug-metadata -DFLUTTER_WEB_AUTO_DETECT=true --output-dill
/var/folders/1n/vlfv1d455490yst2p7myk5w40000gn/T/flutter_tools.Yrc4td/flutter_test_compiler.D07Dco/output.dill --packages /Users/philippe.ehlert/dev/project/test/flutter/.dart_tool/package_config.json -Ddart.vm.profile=false -Ddart.vm.product=false --enable-asserts
--track-widget-creation --initialize-from-dill /Users/philippe.ehlert/dev/project/test/flutter/build/test_cache/build/c075001b96339384a97db4862b8ab8db.cache.dill.track.dill --source
/Users/philippe.ehlert/dev/project/test/flutter/.dart_tool/flutter_build/dart_plugin_registrant.dart --source package:flutter/src/dart_plugin_registrant.dart
-Dflutter.dart_plugin_registrant=file:///Users/philippe.ehlert/dev/project/test/flutter/.dart_tool/flutter_build/dart_plugin_registrant.dart
[  +30 ms] <- compile file:///var/folders/1n/vlfv1d455490yst2p7myk5w40000gn/T/flutter_tools.Yrc4td/flutter_test_listener.kqZoOi/listener.dart
[+3846 ms] ../../../../../.pub-cache/git/flare-flutter-7870c1a8d2acb25dca4b7dbbbef0b883e1535588/flare_flutter/lib/flare_render_box.dart:167:26: Warning: Operand of null-aware operation '?.' has type 'SchedulerBinding' which excludes null.
...

This last step (compile file:///var/folders....) consistently takes ~4s to run.

Every time I run flutter test, it creates a new folder inside /var/folders/1n/vlfv1d455490yst2p7myk5w40000gn/T/.
In this example, the folder name is flutter_tools.Yrc4td. If I ls this directory there are multiple folders called flutter_tools.a_hash.

Is there a way to reuse the same files that were built previously instead of creating new ones every time I run a test?

There have been various improvements here over the years, but if you're encountering issues with flutter test unrelated to the initial snapshot generation on first run, please file another issue. I'm going to close this issue as the original problem has been resolved.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

A cold-start "flutter test" takes multiple seconds #17130

16 remaining items

Tests

Results

Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A cold-start "flutter test" takes multiple seconds #17130

Description

Activity

zoechi commented on Jan 18, 2019

jonahwilliams commented on Feb 10, 2019

jonahwilliams commented on Feb 10, 2019

jonahwilliams commented on Feb 10, 2019

jonahwilliams commented on Feb 10, 2019

jonahwilliams commented on Feb 11, 2019

aam commented on Feb 11, 2019

jonahwilliams commented on Feb 11, 2019

jonahwilliams commented on Mar 19, 2019

jonahwilliams commented on Mar 20, 2019

gsouf commented on Mar 23, 2020

16 remaining items

SabianF commented on May 29, 2022

Tests

Results

Questions

jensjoha commented on Jun 1, 2022

SabianF commented on Jun 1, 2022

jensjoha commented on Jun 2, 2022

SabianF commented on Jun 2, 2022

jensjoha commented on Jun 3, 2022

SabianF commented on Jun 3, 2022

DanTup commented on Jun 6, 2022

philippeehlert commented on Dec 28, 2022

bkonyi commented on Jun 27, 2025

github-actions commented on Jul 11, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions