Integration test TimeoutException after 0:12:00.000000: Test timed out after 12 minutes

@ductranit

Description

I have the exact same issue as @ductranit in #103064. I'm getting

12:00 +0 -1: loading /Users/runner/work/sharezone-app/sharezone-app/app/integration_test/app_test.dart [E]                                                                                             
  TimeoutException after 0:12:00.000000: Test timed out after 12 minutes.
  package:test_api/src/backend/invoker.dart 333:28  Invoker._handleError.<fn>
  dart:async/zone.dart 1418:47                      _rootRun
  dart:async/zone.dart 1328:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 331:10  Invoker._handleError
  package:test_api/src/backend/invoker.dart 287:9   Invoker.heartbeat.<fn>.<fn>
  dart:async/zone.dart 1426:13                      _rootRun
  dart:async/zone.dart 1328:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 286:38  Invoker.heartbeat.<fn>
  dart:async-patch/timer_patch.dart 18:15           Timer._createTimer.<fn>
  dart:isolate-patch/timer_impl.dart 395:19         _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 426:5          _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 192:12      _RawReceivePortImpl._handleMessage

https://github.com/SharezoneApp/sharezone-app/runs/6866573231?check_suite_focus=true

I'm using the flutter test command. Building my app takes ~20 minutes. But the flutter test command times out after 12 minutes. So even before finish building my app, the command times out.

Using the timeout argument (flutter test --timeout none) changes nothing.

Workaround

1) Use `flutter drive`

My workaround is to use the flutter drive command:

flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/app_test.dart

The flutter drive command has no timeout. Therefore, the app finishes after ~20 minutes of building and all tests can pass.

Using v2.10.5 of Flutter.

2) Use `flutter build` and then `flutter test`

Another workaround is to run flutter build before running flutter test. Therefore, the build for flutter test should hopefully be faster because of the caching.

Edit: I have reproducible repository. It uses Flutter v3.3.4 and has cloud_firestore as a dependency to increase the build duration: https://github.com/nilsreichardt/playground/tree/integration-test-timeout

Running flutter test integration_test --timeout none on a slow machine (like CI) will fail after 12 minutes: https://github.com/nilsreichardt/playground/actions/runs/3218752730

flutter test integration_test --timeout none: https://github.com/nilsreichardt/playground/actions/runs/3218752730
flutter test integration_test --timeout 3600s: https://github.com/nilsreichardt/playground/actions/runs/3218848669
Use timeout parameter the testWidgets method: https://github.com/nilsreichardt/playground/actions/runs/3218841600
Use @Timeout(Duration(hours: 1)): https://github.com/nilsreichardt/playground/actions/runs/3218844853

Steps to reproduce (we are using Firestore to increase the build duration):

Run these commands

flutter create integration_test_timeout
cd integration_test_timeout
flutter pub add cloud_firestore
flutter pub add firebase_core

Adjust minimum os version, for macOS: Set platform :osx, '10.12' in macos/Podfile
Add a basic integration

This is working as intended, see https://github.com/dart-lang/test/tree/master/pkgs/test#timeouts about configuring timeouts in your test.

@christopherfujino

@christopherfujino Thanks your quick reply. But even with @Timeout(Duration(minutes: 60)) in the app_test.dart (there are my integration tests located) times out the flutter test command after 12 minutes.

https://github.com/SharezoneApp/sharezone-app/blob/dc6b31e65d77908b0878cbcafa9c77908f1273cf/app/integration_test/app_test.dart#L9

Can we reopen this issue?

Hmm, I'm guessing you're referring to https://github.com/SharezoneApp/sharezone-app/runs/6878674902?check_suite_focus=true ? I'm not sure what's happening here....

Hmm, I'm guessing you're referring to SharezoneApp/sharezone-app/runs/6878674902?check_suite_focus=true ?

Correct 👍

Anything I can do to provide more information?

I'm having the same issue, neither the --timeout command line option for flutter test, the annotation nor the timeout parameter for testWidgets are working.

`@Timeout(Duration(minutes: 20))
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

void main() {
final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized();

testWidgets("failing test example", (WidgetTester tester) async {
expect(2 + 2, equals(5));
}, timeout: Timeout(Duration(minutes: 20)));
}`

I have added to reproduce it in the ticket description 👍

We got the same issue after changing the target to API 33 - Android 13. The app is not installed automatically anymore on Android 13 devices

Same here, cmd arguments is not taken in consideration

We got the same issue after changing the target to API 33 - Android 13. The app is not installed automatically anymore on Android 13 devices

Hello! Did you manage to find a workaround for this issue?

#99246 (comment) is maybe related

@christopherfujino

@christopherfujino Sorry to tag you, and I'm not a fan of making pressure, but we suffer from this problem a lot (we can't use the workaround anymore). I really think this issue should get more attention. It has almost 40 upvotes, has been referenced many times, has been open for more than 2.5 years and even Google is suffering from it (see google/webcrypto.dart#119).

@nilsreichardt

@nilsreichardt I no longer work on this, so please do not tag me further.

@christopherfujino

@christopherfujino Sorry for an additional tag after your above comment but, as you are a member of the flutter org, could you at least please tag someone who might be able to further the conversation and status of this issue?

I found a semi fix for this issue and opened a PR: #164437.

A screenshot of running flutter test integration_test --ignore-timeouts which surpasses the previous timeout of 12 minutes.

The root cause of this issue is in the Dart test_api package which is used by the dart test command. In the dart test command is a separate timeout for loading the test suite (see https://github.com/dart-lang/test/blob/db8cf091506a67eba1e523215e0e49a0db7cd1fd/pkgs/test_core/lib/src/runner/load_suite.dart#L23-L29) which times out when the compiling takes too long. The dart test command has --ignore-timeouts flag which bypasses this timeout. The PR just adds this flag also to the flutter test command and passes it to the dart test command.

The downside of this fix it that your test will never time out (e.g. you have a failed test because a button isn't visible). Therefore, you should always set timeouts for the host machine, e.g. using timeout-minutes in GitHub Actions (https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes).

I also opened a ticket to make this load test suite timeout configurable so that we can implement a proper fix in the future: dart-lang/test#2463

Add `--ignore-timeouts` flag for `flutter test` command (#164437)

The PR #164437 has been merged and now this issue has a proper workaround 🎉

You can add the --ignore-timeouts flag to your test command to ignore the timeout during build times.

flutter test integration_test --ignore-timeouts

However, using this flag has a side effect: it will ignore all other timeouts in a test. This means that if an integration test would fail because a button isn't displayed, that test will run forever because there is no timeout to look for that button.

Therefore, when using this flag in CI, I would recommend always adding a timeout for the host machine, e.g. using "timeout-minutes" in the case of GitHub actions:

jobs:
  integration-test:
    runs-on: macos-latest
    timeout-minutes: 60 # <----- Add this timeout to your workflow
    steps:
    - uses: actions/checkout@v2
    - name: Install Flutter
      uses: subosito/flutter-action@v1
      with:
        channel: master
    - name: Run Integration Test
      working-directory: integration_test_timeout 
      run: flutter test integration_test -d macos --ignore-timeouts

Proof that --ignore-timeouts fixes this problem: https://github.com/nilsreichardt/playground/actions/runs/14207274819/job/39807662668

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Add `--ignore-timeouts` flag for `flutter test` command (flutter#164437)

Navigation Menu

Repository navigation

Integration test TimeoutException after 0:12:00.000000: Test timed out after 12 minutes #105913

Description

Workaround

1) Use `flutter drive`

2) Use `flutter build` and then `flutter test`

31 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Integration test TimeoutException after 0:12:00.000000: Test timed out after 12 minutes #105913

Description

Description

Workaround

1) Use flutter drive

2) Use flutter build and then flutter test

Activity

christopherfujino commented on Jun 13, 2022

nilsreichardt commented on Jun 14, 2022

nilsreichardt commented on Jun 16, 2022

christopherfujino commented on Jun 16, 2022

nilsreichardt commented on Jun 16, 2022

cladelpino commented on Jul 18, 2022

himeshp commented on Sep 15, 2022

nilsreichardt commented on Oct 10, 2022

lhimo commented on Nov 2, 2022

tomas-carv-com commented on Nov 15, 2022

31 remaining items

MortadhaFadhlaoui commented on Nov 14, 2023

vodemn commented on Mar 5, 2024

vodemn commented on Mar 12, 2024

nilsreichardt commented on Jan 4, 2025

christopherfujino commented on Jan 6, 2025

atmask commented on Jan 15, 2025

nilsreichardt commented on Mar 2, 2025

nilsreichardt commented on Apr 1, 2025

github-actions commented on Apr 16, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1) Use `flutter drive`

2) Use `flutter build` and then `flutter test`