Skip to content

Timeouts are not ignored when debugging tests inside Pub Workspaces #5620

Closed
Closed
Timeouts are not ignored when debugging tests inside Pub Workspaces#5620
Labels
in testingRelates to test execution of Dart/Flutter tests for end usersis bug
Milestone
v3.122.0

Description

@FMorschel
FMorschel
opened on Jul 30, 2025

Describe the bug
When running SDK tests, I constantly see test timeout exceptions when debugging them.

To Reproduce
Steps to reproduce the behavior:

  1. Go to any test_reflective_loader test like pkg/analysis_server/test/src/services/correction/fix/remove_async_test.dart
  2. Add:
@Timeout(Duration(seconds: 5))
library;
import 'package:test/test.dart';

//...

@reflectiveTest
class RemoveAsyncIllegalReturnTest extends FixProcessorTest {
  @override
  FixKind get kind => DartFixKind.REMOVE_ASYNC;

  @soloTest
  void test_a() {
    print(''); // B1
    print(''); // B2
  }

  @soloTest
  Future<void> test_b() async {
    print(''); // B3
    await Future.delayed(Duration.zero);
    print(''); // B4
  }
//...
  1. Run the test and wait more than five seconds between B1 and B2, see no problems
  2. Run the test and wait more than five seconds between B3 and B4, see timeout

Expected behavior
Test timeout should be disabled when the debugger is running.

Please complete the following information:

Workspace Environment 
Dart Code extension: 3.115.20250728
Flutter extension: 3.115.20250701 (activated)

App: Visual Studio Code
App Host: desktop
Version: win 1.102.3

Workspace type: Dart (LSP) (Multiroot)
Workspace name: sdk (Workspace)

Dart (3.10.0-edge.2bd721707305288f80bcaeec1834cdfcb71885b7): C:\Users\felip_0vh5fa6\dart
Flutter (3.32.7): C:\Users\felip_0vh5fa6\.puro\envs\stable\flutter (No device)
Output from 'dart info'

C:\Users\felip_0vh5fa6\dart\bin\dart.exe info

If providing this information as part of reporting a bug, please review the information
below to ensure it only contains things you're comfortable posting publicly.

General info

  • Dart 3.10.0-edge.2bd721707305288f80bcaeec1834cdfcb71885b7 (main) (Fri Jul 25 09:44:17 2025 -0700) on "windows_x64"
  • on windows / "Windows 11 Pro" 10.0 (Build 26100)
  • locale is pt-BR

Project info

  • sdk constraint: '^3.7.0'
  • dependencies: _fe_analyzer_shared, analysis_server_plugin, analyzer, analyzer_plugin, args, collection, convert, crypto, dart_style, dtd, http, json_rpc_2, language_server_protocol, linter, memory_usage, meta, path, pub_semver, stream_channel, telemetry, unified_analytics, vm_service, watcher, yaml, yaml_edit
  • dev_dependencies: analyzer_testing, analyzer_utilities, cli_util, lints, logging, matcher, test, test_descriptor, test_reflective_loader, web_socket_channel

Process info

Memory CPU Elapsed time Command line
0 MB -- dart.exe
0 MB -- dart.exe
0 MB -- dart.exe
0 MB -- dart.exe
0 MB -- dart.exe
0 MB -- dart.exe

Activity

  1. FMorschel
    added
    is bug
    on Jul 30, 2025
  2. DanTup
    added this to the v3.118.0 milestone on Aug 7, 2025
  3. DanTup
    added
    in testingRelates to test execution of Dart/Flutter tests for end users
    on Aug 7, 2025
  4. DanTup
    modified the milestones: v3.118.0, v3.120.0 on Aug 28, 2025

  5. DanTup commented on Sep 2, 2025

    @DanTup
    DanTup
    on Sep 2, 2025
    Member

    @FMorschel are you running with Run or Debug?

    I made a simple sample like this:

    @Timeout(Duration(seconds: 1))
library;

import 'package:test/test.dart';
import 'package:test_reflective_loader/test_reflective_loader.dart';

void main() {
  defineReflectiveSuite(() {
    defineReflectiveTests(FooTest);
  });
}

@reflectiveTest
class FooTest {
  Future<void> test_one() async {
    print('');
    await Future.delayed(Duration.zero);
    print('');
  }

  Future<void> test_two() async {
    print('');
    await Future.delayed(const Duration(seconds: 5));
    print('');
  }

  Future<void> test_three() async {
    print('');
    await Future.delayed(Duration.zero);
    print('');
  }
}

    When running with Debug, I do not get the timeout (because we suppress timeouts). But if using Run, we do (because we don't suppress it). This seems reasonable behaviour to me, but I don't know if it matches what you're seeing.

    When running in debug mode, if I Capture Debugging Logs and then inspect the launch request, I see this:

            "toolArgs": [
            "--ignore-timeouts"
        ],

    This is what controls it, and it appears that test_reflective_loader is just passing the timeout through to pkg:test and therefore it works the same.

  6. DanTup
    added
    awaiting infoRequires more information from the customer to progress
    on Sep 2, 2025

  7. FMorschel commented on Sep 2, 2025

    @FMorschel
    FMorschel
    on Sep 2, 2025
    ContributorAuthor

    Well, I do get a timeout when running on debug because I have breakpoints, and they are my problem. While running, they don't get triggered. I'll see about a screen recording later today to share with you. I can also capture debugging logs to see if that does show up.

  8. DanTup commented on Sep 2, 2025

    @DanTup
    DanTup
    on Sep 2, 2025
    Member

    Well, I do get a timeout when running on debug because I have breakpoints, and they are my problem

    If I change the delay to Duration.zero in the code and instead wait on a breakpoint there, I still don't see a timeout. Is this different to what you see?

    bp.mp4

  9. FMorschel commented on Sep 2, 2025

    @FMorschel
    FMorschel
    on Sep 2, 2025
    ContributorAuthor

    Here is a screen recording of what I see:

    20250902_123228.mp4

    Without any breakpoints:

    Image

    But great idea to capture the logs, the toolArgs is empty:

    Dart-Code-Log-2025-08-02 12-35-02.txt

  10. DanTup commented on Sep 2, 2025

    @DanTup
    DanTup
    on Sep 2, 2025
    Member

    Oh, I see the problem.. this is what I get for building isolated repros instead of testing where you did... We still have some code that considers the Dart SDK as not supporting "dart test", so we never add that arg here (which is obviously silly given that we are spawning the "dart test" debug adapter). Probably this just needs a bit of code deleting :-)

  11. DanTup
    removed
    awaiting infoRequires more information from the customer to progress
    on Sep 2, 2025
  12. DanTup
    changed the title [-]Test timeouts with `test_reflective_loader`[/-] [+]Timeouts are not ignored when debugging tests inside the Dart SDK[/+] on Sep 2, 2025

  13. FMorschel commented on Sep 2, 2025

    @FMorschel
    FMorschel
    on Sep 2, 2025
    ContributorAuthor

    this is what I get for building isolated repros instead of testing where you did...

    I didn't even try because last time I did that, the published version of test_reflective_loader was still before our recent changes, so it had nothing of the behaviour I wanted to reproduce 😅.

  14. DanTup
    closed this as completedin e3b8259on Sep 3, 2025

  15. 3 remaining items

  16. DanTup
    added a commit that references this issue on Sep 3, 2025

    Don't flag Dart SDK as disallowing "dart test"

    e3b8259
  17. DanTup
    modified the milestones: v3.120.0, v3.122.0 on Sep 25, 2025
  18. DanTup
    changed the title [-]Timeouts are not ignored when debugging tests inside the Dart SDK[/-] [+]Timeouts are not ignored when debugging tests inside Pub Workspaces[/+] on Oct 20, 2025
  19. DanTup
    removed
    blocked on dart / flutterRequires a change in Dart or Flutter to progress
    on Oct 20, 2025
  20. DanTup
    closed this as completedin 57c4139on Oct 20, 2025

  21. FMorschel commented on Oct 20, 2025

    @FMorschel
    FMorschel
    on Oct 20, 2025
    ContributorAuthor

    Will this be out on a pre-release version? Not necessary, but I'd be fine with testing it out before it gets to the full version and it would help me a lot on the SDK contributions. Thanks!

  22. DanTup commented on Oct 20, 2025

    @DanTup
    DanTup
    on Oct 20, 2025
    Member

    Yeah, I can do one soon - there may be a few other things worth getting in first though (for ex. the workaround for "pub get" prompts at #5760)

  23. DanTup commented on Oct 22, 2025

    @DanTup
    DanTup
    on Oct 22, 2025
    Member

    I forgot to mention, I did a pre-release yesterday :)

  24. FMorschel commented on Oct 22, 2025

    @FMorschel
    FMorschel
    on Oct 22, 2025
    ContributorAuthor

    Thanks a lot! I'll be sure to update things here.

  25. FMorschel commented on Oct 23, 2025

    @FMorschel
    FMorschel
    on Oct 23, 2025
    ContributorAuthor

    I still seem to be hitting timeouts when working on the SDK. I'm at dart-lang/sdk@1c0c66f from yesterday with gclient sync done.

    Dart-Code-Log-2025-09-04 11-48-38.txt

    Workspace Environment 
    Dart Code extension: 3.121.20251022
Flutter extension: 3.121.20250930 (activated)

App: Visual Studio Code
App Host: desktop
Version: win 1.105.1

Workspace type: Dart (LSP) (Multiroot)
Workspace name: sdk (Workspace)

Dart (3.11.0-edge.b24698b9b5d445fd92eb32486a5a887b9540cd0f): C:\Users\felip_0vh5fa6\dart
Flutter (3.35.5): C:\Users\felip_0vh5fa6\.puro\envs\stable\flutter (No device)
    Output from 'dart info'

    C:\Users\felip_0vh5fa6\dart\bin\dart.exe info

    If providing this information as part of reporting a bug, please review the information
    below to ensure it only contains things you're comfortable posting publicly.

    General info

    • Dart 3.11.0-edge.b24698b9b5d445fd92eb32486a5a887b9540cd0f (main) (Thu Oct 23 04:58:29 2025 -0700) on "windows_x64"
    • on windows / "Windows 11 Pro" 10.0 (Build 26200)
    • locale is pt-BR

    Process info

    Memory CPU Elapsed time Command line
    0 MB -- dart.exe
    0 MB -- dart.exe
    0 MB -- dart.exe
    0 MB -- dart.exe
    0 MB -- dart.exe
    0 MB -- dart.exe
    0 MB -- dart.exe

  26. DanTup commented on Oct 23, 2025

    @DanTup
    DanTup
    on Oct 23, 2025
    Member

    @FMorschel can you check what this setting is set to? Edit: Apaprently GH doesn't like vscode://settings/dart.suppressTestTimeouts links :-)

    Image

    And are you able to reload VS Code ("Reload Window" or "Restart Extension host") and then repro and grab a new log? I can't see us trying to run dart test --version or dart test --help in your log, but I'm not sure why.

  27. FMorschel commented on Oct 23, 2025

    @FMorschel
    FMorschel
    on Oct 23, 2025
    ContributorAuthor

    Hmmm... It was on never by default. Is this desired behaviour? Why?

    Edit: Changing that to debug does solve the problem. Thanks!

  28. DanTup commented on Oct 23, 2025

    @DanTup
    DanTup
    on Oct 23, 2025
    Member

    Oh, I didn't notice that when I took the screenshot, thanks!

    I did some digging and found:

    #3687 (comment)

    I've added a new setting: dart.suppressTestTimeouts. It's an enum of never, always, debug. It will inject --timeout 1d into the toolArgs (although note I found that Flutter test seems to already disable timeouts when running in debug mode, so it's not really doing much there - expect perhaps shortening it from unlimited to 1d).

    For now, the default is never because I'd like to ship this as opt-in. I'll consider changing it to debug in a future release.

    Looks like I never did this (before the flag was supported, we would set the timeout to 1d... maybe I didn't want to default that!).

    I will use this issue to track changing the default :)

  29. DanTup commented on Oct 23, 2025

    @DanTup
    DanTup
    on Oct 23, 2025
    Member

    Oh wait, this issue was something else - I'll file a new issue for that instead 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    in testingRelates to test execution of Dart/Flutter tests for end usersis bug

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions