Gitlab + Flutter CI: Testing very slow

I currently have a Flutter project with over 700 tests which has turned my CI from 15min to almost an hour in the last couple months of development. I've been trying to find ways to reduce the time, but comparing the test times between CI and local, I think there might be an issue with either Gitlab or Flutter (so raising issues with both).

LOCAL

PS C:\Users\lr\Documents\GitHub\hive-manager> flutter test --coverage
04:04 +707: All tests passed!

GITLAB CI

$ flutter test --coverage
45:30 +707: All tests passed!

It is definitely the test phase of the pipeline that is causing issues as when I look at each jobs time there is a noticeable difference:

code_quality - 00:05:24
test - 00:48:47
coverage - 00:02:06 (82.2%)
semantic-version - 00:01:20

additionally, I've tried removing the coverage for certain pipelines to reduce the time which has helped a little but there is still a noticeable difference between local and CI:
LOCAL

PS C:\Users\lr\Documents\GitHub\hive-manager> flutter test
01:34 +707: All tests passed!

GITLAB CI

$ flutter test
08:31 +707: All tests passed!

I've also run the tests in verbose which show nothing out of the ordinary apart from very slow times.
raw.txt

and finally, I've looked at the report.xml in Gitlab and can see that while each test takes ~2sec, the total time is only around 2min despite the CI taking well over 40min.

below is the .gitlab-ci.yml

stages: # List of stages for jobs, and their order of execution
  - analyze
  - test
  - coverage
  - semantic-version

default:
  image: cirrusci/flutter:latest

cache:
  paths:
    - /flutter/bin/cache/dart-sdk

code_quality:
  stage: analyze
  before_script:
    - pub global activate dart_code_metrics
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
  script:
    - flutter --version
    - flutter analyze
    - metrics lib -r codeclimate  > gl-code-quality-report.json
  rules:
    - if: $CI_COMMIT_BRANCH == "develop" && $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      allow_failure: true

  artifacts:
    reports:
      codequality: gl-code-quality-report.json

test-all:
  stage: test
  script:
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
    - pub global activate junitreport
    - flutter test --machine --coverage | tojunit -o report.xml
  only:
    refs:
      - main
      - develop
  artifacts:
    paths:
      - coverage
    reports:
      junit: report.xml

test-dev:
  stage: test
  script:
    - export PATH="$PATH":"$HOME/.pub-cache/bin"
    - pub global activate junitreport
    - flutter test --machine | tojunit -o report.xml
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  artifacts:
    reports:
      junit: report.xml

coverage: # This job runs in the test stage.
  stage: coverage # It only starts when the job in the build stage completes successfully.
  script:
    - lcov --summary coverage/lcov.info
    - lcov --remove coverage/lcov.info
      "lib/config/*"
      "lib/application/l10n/l10n.dart"
      "lib/application/l10n/**/*"
      "lib/domain/repositories/*"
      "lib/injection.config.dart"
      "lib/presentation/routes/*"
      "lib/infrastructure/repositories/firebase_injectable_module.dart"
      "**/mock_*.dart"
      "**/*.g.dart"
      "**/*.gr.dart"
      "**/*.freezed.dart"
      "**/*.mocks.dart"
      "**/*.config.dart"
      -o coverage/clean_lcov.info
    - genhtml coverage/clean_lcov.info --output=coverage
    - curl -Os https://uploader.codecov.io/latest/linux/codecov
    - chmod +x codecov
    - ./codecov -t $CODECOV_TOKEN
    - mv coverage/ public/
  only:
    refs:
      - main
      - develop
  coverage: '/lines\.*: \d+\.\d+\%/'
  artifacts:
    paths:
      - public

semantic-version:
  image: node:16
  stage: semantic-version
  only:
    refs:
      - main
      - develop
  script:
    - touch CHANGELOG.md
    - npm install @semantic-release/gitlab @semantic-release/changelog
    - npx semantic-release
  artifacts:
    paths:
      - CHANGELOG.md

See #86722

@lloydrichards

@lloydrichards
Can you provide your flutter doctor -v on which you are experiencing this behavior ?

Similar issue / scenario reported in linked issue above was fixed per this comment: #86722 (comment)

LOCAL

PS C:\Users\lr\Documents\GitHub\hive-manager> flutter doctor -v
[√] Flutter (Channel stable, 2.10.3, on Microsoft Windows [Version 10.0.22000.556], locale en-US)
    • Flutter version 2.10.3 at C:\Users\lr\Documents\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 7e9793dee1 (3 weeks ago), 2022-03-02 11:23:12 -0600
    • Engine revision bd539267b4
    • Dart version 2.16.1
    • DevTools version 2.9.2

[√] Android toolchain - develop for Android devices (Android SDK version 30.0.3)
    • Android SDK at C:\Users\lr\AppData\Local\Android\Sdk
    • Platform android-31, build-tools 30.0.3
    • ANDROID_HOME = C:\Users\lr\AppData\Local\Android\Sdk
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6842174)
    • All Android licenses accepted.

[√] Chrome - develop for the web
    • Chrome at C:\Program Files\Google\Chrome\Application\chrome.exe

[√] Visual Studio - develop for Windows (Visual Studio Community 2022 17.0.6)
    • Visual Studio at C:\Program Files\Microsoft Visual Studio\2022\Community
    • Visual Studio Community 2022 version 17.0.32126.317
    • Windows 10 SDK version 10.0.20348.0

[√] Android Studio (version 4.2)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin can be installed from:
       https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
       https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.8+10-b944.6842174)

[√] VS Code (version 1.65.2)
    • VS Code at C:\Users\lr\AppData\Local\Programs\Microsoft VS Code
    • Flutter extension version 3.37.20220301

[√] Connected device (3 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.22000.556]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 99.0.4844.82
    • Edge (web)        • edge    • web-javascript • Microsoft Edge 99.0.1150.39

[√] HTTP Host Availability
    • All required HTTP hosts are available

• No issues found!

GITLAB CI

$ flutter doctor -v
[✓] Flutter (Channel unknown, 2.10.3, on Ubuntu 20.04.3 LTS 5.4.109+, locale en_US.UTF-8)
    • Flutter version 2.10.3 at /sdks/flutter
    • Upstream repository unknown
    • Framework revision 7e9[79](https://gitlab.com/vatorex/hive-manager/-/jobs/2251365918#L79)3dee1 (3 weeks ago), 2022-03-02 11:23:12 -0600
    • Engine revision bd53[92](https://gitlab.com/vatorex/hive-manager/-/jobs/2251365918#L92)67b4
    • Dart version 2.16.1
    • DevTools version 2.9.2
[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /opt/android-sdk-linux
    • Platform android-30, build-tools 30.0.2
    • ANDROID_HOME = /opt/android-sdk-linux
    • ANDROID_SDK_ROOT = /opt/android-sdk-linux
    • Java binary at: /usr/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.13+8-Ubuntu-0ubuntu1.20.04)
    • All Android licenses accepted.
[✗] Chrome - develop for the web (Cannot find Chrome executable at google-chrome)
    ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/linux#android-setup for detailed instructions).
[!] Connected device
    ! No devices available
[✓] HTTP Host Availability
    • All required HTTP hosts are available
! Doctor found issues in 3 categories.

From the linked thread it seems like this has been addressed and fixed (unless I'm reading it wrong). Am currently using flutter_test in dev dependencies, but without a version:

dev_dependencies:
  flutter_test:
    sdk: flutter
  integration_test:
    sdk: flutter

Thanks for the update. I am keeping it open and labeling for further insights from the team.

@liamappelbe

/cc @liamappelbe

[vm] Switch representation of line_starts to allow binary searching

Revert "[vm] Skip filtered class earlier in source report."

Use `coverage.collect`'s `coverableLineCache` param to speed up cover…

Reverts "Use `coverage.collect`'s `coverableLineCache` param to speed…

@mraleph

In @mraleph 's stress test, searching this array accounts for the vast majority of coverage time, so we'll get a big speed up if I change the representation. I'll work on this soon.

Friendly ping - did we end up doing this? If so, should we close this issue for now?

Yeah, that was implemented a while ago. I think we can close this issue.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

Gitlab + Flutter CI: Testing very slow #100751

42 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Gitlab + Flutter CI: Testing very slow #100751

Description

Activity

TahaTesser commented on Mar 25, 2022

darshankawar commented on Mar 25, 2022

lloydrichards commented on Mar 25, 2022

darshankawar commented on Mar 28, 2022

darshankawar commented on Mar 28, 2022

42 remaining items

matanlurey commented on Nov 20, 2024

liamappelbe commented on Nov 20, 2024

github-actions commented on Dec 5, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions