tests/run-tests.py: Add automatic retry for known-flaky tests. by andrewleech · Pull Request #18861 · micropython/micropython

andrewleech · 2026-02-23T19:35:42Z

Summary

The unix port CI workflow on master fails ~18% of the time due to a handful of intermittent test failures (mostly thread-related). This has normalised red CI to the point where contributors ignore it, masking real regressions.

I added a flaky-test allowlist and retry-until-pass mechanism to run-tests.py. When a listed test fails, it's re-run up to 2 more times and passes on the first success. The 8 tests on the list account for essentially all spurious CI failures on master.

tools/ci.sh is updated to remove --exclude patterns for tests now handled by the retry logic, so they get actual coverage again. thread/stress_aes.py stays excluded on RISC-V QEMU where it runs close to the 200s timeout and retries would burn excessive CI time.

Retries are on by default; --no-retry disables them. The end-of-run summary reports tests that passed via retry separately from clean passes.

Testing

Full CI passed on fork (all workflows green). The codecov upload step is also gated on CODECOV_TOKEN being available so it skips cleanly on forks.

Trade-offs and Alternatives

Retry-until-pass can mask a test that has become intermittently broken. The allowlist is deliberately small and each entry documents a reason and optional platform restriction. An alternative would be to mark these tests as expected-fail, but that removes coverage entirely which is worse than the current situation.

The test result status strings ("pass", "fail", "skip") are bare string literals throughout run-tests.py — the retry code follows this existing convention. Worth considering a follow-up to consolidate these into constants for the whole file.

Generative AI

I used generative AI tools when creating this PR, but a human has checked the code and is responsible for the description above.

github-actions · 2026-02-23T21:41:24Z

Code size report:

Reference:  rp2/boards/WAVESHARE_RP2040_ZERO: Add Waveshare RP2040 Zero board. [ad054fc]
Comparison: tests/run-tests.py: Ignore known-flaky test failures. [merge of 4319927]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
      esp32:    +0 +0.000% ESP32_GENERIC
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

projectgus

Thanks for putting this together, @andrewleech!

It looks fine to me, although the implementation feels like yet another way that run-tests.py is due for a refactor as the simple script-y style is starting to buckle under the load of features added to it!

However that's a problem for another day, and having a flaky test retry mechanism seems pretty valuable.

Only one thing I'm worried about: the Checks list seems to not include any of the Unix port checks running? Do you know what that might be? I was going to go and mash "retry" on them a few times to see how the flaky check works in practice.

Would also add, there is another possible alternative approach here which is to improve these flaky tests until they are no longer flaky. I think we should keep this as an eventual goal, but the fact they've been flaky for some years now suggests it's not a viable short-term goal.

andrewleech · 2026-02-25T02:23:23Z

Thanks @projectgus I've got a few minor cleanups to make still based on feedback in andrewleech#6 ...

Only one thing I'm worried about: the Checks list seems to not include any of the Unix port checks running? Do you know what that might be? I was going to go and mash "retry" on them a few times to see how the flaky check works in practice.

I'm not quite sure what you mean by Checks list here?
Most of the tests in the FLAKY_TESTS dict are allowed to retry on all platforms (including unix) except a couple that were only excluded on particular platforms originally.

Would also add, there is another possible alternative approach here which is to improve these flaky tests until they are no longer flaky.

Oh yes that's absolutely preferential, ideally even with this flaky retry handling the tests should get fixed to eventually empty the list here.

I believe it's not just the tests that are flaky though... threads in particular I think have a/some bugs on unix port at least that the tests correctly detect, some kind of intermittent race condition that's hard to track down. I've tried to look into this in the past but not got to the bottom of it. The other tests, not sure, but the AI investigation into these failing tests did purpose some theories (in the report in a gist linked from my discussion post) that I haven't reviewed yet.

projectgus · 2026-02-25T03:34:06Z

I believe it's not just the tests that are flaky though... threads in particular I think have a/some bugs on unix port at least that the tests correctly detect, some kind of intermittent race condition that's hard to track down.

Oh for sure, I should have said to "fix whatever is making the tests fail".

I'm not quite sure what you mean by Checks list here?

Sorry, wasn't clear: I meant that if I click the Checks tab in this PR then I can't see the workflow run for the unix port in the list of workflows... It's like it somehow hasn't run it?

As it happens, though, I spoke with @dpgeorge about this just now and he prefers if we ignore failures on these tests instead of retrying. He made the reasonable point that most bugs will be masked in either case, so we may as well do it the simpler way. Do you want to update this PR? If not then I'm happy to throw up a quick one to add some more ignore logic.

Josverl · 2026-02-25T05:25:34Z

think we should keep fixing the flaky tests as an eventual goal, but the fact they've been flaky for some years now suggests it's not a viable short-term goal.

Should we create issues for each of these flaky test, or just keep them as a TODO comment in the test or test runner?

I'd prefer explicit issues for sake of visibility and possible connections to real-world occurrences.

andrewleech · 2026-02-25T19:04:40Z

Updated to ignore failures instead of retrying, as suggested.

The FLAKY_TESTS dict stays the same but now after all tests complete, failures on listed tests get reclassified from "fail" to "ignored". They still run and the output is reported, but they don't affect the exit code. Ended up being quite a bit simpler than the retry approach.

The --exclude lists in ci.sh for flaky tests are removed so those tests actually run now, except thread/stress_aes.py on RISCV64 which is still excluded because it takes ~180s against a 200s timeout and burning that CI time just to ignore the failure isn't great.

@Josverl yeah creating individual issues for each is a good idea, it'd make it easier to track and reference from the FLAKY_TESTS dict as well, though they might end up lost in the backlog?

Force pushed, old version backed up at flaky-test-retry-v1-backup if anyone wants to compare.

projectgus · 2026-02-26T01:22:14Z

@@ -93,6 +93,7 @@ jobs:
        (cd ports/unix && gcov -o build-coverage/py ../../py/*.c || true)
        (cd ports/unix && gcov -o build-coverage/extmod ../../extmod/*.c || true)
    - name: Upload coverage to Codecov
+      if: ${{ secrets.CODECOV_TOKEN != '' }}


Ah, this is why the unix port checks aren't running in this PR:

https://github.com/micropython/micropython/actions/runs/22411608686

Invalid workflow file: .github/workflows/ports_unix.yml#L1 (Line: 96, Col: 11): Unrecognized named-value: 'secrets'. Located at position 1 within expression: secrets.CODECOV_TOKEN != ''

Ah, whoops. I'll have to figure out a better way to fix this (or leave it for another PR).

Basically this step always fails if you run actions on a fork, so I'm hoping to gate it cleanly so forks can properly test things before submitting a pr

codecov · 2026-03-03T10:55:34Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.45%. Comparing base (ad054fc) to head (4319927).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #18861   +/-   ##
=======================================
  Coverage   98.45%   98.45%           
=======================================
  Files         175      175           
  Lines       22635    22635           
=======================================
  Hits        22286    22286           
  Misses        349      349

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

projectgus

Thanks @andrewleech!

Reclassify failures of tests listed in flaky_tests_to_ignore as "ignored" instead of retrying them. Ignored tests still run and their output is reported, but they don't affect the exit code. The ci.sh --exclude lists for these tests are removed so they run normally. Signed-off-by: Andrew Leech <[email protected]>

projectgus reviewed Feb 24, 2026

View reviewed changes

Comment thread tools/ci.sh Outdated

andrewleech force-pushed the flaky-test-retry branch 2 times, most recently from 5571e56 to cbc5529 Compare February 25, 2026 19:04

projectgus reviewed Feb 26, 2026

View reviewed changes

dpgeorge reviewed Feb 26, 2026

View reviewed changes

Comment thread tests/run-tests.py Outdated

andrewleech force-pushed the flaky-test-retry branch from cbc5529 to 36ae4eb Compare February 27, 2026 10:04

andrewleech mentioned this pull request Mar 1, 2026

[TRIAGE TEST #18868] tests: cmdline/repl_lock.py and repl_cont.py intermittent failures andrewleech/micropython#12

Closed

mpy-reviewer Bot mentioned this pull request Mar 1, 2026

[TRIAGE TEST 18870] tests: thread/stress_heap.py intermittent failure on macOS andrewleech/micropython#17

Open

dpgeorge added the tests Relates to tests/ directory in source label Mar 1, 2026

dpgeorge reviewed Mar 2, 2026

View reviewed changes

Comment thread tools/ci.sh Outdated

andrewleech force-pushed the flaky-test-retry branch 2 times, most recently from 9cfef37 to 032aa59 Compare March 3, 2026 10:51

andrewleech force-pushed the flaky-test-retry branch 2 times, most recently from 5ed226e to 06ba55a Compare March 4, 2026 09:58

dpgeorge reviewed Mar 9, 2026

View reviewed changes

Comment thread tests/run-tests.py Outdated

andrewleech force-pushed the flaky-test-retry branch from 06ba55a to 56d2d12 Compare March 10, 2026 02:35

projectgus approved these changes Mar 11, 2026

View reviewed changes

dpgeorge force-pushed the flaky-test-retry branch from 56d2d12 to 4319927 Compare March 11, 2026 03:58

dpgeorge merged commit 4319927 into micropython:master Mar 11, 2026
76 checks passed

projectgus mentioned this pull request Mar 25, 2026

github: Avoid unnecessary failures when Actions run in forks #18998

Merged

Navigation Menu

Repository navigation

tests/run-tests.py: Add automatic retry for known-flaky tests. - #18861

tests/run-tests.py: Add automatic retry for known-flaky tests.#18861
dpgeorge merged 1 commit into
micropython:masterfrom
andrewleech:flaky-test-retry

andrewleech commented Feb 23, 2026

Uh oh!

github-actions Bot commented Feb 23, 2026 •

edited

Loading

Uh oh!

projectgus left a comment

Uh oh!

Uh oh!

andrewleech commented Feb 25, 2026

Uh oh!

projectgus commented Feb 25, 2026

Uh oh!

Josverl commented Feb 25, 2026

Uh oh!

andrewleech commented Feb 25, 2026 •

edited

Loading

Uh oh!

projectgus Feb 26, 2026 •

edited

Loading

Uh oh!

andrewleech Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Mar 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

projectgus left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

andrewleech commented Feb 23, 2026

Summary

Testing

Trade-offs and Alternatives

Generative AI

Uh oh!

github-actions Bot commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

projectgus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrewleech commented Feb 25, 2026

Uh oh!

projectgus commented Feb 25, 2026

Uh oh!

Josverl commented Feb 25, 2026

Uh oh!

andrewleech commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

projectgus Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewleech Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

projectgus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

github-actions Bot commented Feb 23, 2026 •

edited

Loading

andrewleech commented Feb 25, 2026 •

edited

Loading

projectgus Feb 26, 2026 •

edited

Loading

codecov Bot commented Mar 3, 2026 •

edited

Loading