Skip to content

CI flakiness inventory: Aug 11-Sep 8, 2026 #12704

Closed
Closed
CI flakiness inventory: Aug 11-Sep 8, 2026#12704
#12718 (+2)
Labels
needs-triageIssues that need to be assigned.

Description

@simonrozsival
simonrozsival
opened on Sep 8, 2026

Android framework version

net11.0-android (Preview)

Affected platform version

dotnet/android pull-request CI, public Azure DevOps dotnet-android pipeline, 2026-08-11 through 2026-09-08.

Description

This issue tracks flaky infrastructure and flaky tests observed across 242 PR pipeline runs from 2026-08-11 through 2026-09-08. Of these, 241 runs completed (123 red and 118 green); 1 run was still active at collection time.

Frequency is measured as build incidents / 241 completed PR runs and as the average incidents per week over this four-week interval. A build incident is one unique root signature in one build; duplicate configuration annotations are collapsed. Independent roots in the same build are counted separately. Failures determined to be caused by the PR are intentionally omitted from this issue.

The first PR pipeline run returned within the interval started at 2026-08-29 02:03 UTC; the Azure build query returned no PR runs from 2026-08-11 through 2026-08-29 before that run.

Summary: 18 infrastructure signatures / 71 incidents and 10 flaky-test signatures / 32 incidents.

Earlier inventories and focused trackers

Recently merged fixes and recurrence

Failure family Merged change Evidence after merge Assessment
GetObjectArray peer identity #12650 on Sep 3 15:45 UTC Last failure was Sep 3 09:25 UTC; 0 recurrences in 71 later completed builds. Earlier mitigations #12271 and #12357 had not eliminated it. Likely fixed.
InstallAndroidDependenciesTest missing SharpZipLib/Polly #12649 on Sep 3 04:59 UTC The only current occurrence preceded the merge; 0 recurrences in 107 later completed builds. Earlier work #11974, #12059, and #12580 addressed separate retry/network paths. Likely fixed.
CheckSignApk stale JAVA_HOME #12603 on Sep 1 13:54 UTC The only current occurrence preceded the merge; 0 recurrences in 167 later completed builds. Likely fixed.
BuildReleaseArm64 apkdiff timeout #12515 on Sep 1 10:14 UTC Both real occurrences preceded the change from 30 to 120 seconds; 0 recurrences in 184 completed builds queued after merge. Build 1575474 was a false attribution: the test passed and MAUI Helix caused the red build. Likely fixed.
BootstrapTasks libMono.Unix path evaluation #12231 on Jul 27 12:28 UTC The only occurrence was PR #12611 against a release branch that diverged before the fix. 313 other scannable completed builds after merge had zero exact recurrences. Fixed on main; release-branch exception.
BindLibrary generated MemorySafetyRulesAttribute compile #12700 on Sep 7 12:16 UTC, carrying dotnet/roslyn#85131 The only failure was already running with the older SDK when #12700 merged; 0 recurrences in 7 verified completed builds based directly on the fixed SDK merge. Likely fixed.
MAUI R2R Helix #12641 on Sep 3 16:00 UTC The fork-token 401 path was fixed, but direct device-lab failures continued through Sep 8. Partially fixed; #12658 remains ongoing.
Shared-process JNI GREF/peer assertions #12032 and #12037 in July Those PRs disabled two specific assertions. The enabled String overload and JavaSideActivation still failed in this interval. Ongoing under #12031.
Logcat capture pressure/hangs #12471 on Aug 21 and #12620 on Sep 2 Capture timeouts still occurred Sep 4 and Sep 7. Mitigated, not fixed.
Previous inventory: performance and WearOS noise #11249 and #11257 on May 1 Those failure families are absent from this interval. Fixes are still holding.
Previous FastTiming logcat race #12492 on Aug 25 No unrelated-PR recurrence was classified as flaky. One later failure occurred on a PR directly changing FastTiming and was classified PR-related. Original flake fix appears to be holding.

First-attempt and unchanged-rerun reliability

Unit: one PR plus exact source SHA. Sequences containing any PR-related incident are excluded, so a later build is an unchanged rerun rather than a code-change build.

Outcome Sequences Share
Green on first attempt 109 70.3%
- clean green with no recorded flake 91 58.7%
- green with non-gating flake(s) 18 11.6%
Red solely from flakiness 46 29.7%

Of the 46 flake-red first attempts, 32 were infrastructure-only, 8 test-only, and 6 contained both.

Unchanged-rerun metric Result
Flake-red sequences actually rerun at the same SHA 5 / 46 (10.9%)
Eventually green when rerun unchanged 4 / 5 (80.0%)
Median reruns to green among recovered sequences 1
Mean reruns to green among recovered sequences 1.25

Three recovered on the first rerun; one recovered on the second. One additional unchanged sequence was rerun once but hit another flake and remained red.

Recovered unchanged-SHA chains:

Most flake-red sequences were not rerun unchanged, so they are censored rather than counted as failed recoveries.

Flaky infrastructure

Failure First seen Last seen Frequency Red / non-gating green PRs Evidence
MAUI R2R Helix work item/submission failure 2026-08-30 18:14 UTC (1574227) 2026-09-08 07:04 UTC (1586410) 35 / 241 (14.5%; 8.75/week) 26 / 9 29 Status: Ongoing. Fork authentication was fixed by #12641, but direct device-lab failures continued through Sep 8 and are tracked by #12658. Recurring Helix work-item or submission failures crossed unrelated PRs and sometimes appeared as non-gating failures in green builds. Confidence: high.
Gradle daemon/cache/tooling failure 2026-08-30 18:14 UTC (1574227) 2026-09-02 21:26 UTC (1578693) 11 / 241 (4.6%; 2.75/week) 11 / 0 9 Status: Quiet since Sep 2; no merged fix was identified. Gradle daemons, cache transforms, or shared Java helper builds failed across unrelated PRs; unchanged retries later passed for this family. Confidence: medium.
MAUI restore requested an unpublished Microsoft.Extensions.Logging build 2026-08-31 19:40 UTC (1575484) 2026-09-04 06:10 UTC (1582480) 4 / 241 (1.7%; 1.00/week) 4 / 0 4 Status: Transient publication/feed lag; quiet since Sep 4 without a repository fix. NU1102 requested a dotnet11 package version newer than the nearest version available on the feed. Confidence: high.
Hosted agent disconnected or was deprovisioned 2026-08-31 16:56 UTC (1575255) 2026-09-07 07:14 UTC (1585360) 3 / 241 (1.2%; 0.75/week) 3 / 0 3 Status: Ongoing/recent; last observed Sep 7. Azure stopped hearing from the agent or the remote provider cancelled the machine. Confidence: high.
Azure artifact download had no route to host 2026-09-04 10:50 UTC (1582867) 2026-09-04 17:15 UTC (1583384) 2 / 241 (0.8%; 0.50/week) 2 / 0 2 Status: Transient cluster on Sep 4; no later recurrence in the interval. DownloadPipelineArtifact could not connect to dev.azure.com:443; one occurrence also timed out publishing telemetry. Confidence: high.
Logcat capture task timed out 2026-09-04 15:31 UTC (1583273) 2026-09-07 11:18 UTC (1585745) 2 / 241 (0.8%; 0.50/week) 2 / 0 2 Status: Ongoing/recent. It recurred after #12471 bounded capture and after #12620 reduced reference logcat traffic. One or more post-test capture-logcat tasks hit the task timeout. Confidence: high.
System.NetTests.SslTest.VerifyTrustedCertificates 2026-08-31 09:46 UTC (1574749) 2026-09-03 22:24 UTC (1581249) 2 / 241 (0.8%; 0.50/week) 1 / 1 2 Status: Quiet since Sep 3; no merged fix was identified. Certificate/network validation failed in an unrelated PR and did not gate the otherwise-successful build. Confidence: medium.
dotnet-test-slicer could not find its required .NET runtime 2026-08-31 07:47 UTC (1574631) 2026-08-31 13:40 UTC (1574989) 2 / 241 (0.8%; 0.50/week) 2 / 0 2 Status: Quiet since Aug 31; no repository fix was identified. The hosted macOS image launched dotnet-test-slicer without DOTNET_ROOT or a discoverable runtime. Confidence: high.
ADB install failed with a broken pipe 2026-09-01 04:31 UTC (1576002) 2026-09-01 04:31 UTC (1576002) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single transient occurrence on Sep 1. Device installation failed in adb client/server communication rather than in product compilation. Confidence: high.
Android toolchain cache download was rate-limited 2026-09-02 14:03 UTC (1578771) 2026-09-02 14:03 UTC (1578771) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single transient HTTP 429 occurrence on Sep 2. The shared cache step failed with HTTP 'Too Many Requests'. Confidence: high.
BootstrapTasks could not locate libMono.Unix.so 2026-09-01 04:45 UTC (1576007) 2026-09-01 04:45 UTC (1576007) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Fixed by #12231 on main. The only occurrence came from a release branch that diverged before the fix; 313 other scannable completed builds after merge had zero exact recurrences. The release branch lacked #12231, so BootstrapTasks evaluated an empty PkgMono_Unix path and attempted a root-relative native-library copy. Confidence: high.
Git submodule checkout failed with exit code 128 2026-08-29 02:03 UTC (1573456) 2026-08-29 02:03 UTC (1573456) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single transient checkout occurrence on Aug 29. Repository checkout failed before product code was built. Confidence: high.
Hosted agent DNS failure while installing the .NET SDK 2026-09-02 07:35 UTC (1578195) 2026-09-02 07:35 UTC (1578195) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single transient DNS occurrence on Sep 2. curl exited with code 6 while the SDK installation step resolved its download host; the same SHA later passed. Confidence: high.
Hosted job exceeded the 240-minute limit 2026-09-02 21:21 UTC (1579664) 2026-09-02 21:21 UTC (1579664) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single occurrence on Sep 2, downstream of workload download stalls. The macOS job was terminated by Azure after the maximum hosted-job duration. Confidence: high.
NuGet workload-pack downloads timed out 2026-09-02 21:21 UTC (1579664) 2026-09-02 21:21 UTC (1579664) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single transient feed/download occurrence on Sep 2. Multiple package downloads received no data for 60 seconds and the operation was cancelled. Confidence: high.
Pipeline failed before any job or timeline record started 2026-09-04 21:12 UTC (1583713) 2026-09-04 21:12 UTC (1583713) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single Azure service/startup occurrence on Sep 4. The build began and ended at the same instant, with no GitHub check and no Azure timeline. Confidence: high.
Xamarin.Android.Build.Tests.BuildTest.Desugar(True,"",CoreCLR) 2026-09-01 22:23 UTC (1577672) 2026-09-01 22:23 UTC (1577672) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Single external-download timeout on Sep 1. HttpClient timed out while the test fetched external Android artifacts; this is a network dependency failure. Confidence: high.
Xamarin.Android.NetTests.AndroidMessageHandlerTests.HttpContentStreamIsRewoundAfterCancellation 2026-09-07 07:02 UTC (1584741) 2026-09-07 07:02 UTC (1584741) 1 / 241 (0.4%; 0.25/week) 0 / 1 1 Status: Ongoing/recent; non-gating socket failure observed Sep 7. The failure was an IOException 'Broken pipe' during a green build, indicating transient socket infrastructure. Confidence: high.

Flaky tests

Failure First seen Last seen Frequency Red / non-gating green PRs Evidence
Android.RuntimeTests.JnienvArrayMarshaling.GetObjectArray 2026-08-29 04:44 UTC (1573562) 2026-09-03 09:25 UTC (1579507) 14 / 241 (5.8%; 3.50/week) 11 / 3 12 Status: Likely fixed by #12650: last failure preceded the merge, with 0 recurrences in 71 completed builds afterward. JNI object identity intermittently mismatched across many unrelated PRs, including green builds and unchanged retries. Confidence: high.
Xamarin.Android.JcwGenTests.BindingTests.JavaSideActivation 2026-09-01 22:13 UTC (1577025) 2026-09-07 13:57 UTC (1583153) 4 / 241 (1.7%; 1.00/week) 2 / 2 4 Status: Ongoing; last observed Sep 7 and covered by open tracker #12031. The GREF assertion recurred across unrelated PRs and in non-gating green builds. Confidence: high.
Xamarin.Android.Build.Tests.InstallAndRunTests.GradleFBProj(False,CoreCLR) 2026-09-01 18:37 UTC (1577327) 2026-09-04 14:38 UTC (1583179) 3 / 241 (1.2%; 0.75/week) 3 / 0 3 Status: Quiet since Sep 4; no merged fix was identified. The device test failed across unrelated PRs, and build 1577702 passed unchanged as build 1578074. Confidence: high.
Java.InteropTests.JniTypeUtf8Test.TryFindClass_String_DoesNotLeakGlobalRefs 2026-08-31 13:40 UTC (1574982) 2026-09-01 10:15 UTC (1576388) 2 / 241 (0.8%; 0.50/week) 2 / 0 2 Status: Still unresolved under #12031. #12032 disabled the different UTF-8 overload, not this String overload. The global-reference count assertion failed on unrelated native and typemap PRs. Confidence: high.
Xamarin.Android.Build.Tests.BuildTest2.BuildReleaseArm64(True,CoreCLR,True) 2026-08-31 13:40 UTC (1574985) 2026-08-31 20:14 UTC (1575529) 2 / 241 (0.8%; 0.50/week) 2 / 0 2 Status: Likely fixed by #12515: the two real occurrences used the old 30-second apkdiff timeout, with 0 recurrences in 184 completed post-merge builds. Build 1575474 was removed as a false attribution because the test passed and MAUI Helix caused the red build. The apkdiff process timed out under the former 30-second limit before producing a size-regression verdict. Confidence: medium.
Xamarin.Android.Build.Tests.InstallAndRunTests.DotNetRunBenchmarkDotNet 2026-08-31 07:54 UTC (1573240) 2026-09-03 11:11 UTC (1580340) 2 / 241 (0.8%; 0.50/week) 0 / 2 2 Status: Quiet since Sep 3; no merged fix was identified. The benchmark test failed only as a non-gating result in otherwise-green unrelated builds. Confidence: high.
Xamarin.Android.Build.Tests.InstallAndRunTests.GradleFBProj(True,NativeAOT) 2026-08-31 07:52 UTC (1573440) 2026-09-07 13:34 UTC (1583858) 2 / 241 (0.8%; 0.50/week) 0 / 2 2 Status: Ongoing/recent; last observed as a non-gating failure Sep 7. The device test appeared as a non-gating failure in otherwise-green unrelated builds. Confidence: high.
Xamarin.Android.Build.Tests.AndroidDependenciesTests.InstallAndroidDependenciesTest("GoogleV2",CoreCLR) 2026-09-02 21:22 UTC (1579349) 2026-09-02 21:22 UTC (1579349) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Likely fixed by #12649: 0 recurrences in 107 completed builds after the runtime-dependency fix. The test host could not load ICSharpCode.SharpZipLib; the next build passed after only merging main. Confidence: high.
Xamarin.Android.Build.Tests.AndroidGradleProjectTests.BindLibrary(True,"Debug",True,CoreCLR) 2026-09-07 11:30 UTC (1584520) 2026-09-07 11:30 UTC (1584520) 1 / 241 (0.4%; 0.25/week) 0 / 1 1 Status: Likely fixed by #12700, which flowed the Roslyn fix from dotnet/roslyn#85131. The only failure used the older SDK; 0 recurrences in 7 verified post-merge SDK exposures. The generated binding compile hit the Roslyn MemorySafetyRulesAttribute decoding race fixed upstream by dotnet/roslyn#85131. Confidence: high.
Xamarin.Android.Build.Tests.PackagingTest.CheckSignApk(False,False,NativeAOT) 2026-08-29 06:09 UTC (1573574) 2026-08-29 06:09 UTC (1573574) 1 / 241 (0.4%; 0.25/week) 1 / 0 1 Status: Likely fixed by #12603: 0 recurrences in 167 completed builds after the SDK-state restoration fix. The test inherited an invalid JAVA_HOME from a previous test, indicating test isolation/order dependence. Confidence: high.

Steps to Reproduce

  1. Enumerate all dotnet-android PR builds started during the interval.
  2. Read GitHub aggregate-check annotations for failed-test names and messages.
  3. Read the Azure timeline and failed task logs for infrastructure and failures omitted from aggregate annotations.
  4. Collapse duplicate configuration annotations to one signature per build, then group identical signatures across builds.
  5. Classify signatures using unchanged retry success, occurrence across unrelated PRs, green non-gating failures, task-log root causes, and overlap with PR changes.

Did you find any workaround?

Rerunning an unchanged SHA cleared several of these failures. That is only a tactical workaround; the recurring signatures above should be fixed or quarantined independently.

Relevant log output

Each row links the first and last Azure build where the signature was observed. The underlying catalog retains every affected build and PR.

Activity

  1. simonrozsival
    added
    needs-triageIssues that need to be assigned.
    on Sep 8, 2026

  2. simonrozsival commented on Sep 8, 2026

    @simonrozsival
    simonrozsival
    on Sep 8, 2026
    MemberAuthor

    CI flake remediation plan

    Goal

    Reduce the current 29.7% flake-red first-attempt rate by assigning one independent
    dotnet/android session and pull request to each actionable failure family.

    Each child session must:

    1. Re-read issue CI flakiness inventory: Aug 11-Sep 8, 2026 #12704 and the linked Azure examples.
    2. Reproduce or establish the root cause from logs before changing behavior.
    3. Prefer a root-cause fix; quarantine or retry only when the external dependency
      cannot be made deterministic.
    4. Add focused coverage where practical.
    5. Create a draft pull request whose title starts with [flaky-ci], has the
      flaky-ci label, and references CI flakiness inventory: Aug 11-Sep 8, 2026 #12704 plus any focused tracker.
    6. Report the PR and remaining uncertainty to this coordinator.

    Wave 1: highest impact

    Workstream Current signal Intended PR
    MAUI R2R Helix device-lab failures 35 incidents; ongoing; #12658 Add device preflight, cleanup/recovery, and actionable machine/device diagnostics without hiding real APK failures.
    Gradle daemon/cache/tooling failures 11 incidents Isolate shared Gradle state and make known daemon/cache corruption recoverable without retrying real Gradle build errors.
    JavaSideActivation GREF assertion 4 incidents; ongoing; #12031 Replace exact process-global GREF equality with an isolated or operation-specific leak assertion.
    GradleFBProj device tests 5 incidents across CoreCLR and NativeAOT variants Stabilize app startup/device readiness and retain meaningful Gradle/Facebook project coverage.
    Logcat capture timeouts 2 incidents after prior mitigations Make capture bounded and non-gating while preserving failure diagnostics.
    MAUI package publication lag 4 incidents Make MAUI restore tolerate short dependency-publication skew without masking persistent package resolution errors.

    Wave 2: remaining actionable failures

    Workstream Current signal Intended PR
    JniTypeUtf8 String GREF assertion 2 incidents; open under #12031 Isolate or redesign the enabled String-overload leak assertion.
    BuildReleaseArm64 APK-size baseline 3 incidents Make the comparison deterministic and explain real size regressions without stale-baseline flakes.
    dotnet-test-slicer runtime discovery 2 incidents Ensure the tool runs with an explicit compatible runtime on hosted macOS agents.
    ADB install broken pipe 1 incident Retry only recognized transport breakage and preserve real install failures.
    Hosted dependency downloads cache HTTP 429 plus NuGet workload-pack stalls/job timeout Add bounded retries/backoff for transient package acquisition and avoid consuming the full hosted-job limit.
    Desugar external artifact timeout 1 incident Remove or harden the external network dependency, preferably through existing mirrors/caches.
    Missing libMono.Unix test artifact 1 incident Correct the test runtime copy-local dependency closure.
    AndroidMessageHandler broken pipe 1 non-gating incident Make the cancellation test server/client teardown deterministic without weakening assertions.
    BenchmarkDotNet device test 2 non-gating incidents Stabilize benchmark app startup/execution and improve failure diagnostics.
    AndroidGradleProject BindLibrary 1 non-gating incident Reproduce and remove the nondeterministic Gradle/binding fixture failure.
    System.NetTests VerifyTrustedCertificates 2 incidents Stabilize certificate/environment assumptions while preserving the trust validation coverage.

    Verify, but do not create duplicate PRs

    Failure Existing fix Verification
    GetObjectArray peer identity #12650 No recurrence in 71 completed post-merge builds.
    InstallAndroidDependencies missing SharpZipLib/Polly #12649 No recurrence in 107 completed post-merge builds.
    CheckSignApk stale JAVA_HOME #12603 No recurrence in 167 completed post-merge builds.
    Original FastTiming logcat race #12492 No unrelated-PR recurrence in the current catalog.

    External-only incidents

    Do not create speculative repository PRs for hosted-agent deprovisioning, Azure
    no-route-to-host, DNS outages, pipeline startup failure, or isolated git checkout
    failure. Keep them in #12704 for frequency tracking and escalate to the service/device-
    lab owners when they recur.

    Coordination

    • All 17 implementation sessions branch independently from main.
    • Sessions start in plan mode; the coordinator approves only plans that identify a
      concrete root cause and a reviewable fix.
    • Overlapping pipeline files may produce merge conflicts; PRs remain independent and
      will be rebased after higher-impact fixes merge.
    • Prioritize Wave 1 review and merge order by expected incident reduction.
  3. jonathanpeppers
    added 2 commits that reference this issue on Sep 8, 2026

    [ci] Pin test slicer to repo-local runtime (#12706)

    ea3b51b

    [tests] use embedded jars for desugar test (#12710)

    18479db
  4. simonrozsival
    added a commit that references this issue on Sep 10, 2026

    [flaky-ci] Isolate JavaSideActivation GREF leak check (#12716)

    5986770
  5. simonrozsival
    closed this as completedin #12711on Sep 10, 2026
  6. simonrozsival
    added 2 commits that reference this issue on Sep 10, 2026

    [flaky-ci] Stabilize BenchmarkDotNet device test results (#12718)

    4e78319

    [flaky-ci] Classify pre-TLS failures in trusted certificate test (#12714

    263fe5c
  7. pull
    added a commit that references this issue on Sep 10, 2026

    [flaky-ci] Retry transient ADB install failures (dotnet#12711)

    b03157a
  8. simonrozsival
    added 4 commits that reference this issue on Sep 15, 2026

    [flaky-ci] Stabilize AndroidMessageHandler cancellation rewind test (#…

    4f82a63

    [flaky-ci] Isolate TryFindClass JNI leak checks (#12709)

    66a289b

    [flaky-ci] Make logcat capture bounded and non-gating (#12708)

    b9dfd0c

    [flaky-ci] Isolate Gradle state in CI (#12707)

    52d36ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageIssues that need to be assigned.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions