[flaky-ci] Isolate Gradle state in CI by simonrozsival · Pull Request #12707 · dotnet/android

simonrozsival · 2026-09-08T09:00:21Z

Summary

give each CI job a clean GRADLE_USER_HOME under $(Agent.TempDirectory) while caching only Maven dependencies and Gradle wrapper distributions
order java-source-utils before manifestmerger, extending the existing manifestmerger → r8 → proguard-android chain so shared-wrapper Gradle invocations do not overlap
preserve high-importance Gradle output for the previously opaque Xamarin.Android.LibraryProjectZip-LibBinding failure

Root cause

The nine Windows incidents in builds 1574227, 1574359, 1575505, 1576382, 1576423, 1576487, 1576567, 1578693, and 1578694 all failed while manifestmerger and java-source-utils ran concurrently. Although both use --no-daemon, Gradle starts a single-use daemon and both processes shared C:\Users\cloudtest\.gradle\daemon\9.5.0. The failing stack ends in daemon registry removal and file-lock contention handling with Timeout waiting to connect to the Gradle daemon.

Build 1575255 has a separate macOS Kotlin ClasspathEntrySnapshotTransform/KotlinCompilerVersion failure and is intentionally not covered by a generic retry. Build 1576677 did not retain Gradle stdout/stderr, so this change improves that target's diagnostics without assuming the same cause.

References #12704.

Approach

The pipeline now creates a fresh Gradle user home for each job. Only caches/modules-2 and wrapper/dists are restored across jobs; mutable transforms, project caches, and daemon registry state remain job-local. Cache failures continue to fail normally rather than being recovered or allowlisted.

The new ProjectReference is intentionally an ordering dependency rather than an assembly dependency. It ensures java-source-utils finishes before manifestmerger; the existing project references then order manifestmerger, r8, and proguard-android.

Validation

parsed the changed Azure YAML and MSBuild XML files
evaluated the manifestmerger project references through MSBuild
exercised the Gradle cache-directory setup with PowerShell on macOS
built java-source-utils with a cold isolated Gradle home and reran it warm
built manifestmerger twice with the same isolated Gradle home
verified the binding project preprocesses with high-importance Gradle stdout and stderr

Prevent concurrent single-use Gradle daemons from sharing the manifestmerger/java-source-utils registry, keep pipeline Gradle state job-local, and persist only dependency downloads and wrapper distributions. Preserve high-importance Gradle output for the previously opaque binding fixture failure. Co-authored-by: Copilot App <[email protected]>

Use missing-safe cache state recovery and inspect the current Azure job timeline so only explicitly recovered optional cache failures remain non-gating. Co-authored-by: Copilot App <[email protected]>

Poll the current-job timeline within a wall-clock bound before applying recovered optional-task exemptions, so delayed task results cannot be masked. Co-authored-by: Copilot App <[email protected]>

Use HttpClient full-response cancellation against the remaining wall-clock budget and reject overdue or unknown timeline results. Co-authored-by: Copilot App <[email protected]>

simonrozsival · 2026-09-08T14:24:53Z

/review

github-actions · 2026-09-08T14:26:50Z

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12707

github-actions

❌ Reject

Found 2 errors:

Nested issue gates do not preserve the reusable templates’ custom xaSourcePath, causing script-not-found failures for non-default checkouts.
The new HTTP integration test expects Azure log masking in raw loopback request bytes and therefore fails when run.

CI is green for Azure DevOps build 1587136. The Gradle cache isolation and fail-closed timeline logic are otherwise thoughtfully defensive, with good coverage of delayed and malformed timeline responses.

Generated by Android PR Reviewer for #12707 · gpt56 · 200.8 AIC · ⌖ 19.5 AIC · ⊞ 25.7K
_{Comment /review to run again}

Preserve custom checkout paths in nested test templates and parse the actual authorization header in the loopback fixture. Co-authored-by: Copilot App <[email protected]>

simonrozsival · 2026-09-08T20:35:33Z

/review

github-actions · 2026-09-08T20:37:17Z

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12707

github-actions

⚠️ Needs Changes

Findings: 0 errors, 0 warnings, 1 suggestion.

The Gradle state isolation, cache-recovery allowlist, timeline gate, and project-reference serialization are coherent with the surrounding build graph. I left one inline suggestion to cover the explicit non-success HTTP response path.

CI is currently red: build 1587818 failed in the Linux Build, macOS Build, and Windows Build & Smoke Test lanes. The available check metadata does not expose enough Azure log detail to attribute those failures to this change, but the PR is not merge-ready until those lanes are green.

Generated by Android PR Reviewer for #12707 · gpt56 · 240.9 AIC · ⌖ 8.84 AIC · ⊞ 25.7K
_{Comment /review to run again}

Co-authored-by: Copilot App <[email protected]>

Exercise the explicit non-success response path in the fail-closed Azure timeline gate fixture. Co-authored-by: Copilot App <[email protected]>

Copilot

Copilot review overview

🟡 Changes recommended

The newly added PowerShell test fixtures for the gate/recovery logic are not currently wired to run in CI, reducing confidence that future edits won’t reintroduce failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 · 2

New issues introduced by this change (3)

Severity	Finding
	`build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1` — ⚠️ These PowerShell test fixtures don’t appear to be executed anywhere in the pipeline (no YAML…
	`build-tools/automation/yaml-templates/cache-gradle.yaml` — 💡 Similar to `fail-on-issue.yaml`, this direct path concatenation can be fragile if `xaSourcePath`…
	`build-tools/automation/yaml-templates/fail-on-issue.yaml` — 💡 Using string concatenation with `.../${{ parameters.xaSourcePath }}/...` is a bit brittle across…

What changed in this PR

This PR aims to reduce CI flakiness by isolating Gradle state per job (while still caching the “download-only” bits) and by adding a stricter, timeline-based gate that only allows known-recovered optional cache failures to remain non-gating.

Changes:

Isolates GRADLE_USER_HOME under $(Agent.TempDirectory) and caches only caches/modules-2 plus wrapper distributions, with recovery when cache restore is incomplete.
Serializes Gradle-wrapper–invoking projects by extending the project reference chain (preventing concurrent single-use daemons from racing).
Improves diagnostics by preserving high-importance Gradle stdout/stderr for the binding zip test target and by replacing the simple “SucceededWithIssues => fail” gate with a timeline-aware allowlist gate.

File	Description
tests/CodeGen-Binding/Xamarin.Android.LibraryProjectZip-LibBinding/Xamarin.Android.LibraryProjectZip-LibBinding.targets	Elevates Gradle stdout/stderr importance to improve failure diagnostics.
src/manifestmerger/manifestmerger.csproj	Adds a project reference to serialize Gradle wrapper usage with `java-source-utils`.
build-tools/automation/yaml-templates/variables.yaml	Sets job-level `GRADLE_USER_HOME` default under `$(Agent.TempDirectory)`.
build-tools/automation/yaml-templates/stage-msbuild-emulator-tests.yaml	Plumbs `xaSourcePath` through to the issue gate template.
build-tools/automation/yaml-templates/run-msbuild-tests.yaml	Plumbs `xaSourcePath` through to the issue gate template.
build-tools/automation/yaml-templates/fail-on-issue.yaml	Replaces inline gating with `FailOnUnexpectedIssues.ps1` and provides OAuth token env.
build-tools/automation/yaml-templates/cache-gradle.yaml	Creates isolated Gradle home, caches only download artifacts, and invokes recovery validation.
build-tools/automation/yaml-templates/build-linux-steps.yaml	Updates cleanup commentary to reflect new Gradle home location.
build-tools/automation/scripts/RecoverGradleCache.ps1	Resets cache dirs to a verified-empty state when cache restore state is missing/invalid.
build-tools/automation/scripts/RecoverGradleCache.Tests.ps1	Adds fixtures validating cache-reset and allowlist emission behavior.
build-tools/automation/scripts/FailOnUnexpectedIssues.ps1	Implements timeline-based gate to fail closed unless only allowlisted recovered tasks are in issue state.
build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1	Adds fixtures validating timeline polling, allowlisting, and fail-closed behavior.
build-tools/automation/azure-pipelines.yaml	Passes `xaSourcePath` to the gate template for correct script resolution.
build-tools/automation/azure-pipelines-public.yaml	Passes `xaSourcePath` to the gate template for correct script resolution.
build-tools/automation/azure-pipelines-internal.yaml	Passes `xaSourcePath` to the gate template for correct script resolution.

Execute the PowerShell regression fixtures once in the Linux build and resolve script paths with Join-Path across platforms. Co-authored-by: Copilot App <[email protected]>

Co-authored-by: Copilot App <[email protected]>

Keep Gradle state job-local and cache only dependency and wrapper downloads. Remove cache recovery and timeline-gating machinery. Co-authored-by: Copilot App <[email protected]>

Describe the artificial java-source-utils to manifestmerger dependency without implying that one reference establishes the entire Gradle build chain. Co-authored-by: Copilot App <[email protected]>

Fail manifestmerger builds deterministically if the artificial java-source-utils ordering reference is removed. Co-authored-by: Copilot App <[email protected]>

simonrozsival · 2026-09-21T15:20:51Z

/review

github-actions · 2026-09-21T15:22:33Z

❌ Android PR Reviewer failed. Please review the logs for details.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Generated by Android PR Reviewer for #12707

jonathanpeppers · 2026-09-21T18:53:57Z

+
+  <Target Name="_ValidateGradleProjectOrdering" BeforeTargets="_BuildManifestMerger">
+    <Error
+        Condition=" '@(ProjectReference->WithMetadataValue('Filename', 'java-source-utils')->Count())' == '0' "
+        Text="manifestmerger must reference java-source-utils to serialize their Gradle invocations."
+    />
+  </Target>


I think you can probably remove this target and just rely on the ProjectReference

jonathanpeppers

Just the one comment, but it's not blocking.

simonrozsival added the flaky-ci Intermittent CI failures and work to improve CI reliability label Sep 8, 2026

simonrozsival and others added 3 commits September 8, 2026 13:06

Harden recovered Gradle cache handling

7ce2270

Use missing-safe cache state recovery and inspect the current Azure job timeline so only explicitly recovered optional cache failures remain non-gating. Co-authored-by: Copilot App <[email protected]>

Wait for complete Azure task timelines

277f95b

Poll the current-job timeline within a wall-clock bound before applying recovered optional-task exemptions, so delayed task results cannot be masked. Co-authored-by: Copilot App <[email protected]>

Bound Azure timeline response reads

7ccc1e3

Use HttpClient full-response cancellation against the remaining wall-clock budget and reject overdue or unknown timeline results. Co-authored-by: Copilot App <[email protected]>

github-actions Bot reviewed Sep 8, 2026

View reviewed changes

Comment thread build-tools/automation/yaml-templates/fail-on-issue.yaml Outdated

Comment thread build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1 Outdated

Forward source paths to issue gates

1cabcf4

Preserve custom checkout paths in nested test templates and parse the actual authorization header in the loopback fixture. Co-authored-by: Copilot App <[email protected]>

github-actions Bot reviewed Sep 8, 2026

View reviewed changes

Comment thread build-tools/automation/scripts/FailOnUnexpectedIssues.ps1 Outdated

simonrozsival and others added 2 commits September 9, 2026 11:09

Merge origin/main into Gradle CI isolation

64c76ab

Co-authored-by: Copilot App <[email protected]>

Cover timeline HTTP status failures

b511a6b

Exercise the explicit non-success response path in the fail-closed Azure timeline gate fixture. Co-authored-by: Copilot App <[email protected]>

simonrozsival marked this pull request as ready for review September 10, 2026 10:41

simonrozsival requested a review from jonathanpeppers as a code owner September 10, 2026 10:41

Copilot AI lite review requested due to automatic review settings September 10, 2026 10:41

Copilot started reviewing on behalf of simonrozsival September 10, 2026 10:41 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread build-tools/automation/scripts/FailOnUnexpectedIssues.Tests.ps1 Outdated

Comment thread build-tools/automation/yaml-templates/cache-gradle.yaml Outdated

Comment thread build-tools/automation/yaml-templates/fail-on-issue.yaml Outdated

simonrozsival and others added 5 commits September 10, 2026 15:11

Run cache recovery fixtures in CI

0b0359b

Execute the PowerShell regression fixtures once in the Linux build and resolve script paths with Join-Path across platforms. Co-authored-by: Copilot App <[email protected]>

Merge origin/main into Gradle CI isolation

b92f80a

Co-authored-by: Copilot App <[email protected]>

[ci] Simplify Gradle state isolation

b86dfbb

Keep Gradle state job-local and cache only dependency and wrapper downloads. Remove cache recovery and timeline-gating machinery. Co-authored-by: Copilot App <[email protected]>

Clarify Gradle project ordering

004e7ff

Describe the artificial java-source-utils to manifestmerger dependency without implying that one reference establishes the entire Gradle build chain. Co-authored-by: Copilot App <[email protected]>

Guard Gradle project ordering

f1dc64d

Fail manifestmerger builds deterministically if the artificial java-source-utils ordering reference is removed. Co-authored-by: Copilot App <[email protected]>

github-actions Bot added the needs-review label Sep 21, 2026

simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Sep 21, 2026

jonathanpeppers reviewed Sep 21, 2026

View reviewed changes

jonathanpeppers approved these changes Sep 21, 2026

View reviewed changes

simonrozsival merged commit 52d36ee into main Sep 22, 2026
44 checks passed

simonrozsival deleted the simonrozsival-gradle-ci-state-isolation branch September 22, 2026 12:02

simonrozsival mentioned this pull request Sep 26, 2026

CI flakiness inventory: Sep 19-26, 2026 (follow-up to #12704) #12929

Open

5 tasks

Conversation

simonrozsival commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root cause

Approach

Validation

Uh oh!

simonrozsival commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

❌ Reject

Uh oh!

Uh oh!

Uh oh!

simonrozsival commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

⚠️ Needs Changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonrozsival commented Sep 21, 2026

Uh oh!

github-actions Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jonathanpeppers Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

jonathanpeppers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simonrozsival commented Sep 8, 2026 •

edited

Loading

github-actions Bot commented Sep 8, 2026 •

edited

Loading

github-actions Bot commented Sep 8, 2026 •

edited

Loading

github-actions Bot commented Sep 21, 2026 •

edited

Loading