Skip to content

[MSBuild] Add retained typemap rule generation pipeline - #12827

Merged
simonrozsival merged 16 commits into
mainfrom
simonrozsival-typemap-msbuild-pipeline
Oct 4, 2026
Merged

simonrozsival merged 16 commits into
mainfrom
simonrozsival-typemap-msbuild-pipeline

Conversation

@simonrozsival

@simonrozsival simonrozsival commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Why

Layer 3 of 4 in native stack #12893, depending on #12825. The retained typemap adapters need a shared incremental MSBuild pipeline that consumes exact producer outputs rather than discovering stale or pre-trim candidates.

Changes

  • Add Microsoft.Android.Sdk.TypeMap.Proguard.targets with shared key extraction, class/member rule generation, NativeAOT opt-in, complete-configuration override handling, and incremental output tracking.
  • Return exact per-RID native-object and LLVM tool paths plus the complete linked typemap inventory from the assembly-resolution inner build.
  • Expand the standalone TypeMapProguardTests to exercise actual production tasks and explicit imports of the new pipeline.

The pipeline is available but not imported by product targets until Layer 4. This layer does not change existing R8/D8 configuration, resource flags, legacy typemap gates, or DGML behavior. All content is carved from the frozen feature snapshot; only the three assigned files change.

Validation

36 passed, 0 failed, 0 skipped:

dotnet test tests/Microsoft.Android.Sdk.TrimmableTypeMap.Tests/Microsoft.Android.Sdk.TrimmableTypeMap.Tests.csproj \
  --filter 'FullyQualifiedName~TypeMapProguardTests' \
  -p:_AndroidTreatWarningsAsErrors=true \
  -p:_NativeAotLlvmReadObjPath=<NDK>/toolchains/llvm/prebuilt/<host>/bin/llvm-readobj

Coverage includes real native object groups, RID unions, linked DLLs and empty stubs, runtime/mode/input changes, missing files, deleted outputs, relative task-assembly paths, complete overrides, and exact producer metadata. The run used the existing NDK llvm-readobj and adjacent llvm-objdump/clang, without a full SDK build or new tooling.

Three frozen test methods are temporarily deferred because they require the existing-target edits in Layer 4:

  • PlatformConfigurationSeparatesCoreClrOptimizationFromObfuscation
  • TypemapInputsDoNotRequestGraphsOrChangeParallelism
  • NdkDependencyRequiresNativeObjectOptIn

Restoring those methods in Layer 4 makes the test file identical to the frozen source.

Copilot AI lite review requested due to automatic review settings September 18, 2026 15:25
@simonrozsival simonrozsival added the r8-rules Typemap-derived ProGuard and R8 rules label Sep 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Generated typemap outputs are missing from Dalvik inputs, risking stale dex output, and manifest generation is not incremental.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity

Open (1)
What changed in this PR

Adds a retained typemap ProGuard generation pipeline with producer metadata propagation and integration coverage.

Changes:

  • Adds shared key extraction and ProGuard rule-generation targets.
  • Propagates exact per-RID producer and tool paths.
  • Expands pipeline and incremental-build tests.
File Summary
tests/​Microsoft.Android.Sdk.TrimmableTypeMap.Tests/​Generator/​TypeMapProguardTests.cs Tests production tasks and MSBuild integration.
src/​Xamarin.Android.Build.Tasks/​Microsoft.Android.Sdk/​targets/​Microsoft.Android.Sdk.TypeMap.Proguard.targets Defines the shared extraction and rule-generation pipeline.
src/​Xamarin.Android.Build.Tasks/​Microsoft.Android.Sdk/​targets/​Microsoft.Android.Sdk.AssemblyResolution.targets Returns producer metadata from inner builds.

@simonrozsival
simonrozsival added this pull request to stack #12830 September 18, 2026 19:14
@simonrozsival
simonrozsival force-pushed the simonrozsival-typemap-msbuild-pipeline branch from c84b518 to 0742417 Compare September 21, 2026 15:42
@simonrozsival

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@simonrozsival
simonrozsival force-pushed the simonrozsival-typemap-msbuild-pipeline branch from 0742417 to 51fe93d Compare September 21, 2026 15:55
@simonrozsival

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@simonrozsival
simonrozsival force-pushed the simonrozsival-typemap-msbuild-pipeline branch from 51fe93d to ee4d1b9 Compare September 22, 2026 22:02
@simonrozsival
simonrozsival removed this pull request from stack #12830 September 23, 2026 07:16
@simonrozsival
simonrozsival force-pushed the simonrozsival-typemap-msbuild-pipeline branch from ee4d1b9 to 4aa80bc Compare September 23, 2026 07:43
@simonrozsival
simonrozsival added this pull request to stack #12893 September 23, 2026 07:45
Base automatically changed from simonrozsival-typemap-nativeaot-object-adapter to main September 26, 2026 14:57
@simonrozsival
simonrozsival force-pushed the simonrozsival-typemap-msbuild-pipeline branch from c0f720c to c0f8419 Compare September 26, 2026 15:01
@simonrozsival

Copy link
Copy Markdown
Member Author

@dalexsoto review

@simonrozsival simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Sep 26, 2026

@dalexsoto dalexsoto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two issues in the staged typemap pipeline need fixing before it can be imported:

  1. src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.TypeMap.Proguard.targets:4,58-60,99-102: a trimmed R8 CoreCLR build with AndroidTypeMapImplementation=llvm-ir selects ExtractTypeMapKeysFromLlvmIr, but no task by that name is compiled into $(_MicrosoftAndroidBuildTasksAssembly) at this head. This path fails with MSB4036 instead of generating typemap keys. Please supply and test the extractor, or keep this branch inactive until it exists.

  2. src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.TypeMap.Proguard.targets:37-45: the NativeAOT enabled and disabled targets share proguard_project_references.cfg. On a true -> false change to _AndroidEnableTypemapR8Trimming, the ACW map and production _AndroidBuildPropertiesCache do not change (the cache does not record this flag). The fallback therefore skips as up-to-date and retains the narrower enabled-mode rules, defeating the opt-out. The new transition test masks this by using a synthetic cache that does record the flag. Please include the mode in fallback invalidation and test against production-equivalent cache contents.

I treated the product import and Dalvik/R8 wiring as intentionally deferred to Layer 4; these are defects in the staged pipeline itself.

@simonrozsival

Copy link
Copy Markdown
Member Author

@dalexsoto Thanks for catching both of these. They are addressed now:

  1. I removed the LLVM-IR path instead of adding the missing extractor. The ProGuard pipeline no longer registers or invokes ExtractTypeMapKeysFromLlvmIr, no longer selects an llvm-ir key kind, and no test case selects the llvm-ir representation. The pipeline is now limited to the trimmable CoreCLR and NativeAOT representations.

  2. The NativeAOT legacy fallback now has a dedicated typemap.legacy-proguard.inputs state file containing _AndroidEnableTypemapR8Trimming, written with WriteOnlyWhenDifferent. That file is part of the fallback target inputs, so a true to false or unset transition invalidates proguard_project_references.cfg even when the production build-properties cache is unchanged. The test fixture cache is now constant production-like content, so the transition test no longer masks the issue.

The fixes are in 8a969c8fb4 and 57aabea892.

@simonrozsival

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12827

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Needs Changes

Found 1 error and 1 warning in the new MSBuild pipeline: the complete ProguardConfigFiles override still executes an unused typemap generation path, and the member-rule target uses AfterTargets despite an available dependency extension point.

The producer metadata flow and focused incremental coverage are otherwise well designed. All 44 reported CI checks are green.

Generated by Android PR Reviewer for #12827 · copilot · gpt56 · 139.9 AIC · ⌖ 14 AIC · ⊞ 26.5K
Comment /review to run again

simonrozsival and others added 6 commits September 30, 2026 09:16
Skip generated typemap rules when custom ProGuard configuration replaces them, and order member generation through the Dalvik dependency chain.

Co-authored-by: Copilot App <[email protected]>
Generate scoped member rules for NativeAOT and share mode-state inputs between modern and legacy ProGuard writers so mode round trips cannot reuse stale rules.

Co-authored-by: Copilot App <[email protected]>
@simonrozsival
simonrozsival force-pushed the simonrozsival-typemap-msbuild-pipeline branch from ef2c47c to ea47cc1 Compare September 30, 2026 07:16
@simonrozsival simonrozsival removed the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Sep 30, 2026
Use the layout-aware Android SDK tools directory so the NativeAOT R8 regression locates build tasks in both local and CI workload installations.

Co-authored-by: Copilot App <[email protected]>
@simonrozsival simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Sep 30, 2026
@jonathanpeppers

Copy link
Copy Markdown
Member

/review

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

✅ Android PR Reviewer completed successfully!

Generated by Android PR Reviewer for #12827

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM

Findings: 0 errors · 0 warnings · 1 suggestion

The pipeline correctly keeps exact producer outputs, runtime/mode state, custom-configuration gating, and Dalvik incrementality aligned. The new target is intentionally not imported by product targets in this layer, matching the PR description. All 44 reported checks are green. I left one non-blocking inline suggestion to add equivalent end-to-end CoreCLR/R8 coverage for the NativeAOT constructor-retention regression test.

Generated by Android PR Reviewer for #12827 · copilot · gpt56 · 150 AIC · ⌖ 11.2 AIC · ⊞ 26K
Comment /review to run again

@jonathanpeppers

Copy link
Copy Markdown
Member

@dalexsoto review

simonrozsival and others added 2 commits October 1, 2026 21:39
Migrate retained typemap target regressions into the existing NUnit build-task suite after main removed the standalone test project, preserving native-object and incremental-build coverage.

Co-authored-by: Copilot App <[email protected]>
Include the latest legacy-tooling removal after the prepared main merge; retained typemap target and ProGuard generator regressions pass against the updated dependency layout.

Co-authored-by: Copilot App <[email protected]>

@dalexsoto dalexsoto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previously reported extractor, NativeAOT member-retention, mode-invalidation, override and ordering concerns are resolved in current source. Product import and the stated Layer-4-dependent edits/tests remain accepted staging, not blockers. The complete eight-file review and separate integration pass identify one defect in the explicitly imported CoreCLR pipeline.

Keep linked-assembly extraction inactive when CoreCLR ILLink is disabled (TypeMap.Proguard.targets:125-128). With CoreCLR, PublishTrimmed=true, AndroidLinkTool=r8 and RunILLink=false, the new activation still invokes linked-DLL extraction, and the new inner-build inventory condition:85-91 still advertises typemap DLLs under $(IntermediateOutputPath)linked.

The repository's pinned SDK linker target excludes RunILLink=false, so on clean intermediates these linked DLLs are not produced. The existing CoreCLR publish adapter:317-320 can publish generated typemaps without linked copies; that does not create the linked paths advertised by this new inventory. Assembly extraction:33-44 opens those paths and reports missing files as XA4327, failing this non-linking configuration instead of leaving extraction inactive.

Exclude RunILLink=false from the CoreCLR activation and linked-inventory conditions and add a clean-intermediates case. Keep strict missing-input validation and exact linked producer contracts; do not substitute stale/pre-trim discovery. Do not apply this exclusion to NativeAOT, where ILC performs trimming.

This activation/producer mismatch is established by immutable source tracing, not an executed MSBuild reproduction. The observed CI failure in build1620088 is separately verified as a submodule checkout receive timeout before build/tests, and is not the reason for this source finding.

Gate linked typemap inventory and CoreCLR rule activation on RunILLink, preserve NativeAOT extraction, and cover both runtimes' JNI-only constructor retention through R8.

Co-authored-by: Copilot App <[email protected]>
@simonrozsival

Copy link
Copy Markdown
Member Author

@dalexsoto Fixed the disabled-linker activation/producer mismatch in da429bbb5e.

Both CoreCLR activation conditions (_AndroidConfigureTypeMapProguard and the class-generation entry target) now require RunILLink != false, and the inner-build linked typemap inventory uses the same guard. NativeAOT is unchanged: ILC extraction and exact native-object metadata remain active even with RunILLink=false. Missing-input validation remains strict; no pre-trim/stale-file fallback was added.

I reproduced the defect with clean-output target fixtures before changing the production conditions: both CoreCLR activation cases and the disabled-linker inventory case failed. They now pass, asserting that no keys, manifests, class/member rules, or modern-policy activation are produced despite missing linked DLLs and a deliberately missing task assembly. Producer tests cover blank/true/false linker states, and a real NDK-backed NativeAOT case proves RunILLink=false does not disable native extraction.

Validation: 46 focused target/generator cases passed with 0 skips. The separate CoreCLR/R8 coverage suggestion is addressed too: both CoreCLR and NativeAOT end-to-end builds pass and retain the JNI-only (I)V constructor in classes.dex.

@dalexsoto dalexsoto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The disabled-ILLink activation/producer mismatch is fixed, and the earlier NativeAOT retention, mode-state, override and ordering findings remain resolved. The declared Layer-3 product-import deferral is accepted. The complete current eight-file review and separate integration/completeness pass identify one remaining incremental-contract blocker.

Track CoreCLR opt-out as a Dalvik input change (TypeMap.Proguard.targets:125-129). After an enabled trimmed CoreCLR/R8 build, a no-clean rebuild with _AndroidEnableTypemapR8Trimming=false skips the modern configuration/class/member targets before their dependencies. The effective configuration/file list changes, but the only mode-file writer is NativeAOT-only (lines 26-38). CoreCLR therefore does not update a policy sentinel or register an inactive-mode change with Dalvik.

The production properties cache does not record this flag, and the resolved-assembly hash ignores changed producer metadata. The real Dalvik inputs contain user @(ProguardConfiguration), not the internal selected configuration list; removing the modern input registrations does not invalidate the existing timestamp-based Dalvik stamp. When the remaining producer/consumer inputs retain their timestamps, R8 can skip and leave classes.dex from the previously selected keep configuration. The baseline CoreCLR configuration writer can also remain up to date; this finding does not assume that opt-out necessarily generates fresh legacy contents. This is pinned static source proof, not an executed rebuild or a claim that every flag flip skips R8.

Persist the effective CoreCLR policy mode even when modern extraction is inactive, include that state file in Dalvik inputs, and add a no-clean enabled/disabled/enabled consumer regression using unchanged linked DLLs and production-like cache/input contracts. Do not invoke inactive extraction/tasks or require Layer-4 product imports to fix this staged-pipeline contract.

Review-completeness correction: this gap already existed at the previously reviewed 2b4d048e0fb03db209b38399b6271c6eb7599908 head and was missed in the earlier complete review. It was not introduced by da429; the separate disabled-ILLink fix is valid.

Persist the effective CoreCLR ProGuard policy independently of active extraction and include it in Dalvik inputs. Cover no-clean opt-out, disabled ILLink, and custom configuration round trips with unchanged linked assemblies and cache timestamps.

Co-authored-by: Copilot App <[email protected]>
@simonrozsival

Copy link
Copy Markdown
Member Author

@dalexsoto Fixed the CoreCLR opt-out Dalvik invalidation gap in 32bfcc7875.

A new CoreCLR policy-state writer runs directly through _CompileToDalvikDependsOnTargets, independently of the conditional modern generation targets. It records the effective _AndroidUseTypeMapProguardConfiguration value in typemap.proguard-mode.inputs, preserves its timestamp with WriteOnlyWhenDifferent, registers it in FileWrites, and always includes it in the CoreCLR outer Dalvik input contract. It depends only on policy configuration—not Java stubs, linked inputs, or extraction/tasks—so opt-out and other inactive policies still invalidate the consumer without invoking modern tasks. Active class generation also consumes this state input. NativeAOT retains its existing shared mode-state behavior.

I reproduced the reported defect before changing the target: the no-clean enabled-to-disabled invocation kept the previous Dalvik stamp. The regression now covers enabled/disabled/enabled with unchanged linked DLL and production-like cache timestamps, asserts the consumer's selected policy/member list on each transition, and checks that repeated same-mode invocations preserve both the policy-file timestamp and the Dalvik stamp. Disabled invocations use a deliberately missing modern task assembly and verify the key output remains unchanged. The same round-trip checks also cover RunILLink=false and the complete ProguardConfigFiles override.

Validation: 49 focused target/generator cases passed, 0 failed or skipped, including the existing NDK-backed NativeAOT cases. Target XML and git diff --check pass. No Layer-4 product imports or stale/pre-trim discovery were added.

@dalexsoto dalexsoto left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CoreCLR opt-out invalidation blocker is resolved: the policy-state writer now runs directly in the outer Dalvik dependency chain even when modern generation is inactive, updates a surviving consumer input with write-only-when-different semantics, and registers it for Clean. Re-enabling policy also invalidates class generation; disabled-linker/full-override paths avoid modern task invocation, and NativeAOT's shared-state behavior is preserved.

The complete eight-file/current-hunk review, separate producer-to-actual-Dalvik integration pass and final additional-blocker sweep found no remaining high-confidence source blocker. The declared Layer-3/Layer-4 staging and previously accepted policy decisions remain accepted. This is immutable source/contract review,not independently executed private R8,NDK,build or device validation.

@simonrozsival
simonrozsival merged commit 9614a8f into main Oct 4, 2026
44 checks passed
@simonrozsival
simonrozsival deleted the simonrozsival-typemap-msbuild-pipeline branch October 4, 2026 20:43
simonrozsival added a commit that referenced this pull request Oct 7, 2026
## Summary

Layer 4 of 4 in native stack #12893, depending on #12827 (`simonrozsival-typemap-msbuild-pipeline`).

Activate the shared retained-typemap rule pipeline so eligible CoreCLR and explicitly opted-in NativeAOT R8 builds derive Java class roots from retained typemap keys rather than keeping every ACW. This removes the dependency on large NativeAOT DGML graphs while preserving runtime/JNI roots and user-authored Java source retention.

- Import the shared MSBuild pipeline, wire R8 flags/resources and incremental inputs, and gate legacy configuration generation. CoreCLR keeps scoped JNI-facing members and optimization while disabling renaming; NativeAOT retains class-wide member preservation and remains opt-in for NDK-backed object inspection.
- Remove the old DGML producer, resource, diagnostics implementation, and graph-specific test fixtures; retain the interface-collection runtime test.
- Add R8, build, and scoped-member device coverage; restore the three deferred `TypeMapProguardTests` methods and their attributes (17 cases); document retention behavior and retired diagnostics.

## Validation

```sh
dotnet test tests/Microsoft.Android.Sdk.TrimmableTypeMap.Tests/Microsoft.Android.Sdk.TrimmableTypeMap.Tests.csproj \
  -v minimal \
  --filter 'FullyQualifiedName~TypeMapProguardTests|FullyQualifiedName~ExtractTypeMapKeys' \
  -p:_AndroidTreatWarningsAsErrors=true \
  -p:_NativeAotLlvmReadObjPath="$NDK_LLVM_BIN/llvm-readobj"
```

Passed **294/294**, **0 skipped**: 53 pipeline/generator cases (including all 17 restored cases) and 241 retained-key adapter cases. Used the supplied existing Android NDK `llvm-readobj` and adjacent `llvm-objdump`/`clang` executables read/execute-only.

All six changed XML targets/resources parse successfully with `xmllint --nonet --noout`, including XML comments. `git diff --check` passes.

**Not run:** `Xamarin.Android.Build.Tests` R8/build cases and `MSBuildDeviceIntegration` device cases. This worktree has neither a local built Android SDK (`bin/Debug/dotnet/dotnet` or Release) nor the corresponding test assemblies. No full SDK bootstrap or new tooling installation was performed for this frozen split.

## Frozen split integrity

Exactly 28 paths, +601/-1081, relative to #12827. The final tree is exactly `168821457836836b55368bb7ad53c2bfd9833cb4`: frozen source `87bd3f8a8b512859339e5446c1f86709a6c28c3b` with the base XA1037 documentation/localization/JavaSourceUtils changes preserved. `TypeMapProguardTests.cs` is byte-identical to the frozen source. No non-English localization files changed.

Native stack #12893 is registered with #12824, #12825, #12827, and this PR in order. The closed LLVM adapter #12823 is not in this stack.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r8-rules Typemap-derived ProGuard and R8 rules ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants