[Microsoft.Android.Build.Tasks] Write assembly-store ELF libraries in managed code by simonrozsival · Pull Request #12962 · dotnet/android

simonrozsival · 2026-09-30T22:22:37Z

Part of #12940.

Assembly stores need valid, ABI-specific ELF libraries so Android can package them under lib/<abi>/libassembly-store.so and Google Play can split an AAB by ABI. Today, producing these data-only libraries invokes .S -> llvm-mc -> .o -> ld. Write them directly in managed code instead, avoiding those external processes for store wrapping without changing the raw store format or packaging contract.

Implementation

Add the framework-only Microsoft.Android.Tasks.AssemblyStoreElfWriter and a uniquely named <WrapAssemblyStoresAsSharedLibraries/> task in the modern .NET build-task assembly. Neither the writer nor its focused test project adds a managed ELF-parser dependency.
Produce little-endian ELF32/ELF64 ET_DYN images with one read-only load segment, read-only dynamic metadata, a non-executable stack, and a single default-visible global object symbol, _assembly_store. Preserve ARM EABI5/base-softfp flags and 16 KiB alignment for 64-bit ABIs / 4 KiB for 32-bit ABIs.
Stream-copy store bytes unchanged. Keep section names and section headers after the payload, outside PT_LOAD, so stripping can rebuild non-allocated metadata safely.
Dispatch stores unconditionally to the modern task using the existing store setting, with no LLVM fallback or new feature switch. Preserve the existing incremental packaging gate, file tracking, cleanup, and the legacy discrete assembly/PDB/config wrapper path.

Trusted native store contract

Current main already resolves only _assembly_store and calls configure_from_payload(pointer, path). This PR preserves that API and all existing runtime behavior and checks; the native source delta is only a comment clarifying the contract.

The XABA v3 payload is intentionally trusted build-generated data. There is no _assembly_store_end, external payload length, replacement size field, or runtime ELF-header walk. The standard ELF symbol size describes the payload for inspection only; the native loader does not consume it. This is not an untrusted-input parsing boundary.

Standalone scope

Updated by a normal merge of main at c35205f1db840afb260678efaf83ef034afec51e, including the upstream Debug-scanner fix in #12976. #12895 is not a prerequisite.

Other application-specific LLVM generation and bundled tool usage remain intact; this does not eliminate all CoreCLR LLVM/binutils requirements. The writer diff has no bootstrap/environment/runtime-property changes, runtime-owned decompression/state changes, DSO-cache or JNI-remapping changes, typemap/marshal removal, libxamarin-app.so removal, NDK/linker-policy changes, or tool-distribution pruning.

NDK-based inspection tests

Use the official Android NDK's llvm-readobj, llvm-nm, llvm-strip, and llvm-objcopy, without adding ELFSharp to the modern test project. A small test-only helper is shared with the APK/AAB regression tests; no native tools are invoked by the managed writer. Existing ELFSharp consumers elsewhere in the repository are unchanged.

llvm-readobj supplies ELF header/section/symbol/program-header metadata, and llvm-objcopy extracts payload, dynamic-table, string-table, and hash bytes. The independent symbol-and-segment extraction checks remain, including before/after stripping. Extraction uses a disposable output copy so the original library is not rewritten.

Resolve the toolchain on the executing test host, in this order: TEST_ANDROID_NDK_PATH, ANDROID_NDK_LATEST_HOME, that user's android-toolchain/ndk installation, then a build-time AndroidNdkDirectory only if it still exists. This follows the existing test discovery convention and supports DLLs built on macOS and transferred to Windows/Linux.

The four packaging-test failures in build 1620483 selected the embedded macOS builder path, /Users/cloudtest/android-toolchain/ndk. The failing Linux and Windows agents had already successfully installed NDK r28c in the existing setup step before testing. The fix uses those installations; it does not add a CI download/install step or weaken inspection assertions.

Native inspection cases remain tagged RequiresAndroidNdk. An absent or explicitly invalid toolchain still fails with a diagnostic; the stream/input-validation cases need no native tools. Five new discovery regressions cover runtime override, executing-host precedence over even an existing builder path, stale builder paths, explicit invalid configuration, and a valid local build fallback.

Bounded process capture

The .NET 11 native-tool test runner uses Process.RunAndCaptureText to drain stdout and stderr together, with one 30-second timeout covering output capture and process exit. Timeout exceptions and canceled exit status are failures, not successful partial results. There are no unbounded manual exit waits.

The helper is also linked into the existing net10.0 packaging tests, where the new API is unavailable. That build uses cancellation-aware concurrent reads and a cancellation-bounded combined wait, with the same whole-operation deadline. No target frameworks or product dependencies change.

Five shared process regressions exercise output exceeding pipe capacity on both streams, nonzero exit diagnostics, a process that never closes output, a process that closes both streams but does not exit, and an exited parent whose child retains inherited pipes. The two POSIX-specific pipe-lifecycle cases are excluded on Windows. The identical regression sources are compiled into both test assemblies.

Review follow-up

Preserve already-written wrappers and cleanup directories when a later item lacks Abi: break the wrapping loop, finalize outputs, and return failure through the logged XA4234. A three-item regression checks the earlier output remains tracked and later stores are not written.

Add NativeLibraryLoadTests.AssemblyStoreDlopenResolvesOriginalPayload to the existing device-integration suite. It builds and installs a Release CoreCLR app through production assembly-store packaging, then uses bionic dlopen("libassembly-store.so", RTLD_NOW | RTLD_LOCAL) and dlsym("_assembly_store"). The complete resolved payload's SHA-256 must equal the original raw build-generated store. The test verifies process bitness and the manifest mode and covers both extracted filesystem libraries and APK-embedded libraries.

The regression has 32-bit and 64-bit cases, selecting a supported CoreCLR ABI from the target. Locally, both loading modes passed on exclusive session-owned ARM64 emulators running API 29 and API 35. Both installed system images expose only arm64-v8a; their two 32-bit cases were explicitly skipped. The 32-bit cases run where armeabi-v7a is available and are not claimed as locally validated. The temporary emulators were stopped and deleted afterward; no shared emulator or physical device was used.

Validation

Coverage	Result
Combined modern writer + process + discovery tests	89 passed, 0 failed/skipped, compiled with the exact stale CI builder path and run with an execution-host NDK override
Combined net10.0 raw-store/packaging + process + discovery tests	14 passed, 0 failed/skipped, including actual arm/arm64/x64 APK/AAB builds and incremental checks; compiled with the same stale CI path
NDK environment fallback	24 native round-trip cases passed using only `ANDROID_NDK_LATEST_HOME`, without `TEST_ANDROID_NDK_PATH`
Executing user's standard installation	14 host cases passed with both NDK environment variables unset, including actual three-ABI APK/AAB builds
ELF layout and writer task coverage	All four encodable ABIs, layout/hash/dynamic metadata, stream and size/error handling, ABI archive/cleanup metadata, and partial-output failure handling retained
Independent official NDK inspection	All 24 symbol-based payload cases passed across arm/arm64/x64, four sizes, before/after stripping, plus section extraction
Actual bionic loader, API 29 and API 35 ARM64	2 passed per system image, 0 failed; 2 unsupported 32-bit cases skipped per image. Both extracted and embedded modes match the complete original payload
Modern test dependency graph	No ELFSharp entry in resolved NuGet assets or the output `.deps.json`
Upstream #12976 regression coverage after the main merge	15 focused host cases previously passed; original Debug and JNI-reference-leak test-app builds succeed

Exact current-head local commands, with both test assemblies deliberately built using -p:AndroidNdkDirectory=/Users/cloudtest/android-toolchain/ndk and runtime TEST_ANDROID_NDK_PATH pointing at the installed private NDK:

./dotnet-local.sh test src/Microsoft.Android.Build.Tasks/Tests/Microsoft.Android.Build.Tasks.Tests/Microsoft.Android.Build.Tasks.Tests.csproj --disable-build-servers -nr:false -p:UseSharedCompilation=false -p:AndroidNdkDirectory=/Users/cloudtest/android-toolchain/ndk -v minimal --filter "FullyQualifiedName~NativeToolTestHelperTests|FullyQualifiedName~AssemblyStoreElfWriterTests|FullyQualifiedName~DlopenAssemblyStoreGeneratorTests|FullyQualifiedName~WrapAssemblyStoresAsSharedLibrariesTests" — 89 passed, 0 skipped.
./dotnet-local.sh test bin/TestDebug/net10.0/Xamarin.Android.Build.Tests.dll -v minimal --filter "FullyQualifiedName~NativeToolTestHelperTests|FullyQualifiedName~PackagingTest.ManagedAssemblyStoreElfWrappers|FullyQualifiedName~Tasks.CreateAssemblyStoreTests" — 14 passed, 0 skipped, repeated successfully with both NDK environment variables unset.
./dotnet-local.sh test bin/TestDebug/net11.0/Microsoft.Android.Build.Tasks.Tests.dll -v minimal --filter "FullyQualifiedName~DlopenAssemblyStoreGeneratorTests.NativeToolsCanExtractPayloadBySymbol" — 24 passed, 0 skipped with only ANDROID_NDK_LATEST_HOME set.
./dotnet-local.sh test bin/TestDebug/MSBuildDeviceIntegration/net10.0/MSBuildDeviceIntegration.dll -v minimal --filter "FullyQualifiedName~NativeLibraryLoadTests.AssemblyStoreDlopenResolvesOriginalPayload" — 2 passed, 2 unsupported-ABI skips on each of API 29 and API 35, with ADB_TARGET and ANDROID_SERIAL explicitly selecting the session-owned emulator.

Earlier validation also built CoreCLR and NativeAOT native hosts for arm/arm64/x64 in Debug and Release, configured the local workload with make prepare / make leeroy, verified nine real bundletool ABI/config split APKs preserve their matching store libraries, and verified all six real APK/AAB store payloads survive native strip/objcopy round trips.

Validation uses this worktree's main-pinned SDK (12.0.100-alpha.1.26477.101) and existing installed Android SDK/NDK/JDK tooling, not PoC task/native outputs. No product/framework/runtime pins are overridden. The stale NDK metadata is a test-artifact relocation simulation, not a product build pin override.

Pull Request
title and
description
should follow the
commit-messages.md workflow documentation, and in particular should include:

Useful description of why the change is necessary.
Links to issues fixed
Unit tests

… managed code Emit data-only ELF32/64 wrappers in modern build tasks and dispatch assembly stores to them without llvm-mc or ld. Preserve discrete payload wrapping, per-ABI archive placement, incremental packaging and the trusted single-symbol native store API. Cover ELF layout, official native-tool strip round trips and compressed APK/AAB packaging. Co-authored-by: Copilot App <[email protected]>

Copilot

Copilot review overview

🔵 Needs a closer look

Handwritten ABI-specific ELF generation affects runtime packaging and warrants final human review alongside the unresolved error-path output issue.

Review effort: Balanced
Findings: 1

Open (1)

Preserve partial outputs when a later item lacks Abi · New

What changed in this PR

Replaces external assembly-store assembly/link steps with managed generation of ABI-specific ELF libraries.

Changes:

Adds a managed ELF writer and modern MSBuild wrapper task.
Preserves ABI packaging, cleanup, and legacy discrete-assembly handling.
Adds extensive ELF, packaging, stripping, and incremental-build tests.

File	Description
`Xamarin.Android.Common.targets`	Routes stores through the modern task.
`Utilities/DlopenAssemblyStoreGenerator.cs`	Removes the native-tool implementation.
`PackagingTest.cs`	Tests packaged stores and incrementality.
`WrapAssembliesAsSharedLibraries.cs`	Retains only discrete-file wrapping.
`CollectNativeFilesForArchive.cs`	Removes obsolete store cleanup.
`assembly-store.hh`	Documents the trusted payload contract.
`TaskItemExtensions.cs`	Generalizes helper documentation.
`DlopenAssemblyStoreGenerator.cs`	Adds managed wrapper generation.
`AssemblyStoreElfWriter.cs`	Implements ELF32/ELF64 output.
`WrapAssemblyStoresAsSharedLibrariesTests.cs`	Tests task outputs and errors.
`NativeToolTestHelper.cs`	Locates NDK inspection tools.
`Microsoft.Android.Build.Tasks.Tests.csproj`	Adds ELF test dependencies.
`DlopenAssemblyStoreGeneratorTests.cs`	Validates symbols and stripping.
`AssemblyStoreElfWriterTests.cs`	Verifies ELF layout and edge cases.
`WrapAssemblyStoresAsSharedLibraries.cs`	Adds the modern wrapping task.
`Microsoft.Android.Build.Tasks.csproj`	Links required shared helpers.
`ApkSharedLibraries.md`	Documents the managed ELF layout.

Integrate c35205f, including the trimmable Debug scanner fix from PR #12976, while preserving the standalone managed assembly-store writer changes. Co-authored-by: Copilot App <[email protected]>

simonrozsival · 2026-10-01T19:51:08Z

@dalexsoto review

Remove the new ELFSharp dependency from modern writer tests. Use official NDK readobj, nm, strip and objcopy for ELF metadata and byte-exact section round trips, sharing the test-only helper with APK/AAB coverage. Keep stream/input validation tool-free and require the NDK explicitly for native inspection cases. Co-authored-by: Copilot App <[email protected]>

simonrozsival · 2026-10-01T21:43:22Z

/review

github-actions · 2026-10-01T21:44:24Z

✅ Android PR Reviewer completed successfully!

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

azcliprod.blob.core.windows.net

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "azcliprod.blob.core.windows.net"

See Network Configuration for more information.

Generated by Android PR Reviewer for #12962

github-actions

⚠️ Needs Changes — 0 errors, 1 warning, 0 suggestions in this pass.

The managed writer is carefully bounded and the static ELF/layout, strip round-trip, packaging, and incremental coverage are strong. The remaining gap is validation through the actual Android dlopen/dlsym boundary; offline LLVM acceptance does not establish bionic compatibility. The previously posted partial-output/cleanup concern at WrapAssemblyStoresAsSharedLibraries.cs:40 also remains unresolved.

CI is still running: the completed checks are green, with the platform builds still in progress and the aggregate pipeline queued.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

azcliprod.blob.core.windows.net

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "azcliprod.blob.core.windows.net"

See Network Configuration for more information.

Generated by Android PR Reviewer for #12962 · copilot · gpt56 · 163.6 AIC · ⌖ 11.3 AIC · ⊞ 26K
_{Comment /review to run again}

Finalize wrapper and cleanup outputs after a later store lacks Abi metadata, retaining XA4234 failure reporting and tracking every file already written. Add Android device coverage through the production store packaging and bionic dlopen/dlsym path. Compare the full loaded payload digest with the original raw store for embedded and extracted libraries, with 32-bit and 64-bit cases selected when supported by the target. Co-authored-by: Copilot App <[email protected]>

Use Process.RunAndCaptureText to multiplex native tool stdout and stderr with one timeout covering output capture and process exit. Treat timed-out exit status as failure and include the executable in timeout diagnostics. Keep the shared net10 packaging helper compatible with cancellation-aware concurrent reads and a bounded combined wait. Cover pipe-capacity output, nonzero exits, output/exit timeouts and inherited pipes in both test assemblies. Co-authored-by: Copilot App <[email protected]>

dalexsoto

One current blocker remains after the complete 20-file review: the new NDK tool lookup is not portable across CI hosts.

Build-machine NDK path breaks Windows/Linux packaging tests. NativeToolTestHelper.cs:18-20 exclusively reads the AndroidNdkDirectory assembly metadata emitted by both test projects. CI builds these DLLs on macOS and transfers them to Windows/Linux without rebuilding. Artifact 49182704 contains /Users/cloudtest/android-toolchain/ndk in that attribute, with the relevant source checksums matching this head. The executing agents therefore assert that the macOS builder's directory exists instead of selecting their own installed NDK.

This accounts for all four new ManagedAssemblyStoreElfWrappers failures in build 1620483: Linux APK/AAB (run 44889610, results 100007/100044), Windows AAB (44891428/100159), and Windows APK (44891494/100209). Please prefer the executing agent's NDK configuration using the existing runtime-discovery convention, such as TEST_ANDROID_NDK_PATH / ANDROID_NDK_LATEST_HOME, before any compile-time fallback. The accepted policy of explicitly failing when inspection tools are missing can remain; selecting the wrong host's toolchain is the independent source-related defect.

The partial-output tracking and production bionic-loader regression follow-ups are fixed at this head. The trusted XABA payload contract remains accepted. The emulator checkout failure is separately established infrastructure, not a code finding; no other current high-confidence blocker was found.

Prefer TEST_ANDROID_NDK_PATH, ANDROID_NDK_LATEST_HOME and the executing user's standard android-toolchain/ndk installation before a valid build-machine fallback. Test DLLs transferred from macOS must not select the builder's NDK on Linux or Windows. Keep native inspections and explicit prerequisite failures, using the NDK already supplied by existing CI setup. Add relocated-artifact discovery regressions without adding ELFSharp or new CI installation steps. Co-authored-by: Copilot App <[email protected]>

simonrozsival · 2026-10-02T07:07:10Z

Fixed the executing-host NDK lookup in 67e17a8. The helper now prefers TEST_ANDROID_NDK_PATH, then ANDROID_NDK_LATEST_HOME, then the executing user profile’s android-toolchain/ndk; a build-time metadata directory is only a final fallback if it exists. Explicitly invalid or missing toolchains still fail rather than skipping assertions.

I checked the actual failing Linux/Windows setup logs from build 1620483: both agents had successfully installed NDK r28c through the existing androidsdk.csproj setup before the tests ran. This needs no new CI download/install step, so the NDK-based checks are retained rather than adding ELFSharp.

Validated both test assemblies with the exact stale /Users/cloudtest/android-toolchain/ndk attribute: 89/89 modern and 14/14 net10 cases pass using the runtime override, including the APK/AAB regressions. The 24 native round-trip cases also pass with only ANDROID_NDK_LATEST_HOME configured, and all 14 host cases pass with both NDK variables unset using the executing user-profile installation. Five discovery regressions cover the lookup ordering and relocated-artifact cases. New public build 1620809 is running; I am not claiming CI green yet.

simonrozsival · 2026-10-02T13:09:59Z

@dalexsoto review

dalexsoto

The executing-host NDK lookup fixes the cross-host test-artifact failure while preserving explicit missing/invalid-toolchain errors. The complete 20-file correctness and integration review leaves no blocking issues: ELF/payload, ABI packaging, incremental cleanup and bounded process capture remain consistent, and the earlier partial-output and production bionic-regression findings stay fixed. The trusted build-generated payload contract and retained discrete-wrapper/tooling boundaries remain accepted.

simonrozsival requested a review from jonathanpeppers as a code owner September 30, 2026 22:22

Copilot AI balanced review requested due to automatic review settings September 30, 2026 22:22

Copilot started reviewing on behalf of simonrozsival September 30, 2026 22:23 View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Comment thread src/Microsoft.Android.Build.Tasks/Tasks/WrapAssemblyStoresAsSharedLibraries.cs Outdated

simonrozsival mentioned this pull request Oct 1, 2026

Plan: remove app-build LLVM IR and the bundled binutils dependency #12940

Open

Merge main into assembly-store ELF writer

963343b

Integrate c35205f, including the trimmable Debug scanner fix from PR #12976, while preserving the standalone managed assembly-store writer changes. Co-authored-by: Copilot App <[email protected]>

github-actions Bot reviewed Oct 1, 2026

View reviewed changes

Comment thread ...d.Build.Tasks/Tests/Microsoft.Android.Build.Tasks.Tests/DlopenAssemblyStoreGeneratorTests.cs

simonrozsival and others added 2 commits October 2, 2026 00:22

dalexsoto requested changes Oct 2, 2026

View reviewed changes

simonrozsival requested a review from dalexsoto October 2, 2026 11:58

simonrozsival added the ready-to-review This PR is ready to review/merge, I think any CI failures are just flaky (ignorable). label Oct 2, 2026

dalexsoto approved these changes Oct 3, 2026

View reviewed changes

simonrozsival merged commit 3d1ca9a into main Oct 3, 2026
44 checks passed

simonrozsival deleted the simonrozsival-assembly-store-elf-writer branch October 3, 2026 20:22

Navigation Menu

Repository navigation

[Microsoft.Android.Build.Tasks] Write assembly-store ELF libraries in managed code - #12962

[Microsoft.Android.Build.Tasks] Write assembly-store ELF libraries in managed code#12962
simonrozsival merged 6 commits into
mainfrom
simonrozsival-assembly-store-elf-writer

simonrozsival commented Sep 30, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

simonrozsival commented Oct 1, 2026

Uh oh!

simonrozsival commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment

Uh oh!

Uh oh!

dalexsoto left a comment

Uh oh!

simonrozsival commented Oct 2, 2026

Uh oh!

simonrozsival commented Oct 2, 2026

Uh oh!

dalexsoto left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

simonrozsival commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Implementation

Trusted native store contract

Standalone scope

NDK-based inspection tests

Bounded process capture

Review follow-up

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Uh oh!

simonrozsival commented Oct 1, 2026

Uh oh!

simonrozsival commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dalexsoto left a comment

Choose a reason for hiding this comment

Uh oh!

simonrozsival commented Oct 2, 2026

Uh oh!

simonrozsival commented Oct 2, 2026

Uh oh!

dalexsoto left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

simonrozsival commented Sep 30, 2026 •

edited

Loading

github-actions Bot commented Oct 1, 2026 •

edited

Loading