Add native dll resolver by iSazonov · Pull Request #11032 · PowerShell/PowerShell

Ilya (iSazonov) · 2019-11-11T09:56:09Z

PR Summary

Resolve #9488

Add a native dll resolver to discover and load native dlls based on OS platform and architecture.

PR Context

Releated #3091

If a managed dll has native dependencies the handler will try to find these native dlls.
1. Gets the managed.dll location (folder)
2. Based on OS name and architecture name builds subfolder name where it is expected the native dll resides:
3. Loads the native dll

    ///     managed.dll folder
    ///                     |
    ///                     |--- 'win-x64' subfolder
    ///                     |       |--- native.dll
    ///                     |
    ///                     |--- 'win-x86' subfolder
    ///                     |       |--- native.dll
    ///                     |
    ///                     |--- 'win-arm' subfolder
    ///                     |       |--- native.dll
    ///                     |
    ///                     |--- 'win-arm64' subfolder
    ///                     |       |--- native.dll
    ///                     |
    ///                     |--- 'linux-x64' subfolder
    ///                     |       |--- native.so
    ///                     |
    ///                     |--- 'linux-x86' subfolder
    ///                     |       |--- native.so
    ///                     |
    ///                     |--- 'linux-arm' subfolder
    ///                     |       |--- native.so
    ///                     |
    ///                     |--- 'linux-arm64' subfolder
    ///                     |       |--- native.so
    ///                     |
    ///                     |--- 'osx-x64' subfolder
    ///                     |       |--- native.dylib

PR Checklist

Rain Sallow (/u/ta11ow) (vexx32) · 2019-11-11T11:07:09Z

This looks super promising, wow! Nice work! 😍

I have one suggestion just for compatibility; prior to 6.2 (or 6.1? Don't recall exactly) pwsh was able to search the location the library was in for native DLLs (though not subfolders). As part of one of the changes in 6.2, that was removed, and this seems like a sensible place to add it.

Can we have it also check the same folder as the managed library so that anything that was using the old behaviour works again as well?

Ilya (iSazonov) · 2019-11-11T12:35:06Z

Can we have it also check the same folder as the managed library so that anything that was using the old behaviour works again as well?

Yes, we can. I think first we need to have a standard (RFC?) how we search, order is important. Also we should consider security compliance.

Rain Sallow (/u/ta11ow) (vexx32) · 2019-11-11T12:54:12Z

/cc Paul Higinbotham (@PaulHigin) Steve Lee (@SteveL-MSFT)

Rain Sallow (/u/ta11ow) (vexx32) · 2019-11-11T15:04:25Z

Ilya (@iSazonov) we may want to look into whether there's a way to pull the acceptable RIDs from dotnet / if this is available from a .NET API rather than having a hardcoded set here. 🙂

Ilya (iSazonov) · 2019-11-11T15:36:47Z

Rain Sallow (/u/ta11ow) (@vexx32) .Net Core has not API to expose RIDs.
We could support all RIDs from https://docs.microsoft.com/en-us/dotnet/core/rid-catalog
but it will very complicate the resolver.
I hope the proposed solution covers most of use cases. Notice, that key here is that we calculate path based on managed assembly location: if the assembly is right assembly for OS, arch, runtime then we expect that native assembly also is right assembly and is in right place. I hope other scenarios is edge cases that we could delegate third-party project developers who could use DllImportResolver callbacks directly.

.NET Core Runtime IDentifier (RID) catalog
Learn about the Runtime IDentifier (RID) and how RIDs are used in .NET Core.

Rain Sallow (/u/ta11ow) (vexx32) · 2019-11-11T15:56:02Z

Ilya (@iSazonov) yeah I wondered. The reason I ask is that some RIDs include OS version as well as architecture, so we might be missing (for example) a folder with win7-x64 quite easily. Perhaps we could for the moment solve that problem with use of wildcards?

Steve Lee (SteveL-MSFT) · 2019-11-11T16:38:53Z

It makes sense to follow the .NET RID hierarchy so that it's consistent and predictable. It doesn't have to be an exact match, but I can image build scripts that create the native folder under the RID passed to msbuild. So for macOS, you're just as likely to see osx-x64 folder.

Ilya (iSazonov) · 2019-11-11T17:28:30Z

Will replace "osx" with "osx-x64".

Ilya (iSazonov) · 2019-11-11T17:39:46Z

The reason I ask is that some RIDs include OS version as well as architecture, so we might be missing (for example) a folder with win7-x64 quite easily. Perhaps we could for the moment solve that problem with use of wildcards?

It is just a case I name as "edge". Very unlikely to be needed. I believe any developer prefer to create portable solutions.
Also Core does not have a reliable way to determine OS/kernel version.
So I suggest only use portable RIDs. At least until we get feedback.

Rain Sallow (/u/ta11ow) (vexx32) · 2019-11-11T17:50:23Z

🤔 along with this, would it perhaps be feasible to expose a public API to allow module authors to add specific paths to search when working in those edge cases?

(maybe that's a future discussion 😆)

Ilya (iSazonov) · 2019-11-12T03:02:37Z

to expose a public API

The API is already in .Net Core. See DllImportResolver callbacks reference above. I guess it is possible to use even script delegate for the callback.

Dongbo Wang (daxian-dbw)

Can you please add one test?
I guess you can fake a native library file (renaming a text file to native.dll), then the resolution should succeed but loading the native library will fail with a BadImageFormatException -- An attempt was made to load a program with an incorrect format. (0x8007000B).

…text.cs Co-Authored-By: Dongbo Wang <[email protected]>

Travis Plunk (TravisEz13) · 2019-12-03T19:34:51Z

Ilya (@iSazonov) Please don't use 7.0-consider for PRs... I'll try to update the descriptions to say which milestones are for issues and which are for PRs.

Dongbo Wang (daxian-dbw)

LGTM

Ilya (iSazonov) · 2019-12-06T05:22:38Z

Added a reference to new doc issue.

Deleted user (ghost) · 2019-12-16T21:37:40Z

🎉v7.0.0-rc.1 has been released which incorporates this pull request.:tada:

Handy links:

Release Notes

Add native dll resolver

ab13ca9

Ilya (iSazonov) added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Nov 11, 2019

Ilya (iSazonov) added this to the 7.0.0-preview.6 milestone Nov 11, 2019

Ilya (iSazonov) requested review from Steve Lee (SteveL-MSFT), Aditya Patwardhan (adityapatwardhan), Dongbo Wang (daxian-dbw) and Rain Sallow (/u/ta11ow) (vexx32) November 11, 2019 09:56

Ilya (iSazonov) added the Documentation Needed in this repo Documentation is needed in this repo label Nov 11, 2019

PoshChan-Bot (PoshChan) reviewed Nov 11, 2019

View reviewed changes

Comment thread src/System.Management.Automation/CoreCLR/CorePsAssemblyLoadContext.cs

PoshChan-Bot (PoshChan) reviewed Nov 11, 2019

View reviewed changes

Comment thread src/System.Management.Automation/CoreCLR/CorePsAssemblyLoadContext.cs

Steve Lee (SteveL-MSFT) assigned Dongbo Wang (daxian-dbw) Nov 11, 2019

Replace "osx" with "osx-x64"

c83e8a6

Fix CodeFactor add Codacy issues

4bddf37

Dongbo Wang (daxian-dbw) reviewed Nov 13, 2019

View reviewed changes

Comment thread src/System.Management.Automation/CoreCLR/CorePsAssemblyLoadContext.cs Outdated

This was referenced Nov 14, 2019

Add more Extra variables to be allowed to be referenced in module manifest #9908

Closed

XPLAT: RequiredAssemblies must automatically consider lib\framework #3091

Closed

Rain Sallow (/u/ta11ow) (vexx32) mentioned this pull request Nov 21, 2019

Q&A for PowerShell Community Call for Nov 21, 2019 PowerShell/PowerShell-RFC#235

Closed

Rain Sallow (/u/ta11ow) (vexx32) approved these changes Nov 21, 2019

View reviewed changes

Dongbo Wang (daxian-dbw) reviewed Nov 21, 2019

View reviewed changes

Ilya (iSazonov) and others added 3 commits November 28, 2019 07:58

Update src/System.Management.Automation/CoreCLR/CorePsAssemblyLoadCon…

c4653b3

…text.cs Co-Authored-By: Dongbo Wang <[email protected]>

Update comment

d28ab8c

Use ToString().ToLowerInvariant()

f1207bd

Ilya (iSazonov) mentioned this pull request Nov 30, 2019

macOS can't use New-PSSession to Connect to O365 from PowerShell Host - WSMAN client library not found #10600

Closed

Ilya (iSazonov) removed this from the 7.0-Consider milestone Dec 4, 2019

Dongbo Wang (daxian-dbw) approved these changes Dec 6, 2019

View reviewed changes

Dongbo Wang (daxian-dbw) added this to the rc.1-consider milestone Dec 6, 2019

Ilya (iSazonov) mentioned this pull request Dec 6, 2019

Native dll resolver MicrosoftDocs/PowerShell-Docs#5196

Closed

Ilya (iSazonov) removed the Documentation Needed in this repo Documentation is needed in this repo label Dec 6, 2019

Dongbo Wang (daxian-dbw) merged commit 95c472a into PowerShell:master Dec 9, 2019

Ilya (iSazonov) deleted the add-native-dll-resolver branch December 10, 2019 04:39

Steve Lee (SteveL-MSFT) modified the milestones: rc.1-consider, rc.1-approved Dec 10, 2019

Travis Plunk (TravisEz13) modified the milestones: rc.1-approved, 7.0.0-rc.1 Dec 11, 2019

Travis Plunk (TravisEz13) pushed a commit that referenced this pull request Dec 11, 2019

Add native binary resolver (#11032)

5df5ac8

Joey Aiello (joeyaiello) mentioned this pull request Dec 12, 2019

Document change in assembly loading behaviour MicrosoftDocs/PowerShell-Docs#4249

Closed

3 of 8 tasks

ghost mentioned this pull request Dec 16, 2019

Loading Native Libraries is broken #9488

Closed

Rain Sallow (/u/ta11ow) (vexx32) mentioned this pull request Jan 20, 2020

New-WordCloud is non-functional in PowerShell 7 release candidates vexx32/PSWordCloud#29

Closed

Ilya (iSazonov) mentioned this pull request Jan 30, 2020

Loading nuget with referenced modules - Weird behavior #9846

Closed

Steve Lee (SteveL-MSFT) mentioned this pull request Mar 14, 2020

It's Really Hard To Package Native Libraries for PowerShell Modules #8861

Closed

Matthew Leibowitz (mattleibow) mentioned this pull request Apr 25, 2020

[QUESTION] Loading difficulties for xplat pwsh modules mono/SkiaSharp#783

Closed

refactorsaurusrex mentioned this pull request May 26, 2020

I'm missing a native binary refactorsaurusrex/journal-cli#60

Closed

Steve Lee (SteveL-MSFT) mentioned this pull request Oct 7, 2021

Add processor architecture information in $PSVersionTable #16209

Closed

Travis Plunk (TravisEz13) mentioned this pull request Oct 12, 2022

Back-port to 7.0: Update list of PowerShell team members in release tools #18266

Merged

22 tasks

Steve Lee (SteveL-MSFT) mentioned this pull request Dec 13, 2023

Fix native library resolver to handle cases with extension #20912

Open

7 of 22 tasks

Thatgfsj (Thatgfsj) pushed a commit to Thatgfsj-contribs/PowerShell that referenced this pull request Aug 6, 2026

Add native binary resolver (PowerShell#11032)

bd35bfd

Conversation

Ilya (iSazonov) commented Nov 11, 2019 • edited by daxian-dbw Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

Uh oh!

Uh oh!

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019

Uh oh!

Ilya (iSazonov) commented Nov 11, 2019

Uh oh!

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019

Uh oh!

Ilya (iSazonov) commented Nov 11, 2019 • edited by unfurl-links Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Steve Lee (SteveL-MSFT) commented Nov 11, 2019

Uh oh!

Ilya (iSazonov) commented Nov 11, 2019

Uh oh!

Ilya (iSazonov) commented Nov 11, 2019

Uh oh!

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ilya (iSazonov) commented Nov 12, 2019

Uh oh!

Uh oh!

Dongbo Wang (daxian-dbw) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Travis Plunk (TravisEz13) commented Dec 3, 2019

Uh oh!

Dongbo Wang (daxian-dbw) left a comment

Choose a reason for hiding this comment

Uh oh!

Ilya (iSazonov) commented Dec 6, 2019

Uh oh!

Deleted user (ghost) commented Dec 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Ilya (iSazonov) commented Nov 11, 2019 •

edited by daxian-dbw

Loading

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019 •

edited

Loading

Ilya (iSazonov) commented Nov 11, 2019 •

edited by unfurl-links Bot

Loading

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019 •

edited

Loading

Rain Sallow (/u/ta11ow) (vexx32) commented Nov 11, 2019 •

edited

Loading