It's Really Hard To Package Native Libraries for PowerShell Modules

TylerLeonhardt

OK, so... this is an issue that's been around for a long time, I'm sure. I've not really had reason to hit it before. And now I have.

Scenario

I'm trying to package a module with a dependency that does p/invoking via native runtime libraries. The library is SkiaSharp. This library cannot be used without native runtimes.

I'd like to be able to handle native runtimes correctly from a compiled PowerShell cmdlet / module.

Problem

The issue then becomes that these native runtimes are different on each platform. Each platform expects that the libraries will be in the same folder as the SkiaSharp.dll. This can't be done for every platform at once, as several of the platform-specific libraries have the same name.

The folder structure after running dotnet publish looks like this:

Ignoring the tizen runtimes (I think they're for Android or something? not sure), a few of the libraries have the same filenames, namely the Windows libraries for the two architectures. A temporary workaround is to put as much as possible in the same folder and ignore that x86 Windows still exists.

Attempted Solutions

We have tried to work around the problem by selectively importing the native libraries, but this is entirely impossible with PowerShell as they throw Bad IL format errors. We have tried:

Import-Module .\bin\Debug\netstandard2.0\publish\runtimes\<platform>\native\<runtime_file>
Add-Type -Path .\bin\Debug\netstandard2.0\publish\runtimes\<platform>\native\<runtime_file>
[System.Reflection.Assembly]::LoadFile((Resolve-Path ".\bin\Debug\netstandard2.0\publish\runtimes\<platform>\native\<runtime_file>"))

Possible Solutions

Ability to specify dependencies such as these by relative path in a PSD1 on a per-platform basis, which is respected with Import-Module and loaded correctly.
Some kind of handling in PowerShellGet for native runtimes on a per-platform basis? Possibly handled via NuGet in terms of pulling down the exact runtime required on a specific platform.

There's probably a neater solution I'm missing, feel free to add any suggestions.

Files

Source & module files for reference, if you should like to attempt anything with them:
https://github.com/vexx32/PSWordCloud/tree/Cmdlet/PSWordCloud

Running PSWordCloud

To use:

Publish the source files for this module with dotnet publish
Import the PSM1 file in the linked folder with Import-Module
Attempt to pipe some text into New-WordCloud -Path .\test.svg

This will work on Windows thanks to the PSM1 manually modifying $env:Path to add all the native runtime folders. This solution is terribly messy and absolutely makes a right royal mess of $env:Path for anyone importing the module. I would like to avoid this.

It attempts to do similar on Mac OS and Linux, but in these cases it seems the path for native runtime libraries isn't searched correctly. The only currently available workaround is manually copying these libraries to the same folder as SkiaSharp.dll which is... less than great.

/cc TylerLeonhardt Dongbo Wang (@daxian-dbw) Steve Lee (@SteveL-MSFT)

Here's the issue I opened for PSGet to support runtimes: https://github.com/PowerShell/PowerShellGet/issues/273

For now, you can have different managed wrappers using PInvoke for each RID and selectively load the right one based on the platform. This means you'd ship all the native libraries you support and have multiple copies of the wrapper (one for each RID).

Awesome! Do you have a reference / example for how that would be done?

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) Here's an example from Joel Bennett (@Jaykul).

Just omit the extension in the DllImport attribute and they'll be loaded based on those environment variables. Make sure those are set before any p/invoke method is invoked.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) Did adding the paths to the variables not work?

Matthew Leibowitz (@mattleibow)

Matthew Leibowitz (@mattleibow) nope. Let me go back a little ways in my commit history for the file I was using here...

This was the original from Joel Bennett (@Jaykul) (linked by Patrick Meinecke (@SeeminglyScience) above). Ah! Patrick, I was using that script under the assumption that Skia's own DLLImport methods most likely would handle the native import, especially since placing the native runtime files in the same folder as the SkiaSharp.dll allowed it to load flawlessly.

I noted that the error message on Mac systems (TylerLeonhardt may still have a copy of the exact error or can get it for us; he was helping me test it on Mac) seems to point to a different library path than Joel had noted in his script, and so I ended up with this script where I added the paths to both variables:

https://github.com/vexx32/PSWordCloud/blob/d01d38c7b837639c5070edee07a44b3d92e10571/Cmdlet/PSWordCloud.psm1

Setting the library paths didn't work at all, although moving the .dylib file to the same folder as SkiaSharp.dll did. The library paths don't seem to be respected in all cases, despite being directly referenced in the error messages. I haven't yet attempted changing the recorded environment variables outside the PowerShell process to accomplish this, but it may be necessary to have that work, I'm not sure.

Currently I'm working around this as you can see here (module file) and here build file. Basically I execute the build script to compile the DLLs and copy them from the output folder into a subtree that ends up looking like this:

PSWordCloud/
  |- win-x64/
    |- SkiaSharp.dll
    |- libSkiaSharp.dll
  |- win-x86/
    |- SkiaSharp.dll
    |- libSkiaSharp.dll
  |- linux-x64/
    |- SkiaSharp.dll
    |- libSkiaSharp.so
  |- osx/
    |- SkiaSharp.dll
    |- libSkiaSharp.dylib
  |- PSWordCloudCmdlet.dll
  |- PSWordCloud.psd1
  |- PSWordCloud.psm1

As you can see, there will be a good bit of duplicating the main SkiaSharp.dll, but this way I can selectively pick which folder to Add-Type the file from when importing the PowerShell module on a given platform.

The duplication is necessary, since I am unable to Add-Type the libSkiaSharp.* files directly; on Windows they give me Bad IL format errors, and I imagine a similar error on other OSes as well, though I haven't had a chance to try that and don't really consider it necessary.

Thus far I have tested Mac and Windows, and the library paths have only been respected on Windows (probably because they're literally just pulled from %PATH% and used as-is).

Duplicating the managed SkiaSharp.dll might not be to bad as that is a fairly small file. You won't be able to load the libSkiaSharp.* directly as that is machine code, not IL. But, you don't have to worry about it as the managed SkiaSharp.dll will load it automatically via P/Invoke.

The real issue here is that you are publishing a .NET Core app, and everything is in the right place. The native files are supposed to be under the runtimes folder. PowerShell is just not respecting those rules. If you were to compile and publish a .NET Core console or web app, the .NET Core runtime would know to look in those folders.

That is why I would rather fix this on the PowerShell side as they need to correctly look at the runtimes folder.

If you have a look at this file: https://github.com/PowerShell/PowerShell/blob/master/docs/building/internals.md#native-components you can see PowerShell itself uses this folder structure, it just appears to no understand it with modules.

Yeah, it would be much better to fix this in PowerShell or PowerShellGet, and have the floor open for more xplat native code extensibility.

People keep asking for PowerShell to do this: #3091 #6642 #6724

We can't keep pushing this stuff off on module authors -- not only is it painful, it also produces a lot of copies of everything. We also can't put it purely on PowerShellGet, unless we really think the right solution is to build NPM-like warrens of dependencies inside every module folder.

However, if we do think that's the right solution, we should at least do it the way NPM does: the package manager needs to support assemblies as dependencies -- not require us to re-package them in modules.

Personally, I think the NPM way is wrong, and the dotnet build way is a better model: a single copy of the assemblies which your session can go fetch from a central location when it's needed.

This means PowerShell needs to do the heavy lifting #7259

We would add nuget package references to RequiredAssemblies, and

PowerShellGet would download those dependencies
PowerShell would go find them and load the right one

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32) this should be resolved by #11032?

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT) yep, can confirm this appears to be working correctly! Expect to see more PSWordClouds soon. 😂

So how do you Package Native Libraries for PowerShell Modules, now when this change is in?

Andrew Savinykh (@AndrewSav)

Andrew Savinykh (@AndrewSav) see here:

https://docs.microsoft.com/en-us/powershell/scripting/learn/writing-portable-modules?view=powershell-7#dependency-on-native-libraries

Essentially, pack all the native runtimes into your module as needed, and put them in folders named after the runtime identifier for the platform.

Writing Portable Modules - PowerShell

Navigation Menu

Repository navigation

It's Really Hard To Package Native Libraries for PowerShell Modules #8861

Scenario

Problem

Attempted Solutions

Possible Solutions

Files

Running PSWordCloud

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

It's Really Hard To Package Native Libraries for PowerShell Modules #8861

Description

Scenario

Problem

Attempted Solutions

Possible Solutions

Files

Running PSWordCloud

Activity

SteveL-MSFT commented on Feb 11, 2019

vexx32 commented on Feb 11, 2019

SeeminglyScience commented on Feb 11, 2019

mattleibow commented on Feb 11, 2019

vexx32 commented on Feb 11, 2019

mattleibow commented on Feb 15, 2019

vexx32 commented on Feb 15, 2019

Jaykul commented on Feb 15, 2019

SteveL-MSFT commented on Mar 14, 2020

vexx32 commented on Mar 14, 2020

AndrewSav commented on Mar 14, 2020

vexx32 commented on Mar 14, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions