Repository navigation
Loading Native Libraries is broken #9488
Description
Attempting to load a module that requires p/invoke from a third-party native library fails to load the library from the same folder as the DLL doing the p/invoking.
This used to work in PowerShell Core 6.1.0 but regressed between 6.2.0-preview1 and 6.2.0-preview2. It still works in Windows PowerShell.
The native library it's failing to locate is located in the same folder as the SkiaSharp.dll that is trying to locate it. See #8861 (comment) and that issue thread for why it was done in that fashion.
Steps to reproduce
Install-Module PSWordCloud
New-WordCloudExpected behavior
cmdlet New-WordCloud at command pipeline position 1
Supply values for the following parameters:
InputObject:
Actual behavior
new-wordcloud : The type initializer for 'PSWordCloud.WCUtils' threw an exception.
At line:1 char:1
+ new-wordcloud
+ ~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], TypeInitializationException
+ FullyQualifiedErrorId : System.TypeInitializationException
Querying for the base exception with $Error[0].Exception.GetBaseException() | Format-List * -F yields the following:
Message : Unable to load DLL 'libSkiaSharp' or one of its dependencies: The specified module could not be
found. (Exception from HRESULT: 0x8007007E)
TypeName :
Data : {}
InnerException :
TargetSite : IntPtr sk_fontmgr_ref_default()
StackTrace : at SkiaSharp.SkiaApi.sk_fontmgr_ref_default()
at SkiaSharp.SKFontManager.get_Default()
at PSWordCloud.WCUtils..cctor()
HelpLink :
Source : SkiaSharp
HResult : -2146233052
Environment data
Name Value
---- -----
PSVersion 6.2.0
PSEdition Core
GitCommitId 6.2.0
OS Microsoft Windows 10.0.17763
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Note: The same was tested in a Mac environment (thanks Stephen Valdinger (@steviecoaster)!) and also failed in the same way. I suspect it will be identical on Linux distros as well.
Activity
[-]My bug report[/-][+]Loading Native Libraries is broken[/+]SteveL-MSFT commentedon Apr 27, 2019
Aditya Patwardhan (@adityapatwardhan) can you take a look at this?
iSazonov commentedon Apr 27, 2019
You could look used paths with
procmonutility and then try to fix with HintPathvexx32 commentedon Apr 27, 2019
I'll give that a look but given this isn't a library I'm loading directly (it's loaded by the main SkiaSharp.dll itself) I'm not that confident in that doing too much. The fact that there are four possible library files to load depending on platform also complicates things.
But regardless, this doesn't seem like an intentional change so the regression should probably be addressed.
iSazonov commentedon Apr 27, 2019
MSFT team works hard on vNext p/invokes in .Net Core 3.0 and perhaps they back-ported something to 2.1. I don't remember that we changed anything in PowerShell related dll loading.
vexx32 commentedon Apr 27, 2019
There was #8073 and several
Internal-marked changes in 6.2.0-preview2. I have no idea what else may have changed during those. 😕iSazonov commentedon Apr 27, 2019
The PR change is to load module dll from module folder first then from GAC. Your problem is that the module dll reference second dll and auto dll loading (it does Core not PowerShell ) doesn't found the dll. My suggestion is to explicitly add hintpath to the dll in project file.
vexx32 commentedon Apr 27, 2019
Ilya (@iSazonov) I added some HintPath references to my csproj, but all I get is a warning when I
dotnet publish. There is no change in behaviour of the module once loaded.From a bit of reading up on it, it looks like the HintPath is used to reference external assemblies that aren't included in the project build itself. The SkiaSharp assemblies are included, so adding a HintPath for compile/build doesn't affect the Skia DLLs or the paths they look up at runtime.
For reference, this is my csproj:
I added the following lines to no effect (other than a build/publish time warning that they can't be found, because, well, they don't exist until after the publish is completed anyway):
From what I've read on HintPath, it doesn't seem to have any effect on what paths are searched when loading a DLL, or that DLL tries to load other DLLs for p/invoke purposes.
Jaykul commentedon Apr 28, 2019
That HintPath is a compile-time thing only.
vexx32 commentedon Apr 28, 2019
The way you say that makes it sound like there's another I missed?
iSazonov commentedon Apr 28, 2019
Rain Sallow (/u/ta11ow) (@vexx32) If you have simple repo code you could ask in CoreCLR repository.
vexx32 commentedon Apr 28, 2019
I'm unsure if I can rule out a Powershell-specific change just yet. I might have to put a small test together for a console app and see how that goes.
vexx32 commentedon Apr 28, 2019
Thanks to a tip from Joel Bennett (@Jaykul), I've managed to put together a stop-gap solution. Executing this from the PSM1 before importing the cmdlet and the main Skia DLL seems to resolve the issue.
However, the fact that it requires additional p/invoke to make this work is kind of ridiculous. I'm also very confused as to what changed and removed the loaded SkiaSharp.dll's own directory path from the standard search locations as documented here.
53 remaining items
vexx32 commentedon Jun 29, 2019
Currently the folder structure of PSWordCloud is not general, no. That can be changed, of course.
Yeah, I can absolutely put together a simple test bed for that. Pester is a bit tricky to do, just based on the nature of the output I suppose, but I think when it fails it produces an exception you can catch, though that exception is thrown from the type initializer if I recall correctly.
Would it be appropriate to have the test itself pull the Skia DLLs via dotnet, or would it be better to package them in the repo? I guess if we pull them each time it's not so big a deal if it's flagged as a feature test once it's no longer pending? 🤔
iSazonov commentedon Jun 29, 2019
If you could implement a test nodule with indirect dependence with Skia dlls it would be nice. And yes we could install Skia from nuget. Although the test would be more reliable if we directly put the dlls with the test module in
our test module folderan assert folder.It seems Nuget best practice is to put native dlls in RID named folders. Yes?
vexx32 commentedon Jun 29, 2019
Yep, I can do that! I'll have a look at it tommorow. 😊
iSazonov commentedon Jun 29, 2019
I updated my comment - please put the test module in an test assert folder.
vexx32 commentedon Nov 11, 2019
Rob Holt (@rjmholt) I made some attempts to use the code you provided to make it work for PSWordCloud, but it's still not working at all. The load operation completes successfully, but loading the managed SkiaSharp.dll after the unmanaged library still fails to make use of the loaded DLL on Unix platforms. It does manage to work for Windows, though, somehow.
See my WIP branch here https://github.com/vexx32/PSWordCloud/tree/FixPInvokes -- run
build.ps1to build the module and put all the files in the right places. It'll import the module for you, too. CallingNew-WordCloudafter running the build works on Windows, but fails completely on Mac OS.Patrick Meinecke (@SeeminglyScience) you see anything I did wrong there? :/
vexx32 commentedon Nov 11, 2019
Steve Lee (@SteveL-MSFT) what's the plan for handling native libs at present? 😕