Repository navigation
XPLAT: RequiredAssemblies must automatically consider lib\framework #3091
Description
Activity
That sounds pretty complicated - why shouldn't a module author just target
netstandard1.3and only depend on assemblies that also target an appropriate version ofnetstandard?Take a look at how msbuild is thinking about this issue - it's really a similar thing: dotnet/msbuild#1542
I think we'll need to do the same for PowerShell - ship the facade assemblies alongside powershell.exe in
$env:SystemRoot\System32\WindowsPowerShell\v1.0and, if applicable,$env:SystemRoot\SysWow64\WindowsPowerShell\v1.0- we would need to do that for V3 and V5.1 at a minimum, maybe V4 and V5 as well.First of all, .netstandard1.3? why 1.3? I thought we were on 1.6?
Now, I'm going to go ahead and (pretend to?) accept as a given that you can find some way of shipping all "the" facades into the old versions of PowerShell on all OSes (for no reason but forward compatibility?)...
That sounds like a great plan!
I actually thought the whole "facades" thing wasn't going to work until .netstandard2, but if it's an option now, that sounds great. Does that mean you're going to ship everything in PowerShell core, too -- even if you're not using it yourselves?
I mean, otherwise, you're only covering the basics, and you have barely dented the problem. However, regardless:
-
PowerShell modules are still going to have half a metric gigabyte of dependencies, ,which they only need for Core. We can make a list of what you've shipped to reduce what gets packaged, right, but still ...
-
What do I do with "native" libraries? I mean, even
SystemandSystem.Net.Httpare compiled to native, but have you seenSystem.Security.Cryptography?
The other day I tried AngleSharp. It has no dependencies on .Net 4.5, but when I ran...
Install-Package AngleSharp -Source https://www.nuget.org/api/v2/ -Provider NuGet -Destination .
Since package manager's motto is "be prepared," I managed to get back 318,011,877 bytes worth of stuff.
Including these:
runtime.debian.8-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.fedora.23-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.fedora.24-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.native.System.Security.Cryptography.Apple.4.3.0 runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.opensuse.13.2-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.opensuse.42.1-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.Apple.4.3.0 runtime.osx.10.10-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.rhel.7-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.ubuntu.14.04-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.ubuntu.16.04-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0 runtime.ubuntu.16.10-x64.runtime.native.System.Security.Cryptography.OpenSsl.4.3.0Now, for Cryptography, you guys might choose to ship it in PowerShell Core, but my question is, in general -- how do I load the right library, when it's just not cross-platform?
-
Jason Shirk (@lzybkr) We (third parties) must be able to produce modules like this one:
We either need a way to ship "Core" and "Full" modules to the gallery (and automatically install the right one), or we need a solution for loading the right one from a module package containing both.
Not to mention that if you build a binary module you have to take a dependency on System.Management.Automation, which is cross compiled -- forcing the binary module to cross-compile.
It's therefore impossible, as things stand, to ship a cross-platform binary module the way you describe. We have to package both modules as submodules into one module package, and use a script module facade to do OS detection and load the right one.
By the time .Net 2 ships, and a better way gets sorted out, we're going to be knee deep in hacks for this.
Joel Bennett (@Jaykul) Jason did some early work with portable modules successfully compiling PSReadline to be cross platform and supporting PSv3 #3095
Reacted by Joel BennettThanks Steve Lee (@SteveL-MSFT), it's good to see that effort -- although somewhat discouraging.
More questions arise from that:
- Since that document is from 3 weeks ago, has any decision been reached about whose problem this is?
- I assume that even if the problem is passed to the PowerShellGallery team, the PowerShell core team still owns producing new reference assemblies -- should I open a new issue to track that?
- If the problem is passed to the PowerShellGallery team, will the solution be dictated to them, or will they be free to choose to solve this in packaging (as in, sxs folders in the package, and only the appropriate one is extracted)?
- Does the story for shipping an updated PowerShellGet module require that every module author who needs cross-platform binary modules will have to specify the minimum PowerShellGet version to successfully install the module on Windows?
I'm picturing something like this in every github repo:
To install
<MyXplatModule>on Windows, you must first update to the latest PowerShellGetUpdate-Module PowerShellGet Install-Module MyXplatModule
Joel Bennett (@Jaykul) what may not be immediately obvious (and we covered this briefly on the last Community Call) is that we're working through pruning the set of api's we would publish as
PowerShell Standard 1.0(name to be decided later) and this work isn't done yet (as part of the PR I ref'd above).Expectation is that once this is complete, we'd have a new ref assembly you would use for cross platform (and cross PS version) compatible PowerShell modules (we're calling Portable Modules for now).
Intent is that module authors only build one assembly that works where they need it to work. Assume for now, that
PowerShell Standard 1.0is targeting PSv3,2.0is PSv5+. There should be no changes needed for PSGet although we will probably need some changes to PSGallery for new tags yet to be defined.Reacted by Joel BennettJoel Bennett (@Jaykul) are you referring to native assemblies as well?
Steve Lee (@SteveL-MSFT) How is defining a PS standard going to help? This would solve the problem for the reference to System.Management.Automation.dll but the references to the .net standard shims are still a problem.Reacted by Joel BennettAtanas Atanasov (@atanasa) - you're right, as I mention in https://github.com/lzybkr/PowerShell/blob/34ed215a6671f5cc234f58cfed04504890f91159/docs/portable-modules/portable-modules.md (part of #3095) - there are 2 parts of the story - a baseline (standard) System.Management.Automation.dll (and probably a few other dlls) and the .net facade assemblies.
Jason Shirk (@lzybkr) have you considered packing the shims in a set of modules in the Gallery maintained by Microsoft while portable modules specify the dependency in their manifest as a required module rather than custom tags? To me this looks like the classic problem of sharing a dynamic library and a good solution is a package manager. MSBuild does not have a package manager (yet) but PowerShell does.
Can our package manager express the dependency correctly? It's only a dependency with desktop CLR - you wouldn't want them installed on non-Windows machines or on Nano Server, and you can't have them loaded in Core PowerShell, they should only be loaded in Windows PowerShell.
- added this to the milestone
on Apr 20, 2017 - modified the milestones: ,
on May 15, 2017 12 remaining items
The compile-time-only PowerShellStandard.Library goes a long way to helping with this -- although the docs need to show the right way for people to avoid shipping the library:
<PackageReference Include="PowerShellStandard.Library" Version="5.1.0-RC1"> <IncludeAssets>Compile</IncludeAssets> <ExcludeAssets>Runtime</ExcludeAssets> </PackageReference>
However, even in straightforward Windows-only projects we still have very uncomfortable problems using things like ... System.Drawing.Common
Note that's not a "native" binary, and the problems occur just going from Windows PowerShell to PowerShell Core on Windows -- Microsoft is shipping completely different assemblies for .Net 4x and .Net Standard, so when I take a reference on it, I have to cross-compile and ship separate sets of assemblies. The .NET Standard one works fine in PowerShell Core, but if I load it in Windows PowerShell, I get
System.Drawing is not supported on this platformerrors.I know (now) that I can put conditional logic in a manifest to reference the right assemblies, but that's something I have to hand write and get it right.
Joel Bennett (@Jaykul) what version of .NET Framework are you using? Prior to 4.7.2, you had to include the netstandard.dll facade assembly as it wasn't part of .NET Framework. Also, if using Windows Compatibility Pack for .NET, you have to also include those facade assemblies for .NET Framework.
The SDK docs should provide guidance on not accidentally shipping the ref assembly. cc Aditya Patwardhan (@adityapatwardhan) , James Truher (@JamesWTruher)
Reacted by Joel BennettThat's a complicated question, Steve Lee (@SteveL-MSFT) 😁
Right this minute (where I'm trying to reference System.Drawing.Common), I have ended up targeting .NetStandard 2.0 and .Net 4.7.2 in my WindowsConsoleFonts project. I'm conditionally loading the RootModule in the manifest, and it works in both Windows PowerShell (with the 4.x binary) and PowerShell Core (on Windows) with the .Net Standard binary.
Note that I could target 4.6.2 (as System.Drawing.Common does), but it would make no difference to this problem -- and the build ends up with something like 96 extra binary assemblies, and since I don't understand what portion of my users might be helped by that, I'm just going to target the latest for now and wait for people to complain, because I'm already having to ship my binaries twice, and I'd rather not ship 30x the binaries.
Joel Bennett (@Jaykul) Another option is to do what PSReadLine does which is to build for
net472only and test on PSCore6, and ship just that assembly (and dependent assemblies). In this case, you wouldn't separately build fornetstandard2.0.MSIL is compatible between .Net Framework and .Net Core. .Net Standard really only provides some assurance that the APIs you're using are compatible in multiple CLRs. If you test in both Windows PowerShell and PowerShell Core, you could get away with just building for
net472. There are exceptions, of course, if you start using assemblies that we don't include with PSCore6.I tested a build of your module against
net472(modifying the psd1) on both 5.1 and 6.2-preview.1 and works in both (well, only tried Get and Set)That's interesting. You're totally right, that works when all I need is Windows.
So is this being addressed in 7, or are we just going to keep ignoring the problem?
🙏 I know it's not an easy problem, but it would make working with xplat native APIs significantly easier if we can get this resolved. GraphicalTools has probably run into this problem at least a handful of times, I'd wager, and the workarounds aren't particularly pretty.
Perhaps #11032 resolves the issue.
In order to support cross-platform modules, we need to support dependent assembly packages properly, the way that
dotnet builddoes -- we need to be able to either:To me, it's perfectly obvious that a single machine-wide .nuget folder -- which PowerShell could un-pack from on demand -- would be the ideal solution, requiring the least amount of disk space and work on the part of developers. It would require the NuGet package provider for PowerShell PackageManagement be updated, and significant changes to the core to do the work of unpacking and loading the right assemblies...
Short of that, a .nuget folder (or "assemblies" or whatever) in my PowerShell folder would be a good fall-back option. Same basic story.
Failing that, the fallback option would be to allow module authors to include the actual nupkg files in their modules. It requires almost all of the same development, and you end up with a lot more copies of the assemblies, but since it doesn't require a working PackageManagement provider (at least, not at runtime), it's somewhat simpler, and self-contained.
As a final, worst-case scenario, we could require module authors to actually unpack the nupkg files. That is, we could require each module author to
Install-Package Dependency -Destination MyModuleFolderand put thelibfolder from the nuget packages directly within their module. This way PowerShell is only responsible to load the right assembly based on the platform, but we still get cross-platform modules.Backwards compatibility
Note that for the sake of making modules compatible with PowerShell 5.x and lower, all of these methods require cutting off support at PowerShell 3 (.Net 4) and shipping the .Net 4 assemblies loose in the root folder of the module. As long as the manifest just says: RequiredAssemblies = "Whatever.dll", legacy
Windows PowerShellonly looks in the GAC or in the module folder.