Repository navigation
Tests related to -SslProtocol in web cmdlets are failing on macOS daily runs #5590
Description
Activity
I noticed that the openssl version is different between the last passing daily and the next failing daily.
1.0.2l (failed): https://travis-ci.org/PowerShell/PowerShell/jobs/309351359#L142
1.0.2j (passed): https://travis-ci.org/PowerShell/PowerShell/jobs/308273280#L174That seems to coincide with the changes in the default macos image on Travis CI https://blog.travis-ci.com/2017-11-21-xcode8-3-default-image-announce
This is the kind of weird behavior that happens on a systems with the non-opnessl libcurl.
brew has 1.0.2m available. I have that on my macOS laptop. I will test master and see if this repros
Mark Kraus (@markekraus) Thanks for looking into it!
I can't repro this with openssl 1.0.2l or 1.0.2m. I noticed they also added [email protected] to the image. I installed this and and tested and still cant' repro this. I'm on macOS 10.13.1 and another change they made was the move to 10.12.6. So i can't duplicate this environment entirely.
I'm able to reproduce the failure:
> $env:PATH = "/Users/psbuildacct/PowerShell/test/tools/WebListener/bin/:" + $env:PATH > Import-Module /Users/psbuildacct/PowerShell/test/tools/Modules/HttpListener > Import-Module /Users/psbuildacct/PowerShell/test/tools/Modules/WebListener > $response = Start-HttpListener -Port 8080 > $WebListener = Start-WebListener > $Uri = Get-WebListenerUrl -Test 'Get' -Https -SslProtocol Tls11 > Invoke-WebRequest -Uri $Uri -SslProtocol Tls11 -SkipCertificateCheck Invoke-WebRequest : SSL connect error At line:1 char:1 + Invoke-WebRequest -Uri $Uri -SslProtocol Tls11 -SkipCertificateCheck + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + CategoryInfo : InvalidOperation: (Method: GET, Re...rShell/6.0.0 }:HttpRequestMessage) [Invoke-WebRequest], HttpRequestException + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommandThis happens on macOS
10.12.6with openssl1.0.2minstalled.
Here are the exception call stack and error messages:PS:19> $er.Exception.Message An error occurred while sending the request. PS:20> $er.Exception.StackTrace at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.<FinishSendAsyncUnbuffered>d__59.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(HttpClient client, HttpRequestMessage request, Boolean stripAuthorization) in /Users/psbuildacct/PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/WebRequestPSCmdlet.CoreClr.cs:line 457 at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord() in /Users/psbuildacct/PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/CoreCLR/WebRequestPSCmdlet.CoreClr.cs:line 550 PS:21> $er.Exception.InnerException.Message SSL connect error PS:22> $er.Exception.InnerException.StackTrace at System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error) at System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1 easyWrapper, CURLcode messageResult)I have no idea about the root cause. For now I will disable
-SslPrototoltests.One more thing I want to bring up is that our macOS CI actually uses openssl
1.0.2m. The bootstrap script runsbrew install curl --with-openssl, which installs the1.0.2m. See the CI screenshot below:
yea I missed where it was installing 1.0.2m
The question for me is whether this is WebListener not getting the right TLS applied, or the web cmdlets (HttpClient) not selecting the right tls setting.. or both. I don't have a good way to test that except maybe verify with curl that limiting those protocols to those ports works against web listener. I could not find any public web services that had Tls 1.0, 1.1, or 1.2 only.
I have been trying several things on travis CI, but it takes an eternity for macOS builds to start and finish. can you do a
brew list --versionsin your environment?I'm waiting on a build to start that runs
brew remove [email protected]before boot strapping. I can't reproduce an issue with it installed on my system with it installed... but maybe it's an issue for 10.12.6?but it takes an eternity for macOS builds to start and finish
Yes, it's really a pain to verify anything with the Travis CI mac builder.
can you do a brew list --versions in your environment
Sure, here is the output:
$ brew list --versions cmake 3.6.2 curl 7.54.0 hub 2.2.9 makedepend 1.0.5 openssl 1.0.2j 1.0.2k 1.0.2m pkg-config 0.29.1_2 0.29.2but maybe it's an issue for 10.12.6?
That is a possibility -- I never ran those tests on the macOS I'm using now.
If a consumer has a slightly different version of the library, PowerShell will not work. Can we inplace such sensitive libraries in PowerShell Core folder?
Ilya (@iSazonov) This does not appear to be related to the libcurl library or any other. As far as I can tell, this is an issue between CoreFX and macOS 10.12.6. The same library versions and build process works on 10.12.5 and 10.13.1, but not 10.12.6.
It could also be that the issue is on the WebListener side. there could be something in the 10.12.6 environment that is causing kestrel to bind the wrong protocols to the ports.
issue between CoreFX and macOS 10.12.6
I guess that the Issue is that CoreFX uses libcurl. We inplace CoreFX libraries in PowerShell Core folder. I guess we could put there a libcurl of needed version.
Ilya (@iSazonov) Again, the libcurl is not the issue here. the exact same libcurl library version works fine on macOS 10.12.5 and macOS 10.13.1.
I would caution against packaging libcurl with the project. My experience is that it is one of the least portable Linux libraries.
We already have a step in the build process that ensure the right version is installed. i mistakenly thought other wise earlier in digging into this issue.
Mark Kraus (@markekraus) Thanks for clarify!
Can you test Kestrel by browser locally on MacOS? (On Windows in IE we can select TLS versions.)Ilya (@iSazonov) the problem I have is that I cannot reproduce this in my mac environment. Dongbo Wang (@daxian-dbw) was able to on his 10.12.6 system.
But yes, you can test WebListener in a browser
Import-Module ./build.psm1 Publish-PSTestTools $WebListener = Start-WebListener
Then use the
Get-WebListenerUrlto get the various URLs.The problem is that this all needs to be don on an environment that is like the one in Travis CI.
Before Dongbo Wang (@daxian-dbw) opened #5605 I was trying to do some troubleshooting in Travis CI, but it started taking 4-5 hours to start the macOS builds. 😭
Best way to test would be to use
openssl s_clientagainst the WebListener on you can see what TLS version is used in the output.echo -e "GET / HTTP/1.1\nHost: localhost\n\n" | openssl s_client -connect localhost:8084 -ign_eof 2>&1 | grep 'TLSv' | grep 'Protocol' echo -e "GET / HTTP/1.1\nHost: localhost\n\n" | openssl s_client -connect localhost:8085 -ign_eof 2>&1 | grep 'TLSv' | grep 'Protocol' echo -e "GET / HTTP/1.1\nHost: localhost\n\n" | openssl s_client -connect localhost:8086 -ign_eof 2>&1 | grep 'TLSv' | grep 'Protocol'
should be
Protocol : TLSv1.2 Protocol : TLSv1.1 Protocol : TLSv1This may be related to libcurl after all.
I have done a ton of testing on my mac and I have discovered that no matter what brew version of curl/libcurl I have installed and no matter how I try to trick pwsh into using the correct lubcurl, it keeps using the system supplied one in /usr/lib/, which uses LiberSSL.
I was testing if the newer version of libcurl and dotnet sdk resolved the inability to use certificate authentication on macOS and it spits this at me:
The handler does not support client authentication certificates with this combination of libcurl (7.54.0) and its SSL backend ("LibreSSL/2.0.20").Which is odd considering I have curl 7.57.0 (with OpenSSL) installed by brew.
curl --version
curl 7.57.0 (x86_64-apple-darwin17.2.0) libcurl/7.57.0 OpenSSL/1.0.2m zlib/1.2.11 Release-Date: 2017-11-29 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxyYou can reproduce the error with:
Import-Module ./build.psm1 Publish-PSTestTools $Url = Get-WebListenerUrl -Test Cert $certificate = Get-WebListenerClientCertificate $result = Invoke-WebRequest -Uri $url -Certificate $certificate -SkipCertificateCheck
There hasn't been any way that I have found to see what version of libcurl is using without making it throw an error or some kind.
I'm used to compilers having away to include the correct library when a system has multiple, but I have no clue how do to that with
dotnet. I haved tried settingLD_LIBRARY_PATHto/usr/local/opt/curl/lib, I'v tried settingLDFLAGSto-L/usr/local/opt/curl/lib, i have tred copying the correct libcurl files to the same directory aspwsh.... no matter what i do it sill choses the system libcurl files.Update: The other comments are also useful.
Ilya (@iSazonov) thanks. I'm not sure that plays into this, but at least that pointed me in a direction where I can test this. https://github.com/markekraus/PowerShell/blob/TestMacosFails/test/powershell/Libcurl.Tests.ps1
This is the build job (still pending) https://travis-ci.org/markekraus/PowerShell/builds/311234209
Yep, The reason why this is failing is because it is using the wrong libcurl:
https://travis-ci.org/markekraus/PowerShell/builds/311234209#L2884
LIBCURl===================================================== LIBCURL GetSslVersionDescription(): SecureTransport LIBCURL GetSupportedFeatures(): CURL_VERSION_IPV6, CURL_VERSION_SSL, CURL_VERSION_LIBZ, CURL_VERSION_NTLM, CURL_VERSION_ASYNCHDNS, CURL_VERSION_SPNEGO, CURL_VERSION_LARGEFILE, CURL_VERSION_NTLM_WB, CURL_VERSION_GSSAPI, CURL_VERSION_KERBEROS5, CURL_VERSION_UNIX_SOCKETS LIBCURL GetSupportsHttp2Multiplexing(): False LIBCURL GetVersionDescription(): 7.54.0 LIBCURl=====================================================Mark Kraus (@markekraus) Great work! Could you get a path to the curl? Can reflection help?
Maybe related #4029
Ilya (@iSazonov) from what I can tell, .NET calls the
System.Net.Http.Nativeassembly (link) and not directly to libcurl. That assembly either contains links to the system libcurl or has that libcurl baked into it. I'm not sure what voodoo magic it is doing in that assembly. I'm not sure how you generate that assembly other than to obtain it through nuget. This part of the problem is out of my depth as I am very very unfamiliar with the dotent build process. We need someone who knows what they're doing in that knowledge area to take a look.re: #4029 I suspected that at first there was some macOS/libcurl/CoreFX issue at play, but I went digging into the code and was not sure it is even using libcurl for any of that. Looks to me like they are missing the brew version of openssl.
It seems both Issues related with wrong library version (curl and OpenSSL/LibreSSL).
So questions is:- Where is the root of the problem? How CoreFX looks for the libraries?
- How can we make it stable on all platforms and distributions?
Dongbo Wang (@daxian-dbw) Steve Lee (@SteveL-MSFT) Can you ping CoreFX team?
Mark Kraus (@markekraus) Ilya (@iSazonov) I remember seeing a PR in CoreFx about removing OpenSSL dependencies on macOS, and digged into the CoreFx PRs, this is what I found: Remove OpenSSL dependency on macOS (dev/apple_crypto -> master).
Quoted from the PR description:
only the types whose name includes "OpenSsl" (e.g. RSAOpenSsl) will use OpenSSL on macOS. Instead, everything is provided by Security.framework.
If I understand that PR correctly, previously they were directly using OpenSSL in the project for a bunch of operations and removed that. I don't think this changes anything regarding the issue at hand. You should still be able to plug in the
libcurlof your choice as HttpClient is just wrapping it. If 'libcurl' is using OpenSSL then you get the supported features such as Certificate authentication and not-brokenHttpClientHandler.SslProtocols.What is happening instead is that "System.Net.Http.Native" is only using the system provided libcurl. on 10.13.1 that is libcurl+LibreSSL, on 10.12.6 that is libcurl+SecureTransport. We have a step in the build process to install curl with openssl, but instead of it actually using that libcurl, it is favoring the system libcurl.
The brew formula says it all:
==> Caveats This formula is keg-only, which means it was not symlinked into /usr/local, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble. If you need to have this software first in your PATH run: echo 'export PATH="/usr/local/opt/curl/bin:$PATH"' >> ~/.bash_profile For compilers to find this software you may need to set: LDFLAGS: -L/usr/local/opt/curl/lib CPPFLAGS: -I/usr/local/opt/curl/include For pkg-config to find this software you may need to set: PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfigThis at least explains why a change in the OS version by Travis CI would affect the web cmdlets tests: new OS version means a different libcurl version and possibly even a different crypto provider (in the case of 10.12-to-10.13).
That means that the features available and working are beholden to the underlying OS unless we have some way to specify or include the libcurl that is called to ensure it is the one installed from brew.
Mark Kraus (@markekraus) Thanks for figuring out the root cause!
We have a step in the build process to install curl with openssl, but instead of it actually using that libcurl, it is favoring the system libcurl.
The step in our bootstrapping script that installs
curlandopensslis actually the leftover from old days when .NET Core had a hard dependency onopenssl. The PR name"Remove OpenSSL dependency on macOS"seems to suggest that we don't need this step anymore 😄You should still be able to plug in the libcurl of your choice as HttpClient is just wrapping it.
Mark Kraus (@markekraus) Can you please open an issue in CoreFx to check with the CoreCLR team whether it's supported for users to plug in the
libcurlof their choices forHttpClientto use?I have one more thing to try (setting the DYLD_LIBRARY_PATH env var before starting and compiling) and if that doesn't work I will open an issue.
Reacted by Dongbo Wangalso:
seems to suggest that we don't need this step anymore
Except that I think it is still being used by other parts of PowerShell, like omi https://github.com/PowerShell/psl-omi-provider/blob/master/src/CMakeLists.txt 😃
🎉 YES!! 🎉
Addingexport DYLD_LIBRARY_PATH=/usr/local/opt/curl/lib:${DYLD_LIBRARY_PATH};to the .travis.yml made pwsh pickup the correct libcurl. Now I just need to test that it doesn't break anything.
https://travis-ci.org/markekraus/PowerShell/builds/311469327#L2912Reacted by Dongbo Wang and Steve Lee
Same failures from 2 macOS daily builds. And we got the similar failures for
Invoke-RestMethod -SslProtocol.