Repository navigation
Tests should not rely on external websites which lead to random failures #2504
Description
Today https://httpbin.org/ is down the whole day.
It caused two daily builds to fail with 43 tests for Invoke-WebRequest and Invoke-RestMethod.
https://ci.appveyor.com/project/PowerShell/powershell-f975h/build/6.0.0-alpha.10-26
https://ci.appveyor.com/project/PowerShell/powershell-f975h/build/6.0.0-alpha.11-27
We should replace httpbin with eaver our own hosted version or localhost server, i.e.
https://github.com/Runscope/httpbin
localhost server will introduce unnecessary testing dependencies, not sure which one would be a better solution.
cc Dongbo Wang (@daxian-dbw) Francisco Gamino (@Francisco-Gamino)
Activity
daxian-dbw commentedon Oct 19, 2016
I prefer to host the service in cloud ourselves. I got it hosted at Heroku, but still not clear if the free plan would be sufficient.
I tried to get it running on Azure using MSDN subscription, but failed and I didn't get the time to resolve it.
SteveL-MSFT commentedon May 22, 2017
Move tests to use httplistener where we have more control and also suppress progress for the tests
SteveL-MSFT commentedon Jun 22, 2017
Also review for other external websites being used beyond httpbin.org
[-]https://httpbin.org/ is unreliable for web requests testing [/-][+]Tests should not rely on external websites which lead to random failures[/+]markekraus commentedon Aug 31, 2017
Travis Plunk (@TravisEz13) I plan to move all of the HTTPS tests to WebListener (#4719).
For HTTP (no-S) tests that are currently pointed to httpbin I plan to pick the one (possibly /get), create the same functionality in WebListener, and move all the tests that use it to WebListener. Before I begin, though, I'd like to just confirm that this is the right path to take. I figured I would start moving one httpbin target URL at a time and then work on moving the HttpListener tests one URL at a time. The end goal being that all HTTP/HTTPS tests will run against WebListener and HttpListener can be retired.
SteveL-MSFT commentedon Aug 31, 2017
Mark Kraus (@markekraus) sounds right to me. Thanks!
markekraus commentedon Sep 1, 2017
Steve Lee (@SteveL-MSFT) How should I do the PR's for these? Should I create a One PR for adding the WebListener functionality and another for switching the tests it replaces? Or just lump the WebListener functionality in with the test changes in a single PR for each URL replacement?
SteveL-MSFT commentedon Sep 1, 2017
Mark Kraus (@markekraus) I think it would be better to have both the WebListener changes and test code changes together for each URL replacement so the reviewers can understand what's happening on both the client side and server side
markekraus commentedon Dec 10, 2017
after #5665 is merged, there is only 4 tests which rely on http://http.lee.io/method, all other tests rely on WebListener or HttpListener.
TravisEz13 commentedon Dec 20, 2017
Mark Kraus (@markekraus) Thank you for all your hard work!
And everyone else that helped review these.
iSazonov commentedon Dec 20, 2017
Mark Kraus (@markekraus) Many thanks! 🍻