Repository navigation
Maybe test SAMP and other networking code with mock sockets? #2126
Description
A lot of the "random" test failures on Travis of late are mostly, I think, due to interactions with OS sockets over which we don't have full control, and which can lead to unpredictable behavior.
I wonder if it wouldn't be better to just mock the entire socket module for these purposes, using a lightweight Python replacement for sockets without touching real OS level sockets. For the most part I don't think we require any advanced socket features so this should be reasonably straightforward.
There is one project already called "mocket" which provides this: https://github.com/mocketize/python-mocket, though it doesn't seem to have had much active development in the last year. It still looks okay though. There's also a mock_socket.py hanging out in Python itself: http://hg.python.org/cpython/file/718cddfa0292/Lib/test/mock_socket.py
This is not a normally installed module, but it could just be copied verbatim with the proper attribution. Or, depending on what we determine our actual needs to be, it may be straightforward enough to roll our own.
This isn't something I'd want to put a lot of effort into if it winds up being troublesome. But I think it could really improve the reliability of our tests without implicitly also testing sockets, which really doesn't help us in general.
Activity
astrofrog commentedon Feb 24, 2014
This is a great idea!
cdeil commentedon Apr 12, 2014
For reference: there's a bunch of "random" travis-ci Astropy test errors reported in Issue #2064.
It would be really nice to get rid of those.
cdeil commentedon Apr 12, 2014
I'm not sure this is a good idea, because the travis-ci boxes are just normal Ubuntu machines, so if there are "random" test errors there, they will very likely also occur for Astropy users.
If that happens in interactive use it's confusing and annoying, it if crashes an analysis pipeline on the weekend it costs time and money.
Which Astropy modules besides
astropy.vo.sampuse sockets?Is it possible to rewrite those to avoid or better handle the socket errors?
If not, then I think we should put a warning in the docs in addition to mockig out or xfailing the unit tests with "random" socket failures.
(I'm not familiar with sockets, so I won't be helpful with solving this issue at all, but I wanted to mention this as a user's perspective.)
astrofrog commentedon Apr 12, 2014
@cdeil - I agree that fixing the issue in the first place is what would be needed, but one advantage of a mock socket module would be to actually be able to more easily debug what is going on and why things are failing. I've spent hours on trying to understand if there are race conditions, and I found a few, but then I got stuck. I think the issues could be related to multi-threading and the balancing between the threads, but I couldn't figure out anything beyond that.
embray commentedon Apr 14, 2014
Yes, I was just going to add, a lot of these failures don't seem to be caused by bugs the code or even any realistic use cases. But rather by the fact that it's running on a machine that many other peoples' tests are running on simultaneously and I suspect that by requesting lots of sockets from the system the tests are just running up against system resource limits that wouldn't happen in a normal case. In particular, most of the failures appear to be timeout errors.
A decent enough mock socket library would reproduce all the functionality that matters but without actually using up any system resources like real sockets.
(That's not to say there are no bugs in the code, but the problems we're having seem more likely to be related to the extreme conditions under which the tests are being run.)
astrofrog commentedon Jun 15, 2017
I'm going to go ahead and close this - even though this could potentially be nice in a general sense, the only failures that occurred were related to the HTTPS/SSL tests, and SSL support has been removed in #6201. We can always re-open in future if we run into similar issues, but for now the problem is solved.