Rerun flaky tests that fail in setup - #148

I think this new condition should only apply to the failure case (not the success case).

So maybe we can rewrite the following few lines like this:

if report.when == self._PYTEST_WHEN_CALL and report.outcome == self._PYTEST_OUTCOME_PASSED:
    if self._should_handle_test_success(item):
        log = False
elif report.when in self._PYTEST_WHENS and report.outcome == self._PYTEST_OUTCOME_FAILED:
    err, name = self._get_test_name_and_err(item, when)
    if self._will_handle_test_error_or_failure(item, name, err):
        log = False

I think that should still allow rerunning tests that fail in setup, and it won't erroneously count every test success as having passed twice.

I'm not sure what you mean about counting every test success as having passed twice. Here's output from running one test in which the setup failed once, then passed:

`
[sscadmin@57a98df50c92 ~]$ pytest -v ~/test_a.py
========================================================================== test session starts ===========================================================================
platform linux2 -- Python 2.7.17, pytest-4.6.11, py-1.9.0, pluggy-0.13.1 -- /home/sscadmin/.local/share/virtualenvs/flaky/bin/python2.7
cachedir: .pytest_cache
rootdir: /home/sscadmin
plugins: cov-2.10.0, forked-1.2.0, xdist-1.32.0, flaky-3.6.1
collected 1 item

test_a.py::test_foo PASSED [100%]
===Flaky Test Report===

test_foo failed (2 runs remaining out of 3).
<type 'exceptions.AssertionError'>
assert 0.615803923129647 < 0.3

• where 0.615803923129647 = <built-in method random of Random object at 0x81b6f0>()
• where <built-in method random of Random object at 0x81b6f0> = random.random
  [<TracebackEntry /home/sscadmin/test_a.py:8>]
  test_foo passed 1 out of the required 1 times. Success!

===End Flaky Test Report===

======================================================================== 1 passed in 0.04 seconds ========================================================================
(
`

I think you're right! Sorry about the confusion.

Thanks!