Use incremental mode in python evaluation test cases #1412

Activity

1. gvanrossum commented on Apr 20, 2016

   

   gvanrossum

   on Apr 20, 2016

   Member

   More actions

   I looked into this for a few minutes and one problem is that currently each test is run in a clean directory. There currently is no way to change the name of the cache directory, and it's always .mypy_cache relative to the current directory. So we'd need something for this. (I've filed #1414 to track it.)
2. gvanrossum

   added

   topic-incremental

   on Apr 20, 2016
3. ilevkivskyi

   added

   topic-tests

   priority-1-normal

   on May 18, 2018
4. emmatyping commented on May 18, 2018

   

   emmatyping

   on May 18, 2018

   Member

   More actions

   I just tried making the pythoneval tests use a shared cache directory via this diff:

   diff --git a/mypy/test/testpythoneval.py b/mypy/test/testpythoneval.py
   index cf4cbafd..9c207d5d 100644
   --- a/mypy/test/testpythoneval.py
   +++ b/mypy/test/testpythoneval.py
   @@ -14,7 +14,7 @@ import os
    import os.path
    import re
    import sys
   -
   +from tempfile import TemporaryDirectory
    import pytest  # type: ignore  # no pytest in typeshed
   
    from typing import List
   @@ -37,12 +37,12 @@ class PythonEvaluationSuite(DataSuite):
                 'pythoneval-asyncio.test']
        base_path = test_temp_dir
        optional_out = True
   -
   +    cache_dir = TemporaryDirectory()
        def run_case(self, testcase: DataDrivenTestCase) -> None:
   -        test_python_evaluation(testcase)
   +        test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache'))
   
   
   -def test_python_evaluation(testcase: DataDrivenTestCase) -> None:
   +def test_python_evaluation(testcase: DataDrivenTestCase, cache_dir: str) -> None:
        """Runs Mypy in a subprocess.
   
        If this passes without errors, executes the script again with a given Python
   @@ -71,6 +71,7 @@ def test_python_evaluation(testcase: DataDrivenTestCase) -> None:
        with open(program_path, 'w') as file:
            for s in testcase.input:
                file.write('{}\n'.format(s))
   +    mypy_cmdline.append('--cache-dir={}'.format(cache_dir))
        output = []
        # Type check the program.
        out, err, returncode = api.run(mypy_cmdline)
   

   However I get a failure (consistently) with:

   _______________ testCanConvertTypedDictToAnySuperclassOfMapping _______________
   [gw8] win32 -- Python 3.6.5 C:\Python36\python.exe
   data: C:\Users\ethanhs\Documents\mypy\test-data\unit\pythoneval.test:1081:
   ..\..\..\..\Documents\mypy\mypy\test\testpythoneval.py:42: in run_case
       test_python_evaluation(testcase, os.path.join(self.cache_dir.name, '.mypy_cache'))
   ..\..\..\..\Documents\mypy\mypy\test\testpythoneval.py:92: in test_python_evaluation
       testcase.file, testcase.line))
   ..\..\..\..\Documents\mypy\mypy\test\helpers.py:98: in assert_string_arrays_equal
       raise AssertionError(msg)
   E   AssertionError: Invalid output (C:\Users\ethanhs\Documents\mypy\test-data\unit\pythoneval.test, line 1081)
   ---------------------------- Captured stderr call -----------------------------
   Expected:
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: error: Incompatible...
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Following mem...
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Expected:
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def __iter__(self) -> Iterator[int] (diff)
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Got:
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def __iter__(self) -> Iterator[str] (diff)
   Actual:
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: error: Incompatible...
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note: Following mem...
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Expected:
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def () -> Iterator[int] (diff)
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:     Got:
     _testCanConvertTypedDictToAnySuperclassOfMapping.py:11: note:         def () -> Iterator[str] (diff)
   
   Alignment of first line difference:
     E: ...y:11: note:         def __iter__(self) -> Iterator[int]
     A: ...y:11: note:         def () -> Iterator[int]
                                   ^
   

   this appears to be an incremental bug?
5. ilevkivskyi commented on May 18, 2018

   

   ilevkivskyi

   on May 18, 2018

   Member

   More actions

   Yes, it looks like name of a method is lost during (de-)serialisation. If you have time and desire, you could fix this yourself, otherwise create a separate issue about this.
6. ilevkivskyi commented on May 19, 2018

   

   ilevkivskyi

   on May 19, 2018

   Member

   More actions

   @ethanhs I think the fail you see is the same as #4772, where also method name disappears after loading from cache
7. emmatyping commented on May 19, 2018

   

   emmatyping

   on May 19, 2018

   Member

   More actions

   @ilevkivskyi that makes sense. I'm afraid I'm not familiar enough with the intricacies of incremental to debug this one. I'll bump the priority of that issue up however.
8. ilevkivskyi commented on May 24, 2018

   

   ilevkivskyi

   on May 24, 2018

   Member

   More actions

   @ethanhs #5109 should unblock this.
9. ilevkivskyi

   added a commit that references this issue on May 26, 2018

   Preserve the name of first argument in methods when serializing (#5109)

   ...

   c9a5d56
10. ilevkivskyi

    closed this as completedin #5114on May 26, 2018
11. ilevkivskyi

    added a commit that references this issue on May 26, 2018

    Use a shared incremental cache for pythoneval tests (#5114)

    ...

    This issue will close once commit d3b32b7 is merged into the 'master' branch.

    d3b32b7