Skip to content

Cut out evaluation in the "pythoneval" tests #1671

Closed

Description

@gnprice

The "pythoneval" tests are by far the biggest contributor to the time it takes to run our test suite. For each of 171 sample programs, we type-check them and then run them through the Python interpreter.

From discussion in #1668, running the programs is no longer important for our testing. So cut it out. This will mean

  • cutting out that step in testpythoneval.py,
  • cutting out the program output (as opposed to the type-checker output) from all the test cases, and
  • cutting out entirely any cases that no longer seem to have any meaningful content when they're only for the type-checker.

Some of the test cases may exercise features of typing in interesting ways -- any of those should be moved to the typing repo's own test suite if they aren't covered there already.

After this is done, we can probably get a big further speedup by folding the remaining test cases into just more cases for testcheck.py, so that we're not forking a mypy subprocess for each one. Will make a separate issue for that.

Activity

  1. added a commit that references this issue on Sep 27, 2017
    98c0f33
  2. emmatyping commented on May 27, 2018

    @emmatyping
    Member

    I just tested completely removing this and it doesn't seem to actually be that big of a runtime hit. WIth Guido's points about checking real code: #4025 (comment), I think we can just close this, the majority of the runtime seems to be in type checking the files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions