Skip to content

Support PEP 612 in typing_extensions (Python 3) - #774

Merged
JelleZijlstra merged 14 commits into
python:masterfrom
Fidget-Spinner:pep612
Apr 10, 2021
Merged

JelleZijlstra merged 14 commits into
python:masterfrom
Fidget-Spinner:pep612

Conversation

@Fidget-Spinner

Copy link
Copy Markdown
Member

Runtime correctness is zero due to wacky list workarounds. Unfortunately, I'm not enough of an MRO/metaclass wizard to know how to trick isinstance(arg, list) in Callable while still returning a typing._GenericAlias.

The first patch is for Python3. I'm mildly reluctant to write one for Python2 since it's EOL-ed for a year already, but we'll see ;).

@Fidget-Spinner Fidget-Spinner changed the title Support PEP 612 for typing_extensions in Python 3 Support PEP 612 in typing_extensions (Python 3) Jan 2, 2021
@Fidget-Spinner

Copy link
Copy Markdown
Member Author

@gvanrossum Would you like to see Python 2 supported as well? It's possible to have runtime correctness for Python 2 since I can edit the typing PyPI module as well, though that would require more work.

@gvanrossum

Copy link
Copy Markdown
Member

Let’s skip Python 2. Sorry for the slow review, will get to it soon.

@Fidget-Spinner

Fidget-Spinner commented Jan 11, 2021 •

Copy link
Copy Markdown
Member Author

Let’s skip Python 2. Sorry for the slow review, will get to it soon.

No worries, I didn't mean to rush and I hope it didn't come off that way. Please take your time :).

@gvanrossum

gvanrossum commented Jan 11, 2021 via email

Copy link
Copy Markdown
Member

@gvanrossum

Copy link
Copy Markdown
Member

@JelleZijlstra Have you reviewed this? I have sadly dropped the ball (again). If you approve I will land it.

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for letting this linger! I took a look at the code and I have some suggested improvements.

Comment thread typing_extensions/src_py3/typing_extensions.py
Comment thread typing_extensions/src_py3/typing_extensions.py Outdated
Comment thread typing_extensions/src_py3/typing_extensions.py
Comment thread typing_extensions/src_py3/typing_extensions.py
Comment thread typing_extensions/src_py3/typing_extensions.py
Comment thread typing_extensions/src_py3/typing_extensions.py
@JelleZijlstra

Copy link
Copy Markdown
Member

Also, I noticed we're not running 3.9 in CI, so I opened #796 to add it.

@Fidget-Spinner
Fidget-Spinner marked this pull request as draft April 6, 2021 15:23
@Fidget-Spinner

Copy link
Copy Markdown
Member Author

@JelleZijlstra - Thanks! I really appreciate the reviews. I marked the PR as draft because you gave me some other ideas to improve the runtime information. So I'm working on them now.

This PR was originally just to implement all the basic stuff required for type checkers, which is why it's so barren in terms of runtime information.

@Fidget-Spinner
Fidget-Spinner marked this pull request as ready for review April 6, 2021 16:54
@Fidget-Spinner

Copy link
Copy Markdown
Member Author

@JelleZijlstra this should be ready now. Please take a look when you have the time. Thanks!

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I checked out the code and played around with it some more and I have two suggestions to make the new types a bit more usable at runtime.

Comment thread typing_extensions/src_py3/typing_extensions.py
Comment thread typing_extensions/src_py3/typing_extensions.py
Note that only parameter specification variables defined in global scope can
be pickled.
"""
args = object()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit unhappy with this because it makes the runtime annotations just look like this:

In [11]: takes_int_str.__annotations__
Out[11]: 
{'args': <object at 0x7ff1831d5f30>,
 'kwargs': <object at 0x7ff1831d5f60>,
 'return': ~R}

However, it's the exact same in CPython, so it makes sense to follow the same principle here. I'll plan to submit a change to CPython to make this instead return some helper object pointing back to the ParamSpec; if that gets accepted we can apply the same change here.

For now no change is required here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was definitely something I overlooked when I implemented the CPython version :(. How I'd do it in typing.py:

  1. Create a _ParamSpecGenericAlias.
  2. __origin__ is the ParamSpec it belongs to, __args__ is just a tuple with 1 element - "args" or "kwargs".
  3. Override the __repr__ to print ~T.(args|kwargs) instead of the default ~T[args] found in genericaliases.

I look forward to seeing your PR!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was thinking something similar, except with different classes for args and kwargs, but either way should work.

Comment thread typing_extensions/src_py3/typing_extensions.py Outdated
@JelleZijlstra

Copy link
Copy Markdown
Member

Looks like this broke CI because ParamSpec objects aren't hashable. I'm guessing we could make them hashable by overriding __hash__ and hoping that nobody will actually modify the underlying list.

@Fidget-Spinner

Copy link
Copy Markdown
Member Author

Hmm the maximum recursion depth exceeded errors due to _tp_cache in _concatenate_getitem are particularly hairy.

Comment thread typing_extensions/src_py3/typing_extensions.py Outdated
@JelleZijlstra

Copy link
Copy Markdown
Member

Hmm the maximum recursion depth exceeded errors due to _tp_cache in _concatenate_getitem are particularly hairy.

It reproduces only if some other test runs before it, not if you just run the line that fails on its own. I can spend some time later today trying to figure out where it's going wrong.

@JelleZijlstra

Copy link
Copy Markdown
Member

I tracked it down to ParamSpec("P") == ParamSpec("P2") failing. It works if you add def __eq__(self, other): return self is other. Comparing by identity seems reasonable to me because that's also how TypeVars work: TypeVar("T") == TypeVar("T") returns False.

@Fidget-Spinner

Copy link
Copy Markdown
Member Author

I tracked it down to ParamSpec("P") == ParamSpec("P2") failing. It works if you add def __eq__(self, other): return self is other. Comparing by identity seems reasonable to me because that's also how TypeVars work: TypeVar("T") == TypeVar("T") returns False.

Thank you for digging out the cause. I had forgotten that dictionary keys need to implement both __hash__ and __eq__ to work properly. This was a good refresher.

@JelleZijlstra JelleZijlstra left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Could you also add some unit tests doing == and hash() on ParamSpec and Concatenate? We should make sure they keep working if we ever refactor these classes here. With those changes this should be good to go.

Comment thread typing_extensions/src_py3/typing_extensions.py
@Fidget-Spinner

Copy link
Copy Markdown
Member Author

Thanks! Could you also add some unit tests doing == and hash() on ParamSpec and Concatenate? We should make sure they keep working if we ever refactor these classes here. With those changes this should be good to go.

Thanks for the reminder, that slipped my mind.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants