Skip to content

How do I type a generic factory function? #6073

Closed
Closed
How do I type a generic factory function?#6073
Labels
featureneeds discussionpriority-1-normal

Description

@njsmith
njsmith
opened on Dec 15, 2018

[Not sure if this is the right place to file issues like this, feel free to tell me to go somewhere else :-)]

I have two generic types:

class SendChannel(Generic[T]):
    ...

class ReceiveChannel(Generic[T]):
    ...

I have a factory function that creates a linked pair of these objects:

def open_channel_pair(buffer_size: int) -> Tuple[SendChannel[T], ReceiveChannel[T]]:
    # ... construct some objects ...
    return (s, r)

I would like to allow people to call it like:

s, r = open_channel_pair[str](0)
# Now 's' is a SendChannel[str], and 'r' is a ReceiveChannel[str]

This works for the factory callables built into the language, e.g. calling SendChannel[str]() returns an object of type SendChannel[str]. This is basically the same thing, but since I have to construct two objects in a single operation, the regular thing doesn't work.

I did come up with a sneaky hack:

# Complete cut-and-pasteable example
from typing import Generic, TypeVar, Tuple

T = TypeVar("T")

class SendChannel(Generic[T]):
    pass

class ReceiveChannel(Generic[T]):
    pass

class open_channel_pair(Tuple[SendChannel[T], ReceiveChannel[T]]):
    def __new__(self, buffer_size: int):
        return (SendChannel[T](), ReceiveChannel[T]())

    # Note: this function is unreachable, but we have to define it, and keep
    # its signature in sync with __new__'s signature, to make mypy happy.
    def __init__(self, buffer_size: int) -> None:
        assert False

s, r = open_channel_pair[str](0)
# Confirms that it works:
reveal_type(s)
reveal_type(r)

This is gross, though. Is there any way to define a regular callable that has the same type as a generic class constructor, but is less gross than this?

What even is the type of a generic class constructor? reveal_type(SendChannel) prints def [T] () -> SendChannel[T`1], which is a very strange looking type.

Activity

  1. ilevkivskyi commented on Dec 16, 2018

    @ilevkivskyi
    ilevkivskyi
    on Dec 16, 2018
    Member

    This is not really mypy's problem. This just doesn't work at runtime. From mypy perspective it is totally fine. For example, as you can see in your initial version of the code:

    s, r = open_channel_pair[str](0) # Type application is only supported for generic classes
reveal_type(s)  # Revealed type is 'SendChannel[builtins.str*]'
reveal_type(r)  # Revealed type is 'ReceiveChannel[builtins.str*]'

    the first error is bogus, because it is actually supported in mypy, just not at runtime. I think we should just remove this error, so that one can at least use cast('open_channel_pair[str]', open_channel_pair)(0).

    Later we can support a safer version of cast for such cases (we already discussed this), or you can try discussion whether this can be allowed at runtime on CPython forums (a trivial __getitem__ returning self on types.FunctionType).

    What even is the type of a generic class constructor? reveal_type(SendChannel) prints def [T] () -> SendChannel[T`1], which is a very strange looking type

    What is wrong with this type? It looks totally fine to me.

  2. ilevkivskyi
    added
    feature
    priority-1-normal
    on Dec 16, 2018

  3. njsmith commented on Dec 16, 2018

    @njsmith
    njsmith
    on Dec 16, 2018
    Author

    I think we should just remove this error, so that one can at least use cast('open_channel_pair[str]', open_channel_pair)(0).

    It seems like this might need an update to PEP 484? And ideally some more details on exactly what the semantics are :-).

    or you can try discussion whether this can be allowed at runtime on CPython forums (a trivial __getitem__ returning self on types.FunctionType).

    It's also easy to hack a __getitem__ in without changing the language:

    class generic_function:
    def __init__(self, fn):
        self._fn = fn

    def __call__(self, *args, **kwargs):
        return self._fn(*args, **kwargs)

    def __getitem__(self, _):
        return self

@generic_function
def open_channel_pair(...):
    ...

    ...except that then I can't figure out how to type generic_function without breaking mypy further.

    I tried the blunt hammer of generic_function = cast("Callable[[T], T]", _generic_function) (i.e. "this decorator just returns whatever it gets, please ignore it entirely"), but mypy doesn't ignore it – I get:

    error: Value of type "Callable[[int], Tuple[SendChannel[T], ReceiveChannel[T]]]" is not indexable

    and both revealed types are Any. So apparently passing the magic generic function type through the identity function strips off the magic.

    What is wrong with this type? It looks totally fine to me.

    It's not... type syntax? Like I can definitely not write my_factory: def [T] () -> SendChannel[T`1] = ..., right? I guess I could stick it in a string, but I don't remember seeing anything like that in PEP 484...

    I guess in PEP 484 it is normal that you can't point to a function and say "what's the type of this?" (e.g., anything with keyword arguments also has an inexpressible type). It's just not what what other type systems have trained me to expect.

  4. njsmith commented on Dec 16, 2018

    @njsmith
    njsmith
    on Dec 16, 2018
    Author

    I found another way to write this that works:

    class open_channel_pair_type:
    def __call__(self, buffer_size: int) -> Tuple[SendChannel[Any], ReceiveChannel[Any]]:
        # ... insert real implementation here ...
        return (SendChannel(), ReceiveChannel())

    def __getitem__(self, _: Type[T]) -> Callable[[int], Tuple[SendChannel[T], ReceiveChannel[T]]]:
        return self

open_channel_pair = open_channel_pair_type()

    I guess this is less gross than abusing __new__, though it still has a lot of repetition, and is going to get even wordier if we ever make buffer_size optional or add kwonly arguments.

  5. ilevkivskyi commented on Dec 18, 2018

    @ilevkivskyi
    ilevkivskyi
    on Dec 18, 2018
    Member

    I guess in PEP 484 it is normal that you can't point to a function and say "what's the type of this?" (e.g., anything with keyword arguments also has an inexpressible type). It's just not what what other type systems have trained me to expect.

    The syntax of reveal_type() is currently not standardized in any way (and unfortunately not documented). There is a proposal python/typing#277, but it didn't get any traction.

    It's also easy to hack a __getitem__ in without changing the language:

    Requiring a decorator on every generic function is a big change. We can of course clearly document that it is optional, and is only required if one wants to subscript the function. In any case, I think this requires a wider discussion.

  6. ilevkivskyi
    added
    needs discussion
    on Dec 18, 2018

  7. njsmith commented on Dec 19, 2018

    @njsmith
    njsmith
    on Dec 19, 2018
    Author

    Requiring a decorator on every generic function is a big change. We can of course clearly document that it is optional, and is only required if one wants to subscript the function. In any case, I think this requires a wider discussion.

    Oh yeah, I just meant to use it on that subset of generic functions where you explicitly specify the type when calling. Not sure if there's a name for that.

    Is there a better place to have this discussion for people to see it, or...?

  8. gvanrossum commented on Dec 19, 2018

    @gvanrossum
    gvanrossum
    on Dec 19, 2018 via email
    Member

  9. oremanj commented on Feb 6, 2019

    @oremanj
    oremanj
    on Feb 6, 2019
    Contributor

    @njsmith 's most recent suggestion doesn't work for "special forms", because they can't be inferred as the T in Type[T].. Not sure if this is a bug in mypy or intended behavior.

    from typing import Generic, TypeVar, List, Tuple, Callable, Any, Type, Awaitable

T = TypeVar("T")

class SendChannel(Generic[T]): ...
class ReceiveChannel(Generic[T]): ...

class open_channel_pair_type:
    def __call__(self, buffer_size: int) -> Tuple[SendChannel[Any], ReceiveChannel[Any]]:
        # ... insert real implementation here ...
        return (SendChannel(), ReceiveChannel())

    def __getitem__(self, _: Type[T]) -> Callable[[int], Tuple[SendChannel[T], ReceiveChannel[T]]]:
        return self

open_channel_pair = open_channel_pair_type()

reveal_type(open_channel_pair[int](5))
reveal_type(open_channel_pair[List[int]](10))
reveal_type(open_channel_pair[Tuple[int, str]](10))
reveal_type(open_channel_pair[Callable[[], Awaitable[None]]](10))

    shows that int and List[int] work, but the tuple and callable forms don't.

    t.py:18: error: Revealed type is 'Tuple[t.SendChannel[builtins.int*], t.ReceiveChannel[builtins.int*]]'
t.py:19: error: Revealed type is 'Tuple[t.SendChannel[builtins.list*[builtins.int*]], t.ReceiveChannel[builtins.list*[builtins.int*]]]'
t.py:20: error: Revealed type is 'Tuple[t.SendChannel[Any], t.ReceiveChannel[Any]]'
t.py:21: error: Revealed type is 'Tuple[t.SendChannel[Any], t.ReceiveChannel[Any]]'

  10. njsmith commented on Feb 8, 2019

    @njsmith
    njsmith
    on Feb 8, 2019
    Author

    @oremanj Oh, that is a subtle flaw indeed, nice catch.

    I just tried it, and it looks like my original version (the class open_channel_pair(Tuple[...]) one) does work in all your tricky cases, so that's still an option.

    @ilevkivskyi's trick of simply writing def open_channel_pair() -> Tuple[...] also works with these tricky cases, if we ignore the "Type application is only supported for generic classes" error.

    Probably someone should start a discussion about this on typing-sig, basically suggesting making the def version legal (probably with some decorator to allow the __getitem__ syntax). Unfortunately I've been feeling exhausted at the idea of signing up for yet another mailing list just to have one conversation :-/. It'd be easier if it were a category on discourse, though I don't suppose it will change just for me! I'll get around to it eventually...

  11. JukkaL commented on Jan 29, 2020

    @JukkaL
    JukkaL
    on Jan 29, 2020
    Collaborator

    Closing this issue since the consensus seems to be that this should be discussed on typing-sig@.

  12. JukkaL
    closed this as completedon Jan 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureneeds discussionpriority-1-normal

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions