Function returning a generic function

Mypy doesn't support a function that tries to return a generic callable. I wonder if mypy could make this code work properly:

from typing import TypeVar, Callable, Any

T = TypeVar('T', bound=Callable[..., Any])

def deco() -> Callable[[T], T]:
    def inner(x):
        return x
    return inner

@deco()
def g(x: str) -> int: ...

g('') # should be fine? currently "Argument 1 to "g" has incompatible type "str"; expected None"
g(1) # should be an error

This is related to my comments at #1317, but maybe worth keeping open as a separate issue.

As Jukka and I discussed at PyCon, this situation where a type variable appears only in the result type of a generic callable could be handled by "floating in" the type variable to the result type: instead of

def [T] () -> def (T) -> T

give deco the type

def () -> def [T] (T) -> T

I wonder how often this arises in practice?

The only case where I remember having seen this is in a decorator that is used like this:

@memoize(cache_size=200)
def do_stuff(): ...

The decorator, when called, returns a generic function T -> T that preserves the signature of the decorated function.

At Zulip, we're using such decorator-returning functions a lot. One of these decorator-returning functions, named cache_with_keys, is applied to dozens of functions, many of which are called hundreds of times (see zulip/zulip#1348). It'll be great to see this issue fixed since a lot of our code is not properly type checked because of this.

I have another example from our codebase:

@app.route('/')
@rate_limit(30, timedelta(seconds=60))
def index() -> Response:
    # ...

Both of the decorators returned by app.route() and rate_limit() at lest formally preserve the signature of the callable being decorated.

Sadly I don't have enough knowledge to help with this myself.

This seems like something we'll really want to fix, but it's not immediately clear yet what the fix looks like.

As a workaround, rewriting nested functions to a class seem to work:

from functools import wraps
from typing import TypeVar, Callable, Any, cast

TFun = TypeVar('TFun', bound=Callable[..., Any])

class log_calls:
    def __init__(self, prefix: str) -> None:
        self.prefix = prefix

    def __call__(self, f: TFun) -> TFun:
        prefix = self.prefix
        @wraps(f)
        def wrapper(*args, **kwargs) -> Any:
            print('{}: {!r} {!r}'.format(prefix, args, kwargs))
            return f(*args, **kwargs)
        return cast(TFun, wrapper)

@log_calls('Calling foo')
def foo(x: str) -> str:
    return x

reveal_type(foo)

Oooh, very clever! I can't say I personally understand how it works, and we
should still fix the original error, but this looks like a useful
workaround. Thanks!

On Sat, Oct 15, 2016 at 4:28 AM, Yegor Roganov [email protected]
wrote:

As a workaround, rewriting nested functions to a class seem to work:

from functools import wrapsfrom typing import TypeVar, Callable, Any, cast

TFun = TypeVar('TFun', bound=Callable[..., Any])
class log_calls:
def init(self, prefix: str) -> None:
self.prefix = prefix
def __call__(self, f: TFun) -> TFun:
    prefix = self.prefix
    @wraps(f)
    def wrapper(*args, **kwargs) -> Any:
        print('{}: {!r} {!r}'.format(prefix, args, kwargs))
        return f(*args, **kwargs)
    return cast(TFun, wrapper)
@log_calls('Calling foo')def foo(x: str) -> str:
return x

reveal_type(foo)

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1551 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACwrMntwUxGKXBJXiFRnuVKRAye7x5Rxks5q0LjDgaJpZM4IhSDU
.

--Guido van Rossum (python.org/~guido)

Exclude boto from tests due to broken Liskov Substitution Principle

Revert "Make all single-constraint TypeVars to use bounds" (#806)

This looks fun and important. It'll be a good warm-up to all the rest of the features we need to get decorators working right. claim

Make all single-constraint TypeVars use bound= insteads (plus hack) (#…

Make TypeQuery more general, handling nonboolean queries.

I just opened a pull request #3113 that fixes this. It's still WIP, but I should link this issue in.

Make TypeQuery more general, handling nonboolean queries. (#3084)

@sixolet

@sixolet, Is this fixed by your #3113?

This is the email I meant to reply to saying yes I think it's fixed. Sorry, I was getting off a plane on 4h sleep.

NP! We've had quite a busy week. I'll tell you the stories some time.

I still have this issue (and it is fixed by the class hack) on mypy 1.14.1.

For context, numpy "ufuncs" can be applied to anything that is compatible with a certain protocol, but the type hints don't work. The xarray types DataArray and Dataset support this protocol, and e.g. np.sqrt(some_dataset) returns a Dataset, but the type hints say the result is a numpy array.

I can fix this for an individual numpy ufunc, but my attempts to make a function returning a generic function with the correct type hints lead me here.

from collections.abc import Callable
from typing import cast, TypeVar

import numpy as np
import xarray as xr

XrData = TypeVar("XrData", xr.DataArray, xr.Dataset)
XrOpType = Callable[[XrData], XrData]

# this gives the correct type hints
def xr_sqrt(data: XrData) -> XrData:
    result = np.sqrt(data)
    if isinstance(data, xr.DataArray):
        return cast(xr.DataArray, result)
    if isinstance(data, xr.Dataset):
        return cast(xr.Dataset, result)

# this doesn't work
def xr_types(ufunc: np.ufunc) -> XrOpType:
    def xr_typed_func(data: XrData) -> XrData:
        result = ufunc(data)
        if isinstance(data, xr.DataArray):
            return cast(xr.DataArray, result)
        if isinstance(data, xr.Dataset):
            return cast(xr.Dataset, result)
    return xr_type_func  # reveal_type says this maps XrData to XrData

bad_xr_sqrt = xr_types(np.sqrt)  # reveal_type says this maps Any to Any

# class hack works
class XrTypesHack:
    def __init__(self, ufunc: np.ufunc) -> None:
        self.ufunc = ufunc

    def __call__(self, data: XrData) -> XrData:
        result = self.ufunc(data)
        if isinstance(data, xr.DataArray):
            return cast(xr.DataArray, result)
        if isinstance(data, xr.Dataset):
            return cast(xr.Dataset, result)

xr_sqrt_hack = XrTypesHack(np.sqrt)  # reveal_type(xr_sqrt_hack.__call__) says this maps XrData to XrData

Navigation Menu

Repository navigation

Function returning a generic function #1551

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Function returning a generic function #1551

Description

Activity

rwbarton commented on May 18, 2016

rwbarton commented on Jun 13, 2016

JukkaL commented on Jun 13, 2016

sharmaeklavya2 commented on Jul 22, 2016

jstasiak commented on Aug 10, 2016

gnprice commented on Aug 11, 2016

roganov commented on Oct 15, 2016

gvanrossum commented on Oct 15, 2016

sixolet commented on Mar 24, 2017

sixolet commented on Apr 3, 2017

gvanrossum commented on Apr 21, 2017

sixolet commented on Apr 22, 2017

gvanrossum commented on Apr 22, 2017

brendan-m-murphy commented on Feb 27, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions