Extend __numpy_ufunc__ to most numpy.core.multiarray functions?

The __numpy_ufunc__ mechanism likely is easily extended to all relevant numpy.core.multiarray functions. This would also enable several pure-Python Numpy functions (e.g. hstack, vstack, ...) to work transparently with 3rd party classes --- as long as array/asarray are not called, things should work out transparently.

It might also make sense to add a helper function to multiarraymodule so that the override mechanism can be invoked from Python side. This could be a sensible alternative e.g. in the case of linear algebra functions.

The list of functions is not impossibly big: (see here and here)

They aren't ufuncs, though... The nice thing about ufuncs is that they all
have similar (and introspectable) calling conventions, so it's quite
reasonable in many cases to have a single code path that implements
overrides for all of them.

Maybe this isn't a big deal, and we should rename it numpy_override or
something and tell people to do isinstance(callable, np.ufunc) if they want
a generic ufunc overload? But that doesn't really give a good way to handle
things like reductions.
Or we could keep numpy_ufunc for just ufuncs, and add numpy_misc
for everything else, and everyone would implement it as a giant if chain.
Or, most similar to the python standard, have a different numpy
override for each function.

I don't see how anyone can reasonably write a single code path that handles
np.hstack and np.mean, for example. Anyone overloading these functions is
going to have to go through the list one by one in any case.

The list of functions is not impossibly big: (see
herehttps://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/multiarraymodule.c#L3606and
herehttps://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/methods.c#L2284
)

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/4164#issuecomment-31470993
.

Yes, __numpy_override__ might be better.

Since the user can always implement a whatever dispatch mechanism required, I don't see great added value in a separate __numpy_misc__.

It's true that implementing all of the functions in a generic way is not trivial, but I think making it possible to do it at all is preferable to not allowing doing it (or silently casting to ndarrays dropping all metadata).

Note also that both mean and hstack are composite functions. The first one resolves to add.reduce and true_divide, and the second to concatenate.

One thing that might be desirable, however, is a "treat-this-object-as-an-ndarray" return value. However, this case can be handled via a generic fallback, such as:

def __numpy_ufunc__(self, ufunc, method, i, inputs, **kwargs):
    inputs = list(inputs)
    inputs[i] = self.toarray()
    ret = getattr(ufunc, method)(*inputs, **kwargs)
    # (*) whatever additional computation
    return ret

The quantities project, for example, could add to (*) the computation for the unit of the result:

COMMON_UNIT_UFUNCS = [np.partition, np.concatenate, np.add, np.subtract, np.lexsort]
...
    if ufunc in COMMON_UNIT_UFUNCS:
        unit = compute_common_unit(inputs)
    ...

I would like a way to be able to override non-ufunc functions, so +1 from me!

I'd love to have a way to override other functions, though I also see the point that one should keep the mechanism simple. Maybe it would make sense to start with functions that have consistent calling sequence? I.e., follow up np.dot with np.inner, np.outer, etc.

@pv

@pv - what can we do to try and get this idea moving forward? Is this something that requires a NEP?

@pv

A good start would be for someone to sit down and make a list of all the
functions, and make a concrete proposal about how to handle each of them.

A NEP would be a good format to write down such a proposal.

On Mon, Mar 31, 2014 at 4:04 PM, Thomas Robitaille <[email protected]

wrote:

@pv https://github.com/pv - what can we do to try and get this idea
moving forward? Is this something that requires a NEP?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4164#issuecomment-39098849
.

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

The thing that needs to be decided is the extended call signature to __numpy_ufunc__ (or __numpy_override__ if we a separate method is needed). Once that is done, I'd guess this would be quite straightforward to implement.

Now that __numpy_ufunc__ is within sight, shall we try to reboot this one? I'll miss the SciPy developers meeting, but I'll be sticking around through the sprints and would be happy to collaborate on a proposal/prototype.

I think I added this somewhere else, but for reference, I used __numpy_ufunc__ internally to help overwrite all the ndarray methods (some of which are ufuncs, but not all); it turned out much of the boiler-plate code was the same (see https://github.com/astropy/astropy/pull/2583/files#diff-e5e9598ff2150e7d4487df941f76f6d0R1171)

Here's the list of functions to wrap that I made during the SciPy sprints:

Things to wrap:

Misceleanous duck-array stuff:
- concatenate & stack
- insert
- broadcast_to (broadcast_arrays is implemented in terms of this)
- select & choose (see also 3-arg where below)
- reshape
- take
- einsum (dask.array can actually do this)
Aggregation functions (that don’t already call the method). These could be gufuncs if we had an axis argument, so perhaps __numpy_ufunc__ would be appropriate here?
- nan-skipping: nanmean, nansum…
- median
- percentile
- searchsorted
Other things that should be (g)ufuncs but aren’t -- these could also use __numpy_ufunc__?
- 3-argument where
- around & isclose -- these take optional arguments
- matmul (dot?)
- sort/argsort
- tensordot

Most of these could use something quite similar to __numpy_ufunc__, even if they aren't quite ufuncs. A few of things have signatures that are quite different (e.g., concatenate and stack), so they would need a different approach -- possibly special __numpy_concatenate__ and __numpy_stack__ methods.

Have we considered adding a simple multiple-dispatch mechanism to numpy in addition to this approach for ufuncs. The rest of the numpy functions could be multiple-dispatch enabled.

It would be very simple to implement a multiple-dispatch approach that just raises an error if there is no concrete implementation registered for the type.

R does this with S4 and Julia does this as well. Python also uses flavors of multiple-dispatch in various ways in copy_reg and pickle. The numpy ufunc itself uses multiple-dispatch under the covers.

I think it makes the most sense for NumPy to implement a simple multiple-dispatch concept which would be very easy to communicate to downstream developers.

@shoyer

Travis-- you should talk to @shoyer about this while you're in the same
room, he and I just spent a few hours talking about related topics :-)
On Sep 19, 2015 5:49 PM, "Travis E. Oliphant" [email protected]
wrote:

Have we considered adding a simple multiple-dispatch mechanism to numpy in
addition to this approach for ufuncs. The rest of the numpy functions could
be multiple-dispatch enabled.

It would be very simple to implement a multiple-dispatch approach that
just raises an error if there is no concrete implementation registered for
the type.

R does this with S4 and Julia does this as well. Python also uses flavors
of multiple-dispatch in various ways in copy_reg and pickle. The numpy
ufunc itself uses multiple-dispatch under the covers.

I think it makes the most sense for NumPy to implement a simple
multiple-dispatch concept which would be very easy to communicate to
downstream developers.

—
Reply to this email directly or view it on GitHub
#4164 (comment).

We had more discussions about this at the BIDS meeting this past weekend. Our notes from the meeting are on Hackpad: https://bids.hackpad.com/Duck-arrays-in-NumPy-6MgLmEnkNhT

Based on these discussions, I plan to make a concrete proposal for __array_concatenate__ (as a NEP) in the near future. Once we have that figured out, it should give us a template for moving forward.

@shoyer

@shoyer - am intrigued. One suggestion: I think more fundamental than concatenation is the underlying combination of two things: (i) the ability to generate an output array of possibly new shape with otherwise identical generalized dtype (type, units, whatever a duck array needs); and (ii) insertion. This because really concatenation is nothing more than:

out = generalized_empty_like(arrays[<suitable-pick-for-base>], new_shape)
for array, slice in arrays, <suitable-slices>:
    out[slice] = array

where <suitable-pick> should be the same one that would be picked for any other case where there is a list of various arrays.

The above seems to suggest one doesn't generally need a __array_concatenate__ method as much as a __empty_like__ method that overrides how the output array is created -- something that will be needed anyway to construct arrays for other operations such as arithmetic. (And of course one has to take care with __setitem__ -- in the case of Quantity this implies ensuring unit conversion on top of possible float/int, etc., conversion.)

p.s. This thread may be the wrong place to comment; if so, do direct me elsewhere.

@shoyer

I think there are some notes on empty_like in that hackpad as well. But
note that one issue with the setitem approach is that some duck arrays will
probably be immutable. (E.g. dask can implement concatenate trivially, but
doesn't currently support setitem.)
On Sep 22, 2015 12:53 PM, "Marten van Kerkwijk" [email protected]
wrote:

@shoyer https://github.com/shoyer - am intrigued. One suggestion: I
think more fundamental than concatenation is the underlying combination of
two things: (i) the ability to generate an output array of possibly new
shape with otherwise identical generalized dtype (type, units, whatever a
duck array needs); and (ii) insertion. This because really concatenation is
nothing more than:

out = generalized_empty_like(arrays[], new_shape)
for array, slice in arrays, :
out[slice] = array

where should be the same one that would be picked for any
other case where there is a list of various arrays.

The above seems to suggest one doesn't generally need a
array_concatenate method as much as a empty_like method that
overrides how the output array is created -- something that will be needed
anyway to construct arrays for other operations such as arithmetic. (And of
course one has to take care with setitem -- in the case of Quantity
this implies ensuring unit conversion on top of possible float/int, etc.,
conversion.)

p.s. This thread may be the wrong place to comment; if so, do direct me
elsewhere.

—
Reply to this email directly or view it on GitHub
#4164 (comment).

(E.g. dask can implement concatenate trivially, but doesn't currently support setitem.)

Maybe not __setitem__ but surely something similar, since otherwise how can one construct a new array from many pieces?

But I guess my main point would still be that concatenate requires creating a new array and insertion, both of which are fundamental to any array operation which results in a new array, so one might as well start there.

@shoyer

Maybe not setitem but surely something similar, since otherwise how
can one construct a new array from many pieces?

You can create a dask array however you like, but once it's created you
can't modify it further.
.
Another issue with the empty_like approach is that if you concatenate a
mixture of different types then it isn't clear what the output type should
be. (E.g. sparse_array.array_concatenate might want to choose whether
to return a sparse or dense array depending on how many non-zeros the
result had -- concat(1x10 sparse + 1x1000000 dense) -> dense, concat(1x10
dense + 1x1000000 sparse) -> sparse.)
.
Presumably @shoyer will include these kinds of examples in his NEP's
rationale section :-).
On Sep 22, 2015 1:16 PM, "Marten van Kerkwijk" [email protected]
wrote:

(E.g. dask can implement concatenate trivially, but doesn't currently
support setitem.)

Maybe not setitem but surely something similar, since otherwise how
can one construct a new array from many pieces?

But I guess my main point would still be that concatenate requires
creating a new array and insertion, both of which are fundamental to any
array operation which results in a new array, so one might as well start
there.

—
Reply to this email directly or view it on GitHub
#4164 (comment).

Fair enough -- I'll wait for the NEP!

Yes, I think supporting immutable arrays is an important use cases. Immutability is pretty common for objects intended to be used in a distributed context.

It would be nice to be able to support duck arrays more generically when possible by defining NumPy's functions on top of more generic operations. For example, stack and its variants can be written in terms of duck array operations on stack. Likewise, it's certainly a good idea to support a generic __array_empty_like__ which np.concatenate could be built out of. This is particularly useful for duck arrays that adhere very closely to the design of numpy's ndarray. However, that doesn't preclude checking for more specific overrides like __array_concatenate__.

Can we close this? We have now NEP 18 __array_function__ and NEP 18 duck arrays

Agreed, closing.

Navigation Menu

Repository navigation

Extend __numpy_ufunc__ to most numpy.core.multiarray functions? #4164

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Extend __numpy_ufunc__ to most numpy.core.multiarray functions? #4164

Description

Activity

pv commented on Jan 2, 2014

njsmith commented on Jan 2, 2014

pv commented on Jan 2, 2014

astrofrog commented on Jan 2, 2014

mhvk commented on Jan 6, 2014

astrofrog commented on Mar 31, 2014

njsmith commented on Mar 31, 2014

pv commented on Mar 31, 2014

shoyer commented on Jun 16, 2015

mhvk commented on Jun 16, 2015

shoyer commented on Aug 3, 2015

teoliphant commented on Sep 20, 2015

njsmith commented on Sep 20, 2015

shoyer commented on Sep 22, 2015

mhvk commented on Sep 22, 2015

njsmith commented on Sep 22, 2015

mhvk commented on Sep 22, 2015

njsmith commented on Sep 22, 2015

mhvk commented on Sep 22, 2015

shoyer commented on Sep 22, 2015

mattip commented on Oct 3, 2018

mhvk commented on Oct 3, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions