NumPy ignores __array_priority__ when the right-hand-side operand is a subclass of ndarray #4766

Activity

1. pv commented on Jun 2, 2014

   

   pv

   on Jun 2, 2014

   Member

   More actions

   Does that change break gh-3502, gh-3503, or gh-3375?
2. pv commented on Jun 2, 2014

   

   pv

   on Jun 2, 2014

   Member

   More actions

   __array_priority__ initially existed only for the umath machinery to decide what __array_wrap__ and __array_prepare__ routines it should call, and which array subtype should be returned.

   Unfortunately, there is the second problem (as explained in gh-4709) that Python's operator override system has difficulties with the case superclass + two subclasses --- ie. what should the superclass __op__ do to allow sub1 op sub2 and sub2 op sub1 work as expected?

   In principle, this problem has nothing to do with __array_priority__ at all --- it's a generic Python programming best-practices question (but one where the best practice seems very unclear). Some of the code paths were later on adapted to use __array_priority__, but as noted in gh-4709 problems still crop up. It seems to me that just the change from Check to CheckExact fails to address some of the issues encountered in gh-4709...
3. njsmith commented on Jun 2, 2014

   

   njsmith

   on Jun 2, 2014

   Member

   More actions

   Does numpy_ufunc solve your original problem that prompted the bug
   report?
   On 2 Jun 2014 11:03, "Mads R. B. Kristensen" [email protected]
   wrote:

   The following code does not output "sub_class radd" as expected since
   NumPy ignores array_priority when the right-hand-side operand is a
   subclass of ndarray.

   import numpy as npclass sub_class(np.ndarray):
   array_priority = 1000
   def add(self, rhs):
   print "sub_class add"
   def radd(self, lhs):
   print "sub_class radd"sub = sub_class((2,2))np.float32(42.0) + sub

   The problem is line 282 in number.c
   https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/number.c#L282
   where a branch uses PyArray_Check() to check the right-hand-side operand
   m2. If PyArray_Check(m2) returns True, which is the case when m2 is a
   subclass of ndarray, array_priority is ignored.

   Now, this is not a problem when the left-hand-side operand is an ndarray
   since in CPython a subclass always takes precedence over its superclass.
   However, a NumPy scalar is not a sub-/super-class of ndarray as the example
   code above illustrates.

   There is a very simple fix: change PyArray_Check(m2) to
   PyArray_CheckExact(m2). However, this fix has the side-effect that the
   operation will fail with NotImplementedError if the subclass doesn’t
   implement the operator, which is a change to the current behavior.

   —
   Reply to this email directly or view it on GitHub
   #4766.
4. pv commented on Jun 2, 2014

   

   pv

   on Jun 2, 2014

   Member

   More actions

   @njsmith: it will probably solve it, but the execution will end up calling it, due to the aggressive implementation of the base class binop
5. madsbk commented on Jun 3, 2014

   

   madsbk

   on Jun 3, 2014

   ContributorAuthor

   More actions

   @pv: Actually, I believe that changing Check to CheckExact will fix the issue in gh-4709 where overloading __eq__ doesn’t capture np.int64(10) == q when combined with your commit da40eba. However, I do not know if it address the issues with masked arrays and matrices but I would be very surprised if it makes it worse.

   Anyhow, using __numpy_ufunc__ could solve my issue if both ndarray and all scalar types call __numpy_ufunc__.
   Note that in order for __numpy_ufunc__ to be called by all scalar types, the two _convert_to_ctype() functions in scalarmathmodule.c.src should return -2 (line 559 and 611). The user can trigger this by assigning a __array_priority__ greater than NPY_PRIORITY or by making sure that PyArray_ScalarFromObject() fails. In my opinion, a more elegant solution is simply to use the standard ufunc implementation instead of having a specialized scalar-scalar implementation.

   Thanks
6. seberg

   added

   57 - Close?Issues which may be closable unless discussion continuedIssues which may be closable unless discussion continued

   and removed

   57 - Close?Issues which may be closable unless discussion continuedIssues which may be closable unless discussion continued

   on Jan 5, 2019
7. madsbk

   closed this as completedon Aug 2, 2019