BUG: Masked scalar comparison returns float #4332

Activity

1. abalkin commented on Feb 20, 2014

   

   abalkin

   on Feb 20, 2014

   ContributorAuthor

   More actions

   Apparently the problem is with this logic

               if result.shape == () and m:
                   return masked

   in MaskedArray.__array_wrap__. Any scalar result from ufunc gets replaced with the masked singleton which has

   >>> numpy.ma.masked.dtype
   dtype('float64')

   So the issue is not limited to comparison operators:

   >>> -numpy.ma.array(5, mask=True) is numpy.ma.masked
   True
2. abalkin

   added a commit that references this issue on Feb 20, 2014

   BUG: Non-bool result when comparing ma scalars.

   ...

   This issue will close once commit 6d569ae is merged into the 'master' branch.

   6d569ae
3. pierregm commented on Feb 22, 2014

   

   pierregm

   on Feb 22, 2014

   Contributor

   More actions

   The whole idea is indeed to return a specific value, the masked singleton, however it is defined. That way, one can test whether an item of a ndarray is masked by simply checking item is masked.

   The reason why it worked with oldnumeric is that the masked singleton was defined as np.ma.MaskedArray(False, dtype=bool). I'd be more willing to have this changed back than having __array_wrap__ no longer returning the np.ma.masked value. I'm afraid the current PR is such a redical change it will break things down the road.
4. abalkin commented on Feb 22, 2014

   

   abalkin

   on Feb 22, 2014

   ContributorAuthor

   More actions

   The reason why it worked with oldnumeric is that the masked singleton
   was defined as np.ma.MaskedArray(False, dtype=bool).

   No,

    >>> numpy.oldnumeric.ma.masked.dtype
    dtype('int32')
   

   but

    >>> (numpy.oldnumeric.ma.masked > numpy.oldnumeric.ma.masked).dtype
    dtype('bool')
   
5. abalkin commented on Feb 22, 2014

   

   abalkin

   on Feb 22, 2014

   ContributorAuthor

   More actions

   Would it be less radical if we preserve the existing behavior when result dtype is float, but return the correct type otherwise?
6. pierregm commented on Feb 23, 2014

   

   pierregm

   on Feb 23, 2014

   Contributor

   More actions

   @abalkin Can you give an example of what you have in mind?
7. abalkin commented on Feb 23, 2014

   

   abalkin

   on Feb 23, 2014

   ContributorAuthor

   More actions

   @pierregm

   --- a/numpy/ma/core.py
   +++ b/numpy/ma/core.py
   @@ -2841,7 +2841,7 @@ class MaskedArray(ndarray):
                        # Don't modify inplace, we risk back-propagation
                        m = (m | d)
                # Make sure the mask has the proper size
   -            if result.shape == () and m:
   +            if result.shape == () and m and m.dtype == masked.dtype:
                    return masked
                else:
                    result._mask = m

   I am not sure backward compatibility is such a big concern here. I understand that some people like a[i] is ma.masked idiom, but I am yet to see anyone writing a.sum() is not ma.masked.

   In the long run, I would really like to see masked singleton use deprecated. As numpy type system getting reacher, it will cause more and more problems.
8. charris

   added

   Defect

   on Feb 24, 2014
9. charris commented on Feb 24, 2014

   

   charris

   on Feb 24, 2014

   Member

   More actions

   Labelling a bug until the conversation concludes.
10. pierregm commented on Feb 25, 2014

    

    pierregm

    on Feb 25, 2014

    Contributor

    More actions

    Returning the masked singleton was a design choice since numarray. I understand it can be frustrating, that a better solution needs to be implemented (once a consensus will be reached on missing/masked data...), that testing x is np.ma.masked is a bit weird (on another hand, testing x is None is quite common)... Nevertheless, I would first try to set np.ma.masked=MaskedArray(False,mask=True, dtype=bool), see whether it breaks anything in numpy and matplotlib before using your solution (because there's a good chance that m.dtype will never be masked.dtype in a generic case).
    Note #1: OK, I see the comments in L5673/L5764 about precedence, so there must have been a good reason at the time, but I can't remember which one..
    Note #2: another possibility with masked scalars is to have a comparison return False all the time (like np.nan does). But that's a radical change too...
11. abalkin commented on Feb 25, 2014

    

    abalkin

    on Feb 25, 2014

    ContributorAuthor

    More actions

    I would first try to set np.ma.masked=MaskedArray(False,mask=True, dtype=bool), see whether it breaks anything

    Do you recall the reason for changing ma.masked dtype from int32 to float between oldnumeric.ma and numpy.ma?
12. mattip

    removed

    priority: normal

    on Oct 21, 2018