Implement MaskedQuantity

@mhvk

This was originally done as part of #1583 but that was closed, so just opening a general issue to keep track of this.

@mhvk @taldcroft - is this currently in any of the pull requests?

MaskedQuantity is not currently part of the generalized columns work.

@mhvk

@mhvk - did you want to have a go at this since you had already done some work on it here:

https://github.com/mhvk/astropy/blob/69ea9bb4480f0d4ae6daf989057c55e7624baf7b/astropy/units/quantity.py

?

@astrofrog

@astrofrog - thanks for setting this up separately. I've just been thinking about this yesterday, as I wondered whether the more automated approach used for Column might also work here, i.e., rather than explicitly defining masked versions of Quantity, Angle, Column, etc., might it be possible to make a Masked constructor. Indeed, if one looks at the MaskedArray, that seems to have been the idea: in principle, it allows one to insert any ndarray subclass, with possibly required attributes kept in _optinfo. Unfortunately, like quite a few other aspects, the implementation works only partially. But relatively small changes can help; e.g.:

In [1]: import astropy.units as u; import numpy as np
In [2]: q = u.Quantity(np.arange(10.), u.m)
In [3]: np.ma.MaskedArray(q).data
Out[3]: <Quantity [ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.] (Unit not initialised)>
In [4]: %paste
class myma(np.ma.MaskedArray):
    def _get_data(self):
        data = super(myma, self)._get_data()
        if hasattr(data, '__dict__'):
            data.__dict__.update(self._optinfo)
        return data
    _data = property(fget=_get_data)
    data = property(fget=_get_data)
## -- End pasted text --
In [5]: myma(q).data
Out[5]: <Quantity [ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.] m>

Otherwise, the main issue I found is that __str__ and __repr__ are set up assuming pure ndarray behaviour, and these will need to be overwritten as well (or one has to taken their approach into account in one's own __setitem__ and __str__).

Still, the above made me wonder whether for internal use we should simply make our own MaskedArray subclass in which we override some of the behaviour; this helps make sure we do not have to take the peculiarities into account in every subclass we construct (the comments in MaskedColumn show this has been a fun aspect...), and will also make it easier to send PRs to numpy (I saw @taldcroft had a few, and I've been making some myself as well).

Aside: from my last PR it is clear the numpy people would like to move MaskedArray to the new __array_ufunc__ functionality (which will be a big benefit for Quantity as well). This is a chance to help get the implementation right.

I guess we are going to have to punt on this - re-assigning to 1.0.0

Just for the record, I just found again that numpy.ma and Quantity, don't play nicely:

In [1]: from astropy import units as u

In [2]: np.arange(5)*u.km
Out[2]: <Quantity [ 0., 1., 2., 3., 4.] km>

In [3]: np.ma.arange(5)*u.km
Out[3]: 
masked_array(data = [<Quantity 0.0 km> <Quantity 1.0 km> <Quantity 2.0 km> <Quantity 3.0 km>
 <Quantity 4.0 km>],
             mask = False,
       fill_value = ?)

In [5]: (np.arange(5)*u.km).unit
Out[5]: Unit("km")

In [6]: (np.ma.arange(5)*u.km).unit
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-940865407535> in <module>()
----> 1 (np.ma.arange(5)*u.km).unit

AttributeError: 'MaskedArray' object has no attribute 'unit'

Given that we might not even have a solution in time for 1.0.0 I wonder if we should add a big warning in the documentation ("Quantities and masked arrays are not copatible. Use at your own risk.") and/or print a warning on screen every time an operation mixes units and masked arrays.

Yes, this is definitely not going to be ready for 1.0 and I'll move this to future: it needs a rather big overhaul of MaskedArray. I'd support a PR with a warning in the documentation (though I don't think we ever promised that Quantity would work with masked arrays).

For the record, the behaviour you are seeing is the subject of numpy/numpy#3907, but there has been little progress on that (or on my other PRs, numpy/numpy#4586 and numpy/numpy#4617). Partially on logical grounds, though: MaskedArray is rather a mess and should be overhauled more generally, likely using the __numpy_ufunc__ mechanism. But of course that does not help us much...

I agree, we never promised that it would work. But I usually operate
like this: Quantity works with numpy arrays. So I assume it also works
with numpy masked arrays.

@hamogu

I just tried to use quantities as a basis for masked arrays too, because I also normally operate as @hamogu says.

Working:

statistical summary functions appear to work out numerically (i.e., Out[43] below correctly excludes the masked value)
I can retrieve a correct .data attribute from the masked quantity
I can retrieve a correct .unit attribute from the result of a statistical summary function (Out[46], below)

Not working:

I cannot print/display a masked quantity when any values are masked. ("TypeError: ufunc 'isfinite' not supported for the input types...")
I cannot retrieve a .unit attribute directly from the masked quantity ("AttributeError: 'MaskedArray' object has no attribute 'unit'...")

A workaround for both the above issues is to print (or get the .unit attribute of) my_masked_quantity.data instead of my_masked_quantity.

Sketchy:

Format of a masked quantity is bizzare (Out[39])

Canopy tells me I'm using astropy 0.4.1. Looks like I just need to avoid printing masked quantities directly, other than that, they appear to work for my purposes....

x = [2,3,4,5] * u.deg_C

x
Out[37]: <Quantity [ 2., 3., 4., 5.] deg_C>

y = ma.array(x)

y
Out[39]: 
masked_<Quantity [ 2., 3., 4., 5.] deg_C>(data = [ 2.  3.  4.  5.] deg_C,
                                          mask = False,
                                    fill_value = 1e+20)

y[2] = ma.masked

y
...
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' 

x
Out[42]: <Quantity [ 2., 3., 4., 5.] deg_C>

y.sum()
Out[43]: <Quantity 10.0 deg_C>

y.mask
Out[44]: array([False, False,  True, False], dtype=bool)

y.data
Out[45]: <Quantity [ 2., 3., 4., 5.] deg_C>

y.sum().unit
Out[46]: Unit("deg_C")

y.unit
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-47-9a6034f374de> in <module>()
----> 1 y.unit

AttributeError: 'MaskedArray' object has no attribute 'unit'

@hamogu

@hamogu, @bnordgren - it would probably be good both to document this and to have an independent look at the numpy PRs I mentioned above, to see how much of the issues are solved by them.

I did make two attempts to get things to work better. Probably the most significant one is https://github.com/mhvk/astropy/compare/util-masked-array?expand=1, where I try to implement a MaskedArray class that would work with Quantity and its subclasses (note that it uses #2327, hence the PR looks more complicated than it really is; probably the most important is the test file, https://github.com/mhvk/astropy/blob/d1fc53a509444475b634f1c8001c5fd3e4125e40/astropy/utils/compat/numpy/tests/test_masked_quantity.py). This comes with quite a few tests and just having a masked array mostly works, but I got somewhat stuck at the proper mixin classes. I won't have time in the near future to work on this, so point you to this mostly in case you do have time and feel like working on it. I should perhaps add that I'm not sure this is the best way forward. Possibly, it is better to just write a compatible MaskedArray class from scratch...

I'm unqualified to speak to the best path forward for astropy internals.

However, I can document. Further playing today led me to the following synopsis (which covers all the above):

Creation order matters. Creating a quantity using a masked array yields a different result than creating a masked array using a quantity.
My workaround of using the .data attribute is only possible if a masked array is created using a Quantity object. The "dtype" of the result is preserved.
Using a masked array to create a quantity object is very broken. The result is a list of discrete quantity types, the .dtype of the result is set to "object". It seems impossible to get a "normal" quantity object back once this happens.

@bsipocz

This was requested again at an Astropy workshop @bsipocz and I did.

@mhvk

@mhvk - What is the status of this? Both example snippets from above in #1852 (comment) and #1852 (comment) seem to work (except y.units is still raising the AttributeError, but y.data.units works).

See #6005 - not feature-complete, though.

See also #6031 for a proof of concept using NaN.

👏 👏 👏

That's what I call closing a long-standing issue!

Navigation Menu

Repository navigation

Implement MaskedQuantity #1852

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implement MaskedQuantity #1852

Description

Activity

taldcroft commented on Dec 1, 2013

astrofrog commented on Dec 1, 2013

mhvk commented on Dec 1, 2013

astrofrog commented on May 3, 2014

hamogu commented on Nov 3, 2014

mhvk commented on Nov 3, 2014

hamogu commented on Nov 4, 2014

bnordgren commented on Nov 6, 2014

mhvk commented on Nov 6, 2014

bnordgren commented on Nov 19, 2014

astrofrog commented on Apr 28, 2016

bsipocz commented on Jun 27, 2017

mhvk commented on Jun 27, 2017

taldcroft commented on Nov 21, 2017

pllim commented on Apr 13, 2021

hamogu commented on Apr 13, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions