self._mask is undefined for MaskedColumn

See traceback and ipdb output at the bottom of this example.

http://nbviewer.ipython.org/github/AASJournals/Tutorials/blob/master/3D_Objects/HYG_Database/HYG.ipynb

I'm at my limit to debug what is actually going on.

But if you ask me why I'm using masked column at all then i'd have to ask you (astropy) why astropy is using masked column for reading non-sparse (non-null containing or flagged) columns in the first place.

@augustfly

@augustfly - could you provide the exact code you used to convert the table from CSV to FITS?

@astrofrog

It actually doesn't matter @astrofrog -- if you look in the repo you can see the original csv file. If I rerun the notebook with the csv instead of the fits file then one gets the exact same error (and the same import as MaskedColumns).

But if you want to know it was topcat!

Indeed, when I loaded the csv with Table.read I got masked columns. For comparison, I loaded up the same file with pandas and looking at the ra column I didn't see any missing data or anything.

I'm going to hazard a guess that some columns are missing data so the entire table is then a masked table. However, since the mix-in columns, this is no longer technically necessary (we could have a mix of Column and MaskedColumn).

In any case, there's an underlying bug here that needs to be addressed.

@augustfly

Yes, some columns are missing data which (currently) makes all columns be MaskedColumn. There is something funky going on deep in the bowels of masked_array, because a MaskedColumn (straight from Table) most definitely does have a _mask attribute:

t = Table.read('https://github.com/AASJournals/Tutorials/raw/master/3D_Objects/HYG_Database/hygdata_v3.csv', format='ascii.csv')
t['dec']._mask
Out[28]: array([False, False, False, ..., False, False, False], dtype=bool)

@augustfly - if you'd like to just keep on going without worrying about missing data, just do:

data = data.filled()

Then everything is a normal Column, and any missing values have been set to the fill value. If you weren't aware that some data elements are missing then you probably won't mind this.

Here is a minimal example that replicates the problem:

In [42]: from astropy.table import MaskedColumn

In [43]: m = MaskedColumn([1])

In [44]: np.concatenate([m]).ravel()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-44-a2ed010a5af0> in <module>()
----> 1 np.concatenate([m]).ravel()

/Users/aldcroft/anaconda/envs/astropy/lib/python2.7/site-packages/numpy/ma/core.pyc in ravel(self)
   4065         r = ndarray.ravel(self._data).view(type(self))
   4066         r._update_from(self)
-> 4067         if self._mask is not nomask:
   4068             r._mask = ndarray.ravel(self._mask).reshape(r.shape)
   4069         else:

AttributeError: 'MaskedColumn' object has no attribute '_mask'

[EDIT]: updated example

@taldcroft

Thanks @taldcroft for the .filled() that is perfect to work around.

Yes, instantiated objects have _mask but note in my notebook's ipdb results: MaskedColumn objects do not have self references to _mask until the object is created. I'm probably displaying my ignorance here, but:

In [3]: '_mask' in dir(MaskedColumn)
Out[3]: False

Also, there is some obvious buggy behavior related to normal masked arrays and concatentation:

In [20]: m = np.ma.arange(2)

In [21]: m.mask = [False, True]

In [22]: m
Out[22]: 
masked_array(data = [0 --],
             mask = [False  True],
       fill_value = 999999)

In [23]: np.concatenate([m, m])
Out[23]: 
masked_array(data = [0 1 0 1],
             mask = False,
       fill_value = 999999)

In [24]: np.c_[m.ravel(), m.ravel()]  # this is similar to where @augustfly's crash occurs
Out[24]: 
masked_array(data =
 [[0 0]
 [1 1]],
             mask =
 False,
       fill_value = 999999)

In both cases the code runs through but does not propagate the mask correctly. I guess that's why there is a np.ma.concatenate method.

This is making some sense now. np.concatenate is (effectively) used in the mayavi code (points = np.c_[x, y, z], and np.concatenate does not propagate the input subclass attributes. This makes sense because it doesn't have any concept of how to merge the subclass attributes for each of the input arrays.

The object coming out of np.c_ is made to be the right class with a view() but this is a bit of a sham.

Final upshot is that there is no bug in astropy, numpy, or mayavi. However, you can't use MaskedColumn (or MaskedArray) to hold the coordinate values. In theory mayavi could be patched to look for masked arrays and use np.ma.concatenate instead, but this doesn't seem all that useful.

@augustfly

@augustfly et al. - closing this. Feel free to reopen if you think I've missed something.

@taldcroft

Whew! I'm glad you understand that @taldcroft!

Thanks again for the filled() trick. I still agree with @astrofrog that why these columns are masked because other columns could be masked doesn't make sense and seems like a leftover of Table pre-v1.0.

Finally, I ❤️ masked arrays but only if I make them. astropy guessing and using arbitrary real number fill values just gets my 🐐! But you might have guessed that.

The real numbers for the fill values are actually set by Numpy btw (not that it makes it any better).

Navigation Menu

Repository navigation

self._mask is undefined for MaskedColumn #3662

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

self._mask is undefined for MaskedColumn #3662

Description

Activity

astrofrog commented on Apr 3, 2015

augustfly commented on Apr 3, 2015

embray commented on Apr 3, 2015

astrofrog commented on Apr 3, 2015

taldcroft commented on Apr 3, 2015

taldcroft commented on Apr 3, 2015

augustfly commented on Apr 3, 2015

taldcroft commented on Apr 3, 2015

taldcroft commented on Apr 3, 2015

taldcroft commented on Apr 3, 2015

augustfly commented on Apr 3, 2015

astrofrog commented on Apr 3, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions