Consider allowing MaskedColumn and Column in the same Table

Currently if any column in a table is a MaskedColumn, then all other columns are "upgraded" to be masked as well. This happens on initialization and if any column in an existing table is converted from Column to MaskedColumn. This is entirely a holdover from the original implementation of table using a numpy structured array.

In the current architecture there is really no reason to do this. I suspect the implementation of this would be relatively easy. The main reservation is about stability and breaking code.

The main API changes would be:

In order to automatically add new columns as MaskedColumn, one would need to supply masked=True when creating the table. Currently this happens if any of the columns are masked.
Adding a MaskedColumn to an unmasked table would not automatically "upgrade" other columns (this is a good thing to me).

The Table.masked attribute would still reflect if there are any MaskedColumns that have masked elements, so no change there.

In order to minimize breakage, what about making it that if a user tries to set the mask on a Column or access it, it automatically upgrades it, with a warning? This could potentially be a deprecation warning if we'd want to then remove that.

BTW, the auto-upgrade is already a bit broken from my perspective (see below). If you add a MaskedColumn with no elements masked, it gets downgraded to Column. Maybe these auto-transforms are just confusing (a remnant of the old structured array container) and it's better to bite the bullet and tidy this up.

The only required code fix for compatibility with the current behavior would be setting masked=True when creating the table, and that is probably already the case most of the time where users actually want masked columns.

In [2]: t = simple_table()

In [3]: t['d'] = MaskedColumn([1,2,3])

In [4]: t
Out[4]: 
<Table length=3>
  a      b     c     d  
int64 float64 str1 int64
----- ------- ---- -----
    1     1.0    c     1
    2     2.0    d     2
    3     3.0    e     3

In [5]: t['d'].mask[1] = True
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-6b9a29babb9a> in <module>()
----> 1 t['d'].mask[1] = True

AttributeError: 'Column' object has no attribute 'mask'

In [6]: t['d'] = MaskedColumn([1,2,3], mask=[True, False, True])
INFO: Upgrading Table to masked Table. Use Table.filled() to convert to unmasked table. [astropy.table.table]

Ok that sounds good - all I'd suggest is that in the case where a column would have got auto-upgraded but isn't, it might be worth having a warning saying that masked=True should be specified.

BTW this is a repeat of #3665.

@taldcroft

@taldcroft , then we should close one of them?

There can be only one!

Navigation Menu

Repository navigation

Consider allowing MaskedColumn and Column in the same Table #7496

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Consider allowing MaskedColumn and Column in the same Table #7496

Description

Activity

astrofrog commented on May 24, 2018

taldcroft commented on May 24, 2018

astrofrog commented on May 24, 2018

taldcroft commented on May 31, 2019

pllim commented on May 31, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions