Allow mixing Column and MaskedColumn in Table and use this for Table.read() #3665
Description
#3662 is an example of getting sidetracked from doing science because a table column that wasn't actually missing data is still a MaskedColumn when other table columns have missing data. Masked arrays don't always play well with other tools.
Currently a Table uses the masked attribute to not only determine what class to use for creating new columns, but also whether to convert a Column to MaskedColumn or vice-versa.
This issue to explore changing this behavior so that columns are more autonomous. For instance, when creating a new column in an existing table it might make sense to only use MaskedColumn when necessary.
From a feasibility perspective this probably isn't too difficult. The real issue will be ensuring that there isn't massive code breakage. Right now if you do:
t = Table(masked=True)
t['a'] = [1, 2]
then you know that t['a'] has a mask attribute. So where do we draw the line?
Activity
embray commentedon Apr 3, 2015
Don't most software bugs/misbehaviors count as getting sidetracked from doing science? :)
astrofrog commentedon Apr 3, 2015
To me:
means that all columns will be masked (and has to, for backward-compatibility). So it's more that if we add a masked column to a non-masked table, it would then work rather than crash.
taldcroft commentedon Apr 3, 2015
@astrofrog - I agree with you on keeping back compatibility.
One idea is to allow
masked=Nonewhich never does any conversions beyond going toColumnorMaskedColumnwhen necessary. The default now ismasked=Nonebut we would change that tomasked=Falseto match the current behavior.taldcroft commentedon Jun 3, 2019
Superceded by #7496