Repository navigation
filtering masked tables resets column fill_value #4707
Description
filtering a masked table resets the fill value to 1:
In [1]: from astropy.table import Table
In [2]: a = Table(masked=True)
In [3]: a['X'] = [0,1,2,3]
In [4]: a['Y'] = [0,1,2,3]
In [5]: a['Y'].mask = [False, False, False, True]
In [6]: print a['Y'].fill_value
999999
In [7]: a
Out[7]:
<Table masked=True length=4>
X Y
int64 int64
----- -----
0 0
1 1
2 2
3 --
In [8]: a = a[a['Y']<3]
In [9]: print a['Y'].fill_value
1
This is dangerous because it reset the fill value to a valid value in the table, which breaks how masked values are tracked in FITS-format tables (I'll submit a separate issue about that). But filtering shouldn't reset the fill value in the first place.
In [10]: a
Out[10]:
<Table masked=True length=3>
X Y
int64 int64
----- -----
0 0
1 1
2 2
In [11]: a.write('blat.fits', overwrite=True)
In [12]: b = Table.read('blat.fits')
In [13]: b
Out[13]:
<Table masked=True length=3>
X Y
int64 int64
----- -----
0 0
1 --
2 2
The end result is that after filtering my masked table, round-tripping it to a FITS file gives me back a different table, with one of my previously unmasked values now masked.
Activity
pllim commentedon Mar 21, 2016
Maybe less confusing to just address this in #4708 since they are related?
sbailey commentedon Mar 21, 2016
Fine to treat this as a single ticket if both #4707 and #4708 are fixed simultaneously. But they are separate issues — resetting the fill value when filtering could be problematic even if no I/O is ever performed with the table. It may be possible to usefully fix this ticket by itself even if the FITS I/O issue #4708 requires more work.
pllim commentedon Mar 21, 2016
Ok. Re-opening. Thanks for the clarifications.
saimn commentedon Oct 3, 2017
Seems to be fixed, maybe it was a Numpy issue with subclasses ?
astropy-bot commentedon Mar 8, 2018
Hi humans 👋 - this issue was labeled as Close? approximately 5 months ago. So..... any news? 🗞️
If you think this issue should not be closed, a maintainer should remove the Close? label - otherwise, I'm just gonna have to close this issue in a month. Your time starts now! Tick tock 🕙
If you believe I commented on this issue incorrectly, please report this here
saimn commentedon Mar 8, 2018
Ok, closing as I still cannot reproduce.
sbailey commentedon Mar 8, 2018
I confirm that this problem still exists with astropy/2.0.2 + numpy 1.11.1 but it is fixed with astropy/2.0.2 + numpy/1.13.1 . OK with me to leave it closed.