Bug in Table with unicode sandwich when dealing with masked columns

@taldcroft

When comparing masked columns in bytes and unicode, the result is different when comparing two unicode columns compared to comparing a unicode and a bytes column:

In [1]: from astropy.table import Table
   ...: 

In [2]: t1 = Table(masked=True)
   ...: t1['a'] = ['z', 'b', 'c']
   ...: t1['a'].mask = [1, 0, 0]
   ...: t2 = Table(masked=True)
   ...: t2['a'] = ['a', 'b', 'c']
   ...: 

In [3]: t2['a'] == t1['a']
Out[3]: 
masked_array(data = [-- True True],
             mask = [ True False False],
       fill_value = True)

In [4]: t1 = Table(masked=True)
   ...: t1['a'] = ['z', 'b', 'c']
   ...: t1['a'].mask = [1, 0, 0]
   ...: t2 = Table(masked=True)
   ...: t2['a'] = [b'a', b'b', b'c']
   ...: 

In [5]: t2['a'] == t1['a']
Out[5]: 
masked_array(data = [False True True],
             mask = [False False False],
       fill_value = True)

In the second case, the first mask item should be True. This is a separate issue from #6838 (but I had to do t2['a'] == t1['a'] instead of t1['a'] == t2['a'] to avoid running into that issue).

cc @taldcroft @mhvk

This bug is because np.char.encode does not respect masked array, and is fixable. Along the way I discovered another upstream bug in numpy MaskedArray that propagates to MaskedColumn. Note that this issue only seems to apply to string dtype, while numeric dtypes seem OK (output is masked correctly).

In [24]: m1 = np.ma.MaskedArray(['a', 'b'], mask=[1, 0])

In [25]: m2 = np.ma.MaskedArray(['a', 'b'])

In [26]: m1 == m2
Out[26]: 
masked_array(data = [-- True],
             mask = [ True False],
       fill_value = True)

In [27]: m1 < m2
Out[27]: array([False, False], dtype=bool)

In [28]: m1 > m2
Out[28]: array([False, False], dtype=bool)

In [29]: m1 >= m2
Out[29]: array([ True,  True], dtype=bool)

In [30]: m1 <= m2
Out[30]: array([ True,  True], dtype=bool)

Navigation Menu

Repository navigation

Bug in Table with unicode sandwich when dealing with masked columns #6839

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bug in Table with unicode sandwich when dealing with masked columns #6839

Description

Activity

taldcroft commented on Nov 26, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions