Bug in Table with unicode sandwich related to order of comparison

@taldcroft

The following example demonstrates an issue when comparing a string column with a bytes column:

In [64]: t1 = Table(masked=True)

In [65]: t1['a'] = ['a', 'b', 'c']

In [66]: t2 = Table(masked=True)

In [67]: t2['a'] = [b'a', b'b', b'c']

In [68]: t1['a'] == t2['a']
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-68-372e9d1afaab> in <module>()
----> 1 t1['a'] == t2['a']

~/Dropbox/Code/Astropy/astropy/astropy/table/column.py in _compare(self, other)
    938             if self.dtype.char == 'S':
    939                 other = self._encode_str(other)
--> 940             return getattr(self.data, oper)(other)
    941         return _compare
    942 

~/miniconda3/envs/dev/lib/python3.6/site-packages/numpy/ma/core.py in __eq__(self, other)
   3909         else:
   3910             odata = filled(other, 0)
-> 3911             check = self.filled(0).__eq__(odata).view(type(self))
   3912             if self._mask is nomask:
   3913                 check._mask = omask

AttributeError: 'NotImplementedType' object has no attribute 'view'

In [69]: t2['a'] == t1['a']
Out[69]: 
masked_array(data = [True True True],
             mask = [False False False],
       fill_value = True)

The expected result is that t1['a'] == t2['a'] should be equal to t2['a'] == t1['a']

cc @taldcroft @mhvk

This looks like an upstream bug in MaskedArray. I put in a print statement and verified that for Column the comparison of U to S string raises a NotImplemented error so the compare order gets swapped and everything works.

In [6]: Column(['a']) == Column([b'b'])
a b
b a
Out[6]: array([False], dtype=bool)

But for MaskedArray that NotImplemented is not respected and instead the code is assuming success and trying to take the view of it, which fails.

In [7]: MaskedColumn(['a']) == MaskedColumn([b'b'])
a b
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-7-e17234b58113> in <module>()
----> 1 MaskedColumn(['a']) == MaskedColumn([b'b'])

/Users/aldcroft/git/astropy/astropy/table/column.py in _compare(self, other)
    939             if self.dtype.char == 'S':
    940                 other = self._encode_str(other)
--> 941             return getattr(self.data, oper)(other)
    942         return _compare
    943 

/Users/aldcroft/anaconda3/envs/astropy/lib/python3.5/site-packages/numpy/ma/core.py in __eq__(self, other)
   3890         else:
   3891             odata = filled(other, 0)
-> 3892             check = self.filled(0).__eq__(odata).view(type(self))
   3893             if self._mask is nomask:
   3894                 check._mask = omask

AttributeError: 'NotImplementedType' object has no attribute 'view'

In [8]: debug
> /Users/aldcroft/anaconda3/envs/astropy/lib/python3.5/site-packages/numpy/ma/core.py(3892)__eq__()
   3890         else:
   3891             odata = filled(other, 0)
-> 3892             check = self.filled(0).__eq__(odata).view(type(self))
   3893             if self._mask is nomask:
   3894                 check._mask = omask

ipdb> self.filled(0).__eq__(odata)
NotImplemented

So we need another MaskedArray workaround.

@taldcroft

@taldcroft, @astrofrog - I was looking at what to report upstream, but it turns out my own work on making masked array comparisons more reliable (numpy/numpy#8590) also solved this problem. As far as I can see the problem doesn't exist for numpy 1.13 (didn't test earlier versions...). What version of numpy are you on?

Merge pull request #6899 from taldcroft/table-fix-6838

@mhvk

@mhvk - are you sure about that? See #6899 and below using current astropy master and numpy 1.13.1. It does not give an exception but doesn't give the right answer either.

In [6]: MaskedColumn(['a', 'b']) == MaskedColumn([b'b', b'b'])
Out[6]: False

In [7]: MaskedColumn([b'a', b'b']) == MaskedColumn(['b', 'b'])
Out[7]: 
masked_array(data = [False  True],
             mask = False,
       fill_value = True)

Ah, so much for my trials - I was looking for an exception, not whether the answer was even right...

BUG: Allow MaskedArray.__eq__ and __ne__ to return NotImplemented.

See numpy/numpy#10104 for the proposed upstream fix.

Merge pull request #6899 from taldcroft/table-fix-6838

I should add here that I closed the upstream numpy fix, as I became convinced it was wrong. The underlying problem is that MaskedArray expects that data is the unmasked object, but for a MaskedColumn we get out a MaskedArray object - rather unfortunate that both classes use data differently. I need to check whether in MaskedArray we may be able to get away with using the private _data attribute; otherwise, we'll need to override __eq__ and __ne__.

Navigation Menu

Repository navigation

Bug in Table with unicode sandwich related to order of comparison #6838

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bug in Table with unicode sandwich related to order of comparison #6838

Description

Activity

taldcroft commented on Nov 26, 2017

mhvk commented on Nov 27, 2017

taldcroft commented on Nov 27, 2017

mhvk commented on Nov 27, 2017

mhvk commented on Nov 27, 2017

mhvk commented on Dec 11, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions