Repository navigation
Bug in Table with unicode sandwich related to order of comparison #6838
Description
Activity
- changed the title
[-]Bug in Table with unicode sandwich comparison[/-][+]Bug in Table with unicode sandwich related to order of comparison[/+]on Nov 12, 2017 This looks like an upstream bug in
MaskedArray. I put in a print statement and verified that forColumnthe comparison of U to S string raises a NotImplemented error so the compare order gets swapped and everything works.In [6]: Column(['a']) == Column([b'b']) a b b a Out[6]: array([False], dtype=bool)But for MaskedArray that
NotImplementedis not respected and instead the code is assuming success and trying to take the view of it, which fails.In [7]: MaskedColumn(['a']) == MaskedColumn([b'b']) a b --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-7-e17234b58113> in <module>() ----> 1 MaskedColumn(['a']) == MaskedColumn([b'b']) /Users/aldcroft/git/astropy/astropy/table/column.py in _compare(self, other) 939 if self.dtype.char == 'S': 940 other = self._encode_str(other) --> 941 return getattr(self.data, oper)(other) 942 return _compare 943 /Users/aldcroft/anaconda3/envs/astropy/lib/python3.5/site-packages/numpy/ma/core.py in __eq__(self, other) 3890 else: 3891 odata = filled(other, 0) -> 3892 check = self.filled(0).__eq__(odata).view(type(self)) 3893 if self._mask is nomask: 3894 check._mask = omask AttributeError: 'NotImplementedType' object has no attribute 'view' In [8]: debug > /Users/aldcroft/anaconda3/envs/astropy/lib/python3.5/site-packages/numpy/ma/core.py(3892)__eq__() 3890 else: 3891 odata = filled(other, 0) -> 3892 check = self.filled(0).__eq__(odata).view(type(self)) 3893 if self._mask is nomask: 3894 check._mask = omask ipdb> self.filled(0).__eq__(odata) NotImplementedSo we need another MaskedArray workaround.
@taldcroft, @astrofrog - I was looking at what to report upstream, but it turns out my own work on making masked array comparisons more reliable (numpy/numpy#8590) also solved this problem. As far as I can see the problem doesn't exist for numpy 1.13 (didn't test earlier versions...). What version of numpy are you on?
- added a commit that references this issue
on Nov 27, 2017 @mhvk - are you sure about that? See #6899 and below using current astropy master and numpy 1.13.1. It does not give an exception but doesn't give the right answer either.
In [6]: MaskedColumn(['a', 'b']) == MaskedColumn([b'b', b'b']) Out[6]: False In [7]: MaskedColumn([b'a', b'b']) == MaskedColumn(['b', 'b']) Out[7]: masked_array(data = [False True], mask = False, fill_value = True)Ah, so much for my trials - I was looking for an exception, not whether the answer was even right...
- added a commit that references this issue
on Nov 27, 2017 See numpy/numpy#10104 for the proposed upstream fix.
- added a commit that references this issue
on Nov 28, 2017 I should add here that I closed the upstream numpy fix, as I became convinced it was wrong. The underlying problem is that
MaskedArrayexpects thatdatais the unmasked object, but for aMaskedColumnwe get out aMaskedArrayobject - rather unfortunate that both classes usedatadifferently. I need to check whether inMaskedArraywe may be able to get away with using the private_dataattribute; otherwise, we'll need to override__eq__and__ne__.
The following example demonstrates an issue when comparing a string column with a bytes column:
The expected result is that
t1['a'] == t2['a']should be equal tot2['a'] == t1['a']cc @taldcroft @mhvk