Repository navigation
.view no longer works on field selections of structured arrays #10409
Description
See this SO question:
>>> some_data = np.array([('foo', 3.5, 2.15), ('bar', 2.8, 5.3), ('baz', 1.2, 3.7)],
dtype=[('col1', '<U20'), ('A', '<f8'), ('B', '<f8')])
>>> ab = some_data[['A', 'B']]>>> np.__version__
'1.13.3'
>>> ab.dtype
dtype([('A', '<f8'), ('B', '<f8')])
>>> ab.view(('<f8', 2))
FutureWarning: Numpy has detected that you may be viewing or writing to an array returned by selecting multiple fields in a structured array.
This code may break in numpy 1.13 because this will return a view instead of a copy -- see release notes for details.
array([[ 3.5 , 2.15],
[ 2.8 , 5.3 ],
[ 1.2 , 3.7 ]])vs
>>> np.__version__
'1.15.0.dev...'
>>> ab.dtype
dtype({'names':['A','B'], 'formats':['<f8','<f8'], 'offsets':[80,88], 'itemsize':96})
>>> ab.view(('<f8', 2))
ValueError: Changing the dtype to a subarray type is only supported if the total itemsize is unchangedSince we now return a view rather than a copy from arr[fields], we end up with a dtype with padding. This prevents .view being called, even though the memory layout is sufficient to allow it
Activity
ahaldane commentedon Jan 17, 2018
Yeah, I explicitly mentioned this in the 1.13 and 1.14 release notes. Actually, since we have not provided a
pack_fieldsmethod it is difficult for downstream users to update their code to get the old behavior: I originally thought the user could just replacearr[fields']byarr[fields].copy(), but that does not remove padding bytes either.8 remaining items
ahaldane commentedon Jun 11, 2018
Right, but it still bothers me that
arr[['a', 'c']].view((np.int16, 2))would still fail, which would be unexpected so someone who found that[['a', 'b']]worked. It might be better to force the user to writerepack_fields(arr[['a', 'c']]).view((np.int16, 2))in all 3 cases.Also, I'm pretty sure that even with this
trim_fieldsbehavior the view often isn't allowed because the resulting array is not C-contiguous. Compare to:charris commentedon Jun 11, 2018
@ahaldane Is this now fixed for 1.15?
ahaldane commentedon Jun 11, 2018
Yes, I just checked master. Master now gives the 1.13 behavior, just with an updated futurewarning.
I'm not sure we should close, since this issue will pop up again in 1.16.
charris commentedon Jun 11, 2018
Good, I'll just update the milestone :).
mattip commentedon Jul 6, 2018
I ran across code in
mapping.cthat is marked for removal in 1.16, but could not find a pull request that is waiting for a merge. It seems there is work to be done here?ahaldane commentedon Jul 7, 2018
Yes, that's on my TODO list. See discussion in #10411, essentially we need to fix #8100 first.
I should have some time to work on it starting in two weeks, but feel free to take a shot at it before. There is already some work on it in #10931 but as I comment there a more extensive fix is needed.
mattip commentedon Aug 14, 2018
We have returned to the 1.13 behavior now. PR #10411 was merged, issue #8100 is closed. Can we close this or do we want to explore returning views again?
ahaldane commentedon Aug 17, 2018
Hmm in my opinion #8100 was mistakenly closed, since I accidentally used the word "fix" in a sentence in #10411's description.
I am also not sure we've made a decision about structured views. As far as I am aware (I was out for a bit so may have missed it) I think we are still considering turning it into a view. #10411 just pushed off the decision/implementation.