Repository navigation
Python nditer should reinsert removed axes into its iteration results #9808
Description
When iterating using C, all that's returned is a pointer to the start of the array, and it's the users job to index it with their own memory of its strides and dimensions (like what happens in the gufunc inner loops).
However, the python nditer[i] API always returns 0d arrays, and does not produce a view over the axes that were removed.
My feeling is that this:
a = np.array([[1, 2], [3, 4]])
b = np.array([[10, 20], [30, 40]])
it = np.nditer((a, b), op_axes=[[0], [0]])
while not it.finished:
print(i.value)
it.iternext()should output
(array([1, 2]), array([10, 20]))
(array([3, 4]), array([30, 40]))whereas it outputs at the moment
(array(1), array(10))
(array(3), array(30))Can someone with a better understanding of NdIter confirm that my intuition is correct here?
Activity
seberg commentedon Oct 2, 2017
In some sense, I don't think that the python side API was ever really aimed for series usage outside of testing, understanding the C-side and some stuff in the direction of numexpr. In some sense I guess it just mirrors the C-API too much.
In this case, yes, one could try to change it, does not even change the only real thing of interest here in some sense (the pointer to the data), so it might even be safe for someone actually using it....
But overall, I think I would rather have a
ndarray.iteraxis(axis=None)or whatever new iterator then try to make nditer nice enough to actually advertise. But I suppose you want to do some advanced stuff inside numpy itself, so not sure what you need.mhvk commentedon Oct 2, 2017
Hmm, it may not be intended, but we're definitely using
nditerin astropy. The cases I can think of all iterate over all axes, so it is not an issue, but @eric-wieser's expectation seems reasonable.seberg commentedon Oct 2, 2017
I think it is not unintended to use it from python, or bad habit, but I personally believe it has too many quirks to call it a reasonable python API. So it is a very advanced feature and probably not used a lot.
eric-wieser commentedon Oct 2, 2017
This was my use case, and I found it very confusing that the removed axes just seemed to dissapear.
WillAyd commentedon May 20, 2020
AFAICT this behavior can be traced to here:
numpy/numpy/core/src/multiarray/nditer_pywrap.c
Line 2001 in 29f0b53
I've noticed if hard-coding
You get the expected result above. Obviously that's not a real solution...tried to leverage the
innerloopsizeptrandinnerstridesbut they appear to segfault on access from the!has_external_loopbranch so I assume there's some state management that those are coupled with.Will try to dig further and see if there is a solution here, just sharing above in the interim in case its of use to anyone else
seberg commentedon May 20, 2020
That looks right, if
op_axesis used, this could be modified. Maybe we should just do that with a big release notes warning? The current method is pretty useless after all, and if you usenp.stride_tricksor similar on the (assumed 0D) output array to get the old one back, then it would not actually break if we change this here...eric-wieser commentedon May 20, 2020
I think changing this is fine. I think I remember trying to fix this in the past, and not doing so because it was hard, not because it might break compatibility.
WillAyd commentedon May 21, 2020
Cool I'll give it a look over the next few days and see if I can push a PR
WillAyd commentedon May 29, 2020
Any pointers on how to introspect the size of the operands at this point during iteration? I think the stride information is accessible through the
**dtypesstruct member ofNewNpyArrayIterObject_tagseberg commentedon May 29, 2020
It is a bit tricky, but basically, I do not think the NpyIter will help you at all. I do not even see that it holds on to the
op_axesand you do not really need to know the iteration order.What I think you have to do is extract where the
-1elements are, and store their dimensions and strides and the number... Probably on the python side, unless you want to add this as a helper to NpyIter itself (which is maybe not implausible).eric-wieser commentedon May 29, 2020
This sounds like the issue that stumped me. I think my conclusion was that tracking that state was the caller's problem, but I don't remember too well.