Repository navigation
BUG: np.unique with return_inverse and axis specification yields a wrong shape #26738
Description
Describe the issue:
In NumPy v2.0.0, when I specify return_inverse=True and axis for a 2d array, np.unique will return a wrongly deep-nested array for the inverse.
Reproduce the code example:
import numpy as np
a = np.array([[3, 1], [1, 2], [3, 1]])
u, inv = np.unique(a, return_inverse=True)
a1 = u[inv]
assert np.all(a1 == a) # True
u, inv = np.unique(a, return_inverse=True, axis=0)
a2 = u[inv.flatten()]
a3 = u[inv]
assert np.all(a2 == a) # True
assert np.all(a3 == a) # FalseError message:
No response
Python and NumPy Versions:
- NumPy: 2.0.0
- Python: 3.9.13 (main, Aug 25 2022, 23:26:10)
[GCC 11.2.0]
Runtime Environment:
No response
Context for the issue:
No response
Activity
seberg commentedon Jun 18, 2024
This was intentional: https://numpy.org/devdocs/release/2.0.0-notes.html#np-unique-return-inverse-shape-for-multi-dimensional-inputs
It is now compatible with
np.take_along_axes.But we may have underestimated the impact of this choice, and whether it is even the right one, since previously it was compatible with
np.take.The reason for this compatibility is that unlike
argmin/argmaxthe unspecified axes are all part of the operation and not broadcast axes.To clarify, let me invent a different operation that behaves similar to
return_inverse:The
sumoperation there uses all but the specified axes.That means that both choices make sense, since using
takeis a correct choice if there are no broadcast axes in the operation andtake_along_axesis the right choice if there are only broadcast axes.One option is just to undo this for
uniquebut keep the take-along-axes behavior for the new ones. When the change was made, it sounded to me like the current behavior doesn't make sense. But it seems like it does because the other axes are not broadcast-axes forunique(and I guess never will be).Unless we could even reconsider it for the
unique_*?CC @jakevdp, @rgommers.
seberg commentedon Jun 18, 2024
Ah, one thing I missed and this does indeed improve, is that for
axis=Nonereturning this shape does simplify reconstructing the original array.Although, interestingly if paired with
np.take()not withnp.take_along_axes().seberg commentedon Jun 18, 2024
@nabenabe0928 I forgot to mention that the work-around is to add a
.reshape(-1)to the result, I think. And whatever happens, there is a small chance it could become a no-op, but it'll keep working on any version.nabenabe0928 commentedon Jun 19, 2024
@seberg
Thank you for the information, I completely had missed the context!
Tbh, I did not know
np.takeandnp.take_along_axiseither.This breaking change was a bit troublesome for me because I was using
np.unique(..., return_inverse=True)as a lexicographic sorting function and its inverse array.With this change, the inverse function looked a bit unnatural when specifying an axis although I agree that the behavior for
axis=Noneis now much more natural.Note
I know that there is
np.lexsort, but in my usecase, the shape of an array is(N, M)where bothNandMcan change arbitrarily and I wanted to make the array unique as well, sonp.uniquewas syntactically cleaner and more efficient.Anyways, given the context that the change is intended, I will simply use
reshapeorflattento avoid the problem.jakevdp commentedon Jun 21, 2024
For what it's worth, this was the kind of issue I had in mind when I commented here on the original PR: #25553 (comment), but the NumPy team together decided that the breaking change was warranted: #25553 (comment)
I'm not sure whether there are any public notes from that discussion.
seberg commentedon Jun 21, 2024
Right, the thing that I misunderstood, is that I had the impression that new behavior was chosen because the old behavior was very strange. But that seems wrong, I am not even sure why the new behavior is in any serious way better right now (besides for
axis=Nonemaybe and I am not sure how much it matters).Dunno, its public, but of there are unlikely to be detailed enough notes.
seberg commentedon Jul 13, 2024
Closing, this is reverted. Note that the shape did change in 2.0 for the first case you mentioned and will stay changed. The first example actually didn't work previously.
Upgrade Numpy dependency to >=2.0.1
Upgrade Numpy dependency to >=2.0.1