You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It is by design – the idea is that numpy's default int type matches the range of python 2's int, which in turn matches the platform C compiler's long.
Whether this is a good design is another question, especially since python 3 has eliminated this. There have been intermittent discussion about changing it before that you can probably dig up – especially the confusing and error prone way the default is 32 bits on win64.
I suppose one way to move that discussion forward would be to test whether any major packages break if you do make that change.
Changing the default int type on windows 64 to 64 bit would imo be an important enough change to warrant breaking software.
The current behavior just causes too many bugs.
That the default int type on 32 bit is 32 bit int is probably not so bad, as it does at least cover the full addressable range and changing it could have performance impact.
In my experience, if a Python library of moderate complexity that uses NumPy does not run Windows specific tests, it probably broken for this reason.
Reacted by Thomas Mansencal, Lawrence B., Jonathan Cooper, Sylvain Marié, Joris Van den Bossche, Bas van Beek, Mitar, Fabien Maussion, diabolic8733, Lucas Morin and 5 more
Perhaps we should drop this default at the same time as python 2, since the sole reason for defaulting to np.int_ was that it matched the size of builtins.int, which in python 3 is not even true.
Reacted by Thomas Mansencal, Stephan Hoyer and Lucas Morin
Ideally numpy should behave the same way across platforms. A colleague of mine uses Windows and recently had to spend some time trying to figure out why a program was yielding different results on his machine than on my Mac. IMO performance considerations pale in comparison to getting correct and consistent results.
Reacted by Thomas Mansencal, Jan Sellner, Quang Vu, peterbell10, Laurent Faucheux, Lucas Morin, Mesmer, Alberto Pasqualetto and Hugues Hoppe
Is there any runtime workaround a user could execute, before their other code, to force numpy-on-Windows default types to the same widths as elsewhere? (Perhaps, a data-driven, tamperable mapping of Python types to numpy types?)
As a fresh example of some of the resulting craziness, specifically asking for a array of a type compatible with type(2**32) results in an array that can't store 2**32:
2020-07-07T06:53:20.9528159Z def testTiny(self):
2020-07-07T06:53:20.9528423Z a = np.empty(1, dtype=type(2**32))
2020-07-07T06:53:20.9529046Z > a[0] = 2**32
2020-07-07T06:53:20.9529318Z E OverflowError: Python int too large to convert to C long
@gojomo I'm not sure that's a right approach anyway. On python 3 type(2**32) is guaranteed to be int, so that's just a more complicated way of saying dtype=int. If you're using a literal like that anyway you could of course use explicit dtype=np.int64.
To make it more dynamic, does dtype=np.array(2**32).dtype work? (Odds are there are even more idiomatic ways to do this.)
EDIT: np.empty_like(2**32, shape=...) is probably it, assuming that works.
No, I had a PR to add one, maybe I can open that again now that we decided to start the deprecation on some of the aliases: #16535
So either use dtype=np.intp which gives you 32bit on 32bit systems and 64bit on 64bit systems, or use dtype=np.int64 to begin with. That PR made dtype=np.intp the default, which is the simpler change, because intp is fairly common in NumPy already.
@adeak My snippet's not a literal example; my actual issue is that I've got a list of many ints, which eventually reach 2**32, but a numpy array typed based on the first int breaks on Windows when it reaches 2**32, but works everywhere else.
(I was hoping the snippet highlighted some of the on-the-face absurdity of the Python-to-numpy interaction: shouldn't a reported type for a specific number specifically-communicate a corresponding type wide enough to store it? But I suppose Python is an equal contributor to the problem, as 2**65 & 2**129 have the same problem of reporting as simple int. So it's more a brain-teaser than a guide to better behavior.)
I'd answer the "is numpy's choice a good design?" question in @njsmith's 2017 comment as: "Reasonable way back when, but not anymore, with Python3, & the primacy of 64bit systems, and Microsoft's own phasing-out of WIndows 10's support for 32-bit systems."
Traffic on this issue since looks like it has referenced many places this has caused problems for people, but not yet any extant examples of code that'd break with a changed default. (There's probably some, somewhere.)
If the plunge of changing the default in one swoop is too risky, a call that opts-in to some minimum-width default (or user-chosen default) for all subsequent mappings of Python's int might help. (And then at some later date with warning to Windows users, change the default, but give laggards an option to change it back for a while.)
Don't hesitate to get in touch with me. There are too many things for the core team to push, so someone helping championing such change makes it much likely to happen!
The problem was that while with MacOs and Linux, int64 is the default, for Windows it is int32, even if it is a 64 bit operating system. My solution was to basically replace all Numpy functions with their corresponding Python functions, like numpy.multiply.reduce with prod and numpy.ceil with ceil. I could also have specified dtype=np.int64 or dtype=np.int64 but that would be quite explicit and hopefully not necessary anymore in the future.
If this is API breaking or so, maybe it would be something for Numpy 2.0, wouldn't it?
@fmaussion writing a brief NEP would be good. At this point, we should probably consider including such change in a major release (but it might still be good to summarize things in a NEP!), since I hope that isn't too far off.
I also suspect that the sane choice is probably (unfortunately) to switch to intp as default (i.e. 64bit on 64bit windows and not attempting any change e.g. on 32bit linux). But NEP would be the place to summarize that.
I added a switch for "use NumPy 2 behavior" very recently, so once there is some general consensus to push for this, there is also a path to start implementing it as planned for the major release.
I have no bandwidth either, but if you need someone to write that NEP, I can give it a shot.
I think that this issue should be written in the numpy.array documentation page, because the phrase "[...] NumPy will try to use a default dtype that can represent the values (by applying promotion rules when necessary.)" is misleading.
This has been addressed in NumPy 2.0 by changing the default integer type to intp, which has the practical consequence of changing the default on 64-bit Windows from int32 to int64. See:
This solves the key issue of Windows behaving differently than other platforms; after the change all 64-bit platforms return int64 for np.array([1]).dtype, and all 32-bit platforms return int32.
This issue has been addressed to the extent possible, and I don't think we'll be making another change to it after this, so I'll close this issue. Thanks all!
Reacted by Andras Deak, Naoyuki Yamada, Josh Moore and Przemyslaw Wysocki
Activity
njsmith commentedon Jul 26, 2017
It is by design – the idea is that numpy's default int type matches the range of python 2's
int, which in turn matches the platform C compiler'slong.Whether this is a good design is another question, especially since python 3 has eliminated this. There have been intermittent discussion about changing it before that you can probably dig up – especially the confusing and error prone way the default is 32 bits on win64.
I suppose one way to move that discussion forward would be to test whether any major packages break if you do make that change.
pv commentedon Jul 26, 2017
One thing that may break is if someone is using dtype=int and assumes this is somehow related to C long type...
juliantaylor commentedon Sep 1, 2017
Changing the default int type on windows 64 to 64 bit would imo be an important enough change to warrant breaking software.
The current behavior just causes too many bugs.
That the default int type on 32 bit is 32 bit int is probably not so bad, as it does at least cover the full addressable range and changing it could have performance impact.
shoyer commentedon Mar 20, 2018
We should seriously consider changing this.
In my experience, if a Python library of moderate complexity that uses NumPy does not run Windows specific tests, it probably broken for this reason.
thomasmansencal commentedon Sep 15, 2018
@shoyer : We ran into this exact problem on Windows with @MichaelMauderer on colour-science/colour#431.
I was assuming incorrectly that
np.int_was platform independent.eric-wieser commentedon Sep 15, 2018
Perhaps we should drop this default at the same time as python 2, since the sole reason for defaulting to
np.int_was that it matched the size ofbuiltins.int, which in python 3 is not even true.lawrence858 commentedon Oct 10, 2018
Ideally numpy should behave the same way across platforms. A colleague of mine uses Windows and recently had to spend some time trying to figure out why a program was yielding different results on his machine than on my Mac. IMO performance considerations pale in comparison to getting correct and consistent results.
gojomo commentedon Jul 7, 2020
Is there any runtime workaround a user could execute, before their other code, to force numpy-on-Windows default types to the same widths as elsewhere? (Perhaps, a data-driven, tamperable mapping of Python types to numpy types?)
As a fresh example of some of the resulting craziness, specifically asking for a array of a type compatible with
type(2**32)results in an array that can't store2**32:adeak commentedon Jul 7, 2020
@gojomo I'm not sure that's a right approach anyway. On python 3
type(2**32)is guaranteed to beint, so that's just a more complicated way of sayingdtype=int. If you're using a literal like that anyway you could of course use explicitdtype=np.int64.To make it more dynamic, does
dtype=np.array(2**32).dtypework? (Odds are there are even more idiomatic ways to do this.)EDIT:
np.empty_like(2**32, shape=...)is probably it, assuming that works.seberg commentedon Jul 7, 2020
No, I had a PR to add one, maybe I can open that again now that we decided to start the deprecation on some of the aliases: #16535
So either use
dtype=np.intpwhich gives you 32bit on 32bit systems and 64bit on 64bit systems, or usedtype=np.int64to begin with. That PR madedtype=np.intpthe default, which is the simpler change, becauseintpis fairly common in NumPy already.adeak commentedon Jul 7, 2020
I was thinking that if NEP 31 ever happens, it would also make this kind of replacing defaults easily opt-in.
gojomo commentedon Jul 7, 2020
@adeak My snippet's not a literal example; my actual issue is that I've got a list of many ints, which eventually reach
2**32, but a numpy array typed based on the first int breaks on Windows when it reaches2**32, but works everywhere else.(I was hoping the snippet highlighted some of the on-the-face absurdity of the Python-to-numpy interaction: shouldn't a reported type for a specific number specifically-communicate a corresponding type wide enough to store it? But I suppose Python is an equal contributor to the problem, as
2**65&2**129have the same problem of reporting as simpleint. So it's more a brain-teaser than a guide to better behavior.)I'd answer the "is numpy's choice a good design?" question in @njsmith's 2017 comment as: "Reasonable way back when, but not anymore, with Python3, & the primacy of 64bit systems, and Microsoft's own phasing-out of WIndows 10's support for 32-bit systems."
Traffic on this issue since looks like it has referenced many places this has caused problems for people, but not yet any extant examples of code that'd break with a changed default. (There's probably some, somewhere.)
If the plunge of changing the default in one swoop is too risky, a call that opts-in to some minimum-width default (or user-chosen default) for all subsequent mappings of Python's
intmight help. (And then at some later date with warning to Windows users, change the default, but give laggards an option to change it back for a while.)11 remaining items
fmaussion commentedon Feb 15, 2023
Thanks @seberg this sounds reasonable - I'll see if I can find the bandwith to get things started but I'll certainly need help.
seberg commentedon Feb 15, 2023
Don't hesitate to get in touch with me. There are too many things for the core team to push, so someone helping championing such change makes it much likely to happen!
njh219 commentedon Feb 15, 2023
xor2k commentedon Feb 24, 2023
Hi everybody! I just experienced this problem in my project
npy-append-array, comparexor2k/npy-append-array#6
The problem was that while with MacOs and Linux,
int64is the default, for Windows it isint32, even if it is a 64 bit operating system. My solution was to basically replace all Numpy functions with their corresponding Python functions, likenumpy.multiply.reducewithprodandnumpy.ceilwithceil. I could also have specifieddtype=np.int64ordtype=np.int64but that would be quite explicit and hopefully not necessary anymore in the future.If this is API breaking or so, maybe it would be something for Numpy 2.0, wouldn't it?
xor2k commentedon Feb 24, 2023
I have no bandwidth either, but if you need someone to write that NEP, I can give it a shot.
joaoe commentedon Apr 17, 2023
hi.
This issue causes serious compatibility problems between Windows 64 and Linux/Mac. E.g., another one unionai-oss/pandera#726
albertopasqualetto commentedon Aug 18, 2023
I think that this issue should be written in the numpy.array documentation page, because the phrase "[...] NumPy will try to use a default dtype that can represent the values (by applying promotion rules when necessary.)" is misleading.
Fix number representation bug on Windows
Fix number representation bug on Windows
Fix number representation bug on Windows
fix: fix ray test broken on windows (#2188)
rgommers commentedon Jun 24, 2024
This has been addressed in NumPy 2.0 by changing the default integer type to
intp, which has the practical consequence of changing the default on 64-bit Windows fromint32toint64. See:This solves the key issue of Windows behaving differently than other platforms; after the change all 64-bit platforms return
int64fornp.array([1]).dtype, and all 32-bit platforms returnint32.This issue has been addressed to the extent possible, and I don't think we'll be making another change to it after this, so I'll close this issue. Thanks all!