Repository navigation
dtype keyword for np.random.randint and np.random.rand #6790
Description
As brought up at the mailinglist, this code ates up much memory:
spectrumArray = np.random.randint(0,255, (2**20,2**12)).astype(np.uint8)
It has also crossed my mind that np.random.randint and np.random.rand could use an extra 'dtype' keyword. It didn't look easy to implement though.
See this thread on the mailinglist: [Numpy-discussion] array of random numbers fails to construct
Activity
njsmith commentedon Dec 8, 2015
This came up on the mailing list as a feature request from someone who
wanted to generate many gigabytes of random uint8s, so couldn't afford the
memory overhead of generating uint64s and then calling astype.
On Dec 8, 2015 01:26, "Sebastian" [email protected] wrote:
gfyoung commentedon Dec 20, 2015
How is the
dtypeparameter processed into C data types (e.g.longorlong long) when creating anarrayormatrixobject? I have been knee-deep inrandintcode, and as a first thought, is there anything wrong with initializingrvandarraydepending on the requesteddtype?jaimefrio commentedon Dec 20, 2015
Take a look here. When numpy C code wants to cover all the possible integer types, it typically has versions for:
npy_bytenpy_shortnpy_intnpy_longnpy_longlongplus their unsigned counterparts.
The simplest solution would be to leave the generation of the random numbers alone, i.e. make no changes to
rk_interval, and simply cast the result to the requested type, one item at a time, rather than the whole array at once. This will not be optimal in some senses, but it will keep a lot of behavior consistent with the current one, which may not be a strict requirement, but is certainly not a bad thing.I'm guessing the Cython way of doing this would be to move the loop copying the random numbers into the output array here, into a separate function defined with fused types. If this doesn't make sense, ask and I'll try to explain...
gfyoung commentedon Dec 20, 2015
Thanks for the link to the
numpyC data types. That's going to be quite useful.Fused types, interesting...I've never used them before, but the DOC seems to be cautioning us against using them for the time being. Thus, it seems best to go the less Cythonic route for now. At least the functionality would be in place, and if necessary, another PR could be put up to refactor the code once fused types are more stable.
jaimefrio commentedon Dec 20, 2015
Scipy uses fused types in several places. We are different projects, but close cousins to say the least: unless someone else makes a compelling argument against their use in NumPy, I think you can assume that it is safe to use them.
Unless you have a better alternative, I think it's either an if-elif monster with 10 branches, copying the bytes one at a time, or fused types. The first is ugly, the second will be slower, but perhaps not noticeably so, depending on how slow
jaimefrio commentedon Dec 20, 2015
<hit send to soon, continuing>the actual random number generation is.
What do you have in mind.
gfyoung commentedon Dec 20, 2015
Ah! Okay, that's good to know. I was initially going by the DOC, which warned against using them. However, if
scipyhas been able to use them without harm, then they certainly merit more consideration as a possible implementation.gfyoung commentedon Dec 20, 2015
Also, it occurs to me that one might also want to expand
numpy.random.RandState.tomaxintto account fordtype. But I'm getting ahead of myself here. Let me see howrandintandrandgo first.gfyoung commentedon Jan 3, 2016
Just curious, does anyone know of a significant use case for specifying
dtypeforrand? Otherwise, I think this issue is good to close as well.charris commentedon Jan 3, 2016
Someone might want floats instead of doubles, that would about double the speed.
charris commentedon Jan 3, 2016
That's probably an easier project than randint. Might raise the topic on the list for discussion.
gfyoung commentedon Jan 4, 2016
Sent out email here. I'll try putting together a PR for this once I start getting feedback.
13 remaining items
charris commentedon Jan 4, 2017
Look at how it is done for integers. C++ is a no go.
NumesSanguis commentedon Feb 8, 2019
Is someone still working on this feature? Since I'm dealing with audio and deep learning, I wanted to add white noise to a signal. For DL, float64 is overkill, so it would be nice if we could generate a uniform distribution as float32 from the start with:
bashtage commentedon Apr 11, 2019
xref #13163 . This is fixed in the randomgen branch.
mattip commentedon May 29, 2019
Closing, the new API supports dtype. Please reopen if needed.