Repository navigation
Some arrays for which arr.size * arr.dtype.itemsize > max_ssize_t currently not recognized as invalid #7451
Description
The example is a little convoluted, but fails consistently:
import numpy as np
from numpy.lib.stride_tricks import as_strided
a = np.ones((1, 1))
len_ = np.iinfo(np.intp).max // 2 + 1
b = as_strided(a, shape=(1, len_), strides=(0, 0))
# next line segfaults
b.ravel()Not sure what may be going on, but it sure doesn't happen if the strided view is much smaller.
Activity
mhvk commentedon Mar 22, 2016
Odd, does not segfault for me (1.10.4, 1.11.0.dev0+2ec0db0; Debian amd64).
jaimefrio commentedon Mar 23, 2016
Of course you don't, because I left the offending line out: a call to
b.ravel()at the end... Have edited my original post.seberg commentedon Mar 23, 2016
There is another odd thing here (and I get a MemoryError if I avoid it). The array should not be created in the first place. I guess it might be a separate issue though. For the original array
arr.size * arr.dtype.itemsizeis larger thenssize_there (my old 1.8.2 fails onb =line).AFAIK, that is not considered a valid array.
mhvk commentedon Mar 23, 2016
Indeed, now it "works"...
seberg commentedon Mar 23, 2016
Git blame suggests the offending commit is 1475cfa, the
sd * sizelater on, will go off into integer wraparound and should be checked (possibly start off with size = sd, maybe with an sd > 0 check -- not sure that can happen or not, somewhat think not, but...). Anyway, this also segfaults for copy, etc. as well, so nothing to do with ravel, just with malloc'ing empty arrays when the array is larger then legally possible.[-]Segfault with np.ravel[/-][+]Some arrays for which `arr.size * arr.dtype.itemsize > max_ssize_t` currently not recognized as invalid[/+]jaimefrio commentedon Mar 23, 2016
I guess we have a winner then... The following fails with a memory error:
The difference is that
(np.iinfo(np.intp).max // 2 + 1) * 8is a power of two, so when it overflows anint64, it results in a calculated array size of zero bytes.np.iinfo(np.intp).max // 2 * 8is a power of two minus one, and thus has a binary representation full of ones, so even though it also overflows, it results in some ridiculously large amount of memory that cannot possibly be allocated.jaimefrio commentedon Mar 23, 2016
Ah, hadn't seen your message when I sent mine. Yes, that seems to be it.
BUG: fix array too big error for wide dtypes.