ENH/API: Change flexible types to indicate resizability with elsize == -1 - #8970

this should be just Py_BuildValue('(Oi)', type, 1)

Yeah, I did it this way for speed - perhaps premature optimization

type needs casting to PyObject then.
The error checking above, while it won#t happen in practice may throw errors with static reference checker

also for speed PyTuple_Pack is usually good

Good catch. Think I might go for Py_BuildValue anyway here, since creating zeros of string type doesn't sound like something that should be a bottleneck

I'd be surprised if what I have here is slower than PyTuple_Pack, since that at best does a loop with these lines in

PyTuple_Pack is (marginally) slower than directly setting a few entries but faster than Py_BuildValue if you already have objects.
Here it would be useful to save some lines of code, this doesn't seem to be a performance relevant path to me.

Alright, changed to use Py_BuildValue("Ni", (PyObject*)type, 1)

adding and example and dtypes is better, e.g. unsized dtypes (``S``) from sized-to-zero dtypes (``S0``)

it should also mention that the python-side returns None for unsized dtypes.

Is this too breaking a change? Any C code that constructs flexible dtypes manually is in for a bad time - but hopefully nothing ever should, since PyArray_DescrFromTypeObject(Py_String) etc already do the right things

yes it previously returned a number now it doesn't. Likely not a big deal in practice but it should be mentioned.

Sorry, I mean from the C api perspective. But yes, I agree it at the very least needs to be in the release notes

Yes it has the potential to break others code that assumes itemsize is always positive.
But at least it should be an easy to detect error as it will probably crash by trying to allocate 2GiB of memory.

I was thinking code that assumed setting elsize to 0 would allow it to resize would be surprised when it no longer does

does this work with overlapping input and output?

These don't overlap - tuple contains a copy of the value in type, not a pointer to the variable &type

really? I though setitem will just increase the reference count of type by 1 not actually make a copy of it.

SET_ITEM actually doesn't touch the reference count at all - it steals the reference.

But SetItem copies type but increases the reference count of *type. We would be free to set type = NULL after adding it to the tuple, and the tuple would still contain the right thing

yes but PyArray_DescrConverter would still be working inplace, its output type is the same object as its input type. It probably works fine but that should be checked.

No it isn't, because PyArray_DescrConverter does essentially *(&type) = CreateNewDescr. It never touches the original object - because normally that original object is NULL

if it does this it doesn't work because creating a new type into type overwrites the input before it is read.

oh it does work as its &type not type never mind then

No it doesn't, because the input is no longer the type variable - it's the copy of the pointer the type variable held, which is stored inside tuple. So for example

type = (PyObject *) 0x00001230
Tuple_SETITEM(tuple, 0, type);
Tuple_GETITEM(tuple, 0);  // returns (PyObject *) 0x00001230
type = NULL
Tuple_GETITEM(tuple, 0);  // still returns (PyObject *) 0x00001230 - we didn't store a pointer to type

your mul_with_overflow are wrong, its *result, a, b

Alarming that runtest.py -b on windows does not warn me about that

our travis warning check catches it

Indeed, but that's pretty late in the pipeline, and it runs a bunch of other tests first