PyArray_DescrConverter inconsistency with empty arrays in records (Trac #1600)

Original ticket http://projects.scipy.org/numpy/ticket/1600 on 2010-09-02 by trac user steveb, assigned to unknown.

When a record type is defined using a dictionary, dtype doesn't allow zero length arrays.

np.dtype([('val1', np.float32, (0,))]) # OK
np.dtype({'names': ['val1'], 'formats':[(np.float32, (0,))]}) # Throws TypeError

It isn't clear to me why they shouldn't be allowed in this case.

Attachment added by trac user steveb on 2010-09-02: create_from_dict_elsize.patch

Not sure about the patch, it hasn't been applied.

Disallowing size-0 fields in structured arrays is intentional, the patch is the wrong solution. Size-0 fields cause segfaults, as seen in #473, #4955, #2585, #1901.

The problem here is not that the dict form of specification disallows size-0 fields, but rather that the list form of specification should not allow them but does!

Now, probably the ideal solution would be to get rid of the segfaults and allow 0-sized fields. However that seems tricky. Consider how this problem was solved for unstructured types: You can create size-0 types, but if you ever try to create an array with one it gets automatically modified to be size-1. We can't do that with structured array fields since it would change all subsequent field offsets.

>>> d = dtype('S')
>>> d, d.itemsize
(dtype('S'), 0)
>>> a = np.zeros(2, dtype=d)
>>> a.dtype, a.dtype.itemsize
(dtype('S1'), 1)

I guess a valid alternative solution would be to go around and fix all the
places that crash given size zero dtypes.
On Aug 1, 2015 7:51 AM, "ahaldane" [email protected] wrote:

Disallowing size-0 fields in structured arrays is intentional, the patch
is the wrong solution. Size-0 fields cause segfaults, as seen in #473
#473, #4955
#4955, #2585
#2585, #1901
#1901.

The problem here is not that the dict form of specification disallows
size-0 fields, but rather that the list form of specification should not
allow them but does!

Now, probably the ideal solution would be to get rid of the segfaults and
allow 0-sized fields. However that seems tricky. Consider how this problem
was solved for unstructured types: You can create size-0 types, but if you
ever try to create an array with one it gets automatically modified to be
size-1. We can't do that with structured array fields since it would change
all subsequent field offsets.

d = dtype('S')
d, d.itemsize
(dtype('S'), 0)
a = np.zeros(2, dtype=d)
a.dtype, a.dtype.itemsize
(dtype('S1'), 1)

—
Reply to this email directly or view it on GitHub
#2196 (comment).

@ahaldane

I was under the impression that zero-size dtypes are supported now. They used to just not work at all (and indeed lead to segfaults). But I do have code that relies on this, and at least as of Numpy 1.6 that code has been working (previously I had to rely on gnarly workarounds for zero-width fields).

Maybe the fact that it happens to be working for my case is a happy accident though.

Nonetheless the point stands that it's possible to create a record dtype with zero-width fields using the list-based constructor, but not with the dict-based constructor. That inconsistency is a bug no matter what.

As for @ahaldane's example above, I think it's more of a special case. In fact in other cases it is disallowed:

>>> np.zeros(2, dtype=np.dtype('|V0'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Empty data-type

But in the case of empty strings it happens to work:

>>> np.zeros(2, dtype=np.dtype('S'))
array([b'', b''], 
      dtype='|S1')

But that only works because as a special case for strings it converts an 'S0' to 'S1'. It's not using a zero-width dtype anymore:

>>> np.zeros(2, dtype=np.dtype('S')).itemsize
1

So I think that's beside the point.

Where I've used this is in cases (which I agree are completely absurd, but believe me when I say there's a use-case):

>>> np.dtype(('f8', (0,)))
dtype(('<f8', (0,)))

This is an odd-looking case, but it actually ends up being treated as equivalent to '|V0', just with dtype('float64') as its .base instead of dtype('V'). This works in most cases though, and I think cases where it doesn't work should be fixed, instead of saying it's illegal (which currently it isn't as far as construction is concerned...)

@embray

@embray: I don't see any reason why we can't make zero-width fields inside record arrays legal and supported, but it's quite possible that you're the only one who cares enough to put the effort into making them work :-).

Going through the above issues one by one now--will update this comment as I go.

memory error with strings in structured arrays #473 -- still an issue. A bug clearly not the fault of zero-width arrays though. Should be easy to fix with a check in the appropriate place. This is a problem for 'S0' but 'V0' won't allow this--assignment to it results in TypeError: Empty data-type Should probably do the same for empty string fields (since there is only one valid "empty string" anyhow there's no reason to assign to it--or if nothing else it could fail except for assignment of empty strings).
Structured array fields with str dtype have length 0, not 1 #4955 -- looks like probably a symptom of the same bug behind memory error with strings in structured arrays #473, but demonstrated in such a way that at least happily does not cause the interpreter to crash.
data leakage with S0 in record dtype (Trac #1992) #2585 -- this is actually a duplicate (or I guess the other way around) of Structured array fields with str dtype have length 0, not 1 #4955. It also notes that there's some ambiguity as to what np.dtype('S') produces, but I don't think there is. In all cases, including in structured arrays, it results in a zero-width string type. What is surprising is that np.zeros(), and probably similar functions, promote it to 'S1'. But there's no ambiguity in the construction of the dtype itself.
Bug in rec.fromarrays in handling length-0 strings (Trac #1303), now with segfault. #1901 -- didn't segfault for me on Numpy 1.9.1, but could just be luck. Looks to be almost certainly a symptom of the same bug causing the previous two.

@njsmith

I think above I was worried about what would be returned if you subscripted a 0-sized array (to get an element). But I think I was just confused - you'd get en empty string, or an empty buffer, or an empty array. So I agree with @njsmith.

Also back then I hadn't figured out how to use gdb to diagnose segfaults, so tracking down the ones I linked above was daunting. Seems very doable now!

but it's quite possible that you're the only one who cares enough to put the effort into making them work

Maybe so... though at least other people have reported these issues too (that's how I found this one--went looking to see if it was reported before doing so myself). I actually ran into this a long time ago but got too distracted to report it at the time.

I'd happily take a look at fixing this. Almost all of those issues are probably symptoms of the same one or two bugs.

And I'm an adept gdb user :), so since I can reproduce the crash in #473 it shouldn't be hard to track down. I'll have a look...

Actually the dtype constructor blows up on zero-width fields *except*…

BUG: Fix numerous bugs related to zero-width string arrays (numpy#473, …

Actually the dtype constructor blows up on zero-width fields *except*…

Navigation Menu

Repository navigation

PyArray_DescrConverter inconsistency with empty arrays in records (Trac #1600) #2196

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

PyArray_DescrConverter inconsistency with empty arrays in records (Trac #1600) #2196

Description

Activity

numpy-gitbot commented on Oct 19, 2012

charris commented on Feb 20, 2014

ahaldane commented on Aug 1, 2015

njsmith commented on Aug 1, 2015

embray commented on Oct 7, 2015

njsmith commented on Oct 7, 2015

embray commented on Oct 7, 2015

ahaldane commented on Oct 7, 2015

embray commented on Oct 7, 2015

embray commented on Oct 7, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions