Skip to content

memory error with strings in structured arrays #473

Closed
Closed
memory error with strings in structured arrays#473

Description

@jakevdp
jakevdp
opened on Oct 1, 2012

The following code writes to memory blocks beyond the limits of the array, resulting in memory errors in both released and dev version of numpy (I've tried 1.6.2 and 1.8.0.dev):

In [1]: import numpy as np

In [2]: dtype = [('I', int), ('S', str)]

In [3]: x = np.zeros(3, dtype=dtype)

In [4]: x['S'] = ['a', 'b', 'c']

The problem (I believe) is that passing str to dtype results in a zero-length string, and this is not checked on assignment.

Activity

  1. glimmercn
    added 7 commits that reference this issue on Apr 4, 2013

    BUG: solve issue numpy#2649.

    cfc78fa

    BUG: fix 2649.

    a86fe7c

  2. embray commented on Oct 8, 2015

    @embray
    embray
    on Oct 8, 2015
    Contributor

    By taking the example in this issue a little further we demonstrate the root of this issue. For some reason when you call np.zeros (and probably other functions) with a zero-width string type, it replaces the dtype with a one-width string (in the case of unicode this means 'U1', so 4 bytes):

    >>> x = np.zeros(3, dtype='S0')
>>> x
array([b'', b'', b''], 
      dtype='|S1')

    Unexpected, but whatever. If backward compatibility is a concern I wouldn't suggest changing this. However, when using a structured dtype that includes a zero width string no such conversion occurs:

    >>> dt = np.dtype([('I', 'int64'), ('S', bytes)])
>>> z = np.zeros(3, dtype=dt)
>>> z
array([(0, b''), (0, b''), (0, b'')], 
      dtype=[('I', '<i8'), ('S', 'S')])
>>> z.itemsize
8

    But when accessing the 'S' field, the array returned again promotes the dtype to 'S1':

    >>> z['S']
array([b'', b'', b'Q'], 
      dtype='|S1')

    This is definitely a bug and should be changed.

  3. charris commented on Oct 8, 2015

    @charris
    charris
    on Oct 8, 2015
    Member

    Looks like there will be a 1.10.1 real soon now on account of the reported problems with msvc 2008, so it would be good to get a fix in. I assume this impacts astropy?

  4. embray commented on Oct 8, 2015

    @embray
    embray
    on Oct 8, 2015
    Contributor

    @charris Fortunately this impacts astropy only tangentially. I've never run into it myself. It could in principle though. Honestly #2196 is more impactful in that it's pretty inconvenient. Though I can imagine a case where this bug would bite too in certain FITS files (albeit obscure).

    So I'll investigate how difficult this is to fix, and it would be great to get a fix in 1.10.1 if I can come up with a reasonable patch. But it's not that important either.

  5. ahaldane commented on Oct 8, 2015

    @ahaldane
    ahaldane
    on Oct 8, 2015
    Member

    Well, this bug is involved in quite a few segfault issues which would be nice to fix.

    @embray, that sounds like the problem, nice find. Do you happen to know where the type conversion happens? I think I looked for it once but didn't find it.

  6. embray commented on Oct 8, 2015

    @embray
    embray
    on Oct 8, 2015
    Contributor

    It's in PyArray_NewFromDescr_int. There are a few different solutions to this I'm thinking of. Will try one and see how it pans out...

  7. embray
    added a commit that references this issue on May 31, 2016

    BUG: Fix numerous bugs related to zero-width string arrays (numpy#473, …

    0bf907a

  8. ahaldane commented on Jun 2, 2016

    @ahaldane
    ahaldane
    on Jun 2, 2016
    Member

    Fixed by #6430

  9. ahaldane
    closed this as completedon Jun 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions