Skip to content

_dtype_from_pep3118 is overly strict on prefixes #9049

Open

Description

@bmerry

In numpy 1.12.1, _dtype_from_pep3118 requires that modifiers are specified in the order shape then byte order then count, with at most one of each. The PEP is a bit vague, but it seems like modifiers ought to be interpreted recursively i.e. a modifier can be followed by any valid PEP 3118 specifier.

In practice this can lead to certain types failing to round-trip through a memoryview e.g.:

>>> dt = np.dtype({'formats': [np.dtype((np.dtype((np.int32, (3,))), (2,)))], 'names': ['foo']})
>>> a = np.empty(0, dt)
>>> m = memoryview(a)
>>> np.array(m)
NotImplementedError: memoryview: unsupported format T{(2)(3)i:foo:}

It also causes problems if one puts an endianness specifier at the start of the string (which should always be permitted, since the original struct module allows it). I haven't managed to get numpy to generate such a format string when wrapping an array into a memoryview, but it's causing me some problems while developing new features for pybind11:

>>> np.core._internal._dtype_from_pep3118('<(3)i')
ValueError: Unknown PEP 3118 data type specifier '(3)i'

Activity

  1. eric-wieser commented on May 4, 2017

    @eric-wieser
    Member

    Seems odd to me that we implement _buffer_format_string in C, but _dtype_from_pep3118, the inverse, in python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions