Skip to content

ctypeslib test_struct_array_pointer test failing with Python 3.12.0rc1 #24399

Description

@rgommers

This is showing up in all the Python 3.12 wheel build jobs now, see gh-24398 (that tests with an empty commit on top of main). I think it came in with the update to cibuildwheel 2.15.0 in gh-24372 two days ago; the wheel build jobs were not actually run on that PR.

   ____________________ TestAsArray.test_struct_array_pointer _____________________
  
  self = <numpy.tests.test_ctypeslib.TestAsArray object at 0x117f99a90>
  
      def test_struct_array_pointer(self):
          from ctypes import c_int16, Structure, pointer
      
          class Struct(Structure):
              _fields_ = [('a', c_int16)]
      
          Struct3 = 3 * Struct
      
          c_array = (2 * Struct3)(
              Struct3(Struct(a=1), Struct(a=2), Struct(a=3)),
              Struct3(Struct(a=4), Struct(a=5), Struct(a=6))
          )
      
          expected = np.array([
              [(1,), (2,), (3,)],
              [(4,), (5,), (6,)],
          ], dtype=[('a', np.int16)])
      
          def check(x):
              assert_equal(x.dtype, expected.dtype)
              assert_equal(x, expected)
      
          # all of these should be equivalent
          check(as_array(c_array))
          check(as_array(pointer(c_array), shape=()))
  >       check(as_array(pointer(c_array[0]), shape=(2,)))
  
  Struct     = <class 'numpy.tests.test_ctypeslib.TestAsArray.test_struct_array_pointer.<locals>.Struct'>
  Struct3    = <class 'numpy.tests.test_ctypeslib.Struct_Array_3'>
  Structure  = <class '_ctypes.Structure'>
  c_array    = <Struct_Array_3_Array_2 object at 0x121ddd9d0>
  c_int16    = <class 'ctypes.c_short'>
  check      = <function TestAsArray.test_struct_array_pointer.<locals>.check at 0x10239f100>
  expected   = array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')])
  pointer    = <built-in function pointer>
  self       = <numpy.tests.test_ctypeslib.TestAsArray object at 0x117f99a90>
  
  ../venv-test/lib/python3.12/site-packages/numpy/tests/test_ctypeslib.py:241: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  ../venv-test/lib/python3.12/site-packages/numpy/tests/test_ctypeslib.py:236: in check
      assert_equal(x, expected)
          expected   = array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')])
          x          = array([(1,), (2,), (3,)], dtype=[('a', '<i2')])
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  
  args = (<built-in function eq>, array([(1,), (2,), (3,)], dtype=[('a', '<i2')]), array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')]))
  kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}
  
      @wraps(func)
      def inner(*args, **kwds):
          with self._recreate_cm():
  >           return func(*args, **kwds)
  E           AssertionError: 
  E           Arrays are not equal
  E           
  E           (shapes (3,), (2, 3) mismatch)
  E            x: array([(1,), (2,), (3,)], dtype=[('a', '<i2')])
  E            y: array([[(1,), (2,), (3,)],
  E                  [(4,), (5,), (6,)]], dtype=[('a', '<i2')])
  
  args       = (<built-in function eq>, array([(1,), (2,), (3,)], dtype=[('a', '<i2')]), array([[(1,), (2,), (3,)],
         [(4,), (5,), (6,)]], dtype=[('a', '<i2')]))
  func       = <function assert_array_compare at 0x10dba2de0>
  kwds       = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}
  self       = <contextlib._GeneratorContextManager object at 0x10d9b9af0>

This will block uploading of 3.12 wheels, so it needs either investigating or xfail-ing to deal with later. Cc @charris for visibility.

Activity

  1. added this to the 1.26.0 release milestone on Aug 11, 2023
  2. charris commented on Aug 11, 2023

    @charris
    Member

    Curious error.

    There has been a fair amount of churn in ctypes in the past couple of weeks of Python 3.12.0 development, we may want to open an issue.

  3. seberg commented on Aug 12, 2023

    @seberg
    Member

    On first sight, the test does look right so I think we should indeed upstream as it seems like a likely Python bug. I don't have python 3.12 ready right now, but one remaining check might to reproduce it via memoryview() (digging into our ctypeslib.as_array()):

    p_arr_type = ctypes.POINTER(_ctype_ndarray(obj._type_, shape))
    obj = ctypes.cast(obj, p_arr_type).contents
    asarray(obj)
    

    compared to using memoryview(obj) at the end there (and then maybe trying to figure out where the shape info got dropped)? @eric-wieser might actually be interested in it, since I think some of this may be based on his Python PRs years ago.

  4. mattip commented on Aug 13, 2023

    @mattip
    Member

    This will block uploading of 3.12 wheels, so it needs either investigating or xfail-ing to deal with later.

    +1 for xfailing for now, there is time before the actual 3.12 release and it would be good to get going with rc1 wheels.

  5. mattip commented on Aug 14, 2023

    @mattip
    Member

    The tests is checking unsafe behaviour in ctypes.cast: given a 2d ctypes array of c_array = some_class([[1, 2, 3], [4, 5, 6]]), what happens when we cast pointer(c_array[0]) back to type(c_array)?

    The test is casting a data pointer to the same p_arr_type (the type of c_array), which will always be a 2x3 array. _ctype_ndarray() is producsing the correct 2x3 ctypes type. The cast also works well, even when the rhs obj is c_array[0] the cast is a 3x2 object. But when taking the contents, something happens and only the original c_array[0] content is returned.

  6. mattip commented on Aug 14, 2023

    @mattip
    Member

    Here is a smaller reproducer:

    from ctypes import Structure, pointer, c_int16, cast, POINTER
    
    class Struct(Structure):
        _fields_ = [('a', c_int16)]
    
    Struct3 = 3 * Struct
    
    c_array = (2 * Struct3)(
        Struct3(Struct(a=1), Struct(a=2), Struct(a=3)),
        Struct3(Struct(a=4), Struct(a=5), Struct(a=6))
    )
    
    assert c_array[0][0].a == 1
    
    p_obj = cast(pointer(c_array), POINTER(pointer(c_array)._type_))
    obj = p_obj.contents
    # Works on 3.12, obj is 2d
    assert obj[0][0].a == 1
    
    p_obj = cast(pointer(c_array[0]), POINTER(pointer(c_array)._type_))
    obj = p_obj.contents
    # Fails on 3.12, obj is 2d
    assert obj[0][0].a == 1
  7. mattip commented on Aug 14, 2023

    @mattip
    Member

    Was this failing in the 3.12 alpha/beta releases? I think it may have to do with a PR that went in to rc1 just 2 weeks ago.

  8. rgommers commented on Aug 14, 2023

    @rgommers
    MemberAuthor

    I don't think it was failing before. We don't have a regular 3.12-dev CI job, so it's hard to be sure. But my impression is that it happened recently indeed.

  9. ZupoLlask commented on Sep 16, 2023

    @ZupoLlask

    I guess this is now fixed, as Python team reverted the commit causing this issue, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions