Skip to content

recfunctions.append_fields fails on arrays containing objects (Trac #1751) #2346

Closed

Description

@numpy-gitbot

Original ticket http://projects.scipy.org/numpy/ticket/1751 on 2011-02-28 by trac user aickley, assigned to unknown.

The problem is that append_fields calls array.view() which does not accept arrays containing objects (See [http://projects.scipy.org/numpy/ticket/674])

Test case (to include in numpy/lib/tests/recfunctions.py):

class TestAppendFieldsObj(TestCase):
    """
    Test append_fields with arrays containing objects
    """
    def setUp(self):
        from datetime import date
        self.data = dict(obj=date(2000, 1, 1))

    def test_append_to_objects(self):
        "Test append_fields when the base array contains objects"
        obj = self.data['obj']
        x = np.array([(obj, 1.), (obj, 2.)], dtype=[('A', object), ('B', float)])
        y = np.array([10, 20], dtype=int)
        test = append_fields(x, 'C', data=y, usemask=False)
        control = np.array([(obj, 1.0, 10), (obj, 2.0, 20)],
                           dtype=[('A', object), ('B', float), ('C', int)])
        assert_equal(test, control)
    #
    def test_append_with_objects(self):
        "Test append_fields when the appended data contains objects"
        obj = self.data['obj']
        x = np.array([(10, 1.), (20, 2.)], dtype=[('A', int), ('B', float)])
        y = np.array([obj, obj], dtype=object)
        test = append_fields(y, 'C', data=y, dtypes=object, usemask=False)
        control = np.array([(10, 1.0, obj), (20, 2.0, obj)],
                           dtype=[('A', int), ('B', float), ('C', object)])
        assert_equal(test, control)






======================================================================
ERROR: Test append_fields when the base array contains objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", line 406, in test_append_to_objects
    test = append_fields(x, 'C', data=y, usemask=False)
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 629, in append_fields
    base = merge_arrays(base, usemask=usemask, fill_value=fill_value)
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 399, in merge_arrays
    return seqarrays.view(dtype=seqdtype, type=seqtype)
TypeError: Cannot change data-type for object array.

======================================================================
ERROR: Test append_fields when the appended data contains objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", line 416, in test_append_with_objects
    test = append_fields(y, 'C', data=y, dtypes=object, usemask=False)
  File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 627, in append_fields
    for (a, n, d) in zip(data, names, dtypes)]
TypeError: Cannot change data-type for object array.

----------------------------------------------------------------------
Ran 33 tests in 0.062s

Regarding the first failure, append_fields calls merge_arrays on the base array:

base = merge_arrays(base, usemask=usemask, fill_value=fill_value)

Actually this line can be completely removed without breaking existing unit tests. I don't like this call because if we are calling merge_array on a single array we are are not actually merging anything, but rather making sure that the array conforms to certain criteria. Maybe this code should be refactored out from merge_arrays...

Activity

  1. numpy-gitbot commented on Oct 19, 2012

    @numpy-gitbot
    Author

    @rgommers wrote on 2011-03-02

  2. added 10 commits that reference this issue on Jun 4, 2015
    907c541
    b7fece9
    4e3841c
    8171464
    34bdab2
    00a31fe
    432ddcd
    f3903e3
    958bf63
    e2cd6fa
  3. ahaldane commented on Oct 29, 2015

    @ahaldane
    Member

    Reopening this after the partial backtrack in #6562.

    Now it is partially fixed, but appending object fields still doesn't work. Once it is fixed, remember to add back the test removed in #6562.

    I suspect this will be fixed by #6053.

  4. hpaulj commented on Apr 6, 2016

    @hpaulj

    This

    base = merge_arrays(base, usemask=usemask, fill_value=fill_value)
    

    is also causing problems if base is a single element masked array.

    http://stackoverflow.com/questions/36440557/typeerror-when-appending-fields-to-a-structured-array-of-size-one

    If len(base) is 1, merge_arrays calls np.asanyarray([base][0]). In the case of masked structure array this returns a void array element, which gives a subsequent ravel a headache.

    Simply dropping this merge_arrays call might the solution, if it's sole purpose is to cleanup some obscure way of passing a base to append_fields.

    (I'm not familiar enough with the numpy issues to know whether adding this to this old issue is the best choice, or whether it would be better to raise a new issue.)

    I ran the test_recfunctions.py on this modification, and it ran fine. So apparently this base 'cleanup' is not needed for the test cases.

  5. 14 remaining items

  6. eric-wieser commented on Jul 4, 2017

    @eric-wieser
    Member

    This is also fixed in #8514

  7. added a commit that references this issue on Oct 23, 2017
    ae582a8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions