Repository navigation
recfunctions.append_fields fails on arrays containing objects (Trac #1751) #2346
Copy link
Copy link
Closed
Milestone
Description
Original ticket http://projects.scipy.org/numpy/ticket/1751 on 2011-02-28 by trac user aickley, assigned to unknown.
The problem is that append_fields calls array.view() which does not accept arrays containing objects (See [http://projects.scipy.org/numpy/ticket/674])
Test case (to include in numpy/lib/tests/recfunctions.py):
class TestAppendFieldsObj(TestCase):
"""
Test append_fields with arrays containing objects
"""
def setUp(self):
from datetime import date
self.data = dict(obj=date(2000, 1, 1))
def test_append_to_objects(self):
"Test append_fields when the base array contains objects"
obj = self.data['obj']
x = np.array([(obj, 1.), (obj, 2.)], dtype=[('A', object), ('B', float)])
y = np.array([10, 20], dtype=int)
test = append_fields(x, 'C', data=y, usemask=False)
control = np.array([(obj, 1.0, 10), (obj, 2.0, 20)],
dtype=[('A', object), ('B', float), ('C', int)])
assert_equal(test, control)
#
def test_append_with_objects(self):
"Test append_fields when the appended data contains objects"
obj = self.data['obj']
x = np.array([(10, 1.), (20, 2.)], dtype=[('A', int), ('B', float)])
y = np.array([obj, obj], dtype=object)
test = append_fields(y, 'C', data=y, dtypes=object, usemask=False)
control = np.array([(10, 1.0, obj), (20, 2.0, obj)],
dtype=[('A', int), ('B', float), ('C', object)])
assert_equal(test, control)
======================================================================
ERROR: Test append_fields when the base array contains objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", line 406, in test_append_to_objects
test = append_fields(x, 'C', data=y, usemask=False)
File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 629, in append_fields
base = merge_arrays(base, usemask=usemask, fill_value=fill_value)
File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 399, in merge_arrays
return seqarrays.view(dtype=seqdtype, type=seqtype)
TypeError: Cannot change data-type for object array.
======================================================================
ERROR: Test append_fields when the appended data contains objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/tests/test_recfunctions.py", line 416, in test_append_with_objects
test = append_fields(y, 'C', data=y, dtypes=object, usemask=False)
File "/home/kolpakov/lib/python2.6/site-packages/numpy/lib/recfunctions.py", line 627, in append_fields
for (a, n, d) in zip(data, names, dtypes)]
TypeError: Cannot change data-type for object array.
----------------------------------------------------------------------
Ran 33 tests in 0.062s
Regarding the first failure, append_fields calls merge_arrays on the base array:
base = merge_arrays(base, usemask=usemask, fill_value=fill_value)
Actually this line can be completely removed without breaking existing unit tests. I don't like this call because if we are calling merge_array on a single array we are are not actually merging anything, but rather making sure that the array conforms to certain criteria. Maybe this code should be refactored out from merge_arrays...
Reactions are currently unavailable
Activity
numpy-gitbot commentedon Oct 19, 2012
@rgommers wrote on 2011-03-02
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ENH: structured datatype safety checks
ahaldane commentedon Oct 29, 2015
Reopening this after the partial backtrack in #6562.
Now it is partially fixed, but appending object fields still doesn't work. Once it is fixed, remember to add back the test removed in #6562.
I suspect this will be fixed by #6053.
hpaulj commentedon Apr 6, 2016
This
is also causing problems if
baseis a single element masked array.http://stackoverflow.com/questions/36440557/typeerror-when-appending-fields-to-a-structured-array-of-size-one
If
len(base)is 1,merge_arrayscallsnp.asanyarray([base][0]). In the case of masked structure array this returns avoidarray element, which gives a subsequentravela headache.Simply dropping this
merge_arrayscall might the solution, if it's sole purpose is to cleanup some obscure way of passing abasetoappend_fields.(I'm not familiar enough with the numpy issues to know whether adding this to this old issue is the best choice, or whether it would be better to raise a new issue.)
I ran the
test_recfunctions.pyon this modification, and it ran fine. So apparently thisbase'cleanup' is not needed for the test cases.14 remaining items
eric-wieser commentedon Jul 4, 2017
This is also fixed in #8514
MAINT: improve struct assignment, allow multi fields views