Repository navigation
New in NumPy 1.14: failures in PyTables #10344
Description
In the following snippet:
import numpy as np
data_proper = [(456, b'dbe', 1.2), (2, b'ded', 1.3)]
data_improper = [[456, b'dbe', 1.2], [2, b'ded', 1.3]]
descr = [('f0', '<i4'), ('f1', 'S3'), ('f2', '<f8')]
dtype = np.dtype((np.record, descr))
np.array(data_improper, dtype=dtype)Execution of the last line in 1.13 used to raise:
TypeError: a bytes-like object is required, not 'int'
which allowed the logic in numpy/core/records.py#L676 to recover and reinterpret the line as np.array(data_proper, dtype=dtype).
The trouble is that in NumPy 1.14
ValueError: invalid literal for int() with base 10: b'dbe'
is being raised instead, which results in the failure.
This is a gist of the issue behind numerous test failures in pyTables when used with NumPy 1.14.
Activity
charris commentedon Jan 8, 2018
The error comes from cPython, but where it actually originates is hard to tell, the traceback has been truncated too far. I think we will need a bisect on this.
eric-wieser commentedon Jan 8, 2018
Perhaps the easy fix is to just to catch
ValueErrorinrec.arrayas well?charris commentedon Jan 8, 2018
@eric-wieser I'd worry that that might not be the only place affected. I think we need to know just what changed.
eric-wieser commentedon Jan 8, 2018
I think the simplest reproduction is:
However, the new behavior seems kinda consistent, when compared to:
and
charris commentedon Jan 8, 2018
For reference, the 1.13.3 error is
TypeError: expected a readable buffer object.charris commentedon Jan 8, 2018
I do think the catch in
records.pyis too narrow, too dependent on underlying implementation details, so that needs a fix.eric-wieser commentedon Jan 8, 2018
Frankly, I think we should deprecate that code path entirely. If you pass malformed data, expecting numpy to guess what you meant seems undesirable
charris commentedon Jan 8, 2018
The change came in with #6053. I don't think this is so much a bug, as a change in behavior.
charris commentedon Jan 8, 2018
I think we need a bigger discussion on the way forward, so pinging @ahaldane . IIRC, h5py was also affected by this change.
ahaldane commentedon Jan 8, 2018
I'm reading along. Following on what @eric-wieser said, the change in behavior was to allow scalars to broadcast to all fields during array creation, not just during assignment. The difference in behavior of these two lines seemed inconsistent in 1.13:
Furthermore, in 1.13:
Why should the first line fail, but the second succeed?
So my feeling is that the special casing in
records.pyisn't ideal, but maybe there is away to keep things back-compatible, I'm not sure yet.By the way, I will be at a conference later this week, so may be away from my computer.
charris commentedon Jan 8, 2018
The problem seems isolated to the
record.pymodule, so for 1.14 addingValueErrorto the list of caught errors is probably an adequate fix. @eric-wieser's point about trying to fix up malformed input is the long term problem, and we probably need more time than 1.14 allows to deal with that.oleksandr-pavlyk commentedon Jan 8, 2018
@charris Thank you. The improper record array object input to
numpy.record.arraywas all coming from hard-coded list of lists in pyTable's test files, so it is easy, albeit tedious, to modify. I am prepping a PR for pyTable now.replaced hand-coded list of lists Python object inputs to numpy.recor…
17 remaining items
BUG: 1.14 multifield-indexing adds padding bytes
BUG: 1.14 multifield-indexing adds padding bytes
BUG: 1.14 multifield-indexing adds padding bytes
BUG: 1.14 multifield-indexing adds padding bytes
charris commentedon Feb 8, 2018
OK, made the suggested fix in #10543. Note that the @ahaldane reverted the causal changes for 1.14.1, but getting a FutureWarning out there early is probably a good thing.
DEP: Issue FutureWarning when malformed records detected.
DEP: Issue FutureWarning when malformed records detected.
DEP: Issue FutureWarning when malformed records detected.
MAINT: Fixed test failures with Python 2.7 and NumPy 1.14.3