numpy.fromfile does not accept StringIO object (Trac #1634)

Original ticket http://projects.scipy.org/numpy/ticket/1634 on 2010-10-11 by trac user foobaron, assigned to unknown.

aArray = numpy.fromfile(signalfile, numpy.dtype('u1'), number_of_bytes)

signalfile = <StringIO.StringIO instance at 0xa0008ec>1
-->
IOError: first argument must be an open file

used python version: 2.6.5

@pv

@pv wrote on 2010-10-28

Ditto for tofile.

@cgohlke

@cgohlke @pv Is this fixed now?

I checked this is still the case in NumPy 1.9 and both Python 2 and 3.

Still present; the number parsing in numpyos needs to read from an OS-level file descriptor. Fix nontrivial (file-like objects require you to go through python to get characters, and don't provide ungetc).

Is there a workaround or something?

Error still not fixed

@cgohlke

And it's broken @cgohlke 's work-around because now it throws TypeError instead of IOError.

If it helps anyone else, I was also hitting this same problem with a third-party library using numpy.fromfile. I really wanted to stream an in-memory byte string to it rather than writing a temporary file to disk.

I finally found that tempfile.SpooledTemporaryFile does the trick. You can create a SpooledTemporaryFile, write the bytes to it (in memory) and supply this to numpy.fromfile without errors...in my experience.

Actually, I need to amend my last comment. It looked like everything was working, but I suspect a file intermediate is still being used because the documentation for SpooledTemporaryFile says that the contents will be written to disk if fileno() is called, and I've seen that this is exactly what's happening with numpy.fromfile. So it's probably not avoiding the to disk problem. YMMV

glad to have found this issue... I wanted to load an array from a compressed file and thought np.fromfile(...LZMAFile...) would work. it doesn't, it uses fileno() of the compressed file and interprets the compressed data rather than the uncompressed one.

plain = bytes.fromhex("76543210")
compressed = lzma.compress(plain)
lzfile = lzma.open(io.BytesIO(compressed))
print(x.read().hex())
x.seek(0)
print(np.fromfile(x, dtype=np.uint8)) # this fails because the file object has no fileno method

I think this would be a really useful feature to have. The only workaround would be to .read() from that file and pass the bytes object to np.frombuffer(). I'm unsure if that's efficient.

@crackwitz

@crackwitz using np.frombuffer should be just as efficient. If you check the result from it, you will see that its res.flags shows that it is readonly, which is because the bytes object returned by file.read() is readonly. So that is the only thing you are losing speed wise: if you want a writeable array, you will need to make a single copy (there may be some small efficiency difference due to different methods of reading and small overheads if you are loading small files).

For such simple cases, readinto is also an option:

b = np.empty(len(plain), dtype=np.uint8)
x.readinto(b)

@cgohlke

@cgohlke nice. Where in the documentation should we add that to make it more discoverable?

Just tried this recently and got the following error.

In [1]: import io                                                               

In [2]: import numpy as np                                                      

In [3]: b = b"\x00\x01\x02\x03\x04"                                             

In [4]: f = io.BytesIO(b)                                                       

In [5]: a = np.fromfile(f, dtype="u1")                                          
---------------------------------------------------------------------------
UnsupportedOperation                      Traceback (most recent call last)
<ipython-input-5-186bca0dc811> in <module>
----> 1 a = np.fromfile(f, dtype="u1")

UnsupportedOperation: fileno

Any news on this? Is switching from np.fromfile(f, ...) to np.frombuffer(f.read(x), ...) the only standard, non-disk option?

Also, the title should change to reflect that io.BytesIO is not accepted either. This is not an issue with just text streams, but with any stream that is not an actual on-disk local file.

For those in need of a workaround to this limitation, here's a wrapper based on Teledyne-Marine/pyread7k#14's solution:

AnyFile = typing.Union[str, bytes, os.PathLike, io.IOBase]

def numpy_fromfile(file: AnyFile, dtype: numpy.typing.DTypeLike = float, count: int = -1):
    """numpy.fromfile() wrapper, handling io.BytesIO file-like streams.

    Numpy requires open files to be actual files on disk, i.e., must support
    file.fileno(), so it fails with file-like streams such as io.BytesIO().

    If numpy.fromfile() fails due to no file.fileno() support, this wrapper
    reads the required bytes from file and redirects the call to
    numpy.frombuffer().

    See https://github.com/numpy/numpy/issues/2230
    """
    try:
        return numpy.fromfile(file, dtype=dtype, count=count)
    except io.UnsupportedOperation as e:
        if not (e.args and e.args[0] == 'fileno' and isinstance(file, io.IOBase)):
            raise  # Nothing I can do about it
        dtype = numpy.dtype(dtype)
        buffer = file.read(dtype.itemsize * count)
        return numpy.frombuffer(buffer, dtype=dtype, count=count)

Can anyone verify if this a proper workaround?

@MestreLion

thanks @MestreLion!
This works for me.

Navigation Menu

Repository navigation

numpy.fromfile does not accept StringIO object (Trac #1634) #2230

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

numpy.fromfile does not accept StringIO object (Trac #1634) #2230

Description

Activity

numpy-gitbot commented on Oct 19, 2012

charris commented on Jan 16, 2014

astrojuanlu commented on Nov 13, 2014

aarchiba commented on Aug 26, 2015

josechavezm commented on Jun 22, 2017

nikita6187 commented on Jul 27, 2017

LeeKamentsky commented on Aug 11, 2017

drkeoni commented on Sep 12, 2018

drkeoni commented on Sep 12, 2018

crackwitz commented on Feb 18, 2019

seberg commented on Feb 18, 2019

cgohlke commented on Feb 19, 2019

mattip commented on Feb 19, 2019

jakirkham commented on Nov 15, 2019

MestreLion commented on Oct 21, 2021

MestreLion commented on Oct 22, 2021

MestreLion commented on Oct 22, 2021

marscher commented on Aug 29, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions