Check file type annotations against open mode

I have a function argument whose type I want to limit to a file object that's open for binary reading — i.e. I want only an instance of io.BufferedReader as returned by open() when mode="rb".

from typing import IO


def parser(file_obj: IO[bytes]) -> None:
    pass

As I would expect, mypy runs without complaint on this code:

parser(open("example.bin", "rb"))

But it doesn't output an error for this:

parser(open("example.txt", "rt"))

In that case, I would want an error along the lines of Argument 1 to "parser" has incompatible type IO[str]; expected "IO[bytes]".

This would probably have to wait for some kind of "dependent types", unless we decide that `open()` is so important we write custom code for it in mypy. (Which IMO isn't totally unreasonable given that it's a builtin.)

I think that special casing open and a few other stdlib functions would be totally reasonable. We already special case the string % operator so there is even a precedent.

Hello,

I was trying to type check if the file was open in binary mode too, and noticed a few things (and a workaround right at the end of all this)...

mypy -2 doesn't seem to know that io.BufferedReader is there.

io.BufferedReader
afile = io.open('/etc/passwd', 'rb')

def read_data(f):
    # type: (io.BufferedReader) -> bytes
    return f.read()

bytes_issue.py:2: error: "module" has no attribute "BufferedReader"
bytes_issue.py:5: error: Name 'io.BufferedReader' is not defined

If I run mypy in python3 mode, it passes.

Also if I change it to use _io, it works under mypy -2, but not in python3 mode.

_io.BufferedReader
afile = _io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (_io.BufferedReader) -> bytes
    return f.read()

With mypy in 3 mode I get:

bytes_issue.py:1: error: Cannot find module named '_io'

In python python2.7 and 3.6, the types for io.open are like so.

>>> type(io.open('/etc/passwd', 'r'))
<type '_io.TextIOWrapper'>
>>> type(io.open('/etc/passwd', 'rb'))
<type '_io.BufferedReader'>

Hacking about some more I make this mess below which does let me check that every file obj passed into the object is a BufferedReader. Of course, I need to manually cast every instance... not at all nice.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)
if PY3:
    from io import BufferedReader
else:
    from _io import BufferedReader
afile = io.open('/etc/passwd', 'rb')
def read_data(f):
    # type: (BufferedReader) -> bytes
    return f.read()
read_data(cast(BufferedReader, afile))
#The following code will give a type error, as I haven't cast it to a binary file.
#another_file = io.open('/etc/passwd', 'r')
#read_data(afile)

Finally I decide to have a separate open function wrapper for every mode. This way I can use a cast inside it, and the type checker works. Any file not using my custom wrappers also fails - which is great because I know where I have to track things down to check them.

import io, sys
from typing import cast
PY3 = sys.version_info > (3,)

if PY3:
    from io import BufferedReader, TextIOWrapper
else:
    from _io import BufferedReader, TextIOWrapper

def my_io_open_r(fname): # type: (str) -> TextIOWrapper
    return cast(TextIOWrapper, io.open(fname, 'r'))
def my_io_open_rb(fname): # type: (str) -> BufferedReader
    return cast(BufferedReader, io.open(fname, 'rb'))

def read_data(f): # type: (BufferedReader) -> bytes
    return f.read()

read_data(my_io_open_rb('/etc/passwd'))

# below fails, as it is opened with 'r'
# read_data(my_io_open_r('/etc/passwd'))

# below also fails type check with a got IO[Any] error.
# read_data(io.open('/etc/passwd', 'r'))

I think this technique would work anywhere you have a factory pattern where the return types are dependent on logic inside the function, or the parameters. As long as number of combinations are not too big that is! (open_rb, open_r, open_w, open_wb)

cheers,

#3299 has fixed this for calls where the mode argument is either a literal or not explicitly provided.

@illume

Actually some of @illume's issues (e.g. io.BufferedReader doesn't exist in PY2) are pure typeshed issues. @illume Could you open a new issue in typeshed about this (and the other issues you've found related to missing classes in io.pyi)?

Navigation Menu

Repository navigation

Check file type annotations against open mode #2337

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Check file type annotations against open mode #2337

Description

Activity

gvanrossum commented on Oct 26, 2016

JukkaL commented on Oct 26, 2016

illume commented on Feb 18, 2017

JukkaL commented on May 31, 2017

gvanrossum commented on May 31, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions