Skip to content

Support for Path objects in io packages - #4606

Merged
taldcroft merged 7 commits into
astropy:masterfrom
anchitjain1234:fix-4412
Mar 1, 2016
Merged

taldcroft merged 7 commits into
astropy:masterfrom
anchitjain1234:fix-4412

Conversation

@anchitjain1234

Copy link
Copy Markdown
Contributor

For #4412.
Right now added support for io.ascii by modifying get_readable_fileobj in utils because io.ascii uses get_readable_fileobj to read.
I would also try to add support for Path objects in other io packages.

@anchitjain1234

Copy link
Copy Markdown
Contributor Author

@taldcroft Is this the correct approach? I tried to follow yours as in #4405.

@anchitjain1234 anchitjain1234 changed the title Added support for Path objects in get_readable_fileobj in utils Support for Path objects in io packages Feb 13, 2016
By modifying get_readable_fileobj in utils because io.ascii uses get_readable_fileobj to read.
@taldcroft

Copy link
Copy Markdown
Member

@anchitjain1234 - thanks, this looks promising! If you do a census throughout the project of places that use get_readable_fileobj, I would be nice to see demonstration through testing of this fix being more broadly applicable than just io.ascii. I'm not sure off-hand how io.fits and io.votable handle the front-end task of handling the "input" file, which might be one of many different object types.

@taldcroft taldcroft self-assigned this Feb 15, 2016
@taldcroft taldcroft added this to the v1.2.0 milestone Feb 15, 2016
Also modified docstrings of io.votable.validate and io.ascii.read for indicating capability to read Path objects
Also modified docstring of fitsopen in hdulist.py
@anchitjain1234

Copy link
Copy Markdown
Contributor Author

If you do a census throughout the project of places that use get_readable_fileobj, I would be nice to see demonstration through testing of this fix being more broadly applicable than just io.ascii.

@taldcroft I grepped the whole project to find the usage of get_readable_fileobj. Check results here.
From the above results, I think I have covered through tests io.ascii, io.votable and utils.
But I am not sure if vo also needs to be tested? because it uses get_readable_fileobj to get data from URL only

I'm not sure off-hand how io.fits and io.votable handle the front-end task of handling the "input" file

I handled io.fits directly in _File class and io.votable is automatically accepting pathlib.Path object because it uses utils.xml.iterparser which in turn uses get_readable_fileobj.

I also added a separate test for votable.validate because it uses get_readable_fileobj directly to get content.

@anchitjain1234

Copy link
Copy Markdown
Contributor Author

@taldcroft Could you please check the changes?

Comment thread astropy/io/fits/file.py Outdated
else:
self.simulateonly = False
# If fileobj is of type pathlib.Path
if PY3 and HAS_PATHLIB and isinstance(fileobj, Path):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to check PY3, all that matters is HAS_PATHLIB to make sure the second logical test will execute properly.

@taldcroft

Copy link
Copy Markdown
Member

@anchitjain1234 - looks pretty good, thanks!

@saimn

saimn commented Feb 23, 2016

Copy link
Copy Markdown
Contributor

Also pathlib can be installed on PY2: https://pypi.python.org/pypi/pathlib2/
So it is better to avoid testing for PY3 as @taldcroft said.

@anchitjain1234

Copy link
Copy Markdown
Contributor Author

@taldcroft Please check the changes.

Comment thread astropy/io/registry.py Outdated
if six.PY3:
# path might be a pathlib.Path object if HAS_PATHLIB,
# so coerce to a regular string.
if HAS_PATHLIB:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It occurs to me that in theory a user might be passing a unicode file name on Python 2 with the backport of pathlib independently installed. Then conversion with str() would be a problem. So maybe you need to also check isinstance(args[0], pathlib.Path)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@taldcroft If I understand correctly, you meant to say for example a file named x©.fits on Python 2 with the backport of pathlib independently installed would be a problem?
I cant understand why would it be a problem?
I also tested it on my system on Python 2 and Python 3 where it works correctly.

@taldcroft

Copy link
Copy Markdown
Member

@anchitjain1234 - looks good! Now you just need to add an entry to CHANGES.rst describing this update. You can put [skip ci] in the commit message to avoid re-running CI testing.

@anchitjain1234

Copy link
Copy Markdown
Contributor Author

@taldcroft In Python 2 pathlib is pathlib 2 therefore import pathlib would not work. So do I need to add import pathlib2 statements also?

@saimn

saimn commented Feb 29, 2016

Copy link
Copy Markdown
Contributor

@anchitjain1234 - Even if the package name is pathlib2, it will import as pathlib.

@taldcroft - Maybe the extra dependency should be added to a Python2 build in Travis ?

@anchitjain1234

Copy link
Copy Markdown
Contributor Author

@taldcroft Changelog entry added.

@taldcroft

Copy link
Copy Markdown
Member

@saimn - I think that @anchitjain1234 has a point. If pathlib2 is installed the code still needs to explicitly do:

import pathlib2 as pathlib

I was hoping that pip install pathlib2 did a magic rename so that import pathlib would work on Py2, but I just tried and that is not the case. So this leads to the uglyness:

try:
   import pathlib
except ImportError:
    try:
        import pathlib2 as pathlib
.. etc

I think we should just merge this current version, which addresses the stated issue of providing support for bona-fide pathlib.Path objects. Supporting the Python2 backport can be a separate issue, but I think it will need to go along with refactoring the HAS_PATHLIB logic into a separate single entity.

@saimn

saimn commented Feb 29, 2016

Copy link
Copy Markdown
Contributor

@taldcroft - Ah sorry, I tested this on a virtualenv, that I thought was based on PY2 but it seems not. The backport provides indeed a pathlib2 package, and this is intentional : jazzband/pathlib2#8 (Probably for good reasons even if it is not handy).

taldcroft added a commit that referenced this pull request Mar 1, 2016
Support for Path objects in io packages
@taldcroft
taldcroft merged commit a447a48 into astropy:master Mar 1, 2016
@taldcroft

Copy link
Copy Markdown
Member

Thanks @anchitjain1234 ! And @saimn for useful review and input.

@anchitjain1234
anchitjain1234 deleted the fix-4412 branch April 8, 2016 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants