Table.read for Path objects and FITS files by taldcroft · Pull Request #4405 · astropy/astropy

taldcroft · 2015-12-18T01:07:38Z

I got this error message (see also #4165):

  File "code/hgps/figures/hgps_associations_image/hgps_associations_image.py", line 54, in draw_snrs
    table = Table.read(filename, format='fits')
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/table/table.py", line 2261, in read
    return io_registry.read(cls, *args, **kwargs)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/registry.py", line 332, in read
    data = reader(*args, **kwargs)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/connect.py", line 137, in read_table_fits
    hdulist = fits_open(input)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 138, in fitsopen
    return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 280, in fromfile
    save_backup=save_backup, cache=cache, **kwargs)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 801, in _readfrom
    ffo = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/file.py", line 143, in __init__
    self._open_filelike(fileobj, mode, clobber)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/file.py", line 455, in _open_filelike
    % self.mode)
OSError: File-like object does not have a 'write' method, required for mode 'ostream'.

The reason is that I used a pathlib.Path object, which doesn't work for filenames with Astropy.

I don't know if it would be desirable to make Astropy table and fits work with pathlib.Path objects in addition to filename strings.

But I noticed this, which maybe could be considered a bug?

from astropy.table import Table
from pathlib import Path

table = Table()
table['a'] = [1, 2, 3]

filename = Path('table_fits_test.fits.gz')
table.write(filename, format='fits', overwrite=True)
table = Table.read(filename, format='fits')
table.pprint()

$ python table_fits_test.py 
Traceback (most recent call last):
  File "table_fits_test.py", line 8, in <module>
    table.write(filename, format='fits', overwrite=True)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/table/table.py", line 2278, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/registry.py", line 380, in write
    writer(data, *args, **kwargs)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/connect.py", line 300, in write_table_fits
    table_hdu.writeto(output)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/hdu/base.py", line 1765, in writeto
    checksum=checksum)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 685, in writeto
    fileobj = _File(fileobj, mode='ostream', clobber=clobber)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/file.py", line 143, in __init__
    self._open_filelike(fileobj, mode, clobber)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/file.py", line 448, in _open_filelike
    self._overwrite_existing(clobber, fileobj, False)
  File "/Users/deil/Library/Python/3.4/lib/python/site-packages/astropy-1.1.dev14232-py3.4-macosx-10.11-x86_64.egg/astropy/io/fits/file.py", line 386, in _overwrite_existing
    raise IOError("File %r already exists." % self.name)
OSError: File 'table_fits_test.fits.gz' already exists.

The overwrite=True argument to Table.write is not recognised in this case!?
Can this behaviour or the error messages when pathlib.Path objects are passed be improved?

taldcroft · 2015-12-18T01:08:17Z

Code attached.

cc: @embray @astrofrog

taldcroft · 2015-12-18T01:10:32Z

Note that this only applies to the I/O registry and not the underlying FITS layer.

mhvk · 2015-12-18T16:26:29Z

Looks good!

astrofrog · 2015-12-18T16:38:57Z

+1 (but needs a changelog entry)

Allow pathlib.Path objects for registry read / write file names

embray · 2015-12-21T23:20:43Z

Please don't merge PRs (especially for bug fixes!) without applying a milestone.

taldcroft · 2015-12-22T02:11:02Z

@embray - sorry (again, sigh). But in fact I think the current classification of this PR should be Enhancement, not Bug, and that the milestone should be 1.2. Certainly should have discussed this before merge. Can I blame the reviewers?? 😄

It does not fix the underlying issue in io.fits (and io.ascii, and probably votable) which causes unexpected behavior if a Path object gets passed (#4412). Instead this PR just adds the ability to pass a Path object by converting to a string before the low-level I/O functions are called. Along the way it does prevent the original issue here from occurring.

embray · 2015-12-23T15:47:21Z

I'm not super worried about how it's "classified". Just important to add the milestone so that changes can get sorted correctly into their targeted releases.

taldcroft · 2015-12-23T16:21:08Z

@embray - ok I've changed this to an enhancement for 1.2.0. This is another lesson for me to be more cautious about converting an issue to a PR. In this case a new PR would have worked out better.

embray · 2015-12-23T16:57:32Z

I think it should still be where it already was. Missing support for pathlib can be considered a bug.

embray · 2015-12-23T16:58:44Z

Besides if you're going to change the milestone you would also have to move the changelog entry. I think it's fine as it is. I was just nagging about the milestone for the purpose of reminding.

Allow pathlib.Path objects for registry read / write file names

astrofrog added table io.fits Bug labels Dec 17, 2015

Allow pathlib.Path objects in registry read / write

bc497ae

Update CHANGES.rst [skip ci]

d627d86

taldcroft added a commit that referenced this pull request Dec 19, 2015

Merge pull request #4405 from taldcroft/registry-path

51ae92c

Allow pathlib.Path objects for registry read / write file names

taldcroft merged commit 51ae92c into astropy:master Dec 19, 2015

taldcroft deleted the registry-path branch December 19, 2015 16:28

embray added this to the v1.1.1 milestone Dec 21, 2015

embray added the Affects-release label Dec 21, 2015

taldcroft mentioned this pull request Dec 22, 2015

Path objects not supported in low-level I/O packages #4412

Closed

taldcroft added Enhancement and removed Bug labels Dec 23, 2015

taldcroft modified the milestones: v1.2.0, v1.1.1 Dec 23, 2015

taldcroft added the io.registry label Dec 23, 2015

embray modified the milestones: v1.1.1, v1.2.0 Dec 23, 2015

taldcroft added a commit that referenced this pull request Jan 7, 2016

Merge pull request #4405 from taldcroft/registry-path

b377005

Allow pathlib.Path objects for registry read / write file names

anchitjain1234 mentioned this pull request Feb 13, 2016

Support for Path objects in io packages #4606

Merged

Navigation Menu

Repository navigation

Table.read for Path objects and FITS files - #4405

Table.read for Path objects and FITS files#4405
taldcroft merged 2 commits into
astropy:masterfrom
taldcroft:registry-path

taldcroft commented Dec 18, 2015

Uh oh!

taldcroft commented Dec 18, 2015

Uh oh!

taldcroft commented Dec 18, 2015

Uh oh!

mhvk commented Dec 18, 2015

Uh oh!

astrofrog commented Dec 18, 2015

Uh oh!

embray commented Dec 21, 2015

Uh oh!

taldcroft commented Dec 22, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

taldcroft commented Dec 23, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

taldcroft commented Dec 18, 2015

Uh oh!

taldcroft commented Dec 18, 2015

Uh oh!

taldcroft commented Dec 18, 2015

Uh oh!

mhvk commented Dec 18, 2015

Uh oh!

astrofrog commented Dec 18, 2015

Uh oh!

embray commented Dec 21, 2015

Uh oh!

taldcroft commented Dec 22, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

taldcroft commented Dec 23, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

embray commented Dec 23, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants