Skip to content

Fix a bug that caused writing to FITS of Table objects that had index columns to fail - #8077

Merged
taldcroft merged 2 commits into
astropy:masterfrom
astrofrog:fix-table-io-time-index
Oct 23, 2019
Merged

taldcroft merged 2 commits into
astropy:masterfrom
astrofrog:fix-table-io-time-index

Conversation

@astrofrog

@astrofrog astrofrog commented Nov 5, 2018 •

Copy link
Copy Markdown
Member

Edit: THIS REQUIRES #9009, keep open while that is addressed.

Before this fix, the following failed:

>>> from astropy.time import Time
>>> from astropy.table import Table
>>> time = Time(['2016-03-22T12:30:31', '2016-03-22T12:30:38', '2016-03-22T12:34:40'])
>>> table = Table()
>>> table['time'] = time
>>> table.add_index('time')
>>> table.write('test.fits', overwrite=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/table.py", line 2592, in write
    io_registry.write(self, *args, **kwargs)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/registry.py", line 560, in write
    writer(data, *args, **kwargs)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/fits/connect.py", line 388, in write_table_fits
    table_hdu = table_to_hdu(input, character_as_bytes=True)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/fits/convenience.py", line 470, in table_to_hdu
    table, hdr = time_to_fits(table)
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/io/fits/fitstime.py", line 533, in time_to_fits
    newtable.replace_column(col.info.name, Column(jd12, unit='d'))
  File "/Users/tom/Dropbox/Code/Astropy/astropy/astropy/table/table.py", line 1796, in replace_column
    raise ValueError('cannot replace a table index column')
ValueError: cannot replace a table index column

The solution is to remove the index columns.

@taldcroft - is there a better way to just remove all indices from a table?

@astropy-bot

astropy-bot Bot commented Nov 5, 2018 •

Copy link
Copy Markdown

Check results are now reported in the status checks at the bottom of this page.

@astropy astropy deleted a comment from astropy-bot Bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot Bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot Bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot Bot Nov 5, 2018
@astropy astropy deleted a comment from astropy-bot Bot Nov 5, 2018
@saimn

saimn commented Nov 5, 2018

Copy link
Copy Markdown
Contributor

It would be more generic to handle this case in Table.replace_column instead of raising an error. Why not dropping the index there, maybe with a keyword or warning ?

@astrofrog

Copy link
Copy Markdown
Member Author

@taldcroft - do you have any thoughts on this?

@astrofrog

astrofrog commented Nov 7, 2018 •

Copy link
Copy Markdown
Member Author

@saimn - on further thoughts, I guess my thinking is that Table has already made a choice to raise a clear exception in this use case of replace_column, and this PR fixes a bug in the FITS code that uses that. Changing the behavior in Table would actually be an API change. So I personally think we should just treat this as a bug fix and not modify Table.

I guess we could also add a keyword argument to control this but again this would be a change in API and I'd like to get this fix in v3.1.

@taldcroft

Copy link
Copy Markdown
Member

This looks fine as a bug fix. Ideally the right answer is to serialize the index so it round-trips, which would make generating the index more useful since it would persist. In theory the machinery is mostly there, but it probably would not be trivial in the end.

@saimn

saimn commented Nov 7, 2018

Copy link
Copy Markdown
Contributor

@astrofrog - I can understand your reasoning, but for me it is a Table thing, not FITS or Time. Looking back in time, #4090 (comment) shows - I think - that there is no real reason for raising an error in this case.
I don't know what would the good solution when replacing a column with an index (recreating all the indexes using this column may be tricky), but a simple solution could be to add a drop_index keyword, which would not change the default behavior.
And maybe another option would be to remove the use of replace_colum, I'm not sure if it is necessary here ? At least replacing it with newtable[col.info.name] = jd12 does not cause a failure in the unit tests.

@saimn

saimn commented Nov 7, 2018

Copy link
Copy Markdown
Contributor

@taldcroft - the serialization is not the issue (it seems that indexes are dropped when writing tables to fits), the crash happens when using replace_column here:

# The following is necessary to deal with multi-dimensional ``Time`` objects
# (i.e. where Time.shape is non-trivial).
jd12 = np.array([col.jd1, col.jd2])
# Roll the 0th (innermost) axis backwards, until it lies in the last position
# (jd12.ndim)
jd12 = np.rollaxis(jd12, 0, jd12.ndim)
newtable.replace_column(col.info.name, Column(jd12, unit='d'))

@codecov

codecov Bot commented Nov 8, 2018 •

Copy link
Copy Markdown

Codecov Report

Merging #8077 into master will decrease coverage by 0.03%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #8077      +/-   ##
==========================================
- Coverage   86.96%   86.92%   -0.04%     
==========================================
  Files         408      383      -25     
  Lines       61428    57826    -3602     
  Branches     1096     1056      -40     
==========================================
- Hits        53422    50267    -3155     
+ Misses       7383     6945     -438     
+ Partials      623      614       -9
Impacted Files Coverage Δ
astropy/io/fits/fitstime.py 84.5% <100%> (+0.15%) ⬆️
astropy/convolution/__init__.py 62.5% <0%> (-37.5%) ⬇️
astropy/wcs/__init__.py 50% <0%> (-16.67%) ⬇️
astropy/visualization/wcsaxes/coordinates_map.py 86.88% <0%> (-10.18%) ⬇️
astropy/modeling/mappings.py 84.31% <0%> (-8.79%) ⬇️
astropy/convolution/src/convolve.c 80.66% <0%> (-8.36%) ⬇️
astropy/wcs/src/unit_list_proxy.c 53.96% <0%> (-6.91%) ⬇️
astropy/visualization/units.py 68.08% <0%> (-5.5%) ⬇️
astropy/modeling/separable.py 90.62% <0%> (-5.26%) ⬇️
astropy/constants/__init__.py 94.87% <0%> (-5.13%) ⬇️
... and 277 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 253ea74...af611a3. Read the comment docs.

@astrofrog astrofrog modified the milestones: v3.1, v3.2 Nov 13, 2018
@astrofrog

Copy link
Copy Markdown
Member Author

Ok let's delay this for now, I don't have time to work on it more for 3.1

@astrofrog astrofrog modified the milestone: v3.2 Nov 22, 2018

@taldcroft taldcroft left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in discussion comment.

@astropy-bot

astropy-bot Bot commented Jun 15, 2019

Copy link
Copy Markdown

Hi humans 👋 - this pull request hasn't had any new commits for approximately 7 months. I plan to close this in a month if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

@astropy-bot

astropy-bot Bot commented Jul 17, 2019

Copy link
Copy Markdown

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.

@taldcroft

Copy link
Copy Markdown
Member

I re-opened this as I think that this approach is OK and fixes a real issue.

The only outstanding problem is the comment above that the current code actually corrupts the original table by removing indices on newtable. That's basically a bug that I plan to fix in #9009.

@bsipocz

bsipocz commented Jul 17, 2019

Copy link
Copy Markdown
Member

@astrofrog - could you please rebase and address the comment to make it ready to be merged?

@taldcroft

Copy link
Copy Markdown
Member

@bsipocz - I think you just missed that this would be best done by waiting on a fix for #9009, at which point the table copy (with copy=False) will do the right thing.

However, there definitely should be a test that this operation does not corrupt the original table. That will fail right now, but will start working once #9009 is fixed.

@bsipocz

bsipocz commented Jul 17, 2019

Copy link
Copy Markdown
Member

Oh, indeed. Then I add the keep-open here, as either that or a rebase that makes sure this doesn't get closed again when the bot runs next time.

@bsipocz bsipocz modified the milestones: v3.2.2, v4.0 Sep 27, 2019
@taldcroft
taldcroft force-pushed the fix-table-io-time-index branch from 60d051e to 1ef279e Compare October 22, 2019 17:41
@taldcroft
taldcroft force-pushed the fix-table-io-time-index branch from 1ef279e to af611a3 Compare October 22, 2019 17:45
@taldcroft

Copy link
Copy Markdown
Member

@astrofrog et al. - I think should be good now, please review. The crux lines of code depend on table improvements in 4.0-dev, so I don't recommend trying to backport.

@bsipocz bsipocz removed the keep-open label Oct 22, 2019
@bsipocz

bsipocz commented Oct 22, 2019

Copy link
Copy Markdown
Member

@taldcroft - There are no more backports except critical fixes like the one we need for iers :)

@bsipocz

bsipocz commented Oct 22, 2019

Copy link
Copy Markdown
Member

@taldcroft - #9009 still seems to be relevant, is that not an issue for this any more?

@taldcroft

Copy link
Copy Markdown
Member

#9009 is still relevant because it will make it possible to improve the implementation here to avoid making any copies. Right now it does copy any mixin columns that have indices. So I put in a breadcrumb in #9009 to come back here and have a look.

@taldcroft
taldcroft merged commit 04e6d66 into astropy:master Oct 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants