Skip to content

Numpy dev fixes for subarray dtype - #10818

Closed
mhvk wants to merge 2 commits into
astropy:masterfrom
mhvk:numpy-dev-subarray-dtype-fixes
Closed

mhvk wants to merge 2 commits into
astropy:masterfrom
mhvk:numpy-dev-subarray-dtype-fixes

Conversation

@mhvk

@mhvk mhvk commented Oct 8, 2020

Copy link
Copy Markdown
Contributor

fixes #10815

Not very happy with the fix for Column, but could not think of a better way. Ideally, this gets fixed better by numpy. See numpy/numpy#17511.

mhvk added 2 commits October 8, 2020 17:24
This works around what arguably is a bug in numpy < 1.20, and which
raises a DeprecationWarning in numpy 1.20-dev.

Also has the benefit of the new code being more explicit.
…dtype.

This was something that happened to work in numpy < 1.20, but actually
led to a Column with arguably the wrong dtype.  In numpy 1.20-dev,
this leads to DeprecationWarning.  Unfortunately, it seems not possible
to avoid special-casing the empty list.
@pllim

pllim commented Oct 8, 2020

Copy link
Copy Markdown
Member

I thought I saw multiple occurrence of the thingy in votable but I'll double check again to be sure. If nothing is amiss, I'll approve. Thank you so much!

Comment thread astropy/table/column.py
# numpy 1.20-dev this is giving a DeprecationWarning.
# TODO: Adjust when/if the numpy behaviour behaviour changes.
# Hopefully, this clause can again be removed. NUMPY_LT_1_20.
self_data = np.empty(0, dtype=dtype)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other solution that is possible if you know that the list represents effectively a 1-D dataset is to use:
np.empty(len(data), dtype=dtype) and then assign to that result. That does have slightly different semantics though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that might indeed work. But I think the standard case should just go through np.array(...), since Column is an ndarray subclass.

The whole thing does make one wonder whether the new behaviour in numpy is in fact an improvement, since it doesn't seem to move to the "more logical" state - re-opening the numpy issue.

@mhvk
mhvk marked this pull request as draft October 8, 2020 23:07
@mhvk

mhvk commented Oct 8, 2020

Copy link
Copy Markdown
Contributor Author

I've temporarily converted this to a draft, pending further discussion over at the numpy issue.


self._base = base
self.default = np.array([], dtype=self._base.format)
self.default = np.empty(0, dtype=self._base.format)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is the only case where [] is used with self._base.format. I think we should wait till Numpy wheel picks up that patch (and hence fails our numpy-dev job on master) and then make sure the numpy-dev job here passes with these changes. Otherwise, LGTM on the io.votable side. Thank you!

Comment thread astropy/table/column.py
@pllim pllim mentioned this pull request Oct 9, 2020
@astrofrog astrofrog modified the milestones: v4.0.2, v4.0.3 Oct 10, 2020
@pllim

pllim commented Oct 12, 2020

Copy link
Copy Markdown
Member

The failure is on master now (example log: https://travis-ci.org/github/astropy/astropy/jobs/734945598). I restarted the numpy-dev job here. FYI.

@pllim

pllim commented Oct 12, 2020

Copy link
Copy Markdown
Member

On master:
Numpy: 1.20.0.dev0+07a54df failed

Here:
Numpy: 1.20.0.dev0+07a54df passed 🎉

@seberg

seberg commented Oct 12, 2020

Copy link
Copy Markdown
Contributor

Well, I will try to fix up NumPy to reduce the noise level of this change (for astropy, probably that effectively means no noise), but if the noise comes it might be more annoying :) (but also indicate buggy behaviour).

@pllim pllim added the Upstream Action Required Was: Upstream Fix Required label Oct 12, 2020
@pllim

pllim commented Oct 12, 2020

Copy link
Copy Markdown
Member

@seberg , thank you! Please keep us updated. 😄

@astrofrog astrofrog modified the milestones: v4.0.3, v4.0.4 Oct 14, 2020
@pllim

pllim commented Oct 29, 2020

Copy link
Copy Markdown
Member

@mhvk , does this mean we can close this without merge?

@mhvk

mhvk commented Oct 29, 2020

Copy link
Copy Markdown
Contributor Author

Yes, hopefully, but let's do it only once our numpy-dev tests pass again

@mhvk
mhvk deleted the numpy-dev-subarray-dtype-fixes branch November 9, 2020 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

numpy-dev using a dtype with a subarray field is deprecated

5 participants