Skip to content

Shape changing mixin - #4123

Merged
mhvk merged 12 commits into
astropy:masterfrom
mhvk:shape-changing-mixin
Jul 27, 2016
Merged

mhvk merged 12 commits into
astropy:masterfrom
mhvk:shape-changing-mixin

Conversation

@mhvk

@mhvk mhvk commented Sep 3, 2015 •

Copy link
Copy Markdown
Contributor

EDIT: this is now a simplied follow-up of #4096, where a new ShapedLikeNDArray mixin class is created, which is used in Time and in the various coordinate representations. To also use it in frames and coordinates will require that the frame and coordinate attributes are properly broadcast at creating time.

OLD TEXT FOR REFERENCE
This is a first cut at #4096, where following @embray's suggestion the shape-changing methods from Time are split off to a new ShapedLikeNDArray mixin class and used in Coordinates. With it,

In [1]: from astropy.coordinates import SkyCoord; import astropy.units as u, numpy as np

In [2]: sc = SkyCoord(ra=np.arange(9)*u.deg, dec=np.arange(10, 19)*u.deg)

In [3]: sc.reshape(3,3).transpose()
Out[3]: 
<SkyCoord (ICRS): (ra, dec) in deg
    [[(0.0, 10.0), (3.0, 13.0), (6.0, 16.0)],
     [(1.0, 11.0), (4.0, 14.0), (7.0, 17.0)],
     [(2.0, 12.0), (5.0, 15.0), (8.0, 18.0)]]>

Comments very welcome (as are PRs with test cases; I'd prefer not to have to write them myself; was boring enough for Time!). cc: @taldcroft, @eteq.

Note that there are no test cases yet, though it seems all regular tests pass (which means that __getitem__ is correctly done). A̶l̶s̶o̶,̶ ̶a̶ ̶p̶r̶o̶b̶l̶e̶m̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶̶s̶c̶.̶T̶̶ ̶d̶o̶e̶s̶ ̶n̶o̶t̶ ̶w̶o̶r̶k̶ (EDIT: this is solved by the last commit)

@mhvk mhvk mentioned this pull request Sep 3, 2015
@mhvk

mhvk commented Sep 4, 2015

Copy link
Copy Markdown
Contributor Author

Note that the sphinx build fails, since somehow the references to Time.take, etc., cannot be found any more now they come from the mixin class. Anybody know how to solve this?

EDIT: Oddly, I cannot reproduce this locally.

Comment thread astropy/utils/misc.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you can make _replicate an abstractmethod?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I thought about making it an ABC as well. It has the advantage that I could also define ndim and shape as abstract properties, since these are needed for some of the methods.

@embray

embray commented Sep 11, 2015

Copy link
Copy Markdown
Member

As for the docs build, I think there's a setting somewhere you can tweak for the API docs for whether or not class methods should be inherited. I need to check exactly where you change that...

@mhvk

mhvk commented Sep 21, 2015

Copy link
Copy Markdown
Contributor Author

@embray - I tried changing ShapedLikeNDArray to an abstract base class, as this seemed much more logical, but run into problems with coordinates.representation.BaseRepresentation already using a metaclass which is based on type (while mine is based on object). It seems difficult to have two metaclasses which are not subclasses of the same class. Any suggestions?

@embray

embray commented Sep 21, 2015

Copy link
Copy Markdown
Member

You'll have to show me what you mean.

@mhvk

mhvk commented Sep 21, 2015

Copy link
Copy Markdown
Contributor Author

Fair enough -- see the branch at https://github.com/astropy/astropy/compare/master...mhvk:shape-changing-mixin-abc?expand=1

With that,

In [1]: from astropy.coordinates import SkyCoord
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-a44fc1d186bf> in <module>()
----> 1 from astropy.coordinates import SkyCoord

/data/mhvk/venv/astropy-dev/lib/python3.4/site-packages/astropy-1.1.dev13343-py3.4-linux-x86_64.egg/astropy/coordinates/__init__.py in <module>()
     11 from .errors import *
     12 from .angles import *
---> 13 from .baseframe import *
     14 from .distances import *
     15 from .earth import *

/data/mhvk/venv/astropy-dev/lib/python3.4/site-packages/astropy-1.1.dev13343-py3.4-linux-x86_64.egg/astropy/coordinates/baseframe.py in <module>()
     26                      OrderedDescriptorContainer)
     27 from .transformations import TransformGraph
---> 28 from .representation import (BaseRepresentation, CartesianRepresentation,
     29                              SphericalRepresentation,
     30                              UnitSphericalRepresentation,

/data/mhvk/venv/astropy-dev/lib/python3.4/site-packages/astropy-1.1.dev13343-py3.4-linux-x86_64.egg/astropy/coordinates/representation.py in <module>()
     58 
     59 @six.add_metaclass(MetaBaseRepresentation)
---> 60 class BaseRepresentation(ShapedLikeNDArray):
     61     """
     62     Base Representation object, for representing a point in a 3D coordinate

/data/mhvk/venv/astropy-dev/lib/python3.4/site-packages/astropy-1.1.dev13343-py3.4-linux-x86_64.egg/astropy/extern/bundled/six.py in wrapper(cls)
    726             for slots_var in slots:
    727                 orig_vars.pop(slots_var)
--> 728         return metaclass(cls.__name__, cls.__bases__, orig_vars)
    729     return wrapper
    730 

TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

@embray

embray commented Sep 22, 2015

Copy link
Copy Markdown
Member

Oh I see what you're saying. For that to work you have to make MetaBaseRepresentation a subclass of abc.ABCMeta. But yeah, it may not be worth the trouble right now.

@mhvk

mhvk commented Sep 22, 2015

Copy link
Copy Markdown
Contributor Author

@embray - After a few more trials, it seems it is indeed as simple as making MetaBaseRepresentation be a subclass of abc.ABCMeta -- and the same for OrderedDescriptorContainer. I've now put it here; have a look if you think it makes sense to do it this way.

@embray

embray commented Sep 22, 2015

Copy link
Copy Markdown
Member

Yeah, the problem with that now is that everything that uses OrderedDescriptorContainer now also becomes an ABC. Which I think is probably mostly harmless, but it is odd. This is actually an acknowledged problem with Python metaclasses, and there are a few PEPs in the work to add capabilities to Python classes so that some of the most common problems solved by metaclasses can be accomplished also without them.

But I'm fine with seeing how that goes for now.

@mhvk

mhvk commented Sep 24, 2015

Copy link
Copy Markdown
Contributor Author

@embray - I moved the abc.ABCMeta inheritance from OrderedDescriptorContainer to where it is actually needed. I also tried to solve the issues with sphinx no longer finding the links to Time.take, etc., but couldn't. It is hard to experiment, since locally it actually works fine. If you have any suggestion, let me know...

@embray

embray commented Sep 28, 2015

Copy link
Copy Markdown
Member

Okay, sounds good. Odd about the docs. I'll see if I can reproduce that.

@mhvk

mhvk commented Sep 29, 2015

Copy link
Copy Markdown
Contributor Author

@eteq, @taldcroft, @astrofrog - I think enabling multidimensional coordinates and allowing their shape to change would be nice to get in for 1.1, but I don't think I'll get around to adding test cases. Any of you have a change? Obviously, one can use those in Time as a basis, to test that views are taken when they are supposed to, etc.

@embray - any luck with figuring out the reason the docs don't link correctly?

@taldcroft

Copy link
Copy Markdown
Member

@mhvk - sorry, I'm way behind just trying to get all my own PRs and assigned PRs wrapped up.

@eteq

eteq commented Oct 2, 2015

Copy link
Copy Markdown
Member

@mhvk - I see your point and can try to give it a shoot (especially given its been done for some time...).

Just so we're clear, it's basically time/tests/test_methods.py that you're thinking about here, right?

@mhvk

mhvk commented Oct 2, 2015

Copy link
Copy Markdown
Contributor Author

@eteq - yes, it is. Not sure one would need to be as thorough as that one, given that the same code is used, but I think it is especially important to check that view and copies are done appropriately (e.g., for the representations, I needed to add copy=False in _replicate, but I may have missed similar instructions elsewhere, not knowing coordinates all that well).

@eteq

eteq commented Oct 5, 2015

Copy link
Copy Markdown
Member

sadly, I haven't had time to get tests in on this, and it appears to now need a rebase. So I think we're going to have to triage it to 1.2 😢

@mhvk

mhvk commented Oct 5, 2015

Copy link
Copy Markdown
Contributor Author

Yikes, that merge conflict made me realise a change introduced by #3915 that I think is rather in the wrong place (#4222...). Anyway, for now just kept it. Hopefully things still work.

@embray
embray force-pushed the master branch 2 times, most recently from 63540e0 to 34f4c29 Compare October 5, 2015 21:18
@alexrudy

Copy link
Copy Markdown
Contributor

A minor comment: With numpy arrays, it is possible to set the shape attribute to change the shape if that wouldn't change the memory layout of the underlying array (see http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html). Perhaps that isn't a guarantee we want to keep, but I use somearray.shape = (10,10) all the time, and it would be nice if astropy array-like objects allowed that as well.

@mhvk

mhvk commented Oct 13, 2015

Copy link
Copy Markdown
Contributor Author

@alexrudy - I did not yet implement shape-setting for coordinates, as it was not something that was as easy to do with the mixin class proposed here as the ndarray methods, which all rely on a _replicate method that always returns a new instance. The latter ensures also that coordinate instances remain strictly immutable. It would be quite easy, though, to add shape setting, and personally I think it is a good idea (indeed, for Time I had already done so). @eteq - what do you think? It does keep objects immutable in the sense that perhaps matters most, that their data is not changed at all.

@mhvk mhvk changed the title WIP Shape changing mixin Shape changing mixin Nov 2, 2015
Comment thread astropy/utils/misc.py Outdated
return self._replicate('squeeze', *args, **kwargs)

def take(self, indices, axis=None, mode='raise'):
"""Return a new instance formed from the elements the given indices.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'from the elements at the given indices'

@StuartLittlefair

StuartLittlefair commented Jul 13, 2016 •

Copy link
Copy Markdown
Contributor

Hi @mhvk. I think this looks very nice, though it took me a while to wrap my head around the metaclass magic...

A couple of extra comments:

  1. You still define ndim, size and isscalar explicitly for Time objects. That's not necessary any more, is it?
  2. Reshaping a Time object to ndim greater than 1 breaks light_travel_time, with the following error:
...
/usr/local/lib/python3.5/site-packages/astropy/coordinates/builtin_frames/icrs_cirs_transforms.py in gcrs_to_icrs(gcrs_coo, icrs_frame)
    171     astrom = erfa.apcs13(jd1, jd2, pv)
    172 
--> 173     i_ra, i_dec = aticq(gcrs_ra, gcrs_dec, astrom)
    174 
    175     if gcrs_coo.data.get_name() == 'unitspherical' or gcrs_coo.data.to_cartesian().x.unit == u.one:

/usr/local/lib/python3.5/site-packages/astropy/coordinates/builtin_frames/utils.py in aticq(ri, di, astrom)
    184 
    185     # Bias-precession-nutation, giving GCRS proper direction.
--> 186     ppr = erfa.trxp(astrom['bpn'], pos)
    187 
    188     # Aberration, giving GCRS natural direction

/usr/local/lib/python3.5/site-packages/astropy/_erfa/core.py in trxp(r, p)
  24006 
  24007     #Create the output array, based on the broadcasted shape, adding the generated dimensions if needed
> 24008     broadcast = numpy.broadcast(numpy.int32(0.0), numpy.int32(0.0), r_in[...,0,0], p_in[...,0])
  24009     trp_out = numpy.empty(broadcast.shape + (3,), dtype=numpy.double)
  24010 

ValueError: shape mismatch: objects cannot be broadcast to a single shape

It might be best to merge this PR, and create an issue for that problem to be fixed in a separate PR, especially as this doesn't work in the master branch either.

@mhvk
mhvk force-pushed the shape-changing-mixin branch from fdfef92 to 15106c1 Compare July 21, 2016 15:51
@mhvk

mhvk commented Jul 21, 2016

Copy link
Copy Markdown
Contributor Author

@StuartLittlefair - indeed, it is not necessary to redefine ndim, etc., any more. I had kept them for speed, but on second thought, I think it makes more sense to just use the defaults.

On the ndim > 1 issue for Time: I would think this is already broken in current master too, correct? If so, could you raise a new issue? If not, then I think I should fix it here.

@StuartLittlefair

Copy link
Copy Markdown
Contributor

@mhvk - yes it's currently broken in master too. I'll raise a new issue now...

@mhvk
mhvk force-pushed the shape-changing-mixin branch from 91fa8c1 to caa6f93 Compare July 22, 2016 10:49
@mhvk

mhvk commented Jul 22, 2016

Copy link
Copy Markdown
Contributor Author

@StuartLittlefair - OK, I removed the unnecessary redefinitions of ndim, size and isscalar, and also added a shape setter (plus made that for Time safer).

I liked your simple _apply alternative to _replicate, so use that throughout now.

It should be ready to go in now!

@mhvk

mhvk commented Jul 23, 2016

Copy link
Copy Markdown
Contributor Author

@eteq, @taldcroft, @StuartLittlefair - any last comments on the shape-changing PR?

@StuartLittlefair

Copy link
Copy Markdown
Contributor

Nothing more to add from me @mhvk!

@mhvk

mhvk commented Jul 27, 2016

Copy link
Copy Markdown
Contributor Author

I'm merging this, since it is holding up further work on shape-awareness and broadcasting in coordinates (and therewith fixing, e.g., the bug with light travel times for multidimensional time instances -- #5191).

p.s. the merging travis build will likely fail due to #5198.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants