Significant increase in import time for several sub-packages #9555

Activity

1. astrofrog

   added

   Performance

   on Nov 7, 2019
2. pllim

   added

   nddata

   coordinates

   utils.iers

   on Nov 7, 2019
3. pllim commented on Nov 7, 2019

   

   pllim

   on Nov 7, 2019

   Member

   More actions

   For coordinates, one should also review whether that same PR sped up other stuff or not. If it sped up some calculations significantly, import time increase is a small price to pay.

   Unfortunately, no benchmark is available for nddata computations.
4. mhvk commented on Nov 7, 2019

   

   mhvk

   on Nov 7, 2019

   Contributor

   More actions

   @astrofrog - not completely sure, but the problem with the auto-update of the leap seconds is that the moment you use any time (like setting the J2000 epoch in the coordinate frames), it will check whether the leap second table is up to date, which means importing utils.iers and therefore table and thus all of io. I didn't like this much but saw no easy alternative. Options:
   0. Not worry; generally people will use tables/io at some point anyway.

   1. Only check for leap seconds when one gets a time beyond the last leap second built into erfa. I thought about this a bit, but concluded it was quite invasive within Time (obviously, I may be wrong).
   2. Read in the leap seconds with np.loadtxt or so. But then, we have this nice table interface and I think it is logical to expect leap seconds to be presented as a Table (and it to be part of utils.iers).
   3. Avoid importing all of io in table (right now, done to ensure the lists of readers and docstrings are complete).

   Personally, I prefer (3), but this may not be trivial either; @taldcroft and @MSeifert04 did quite a nice job to make the reader registration and docstring autogeneration work, but perhaps it can somehow be done more lazily.
5. MSeifert04 commented on Nov 7, 2019

   

   MSeifert04

   on Nov 7, 2019

   Last edited by MSeifert04

   Contributor

   More actions

   I haven't had time to look but what in nddata requires time? Is it the from astropy.coordinates import SkyCoord in utils? Or the units import?
6. mhvk commented on Nov 7, 2019

   

   mhvk

   on Nov 7, 2019

   Contributor

   More actions

   coordinates does need Time (for the epoch/equinox definitions), units does not.
7. astrofrog commented on Nov 7, 2019

   

   astrofrog

   on Nov 7, 2019

   MemberAuthor

   More actions

   Regarding:

   Avoid importing all of io in table (right now, done to ensure the lists of readers and docstrings are complete).

   it just occurred to me that one way around this would be to actually move all the connect.py files (which is where we defines readers/writers) to the astropy.table module and make sure that the imports to various io modules then happens inside the readers/writers as needed. I'm not sure why I didn't think of that before, but that could speed things up a lot?
8. mhvk commented on Nov 7, 2019

   

   mhvk

   on Nov 7, 2019

   Contributor

   More actions

   That sounds very reasonable, though probably good to check the timing (IIRC that is quite easy with python 3.7), just to be sure it is in fact io that is the part that adds most time, not table itself (a quick check with python3 -X importtime -c 'import astropy.nddata' does suggest this is the case).

   For nddata, it may also not be crazy to import coordinates locally (though, then again, if one almost always ends up using it anyway, one might as well get it upfront).