Skip to content

BUG: Fix leap second table download triggered on import - #10125

Closed
pllim wants to merge 4 commits into
astropy:masterfrom
pllim:jd-no-download
Closed

pllim wants to merge 4 commits into
astropy:masterfrom
pllim:jd-no-download

Conversation

@pllim

@pllim pllim commented Apr 8, 2020

Copy link
Copy Markdown
Member

Description

This pull request is to address the problem of leap-second table being downloaded on module import, causing caching problems in cluster computing sharing the same cache.

Disclaimer: I did not benchmark the performance hit. Is performance even an issue we need to consider for this patch?

Address the problem reported in #9479 (comment)

@pllim pllim added this to the v4.0.2 milestone Apr 8, 2020
@pllim
pllim requested review from adrn, eteq, mhvk and weaverba137 April 8, 2020 15:07
@pllim pllim mentioned this pull request Apr 8, 2020
4 tasks

@weaverba137 weaverba137 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Are those module-level variables used elsewhere? Or do they just need to be global variables in that one module?

@pllim

pllim commented Apr 8, 2020

Copy link
Copy Markdown
Member Author

Only used in that module.

@mhvk mhvk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really all that is needed? If so, that's very nicely found!

Inline I suggest an even simpler solution: just insert the numerical values of the jd... (low-tech but fast and reliable!)

Comment thread CHANGES.rst Outdated
@mhvk

mhvk commented Apr 8, 2020

Copy link
Copy Markdown
Contributor

If this works, I think it may be possible to do the same in time.formats, so that even importing the time module wouldn't download the LeapSecond table immediately...

@pllim

pllim commented Apr 8, 2020

Copy link
Copy Markdown
Member Author

Re: #9479 (comment) and the comments above -- Would be nice if @weaverba137 can take this PR for a spin and see where it crashes next.

I'd rather we fix a real use case than go hunting for all Time() calls.

@weaverba137

Copy link
Copy Markdown
Member

I will try to run a test when I can. I'm pretty slammed today.

@pllim

pllim commented Apr 8, 2020

Copy link
Copy Markdown
Member Author

These might be in scope too:

astropy/coordinates/builtin_frames/lsr.py:J2000 = Time('J2000')
astropy/coordinates/builtin_frames/utils.py:EQUINOX_J2000 = Time('J2000', scale='tt')
astropy/coordinates/builtin_frames/utils.py:EQUINOX_B1950 = Time('B1950', scale='tt')
astropy/coordinates/builtin_frames/utils.py:DEFAULT_OBSTIME = Time('J2000', scale='tt'

@mhvk

mhvk commented Apr 8, 2020

Copy link
Copy Markdown
Contributor

Of course, those are a lot trickier, since an actual time instance is wanted. I'm starting to think the leap second loading itself should be moved away from __new__ to where it might actually get used (I don't quite recall why I didn't start with that...)

@pllim

pllim commented Apr 8, 2020

Copy link
Copy Markdown
Member Author

The one in LSR has a vague "for speed" comment but it is not used in the module. Maybe @adrn can explain why it was added in #6219 and whether it is still relevant.

# For speed
J2000 = Time('J2000')

@pllim

pllim commented Apr 8, 2020

Copy link
Copy Markdown
Member Author

Re: #10125 (comment)

Ah, @mhvk , if you can nip it in the bud, then maybe this PR won't be needed. I'll put this on hold.

@adrn

adrn commented Apr 8, 2020

Copy link
Copy Markdown
Member

The one in LSR has a vague "for speed" comment but it is not used in the module. Maybe @adrn can explain why it was added in #6219 and whether it is still relevant.

I don't remember. Try removing and see if tests still pass? 🤷‍♂

@mhvk

mhvk commented Apr 8, 2020

Copy link
Copy Markdown
Contributor

@pllim - not sure that I'll succeed - I don't remember much about my trials, but do remember it was an enormous hassle to avoid circular dependencies...

@bsipocz

bsipocz commented Apr 9, 2020 •

Copy link
Copy Markdown
Member

Ah, @mhvk , if you can nip it in the bud, then maybe this PR won't be needed. I'll put this on hold.

What about going gradually, and get this in now and instead of holding up the PR, just revert if/when the other approach works out and gets in?

@mhvk

mhvk commented Apr 9, 2020

Copy link
Copy Markdown
Contributor

I'm fine with replacing the jds here with actual values (no need for the lazy loading), but think it is unlikely it actually solves the problem, so the changelog entry may need to be adjusted.

@pllim

pllim commented Apr 9, 2020

Copy link
Copy Markdown
Member Author

replacing the jds here with actual values

That works for me. Will the values ever change due to leap seconds? Maybe we can even include it in constants and be done with it. I see a few flavors of Time(epoch) calls across the package.

@pllim

pllim commented Apr 9, 2020 •

Copy link
Copy Markdown
Member Author

It is too bad we cannot use PEP 562 yet. The ones in builtin_frames/utils.py are tricky.

Also, I don't think this is worth it as it would be hard to maintain going forward: https://www.themetabytes.com/2018/06/19/lazy-loading-modules-in-python/

@pllim

pllim commented Apr 9, 2020 •

Copy link
Copy Markdown
Member Author

This is looking pretty hacky... 😬 But at least tests passed with this patch for me, on Windows no less.

@mhvk

mhvk commented Apr 9, 2020

Copy link
Copy Markdown
Contributor

Great, now at least it is clear where the problems lie. Will try to find some time to see if I can instead move the download further down.

@taldcroft

Copy link
Copy Markdown
Member

Probably the coffee still isn't working, but I'm confused about this approach and I don't see how this is preventing making new Time objects at import time, which if I understand can trigger a download. I made a file lazy.py:

def func(arg):
    print(f'func called: {arg}!')
    return arg

class TimeAttribute:
    def __init__(self, arg):
        print(f'In TimeAttribute: {arg}')

class Blah:
    attr = TimeAttribute(arg=func('first'))

    def meth(arg=func('second')):
        pass

Then

>>> import lazy
func called: first!
In TimeAttribute: first
func called: second!

@pllim

pllim commented Apr 10, 2020 •

Copy link
Copy Markdown
Member Author

@taldcroft , I didn't know Python actually runs the nested code on import. 🤔 I'll have to double check.

UPDATE: You are right! 😱 I gotta think about this some more.

@pllim

pllim commented Apr 10, 2020 •

Copy link
Copy Markdown
Member Author

I think the trick is to make them class properties. If I changed @taldcroft 's lazy.py to the following:

class Blah:
    @property
    def attr(self):
        return TimeAttribute(arg=func('first'))

    def meth(arg=func('second')):
        pass

I only get:

>>> import lazy
func called: second!

But this doesn't solve the module level caching.

@taldcroft

Copy link
Copy Markdown
Member

I do hope that @mhvk can find a lower-level solution. It does not seem like there should be any requirement to access the latest leap seconds just to create a Time object. It's only for converting.

A lower-level solution also helps other projects that may be creating Time objects on import. It seems that importing any part of astropy should never trigger or rely on network access.

@pllim

pllim commented Apr 10, 2020

Copy link
Copy Markdown
Member Author

Well, I am stuck anyway. I went down the rabbit hole of frame attributes. Changing them to class property broke everything. There is some magic going on with class level attributes and there is no easy way to lazy load them. 😢

@pllim

pllim commented Apr 10, 2020

Copy link
Copy Markdown
Member Author

I am closing this PR. I think this is a dead end. Sorry for the noise!

@pllim pllim closed this Apr 10, 2020
@pllim
pllim deleted the jd-no-download branch April 10, 2020 17:24
@pllim pllim removed this from the v4.0.2 milestone Apr 10, 2020
@mhvk

mhvk commented Apr 10, 2020

Copy link
Copy Markdown
Contributor

@pllim - thanks for the valiant attempt! I agree that there's a need for a lower-level solution, and it should be do-able (though it is not like I didn't try initially; am hoping that a fresh look will help...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants