Skip to content

Improve AltAz to RaDec accuracy tests - #3296

Merged
eteq merged 2 commits into
astropy:masterfrom
cdeil:radec-altaz-tests
Jan 26, 2015
Merged

eteq merged 2 commits into
astropy:masterfrom
cdeil:radec-altaz-tests

Conversation

@cdeil

@cdeil cdeil commented Jan 14, 2015

Copy link
Copy Markdown
Member

Implements #3244

cc @astrofrog @eteq

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyone has an idea what the origin of the 31 arcsec difference is?
I'm not sure what the humidity and wavelength is for the PyEphem result ... that might be it.
And this is a rather extreme example at 30 km.

I'll now try and fix the other examples.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cdeil - can you make a plot of the offset for many different inputs to see if it is always in the same direction, or always around 31 arcsec? I.e. is it systematic?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll do that over in coordinates-benchmarks ...

@embray embray added coordinates Affects-dev PRs and issues that do not impact an existing Astropy release labels Jan 14, 2015
@astrofrog

Copy link
Copy Markdown
Member

@cdeil - is this good to merge for now, and we can update later if we can get a better accuracy?

@cdeil

cdeil commented Jan 15, 2015

Copy link
Copy Markdown
Member Author

No, there's still a bunch of skipped tests that I want to fix ... I'll finish this today, OK?

@astrofrog

Copy link
Copy Markdown
Member

@cdeil - sounds good!

@cdeil
cdeil force-pushed the radec-altaz-tests branch from 320ef64 to ed3378e Compare January 18, 2015 14:05
@cdeil

cdeil commented Jan 18, 2015

Copy link
Copy Markdown
Member Author

@astrofrog - I'm afraid understanding the discrepancies here and the work on coordinates-benchmark are taking longer than expected (see astropy/coordinates-benchmark#42).

There's one test here, test_against_pyephem, which agrees with ephem at the 3 milli-arcsec level!
Because of this I strongly suspect that the astropy.coordinates implementation is OK and the large discrepancies in the other tests are simple mistakes in input or output values.

If someone else has time to try to try to find these errors and debug these tests, great!

But if not, I think it would be OK to merge this as-is and continue with the precision checks over in coordinate-benchmarks in the coming months ...

@cdeil cdeil changed the title WIP: Fix RaDec to AltAz coordinate conversion tests Improve AltAz to RaDec accuracy tests Jan 18, 2015
@cdeil

cdeil commented Jan 18, 2015

Copy link
Copy Markdown
Member Author

I looked at the wrong assert, the difference against the ephem reference results is still 31 arcsec in test_against_pyephem ... still, IMO it's acceptable to merge this as-is, release 1.0 and continue the investigation later.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the example .pro file, it is 10 PM local time, while you enter a UT. It should be '2041-12-26 06:00:00'.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I need to use '2041-12-26 05:00:00' to get the right answer...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for finding this bug ... fixed.

@mhvk

mhvk commented Jan 18, 2015

Copy link
Copy Markdown
Contributor

@cdeil - it seems I at least found the reason for the egregious difference with the idl routine. But I find it rather worrying that the other results are so far off -- this kind of stuff should be good to an arcsec or so...

@astrofrog

Copy link
Copy Markdown
Member

Is it going to be something related to leap seconds? If the 30 arcsecond shift is always systematic it could be a 2s difference in the time (easily done with a couple of leap seconds). I think it's important to figure out if the differences are systematic or random for this reason.

@astrofrog astrofrog added this to the v1.0.0 milestone Jan 21, 2015
@cdeil
cdeil force-pushed the radec-altaz-tests branch from ed3378e to 5e5514c Compare January 21, 2015 15:33
@cdeil

cdeil commented Jan 21, 2015

Copy link
Copy Markdown
Member Author

I've updated this PR with the fixes pointed out by @mhvk .
The difference with the IDL routine is now 2.6285 arcsec for unknown reasons.
I tried adding and subtracting a few seconds from the observation time and the difference is always larger ... that's not it.

I'll try to find time tomorrow to add some more extensive tests to
https://github.com/astropy/coordinates-benchmark tomorrow, but I'm very short on time this week, so anyone interested in this ... pull requesests welcome!

@astrofrog

Copy link
Copy Markdown
Member

@cdeil is the 2.6285" error systematic or just the largest error you found?

@cdeil

cdeil commented Jan 21, 2015

Copy link
Copy Markdown
Member Author

@astrofrog - I have not done any systematic testing ... I will start this in coordinates-benchmark tomorrow. 2.6 arcsec is the difference I get for this one example: https://github.com/astropy/astropy/pull/3296/files#diff-3f09c706ec1d34b00951a929e4c0e893R57

@astrofrog

Copy link
Copy Markdown
Member

@cdeil - would you like us to merge this now, or do you want to wait for the systematic testing? This is already a lot better than what's in the core, but if you think it could get better in the next few days I can hold off from merging.

@cdeil

cdeil commented Jan 21, 2015

Copy link
Copy Markdown
Member Author

Might as well wait a few days then in case I or someone else finds more issues / improved understanding with these numbers ... of course don't let this hold up the 1.0 branching. :-)

@eteq

eteq commented Jan 22, 2015

Copy link
Copy Markdown
Member

prompted by @astrofrog's comment, I went and tracked down where pyephem gets its leap seconds from: it appears to be a hard-coded table that ends in 2011... So anything after that might by wrong in pyephem.

@eteq

eteq commented Jan 26, 2015

Copy link
Copy Markdown
Member

An update on some of this: as an independent check I've written my own tests to compare against pyephem, pytpm, ho2eq/eq2hor, and a c routine that uses erfa (which really ought to be the same because most of astropy uses that directly or indirectly).

After much struggling to try to convince them all to not do atmospheric refraction (because the models there are all quite different), I'm now getting them to match astropy to ~2 arcsec (with the exception of eq2hor ... but it doesn't even match hor2eq, so I'm not so worried that it's the only outlier...).

What I've done all requires external packages, though, of course (along the lines of the benchmark repo, but much hackier). So probably it shouldn't be added here. But it gives me much more confidence that the situation is pretty much ok.

@cdeil

cdeil commented Jan 26, 2015

Copy link
Copy Markdown
Member Author

@eteq - Would you be OK to merge this now?

I think it's an improvement over what's in master now and as far as we'll get for 1.0 ... if there's any comments that are quick to address I can do that today.

And would you mind sharing your test scripts somewhere on Github and make a new issue in coordinates-benchmark that they should be incorporated?
I'm sure it's easier to start implementing some transformation for a new tool from a hacky, but working script than to start from scratch ... the hard part is always finding the right functions and parameters.

@eteq

eteq commented Jan 26, 2015

Copy link
Copy Markdown
Member

@cdeil - yeah, I will give it one last look and then merge it.

The tests I did are in https://github.com/eteq/astropy_astrometry_tests - that includes data files so all the comparisons should be possible even if you don't have IDL or want to bother with compiling against ERFA.

Also, I found a newer version of hor2eq, and with that it is now both consistent with eq2hor and within 2" of astropy. So I think we're all set!

@eteq

eteq commented Jan 26, 2015

Copy link
Copy Markdown
Member

I'll go ahead and merge this, but I'm going to issue another PR that builds on it some. It's not necessarily RC critical, but will be good to have in before the actual release.

eteq added a commit that referenced this pull request Jan 26, 2015
Improve AltAz to RaDec accuracy tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Affects-dev PRs and issues that do not impact an existing Astropy release coordinates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants