Update IERS B - #4933

@taldcroft - this is very weird: astropy.utils.iers.tests.test_iers.TestIERS_AExcerpt.test_simple fails just opening the IERS_A_EXCERPT. I now tried locally and see it there too. I suspect it has something to do with IERSAuto, so thought I would alert you first, in case you have a simple idea. (If not, I'll look into it in more detail tomorrow; have to run now...).

I can't reproduce that:

astropy/utils/iers/tests/test_iers.py::TestBasic::test_simple PASSED
astropy/utils/iers/tests/test_iers.py::TestBasic::test_open_filename PASSED
astropy/utils/iers/tests/test_iers.py::TestBasic::test_open_network_url PASSED
astropy/utils/iers/tests/test_iers.py::TestIERS_AExcerpt::test_simple PASSED
astropy/utils/iers/tests/test_iers.py::TestIERS_A::test_simple SKIPPED
astropy/utils/iers/tests/test_iers.py::TestIERS_Auto::test_simple SKIPPED

@mhvk - I'm a little confused about the justification, namely allowing "simple" tests to run without remote data. Which specific tests are you talking about?

@taldcroft - sorry, the commit message made more sense as part of #4920 (where I started it, but then took it out): it calculates some ephemeris for a 2014 date, which causes IERS A to be downloaded since IERS B shipped along with astropy is so far out of date.

But what I don't understand is how updating IERS B could have affected this. Unless it is that now the excerpt that is used in test_simple overlaps in time with the new IERS B, and this somehow causes IERS_A.open() to do something funny.

But even stranger that you cannot reproduce this...

@mhvk - I am able to reproduce now. I'll look into this.

@mhvk - the problem is because the IERS-A excerpt is entirely contained within the new IERS-B file. I fixed the broadcasting exception but other tests are failing because with the new logic of patching in IERS-B values where available, now the entire IERS-A excerpt is coming from IERS-B.

I think I can update the logic in _combine_a_b_columns to only change existing IERS-B values in the current table instead of introducing new ones as well. In realistic usage this should not change anything since a downloaded IERS-A finals file will always have newer "B" values than what ships with astropy. That is the case that was already tested alot.

I should have a PR to your branch soon.

Ah, I figured it was something like that. But still don't quite understand it: I thought IERS_A itself would do no patching whatsoever, but rather that that was all done in IERSAuto. I do like the idea that one can use an IERS_A table as is. If it is difficult to do in IERSAuto, then I'd also be fine with an option in IERS_A that turns patching on and off (default off, but set to on by IERSAuto?).

I now see you already have a commit that fixes the error, so I'll go ahead and merge that so it can be tested and this can go into master. I can then add a substitute_iers_b argument or so in a separate PR.

Yes, you could factor out this bit into a separate private method _substitute_iers_b which is pass for IERS_A and the real thing below for IERS_Auto. I think that would be a good thing, in keeping with IERS_A just doing what you asked for and IERS_Auto doing stuff under the hood to make everything work optimally.

        # IERS-A has IERS-B values included, but for reasons unknown these do not
        # match the latest IERS-B values.  See comments in #4436 for details.  So
        # here we use the bundled astropy IERS-B table to overwrite the values
        # in the downloaded IERS-A table.
        iers_b = IERS_B.open()
        # Substitute IERS-B values for existing B values in IERS-A table
        mjd_b = table['MJD'][~table['UT1_UTC_B'].mask]
        i0 = np.searchsorted(iers_b['MJD'].value, mjd_b[0], side='left')
        i1 = np.searchsorted(iers_b['MJD'].value, mjd_b[-1], side='right')
        iers_b = iers_b[i0:i1]
        n_iers_b = len(iers_b)
        # If there is overlap then replace IERS-A values from available IERS-B
        if n_iers_b > 0:
            # Sanity check that we are overwriting the correct values
            if not np.allclose(table['MJD'][:n_iers_b], iers_b['MJD'].value):
                raise ValueError('unexpected mismatch when copying IERS-B values into IERS-A table')
            # Finally do the overwrite
            table['UT1_UTC_B'][:n_iers_b] = iers_b['UT1_UTC'].value
            table['PM_X_B'][:n_iers_b] = iers_b['PM_x'].value
            table['PM_Y_B'][:n_iers_b] = iers_b['PM_y'].value

OK, I made the change following your suggestion. I also had to undo the changes to the test file, since you were now testing against IERS_B values...

Looks good to me assuming tests pass.

I killed the tests to make _substitute_iers_b a classmethod in both classes, and to add a note that the TestIERS_AExcerpt is also a regression test for this PR; will merge if the restarted tests pass.

backported to 1.2 in 64c6b62