One failing io.fits test on Numpy 1.10.1 - #4266

I believe that Numpy 1.10.1 also forces specutils to fail
one of its tests. I suspect the two are related.

And, it does affect me so I suspect this is a significant issue.
Have moved back to numpy 1.9.3

@profxj @embray - I've been noticing chatter on the numpy and other mailing lists about some serious 1.10.0 and 1.10.1 bugs. So it might be that 1.10.2 will just fix all of this? Might be worth waiting to see before investing too much time into track down obscure issues.

We'll see. I'm not entirely sure it's a bug in Numpy--I forgot a lot of stuff over vacation.

This is continuing to cause problems, especially among those who
use conda where numpy defaults to 1.10 (sensibly, I suppose).

And I expect that specutils is still failing its tests.

@profxj What specifically is causing problems? This specific issue is about one test that was failing, but is there some other failure you can point to? Astropy v1.0.6 should have resolved all common issues with Numpy 1.10.

@profxj In other words, please post some specific code that's failing.

I see now that it is a bug in specutils (I think) via numpy
and not astropy io.fits

Here is the trouble in specutils:
astropy/specutils#123

as evidence by this simple snippet.

In [1]: import specutils

In [2]: specutils.__version__
Out[2]: u'0.2.dev0591'

In [3]: wave = np.arange(3000,5000)*u.AA

In [4]: flux = np.ones(wave.size)

In [5]: spec = specutils.Spectrum1D.from_array(wave,flux)

In [6]: spec.dispersion
Out[6]: <Quantity [   nan, 3001., 3002.,...,  4997., 4998., 4999.] Angstrom>

@profxj That looks like an issue to raise in specutils.

Had to rebase.

@embray - don't feel like a FITS expert by any means, but I think this all looks good.

The issue here was that when writing a table out to a FITS file, depending on where the table data came from in the first place any number of the columns may have to be byte-swapped individually. This also means assigning a new dtype to the data with the relevant columns byte-swapped.

Unfortunately, teasing apart a multi-field dtype and reassembling it correctly can be something of a pain, since there are a few applications in FITS where fields need to be able to have arbitrary offsets, and Numpy makes it less easy than it should be to create a new multi-field dtype from an existing one with the correct offsets maintained as well. The code fixed in this PR was not careful to preserve the offsets in a case where they mattered. Numpy < 1.10 didn't care, but Numpy 1.10 was somehow catching the inconsistency (I forget exactly what the error was though...)

@embray - OK, thanks for the explanation. Nice to be fully OK with 1.10 now! (though one needs 1.10.2 to avoid some of the numpy slowdowns).