test_connect.py failure on mips, powerpc, s390x, sparc

On mips, powerpc, s390x, and sparc, the Debian package build for version 0.3 fails in the testing step on python-2.7.5 with errors like:

_________________________ TestSingleTable.test_simple __________________________
>       assert equal_data(t1, t2)
E       assert equal_data(<Table rows=4 names=('a','b','c')>\narray([(1, u'a', 2.3), (2, u'b', 4.5), (3, ...6.7), (4, u'd', 8.9)], \n      dtype=[('a', '>i4'), ('b', '>U1'), ('c', '>f8')]),
                          <Table rows=4 names=('a','b','c')>\narray([(1, '\x00\x00\x00a', 2.3), (2, '\x00...'\x00\x00\x00d', 8.9)], \n      dtype=[('a', '>i4'), ('b', 'S4'), ('c', '>f8')]))

astropy/io/fits/tests/test_connect.py:38: AssertionError

The assertion errors are in:
astropy/io/fits/tests/test_connect.py, lines 38, 50, 75, 84, 97, 111, 148, 164.

Also, there are doctest failures:

___________________ [doctest] astropy.table.np_utils.hstack ____________________
423     --------
424 
425     To stack two arrays horizontally (by columns) do::
426 
427       >>> from astropy.table import np_utils
428       >>> t1 = np.array([(1, 2),
429       ...                (3, 4)], dtype=[('a', 'i4'), ('b', 'i4')])
430       >>> t2 = np.array([(5, 6),
431       ...                (7, 8)], dtype=[('c', 'i4'), ('d', 'i4')])
432       >>> np_utils.hstack([t1, t2])
Expected:
    array([(1, 2, 5, 6),
           (3, 4, 7, 8)],
          dtype=[('a', '<i4'), ('b', '<i4'), ('c', '<i4'), ('d', '<i4')])
Got:
    array([(1, 2, 5, 6), (3, 4, 7, 8)], 
          dtype=[('a', '>i4'), ('b', '>i4'), ('c', '>i4'), ('d', '>i4')])

astropy/table/np_utils.py:432: DocTestFailure

and similarly in line 320, and another here:

______________________ [doctest] astropy.table.table.Row _______________________
880     """A class to represent one row of a Table object.
881 
882     A Row object is returned when a Table object is indexed with an integer
883     or when iterating over a table::
884 
885       >>> table = Table([(1, 2), (3, 4)], names=('a', 'b'),
886       ...               dtype=('int32', 'int32'))
887       >>> row = table[1]
888       >>> row
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,3 @@
     <Row 1 of table
      values=(2, 4)
    - dtype=[('a', '<i4'), ('b', '<i4')]>
    + dtype=[('a', '>i4'), ('b', '>i4')]>

astropy/table/table.py:888: DocTestFailure

On i386, amd64, arm, ia64, kfreebsd, and hurd, the tests all pass.

Build log URLs:

The joys of also testing the documentation... I think # doctest: +SKIP would be appropriate here, unless/until one can think of an automated way to ignore the ≶ signs

p.s. Many thanks for packaging astropy for Debian! Whenever I look at installation instructions, I can't but marvel at how much better a life I have with apt-get and aptitude -- but I'm well aware this takes real effort.

This may help for the last two errors, but the first seems to be different. However, it looks like a weird endianess problem?

We already have an automated way to ignore the pipe signs, e.g. "|S9", that were removed in Numpy 1.8, so I think we just need to extend that for this case.

As for the first error, I've seen that before and I think it's a Numpy bug, fixed in recent versions. What version of Numpy are we testing against here? I might try to fire that up in a PPC VM or something to investigate further.

python-numpy 1.7.1 on all machines (litte endian as well as big endian). This is the latest version available in Debian unstable/testing. The latest version available upstream is 1.8.0.

Thanks -- I ultimately saw that in the logs (sorry). I'll get some big endian VM fired up and see what I can find there.

Partial fix for astropy#1849. Ignore the endianess markers in Numpy d…

@embray

I want to bring the first error to @embray's attention, as it appears to be astropy.io.fits related.

The test is:

   def setup_class(self):
        self.data = np.array(list(zip([1, 2, 3, 4],
                                      ['a', 'b', 'c', 'd'],
                                      [2.3, 4.5, 6.7, 8.9])),
                             dtype=[('a', int), ('b', 'U1'), ('c', float)])

    def test_simple(self, tmpdir):
        filename = str(tmpdir.join('test_simple.fits'))
        t1 = Table(self.data)
        t1.write(filename, overwrite=True)
        t2 = Table.read(filename)
        assert equal_data(t1, t2)

so, it writes out a FITS file, containing a Unicode column, and reads it in. The read-in table treats the column as string, not unicode. I think this works "by accident" on little-endian platforms, because Numpy ignores any trailing null characters in the string. On big-endian platforms, the null characters appear at the front.

So, a question to the FITS experts: does it even make sense to support these unicode columns in FITS? If not, I think we need to handle the conversion to ascii strings correctly, which probably implies a data copy to shrink the width of the column, or at least a rewrite of the string column's data.

I copied the current pytest_plugins.py to the 0.3 sources, marked the test_connect.py (as in #1855 ) issues as "xfail", and did a retry. Now the doctest problems above do not appear; however I still have another issue on Python3:

astropy/io/votable/tests/vo_test.py:204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

_python_based = False, binary_mode = 1

    def _test_regression(_python_based=False, binary_mode=1):
[...]
        if sys.byteorder == 'big':
            new_dtypes = []
            for dtype in dtypes:
                dtype = list(dtype)
>               dtype[1] = dtype[1].replace(b'<', b'>')
E               TypeError: Can't convert 'bytes' object to str implicitly

(this is actually vo_test.py, line 130)

Full log (for powerpc)

This was hidden in the old patch since we didn't even come to the python3 tests.

@olebole

@olebole: I've added a fix for the issue above to #1855.

@mdboom

I can confirm that the patch #1855 solves the endianess problems for the doctests. When I also set the mentioned tests in astropy/io/fits/tests to xfail, the compilation runs fine on big endian machines (except s390x, see #1885).

@mdboom: I couldn'd find the patch; however I just removed the "b" specifiers.

@embray

Sorry about that -- I added to the patch to an already-merged pull request, which obviously doesn't work.

This is the patch, which should be more or less equivalent to what you did. --I'll go ahead and merge it directly on-- master.

--- a/astropy/io/votable/tests/vo_test.py
+++ b/astropy/io/votable/tests/vo_test.py
@@ -127,7 +127,7 @@ def _test_regression(_python_based=False, binary_mode=1):
         new_dtypes = []
         for dtype in dtypes:
             dtype = list(dtype)
-            dtype[1] = dtype[1].replace(b'<', b'>')
+            dtype[1] = dtype[1].replace(str('<'), str('>'))
             new_dtypes.append(tuple(dtype))
         dtypes = new_dtypes
     assert table.array.dtype == dtypes

It would be nice to get to the bottom on the FITS failures, but xfail might be a good short term solution. I have no idea how deeply nested that bug is, or even if we can work around it (as I think the true location of the bug is in Numpy). @embray.

EDIT: I'll make a pull request for this instead, so it will get merged in 0.3.1.

Are any of these still an issue? #1887 is reported to have fixed some of them, but are there still failures elsewhere? Or are they just being xfailed, which is fine too?

@olebole

@olebole - are you still seeing these failures?

@sergiopasra

That does not appear anymore, and can be closed. @sergiopasra what about Fedora?

@olebole

The problem with ppc64 seems to be fixed in Fedora @olebole

@sergiopasra

@sergiopasra @olebole - thanks for confirming!

Navigation Menu

Repository navigation

test_connect.py failure on mips, powerpc, s390x, sparc #1849

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

test_connect.py failure on mips, powerpc, s390x, sparc #1849

Description

Activity

mhvk commented on Nov 30, 2013

olebole commented on Dec 1, 2013

mdboom commented on Dec 2, 2013

olebole commented on Dec 2, 2013

mdboom commented on Dec 2, 2013

mdboom commented on Dec 2, 2013

olebole commented on Dec 10, 2013

mdboom commented on Dec 10, 2013

olebole commented on Dec 11, 2013

mdboom commented on Dec 11, 2013

embray commented on Apr 30, 2014

10 remaining items

astrofrog commented on Feb 9, 2016

olebole commented on Feb 9, 2016

sergiopasra commented on Feb 12, 2016

astrofrog commented on Feb 12, 2016

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions