speed issues after install/upgrade of astropy and numpy

I recently upgraded astropy using pip, which installed version 1.0.5. It also caused an upgrade of numpy to 1.10.0. I then noticed that my use of fitsio was dramatically slower. I'm using fitsio to read a binary table HDU (single dish convention) and using astropy.fits.io (pyfits) to write out a random group FITS file. The fitsio speed issue is on a row iterator through that binary table with an iter_row_buffer size of 1000 rows [but that value isn't that important for values > 50 in previous tests I've done on this file]. The speed increased by close to 2 orders of magnitude (< 1 s to iterate through the file before to > 20 s now; that's just code involving just the fitsio row iterator - no direct use of astropy/pyfits or numpy).

I tried upgrading fitsio using pip (which brought it to 0.9.7 from 0.9.3 and also triggered a upgrade of numpy to 1.10.0.post2, not exactly the same as 1.10.0 that upgrading astropy triggered) with no change in speeds. I finally tried installing from the tar of the 0.9.8rc release candidate. No change.

Have you seen or heard of anyone else with this problem? Is this pilot error on my part? Any suggestions on what I should try next? Thanks.

I wonder if the file written by pyfits has changed in some way? Do you have an older file to check?

The file being read in the test code hasn't changed.

Do you mind posting the file somewhere for testing? If you don't want it made public I can send you my email.

I'll try and do that on Monday. I'm not in a place today where that's
easy to do. It's a fairly large file, but I can probably trim some
rows and still show the effect, so I'll try that.

Bob

On 10/9/2015 11:22 AM, Erin Sheldon wrote:

Do you mind posting the file somewhere for testing? If you don't want
it made public I can send you my email.

—
Reply to this email directly or view it on GitHub
#58 (comment)
Bug from
https://github.com/notifications/beacon/AAK_smpCnI-kNw6EloOB1WX7tqmHMNdMks5o59M-gaJpZM4GMIvD.gif

Also, I reverted fitsio, astropy, numpy (and pyfits, and reinstalled scipy although I don't think either are involved) to the versions that I had when things went faster and things went faster again.

Why don't you send me your e-mail and I'll mail it to you. It's about 22 MB.

Oh, and my e-mail is bgarwood at nrao dot edu

Thanks for the file and test script. On my laptop your test runs in 0.05s on the first run, without the file cached in memory. It runs in 0.01 seconds on the second run, with the file cached. This may be a file system related issue. What file system are you using? Are you using a distributed file system such as gpfs or lustre?

On 10/12/2015 12:00 PM, Erin Sheldon wrote:

Thanks for the file and test script.

On my laptop your test runs in 0.05s on the first run, without the
file cached
in memory. It runs in 0.01 seconds on the second run, with the file
cached.

This may be a file system related issue. What file system are you
using? Are
you using a distributed file system such as gpfs or lustre?

—
Reply to this email directly or view it on GitHub
#58 (comment)
Bug from
https://github.com/notifications/beacon/AAK_sgHtkICHj39Ay63cjdm1pRQ9V0jRks5o69CIgaJpZM4GMIvD.gif

It's on a local disk, although I access it via an NFS mount point. When
I use the local mount point the initial run is 0.18s and subsequent runs
are 0.03+-a bit.

The problem is that when I moved some pyfits code around it exposed what
I think must be a bug in pyfits and/or numpy (I was opening a FITS file
with update=True and then getting an error from a low-level numpy
routine about what looks like a missing class data field when I tried to
update something in a random groups data element). So, I did...

pip install --upgrade astropy

(because I'm getting fitsio vi astropy.io.fits now) and it upgraded
numpy before upgrading astropy (presumably because of the dependencies)

At that point, this same script on this same file was suddenly > 2
orders of magnitude slower.

After commenting out a bunch of code I was left with the conclusion that
it was fitsio that had gotten slower after that upgrade and I then wrote
the test script in that ftp area. I tried upgrading fitsio to the
latest using pip and also using the fitsio from the tar file marked as
your most recent release candidate. That didn't help.

I saw the same behavior on an equivalent python installation on my laptop.

When I reverted all of these (astropy, fitsio, numpy) back to the
versions they were at before the upgrade, the speed returned to what it
is now (and what I quote above in this e-mail). The problem is I need
that upgrade to use pyfits the way I'd like to without that causing
fitsio to run slower.

Bob

It's on a local disk, although I access it via an NFS mount point. When
I use the local mount point the initial run is 0.18s and subsequent runs
are 0.03+-a bit.

The problem is that when I moved some pyfits code around it exposed what
I think must be a bug in pyfits and/or numpy (I was opening a FITS file
with update=True and then getting an error from a low-level numpy
routine about what looks like a missing class data field when I tried to
update something in a random groups data element). So, I did...

pip install --upgrade astropy

(because I'm getting fitsio vi astropy.io.fits now) and it upgraded
numpy before upgrading astropy (presumably because of the dependencies)

Do you mean you are getting pyfits from astropy.io.fits?

When I reverted all of these (astropy, fitsio, numpy) back to the
versions they were at before the upgrade, the speed returned to what it
is now (and what I quote above in this e-mail). The problem is I need
that upgrade to use pyfits the way I'd like to without that causing
fitsio to run slower.

OK, I'm using anaconda on all my machines, which is still at numpy 1.9.

If numpy is the cause we need to do a test with that. Does anyone else
reading this have access to that version? If not, I'll try to put something
together.

Yes, pyfits from astropy, sorry.

I have a newer setup if you want me to test things...

On 10/12/15, Matthew R. Becker [email protected] wrote:

I have a newer setup if you want me to test things...

please do! I was probably going to make a new virtual env. and
install the newest python....

I don't see any obvious problems in the 1.10 release notes, but maybe
I missed something

https://github.com/numpy/numpy/blob/master/doc/release/1.10.0-notes.rst

On 10/12/15, Erin Sheldon [email protected] wrote:

On 10/12/15, Matthew R. Becker [email protected] wrote:

I have a newer setup if you want me to test things...

please do! I was probably going to make a new virtual env. and
install the newest python....

Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com

I didn't see anything either. I also didn't see anything that looked related to a fix for the bug I saw.

The other oddity was that the upgrade of astropy resulted in numpy-1.10.0 and then I did an upgrade of fitsio in the hopes it would fix it and that caused an upgrade of numpy to 1.10.0post2 (from memory, the exact string might be a bit different). There didn't seem to be versions of all 3 that made pip happy that there was nothing left to upgrade.

I may try a clean python install with just numpy and fitsio to see if the problem remains. Oh, and I think this involves python 2.7.

On October 12, 2015 8:42:56 PM EDT, Erin Sheldon [email protected] wrote:

I don't see any obvious problems in the 1.10 release notes, but maybe
I missed something

https://github.com/numpy/numpy/blob/master/doc/release/1.10.0-notes.rst

On 10/12/15, Erin Sheldon [email protected] wrote:

On 10/12/15, Matthew R. Becker [email protected] wrote:

I have a newer setup if you want me to test things...

please do! I was probably going to make a new virtual env. and
install the newest python....

Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com

Reply to this email directly or view it on GitHub:
#58 (comment)

I think this is a numpy issue. The fitsio.test.test() suite is unbelievably slow after moving to 1.10

this seems to be related to simple access of arrays with fields by name, which is all over fitsio.

We need to put together a simpler test case and submit a numpy issue.

Thanks for confirming this. Is it the extraction of the SCAN value by name inside the iterator loop that's slowing it down in my test or is it happening deeper in fitsio than that? Can I work around it here by using column numbers? I'll also refactor what I was trying to do with pyfits to see if there's an alternative that does what I need and works with the earlier version. I suppose I could also see if any of the intermediate versions of numpy and astropy fixed the bug that tripped me up without slowing fitsio down, but I suspect I won't get lucky following that path. It's almost enough to get me curious to see what it would take to add support for groups to fitsio (not quite there yet, but it's close).

Hi all,

I can also confirm the slow behavior. I have pasted below the simplest script I could think of that illustrates the issue.

#!/usr/bin/env python
import os
import time
import sys
import numpy as np

def test(N=100000,verbose=False):
    d = np.zeros(1,dtype=[('col','f8')])

    t0 = time.time()
    for i in xrange(N):
        d['col'] += i
    t0 = time.time() - t0

    if verbose:
    print 'numpy version:',np.version.version
        print 'time: %g' % t0

if __name__ == "__main__":
    if len(sys.argv) > 1:
    N = int(sys.argv[1])
    test(N=N,verbose=True)
    else:
    test(verbose=True)

Here are the times:

numpy version: 1.9.3
time: 0.262786

numpy version: 1.10.1
time: 3.57254

Yep, I see the same with your script

On 10/13/15, Matthew R. Becker [email protected] wrote:

Hi all,

I can also confirm the slow behavior. I have pasted below the simplest
script I could think of that illustrates the issue.

#!/usr/bin/env python
import os
import time
import sys
import numpy as   np

def test(N=100000,verbose=False):
    d = np.zeros(1,dtype=[('col','f8')])

    t0 = time.time()
    for i in xrange(N):
        d['col'] +=   i
    t0 = time.time() - t0

    if verbose:
  print 'numpy version:',np.version.version
        print 'time: %g' % t0

if __name__ == "__main__":
    if len(sys.argv) > 1:
  N = int(sys.argv[1])
  test(N=N,verbose=True)
    else:
  test(verbose=True)

Here are the times:

numpy version: 1.9.3
time: 0.262786

numpy version: 1.10.1
time: 3.57254

Reply to this email directly or view it on GitHub:
#58 (comment)

Erin Scott Sheldon
Brookhaven National Laboratory erin dot sheldon at gmail dot com

OK. I profiled it and the same functions are causing issues. Submitting a numpy issue now.

You can track the progress of this here

numpy/numpy#6467

Looks like numpy v1.10.2 is out and it appears everything runs normally.

I'm going to close this issue. The new numpy works fine, and most people are moving past the problematic version of numpy (e.g. anaconda)

Navigation Menu

Repository navigation

speed issues after install/upgrade of astropy and numpy #58

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

speed issues after install/upgrade of astropy and numpy #58

Description

Activity

esheldon commented on Oct 9, 2015

bgarwood commented on Oct 9, 2015

esheldon commented on Oct 9, 2015

bgarwood commented on Oct 9, 2015

bgarwood commented on Oct 9, 2015

bgarwood commented on Oct 12, 2015

bgarwood commented on Oct 12, 2015

esheldon commented on Oct 12, 2015

bgarwood commented on Oct 12, 2015

esheldon commented on Oct 12, 2015

bgarwood commented on Oct 12, 2015

beckermr commented on Oct 12, 2015

esheldon commented on Oct 12, 2015

esheldon commented on Oct 13, 2015

bgarwood commented on Oct 13, 2015

esheldon commented on Oct 13, 2015

bgarwood commented on Oct 13, 2015

beckermr commented on Oct 13, 2015

esheldon commented on Oct 13, 2015

beckermr commented on Oct 13, 2015

beckermr commented on Oct 13, 2015

beckermr commented on Dec 17, 2015

esheldon commented on Aug 11, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions