Change in behavior in setuptools 0.6c11 is causing issues on certain setups - #396

It looks like others have had this issue: pandas-dev/pandas#1806

Suggested patch in setuptools, though seems to have fallen on deaf ears? http://mail.python.org/pipermail/distutils-sig/2012-June/018647.html

Use distribute.

@astrofrog, was the Extension instance you were looking at here from distribute or vanilla setuptools? (maybe try inspect.getfile(ext.__class__) and see if distribute is in the path?) I would have thought it already should be distribute, because we use distribute_setup.py...

It's vanilla setuptools for some reason. I thought we were automatically using distribute too, but it looks like in some cases it can switch to setuptools. The output of inspect.getfile is:

/Volumes/Raptor/tmp/tt2/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/extension.py

Also, to test this out, I had to put this in my dependencies file for pip

-e git+http://github.com/astrofrog/astropy.git@fix-pip#egg=astropy

so the print statements are in one of my branches. Exploring this more, the issue is occurring when using virtualenv, without the --distribute option. If I create a virtualenv with the --distribute option, then it works fine.

However, I think this wasn't an issue with older versions of setuptools - as indicated in the link for pandas in one of my above comments, this is a change that has also affected other projects.

I feel like something has to be done at some level, because there is a change in behavior that is messing things up. Any ideas on how we should proceed?

Actually, I don't fully understand how this is "new"/changed behavior. Isn't 0.6c11 pretty old? If I look at http://pypi.python.org/pypi/setuptools/0.6c11, it seems to say this was most recently updated in 2009, so it should be the "standard" setuptools?

Anyway, though, the point of using distribute is to get around the fact that setuptools is no longer updated (and not py 3.x compatible, anyway). So while we could do a workaround like that pandas fix (maybe? Our scheme is slightly different and I haven't thought about it terribly closely), I'm not sure we want to. A better fix might be to figure out why distribute is not being used. distribute_setup.py is supposed to download the appropriate distribute and use it regardless of which setuptools is installed, and that is triggered in the first two lines of setup.py, right? Does pip somehow skip that?

It seems that when installing with pip, if distribute cannot be found, setuptools is used. This is an extract from the installation log with pip from a standard virtualenv (which still defaults to setuptools):

  Running setup.py develop for astropy
    /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'use_2to3

(and use_2to3 is a distribute thing) so this shows that vanilla setuptools is being used. Curiously, if distribute is present, albeit a version older than 0.6.28 (which is what distribute_setup.py tries to install), then I also get an error:

  Running setup.py egg_info for package astropy
    The required version of distribute (>=0.6.28) is not available,
    and can't be installed while this script is running. Please
    install a more recent version first, using
    'easy_install -U distribute'.

(which, incidentally, was a good reason to not update distribute_setup.py to the latest and greatest I guess). So the bottom line is that it seems that somehow, if distribute is not installed at all, then vanilla setuptools is used instead. The reason this is an issue is that basically if someone sets up a default virtualenv (i.e. with setuptools), as it stands, 0.2 will not install (0.1 did not have this issue). Of course, one option is to just tell people they have to run virtualenv with --distribute, but that doesn't seem ideal to me.

Interesting, so in the case of a virtualenv set up with setuptools, distribute_setup.py does add distribute at the front of sys.path, but once we get back into setup.py, if setuptools is imported, it's still the vanilla one. I'm investigating...

This is definitely a pip vs non-pip issue. When using pip, after

use_setuptools()

if I do

import setuptools
print setuptools.__file__

it points to the vanilla setuptools. If not using pip, it points to the distribute setuptools.

I've summarized the issue on stackoverflow:

http://stackoverflow.com/questions/12572759/issue-with-pip-distribute-and-setuptools

where I hope someone will have some insight...

I'll need to spend some time catching up on this later, but in sort if you already have setuptools installed then distribute isn't used (because the import setuptools succeeds).

Scratch what I just said--I forgot that we do call into distribute_setup.use_setuptools() by default. I'm just out the door so I'm out of time to look at this for now, but I think that use_setuptools() does not automatically override an existing setuptools installation. I think there's something like distribute_setup.use_setuptools(replace=True) that does this, however.

The call signature for use_setuptools is:

def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
                   to_dir=os.curdir, download_delay=15, no_fake=True):

so I don't see anything that would help us here... It does look like use_setuptools overrides setuptools, just not when using pip. See the StackOverflow question for a simple example.

@iguananaut - I have a suggestion for fixing this in Astropy without hacking too much. We could change:

        if release or not HAVE_CYTHON:
            # Replace .pyx with C-equivalents, unless c files are missing
            for jdx, src in enumerate(extension.sources):
                if src.endswith('.pyx'):
                    pyxfn = src
                    cfn = src[:-4] + '.c'
                elif src.endswith('.c'):
                    pyxfn = src[:-2] + '.pyx'
                    cfn = src
                if os.path.isfile(pyxfn):
                    if os.path.isfile(cfn):
                        extension.sources[jdx] = cfn
                    else:
                        msg = (
                            'Could not find C file {0} for Cython file '
                            '{1} when building extension {2}. '
                            'Cython must be installed to build from a '
                            'git checkout'.format(cfn, pyxfn,
                                                  extension.name))
                        raise IOError(errno.ENOENT, msg, cfn)

to

        # Replace .pyx with C-equivalents, unless c files are missing
        for jdx, src in enumerate(extension.sources):
            if src.endswith('.pyx'):
                pyxfn = src
                cfn = src[:-4] + '.c'
            elif src.endswith('.c'):
                pyxfn = src[:-2] + '.pyx'
                cfn = src
            if os.path.isfile(pyxfn):
                if os.path.isfile(cfn):
                    extension.sources[jdx] = cfn
                else:
                    if release or not HAVE_CYTHON:
                        msg = (
                            'Could not find C file {0} for Cython file '
                            '{1} when building extension {2}. '
                            'Cython must be installed to build from a '
                            'git checkout'.format(cfn, pyxfn,
                                                  extension.name))
                        raise IOError(errno.ENOENT, msg, cfn)
                    else:
                        extension.sources[jdx] = pyxfn

which seems to do the trick. Any thoughts on this?

By the way, the simplest way to reproduce the issue and test the fix above is to install setuptools 0.6c11 and comment out use_setuptools in setup.py.

@astrofrog - unless I'm misunderstanding, your suggestion appears to be a change from the original behavior.

In the current master, if you are in development mode and have cython, it will always use the .pyx files, regardless of whether .c files are present. The change you suggest makes it so that if both the .c and .pyx files are present, it will prefer the .c file over the .pyx file. Cython puts .c files in the source directory along side the .pyx files, so developers working on .pyx code will have to manually delete the .c files every time with this change.

I altered your idea to fix this, and it seems to work:

            # Replace .pyx with C-equivalents, unless c files are missing
            for jdx, src in enumerate(extension.sources):
                if src.endswith('.pyx'):
                    pyxfn = src
                    cfn = src[:-4] + '.c'
                elif src.endswith('.c'):
                    pyxfn = src[:-2] + '.pyx'
                    cfn = src

                if os.path.isfile(pyxfn):
                    if HAVE_CYTHON and not release:
                        extension.sources[jdx] = pyxfn
                    else:
                        if os.path.isfile(cfn):
                            extension.sources[jdx] = cfn
                        else:
                            msg = (
                                'Could not find C file {0} for Cython file '
                                '{1} when building extension {2}. '
                                'Cython must be installed to build from a '
                                'git checkout'.format(cfn, pyxfn,
                                                      extension.name))
                            raise IOError(errno.ENOENT, msg, cfn)

I've placed a branch with this at https://github.com/eteq/astropy/tree/setuptools0.6c11-fix - shall I attach it to this issue?

Seems fine, though I don't fully understand the issue. I don't usually have setuptools installed anymore except in a few venvs for testing.

Looks like stackoverflow doesn't understand the issue either ;-) The bottom line is just that setuptools will automatically convert .pyx filenames to .c if pyrex is not available, even if cython is available... I think @eteq's fix will work, but I will double check tomorrow (I am traveling right now).

EDIT: and of course, the real issue is that pip somehow causes vanilla setuptools to load.

maybe we should leave an issue at https://github.com/pypa/pip/issues (in addition to our internal workaround)? I don't see exactly this, although I only glanced over the (rather lengthy) list...

@eteq - regarding your previous comment, please feel free to attach your suggested fix to this issue, and I'll check it thoroughly tomorrow.

Code is attached (oh, and ignore the travis build status warning - that's because it's on for my fork and off for the main astropy repo)

I tested this and it works properly with pip now - merging!