RFC: PyPI packaging guidelines

Let's discuss conventions and guidelines when publishing modules on PyPI - for "stdlib" (#405, https://github.com/micropython/micropython-lib), and in general.

Terminology: https://docs.python.org/3.3/distutils/introduction.html#distutils-specific-terminology

Proposal: if distribution wants to show strong affinity for being used with MicroPython (that's in particular 100% the case for micropython-lib), it's name should start with a standard prefix. Given our existing choices for long, fully explicit names, the prefix almost certainly should be "micropython-". That's long, but fully self-describing. So, last chance for a big crowd to shout "It's too long" and propose something shorter.

Aren't you the big proponent for short names? :)

After using it for some time, I agree that typing ./micropython is a bit long. Still, there is no better option, and having, eg, "upython-" as a prefix is not very descriptive.

Yeah, now that it's chosen, need to be consistent. Ok, here we go: https://pypi.python.org/pypi/micropython-unittest/0.0.1 released from https://github.com/micropython/micropython-lib/tree/master/unittest

Attempts to bend pip to install it: http://forum.micropython.org/viewtopic.php?f=3&t=55 (warning: thoughtstream-style).

After using it for some time, I agree that typing ./micropython is a bit long. Still, there is no better option, and having, eg, "upython-" as a prefix is not very descriptive.

Umm. You know about tab completion right?

./mTAB

If you ask nicely you could probably get a Trove classifier for your implementation.

Umm. You know about tab completion right?

Yes! But sometimes I like to type the whole thing... :)

Following RFC #405 (comment) , I started to push patched CPython stdlib modules on PyPI. This includes first module with dependecy on another module, here's how to do it: https://github.com/micropython/micropython-lib/blob/master/copy/setup.py . Using dependencies requires setuptools; setuptools are not part of Python stdlib and should be installed separately (but of course it's usually preinstalled on any distro with decent Python support). I initially dropped initial attempts of using setuptools in favor of distutils exactly because distutils are in stdlib (and IMHO has better docs). But I wonder if we should just consistently use setuptools after all.

Example of installing from PyPI (assuming you have working pip on the host already):

$ cd tools/
$ ./pip-micropython install micropython-copy
...
$ ../unix/micropython
>>> import copy
>>> copy.copy({1:2})
{1: 2}

I went on and submitted all modules we currently have in micropython-lib (all of them are very rough). Current list is here: https://pypi.python.org/pypi?%3Aaction=search&term=micropython&submit=search

Please join the testing, discussion, and setting up guidelines and best practices for MicroPython module publishing!

I remember reading (a while ago) that setuptools was being deprecated...
So I've just done some googling and found there's lots of outdated info about python packaging (seems there were many false starts), but found that https://python-packaging-user-guide.readthedocs.org/en/latest/current.html is the current state of play, and it seems as though setuptools is again the (currently) recommended option.

FYI, having bleed-to-metalevel problem with packaging some standard modules: https://bitbucket.org/pypa/setuptools/issue/187/

The last issue was worked around in 06526e7, it's now possible to install micropython-re-pcre. Please help test.

Dunno if it's a problem or not, but I just noticed that if I tools/pip-micropython install micropython-re-pcre and then tools/pip-micropython install micropython-json, then installing micropython-json seems to re-install micropython-re-pcre as well?

FYI, having bleed-to-metalevel problem with packaging some standard modules: https://bitbucket.org/pypa/setuptools/issue/187/

Ok, so the underlying issue here was that there was no way to preclude putting a current (or more exactly, running script's) directory into sys.path. That's pretty annoying and serious issue, and I kinda even wanted to report it to CPython bugtracker. But now I was re-reading 3.4 changes, and saw:

The python command has a new option, -I, which causes it to run in “isolated mode”, which means that sys.path contains neither the script’s directory nor the user’s site-packages directory, and all PYTHON* environment variables are ignored (it implies both -s and -E).

So, that's it, but perhaps too much for a case in hand (only not forcing script dir to sys.path is needed for case at hand).

@lurch

@lurch: Ack. Apparently, that's the effect of --root switch: pip thinks it installs stuff in virtualenv, and looks for existing packages there. But behind pip's back, we tell setuptools to put stuff under quite different path (--root), voila.

Btw, I had to bring in virtualenv exactly to make pip never consider any package as already installed on your system. Because otherwise it will refuse to install it again - in quite different location, where it actually doesn't exist. Actually, there's --ignore-installed option for pip, but it doesn't work as expected.

So, it's kinda impasse - we need that option to make packages robustly install, and that exactly causes them to always install from scratch. Wanna help with that? ;-). Warning: that's a trap for someone who wants to do stuff pedantically - the only outcome can be desire to rewrite it all from scratch. That's exactly what we can't/shouldn't do, so target should be the simplest and quickest solution (likely, workaround) satisfying requirements.

Afraid I've never looked into how pip works, and I've never even (knowingly) used a virtualenv.
But maybe I'll come back to it if I get bored and run out of files to 'make more pythonic' ;-)

I don't understand how "upy" is not descriptive. What else is called upy?

What would you do if you wanted to use SQLite3 on pyboard to indicate the absence of the "import ffi" module?

@PakchoiFood your comment doesn't make sense for this issue, did you post in error?

uPy-specific packages have been published using the micropython- prefix for the package name, and upip has been working well for quite some time on unix and (wifi-enabled) baremetal ports. So closing this issue.

Yeah, but this ticket was supposed to be open, so people were able to find it and learn how to make a uPy-compatible package.

people were able to find it and learn how to make a uPy-compatible package.

IMO that's more suitable for documentation.

But we don't have such documentation, and this ticket is all we have.

But we don't have such documentation, and this ticket is all we have.

The README.md of the micropython-lib repo has some documentation, and points to this ticket for further reading. I don't see the need to keep this one open (it's still searchable and commentable if necessary). Feel free to take bits from here and add to the micropython-lib README.md if you think that's useful.

Merge pull request micropython#413 from dhalbert/fix_round

Navigation Menu

Repository navigation

RFC: PyPI packaging guidelines #413

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

RFC: PyPI packaging guidelines #413

Description

Activity

dpgeorge commented on Apr 1, 2014

pfalcon commented on Apr 1, 2014

dhylands commented on Apr 1, 2014

dholth commented on Apr 3, 2014

dpgeorge commented on Apr 3, 2014

pfalcon commented on Apr 6, 2014

pfalcon commented on Apr 6, 2014

pfalcon commented on Apr 6, 2014

lurch commented on Apr 9, 2014

pfalcon commented on Apr 12, 2014

pfalcon commented on Apr 12, 2014

lurch commented on Apr 14, 2014

pfalcon commented on Apr 14, 2014

pfalcon commented on Apr 14, 2014

lurch commented on Apr 15, 2014

vitiral commented on Mar 25, 2015

LinJing0511 commented on Jun 27, 2017

dpgeorge commented on Jun 27, 2017

dpgeorge commented on Sep 8, 2017

pfalcon commented on Sep 8, 2017

dpgeorge commented on Sep 10, 2017

pfalcon commented on Sep 10, 2017

dpgeorge commented on Sep 10, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions