Repository navigation
Adding musl c library to micro python - musl branch #9
Description
Hi All,
I've created a new branch 'musl' to import the musl c library
The musl c library has a MIT license, which works nicely with incorporating it into micro python. I'll be looking at adding some (obviously not all the code) of the features like printf, UTF-8, math, regex, etc. to micro python.
I've imported the entire library (including parts that probably won't be used), including history and placed it in a directory called musl. You should be able to track changes back in time in the musl project with
git log --follow
--follow flag is needed because I moved all the files from the root to the musl directory.
Anyways, lots of changes will be made, this branch probably will be unstable at times. If there is any other features that you think might be important let me know. Feel free to comment, hack away or review as this little adventure begins!
Hagen
Activity
dpgeorge commentedon Dec 29, 2013
Thanks Hagen!
It would be great if we can merge this (or parts of it) soon into the master branch.
Does the musl regex implement the same syntax and semantics as Python needs for its regex module?
dpgeorge commentedon Dec 29, 2013
Also, how do we keep in sync with the musl development? It would be nice if we didn't modify musl (as much as that's possible) but instead just started using the parts of their library as necessary. That way we can pull in any new changes they make.
hagenkaye commentedon Dec 29, 2013
That's a good question about regex, I'll have to look into that to know the answer. Seems to me if it doesn't implement the same then it's probably not a good idea to bring in this feature.
I'll try to code in small milestones that would be easy to merge back into the master.
I've struggled with the idea of keeping in sync with musl development. My only experience in this, is with svn, which is not pretty. The model we use is to keep a separate 'Vendor' tree and then hand merge local changes into a 'Local Vendor' tree anytime one of the 'Vendor' libraries has a significant update.
I'm not a git expert, so what I did was just import the repo along with history. I guess we could consider this a branch of some sorts. The best idea I have now is to hand merge any significant changes to musl master into the micro python 'branch'. Unless there is a better 'git' way of doing this. I'm open to any suggestions.
The last bit of work I did with a C library was to enhance the Bionic C library on the Android phones to add wide character support and other items that were missing, sort of the reverse of what I'm trying to do here. From my experience doing that, the C library tends to be like a sweater and when you pull on a thread a lot more comes along with it. So I believe we'll have to do some local changes so the whole sweater doesn't come along with the threads we are pulling.
pfalcon commentedon Dec 29, 2013
Indeed, why import big 3rd-party libs into MicroPython? Why exactly musl? There are/can be lot of C libs around - it's nice to test them for uPy compilation, but why force specific one?
What would be useful is to limit core MicroPython usage to specific subset of C library, like for example no stdio (POSIX file i/o should be enough). But that won't help with 3rd-party extension modules anyway.
hagenkaye commentedon Dec 30, 2013
Generally the C library in the open source world comes from two historical sources - GNU and BSD - the difference being the license model. Since micro python is MIT licensed, the most compatible license to use is the c library that originates from the BSD source.
The musl c library is a carefully crafted version of the c library that carries the MIT license. By bringing in a snapshot of this library (all of the code) into micro python a paper trail has been created of where this source originally came from. Now we can pick and choose what portions of the C library we can repurpose for micro python rather than write code from scratch.
The only other library I can think of that would fit the license would be the Bionic C library which caries the Apache License. However this library has already been stripped down to not include UTF8 support, so there would be quite a bit of work getting the proper code to bring that support back in.
There is a lot of value of re-using this code, even something as simple as a printf function has a lot of nuances that have already been figured out by teams of programmers before us.
pfalcon commentedon Dec 30, 2013
But what's the use case of crafting C library "just for MicroPython"? Currently, MicroPython can be built with any C lib, and 2 actual targets have good libs already - for Cortex-M bare metal that's apparently newlib, for POSIX system (codenamed "unix") - that's host libc, which for most people would be glibc. So, current usecases are covered, what are yours?
hagenkaye commentedon Dec 30, 2013
newlib and glibc are GPL license. Since the micro python code would be statically linking in these libraries, the license of micro python would change from it's existing MIT license which has an overall attractiveness that out weighs just statically linking in a GPL license library.
dpgeorge commentedon Dec 30, 2013
Okay, well, it seems we need to have a think about the issue of a C library.
The main thing we need is UTF-8 support. What is the standard C lib way of doing UTF-8?
For regex, it would be nice to take it from somewhere, but it needs to be exactly Python's regex. We might be able to use the Python code, if it has the correct license.
Neon22 commentedon Dec 30, 2013
regex lib comparisons with licenses:
The pypi module re is a replacement (but not exact match) for inbuilt and is licensed under the PSFL. Is it compatible with the MIT ?
hagenkaye commentedon Dec 30, 2013
I fear I've scared people by talking of tailoring a c library. It's really not that scary, but it is a bit of work. There are lots of things to consider when trying to answer a question like adding UTF-8 support and such. The first that comes to mind is simple things like a c function isalpha()
In the ASCII world this is a-f,A-F but start dipping into the UTF-8 world and now is isalpha() true for é or ö. This is the 'thread' that starts to unravel the 'sweater'. Now you need to bring in wide character support, locale, etc.
It's really not that hard to bring in a C library, but you need to know how much of the sweater you want to bring in.
Hope this makes sense...
pfalcon commentedon Dec 30, 2013
Hello,
On Sun, 29 Dec 2013 16:24:30 -0800
hagenkaye [email protected] wrote:
Newlib is a hack of BSD libc, with most of the hacks themselves
released as BSD: https://sourceware.org/newlib/COPYING.NEWLIB
Glibc is _L_GPL. Yes, it does NOT have static linking exception, like
some libs do. But It makes no sense to static-link Glibc anyway,
because it's bloat. Glibc users use it dynamically and don't have
problems with it.
So, there's no real need to fight windmills. There're fun reasons to
use other libc's and I understand that kind of fun, and will be doing it
to MicroPython like I did to some other projects - unless you beat me on
it ;-). But for that kind of fun, there's no need to do large random
code drops into MicroPython repository. Use the alternative libc's as
they are. If there're good reasons to not use them as they are, fork
them into your own repos, and use that. When you have some results,
create a wiki page titled "How to link a uPy with arbitrary alternative
libc". Well, that's how I'd do it, YMMV.
Best regards,
Paul mailto:[email protected]
pfalcon commentedon Dec 30, 2013
Btw, it would be nice to open separate ticket for regex library - it does not have to come from libc.
pfalcon commentedon Dec 30, 2013
Custom-tailored C libs is not scary, it's lotsa fun. What's scary is random code drops into main repo ;-). Let's try to keep the project clean and lean. Custom-tailored lib can start as a separate repo. It can also start from STM code as pointed by @dpgeorge . Or it can start from another lib, but not with approach "drop lot of iunk, then try to clean it up", but with "take one useful piece, then pull useful dependencies and prune useless". In that regard, the best thing about musl is the full match of license name. Because otherwise it "tries to be correct and modern", which means threads supports, and thread-safety is a gore (when you don't need it). So, there're so many nice choices and tries to do before landing code in the main repo ;-).
hagenkaye commentedon Dec 30, 2013
I'll respectively disagree with the pull in everything and prune approach. As this method definitively shows where source code came from (from that hash of the entire source tree as the first commit).
Anyways, I guess I could put this in a different repo, but I prefer just to create a branch as this allows everyone to see the code in one place easily. I'll let Damien decide if anything I do for fun is worthy of merging back to master.
12 remaining items
Merge pull request micropython#9 from micropython/master
Merge pull request micropython#9 from adafruit/ticks_ms
Merge pull request micropython#9 from emfcamp/list-default-inputs
Merge pull request micropython#9 from pmp-p/patch-14
Merge pull request micropython#9 from adafruit/main
Merge pull request micropython#9 from robert-hh1/w600_i2c
Merge pull request micropython#9 from kwagyeman/openmv
Remove 'manifest.py' file, useless as set as default
docs: Mark Issue #9 as resolved in CLAUDE.local.md.