Repository navigation
Extending stream method inventory to avoid (minimize) memory allocation #2180
Description
Activity
Discussion:
.write(data[, offset[, len]])
Something in queue for couple of year or so. Pretty obvious extension of the existing write method. Definitely should be done, saves need to make a slice or memoryview.
.writev()
This low-hanging-fruit way to optimize a bit minimize-allocation transformations like: micropython/micropython-lib@d036673
.writefmt()
This would be more powerful form of .writev() so, if implemented, would supersede it. .writefmt() allows to potentially achieve zero-allocation formatted output.
As mentioned above, the main requirements for adding these is minimize memory allocation (=> garbage production => fragmentation). Granted, it will be hard to achieve complete zero allocation in a Python code which works with strings. So, are there alternatives? Sure, if we get real-time compacting garbage collector, many memalloc problems would be alleviated. But it's quite a long way to it so far, and anyway, as we know, the only way to resolve all memalloc issues is to avoid memory allocation. Giving users (e.g. library writers) such an alternative is valuable on its own and covers usecases which compacting GC won't. E.g. .write(data[, offset[, len]]) is an obvious and low-hanging performance optimization. .writefmt() will allow to write amount of output larger than physical memory.
.readline_into(buf)
The immediate usecase for this is HTTP parsing, and then alternative is to implement complete HTTP parser in C. But then it will be only an HTTP parser (albeit very efficient), and won't be able to optimize an arbitrary line-based processing (but then amount of optimize they get from .readline_into(buf) is also pretty minor, as any non-trivial processing will require splitting which will alloc memory).
@ncoghlan: If you have interest/time, feel free to share how awful these look from your perspective ;-). .write(data, offset, len) is something about I wanted to post to python-ideas for a long time (hint: for us it's critical that new args stay non-kw-only).
These feel too low level to be adding them to the general purpose stream APIs, rather than the more C-inspired file descriptor based APIs in the OS module.
For the proposed change to write(), does MicroPython currently implement the os.pwrite() and os.pread() APIs?
- https://docs.python.org/3/library/os.html#os.pread
- https://docs.python.org/3/library/os.html#os.pwrite
Similarly, os.writev already exists for use with raw file descriptors: https://docs.python.org/3/library/os.html#os.writev
The rationale for that object-vs-descriptor split is to make it clear that these aren't general purpose APIs every stream (e.g. io.BytesIO) is expected to implement - instead, they're intended for working optimally with the underlying OS (in the case of CPython) or hardware (in the case of MicroPython).
too low level
Indeed, they are - as mentioned, they come from a requirement which no Python programmer is usually concerned with - minimize memory allocation (we don't want Python programmers to be concerned with that too much, but library maintainers may want to do that anyway).
instead, they're intended for working optimally with the underlying OS (in the case of CPython) or hardware (in the case of MicroPython).
Well, not how it's treated in MicroPython, "os" is really an OS, and if there's no OS, "os" module can only emulate some things, at the basic level. OS file descriptors and Python streams are also at the different conceptual levels and have different "polymorhic" power, for example you can't use os.pwrite() on io.BytesIO, right? You could say that for MicroPython bare-metal we could implement io.BytesIO in terms of "os" "file descriptors" (using a special file name for example). But that idea breaks down when we move MicroPython-unix. There, "os" is a real underlying POSIX OS (and we won't add a layers on top of it to make io.Bytes still possible in terms of it - we develop Python implementation, not an OS ;-) ).
Summing up, what's in "os" module is dictated by the underlying OS. CPython's "os" has pread/pwrite/writev only because there're such POSIX functions. OTOH, I want to added the methods above on the Python level, regardless of what's there on OS level.
Thanks for the comments though, it's a useful perspective to keep in mind!
I think the additional context I'm missing is whether or not you're proposing to expand the API of the default streams returned by builtin
open(), or instead define an opt-in "MicroStreams" API that exposes additional low level interfaces (so somebody would have to do something likefrom ustreams import opento get the MicroPython specific features).In the latter case, you're pretty much free to do what you want, with the only risk being that later changes and additions to the standard APIs may conflict with the MicroPython extensions.
In the former case, you'd be on much shakier ground without getting the changes approved for CPython first, since you'd now have a situation where people could write code that wasn't visibly MicroPython specific, but wouldn't run on any version of the reference Python implementation.
The standard stream API does support the notion of optional methods (by way of https://docs.python.org/3/library/io.html#io.UnsupportedOperation ) so if you do want to make these features available by default in MicroPython, I'd suggest proposing them as new optional methods/behaviours for
io.RawIOBasewith concrete implementations based onos.pwriteandos.writev. If you do go down that path, I'd suggest filing each of them as a separate RFE, since the optionality and implementation strategy would differ for each of them.I think the additional context I'm missing is whether or not you're proposing to expand the API of the default streams returned by builtin open()
This. Or more specifically, any object adhering to Python's "stream protocol" may choose to expose these methods (and well, all which already expose "write", will automagically get 3-arg variant). This includes not just file streams from open(), but network sockets, etc. Implementation-wise, these methods are completely generic and work in terms of the C-level stream protocol (which is based around POSIX-like read() and write() operations).
later changes and additions to the standard APIs may conflict with the MicroPython extensions.
We hope that CPython will stay conservative ;-).
but wouldn't run on any version of the reference Python implementation.
In that case, CPython would have done that, not us ;-).
The standard stream API does support the notion of optional methods (by way of https://docs.python.org/3/library/io.html#io.UnsupportedOperation ) so if you do want to make these features available by default in MicroPython
Thanks, when I find time, I'll start with posting 3-arg .write() idea to python-ideas. Though I give 99% that what it'll come to is that "we might not be against it if new args are kw-only args", and that's exactly not we (well, I) would like, as I mentioned in the first method. I really wish that people treat kw-only args as a good idea for user-level apps and libs, and not so good idea for core and around-core features...
but wouldn't run on any version of the reference Python implementation.
In that case, CPython would have done that, not usI know you had the smiley there, but I want to be explicit that that's not the way the language design process (or the Python trademark) work - regardless of whether they're a community project or a commercial operation, folks claiming to provide a Python implementation should not go beyond the reference implementation, except in ways that anyone is free to go beyond the documented API (e.g. adding new modules) or when the deviation is a core part of the reason the alternate implementation exists (e.g. green thread semantics in Stackless Python).
"Allocation free" byte-copying APIs in MicroPython skirt that boundary (since they could be claimed as part of MicroPython's reason for existence), but in most cases I expect you'll be able to make the case for those as POSIX-only io.RawIOBase APIs in CPython as well. The key in making that case will likely be to document all the allocations (and resulting potential memory fragmentation) you require when using the standard API, and how the proposed API enhancements let you avoid those allocations without requiring that developers switch to a completely different IO model.
I think it's a slippery slope defining core methods like these that deviate from CPython. Because then 1) all uPy libraries and user code run only on uPy and can't be tested with CPy (unless you provide a ton of adaptor modules); 2) people have to essentially learn a new language (to both read and write uPy code).
The problem of memory fragmentation is a big problem and needs a plan for a proper solution. I would much prefer a solution that allows people to bring their existing knowledge and write "normal' Python, rather than write in uPy dialect.
A 3-arg write is a nice addition, but I'd really want to see it adopted upstream before we start to use it heavily. But note that allocating a memoryview object will always succeed (if there's non-zero RAM available) because it takes only 1 GC block.
Because ...
Because we can't deliver a nicely working esp8266 port, let me give the same link as I gave in micropython/micropython-lib#77 : http://forum.micropython.org/viewtopic.php?f=16&t=2016 .
can't be tested with CPy (unless you provide a ton of adaptor modules)
So, it's possible to test it on CPy after all (though I don't think anyone would bother). As for "ton", only few might be required, again, if someone would bother to write them.
people have to essentially learn a new language
They're not intended for "people", they're intended for "people who want to run Python where CPython couldn't go even remotely". For example, there's completely no need to integrate assembler into Python, and people should not use it. But people who really need it, well appreciate that it's there, an don't care that all other people should not be using it at all.
The problem of memory fragmentation is a big problem and needs a plan for a proper solution.
I don't know what plan it requires, it's all clear: it requires 1 man-year (+/-) to be implemented, so now there should be a party coming which needs to such extent as to provide support for such effort. While there's a wait for such an event, users of existing ports should not suffer from inability to do things or grow perception that the maintainers don't care about the problem.
A 3-arg write is a nice addition, but I'd really want to see it adopted upstream before
Cool! Other MicroPython developers appear so rarely on the upstream mailing lists, so it's really nice to hear that I can strike it off my list. But I don't think there should be that "before". Upstream likely don't care about that, they're busy with breaking compatibility with previous versions (PEP467) and other important things.
But note that allocating a memoryview object will always succeed (if there's non-zero RAM available) because it takes only 1 GC block.
Yes, it will take 1 GC block from a contiguous 4K area, so next allocation of 4K will fail (following the forum post above).
Because we can't deliver a nicely working esp8266 port, let me give the same link as I gave in
Easy thing to help here is to enable frozen bytecode, ie #2067. Then we can have much larger built-in libraries, and add features without concern for it not fitting in RAM.
As for "ton", only few might be required, again, if someone would bother to write them.
All streams need to be changed so that write works, for example. That's a lot of work. And the point is that we shouldn't need people to bother to do anything to have code compatible under both uPy and CPy.
For example, there's completely no need to integrate assembler into Python, and people should not use it. But people who really need it, well appreciate that it's there, an don't care that all other people should not be using it at all.
That's totally different. Inline assembler is an addition to the language that can be easily abstracted into a separate file, and users of the asm function don't even need to know how it's written. Compared with modifying all stream write methods, that's a much more tightly integrated change.
I don't know what plan it requires, it's all clear: it requires 1 man-year (+/-) to be implemented
Implement what? A compacting GC is not going to work because 1) identifiying pointers is not precise and cannot be made precise without support from the compiler, or totally rewriting the code to have a Python value stack; 2) it introduces significant delays and you can't run irqs during the GC because they may be pointing to memory that is being moved.
There needs to be a different solution, like scoped heaps.
@pfalcon, being pre-emptively antagonistic with python-dev isn't going to help you craft the embedding friendly Python runtime you're seeking - it will just increase the likelihood of proposals being rejected out of hand due to the messenger, rather than the message. Open source programming language design is as much a problem of diplomacy as it is one of technology, so studying successful change proposals like PEP 467 can be useful in that regard: they provide insight into some of the key design considerations (like ease of learning and ease of debugging) that Guido values above and beyond the strict backwards compatibility policies frequently sought by businesses that want to adopt (or continue) a "write it once and then let it bitrot in production without any ongoing maintenance and support" approach to software development projects.
When it comes to effectively making your case for change, as far as I'm aware, I'm the only current core developer with professional embedded development experience, so you won't be able to take it for granted that folks will already appreciate the additional problems that arise when 1 KiB starts counting as "a lot of memory". There's precedent (via the NumPy/SciPy stack) for python-dev standardising features that the reference implementation of the standard library doesn't use, which means you should be able to propose changes for the benefit of MicroPython that always raise UnsupportedOperationError on non-POSIX systems.
That said, the main objection I'd anticipate is a learnability one: any proposal that makes
dir(open("filename.txt", "rb"))ordir(open("filename.txt", "wb"))return additional names necessarily makes Python harder to learn (e.g. code completion in the REPL and IDEs will present more method names, and the additional parameters will appear in call tooltips). Experienced programmers will be readily able to ignore those as being specific to embedded programming, but new programmers don't have those kinds of relevance filters in place yet - they're already suffering from information overload, so anything that is likely to add to that is cause for concern.That means you can anticipate that I won't be the last one to ask "Why isn't a more explicit alternative like
from uio import opensufficient for your use case?". That kind of separation means that the additional complexity is hidden behind an additional activation trigger that makes it much easier for newcomers to Python to ignore, and it means that the need to engage with python-dev would be limited to "Hey, if you ever consider expanding the io.RawIOBase API with POSIX-specific additions, please take compatibility with the MicroPython dialect's additionaluioAPI methods into account". It also means someone (whether the MicroPython project or a third party) could publish something like amicrolibpackage on PyPI that emulated the variousu*MicroPython modules atop existing versions of CPython (and perhaps other runtimes) for local development & testing purposes, rather than having to wait for Python 3.6+ to be published and rolled out through the various redistribution channels.@ncoghlan : I'd like to say (or likely repeat) that I really appreciate you appearing in micropython discussions and answering questions. However:
being pre-emptively antagonistic with python-dev
Sorry, but I'm as "pre-emptively antagonistic" as you (or upstream in general) "not ready to accept a tiny bit of criticism". I'll leave it your judgement who's who. In the meantime, please answer me who of the previous projects like pymite, tinypy, owlpython, or snap talked to upstream? If about none, then d'oh. I did quite a bunch of talking already. Unfortunately, I can't say that I got much support, if any at all. At most it was like "You say we implemented feature you wanted, you provided feedback on that? Didn't hear. And nope, I just did what I had on my mind.". So, sorry for me being skeptic about it. Note that I'm personally not "pissed off" by upstream in any way, though I know there're some who are (I wanted to gave a link, it was funny-titled ;-) post at http://nuitka.net/ , but can't find it now, perhaps it was removed ;-) ).
values above and beyond the strict backwards compatibility policies frequently sought by businesses that want to adopt (or continue) a "write it once and then let it bitrot
That sounds nice (and I love Python for its approach and paradigm!), but there's also big world around. I was astonished some time ago to read "we decided to implement our software in JavaScript because it's one of the most stable languages - code written 15 years ago runs as is nowadays". I was speechless at such arrogant, bold twist. Yet 3 years ago it was pretty customary for JS-enabled sites to misbehave when used with "wrong" browser or break on a browser upgrade, it took incredible handholding when programming in JS (literally as in anecdote of how many JS programmers required to...) and of course changing JS to achieve the state we have now - when JS stuff doesn't break right away, just eats up 100% of CPU time. And yet lack of any reasonable standard library and obduracy of the language allowed JS apologists to issue statements like above. It was easy to counter that with "Python is what's stable and supported. Current version is released in 2008, but previous version is supported at least until 2008". But reading about things like removing the most basic things like
bytes(2), it's hard to keep suggesting to other people that Python is well designed and stable language without risking to be called a liar.Now to more practical part:
When it comes to effectively making your case for change
I don't know how we come to this in this discussion at all. I'm not going to come to CPython and "make a case for change". Well, not all at once, and definitely not before it was tested and tried. So, even the most obvious 3-arg write would be first implemented in MicroPython before seriously proposed for CPython.
any proposal that makes dir(open("filename.txt", "rb")) or dir(open("filename.txt", "wb")) return additional names
That's why it's not going to be proposed for CPython before it's well tested in MicroPython - to not add more stuff than really needed and useful.
e.g. code completion in the REPL and IDEs will present more method names, and the additional parameters will appear in call tooltips
Sorry, but then the most obvious solution is not to add them to CPython! Productive talk would then circle about finding good naming and semantics, and making them "reserved" for CPython. And that's the closest to what I'm talking about. My question to you was essentially "If such functions were added to CPython, what would they be named, what args they should take, etc.". I have no idea why you want to "drag" me to pushing it to CPython, especially bringing up matters like "trademarks" (btw everyone definitely has own criteria for "friendliness", and the above doesn't fit mine well, much less than someone calling me wanting only to break stuff for example).
hidden behind an additional activation trigger
MicroPython can't afford additional activation triggers. CPython can. Actually, it can afford to not implement "general Python3 language" features, just note that such features exist.
"Hey, if you ever consider expanding the io.RawIOBase API with POSIX-specific additions, please take compatibility with the MicroPython dialect's additional uio API methods into account"
It would be better: "Hey, if you ever consider expanding the io.RawIOBase API with any additions, please take compatibility with the MicroPython dialect's additional API methods into account, these are: writefmt(), and additional args to write()".
It also means someone (whether the MicroPython project or a third party) could publish something like a microlib package on PyPI that
Me was doing such stuff: https://pypi.python.org/pypi/micropython-cpython-uasyncio . It can be done for proposed changes too. I don't think it's worth doing that for proposed changes, as code using them is not intended to be run on CPython (but on an easy day, I'll gladly implement it, caveat: I didn't have easy days for couple of years).
That's totally different. Inline assembler is an addition to the language that can be easily abstracted into a separate file, and users of the asm function don't even need to know how it's written. Compared with modifying all stream write methods, that's a much more tightly integrated change.
It's not totally different! Whatever you have in mind for inline assembler, it's the same usage for the methods proposed here. They also intended to be "abstracted into a separate file", e.g. there will be urllib.request which uses .writefmt() for MicroPython, but CPython has own implementation of urllib.request! Or there will be umqtt.simple for MicroPython using it, but as soon as there will be interest from CPython folks to put aside dozen of MQTT libs available for it and use MicroPython's instead, I'll happily rewrite one for CPython.
Implement what? A compacting GC is not going to work because 1) identifiying pointers is not precise and cannot be made precise without support from the compiler, or totally rewriting the code to have a Python value stack; 2) it introduces significant delays and you can't run irqs during the GC because they may be pointing to memory that is being moved.
Yup, it's not exactly easy, but was shown to be doable (by prior art).
There needs to be a different solution, like scoped heaps.
That won't work for pragmatic, imperative Python. But there're growing (have-grown) bunch of LISP hack-alikes trying to do that, just yesterday looked into that: https://github.com/wolfgangj/bone-lisp
It belatedly occurs to me that a similar method to the one TI use (or at least used to use) for their C compiler intrinsics may be applicable here: adopting a prefix convention (such as
_upy_*orupy_*) for any MicroPython specific extensions to the builtins or methods on builtin types.That will free you up to experiment and decide what you want, without risking potential confusion around where the standard Python APIs end and the MicroPython dialect begins, and without needing to engage with the full weight of the core language design process before you're certain you actually want to request language level changes.
.writev(data1, data2, ...)
So, besides .writev(), which would take individual args, it would be useful to have a method which takes an iterable and writes items from it. As semantics is different, I guess it should be a separate method from .writev(). Such a method would be especially useful for async, coroutine-based I/O. I'm immediately adding StreamWriter.awriteiter() to uasyncio.
Closing; see #3382 (comment)
This is continuation of #2056. I'm considering to add following methods for streams:
For write:
For read: