Repository navigation
streams: Support binary-encoded I/O #2543
Description
There's a well-known paradigm of reading/writing binary encoded primitive values from/to stream. This paradigm became well-known with Java, and used in Android, Node.JS, etc. Due to Java heritage, there's big collection of method, one for each data type. Some environments lead it to extremes, for example Node.JS has methods like write32UInt32BE(), read32UInt32BE().
All in all, it's useful paradigm which doesn't have good footing in Python. Instead Python's approach is based on encoding/decoding whole structures with struct module. However, when working with primitive data types that leads to verbose code, and for us, has a drawback of requiring extra memory allocation.
I propose to add binary-encoded I/O support to MicroPython stream classes, byte avoid bloated method space pitfall using Python's native approach to data type encoding, using character codes, as used in standard modules struct and array. Examples:
s.writebin("B", 100) # write a byte
val = s.readbin("<I") # read little-endian uint32
s.writebin("H", 0xaa55) # write using native system byte order, as expected
This extension is intended to be used in standard stream classes (files, sockets) and in stream-like classes of HardwareAPI (I2C, SPI, etc.)
Activity
dpgeorge commentedon Oct 21, 2016
That's certainly a nice idea and seems quite general. In particular
readbinwould be most welcome, to save doing things likes.read(1)[0].But it's obviously very biased towards byte streams. For example, how would it work for something like UART when in 9-bit mode? stmhal has
uart.readchar()anduart.writechar(int). These could becomeuart.readbin('c')anduart.writebit('c', 0x1ff).Also with SPI in 16-bit word mode, that would need special consideration.
deshipu commentedon Oct 21, 2016
A more "pythonic" approach to avoiding memory allocation would be to make those stream methods accept any iterable, not just things that fit in memory in a single chunk -- but I understand it would be much harder to implement.
A note about naming: I think it would make sense to call those methods "write_struct" and "read_struct", giving a hint that they use the same format as the struct module.
I guess 9-bit modes would need an additional format character for the struct? (There is also a 9-bit SPI for some displays, where they use the 9th bit in place of the data/command pin.)
pfalcon commentedon Oct 21, 2016
Glad you like it!
As any general mechanism, it may not cover niche cases. As the examples you give show, those can be covered too if desired.
pfalcon commentedon Oct 21, 2016
Not more than the "array" module should be named "array_struct" just because it uses (almost) the same codes as "struct".
pfalcon commentedon Oct 21, 2016
The proposed methods specifically work with a single scalar value (number). Working with an arbitrary iterable is as easy as:
deshipu commentedon Oct 21, 2016
Ah, I missed the detail that it's supposed to handle a single value only. In that case it's much less useful than I thought. From the initial description I assumed, that it would be basically a replacement for something like:
Which does appear quite often in the code of many of the drivers I wrote. I don't think I would have any use for
s.writebin("B", b), though. From the above, my example code would become:Adn even then it's not exactly equivalent, as the original code is much, much faster (especially with the hardware SPI), which matters a lot when handling things like the RGB displays.
pfalcon commentedon Oct 21, 2016
No. I gave the motivation above - it's to allow step by step construction of a structure by "writing" individual scalar values, with API similar to other languages. And how these other languages do it is simply:
Technically, this is possible, and would be similar to other stream extension methods proposed: #2180 . But while for strings it's definitely useful and the main motivation is minimizing memory allocation, I'm not so sure about such support for binary writing.
As it stands, .writeInt() - like methods are an orthogonal approach to classical Python's using struct module. So, what I propose is to add this orthogonal approach. What you propose is to go step further and make a "dot product" of Python's API paradigm and write-single-value API. The resulting API is more complex, if not heavy-weight or bloated.
Note also that I'm proposing an extra step in each direction - there was no support for alloc-less string formated writing to a string, I propose to add it. There was no support for direct binary encoded writing to a stream, I propose to add it. You propose to make 2 steps at once - write binary values like other languages do, and then write multiple binary values. I'd wary about going forward too much at once. We definitely can and should thing ahead, and as I said, technically it's possible. But I'm not sure it's right way API design wise.
dpgeorge commentedon Oct 22, 2016
I also wondered about reading/writing multiple values. The problem with reading multiples is that then you should always return a tuple of the results, and reading of a single value would necessarily need to return a tuple with 1 item. Not great, since it needs to do an allocation.
Regarding adding these methods SPI: SPI reading requires sending data out on mosi, and the API allows you to specify the output byte. So it seems you'd also need such an (optional) argument to readbin:
And for I2C you need to specify the device address, something like:
deshipu commentedon Oct 22, 2016
I'm not proposing anything. I just wrongly interpreted your initial proposal, because it wasn't entirely clear to me that it was about single values.
Merge pull request micropython#2543 from pewpew-game/support-matrix
dpgeorge commentedon May 18, 2021
Closing, see #3382 (comment)