Repository navigation
RFC: Standardizing on "".format() (vs "" % ()) for micropython-lib #77
Description
With a great sadness I succumb to using "".format() as the default (and only) way to do string formatting in micropython-lib. That means that my dreams of making str.format() optional in uPy won't be realized, and instead str.__modulo__ should be made such.
The reason for that is however obvious on a mid-duration thought: str.format() allows to format with several arguments without allocating intermediate tuple. So simple.
Activity
pfalcon commentedon Jun 10, 2016
But note that I'm now considering rewriting str.format() to work against a stream object instead of vstr, and them add stream.writefmt() method to implement zero-allocation way of streaming formatted strings.
dpgeorge commentedon Jun 11, 2016
But you need to allocate anyway when calling
str.__modulo__since you allocate the created string.This will require many changes to how str.format works, will need a vstr-stream adaptor, introduce extra overhead for stack usage and make it slower.
I don't think it will be easy to make it zero-allocation. There is other aux memory allocated during str.format that would have to be treated a different way. Eg https://github.com/micropython/micropython/blob/master/py/objstr.c#L1052 is hit when you use blank braces {}, and https://github.com/micropython/micropython/blob/master/py/objstr.c#L1078 is hit when you have a format specifier.
pfalcon commentedon Jun 11, 2016
Yes, the talk is about minimizing memory allocations, first of all, in low-hanging-fruit way. The situation we have now is that people start to do something and immediately hit memory errors (e.g. http://forum.micropython.org/viewtopic.php?f=16&t=2016 ). The talk is about letting them do at least one step before hitting them.
Yes, and now that I'm ready to use it, it needs changes anyway (before, I could plan for changes to % instead).
We do have it or should have, if not.
Well, we trade heap for stack allocation. Extra overhead is producing memory garbage and then collecting it.
Anyway, .writefmt() discussion better continue in micropython/micropython#2180
dpgeorge commentedon Jun 14, 2016
I would rather try to optimise % formatting than {} formatting. % is much simpler and easier to optimise, as well as providing a smaller formatting implementation for ports that don't want the bulk of {} formatting.
pfalcon commentedon Jun 14, 2016
That was my logic all these years, and as I wrote above, with a great sadness I give up on that treatment in the face of undeniable facts. What we need to optimize is RAM usage pattern, not code size (because code size optimization process is well settled, always ongoing, and generally obvious - just define out features which can be defined out (that applies to .format() features too)).
Of course, if you plan to work on automatic de-tuplization in uPy compiler, that sounds cool, but I'm afraid there're many more low-hanging fruits to harvest before going to such advanced things.
dpgeorge commentedon Jun 14, 2016
Let's not give up: % is inherently simpler (that's an undeniable fact) and should form the basis of any sprintf-like function that we provide, eg micropython.sprintf.
pfalcon commentedon Jun 14, 2016
I.e. here you propose that we go for micropython-specific solution? I propose stream.writefmt() elsewhere because it's potentially upstream-implementable (in a sense that it's a natural extension of what upstream provides, they won't accept it right away, because they bikeshed other things).
dpgeorge commentedon Jun 14, 2016
I would suggest a set of C-like functions: micropython.{memcpy, memmove, sprintf}. All should work on buffer objects.
pfalcon commentedon Apr 30, 2017
So, I skipped commenting on this, but would like to clarify how I see this: I treat that there's an (implied) "social contract" between upstream and paradigm-shifting forks like ours: we still should try to follow upstream way as diligently as possible. So, if % formatting was found to be not efficient enough, and upstream has {} formatting instead as an upgrade to that, than no matter how I'm personally attached to % formatting, I should try {} formatting first - as is, and then if I still find it not efficient enough, then I should extend this, new formatting API, in a way which could be technically suitable for upstream, instead of inventing own ways to do that (or working on the old formatting API).
So, switching to {} formatting is still a TODO for micropython-lib, and is ("moral right") enabler to start working on things like micropython/micropython#2180.
dpgeorge commentedon May 1, 2017
Remember that {} formatting was found to be cumbersome/verbose and f-strings are now the new thing to use: https://www.python.org/dev/peps/pep-0498/
pfalcon commentedon May 3, 2017
@dpgeorge : I empathize with the irony in reply, but {} is 11 years old: https://www.python.org/dev/peps/pep-3101/ . So, only 10 years left to reconsider f"".
cefn commentedon May 15, 2018
Has there been consideration of f-strings somewhere, or would the effort of implementing/overhead of execution be hilariously bad?
dpgeorge commentedon May 16, 2018
Not much consideration. The discussion can be found here: micropython/micropython#2415 . I didn't look into it much myself so don't know how a possible implementation would look.
nickovs commentedon Jul 6, 2018
I took a look at this and sadly I think that it would be a lot of work to implement, even though the implementation in cPython is very simple. The problem is that MicroPython doesn't maintain a proper
locals()dictionary. If you had a locals dictionary then it would be as simple as compiling f-strings into a regular string followed by a call tostr.format()with the a copy ofglobals()updated withlocals()passed as the keyword arguments. In the absence of an existing implementation oflocals()(and I'm guessing little inclination to create one) this becomes a rather harder.dpgeorge commentedon Jul 6, 2018
It's unlikely we'd be able to support a proper locals() dict, mainly because the code would need to keep track of local variable names (which would be possible but at least would consume memory in the bytecode to hold this table, around 2 bytes per local).
I thought f-strings could be/were implemented in the compiler, breaking up strings and formatting them explicitly in the bytecode. Another way of doing it might be for the compiler to inspect the f-string, work out which locals in referenced by it, and then explicitly build a locals dict to pass to
str.format()-- it can do this because it knows the names of all locals.nickovs commentedon Jul 6, 2018
I like the idea of having the compiler inspect the f-string for locals, build a temporary dictionary of just the ones that are needed and then call the existing formatting code. The details of formatting are potentially complex and best kept in one place but scanning for variable names is comparatively simple. Since you are going to be building the dictionary on the fly if you wanted to save runtime space you could also consider replacing the variable names with shorter tokens, although this would involve more compiler space, time and complexity in return for the smaller compiled code.
More generally, I have been finding that f-strings are becoming one of the biggest cPython features that I miss in MicroPython, which is why I was looking at this in the first place. I appreciate that this is the third formatting option in the language and is not really a substitute for the others, but its use is becoming quite commonplace in Python3 code and it can remove clutter from your source. As such I would love to see this feature added if you can find a way to do it efficiently.
jonnor commentedon Aug 25, 2024
I believe that multiple ways of doing string formatting is in use in the micropython-lib modules. And that it is not a priority to harmonize this. So this issue can probably be closed now, considering no activity for over 5 years.
In MicroPython 1.17 (2021) there was even added support for f-strings, which has since been improved.