Repository navigation
Store ROM qstr strings in a blob with 16-bit offsets #19715
Description
Description
CircuitPython has been on a quest to reduce firmware size, as we have a number of board builds that are close to full. I had Claude Fable look for opportunities. Some are already implemented but I did not know about them, like ulab NDARRAY_BINARY_USES_FUNCTION_POINTER, which saves about 4.5 kB. (adafruit#11403) at the cost of performance.
However, another idea which came up (adafruit#5276) was to store the compile-time qstr pool as a blob of concatenated strings and use an offsets table into it. The strings could be null-terminated, or could keep their length fields. These are the A and B alternatives in adafruit#5276 (comment).
A was implemented in adafruit#11407.
We could submit an upstream PR but you may want to look at this yourselves as well.
Code Size
The size savings on even tiny builds is noticeable: Adafruit Trinket M0, a max-192kB build, saved 1.3 kB. A 512 kB build saved 3.4 kB, etc.
Implementation
I hope the MicroPython maintainers or community will implement this feature
Code of Conduct
Yes, I agree
Activity
Gadgetoid commentedon Sep 18, 2026
Would this not hard limit the qstr pool to under 65kB? (the maximum amount of memory addressable by a 16-bit offset, minus the length of the metadata)
If a 512kB build is already saving 3.4kB (and that's just saving, not total qstr size) then it sounds like we're already in that ballpark.
Either way I guess it composes well with
MP_ERROR_TEXT, since CircuitPython has that too.dhalbert commentedon Sep 18, 2026
@Gadgetoid There are multiple pools, so the 64 kB limit is per pool. I am not sure what you mean by "we're already in the ballpark". Do you mean some pools are close to full? We don't see that:
The error strings are not qstrs, but they can take qstrs as interpolated arguments. So they don't add to the pool sizes. Did you mean something else?
If we overflowed a 64 kB pool, we could go to 32 bit instead of 16 bit pointers, which would still save some space.
dpgeorge commentedon Sep 18, 2026
If performance is not a concern you could do away with the pointers/16-bit offsets altogether, and instead just do a linear search through the qstr blob when you need to convert qstr index to qstr data. Since the strings are null terminated, if you need qstr number N then you just skip past N null bytes. And since there are already multiple qstr pools, you already start at the beginning of the correct pool.
To speed that up a bit, you could store pointer/offset to various "checkpoints" within the big qstr blob (say every 10th qstr), or just split the qstrs up into more
qstr_pool_t.Gadgetoid commentedon Sep 19, 2026
TIL!
And apparently reverse engineering the storage from the saving in my head did not arrive at a reasonable figure. That said 512kB is small potatoes for a build if you bolt on enough C modules. But pools make any concerns moot.
👀 I’m very performance-brained even though we’ve got a couple of builds that have some tens of bytes left before they blow the firmware allocated size. The size saving looks like it would be significant though.