Repository navigation
Compress QSTR pointer pool #5276
Description
Activity
Claude wrote this:
Revisiting this with measurements on the SAMD builds as of 2026-09-15. The post-link step this issue anticipated is no longer needed.
Design
makeqstrdata.pyemits the const pool's strings as one concatenated, NUL-separatedconst char[]blob instead of separate literals. Offsets into the blob are then known to the generator, so it emits aconst uint16_t[]offset table as ordinary compile-time constants. No linker or post-link involvement.- Runtime pools keep the current
const char *qstrs[]layout.qstr_pool_tgets a base pointer and a flag (or a union) so the six accesses inpy/qstr.c(qstr_str(),qstr_data(),qstr_len(),qstr_hash()when hashes are off, and the two compares inqstr_find_strn()) pick the right representation. Sorted-pool binary search works unchanged. mpy-tool.pydoes the same formp_qstr_frozen_const_pool, as a second blob.- Largest blob seen is 20 KB (metro_esp32s3, 2118 qstrs), so 16-bit offsets fit everywhere with room to spare. The generator should assert the limit.
Two variants
- A: also drop the const pool's length array. With one trailing offset entry,
qstr_len(q)isoffsets[q + 1] - offsets[q] - 1. Saves 3 bytes per qstr, but requires blob order to match qstr-id order with no overlap, so strings that are suffixes of other strings can no longer share storage. - B: keep the length array, and have the generator share suffixes the way the linker does today (sort by reversed string, reuse the tail). Saves 2 bytes per qstr.
Measured net saving, after subtracting what the linker currently gets from suffix sharing among qstrs and from merging qstrs with identical C literals elsewhere (which a private blob duplicates: 89 to 534 bytes per board):
Board const qstrs A: drop lengths B: keep lengths, share suffixes trinket_m0 638 ~1.4 KB ~1.2 KB metro_m0_express 941 ~2.0 KB ~1.7 KB metro_m4_express 1651 ~3.5 KB ~3.1 KB pyportal (1771 const + 920 frozen) 2691 ~4.2 KB ~4.8 KB A is simpler and wins on boards without frozen modules. B looks better on pyportal because the linker currently shares 3.4 KB of suffixes there, but some of that is across the frozen and const pools, which separate blobs cannot reproduce, so B's pyportal figure is an upper bound. Either way add roughly 50-100 bytes of code for the per-pool branch in
py/qstr.c.Upstream
MicroPython has not proposed this. It builds on two things they merged in 2023:
MICROPY_QSTR_BYTES_IN_HASH=0(micropython#10758) and sorted const pools (micropython#12678), which is what makes offset-based lookup safe. It is a candidate to send upstream rather than carry as a divergence.Sounds promising! I hadn't thought of generating my own big string.
The QSTR pool is a bunch of 32-bit pointers. We could store them as 16 bit offsets from a base pointer for the whole pool instead.
Doing this for the const pool built into the firmware would save ~1300 bytes of flash but require a post-processing step after the link. The linker places all of strings so it can't be done earlier. I tried to subtract the start of rodata in the initializer list but the compiler complained. Post processing it would require placing the pool as the last thing in flash so that none of the other locations change.
We could probably do the same trick for the in-memory pools too. Micros with <64k RAM will always work because we know the loaded strings will all be in 16-bit range of one another.