Repository navigation
Creating too many qstr's leads to large memory use #2280
Description
Activity
I've narrowed this down a little further. Replacing pickle with json in the above code removes the leak. The following code is the minimum test case I've found for the leak: I suspect the exec statement.
import gc, micropython gc.collect() micropython.mem_info() count = 0 def bar(): global count count += 1 s = repr(('result', str(count))) # pickle.dumps d = {} # pickle.loads. exec("v=" + s, d) # Leak seems to be here for _ in range(1000): bar() gc.collect() micropython.mem_info()
It only leaks if s changes on each iteration - the counter is necessary. In my attempt to find a workround I tried this, which perhaps offers a clue to the cause.
import gc, micropython gc.collect() micropython.mem_info() count = 0 def bar(): global count count += 1 s = repr(('result', str(count))) # pickle.dumps d = {} bytecode = compile("v=" + s, '<string>', 'exec') # leak here # exec(bytecode, d) for _ in range(1000): bar() gc.collect() micropython.mem_info()
Commenting out the exec statement made no difference. The leak occurs in the compile statement. An attempt at using eval also leaks:
import gc, micropython gc.collect() micropython.mem_info() count = 0 def bar(): global count count += 1 s = repr(('result', str(count))) # pickle.dumps return eval(s) for _ in range(1000): a = bar() gc.collect() micropython.mem_info()
produced
mem: total=5613, current=2007, peak=4483 stack: 4992 out of 80000 GC: total: 2072832, used: 3008, free: 2069824 No. of 1-blocks: 51, 2-blocks: 10, max blk sz: 8 8996 mem: total=2143665, current=255899, peak=257448 stack: 4992 out of 80000 GC: total: 2072832, used: 20352, free: 2052480 No. of 1-blocks: 56, 2-blocks: 9, max blk sz: 161Confirmed. The reason for the increased memory usage is string interning: the string you are evaluating is something like
v=('result', '123'), with the 123 changing on each iteration. The'123'is a small string that the parser interns when it parses the code you give to eval/exec/compile. So as iterations go on there the interned strings from previous runs remain and eventually you run out of RAM.A work-around for the above script is to replace
str(count)withcount, since then it's not a string but an integer. But that doesn't help the general case.Reacted by Matt DavisI wonder if string interning could be limited to just string literals that appear in the programs? Would that make sense?
I wonder if string interning could be limited to just string literals that appear in the programs?
As far as the parser is concerned, the input from a file is equivalent to the input from
exec. So there would need to be extra logic to distinguish these.Ah, you are of course, right, in this case this is actually a string literal being evaluated, I missed that, sorry.
Thanks for that. I'll figure out if I can apply the workround in my application.
- added a commit that references this issue
on Dec 18, 2019
Also on Pyboard, ESP8266. Pasted at the REPL:
This produces the following outcome with used increasing: