Creating too many qstr's leads to large memory use #2280

Activity

1. peterhinch commented on Aug 1, 2016

   

   peterhinch

   on Aug 1, 2016

   Last edited by peterhinch

   ContributorAuthor

   More actions

   I've narrowed this down a little further. Replacing pickle with json in the above code removes the leak. The following code is the minimum test case I've found for the leak: I suspect the exec statement.

   import gc, micropython
   gc.collect()
   micropython.mem_info()
   count = 0
   def bar():
       global count
       count += 1
       s = repr(('result', str(count))) # pickle.dumps
       d = {} # pickle.loads.
       exec("v=" + s, d) #  Leak seems to be here
   for _ in range(1000):
       bar()
   gc.collect()
   micropython.mem_info()

   It only leaks if s changes on each iteration - the counter is necessary. In my attempt to find a workround I tried this, which perhaps offers a clue to the cause.

   import gc, micropython
   gc.collect()
   micropython.mem_info()
   count = 0
   def bar():
       global count
       count += 1
       s = repr(('result', str(count))) # pickle.dumps
       d = {}
       bytecode = compile("v=" + s, '<string>', 'exec') # leak here
   #    exec(bytecode, d)
   for _ in range(1000):
       bar()
   gc.collect()
   micropython.mem_info()

   Commenting out the exec statement made no difference. The leak occurs in the compile statement. An attempt at using eval also leaks:

   import gc, micropython
   gc.collect()
   micropython.mem_info()
   count = 0
   def bar():
       global count
       count += 1
       s = repr(('result', str(count))) # pickle.dumps
       return eval(s)
   for _ in range(1000):
       a = bar()
   gc.collect()
   micropython.mem_info()

   produced

   mem: total=5613, current=2007, peak=4483
   stack: 4992 out of 80000
   GC: total: 2072832, used: 3008, free: 2069824
    No. of 1-blocks: 51, 2-blocks: 10, max blk sz: 8
   8996
   mem: total=2143665, current=255899, peak=257448
   stack: 4992 out of 80000
   GC: total: 2072832, used: 20352, free: 2052480
    No. of 1-blocks: 56, 2-blocks: 9, max blk sz: 161
   
2. dpgeorge commented on Aug 4, 2016

   

   dpgeorge

   on Aug 4, 2016

   Member

   More actions

   Confirmed. The reason for the increased memory usage is string interning: the string you are evaluating is something like v=('result', '123'), with the 123 changing on each iteration. The '123' is a small string that the parser interns when it parses the code you give to eval/exec/compile. So as iterations go on there the interned strings from previous runs remain and eventually you run out of RAM.

   A work-around for the above script is to replace str(count) with count, since then it's not a string but an integer. But that doesn't help the general case.

   👍React with 👍1Reacted by Matt Davis
3. deshipu commented on Aug 4, 2016

   

   deshipu

   on Aug 4, 2016

   Contributor

   More actions

   I wonder if string interning could be limited to just string literals that appear in the programs? Would that make sense?
4. dpgeorge commented on Aug 4, 2016

   

   dpgeorge

   on Aug 4, 2016

   Member

   More actions

   I wonder if string interning could be limited to just string literals that appear in the programs?

   As far as the parser is concerned, the input from a file is equivalent to the input from exec. So there would need to be extra logic to distinguish these.
5. deshipu commented on Aug 4, 2016

   

   deshipu

   on Aug 4, 2016

   Contributor

   More actions

   Ah, you are of course, right, in this case this is actually a string literal being evaluated, I missed that, sorry.
6. peterhinch commented on Aug 4, 2016

   

   peterhinch

   on Aug 4, 2016

   ContributorAuthor

   More actions

   Thanks for that. I'll figure out if I can apply the workround in my application.
7. tannewt

   added a commit that references this issue on Dec 18, 2019

   Merge pull request micropython#2280 from xinabox/master

   ...

   83d1aac
8. dpgeorge commented on Jun 23, 2022

   

   dpgeorge

   on Jun 23, 2022

   Member

   More actions

   Closing due to inactivity. Also #4422 tracks a similar issue.
9. dpgeorge

   closed this as completedon Jun 23, 2022