Repository navigation
Improve memory stats #5
Description
With fix to #2, it's now possible to fix alloc stats collection, but instead of throwing a random patch, I'd like to be sure there's good understanding what we can/want to measure.
- We can measure total allocation size. Why, it's useful to know memory throughput of an app. This is what uPy appear to measure currently with total_bytes_allocated. Except that realloc case is not correct. Memory usage grows not by new_num_bytes, but by difference between new and old size. And by logic, this metric is monotonically growing, so, (new_num_bytes - old_num_bytes) < 0 should be handled separately (specifically, ignored).
- Besides total memory allocation, it makes sense to know current memory allocation (that allows to know how much is free in particular). It takes having another var, updated also on m_free (and on realloc w/o any conditions). So, this metric is expected to be 0 at app end and proper interpreter shutdown.
- Finally, it's useful to know peak memory usage we achieved. This will answer question if particular app can run on particular heap (== on particular hardware). This would be just max(current usage).
Hopefully, all these metrics are useful (well, I'd say that 2&3 are more useful than 1). If you agree, I can implement them.
Activity
Neon22 commentedon Dec 29, 2013
That sounds useful. Obviously 3 will vary on a specific run - so maybe if user has a unit test of some kind you could also collect data for statistical memory size ?
Would a specific test be useful for this ?
pfalcon commentedon Dec 29, 2013
I don't think these will be useful for unit testing, it's more useful for (manual) testing of interpreter optimization and for particular apps optimization. Though maybe I just miss your usecase.
dpgeorge commentedon Dec 29, 2013
At the moment the unix version of uPy does not free memory. The GC doesn't work for it. I concentrated more on the STM version, and getting the GC working on that first. The unix version is just to test functionality of the Python bit.
But I agree it would be nice to have proper memory accounting in the unix version so we could test heap memory usage etc. We could also make the GC work on the unix version (not too hard to do). Note that a 64 bit machine is going to use around 2x more memory than the 32 bit MCU, but that can be accounted for.
The first thing to do is get the GC working, since then objects will actually be freed, and the accounting will make sense. At the moment, free is hardly ever called.
Neon22 commentedon Dec 30, 2013
My use case is:
User writes python code and runs it in unix/windows. Code stats report on memory usage. This information tells user that their code will use almost all available target RAM 90% of the time but will occasionally exceed the memory constraints of the target.
Therefore the user chooses to (say) sacrifice code size for speed and leave two functions running interpreted instead of unrolled into C code.
By running the code numerous times or (say) through all test cases defined in test suite, the user is able to build a statistical case for memory useage. Bearing in mind this is probably still not accurate because not all possible routes through the code may be exercised. But it will still be helpful.
pfalcon commentedon Dec 31, 2013
@Neon22: Yeah, that's exactly the usecase I have in mind, thanks for clarification. Another usecase is to test/validate changes to the interpreter itself. For example, in #8 I propose to store hash/length together with a string, by running "before" and "after" thru memory stats we can quantitatively assess the impact (and yes, that requires testcases for various use types).
pfalcon commentedon Jan 4, 2014
Described stats are now collected. It might be useful to expose them to Python side, but that can wait another ticket.
Merge pull request micropython#5 from micropython/master
changed default serial speed from 115200 to 460800 micropython#5
Merge pull request micropython#5 from karl-malakoff/led_read_fix
Merge pull request #5 from mwelling/master
Merge pull request micropython#5 from rreilink/master
Merge pull request #5 from Teknikio/teknikio/bluebird
Merge pull request micropython#5 from adafruit/master
sharpdisplay: Fix memory corruption across soft-reset
Merge pull request micropython#5 from adafruit/main
py/repl: Don't read past the end of import_str.
py/repl: Don't read past the end of import_str.
mimxrt/1170evk: Refactored clock configuration. (#5)
Merge pull request micropython#5 from AnimaGUS-minerva/1.17-riot
Fix 2 CRITICAL buffer overflow vulnerabilities in WebREPL authentication
py/repl: Don't read past the end of import_str.