Repository navigation
Switch to routing allocations through Python's allocators on py35 #4663
Description
Python 3.5's memory allocator system now exposes calloc: http://bugs.python.org/issue21233
...which means that #if we are building for 3.5, we can route all allocations through Python's memory allocator system, and get memory tracing/profiling out of the box.
So we should do this.
Activity
juliantaylor commentedon May 4, 2014
as already mentioned on the mailing list I'm -1 on doing it outside of a debug mode as we don't want to vendor lockin to pythons allocators
if python does not want to extend the tracing api maybe we could ask for a hook to tracemalloc.start which triggers numpy to change its allocators to tracing ones.
njsmith commentedon May 11, 2014
Python-dev seems open to extending the tracing API, and I'm not sure how a
hook like that could work (we can't switch allocators at runtime, can we?).
What would you need to see to switch to +1? A generic "btw I
allocated/freed some memory" API in Python?
On Sun, May 4, 2014 at 1:40 PM, Julian Taylor [email protected]:
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
njsmith commentedon Jun 4, 2014
@juliantaylor: ping
juliantaylor commentedon Jun 5, 2014
right changing allocators at runtime is no good idea (unless we add deletion function pointers to each array object?)
its actually more a -0.5 if a PR shows up and someone wants to merge it I'll not object. We haven't got any special allocators in numpy yet anyway.
njsmith commentedon Jun 5, 2014
OK, good to know.
If we do want to future-proof ourselves by getting a better Python
allocation API, though, then now is the time -- once 3.5 is released it
will be much more of a hassle.
Do you think we should try asking them to also implement another API like
PyMem_NotifyMalloc(const void *, size_t)
PyMem_NotifyRealloc(const void * old_ptr, const void * new_ptr, size_t
new_size)
PyMem_NotifyFree(const void *)
?
On Thu, Jun 5, 2014 at 2:02 PM, Julian Taylor [email protected]
wrote:
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
juliantaylor commentedon Jun 5, 2014
I think that would make sense (+ context argument to allow tracking other types of resources like file descriptors or file based memory maps)
this would allow extensions to call non python-aware code that allocates memory and still track it. E.g. when numpy allocates its arrays via libdynd (assuming its allocator is not pluggable to pythons)
njsmith commentedon Dec 10, 2014
FTR, the CPython bug where discussion of tracemalloc extensions is happening is currently: http://bugs.python.org/issue18835.
ENH: add support for python3.6 memory tracing