peps/pep-0454.txt at main · python/peps

Cannot retrieve contributors at this time

563 lines (377 sloc) 17.9 KB

	PEP: 454
	Title: Add a new tracemalloc module to trace Python memory allocations
	Version: $Revision$
	Last-Modified: $Date$
	Author: Victor Stinner <vstinner@python.org>
	BDFL-Delegate: Charles-François Natali <cf.natali@gmail.com>
	Status: Final
	Type: Standards Track
	Content-Type: text/x-rst
	Created: 03-Sep-2013
	Python-Version: 3.4
	Resolution: https://mail.python.org/pipermail/python-dev/2013-November/130491.html


	Abstract
	========

	This PEP proposes to add a new ``tracemalloc`` module to trace memory
	blocks allocated by Python.


	Rationale
	=========

	Classic generic tools like Valgrind can get the C traceback where a
	memory block was allocated. Using such tools to analyze Python memory
	allocations does not help because most memory blocks are allocated in
	the same C function, in ``PyMem_Malloc()`` for example. Moreover, Python
	has an allocator for small objects called "pymalloc" which keeps free
	blocks for efficiency. This is not well handled by these tools.

	There are debug tools dedicated to the Python language like ``Heapy``
	``Pympler`` and ``Meliae`` which lists all alive objects using the
	garbage collector module (functions like ``gc.get_objects()``,
	``gc.get_referrers()`` and ``gc.get_referents()``), compute their size
	(ex: using ``sys.getsizeof()``) and group objects by type. These tools
	provide a better estimation of the memory usage of an application. They
	are useful when most memory leaks are instances of the same type and
	this type is only instantiated in a few functions. Problems arise when
	the object type is very common like ``str`` or ``tuple``, and it is hard
	to identify where these objects are instantiated.

	Finding reference cycles is also a difficult problem. There are
	different tools to draw a diagram of all references. These tools
	cannot be used on large applications with thousands of objects because
	the diagram is too huge to be analyzed manually.


	Proposal
	========

	Using the customized allocation API from :pep:`445`, it becomes easy to
	set up a hook on Python memory allocators. A hook can inspect Python
	internals to retrieve Python tracebacks. The idea of getting the current
	traceback comes from the faulthandler module. The faulthandler dumps
	the traceback of all Python threads on a crash, here is the idea is to
	get the traceback of the current Python thread when a memory block is
	allocated by Python.

	This PEP proposes to add a new ``tracemalloc`` module, a debug tool
	to trace memory blocks allocated by Python. The module provides the
	following information:

	* Traceback where an object was allocated
	* Statistics on allocated memory blocks per filename and per line
	number: total size, number and average size of allocated memory blocks
	* Computed differences between two snapshots to detect memory leaks

	The API of the tracemalloc module is similar to the API of the faulthandler
	module: ``enable()`` / ``start()``, ``disable()`` / ``stop()`` and
	``is_enabled()`` / ``is_tracing()`` functions, an environment variable
	(``PYTHONFAULTHANDLER`` and ``PYTHONTRACEMALLOC``), and a ``-X`` command line
	option (``-X faulthandler`` and ``-X tracemalloc``). See the `documentation of
	the faulthandler module <http://docs.python.org/3/library/faulthandler.html>`_.

	The idea of tracing memory allocations is not new. It was first
	implemented in the PySizer project in 2005. PySizer was implemented
	differently: the traceback was stored in frame objects and some Python
	types were linked the trace with the name of object type. PySizer patch
	on CPython adds an overhead on performances and memory footprint, even if
	the PySizer was not used. tracemalloc attaches a traceback to the
	underlying layer, to memory blocks, and has no overhead when the module
	is not tracing memory allocations.

	The tracemalloc module has been written for CPython. Other
	implementations of Python may not be able to provide it.


	API
	===

	To trace most memory blocks allocated by Python, the module should be
	started as early as possible by setting the ``PYTHONTRACEMALLOC``
	environment variable to ``1``, or by using ``-X tracemalloc`` command
	line option. The ``tracemalloc.start()`` function can be called at
	runtime to start tracing Python memory allocations.

	By default, a trace of an allocated memory block only stores the most
	recent frame (1 frame). To store 25 frames at startup: set the
	``PYTHONTRACEMALLOC`` environment variable to ``25``, or use the ``-X
	tracemalloc=25`` command line option. The ``set_traceback_limit()``
	function can be used at runtime to set the limit.


	Functions
	---------

	``clear_traces()`` function:

	Clear traces of memory blocks allocated by Python.

	See also ``stop()``.


	``get_object_traceback(obj)`` function:

	Get the traceback where the Python object obj was allocated.
	Return a ``Traceback`` instance, or ``None`` if the ``tracemalloc``
	module is not tracing memory allocations or did not trace the
	allocation of the object.

	See also ``gc.get_referrers()`` and ``sys.getsizeof()`` functions.


	``get_traceback_limit()`` function:

	Get the maximum number of frames stored in the traceback of a trace.

	The ``tracemalloc`` module must be tracing memory allocations to get
	the limit, otherwise an exception is raised.

	The limit is set by the ``start()`` function.


	``get_traced_memory()`` function:

	Get the current size and maximum size of memory blocks traced by the
	``tracemalloc`` module as a tuple: ``(size: int, max_size: int)``.


	``get_tracemalloc_memory()`` function:

	Get the memory usage in bytes of the ``tracemalloc`` module used to
	store traces of memory blocks. Return an ``int``.


	``is_tracing()`` function:

	``True`` if the ``tracemalloc`` module is tracing Python memory
	allocations, ``False`` otherwise.

	See also ``start()`` and ``stop()`` functions.


	``start(nframe: int=1)`` function:

	Start tracing Python memory allocations: install hooks on Python
	memory allocators. Collected tracebacks of traces will be limited to
	nframe frames. By default, a trace of a memory block only stores
	the most recent frame: the limit is ``1``. nframe must be greater
	or equal to ``1``.

	Storing more than ``1`` frame is only useful to compute statistics
	grouped by ``'traceback'`` or to compute cumulative statistics: see
	the ``Snapshot.compare_to()`` and ``Snapshot.statistics()`` methods.

	Storing more frames increases the memory and CPU overhead of the
	``tracemalloc`` module. Use the ``get_tracemalloc_memory()``
	function to measure how much memory is used by the ``tracemalloc``
	module.

	The ``PYTHONTRACEMALLOC`` environment variable
	(``PYTHONTRACEMALLOC=NFRAME``) and the ``-X`` ``tracemalloc=NFRAME``
	command line option can be used to start tracing at startup.

	See also ``stop()``, ``is_tracing()`` and ``get_traceback_limit()``
	functions.


	``stop()`` function:

	Stop tracing Python memory allocations: uninstall hooks on Python
	memory allocators. Clear also traces of memory blocks allocated by
	Python

	Call ``take_snapshot()`` function to take a snapshot of traces
	before clearing them.

	See also ``start()`` and ``is_tracing()`` functions.


	``take_snapshot()`` function:

	Take a snapshot of traces of memory blocks allocated by Python.
	Return a new ``Snapshot`` instance.

	The snapshot does not include memory blocks allocated before the
	``tracemalloc`` module started to trace memory allocations.

	Tracebacks of traces are limited to ``get_traceback_limit()``
	frames. Use the nframe parameter of the ``start()`` function to
	store more frames.

	The ``tracemalloc`` module must be tracing memory allocations to
	take a snapshot, see the ``start()`` function.

	See also the ``get_object_traceback()`` function.


	Filter
	------

	``Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)`` class:

	Filter on traces of memory blocks.

	See the ``fnmatch.fnmatch()`` function for the syntax of
	filename_pattern. The ``'.pyc'`` and ``'.pyo'`` file extensions
	are replaced with ``'.py'``.

	Examples:

	* ``Filter(True, subprocess.__file__)`` only includes traces of the
	``subprocess`` module
	* ``Filter(False, tracemalloc.__file__)`` excludes traces of the
	``tracemalloc`` module
	* ``Filter(False, "<unknown>")`` excludes empty tracebacks

	``inclusive`` attribute:

	If inclusive is ``True`` (include), only trace memory blocks
	allocated in a file with a name matching ``filename_pattern`` at
	line number ``lineno``.

	If inclusive is ``False`` (exclude), ignore memory blocks
	allocated in a file with a name matching ``filename_pattern`` at
	line number ``lineno``.

	``lineno`` attribute:

	Line number (``int``) of the filter. If lineno is ``None``, the
	filter matches any line number.

	``filename_pattern`` attribute:

	Filename pattern of the filter (``str``).

	``all_frames`` attribute:

	If all_frames is ``True``, all frames of the traceback are
	checked. If all_frames is ``False``, only the most recent frame is
	checked.

	This attribute is ignored if the traceback limit is less than ``2``.
	See the ``get_traceback_limit()`` function and
	``Snapshot.traceback_limit`` attribute.


	Frame
	-----

	``Frame`` class:

	Frame of a traceback.

	The ``Traceback`` class is a sequence of ``Frame`` instances.

	``filename`` attribute:

	Filename (``str``).

	``lineno`` attribute:

	Line number (``int``).


	Snapshot
	--------

	``Snapshot`` class:

	Snapshot of traces of memory blocks allocated by Python.

	The ``take_snapshot()`` function creates a snapshot instance.

	``compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False)`` method:

	Compute the differences with an old snapshot. Get statistics as a
	sorted list of ``StatisticDiff`` instances grouped by group_by.

	See the ``statistics()`` method for group_by and cumulative
	parameters.

	The result is sorted from the biggest to the smallest by: absolute
	value of ``StatisticDiff.size_diff``, ``StatisticDiff.size``,
	absolute value of ``StatisticDiff.count_diff``, ``Statistic.count``
	and then by ``StatisticDiff.traceback``.


	``dump(filename)`` method:

	Write the snapshot into a file.

	Use ``load()`` to reload the snapshot.


	``filter_traces(filters)`` method:

	Create a new ``Snapshot`` instance with a filtered ``traces``
	sequence, filters is a list of ``Filter`` instances. If filters
	is an empty list, return a new ``Snapshot`` instance with a copy of
	the traces.

	All inclusive filters are applied at once, a trace is ignored if no
	inclusive filters match it. A trace is ignored if at least one
	exclusive filter matches it.


	``load(filename)`` classmethod:

	Load a snapshot from a file.

	See also ``dump()``.


	``statistics(group_by: str, cumulative: bool=False)`` method:

	Get statistics as a sorted list of ``Statistic`` instances grouped
	by group_by:

	===================== ========================
	group_by description
	===================== ========================
	``'filename'`` filename
	``'lineno'`` filename and line number
	``'traceback'`` traceback
	===================== ========================

	If cumulative is ``True``, cumulate size and count of memory
	blocks of all frames of the traceback of a trace, not only the most
	recent frame. The cumulative mode can only be used with group_by
	equals to ``'filename'`` and ``'lineno'`` and ``traceback_limit``
	greater than ``1``.

	The result is sorted from the biggest to the smallest by:
	``Statistic.size``, ``Statistic.count`` and then by
	``Statistic.traceback``.


	``traceback_limit`` attribute:

	Maximum number of frames stored in the traceback of ``traces``:
	result of the ``get_traceback_limit()`` when the snapshot was taken.

	``traces`` attribute:

	Traces of all memory blocks allocated by Python: sequence of
	``Trace`` instances.

	The sequence has an undefined order. Use the
	``Snapshot.statistics()`` method to get a sorted list of statistics.


	Statistic
	---------

	``Statistic`` class:

	Statistic on memory allocations.

	``Snapshot.statistics()`` returns a list of ``Statistic`` instances.

	See also the ``StatisticDiff`` class.

	``count`` attribute:

	Number of memory blocks (``int``).

	``size`` attribute:

	Total size of memory blocks in bytes (``int``).

	``traceback`` attribute:

	Traceback where the memory block was allocated, ``Traceback``
	instance.


	StatisticDiff
	-------------

	``StatisticDiff`` class:

	Statistic difference on memory allocations between an old and a new
	``Snapshot`` instance.

	``Snapshot.compare_to()`` returns a list of ``StatisticDiff``
	instances. See also the ``Statistic`` class.

	``count`` attribute:

	Number of memory blocks in the new snapshot (``int``): ``0`` if the
	memory blocks have been released in the new snapshot.

	``count_diff`` attribute:

	Difference of number of memory blocks between the old and the new
	snapshots (``int``): ``0`` if the memory blocks have been allocated
	in the new snapshot.

	``size`` attribute:

	Total size of memory blocks in bytes in the new snapshot (``int``):
	``0`` if the memory blocks have been released in the new snapshot.

	``size_diff`` attribute:

	Difference of total size of memory blocks in bytes between the old
	and the new snapshots (``int``): ``0`` if the memory blocks have
	been allocated in the new snapshot.

	``traceback`` attribute:

	Traceback where the memory blocks were allocated, ``Traceback``
	instance.


	Trace
	-----

	``Trace`` class:

	Trace of a memory block.

	The ``Snapshot.traces`` attribute is a sequence of ``Trace``
	instances.

	``size`` attribute:

	Size of the memory block in bytes (``int``).

	``traceback`` attribute:

	Traceback where the memory block was allocated, ``Traceback``
	instance.


	Traceback
	---------

	``Traceback`` class:

	Sequence of ``Frame`` instances sorted from the most recent frame to
	the oldest frame.

	A traceback contains at least ``1`` frame. If the ``tracemalloc`` module
	failed to get a frame, the filename ``"<unknown>"`` at line number ``0`` is
	used.

	When a snapshot is taken, tracebacks of traces are limited to
	``get_traceback_limit()`` frames. See the ``take_snapshot()``
	function.

	The ``Trace.traceback`` attribute is an instance of ``Traceback``
	instance.


	Rejected Alternatives
	=====================

	Log calls to the memory allocator
	---------------------------------

	A different approach is to log calls to ``malloc()``, ``realloc()`` and
	``free()`` functions. Calls can be logged into a file or send to another
	computer through the network. Example of a log entry: name of the
	function, size of the memory block, address of the memory block, Python
	traceback where the allocation occurred, timestamp.

	Logs cannot be used directly, getting the current status of the memory
	requires to parse previous logs. For example, it is not possible to get
	directly the traceback of a Python object, like
	``get_object_traceback(obj)`` does with traces.

	Python uses objects with a very short lifetime and so makes an extensive
	use of memory allocators. It has an allocator optimized for small
	objects (less than 512 bytes) with a short lifetime. For example, the
	Python test suites calls ``malloc()``, ``realloc()`` or ``free()``
	270,000 times per second in average. If the size of log entry is 32
	bytes, logging produces 8.2 MB per second or 29.0 GB per hour.

	The alternative was rejected because it is less efficient and has less
	features. Parsing logs in a different process or a different computer is
	slower than maintaining traces on allocated memory blocks in the same
	process.


	Prior Work
	==========

	* `Python Memory Validator
	<http://www.softwareverify.com/python/memory/index.html>`_ (2005-2013):
	commercial Python memory validator developed by Software Verification.
	It uses the Python Reflection API.
	* `PySizer <http://pysizer.8325.org/>`_: Google Summer of Code 2005 project by
	Nick Smallbone.
	* `Heapy
	<http://guppy-pe.sourceforge.net/>`_ (2006-2013):
	part of the Guppy-PE project written by Sverker Nilsson.
	* Draft PEP: `Support Tracking Low-Level Memory Usage in CPython
	<http://svn.python.org/projects/python/branches/bcannon-sandboxing/PEP.txt>`_
	(Brett Canon, 2006)
	* Muppy: project developed in 2008 by Robert Schuppenies.
	* `asizeof <http://code.activestate.com/recipes/546530/>`_:
	a pure Python module to estimate the size of objects by Jean
	Brouwers (2008).
	* `Heapmonitor <http://www.scons.org/wiki/LudwigHaehne/HeapMonitor>`_:
	It provides facilities to size individual objects and can track all objects
	of certain classes. It was developed in 2008 by Ludwig Haehne.
	* `Pympler <http://code.google.com/p/pympler/>`_ (2008-2011):
	project based on asizeof, muppy and HeapMonitor
	* `objgraph <http://mg.pov.lt/objgraph/>`_ (2008-2012)
	* `Dozer <https://pypi.python.org/pypi/Dozer>`_: WSGI Middleware version
	of the CherryPy memory leak debugger, written by Marius Gedminas (2008-2013)
	* `Meliae
	<https://pypi.python.org/pypi/meliae>`_:
	Python Memory Usage Analyzer developed by John A Meinel since 2009
	* `gdb-heap <https://fedorahosted.org/gdb-heap/>`_: gdb script written in
	Python by Dave Malcolm (2010-2011) to analyze the usage of the heap memory
	* `memory_profiler <https://pypi.python.org/pypi/memory_profiler>`_:
	written by Fabian Pedregosa (2011-2013)
	* `caulk <https://github.com/smartfile/caulk/>`_: written by Ben Timby in 2012

	See also `Pympler Related Work
	<http://pythonhosted.org/Pympler/related.html>`_.


	Links
	=====

	tracemalloc:

	* `#18874: Add a new tracemalloc module to trace Python
	memory allocations <http://bugs.python.org/issue18874>`_
	* `pytracemalloc on PyPI
	<https://pypi.python.org/pypi/pytracemalloc>`_


	Copyright
	=========

	This document has been placed in the public domain.



	..
	Local Variables:
	mode: indented-text
	indent-tabs-mode: nil
	sentence-end-double-space: t
	fill-column: 70
	coding: utf-8
	End: