18 Pymalloc: A Specialized Object Allocator
Pymalloc, a specialized object allocator written by Vladimir Marangozov, was a feature
added to Python 2.1. Pymalloc is intended to be faster than the system malloc() and to have less memory overhead for allocation patterns
typical of Python programs. The allocator uses C's malloc()
function to get large pools of memory and then fulfills smaller memory requests from these
pools.
In 2.1 and 2.2, pymalloc was an experimental feature and wasn't enabled by default; you
had to explicitly enable it when compiling Python by providing the --with-pymalloc
option to the configure script. In 2.3, pymalloc has had further
enhancements and is now enabled by default; you'll have to supply --without-pymalloc
to disable it.
This change is transparent to code written in Python; however, pymalloc may expose bugs
in C extensions. Authors of C extension modules should test their code with pymalloc
enabled, because some incorrect code may cause core dumps at runtime.
There's one particularly common error that causes problems. There are a number of
memory allocation functions in Python's C API that have previously just been aliases for
the C library's malloc() and free(),
meaning that if you accidentally called mismatched functions the error wouldn't be
noticeable. When the object allocator is enabled, these functions aren't aliases of malloc() and free() any more, and
calling the wrong function to free memory may get you a core dump. For example, if memory
was allocated using PyObject_Malloc(), it has to be freed using
PyObject_Free(), not free(). A few
modules included with Python fell afoul of this and had to be fixed; doubtless there are
more third-party modules that will have the same problem.
As part of this change, the confusing multiple interfaces for allocating memory have
been consolidated down into two API families. Memory allocated with one family must not be
manipulated with functions from the other family. There is one family for allocating
chunks of memory and another family of functions specifically for allocating Python
objects.
- To allocate and free an undistinguished chunk of memory use the ``raw memory''
family: PyMem_Malloc(), PyMem_Realloc(),
and PyMem_Free().
- The ``object memory'' family is the interface to the pymalloc facility described
above and is biased towards a large number of ``small'' allocations: PyObject_Malloc, PyObject_Realloc,
and PyObject_Free.
- To allocate and free Python objects, use the ``object'' family PyObject_New(),
PyObject_NewVar(), and PyObject_Del().
Thanks to lots of work by Tim Peters, pymalloc in 2.3 also provides debugging features
to catch memory overwrites and doubled frees in both extension modules and in the
interpreter itself. To enable this support, compile a debugging version of the Python
interpreter by running configure with --with-pydebug.
To aid extension writers, a header file Misc/pymemcompat.h is
distributed with the source to Python 2.3 that allows Python extensions to use the 2.3
interfaces to memory allocation while compiling against any version of Python since 1.5.2.
You would copy the file from Python's source distribution and bundle it with the source of
your extension.
|
