Python Cookbook 2Nd Edition Jun 1002005 [Electronic resources] نسخه متنی

Recipe 17.10. Debugging Memory Problems

Credit: Will Ware

Problem

You're developing C
extensions, and you experience memory problems. You suspect
mismanagement of reference counts and want to check whether your C
extension code is correctly managing reference counts.

Solution

To chase these problems in an optimal way, you need to alter
Python's sources and rebuild Python. Specifically,
add the following function in Objects/object.c,
immediately before the _Py_PrintReferences
function:

void
_Py_CountReferences(FILE *fp)
{
int nr, no;
PyObject *op;
for (nr = no = 0, op = refchain._ob_next;
op != &refchain;
op = op->_ob_next, nr += op->ob_refcnt, no += 1)
{ }
fprintf(fp, "%d refs (%d), %d objs\n", nr, _Py_RefTotal, no);
}

I place the following macros in my C extensions:

#if defined(Py_DEBUG) || defined(DEBUG)
extern void _Py_CountReferences(FILE*);
#define CURIOUS(x) { fprintf(stderr,
 _ _FILE_ _ ":%d ", _ _LINE_ _); x; }
#else
#define CURIOUS(x)
#endif
#define MARKER( )        CURIOUS(fprintf(stderr, "\n"))
#define DESCRIBE(x)     CURIOUS(fprintf(stderr, "  " #x "=%d\n", x))
#define DESCRIBE_HEX(x) CURIOUS(fprintf(stderr, "  " #x "=%08x\n", x))
#define COUNTREFS( )     CURIOUS(_Py_CountReferences(stderr))

To debug, I rebuild Python using make
OPT="-DPy_DEBUG", which causes the code
under Py_TRACE_REFS to be built. My own makefile
for my extensions uses the same trick by including these lines:

debug:
make clean; make OPT="-g -DPy_DEBUG" all
CFLAGS = $(OPT) -fpic -O2 -I/usr/local/include -I/usr/include/python2.3

Discussion

When I'm developing C extensions and running into
memory problems, I find that the typical cause is mismanagement of
reference counts, particularly misuse of Py_INCREF
and Py_DECREF, as well as forgetfulness of the
reference-count effects of functions like
Py_BuildValue,
PyArg_ParseTuple, and
PyTuple/List_SetItem/GetItem. The Python sources
offer help with this problem (search for
Py_TRACE_REFS), and function
sys.getrefcounts in the Python Standard Library is
also helpful. Nevertheless, it's useful to add this
recipe's function in
Objects/object.c just before
_Py_PrintReferences.

Unlike _Py_PrintReferences, this
recipe's _Py_CountReferences
function prints only the totals of all the refcounts and number of
objects in the system, so it can be sensibly called, even in loops
that repeat millions of times, while
_Py_PrintReferences would print out way too much
stuff to be useful. The information printed by
_Py_CountReferences can help you identify errantly
wandering Py_INCREFs and
Py_DECREFs. _Py_CountReferences
plays it safe by performing its own counts of objects references,
which it prints side by side with the
"official" count of references that
Python itself maintains (when compiled for debugging) as global
variable _Py_RefTotal. Should any discrepancy
arise, you know something deeply wrong is going
on.

When I suspect that one of my C-coded functions is responsible for
memory problems, I liberally sprinkle the suspect function with calls
to the COUNTREFS macro. Doing so allows me to keep
track of exactly how many references are being created or destroyed
as I go through my function. This information is particularly useful
in tight loops, in which dumb mistakes can cause reference counts to
grow ridiculously fast. Also, reference counts that shrink too fast
(because of overzealous use of Py_DECREF) can
cause core dumps because the memory for objects that should still
exist has been reallocated for new objects.

See Also

The only documentation in this case is Python's own
source code. Use the source, Luke!