Here is a quick summary of the steps required to prepare a build of Python 2.7 for use with valgrind memcheck tool.

Valgrind is an indispensable tool for discovering memory safety issues, and for quickly tracking down the source of many kinds of memory corruption, especially in C code the user may not be familiar with. If your Python program is crashing with a segmentation fault, executing it using an instrumented build running under valgrind can very often pinpoint the exact cause of the fault, if the fault is due to a memory safety violation.

I needed this setup as today I decided to finalize an old incomplete patch to py-lmdb, to allow passing fixed-size integer keys and values to the database, useful for reducing storage and performance overhead. Today valgrind is useful since with the patch enabled, LMDB appears to disable some of its sanity checks around the size of key/value pointers passed to it, and so in order to maintain memory safety in the binding, I’d like to discover any places where I’ve forgotten to add a corresponding size check.

Grab dependencies

Grab a copy of the Python source tree, and check out the 2.7 branch:

$ sudo apt-get install build-essential valgrind $ pip install Mercurial $ cd /home/dmw/src $ hg clone https://hg.python.org/cpython/ $ cd cpython $ hg checkout 2.7

Build Python

Build a copy of Python with pymalloc disabled. This is required since when pymalloc is enabled, Python asks for large chunks of memory at a time from the C library, and so valgrind is unable to discover allocation boundaries using its magical instrumented malloc and free implementations that it injects into the target process.

Additionally build with Py_DEBUG to cause reference counts to be more thoroughly checked. Do not --enable-shared , as this produces a Python binary that will link to your system-installed libpython.so unless you remember to set LD_LIBRARY_PATH every time you run the debug build.

There is also a --with-valgrind option that prepares a build where pymalloc is disabled at runtime when running under valgrind, but means the Python binary will change behaviour depending on whether it’s running directly, under valgrind, or under gdb, a fabulous source of seeming heisenbugs when the user (me) is half asleep, so I prefer to avoid it.

Finally, set --prefix to a subdirectory so that when we come to install pip and setuptools, these tools are not installed into /usr/local or any other global filesystem path.

$ ./configure --without-pymalloc --with-pydebug --prefix=`pwd`/build $ make -j 16

Test The Build

We do not need to install the built tree in order to make use of it, simply invoke ./python to run it in-place:

$ ./python Python 2.7.12+ (2.7:11a9bca71528, Oct 17 2016, 16:21:52) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> [59613 refs] [27063 refs]

The presence of [59613 refs] indicates the binary you are running was built with --with-pydebug .

Try It Under Valgrind

Now let’s try running the binary by itself under valgrind, to ensure our configuration is sane, and that valgrind is not reporting errors because we screwed up some step.

$ time valgrind ./python -S -c 'print 123' ==12914== Memcheck, a memory error detector ==12914== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==12914== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info ==12914== Command: ./python -S -c print\ 123 ==12914== 123 [10871 refs] ==12914== ==12914== HEAP SUMMARY: ==12914== in use at exit: 470,972 bytes in 2,963 blocks ==12914== total heap usage: 9,362 allocs, 6,399 frees, 1,626,675 bytes allocated ==12914== ==12914== LEAK SUMMARY: ==12914== definitely lost: 0 bytes in 0 blocks ==12914== indirectly lost: 0 bytes in 0 blocks ==12914== possibly lost: 77,979 bytes in 310 blocks ==12914== still reachable: 392,993 bytes in 2,653 blocks ==12914== suppressed: 0 bytes in 0 blocks ==12914== Rerun with --leak-check=full to see details of leaked memory ==12914== ==12914== For counts of detected and suppressed errors, rerun with: -v ==12914== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) real 0m2.201s user 0m2.168s sys 0m0.032s

We see already valgrind is reporting memory has been leaked. This is normal, as there are a variety of small structures allocated by CPython at startup that it never frees, usually because doing so would be too difficult.

You may also notice this Python build is much slower! Not only is valgrind slowing performance quite a lot, but the absense of pymalloc vastly hurts performance of the built interpreter. Why is this? Well at least, pymalloc knows more about the allocation styles that Python needs in comparison to the C library allocator, and so can be specialized for it, but perhaps more importantly, unlike the C allocator, pymalloc knows it will always run under the Global Interpreter Lock, so it does not require any additional mutexes for its data structures.

Running with --leak-check=full should indicate most of these “leaks” have PyType_Ready somewhere in the call stack, indicating they are heap allocations to support the static Py_Type structures that implement a variety of built-in functionality (such as the str object).

Install pip and setuptools

$ ./python -m ensurepip

We can now install any Python packages we need to support our test environment. If your application is complex, it might be desirable to also ./python -m pip install virtualenv along with make install to get a working virtualenv. I do not need it now so we will skip this step.

Build your application with the debug Python build

Since we built a completely new Python interpreter with debugging enabled, its header definitions will no longer match the headers that were in use by any C extensions you may have previously built. This is important, since the critical macros Py_INCREF and Py_DECREF change their implementation significantly in debug mode. For that reason, you must rebuild any third party modules your application is using with the new Python build.

In my case, I am not debugging an application, just an extension module, so I will only rebuild it, and add a little symlink so I do not need to install it to --prefix every time I rebuild it.

$ cd /home/dmw/src/py-lmdb $ git clean -dfx $ /home/dmw/src/cpython/python setup.py build running build running build_py ... [91959 refs] $ cd lmdb $ ln -s ../build/lib.linux-x86_64-2.7-pydebug/lmdb/cpython.so

Trigger behaviour leading to the scenario you wish to debug

Self-explanatory. In my case, running py-lmdb’s tests is sufficient to reach the code paths I suspect might be damaged, so I’ll install py.test and run the tests under vaglrind.

To demonstrate what a valgrind error looks like, I’ve injected some obviously buggy code into cpython.c :

{ printf("hello

"); char *x = malloc(3); setenv("X", x, 0); }

This allocates some uninitialized memory, then attempts to set the X environment variable to the content of that memory. It is a bug since the memory could contain any random junk, and accessing uninitialized memory in C is “undefined behaviour”, i.e. the specification allows for your program to crash, your computer to catch fire, or the moon to come crashing into the earth.

Now let’s see how the error looks under valgrind:

$ /home/dmw/src/cpython/python -m pip install pytest $ PYTHONPATH=. valgrind /home/dmw/src/cpython/python -m pytest tests/txn_test.py ==29422== Memcheck, a memory error detector ==29422== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==29422== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info ==29422== Command: /home/dmw/src/cpython/python -m pytest tests/txn_test.py ==29422== =============================================================================== test session starts ================================================================================ platform linux2 -- Python 2.7.12+, pytest-3.0.3, py-1.4.31, pluggy-0.4.0 rootdir: /home/dmw/src/py-lmdb, inifile: collected 46 items tests/txn_test.py ==29422== Conditional jump or move depends on uninitialised value(s) ==29422== at 0x4C30759: setenv (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==29422== by 0xD1A152F: trans_put (cpython.c:3271) ==29422== by 0x562798: PyCFunction_Call (methodobject.c:85) ==29422== by 0x4D6457: call_function (ceval.c:4350) ==29422== by 0x4D1281: PyEval_EvalFrameEx (ceval.c:2987) ==29422== by 0x4D3C57: PyEval_EvalCodeEx (ceval.c:3582) ==29422== by 0x4D6A31: fast_function (ceval.c:4445) ==29422== by 0x4D6632: call_function (ceval.c:4370) ==29422== by 0x4D1281: PyEval_EvalFrameEx (ceval.c:2987) ==29422== by 0x4D3C57: PyEval_EvalCodeEx (ceval.c:3582) ==29422== by 0x55FE44: function_call (funcobject.c:523) ==29422== by 0x420F28: PyObject_Call (abstract.c:2547) ==29422==

From the stack trace, we can see cpython.c:3271 is mentioned, corresponding to the source line where our erroneous setenv() call appears. It is very obvious from this output that uninitialized data was accessed, and who accessed it.

valgrind –db-attach

There is one final crucial flag to know about for valgrind, and that is --db-attach . This causes valgrind to immediately halt the Python interpreter when an error is detected, and drop you into a gdb debugger shell, where you can inspect the program using a regular debugger with the exact state of memory preserved at the point where the error occurred.

Don’t forget to apt-get install gdb before trying it out.

Happy debugging!

Valgrind catches a huge variety of memory violations, this simple example only demonstrates one. A full tutorial on these errors would be hard to write, as I doubt I’ve even seen them all. As always, better understanding comes through practice, so next you catch a Python segmentation fault, perhaps try this route to see if you can diagnose or fix it for yourself.