77

I am trying to memcheck a C python extension I am writing, but I'm having trouble setting up valgrind to work with python. I would really appreciate some advice. Just for context, this is Ubuntu 13.10, python 2.7.5+, and valgrind 3.8.1.

As per recommendation from Readme.valgrind I did the following.

1) Downloaded the python source with

sudo apt-get build-dep python2.7
apt-get source python2.7

2) Applied the code patch, i.e. "Uncomment Py_USING_MEMORY_DEBUGGER in Objects/obmalloc.c".

3) Applied the suppression patch, i.e. "Uncomment the lines in Misc/valgrind-python.supp that suppress the warnings for PyObject_Free and PyObject_Realloc"

4) Compiled python with 

./configure --prefix=/home/dejan/workspace/python --without-pymalloc
make -j4 install

Note that I did both 2 and 3, while README.valgrind says to do 2 or 3... more can't hurt.

Now, let's test this on some sample python code in test.py

print "Test"

Let's run valgrind on python with this script

valgrind --tool=memcheck --leak-check=full --suppressions=python2.7-2.7.5/Misc/valgrind-python.supp bin/python test.py

Unexpectedly, there is still loads of reports from valgrind, with the first one being (and many more following) 

==27944== HEAP SUMMARY:
==27944==     in use at exit: 857,932 bytes in 5,144 blocks  
==27944==   total heap usage: 22,766 allocs, 17,622 frees, 4,276,934 bytes allocated
==27944== 
==27944== 38 bytes in 1 blocks are possibly lost in loss record 24 of 1,343
==27944==    at 0x4C2A2DB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==27944==    by 0x46B8DD: PyString_FromString (stringobject.c:143)
==27944==    by 0x439631: PyFile_FromFile (fileobject.c:157)
==27944==    by 0x4E9B4A: _PySys_Init (sysmodule.c:1383)
==27944==    by 0x4E29E9: Py_InitializeEx (pythonrun.c:222)
==27944==    by 0x4154B4: Py_Main (main.c:546)
==27944==    by 0x577DDE4: (below main) (libc-start.c:260)

Am I doing something wrong? Is there a way to valgrind a python script that doesn't leak and get clean valgrind output?

Improve this question
3
  • Have you seen stackoverflow.com/questions/1519276/… Commented Nov 21, 2013 at 5:22
  • @VooDooNOFX I've searched stackoverflow and beyond and all answers basically say that I should do what I described I did -- compile python or use a supress. This does get rid of the custom allocation problematic reports such as PyObject_Free (this is the one in the question you linked), but the reports I'm getting are not of that sort. Commented Nov 21, 2013 at 5:28
  • In order to test my c extension, I compiled debug python with instructions provided in the Answer. And I rebuild my extension with debug version python, and it got failure result. I used setuptools extension build the extension, and it works well under system environment. The build flags are very different. Commented Dec 14, 2016 at 8:50

2 Answers 2

Reset to default
71

Since python 3.6, there's a PYTHONMALLOC environment variable which is available in release builds, without needing to recompile.

PYTHONMALLOC=malloc python3 foobar.py

This will disable pymalloc and just use the libc malloc directly, making it valgrind-friendly. This is equivalent to --without-pymalloc (and it is just as slow)

If valgrind is too slow, other values can be helpful. PYTHONMALLOC=debug and PYTHONMALLOC=malloc_debug add debug hooks on top of the default and the libc allocators respectively. Their effects, from the docs:

  • Newly allocated memory is filled with the byte 0xCB
  • Freed memory is filled with the byte 0xDB
  • Detect violations of the Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
  • Detect writes before the start of a buffer (buffer underflows)
  • Detect writes after the end of a buffer (buffer overflows)
  • Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.

This will catch some uninitialized reads, some use after free, some buffer under/overflows, etc, but won't report leaks and won't touch memory that isn't allocated through python (When using glibc, the MALLOC_PERTURB_ and MALLOC_CHECK_ environment variables might help there)

See also:

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

57

I found the answer here.

Python also needs to be compiled in debug mode, i.e. 

./configure --prefix=/home/dejan/workspace/python --without-pymalloc --with-pydebug --with-valgrind

In addition, numpy has a suppresion file that gets rid of the extra valgrind warnings.

Improve this answer

6 Comments

Thanks for doing the research for everyone! Do you know if one should recompile python once done using valgrind? And, more importantly - how to do it?
What works for me is compiling/installing a valgrind-compatible version of python, then basing a virtualenv on that. That gives me the convenient activate script. Even better with virtualenvwrapper
does this make every python command call run with valgrind? I'm noticing a significant slow down installing packages with pip
@JamesRobertAlbert From the README: Note: configuring python --without-pymalloc makes Python run much slower, especially when running under Valgrind. ... It seems to take about 5 times longer to run --without-pymalloc
@YairDaon you don't have to recompile Python because you create a new interpreter. Related : stackoverflow.com/questions/36697863/….
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.