How do I debug a built-in Python command, package or module?

Question

I would like to debug some of the basic packages that come with the Python install and/or are built-in packages, including pip and venv.

The desire comes from an error message of file permissions (unable to access a file with an "unprintable file name") some of my team is getting running these commands - see this question for details.

Question

How do you debug the Python source code when trying to catch issues in the main python executable, or when directly running a base python module (see following examples for pip and venv)?

$ python -m pip install --upgrade
$ python -m venv .venv

If it matters, my environment is VSCode, where I am happily able to engage the debugger on any custom script I have written, using the built-in debugger that interacts (I assume) with the main Microsoft Python extension.

pip is a pre-built binary, so if you want to debug that it's most likely going to involve modifying its source code directly and rebuilding the executable. For venv, the source appears to be located in the __init__.py file under <python_install>\lib\venv. In there, you could add import pdb; pdb.set_trace() to start the Python DeBugger (or just breakpoint(), which should work in VSCode). — b_c
– b_c, Commented Jan 20, 2020 at 19:17
@b_c: The pip 'executable' is just a Python script that uses the pkg_resources library to (indirectly) import the pip._internal.cli.main module and runs the main() function in it. Open the path that which pip points to and you'll see it's just text you can add a debug statement to. — Martijn Pieters
– Martijn Pieters, Commented Jan 20, 2020 at 19:18
@MartijnPieters Good to know! I didn't see a pip module when I looked (wasn't expecting it to be in site-packages) and I didn't consider python -m pip would suggest that it does exist as a module somewhere. — b_c
– b_c, Commented Jan 20, 2020 at 20:21

Brett Cannon · Accepted Answer · 2020-01-21 23:24:09Z

11

You will need to set "justMyCode": false in your launch.json for the debugger to trace into third-party code.

answered Jan 21, 2020 at 23:24

Brett Cannon

16.4k3 gold badges54 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Martijn Pieters · Accepted Answer · 2020-01-20 19:27:29Z

6

Start by looking at the source code for those modules; the -m switch looks for a package or module to import first. If it's a package, then Python imports the __main__ module in that package and runs it as the main script. If it is a module, the module itself is imported and run as __main__.

Usually the code is structured such that a function is called you can import directly too. You can then just write a bit of code that imports the same function and calls it the same way the __main__ module would. From there on out it is trivial to run this under a debugger.

E.g. pip is a package, so python -m pip will import pip.__main__ and run that as a script. This then triggers:

from pip._internal.cli.main import main as _main  # isort:skip # noqa

if __name__ == '__main__':
    sys.exit(_main())

to be run. You can do the same in VSCode; import pip._internal.cli.main.main and call it.

You can find the source code for these modules by just importing them and printing out the resulting object:

python -c "import pip; print(pip)"

The representation of a module, if loaded from disk, will include it's filename. If the filename ends in /__init__.py it's a package, so you can also double-check that the __main__.py file exists:

python -c "import pip.__main_; print(pip.__main__)"

You can do the same for the venv module. This one is part of the Python standard library, so the documentation actually links directly to the source code, and the venv.__main__ module just imports venv.main() and calls it.

edited Jan 20, 2020 at 19:27

answered Jan 20, 2020 at 19:13

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

3 Comments

LightCC Over a year ago

Thanks! So find and download the source code - do I also need the "debug binaries" from the main Python installation? I recall checkboxes for those during the original Python installs (on Windows at least).

Martijn Pieters Over a year ago

@LightCC: python -c "import pip; print(pip)" will give you the path to the source code on your harddisk.

Martijn Pieters Over a year ago

@LightCC: you don't need the Python debug build, no. Nothing here is a binary extension, it's all just Python code.

Collectives™ on Stack Overflow

How do I debug a built-in Python command, package or module?

Question

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Question

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related