Extending & Embedding Python Using C

Extending & Embedding Python Using C - A Module Using Linux

Written by Mike James

Monday, 06 May 2024

Article Index
Extending & Embedding Python Using C - A Module Using Linux
GCC and VS Code
Building a Python Extension
Testing the Module
Multimodal Debugging

Page 5 of 5

Multimodal Debugging

This is an advanced topic and you can skip it and come back to it when you need to debug an extension – you will need to find out how run a debugger on an extension at some point. Of course, the problem is that there are two programming languages involved – Python and C. We need to use two debuggers we need multimodal debugging.

The Python debugger can be invoked in the usual way when the Python program is run, but the C debugger can only be invoked when our C program is doing something. It has to be attached to the Python system running in response to starting the Python program. Of course, we can’t attach the C debugger until the Python program is running and, if we just let the Python program run, it will end before we have a chance to attach the debugger. The solution is that we have to pause the Python program while it is debugging, then start the C debugger and finally start the Python program again.

We need to set up VS Code with two debugging profiles. Using the standard gdb debugger the launch.json file is:

{
    "version": "0.2.0",
    "configurations": [
      {
          "name": "Python: Current File",
          "type": "python",
         "request": "launch",
          "program": "${file}",
          "console": "integratedTerminal"
      },
      {
          "name": "(gdb) Attach",
          "type": "cppdbg",
          "request": "attach",
          "program": "/usr/local/bin/python3.11",
          "processId": "${command:pickProcess}",
          "MIMode": "gdb",
          "setupCommands": [
              {
        "description": "Enable pretty-printing for gdb",
        "text": "-enable-pretty-printing",
        "ignoreFailures": true
             }
          ]
      }
    ]
  }

You have to change "program": to be the path to the Python interpreter. The big problem is that you have to set the processId of the Python process and this isn’t something you can know or set in advance. The solution is that the pickProcess command gives you a drop-down list from which you can select the process and this is not always easy.

With this launch.json the procedure to debug an extension is:

Put a breakpoint in the Python program just before the code in your extension that is to be executed – it can be in the statement that imports it.
Start the Python program running in debug mode and wait for it to stop on the breakpoint.
Put a breakpoint in the C program where you want it to stop – it has to be somewhere that the Python program will call into.
Go to the Run and Debug view and select gdb Attach, click the run button:
You will next be asked to select the process to attach to. Type P and you will see just the Python programs running:

The one that you want to attach to is the one with the adapter access token and you should select it – it is usually the last one in the list.
Now single-step through the Python program so that it calls your extension.
This should cause the C program to pause at the breakpoint – as long as the Python program calls the code that contains the breakpoint.

You can now use the debugger to step through the code of both programs. Initially it can be difficult to work out how to get to a particular section of the C code, but you quickly get used to it. The only frustrating part is having to repeatedly specify the process Id for the Python program.

Notice that if everything is set up correctly you should be able to single-step and examine calls into the C API.

Summary

Linux is an easy-to-use development system because it generally is preconfigured with the GCC compiler and gdb debugger.
Although you can use the command line or any IDE, there are advantages in using VS Code as it is open source and supports both Python and C.
It is worth checking that everything is installed correctly by creating a Hello World program.
To build a Python extension you have to specify both the include directory and that a shared.so library is to be generated.
To make the Intellisense prompting work correctly you have to define a c_cpp_properties.json file.
Testing an extension is a matter of running a Python program that imports it and then makes use of it.
As you have GCC installed, it is relatively easy to compile the latest version of Python to use.
At some point you are going to need to work out how to debug an extension. This involves running a debugger for the Python program and a separate debugger for the C program.