You can use a subprocess from Python even if it isn't a Python program you want to control. Find out how to use them in this extract from Programmer's Python: Async.

Programmer's Python:
Async
Threads, processes, asyncio & more

Is now available as a print book: Amazon

Contents

1)  A Lightning Tour of Python.

2) Asynchronous Explained

3) Processed-Based Parallelism
         Extract 1 Process Based Parallism
4) Threads
         Extract 1 -- Threads
5) Locks and Deadlock

6) Synchronization

7) Sharing Data
        Extract 1 - Pipes & Queues

8) The Process Pool
        Extract 1 -The Process Pool 1 

9) Process Managers

10) Subprocesses ***NEW!

11) Futures
        Extract 1 Futures, 

12) Basic Asyncio
        Extract 1 Basic Asyncio

13) Using asyncio
        Extract 1 Asyncio Web Client
14) The Low-Level API
       Extract 1 - Streams & Web Clients
Appendix I Python in Visual Studio Code

So far the processes that we have been creating have hosted Python code. That is, the process has a complete copy of the Python interpreter and a Python program ready to run. Sometimes it is useful to be able to run other languages in a separate process and use a Python program to control and make use of it. This is what the subprocess module allows you to do. It is a replacement for low level interaction with foreign code via the os.system and os.spawn modules.

The subprocess module gives you an easier to use approach to running foreign code. However easy the general idea may be, the actual implementation is usually difficult due to the non-standard ways programs can be run under different operating systems. Put simply, the differences between Windows and Linux, or more generally Posix, operating systems are many and time consuming. The simplest situation is only having to target a single operating system. In this case you can adjust parameters to get things to work and not worry about finding something that works across operating systems.

A second problem is the way that programs interact with a human client. The range of ways this can happen is more varied than you might think and it is generally far from regular. Writing code that recognizes when a program has finished sending messages to the user and is now waiting for input is difficult. You are essentially mimicking human behavior, reading the screen and typing appropriate responses. At the extreme this needs AI, but in most cases it only requires simple string manipulation.

It is worth noting that the asyncio module also provides facilities similar to subprocess but modified for asynchronous use. It is worth considering because asyncio doesn’t suffer from the same buffering problems as subprocess. However, it is based on the subprocess module and if you simply want to add external programs to an existing, otherwise non-async, program then subprocess is the way to go.

Running a program

The simplest of the subprocess functions is run. It is built on top of the more flexible Popen class and it works well for simple situations. The run and Popen functions support a very large number of parameters to allow you to customize the way a program is run. These are described in the documentation and most are very specific to particular operating systems and conditions. In this chapter we will deal only with the most important ones.

The run command is useful if you just want to run a program and get a result. The Popen class is useful if you need to run and interact with a program to get a result. In most cases you can start with a simple version:

subprocess.run(command line)

where command line is a list of items that you would use if running the program from the command line. The program specified is run in a new process and the calling process waits for it to complete. You can specify a timeout= parameter if you don’t want to wait more than the specified time. For example, if you want to get a directory listing then under Linux you would use:

dir /

and under Windows:

dir \

To run these programs, under Linux you would use:

import subprocess
subprocess.run(["dir","/"])

and under Windows:

import subprocess
subprocess.run(["dir", "\\"])

Notice the need to escape the backslash.

You can see that each separate item on the command line has to be converted into an item in the list. One of the problems is working out what corresponds to a separate item. The reason that the parameters are specified as items in a list is that the run method processes them to try to make them correct for the system. For example, you cannot use:

subprocess.run(["dir /"])

as this generates the error:

No such file or directory: 'dir /'

The point is that run doesn’t simply submit the items as written, it processes them and the first item is always regarded as the name of the file to be executed.

It can be difficult to work out how to split a command line into suitable items correctly. The easiest option is to use the shlex lexical analysis object. For example:

import shlex
print(shlex.split("dir /"))

displays

['dir', '/']

The lexical analysis is designed for Linux/Unix shells and isn’t guaranteed to work for Windows but the changes needed are usually small.