Interaction

If you want to interact with a subprocess then things become much more complicated. The reason is that most of the programs you are likely to want to run are written in C and this changes the way that they work when you redirect their I/O to a pipe. Some even avoid using stdin and stdout, instead preferring to send data directly to the console. If you are working with such a program there is nothing you can do. Even if you are using a perfectly standard C program there is still a problem. By default, data sent to stdout and stdin is line buffered, that is the buffer is cleared and the data sent to the device a line at a time. In an effort to be more efficient, the standard library code switches to using the available buffers when stdout and stdin are redirected to files or pipes.

This is usually a good idea, but it makes using programs interactively very difficult and sometimes impossible. If a program sends a question to stdout connected to a pipe then the data may not be transmitted to the client because the buffer isn’t full. This can leave the client stalled while it waits for input. If the client then executes an input to get the answer to the question from the client we have deadlock. This is the reason that the documentation warns that reading and writing a pipe is dangerous and you should use communicate.

If you do use communicate then there is no chance of any interactive behavior. For example, consider this very simple script, firstly for Linux:

#!/bin/sh
echo "user name: "
read name
echo $name

Secondly for Windows:

@echo off
echo "User name:"
set /p id=
echo %id%

This simply displays a prompt for a user name, reads it in and then displays it. This is a simple model for part of an interactive sequence from almost any command line program. To automate this we have to read the prompt, perhaps test to see what it is, and then write a suitable name. For example under Windows:

import subprocess
p = subprocess.Popen(["myscript.bat"],
       stdout=subprocess.PIPE,stdin=subprocess.PIPE,
                                            text=True)
ans=p.stdout.readline()
print("message",ans)
p.stdin.write("mike\n")
ans=p.stdout.readline()
print("message",ans)

A Linux version just needs the name of the script changed to ["./myscript.sh"]

If you try this out you will discover that it hangs at the last readline. The first readline works because the output buffer is flushed by the script’s input command (read or set) but the write simply sends the data to the buffer waiting to be sent to the script. As a result the script doesn’t read it and so doesn’t move on to send the data back to the parent and hence the final readline waits forever.

There are a number of solutions to this problem but, as the problem is that the Python buffer isn’t being cleared, the best is to use the flush method:

ans = p.stdout.readline()
print("message",ans)
p.stdin.write("mike\n")
p.stdin.flush()
ans = p.stdout.readline()
print("message",ans)

This makes the program work, but the same situation can occur due to the subprocess not flushing its buffer and then the parent process cannot solve the problem using a flush command as it can flush its own buffers but not those of the subprocess.

The point is that communicating over buffered pipes is difficult to get right and very fragile. You can make thing a little easier by switching to line buffering, i.e. the buffer is flushed each time a complete line is present. To do this you simply set bufsize to 0 and universal_newlines to true:

import subprocess
p = subprocess.Popen(["myscript.bat"],
    stdout=subprocess.PIPE,
      stdin=subprocess.PIPE, bufsize=0,
            universal_newlines=True, text=True)
ans=p.stdout.readline()
print("message",ans)
p.stdin.write("mike\n")
ans=p.stdout.readline()
print("message",ans)

This now works without the parent needing to flush its buffers, but it only works because the script sends a newline after the prompt. If you change the script to read:

(Linux)

#!/bin/sh
echo -n "User name: "
read name
echo $name

(Windows)

@echo off
set /p id= "User name"
echo %id%

Now the prompt “User name” is delivered without a newline before the user’s responses. This, of course, makes the program hang because the first readline never completes as the child process's buffer isn’t flushed.