Programmer's Python Async - Streams & Web Clients
Written by Mike James   
Monday, 07 November 2022
Article Index
Programmer's Python Async - Streams & Web Clients
The Response

Applying asyncio can be harder than you think. Find out how to use streams to implement an asynchronous web client in this extract from my new book Programmer's Python: Async

Programmer's Python:
Threads, processes, asyncio & more

Is now available as a print book: Amazon


1)  A Lightning Tour of Python

Python's Origins, Basic Python, Data Structures, Control Structures – Loops, Space Matters, Conditionals and Indenting, Pattern Matching, Everything Is An Object – References, Functions , Objects and Classes, Inheritance, Main and Modules, IDEs for Python, Pythonic – The Meta Philosophy, Where Next, Summary.

2) Asynchronous Explained

A Single Thread, Processes, I/O-Bound and CPU-Bound, Threads, Locking, Deadlock, Processes with Multiple Threads, Single-Threaded Async, Events,,Events or Threads, Callback Hell, More Than One CPU – Concurrency, Summary.

3) Processed-Based Parallelism

Extract 1 Process Based Parallism
The Process Class, Daemon, Waiting for Processes, Waiting for the First to Complete, Computing Pi, Fork v Spawn, Forkserve, Controlling Start Method, Summary.

4) Threads

Extract 1 -- Threads
The Thread Class, Threads and the GIL, Threading Utilities, Daemon Threads, Waiting for a Thread, Local Variables, Thread Local Storage, Computing Pi with Multiple Threads, I/O-Bound Threads, Sleep(0), Timer Object, Summary.

5) Locks and Deadlock

Race Conditions, Hardware Problem or Heisenbug, Locks, Locks and Processes, Deadlock, Context Managed Locks, Recursive Lock, Semaphore, Atomic Operations, Atomic CPython, Lock-Free Code, Computing Pi Using Locks, Summary.

6) Synchronization

Join, First To Finish, Events, Barrier, Condition Object, The Universal Condition Object, Summary.

7) Sharing Data

Extract 1 - Pipes & Queues
The Queue, Pipes, Queues for Threads, Shared Memory,  Shared ctypes, Raw Shared Memory, Shared Memory, Manager, Computing Pi , Summary.

8) The Process Pool

Waiting for Pool Processes, Computing Pi using AsyncResult, Map_async, Starmap_async, Immediate Results – imap, MapReduce, Sharing and Locking, Summary.

9) Process Managers

The SyncManager, How Proxies Work, Locking, Computing Pi with a Manager, Custom Managers, A Custom Data Type, The BaseProxy, A Property Proxy, Remote Managers, A Remote Procedure Call, Final Thoughts, Summary.

10) Subprocesses

Running a program, Input/Output, Popen, Interaction, Non-Blocking Read Pipe, Using subprocess, Summary.

11) Futures

Extract 1 Futures

12) Basic Asyncio

Extract 1 Basic Asyncio
Callbacks, Futures and Await, Coroutines, Await, Awaiting Sleep, Tasks, Execution Order, Tasks and Futures, Waiting On Coroutines, Sequential and Concurrent, Canceling Tasks, Dealing With Exceptions, Shared Variables and Locks, Context Variables, Queues, Summary.

13) Using asyncio

Extract 1 Asyncio Web Client
Streams, Downloading a Web Page, Server, A Web Server, SSL Server, Using Streams, Converting Blocking To Non-blocking, Running in Threads, Why Not Just Use Threads, CPU-Bound Tasks, Asyncio-Based Modules, Working With Other Event Loops – Tkinter, Subprocesses, Summary.

14) The Low-Level API

Extract 1 - Streams & Web Clients
The Event Loop, Using the Loop, Executing Tasks in Processes, Computing Pi With asyncio, Network Functions,
Transports and Protocols, A UDP Server, A UDP Client, Broadcast UDP, Sockets, Event Loop Implementation, What Makes a Good Async Operation, Summary.

Appendix I Python in Visual Studio Code


So far in our examination of asyncio the only truly asynchronous coroutine we have used has been asyncio.sleep. This frees the main thread to run the event loop, but it doesn’t really achieve very much other than helping explain what is going on. Put another way, without the use of asyncio.sleep we essentially have a synchronous system, even if it does use an event loop. In this chapter we take a look at things that we can do with asyncio that are actually useful and demonstrate an efficiency improvement.

The asyncio module is primarily designed to work with asynchronous network connections. If you want to go beyond this important, but limited, application you need either to use a library that extends asyncio or to create your own asynchronous extensions.

To be clear, asyncio does not help with working with local files asynchronously and it doesn’t provide any methods of dealing with asynchronous user interaction, but it is possible to adapt it to do both and there are existing modules for most similar applications.

What is more surprising is that it doesn’t provide high-level networking facilities. There is no asynchronous download of an HTML page, for example. Indeed until recently, all network interaction was performed at a low level, but in most cases you can ignore the low-level API and its pipes, sockets and transports and simply use streams.


Streams are the high-level coroutine implementation of network connections implemented using sockets. They work in much the same way as standard files, but they are asynchronous which enables you to work with many, hundreds or even thousands of, connections using a single thread.

Like a file, a stream has to be opened:

reader, writer = asyncio.open_connection(host = None, port = None)

There are a great many additional parameters which you can use to customize the connection, but this simple form does for most connections to internet servers.

Opening a stream returns a tuple consisting of a StreamReader and a StreamWriter. These have methods very similar to any of the familiar file objects and, apart from being asynchronous, they work in the same way. If you want to know more about file objects refer to Programmer’s Python: Everything Is Data, ISBN: 978-1871962598.


The following methods are commonly used to read streams:

  • read(n = -1) - reads up to n bytes as a bytes object

The default, n = -1, is to read until the end of the file signal (EOF) is received and return all read bytes.

  • readline() - reads one line, where “line” is a sequence of bytes ending with \n

If EOF is received and \n was not found, the method returns partially read data. If EOF is received and the internal buffer is empty, returns an empty bytes object.

  • readexactly(n) - reads exactly n bytes and raises an IncompleteReadError if EOF is reached before n can be read
    Use the IncompleteReadError.partial attribute to get any partially read data.

  • readuntil(separator = b'\n') - reads data from the stream until separator is found.
    The default is to use \n, i.e. new line, as the separator which makes it the same as readline.

If the amount of data read exceeds the configured stream limit, a LimitOverrunError exception is raised, and the data is left in the internal buffer and can be read again. If EOF is reached before the complete separator is found, an IncompleteReadError exception is raised and the internal buffer is reset.
The IncompleteReadError.partial attribute may contain a portion of the separator.

  • at_eof() - True if buffer is empty and EOF has been signaled. 

Notice that all of the reading methods are coroutines as there may not be enough data ready to satisfy the call. In this case the coroutine is suspended and the main thread is freed. That is calls to functions that read data are asynchronous coroutines. Also notice that while there are references to using EOF to signal the end of a transaction, in general EOF isn’t particularly useful when dealing with sockets. Sockets tend to be left open until they are no longer required and data is usually sent in some sort of format that lets you work out when you have read a useful chunk of data that can be processed. Generally, if you wait for an EOF you will wait a long time until the server times out and closes the socket.

Last Updated ( Tuesday, 08 November 2022 )