Applying asyncio can be harder than you think. Find out how to use streams to implement an asynchronous web client in this extract from my new book Programmer's Python: Async

Programmer's Python:
Async
Threads, processes, asyncio & more

Is now available as a print book: Amazon

Contents

1)  A Lightning Tour of Python.

2) Asynchronous Explained

3) Processed-Based Parallelism
         Extract 1 Process Based Parallism
4) Threads
         Extract 1 -- Threads
5) Locks and Deadlock

6) Synchronization

7) Sharing Data
        Extract 1 - Pipes & Queues

8) The Process Pool
        Extract 1 -The Process Pool 1 ***NEW!

9) Process Managers

10) Subprocesses

11) Futures
        Extract 1 Futures, 

12) Basic Asyncio
        Extract 1 Basic Asyncio

13) Using asyncio
        Extract 1 Asyncio Web Client
14) The Low-Level API
       Extract 1 - Streams & Web Clients
Appendix I Python in Visual Studio Code

So far in our examination of asyncio the only truly asynchronous coroutine we have used has been asyncio.sleep. This frees the main thread to run the event loop, but it doesn’t really achieve very much other than helping explain what is going on. Put another way, without the use of asyncio.sleep we essentially have a synchronous system, even if it does use an event loop. In this chapter we take a look at things that we can do with asyncio that are actually useful and demonstrate an efficiency improvement.

The asyncio module is primarily designed to work with asynchronous network connections. If you want to go beyond this important, but limited, application you need either to use a library that extends asyncio or to create your own asynchronous extensions.

To be clear, asyncio does not help with working with local files asynchronously and it doesn’t provide any methods of dealing with asynchronous user interaction, but it is possible to adapt it to do both and there are existing modules for most similar applications.

What is more surprising is that it doesn’t provide high-level networking facilities. There is no asynchronous download of an HTML page, for example. Indeed until recently, all network interaction was performed at a low level, but in most cases you can ignore the low-level API and its pipes, sockets and transports and simply use streams.

Streams

Streams are the high-level coroutine implementation of network connections implemented using sockets. They work in much the same way as standard files, but they are asynchronous which enables you to work with many, hundreds or even thousands of, connections using a single thread.

Like a file, a stream has to be opened:

reader, writer = asyncio.open_connection(host = None, port = None)

There are a great many additional parameters which you can use to customize the connection, but this simple form does for most connections to internet servers.

Opening a stream returns a tuple consisting of a StreamReader and a StreamWriter. These have methods very similar to any of the familiar file objects and, apart from being asynchronous, they work in the same way. If you want to know more about file objects refer to Programmer’s Python: Everything Is Data, ISBN: 978-1871962598.

StreamReader

The following methods are commonly used to read streams:

• read(n = -1) - reads up to n bytes as a bytes object

The default, n = -1, is to read until the end of the file signal (EOF) is received and return all read bytes.

• readline() - reads one line, where “line” is a sequence of bytes ending with \n

If EOF is received and \n was not found, the method returns partially read data. If EOF is received and the internal buffer is empty, returns an empty bytes object.

• readexactly(n) - reads exactly n bytes and raises an IncompleteReadError if EOF is reached before n can be read
  Use the IncompleteReadError.partial attribute to get any partially read data.

• readuntil(separator = b'\n') - reads data from the stream until separator is found.
  The default is to use \n, i.e. new line, as the separator which makes it the same as readline.

If the amount of data read exceeds the configured stream limit, a LimitOverrunError exception is raised, and the data is left in the internal buffer and can be read again. If EOF is reached before the complete separator is found, an IncompleteReadError exception is raised and the internal buffer is reset.
The IncompleteReadError.partial attribute may contain a portion of the separator.

• at_eof() - True if buffer is empty and EOF has been signaled. 

Notice that all of the reading methods are coroutines as there may not be enough data ready to satisfy the call. In this case the coroutine is suspended and the main thread is freed. That is calls to functions that read data are asynchronous coroutines. Also notice that while there are references to using EOF to signal the end of a transaction, in general EOF isn’t particularly useful when dealing with sockets. Sockets tend to be left open until they are no longer required and data is usually sent in some sort of format that lets you work out when you have read a useful chunk of data that can be processed. Generally, if you wait for an EOF you will wait a long time until the server times out and closes the socket.