Tasks

Tasks are coroutines plus futures. To be more exact, Task inherits from Future and keeps a reference to a coroutine that the Future is associated with. Put simply, a Task is what you add to the event loop’s queue and it is a Future plus a coroutine. The coroutine does the work and the Future returns the result.

To add a Task to the event loop you need to use the:

asyncio.create_task(coroutine, args, name = None)

This adds the coroutine to the current event loop as a Task, passing it any args you may specify and giving it a name if you specify one. The function adds the coroutine to the event loop queue ready to be executed. It doesn’t actually get to run until the main thread is free to return to the event loop and run the tasks that it finds there. This only happens when the currently executing coroutine awaits an asynchronous coroutine or terminates.

The asyncio.create_task function returns a Task which, as already explained, is a future-like object which resolves with the result when the coroutine ends. Anything that is added to the event loop has to behave like a Future as it is needed for the await to retrieve a result. For example, if we create a coroutine that prints a range of numbers then this can be added to the event loop within main:

import asyncio
async def count(n):
    for i in range(n):
        print(i)
    return n

async def main(myValue):
    t1=asyncio.create_task(count(10))  
    print("Hello Coroutine World")
    await asyncio.sleep(5)
    return myValue

result= asyncio.run(main(42))
print(result)

The count coroutine is added to the event loop before the print, but it doesn’t get to run until main awaits sleep for 5 seconds and so frees the thread. If you take out the await on sleep then the count coroutine gets to run when main finishes. If you put the await before the print then count gets to print its values before the Hello Coroutine World is displayed. When count runs depends on when main is allowed to run the event loop.

If you look at the program again you will see that count returns a result, but the result isn’t used in the program. How can we get a result from a Task? The simple answer is that we wait for its Future to resolve and for await to return its result after consuming the Future, for example:

async def main(myValue):
    t1 = asyncio.create_task(count(10))   
    print("Hello Coroutine World")
    await asyncio.sleep(5)
    result = await t1
    print(result)
    return myValue

In this case await t1 returns result which is displayed.

At this point you should be wondering why we bothered creating a Task and adding it to the event loops’s queue? Why not just use await count? That is:

async def main(myValue):
    print("Hello Coroutine World")
    await asyncio.sleep(5)
    result= await count(10)
    print(result)
    return myValue

This produces the same answer, but what happens under the hood is very different. The first version adds count to the event loop’s queue and when main sleeps for five seconds the thread is freed and t1 is allowed to execute. After the five seconds is up the await returns at once because t1 is resolved and there is a result which is displayed immediately. The second version doesn’t add count to the queue and so nothing happens while main waits for five seconds. When the wait is up the await starts count running to get the result which is then printed. In other words, when you await a Task the Task might already have run and have a result. If it hasn’t already resolved then it is taken from the queue and run just like a coroutine. That is:

• when you await a coroutine it starts running to completion

• when you await a Task it only starts running if it hasn’t already resolved

In both cases the thread might be released and tasks run before the coroutine or the Task completes.

You will sometimes see instructions like:

    value = await asyncio.create_task(count(10)) 

This adds the coroutine to the queue as a Task and then immediately awaits it, which of course, starts it running. There is no point in doing this and it is entirely equivalent to:

    value = await count(10) 

In general:

await asyncio.create(coroutine)

is the same as:

await coroutine

To summarize:

• asyncio.create_task(coroutine) runs the coroutine at a later time. It adds it to the event loop’s queue as a Task for execution when the thread is free

• await coroutine runs the coroutine immediately

• await Task runs the associated coroutine if the associated Future hasn’t ready resolved

• asyncio.create_task(coroutine) is equivalent to await coroutine

Because it is often used in older examples, it is also worth mentioning,:

asyncio.ensure_future(awaitable)

This takes a coroutine, Future or any awaitable object, converts it and “ensures” that it is a future-like object and adds it to the event queue. In practice, this means that the coroutine is converted into a Task and added to the queue. In other words, it is the equivalent of create_task.

In Chapter but not included in this extract

• Execution Order
• Tasks and Futures
• Waiting On Coroutines
• Sequential and Concurrent
• Canceling Tasks
• Dealing With Exceptions
• Shared Variables and Locks
• Context Variables
• Queues

Summary

• The asyncio module provides single-threaded multi-tasking.

• The callback is the most common way of implementing single-threaded multi-tasking but it has many disadvantages. A better method is to use a Future and an even better method is to use await.

• A coroutine is a function that can be suspended and resumed by the use of the await instruction.

• A coroutine has to be run in conjunction with an event loop. The asyncio.run creates an event loop and runs a coroutine using it.

• A Task is a Future plus a coroutine and it is what is added to the event loop’s queue using asyncio.create_task. The Task is run when the thread becomes free.

• When you await a coroutine it starts running to completion.

• When you await a Task it only starts running if it isn’t already completed.

• The await always returns the result of the coroutine/Task, including any exceptions that might have occurred.

• If you don’t await a Task its result and any exceptions are ignored.

• You can use wait_for as a version of await with a timeout.

• The wait coroutine can be used to wait for the first to complete, the first to raise an exception or for all complete.

• Task coroutines can be executed in sequential order by awaiting each one in turn. They can be run concurrently by adding them to the queue or by using the gather coroutine.

• A Task can be canceled and this sends the CancelledError exception to the Task.

• A Task returns any exceptions to the awaiting coroutine – these can be raised or processed.

• Locks are less useful for coroutines because unless the thread is released they are atomic. If a race condition can occur there are asynchronous equivalents of all of the standard synchronization objects.

• Shared global variables cannot be protected against race conditions using thread-local variables as only a single thread is in operation. Instead we need to use context variables.

• There are asynchronous equivalents of the Thread queue objects.

Programmer's Python:
Async
Threads, processes, asyncio & more

Is now available as a print book: Amazon

Contents

1)  A Lightning Tour of Python

Python's Origins, Basic Python, Data Structures, Control Structures – Loops, Space Matters, Conditionals and Indenting, Pattern Matching, Everything Is An Object – References, Functions , Objects and Classes, Inheritance, Main and Modules, IDEs for Python, Pythonic – The Meta Philosophy, Where Next, Summary.

2) Asynchronous Explained

A Single Thread, Processes, I/O-Bound and CPU-Bound, Threads, Locking, Deadlock, Processes with Multiple Threads, Single-Threaded Async, Events,,Events or Threads, Callback Hell, More Than One CPU – Concurrency, Summary.

3) Processed-Based Parallelism

        Extract 1 - Process Based Parallism

The Process Class, Daemon, Waiting for Processes, Waiting for the First to Complete, Computing Pi, Fork v Spawn, Forkserve, Controlling Start Method, Summary.

4) Threads

        Extract 1 -- Threads

The Thread Class, Threads and the GIL, Threading Utilities, Daemon Threads, Waiting for a Thread, Local Variables, Thread Local Storage, Computing Pi with Multiple Threads, I/O-Bound Threads, Sleep(0), Timer Object, Summary.

5) Locks and Deadlock

Race Conditions, Hardware Problem or Heisenbug, Locks, Locks and Processes, Deadlock, Context Managed Locks, Recursive Lock, Semaphore, Atomic Operations, Atomic CPython, Lock-Free Code, Computing Pi Using Locks, Summary.

 6) Synchronization

Join, First To Finish, Events, Barrier, Condition Object, The Universal Condition Object, Summary.

 7) Sharing Data

The Queue, Pipes, Queues for Threads, Shared Memory,  Shared ctypes, Raw Shared Memory, Shared Memory, Manager, Computing Pi , Summary.

8) The Process Pool

Waiting for Pool Processes, Computing Pi using AsyncResult, Map_async, Starmap_async, Immediate Results – imap, MapReduce, Sharing and Locking, Summary.

9) Process Managers

The SyncManager, How Proxies Work, Locking, Computing Pi with a Manager, Custom Managers, A Custom Data Type, The BaseProxy, A Property Proxy, Remote Managers, A Remote Procedure Call, Final Thoughts, Summary.

10) Subprocesses

Running a program, Input/Output, Popen, Interaction, Non-Blocking Read Pipe, Using subprocess, Summary.

11) Futures

Futures, Executors, I/O-Bound Example, Waiting On Futures, Future Done Callbacks, Dealing With Exceptions, Locking and Sharing Data, Locking and Process Parameters, Using initializer to Create Shared Globals, Using a Process Manager to Share Resources, Sharing Futures and Deadlock, Computing Pi with Futures, Process Pool or Concurrent Futures, Summary.

12) Basic Asyncio

        Extract 1 Basic Asyncio

Callbacks, Futures and Await, Coroutines, Await, Awaiting Sleep, Tasks, Execution Order, Tasks and Futures, Waiting On Coroutines, Sequential and Concurrent, Canceling Tasks, Dealing With Exceptions, Shared Variables and Locks, Context Variables, Queues, Summary.

13) Using asyncio

   Extract 1 Asyncio Web Client

Streams, Downloading a Web Page, Server, A Web Server, SSL Server, Using Streams, Converting Blocking To Non-blocking, Running in Threads, Why Not Just Use Threads, CPU-Bound Tasks, Asyncio-Based Modules, Working With Other Event Loops – Tkinter, Subprocesses, Summary.

14) The Low-Level API

    Extract 1 - Streams & Web Clients

The Event Loop, Using the Loop, Executing Tasks in Processes, Computing Pi With asyncio, Network Functions,
Transports and Protocols, A UDP Server, A UDP Client, Broadcast UDP, Sockets, Event Loop Implementation, What Makes a Good Async Operation, Summary.

Appendix I Python in Visual Studio Code

<ASIN:1871962765>

<ASIN:1871962749>

<ASIN:1871962595>