It's that time of year - a new Python is available for us to use rather than just test out. Python 3.12 isn't obviously groundbreaking, but it could be if the next release enthusiastically builds on its new features.

There are lots of small changes in 3.12 that you can read about in the release notes, but what is really interesting is a new feature that most of us won't be able to use for a while - in fact not until 3.13 if it all works out. But it is so important that it is worth explaining in detail.

The introduction of sub-interpreters with a per-interpreter GIL has the potential to make moot the point about removing the GIL. To understand what is going on you have to appreciate that the problem with the GIL, Global Interpreter Lock, is exactly that - it is a lock on the Python interpreter. This means that only a single thread can be executing the Python interpreter at any given moment. So even if you have 100 cores available to do work, and you have used Python threads, your Python program is executing a single instruction at a time. Indeed this makes the idea of Python threads useful only for threads that are I/O bound. In this case you can have N threads and, as long as N-1 of them are waiting for something to happen, everything is effective as it possibly can be.

The obvious solution is to get rid of the GIL, which is something that the Python team have decided to do - if at all possible. But even if the GIL can be removed there is going to be lots of sofware that needs it to work correctly. Removing the GIL is always going to be an optional extra.

There is a second, less obvious, solution - keep the GIL, just get more interpreters. This is the idea of the sub-interpreter and you can already make use of this idea, but only if you use Python processes rather than Python threading. If you create a new Python process then you get a new operating system process which is isolated from the original program. It can run Python without worrying about the GIL because it has its own Python intepreter installed in the process and that interpreter has its own GIL. Using this approach if you have 100 cores you can have 100 parts of your Python programs running at the same time - but each one only executes a single thread at any given moment.

Of course, the downside of this approach is that you have 100 copies of the Python interpreter loaded and hence some wasted memory - but memory is cheap. A much bigger problem is that each interpreter is sealed inside a process box, which makes getting data between them difficult and slow. The advantage of threads is that they naturally share the same process space and hence have access to the same globals. This too has a problem in that you have to control access to shared variables and this usually means locking. Interprocess communication is usually via pipes or similar objects and these often don't need locking.

So the big new deal in 3.12 is the ability to create and run multiple Python interpreters, each with their own GIL, within the same process - indeed even on the same thread. The sub-interpreter also has independent copies of all modules loaded, with the proviso that the modules explicitly support sub-interpreters.

What this means is that you can execute multiple threads in multiple interpreters, each with its own GIL and with shared variables. Apart from the overhead of having to load modules again, the only real cost is the size of the sub-interpreter which isn't an issue on desktop machines - but might be for IoT systems.

This all sounds revolutionary - we can make Python programs faster and still rely on the GIL to keep us out of trouble. This is true, but Python 3.12 hasn't completely added everything that is needed. In particular, you can only use sub-interpreters from the C API. This means you can write a C extension and load sub-interpreters and use them to run Python programs and then, presumably use C to collect the results. This is fun, but it is hardly going to impact the average Python programmer. At best it allows C extension writers to speed up their creations.

What we really need are the new Python commands that allow the creation of sub-interperters from a Python program. This would make the sub-interpreter approach to async as fast as the current process-based approach, but with the benefits of being lightweight from the operating system's point of view and having shared memory.

So the revolution is to be in Python 3.13, when we are also promised the removal of the GIL. Which will happen and which will turn out to be practical?

Life without the GIL or Life with multiple GILs - only 3.13 will let us know.

• Mike James is the author of the Programmer's Python: Something Completely Different series of books which set out to show how Python diverges from other programming languages and how its special features deserve our attention. The most recent volume, Programmer’s Python: Async  on asynchronous and concurrent Python explores the problems posed by the GIL in detail. His latest book Extending Python Using C is also highly relevant to this innovation.

More Information

https://docs.python.org/3.12/whatsnew/3.12.html

Related Articles

Goodbye GIL - But Will It Make Python Faster?

Python 3.11 Goes Faster

Python 3.11 Released

Guido And Microsoft Want To Make Python x2 Faster

Microsoft Now Visionary Sponsor Of Python

What Makes Python Great & Greater

Guido van Rossum Quits As Python BDFL 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info

<ASIN:B0CK3X93KF>

<ASIN:1871962765>

<ASIN:1871962749>

<ASIN:1871962757>