[Python] Learn about asynchronous programming and event loops

About this article

I'd like to say that I can't really talk about "what is asynchronous programming?" This article describes what we have learned about one of the elements of asynchronous programming: collaborative multitasking, Python's ʻasync`, and event loops.

Collaborative multitasking and asynchronous I / O

Collaborative multitasking is a central element of asynchronous programming and is like an idea.

When a computer multitasks, it does not use the context switch of the OS. When each process enters the wait state, the process itself voluntarily releases control. Gives control to many other processes running at the same time. (Comparable to the meaning of ** cooperative **) Control here refers to suspending and resuming a process, and allocating and releasing resources.

It seems that all processes need to be coordinated in order to multitask smoothly.

For application-level collaborative multitasking

Where do you do collaborative multitasking? Rather than coordinating multiple processes or threads ** All multitasking is executed in one process or thread. ** **

Who controls the task? Control of multiple tasks is limited to one function, which manages the coordination of tasks.

Key issues in collaborative multitasking

In collaborative multitasking, the most important issue is the timing of releasing control **. The timing of release is similar to the behavior of threads. Where? ⇒ Many asynchronous applications give control to the event loop or scheduler at the time of I / O instruction. Releases control when waiting for I / O processing.

What's the difference? ⇒ ** Thread ** is due to a system-level thread, ** OS can interrupt the running thread at any time ** and pass control to other threads. In ** asynchronous programming **, ** tasks cannot be interrupted by an event loop. ** (Non-preemptive multitasking)

What you have to keep (probably) minimal about asynchronous programming

Python runs on the OS, competing for other processes and resources. In other words, the OS controls all processes. In the case of an asynchronous application, processing is interrupted by a scheduler interrupt, but when control is returned, it resumes from the interrupted location. This is not always the case with multithreading and multiprocessing. Also, For ** multi-processes and threads **, the task to be restarted is determined by the OS scheduler **. In ** asynchronous programming **, the application ** decides which task to restart.

ʻAsync and ʻawait in Python

Reserved words ʻasync and ʻawait

ʻAsync is used before the def statement to define a new coroutine (parallel task). Execution of coroutine functions is interrupted and resumed depending on the defined situation. Even if the function defined by ʻasync is called, the function is not executed on the spot and a ** coroutine object ** is returned. The following is an implementation example.

>>> async def asyc_hello():
...   print("Hello")
...
>>> asyc_hello()
<coroutine object asyc_hello at 0x000001D3D021C748>

You can see that ʻasyc_hello ()returns a coroutine object instead of the standard output value ofprint ("Hello ")`. So what is a coroutine object? How should I handle it? I will explain this area roughly. You have to create something to execute a coroutine object. It's an ** event loop **. Below is the code that created the event loop and executed the coroutine object.

>>> import asyncio
>>> async def asyncio_hello():
...     print("Hello")
...
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(asyncio_hello())
Hello
>>> loop.close()

Create an event loop with ʻasyncio.get_event_loop () Execute the coroutine object byrun_until_complete (asyncio_hello ())`. Next, I will explain the basics of what the event loop is.

About the event loop

This article does not explain terms related to event loops. Instead of explaining event queues, event dispatchers, event handlers, callback functions, etc., I focused on what the event loop mechanism is doing and expressed what I learned in the figure. Below is the figure. For convenience, we changed the expression of event to request. (For convenience, it's easier to get an image of that personally) イベントループ完成図1.png

About coroutine objects

Let's return to the handling of coroutine objects. Coroutine objects will be executed inside the event loop. Also, it is queued in the event loop and the coroutine object does nothing until its turn comes. The following source code is the code that prepares only one simple coroutine and executes the event loop.

asyncprint.py


import asyncio

async def print_numeber(number):
    print(number)

if __name__ == "__main__":
    loop = asyncio.get_event_loop()

    loop.run_until_complete(
        asyncio.wait([
            print_numeber(number)
            for number in range(10)
        ])
    )
    loop.close()
$ python src/asyncpritn.py
2
4
8
6
9
5
1
3
7
0

This is the basic flow of the above code.

  1. Create an event loop: ʻasyncio.get_event_loop () `
  2. Create or add a task: ʻasyncio.get_event_loop (). create_task () ʻorʻasyncio.wait ()`
  3. Run the event loop: ʻasyncio.get_event_loop (). run_until_complete () `
  4. Explicitly close the event loop: ʻasyncio.get_event_loop (). close () `

Next, it is an example when the reserved word ʻawait is added to ʻasyncio.wait ().

corowait.py


import time 
import random
import asyncio

async def waiter(name):
    for _ in range(4):
        time_to_sleep = random.randint(1,3)/4
        time.sleep(time_to_sleep)
        print(
            "{}Is{}Waited for a second"
            "".format(name, time_to_sleep)
        )

async def main():
    await asyncio.wait([waiter("foo"), waiter("bar")])

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()
$time python corowait.py
foo is 0.Waited 5 seconds
foo is 0.Waited 25 seconds
foo is 0.Waited 75 seconds
foo is 0.Waited 25 seconds
bar is 0.Waited 25 seconds
bar is 0.Waited 75 seconds
bar is 0.Waited 75 seconds
bar is 0.Waited 25 seconds

real    0m4.416s
user    0m0.130s                                                                                      sys     0m0.013s  

ʻAwaitwaits for the coroutine to return execution, then releases control and passes it to the event loop until it finishes executing. Here, the processing is blocked by thetime.sleep () function. Therefore, it becomes a synchronous process and the processes are executed in order. In Python, there is a ʻasycio.sleep () function to change blocking processing to non-blocking processing and asynchronous processing. By using this, it is possible to execute asynchronous processing. Below is a sample code.

cowait_improved.py


import time 
import random
import asyncio

async def waiter(name):
    for _ in range(4):
        time_to_sleep = random.randint(1,3)/4
        await asyncio.sleep(time_to_sleep)
        print(
            "{}Is{}Waited for a second"
            "".format(name, time_to_sleep)
        )

async def main():
    await asyncio.wait([waiter("foo"), waiter("bar")])

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()
$time python corowait_improved.py
foo is 0.Waited 25 seconds
bar is 0.Waited 75 seconds
foo is 0.Waited 75 seconds
bar is 0.Waited 75 seconds
foo is 0.Waited 75 seconds
bar is 0.Waited 25 seconds
foo is 0.Waited 25 seconds
bar is 0.Waited 25 seconds

real    0m2.442s
user    0m0.161s                                                                                      sys     0m0.017s  

The functions foo and bar are executed alternately, and the processing speed is improved. It means that the coroutine has released control cooperatively.

References

https://www.atmarkit.co.jp/ait/articles/1103/23/news101_2.html Introduction to Python3 Expert Python Programming Revised 2nd Edition

Recommended Posts

[Python] Learn about asynchronous programming and event loops
Learn about programming
About Python for loops
Getting Started with python3 # 2 Learn about types and variables
Programming with Python and Tkinter
About python objects and classes
About Python variables and objects
Paiza Python Primer 1 Learn Programming
About Python, len () and randint ()
About Python and regular expressions
About Python and os operations
Python # About reference and copy
About Python sort () and reverse ()
[Introduction to Python3 Day 1] Programming and Python
About python dict and sorted functions
About dtypes in Python and Cython
About Python pickle (cPickle) and marshal
[Python] About Executor and Future classes
About Python, from and import, as
Compare Python and JavaScript array loops
Avoid nested loops in PHP and Python
Learn Python and AI related English words. .. ..
Learn math and English through programming (Part 1)
A story about Python pop and append
Learn dynamic programming in Python (A ~ E)
Talking about old and new Python classes
About _ and __
Learn math and English by programming (Part 2)
Talking about Python class attributes and metaclasses
What is "functional programming" and "object-oriented" in Python?
Think about depth-priority and width-priority searches in Python
About the difference between "==" and "is" in python
A story about modifying Python and adding functions
About the * (asterisk) argument of python (and itertools.starmap)
About shallow and deep copies of Python / Ruby
About python slices
About python comprehension
Python programming note
About Python tqdm.
About python, class
About python inheritance
Programming in python
About python, range ()
About python reference
About Python decorators
[Python] About multi-process
Learn python gestures
Python asynchronous processing ~ Full understanding of async and await ~
About creating and modifying custom themes for Python IDLE
Paiza Python Primer 2: Learn Conditional Branching and Comparison Operators
Learn Python asynchronous processing / coroutines by comparing with Node.js
Learn about python's print function and strings for beginners.
[Python] Chapter 01-02 About Python (Execution and installation of development environment)