Learn Python asynchronous processing / coroutines by comparing with Node.js

Python is a style language that implements asynchronous processing by importing the ʻasyncio` module and defining coroutines. A ** coroutine ** is a structure that allows a process to be interrupted and then resumed, in contrast to a ** subroutine **, which is a structure that does not stop until the process is completed.

This time, let's focus on how Python's asynchronous processing is written differently from Node.js's Promise and ʻasync / await`, which I've been familiar with for a long time. think.

The Python source code generally conforms to the official documentation Coroutines and Tasks.

Coroutine

In Python, the ʻasync def` statement defines a coroutine.

Python


import asyncio

async def main():
    print('hello')
    await asyncio.sleep(1)
    print('world')

asyncio.run(main())

This is an example of printing "world" one second after printing "hello". You can use ʻawait in coroutines. This is a typical example of asynchronous processing, where ʻawait asyncio.sleep (1) waits for the resolution of the processing for 1 second before starting the next processing.

What about Node.js? First of all, Node.js doesn't have a built-in async function like Python's ʻasyncio.sleep () `, so

function sleep(sec) {
  return new Promise(resolve => {
    setTimeout(resolve, timeout=sec*1000);
  })
}

(Hereafter, the definition of this sleep function is omitted in the Node.js source code). Then you can write as follows.

Node.js


async function main() {
  console.log('hello');
  await sleep(1);
  console.log('world');
}

main();

Comparing the two, Python's coroutine asynchronous processing does not simply call the top-level entry point main (), but ʻasyncio.run (main ()), like ʻasyncio. The difference is that you have to be careful about running it in the argument of .run () .

Serial processing

Arranging awaits will result in serial processing.

Python


import asyncio
import time

async def say_after(delay, what):
    await asyncio.sleep(delay)
    print(what)

async def main():
    print(f"started at {time.strftime('%X')}")

    await say_after(1, 'hello')
    await say_after(2, 'world')

    print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

This is an example of waiting for 1 second and then printing "hello", then waiting for another 2 seconds and then printing "world". Written in Node.js, it will look something like this:

Node.js


async function say_after(delay, what) {
  await sleep(delay);
  console.log(what);
}

async function main() {
  console.log(`started at ${new Date().toTimeString()}`);

  await say_after(1, 'hello');
  await say_after(2, 'world');

  console.log(`finished at ${new Date().toTimeString()}`);
}

main();

It looks the same, so it's easy to understand.

Concurrency

In Python, Tasks allow you to run coroutines in parallel.

Python


import asyncio
import time

async def say_after(delay, what):
    await asyncio.sleep(delay)
    print(what)

async def main():
    task1 = asyncio.create_task(
        say_after(1, 'hello'))

    task2 = asyncio.create_task(
        say_after(2, 'world'))

    print(f"started at {time.strftime('%X')}")

    await task1
    await task2

    print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

In this example, unlike the previous example, the operation of "waiting for 2 seconds and then outputting" world "" is performed at the same time as the operation of "waiting for 1 second and then outputting" hello "", so 1 than the previous example. It will end a second earlier. In Python, you can wrap a coroutine as a Task with ʻasyncio.create_task ()` and schedule its execution.

Written in Node.js, it will look something like this:

Node.js


async function say_after(delay, what) {
  await sleep(delay);
  console.log(what);
}

async function main() {
  const task1 = say_after(1, 'hello');
  const task2 = say_after(2, 'world');

  console.log(`started at ${new Date().toTimeString()}`);

  await Promise.all([task1, task2]);

  console.log(`finished at ${new Date().toTimeString()}`);
}

main();

For Node.js, you can use Promise.all ().


Let's look at another example.

Python


import asyncio

async def factorial(name, number):
    f = 1
    for i in range(2, number + 1):
        print(f"Task {name}: Compute factorial({i})...")
        await asyncio.sleep(1)
        f *= i
    print(f"Task {name}: factorial({number}) = {f}")

async def main():
    await asyncio.gather(
        factorial("A", 2),
        factorial("B", 3),
        factorial("C", 4),
    )

asyncio.run(main())

This is a function that calculates the factorial by delaying the multiplication by 1 second each time. This time, we see ʻasyncio.gather () `, which also schedules the argument coroutine as a Task.

Written in Node.js, it will look something like this:

Node.js


async function factorial(name, number) {
  let f = 1;
  for (let i=2;i<=number;++i) {
    console.log(`Task ${name}: Compute factorial(${i})...`);
    await sleep(1);
    f *= i;
  }
  console.log(`Task ${name}: factorial(${number}) = ${f}`);
}

async function main() {
  await Promise.all([
    factorial("A", 2),
    factorial("B", 3),
    factorial("C", 4)
  ]);
}

main();

time out

In Python asynchronous processing, you can use ʻasyncio.wait_for ()` to treat it as a timeout if the processing is not completed in a certain amount of time.

Python


import asyncio

async def eternity():
    await asyncio.sleep(3600)
    print('yay!')

async def main():
    try:
        await asyncio.wait_for(eternity(), timeout=1.0)
    except asyncio.TimeoutError:
        print('timeout!')

asyncio.run(main())

For example, in the example above, the function ʻeternity ()` that sleeps for 3600 seconds actually times out after 1 second without waiting for 3600 seconds.

I don't think there is a concise way to implement this in Node.js (as far as I know).

Certainly, if you use Promise.race (),

await Promise.race(eternity(), timeout(1.0))
  .catch(err => {
    console.log('timeout!');
  })

(Timeout (sec) is a function that returns reject after sec seconds). However, in this implementation, after 1 second, "timeout!" Is displayed, but ʻeternity ()` continues to wait.

See also: Use Promise.race to Timeout Promises

Caveat: Cleanup The timeout does not cause the other promise to clean up or cancel. For example, if a database-write promise were to be Promise.race ‘d against a timeout, and if the timeout completed first, then the database write operation would still be running and may (eventually) succeed, but the rest of your application will think it failed. Make sure to introduce logic to cancel the operation in case your application initiates a timeout. Handling cleanup and canceling logic is a tricky subject, as the feature is not built into JavaScript.

With that in mind, it's quite convenient in Python to be able to write a concise description of asynchronous processing interruptions.

Recommended Posts

Learn Python asynchronous processing / coroutines by comparing with Node.js
Learn Python with ChemTHEATER
Communication processing by Python
Image processing with Python
100 Language Processing with Python Knock 2015
"Apple processing" with OpenCV3 + Python3
Acoustic signal processing with Python (2)
Acoustic signal processing with Python
Asynchronous processing (threading) in python
[Python] Image processing with scikit-image
Improve your productivity by processing huge Excel files with Python
[Python] Easy parallel processing with Joblib
100 Language Processing Knock with Python (Chapter 1)
Paiza Python Primer 3: Learn Loop Processing
Learn search with Python # 2bit search, permutation search
Learn Python by drawing (turtle graphics)
100 Language Processing Knock with Python (Chapter 3)
Image processing with Python 100 knocks # 3 Binarization
[Python] Asynchronous request with async / await
Image processing with Python 100 knocks # 2 Grayscale
100 Language Processing Knock Chapter 1 by Python
Basics of binarized image processing with Python
Get property information by scraping with python
Image processing with Python 100 knock # 10 median filter
Getting Started with python3 # 1 Learn Basic Knowledge
Reintroduction to Python Decorators ~ Learn Decorators by Type ~
Socket communication and multi-thread processing by Python
Image processing by Python 100 knock # 1 channel replacement
Save video frame by frame with Python OpenCV
Image processing with Python 100 knocks # 8 Max pooling
Periodically perform arbitrary processing with Python Twisted
Let Heroku do background processing with Python
100 Language Processing Knock with Python (Chapter 2, Part 2)
Grayscale by matrix-Reinventor of Python image processing-
3. Natural language processing with Python 2-1. Co-occurrence network
Image processing with Python 100 knock # 12 motion filter
100 image processing by Python Knock # 6 Color reduction processing
3. Natural language processing with Python 1-1. Word N-gram
Learn Python! Comparison with Java (basic function)
100 Language Processing Knock with Python (Chapter 2, Part 1)
Drawing with Matrix-Reinventor of Python Image Processing-
Easy image processing in Python with Pillow
Organize data divided by folder with Python
Learn the design pattern "Singleton" with Python
Image processing with Python 100 knocks # 7 Average pooling
Light image processing with Python x OpenCV
Learn the design pattern "Facade" with Python
Image processing with Python 100 knocks # 9 Gaussian filter
Getting started with Python with 100 knocks on language processing
Asynchronous processing in Python: asyncio reverse lookup reference
Stock number ranking by Qiita tag with python
3. Natural language processing with Python 2-2. Co-occurrence network [mecab-ipadic-NEologd]
How to do multi-core parallel processing with python
Image processing from scratch with python (5) Fourier transform
Draw a graph by processing with Pandas groupby
[Python] I played with natural language processing ~ transformers ~
Image processing from scratch with python (4) Contour extraction
What are you comparing with Python is and ==?
Asynchronous processing with Arduino (Asynchronous processing of processing requests from Linux)
Image processing by Python 100 knock # 11 smoothing filter (average filter)
Parallel processing with no deep meaning in Python