Python 3 multi-process Pool methods should use imap_unordered

There are many methods of multi-process Pool in Python3 and I didn't know which one to use, so I looked it up.

map function

The map function blocks until all results are available.

When I run the code

map.py


from multiprocessing import Pool
from time import sleep
import time

start = time.time()

def do(waitTime):
    print("do waitTime:{}From the beginning{}Seconds have passed".format(waitTime,time.time() - start))
    sleep(waitTime)
    return waitTime
 
if __name__ == '__main__':
    waitTimes = [3,2,1]
    with Pool(10) as p:
        for result in p.map(do,waitTimes):
            print("result waitTime:{}From the beginning{}Seconds have passed".format(result,time.time() - start))

Result is

do waitTime:3 0 from the start.035417079925537 11 seconds have passed
do waitTime:2 0 from the start.03555130958557129 seconds have passed
do waitTime:1 from the start 0.03568387031555176 seconds have passed
result waitTime:3 from the beginning 3.0372514724731445 seconds have passed
result waitTime:2 from the beginning 3.0373241901397705 seconds have passed
result waitTime:1 from the beginning 3.037338972091675 seconds have passed

for result in p.map (do, waitTimes): The return value of the line is list type ** The for statement will not be executed until all processes are finished **

imap function

The imap function is a lazy evaluation version of the map function. The return value of the imap function returns an iterator. If there is a process whose return value is fixed, the for statement can be executed without waiting for the completion of other processes. However, if the first process is not terminated due to the restriction that it must be returned in the same order as it was passed, the for statement will not be executed indefinitely even if other processing is completed.

imap.py


from multiprocessing import Pool
from time import sleep
import time

start = time.time()
 
def do(waitTime):
    print("do waitTime:{}From the beginning{}Seconds have passed".format(waitTime,time.time() - start))
    sleep(waitTime)
    return waitTime
 
if __name__ == '__main__':
    waitTimes = [3,2,1]
    with Pool(10) as p:
        for result in p.imap(do,waitTimes):
            print("result waitTime:{}From the beginning{}Seconds have passed".format(result,time.time() - start))
do waitTime:3 0 from the start.03740239143371582 seconds have passed
do waitTime:2 0 from the start.03748369216918945 seconds have passed
do waitTime:1 from the start 0.0376131534576416 seconds have passed
result waitTime:3 from the beginning 3.041029930114746 seconds have passed
result waitTime:2 from the beginning 3.041118621826172 seconds have passed
result waitTime:1 from the beginning 3.0411417484283447 seconds have passed

imap_unordered function

It is the same as imap (), except that the order of the results returned by the iterator is considered to be arbitrary.

imap.py


from multiprocessing import Pool
from time import sleep
import time

start = time.time()
 
def do(waitTime):
    print("do waitTime:{}From the beginning{}Seconds have passed".format(waitTime,time.time() - start))
    sleep(waitTime)
    return waitTime
 
if __name__ == '__main__':
    waitTimes = [3,2,1]
    with Pool(10) as p:
        for result in p.imap_unordered(do,waitTimes):
            print("result waitTime:{}From the beginning{}Seconds have passed".format(result,time.time() - start))
do waitTime:3 0 from the start.03511857986450 195 seconds have passed
do waitTime:2 0 from the start.035273075103759766 seconds have passed
do waitTime:1 from the start 0.035429954528808594 seconds have passed
result waitTime:1 from the beginning 1.0369133949279785 seconds have passed
result waitTime:2 from the beginning 2.0377516746520996 seconds have passed
result waitTime:3 from the beginning 3.038750171661377 seconds have passed

Oops! If there is a process that has completed processing, it will be executed without waiting for other processing!

Summary

Python 3 multi-process Pool methods should use imap_unordered

Recommended Posts

Python 3 multi-process Pool methods should use imap_unordered
[Python] About multi-process
You should know if you use Python! 10 useful libraries
Three Reasons Why Machine Learning Learners Should Use Python
Use thingsspeak from python
Use config.ini in Python
Use fluentd from python
Use dates in Python
Convenient Python methods, etc.
[python] class basic methods
Use Valgrind in Python
Use MySQL from Python
Use mecab with Python3
Use LiquidTap Python Client ③
Use DynamoDB with Python
Multi-process asynchronously with python
Use Python 3.8 with Anaconda
[Python] format methodical use
Use python with docker
Use MySQL from Python
Use LiquidTap Python Client ②
Use BigQuery from python.
Use profiler in Python
Use mecab-ipadic-neologd from python
Use LiquidTap Python Client ①
[Python] DI (Dependency Injection) container Which one should I use?