There are many methods of multi-process Pool in Python3 and I didn't know which one to use, so I looked it up.
The map function blocks until all results are available.
When I run the code
map.py
from multiprocessing import Pool
from time import sleep
import time
start = time.time()
def do(waitTime):
print("do waitTime:{}From the beginning{}Seconds have passed".format(waitTime,time.time() - start))
sleep(waitTime)
return waitTime
if __name__ == '__main__':
waitTimes = [3,2,1]
with Pool(10) as p:
for result in p.map(do,waitTimes):
print("result waitTime:{}From the beginning{}Seconds have passed".format(result,time.time() - start))
Result is
do waitTime:3 0 from the start.035417079925537 11 seconds have passed
do waitTime:2 0 from the start.03555130958557129 seconds have passed
do waitTime:1 from the start 0.03568387031555176 seconds have passed
result waitTime:3 from the beginning 3.0372514724731445 seconds have passed
result waitTime:2 from the beginning 3.0373241901397705 seconds have passed
result waitTime:1 from the beginning 3.037338972091675 seconds have passed
for result in p.map (do, waitTimes):
The return value of the line is list type ** The for statement will not be executed until all processes are finished **
The imap function is a lazy evaluation version of the map function. The return value of the imap function returns an iterator. If there is a process whose return value is fixed, the for statement can be executed without waiting for the completion of other processes. However, if the first process is not terminated due to the restriction that it must be returned in the same order as it was passed, the for statement will not be executed indefinitely even if other processing is completed.
imap.py
from multiprocessing import Pool
from time import sleep
import time
start = time.time()
def do(waitTime):
print("do waitTime:{}From the beginning{}Seconds have passed".format(waitTime,time.time() - start))
sleep(waitTime)
return waitTime
if __name__ == '__main__':
waitTimes = [3,2,1]
with Pool(10) as p:
for result in p.imap(do,waitTimes):
print("result waitTime:{}From the beginning{}Seconds have passed".format(result,time.time() - start))
do waitTime:3 0 from the start.03740239143371582 seconds have passed
do waitTime:2 0 from the start.03748369216918945 seconds have passed
do waitTime:1 from the start 0.0376131534576416 seconds have passed
result waitTime:3 from the beginning 3.041029930114746 seconds have passed
result waitTime:2 from the beginning 3.041118621826172 seconds have passed
result waitTime:1 from the beginning 3.0411417484283447 seconds have passed
It is the same as imap (), except that the order of the results returned by the iterator is considered to be arbitrary.
imap.py
from multiprocessing import Pool
from time import sleep
import time
start = time.time()
def do(waitTime):
print("do waitTime:{}From the beginning{}Seconds have passed".format(waitTime,time.time() - start))
sleep(waitTime)
return waitTime
if __name__ == '__main__':
waitTimes = [3,2,1]
with Pool(10) as p:
for result in p.imap_unordered(do,waitTimes):
print("result waitTime:{}From the beginning{}Seconds have passed".format(result,time.time() - start))
do waitTime:3 0 from the start.03511857986450 195 seconds have passed
do waitTime:2 0 from the start.035273075103759766 seconds have passed
do waitTime:1 from the start 0.035429954528808594 seconds have passed
result waitTime:1 from the beginning 1.0369133949279785 seconds have passed
result waitTime:2 from the beginning 2.0377516746520996 seconds have passed
result waitTime:3 from the beginning 3.038750171661377 seconds have passed
Oops! If there is a process that has completed processing, it will be executed without waiting for other processing!
Python 3 multi-process Pool methods should use imap_unordered
Recommended Posts