--Speaking of python parallel processing, the multiprocess module --Use map when you want the result in a list ――But it seems that you can't pass multiple arguments in map ――For numerical experiments such as machine learning, I often experiment with different arguments, so one is a little. .. --In Python 2.X, in order to pass multiple arguments to map, it was necessary to define a wrapper function and write it to be passed, but since Python 3.3, it is easier to write with starmap. --This is my memo on how to use starmap.
#!/usr/bin/env python3
from itertools import product
from multiprocessing import Pool
#Functions you want to process in parallel
def multi_test(arg1, arg2, arg3):
return(arg1+arg2+arg3)
#Decide the number of parallels and prepare Pool
pool = Pool(3)
#Prepare argument variations with a list of tuples
multi_args = []
for iter1, iter2, iter3 in product(range(1,4), range(1,4), range(1,4)):
multi_args.append((iter1,iter2,iter3))
print(multi_args)
[(1, 1, 1), (1, 1, 2), (1, 1, 3), (1, 2, 1), (1, 2, 2), (1, 2, 3), (1, 3, 1), (1, 3, 2), (1, 3, 3), (2, 1, 1), (2, 1, 2), (2, 1, 3), (2, 2, 1), (2, 2, 2), (2, 2, 3), (2, 3, 1), (2, 3, 2), (2, 3, 3), (3, 1, 1), (3, 1, 2), (3, 1, 3), (3, 2, 1), (3, 2, 2), (3, 2, 3), (3, 3, 1), (3, 3, 2), (3, 3, 3)]
#Parallel processing execution
results = pool.starmap(multi_test, multi_args)
print(results)
[3, 4, 5, 4, 5, 6, 5, 6, 7, 4, 5, 6, 5, 6, 7, 6, 7, 8, 5, 6, 7, 6, 7, 8, 7, 8, 9]
――I was able to write it simply. I've just started using it, so please tell me what's wrong.
Recommended Posts