It is a memorandum.
2020/5/21 Added description about python3.8
sample.py
import os
def hoge(filename):
#Read the file and do some processing.
return data
data = [hoge(filename) for filename in os.listdir(".")]
I would like to do something like that, but if I turn this code as it is, it will use only one core, so it is a waste of time. I want to do it in parallel.
sample.py
from multiprocessing import Pool
import os
def hoge(filename):
#Read the file and do some processing.
#This function can be left as it is
return data
with Pool() as p:
data = p.map(hoge, os.listdir("."))
Works in parallel with.
sample.py
from tqdm import tqdm
from multiprocessing import Pool
import os
with Pool() as p:
imap = p.imap(hoge, os.listdir("."))
data = list(tqdm(imap, total=len(os.listdir("."))))
It's stuck for some reason. This is a provisional response,
sample.py
from multiprocessing import Pool
import os
def hoge(filename):
#Read the file and do some processing.
#This function can be left as it is
return data
if __name__=="__main__":
with Pool() as p:
data = p.map(hoge, os.listdir("."))
It seems to work by doing. However, it seems that the `` `datavariable created with much effort can be referenced only within the scope ofif name ==" main "```.
Also, previously it was possible to execute in parallel on Jupyter, but it is not possible in Python 3.8.
Recommended Posts