[PYTHON] Parallel computing (pathos) when dealing with objects that cannot be pickled

Overview

We will introduce the solution when the following error is encountered when performing process parallel calculation using joblib and multiprocessing. The cause is that it cannot be pickled, which is used for interprocess communication.

error1}

TypeError: cannot pickle 'SwigPyObject' object

error2}

PicklingError: Could not pickle the task to send it to the workers.

Solution

Use pathos. You can install it with pip install pathos.

code}

# some_func, some_Please create a list appropriately
from pathos.multiprocessing import ProcessingPool
pool = ProcessingPool(nodes=4)
result_list = pool.map(lambda x: some_func(x), some_list)

bonus

If it is pickle at the time of saving, it can be solved by using dill.

reference

• https://pathos.readthedocs.io/en/latest/
• https://stackoverflow.com/questions/19984152/what-can-multiprocessing-and-dill-do-together
• http://nishidy.hatenablog.com/entry/2016/02/01/225750