Parallel computing with iPython notebook
Using map in multiprocessing.Pool in iPython I don't want to use it as much as possible because Python becomes a zombie when the kernel is stopped. Therefore, parallelize with the default function of iPython.
Here is the minimum usage of parallelization by iPyhton. See [Parallel Computing with IPython] for details.
Postscript
By default, ipython notebook has moved to jupyter You are now required to pip install ipyparallel on the cluster startup screen. (It has been migrated for a long time, but I will add it by this time 2016/1/19)
Install command
pip install ipyparallel
Furthermore, it will not be effective unless you add the following line to jupyter_notebook_config.py. .. ..
c.NotebookApp.server_extensions.append('ipyparallel.nbextension')
Although jupyter_notebook_config.py is in .jupyter in your home directory. .. .. .. It seems that it is not generated by default unless you execute the following one line. .. ..
jupyter notebook --generate-config
Start the cluster
Select Clusters from the screen when iPython notebook is started. Set and start the number of engines corresponding to the profile specified when notebook is started as shown below.
Code execution
import numpy as np
from IPython import parallel
clients = parallel.Client()
#Synchronous execution(No control is returned during parallel computing)
clients.block = True
view = clients.load_balanced_view()
dview = clients[:]
#cluster(engine?)Check the number of
print clients.ids
def func(n):
return np.arange(n)
#cluster(engine?)To import numpy
dview.execute('import numpy as np')
#Run func in parallel
view.map(func, np.arange(3))
References
- Parallel Computing with IPython [Parallel Computing with IPython]: http://www.astro.washington.edu/users/vanderplas/Astr599/notebooks/21_IPythonParallel
Recommended Posts