[PYTHON] Parallel processing with multiprocessing

#! /usr/local/bin/python
# -*- coding:utf-8

import multiprocessing

def plus_data(num):
    return num+1

def multi_plus_one(before_list):
    pool = multiprocessing.Pool(processes=4)
    return pool.map(plus_data, before_list)

if __name__ == "__main__":
    before_list = range(10)
    print before_list
    #All on the elements of the list+1
    print multi_plus_one(before_list)

Like this. If you just want to calculate in parallel, maybe multiprocessing.Process

#! /usr/local/bin/python
# -*- coding:utf-8

import multiprocessing

def plus_data(num, plus_num):
    return num + plus_num

#Wrapper for multi pool
def wrapper_plus_data(args):
    return plus_data(*args)

def multi_plus_one(before_list):
    pool = multiprocessing.Pool(processes=4)
    return pool.map(wrapper_plus_data, before_list)

if __name__ == "__main__":
    before_list = [(i, 2) for i in range(10)]
    #All on the elements of the list+2
    print multi_plus_one(before_list)

If you want to pass multiple arguments, do something like this

Recommended Posts

Parallel processing with multiprocessing
Parallel processing with local functions
Parallel processing with Parallel of scikit-learn
[Python] Easy parallel processing with Joblib
Python parallel processing (multiprocessing and Joblib)
Image processing with MyHDL
How to do multi-core parallel processing with python
Processing datasets with pandas (1)
Processing datasets with pandas (2)
Image processing with Python
Parallel processing with no deep meaning in Python
SMP parallel with OpenMP
Image Processing with PIL
Image processing with Python (Part 2)
100 Language Processing with Python Knock 2015
Image processing with PIL (Pillow)
"Apple processing" with OpenCV3 + Python3
Acoustic signal processing with Python (2)
Blender Modal Operator parallel processing
Acoustic signal processing with Python
Image processing with Python (Part 1)
Image processing with Python (Part 3)
Data processing tips with Pandas
[Python] Image processing with scikit-image
Real-time image processing basics with opencv
100 Language Processing Knock with Python (Chapter 1)
Try audio signal processing with librosa-Beginner
100 Language Processing Knock with Python (Chapter 3)
Image processing with Python 100 knocks # 3 Binarization
IPython cluster stupid (distributed parallel processing)
Path processing with takewhile and dropwhile
Python parallel / parallel processing sample code summary
Read files in parallel with Python
Image processing with Python 100 knocks # 2 Grayscale
Arithmetic processing with Chinese numeral class
Easy parallel execution with python subprocess
Creating an exe file with Python PyInstaller: PC freezes in parallel processing
Receive a list of the results of parallel processing in Python with starmap
Sharing list type data between processes by parallel processing using Python Multiprocessing
Basics of binarized image processing with Python
Image processing with Python 100 knock # 10 median filter
100 Language Processing Knock 2020 with GiNZA v3.1 Chapter 4
Create an image processing viewer with PySimpleGUI
Image processing with Python 100 knocks # 8 Max pooling
Periodically perform arbitrary processing with Python Twisted
Let Heroku do background processing with Python
100 Language Processing Knock with Python (Chapter 2, Part 2)
Image processing with Python & OpenCV [Tone Curve]
Image processing with Python 100 knock # 12 motion filter
3. Natural language processing with Python 1-1. Word N-gram
Smoother pipeline processing with Luigi! Introducing gokart
100 Language Processing Knock with Python (Chapter 2, Part 1)
Drawing with Matrix-Reinventor of Python Image Processing-
Easy image processing in Python with Pillow
Image processing with Python 100 knocks # 7 Average pooling
I tried natural language processing with transformers.
Light image processing with Python x OpenCV
Image processing with Lambda + OpenCV (gray image creation)
Batch processing forms with Azure Form Recognizer
Example of efficient data processing with PANDAS
Image processing with Python 100 knocks # 9 Gaussian filter