[PYTHON] parallelization of class method

Class method parallelization and process monitoring

In python, parallelization can be easily realized by using multiprocessing.Pool of multiprocessing which is a parallelization library, but it cannot be used for parallelization such as class method. In this case, there is no choice but to parallelize using multiprocessing.Process, but if you run processes for each number of CPUs, when you run more processes than the number of CPUs, the CPU of the process that ended earlier will be freed and it will be wasted. It ends up. I wondered if I could run a new process from a CPU that ended up like SGE, so I forcibly created a script for it. Maybe there is a better way.

An example of a parallel process management script when parallelizing the class Method.

import multiprocessing as mp
ps = []
for i in range(1000):
    ps.append(mp.Process(target=hoge,args=(i,))) #Generate 1000 target functions or methods

finishedList       = []
presentindex       = 0
for p in ps[0:mp.cpu_count()]:
    p.start()
    presentindex += 1

while 1:
    time.sleep(1)#Monitor every second.
    for i, p in enumerate(ps[:presentindex]):
        if p.is_alive() or i in finishedList:#Judgment of end of process p
            pass
        else:
            finishedList.append(i)#Save index of terminated process
            if presentindex < len(ps):
                ps[presentindex].start()
                presentindex += 1
                print(i)

    if len(finishedList) == len(ps):#Exit if the number of terminated processes and the number of created processes are the same
        break