[PYTHON] About "AssertionError: daemonic processes are not allowed to have children"

environment

problem

I'm running in parallel using multiprocessing.pool.Pool. In the cube_for_row function running in parallel, when I run it in parallel, I get" AssertionError: daemonic processes are not allowed to have children ".

main.py


from typing import List
import multiprocessing


def cube(value: float) -> float:
    return value * value * value


def cube_for_row(row: List[float]) -> List[float]:
    with multiprocessing.Pool() as pool:
        return pool.map(cube, row)


def cube_for_matrix(matrix: List[List[float]], is_parallel:bool) -> List[List[float]]:
    if is_parallel:
        with multiprocessing.Pool() as pool:
            return pool.map(cube_for_row, matrix)
    else:
        return [cube_for_row(row) for row in matrix]


def main():
    matrix = [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]]
    result1 = cube_for_matrix(matrix, False)
    print(result1)
    # OK
    result2 = cube_for_matrix(matrix, True)
    print(result2)
    # NG
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/vagrant/Downloads/run.py", line 8, in cube_for_row
    with multiprocessing.Pool() as p:
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 212, in __init__
    self._repopulate_pool()
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "/home/vagrant/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
AssertionError: daemonic processes are not allowed to have children
"""

Cause

The error message says "Daemon process cannot have child process". The function I'm passing to multiprocessing.pool.Pool seems to be running in a daemon process.

The multiprocessing.Process.daemon documentation states that "daemon processes cannot create child processes".

A Boolean flag that indicates whether it is a daemon process. This attribute must be set before start () is called. The initial value is inherited from the creating process. When a process terminates, it attempts to terminate all of its child daemon processes.

Note that the daemon process cannot create child processes. If it can be created, the child process of the daemon process may become an orphan when the parent process of the daemon process terminates. Furthermore, daemon processes are regular processes, not Unix daemons or services, and are terminated (and not joined) when a non-daemon process terminates.

Temporary solution 1

Since it is an "Assertion Error", it can be ignored.

$ python -O main.py
[[1, 8, 27, 64], [125, 216, 343, 512], [729, 1000, 1331, 1728]]

If you just ignore it, it seems that a zombie process will be born.

Temporary solution 2

For daemon processes, do not process in parallel.

def cube_for_row(row: List[float]) -> List[float]:
    """If it can be executed in parallel, execute it in parallel"""
    process = multiprocessing.current_process()
    if process.daemon:
        return [cube(v) for v in row]
    else:
        with multiprocessing.Pool() as pool:
            return pool.map(cube, row)


Recommended Posts

About "AssertionError: daemonic processes are not allowed to have children"
To people who are "recruiting but not recruiting"