[python, multiprocessing] Behavior for exceptions when using multiprocessing

Although multiprocessing is used for parallel processing in Python, the behavior when a child process is created by multiprocessing has some differences from the behavior of a normal function call.

Preparation

This time, the function sleep_bug () is used for simplicity. This is a function that executes 1 / 0 at ʻi == 5` to generate an error in order to intentionally generate an error during execution.

import time

def sleep_bug():
    for i in range(10):
        print('sleeping %d' % i)
        if i == 5:
            1/0
        time.sleep(1)
    return i

sleep_bug()

 output
'''
sleeping 0
sleeping 1
sleeping 2
sleeping 3
sleeping 4
sleeping 5
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-44-d9f02a4cf7f3> in <module>
----> 1 sleep_bug()

<ipython-input-41-26bb27998e63> in sleep_bug()
     12         print('sleeping %d' % i)
     13         if i==5:
---> 14             1/0
     15         time.sleep(1)
     16 

ZeroDivisionError: division by zero
'''

If the child process makes an error, the parent process will continue to run.

In a normal function call, if an error occurs in the called function, the program will stop there. However, if you let a child process created using Pool execute sleep_bug (), the child process will stop with an error, but the parent process will proceed to the end without any error.

from multiprocessing import Pool
p = Pool(1)
r = p.apply_async(sleep_bug)
p.close()
 p.join () # Wait for the child process to finish.
print('Done')

 output
'''
sleeping 0
sleeping 1
sleeping 2
sleeping 3
sleeping 4
sleeping 5
Done
'''

Use r.get () to make the error propagate to the parent process when the child process stops due to an error. r.get () is a function that normally waits for the child process to terminate and outputs the return value of the child process, but if an error occurs in the child process, r.get () Will throw an exception and the parent process will also stop there.

from multiprocessing import Pool
p = Pool(1)
r = p.apply_async(sleep_bug)
p.close()
output = r.get()
print('Done %d' % output)

# output
'''
sleeping 0
sleeping 1
sleeping 2
sleeping 3
sleeping 4
sleeping 5
---------------------------------------------------------------------------
RemoteTraceback                           Traceback (most recent call last)
RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "<ipython-input-41-26bb27998e63>", line 14, in sleep_bug
    1/0
ZeroDivisionError: division by zero
"""

The above exception was the direct cause of the following exception:

ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-50-fb8f5892e1a7> in <module>
      3     r = p.apply_async(sleep_bug)
      4     p.close()
----> 5     output = r.get()
      6     print('Done %d' % output)

/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py in get(self, timeout)
    655             return self._value
    656         else:
--> 657             raise self._value
    658 
    659     def _set(self, i, obj):

ZeroDivisionError: division by zero
'''

If you use Process, an error will be displayed, but the parent process will proceed to the end.

from multiprocessing import Process
p = Process(target=sleep_bug)
p.start()
p.join()
print('Done')

# output
'''
sleeping 0
sleeping 1
sleeping 2
sleeping 3
sleeping 4
sleeping 5
Process Process-35:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-41-26bb27998e63>", line 14, in sleep_bug
    1/0
ZeroDivisionError: division by zero
Done
'''

The child process keeps running even if the parent process stops

In the script below, the child process waits 2 seconds while it is running, and the parent process terminates itself there with sys.exit (). As shown in the execution example below, even if the parent process stops in the middle, the child process continues to run.

from multiprocessing import Pool
p = Pool(1)
r = p.apply_async(sleep)
p.close()
 r.wait (2) #wait 2 seconds for child process
sys.exit()

# output
'''
sleeping 0
sleeping 1
An exception has occurred, use %tb to see the full traceback.

SystemExit
sleeping 2
sleeping 3
sleeping 4
sleeping 5
sleeping 6
sleeping 7
sleeping 8
sleeping 9
'''