Take a look at the Python built-in exception tree structure

Introduction

When implementing a custom exception for my module, I was interested in how the tree structure of built-in exceptions differs between Python 2 and 3. So I decided to investigate the difference in the tree structure of each version.

In conclusion, it turned out that "the tree structure of exceptions differs greatly depending on the version even in the 3rd system, not in the first place the distinction between the 2nd system and the 3rd system". Please note that if you think that the tree structure of exceptions is the same in 3 series, you may have unexpected troubles.

Code used for this survey

print_exc_tree.py


#!/usr/bin/env python

from __future__ import print_function
import platform


def classtree(cls, depth=0):
    if depth == 0:
        prefix = ''
    else:
        prefix = '.' * (depth * 3) + ' '
    if cls.__name__.lower() == 'error':
        print('{0}{1} ({2})'.format(prefix, cls.__name__, cls))
    else:
        print('{0}{1}'.format(prefix, cls.__name__))
    for subcls in sorted(cls.__subclasses__(), key=lambda c: c.__name__):
        classtree(subcls, depth+1)


if __name__ == '__main__':
    print('Python Version: {0}'.format(platform.python_version()))
    print()
    classtree(BaseException)

2.7.12 V.S. 3.5.2

Compare the latest two at the time of writing. Basically, it should be OK if you are aware of this difference in tree structure.

$ pyenv shell 2.7.12
$ python print_exc_tree.py
Python Version: 2.7.12

BaseException
... Exception
...... Error (<class 'locale.Error'>)
...... StandardError
......... ArithmeticError
............ FloatingPointError
............ OverflowError
............ ZeroDivisionError
......... AssertionError
......... AttributeError
......... BufferError
......... EOFError
......... EnvironmentError
............ IOError
............... ItimerError
............ OSError
......... ImportError
............ ZipImportError
......... LookupError
............ CodecRegistryError
............ IndexError
............ KeyError
......... MemoryError
......... NameError
............ UnboundLocalError
......... ReferenceError
......... RuntimeError
............ NotImplementedError
......... SyntaxError
............ IndentationError
............... TabError
......... SystemError
............ CodecRegistryError
......... TypeError
......... ValueError
............ UnicodeError
............... UnicodeDecodeError
............... UnicodeEncodeError
............... UnicodeTranslateError
...... StopIteration
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... _OptionError
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

$ pyenv shell 3.5.2
$ python print_exc_tree.py
Python Version: 3.5.2

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... EOFError
...... Error (<class 'locale.Error'>)
...... ImportError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... OSError
......... BlockingIOError
......... ChildProcessError
......... ConnectionError
............ BrokenPipeError
............ ConnectionAbortedError
............ ConnectionRefusedError
............ ConnectionResetError
......... FileExistsError
......... FileNotFoundError
......... InterruptedError
......... IsADirectoryError
......... ItimerError
......... NotADirectoryError
......... PermissionError
......... ProcessLookupError
......... TimeoutError
......... UnsupportedOperation
...... ReferenceError
...... RuntimeError
......... BrokenBarrierError
......... NotImplementedError
......... RecursionError
......... _DeadlockError
...... StopAsyncIteration
...... StopIteration
...... StopTokenizing
...... SubprocessError
......... CalledProcessError
......... TimeoutExpired
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TokenError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... ResourceWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... _OptionError
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

I'm a little worried about sre_constants.error, but it seems to be used when the regular expression is ambiguous (Reference)

Python 3.5.2 (default, Jul 29 2016, 11:13:25)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.compile("*kittens*")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dmiyakawa/.pyenv/versions/3.5.2/lib/python3.5/re.py", line 224, in compile
    return _compile(pattern, flags)
  File "/Users/dmiyakawa/.pyenv/versions/3.5.2/lib/python3.5/re.py", line 293, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/Users/dmiyakawa/.pyenv/versions/3.5.2/lib/python3.5/sre_compile.py", line 536, in compile
    p = sre_parse.parse(p, flags)
  File "/Users/dmiyakawa/.pyenv/versions/3.5.2/lib/python3.5/sre_parse.py", line 829, in parse
    p = _parse_sub(source, pattern, 0)
  File "/Users/dmiyakawa/.pyenv/versions/3.5.2/lib/python3.5/sre_parse.py", line 437, in _parse_sub
    itemsappend(_parse(source, state))
  File "/Users/dmiyakawa/.pyenv/versions/3.5.2/lib/python3.5/sre_parse.py", line 638, in _parse
    source.tell() - here + len(this))
sre_constants.error: nothing to repeat at position 0

Old Python 3 exception system

If you investigate, you can see that this tree structure has changed significantly depending on the version even in the 3rd system. It may seem a bit verbose, but let's look at 3.0, 3.1, 3.2, 3.3, 3.4 in order.

$ pyenv shell 3.0.1
$ python print_exc_tree.py
Python Version: 3.0.1

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... EOFError
...... EnvironmentError
......... IOError
............ BlockingIOError
............ ItimerError
............ UnsupportedOperation
......... OSError
...... ImportError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... ReferenceError
...... RuntimeError
......... NotImplementedError
...... StopIteration
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... error (<class '_thread.error'>)
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

$ pyenv shell 3.1.5
$ python print_exc_tree.py
Python Version: 3.1.5

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... EOFError
...... EnvironmentError
......... IOError
............ BlockingIOError
............ ItimerError
............ UnsupportedOperation
......... OSError
...... Error (<class 'locale.Error'>)
...... ImportError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... ReferenceError
...... RuntimeError
......... NotImplementedError
...... StopIteration
...... StopTokenizing
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TokenError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

$ pyenv shell 3.2.6
$ python print_exc_tree.py
Python Version: 3.2.6

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... CalledProcessError
...... EOFError
...... EnvironmentError
......... IOError
............ BlockingIOError
............ ItimerError
............ UnsupportedOperation
......... OSError
...... Error (<class 'locale.Error'>)
...... ImportError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... PickleError
......... PicklingError
......... UnpicklingError
...... PickleError
......... PicklingError
......... UnpicklingError
...... ReferenceError
...... RuntimeError
......... NotImplementedError
...... StopIteration
...... StopTokenizing
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TokenError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... ResourceWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... _OptionError
...... _Stop
...... error (<class 'sre_constants.error'>)
...... error (<class '_thread.error'>)
...... error (<class 'select.error'>)
...... error (<class 'struct.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

$ pyenv shell 3.3.6
$ python print_exc_tree.py
Python Version: 3.3.6

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... EOFError
...... Error (<class 'locale.Error'>)
...... ImportError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... OSError
......... BlockingIOError
......... ChildProcessError
......... ConnectionError
............ BrokenPipeError
............ ConnectionAbortedError
............ ConnectionRefusedError
............ ConnectionResetError
......... FileExistsError
......... FileNotFoundError
......... InterruptedError
......... IsADirectoryError
......... ItimerError
......... NotADirectoryError
......... PermissionError
......... ProcessLookupError
......... TimeoutError
......... UnsupportedOperation
...... ReferenceError
...... RuntimeError
......... NotImplementedError
......... _DeadlockError
...... StopIteration
...... StopTokenizing
...... SubprocessError
......... CalledProcessError
......... TimeoutExpired
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TokenError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... ResourceWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... _OptionError
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

$ pyenv shell 3.4.3
$ python print_exc_tree.py
Python Version: 3.4.3

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... EOFError
...... Error (<class 'locale.Error'>)
...... ImportError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... OSError
......... BlockingIOError
......... ChildProcessError
......... ConnectionError
............ BrokenPipeError
............ ConnectionAbortedError
............ ConnectionRefusedError
............ ConnectionResetError
......... FileExistsError
......... FileNotFoundError
......... InterruptedError
......... IsADirectoryError
......... ItimerError
......... NotADirectoryError
......... PermissionError
......... ProcessLookupError
......... TimeoutError
......... UnsupportedOperation
...... ReferenceError
...... RuntimeError
......... BrokenBarrierError
......... NotImplementedError
......... _DeadlockError
...... StopIteration
...... StopTokenizing
...... SubprocessError
......... CalledProcessError
......... TimeoutExpired
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TokenError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... ResourceWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... _OptionError
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

I've just investigated the details, so I'll leave it here, but here I will pay particular attention to the fact that ʻIOError has disappeared in Python 3.3, and that the inheritance structure around ʻOSError has changed drastically. It is also one of the major differences between the current 2 and 3 systems.

In Python 3.3, IOError disappeared and OSError was sorted out.

This is a change based on "PEP 3151 --Reworking the OS and IO exception hierarchy". You can see the details by reading the same PEP, but it seems that one major factor is that the error at the time of file operation was split between ʻIOError and ʻOSError.

>>> os.remove("fff")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 2] No such file or directory: 'fff'

>>> open("fff")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: 'fff'

There is no big difference between 3.3, 3.4 and 3.5 at this level. At the tree level, it looks like new exception classes have been added (BrokenBarrierError in 3.3 → 3.4, RecursionError and StopAsyncIteration in 3.4 → 3.5).

However, the change in the behavior of each function is not known in the tree structure, so if you are interested, please read the Changelog etc. carefully, including the difference in minor versions.

Execution result in 3.6.2

Supplement (2017-08-05): In the previous version, it was 3.6.0b2, but since the official version is released in 3.6, it will be 3.6.2. I will not revise the whole sentence for now.

Following the above flow, I also examined the 3.6.2 tree structure currently under development.

Python Version: 3.6.2

BaseException
... Exception
...... ArithmeticError
......... FloatingPointError
......... OverflowError
......... ZeroDivisionError
...... AssertionError
...... AttributeError
...... BufferError
...... EOFError
...... Error (<class 'locale.Error'>)
...... ImportError
......... ModuleNotFoundError
......... ZipImportError
...... LookupError
......... CodecRegistryError
......... IndexError
......... KeyError
...... MemoryError
...... NameError
......... UnboundLocalError
...... OSError
......... BlockingIOError
......... ChildProcessError
......... ConnectionError
............ BrokenPipeError
............ ConnectionAbortedError
............ ConnectionRefusedError
............ ConnectionResetError
......... FileExistsError
......... FileNotFoundError
......... InterruptedError
......... IsADirectoryError
......... ItimerError
......... NotADirectoryError
......... PermissionError
......... ProcessLookupError
......... TimeoutError
......... UnsupportedOperation
...... ReferenceError
...... RuntimeError
......... BrokenBarrierError
......... NotImplementedError
......... RecursionError
......... _DeadlockError
...... StopAsyncIteration
...... StopIteration
...... StopTokenizing
...... SubprocessError
......... CalledProcessError
......... TimeoutExpired
...... SyntaxError
......... IndentationError
............ TabError
...... SystemError
......... CodecRegistryError
...... TokenError
...... TypeError
...... ValueError
......... UnicodeError
............ UnicodeDecodeError
............ UnicodeEncodeError
............ UnicodeTranslateError
......... UnsupportedOperation
...... Verbose
...... Warning
......... BytesWarning
......... DeprecationWarning
......... FutureWarning
......... ImportWarning
......... PendingDeprecationWarning
......... ResourceWarning
......... RuntimeWarning
......... SyntaxWarning
......... UnicodeWarning
......... UserWarning
...... _OptionError
...... error (<class 'sre_constants.error'>)
... GeneratorExit
... KeyboardInterrupt
... SystemExit

$ diff /tmp/352.txt /tmp/362.txt
1c1
< Python Version: 3.5.2
---
> Python Version: 3.6.2
14a15
> ......... ModuleNotFoundError
65a67
> ...... Verbose

There are two new exceptions, but it doesn't seem to be an essential change either.

Click here for information about 3.6: https://docs.python.org/3.6/whatsnew/3.6.html (Because it is updated with the latest information of 3.6, it is not necessarily 3.6.2 above!)

Supplement 2 (2017-12-12): The difference between 3.6.2 and the latest 3.6.7 at the time of update is shown below.

$ diff 362.txt 367.txt
1c1
< Python Version: 3.6.2
---
> Python Version: 3.6.7

Execution result in 3.7.1

$ diff 367.txt 371.txt
1c1
< Python Version: 3.6.7
---
> Python Version: 3.7.1
80c80
< ...... error (<class 'sre_constants.error'>)
---
> ...... error (<class 're.error'>)

There is no big difference.

Reference: https://docs.python.org/3.7/whatsnew/3.7.html

If you want to create an exception system with your own module

If you use Python runtime exceptions directly in your module, you will not be able to distinguish your module from Python runtime errors (eg RuntimeError or ʻOSError`). Basically, it is a common practice to create a base exception in your own module and create an exception class tree starting from that. This is a concept often found in other languages that have introduced the idea of exceptions.

If you create your own tree-structured exception system, Python recommends inheriting ʻException` or its subclasses as the base exception class (http://docs.python.jp/3/library/exceptions). .html).

You can subclass the built-in exception class to define new exceptions. We recommend that new exceptions be derived from Exception or its subclasses instead of BaseException. For more information on defining exceptions, see the User-Defined Exceptions section of the Python tutorial.

Serpent: Comparison with Ruby

A word about another language, Ruby.

According to "Effective Ruby", StandardError is a good base class for creating custom exceptions. There is a class with the same name in the 2nd series, but there is no exception class in the Python 3 series in the first place, so consider ʻException` and its subclass.

Be aware of subtle differences in conventions, rules, and conventions when switching between multiple programming languages.

reference

update

2016-09-14

2016-10-24

2016-10-25

Looking back, it might have been kinder to use cls.__module__! ='builtins' than cls .__ name__. Lower () =='error' in the above validation code branch. not. We plan to do so the next time we make a major fix.

The title says "Built-in exception tree structure", but the displayed tree structure is actually a builtins (built-in) exception class and ʻio.UnsupportedOperation that does not require any import like ʻException. There are some exception classes such as that cannot be read without importing some module. So, for example, when you think "Hmm, what is Verbose "and look it up?

Python 3.6.0b2 (default, Oct 25 2016, 09:04:16)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> help(Verbose)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Verbose' is not defined
>>> import sre_parse
>>> help(sre_parse.Verbose)
(Help is displayed)

And cause confusion.

2017-08-05

We adopted 3.6.2 instead of 3.6.0b2. For reference, the result of 3.7-dev is posted.

2018-12-12

2020-03-29

The sequel is a separate article.

https://qiita.com/amedama/items/e1a605c004d1e2276778

At the same time, I wrote about the above " StopTokenizing SubprocessError disappears from the exception list". The point is, "There is no way for parents to know their children because they are not loaded at runtime startup", but it does not seem to disappear from the implementation.

Recommended Posts

Take a look at the Python built-in exception tree structure
Take a look at the built-in exception tree structure in Python 3.8.2
Take a closer look at the Kaggle / Titanic tutorial
I took a quick look at the fractions package that handles Python built-in fractions.
Let's take a look at the Scapy code. How are you processing the structure?
[Go] Take a look at io.Writer
Let's take a look at the feature map of YOLO v3
[Python] Take a screenshot
Take a peek at the processing of LightGBM Tuner
Tasks at the start of a new python project
Take a look at profiling and dumping with Dataflow
Don't take an instance of a Python exception class directly as an argument to the exception class!
Take a screenshot in Python
Try using the Python web framework Django (2) --Look at setting.py
Challenge image classification by TensorFlow2 + Keras 2 ~ Let's take a closer look at the input data ~
When learning a programming language, it's a good idea to first look at the family tree of the programming language.
I took a closer look at why Python self is needed
Let's take a look at the forest fire on the west coast of the United States with satellite images.
Let's take a look at the Scapy code. Overload of special methods __div__, __getitem__ and so on.
What is the XX file at the root of a popular Python project?
[Mac] Build a Python 3.x environment at the fastest speed using Docker
Get a datetime instance at any time of the day in Python
Make a decision tree from 0 with Python and understand it (4. Data structure)
Review the tree structure and challenge BFS
Output tree structure of files in Python
Write the test in a python docstring
Search the maze with the python A * algorithm
[Python] [Windows] Take a screen capture in Python
Run the Python interpreter in a script
[python] [meta] Is the type of python a type?
Call the python debugger at any time
Draw a tree in Python 3 using graphviz
Python exception handling a little more convenient
Python Basic Course (at the end of 15)
The story of blackjack A processing (python)
[Python] A progress bar on the terminal
Take the Python3 Engineer Certification Basic Exam
[Python] A program that rounds the score
Algorithm learned with Python 11th: Tree structure
Let's look at a differential equation that cannot be solved normally in Python
Turn multiple lists with a for statement at the same time in Python