Output log in JSON format with Python standard logging

How to use logging

First of all, the basics. Write the log on the library side and Decide how to handle the log on the client side.

Library side

First, add the following 4 lines to the beginning of each file

from logging import getLogger, DEBUG, NullHandler
logger = getLogger(__name__)
logger.addHandler(NullHandler())
logger.setLevel(DEBUG)

This will output to logger with .debug, .info, etc.

logger.info("Enter special block")

Since NullHandler is registered, this log will be discarded if the client does not capture it.

Client side

The name of the logger on the library side is registered with __name__. For example, it is registered with a name such as mod.submod1, mod.submod2. Since it is layered with ., you can receive the logs of both submodules by referencing it with mod. If no name is given, all logs will be collected.

--Output all logs to standard error output (sys.stderr)

import logging
logger = logging.getLogger()
logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)

--Output the log of INFO or higher of mod module to a file (mod.log)

import logging
logger = logging.getLogger("mod")
logger.addHandler(logging.FileHandler("mod.log", mode="w"))
logger.setLevel(logging.INFO)

Of course, it is possible to prepare a function on the library side that makes it easy for the client to execute these codes.

Output in JSON

Considering the subsequent analysis, I want a structured log. Actually, you can give a dictionary object such as logger.info:

logger.info({
  "key": value,
  "homhom": 0,
})

If you do not set it in particular, the result of printing this normally will be output, so it looks the same as JSON, but It's not official JSON, so it's hard to parse it. That's where python jsonlogger comes in. This will be reformatted as JSON when output on the client side.

To output to the above file:

import logging
from pythonjsonlogger import jsonlogger
h = logging.FileHandler("mod.log", mode="w")
h.setFormatter(jsonlogger.JsonFormatter())
logging.getLogger("mod").addHandler(h)

This will output a valid JSON log for each row. So log

with open("mod.log") as f:
    df = pd.DataFrame([json.loads(l) for l in f])

You can easily pour it into pandas like this.