[Python] 3 types of libraries that improve log output a little [logzero, loguru, pyrogrus]

This is an introduction of the library for python log output. Introducing three types of sample code and how to use them, which are libraries logzero, loguru, and pyrogrus that can manage log output more easily and conveniently than standard logging.

Sample code

We have prepared sample code for each library. Please refer to it together with the usage described later.

logzero

Documentation: https://github.com/metachris/logzero, https://logzero.readthedocs.io/en/latest/

A library that can be used instead of logging. An easy-to-use version of logging.

image.png

How to use logzero

Use with default settings

from logzero import logger

logger.trace("sample trace level log message")
logger.debug("sample debug level log message")
logger.info("sample info level log message")
logger.warning("sample warn level log message")
logger.error("sample error level log message")
logger.critical("sample critical level log message")

Console output log level setting

logzero.loglevel(logging.INFO)

Log file output settings

#Output settings
logzero.logfile(
    '/var/log/logzero.log',  #Log file path
    loglevel=logging.ERROR,   #Log level output to file
    maxBytes=1e6,            #Maximum file size
    backupCount=3            #Number of generations of old files to keep
)

#Disable output settings
logzero.logfile(None)

Log format

The part from % (color) s to% (end_color) s is colored according to the log level.

log_format = '%(color)s[%(levelname)1.1s %(asctime)s %(name)s %(module)s %(funcName)s:%(lineno)d]%(end_color)s %(message)s'
formatter = logzero.LogFormatter(fmt=log_format)
logzero.formatter(formatter)

Create logger instance

#Create logger instance
log_format = '%(color)s[%(levelname)1.1s %(asctime)s %(name)s %(module)s %(funcName)s:%(lineno)d]%(end_color)s %(message)s'
formatter = logzero.LogFormatter(fmt=log_format)
custom_logger = logzero.setup_logger(
    name=__name__,
    logfile="/var/log/logzero.log",
    formatter=formatter,
    maxBytes=1000000,
    backupCount=3,
    level=logging.INFO,
    fileLoglevel=logging.ERROR,
    disableStderrLogger=False,
)

#Log output
custom_logger.debug("sample class custom logger debug level log message")
custom_logger.info("sample class custom logger info level log message")
custom_logger.warning("sample class custom logger warn level log message")
custom_logger.error("sample class custom logger error level log message")

loguru

Documentation: https://github.com/Delgan/loguru, https://loguru.readthedocs.io/en/stable/index.html

It's different from logging, but it's easy and intuitive to use.

image.png

How to use loguru

Add log output destination

By default, it is output only to stderr. If you want to output to a file, do logger.add as follows.

logger.add("/var/log/loguru_sample2.log")

Settings such as log rotation

logger.add("file_1.log", rotation="500 MB")    #Rotation by file size
logger.add("file_2.log", rotation="12:00")     #Rotation at specified time every day
logger.add("file_3.log", rotation="1 week")    #Weekly rotation
logger.add("file_A.log", retention=3)  #Hold 3 generations
logger.add("file_B.log", retention="10 days")  #Hold for 10 days
logger.add("file_Y.log", compression="zip")    #Compress with zip

For other settings, see https://loguru.readthedocs.io/en/stable/api/logger.html#file. This logger.add has some configurable parameters such as log level. Please refer to https://loguru.readthedocs.io/en/stable/api/logger.html?highlight=logger.add#loguru._logger.Logger.add for details. * It is recommended to set False for diagnose and backtrace.

--Example of logger.add

logger.add(
    "/var/log/sample.log",
    rotation="12:00",
    format=log_format,
    diagnose=False,
    backtrace=False,
    level=LOG_LEVEL
)

Formatting

You can color the log by writing it as <green> ～～～ </ green>. You can change the color with <level> depending on the severity. Log messages are associated with record. record is a dictionary type and by embedding key in format like {message} It can be embedded in the log. What kind of key you have https://loguru.readthedocs.io/en/stable/api/logger.html See "The record dict" in.

log_format = "<green>{time:YYYY-MM-DDTHH:mm:ss}</green> <level>{level} {message}</level>"

#If you want to set the console output, you need to turn off the default setting first.
#Previously logger.Note that the added one will also disappear.
logger.remove()
logger.add(sys.stderr, format=log_format)

Add a value not in record to the format

You can define it as an extra dict in the format, like {extra [extra_value]}, and pass it as a bind or message argument.

logger.remove()
log_format = "<green>{time}</green>: <level>{level} {message}</level>: {extra[extra_value]}"
logger.add(
    sys.stdout,
    format=log_format,
    serialize=True
)

logger.bind(extra_value="some_extra_value").info("serialize message 01")
logger.info("serialize message 02", extra_value="some_extra_value")

Output in JSON

You can make the output json as serialize = True.

logger.remove()
log_format = "<green>{time}</green>: <level>{level} {message}</level>"
logger.add(
    sys.stdout,
    format=log_format,
    serialize=True
)

Change the log level of Exception

try:
    raise ValueError("error!")
except ValueError as e:
    logger.opt(exception=True).critical("Change the log level of Exception")

Color a part of the log

logger.remove()
logger.add(sys.stderr, format="{time} {level} {message}", level="INFO")
logger.opt(colors=True).info("Color the log<blue>colors</blue>")

Add the information stored in record to log

Log messages are associated with record. record is a dictionary type and what kind of key it has https://loguru.readthedocs.io/en/stable/api/logger.html See "The record dict" in.

logger.opt(record=True).info("Add the information stored in record to log(eg. {record[thread]})")

Log output ignoring format

logger.opt(raw=True).info("Log output ignoring format\n")

Display parent (caller) information in the log

The record information used in the log message is that of the parent (caller). In the case of the following example, the log is output in child_func (), but {function" like parent_func which displays parent information in the log 2020-05-30T18: 54: 32.505956 + 0000 INFO } Enter parent_func.

logger.remove()
logger.add(sys.stderr, format="{time} {level} {message} {function}", level="INFO")
def child_func():
    logger.opt(depth=1).info("Display parent information in the log")

def parent_func():
    child_func()
parent_func()

Use variables only for message

Normally, when you use a variable in a log message, its value is stored in record ['extra']. If capture = False is set, it will not be stored.

logger.remove()
logger.add(sys.stderr, format="{time} {level} {message} {function}", level="INFO", serialize=True)
logger.opt(capture=False).info("{dest}Use variables only for message", dest="extra")

Output example

# capture=If False
{
    ...
    "record": {
        ...
        "extra": {},
        ...
        "message": "Keyword arguments not added to extra dict",
        ...
    }
}

# capture=If True
{
    ...
    "record": {
        ...
        "extra": {"dest": "extra"},
        ...
        "message": "Keyword arguments not added to extra dict",
        ...
    }
}

lazy: Control heavy function execution

Only execute the function when the loglebel is appropriate.

def test_lazy():
    print('exec test_razy')
    return 'exec test_razy'

logger.remove()
logger.add(sys.stderr, format="{time} {level} {message}", level="INFO")

#The log level to be displayed is set to INFO.
#At this time lazy=False(Default), Will not show the following DEBUG level logs, but test_lazy()The function will be executed.
logger.opt(lazy=False).debug("DEBUG LEVEL LOG: {x}", x=test_lazy())

# lazy=Set to True and use lambda as follows.
#In this case, no log is output and test_lazy()Is not executed either.
logger.opt(lazy=True).debug("DEBUG LEVEL LOG: {x}", x=lambda: test_lazy())

#If you change to DEBUG level and execute as follows
#The execution result of lambda is stored in x
logger.remove()
logger.add(sys.stderr, format="{time} {level} {message}", level="DEBUG")
logger.opt(lazy=True).debug("DEBUG LEVEL LOG: {x}", x=lambda: test_lazy())

pylogrus

Documentation: https://github.com/vmig/pylogrus

A library that extends logging so that it can be used like logrus of golang. The basic usage is the same as lgging.

image.png

How to use pylogrus

Basic output and output to file

Same as logging except that logging.setLoggerClass (PyLogrus) is executed first and the TextFormatter of pylogrus is used.

import logging
from pylogrus import PyLogrus, TextFormatter

logging.setLoggerClass(PyLogrus)
logger = logging.getLogger(__name__)  # type: PyLogrus
logger.setLevel(logging.DEBUG)

formatter = TextFormatter(datefmt='Z', colorize=True)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)

ch = logging.FileHandler('/var/log/py_logrus_sample2.log')
ch.setLevel(logging.ERROR)
ch.setFormatter(formatter)
logger.addHandler(ch)

logger.debug("DEBUG MESSAGE")
logger.info("INFO MESSAGE")
logger.warning("WARNING MESSAGE")
logger.error("ERROR MESSAGE")

Prefix the message

# [2020-05-31T13:12:14.428Z]    DEBUG [API] DEBUG MESSAGE
#Is output as
logger = logger.withPrefix("[API]")
logger.debug("DEBUG MESSAGE")

Add a field to the message

# [2020-05-31T13:12:14.428Z]     INFO INFO MESSAGE; error_code=404
#Is output as
logger.withFields({'error_code': 404}).info("INFO MESSAGE")

Output in JSON format

Use JsonFormatter with the fields to enable.

enabled_fields = [
    ('name', 'logger_name'),
    ('asctime', 'service_timestamp'),
    ('levelname', 'level'),
    ('threadName', 'thread_name'),
    'message',
    ('exception', 'exception_class'),
    ('stacktrace', 'stack_trace'),
    'module',
    ('funcName', 'function')
]
formatter = JsonFormatter(datefmt='Z', enabled_fields=enabled_fields, indent=2, sort_keys=True)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)

logger.debug("DEBUG MESSAGE")

Summary

We introduced three types: logzero, loguru, and pyrogrus. There is standard logging in python, but I think you should use these when you want to easily output a nice log. Personally, I found loguru easy to handle, but if you are used to logging, I recommend using logzero and pyrogrus.

Recommended Posts

[Python] 3 types of libraries that improve log output a little [logzero, loguru, pyrogrus]
[python] Create a list of various character types
Output in the form of a python array
Simple comparison of Python libraries that operate Excel
[Python] A program that counts the number of valleys
Python that merges a lot of excel into one excel
A quick comparison of Python and node.js test libraries
Super simple: A collection of shells that output dates
[Python] A program that compares the positions of kangaroos.
[python] [Gracenote Web API] A little customization of pygn
Basics of python: Output
I just changed the sample source of Python a little.
[Python] A program that finds the most common bird types
A set of script files that do wordcloud in Python3
A Python script that compares the contents of two directories
[Django] A brief summary of the log output function so that even beginners can understand it.