I had the opportunity to implement Logging, so I'll summarize what I did.
I used the print statement to check the process in a simple and debug-like manner, but I think that this implementation made it look cool after covering all of these things.
I don't think it's enough to put it on the Advent calendar, but ...
I think that there are projects that are actually operated with print statements, and I think that there are various logging methods. This time, I referred to this Qiita article and implemented it with cool ** getLogger **. It was very helpful. (Thanks to amedama)
log_accessor.py
from logging import getLogger, StreamHandler, FileHandler, basicConfig, Formatter, handlers
import datetime
■ Class definition
log_accessor.py
class App_log:
"""
When you create an instance of this class,
Named logging.A Logger instance is created and
You can output the log to the specified file.
"""
logger = None
def __init__(self, module_name, script_name, output_dir, logLevel='DEBUG' isTest=False):
"""
module_name:str type (specify the execution module name)
script_name:str type (specify the execution script name)
output_dir:str type (Log output directory)
logLevel:str type (specify Log level) (Default:'DEBUG'）
is_Test:bool type (when outputting Log to a terminal etc.'True'）（Default:false）
"""
log_dir = f'{output_dir}/logs'
self.logger = _get_module_logger(module_name, script_name, log_dir, logLevel, isTest)
def debug_log(self, msg):
"""
Output the values of variables in the script
"""
self.logger.debug(msg)
def info_log(self, msg):
"""
Output the start and end of the execution script
"""
self.logger.info(msg)
def warning_log(self, msg):
"""
Output when there is a possibility that the processing result is not the expected processing result although it is not an error
"""
self.logger.warning(msg)
def error_log(self, msg):
"""
Output when an error occurs
"""
self.logger.error(msg)
■ Function
log_accessor.py
def _get_module_logger(module_name, script_name, log_dir, logLevel, isTest):
"""
ret:logging.Logger
There are 3 types of handler
error_handler:Outputs a log of warning level or higher to a file common to all programs
info_handler:Output the log of the level specified by parameter to a file for each module
process_handler:Output log of info or higher level to a file common to all programs
StreamHandler:info_Output the contents of handler to standard output (isTest=True）
"""
day = datetime.date.today().strftime('%Y%m')
# Get Logger
logger = getLogger(f'{module_name}.{script_name}')
logger.setLevel('DEBUG')
# Create Handler -(error_handler)
error_handler = FileHandler(filename=f'{log_dir}/all_app_error_{day}.log', encoding='utf-8')
# Set LogLevel -(error_handler)
error_handler.setLevel('WARNING')
# Create Formatter -(error_handler)
fmt = Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# Set Formatter -(error_handler)
error_handler.setFormatter(fmt)
# Create Handler -(info_handler)
# info_handler = FileHandler(filename=f'{log_dir}/{module_name}_{day}.log')
info_handler = handlers.RotatingFileHandler(
filename=f'{log_dir}/{module_name}.log',
maxBytes=1048576, # 1MB
backupCount=5,
encoding='utf-8'
)
# Set LogLevel -(info_handler)
info_handler.setLevel(logLevel)
# Set Formatter -(info_handler)
info_handler.setFormatter(fmt)
# Create Handler -(process_handler)
process_handler = FileHandler(filename=f'{log_dir}/all_app_process_{day}.log', encoding='utf-8')
# Set LogLevel -(process_handler)
process_handler.setLevel('INFO')
# Create Formatter -(process_handler)
fmt = Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# Set Formatter -(process_handler)
process_handler.setFormatter(fmt)
# Logger - Set Handler
logger.addHandler(error_handler)
logger.addHandler(info_handler)
logger.addHandler(process_handler)
# Create Handler -(StreamHandler - basicConfig)
# Logger - Set Handler -(StreamHandler - basicConfig)
if isTest:
stream_handler = StreamHandler()
stream_handler.setFormatter(fmt)
logger.addHandler(stream_handler)
return logger
■ Import
xxxxx.py
# import logging
from util.log_accessor import App_log
■ Instance generation
xxxxx.py
# Set Logging
module_name = os.path.split(package_dir)[-1]
script_name = os.path.splitext(os.path.basename(__file__))[0]
app_log_ut = App_log(module_name=module_name,
script_name=script_name,
output_dir=project_dir, logLevel='DEBUG', isTest=True)
■ Built-in Logging
xxxxx.py
self.logging.info_log(f'Start xxxxx')
~~~~~~~~
self.logging.info_log(f'End xxxxx')
■ info_handler
TEST.log
2020-12-17 12:54:02,361 - TEST.xxxxx - INFO - Start xxxxx.edit
2020-12-17 12:54:02,400 - TEST.xxxxx - DEBUG - Start method1
2020-12-17 12:54:02,405 - TEST.xxxxx - DEBUG - df.shape (288, 19)
2020-12-17 12:54:02,405 - TEST.xxxxx - DEBUG - END method1
2020-12-17 12:54:02,406 - TEST.xxxxx - DEBUG - Start method2
2020-12-17 12:54:02,409 - TEST.xxxxx - DEBUG - min:1 max:120
2020-12-17 12:54:02,409 - TEST.xxxxx - DEBUG - df2.shape (720, 8)
2020-12-17 12:54:02,409 - TEST.xxxxx - DEBUG - End method2
2020-12-17 12:54:02,409 - TEST.xxxxx - INFO - END xxxxx.edit
■ process_handler
all_app_process_202012.log
2020-12-17 12:54:02,361 - TEST.xxxxx - INFO - Start xxxxx.edit
2020-12-17 12:54:02,409 - TEST.xxxxx - INFO - END xxxxx.edit
2020-12-17 23:09:13,953 - TEST.yyyyy - INFO - Start yyyyy.edit
2020-12-17 23:27:38,536 - TEST.yyyyy - INFO - END yyyyy.edit
I'm still studying Python, so I think there are many things to do. Please point out ...
Recommended Posts