[PYTHON] How to deal with the problem that Japanese characters are garbled when outputting logs using JSON log formatter

I used JSON log formatter because I want to get the log in JSON, but for some reason Japanese (double-byte characters) is garbled.

environment

Windows10 Home 10.0.18362 Build 18362 Python 3.8.3 JSON-log-formatter 0.3.0

Failure example

Execute the following program.

json_log_test.py

import logging
import json_log_formatter

formatter = json_log_formatter.JSONFormatter()

json_handler = logging.FileHandler(filename='my-log.json', encoding='utf-8')#UTF-Output at 8
json_handler.setFormatter(formatter)

logger = logging.getLogger('my_json')
logger.addHandler(json_handler)
logger.setLevel(logging.INFO)

logger.info('This string is garbled', extra={'extra': 'This string is also garbled'})

The output log file "my-log.json" is as follows. (Shaped for easy viewing)

my-log.json

{
    "extra": "\u3053\u306e\u6587\u5b57\u5217\u3082\u6587\u5b57\u5316\u3051\u3059\u308b",
    "message": "\u3053\u306e\u6587\u5b57\u5217\u304c\u6587\u5b57\u5316\u3051\u3059\u308b",
    "time": "2020-10-21T02:29:53.275011"
}

The extra and message are garbled.

solution

Rewrite the "to_json" method of "\ _ \ _ init__.py" in the "json_log_formatter" directory as follows.

__init__.py

    def to_json(self, record):
        """Converts record dict to a JSON string.

        It makes best effort to serialize a record (represents an object as a string)
        instead of raising TypeError if json library supports default argument.
        Note, ujson doesn't support it.

        Override this method to change the way dict is converted to JSON.

        """
        try:
            #return self.json_lib.dumps(record, default=_json_serializable)#Before rewriting
            return self.json_lib.dumps(record, ensure_ascii=False, default=_json_serializable)#After rewriting
        # ujson doesn't support default argument and raises TypeError.
        except TypeError:
            #return self.json_lib.dumps(record)#Before rewriting
            return self.json_lib.dumps(record, ensure_ascii=False)#After rewriting

Add ensure_ascii = False to the argument of the dumps method that converts the dictionary to JSON.

When I execute "json_log_test.py" again and check the output file, the characters are not garbled this time.

my-log.json

{
    "extra": "This string is also garbled",
    "message": "This string is garbled",
    "time": "2020-10-21T02:31:16.704940"
}

Reference site

Memo when Japanese characters are garbled with json.dumps () of Python --Tile memorandum -pygo https://cortyuming.hateblo.jp/entry/20140920/p2 JSON-log-formatter · PyPI https://pypi.org/project/JSON-log-formatter/