[PYTHON] Handle structured logs with GCP Cloud Logging

What is a structured log?

A log output in a data format such as JSON as shown below. GCP automatically parses JSON-formatted logs for use in search queries, which is convenient. (Confirmed in Cloud Run) Line breaks are also escaped as JSON, which makes it easier to handle multi-line logs as a side effect.

{"level": "INFO", "message": "hello\nworld", "timestamp": "2020-07-01 00:00:00"}

Sample to output log in JSON with Python

In Python, you can output the log in JSON format by setting your own formatter in logger.

$ python -V
Python 3.8.0

logger.py

import sys
import logging
import json
import traceback


class JsonFormatter(logging.Formatter):
  def format(self, log):
    return json.dumps({
      "level": log.levelname,
      "message": log.msg,
      "timestamp": self.formatTime(log, self.datefmt),
      "traceback": traceback.format_exc() if log.exc_info else []
    })
    

formatter = JsonFormatter(datefmt="%Y-%m-%d %H:%M:%S")

stream = logging.StreamHandler(stream=sys.stdout)
stream.setFormatter(formatter)

logger = logging.getLogger('your-logger-name')
logger.setLevel(logging.INFO)
logger.addHandler(stream)

Load and use the set logger.

sample.py

from logger import logger


logger.info("hello\nworld")
logger.info({"foo": "foo", "boo": "boo"})

The following log is output.

{"level": "INFO", "message": "hello\nworld", "timestamp": "2020-07-01 00:00:00", "traceback": []}
{"level": "INFO", "message": {"foo": "foo", "boo": "boo"}, "timestamp": "2020-07-01 00:00:00", "traceback": []}

Cloud Logging

You can search from GCP Cloud Logging-> Log Viewer screen. The data output as a log as shown below is stored in jsonPayload, and the key can be specified on the query.

resource.type="cloud_run_revision"
jsonPayload.level="INFO"
jsonPayload.message.foo="foo"

The log will appear as a search result as shown below

{
  "insertId": "xxxxxxxxxxxxxxxxxxxxxxxx",
  "jsonPayload": {
    "message": {"foo": "foo", "boo": "boo"},
    "traceback": [],
    "timestamp": "2020-07-07 15:33:28",
    "level": "INFO"
  },
  ...
}