[PYTHON] Manage your Amazon CloudWatch loggroup retention with AWS Lambda

Hello, this is risako of stremapack. After a long rainy season, summer has arrived: sunny: You need to be more careful not to get heat stroke with a mask while being careful of the corona: mask: I will spend this summer taking care of my physical condition.

What I want to do this time

The theme this time is "Let's manage Amazon CloudWatch log group retention period with AWS Lambda"! When using Amazon ECS or AWS Lambda (hereinafter Lambda), logs continue to accumulate in Amazon CloudWatch (hereinafter CoudWatch). What's more, the CloudWatch LogGroup retention period is set to "Never expire" by default, so logs will remain forever and the amount will increase. This time, in order to reduce the amount of money as much as possible, I would like to introduce a mechanism in Lambda that automatically changes the retention period of the newly created log group to "1 month"!

This item

• IAM role
• IAM policy
• Amazon CloudWatch LogGroup
• AWS Lambda
• Amazon EventBridge

let's try it

IAM Role settings

The Role name should be ** cloudwatchtest-lambda-role **.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

IAM Policy settings

The Policy name is ** cloudwatchtest-lambda-policy **. You have been granted permission to operate CloudWatch.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:PutRetentionPolicy",
                "logs:CreateLogGroup",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]

Lambda settings

The Lambda function was created below. ** Function name: change_Retention_for_cloudwatch-logs Runtime: Python 3.8 **

Function code I want to change the retention period of the newly created log group to ** 1 month (30 days) **, so set retentionInDays = 30. Here you specify the desired retention period. Example: 1.3.5.7.60

import boto3

logs = boto3.client('logs')

def lambda_handler(event, context):

    loggroupname = event['detail']['requestParameters']['logGroupName']

    try:
        response = logs.put_retention_policy(
            logGroupName = loggroupname,
            retentionInDays = 30
        )

    except Exception as e:
        print(e)

Event Bridge settings

This time, I want to move the Lambda created above with the trigger that CloudWatch Log Group is newly created, so let's have it detected by the rules of Event Bridge. Now let's create a rule.

The pattern definition is set below. ** Event pattern Event Match Pattern: Custom Pattern **

Put the code to be defined in the square on the right side.

For the target, specify the Lambda function you created earlier.

This completes the settings! Let's move

If the retention period of the newly created Cloudwatch LogGroup is one month, the operation check is OK! Create a new one from "Create Log Group" in the CloudWatch console.

** Immediately after creation **

** After a few minutes **

The retention period is not reflected immediately after creating a new one, but you can confirm that the retention period has been set correctly. This seems to prevent the log from accumulating and unknowingly increasing the price: ok_hand:

reference

・ What is Amazon Event Bridge? https://docs.aws.amazon.com/ja_jp/eventbridge/latest/userguide/what-is-amazon-eventbridge.html ・ What is Amazon CloudWatch Logs? https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html · How do I find out why I was charged for CloudWatch and then reduce future charges? https://aws.amazon.com/jp/premiumsupport/knowledge-center/cloudwatch-understand-and-reduce-charges/?nc1=h_ls