[PYTHON] Wait for Aurora Serverless to wake up

For example, if you regularly pull data from somewhere with AWS Lambda + Amazon CloudWatch Events and put it in Amazon RDS appropriately, it costs at least several thousand yen / month for a normal Provisioned instance. However, if you use Aurora Serverless, it will sleep without permission when you are not using it, so it may be cheap. [^ pricing] [^ serverless]

[^ pricing]: I tried to calculate the usage cost of Amazon Aurora Serverless | Developers.IO [^ serverless]: What to watch out for when installing Aurora Serverless | Developers.IO

However, there is one problem with this idea: ** Aurora Serverless is slow to happen **.

Aurora Serverless, which was sleeping when trying to connect from a DB client, starts booting, but sometimes the client times out before it can accept the connection.

Therefore, when using it, you should send a request to start Aurora Serverless (increase Capacity) in advance, and connect from the DB client when it starts (Capacity becomes 0 or more).

Implementation of Lambda Function

Try implementing a Lambda Function that waits for Aurora Serveless to occur and then does nothing.

The runtime used Python 3.8.

import asyncio
import boto3


def is_aurora_serverless_up(client, identifier: str) -> bool:
    """Returns whether Aurora Serverless is running"""
    response = client.describe_db_clusters(DBClusterIdentifier=identifier)
    assert response['ResponseMetadata']['HTTPStatusCode'] == 200
    assert len(response['DBClusters']) > 0
    assert response['DBClusters'][0]['EngineMode'] == 'serverless'
    return response['DBClusters'][0]['Capacity'] > 0


async def wake_aurora_serverless_up(client, identifier: str, capacity: int = 2):
    """Start Aurora Serverless"""
    if is_aurora_serverless_up(client, identifier):
        return
    response = client.modify_current_db_cluster_capacity(DBClusterIdentifier=identifier, Capacity=capacity)
    assert response['ResponseMetadata']['HTTPStatusCode'] == 200
    for i in range(10):
        await asyncio.sleep(i ** 2)
        if is_aurora_serverless_up(client, identifier):
            return
    raise TimeoutError()


async def main():
    client = boto3.client('rds')
    await wake_aurora_serverless_up(client, 'mycluster')


def lambda_handler(event, context):
    asyncio.get_event_loop().run_until_complete(main())

What I need to do later.

--Because you need the privileges of rds: DescribeDBClusters and rds: ModifyCurrentDBClusterCapacity to execute, create an IAM Role like that and assign it to the Lambda Function. --If the default value of the Lambda Function timeout setting is 3 seconds, it will never end, so set it to 3 minutes.

This time it is implemented in asyncio, but it doesn't make much sense because it is only used in asyncio.sleep. You can use time.sleep as usual. Implementing with asyncio has the advantage that asyncpg can be selected as the PostgreSQL client.

Try to run

If you try it, you will find the following.

--It takes about 15 seconds for Aurora Serverless to start --Even if you start it (capacity> 0), you cannot connect from the DB client immediately, and wait another 10 seconds from there.

It hasn't changed much since I waited after all, but this is fine because the DB client no longer times out.

Finally

After doing so far, I realized that "Isn't this solved by just setting the connection timeout of the DB client longer?"

import psycopg2

with psycopg2.connect('postgresql://...', connect_timeout=120) as conn:
    ...

(In case of psycopg2) Now it works fine.

Ah, yes ...