[PYTHON] Notes on accessing SQS from AWS VPC Lambda via endpoint

Conclusion

The endpoint_url option is required when deploying Lambda in a VPC on AWS and using boto3 SQS via an endpoint.

import boto3
sqs = boto3.client("sqs", endpoint_url="https://sqs.ap-northeast-1.amazonaws.com")
sqs.send_message(...)

Without this option, you will not be able to connect and will time out.

Details

Make a note of the mess when deploying Lamba to a VPC on AWS and using SQS with a private connection. For details, please refer to the following site. When I tried it, the symptom only occurred when using boto3 on Lambda.

Under my conditions, Lambda's boto3 was version 1.14.48. Also, the version I tried with EC2 was 1.16.8, but the same phenomenon occurred here as well.

1. Precautions when using SQS VPC Endpoint with AWS CLI etc .-- https://blog.serverworks.co.jp/tech/2019/03/18/sqsvpcendpoint/
2. Unable Connect do SQS if using a VPC -- https://github.com/boto/boto3/issues/1900

According to the article in 1., the AWS CLI (including boto3 to see this phenomenon) ** tries to connect to the legacy endpoint even if QueueUrl is specified when connecting to SQS **. It seems to be a problem. When I create an SQS endpoint in a VPC, sqs.ap-northeast-1.amazonaws.com (current endpoint) is assigned a private IP, but ap-northeast-1.queue.amazonaws.com (legacy end) Point) remains the public IP. In that state, when I try to use SQS with boto3, it seems that the cause is that I tried to use a legacy endpoint when connecting, but I could not connect and timed out.

The solution using Session is shown in 2. GitHub, but if you specify the current endpoint in the ʻendpoint_urloption as above when executingboto3.client ()`, the problem is solved. For reference, the solution code by Session in the article on GitHub is as follows.

import boto3
session = boto3.Session()
sqs_client = session.client(
    service_name='sqs',
    endpoint_url='https://sqs.ap-northeast-1.amazonaws.com',
)
sqs_client.send_message(...)

... I got stuck for about 2 hours.