[PYTHON] I can't send emails from Lambda in a VPC with Boto3 using a VPC endpoint for SES

at first

This article describes the details of the verification of the title.

It was an opportunity to write, but recently I have been playing with VPC for the first time in a while. I didn't have any memories of going smoothly with the configuration using VPC, and I was really into the swamp.

I've summarized the points, so I hope it helps someone.

If you want to know only the conclusion, please skip to the end

Background to the verification

** Send an email from Lambda in VPC ** I just wanted to do this

The architecture is as follows RDS → Lambda (in VPC) → SES → User

It's an architecture you don't often see Since the release of RDS Proxy, the combination of Lambda and RDS has been used more often. DynamoDB alone has some pains, so I think this architecture will increase in the future.

In implementing it, I was investigating how to get out of the VPC Then, in May 2020, VPC Endpoint for SES Has been released

I tried this not to finish the implementation easily, but Lambda timed out This is the beginning of a long verification ...

Verification content

It was displayed on CloudWatch as follows

Task timed out after 30.03 seconds

I can't seem to get out of the VPC

1. Doubt SES settings

First of all, it is confirmed by sending a test whether SES verification is performed correctly. I was in a sandbox environment, so I thought it might have some impact.

You can easily send a test from the console Procedure here

The email was sent from the sender to the destination as expected

We have confirmed that the transmission can be performed correctly from the SES console, so look for another cause.

2. Around authority

I thought that it was a problem of role or security group setting, so I set up Gabagaba first Check if you can get out of the VPC for the time being

1. Allow all communication inbound and outbound for security groups
2. Add the following policy to the role

"ses:SendEmail","ses:SendRawEmail"Allowed from all resources
AWSLambdaVPCAccessExecutionRole
VPCFullAccess
SESFullAccess

Still the timeout did not improve

3. Logic problem

Next, I reviewed the logic to send to SES. I was using Boto3 with python3.8 The source is based on here

In order to isolate whether there is a problem with the logic or a problem with the setting, I first created Lambda outside the VPC and sent it with the same source code

Then I was able to send an email from Lambda outside the VPC Previous verification has shown that the source is fine and probably a VPC related issue.

4. Specify the endpoint?

When using SQS from Lambda in VPC, it is necessary to specify the endpoint There was no description in the procedure, but I suspected that SES would also need to specify the end point in the same way.

In boto3, you can specify the endpoint with an argument when creating a client Official

I tried adding the URL of the VPC endpoint

import boto3
ses_client = boto3.client('ses', region_name='ap-northeast-1', endpoint_url='com.amazonaws.region.email-smtp')

When I tried sending here, the following error occurred

[ERROR] ValueError: Invalid endpoint: com.amazonaws.ap-northeast-1.email-smtp

Seems invalid as an endpoint

5. Lambda (inside VPC) → Lambda (outside VPC) → SES

It's getting closer to shit I thought that VPC endpoints for SES seemed to be unusable, so I decided to take a different approach.

[Lambda supports Private Link] in October 2020 (https://aws.amazon.com/jp/about-aws/whats-new/2020/10/aws-lambda-now-supports-aws-privatelink/) was doing

When I tried this, I was able to send an email without any problems But not smart ...

6. SMTP interface ...?

Foreigners can do Send Email Using Amazon SES SMTP Interface Yo! I found that he was telling me

You will need a username and password for each region.

Hmm subtle ... I want to send it with boto3 ...

7. Contact support

I decided that I couldn't do anything with my own knowledge, so I made an inquiry. Below is an excerpt of the answer

VPC endpoints are supported by the "1. Use SES SMTP interface" method. On the other hand, the method of using the send_email method of Boto3 that the customer is trying to implement is "2. Use SES API (AWS CLI, AWS SDK)", and it does not support VPC endpoints. In case of "2. Use SES API (AWS CLI, AWS SDK)", please use one of the following. -Start Lambda outside the VPC -Lambda is started in the VPC and NAT Gateway is installed so that Lambda can connect to the Internet.

seriously!!! I didn't say "Boto3" anywhere !!! I felt that, but I'm glad I was able to resolve it safely. Thanks to the support

After that, I tried the method with NAT Gateway and confirmed that the email arrived.

Conclusion

• If you want to send mail from Lambda in VPC using Boto3, you need to connect to the Internet using NAT gateway.
• Boto3 cannot be used (timeout) when using VPC endpoint for SES
• You must use the Amazon SES SMTP Interface (https://docs.aws.amazon.com/ja_jp/ses/latest/DeveloperGuide/send-email-smtp.html) when using a VPC endpoint for SES. There is

combination

Strong to ask support