[PYTHON] Create Cognito user list in S3 with SQS Deploy queue function and API to Lambda with SAM
The title is inconvenient in Japanese, but I will set it in order.
Build a function on Lambda that automatically creates a CSV file of the user list managed by AWS Cognito on S3. Set SQS as the trigger to activate that Lambda, and implement the API to add the message queue to SQS in another Lambda so that HTTP requests can be received via API Gateway.
- Cognito: User management service. You can easily implement multi-factor authentication (MFA) and SNS linkage.
- S3: Cloud storage. You can also version and create download URLs that can be used temporarily.
- SQS: Manage queues on AWS. It is used for asynchronous processing.
- Lambda: Serverless function execution environment
- API Gateway: Provides an API interface for Lambda and EC2.
The configuration diagram is as follows.
AWS-CLI installation and IAM configuration
$ brew install awscli
$ aws configure
AWS Access Key ID [None]: ************
AWS Secret Access Key [None]: ************************
Default region name [None]: ap-northeast-1
Default output format [None]: json
SAM installation
- Docker is also required, so please install and start it.
$ brew tap aws/tap
$ brew install aws-sam-cli
$ sam --version
SAM CLI, version 1.0.0
Initialize SAM project
This time, we will implement it in Python 3.8 based on the Hello World template.
% sam init
Which template source would you like to use?
1 - AWS Quick Start Templates
2 - Custom Template Location
Choice: 1
Which runtime would you like to use?
1 - nodejs12.x
2 - python3.8
3 - ruby2.7
・ ・ ・
Runtime: 2
Project name [sam-app]: sample-app
Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git
AWS quick start application templates:
1 - Hello World Example
2 - EventBridge Hello World
・ ・ ・
Template selection: 1
-----------------------
Generating application:
-----------------------
Name: sample-app
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .
Next steps can be found in the README file at ./sample-app/README.md
Creating a Cognito user pool
This time, I created it with the default settings for the time being, and manually added one suitable user. Use the created user pool ID when creating the file.
Implementation
I implemented the main files as follows.
app.py
import json
import requests
import boto3
from datetime import datetime
import pandas as pd
def lambda_handler(event, context):
try:
sqs_client = boto3.client("sqs")
#Once you deploy SQS, a queue will be automatically created in SQS, so set it.
queue_url = "https://sqs.ap-northeast-1.amazonaws.com/********/**********"
print(queue_url)
now = datetime.now()
date_str = now.strftime('%Y/%m/%d-%H:%M:%S')
sqs_response = sqs_client.send_message(
QueueUrl=queue_url,
MessageBody=json.dumps(date_str)
)
except requests.RequestException as e:
# Send some context about this error to Lambda Logs
print(e)
raise e
return {
"statusCode": 200,
"body": json.dumps({
"message": "hello world"
}),
}
def csv_create_handler(event, context):
for record in event['Records']:
payload=record["body"]
date_str = str(payload)
s3 = boto3.resource('s3')
bucket = s3.Bucket('arkth-user-list-files')
cognito_client = boto3.client('cognito-idp')
response = cognito_client.list_users(
UserPoolId = 'ap-northeast-***********',
AttributesToGet = ['email','sub']
)
data = []
for user in response["Users"]:
data.append([user['Username'], user['Enabled']])
Coulum = ['Username', 'Enabled']
df = pd.DataFrame(data, columns=Coulum)
df_csv = df.to_csv(index=None)
objkey = 'user-list.csv'
print(objkey)
putobj = bucket.Object(objkey)
putobj.put(Body=df_csv)
def support_datetime_default(obj):
if isinstance(obj, datetime):
return obj.isoformat()
raise TypeError(repr(obj) + " is not JSON serializable")
template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sample-app
Sample SAM Template for sample-app
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 3
Runtime: python3.8
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello_world/
Handler: app.lambda_handler
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
UsersCsv:
Type: AWS::SQS::Queue
Properties:
VisibilityTimeout: 60
CsvCreateFunction:
Properties:
CodeUri: hello_world/
Handler: app.csv_create_handler
Events:
MySQSEvent:
Type: SQS
Properties:
Queue: !GetAtt UsersCsv.Arn
BatchSize: 10
Type: AWS::Serverless::Function
requirement.txt
requests
boto3
pandas
SAM build
$ sam build
Building function 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Building function 'CsvCreateFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
Build Succeeded
Built Artifacts : .aws-sam/build
Built Template : .aws-sam/build/template.yaml
Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided
Launch SAM locally
$ sam local start-api
Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-09-05 22:11:22 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
An error will occur because the SQS endpoint is dummy, but access http://127.0.0.1:3000/hello from a browser and check if there is a Syntax Error etc.
Deploy SAM
The first time you deploy with --guided, samconfig.toml is created interactively.
$ sam deploy --guided
Configuring SAM deploy
======================
Looking for samconfig.toml : Not found
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: sample-app
AWS Region [us-east-1]: ap-northeast-1
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]: y
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: Y
HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
Save arguments to samconfig.toml [Y/n]: Y
・ ・ ・ ・
・ ・ ・ ・
・ ・ ・ ・
It will take some time, but IAM and Api Gateway will be set up and deployed appropriately.
Environmental setting
-
Rewrite the SQS endpoint in app.py with the created SQS endpoint.
-
Grant the IAM Role assigned to the file creation Lambda function permission to access SQS and S3.
-
Set the created SQS in the Destination of the Lambda function of the API as shown below. It may be possible to set this with SAM, but this time I did it manually.
Then build & deploy again.
$ sam build
$ sam deploy
Operation check
Access the API Gateway endpoint set in the Lambda function of Hello World with a browser.
After that, when you access the S3 Bucket, the following CSV file is created.
Recommended Posts