[Lambda] I tried to incorporate an external module of python via S3

Overview

When I made a disny monitoring bot using lambda, I wanted to execute it regularly using lambda, so I decided to use aws lambda. At this time, it was necessary to incorporate line-sdk-bot into lambda, so I will describe the method.

Local settings

First, create a docker-lambda directory.

mkdir docker-lambda
cd docker-lambda

The structure of the directory is as follows.

docker-lambda ├── Dockerfile ├── deploy.sh └── requirements.txt

Each file is configured as follows.

The Dockerfile is below.

Dockerfile


FROM lambci/lambda:build-python3.7
ENV AWS_DEFAULT_REGION ap-northeast-1

ADD . .

CMD pip install -r requirements.txt -t python/lib/python3.7/site-packages/ && \
    zip -r linetest.zip ./python 

In lambda, I wanted to put line-bot-sdk, pandas, numpy, requests, boto3, so I created requirements.txt as follows.

requirements.txt


line-bot-sdk
pandas
numpy
requests
boto3

Finally, the sh file is configured as follows for execution. docker build -t lambda_test . Regarding, once you have created an image, you can comment it out.

deploy.sh


#First time only container launch
docker build -t lambda_test .
#Execute container
docker run -v "$PWD":/var/task lambda_test
#Granting authority(Make it possible to read with lambda)
chmod -R 755 ./*

After creating each file, execute the following in the terminal.

terminal


sh deploy.sh

output


 adding: python/lib/python3.7/site-packages/urllib3-1.26.2.dist-info/RECORD (deflated 62%)
 adding: python/lib/python3.7/site-packages/urllib3-1.26.2.dist-info/WHEEL (deflated 14%)
 adding: python/lib/python3.7/site-packages/urllib3-1.26.2.dist-info/top_level.txt (stored 0%)

When the execution is completed in this way, a folder called python and a zip file called linetest.zip are completed. Here linetest.zip is a zipped version of python. External modules can be used by incorporating the created zip file as a layer of lambda.

ls


Dockerfile              lambda_function.py      python
deploy.sh               linetest.zip            requirements.txt

Creating a layer

Go to the aws console and select lambda → layer → create layer.

image.png

Name: Favorite layer name (so you can see what the layer is for later) Description: Write a description of what modules are included upload: <= 10MB → Upload zip file

10MB → Upload via s3 Runtime: Select python3.7 this time License: You don't have to write anything

image.png

Since the size of the zip file exceeds 10MB this time, upload the zip file via S3.

Service → s3 → Create bucket

Bucket name: Anything is fine Region: Match to lambda (this time unified in Tokyo) Packet settings: block all Packet versioning: Disable Default encryption: disable Advanced settings: unchanged

↓ Creating a bucket

FireShot Capture 109 - S3 Management Console - s3.console.aws.amazon.com.png

After creating the bucket, select the created bucket. (This time I created a packet called pypy-test) Upload → Add file → Upload the zip file created earlier

image.png

If you select the uploaded zip file, you can see the object URL, so copy it.

Go back to lambda again and select the layer. Paste the object URL at the upload location.

Creating a function

lambda → function → add function

How to make: Create from scratch Function name: Appropriate Runtime: python3.7 ↓ Creating a function

image.png

What I was addicted to here

Please note that s3 cannot be used unless the region is Tokyo. At first it should have been North Africa. .. ..

lamnda.png

Select a function and select Add Layer. Select a custom layer, select the layer you created and click Add. (If you don't see the layer you created here, check that the runtime may not be there.)

image.png

Finally, set up API Gateway for testing. Select Trigger → API Gateway → Create API → HTTP API → Security: Open → Add

image.png

This time, check if numpy can be used for testing. Rewrite the lambda code as follows.

image.png

Select a test to make sure it runs successfully.

image.png

If you select API Gateway and jump from the API endpoint to the url, you can see that the result of sin (1) is returned as shown below.

FireShot Capture 115 - disny - Lambda - ap-northeast-1.console.aws.amazon.com.png

image.png

From the next time onward, I will apply this to create a linebot.

Reference article

Chmod settings

-About local settings of lambda

-How to upload python module to lambda

Recommended Posts

[Lambda] I tried to incorporate an external module of python via S3
I tried to summarize how to use matplotlib of python
I tried to implement an artificial perceptron with python
I tried to get an AMI using AWS Lambda
[Python] I tried to get Json of squid ring 2
I tried to summarize the string operations of Python
I tried to find the entropy of the image with python
[Python] I tried to visualize the follow relationship of Twitter
I tried to implement blackjack of card game in Python
I tried to touch Python (installation)
I tried to create an environment of MkDocs on Amazon Linux
I tried to make a regular expression of "amount" using Python
I tried to create a list of prime numbers with python
I tried to make a regular expression of "date" using Python
I tried to fix "I tried stochastic simulation of bingo game with Python"
[Python / AWS Lambda layers] I want to reuse only module in AWS Lambda Layers
I tried to make an analysis base of 5 patterns in 3 years
[Python] Simple Japanese ⇒ I tried to make an English translation tool
I tried to improve the efficiency of daily work with Python
I tried to automatically collect images of Kanna Hashimoto with Python! !!
I tried to make an image similarity function with Python + OpenCV
I tried to summarize Python exception handling
I tried to implement PLSA in Python
I tried to implement permutation in Python
I tried to implement PLSA in Python 2
I tried to implement ADALINE in Python
I tried to implement PPO in Python
Connect to s3 with AWS Lambda Python
[Python] I tried to calculate TF-IDF steadily
I tried to touch Python (basic syntax)
[Python] I tried to summarize the set type (set) in an easy-to-understand manner.
I want to color a part of an Excel string in Python
I tried to streamline the standard role of new employees with Python
[Python] I was addicted to not saving internal variables of lambda expressions
I tried to get the movie information of TMDb API with Python
I tried to implement a card game of playing cards in Python
I tried to output the rpm list of SSH login destination to an Excel sheet with Python + openpyxl.
[Python / DynamoDB / boto3] List of operations I tried
I tried hundreds of millions of SQLite with python
[Python] I tried to judge the member image of the idol group using Keras
I tried to notify slack of Redmine update
I tried to get an image by scraping
I was able to recurse in Python: lambda
I tried to find 100 million digits of pi
I tried to build an environment of Ubuntu 20.04 LTS + ROS2 with Raspberry Pi 4
I tried to touch the API of ebay
I tried to get CloudWatch data with Python
I tried debugging from Python via System Console
I tried to correct the keystone of the image
[Python of Hikari-] Chapter 08-04 Module (Installation of external library)
I tried to output LLVM IR with Python
I tried to easily visualize the tweets of JAWS DAYS 2017 with Python + ELK
I tried to automate the 100 yen deposit of Rakuten horse racing (python / selenium)
I tried to implement TOPIC MODEL in Python
I tried to detect an object with M2Det!
I tried using the Datetime module by Python
I tried to automate sushi making with python
I want to get / execute variables / functions / classes of external files from Python
Summary of points I was addicted to running Selenium on AWS Lambda (python)
[AWS / Lambda] How to load Python external library
I tried adding a Python3 module in C