[AWS / Lambda] How to load Python external library

Overview

AWS lambda cannot load external libraries such as pandas and numpy in the initial state. There are two ways to load it, the first is to upload the installed library and executable file to lambda as a whole in a zip file, and the second is to load it as a layer with lambda. The latter method is recommended, but in this article, we have summarized these methods as tips together with the aspect of code management.

environment

MacOS Catalina docker installed AWS CLI installed

Introduction

Since it is not good for code management to develop lambda on the browser, we will make it possible to develop locally. In other words, it takes the form of working locally so that it can be managed on gitHub and uploading code changes to lambda at any time. So I created a deployment environment using docker-lambda. (Please clone if you want to use it) https://github.com/shoda888/local2lambda

First, we need a docker image, so we will create it. (First time only) $ docker build -t shoda888/local2pylambda .

After that, if you give the lambda function name created in advance to the funcname variable of deploy.sh and execute the following, uploading will be done automatically. $ sh deploy.sh

Rewrite lambda_function.py and deploy.sh again to see if it has changed.

How to upload the library to lambda in a zip file

Just give requirements.txt the library name and deploy.sh (pandas, numpy, requests, etc.). A large number of library directories are created in the working directory. Pandas is too heavy and it takes about 5 minutes to upload. When I check it on lambda, the upload is successful, but a warning appears.

The deployment package for the Lambda function "*******" is too large to enable inline code editing. However, you can call the function.

It seems that you can still call the function.

How to organize a library into a layer

It is recommended to leave requirements.txt empty as it is tedious to zip and upload the library every time. I think that it is convenient and reusable if the library is put together in a layer. (However, library version control may be troublesome in the future) Let's see how to organize the library into a layer.

For Pure Python libraries

In the case of a pure Python library, for example, a library composed of 100% Python as shown in the image can easily create a zip file for layer on your Mac without preparing Amazon Linux or ubuntu environment with EC2. 　

If you want to install ask_sdk_core as an example, this is all. (Example of python3.7)

$ mkdir -p build/python/lib/python3.7/site-packages
$ pip3 install ask_sdk_core -t build/python/lib/python3.7/site-packages/
$ cd build
$ zip -r ask_sdk.zip .

Add the generated ask_sdk.zip to layer and adapt the layer in lambda to load the library.

For non-Pure Python libraries

These include numpy and pandas. It seems that it is better to create an Amazon Linux environment with EC2 and zip it. (I felt that I made Layer with ubuntu before, but can ubuntu be used?)

Many people have already done this, so other people's articles will be very helpful. [Add pandas as a Lambda Layer] (https://qiita.com/thimi0412/items/4c725ec2b26aef59e5bd) Common libraries with AWS Lambda Layers

The following is youtube, but layer is made using ubuntu. (Easy to understand) [AWS Lambda Layers for Pandas library] (https://youtu.be/zrrH9nbSPhQ)

Summary

After making it possible to develop Lambda while managing the code locally, the method of loading the external library is summarized as Tips.