[PYTHON] Deploy external modules (numpy, scipy, requests, etc.) to AWS Lambda

I would like to see how to deploy an external module to AWS Lambda (Lambda). Let's take Python as an example.

Execution order

Prepare Amazon Linux 2 (Lambda execution environment). Here, instead of using EC2, we will create a deployment package locally using Docker Image for Lambda.
Deploy to Lambda with source files
How to use Lambda Layer

It doesn't matter which one you choose for * 2.

1. Prepare Amazon Linux 2 (Lambda execution environment)

This step is not necessary for modules written in Pure Python without using C language internally such as requests module. *** However, for modules that depend on C language such as numpy and scipy, it is easier to develop in the same environment as the Lambda execution environment. Even if you zip and deploy numpy installed on windows or Mac, an error will occur. *** ***

Here, we will use the publicly available Docker image *** lambci / lambda: build-python3.7 *** as the environment for Amazon Linux 2.

Create an appropriate directory and create the source code (lambda_function.py) and Dockerfile to be executed by Lambda.

Contents of Dockerfile

FROM lambci/lambda:build-python3.7
CMD ["pip3","install","numpy","-t","/var/task"]

The contents of lambda_function.py are appropriate


import numpy as np

def lambda_handler(event,context):
    print(np.arange(10).reshape(2,5))

Source and Dockerfile creation completed

~$ ls
Dockerfile  lambda_function.py

After that, build and run.

~$ docker image build -t numpy:latest .
~$ docker container run --rm -v ${PWD}:/var/task numpy:latest

*** -v $ {PWD}: / var / task to refer to the external module installed in the container even in the host OS. *** ***

By the way, / var / task is one of the paths that Lambda refers to to import external modules. If you are running in a Docker container, you need to put the external module in this path. As a test, check the path with Lambda as follows.

import sys
def lambda_handler(event,context):
    print(sys.path)
    
"""
python3.Execution result in 7

['/var/task', '/opt/python/lib/python3.7/site-packages', '/opt/python', '/var/runtime', '/var/lang/lib/python37.zip', '/var/lang/lib/python3.7', '/var/lang/lib/python3.7/lib-dynload', '/var/lang/lib/python3.7/site-packages', '/opt/python/lib/python3.7/site-packages', '/opt/python']
"""

If the above command is successful, the source and external module (numpy) should be in the same hierarchy. All you have to do is zip it up with your Lambda Layer or source file and deploy. So, please choose either of the following 2.

~$ ls
Dockerfile  bin  lambda_function.py  numpy  numpy-1.18.3.dist-info  numpy.libs

2. How to use Lambda Layer

As mentioned above, create a python directory so that Lambda can load external modules, and move numpy related files and directories to it. If you created the python directory at the beginning of the execution procedure, you do not need to do the following.

~$ mkdir python
~$ mv bin numpy  numpy-1.18.3.dist-info  numpy.libs python

Zip the external module and deploy it to your Lambda Layer.

~$ zip -rq numpy.zip python

Zip the source files and deploy with the Lambda console or AWS CLI.

~$ zip -q lambda_function.zip lambda_function.py

2. Deploy to Lambda with source files

Zip it up and deploy it with the Lambda console or AWS CLI. When using Lambda Layer, separate the external module and the source file, but this time I will zip them all together. So the zip target is *.

~$ zip -rq numpy.zip *

When deploying to Lambda, it may be too large to be displayed on the screen (numpy is almost invisible), but it can be executed.