[PYTHON] [AWS Lambda] Create a deployment package using the Docker image of Amazon Linux

If you want to use a library that is not in the AWS Lambda runtime environment, you need to download the library source group in the local environment, include it in the deployment package, and upload it to Lambda.

In the case of Python, for example, when creating a deployment package, follow the steps below.

$ virtualenv -p python2.7 venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ zip deploy_package.zip lambda_function.py #Zip the execution script
$ cd venv/lib/python2.7/site-packages
$ zip -r ../../../../deploy_package.zip * #Add libraries installed with pip to zip

If the library you want to use is pure Python code, this package will work on Lambda without any problem, but if the library depends on the function of the OS, Lambda is built and installed in the local environment. It doesn't work even if I upload it above. In the above example, pip install should be done in the same environment as Lambda's execution environment.

Failure example

Create and run a Lambda Function that uses a library called pycrypto.

requirements.txt

pycrypto==2.6.1

lambda_function.py

from Crypto.PublicKey import RSA


def lambda_handler(event, context):
    return {"messagge": "success"}

Prepare the above files and create a deployment package in your local OS X environment.

$ virtualenv -p python2.7 venv
$ source venv/bin/activate
$ pip install -r requirements.txt
$ zip deploy_package.zip lambda_function.py
$ cd venv/lib/python2.7/site-packages
$ zip -r ../../../../deploy_package.zip *

When I upload the created deploy_package.zip to AWS Lambda and execute it, I get the following error. Unable to import module 'lambda_function'

Looking at the full text of the error log, it says ʻUnable to import module'lambda_function': /var/task/Crypto/Util/_counter.so: invalid ELF header. It seems that the header information of the file referenced by pycryptois invalid. The cause is that the environment wherepycrypto` is installed is different from the Lambda execution environment.

Countermeasures

You can work around this issue by installing the library using the Amazon Linux Docker image.

Prepare such a Dockerfile in the same hierarchy as the above file

Dockerfile

FROM amazonlinux:latest

RUN yum -y update && yum -y install gcc python27-devel
RUN cd /tmp && \
    curl https://bootstrap.pypa.io/get-pip.py | python2.7 - && \
    pip install virtualenv
WORKDIR /app
CMD virtualenv -p python2.7 venv-for-deployment && \
    source venv-for-deployment/bin/activate && \
    pip install -r requirements.txt

Running the command in this way creates a Python library code built on Amazon Linux with the name venv-for-deployment.

$ docker build . -t packager
$ docker run --rm -it -v $(PWD):/app packager

After that, create a zip of the deployment package as shown below and upload it to AWS Lambda.

$ zip deploy_package.zip lambda_function.py
$ cd venv-for-deployment/lib/python2.7/site-packages
$ zip -r ../../../../deploy_package.zip * .* #If dotfile is included".*"Also

When executed, the library can be imported and " success " is displayed safely.

Automation

Since there are a lot of commands to hit a little, it is convenient to create a Makefile like this because a zip will be generated just by make.

Makefile

package:
	docker build . -t packager
	docker run --rm -it -v $(PWD):/app packager
	zip deploy_package.zip lambda_function.py
	cd venv-for-deployment/lib/python2.7/site-packages && zip -r ../../../../deploy_package.zip * .*
	echo "Completed. Please upload deploy_package.zip to AWS Lambda"

sample

The sample Lambda Function used this time is in this repository. https://github.com/morishin/python-lambda-function-test

Impressions

AWS Lambda It's convenient, but when you try to do something a little elaborate, you almost cry.