[PYTHON] Docker-lambda and yumda are useful when creating Lambda Layers

Introduction

You can create a layer corresponding to each line time version with the build image of docker-lambda!

$ docker run --rm -v $(pwd):/var/task lambci/lambda:build-python3.8 \
> pip install -r requirements.txt -t python/lib/python3.8/site-packages/

Software packages you want to run on AWS Lambda can be layered with yumda!

$ docker run --rm -v $(pwd)/git-layer:/lambda/opt lambci/yumda:2 yum install -y <package>

Environmental preparation is quite difficult if you do it from scratch

A common caveat with AWS Lambda and Lambda Layers is the handling of native binaries. Because the Lambda runtime environment is based on a particular Amazon Linux environment and kernel version The native binaries used within Lambda must also be compiled in that environment.

It's also currently in a transitional period when Amazon Linux is no longer supported, depending on the version of the runtime used. The difference between being based on Amazon Linux or being based on Amazon Linux 2 is also annoying. For example, in the case of Python, as of March 28, 2020, it is as follows.

AWS Lambda Runtimes https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

It's really troublesome to start EC2 every time you create a package. You can also use the Amazon Linux container image published on Docker Hub. This image contains a minimal set of packages, so It is necessary to additionally install the target runtime environment using this as the base image. When developing for multiple runtimes and versions, manage Dockerfiles and images for each environment I think it's a little difficult to do.

The introduction has become long, but docker-lambda and yumda are useful in such cases.

docker-lambda

What is docker-lambda?

docker-lambda reproduces the AWS Lambda runtime environment as close as possible inside a Docker container. (But not a perfect copy) The container image is published at lambci / lambda on Docker Hub. It was developed as OSS by Michael Hart of AWS Severless Hero.

docker-lambda https://github.com/lambci/docker-lambda

It is also used as the execution environment for Lambda functions in the AWS SAM CLI. When you invoke a Lambda function locally with the SAM CLI, docker-lambda is running internally.

docker-lambda build image

docker-lambda also provides a build image for building and packaging Lambda functions. Build images are provided for each runtime version with image tags such as build-python3.x. Packages that are additionally installed in the build image See https://hub.docker.com/r/lambci/lambda/#build-environment. It also includes the AWS CLI and AWS SAM CLI, so if you want to do it, you can do it in the container.

Example of creating a python3.8 compatible layer

Use lambci / lambda: build-python3.8. Describe the library you want to add to requirements.txt. I'm just adding pandas here.

requirements.txt

pandas

Pip install in the build image to include the library in the Layer.

$ docker run --rm -v $(pwd):/var/task lambci/lambda:build-python3.8 \
> pip install -r requirements.txt -t python/lib/python3.8/site-packages/

$ zip -r pandas-1.0.3.zip ./python > /dev/null

If you want to make multiple versions compatible with each other, use the build image for each version. Do docker run with site-packages and then zip. The directory structure is as follows. You should be aware of the package size limit (250MB) after deployment.

.
`-- python
    `-- lib
        |-- python3.6
        |   `-- site-packages
        |-- python3.7
        |   `-- site-packages
        `-- python3.8
            `-- site-packages

When I check the operation with the following super-simple code using the previous Layer, ...

lambda_function.py

import pandas as pd;

def lambda_handler(event, context):
    pd.show_versions()

I was able to do it!

START RequestId: 813889ca-958c-4b60-ae1d-442870679429 Version: $LATEST

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.8.2.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.14.165-102.205.amzn2.x86_64
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.3
numpy            : 1.18.2
~~Omitted below~~

yumda Use yumda to create software packages that can run in an AWS Lambda environment It can be prepared with the yum command. Like docker-lambda, it is developed as OSS.

yumda – yum for Lambda https://github.com/lambci/yumda

The container image is published at lambci / yumda. An image for the Amazon Linux 2 environment and an image for the Amazon Linux environment are available. nodejs10.x, nodejs12.x, python3.8, java11, ruby2.7 are lambci / yumbda: 2, Other runtime versions use lambci / yumbda: 1.

To see which packages are available for your Amazon Linux 2 environment, run a command similar to the following:

$ docker run --rm lambci/yumda:2 yum list available
Loaded plugins: ovl, priorities
Available Packages
GraphicsMagick.x86_64        1.3.34-1.lambda2                            lambda2
GraphicsMagick-c++.x86_64    1.3.34-1.lambda2                            lambda2
ImageMagick.x86_64           6.7.8.9-18.lambda2                          lambda2
OpenEXR.x86_64               1.7.1-7.lambda2.0.2                         lambda2
OpenEXR-libs.x86_64          1.7.1-7.lambda2.0.2                         lambda2
~~Omitted below~~

Or you can check from the following. https://github.com/lambci/yumda/blob/master/amazon-linux-2/packages.txt

The package is managed in its own repository and you can submit additional requests on GitHub Issues, Anything that already exists in the Amazon Linux core repository and the amazon-linux-extras repository It seems that it is a policy to add.

Please file a GitHub Issue with your request and add the package suggestion label. For now we'll only be considering additions that already exist in the Amazon Linux core repositories, or the amazon-linux-extras repositories (including epel).

Example of creating a Layer with yumda

Here, I will try to execute the git command with the Lmabda function of python3.8.

lambda_function.py

import subprocess

def lambda_handler(event, context):
    return subprocess.check_output(
        "git --version;exit 0",
        stderr=subprocess.STDOUT,
        shell=True
    )

Of course, if you don't use Lambda Layers, the result will be commnad not found.

$ aws lambda invoke --function-name git-layer-test outfile
ExecutedVersion: $LATEST
StatusCode: 200

$ cat outfile
/bin/sh: git: command not found

Create a Layer with yumda.

$ docker run --rm -v $(pwd)/git-layer:/lambda/opt lambci/yumda:2 yum install -y git
$ cd git-layer
$ zip -yr ../git-2.25.0-1.zip . > /dev/null

You can see that git and its dependencies are installed with the following directory structure.

.
|-- bin
|-- etc
|   |-- alternatives
|   |-- pki
|   |-- prelink.conf.d
|   `-- ssh
|-- lib
|   |-- fipscheck
|   `-- nss
|-- libexec
|   |-- git-core
|   `-- openssh
`-- share
    |-- git-core
    `-- licenses

Publish a Layer and set it in your Lambda function.

$ aws lambda publish-layer-version \
> --layer-name git-2-25-0-1 --zip-file fileb://git-2.25.0-1.zip

$ aws lambda update-function-configuration \
>  --function-name git-layer-test \
>  --layers arn:aws:lambda:ap-northeast-1:123456789012:layer:git-2-25-0-1:1

When I ran it, I got a git version!

$ aws lambda invoke --function-name git-layer-test outfile
ExecutedVersion: $LATEST
StatusCode: 200

$ cat outfile
git version 2.25.0

Reference: Use published Lambda Layers

I've written so far, but Lambda Layers is also available in Sevelress Application Repository. Many are registered, and there is also a GitHub repository called λ AWSome Lambda Layers. A list of various layers is put together. If you have what you need for these, it's easy to use them!

that's all. I'm glad if you can use it as a reference.