[PYTHON] Make Lambda Layers with Lambda

Introduction

When I use Lambda, I usually use chalice to create and deploy Lambda functions.

chalice is convenient, but it seems that every time you deploy, you do pip install to collect the necessary package. Therefore, if you are using a heavier package such as pandas, it will take longer to deploy.

Therefore, use the LambdaLayers service of the Lambda that registers the package to be used in advance.

Premise

Prerequisites for creating LambdaLayers.

1. For LambdaLayers, use the file that packages is zipped to make Lambda available in advance Reference /lambda/latest/dg/configuration-layers.html)
2. The folder structure such as the layout of package is also decided.
3. Some packages such as pandas need to be pip install on ʻAmazon Linux` reference
4. It's easy to do on ʻEC2, but it's useless to create it if you don't have an environment such as VPC (VPC` itself costs money).
5. Lambda has an area called / tmp that can be accessed by the program.

This condition (3) was so strict that Lambda sometimes did not recognize it properly with the zip file created locally. You can also do it on ʻEC2in (4), but if it is a private development that uses only serverless, There was a (my) request that I wanted to complete it with onlyLambda` as much as possible because I do not want to spend money.

Implementation

Flow of this implementation

Based on the above premise, this time create the zip file required for LambdaLayers by the following method.

ʻInstall`` packagein the/ tmp area of Lambda, zip it up with zip and upload it to S3 The actual registration ofLambda Layers` is done manually.

Source code

The program to be executed by Lambda is as follows.

lambda_function.py

import json
import subprocess
import zipfile
import boto3
from pathlib import Path


def lambda_handler(event, context):
    #Argument confirmation
    if "package_list" not in event:
        return {"error": "[package_list]Does not exist"}
    package_list = event['package_list']
    if type(package_list) is not list:
        return {"error": "[package_list]Is not a list"}    
    if "s3_bucket" not in event:
        return {"error": "[s3_bucket]Does not exist"}
    if "s3_key" not in event:
        return {"error": "[s3_key]Does not exist"}
    
    #Create a directory called python and install the package there
    mkdir_cmd = "mkdir -p /tmp/python"
    pip_install_cmd = f"pip install -t /tmp/python {' '.join(package_list)}"
    process = (subprocess.Popen(f"{mkdir_cmd} && {pip_install_cmd}", stdout=subprocess.PIPE, shell=True).communicate()[0]).decode('utf-8')
    print('commands...\n'+process)

    #Create Path object
    p = Path("/tmp/python")

    #Add the installed package to the zip
    with zipfile.ZipFile('/tmp/python_lib.zip', 'w', compression=zipfile.ZIP_DEFLATED) as new_zip:
        for f in list(p.glob("**/*")):
            new_zip.write(f, arcname=str(f).replace("/tmp/", ""))

    #Get S3 object and upload
    s3 = boto3.resource('s3')
    bucket = s3.Bucket(event['s3_bucket'])
    bucket.upload_file('/tmp/python_lib.zip', event['s3_key'])
    return {
        "success": f"s3://{event['s3_bucket']}/{event['s3_key']}Output to"
    }

Lambda settings

Please change it according to the package to be installed.

• Memory: 512 [MB]
• timeout： 5[min]
• Role: Grant access to the uploaded S3.

Argument example

When running, pass the following arguments to install the package contained in the package_list. It also uploads the zip file to the locations specified in s3_bucket and s3_key.

event

{
  "package_list": ["pandas", "lxml", "requests", "beautifulsoup4"],
  "s3_bucket": "<bucket name for s3>",
  "s3_key": "lambda_layer/python_lib.zip"
}

If you create a zip file using the above example, a zip file of about 43.1 [MB] will be created.

in conclusion

So, if you copy the above source code to Lambda and execute it, You can now create the zip required to register LambdaLayers.

Note: This time I have no choice but to run shell from Python. Basically, I don't think it's good to call shell from Python, so Let's use it when there is no choice but to use it.