Install python library on Lambda using [/ tmp]

problem

The Lambda deployment package size limit cannot exceed 250 MB.

Solution

Create Lambda Layer Install the library using Lambda's [/ tmp] folder (512M) This time, I will explain method 2.

Create requirements.txt file

requirements-lambda.txt


numpy==1.16.3

create.sh


#!/bin/bash

if [ -d "deploy" ]; then rm -Rf deploy; fi
mkdir deploy
pip install -r requirements-lambda.txt -t deploy/requirements-lambda/
cd deploy/requirements-lambda
rm -r PIL
rm -r Pillow*
zip -9 -r ../lambda-requirements.zip .
cd ..
rm -r requirements-lambda

Unzip to handle program on Lambda (size less than 50MB)

handle.py


import os
import sys
import zipfile

pkgdir = '/tmp/requirements'
zip_requirements = 'lambda-requirements.zip'

if os.environ.get("AWS_EXECUTION_ENV") is not None:
    if not os.path.exists(pkgdir):

        root = os.environ.get('LAMBDA_TASK_ROOT', os.getcwd())
        zip_requirements = os.path.join(root, zip_requirements)
        zipfile.ZipFile(zip_requirements, 'r').extractall(pkgdir)

        sys.path.append(pkgdir)

Download from S3 and Unzip (size is 50MB or more)

handle.py


import boto3
import os
import sys
import zipfile

REQUIREMENTS_BUCKET_NAME = ''
REQUIREMENTS_KEY = ''

pkgdir = '/tmp/requirements'
zip_requirements = '/tmp/lambda-requirements.zip'

sys.path.append(pkgdir)

if os.environ.get("AWS_EXECUTION_ENV") is not None:
    if not os.path.exists(pkgdir):

        s3 = boto3.resource('s3')
        bucket = s3.Bucket(REQUIREMENTS_BUCKET_NAME)
        bucket.download_file(REQUIREMENTS_KEY, zip_requirements)

        zipfile.ZipFile(zip_requirements, 'r').extractall(pkgdir)
        os.remove('zip_requirements')

        sys.path.append(pkgdir)

Recommended Posts

Install python library on Lambda using [/ tmp]
Install Python on CentOS using Pyenv
Install Python on CentOS using pyenv
Install python on WSL
Install Python on Pidora.
Install Scrapy on python3
Install Python on Mac
Install Python 3 on Mac
Install Python3.4 on CentOS 6.6
Install python external library
Install python on windows
Install Python 2.7.3 on CentOS 5.4
Install Python 3.3 on Ubuntu 12.04
Install Python 3.4 on Mac
Install Python 3.6 on Docker
Install Python 3.8.6 on macOS Big Sur using pyenv
Install psycopg2 (pgsql library for python3) on Apple Silicon
Install Python 3.8 on RHEL 8 (AppStream)
Install pygame on python3.4 on mac
Windows10: Install python dlib library
Install Python 3.8 on CentOS 7 (SCL)
Install the 3rd party python library on Cinema 4D
Check types_map when using mimetypes on AWS Lambda (Python)
Install pandas 0.14 on python3.4 [on Mac]
Install OpenCV on Ubuntu + python
Python --Install MySQLDB on EC2
Install Python 3.8 on CentOS 8 (AppStream)
Broadcast on LINE using python
[Python] [Chainer] [Windows] Install Chainer on Windows
Install PyCall on Raspberry PI and try using GPIO's library for Python from Ruby
Install python3 and scientific calculation library on Ubuntu (virtualenv + pip)
How to install python using anaconda
Summary if using AWS Lambda (Python)
Install Python3 on Sakura server (FreeBSD)
Install anaconda on Mac → Add Library
Install Python 3.8 on Ubuntu 20.04 (OS standard)
Install python2.7 on windows 32bit environment
Introducing Python using pyenv on Ubuntu 20.04
Notes on using MeCab from Python
Install xgboost (python version) on Windows
Preparing python using vscode on ubuntu
Install Python on Windows + pip + virtualenv
Install Pytorch on Blender 2.90 python on Windows
Install Python 3.7 Anaconda on MAC, but Python 2
Install python3 on Mac (El Capitan)
Run Python on Schedule on AWS Lambda
Windows10: Install MeCab library in python
Study on Tokyo Rent Using Python (3-2)
Notes on installing Python using PyEnv
Install nginx python postgresql using apt-get
Notes on using rstrip with python.
Install Python 3.9 on Ubuntu 20.04 (OS standard?)
Install Python development environment on Windows 10
Install confluent-kafka for Python on Ubuntu
Study on Tokyo Rent Using Python (3-3)
Install Python framework django using pip
Install Python 2.7 on Ubuntu 20.04 (OS standard?)
Steps to install python3 on mac
install python
Notes for using OpenCV on Windows10 Python 3.8.3.
[AWS] Using ini files with Lambda [Python]