[PYTHON] Prepare an environment to use OpenCV and Pillow with AWS Lambda

I wanted to do something similar to this, so I created a development environment. Comparing Node.js and Python when creating thumbnails using AWS Lambda

Pillow seems to need to be built on EC2. It was okay to launch an instance, but if you use Amazon Linux Container Image and Docker, you can do it locally. thought.

Pillow build

I referred to this area. Try using the image processing library "Pillow" with AWS Lambda [Create a deployment package](https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg- python)

OpenCV build

I referred to here. Procedure to create OpenCV3.0 library available from Python on AWS Lambda

The resulting cv2.so file was as large as 47MB. Lambda function deployment package limits is 50MB, so I'm already full. I removed the unused modules of OpenCV and dieted the so file. Build specific modules OpenCV

Dockerfile.

FROM amazonlinux:latest

RUN yum update -y
RUN yum install python27-devel python27-pip gcc gcc-c++ cmake git zip -y
RUN yum install libjpeg-devel zlib-devel -y
RUN pip install --upgrade pip
RUN pip install virtualenv numpy

ENV HOME /home/lambda-py
RUN mkdir $HOME
WORKDIR $HOME

RUN git clone https://github.com/opencv/opencv.git
WORKDIR $HOME/opencv
WORKDIR $HOME/opencv/build
RUN cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_SHARED_LIBS=NO -D BUILD_opencv_python2=ON -D BUILD_opencv_calib3d=OFF -D BUILD_opencv_objdetect=OFF -D BUILD_opencv_ml=OFF -D BUILD_opencv_features2d=OFF -D BUILD_opencv_stitching=OFF -D BUILD_opencv_videostab=OFF -D BUILD_opencv_flann=OFF -D BUILD_opencv_superres=OFF  -D BUILD_opencv_shape=OFF -D BUILD_opencv_ts=OFF -D BUILD_opencv_video=OFF -D BUILD_opencv_videoio=OFF -D CMAKE_INSTALL_PREFIX=/usr/local ..
RUN make install

WORKDIR $HOME
RUN virtualenv $HOME/venv

If you build with this setting, many modules of OpenCV will be disabled, but the so file will be much smaller.

<Excerpt from build log>
--   OpenCV modules:
--     To be built:                 core imgproc photo imgcodecs highgui python2
--     Disabled:                    calib3d features2d flann ml objdetect shape stitching superres ts video videoio videostab world
--     Disabled by dependency:      -
--     Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev java python3 viz


(venv) bash-4.2# ls -l /usr/local/lib/python2.7/dist-packages/      
total 40544
-rwxr-xr-x 1 root root 41510502 Nov 29 03:54 cv2.so

Make a deploy package

Launch the completed container and create a package (zip file) to deploy to Lambda.

bash-4.2# source $HOME/venv/bin/activate
(venv) bash-4.2# pip install Pillow numpy
(venv) bash-4.2# cp /usr/local/lib/python2.7/dist-packages/cv2.so ~/venv/lib/python2.7/site-packages/
(venv) bash-4.2# cd ~/venv/lib/python2.7/site-packages
(venv) bash-4.2# zip -r9 ~/lambda_function.zip *
(venv) bash-4.2# cd ~/venv/lib64/python2.7/site-packages
(venv) bash-4.2# zip -r9 ~/lambda_function.zip *

The code is edited on the host side and put in a folder mounted in the src folder of the container.

(venv) bash-4.2# cd ~/src
(venv) bash-4.2# zip ~/lambda_function.zip lambda_test.py 

The deploy package is ready!

(venv) bash-4.2# ls -l
total 39736
-rw-r--r--  1 root root 40680005 Nov 29 04:51 lambda_function.zip
drwxr-xr-x 19 root root     4096 Nov 29 03:54 opencv
drwxr-xr-x  4 root root      136 Nov 29 04:50 src
drwxr-xr-x  8 root root     4096 Nov 29 04:39 venv

After that, I think that you can bring this zip file to the host side and deploy it to Lambda using the AWS console or CLI. You can work with the container, but in that case you need to install the AWS CLI.

Recommended Posts

Prepare an environment to use OpenCV and Pillow with AWS Lambda
Building an environment to use CaboCha with google colaboratory
A story about how Windows 10 users created an environment to use OpenCV3 with Python 3.5
Minimum Makefile and buildout.cfg to create an environment with buildout
Install pip in Serverless Framework and AWS Lambda with Python environment
Use PIL and Pillow with Cygwin Python
[AWS] Link Lambda and S3 with boto3
Connect to s3 with AWS Lambda Python
ImportError when trying to use gcloud package with AWS Lambda Python version
[AWS] Create a Python Lambda environment with CodeStar and do Hello World
Prepare an environment to touch grib2 format files with python (Docker edition)
Prepare the environment of Chainer on EC2 spot instance with AWS Lambda
How to crop an image with Python + OpenCV
The easiest way to use OpenCV with python
Pillow environment construction --For Docker + iPython (and OpenCV)
Summary of studying Python to use AWS Lambda
I want to use mkl with numpy and scipy under pyenv + poetry environment
I tried to create an environment to check regularly using Selenium with AWS Fargate
I tried to use Twitter Scraper on AWS Lambda and it didn't work.
How to prepare an environment with different python version and package for each project with pyenv-virtualenv on Amazon Linux
Dynamic HTML pages made with AWS Lambda and Python
Allow Keras 2.0 and OpenCV 3.2 to work in GPU environment
I tried to get an AMI using AWS Lambda
How to loop and play gif video with openCV
Ubuntu 20.04 on raspberry pi 4 with OpenCV and use with python
I want to AWS Lambda with Python on Mac!
Steps to quickly create a deep learning environment on Mac with TensorFlow and OpenCV
Install Ubuntu 20.04 with GUI and prepare the development environment
Make ordinary tweets fleet-like with AWS Lambda and Python
I wrote a script to create a Twitter Bot development environment quickly with AWS Lambda + Python 2.7
Posted as an attachment to Slack on AWS Lambda (Python)
[AWS] What to do when you want to pip with Lambda
It's too easy to use an existing database with Django
[AWS] Try adding Python library to Layer with SAM + Lambda (Python)
Use aggdraw when you want to draw beautifully with pillow
Convert garbled scanned images to PDF with Pillow and PyPDF
I want to use a virtual environment with jupyter notebook!
I just built a virtual environment with AWS lambda layer
Convert video to black and white with ffmpeg + python + opencv
Site monitoring and alert notification with AWS Lambda + Python + Slack
AWS Lambda with PyTorch [Lambda import]
How to use Python lambda
Use AWS interpreter with Pycharm
[Python] Convert CSV file uploaded to S3 to JSON file with AWS Lambda
Create an alias for Route53 to CloudFront with the AWS API
Send images taken with ESP32-WROOM-32 to AWS (API Gateway → Lambda → S3)
How to use jupyter notebook without polluting your environment with Docker
I want to use an external library with IBM Cloud Functions
Build and try an OpenCV & Python environment in minutes using Docker
How to use an external editor for Python development with Grasshopper
The strongest way to use MeCab and CaboCha with Google Colab
Building an environment to run ChainerMN on a GPU instance on AWS
[For Python] Quickly create an upload file to AWS Lambda Layer
How to use Python with Jw_cad (Part 2 Command explanation and operation)
How to build Python and Jupyter execution environment with VS Code
[Python] Try to recognize characters from images with OpenCV and pyocr
How to create a serverless machine learning API with AWS Lambda
I tried to make an image similarity function with Python + OpenCV
[AWS lambda] Deploy including various libraries with lambda (generate a zip with a password and upload it to s3) @ Python
When doing sam build with AWS Lambda Layers, the file hierarchy changes and an import error occurs.
Environment construction of python and opencv