[PYTHON] Prepare an environment to use OpenCV and Pillow with AWS Lambda

I wanted to do something similar to this, so I created a development environment. Comparing Node.js and Python when creating thumbnails using AWS Lambda

Pillow seems to need to be built on EC2. It was okay to launch an instance, but if you use Amazon Linux Container Image and Docker, you can do it locally. thought.

Pillow build

I referred to this area. Try using the image processing library "Pillow" with AWS Lambda [Create a deployment package](https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg- python)

OpenCV build

I referred to here. Procedure to create OpenCV3.0 library available from Python on AWS Lambda

The resulting cv2.so file was as large as 47MB. Lambda function deployment package limits is 50MB, so I'm already full. I removed the unused modules of OpenCV and dieted the so file. Build specific modules OpenCV

Dockerfile.

FROM amazonlinux:latest

RUN yum update -y
RUN yum install python27-devel python27-pip gcc gcc-c++ cmake git zip -y
RUN yum install libjpeg-devel zlib-devel -y
RUN pip install --upgrade pip
RUN pip install virtualenv numpy

ENV HOME /home/lambda-py
RUN mkdir $HOME
WORKDIR $HOME

RUN git clone https://github.com/opencv/opencv.git
WORKDIR $HOME/opencv
WORKDIR $HOME/opencv/build
RUN cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_SHARED_LIBS=NO -D BUILD_opencv_python2=ON -D BUILD_opencv_calib3d=OFF -D BUILD_opencv_objdetect=OFF -D BUILD_opencv_ml=OFF -D BUILD_opencv_features2d=OFF -D BUILD_opencv_stitching=OFF -D BUILD_opencv_videostab=OFF -D BUILD_opencv_flann=OFF -D BUILD_opencv_superres=OFF  -D BUILD_opencv_shape=OFF -D BUILD_opencv_ts=OFF -D BUILD_opencv_video=OFF -D BUILD_opencv_videoio=OFF -D CMAKE_INSTALL_PREFIX=/usr/local ..
RUN make install

WORKDIR $HOME
RUN virtualenv $HOME/venv

If you build with this setting, many modules of OpenCV will be disabled, but the so file will be much smaller.

<Excerpt from build log>
--   OpenCV modules:
--     To be built:                 core imgproc photo imgcodecs highgui python2
--     Disabled:                    calib3d features2d flann ml objdetect shape stitching superres ts video videoio videostab world
--     Disabled by dependency:      -
--     Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev java python3 viz


(venv) bash-4.2# ls -l /usr/local/lib/python2.7/dist-packages/      
total 40544
-rwxr-xr-x 1 root root 41510502 Nov 29 03:54 cv2.so

Make a deploy package

Launch the completed container and create a package (zip file) to deploy to Lambda.

bash-4.2# source $HOME/venv/bin/activate
(venv) bash-4.2# pip install Pillow numpy
(venv) bash-4.2# cp /usr/local/lib/python2.7/dist-packages/cv2.so ~/venv/lib/python2.7/site-packages/
(venv) bash-4.2# cd ~/venv/lib/python2.7/site-packages
(venv) bash-4.2# zip -r9 ~/lambda_function.zip *
(venv) bash-4.2# cd ~/venv/lib64/python2.7/site-packages
(venv) bash-4.2# zip -r9 ~/lambda_function.zip *

The code is edited on the host side and put in a folder mounted in the src folder of the container.

(venv) bash-4.2# cd ~/src
(venv) bash-4.2# zip ~/lambda_function.zip lambda_test.py 

The deploy package is ready!

(venv) bash-4.2# ls -l
total 39736
-rw-r--r--  1 root root 40680005 Nov 29 04:51 lambda_function.zip
drwxr-xr-x 19 root root     4096 Nov 29 03:54 opencv
drwxr-xr-x  4 root root      136 Nov 29 04:50 src
drwxr-xr-x  8 root root     4096 Nov 29 04:39 venv

After that, I think that you can bring this zip file to the host side and deploy it to Lambda using the AWS console or CLI. You can work with the container, but in that case you need to install the AWS CLI.